Unit-level Optimization for SVC Extractor Chang-Ming Lee, Chia-Ying Lee, Bo-Yao Huang, and Kang-Chih Chang Department of Communications Engineering National Chung Cheng University Chiayi, Taiwan changminglee@ee.ccu.edu.tw, jackson99011@gmail.com, joe00450@hotmail.com, jackio816@gmail.com Abstract: - Recently, the developments of video streaming technology are interesting, especially in the network transmission. With various requirements from the user, the issue of different quality of multimedia service is still open. The efficient technique to realize the video streaming must consider the different scalabilities to satisfy the various demands. This paper proposes an extraction refinement for each Network Abstraction Layer Unit (NALU in Scalable Video Coding (SVC with considering both of the video content and the transportlayer information. Therefore the unit-level optimization of the video extraction can be achieved in the video streaming. The granulated decision in terms of the SVC layer effect and the communication bandwidth is more nimble for the dynamic video and transmission environment. Finally, the experimental results show that the unit-level optimization in the SVC extraction can improve the reconstruction quality up to 0.8 db in the comparisons with the Quality-Layer-based extraction. Key-Words: - SVC, Extractor, Temporal Scalability, Quality Scalability, NALU 1 Introduction With the development of the network and video/audio coding techniques, multi-media applications become more and more popular recently, including IPTV, video conferencing, and video surveillance. In addition, numerous kinds of devices, like smart phone, PDA, Pad, desktop, laptop, and HDTV, are developed in this trend. However, these equipments have various functionalities to fit different application scenarios such as the flexibility and the mobility. Especially, the last condition incurs the degradation of service quality and the constraint of energy consumption in the noisy scene. Moreover, the challenges in the heterogeneous networks are exhibited [1] in the hybrid channels (wire and wireless transmissions. To cope with the difficulties in the video communication, SVC (Scalable Video Coding [] is useful to fit different service qualities. The scalability of SVC can be realized by retaining the base layer and removing several enhancement layers according to the layer dependency. Generally, the multi-media transmission involved with the network service always meet the dramatic variety in transmission bandwidth, delay, packet loss, etc. In addition, the capabilities of receiver, processor units and displayer in the client part need to be addressed in the practical application. In this paper, the discussion and study are based on the popular reference software JSVM9.18 [3][4] which has the video codec and several fundamental extractions. Fig. 1 illustrates the framework of SVC system. After the SVC coding, the bit-stream can pass through the extractor [5]. Extractor can provide adjustment of SVC-coded bit-stream with selecting appropriate enhancement layers according to the instrument properties and transmission conditions. Clients can experience higher-quality service with access to more enhancement layers. Three scalabilities in SVC are SNR, Spatial and Spatial scalabilities. The first can provide different visual qualities in the video for the same spatial and temporal resolutions. The second technique can satisfy the various display formats (resolutions. The last one can adjust the display frame-rate. The combination of these three scalabilities in SVC can adapt the video stream to the different application scenarios [6]. SVC Encoder Extractor Low Base Layer Layer 1 Layer Medium High SVC or AVC Decoder SVC Decoder SVC Decoder Fig. 1 The Flow of SVC Codec Display 1 Display Display 3 ISBN: 978-1-61804-140-1 73
The layer information can be partially extracted with respect to the transmission conditions and the layer dependency in each video sequence. The optimization based on the Lagrange cost function is concerned with the distortion (the side effect of discarded layers and rate (the cost of extracted layers. Amonou et al [7] has improved the extraction with the motion information. However, the assumptions about the free of error propagation due to the MV quantization and availability of the perfect reference picture are not practical for the services of broadcasting and sent-on-demand. Besides, the optimization based on the layer decision is not refined to dynamical cases. The similar issues also appear in [8]. Thus, the granularity of SVC extraction needs to be developed to provide more efficient video streaming techniques. The rest of this paper is organized as follows. Section overviews two general extractors. In Section 3, we propose a unit-level optimized extractor. Section 4 provides the simulation results. Finally, Section 5 summarizes our work. Related works Two fundamental extractions provided in JSVM9.18 are introduced in this section. The decision priority would be reviewed with the indexing in the SVC layer structure..1 Basic Extraction In JSVM, Extractor can access the SEI (Supplemental enhancement information in NALU to realize the SVC scalabilities. The configurations of frame-resolutions, frame-rates and framequalities can designate the related enhancement layers for the video processing. Note that the triplet {D T Q} in Fig. can indicate the enhancement layers in the order of resolution, frame-rate, and quality scalabilities, respectively. All units with T=0 can represent I and P frames in the base layer. In the fundamental data structure, ids in base layer are 0s to indicate the lowest available quality. Otherwise, the non-zero triplet can denote the enhancement in different scalabilities. In addition, the layer dependency can be evaluated with analyzing these indices in NALUs. Fig. 3 shows the decision policy in the Basic extraction[6]. According to the coding dependencies in each temporal layer, the units in the same temporal or quality enhancement layer are prioritized with one priority order. Besides, the layer-level decision in the SVC streaming is not adaptive for the transmission bandwidth constraint. D=1 Q= D=1 Q=1 D=1 Q=0 D=0 Q= D=0 Q=1 D=0 Q=0 Prioritization Order Base Layer Enhance Layer Resolution 0 Enhance Layer Resolution 1 0 1 3 T Spatial Quality Base Layer Fig. Paradigm for DTQ layer structure. Temporal Layer 0 Temporal Layer 1 Temporal Layer D0,T0,Q0 D1,T0,Q0 D0,T0,Q0 D0,T0,Q1 D1,T0,Q0 D1,T0,Q1 D0,T1,Q0 D1,T1,Q0 D0,T1,Q0 D0,T1,Q1 D1,T1,Q0 D1,T1,Q1 D0,T,Q0 D1,T,Q0 D0,T,Q0 D0,T,Q1 D1,T,Q0 D1,T,Q1 Fig. 3 An example of layer-level prioritization with the Basic extraction.. Quality Layer Extraction Except the base layers (or level in each temporal level, the prioritization order of each NALU can be determined according to the hybrid relationship from the resolution and quality enhancement layers. This prioritization exhibits the hierarchical distinctions or stratification among various parts of the bit stream in the stream adaptation. The priority information may be conveyed in two ways: either the NALU header (by analyzing the syntax of the triplet of scalability, or the optional Supplemental Information (SEI message. The efficiency of extraction for the SVC-coded stream would be improved, because the prioritization is refined in the QL (Quality-Layer-based extraction. ISBN: 978-1-61804-140-1 74
Prioritization Order Base Layer Enhance Layer Resolution 0 Enhance Layer Resolution 1 Temporal Layer 0 Temporal Layer 1 Temporal Layer D0,T0,Q0 D0,T1,Q0 D0,T,Q0 D1,T0,Q0 D1,T1,Q0 D1,T,Q0 processing to improve the extraction efficiency in the SVC streaming technologies. Therefore, the optimized unit-level decision in the SVC extraction would adapt the video streaming to not only the video content but also the transmission scenarios. To reconstruct the SVC-coded images, the SVC layer recovery (or error concealment should be provided before the video decoder to avoid the side effects incurred by the SVC layer inconsistency in a fixed interval (group of picture or Intra period. The error concealment techniques such as the frame copy or bilinear interpolation would be considered with the correlation between the discarded and the extracted units. Video Input Fig. 4 An example of hybrid layer-level prioritization with the QL extraction. The decision policy of QL extraction is illustrated in Fig.4. To improve the efficiency of video transmission, the decision rule is refined with hybrid-layer prioritization of QL extraction. The priority of base layers is identical in both of Basic and QL extractors. According to the decorrelation among the units with different temporal indices, the performance can be improved by reordering the unit priorities in the resolution or quality enhancement layers. Consequently, QL extraction outperforms Basic extraction in the quality of reconstructed images at the same transmission bitrate. In addition, the improvement of extraction can adapt the prioritization to the dynamic video contents with the granulated layer analysing. 3 Unit-level Refinement for SVC Extraction 3.1 System Architecture The paradigm of SVC system with unit-level extraction is demonstrated in Fig. 5. Instead of the traditional extraction, the relationships between the layers in the SVC bitstream can be determined by the Dependency Analysis in terms of the bitrate of extracted NALUs and layer effects incurred by possible discarded ones. The unit-level optimization realized by lingo [9] considers the information from the channel (e.g., the transmission bandwidth and the Dependency Analysis. More details would be described in Section 3.. Besides, the dynamic information of network conditions or the client s requirements can be adopted in the unit-by-unit Full Bitstream SVC Decoder Video Output SVC Encoder Unit-level Extractor SVC Decoder Video Output Dependency Analysis Unit-level Optimization Channel Information Fig. 5 Paradigm of the optimized extraction with unit-level optimization. 3. Unit-level Optimization The configurations of unit-level optimization involve the transmission conditions in the application and transport layers. Fig. 6 indicates the delivered messages including the analysis of the bitrate of extracted NALUs and layer effects incurred by possible discarded ones in the application layer (from the dependency analysis and bandwidths in the transport layer. Several parameters for the optimization in an intra period are listed as follows. t,q : The layer effect of the unit in quality layer q of frame t. R t,q : The bitrate of the unit in quality layer q of frame t. ISBN: 978-1-61804-140-1 75
x t,q : S t,q : V t,q : e t,q : The decision of unit in quality layer q of frame t (0: non-extracted, 1: extracted. The frame quality with unit loss in the quality layer q (1,...,m+1 of frame t (1,...,n. The maximum q = m+1 means the frame t with all quality layers (units. The quality difference between the layers (units q and q+1 in the frame t which can be formulated as V. t, q ( t, q t, q 1 The difference between the quality layers (units q and q+1 in the frame t which can be formulated as e. t, q ( t, q t, q 1 S S t, S V t, V Application Layer V Application Parameters Layer Effect S Fig. 7 The layer effect for SNR scalability (Q=0,1,. Optimization NALU Bitrate Similarly, can be expressed as follows Transport Parameters Transport Layer Fig. 6 The message passing for the optimization The last three measurements can provide the impact evaluations of each unit (layer in a frame. An example with Q = 0, 1, and is depicted in Fig. 7 for the layer effects in terms of the frame quality and the quality difference. The difference of error is evaluated unit by unit. Take SNR scalability (Q=0,1, for example, S means the frame with all quality layers. S t, and S loss the most two and one quality enhancement layers, respectively. The most damage case S stands for the discard of all layers. All corrupted frames would be reconstructed with error concealment. Assume that the Layer effect t,q for the additional quality layer q= in frame t can be denoted as Additionally, t, ( St, t, (1 t, 1 ( S t, and the MSE different for q=1 can be derived as V t, t, e t, e t, ( t, e t, t, e e e t, e e t, Based on Eqs. ( and (3, t,q can be formulated as (3 t, m Vt, m (4 t, q t, q t, m 1 t, q t, q 1... t, m t, m 1 To address the constraints of unit-level optimization, the bandwidth of transmission channel dominates bitrate of extracted units as expressed in Eq.(5. n m t 0 q 0 R t,q * x t,q R T (5 Therefore, the distortion of reconstructed video sequence can be determined by the impact of layer effect and unit decision. The objective function is to minimize the distortion D expressed in Eq.(6 for each intra period. D t, q * xt, q t q (6 Eventually, the optimization of unit-level extraction can be determined for the temporal and SNR scalabilities as * * t q arg min D,. (7 t,q ISBN: 978-1-61804-140-1 76
4 Experimental Results In the simulation, the test platform includes the software JSVM9.18 and the hardware with Intel Core i7 (3.07GHz and 5G RAM. Table I shows the SVC configurations. The comparisons of RD performance in four sequences (Coastguard, Crew, Mobile, and Harbour are provided in Fig. 8. We can observe that the quality improvement of Basic extraction is not obvious with the incremental bitrate. With the granulated prioritization, QL and the unit-level extractions can perform smooth improvement. The corresponding quality gains in BDPSNR [10] are listed in Table II. Moreover, the unit-level extraction outperforms the QL and Basic ones. In the same transmission rate, the quality improvement can be up to 0.8 db and 1.78 db, respectively. Encoded Frames Table I. The SVC configurations 01 Intra Period 8 Resolution 4CIF GOP size 4 Search Mode Fast QP 3, 6, 0 Structure of Intra Period IBBBP BBB Search Range 3 Table II. Quality comparisons in BDBPSNR ( db between proposed and traditional extractions. Sequences JSVM Basic JSVM QL Coastguard 1.5 0.57 Crew 1.55 0.46 Mobile 1.78 0.57 Harbour 1.41 0.80 5 Conclusions In this paper, we proposed a refined extraction with unit-level optimization according to the dependency analysis to improve the reconstruction quality in video communication. Thus, this granularity of SVC layers can provide the high-efficiency extraction for the streaming. The quality gain can be up to 0.8 db as compared to the QL extraction. In the future, we will consider the robust video system to resist the packet loss in the transmission. Furthermore, this unit-level optimization could be developed with the hybrid ARQ mechanisms while the request signal is available via the feedback channel. Fig. 8 Comparisons of RD performance with three extractions in four sequences ISBN: 978-1-61804-140-1 77
6 Acknowledgment This paper was supported by National Science Council of Taiwan (NSC 101-1-E-194-034. References [1] E. Maani and A. K. Katsaggelos, Unequal Error Protection for Robust Streaming of Scalable Video Over Packet Lossy Networks IEEE Trans. on Circuits and Systems for Video Technology, pp. 407 416, March 010. [] H. Schwarz, D. Marpe, and T. Wiegand, Overview of the scalable video coding extension of the H.64/AVC standard, IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 9, pp. 1103 110, Sep. 007. [3] M. Wien, H. Schwarz, Performance Analysis of SVC, IEEE Transactions on Circuits and Systems for Video Technology., Vol. 17, No. 9, pp.1194-103, September 007. [4] H. Schwarz, J. Vieron, T. Wiegand, M. Wien, A. Eleftheriadis, and V Bottreau, JSVM software, text, and conformance status, Joint Video Team, Doc. JVT-AE031, July, 009. [5] Ye-Kui Wang, Miska M. Hannuksela, Stephane Pateux, and Stephan Wenger, System and Transport Interface of SVC, IEEE Trans. on Circuits and Systems for Video Technology, pp. 1149 1163, Sep. 007. [6] T. C. Thang, J.-G. Kim, J. W. Kang, J.-J Yoo, "SVC adaptation Standard tools and supporting methods," Signal Processing: Image Comm., vol. 4, no. 3, pp. 14-8, Oct. 009. [7] I. Amonou, N Cammas, S. Kervadec, and S. Pateux, Rate-Distortion Optimized Bitstream Extractor for Motion Scalability in Wavelet- Based Scalable Video Coding, IEEE Trans. on Image Processing, pp. 114-113, May 010. [8] I. Amonou, N Cammas, S. Kervadec, and S. Pateux. Optimized Rate-Distortion Extraction with Quality Layers in the Scalable Extension of H.64/AVC, IEEE Trans. on Circuits and Systems for Video Technology, pp. 1186-1193, Sep. 007. [9] Lingo,www.lindo.com/index.php?option=com_ content&view=article&id=8&itemid=4. [10] G. Bjontegaard, Calculation of average PSNR differences between RD-curves, VCEG Contribution VCEG-M33, Austin, TX, Apr. 001. ISBN: 978-1-61804-140-1 78