THE EXPLOSIVE growth of multimedia applications such

Transcription

1 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER Perceptually Scalable Extenion of H.264 Hojin Ha, Jincheol Park, Sanghoon Lee, Member, IEEE, Alan Conrad Bovik, Fellow, IEEE Abtract We propoe a novel viual calable video coding (VSVC) framework, named VSVC H.264/AVC. In thi approach, the non-uniform ampling characteritic of the human eye i ued to modify calable video coding (SVC) H.264/AVC. We exploit the viibility of video content and the calability of the video codec to achieve optimal ubjective viual quality given limited ytem reource. To achieve the larget coding gain with controlled perceptual quality degradation, a perceptual weighting cheme i deployed wherein the compreed video i weighted a a function of viual aliency and of the non-uniform ditribution of retinal photoreceptor. We develop a reource allocation algorithm emphaizing both efficiency and fairne by controlling the ize of the alient region in each quality layer. Efficiency i emphaized on the low quality layer of the SVC. The bit aved by eliminating perceptual redundancy in region of low interet are allocated to lower block-level ditortion in alient region. Fairne i enforced on the higher quality layer by enlarging the ize of the alient region. The imulation reult how that the propoed VSVC framework ignificantly improve the ubjective viual quality of compreed video. Index Term Frequency weighting, H.264/AVC, human viual ytem, perceptual coding, calable video coding, viual attention. I. Introduction THE EXPLOSIVE growth of multimedia application uch a network video broadcating, video-on-demand, and video conference ha energized networked viual communication a an active reearch area. To tranmit voluminou video data over the available bandwidth of network, coniderable effort have been applied to the development of video compreion technique uch a H.261, H.263, H.264, MPEG-1, 2, and 4 [1], [2]. To improve the performance of viual communication ytem, it i neceary to conider both the perceptual Manucript received October 27, 2008; revied April 12, 2010 and December 28, 2010; accepted February 13, Date of publication March 28, 2011; date of current verion November 2, Thi reearch wa upported in part by the Minitry of Knowledge Economy, Korea, under the National HRD upport program for convergence information technology upervied by the National IT Indutry Promotion Agency (NIPA), under Grant NIPA-2010-C , and by the Baic Science Reearch Program through the National Reearch Foundation of Korea funded by the Minitry of Education, Science and Technology, under Grant Thi paper wa recommended by Aociate Editor H. Gharavi. H. Ha i with Digital Media and Communication Reearch and Development Center, Samung Electronic, Suwon , Korea ( hojini@amung.com). J. Park and S. Lee are with the Department of Electrical Engineering, Yonei Univerity, Seoul , Korea ( dewofdawn@yonei.ac.kr; lee@yonei.ac.kr). A. C. Bovik i with the Laboratory for Image and Video Engineering, Department of Electrical and Computer Engineering, Univerity of Texa, Autin, TX USA ( bovik@ece.utexa.edu). Color verion of one or more of the figure in thi paper are available online at Digital Object Identifier /TCSVT /$26.00 c 2011 IEEE quality of the recontituted and diplayed video [14] [24], a well a the eamle delivery and aurance of quality uing calable video coding (SVC) technique to mediate perceptual video quality a a function of bitrate [7] [12]. Recently, a new SVC H.264/AVC ha been developed a an amendment of the H.264 and MPEG-4 part 10 video tandard [4] [6]. The notion of perceptual weighting i enabled by varying the quantization parameter (QP) from 1 to 51 in the tandard [13], [14]. If the QP i contant over all macroblock (MB) in each picture, then the perceptual importance of each MB i regarded to be likewie equal regardle of content. Of coure, thi may reult in MB located in region of high interet or viual attention being quantized to an annoying degree, leading to lo of ubjective quality in the recontituted video. In our propoed viual calable video coding (VSVC) framework, we incorporate perceptual weighting into the reource allocation algorithm. Thi idea i not entirely new. In [33], an improvement of viual quality wa achieved by allocating more encoding bit for region having high perceptual alience. In [15] [21], a non-uniform patial filtering law, called foveation, wa employed to define patial perceptual weight on the MB. Larger weight were applied near preumed viual fixation point, which were repreented at high reolution, while lower weight were aigned to peripheral point. Thi proce of foveation weighting attempt to match the non-uniform denity of photoreceptor over the retina. The local patial bandwidth (LSB) rapidly decreae with ditance from the preumed fixation point(). Fig. 1 i an example of controlling perceptual quality. In one image, the QP i fixed at 40, while in the other, local perceptual quality at the MB level i controlled by varying the QP in the range 34 to 46. In thi example, the 16th frame of the Soccer equence wa ued for viual quality inpection, where the preumed viual fixation point i on the occer player near the right center. It i viually obervable that higher perceptual quality i obtained by controlling the QP a a function of each MB. Thi ugget that adaptive perceptual weighting by patially localized control of the QP i promiing for generally improving the perceptual video quality. Approache that incorporate element of viual perception into video coding have been tudied in [14] [24]. In particular, perceptual quality can be effectively improved by utilizing foveation in MPEG/H.263 video coding, if the fixation point can be acquired or accurately gueed. Reource allocation i accomplihed a a function of the patial ditance from foveation point() [16] [21]. Fixation point election can be directly determined through the ue of eye-tracking, or if that i

2 1668 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER 2011 Fig. 1. Comparion of perceptual quality. (a) Fixed QP allocation for each MB. (b) Dynamic QP allocation for each MB in a picture. inconvenient, by the ue of viual alience meaure baed on color, kin detection, luminance edge, motion, or other viual attractor [28] [31]. Motion and other patiotemporal feature have been ued to identify region of viual importance [22] [28]. In [24], motion information i ued to dicover perceptually ignificant region at low complexity. In our application, the ophiticated motion prediction tool in the H.264/AVC coder ugget that patio-temporal feature are readily available for deciding aeing viual aliency [26] [28]. The approach taken in thi paper i to allocate limited reource to the MB in each frame in order to mediate the perceptual quality of each layer. When the number of target encoding bit for a layer i given, an optimization procedure i formulated to enable block-level reource allocation baed on the video content. Firt, we propoe a perceptual weighting cheme baed on foveation to enable the capture of viible frequencie in video. The cheme conit of two part: viual alience i determined uing motion information [23], [24]; then, the LSB i calculated for each MB baed on a foveation model [15] [21]. Second, we develop a reource allocation algorithm for optimizing perceptual quality by utilizing the computed motionbaed alience to control the calable quality layer in the SVC. The allocation algorithm enhance perceptual quality by allocating more bit to highly localized alient region at the lower quality layer. If degradation occur in le alient region, higher quality will till be attained in more alient region(). At the higher quality layer, the alient region() i expanded. Fairne among MB i fulfilled by allocating more bit to region() of lower aliency at the highet quality layer. Thu, a tradeoff between efficiency and fairne i mediated baed on the number of available bit and the aociated quality layer. We provide imulation where we meaure the performance gain in term of efficiency and fairne relative to conventional algorithm. To meaure efficiency, we utilize the foveated peak ignal-to-noie ratio (FPSNR). To meaure fairne, we plotted the ditribution of mean quare error (MSE) of a frame. II. Sytem Decription A. Motivation Perceptual weighting of MB i an approach that ha been effectively ued for ingle-layer video coding uing rate Fig. 2. Propoed VSVC framework compared to a conventional SVC cheme. (a) Conventional SVC cheme. (b) Propoed VSVC cheme. control [16], [20], [21], [25], [28]. For SVC H.264/AVC coding, we are unaware of any developed approach that, e.g., adaptively elect a QP and allocate bit to each layer baed on perceptual aliency or foveation. Such an approach i feaible and promiing; by uing a rate control algorithm, the number of target bit could be decided, and a contant QP determined for each layer, on a frame-by-frame bai. Thi would require flexibly aigning perceptual weight to each MB, a well a an appropriate bit allocation cheme. To improve the coding performance of each quality layer, our propoed VSVC approach exploit the non-uniform ampling of the eye enory apparatu by controlling the ize of identified alient region acro layer, and by controlling the number of bit for each MB via a rate control algorithm. Fig. 2 how the mechanic of the propoed VSVC framework compared to a conventional approach. At each quality layer, the region indicated by dotted line indicate region of preumed higher alience, while the olid line indicate region of lower alience. The conventional approach hown in Fig. 2(a) applie an equal perceptual weighting acro the layer. However, the human viual repone i higher for lower frequency information, which can be taken into account when electing the quantization level. In the conventional approach, no account of viual attention or patial aignment of viual importance or alience i ued in defining the three quality layer. By comparion, Fig. 2(b) how the variou feature of our propoed VSVC framework. The three quality layer are configured to optimize perceptual quality, i.e., an MB-level rate control mechanim compute the perceptual weight a a function the identified perceptually mot alient region in each quality layer, which are dynamically elected. For the lowet quality layer (layer 0), a mall region of viual importance i maintained over which perceptual quality will be maintained. The region() of preumed perceptual interet i larger in layer 2, and larger till in layer 3. Significant aving in bit allocation in all three layer can be obtained in thi way, with high efficiency in layer 0 and 1. Fair reource allocation can be achieved in layer 2 to maintain the perceptual quality of all MB by expanding the ize of the alient region(). B. Overview of the Propoed VSVC Algorithm Fig. 3 i a block diagram for the propoed VSVC cheme. Fig. 3(b) and (c) how the reult of applying foveation-baed

3 HA et al.: PERCEPTUALLY SCALABLE EXTENSION OF H perceptual weighting on the 35th frame of the Silent video tet clip. The propoed VSVC tructure i baed on the baic SVC H.264 deign, which i claified a a layered video coder. Fig. 3(a) how the typical coder tructure with two patial layer repreented uing a olid line. Since the propoed VSVC algorithm extend the coare-grain Signal-to-noie ratio (SNR) calability, the inter-layer prediction mechanim uing the upampling operation i omitted [34], [35]. The dotted proceing block are added for the propoed SVC algorithm. Next, the LSB for each MB i found by the perceptual weight allocator, which determine motion-baed alience, determine which MB fall within the alient region, and applie foveation-baed perceptual weight. Fig. 3(b) illutrate the outcome of the motion-baed alience model. The face and left hand, both of which are in motion, are elected a region of heightened viual interet. The LSB i decreaed exponentially from the center of each alience region, which we will call the foveation point. The intention i that when a viual fixation fall on the region, the projection of the ditribution of LSB onto the retina will approximately match the non-uniform ditribution of the photoreceptor. A map illutrating the foveation-baed perceptual weighting model i hown in Fig. 3(c). Reource are allocated according to patial placement within the indicated iocontour of the foveation-induced LSB. In thi example, MB in region A are located in a high alient region and are thu finely quantized. MB in region B are located in region of lower aliency reulting in coarer quantization. By conidering viual importance together with bit allocation at each quality layer, the reource allocator eek to optimize efficiency and fairne to achieve improved perceptual video quality. Encoding each quality layer depend on the QP, on motion information, and on identifying alient region. Finally, the encoded bit are multiplexed to produce a calable video bittream. III. Foveation-Baed Perceptual Weighting Allocator We introduce two perceptual model: one i motion-baed and the other i foveation-baed. Uing thee model, it i poible to calculate the LSB for each MB in each quality layer. The perceptual weight are defined over the patial and temporal domain uing the following obervation. 1) Moving object in video are trong attractor of viual attention and tend to draw viual fixation [23], [24], [26], [29], [37], [38]. 2) The eye i enitive to the temporal correlation of moving object [22]. 3) Fixated region in a video are perceived at high reolution via foveal ampling [15], [20], [21], [29]. The motion-baed aliency model exploit the firt and econd aumption. The third aumption i employed in the foveation-baed perceptual weighting model. Fig. 4 diagram the foveation-baed perceptual weight allocator, and how reult from it operation. The weight Fig. 3. Overview of the propoed VSVC cheme. (a) Block diagram of the propoed VSVC cheme. (b) Reult of applying the motion-baed alience model. (c) Depiction of the foveation-baed perceptual weighting cheme. Foveation-induced bandwidth are plotted a io-contour within which the perceptual weight, quantization, and bit allocation are determined. allocation algorithm conit of four tage. Stage A C accomplih the motion-baed aliency determination, while Stage D obtain the foveation-baed perceptual weighting. Thi mean that we regard the alient MB a an extended form of foveation point in the unit of MB rather than in the unit of pixel for obtaining the LSB of MB in each frame. Simulation reult are hown for each tage, which are explained below, uing the 23rd frame of the Soccer video clip. A. Motion-Baed Saliency Model 1) Stage A: Partition Selection: Fig. 4(a) diagram the motion-baed aliency model. The model utilize motion intenity (peed) and the MB partition, both of which can be obtained from the motion etimation (ME) module in the hierarchical B prediction. In H.264/AVC, there exit nine different prediction mode for intra MB and even different prediction mode for inter MB. The block partition of the kth MB in the nth frame, which i denoted a P, i calculated by performing rate-ditortion optimization (RDO)-baed coding mode election algorithm [25], [34] [36]. From Fig. 4(b), it may be oberved that block having mall partition ize are aociated with inhomogeneitie, uch a edge. Block having large partition ize are aociated with homogeneou region [24], [27]. To attempt to ditinguih moving object of interet from the background, peed i employed. For the kth MB in the nth frame, the peed i given by I = (MV x )2 +(MV y )2 (1)

4 1670 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER 2011 Fig. 4. Block diagram and imulation reult for the foveation-baed perceptual weight allocator. (a) Block diagram. (b) Reult of Stage A. (c) Reult of Stage B. (d) Reult of Stage C. (e) Reult of Stage D in Soccer video clip. (f) Reult of Stage D in Stefan video clip. where MV x and MV y are the horizontal and vertical component of motion. Uing I and P, alient region are found in each frame. 2) Stage B: Determination of Salient Region: If P indicate a mall partition ize, the correponding block may occur at or near an object boundary. However, the block may alo belong to the background. Uing only P, it i not poible to determine whether the MB hould be conidered a a alient MB. The MB may be part of a alient region, provided that the peed of the MB i nonzero. However, if the peed i too large, the perceptibility of the moving MB may ignificantly decreae. Hence, an upper bound on the peed of the MB i required, i.e., if the MB belong to a alient region, then it peed mut fall below a threhold [23], [29], [32], [37] 0 <I < k where k i the um of the global mean, μ k, and tandard deviation, δ k, of motion intenitie in the kth frame a follow: k = μ k + δ k (2) where the motion intenity (I l,k ) i calculated from (1). In the cae of an intra-coded MB, the motion intenity of the MB i et to zero ince there i no motion vector. However, for forward, backward, and bidirectional-coding type, motion vector for each MB are utilized for obtaining the motion intenity regardle of the direction of prediction. If the peed of the MB fall outide of thi range, then the MB i regarded a a part of a non-alient region. A binary function A i ued to indicate whether or not the kth MB belong to a alient region; A =1 indicate aliency while A =0 indicate otherwie. The reult of Stage B i exemplified in Fig. 4(c). It may be oberved that edge of the moving occer player and the moving ball are elected a candidate alient region. 3) Stage C: Motion Conitency: Auming that alient region typically urvive acro conecutive frame [22], we define a binary function C that indicate the temporal continuity of motion flow of alient object, a follow: { 1, if W S C = (3) 0, otherwie where W = A n 1,k + A + A n+1,k and S = 2. Baed on thi criterion, moving block having tranient motion (A =1, C = 0) are conidered a non-alient. Thi motion conitency criterion can vary according to the frame rate, e.g., by increaing S at a high frame rate and by diminihing S at a low frame rate. The proceing of Stage C i exemplified in Fig. 4(d). It may be oberved that many block with A =1 are removed in the proce, owing to their low motion conitency. Yet there remain a number of MB that are promiing alient object. Uing A and C, the alient region election algorithm i ummarized. 1) Calculate I and P for the kth MB. 2) Determine whether P < MODE If o, proceed to the next tep. Otherwie, indicate that the MB i not alient by etting A =0.

5 HA et al.: PERCEPTUALLY SCALABLE EXTENSION OF H Fig. 5. Viewing geometry for the foveation model. 3) Determine whether 0 <I < k. If o, indicate that kth MB i a candidate alient MB: A = 1. Otherwie, et A =0. 4) Determine whether W >S. If o, then kth MB i alient: C = 1. Otherwie, it i not C =0. The block ize for the MB partition i elected by performing the RDO-baed coding mode election algorithm. Thu, the block partition i, in fact, dependent on the target bitrate. In mot cae, the higher the target bitrate, the greater the number of block with maller partition mode, o that the number of alient MB increae, and vice vera. However, thi i not critical to the algorithm from the point of view of efficiency and fairne. If the target bitrate i high, it i beneficial that the alient region i larger, o that fairne i guaranteed, and vice vera. Thu, the variation of block ize a a function of the target bitrate i not critical to the following efficiency-fairne (EF) reource allocation algorithm. In the wort cae, no MB in a frame could be elected a a alient MB at a low bitrate. In thi ituation, the center MB of a frame could be candidate alient MB baed on the tendency of human fixation to eek the center of the frame ide [15], [20], [21]. B. Foveation-Baed Perceptual Weighting Model Light reflected from object in the environment pae through the optic of the eye and onto the retinal photoreceptor (cone and rod). We make the very reaonable aumption that the video i bright enough to elicit photopic viion, hence the viual input i dominated by the repone of the cone. The denity of the cone and aociated receptive field neuron i non-uniformly ditributed acro the retinal topography, peaking in denity at the fovea, which alo i centered on the optical axi of the eye. The point on an object urface that project light onto the center of the fovea i correctly termed a point of viual fixation. The term foveation point, which we hall ue, i related, but different. Since our application i preenting video image to human oberver, it i preumed that the point of viual fixation fall on the diplayed video being viewed. A foveation point in a video i a coordinate in pace-time where it i expected that human fixation are likely to be placed. Therefore, foveation point in video are repreented with a high patial reolution. The reolution i made to fall off ytematically away from a foveation point, unle another foveation point i approached. Thi cheme i applicable for multiple foveation point a well. It can be een that there are multiple foveation point in Fig. 4(d). If the LSB for each foveation point overlap, then the larger LSB i choen to determine the perceptual weight. In thi way, the video preentation i made to have high reolution where the oberver viual fixation are known or predicted to be placed, thu eeking to match the ampling capability of the retina. There ha been ueful work done on determining the viual reolution repone (contrat enitivity) a a function of the placement of the timulu on the retinal relative to the fovea, which i known a the retinal eccentricity [39] [41]. Fig. 5 diagram a viewing geometry where p f =(p f x,pf y ) (pixel) indicate a foveation point that i alo a fixation point, v i the viewing ditance from the eye to the diplay image plane, and N i the number of pixel along the horizontal axi. The ditance u from the point p =(p x,p y ) (pixel) to the foveation point p f i u = d( p) / N, where d( p) = p f p 2. The vertical ditance v i defined imilarly. The eccentricity i then defined a the viual angle [20] a follow: e(v, p ) = tan 1 ( u v ) = tan 1 ( d( p ) ). (4) Nv For a given eccentricity, e(v, p), the local patial cut-off frequency (cycle/degree) f c i defined by etting the contrat enitivity to 1.0 (the maximum poible contrat) and i calculated a follow: f c (e(v, p)) = e 2 ln( 1 CT 0 ) α(e(v, p)+e 2 ) where CT 0 i a minimum contrat threhold, e 2 i a halfreolution eccentricity contant, and α i a patial frequency decay contant. In thi model, higher patial frequencie than f c are le viible or inviible. In a diplayed digital image, the effective diplay viual reolution r (pixel/degree) i expreed in term of the viewing ditance v (cm) and diplay reolution N (pixel/cm) a follow: r = pv tan( π 180 ) Nv π 180. (6) The highet diplayable patial frequency i half of r a follow: f d (v) = r 2 Nv π 360. (7) Uing (5) and (7), the local foveated bandwidth of a given MB p k and viewing ditance v i (5) f = min(f c (e(v, p k )),f d (v)). (8) The LSB i explicitly the local foveated bandwidth within which the maximum poible contrat can be recognized a a function of the ditance from a foveation point. Thu, an MB will have a lower LSB when it i further from the foveation point. Likewie, high frequency coefficient tend to be more everely quantized a the QP i increaed. Therefore, in Section IV, we propoe a cheme to adjut local QP a a function of the LSB.

6 1672 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER ) Stage D: Computation of LSB Uing Foveation Model: A hown in Fig. 4(d), there can be multiple foveation point. Thu, multiple LSB overlap. In thi ituation, the larget LSB i choen to determine the perceptual weight a follow: f = max f { f ( d f,k )} (9) where the local foveated bandwidth i expreed a a function of the ditance, d f,k, between the f th foveation point and the kth MB. Uing f, the LSB i obtained for each MB a hown in Fig. 4(e). In Fig. 4(f), the 6th Stefan video tet clip i additionally hown a a reult. Thi proce make it poible to eliminate viual redundancy from non-alient region to improve coding efficiency. 2) Computational Complexity: The foveation-baed perceptual weighting model conit of Stage A D. In Stage A, the root mean quare (RMS) value of directional motion etimate i computed on each MB. Thi value depend on the motion partition. The 4 4 ub-block i the mallet for a MB, in which cae the MB conit of 16 ub-block. Suppoe that all the MB of a video frame have 4 4 ub-block. In thi cae, the maximum number of RMS value to be computed i 16K where K i the total number of MB in the frame. After Stage A i finihed, Stage B and C perform imple threhold comparion to determine the aliency region of all MB. Baed on the elected foveation point, the LSB i then computed. Since the LSB can be obtained uing table look-up there i negligible computational overhead. IV. Optimal Reource Allocation: Efficiency and Fairne The LSB of each MB can be obtained uing the foveation principle outlined in the preceding. In order to adjut video quality a a function of the LSB, the QP i adjuted according to the aliency of each MB. Here, reource allocation cro the layer i developed conidering efficiency. Now let r k (q k ), d k (q k ), and q k be the rate, ditortion, and QP of the kth MB. Let M be the number of MB in a frame, and R n T be the number of target bit for a frame. Given QP q k for the kth MB, then r k (q k ) and d k (q k ) are calculated for each MB uing the RDO algorithm in the reference oftware SVC H.264/AVC [12]. The QP for coding the M MB then compoe a quantization tate vector Q = (q 1,q 2,,q M ). Given the above, we propoe an optimal reource allocation algorithm for each quality layer in the following. For brevity, (v) in (7) and expre it a f. We aume a viewing ditance, v = 40 cm. If the viewing ditance i increaed, then the LSB decreae more lowly from the foveation point, and vice vera. At a greater ditance, a larger region i covered by the angle ubtended by the fovea, cauing the enitivity to be more uniform over a larger part of the image. In thi cae, fairne increae in importance (and vice vera). Thi i reflected in the deign of the LSB model (5). we drop the dependence on v in f A. EF Reource Allocation Algorithm Here we introduce the EF algorithm that eek to achieve a deirable tradeoff between efficiency and fairne in each quality layer. If efficiency i only conidered at the expene of fairne, then MB having large LSB will obtain the mot encoding bit. Other MB having low LSB will likely have very poor perceptual quality. The more efficient the cheme become, the more unfair the enhancement quality layer will become. While thi i the idea of our cheme, it i till important to carefully deign the reource allocator to mediate efficiency while maintaining an appropriate modicum of fairne at each quality layer. The quantity f in (7) i ued to determine the ize of the alient region in each layer. For a given efficiency level l, let EF(l) denote the threhold above which f mut fall for an MB to fall within a alient region, and vice vera. Fig. 6(a) how the layered mechanim of the propoed VSVC framework. Uing efficiency level 9 in Fig. 6(a), the highet efficiency i achieved by etting EF(9) = EF max, which i the larget poible LSB. Thi make it poible to maintain optimal perceptual quality over the mallet alient region. Uing efficiency level 4 in Fig. 6(a), both efficiency and fairne are moderately realized by aigning an intermediate value of the LSB to EF(4), which lead to an enlarged alient region. Finally, uing efficiency level 0 in Fig. 6(a), increaed fairne can be aured by aigning a lower value EF(0). Thi make the alient region a large a poible. The LSB of the MB within the alient region i et to the highet value. We term thi cheme the EF algorithm, repreenting the balanced tradeoff between efficiency and fairne at each layer. To decide the required number of alient region ize, f i mapped onto dicrete level of frequency enitivity, which vary with the frequency indice of the tranform coefficient, the coefficient magnitude, and the block luminance [14], [23], [24], [26]. Let α and β be the indexe of 2-D tranform coefficient in a block. The normalized local patial frequency (cycle/degree) can be expreed a follow: N (α, β) = 1 α2 + β 2N 2 (10) where N (α, β) i normalized by 0.5 [23]. Fig. 7(a) how the contour of N (α, β) a a function of the tranform coefficient index. In our implementation, ten value of N (α, β) are ued to define the ize of the alient region. To modify the perceptual weighting of each MB a a function of quality layer, f i quantized uing the value of N (α, β). Denote the quantized verion of f by ˆf. The quantization proce i a follow: ˆf = max { z N z f } (11) where N = { (ᾱ, N (1, 1),N (1, 2),,N β )}. ᾱ and β are the maximum tranform coefficient indice. In SVC H.264/AVC, ᾱ = 4 and β = 4. For example, if f = 0.19, then ˆf = Thi follow ince for the next level N (α, β) = 0.25, the nearet dicrete value of f i Fig. 7(b) how the contour of ˆf derived from f by uing (11).

7 HA et al.: PERCEPTUALLY SCALABLE EXTENSION OF H Fig. 6. Propoed VSVC framework. (a) Ditribution of reource to achieve efficiency and fairne acro layer in the propoed VSVC framework. (b) Illutration of variation of alient region ize and hape acro the quality layer. (c) Ditribution of w (i) in the MB in an arbitrary image row (eventh row). Fig. 7. ˆf Plot of coefficient indice. (a) Contour of N (α, β) a a function of coefficient index. (b) Contour of ˆf. (d) Relation between g and ˆf. from f. (c) Contour of g from g The value of ˆf are mapped onto integer value denoted in the range [0, 9] a depicted in Fig. 7(c) which i denoted a g. The relation between g a follow: weighting function w ˆf and ˆf i defined by a = w (g ). (12) Fig. 7(d) how the weighting function of w in (12). If g = 9, then the aociated MB obtain the highet dicrete value ˆf ˆf = 0.5. At the other extreme, if g = 0.01 i aigned to the MB. = 0, then lowet value Suppoe that there are L quality layer. Then, for a given quality layer l, the perceptual weighting of each MB in the EF algorithm i fixed by defining ŵ a follow: { ŵ (g w )= ( (L), if g l g (13) w + L 1 ), otherwie where L = 9 in our implementation. Thu, the region with g l become alient and ŵ i decreaed in proportion to the ditance from the alient region. Fig. 6(b) and (c) depict the operation of the EF algorithm. The Soccer tet clip i utilized. The algorithm i illutrated for three efficiency level l =0, 4, 9 with the correponding QP of 42, 36, and 28, while howing the variation in the ize of the alient region. B. EF Reource Allocation The reource allocation algorithm i deigned to minimize the overall ditortion ubject to the target rate contraint uing a rate control algorithm a follow: [ ] M [ŵ min D( Q) = min (g ) d k (q k ) ] (14) Q Q ubject to (13) and ubject to k=1 { M k=1 r k(q k ) R T q min q k q max (15) where l = {0, 1, 2,...,9} and L = 9. The firt contraint i the number of target bit. The econd contraint repreent

8 1674 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER 2011 the feaible range of variation of the QP. A large difference in the QP between adjacent MB may reult in an abrupt change in perceptual quality, which may be noticeable and viually annoying. ForagivenR T, the goal of (14) i to allocate an optimal QP for each MB to minimize the overall perceptual ditortion. Thi problem can be olved by uing a greedy algorithm. At each iteration of the greedy algorithm, each MB i evaluated to determine which can achieve the greatet (weighted) ditortion reduction uing the leat bit, if there are encoding bit available. To conduct thi evaluation, define φ k = ŵ (g ) (d k (q k ) d k (q k 1)). (16) r k (q k 1) r k (q k ) Thi quantity evaluate the gradient of the reduced weighted ditortion that i achieved when q k i decremented by one. Which MB mot effectively enhance the expected perceptual quality via rate control i determined a ˆk = arg max [φ k ]. (17) k If q max qˆk 1 q min, then the QP for the ˆkth MB i decremented by 1. Otherwie, another MB i elected that atifie (17). The current rate allocated to MB ˆk i updated, by etting R um = R um + rˆk (qˆk ). Thi procedure i repeated until R um R T. The iteration algorithm i ummarized a follow. 1) Initialize R T, R um =0,q k = q max for all MB. 2) For each layer l, w (i) i aigned to MB uing (15). 3) Calculate φ k uing (16) for all MB. 4) Find ˆk (17) that atifie q min qˆk q max uing (15). 5) If R um + rˆk (qˆk ) > R T, thi procedure i terminated. Otherwie, go to Step 6. 6) If qˆk <q min, et φˆk = 0 and go to Step 4. 7) Update: qˆk = qˆk 1 and R um = R um + rˆk (qˆk ). 8) Calculate φˆk uing (16) for the ˆkth MB and go to Step 4. Thi iterative procedure i continued until Step 5 i atified. After the procedure i terminated, q k = qk which i the optimal QP of the kth MB. The vector Q conit of the optimal QP value qk for 1 k M. V. Simulation Reult We ue the reference oftware SVC H.264/AVC [12] to demontrate the effectivene of the propoed VSVC algorithm. The propoed VSVC algorithm and the conventional SVC algorithm uing frame level framework (the previou verion of SVC H.264/AVC) are ued for performance comparion. Encoding configuration are a follow. The RDO mode and the loop filter are enabled. The content-baed adaptive binary arithmetic coding option i turned on and variable block ize with a earch range of 32 are utilized for block ME. For calable video, the maximum temporal level i 5 baed on group of picture with 16 frame. All frame except I frame are coded a B frame, which ue forward and backward referencing. Let QP Tmax be the QP at the highet temporal level, which i known prior to coding. The remaining TABLE I Target Bitrate and Ued Bitrate Comparion for Different Quality Layer with Conventional SVC Algorithm, EF(9), and EF(2) Quality Target Ued Bitrate Layer Bitrate Conventional EF(9) EF(2) Stefan BL EL City BL EL Soccer BL EL Silent BL EL BL: bae layer; EL: enhancement layer. QP T for the temporal level 0 T<T max are determined by the reference oftware [12], [34], [35]. We conider SNR calability to improve perceptual quality without including temporal and patial calability. Thu, one patial reolution i encoded for quality layer calability. The parameter q max and q min for each temporal level T are et to QP T + 6 and QP T 6, repectively. The foveation-baed perceptual weighting model i configured auming a block ize of 4 4 for evaluating ˆf. The following tet video clip with a CIF reolution of 30 f/ were ued for the performance comparion: Soccer, Silent, Stefan, and City where 300 frame are ued for each tet video clip. The imulation reult are tudied with repect to two apect: objective perceptual quality and ubjective perceptual quality. A. Objective Viual Quality Evaluation In thi ection, we evaluate the propoed VSVC algorithm at different quality layer. Two different quality layer of the EF algorithm are conidered: l = 2 and 9, which repreent pure fairne and efficiency, repectively. Two type of quality layer are conidered: the bae and the enhancement quality layer, where the value of initial QP Tmax are et to 42 and 36, repectively. Uing the initial QP Tmax, the total ued bitrate i controlled to the target bitrate. Table I compare the target and ued bitrate for each quality layer. To evaluate efficiency in the bae quality layer, we utilize the FPSNR firt developed in [16]. Specifically, the mean quare error weighted by quantized LSB ˆf i defined here a follow: FMSE LSB = 1 M M J ( ˆf ) 2 k=1 k=1 j=1 J j=1 (o k (j) r k (j)) 2 ( ˆf ) 2 (18) where o k (j) ithejth pixel of the kth MB in the original video frame, and r k (j) ithejth pixel of the kth MB in the recontructed image. M and J repreent the total number of MB in a frame and the total number of pixel in an MB, repectively. Then, the overall FPSNR i FPSNR = 10 log FMSE LSB. (19)

9 HA et al.: PERCEPTUALLY SCALABLE EXTENSION OF H TABLE II Average FPSNR and PSNR Comparion for Different Quality Layer with the Conventional SVC Algorithm, EF(9), and EF(2) Scheme Stefan City Soccer Silent Average Conventional FPSNR EF(9) (BL) EF(2) Average Conventional FPSNR EF(9) (EL) EF(2) Average Conventional PSNR EF(9) (BL) EF(2) Average Conventional PSNR EF(9) (EL) EF(2) BL: bae layer; EL: enhancement layer. Table II tabulate the average FPSNR and PSNR comparion for each quality layer. In the bae layer, the EF(9) cheme deliver a higher FPSNR than EF(2) and the conventional SVC cheme (in the range db), ince EF(9) allocate more reource to the identified perceptually important region ince there i a lack of encoding bit. A hown in Fig. 8, EF(9) conitently deliver higher FPSNR value acro frame. Converely, the average traditional PSNR of EF(9) i lower than that of conventional SVC for all of the tet clip in Table II. Thi i expected, ince conventional SVC utilize a reource allocation cheme in order to lower the highet ditortion for each MB in a frame. Yet thi i not an effective meaure of the perceptual quality. A hown in Table II, the EF(9) cheme yield the lowet average FPSNR, ince it make an effort to reduce the ditortion of the mallet alient region. A compared to conventional SVC, EF(2) obtain a higher FPSNR (in the range of ) over all of the tet video clip, indicating that EF(2) provide more efficient perceptual quality improvement than the conventional algorithm. Although the average PSNR of EF(2) i lower than that of conventional SVC, SVC reduce the overall ditortion over the MB without regard to the perceptual relevance of the alient region. B. Subjective Quality Comparion In order to conduct ubjective quality comparion, we ue a video tet clip recontructed by both the propoed VSVC algorithm with the EF algorithm and by conventional SVC. We preent a few exemplary recontructed picture from different quality layer. In addition, we have made everal demo equence available for public download at [44], to enable viual comparion between the reult of conventional SVC and the propoed VSVC cheme. Fig. 8 how a frame from the video clip recontructed uing the bae ( kb/) and enhancement ( kb/) layer in the conventional SVC algorithm, where EF(8) ( kb/) and EF(4) ( kb/) are ued for the bae and enhancement layer. The 98th frame of the City tet video clip i ued. Fig. 8(a) and (b) compare the ubjective quality for the bae layer. When conventional SVC wa ued, noticeable perceptual Fig. 8. Subjective quality comparion for different quality layer on the 98th frame of the City tet video clip. (a) Bae layer in conventional SVC. (b) Bae layer in EF(8). (c) Enhancement layer in conventional SVC. (d) Enhancement layer in EF(4). degradation occur in the middle of the frame owing to the lack of adequate encoding bit. On the contrary, EF(8) preerve detail in the video frame better than doe the conventional SVC cheme. Thi i a good demontration that efficiency i more important than fairne toward improving ubjective video quality, when there i a lack of encoding bit. Fig. 8(c) and (d) compare the ubjective quality for the enhancement quality layer. Since EF(8) allocate more encoding bit to MB having a large perceptual weighting, a noticeable degradation of the ubjective quality occur in the remaining MB. However, the propoed fairne cheme improve the ubjective video quality more effectively than doe the efficiency cheme on the enhancement quality layer. Converely, when conventional SVC i ued, noticeable perceptual degradation occur in the middle of the frame owing to a lack of adequate encoding bit in the bae layer. Fig. 9 how the degree of reduced ditortion over the frame uing conventional SVC, EF(8), and EF(4). The ditortion i defined a the MSE between the original and recontructed image divided by The 98th frame of the City tet video clip i ued. Fig. 9(a) and (b) repreent the ditribution of reource in conventional SVC for different quality layer. Since reource allocation in conventional SVC i baed on the MSE, reource are allocated equally into all MB in the bae and enhancement quality layer. The ditribution of reduced ditortion in the propoed SVC i hown in Fig. 9(c) and (d) for the different quality layer. It may be oberved that EF(8) allocate reource in the perceptually important region with l = 8 in the bae quality layer. In the enhancement quality layer, EF(4) enlarge the range of allocated reource in the ame region with l =4. Fig. 10(a) and (b) how the video clip recontructed uing the bae layer in conventional SVC ( kb/) and in EF(8) ( kb/) with an efficiency level of l = 8, repectively. The 16th frame of the Stefan tet video clip i ued for comparion.

10 1676 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER 2011 Fig. 9. Ditortion and QP ditribution comparion for different quality layer on the 98th frame of the City tet video clip. (a) Amount of reduced ditortion in bae layer of conventional SVC. (b) Amount of reduced ditortion in the enhancement layer of conventional SVC. (c) Amount of reduced ditortion in the bae layer of EF(8). (d) Amount of reduced ditortion in the enhancement layer of EF(4). Fig. 10. Subjective quality comparion on the 14th frame of the Stefan tet video clip. (a) Bae layer in conventional SVC. (b) Bae layer in EF(8). Fig. 11. Subjective quality comparion on the 103th frame of the Silent tet video clip. (a) Enhancement layer in conventional SVC. (b) Enhancement layer in EF(4). and in EF(4) ( kb/) with an efficiency level of l =4, repectively. The 103rd frame of the Silent tet video clip i ued. Baed on the propoed motion-baed aliency model, the region where hand movement for ign language i elected a the alient region. While the enhancement layer in EF(4) focue on the quality improvement of the elected alient region, conventional SVC till aign more bit into the background. Conequently, the viual quality in EF(4) i better than that of conventional SVC owing to the alient regionbaed reource allocation. The propoed fairne cheme in VSVC effectively inhibit perceptual degradation in area of identified perceptual importance by allocating increaed reource to thoe MB. From thee viual quality comparion and the FPSNR analyi, it i apparent that the perceptual weighting cheme of the propoed SVC can efficiently improve the ubjective quality of H.264 compreed video in identified perceptually important region. VI. Concluion We have propoed an MB-level perceptual weighting framework a an extenion to SVC in H.264/AVC. To enable adaptation to the non-uniform reolution of the viual photoreceptor, we developed a foveation-baed perceptual weighting allocator. The propoed VSVC cheme wa developed baed on two eential objective, namely, enforcing efficiency and fairne in the quality layer to improve coding performance. To find an optimal tradeoff between efficiency and fairne, we propoed the EF reource allocation algorithm, which preferentially allocate coding reource to alient region. The ize of the alient region i increaed a the efficiency level i raied. The imulation howed that the VSVC algorithm achieve higher perceptual quality uing FPSNR than doe conventional SVC by db in the different quality layer. In ummary, it can be concluded that the propoed VSVC algorithm i a promiing olution for achieving high-quality and reliable video communication over variable rate channel with good control of QoS. Such algorithm that emphaize perceptual quality a a function of viual importance, aliency, computed or meaured fixation, or other imilar criteria are likely to continue growing in importance, owing to the ongoing increae in diplay ize and video bandwidth, expectation for higher video quality, and generally, increaed ubiquity of video in our daily environment. Uing conventional SVC, a noticeable degradation of quality occur owing to the lack of encoding bit. However, the propoed efficiency algorithm allocate more reource into the alient region to improve the viual quality. While the pectator in the background are more ditorted, the tenni player, which ha been aigned high perceptual importance, i maintained with higher quality. Fig. 11(a) and (b) how the video clip recontructed uing the enhancement layer in the conventional SVC ( kb/) Reference [1] ITU-T and ISO/IEC JTC 1, Generic Coding of Moving Picture and Aociated Audion Information, Part 2: Video, document ITU-T Rec. H.262 and ISO/IEC (MPEG-2 Video), Nov [2] ITU-T and ISO/IEC JTC 1, Advanced Video Coding for Generic Audio- Viual Service, document ITU-T Rec. H.264 and ISO/IEC (MPEG-4 AVC), May [3] H. Schulzrinne, S. Caner, R. Frederick, and V. Jacobon, RTP: A tranport protocol for real-time application, document RFC 1889, Internet Official Protocol Standard (STD 1), Jan

11 HA et al.: PERCEPTUALLY SCALABLE EXTENSION OF H [4] Text of ISO/IEC :2005/FDAM 3 Scalable Video Coding, document N9197, Joint Video Team (JVT) of ISO-IEC MPEG and ITU-T VCEG, Lauanne, Switzerland, Sep [5] ITU-T and ISO/IEC JTC 1, Advanced Video Coding for Generic Audio Viual Service, ITU-T Recommendation H.264 and ISO/IEC (MPEG-4 AVC), Verion 1: May 2003, Verion 2: May 2004, Verion 3: Mar. 2005, Verion 4: Sep. 2005, Verion 5 and Verion 6: Jun. 2006, Verion 7: Apr. 2007, Verion 8 (including SVC extenion): conented in Jul [6] Text of ISO/IEC :2001/PDAM 19 Reference Software for SVC, document N9195, Joint Video Team (JVT) of ISO-IEC MPEG and ITU- T VCEG, Sep [7] S. J. Choi and J. W. Wood, Motion-compenated 3-D ubband coding of video, IEEE Tran. Image Proce., vol. 8, no. 2, pp , Feb [8] H. M. Radha, M. V. D. Schaar, and Y. Chen, The MPEG-4 find-grained calable video coding method for multimedia treaming over IP, IEEE Tran. Multimedia, vol. 3, no. 1, pp , Mar [9] Y. W. Chen and W. A. Pearlman, Three-dimenional ubband coidng of video uing the zerotree method, Proc. SPIE, vol. 2727, pp , Mar [10] J. Park, H. Lee, S. Lee, and A. C. Bovik, Optimal channel adaptation of calable video over a multi-carrier baed multi-cell environment, IEEE Tran. Multimedia Special Iue, vol. 11, no. 6, pp , Oct [11] U. Jang, H. Lee, and S. Lee, Optimal carrier loading control for the enhancement of viual quality over OFDMA cellular network, IEEE Tran. Multimedia, vol. 10, no. 6, pp , Oct [12] J. Reichel, H. Schwarz, M. Wien, and J. Vieron, Joint Scalable Video Model 9 of ISO/IEC :2005/AMC3 Scalable Video Coding, document JVT-X202, Joint Video Team (JVT) of ISO-IEC MPEG and ITU-T VCEG, Geneva, Switzerland, Jul [13] J. Reichel, M. Wien, and H. Schwarz, Scalable video model 3.0, document N6716, ISO/IEC JTC 1/SC 29/WG 11, Palma de Mallorca, Spain, Oct [14] B. Ciptpraet and K. R. Rao, Human viual weighting progreive image tranmiion, in Proc. ICCS, Nov. 1987, pp [15] S. Lee, M. S. Pattichi, and A. C. Bovik, Foveated video quality aement, IEEE Tran. Multimedia, vol. 4, no. 1, pp , Mar [16] S. Lee, M. S. Pattichi, and A. C. Bovik, Foveated video compreion with optimal rate control, IEEE Tran. Image Proce., vol. 10, no. 7, pp , Jul [17] S. Lee and A. C. Bovik, Fat algorithm for foveated video proceing, IEEE Tran. Circuit Syt. Video Technol., vol. 13, no. 2, pp , Feb [18] S. Lee, A. C. Bovik, and Y. Y. Kim, High quality, low delay foveated viual communication over mobile channel, J. Viual Commun. Image Repreentat., vol. 16, no. 2, pp , Apr [19] H. Lee and S. Lee, Compreion gain meaurement by uing ROIbaed data reduction, IEICE Tran. Fundamental, vol. E89-A, no. 11, pp , Nov [20] Z. Wang and A. C. Bovik, Embedded foveation image coding, IEEE Tran. Image Proce., vol. 10, no. 10, pp , Oct [21] Z. Wang, L. Lu, and A. C. Bovik, Foveation calable video coding with automatic fixation election, IEEE Tran. Image Proce., vol. 12, no. 2, pp , Feb [22] K. Yang, C. C. Guet, K. E. Maleh, and P. K. Da, Perceptual temporal quality metric for compreed video, IEEE Tran. Multimedia, vol. 9, no. 7, pp , Nov [23] C. Tang, Spatiotemporal viual coniderationfor video coding, IEEE Tran. Multimedia, vol. 9, no. 2, pp , Feb [24] J. Chen, J. Zheng, and Y. He, Macroblock-level adaptive frequency weighting for perceptual video coding, IEEE Tran. Conumer Electron., vol. 53, no. 2, pp , May [25] J. Lee, Rate-ditortion optimization of parameterized quantization matrix for MPEG-2 encoding, in Proc. ICIP, Oct. 1998, pp [26] Y. F. Ma and H. J. Zhang, A model of motion attention for video kimming, in Proc. ICIP, vol , pp. I [27] C. W. Ngo, T. C. Pong, and H. J. Zhang, Motion analyi and egmentation through patio-temporal lice proceing, IEEE Tran. Image Proce., vol. 12, no. 3, pp , Mar [28] H. B. Yin, Adaptive quanitization in perceptual MPEG video encoder, in Proc. PCS, Apr. 2006, pp [29] R. A. Renink, A model of aliency-baed viual attention for rapid cene analyi, in Proc. ACM 2nd Int. Symp. Smart Graphic, 2002, pp [30] A. B. Waton, G. Y. Yang, J. A. Solomon, and J. Villaenor, Viibility of wavelet quantization noie, IEEE Tran. Image Proce., vol. 6, no. 8, pp , Aug [31] A. B. Waton, J. Hu, and J. F. McGowan, III, DVQ: A digital video quality metric baed on human viion, J. Electron. Imag., vol. 10, no. 1, pp , Aug [32] R. A. Renink, J. K. O. Regan, and J. J. Clark, To ee or not to ee: The need for attention to perceive change in cene, Pychol. Sci., vol. 8, pp , Sep [33] B. Girod, What wrong with mean-quare error? in Digital Image Human Viion, A. B. Waton, Ed. Cambridge, MA: MIT Pre, [34] H. Schwarz, D. Marpe, and T. Wiegand, Overview of the calable video coding extenion of the H.264/AVC tandard, IEEE Tran. Circuit Syt. Video Technol., vol. 17, no. 9, pp , Sep [35] H. Schwarz, D. Marpe, and T. Wiegand, Analyi of hierarchical B picture and MCTF, in Proc. ICME, Jul. 2006, pp [36] I. Amonou, N. Camma, S. Kervadec, and S. Pateux, Optimized rateditortion extraction with quality layer in the calable extenion of H.264/AVC, IEEE Tran. Circuit Syt. Video Technol., vol. 17, no. 9, pp , Sep [37] L. Itti, Quantifying the contribution of low-level aliency human eye movement in dynamic cene, Vi. Cognit., vol. 12, no. 6, pp , Aug [38] L. Itti, C. Koch, and E. Niebur, A model of aliency-baed viual attention for rapid cene analyi, IEEE Tran. Patt. Anal. Mach. Intell., vol. 20, no. 11, pp , Nov [39] W. S. Geiler and J. S. Perry, A real-time foveated multireolution ytem for low bandwidth video communication, Proc. SPIE, vol. 3299, pp , Jul [40] J. G. Robon and N. Graham, Probability ummation and regional variation in contrat enitivity acro the viual field, Vi. Re., vol. 21, no. 3, pp , [41] M. S. Bank, A. B. Sekuler, and S. J. Anderon, Peripheral patial viion: Limited impoed by optic, photoreceptor and receptor pooling, J. Opt. Soc. Am., vol. 8, pp , Nov [42] Final Report from the Video Quality Expert Group on the Validation of Objective Quality Metric for Video Quality Aement. (2008) [Online]. Available: [43] ITU-T, Subjective Video Quality Aement Method for Multimedia Application, document Rec. ITU-T P.910, [44] Demo Sequence. (2010). Perceptually Scalable Extenion of H.264 [Online]. Available: ftp://wirele.yonei.ac.kr Hojin Ha received the B.S. degree from Myongji Univerity, Yongin, Korea, in 1998, the M.S. degree from Hanyang Univerity, Anan, Korea, in 2000, both in control and intrumentation engineering, and the Ph.D. degree in electrical and electronic engineering from Yonei Univerity, Seoul, Korea, in Since 2000, he ha been a Reearch Engineer with Digital Media and Communication (DMC) Reearch and Development Center, Samung Electronic, Suwon, Korea. Hi current reearch interet include multimedia communication, multimedia ignal proceing, peer-topeer networking, and hypertext tranfer protocol adaptive treaming. Jincheol Park wa born in Korea in He received the B.S. degree in information and electronic engineering from Soongil Univerity, Seoul, Korea, in 2006, and the M.S. degree in electrical and electronic engineering from Yonei Univerity, Seoul, in Since 2008, he i puruing the Ph.D. degree from the Wirele Network Laboratory, Yonei Univerity. From June 2010 to June 2011, he wa a Viiting Reearcher in the Laboratory for Image and Video Engineering with Prof. A. C. Bovik at the Univerity of Texa at Autin, Autin. Hi current reearch interet include wirele multimedia communication and video quality aement.

12 1678 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 11, NOVEMBER 2011 Sanghoon Lee (M 05) wa born in Korea in He received the B.S. degree in electrical engineering from Yonei Univerity, Seoul, Korea, in 1989, the M.S. degree in electrical engineering from the Korea Advanced Intitute of Science and Technology, Daejeon, Korea, in 1991, and the Ph.D. degree in electrical engineering from the Univerity of Texa, Autin, in From 1991 to 1996, he wa with Korea Telecom, Seongnam, Korea. From June 1999 to Augut 1999, he wa with Bell Lab, Lucent Technologie, Seoul, working on wirele multimedia communication. From February 2000 to December 2002, he worked on developing real-time embedded oftware and communication protocol for 3G wirele network, Lucent Technologie. In March 2003, he joined the faculty of the Department of Electrical and Electronic Engineering, Yonei Univerity, where he i currently an Aociate Profeor. Hi current reearch interet include image/video quality aement, wirele multimedia communication, multihop enor network, and 4G wirele network. Dr. Lee i an Aociate Editor of the IEEE Tranaction on Image Proceing and an Editor of the Journal of Communication and Network. Currently, he i the Chair of the IEEE Standard Working Group for 3-D quality aement. Alan Conrad Bovik (S 80 M 81 SM 89 F 96) i the Curry/Cullen Trut Endowed Chair Profeor with the Univerity of Texa at Autin, Autin, where he i alo the Director of the Laboratory for Image and Video Engineering. He i a Faculty Member with the Department of Electrical and Computer Engineering and the Center for Perceptual Sytem in the Intitute for Neurocience. Hi current reearch interet include image and video proceing, computational viion, and viual perception. He ha publihed over 500 technical article in thee area and hold two U.S. patent. Hi everal book include the mot recent companion volume, The Eential Guide to Image and Video Proceing (New York: Academic, 2009). He i a regitered Profeional Engineer in the State of Texa and i a frequent conultant to legal, indutrial, and academic intitution. He wa named the SPIE/IS&T Imaging Scientit of the Year for He ha alo received a number of major award from the IEEE Signal Proceing Society, including the Bet Paper Award in 2009, the Education Award in 2007, the Technical Achievement Award in 2005, and the Meritoriou Service Award in He received the Hocott Award for Ditinguihed Engineering Reearch at the Univerity of Texa at Autin, the Ditinguihed Alumni Award from the Univerity of Illinoi at Urbana-Champaign, Urbana, in 2008, the IEEE Third Millennium Medal in 2000, and two journal paper award from the International Pattern Recognition Society in 1988 and He i a Fellow of the Optical Society of America, a Fellow of the Society of Photo- Optical and Intrumentation Engineer, and a Fellow of the American Intitute of Medical and Biomedical Engineering. He ha been involved in numerou profeional ociety activitie, including the Board of Governor of the IEEE Signal Proceing Society from 1996 to 1998, the Co-Founder and Editorin-Chief of the IEEE Tranaction on Image Proceing from 1996 to 2002, an Editorial Board Member of the Proceeding of IEEE from 1998 to 2004, the Serie Editor for Image, Video, and Multimedia Proceing (San Mateo, CA: Morgan and Claypool) from 2003 to preent, and the Founding General Chairman, 1t IEEE International Conference on Image Proceing, held in Autin, TX, in November 1994.