An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2

Size: px

Start display at page:

Download "An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2"

Myron Shaw
5 years ago
Views:

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University,

1 An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University, China (* Corresonding Author) 2 Research Deartment of Hisilicon Semiconductor and Comonent Business Deartment, Huawei Technologies, China Abstract Region-of-Interest () location in videos has many ractical usages in video coding field, such as video content analysis and user exerience imrovement. Although -based coding has been studied widely by many researchers to imrove coding efficiency for video contents, the location itself is seldom coded in video. In this aer, we will introduce our roosed location coding tool which has been adoted in surveillance rofile of AVS2 video coding standard (surveillance rofile). Our tool includes three schemes: direct-coding scheme, differential- coding scheme, and reconstructed-coding scheme. We will illustrate the details of these schemes, and erform analysis of their advantages and disadvantages, resectively. I. Introduction and Related Work Region-of-Interest () location for video sequence is of increasing imortance in many alications. For examle, location can hel decide quantization arameters to enhance the resolution of the interest regions. Secifically, in video conference alications, identified s can be used to guarantee the video qualities of the interested region in order to imroving the user exeriences. Various -based coding methods have been develoed. Chai. D et al. [1] extract face region using skin-color ma to obtain location. Chen et al. [2] use robust skin-color detection to obtain osition for the coding. Menser et al. [3] analyze the video contents by face detection and tracking technology, and then aly them to the alication of coding. Hu et al. [4] categories macroblocks inside one into three different region tyes and then erform rate control by allocating more bits to those region tyes that more interesting. However, these methods all treat location as a reference to decide coding arameters, e.g. quantization arameter, for macroblocks, rather than a art of video contents which can be encoded into the. In our analysis, location contains lots of contents relevant to the corresonding video. It can be used as a efficient way that describes the video contents characteristic. Moreover, location records trajectory of the target objects, which is esecially usefully in surveillance videos. Therefore, it is reasonable to encode location into the video. In the following arts of this aer, we will describe our roosed location coding tool which has been adoted in surveillance rofile of AVS2 video coding standard. We will illustrate three coding schemes, which are direct-coding scheme, temoral differential-coding scheme, and reconstructed- coding scheme. Besides, the analysis and comarison of these schemes will also be resented. The rest of the aer is organized as follows: Section II describes the details of the three schemes in our coding method. Section III shows the exeriment results and analyzes the coding erformances of these schemes. The conclusions are drawn in Section IV. II. Details of Our Coding Tool A. Definitions In our design, we assume s in videos can be bounded by rectangles and the locations of s can be described and coded by the corner oints of these bounding rectangles. According to this assumtion, we define location as: R i = { label i, xi, yi, wi, hi } (1) where R i indicates the i-th in one, label i is the index of R i indicating the s with the same roerties, e.g. the same object: a erson, a car, etc., in the whole video sequences, x i and y i are the horizontal and vertical coordinates of the to-left corner ixel of R i, and w i and h i are the width and height of the R i, resectively. With these five arameters, we can uniquely determine the location and the roerty of one in a. Fig. 1 illustrates the arameters in Eqn. (1). Fig. 1 An examle of s and their arameters in one Then, we define the differentiation of two corresonding s as: D = R R' = { x x', y y', w w', h h'} (2) where R and R are two corresonding s in adjacent encoded s. It will be alied in differential-coding scheme and extension scheme. Before describing the schemes in detail, three imortant oints need to be mentioned: 1) s in videos can either be selected manually or be detected by some state-of-art detection techniques [5]. 2) The order of s in a is ordered by their labels indices. In the roosed method, we allocate label indices according to the order of their aearing time. 3) The location will be added to head of each 's. In the following, we will describe the three roosed schemes, resectively.

2 B. Direct-Coding Scheme In the direct-coding scheme, we encode the raw location directly without any comression. During the rocedure, we rocess the in each indeently, i.e. we can recover the location in one without any additional of other s. More secifically, we first encode the label index difference between the current and the revious one in a (if it is the first in the sequence in one, the difference will be the same as the label number -1). Then, we encode the four arameters of an (Eqn. (1)) directly into the. We encode the s by the above method until the of the sequence in one. And the above rocess will be reeated for each. The detailed rocess of the direct-coding scheme is shown by Algorithm 1 and Fig. 2 shows an examle of using the direct-coding scheme to encode s in a. Algorithm 1: Direct-coding scheme Inut: location in one (assume that there are totally n s) Outut: Bitstream containing encode n into the ; for i = 1 to n R i = the i-th in the sequence, with arameters {label i, x i, y i, w i, h i }; if (i == 1) d i = label i 1; /*label index difference between the current */ /*and the revious in the current */ d i = label i label i-1 1; encode d i, x i, y i, w i, h i into the sequentially; We can see that this scheme is simle and easy to oerate. It is suitable to encode the location when a few of s need to be encoded in the sequence er. 3 0 (,,,) 1 (199,399,200,) 0 (152,149,150,150) Fig. 2 An coding examle of using the direct-coding scheme: assume that the label of R i is Label i=i. C. Differential-Coding Scheme Although direct-coding scheme can encode location into the, when large quantities of s are required to be encoded, the of the location will become huge and non-negligible, and will add a burden to the video. Therefore, we develoed another scheme, named as differential-coding scheme, to further imrove the coding efficiency. In this scheme, we encode the location with the temoral differential coding idea to reduce the coding redundancy. Since temorally neighboring s has high correlation, by encoding the temoral difference of the corresonding location arameters of one, redundancies can be reduced. The detailed rocess of the differential-coding scheme is shown in Algorithm 2. Algorithm 2: Differential-coding Scheme Inut: location in one (assume n s totally) and the in the revious encoded (assume totally m s) Outut: Bitstream containing if (current is intra-coded ) aly direct-coding scheme /* aly differential-coding scheme */ /* encode old s (normal + disaear) */ j = 1; ski = 0; set flag buffer to null; /* Store flags */ encode n into the ; for i = 1 to m R i = the i-th in the sequence in the revious encoded, with arameters {label i, x i, y i, w i, h i}; R j = the j-th in the sequence in the current, with arameters {label j, x j, y j, w j, h j }; if (label j > label i) /* R j disaears */ ush back flag(0) into flag buffer; /* label j ==label i */ D = (x j, y j, w j, h j ) (x j, y j, w j, h j) if (D == (0,0,0,0)) /* R j has same location */ ush back flag(1) into flag buffer; j = j+1; encode ski, flag buffer, D into the sequentially; j = j+1; ski = 0; /* initialize the arameter */ set flag buffer to null; /* initialize the arameter */ /* Before old s, write remains in the flag buffer */ encode ski, flag buffer into the sequentially; /* encode aeared s */ for i = j to n R i = the i-th in the sequence in the current ; if (i == j) encode x i, y i, w i, h i into the sequentially; d i = label i label i-1 1; encode d i, x i, y i, w i, h i into the sequentially; Several things need to be mentioned about the differential-coding scheme: 1) Since there is no reference for the first in a GOP, we aly direct-coding scheme for the first. 2) We define three tyes of s: the which exists in both the revious encoded and the current (normal ); the which exists in the revious encoded but disaear in the current (disaeared ); and the which newly aear in the current (aeared ). Some examles are shown in Fig. 3. 3) We aly different strategies on different tyes. For a normal which has the same location arameters as its corresonding one in the revious (i.e. D in Eqn. (2) is {0, 0, 0, 0}), we allocate a ski flag to indicate such condition instead of encoding four zeros. For other normal s, we encode the four differential numbers in Eqn. (2). Furthermore, for a disaearing, we allocate a disaear

3 flag to indicate such condition. Similarly, for an aeared, we directly encode the four arameters of the s (i.e. x, y, w, h), since no corresonding in the revious encoded can be used as a reference. 4) In our scheme, aearing s are always allocated with a new label and coded at the of the. Fig. 3 shows an examle of using the differential- coding scheme to encode s in a. Previous encoded (,) 3 Total number of s in the current R 1 (200,400) (200,200) R R 3 R2* disaears in the next Ski: the number of skied s (R 1 and R 2) before the next differential coded (R 3) (-1,-1,0,0) Flag buffer: 1: R 1 has the same location with R 1 ; 0: R 2 disaear Current encoded (,) (152,149) R 1 (199,399) R 3 location arameters (osition difference between R 3 and R 3 ) R Ski 200 R 3 R4 aears newly in the current (152,149,150,150) R 4 location arameters (absolute osition of R 4) Fig. 3 An coding examle of using the differential-coding scheme: assume that the label of R i is Label i=i. the reconstructed (i.e., the reconstructed from the current encoded ). Then, the actual locations will be differentiated with these redicted locations and the location difference will be encoded and transmitted. At the decoder side, after reconstruction of the current, the same object detection algorithms are used to derive redicted regions. These redicted locations will be combined with the decoded location differences to recover the actual locations. Since the number of redicted s may not be the same as the actual ones, in order to recisely recover regions, we assign actual with its closest redicted and encode their differences. Furthermore, considering that multile actual s may corresond to one redicted, we also encode the number of the actual s in the. That is, we will encode in the of each redicted art the number of actual s associated with it, followed by the label number and location differences of each actual. A coding examle of reconstructed-coding scheme is shown in Fig. 5, and the above oint will be illustrated visually in the meantime. Algorithm 3 shows the detailed rocess of the reconstructed-coding scheme. D. An Exted Scheme: Reconstructed-Coding Scheme In the differential-coding scheme, we only use the temoral correlation among s for reducing coding redundancy. In ractice, since many s in a can be estimated by object detection and recognition algorithms [5], the location can be further comressed by utilizing the rediction of these algorithms. Therefore, we also roose an exted reconstructedcoding scheme to further reduce coding redundancy. Please notice that this scheme has not been included in coding method in AVS2 standard. The work of the reconstructed-coding scheme is described by Fig. 4. More secifically, at the side, after then encoding of the current, we aly object detection algorithms to derive redicted regions from (52,49,50, 50) (-1,-1,0,0) Fig. 5 An coding examle of using the reconstructed-coding scheme: assume that the label of Label i=i. Inut video of video contents of video constents reconstructed s in current encoded location s in reconstructed reconstructed of outut of location decoder s in current encoded s in reconstructed Fig. 4 The work of Reconstructed-Coding Scheme decoder

Algorithm 3: Reconstructed-coding scheme Inut: location in one (assume n s totally) and the in the reconstructed (assume m s totally) Outut: Bitstream containing

h r i}; k = the number of actual s in the current associated with R r i; encode k and all the labels of the actual s associated with R r i into the sequentially;

if (D == (0,0,0,0)) /* R l and R r i have the same location */ encode ski, D into the (in order); ski = 0; /* initialize the arameter */ E.

length coding (FLC) label values and absolute osition of s. The coded is added in front of the s of a. Fig.

We test the coding erformance on AVS2 surveillance rofile s common test sequences where s are re-defined for each sequence.

Some tested video s and the s are illustrated in Fig. 7.

arameters (QPs). The exeriment results are shown in Table 1.

From Table 1, we can see that if we use the direct-coding scheme, the bit rate is high due to the large number of bits in coding locations.

Considering that large number may exist frequently in many alications such as surveillance and video conference, it is desirable to reduce location signaling bits.

Comared to the direct coding method, the differential method can achieve 2.5% bitrate reduction on average, and u to 6% bitrate reduction. Fig.

Exerimental Results In this section, we show exeriments results of the three coding schemes described in Section II. In the exeriments, we use AVS2 RD6.

In the second exeriment, we further erform exeriments of the exted reconstructed-coding scheme. The number of encoded s for each video sequence is s.

In order to ease comarison, we also include the corresonding results of the direct-coding and the differential-coding schemes in Table 2.

4 Algorithm 3: Reconstructed-coding scheme Inut: location in one (assume n s totally) and the in the reconstructed (assume m s totally) Outut: Bitstream containing encode m into the ; ski = 0; for i = 1 to m (the number of redicted s) R r i = the i-th redicted in the reconstructed, with arameters {label r i, x r i, y r i, w r i, h r i}; k = the number of actual s in the current associated with R r i; encode k and all the labels of the actual s associated with R r i into the sequentially; foreach actual s associated with R r i (R l, l=1,2,,k, with arameters {label r i, x r i, y r i, w r i, h r i}) D = (x l, y l, w l, h l ) (x r i, y r i, w r i, h r i); if (D == (0,0,0,0)) /* R l and R r i have the same location */ encode ski, D into the (in order); ski = 0; /* initialize the arameter */ E. Syntax in the Bitstream In our tool, we use binary (0 and 1) to encode flags, use universal variable length coding (UVLC) [6] to encode difference, and use fixed length coding (FLC) label values and absolute osition of s. The coded is added in front of the s of a. Fig. 6 shows an examle of the created by our differential-coding scheme. We test the coding erformance on AVS2 surveillance rofile s common test sequences where s are re-defined for each sequence. Besides, the object detection algorithm in [5] is utilized to achieve redicted regions in the reconstructed-coding scheme. Some tested video s and the s are illustrated in Fig. 7. In the first exeriment, we erform tests of the direct-coding scheme and the differential-coding scheme with different video sequences under various quantization arameters (QPs). The exeriment results are shown in Table 1. Since fix quantization arameter is used in each exeriment, the PSNR values are the same for different methods and thus only bit rates are listed in Table 1. From Table 1, we can see that if we use the direct-coding scheme, the bit rate is high due to the large number of bits in coding locations. The overhead becomes obvious when the number of s is large. Considering that large number may exist frequently in many alications such as surveillance and video conference, it is desirable to reduce location signaling bits. Comaratively, the bit rates by the differential method are substantially reduced by utilizing the temoral correlation among s. Comared to the direct coding method, the differential method can achieve 2.5% bitrate reduction on average, and u to 6% bitrate reduction. Fig. 8 Some examle video s and the s in the second exeriment (-1,-1,0,0) 0 (152,149,150,150) Fig. 6 An examle of the for differential-coding scheme III. Exerimental Results In this section, we show exeriments results of the three coding schemes described in Section II. In the exeriments, we use AVS2 RD6.1 as the test latform [7]. The icture coding structure is Hierarchical-B (one P with seven B). The details of test conditions can be found at [9]. In the second exeriment, we further erform exeriments of the exted reconstructed-coding scheme. The number of encoded s for each video sequence is s. Table 2 shows the results of the reconstructed-coding scheme on some video sequences as in Fig. 8. In order to ease comarison, we also include the corresonding results of the direct-coding and the differential-coding schemes in Table 2. From Table 2, we can see that the reconstructedcoding scheme can also reduce bit rate by utilizing the redicted regions. Moreover, comaring the reconstructed-coding and differential-coding schemes, we can see that both schemes have their own advantages. If the object detection algorithm is accurate and the redicted s are close to their corresonding actual s, the reconstructed-coding scheme can achieve better results than the differential-coding scheme. Otherwise, the differential-coding scheme creates better erformance. Furthermore, we can also see that when the QP value increases, the erformance of the reconstructed-coding schemes will be decreased since the qualities of the reconstructed s decrease with larger QPs. However, we believe that the effect from QP arameters can be further reduced by utilizing more sohisticated detection techniques [8]. IV. Conclusion Fig. 7 Some examle video s and the s of our exerimental videos (standard AVS video) In this aer, we introduce a coding tool for encoding locations in AVS2 standard. The coding tool has three

5 Table 1 Exeriments on the schemes in the coding tool seq. encoded Bit Rate 30Hz resolution / QP s Original Direct-coding Differential-coding (+1.18%) (+0.25%) BQquare x240 2~ (+2.46%) (+0.52%) (+4.96%) (+1.05%) (+10.11%) (+2.14%) (+0.05%) (+0.01%) Traffic x ~ (+0.08%) (+0.01%) (+0.15%) (+0.02%) (+0.35%) (+0.06%) (+1.62%) (+0.30%) Crossroad x576 30~ (+3.47%) (+0.65%) (+7.93%) (+1.49%) (+17.47%) (+3.28%) (+0.90%) (+0.22%) Office x576 7~ (+1.77%) (+0.43%) (+3.49%) (+0.85%) (+6.92%) (+1.69%) (+0.59%) (+0.15%) Intersection x ~ (+1.12%) (+0.28%) (+2.19%) (+0.56%) (+4.52%) (+1.15%) Table 2 Exeriments on three otional schemes (We encode s for each video sequence) seq. QP resolution / Bit Rate 30Hz Original Direct-coding Differential-coding Reconstructed-coding I ~ (+0.65%) (+0.29%) (+0.25%) I ~ (+4.61%) (+1.93%) (+1.85%) I ~ (+12.82%) 38.16(+6.63%) 38.37(+7.22%) II ~ (+0.76%) (+0.47%) (+0.52%) II ~ (+6.27%) (+2.69%) (+2.75%) II ~ (+13.88%) 40.72(+8.25%) 40.89(+8.68%) III ~ (+6.34%) (+2.54%) (+2.38%) IV ~ (+14.38%) (+7.17%) (+7.62%) V ~ (+26.12%) (+12.26%) (+11.89%) VI ~ (+24.76%) (+10.47%) (+11.21%) different coding schemes which encode locations directly, by using temoral, and by using redicted from object detection algorithms. Exerimental results shows that utilizing temoral and rediction can effectively reduced the overhead for encoding regions. Further work will include combining the schemes of differential-coding and reconstructed-coding such that the coding erformance can be further imroved. Acknowledgements This aer is suorted in art by the following grants: National Science Foundation of China (No , and 61102), the Chinese national 973 roject (2010CB731401), the SMC grant of SJTU, and Shanghai Pujiang Program (12PJ ). References [1] D. Chai, K.N. Ngan, "Face segmentation using skin-color ma in videohone alications," IEEE Trans. Circuits and Systems for Video Technology, vol.9, no.4, , 1999 [2] M.-J.. Chen, M.-C. Chi, C.-T. Hsu and J.-W. Chen, Video Coding Based on H.263+ with Robust Skin-color Detection Technique, IEEE Trans. Consumer Electronics, vol. 49, , [3] B. Menser and M. Brunig, Face Detection and Tracking for Video Coding Alications, IEEE Int l Conf. Signal, Systems and Comuters, vol. 1, , [4] H.-M. Hu, B. Li, W. Lin, W. Li and M.-T. Sun, Region-Based Rate Control for H.264/AVC for Low Bit-Rate Alications, IEEE Trans. Circuits and Systems for Video Technology, vol. 22, , [5] P. Viola and M. Jones, Robust Real-Time Object Detection, Int l J. Comuter Vision, vol. 57, no. 2, , 2004 [6] Y. Itoh, N. Cheung, Universal variable length code for DCT coding IEEE Int l Conf. Image Processing, vol. 1, , 2000 [7] RD6.1,ft:// /incoming/video_codec/AVS2_P2/RD6.1.zi [8] J. Wu, J. M. Rehg, CENTRIST: A Visual Descritor for Scene Categorization IEEE Trans. Pattern Analysis and Machine Intelligence, vol.33, , [9] AVS-N2021, Common test conditions on AVS2-P2 surveillance rofile, December, 2012, Shenzhen, 47 th AVS meeting

Improved Image Super-Resolution by Support Vector Regression

Improved Image Super-Resolution by Support Vector Regression Proceedings of International Joint Conference on Neural Networks, San Jose, California, USA, July 3 August 5, 0 Imroved Image Suer-Resolution by Suort Vector Regression Le An and Bir Bhanu Abstract Suort