REGION-BASED SUPER-RESOLUTION FOR COMPRESSION

Transcription

1 REGION-BASED SUPER-RESOLUTION FOR COMPRESSION D. Barreto 1, L.D. Alvarez 2, R. Molina 2, A.K. Katsaggelos 3 and G.M. Callicó 1 INTERNATIONAL CONFERENCE ON SUPERRESOLUTION IMAGING Theory, Algorithms and Applications August 29-31, 2005 Department of Mathematics, University of Hong Kong 1 Research Institute for Applied Microelectronics, IUMA, University of Las Palmas de Gran Canaria 35017, Spain, dbarreto@iuma.ulpgc.es, gustavo@iuma.ulpgc.es. 2 Department of Computer Science and Artificial Intelligence, University of Granada Granada 18071, Spain, ldac@decsai.ugr.es, rms@decsai.ugr.es. 3 Department of Electrical and Computer Engineering, Northwestern University Sheridan Road, Evanston, IL, ,USA, aggk@ece.nwu.edu.

2 OUTLINE Introduction Problem description System scheme Texture Motion Segmentation Down-sampling patterns Block Matching Motion Estimation SR for Region 1 SR for Region 2 SR for Region 3 Experimental results Conclusions Future work

3 Introduction Super-Resolution (SR) attempts to reconstruct a high resolution (HR) image or video sequence from a set of low resolution (LR) images. Image Sequence Sub-pixel motion LR Static Super-Resolution Images from the same scene Dynamic Video sequence Sub-pixel motion HR

4 One of the most important applications nowadays is video transmission. Before transmission original video sequences are compressed. We propose to down-sample the sequence before compression, especially at low bit-rates. SR can then be used to visualize the video sequence with enough quality at reception. Encoder HR sequence LR sequence Transmission Decoder SR LR decoded sequence SR sequence

5 Problem Description Our purpose is to recover, at reception, the original HR image (sequence). On moving regions, our SR technique will depend on the correctness of the Motion Estimation (ME). On still regions, our SR technique will depend on correct classification between flat and textured areas.

6 System Scheme Seeing the general scheme, we will divide it into two parts: the transmitter and the receiver. Encoder HR sequence LR sequence Transmission Decoder LR decoded sequence SR sequence

7 scheme: D Motion Map LR sequence HR sequence T T Texture-Plain Map TM-map T D TRANSMISSION Threshold selection Motion No motion & Textured No motion & Flat

8 From the HR sequence we obtain two binary maps (block level): Motion/No Motion region: Map with information about the motion within each image. One threshold is needed. Textured/Plain region: Map with information about the texture within each image. One threshold is needed. Using them a new map is generated (TM-map): motion-flat (no motion)-texture (no motion). Using the TM-maps, the HR sequence is down-sampled to obtain the LR sequence. TM-maps and LR sequences are sent to the.

9 scheme: LR decoded sequence TRANSMISSION TM-map extended SR process Motion vectors SR sequence At the receiver once the LR sequence and the TM-map are received, the motion vectors are calculated for the moving regions, and then our SR process is applied. The main novelty is the use of the TM-map.

10 Texture/Motion Segmentation Every image in the sequence will be divided into blocks and they will be labeled according to motion and texture. Areas with motion Region 1 Relevant information from adjacent frames can be used in the SR process. Flat areas with no motion Region 2 The spatial resolution will be increased by interpolation. Textured areas with no motion Region 3 The down-sampling pattern will change from frame to frame in order to have more image information at the receiver. For every frame in the sequence to be super-resolved, new information from three adjacent frames is used.

11 Texture Segmentation: for the texture map, a measure that provides information of the variation of every block is calculated (Intra_SAD). Intra _ SAD = m n i= 1 j= 1 fk ( i, j) μ, m n where μ is the mean value for a block, f the HR image and m x n is the number of blocks. If Intra_SAD Threshold there is texture. Texture No texture

12 Motion Segmentation: the motion map for every frame is based on the difference between such frame and the previous one at block level. Mean _ Diff = Mean f k ( i, j) f k 1( i, j) 1... i= m, j= 1... n where f represents a HR image and m x n is the number of blocks. If Mean_Diff Threshold there is motion. No motion Motion

13 TM-map: Texture-Motion map is calculated using the two binary maps. No motion and texture REGION 3 No motion and flat REGION 2 Motion REGION 1 Texture-Motion map for children2 sequence, frame 4. Motion map Texture map

14 Thresholding is a key point for the method. Depending on the selected values, the improvement can change. Two thresholds are required: Motion/No Motion threshold : to detect motion within the image. Texture/Flat : to discriminate between regions. Thresholds are calculated automatically. Empirical values: Motion threshold = 3*(σ/μ) = variation coefficient σ standar deviation (block based) μ mean (block based) Textured/Flat: threshold = 3

15 Example: No motion and texture Motion No motion and flat Texture-Motion map for sequence interv, frame 4 Motion map Texture map

16 Example: No motion and texture No motion and flat Motion Texture-Motion map for deadline sequence, frame 4 Motion map Texture map

17 Down-sampling pattern The down-sampling pattern is different depending on the type of block. Region 1 (Motion) and Region 2 (Flat and no motion) HR Blocks LR Blocks Region 3 (Texture and no motion) Four frames to be combined if down-sampling factor = 2. The down-sampling pattern changes between frames to reconstruct the HR block perfectly during SR. HR Blocks LR Blocks

18 The segmentation is performed over HR images at a block level, where: BlockSizeH R = downsampling _ factor BlockSizeLR To stress again that depending on the TM-map we classify each block in Region 1, Region 2 or Region 3. The downsampling pattern depends on that and in the frame order Each LR block comes from its corresponding HR block

19 Block-matching Motion Estimation Block matching (BM) divides each frame into blocks of size mxn. Searching in every pixel position of this area is named Full Search BM. Search area: a=(2p+n) 2 if m=n. BM assumes that every pixel in a block has the same MV. SAD Similarity criteria: SAD (Sum of Absolute Differences) m 1 n 1 i, j = gk ( x, y) gl ( x + i, y + j) x= 0 y= 0

20 Sub-pixel block-matching ME: 1. Calculate best motion vector at pixel precision. 2. Search for the best half-pixel motion vector at eight half-pixel positions around the point of minimum given in step Search for the best quarter-pixel motion vector at eight quarter-pixel positions around the point of minimum given in step 2. Integer pixel positions Half pixel positions Quarter pixel positions Best Integer pixel position Best half pixel position Best quarter pixel position

21 Super-Resolution: Region 1 g k-2 g k-1 g k g k+1 g k+2 f k d k,k-2 d k,k-1 d k,k d k,k+1 d k,k+2 After the estimation of the relative motion d between frames g, the samples can be projected onto a HR grid. General steps to follow: Place the current image pixels in their corresponding positions on the HR grid New positions in the HR grid will be filled with values from other frames if there is sub-pixel motion. Remaining empty positions will be filled with interpolated values

22 Region 1 SR example: HR Grid for ¼ pixel precision Image K Image K-1 m.v. (0.5,0.5) Image K-2 m.v. (0.25,0) Image k+1 m.v. (0,0.5) Image k+2 m.v. (0.75,0.25) Interpolations

23 Super-Resolution: Region 2 Region 2 corresponds with areas without motion nor texture (flat). Bilinear interpolation is enough (and simple). Region 2 SR example: HR Grid for ½ pixel precision Image K Interpolations

24 Super-Resolution: Region 3 g k-2 g k-1 g k g k+1 f k 4 different downsampling patterns (if downsampling factor is 2). General steps to follow: Identify the images that can contribute (from one to three). The region will contribute if it has been labeled as 3. Using these images the corresponding positions in the HR grid will be filled. Perfect HR reconstruction is possible (but not always).

25 Region 3 SR example: Block under study has been classified as Region 1 in image g k-2 and as Region 3 in images g k-1,g k and g k+1 g k-2 g k-1 g k g k+1 f k Image K Image K-1 HR Grid for ½ pixel precision Image K-2 Image k+1 Interpolations

26 Experimental results interview sequence PSNR bilinear : db PSNR SR : db Improvement: db Thresholds: M=4.9 T=3

27 Experimental results interview sequence Bilinear Interpolation Original Proposed Superresolution method

28 Experimental results deadline sequence PSNR bilinear : db PSNR SR : db Improvement: db Thresholds: M=3.3 T=3

29 Experimental results deadline sequence Bilinear Interpolation Original Proposed Superresolution method

30 Experimental results children2 sequence PSNR bilinear : db PSNR SR : db Improvement: db Thresholds: M=5.8 T=3

31 Experimental results children2 sequence Bilinear Interpolation Original Proposed Superresolution method

32 Experimental results children2 sequence Bilinear Interpolation Original Proposed Superresolution method

33 Experimental results children2 sequence Bilinear Interp. Proposed SR Original HR REGION 1 motion REGION 2 no motion plain = REGION 3 no motion textured =

34 Conclusions A new SR schema has been presented. A new downsampling pattern that benefits SR process has been described. Experimental results have been shown.

35 Future Work To include compression in this schema. Additional information to be sent to the receiver.