Lec 08 Video Signal Processing I

CS/EE 5590 / ENG 401 Special Topics (17804, 17815, 17803) Lec 08 Video Signal Processing I Motion Estimation and Compensation Zhu Li Course Web: http://l.web.umkc.edu/lizhu/teaching/2016sp.video-communication/main.html Z. Li Multimedia Communciation, 2016 Spring p.1

Lecture 07 Re-Cap Color Space and Sampling Outline Motion Estimation and Compensation Optical Flow (pixel based) Block Based Motion Estimation Fast Block Motion Estimation Z. Li Multimedia Communciation, 2016 Spring p.2

Scalar Quantization Uniform Quantizer & Distortions Distortion Metrics: MSE Uniform Q MSE: d i 2 xˆ x f ( x) dx y x 0 M b i1 b i1 i 2 f ( x) dx 1 1 1 1 d M x dx M 12 12 2 3 2 / 2 Z. Li Multimedia Communciation, 2016 Spring p.3

Scalar Quantization Non-Uniform Scalar Quantization Intuition: denser sampling at higher pdf region Formulation: minimize d() over {b k, y k }, take Lagrangian and via KKT condition: Z. Li Multimedia Communciation, 2016 Spring p.4 x f(x) M k b b k k k k k dx x f x y dx x f x x y b d 1 2 2 1 ) ( ) ( ˆ }), ({ 2 0 ) ( ) ( 0 1 2 1 2 i i i i i i i i i i y y b b f b y b f b y b d i i i i b b b b i i i dx x f dx x f x I X X E y y d 1 1 ) ( ) ( 0 y i is the centroid!

Vector Quantizer A more optimal solution, better approx. R-D info theoretical boundary kmeans() % desired rate R=8; [indx, vq_codebook]=kmeans(x, 2^R); kd-tree implementation [kdt.indx, kdt.leafs, kdt.mbox]=buildvisualwordlist(x, 2^R); [node, prefix_code]=searchvisualwordlist(q, kdt.indx, kdt.leafs); Z. Li Multimedia Communciation, 2016 Spring p.5

DNA Sequence Compression Check out the Matlab bioinformatics toolbox path(path, '../../teaching_video_coding/matlab'); % get dna seq data: human mitochondria genome seq1 = getgenbank('nc_012920','sequenceonly',true); seq_len = length(seq1); % plot its hist basecount(seq1) % A: 5124 % C: 5181 % G: 2169 % T: 4094 h = [5124, 5181, 2169 4094]/seq_len; fprintf('\n estimated entropy=%1.2f, total bits=%d',getentropy(h), getentropy(h)*seq_len); Z. Li Multimedia Communciation, 2016 Spring p.6

Digital Video Basics Frame 1 51 71 91 111 Neighboring frames are usually very similar: Prediction coding will be very efficient. Typical video coding methods: Motion compensated hybrid video coding: most popular approach Block-based motion estimation/compensation Block transform (DCT) Wavelet based video coding (AVC Scalable Video Coding (Shi- Tah Hsiang, Motorola Lab) Model-based video coding: (MPEG4 object based coding) Use analysis/synthesis techniques. Encode model parameters for the decoder to synthesize.

Color Space Conversion RGB to YUV conversion: Kind of backward compatibility with old BW TVs Color does not carry as much info. Z. Li Multimedia Communciation, 2016 Spring p.9

Color Space Conversion Matlab example Play YUV sequences: ffplay -s '1920x1080' -pix_fmt yuv420p -f rawvideo Beauty_1920x1080_120fps_420_8bit_YUV.yuv Access YUV sequences ffmpeg -f rawvideo -s cif -i stefan_cif.yuv -vcodec png stefan_yuv_%03d.png f0 = imread( Stefan_yuv_100.png '); Z. Li Multimedia Communciation, 2016 Spring p.10

Color Space: Down-sampling RGB components of an image have strong correlation. Can be converted to YUV space for better compression. HVS is more sensitive to the details of brightness than color. Can down-sample color components to improve compression. Luma sample Chroma sample YUV 4:4:4 No downsampling Of Chroma MPEG-1 MPEG-2 YUV 4:2:2 YUV 4:2:0 2:1 horizontal downsampling 2:1 horizontal downsampling of chroma components of chroma components 2 chroma samples for 1 chroma sample for every every 4 luma samples 4 luma samples Z. Li Multimedia Communciation, 2016 Spring p.11

YUV 4:2:0 File Format In YUV 4:2:0, number of U and V samples are 1/4 of the Y samples YUV samples are stored separately: Image: YYYY..Y UU U VV V (row by row in each channel) Video: YUV of frame 1, YUV of frame 2, CIF (Common Intermediate format): 352 x 288 pixels for Y, 176 x 144 pixels for U, V QCIF (Quarter CIF): 176 x 144 pixels for Y, 88 x 72 pixels for U, V U V Y Y: 176 x 144 U: 88 x 72 V: 88 x 72 Z. Li Multimedia Communciation, 2016 Spring p.12

Layered Structure for Video Data Video data layers: Sequence layer Group of Picture (GOP) layer Picture (frame) layer Group of Block (GOB) or slice layer Macroblock (MB) layer block layer sub-block layer Block: usually 8x8 pixels (4x4 pixels in H.264) Macroblock (MB): 16 x 16 region Includes 4 luma blocks, 2 chroma blocks. Group of Blocks (GOB) or slice: H.261: 3 rows of 11 MBs (176 x 48 pixels) o QCIF: 176 x 144 3 GOBs o CIF: 352 x 288 12 GOBs Slice: More flexible def. in H.264 Picture layer: one frame GOP layer: Group of frames Sequence: entire video sequence. One GOB 1 2 3 QCIF: 176 x 144 8x8 Y1 Y2 Y3 Y4 An MB: Cr CIF: 352 x 288 Z. Li Multimedia Communciation, 2016 Spring p.13 1 3 5 7 9 11 2 4 6 8 10 12 Cb

Key Idea in Video Coding Predictive coding: Predict each frame from the previous frame(s) and only encode the prediction error: Pred. error has smaller energy and is easier to compress Prediction can be performed at Frame level, Macroblock level, Block level, or even sub-block level. x y Previous frame Current Frame Z. Li, Multimedia Communciation, 2016 Spring p.15

r i f ' Z Y X Motion Field Image velocity of a point moving in the scene v i t Perspective projection: Motion field d v i r dt i f ' 1 f ' r i r o ro r Z ro Zv o vo r Z o o Z r v o t Scene point velocity: Image velocity: o f ' 2 r vo r Z 2 o o r vo dt dr v i i dt Z d o

Optical Flow Optical Flow: OPTICAL FLOW = apparent motion of brightness patterns Ideally a projection of 3d obj motion to 2d image plane

Lucas-Kanade OF estimator Optical Flow I(x, y, t) is the pixel value at location (x,y) on image plane at time t Brightness constancy assumption: dx dy I( x t, y t, t t) I( x, y, t) dt dt Optical Flow Constraint Equation: di dt I x dx dt I y dy dt I 0 t This leads to Lucas-Kanade algorithm (details skipped) Optimal OF [u, v] should satisfy:

Block Based Motion Estimation (ME) For each ME unit (MB or block), find the best match in the previous frame (reference frame) Upper-left corner of the block being encoded: (x0, y0) Upper-left corner of the matched block in the reference frame: (x1, y1) Motion vector (dx, dy): the offset of the two blocks: o (dx, dy) = (x1 x0, y1 y0) = (x1, y1) - (x0, y0) o (x0, y0) + (dx, dy) = (x1, y1) Motion vector need to be sent to the decoder. (x1, y1) Prediction error: e = A B. B (x0, y0) A Z. Li, Multimedia Communciation, 2016 Spring p.19

GOP, I, P, and B Frames I P P P P P P I P P P P P P GOP GOP: Group of pictures (frames). I frames (Key frames): Intra-coded frame, coded as a still image. Can be decoded directly. Used at GOP head, or at scene changes. Allow random access, improves error resilience. P frames: (Inter-coded frames) Predicated from the previous frame. GOP Z. Li, Multimedia Communciation, 2016 Spring p.20

GOP, I, P, and B Frames B frames: Bi-directional interpolated prediction frames Predicted from both the previous frame and the next frame: more flexibilities better prediction. Useful when new objects come into the scene. Before H.264, B frames are not used as reference for future frames: B frames can be coded with lower quality or can be discarded without affecting future frames. Allow temporal scalability Encoding/Decoding order: 1 4 2 3 7 5 6 1 2 3 4 5 6 7 I B B P B B P Display order: 1 2 3 4 5 6 7 Need more buffers Z. Li, Multimedia Communciation, 2016 Spring p.21

Basic Encoder Block Diagram Input frame Intra Inter DCT Pred. error Q Q -1 Entropy Coding Intra Inter Prediction Prediction MC ME I DCT Recon Pred error Memory Recon. Motion vectors Reconstructed Previous frame Motion compensation (MC): get the prediction of the current frame using MV information. Use reconstructed frame in the loop to prevent drifting. Z. Li, Multimedia Communciation, 2016 Spring p.22

Basic Decoder Block Diagram Entropy Decoding Q -1 I DCT Recon Pred error Intra Inter Prediction MC Reconstructed frame Memory Motion vectors Decoder is simpler than the encoder: No need to do motion estimation. Reconstructed Previous frame Z. Li, Multimedia Communciation, 2016 Spring p.23

Motion Estimation For the k-th MB, the prediction error with MV (dx, dy): ( k, dx, dy) x f k n, y k N 1 N 1 i0 j0 : The upper - left coordinate of ( x,y) :The (x, y)- th pixel in Frame n; N : ME block size (16 for MB level ME) f n x i, y j f x dx i, y dy j k k n1 k the k - th MB; p = 1: Sum of Absolute Difference (SAD) p = 2: Sum of Squared Difference (SSD) (-R, -R) Number of MV candidates: (2R+1) x (2R + 1) k p Objective: Find the MV that minimizes the pred error. ( dx *, *) arg min k dyk ( k, dx, dy) ( dx, dy) [ R, R] (R, R) Z. Li, Multimedia Communciation, 2016 Spring p.24

Matlab: motion_estimation.m Simple Matlab implementation Source: motion_estimation.m, getblkmotion.m Block Motion Estimation Search a neighborhood of the block location, looking for the smallest block residual: Z. Li Multimedia Communciation, 2016 Spring p.25

Distribution of SSD/SAD as function of MV It is typically center biased Implications on fast search! B. Girod, EE398B Image Com, Stanford. Z. Li Multimedia Communciation, 2016 Spring p.26

Motion Estimation Example Implemented as int pel resolution exhaustive search in motion_estimation.m Z. Li Multimedia Communciation, 2016 Spring p.27

Motion Compensated Residual Error Motion Vectors predicted residual is smaller than direct frame difference No free lunch, need to code the motion field: 64x64 2D integer Z. Li Multimedia Communciation, 2016 Spring p.28

High Motion Sequences Motion Compensation benefits larger Z. Li Multimedia Communciation, 2016 Spring p.29

Bi-Directional Motion Compensation: B frames Possible temporal prediction for B pictures: 1, 1 1 1 2 2 2 bˆ cˆ ˆ 1 1 2c2 0 : forward prediction 0, 1: backward prediction 0.5: bidirectional prediction, C2 C1 Frame k-1 b Frame k Frame k+1

B Frames Can have better coding efficiency: Average of two predictions reduces the variance. New objects can be better predicted using future frames. Homework-3: Bi-Directional Motion Estimation matlab exercise!

Sub-pixel Motion Estimation The actual motion is almost always not in integer. Optimal (dx, dy) = (3.2, 2.6) Need sub-pixel accuracy in motion estimation Benefits: Smaller prediction error better compression Cost: More complexities More bits spent on motion vectors Used in H.263 and H.264: Half-pixel accuracy 1/4-pixel accuracy Z. Li, Multimedia Communciation, 2016 Spring p.32

Sub-pixel Motion Estimation Need to interpolate the reference frame: Find the best integer MV first, then find the best sub-pixel MV around the best integer MV. Bilinear interpolation: A e B Original samples: A, B, C, D Half-pixel locations: e, f, g, h, x g x h e A B / 2 0.5 C f D f g C D / 2 ( A C ) / 2 0.5 0.5 Longer filters used in H.264. h x ( B D) / 2 A B C 0.5 D / 4 0.5 Z. Li, Multimedia Communciation, 2016 Spring p.33

H.264 Sub-pixel Motion Estimation Six-tap filter for ½-pixel samples: h = [ (E 5F + 20G + 20H 5 I + J) + 16 ] / 32 Similar operation for vi in vertical direction. x = [ (v1 5 v2 + 20 v3 + 20 v4 5 v5 + v6) + 16 ] / 32 E F G h H I J v1 v2 v3 x v4 v5 v6 Bi-linear filter for 1/4-pixel samples

Bi-Linear Interpolation For fractional position pixel value: Matlab: im= bilinearinterpolation(im, 2*size(im)); HW-3: ½ pel motion estimation with bilinear filtering Z. Li Multimedia Communciation, 2016 Spring p.35

Complexity of Motion Estimation Complexity of Motion Estimation is the major portion of video codec compression pipeline If exhaustive search, need prohibitive complexity to complete. For each block, search (2*range+1 x 2*range+1) pixel locations, compute SSD/SAD. If fractional pel motion estimation, need to interpolate images, double the size of complexity for ½ pel, and quadruple the complexity for ¼ pel motion estimation Z. Li Multimedia Communciation, 2016 Spring p.37

Fast Motion Estimation ( k, dx, dy) N 1 N 1 i0 j0 f n x i, y j f x dx i, y dy j k k n1 k k p Full-search motion estimation is time consuming: Each (dx, dy) candidate: N 2 summations If search window size is (2R+1) 2, need (2R+1) 2 x N 2 operations / MB o R=15, N=16: 246016 operations / MB! Fast motion estimation: Use smart search order.(3 Step, Diamond Search) Reduce the number of searched MV candidates. Apply early termination techniques (partial SSD ) Z. Li, Multimedia Communciation, 2016 Spring p.38

SSD Surface for Edge large gradients, all the same large l 1, small l 2 Z. Li Multimedia Communciation, 2016 Spring p.39

SSD Surface for low texture region gradients have small magnitude small l 1, small l 2 Z. Li Multimedia Communciation, 2016 Spring p.40

SSD Surface for High textured region gradients are different, large magnitudes large l 1, large l 2 Z. Li Multimedia Communciation, 2016 Spring p.41

The 3 Step Search ME 3 Step Search Search a 9 point displacement Compute the min SSD/SAD Shrink the size of the search window as it moves to the 2 nd stage Stop at the last search Significant reduction of complexity worst: 27 SSD/SAD computing -7-6 -5-4 -3-2 -1 0 1 2 3 4 5 6 7 Z. Li Multimedia Communciation, 2016 Spring p.42

NTSS(new 3-step search) algorithm NTSS algorithm summary: Decision 1: Min SSD at the center? Decision 2: Min SSD at one of the neighbor of the center? -7-6 -5-4 -3-2 -1 0 1 2 3 4 5 6 7 1st step of NTSS 17 checking points T MV=0 F Decision 1 Decision 2 T 2nd step of NTSS 3 or 5 checking points F 2nd and 3rd step of NTSS (same as in TSS) Z. Li Multimedia Communciation, 2016 Spring p.43

Diamond Search Algorithm To employ two search patterns Large Diamond Search Pattern (LDSP) Small Diamond Search Pattern (SDSP) Z. Li Multimedia Communciation, 2016 Spring p.44

Step 1 The initial LDSP is centered at the origin of the search window, and the 9 checking points of LDSP are tested. If the minimum block distortion (MBD) point calculated is located at the center position, go to Step3, Perform Small DSP search Else,, go to Step 2, perform Large DSP Search Z. Li Multimedia Communciation, 2016 Spring p.45

Diamond Patter Search Summary of Fast Search Ref: [11] Renxiang Li, Bing Zeng, Ming L. Liou, A new three-step search algorithm for block motion estimation, IEEE Trans. Circuits Syst. Video Tech vol.4(4): 438-442 (1994). [top 10 cited T-CSVT paper] [12] S. Zhu, K.-K. Ma, A new diamond search algorithm for fast block-matching motion estimation, IEEE Transactions on Image Processing vol.9(2): 287-290 (2000). Z. Li Multimedia Communciation, 2016 Spring p.46

Motion Compensation a main tool in video coding Pre-HEVC H.261(int pel MC) over JPEG: 50% rate reduction Fractional pel + variable block size: 100+% reduction Z. Li: Multimedia Communication, 2016 Spring p.47

Summary Video Signal is a signal over time Motion Modeling is a key compression efficiency contributor Basic Tricks Block based displacement SSD/SAD computing Fractional pel resolution to improve gains Variable block sizes to better match the object segmentation Fast Motion Estimation Avoid exhaustive search, capitalize the statistics of the SSD surface 3 Step/New 3 Step motion estimation Diamond search. Z. Li Multimedia Communciation, 2016 Spring p.48