Lec 08 Video Signal Processing I

CS/EE 5590 / ENG 401 Multimedia Communication Lec 08 Video Signal Processing I Motion Estimation and Compensation Zhu Li Course Web: http://l.web.umkc.edu/lizhu/ Z. Li Multimedia Communciation, Spring 2017 p.1

Outline Lecture 07 Re-Cap HW-1 Color Space and Sampling Motion Estimation and Compensation Optical Flow (pixel based) Block Based Motion Estimation Fast Block Motion Estimation Z. Li Multimedia Communciation, Spring 2017 p.2

Scalar Quantization Uniform Quantizer & Distortions Distortion Metrics: MSE Uniform Q MSE: d i 2 xˆ x f ( x) dx y x 0 M b i1 b i1 i 2 f ( x) dx 1 1 1 1 d M x dx M 12 12 2 3 2 /2 Z. Li Multimedia Communciation, Spring 2017 p.3

Scalar Quantization Non-Uniform Scalar Quantization Intuition: denser sampling at higher pdf region Formulation: minimize d() over {b k, y k }, take Lagrangian and via KKT condition: Z. Li Multimedia Communciation, Spring 2017 p.4 x f(x) M k b b k k k k k dx x f x y dx x f x x y b d 1 2 2 1 ) ( ) ( ˆ }), ({ 2 0 ) ( ) ( 0 1 2 1 2 i i i i i i i i i i y y b b f b y b f b y b d i i i i b b b b i i i dx x f dx x f x I X X E y y d 1 1 ) ( ) ( 0 y i is the centroid!

Vector Quantizer A more optimal solution, better approx. R-D info theoretical boundary kmeans() % desired rate R=8; [indx, vq_codebook]=kmeans(x, 2^R); kd-tree implementation [kdt.indx, kdt.leafs, kdt.mbox]=buildvisualwordlist(x, 2^R); [node, prefix_code]=searchvisualwordlist(q, kdt.indx, kdt.leafs); Z. Li Multimedia Communciation, Spring 2017 p.5

DNA Sequence Compression Check out the Matlab bioinformatics toolbox path(path, '../../teaching_video_coding/matlab'); % get dna seq data: human mitochondria genome seq1 = getgenbank('nc_012920','sequenceonly',true); seq_len = length(seq1); % plot its hist basecount(seq1) % A: 5124 % C: 5181 % G: 2169 % T: 4094 h = [5124, 5181, 2169 4094]/seq_len; fprintf('\n estimated entropy=%1.2f, total bits=%d',getentropy(h), getentropy(h)*seq_len); Z. Li Multimedia Communciation, Spring 2017 p.6

Homework-1: Arithmetic Coding Quantization/ExpGolomb Binarization Imfilter(im, f1) Res = res(: ) mean(res(: )) ResQ = fix(res/delta); Arithmetic Coding SFU code, thanks for Prof. Jie Liang! Binary Image Adaptive Arithmetic Coding Re-use HEVC BAC DNA sequence coding (bonus) Re-use HEVC BAC Will have Lab session, don t worry. Z. Li Multimedia Communciation, Spring 2017 p.7

Outline Lecture 07 Re-Cap Color Space and Sampling Motion Estimation and Compensation Optical Flow (pixel based) Block Based Motion Estimation Fast Block Motion Estimation Z. Li Multimedia Communciation, Spring 2017 p.8

Digital Video Basics Frame 1 51 71 91 111 Neighboring frames are usually very similar: Prediction coding will be very efficient. Typical video coding methods: Motion compensated hybrid video coding: most popular approach Block-based motion estimation/compensation Block transform (DCT) Wavelet based video coding (AVC Scalable Video Coding (Shi- Tah Hsiang, Motorola Lab) Model-based video coding: (MPEG4 object based coding) Use analysis/synthesis techniques. Encode model parameters for the decoder to synthesize. Z. Li Multimedia Communciation, Spring 2017 p.9

Color Space Conversion RGB to YUV conversion: Kind of backward compatibility with old BW TVs Color does not carry as much info. Z. Li Multimedia Communciation, Spring 2017 p.10

Color Space Conversion Matlab example Play YUV sequences: ffplay -s '1920x1080' -pix_fmt yuv420p -f rawvideo Beauty_1920x1080_120fps_420_8bit_YUV.yuv Access YUV sequences ffmpeg -f rawvideo -s cif -i stefan_cif.yuv -vcodec png stefan_yuv_%03d.png f0 = imread( Stefan_yuv_100.png '); Z. Li Multimedia Communciation, Spring 2017 p.11

Color Space: Down-sampling RGB components of an image have strong correlation. Can be converted to YUV space for better compression. HVS is more sensitive to the details of brightness than color. Can down-sample color components to improve compression. Luma sample Chroma sample YUV 4:4:4 No downsampling Of Chroma MPEG-1 MPEG-2 YUV 4:2:2 YUV 4:2:0 2:1 horizontal downsampling 2:1 horizontal downsampling of chroma components of chroma components 2 chroma samples for 1 chroma sample for every every 4 luma samples 4 luma samples Z. Li Multimedia Communciation, Spring 2017 p.12

YUV 4:2:0 File Format In YUV 4:2:0, number of U and V samples are 1/4 of the Y samples YUV samples are stored separately: Image: YYYY..Y UU U VV V (row by row in each channel) Video: YUV of frame 1, YUV of frame 2, CIF (Common Intermediate format): 352 x 288 pixels for Y, 176 x 144 pixels for U, V QCIF (Quarter CIF): 176 x 144 pixels for Y, 88 x 72 pixels for U, V U V Y Y: 176 x 144 U: 88 x 72 V: 88 x 72 Z. Li Multimedia Communciation, Spring 2017 p.13

Layered Structure for Video Data Video data layers: Sequence layer Group of Picture (GOP) layer Picture (frame) layer Group of Block (GOB) or slice layer Macroblock (MB) layer block layer sub-block layer Block: usually 8x8 pixels (4x4 pixels in H.264) Macroblock (MB): 16 x 16 region Includes 4 luma blocks, 2 chroma blocks. Group of Blocks (GOB) or slice: H.261: 3 rows of 11 MBs (176 x 48 pixels) o QCIF: 176 x 144 3 GOBs o CIF: 352 x 288 12 GOBs Slice: More flexible def. in H.264 Picture layer: one frame GOP layer: Group of frames Sequence: entire video sequence. One GOB 1 2 3 QCIF: 176 x 144 8x8 Y1 Y2 Y3 Y4 An MB: Cr CIF: 352 x 288 Z. Li Multimedia Communciation, Spring 2017 p.14 1 3 5 7 9 11 2 4 6 8 10 12 Cb

Key Idea in Video Coding Predictive coding: Predict each frame from the previous frame(s) and only encode the prediction error: Pred. error has smaller energy and is easier to compress Prediction can be performed at Frame level, Macroblock level, Block level, or even sub-block level. x y Previous frame Current Frame Z. Li Multimedia Communciation, Spring 2017 p.16

Motion Field Image velocity of a point moving in the scene v i t r i f ' Z Y X Perspective projection: Motion field d v i r dt i f ' 1 f ' r i r o ro r Z ro Zv o vo r Z o o Z r v o t Scene point velocity: Image velocity: o f ' 2 r vo r Z 2 o o r vo dt dr v i i dt Z d o Z. Li Multimedia Communciation, Spring 2017 p.17

Optical Flow Optical Flow: OPTICAL FLOW = apparent motion of brightness patterns Ideally a projection of 3d obj motion to 2d image plane Z. Li Multimedia Communciation, Spring 2017 p.18

Lucas-Kanade OF estimator Optical Flow I(x, y, t) is the pixel value at location (x,y) on image plane at time t Brightness constancy assumption: dx dy I( x t, y t, t t) I( x, y, t) dt dt Optical Flow Constraint Equation: di dt I x dx dt I y dy dt I t This leads to Lucas-Kanade algorithm (details skipped) Optimal OF [u, v] should satisfy: 0 Z. Li Multimedia Communciation, Spring 2017 p.19

Block Based Motion Estimation (ME) For each ME unit (MB or block), find the best match in the previous frame (reference frame) Upper-left corner of the block being encoded: (x0, y0) Upper-left corner of the matched block in the reference frame: (x1, y1) Motion vector (dx, dy): the offset of the two blocks: o (dx, dy) = (x1 x0, y1 y0) = (x1, y1) - (x0, y0) o (x0, y0) + (dx, dy) = (x1, y1) Motion vector need to be sent to the decoder. (x1, y1) Prediction error: e = A B. B (x0, y0) A Z. Li Multimedia Communciation, Spring 2017 p.20

GOP, I, P, and B Frames I P P P P P P I P P P P P P GOP GOP: Group of pictures (frames). I frames (Key frames): Intra-coded frame, coded as a still image. Can be decoded directly. Used at GOP head, or at scene changes. Allow random access, improves error resilience. P frames: (Inter-coded frames) Predicated from the previous frame. GOP Z. Li Multimedia Communciation, Spring 2017 p.21

GOP, I, P, and B Frames B frames: Bi-directional interpolated prediction frames Predicted from both the previous frame and the next frame: more flexibilities better prediction. Useful when new objects come into the scene. Before H.264, B frames are not used as reference for future frames: B frames can be coded with lower quality or can be discarded without affecting future frames. Allow temporal scalability Encoding/Decoding order: 1 4 2 3 7 5 6 1 2 3 4 5 6 7 I B B P B B P Display order: 1 2 3 4 5 6 7 Need more buffers Z. Li Multimedia Communciation, Spring 2017 p.22

Basic Encoder Block Diagram Input frame Intra Inter Pred. error DCT Q Q -1 Entropy Coding Intra Inter Prediction Prediction MC ME I DCT Recon Pred error Memory Recon. Motion vectors Reconstructed Previous frame Motion compensation (MC): get the prediction of the current frame using MV information. Use reconstructed frame in the loop to prevent drifting. Z. Li Multimedia Communciation, Spring 2017 p.23

Basic Decoder Block Diagram Entropy Decoding Q -1 I DCT Recon Pred error Motion vectors Intra Inter Prediction MC Reconstructed frame Memory Decoder is simpler than the encoder: No need to do motion estimation. Reconstructed Previous frame Z. Li Multimedia Communciation, Spring 2017 p.24

Motion Estimation For the k-th MB, the prediction error with MV (dx, dy): ( k, dx, dy) x f k n N, y k ( x,y) :The (x, N 1 N 1 i0 j0 : The upper- left coordinate of :MEblock size(16 for MBlevel f n y)- th pixel x i, y j f x dx i, y dy j k k in Frame n; ME) n1 k the k - th MB; p = 1: Sum of Absolute Difference (SAD) p = 2: Sum of Squared Difference (SSD) (-R, -R) Number of MV candidates: (2R+1) x (2R + 1) k p Objective: Find the MV that minimizes the pred error. ( dx *, dy *) argmin k k ( dx, dy) [ R, R] ( k, dx, dy) (R, R) Z. Li Multimedia Communciation, Spring 2017 p.25

Matlab: motion_estimation.m Simple Matlab implementation Source: motion_estimation.m, getblkmotion.m Block Motion Estimation Search a neighborhood of the block location, looking for the smallest block residual: Z. Li Multimedia Communciation, Spring 2017 p.26

Distribution of SSD/SAD as function of MV It is typically center biased Implications on fast search! B. Girod, EE398B Image Com, Stanford. Z. Li Multimedia Communciation, Spring 2017 p.27

Motion Estimation Example Implemented as int pel resolution exhaustive search in motion_estimation.m Z. Li Multimedia Communciation, Spring 2017 p.28

Motion Compensated Residual Error Motion Vectors predicted residual is smaller than direct frame difference No free lunch, need to code the motion field: 64x64 2D integer Z. Li Multimedia Communciation, Spring 2017 p.29

High Motion Sequences Motion Compensation benefits larger Z. Li Multimedia Communciation, Spring 2017 p.30

Bi-Directional Motion Compensation: B frames Possible temporal prediction for B pictures: 1, 1 1 1 2 2 2 bˆ cˆ ˆ 1 1 2c2 0 : forward prediction 0, 1: backward prediction 0.5: bidirectional prediction, C2 C1 b Frame k Frame k+1 Frame k-1 Z. Li Multimedia Communciation, Spring 2017 p.31

B Frames Can have better coding efficiency: Average of two predictions reduces the variance. New objects can be better predicted using future frames. Homework-3: Bi-Directional Motion Estimation matlab exercise! Z. Li Multimedia Communciation, Spring 2017 p.32

Sub-pixel Motion Estimation The actual motion is almost always not in integer. Optimal (dx, dy) = (3.2, 2.6) Need sub-pixel accuracy in motion estimation Benefits: Smaller prediction error better compression Cost: More complexities More bits spent on motion vectors Used in H.263 and H.264: Half-pixel accuracy 1/4-pixel accuracy Z. Li Multimedia Communciation, Spring 2017 p.33

Sub-pixel Motion Estimation Need to interpolate the reference frame: Find the best integer MV first, then find the best sub-pixel MV around the best integer MV. Bilinear interpolation: A e B Original samples: A, B, C, D Half-pixel locations: e, f, g, h, x g x h C f D e f g A B C D / 2 / 2 ( AC) / 2 0.5 0.5 0.5 Longer filters used in H.264. h x ( B D) / 2 0.5 A B C D / 4 0.5 Z. Li Multimedia Communciation, Spring 2017 p.34

H.264 Sub-pixel Motion Estimation Six-tap filter for ½-pixel samples: h = [ (E 5F + 20G + 20H 5 I + J) + 16 ] / 32 Similar operation for vi in vertical direction. x = [ (v1 5 v2 + 20 v3 + 20 v4 5 v5 + v6) + 16 ] / 32 E F G h H I J v1 v2 v3 x v4 v5 v6 Bi-linear filter for 1/4-pixel samples Z. Li Multimedia Communciation, Spring 2017 p.35

Bi-Linear Interpolation For fractional position pixel value: Matlab: im= bilinearinterpolation(im, 2*size(im)); HW-3: ½ pel motion estimation with bilinear filtering Z. Li Multimedia Communciation, Spring 2017 p.36

Complexity of Motion Estimation Complexity of Motion Estimation is the major portion of video codec compression pipeline If exhaustive search, need prohibitive complexity to complete. For each block, search (2*range+1 x 2*range+1) pixel locations, compute SSD/SAD. If fractional pel motion estimation, need to interpolate images, double the size of complexity for ½ pel, and quadruple the complexity for ¼ pel motion estimation Z. Li Multimedia Communciation, Spring 2017 p.38

Fast Motion Estimation ( k, dx, dy) N 1 N 1 i0 j0 f n x i, y j f x dx i, y dy j k k n1 k k p Full-search motion estimation is time consuming: Each (dx, dy) candidate: N 2 summations If search window size is (2R+1) 2, need (2R+1) 2 x N 2 operations / MB o R=15, N=16: 246016 operations / MB! Fast motion estimation: Use smart search order.(3 Step, Diamond Search) Reduce the number of searched MV candidates. Apply early termination techniques (partial SSD ) Z. Li Multimedia Communciation, Spring 2017 p.39

SSD Surface for Edge large gradients, all the same large l 1, small l 2 Z. Li Multimedia Communciation, Spring 2017 p.40

SSD Surface for low texture region gradients have small magnitude small l 1, small l 2 Z. Li Multimedia Communciation, Spring 2017 p.41

SSD Surface for High textured region gradients are different, large magnitudes large l 1, large l 2 Z. Li Multimedia Communciation, Spring 2017 p.42

The 3 Step Search ME 3 Step Search Search a 9 point displacement Compute the min SSD/SAD Shrink the size of the search window as it moves to the 2 nd stage Stop at the last search Significant reduction of complexity worst: 27 SSD/SAD computing -7-6 -5-4 -3-2 -1 0 1 2 3 4 5 6 7 Z. Li Multimedia Communciation, Spring 2017 p.43

NTSS(new 3-step search) algorithm NTSS algorithm summary: Decision 1: Min SSD at the center? Decision 2: Min SSD at one of the neighbor of the center? -7-6 -5-4 -3-2 -1 0 1 2 3 4 5 6 7 1st step of NTSS 17 checking points T MV=0 F Decision 1 Decision 2 T 2nd step of NTSS 3 or 5 checking points F 2nd and 3rd step of NTSS (same as in TSS) Z. Li Multimedia Communciation, Spring 2017 p.44

Diamond Search Algorithm To employ two search patterns Large Diamond Search Pattern (LDSP) Small Diamond Search Pattern (SDSP) Z. Li Multimedia Communciation, Spring 2017 p.45

Step 1 The initial LDSP is centered at the origin of the search window, and the 9 checking points of LDSP are tested. If the minimum block distortion (MBD) point calculated is located at the center position, go to Step3, Perform Small DSP search Else,, go to Step 2, perform Large DSP Search Z. Li Multimedia Communciation, Spring 2017 p.46

Summary of Fast Search Diamond Patter Search Ref: [11] Renxiang Li, Bing Zeng, Ming L. Liou, A new three-step search algorithm for block motion estimation, IEEE Trans. Circuits Syst. Video Tech vol.4(4): 438-442 (1994). [top 10 cited T-CSVT paper] [12] S. Zhu, K.-K. Ma, A new diamond search algorithm for fast block-matching motion estimation, IEEE Transactions on Image Processing vol.9(2): 287-290 (2000). Z. Li Multimedia Communciation, Spring 2017 p.47

Motion Compensation a main tool in video coding Pre-HEVC H.261(int pel MC) over JPEG: 50% rate reduction Fractional pel + variable block size: 100+% reduction Z. Li Multimedia Communciation, Spring 2017 p.48

Summary Video Signal is a signal over time Motion Modeling is a key compression efficiency contributor Basic Tricks Block based displacement SSD/SAD computing Fractional pel resolution to improve gains Variable block sizes to better match the object segmentation Fast Motion Estimation Avoid exhaustive search, capitalize the statistics of the SSD surface 3 Step/New 3 Step motion estimation Diamond search. Z. Li Multimedia Communciation, Spring 2017 p.49