Overview: motion-compensated coding Motion-compensated prediction Motion-compensated hybrid coding Motion estimation by block-matching Motion estimation with sub-pixel accuracy Power spectral density of the motion-compensated prediction error Rate-distortion analysis Loop filter Motion compensated coding with sub-pixel accuracy Rate-constrained motion estimation Bernd Girod: EE398B Image Communication II Motion Compensation no. 1
Motion-compensated prediction previous frame stationary background current frame x t time t moving object y Displacement vector d x d y shifted object Prediction for the luminance signal S(x,y,t) within the moving object: ˆ S (x, y,t) = S(x d x, y d y,t t) Bernd Girod: EE398B Image Communication II Motion Compensation no. 2
Motion-compensated hybrid coder Coder Control Control Data - Decoder Intra-frame DCT Coder Intra-frame Decoder DCT Coefficients Intra/Inter 0 Motion- Compensated Predictor Motion Estimator Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 3
Motion-compensated hybrid decoder Control Data DCT Coefficients Decoder Intra-frame Decoder Intra/Inter 0 Motion- Compensated Predictor Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 4
Block-matching algorithm search range in previous frame S k 1 Subdivide every image into square blocks. Find one displacement vector for each block. Within a search range, find a best match that minimizes an error measure. Intelligent search strategies can reduce computation. block of current frame S k Bernd Girod: EE398B Image Communication II Motion Compensation no. 5
Block-matching algorithm Previous Image Current Image Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 6
Block-matching algorithm Previous Image Current Image... process repeated for another measurement window position. Bernd Girod: EE398B Image Communication II Motion Compensation no. 7
Blockmatching: Matching Criterion Sum of Squared Differences to determine similarity Sum all values in measurement window Current image Previous image SSD( d, d ) = S ( x, y) S ( x + d, y + d ) x y k k 1 x y msmnt window 2 Horizontal shift Vertical shift Alternative matching criteria: SAD (Sum of Absolute Differences), cross correlation,... Only integer pixel shifts are possible Bernd Girod: EE398B Image Communication II Motion Compensation no. 8
Integer Pixel Shifts Previous Image Current Image Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 9
Integer Pixel Shifts 28 42 42 43 44 40 32 20 29 32 22 30 44 45 45 45 42 30 21 26 27 18 35 54 54 54 53 54 53 52 54 53 52 49 53 52 49 31 52 49 31 21 49 31 21 28 31 21 24 21 15 40 63 62 62 63 62 63 59 62 63 59 60 63 59 60 44 59 60 44 33 60 44 33 35 44 33 31 33 26 74 121 120 120 114 120 114 112 120 114 112 111 114 112 111 80 112 111 80 32 111 80 32 23 80 32 22 32 17 79 127 130 130 128 130 128 124 130 128 124 125 128 124 125 88 124 125 88 125 88 24 17 88 24 20 24 13 80 129 131 131 124 131 124 127 131 124 127 124 127 96 127 96 42 127 96 42 29 96 42 28 42 19 50 78 77 77 71 77 71 73 77 71 73 75 71 73 75 63 73 75 63 52 75 63 52 47 63 52 46 52 29 54 53 52 49 31 21 62 63 59 60 44 33 120 114 112 111 80 32 130 128 124 125 88 24 131 124 127 127 96 42 77 71 73 75 63 52 22 37 37 37 39 40 40 41 41 38 25 Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 10
SSD Values Resulting from Blockmatching SSD 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 0 5 Vertical shift d y 10 15 2 6 4 12 10 8 Horizontal shift d x Estimated displacement Integer-pixel accuracy 14 Bernd Girod: EE398B Image Communication II Motion Compensation no. 11
Motion-compensated prediction: example Previous frame Current frame Current frame with displacement vectors Motion-compensated Prediction error Bernd Girod: EE398B Image Communication II Motion Compensation no. 12
Interpolation of the SSD Minimum SSD Sub-pixel Accurate Minimum Fit parabola through 3 >3 points exactly approximately Horizontal shift d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 13
2-d Interpolation of SSD Minimum Paraboloid Perfect fit through 6 points Approximate fit through >6 points SSD Vertical shift d y Horizontal shift d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 14
Blockmatching: search strategies I Full search All possible displacements within the search range are compared. Computationally expensive Highly regular, parallelizable d y d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 15
Blockmatching: search strategies II 2D logarithmic search (Jain + Jain, 1981) Iterative comparison of error measure values at 5 neighboring points Logarithmic refinement of the search pattern if best match is in the center of the 5- point pattern center of search pattern touches the border of the search range 3 5 4 5 5 3 5 5 2 3 2 1 1 1 1 1 2 4 d x d y Bernd Girod: EE398B Image Communication II Motion Compensation no. 16
Blockmatching: search strategies III Computational complexity Example: max. horizontal, vertical displacement = 6, integer-pel accuracy: Block comparisons a b 2D logarithmic full search 18 169 21 169 a - for special vector (2,6) b - worst case Bernd Girod: EE398B Image Communication II Motion Compensation no. 17
Block comparison speed-ups Triangle and Cauchy-Schwarz inequality for SAD and SSE Strategy: S k S k 1 S k S k 1 = S k S k 1 block Compute partial sums for blocks in current and previous frame Compare blocks based on partial sums Omit full block comparison, if partial sums indicate worse error measure than previous best result Performance: > 20x speed-up of full search block matching reported by employing [Lin + Tai, IEEE Tr. Commun., May 97] Sum over 16x16 block Row wise block projection block Column wise block projection block block 2 2 1 1 S S S S S S 2 ( ) = k k 1 k k 1 k k 1 block N block N block block number of terms in sum Bernd Girod: EE398B Image Communication II Motion Compensation no. 18
Hierarchical blockmatching Displacement vector field Block matching current frame previous frame Block matching Filtering and subsampling Block matching Filtering and subsampling Bernd Girod: EE398B Image Communication II Motion Compensation no. 19
Sub-pel accuracy Block matching Displacement vector field with sub-pel accuracy Displacement vector field with integer-pel accuracy Interpolation Block matching current frame previous frame Block matching Filtering and subsampling Block matching Filtering and subsampling Bernd Girod: EE398B Image Communication II Motion Compensation no. 20
Fractional-pixel accuracy Interpolate pixel raster of the reference image to desired fractional pixel accuracy (typically by bi-linear interpolation) Straightforward extension of displacement vector search to fractional accuracy Example: half-pixel accurate displacements dx 4.5 = d y 4.5 Bernd Girod: EE398B Image Communication II Motion Compensation no. 21
Bi-linear Interpolation brightness Interpolated Pixel Value Bernd Girod: EE398B Image Communication II Motion Compensation no. 22
Bi-linear Interpolation (cont.) I(x,y ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 23
Model for performance analysis of an MCP hybrid coder luminance signal S - e R-D optimal intraframe encoder intraframe decoder e motion compensated predictor s displacement estimate true displacement d x d y + x y displacement error Bernd Girod: EE398B Image Communication II Motion Compensation no. 24
Analysis of the motion-compensated prediction error Previous frame Motion-compensated signal ( ) = ( ) ( ) c x s x n x Prediction error x ( ) = ( ) ( ) = s( x) s( x ) + n( x) e x s x c x x Current frame c(x) s(x) x Displacement d x Displacement error x x Bernd Girod: EE398B Image Communication II Motion Compensation no. 25
Analysis of m.c. prediction error (cont.) Motion-compensated prediction error ( ) ( ) ( ) ( ) ( ) ( ) δ( ) δ( ) Power spectrum of prediction error, assuming constant displacement error, statistical independence of s and n x ( ) ( ) ( ) ex= sx cx= sx sx + nx= x x sx+ nx x jω x jω x ( e )( e ) jω x ss ω { e } Φ ( ω) =Φ ( ω) 1 1 +Φ ( ω) ee ss nn ( ) = 2 Φ ( ) 1 Re +Φ ( ω) jω x { e } x Random displacement error, statistically independent from s, n { ( ) } Φ ( ω) = E 2 Φ ( ω) 1 Re +Φ ( ω) ee ss nn jω x { } ( { E e }) = 2 Φ ( ω) 1 Re +Φ ( ω) ss { P ( )} ( ) = 2 Φ ( ω) 1 Re ω +Φ ( ω) ss Bernd Girod: EE398B Image Communication II Motion Compensation no. 26 nn nn nn x
Analysis of m.c. prediction error (cont.) What is P(ω)? jω x ( ω ) = { } P E e x { x } jω ( ) ( ) = p e d = F p Fourier transform of the displacement error pdf! x Same as characteristic function of displacement error, except for sign Extension to 2-d ( { P }) Φ ( ω, ω ) = 2 Φ ( ω, ω ) 1 Re ( ω, ω ) +Φ ( ω, ω ) ee x y ss x y x y nn x y power spectrum of luminance signal Fourier transform of the displacement error pdf p( x, y ) noise spectrum Bernd Girod: EE398B Image Communication II Motion Compensation no. 27
Power spectrum of motion-compensated prediction error 0 π frequency ω x Bernd Girod: EE398B Image Communication II Motion Compensation no. 28
R-D function for MCP with integer-pixel accuracy ( x, y ) T assumed uniformly distributed between x =± 1 2 pel y =± 1 2 line Minimum bit-rate for given SNR ~0.7 bpp Gaussian signal model Φ ss (ω x,ω y ) = A 1+ ω x 2 ω 0 2 +ω y 2 Typical parameters for CIF resolution (352 x 288 pixels) 3 2 Bernd Girod: EE398B Image Communication II Motion Compensation no. 29
Required accuracy of motion compensation p( x, y ) isotropic Gaussian pdf with variance Φ ss (ω x,ω y ) = A 1+ ω 2 2 x +ω y 2 ω 0 Typical parameters for CIF resolution (352 x 288 pixels) Minimum bit-rate for SNR = 30 db σ 2 3 2 Bernd Girod: EE398B Image Communication II Motion Compensation no. 30
Model of MCP hybrid coder with loop filter luminance signal S - e R-D optimal intraframe encoder intraframe decoder spatial filter F motion compensated predictor displacement estimate true displacement d x d y + x y displacement error Bernd Girod: EE398B Image Communication II Motion Compensation no. 31
Motion-compensated prediction error with loop filter Motion-compensated signal cx ()= sx ( x ) nx Previous frame ( ) Prediction error ( ) = s x e x ( ) f( x) c( x) = sx () fx () s( x x )+ fx () n x Impulse response of loop filter () Current frame c(x) s(x) x Displacement d x Displacement error x x Bernd Girod: EE398B Image Communication II Motion Compensation no. 32
Spatial power spectrum of m.c. prediction error with loop filter { } ( ) Φ ( Λ ) =Φ ( Λ ) 1 + F( Λ) 2Re F( Λ) P( Λ ) +Φ ( Λ) F( Λ) 2 2 ee ss nn P(Λ) 2 -D Fourier transform of displacement error pdf F(Λ) 2 -D Fourier transform of f (x, y) Φ uu spatial spectral power density of signal u Λ vector of spatial frequencies (ω x,ω y ) n(x, y) noise Bernd Girod: EE398B Image Communication II Motion Compensation no. 33
Optimum loop filter Wiener filter minimizes prediction error variance Φss ( Λ) Fopt ( Λ ) = P ( Λ) Φ ss ( Λ ) +Φ nn( Λ ) accounts for accuracy of motion compensation Resulting minimum prediction error spectrum accounts for noise 2 Φss ( Λ) Φee( Λ ) =Φss ( Λ) 1 P( Λ) Φss ( Λ ) +Φnn( Λ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 34
Effect of loop filter Moderately accurate motion compensation Very accurate motion compensation Bernd Girod: EE398B Image Communication II Motion Compensation no. 35
Required accuracy of motion compensation with loop filter p( x, y ) isotropic Gaussian pdf with variance σ 2 Minimum bit-rate for SNR = 30 db ~0.8 bpp Bernd Girod: EE398B Image Communication II Motion Compensation no. 36
Practical optimum loop filter design Not practical for loop filter design Φss ( Λ) Fopt ( Λ ) = P ( Λ) Φ ss ( Λ ) +Φ nn( Λ ) Motion compensation accuracy not known To determine Wiener filter from measurements: F opt (Λ) = Φ sc (Λ) Φ cc (Λ) Noise psd not known cross spectrum between s(x,y) and the motion-compensated c(x, y) = r(x d ˆ x, y d ˆ signal y ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 37
Experimental evaluation of fractional-pixel motion compensation ITU-R 601 TV signals, 13.5 MHz sampling rate, interlaced, blockwise motion compensation with blocksize16x16 Zoom Voiture Bernd Girod: EE398B Image Communication II Motion Compensation no. 38
Influence of noise on the performance of MCP Bernd Girod: EE398B Image Communication II Motion Compensation no. 39
Motion Compensation Performance in H.263 500 Bit Rate [kbps] 400 300 200 Intra mode only Integer-pel accuracy 100 1/2-pel accuracy 0 24 30 36 Simulation details: Foreman, QCIF, SKIP=2 Q=4,5,7,10,15,25 PSNR [db] Bernd Girod: EE398B Image Communication II Motion Compensation no. 40
Rate-constrained motion estimation I motion vector rate R m D bit-rate R prediction error rate R e displacement error variance optimum trade-off: D R m = D R e Bernd Girod: EE398B Image Communication II Motion Compensation no. 41
Rate-constrained motion estimation II How to find best motion vector subject to rate constraint? Lagrangian cost function: solve unconstrained problem rather than constrained problem min( D + λr m ) error measure motion vector rate Interpret motion search as ECVQ problem. Bernd Girod: EE398B Image Communication II Motion Compensation no. 42
Rate-constrained Motion Estimation in H.263 Reference Model TMN-10 40 20 PSNR [db] 38 36 34 32 30 28 TMN-10, w/ rate-constrained motion estimation TMN-10, w/o rate-constrained motion estimation (TMN-9) Partial bit-rate [kbps] 15 10 5 TMN-10, w/ rate-constrained motion estimation TMN-10, w/o rate-constrained motion estimation (TMN-9) 26 0 50 100 150 200 Bit Rate [kbps] Simulation details: Foreman, QCIF, SKIP=2 Q=4,5,7,10,15,25 Annexes D+F 0 0 50 100 150 200 Bit Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 43
Video coder control Encoding decisions Coding modes (intra/inter/motion comp.) Block size Motion vectors Quantizer step size Suppression of DCT coefficients Solution Embed rate-constrained motion estimation into mode decision with Lagrangian cost function Couple Lagrange multiplier to quantizer step size Difficulties Joint entropy coding of side information Temporal dependencies due to DPCM structure Input Video - Intra/Inter Decoder 0 Coder Control Intraframe DCT Coder Motion- Compensated Predictor Intraframe Decoder Control Data DCT Coefficients Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 44
History of motion-compensated coding Intraframe coding: only spatial correlation exploited DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992] Conditional replenishment H.120 [1984] (DPCM, scalar quantization) Frame difference coding H.120 Version 2 [1988] Motion compensation: integer-pel accurate displacements H.261 [1991] Half-pel accurate motion compensation MPEG-1 [1993], MPEG-2/H.262 [1994] Variable block-size motion compensation H.263 [1996], MPEG-4 [1999] Complexity increases Bernd Girod: EE398B Image Communication II Motion Compensation no. 45
Efficiency of motion-compensated coding PSNR [db] 38 36 34 32 30 28 State-of-the-Art: Half-pel TMN-10 motion Variable compensation block size (MPEG-1 1993) motion compensation (H.263 1998)? 67 % Integer-pel motion compensation (H.261 1991) Frame difference coding (H.120 1988) Intraframe DCT coding (DCT 1974, JPEG 1992) 26 0 100 200 300 400 500 Foreman 10 Hz, QCIF 100 frames encoded Bit-Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 46
Efficiency of motion-compensated coding PSNR [db] 42 40 Variable block size motion compensation (H.263 1996) Conditional Replenishment (H.120) Mother & Daughter 10 Hz, QCIF 100 frames encoded 38 36 34 32 30 60 % Intraframe DCT coding (JPEG) 28 0 100 200 300 400 Bit-Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 47
Efficiency of motion-compensated coding PSNR [db] 40 38 36 34 32 30 28 26 Variable block size motion compensation (H.263 1996) 35 % Integer-pel motion compensation (H.261 1991) 40 % Intraframe DCT coding (JPEG) Mobile & Calendar 10 Hz, QCIF 100 frames encoded 24 Bit-Rate [kbps] 22 0 500 1000 1500 Bernd Girod: EE398B Image Communication II Motion Compensation no. 48