Overview: motion-compensated coding

Transcription

1 Overview: motion-compensated coding Motion-compensated prediction Motion-compensated hybrid coding Motion estimation by block-matching Motion estimation with sub-pixel accuracy Power spectral density of the motion-compensated prediction error Rate-distortion analysis Loop filter Motion compensated coding with sub-pixel accuracy Rate-constrained motion estimation Bernd Girod: EE398B Image Communication II Motion Compensation no. 1

2 Motion-compensated prediction previous frame stationary background current frame x t time t moving object y Displacement vector d x d y shifted object Prediction for the luminance signal S(x,y,t) within the moving object: ˆ S (x, y,t) = S(x d x, y d y,t t) Bernd Girod: EE398B Image Communication II Motion Compensation no. 2

3 Motion-compensated hybrid coder Coder Control Control Data - Decoder Intra-frame DCT Coder Intra-frame Decoder DCT Coefficients Intra/Inter 0 Motion- Compensated Predictor Motion Estimator Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 3

4 Motion-compensated hybrid decoder Control Data DCT Coefficients Decoder Intra-frame Decoder Intra/Inter 0 Motion- Compensated Predictor Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 4

5 Block-matching algorithm search range in previous frame S k 1 Subdivide every image into square blocks. Find one displacement vector for each block. Within a search range, find a best match that minimizes an error measure. Intelligent search strategies can reduce computation. block of current frame S k Bernd Girod: EE398B Image Communication II Motion Compensation no. 5

6 Block-matching algorithm Previous Image Current Image Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 6

7 Block-matching algorithm Previous Image Current Image... process repeated for another measurement window position. Bernd Girod: EE398B Image Communication II Motion Compensation no. 7

8 Blockmatching: Matching Criterion Sum of Squared Differences to determine similarity Sum all values in measurement window Current image Previous image SSD( d, d ) = S ( x, y) S ( x + d, y + d ) x y k k 1 x y msmnt window 2 Horizontal shift Vertical shift Alternative matching criteria: SAD (Sum of Absolute Differences), cross correlation,... Only integer pixel shifts are possible Bernd Girod: EE398B Image Communication II Motion Compensation no. 8

9 Integer Pixel Shifts Previous Image Current Image Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 9

10 Integer Pixel Shifts Measurement window is compared with a shifted array of pixels in the other image, to determine the best match Rectangular array of pixels is selected as a measurement window Bernd Girod: EE398B Image Communication II Motion Compensation no. 10

11 SSD Values Resulting from Blockmatching SSD Vertical shift d y Horizontal shift d x Estimated displacement Integer-pixel accuracy 14 Bernd Girod: EE398B Image Communication II Motion Compensation no. 11

12 Motion-compensated prediction: example Previous frame Current frame Current frame with displacement vectors Motion-compensated Prediction error Bernd Girod: EE398B Image Communication II Motion Compensation no. 12

13 Interpolation of the SSD Minimum SSD Sub-pixel Accurate Minimum Fit parabola through 3 >3 points exactly approximately Horizontal shift d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 13

14 2-d Interpolation of SSD Minimum Paraboloid Perfect fit through 6 points Approximate fit through >6 points SSD Vertical shift d y Horizontal shift d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 14

15 Blockmatching: search strategies I Full search All possible displacements within the search range are compared. Computationally expensive Highly regular, parallelizable d y d x Bernd Girod: EE398B Image Communication II Motion Compensation no. 15

16 Blockmatching: search strategies II 2D logarithmic search (Jain + Jain, 1981) Iterative comparison of error measure values at 5 neighboring points Logarithmic refinement of the search pattern if best match is in the center of the 5- point pattern center of search pattern touches the border of the search range d x d y Bernd Girod: EE398B Image Communication II Motion Compensation no. 16

17 Blockmatching: search strategies III Computational complexity Example: max. horizontal, vertical displacement = 6, integer-pel accuracy: Block comparisons a b 2D logarithmic full search a - for special vector (2,6) b - worst case Bernd Girod: EE398B Image Communication II Motion Compensation no. 17

18 Block comparison speed-ups Triangle and Cauchy-Schwarz inequality for SAD and SSE Strategy: S k S k 1 S k S k 1 = S k S k 1 block Compute partial sums for blocks in current and previous frame Compare blocks based on partial sums Omit full block comparison, if partial sums indicate worse error measure than previous best result Performance: > 20x speed-up of full search block matching reported by employing [Lin + Tai, IEEE Tr. Commun., May 97] Sum over 16x16 block Row wise block projection block Column wise block projection block block S S S S S S 2 ( ) = k k 1 k k 1 k k 1 block N block N block block number of terms in sum Bernd Girod: EE398B Image Communication II Motion Compensation no. 18

19 Hierarchical blockmatching Displacement vector field Block matching current frame previous frame Block matching Filtering and subsampling Block matching Filtering and subsampling Bernd Girod: EE398B Image Communication II Motion Compensation no. 19

20 Sub-pel accuracy Block matching Displacement vector field with sub-pel accuracy Displacement vector field with integer-pel accuracy Interpolation Block matching current frame previous frame Block matching Filtering and subsampling Block matching Filtering and subsampling Bernd Girod: EE398B Image Communication II Motion Compensation no. 20

21 Fractional-pixel accuracy Interpolate pixel raster of the reference image to desired fractional pixel accuracy (typically by bi-linear interpolation) Straightforward extension of displacement vector search to fractional accuracy Example: half-pixel accurate displacements dx 4.5 = d y 4.5 Bernd Girod: EE398B Image Communication II Motion Compensation no. 21

22 Bi-linear Interpolation brightness Interpolated Pixel Value Bernd Girod: EE398B Image Communication II Motion Compensation no. 22

23 Bi-linear Interpolation (cont.) I(x,y ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 23

24 Model for performance analysis of an MCP hybrid coder luminance signal S - e R-D optimal intraframe encoder intraframe decoder e motion compensated predictor s displacement estimate true displacement d x d y + x y displacement error Bernd Girod: EE398B Image Communication II Motion Compensation no. 24

25 Analysis of the motion-compensated prediction error Previous frame Motion-compensated signal ( ) = ( ) ( ) c x s x n x Prediction error x ( ) = ( ) ( ) = s( x) s( x ) + n( x) e x s x c x x Current frame c(x) s(x) x Displacement d x Displacement error x x Bernd Girod: EE398B Image Communication II Motion Compensation no. 25

26 Analysis of m.c. prediction error (cont.) Motion-compensated prediction error ( ) ( ) ( ) ( ) ( ) ( ) δ( ) δ( ) Power spectrum of prediction error, assuming constant displacement error, statistical independence of s and n x ( ) ( ) ( ) ex= sx cx= sx sx + nx= x x sx+ nx x jω x jω x ( e )( e ) jω x ss ω { e } Φ ( ω) =Φ ( ω) 1 1 +Φ ( ω) ee ss nn ( ) = 2 Φ ( ) 1 Re +Φ ( ω) jω x { e } x Random displacement error, statistically independent from s, n { ( ) } Φ ( ω) = E 2 Φ ( ω) 1 Re +Φ ( ω) ee ss nn jω x { } ( { E e }) = 2 Φ ( ω) 1 Re +Φ ( ω) ss { P ( )} ( ) = 2 Φ ( ω) 1 Re ω +Φ ( ω) ss Bernd Girod: EE398B Image Communication II Motion Compensation no. 26 nn nn nn x

27 Analysis of m.c. prediction error (cont.) What is P(ω)? jω x ( ω ) = { } P E e x { x } jω ( ) ( ) = p e d = F p Fourier transform of the displacement error pdf! x Same as characteristic function of displacement error, except for sign Extension to 2-d ( { P }) Φ ( ω, ω ) = 2 Φ ( ω, ω ) 1 Re ( ω, ω ) +Φ ( ω, ω ) ee x y ss x y x y nn x y power spectrum of luminance signal Fourier transform of the displacement error pdf p( x, y ) noise spectrum Bernd Girod: EE398B Image Communication II Motion Compensation no. 27

28 Power spectrum of motion-compensated prediction error 0 π frequency ω x Bernd Girod: EE398B Image Communication II Motion Compensation no. 28

29 R-D function for MCP with integer-pixel accuracy ( x, y ) T assumed uniformly distributed between x =± 1 2 pel y =± 1 2 line Minimum bit-rate for given SNR ~0.7 bpp Gaussian signal model Φ ss (ω x,ω y ) = A 1+ ω x 2 ω 0 2 +ω y 2 Typical parameters for CIF resolution (352 x 288 pixels) 3 2 Bernd Girod: EE398B Image Communication II Motion Compensation no. 29

30 Required accuracy of motion compensation p( x, y ) isotropic Gaussian pdf with variance Φ ss (ω x,ω y ) = A 1+ ω 2 2 x +ω y 2 ω 0 Typical parameters for CIF resolution (352 x 288 pixels) Minimum bit-rate for SNR = 30 db σ Bernd Girod: EE398B Image Communication II Motion Compensation no. 30

31 Model of MCP hybrid coder with loop filter luminance signal S - e R-D optimal intraframe encoder intraframe decoder spatial filter F motion compensated predictor displacement estimate true displacement d x d y + x y displacement error Bernd Girod: EE398B Image Communication II Motion Compensation no. 31

32 Motion-compensated prediction error with loop filter Motion-compensated signal cx ()= sx ( x ) nx Previous frame ( ) Prediction error ( ) = s x e x ( ) f( x) c( x) = sx () fx () s( x x )+ fx () n x Impulse response of loop filter () Current frame c(x) s(x) x Displacement d x Displacement error x x Bernd Girod: EE398B Image Communication II Motion Compensation no. 32

33 Spatial power spectrum of m.c. prediction error with loop filter { } ( ) Φ ( Λ ) =Φ ( Λ ) 1 + F( Λ) 2Re F( Λ) P( Λ ) +Φ ( Λ) F( Λ) 2 2 ee ss nn P(Λ) 2 -D Fourier transform of displacement error pdf F(Λ) 2 -D Fourier transform of f (x, y) Φ uu spatial spectral power density of signal u Λ vector of spatial frequencies (ω x,ω y ) n(x, y) noise Bernd Girod: EE398B Image Communication II Motion Compensation no. 33

34 Optimum loop filter Wiener filter minimizes prediction error variance Φss ( Λ) Fopt ( Λ ) = P ( Λ) Φ ss ( Λ ) +Φ nn( Λ ) accounts for accuracy of motion compensation Resulting minimum prediction error spectrum accounts for noise 2 Φss ( Λ) Φee( Λ ) =Φss ( Λ) 1 P( Λ) Φss ( Λ ) +Φnn( Λ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 34

35 Effect of loop filter Moderately accurate motion compensation Very accurate motion compensation Bernd Girod: EE398B Image Communication II Motion Compensation no. 35

36 Required accuracy of motion compensation with loop filter p( x, y ) isotropic Gaussian pdf with variance σ 2 Minimum bit-rate for SNR = 30 db ~0.8 bpp Bernd Girod: EE398B Image Communication II Motion Compensation no. 36

37 Practical optimum loop filter design Not practical for loop filter design Φss ( Λ) Fopt ( Λ ) = P ( Λ) Φ ss ( Λ ) +Φ nn( Λ ) Motion compensation accuracy not known To determine Wiener filter from measurements: F opt (Λ) = Φ sc (Λ) Φ cc (Λ) Noise psd not known cross spectrum between s(x,y) and the motion-compensated c(x, y) = r(x d ˆ x, y d ˆ signal y ) Bernd Girod: EE398B Image Communication II Motion Compensation no. 37

38 Experimental evaluation of fractional-pixel motion compensation ITU-R 601 TV signals, 13.5 MHz sampling rate, interlaced, blockwise motion compensation with blocksize16x16 Zoom Voiture Bernd Girod: EE398B Image Communication II Motion Compensation no. 38

39 Influence of noise on the performance of MCP Bernd Girod: EE398B Image Communication II Motion Compensation no. 39

40 Motion Compensation Performance in H Bit Rate [kbps] Intra mode only Integer-pel accuracy 100 1/2-pel accuracy Simulation details: Foreman, QCIF, SKIP=2 Q=4,5,7,10,15,25 PSNR [db] Bernd Girod: EE398B Image Communication II Motion Compensation no. 40

41 Rate-constrained motion estimation I motion vector rate R m D bit-rate R prediction error rate R e displacement error variance optimum trade-off: D R m = D R e Bernd Girod: EE398B Image Communication II Motion Compensation no. 41

42 Rate-constrained motion estimation II How to find best motion vector subject to rate constraint? Lagrangian cost function: solve unconstrained problem rather than constrained problem min( D + λr m ) error measure motion vector rate Interpret motion search as ECVQ problem. Bernd Girod: EE398B Image Communication II Motion Compensation no. 42

43 Rate-constrained Motion Estimation in H.263 Reference Model TMN PSNR [db] TMN-10, w/ rate-constrained motion estimation TMN-10, w/o rate-constrained motion estimation (TMN-9) Partial bit-rate [kbps] TMN-10, w/ rate-constrained motion estimation TMN-10, w/o rate-constrained motion estimation (TMN-9) Bit Rate [kbps] Simulation details: Foreman, QCIF, SKIP=2 Q=4,5,7,10,15,25 Annexes D+F Bit Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 43

44 Video coder control Encoding decisions Coding modes (intra/inter/motion comp.) Block size Motion vectors Quantizer step size Suppression of DCT coefficients Solution Embed rate-constrained motion estimation into mode decision with Lagrangian cost function Couple Lagrange multiplier to quantizer step size Difficulties Joint entropy coding of side information Temporal dependencies due to DPCM structure Input Video - Intra/Inter Decoder 0 Coder Control Intraframe DCT Coder Motion- Compensated Predictor Intraframe Decoder Control Data DCT Coefficients Motion Data Bernd Girod: EE398B Image Communication II Motion Compensation no. 44

45 History of motion-compensated coding Intraframe coding: only spatial correlation exploited DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992] Conditional replenishment H.120 [1984] (DPCM, scalar quantization) Frame difference coding H.120 Version 2 [1988] Motion compensation: integer-pel accurate displacements H.261 [1991] Half-pel accurate motion compensation MPEG-1 [1993], MPEG-2/H.262 [1994] Variable block-size motion compensation H.263 [1996], MPEG-4 [1999] Complexity increases Bernd Girod: EE398B Image Communication II Motion Compensation no. 45

46 Efficiency of motion-compensated coding PSNR [db] State-of-the-Art: Half-pel TMN-10 motion Variable compensation block size (MPEG ) motion compensation (H )? 67 % Integer-pel motion compensation (H ) Frame difference coding (H ) Intraframe DCT coding (DCT 1974, JPEG 1992) Foreman 10 Hz, QCIF 100 frames encoded Bit-Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 46

47 Efficiency of motion-compensated coding PSNR [db] Variable block size motion compensation (H ) Conditional Replenishment (H.120) Mother & Daughter 10 Hz, QCIF 100 frames encoded % Intraframe DCT coding (JPEG) Bit-Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 47

48 Efficiency of motion-compensated coding PSNR [db] Variable block size motion compensation (H ) 35 % Integer-pel motion compensation (H ) 40 % Intraframe DCT coding (JPEG) Mobile & Calendar 10 Hz, QCIF 100 frames encoded 24 Bit-Rate [kbps] Bernd Girod: EE398B Image Communication II Motion Compensation no. 48