Overview: motion estimation Differential methods Fast algorithms for Sub-pel accuracy Rate-constrained motion estimation Bernd Girod: EE368b Image Video Compression Motion Estimation no. 1 Differential motion estimation Assume small displacements : FD(x, y) (d x,d y ) S(x, y) d x x S(x, y) d y y frame-to-frame difference horizontal, vertical gradient of image signal S Aperture problem: 1 equation, but unknowns per pixel. At least (in practice: several) coupled observations required. Linearization inaccurate for displacements > 0.5 pel multigrid methods, iteration Bernd Girod: EE368b Image Video Compression Motion Estimation no. 1
Principle of search range in previous frame S k 1 Subdivide every image into square s. Find one displacement vector for each. Within a search range, find a best match that minimizes an error measure. Intelligent search strategies can reduce computation. of current frame S k Bernd Girod: EE368b Image Video Compression Motion Estimation no. 3 : error measure Mean squared error (sum of squared errors) SSE(d x, d y ) = S k (x, y) S k 1 (x + d x, y + d y ) Sum of absolute differences [ ] SAD(d x, d y ) = S k (x, y) S k 1 (x + d x, y + d y ) Approximately same performance SAD less complex for some architectures Bernd Girod: EE368b Image Video Compression Motion Estimation no. 4
: search strategies I Full search All possible displacements within the search range are compared. Computationally expensive Highly regular, parallelizable d y d x Bernd Girod: EE368b Image Video Compression Motion Estimation no. 5 : search strategies II D logarithmic search (Jain + Jain, 1981) Iterative comparison of error measure values at 5 neighboring points Logarithmic refinement of the search pattern if best match is in the center of the 5- point pattern center of search pattern touches the border of the search range 3 5 4 5 5 5 5 3 3 d y 4 d x Bernd Girod: EE368b Image Video Compression Motion Estimation no. 6 3
: search strategies III Computational complexity Example: max. horizontal, vertical displacement = 6, integer-pel accuracy: comparisons a b D logarithmic full search 18 169 1 169 a - for special vector (,6) b - worst case Bernd Girod: EE368b Image Video Compression Motion Estimation no. 7 comparison speed-ups Triangle inequalities for SAD SSE Strategy: S k S k 1 S k S k 1 = S k S k 1 S k S k 1 1 N Compute partial sums for s in current previous frame Compare s based on partial sums Omit full comparison, if partial sums indicate worse error measure than previous best result Performance: > 0x speed-up of full search reported by employing (Lin + Tai, IEEE Tr. Commun., May 97) Sum over 16x16 Row wise projection Column wise projection S k S k 1 number of terms in sum Bernd Girod: EE368b Image Video Compression Motion Estimation no. 8 = 1 N S k S k 1 4
Hierarchical Displacement vector field current frame previous frame Bernd Girod: EE368b Image Video Compression Motion Estimation no. 9 Sub-pel accuracy Displacement vector field with sub-pel accuracy Displacement vector field with integer-pel accuracy Interpolation current frame previous frame Bernd Girod: EE368b Image Video Compression Motion Estimation no. 10 5
Rate-constrained motion estimation I motion vector D rate R m bit-rate R prediction error rate R e displacement error variance optimum trade-off: D R m = D R e Bernd Girod: EE368b Image Video Compression Motion Estimation no. 11 Rate-constrained motion estimation II How to find best motion vector subject to rate constraint? Lagrangian cost function: solve unconstrained problem rather than constrained problem min( D + λr m ) error measure motion vector rate Interpret motion search as ECVQ problem. Bernd Girod: EE368b Image Video Compression Motion Estimation no. 1 6
Rate-constrained Motion Estimation in H.63 Reference Model TMN-10 40 0 PSNR [db] 38 36 34 3 30 8 w/ rate-constrained motion estimation w/o rate-constrained motion estimation (TMN-9) Partial bit-rate [kbps] 15 10 5 w/ rate-constrained motion estimation w/o rate-constrained motion estimation (TMN-9) 6 0 50 100 150 00 Bit Rate [kbps] 0 0 50 100 150 00 Bit Rate [kbps] Simulation details: Foreman, QCIF, SKIP= Q=4,5,7,10,15,5 Annexes D+F Bernd Girod: EE368b Image Video Compression Motion Estimation no. 13 Summary: motion estimation Differential methods calculate displacement from spatial temporal differences in the image signal. Aperture problem: measurement window required. Differential methods require several iterations. computes error measure for cidate displacements finds best match. Speed up by search methods by clever application of triangle inequality. Hierarchical employs resolution pyramid. with sub-pel accuracy by interpolation of the image signal. Rate-constrained motion estimation using Lagrangian cost function. Bernd Girod: EE368b Image Video Compression Motion Estimation no. 14 7