Module 05 Motion Analysis / Overview. Advanced Video Analysis and Imaging (5LSH0), Module 05. Motion Applications / Video Coding (1)

Transcription

1 1 Module 05 Motion Analysis / Overview 2 Advanced Video Analysis and Imaging (5LSH0), Module 05 Visual Feature Extraction II: Motion Analysis Peter H.N. de With and Dirk Farin ( p.h.n.de.with@tue.nl ) A. Applications and understanding Example applications What is motion? B. Block-based motion estimation as used in MPEG-1/2 and H.264 video coding C. Parametric motion: from affine motion to 3-D projection as used in MPEG-4 sprite coding (background object) Introduction to elementary geometrical transforms. Mod 05 Feature Extraction II: Motion Analysis A. Example Applications and Understanding of Motion 3 Motion Applications / Video Coding (1) High correlation between temporally successive frames. Use motion-compensation to align same image content. Copy image content from previous images into current image to get a prediction image. Code residual image (difference between prediction and current image). 4 1

2 Motion Applications / Video Coding (2) 5 Motion Applicat s / Image registration (1) 6 Residual images without and with motion compensation Remember hybrid coding architecture from 5LSE0! Two images are captured from the same object at different times (e.g., satellite crossing the same area). Changes between images should be identified (changes are visible in residual image). input frame difference to last frame motion-compensated difference to last frame Images must be geometrically aligned. Motion Applicat s / Image registration (2) Medical application, e.g., display the position of blood vessels. Two X-ray images are taken: 1 background image, 1 image after injection of contrast fluid. Difference image shows only blood vessels. Key: the two images have to be aligned since the patient will move. 7 Motion Applic s / Motion segmentation (1) Main problem of segmentation problem is that computer has no semantic understanding. Which areas belong together? [replace with video] 8 2

3 Motion Applic s / Motion segmentation (2) 9 Motion Appl. s / Structure from Motion (1) Estimate the 3-D structure of objects from their motion 10 [FLOWERGARDEN VIDEO] J. Wang and E. Adelson, Representing Moving Images with Layers, IEEE Trans. Image Proc., Vol 3, No. 5, pp , Sep Marc Pollefeys Motion Applicat s / Image morphing (1) 11 Motion Applicat s / Image morphing (2) 12 sequence 1 sequence 2 [BLACK and WHITE video] 3

4 Motion Applicat s / Image morphing (3) sequence 1 13 Motion Applicat s / Image morphing (4) User defines manually important features in the images. Computer interpolates smoothly between these positions. 14 sequence 2 T. Beier: Feature-Based Image Metamorphosis. Computer Graphics (SIGGRAPH), Vol. 26, No. 2, July 1992 Mod 05 Feature Extraction II: Motion Analysis 15 Understanding motion / Motion types 16 Object motion vs. Apparent motion Object motion: real motion, not always visible in the image (rotating sphere). Motion Understanding and Description Apparent motion: observed motion, not always corresponding to real object motion (specular light reflections) Whenever we say motion, we mean apparent motion. Key: we only have the image/video available, not the truth! 4

5 Understanding / Motion field definition (1) Motion field describes movement of a position (x,y) in a first image to a position (x',y')=m(x,y) in a second image 17 Understanding / Motion field definition (2) Each parameterization for m(x,y) can represent a subclass of motions out of all possible motion-fields 18 Properties of motion-models: invertible How should the function m(x,y) be defined? composable Motion Description (1) / Dense motion (pixel-based) Define m(x,y) as a look-up table. Motion of every pixel recorded independently. good for analysis non-rigid motion (e.g., fluids) not good for video coding since coding of the motion-field is expensive Composable (ignoring discretization issues), but generally not invertible. Estimation with optical flow algorithms. B. Horn and B. Schunck: Determining Optical Flow. Artificial Intelligence, Vol. 17, pages , Motion Description (2) / Parametric motion Describe motion field with a small number of parameters. motion induced by camera movement rigid object motion Affine: Quadratic: Projective: Invertibility and composability depend on model. Projective motion will be discussed elsewhere. 20 5

6 Motion Description (3) / Parametric motion Translatorial motion: Scaling only: Rotation only: All together: Affine: Invertible (if {a ij } nonsingular) and composable. 21 Mot. Description (4) / Block-based motion Pixel-based flow fields are complex Parametric motion can only describe a limited set of motions. Subdivide image into blocks and describe motion of each block independently. Use simple motion-model for each block (e.g. translation). Compromise between ability to describe general motion-fields, efficiency to code motion Easy motion estimation. Used in MPEG video coding Motion Description (5) / Mesh-based motion Subdivide image with mesh (e.g., triangular faces). define motion of control points motion within face is interpolated from motion of control points Supports concept of general motion-fields Efficient to code motion information Invertible, (but not easily composable) Problem of Motion Estimation Basic assumption: brightness constancy constraint brightness of corresponding pixels in images I t and I t+1 are equal. Find motion parameters θ that minimize energy in motioncompensated residual image: Number of parameters depends on motion model Can be simple: translation of subimage with 2 parameters only Can be complex: pixel-based motion with 2 parameters per pixel! 24 6

7 Motion Estimation / Measurement Problems Aperture problem How far in neighborhood? 25 Mod 05 Feature Extraction II: Motion Analysis 26 Repetitive patterns To which location? B. Block-based Motion Occlusions Motion of covered/uncovered areas? Block-Based Motion / Principle Partition image into small blocks B (e.g., 16x16) Each block is moved with a simple translation. Translation of a block is specified with motion vector (dx,dy). 27 Block-based motion / Description vector 1 Subdivide image into regular blocks. Search matching block in previous frame. 28 Common misunderstanding: Motion-vector defines where a block in the predicted frame comes from, not where it will move! (Important since it is not invertible.) Block-based motion is not invertible and not composable. 7

8 Block-based motion / Description vector 2 Complex motion is approximated 29 Block-Based Motion / Examples (1) Panning motion 30 reference frame prediction frame Notice zero-vectors in homogeneous areas Block-Based Motion / Examples (2) Fast motion, displacement larger than search-range. 31 Block-Based Motion / Estimation Principle Since blocks get independent parameters, they can also be estimated independently. Find (dx,dy) by minimizing the residual image energy (Mean Square Error): 32 or the Sum of Absolute Differences: Notice vectors along line 8

9 Estimation Error forms Surface 33 Block-based motion Algorithm 1: Full Search (FS) 34 Define a search window Compute E(dx,dy) for every possible dx,dy. Select dx,dy that gives minimum E(dx,dy). Finds optimal solution (not necessarily true motion). Computation intensive! Block-Based Motion / Comparison of FS Results 35 Block-Based Motion / Distribution of Motion-Vectors 36 full search foreman sequence stefan sequence 9

10 Block-based motion Algorithm 2 (1) One-Dimensional Full Search (1DFS) Idea: most motion is pure horizontal or vertical. 37 Block-based motion Algorithm 2 (2) One-Dimensional Full Search (1DFS) 38 Step 1: search pure horizontal motion. Step 2: search pure vertical motion (from best hor. motion). Step 3 & 4: repeat first two steps with half search range from new position. Block-based motion Algorithm 2 (3) One-Dimensional Full Search (1DFS) 39 Block-based motion Algorithm 2 (4) One-Dimensional Full Search (1DFS) 40 Algorithm can also be understood as iterative optimization over an alternating subset of the parameters. Works best if error E is decomposable as E(dx,dy)=f(dx)*g(dy) ( f and g unknown of course) Error isolines: Since searching for x,y together is too complex, hence: keep y fixed, search x, then keep x fixed, search y,... fast convergence case slow convergence case 10

11 Block-Based Motion / Comparison of FS and 1DFS Results 41 Block-based motion Algorithm 3 (1) Gradient Descent (GD) 42 Idea: error function is smooth. Explore neighborhood around current position. Proceed into direction of decreasing error. full search 1-D full search Block-based motion Algorithm 3 (2) Gradient descent (GD) Start at (0,0), compute E in 3x3 neighborhood. 43 Block-based motion Algorithm 3 (3) Gradient descent (GD) 44 If minimum E is in the center, a local minimum has been found => end the search, else: take new minimum as new search center and iterate. Fast for small motions Gets trapped in local minima (especially in textured regions) 11

12 Block-Based Motion / Comparison of FS and GD Results 45 Block-based motion Algorithm 4: Three Step Search (TSS) 46 Idea: error function is smooth. Search on a coarse scale first and then refine to higher accuracy. full search gradient descent Step 1 Step 2 Step 3 Block-Based Motion Comparison of FS and TSS Results 47 Block-based motion Algorithm 5: Predictive Search (1) 48 Idea: motion of neighboring blocks is similar. full search three-step search Start search not at (0,0), but at a vector position computed from the prediction vectors. Often, a median vector is computed: 12

13 Block-based motion Algorithm 5: Predictive Search (2) Predictor lock problem: Along motion discontinuities, the prediction is wrong (e.g., between foreground/background regions). Additionally to prediction vector, also test zero-vector. 49 Block-based motion Algorithm 6 (practical): MVfast - (1) Idea: combination of several techniques. Classify the motion-type depending on previously computed vectors into three classes (low, medium, large). Compute maximum magnitude M of the prediction vectors. Classify according to => low => medium => large and use a different estimation algorithm for each class. 50 Block-based motion Algorithm 6 (practical): MVfast - (2) 51 Block-based motion Algorithm 6 (practical): MVfast - (3) 52 For the area at: Low motion speed Medium motion speed start search at (0,0) and carry out gradient-descent seach with small diamond pattern Start GD search at (0,0), but use large diamond pattern. Search pattern GD-step to the left best match in corner => proceed best match at edge => proceed best match in center => refine and stop 13

14 Block-based motion Algorithm 6 (practical): MVfast - (4) Large motion speed Also use large diamond pattern. Start GD search at median prediction vector position. 53 Block-based motion Algorithm 7: Successive Elimination Search (1) Idea: establish lower bound on the SAD error. Do not test positions that cannot have a lower error. We start with the triangle inequality: Setting gives or 54 f() first image g() second image x,y coordinate in first image x',y' corresponding position in second image Block-based motion Algorithm 7: Successive Elimination Search (2) 55 Block-based motion Algorithm 7: Successive Elimination Search (3) 56 This gives a lower bound for the SAD error Sum over blocks can be computed efficiently by first computing cumulative sums With being the best match so far, we do not have to compute the SAD if Now compute sum over arbitrary area simply as mean value over f mean value over g 14

15 Block-based motion Algorithm 7: Successive Elimination Search (4) 57 Block-based motion Algorithm 7: Successive Elimination Search (5) 58 Cumulative sums can be computed efficiently with an iterative summation scheme Pass 1: A tighter bound leads to more exclusions. Better bound by subdividing block into smaller blocks. Pass 2: More subdivisions lead to tighter bound, but increase the complexity for the exclusion test. Optimum for 16x16 blocks are 4 sub-blocks. Block-based motion Algorithm 7: Successive Elimination Search (6) SAD Computation can be stopped when it exceeds 59 Block-based motion Algorithm 7: Successive Elimination Search (7) Let us denote our error approximation as 60 No large gain if naive summation is used. Better: start with difference of block means and successively approximate the block SAD. The difference between the SAD (E) and our approximation e is (for e<0, the case e>0 is similar): Hence, we can successively approximate the SAD: 15

16 Block-based motion Algorithm 7: Successive Elimination Search (8) Preferred search order. Start with vector that probably gives good SAD bound. 61 Summary of block-based ME algorithms Full-Search: enumeration of all parameter values. 1D-FS: alternate the (x, y) parameters to search. Gradient-Descent: assumes smooth error function without local minima. Three-step-search: search from coarse to fine sampling. Predictive: use context information to start with a better approximation. MVFast: classify the expected problem and use an adapted search algorithm for each class. Successive elimination: exclude parts of search space. 62 Block-Based Motion: Comparison of Results 63 Block-based motion Extensions: sub-pel accuracy (1) 64 full search three-step search 1-D full search gradient descent Object motion is not always a multiple of integer shifts. More accurate motion-estimation requires sub-pixel resolution. MPEG-1/2: half-pel Pixel value at half-pel positions obtained with bi-linear interpolation. MPEG-4 / H.264: quarter pel Half-pel positions using a 8-tap interpolation filter. Quarter-pel position using bi-linear interpolation from half-pel positions. 16

17 Block-based motion Extensions: sub-pel accuracy (2) Error at sub-pel positions is generally lower, as the image is smoothed and noise is filtered Hence, first search on full-pel resolution and then add a half-pel refinement step. 65 Block-based motion Extensions: multiple references (1) Compute motion-vector to several frames and combine the predictions. 66 Block-based motion - Extensions: multiple references vs. accuracy (2) What is better: multiple ref ces (N) or increase accuracy? 67 Block-based motion - Extensions: Variable block size (H.264) Fixed block-size does not adapt to image content. If two objects are visible in one block, their motion cannot be represented with just one motion vector. 68 B. Girod: Why B-Pictures Work: A Theory of Multi-Hypothesis Motion- Compensated Prediction. Int. Conf. on Image Proc., Oct H.264 block sub-divisions 17

18 Block-b. motion / Concluding Remarks Block-motion is a simple approach, but still the most frequently-used technique for video-coding (even for wavelet coders). can approximate any motion easy estimation Block motion can also be used as first step to estimate parametric motion from these vectors. The presented estimation principles can be reused for many other tasks than motion estimation. Parametric Motion Estim. approaches real motion better. 69 Mod 05 Feature Extraction II: Motion Analysis C. Parametric Motion: from affine motion to 3D 70 Parametric Motion / Motivation 71 Parametric motion / System aspects 72 MPEG-4 camera motion compensation. Motion in the image is usually composed of camera motion, rigid object motion, and non-rigid object motion. Camera motion model required to build background sprite. [ insert MPEG-4 stefan sequence 2 ] The camera and rigid objects cannot move arbitrarily. Their motion can be described with some parameters derived from the geometrical setup. We will describe the observed image motion in terms of the 3-D object motion (we do not know what happened..). 18

19 Affine 2D Coordinate Transform (1) 73 Affine 2D Coordinate Transform (2) 74 Express coordinate frame (x,y) in coordinate frame (x',y'). In motion estimation: (x,y) could be the coordinate in the last frame, and (x',y') the corresponding coordinate in the current frame. Special (simple) case 1: translation only A is identity matrix I Affine 2D Coordinate Transform (3) 75 Affine 2D Coordinate Transform (4) 76 Special case 2: scaling only Special case 3: rotation only scaling in x direction scaling in y direction constraint 1: size stays constant constraint 2: coordinate axes are perpendicular Isotropic scaling (rigid motion) A is an orthonormal matrix (rotation matrix R) 19

20 Affine 2D Coordinate Transform (5) 77 Affine 2D Coordinate Transform (6) 78 Rotation matrix for rotation by an angle α Image skew (horizontal) rotation by -α Affine 2D Coordinate Transform (7) 79 Affine 2D Coordinate Transform (8) 80 Concatenation of transforms Computing the inverse of gives gives (the mathematical trick will make this much easier, too) Inverse is possible if A is nonsingular. (Shortly we will introduce a trick to make this much easier) Set of nonsingular affine transforms constitutes a group. 20

21 Affine 2D Coordinate Transform (9) 81 Affine 2D Coordinate Transform (10) 82 Here the trick: augment each point-coordinate with a dummy '1': Concatenation is now very easy (A i are 3x3 matrices) Now, write affine transform as simple matrix multiplication: And the inverse transform is simply the 3x3 matrix inverse. This useful trick will reappear later... Affine 2D Coordinate Transform Example: Transform Sequence (1) What is the transformation between these two images? 83 Affine 2D Coordinate Transform Example: Transform Sequence (2) Break the transformation into sequence of elementary steps. 84 shift to origin rotate shift back 21

22 Affine 2D Coordinate Transform Example: Transform Sequence (3) 85 Affine 3D Coordinate Transform (1) 86 Successive steps: Straight-forward extension to affine 3-D motion Combined transform: Multiplied together: Affine transform / Rotation in 3D Space (1) Elementary rotation matrices: around z-axis around y-axis around x-axis 87 Affine transform / Rotation in 3D Space (2) Euler's rotation theorem: Every 3D rotation can be decomposed into a sequence of three rotations around predefined axes. 88 For example, choose rotation sequence Note: points on rotation axis do not move. The inverse is the transpose: First rotate around z-axis, then x-axis, and finally y-axis. We can describe all 3D rotations by three angles α,β,γ. Rotations are NOT commutative! 22

23 Affine transform / Rotation in 3D Space (3) Inverse of a general 3-D rotation R Decompose R into a sequence of elementary transforms. It follows that 89 Affine transform / Rotation in 3D Space (4) illustration of the rotation sequence 90 and Consequently, the inverse of R is R is orthonormal and Affine transform / Rotation in 3D Space (5) Gimbal lock position. For α=90 degrees, the y and z axes align! Rotation around y can be canceled by rotation around z. Specifying rotation with Euler angles is not unique! 90 degrees same effect 91 3D to 2D Perspective Projection (1) Projection of a 3-D point onto a 2-D image plane simplified pin-hole camera model pin-hole is optical center of the camera for mathematical simplicity (less minus signs), image is in front of pin-hole instead of behind -- focal length f is distance of image plane to optical center 92 23

24 3D to 2D Perspective Projection (2) 93 3D to 2D Perspective Projection (3) 94 Projection of a 3-D point onto a 2-D image plane the intercept theorem gives Problem: projection is a non-linear operation, complicating the mathematical formulation. all points on the ray from the origin to p are Solution: do not use Euclidean geometry. projected to p' References (Block-Matching) 95 References (other) 96 J. Mitchell, W. Pennebaker. MPEG Video Compession Standard, Chapter 13: Motion Estimation. Kluwer. TSS: T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro, Motion-compensated interframe coding for video conferencing. Proc. NTC 81, pp. C , Dec TSS: R. Li, B. Zeng, M. Liou. A New Three-Step Search Algorithm for Block Motion Estimation. IEEE Trans. Circuits Systems Video Technology, Vol. 4, No. 4, Aug GS: L. Liu, E. Feig: A Block-Based Gradient Descent Search Algorithm for Block Motion Estimation in Video Coding, IEEE Trans. Circ. Sys. Video Techn., Vol. 6, No. 4, DFS: M. Chen, L. Chen, T. Chiueh: One-Dimensional Full Search Motion Estimation Algorithm For Video Coding, IEEE Trans. CSVT, Vol. 4, No. 5, Oct SEA: M. Brünig, B. Menser: Fast Full Search Block Matching using Subblocks and Successive Approximation of the Error Measure. SPIE Image and Video Comm. & Proc., Vol. 3974, pp , PMVFAST: A. Tourapis. O. Au, M. Liou: Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) Enhancing Block Based Motion Estimation, SPIE VCIP, Vol. 4210, pp , C. Stiller, J. Konrad. Estimating Motion In Image Sequences: a tutorial on modeling and computation of 2D motion. IEEE Signal. Proc. Magazine, Vol. 16, pp , July B. Horn and B. Schunck: Determining Optical Flow. Artificial Intelligence, Vol. 17, pp , 1981 B. Girod: Why B-Pictures Work: A Theory of Multi-Hypothesis Motion-Compensated Prediction. Int. Conf. on Image Proc., Oct J. Wang and E. Adelson: Representing Moving Images with Layers. IEEE Trans. on Image Processing, Vol. 3, No. 5, pp , Sep T. Beier: Feature-Based Image Metamorphosis. Computer Graphics (SIGGRAPH), Vol. 26, No. 2, July