EE 264: Image Processing and Reconstruction. Image Motion Estimation II. EE 264: Image Processing and Reconstruction. Outline

Transcription

1 Peman Milanar Image Motion Estimation II Peman Milanar Outline. Introduction to Motion. Wh Estimate Motion? 3. Global s. Local Motion 4. Block Motion Estimation 5. Optical Flow Estimation Basics 6. Optical Flow Estimation orn 7. Optical Flow Estimation Kanade 8. onclusions

2 3 Peman Milanar 5.Optical Flow Estimation Basics 4 Video Peman Milanar A ideo is a sequence o rames captured oer time Now our image data is a unction o space and time t

3 3 Peman Milanar 5 he Brightness onstanc Eqn Again I we assume that and are relatiel small and smoothl aring we can use alor epansion: t t Peman Milanar 6 Linearization hus we get: 0

4 7 Algebraic Meaning 0 A B Peman Milanar - + A Deriatie Deriatie B 8 Algebraic Meaning - Eample Peman Milanar D=- B A 4

5 9 Ill-Posed Algebraic Problem Peman Milanar We hae a single equation per each piel with unknowns more unknowns than equations! Additional inormation is required. Eample: We can assume that eer pair o piels share the same ector thus getting equations with unknowns. 0 Geometric Meaning B computing the gradient o we assume a local behaior o tilted plane. Peman Milanar - D D 5

6 Aperture problem Peman Milanar Aperture problem Peman Milanar 6

7 3 Foreground s. Background Peman Milanar Walking. hristo Neimann 4 Foreground s. Background Peman Milanar Not Reall. 7

8 5 he Aperture Eect Optical low ield is not alwas well-deined Peman Milanar 6 onounding Eects Peman Milanar Optical low ield does not alwas match motion ield 0 Intuitiel what does this constraint mean? he component o the low in the gradient direction is determined he component o the low parallel to an edge is unknown 8

9 7 Peman Milanar 8 Oercoming ill-posedness Peman Milanar In order to add more inormation to the sstem o equations we obtained we add an assumption o SMOONESS Actuall some sort o smoothness was alread assumed in the BME. 9

10 9 Peman Milanar Optical Flow Estimation he orn Algorithm 0 orn & Schunck - Basics Peman Milanar he basic idea is to deine a penalt unction that will hae the ollowing ingredients: erm rom the linearized BE equations erm enorcing spatial smoothness on the motion ectors. orn & Schunck approach ields an optimization problem to be minimized iteratiel. 0

11 Peman Milanar orn & Schunck - Penalt erm B A his equation should be written per each piel i.e. or M Peman Milanar orn & Schunck - Penalt erm M M M M M M M M B B B M A M A A Gathering all these equations into one sstem we get: Z

12 3 orn & Schunck - Penalt erm Peman Milanar he irst term in the penalt unction will be E Z his term orces the BE eistence! 4 orn & Schunck - Penalt erm Peman Milanar We can orce smoothness b adding a second term o the orm: E Z S S Where S is the Laplacian operator.

13 3 Peman Milanar 5 orn & Schunck - Minimization 0 S S Z E 0 S S Z E Z S S Z S S Peman Milanar 6 orn & Schunck - Iterations Appling the Steepest Descent algorithm which goes according to the local gradient we get: - he step-size o the SD algorithm j j j j j j j j Z S S Z S S

14 7 orn & Schunck - Algorithm Initialization: hoose 0 = 0 =0 j-th iteration: Update the result b* j j S S j Z j I S S Z and similarl or j. j Peman Milanar j * Multiplication b / /S is a simple operation 8 D=- orn & Schunck - Results Peman Milanar 00 Iterations =00 4

15 9 orn & Schunck Results Peman Milanar E Iteration 30 Peman Milanar Optical Flow Estimation he Kanade Algorithm 5

16 3 Lucas & Kanade - Basics Peman Milanar he basic idea is to orce smoothness b computing a single motion ector per block as in the BME. wo dierences rom the BME: Oerlapping blocks - eer piel gets a ector Using the linearized BE and not dierences hus per each piel we get a small linear set o equations to be soled directl. 3 Lucas & Kanade - Algorithm Peman Milanar For the piel we will use a neighborhood o 5 b 5 block around it. We need to gather the 5 releant equations o the orm: A B 6

17 7 Peman Milanar 33 Lucas & Kanade - Algorithm B A B A B A B A B A Peman Milanar 34 Lucas & Kanade - Algorithm B B B B B A A A A A Z V

18 35 Lucas & Kanade - Algorithm Peman Milanar We want to satis the equation set: Z V Using Least Squares: V opt ArgMin V Z V Z his result implies that we need to inert a b matri er eas! he condition number o relates to the conidence o the results and relects the local teture o the image. 36 Edge Peman Milanar gradients er large or er small large small 8

19 37 Low-teture region Peman Milanar gradients hae small magnitude small small 38 igh-teture region Peman Milanar gradients are dierent large magnitudes large large 9

20 39 Lucas & Kanade - Results Peman Milanar onidence Measure Estimated OF 40 Iteratie Reinement Peman Milanar Iteratie Lucas-Kanade Algorithm. Estimate elocit at each piel b soling Lucas- Kanade equations. Warp towards I using the estimated low ield - use image warping techniques 3. Repeat until conergence his lecture based on notes b G+W 40 0

21 4 Iteratie Reinement Peman Milanar Estimate elocit at each piel using one iteration o Lucas and Kanade estimation Warp one image toward the other using the estimated low ield easier said than done Reine estimate b repeating the process 4 Optical Flow: Iteratie Estimation Peman Milanar estimate update Initial guess: Estimate: 0 using d or displacement here instead o

22 43 Optical Flow: Iteratie Estimation Peman Milanar estimate update Initial guess: Estimate: 0 44 Optical Flow: Iteratie Estimation Peman Milanar estimate update Initial guess: Estimate: 0

23 45 Optical Flow: Iteratie Estimation Peman Milanar 0 46 Optical Flow: Iteratie Estimation Peman Milanar Some Implementation Issues: Warping is not eas ensure that errors in warping are smaller than the estimate reinement Warp one image take deriaties o the other so ou don t need to re-compute the gradient ater each iteration. Oten useul to low-pass ilter the images beore motion estimation or better deriatie estimation and linear approimations to image intensit 3

24 47 Optical Flow: Aliasing Peman Milanar emporal aliasing causes ambiguities in optical low because images can hae man piels with the same intensit. I.e. how do we know which correspondence is correct? actual shit estimated shit nearest match is correct no aliasing nearest match is incorrect aliasing o oercome aliasing: coarse-to-ine estimation. 48 Peman Milanar Reisiting the small motion assumption Is this motion small enough? Probabl not it s much larger than one piel nd order terms dominate ow might we sole this problem? 4

25 49 Reduce the resolution! Peman Milanar 50 oarse-to-ine optical low estimation Peman Milanar u=.5 piels u=.5 piels u=5 piels image u=0 piels image I Gaussian pramid o image Gaussian pramid o image I 5

26 5 oarse-to-ine optical low estimation Peman Milanar run iteratie L-K run iteratie L-K warp & upsample... image J image I Gaussian pramid o image Gaussian pramid o image I 5 Peman Milanar Quick our o Frequenc Domain Methods or Pure ranslational Motion 6

27 7 Peman Milanar 53 Motion Estimation in the -D Frequenc Domain Assume two gien rames: ompute the Normalized ross-spectrum: j e F F * * j j e F e F F F F F Peman Milanar 54 Motion Estimation in the -D Frequenc Domain j e M M M M Θ W Θ W W W ˆ

28 55 Peman Milanar Motion Estimation Using the 3-D Spectrum Gien man rames with tranlational motion: t 0 t t Recipe: works or multiple superimposed translational motions too ompute the 3-D Fourier ransorm Find planes with high energ concentration Equation o a line F t Fo t Equation o a plane 56 onclusions Peman Milanar here are numerous additional algorithms or motion estimation. Perormance has do to with compleit. Pramidal approach can be applied in order to reduce compleit and increase motion dnamic range. Work is still in progress in this ield. 8

29 57 Projection-Based Motion Estimation Note: Motion estimation is er computationall taing. Man applications require etremel eicient motion estimators. Real-time computer ision. Motion compensated ideo coding Goal: ransorm the image data into a lower dimensional space to speed up estimation. Solution: Deelop estimation algorithms using tomographic projections. Peman Milanar 58 Projections and the Radon ransorm he Radon ransorm: Peman Milanar r p p cos sin dd p r p Original Image Projected Image 9

30 59 Motion Under Projections ranslation in an image induces translation in the projections: 0 r p u 0 where u cos sin 0 0 Peman Milanar u 90 u 0 60 Local Motion Estimation Peman Milanar In local motion estimation the simple translation is estimated within blocks o the image sequence. 0 Image Sequence Region Motion -D Motion Estimation Vector Estimate ˆ 0 30

31 6 Peman Milanar Projection-Based Local Motion Estimation Estimate local motions using projections! 90 Projection Estimate 0 ˆ 0 -D Motion Vector For Each Block 0 Projection Estimate 0 ˆ 0 ˆ 0 6 Aine Motion Under Projection Peman Milanar -D Aine Model 0 M Aine motion under projection: u p 0 w w Mw p w cos sin 3

32 Mean Magnitude Error piels 63 Adantages o Using Projections Peman Milanar Reduced algorithmic computational compleit tpicall 5-0 aster. Generic eicienc improing structure. Minimal loss in perormance sometimes improed. 64 Eample o Projection-Based Method Peman Milanar 8 Mean Magnitude Error s SNR D h= -D h= -D h=3 -D h= SNR db Projection-based algorithms can een improe accurac! 3