Precise Multi-Frame Motion Estimation and Its Applications

Transcription

1 Precise Multi-Frame Motion Estimation and Its Applications Peyman Milanfar EE Department Uniersity of California, Santa Cruz Joint wor with Dir Robinson, Michael Elad, Sina Farsiu

2 Motiating Application: Resolution Enhancement from Video The Idea: Diersity + Aliasing Gien multiple low-resolution moing images of a scene (a ideo), generate a high resolution image (or ideo).

3 Motiation Application: (Hollywood Version!)

4 Practical Motiation: (Real Video Enhancement) Motion Estimation Image Reconstruction Resolution enhancement x4 from ideo frames captured by a commercial webcam (3COM Model No.3719)

5 Problem: Gien a pair of frames, we want to estimate the translation Typical Assumptions: Sampled on a finite grid. Sampled aboe Nyquist rate Will discuss aliased case later. Additie white Gaussian Noise Translational Motion Estimation ), ( ), ( ), ( ), ( ), ( ), ( x x e x x f x x f x x e x x f x x f + = + = = 2 1

6 Translational Motion Estimation Optimum Statistical Estimator: Max. Lielihood Correlation Methods max f1( x1 1, x2 2) f2( x1, x2 Direct Maximization 1, 2 x1, x2 Phase-Correlation Nonlinear Least Squares min ( f1( x1 1, x2 2) f2( x1, x2) ) 1, 2 x1, x2 Gradient-Based algorithms Pyramid-Gradient-Based algorithms Direct Minimization Improing to subpixel accuracy Fits a quadratic about the pea of the correlation surface. Gauss-Newton methods, iterated improement Iterating oer scale: pyramid-based methods ) 2

7 Performance Limits in Image Registration How close to the limit are typical methods? Image used At 5 db Independent of underlyling elocity ector Bias

8 Effect of Aliasing How does aliasing affect the ability to estimate translation between sets of images? Little aliasing Lot of aliasing Note false motions.

9 Performance of Aliased Image Obserations: Registration Very little wor addressing registration of aliased image. Performance bound depends on the motion parameters (not true for non-aliased registration) Traditional algorithms designed for nonaliased scenario will fail.

10 Data and Formulation Consider a sequence of noisy, translating images oer time. { } f f = ( 1 1, ) Translate f, + error 1,2 f 1 f 2 2,3 Frame-to-frame motion ectors f N Image formation model: f = Sample [ f ( x, y, t )* h( x, y)] + noise Aliasing Point-spread function

11 Registration of Multiple Video Frames Motion Problem: Gien the frames, estimate ectors Implicit problem: Estimate underlying high resolution image { } 1, Desired unnowns f f = ( 1 1, ) Translate f, + error = Sample [ f ( x, y, t )* h( x, y)] + noise Nuisance Parameter

12 Fusion of Multiple Video Frames Reconstruction Problem: Gien the frames, estimate the high resolution image. (Superresolution) f ( x, y, t) Implicit problem: Estimate the motion ectors Nuisance Parameters f f = ( 1 1, ) Translate f, + error = Sample [ f ( x, y, t )* h( x, y)] + noise Desired unnowns

13 How well can the problem be soled? Fisher Information Matrix (FIM) is partitioned on the motion parameters { 1, } and the high-res (alias-free) atlas image f. Registration Information J J ( ) f f = J }, { 1, T f J J ff Information Correlation Reconstruction Information J J ff - Depends on the set of motions (sampling offsets) and the amount of texture energy in the signal - Depends only on the set of motions

14 CRB for Aliased Image Registration Using Schur decomposition, the CRB for aliased image registration is: Co ({ }) ( 1 T ) J J J J 1 j, f ff f Registration Information Information Loss due to uncertainty about the high resolution image. With just a pair of aliased images, the FIM becomes singular, hence pairwise registration of aliased images is essentially impossible

15 Registering Sets of Images CR Bound (per frame) for multi-frame image registration. Registration CRB (pix/frame) M = 1 M = 2 M = 3 M = 4 M = 5 More aliasing Number of Frames (K+1) No aliasing

16 What to do? Almost all motion estimation algorithms today deal with the case of only two (consecutie) frames at a time. These methods are far from optimal. Proposal: Use multiple frames simultaneously, with care! Pairwise estimation ( Progressie ) Fixed reference estimation ( Anchored )

17 Constraints on Translational Motion Vectors Across Time i, Frame i Frame j Frame i, j j, i,, j i, i = = 0 i, j = + j, j, Linear set of constraints imply that the motion ectors lie in a subspace.

18 An Algorithm Thus Motiated minimize { j } j, f Translate (f j, j, ) p p Can be any penalty function subject to i, = i, j + j,, j = j, With p=2, and linear constraints, we hae a quadratic programming problem. Computationally simpler (but suboptimal) is to project any estimated parameters onto the constraint subspace. With p=1, we can hae a more robust solution.

19 Performance RMSE(Pixels) 10 1 Single Projected p=2, Constrained p=1, Constrained SNR (db)

20 Application to Simultaneous Demosaicing and Resolution Enhancement Bayer Filtered Motion Sequence Single-Frame Demosaicing Image fusion Hi-resolution Demosaicing... OLD NEW

21 Example with Real Data Registering the color-filtered data is nonstandard and difficult in practice. 27 Raw CFA Images

22 Example with Real Data Standard Single-frame demosaicing

23 Example with Real Data Multi-frame Progressie Registration

24 Example with Real Data Multi-frame Anchored Registration

25 Example with Real Data Multi-frame Projected Registration Vectors

26 Example with Real Data Multi-frame Constrained Registration

27 Nice Algebraic Structure { } S j, = Satisfies i,, j i, i = = 0 i, j = + j, j, S is closed under + S is associatie S has an identity element : 0 Eery element of S has an inerse Set of all pairwise motions S between frames is a GROUP

28 A Few Words on Affine Motion Constraints ( M i, Ti,, ) Frame i Frame j Frame (, M i, j, Ti j ) M j, T ) (, j,

29 General Comments The constraints for the affine case are nonlinear. But the algebraic structure persists. Group operation is no longer simple ector addition Algebraic structure also for dense optical flow Here the elements of the algebra (motions) are defined by nonlinear transformations applied to images. Lie Algebra defined by the composition of operators. If operators are differentiable, then a Lie Group.

30 Conclusions Accurate motion estimation is a (ery) hard problem. Registering a pair of aliased frames is an ill-posed problem. Using simultaneous image registration and reconstruction is one possible solution. Motion estimation with constraints (hard or soft) is another alternatie. There are many applications.