Global motion model based on B-spline wavelets: application to motion estimation and video indexing

Size: px

Start display at page:

Download "Global motion model based on B-spline wavelets: application to motion estimation and video indexing"

Lisa Rose
5 years ago
Views:

1 Global motion model based on B-spline wavelets: application to motion estimation and video indexing E. Bruno 1, D. Pellerin 1,2 Laboratoire des Images et des Signaux (LIS) 1 INPG, 46 Av. Félix Viallet, Grenoble Cedex, France 2 ISTG, Université Joseph Fourier, Grenoble, France eric.bruno, denis.pellerin@lis.inpg.fr Abstract This paper describes a framewor to estimate global motion model based on B-spline wavelets. The wavelet-based model allows optical flow to be recovered at different resolution levels from image derivatives. By combining estimation from different resolution levels in a coarse to fine scheme, our algorithm is able to recover a large range of velocity magnitude. Our algorithm is evaluated on artificial and real image sequences and provides accurate optical flows. The wavelet coefficients of the model at low resolution levels also provide features to index video or to recognize activities. Preliminary experiments on video indexing based on coarse wavelet coefficients are presented. As an example, we have considered video sequences containing different human activities. Wavelet coefficients are efficient to give a database partition related to the ind of human activities. 1 Introduction Motion estimation is a fundemental problem in video analysis. The nowledge of motion is required in many applications, such as video compression, video indexing, activity recognition or 3D reconstruction. Parameter models of optical flow aid in estimation by enforcing strong constraints on motion spatial variation within a region or over the whole image. Many authors have used affine or quadratic basis set [2, 8] to model motion. These models are very effective to relate planar transformation but fail when there are two or more different moving regions in the image sequence. These methods generally aim at determining a partition of the moving regions and then applying motion model on each region. Other wors in activity recognition use motion specificities to generate basis functions. In [3], Blac et. al model motion by linear combination of eigenvectors computed from a training set of flow fields (motion discontinuities and non-rigid mouth motion). These methods are very effective to model velocity fields close to the training set but cannot be used to estimate any motions. In our previous wor [4], global motion was expressed as a Fourier series expansion. Such a model allows a large range of motion to be approximated on a few harmonic basis functions. Following the same principle, we now use a set of B-spline wavelet functions. Wavelet series expansion is better than Fourier analysis to approximate piecewise smooth functions, such as optical flow. Then using B-spline wavelets, which are smooth, symmetrical and have a compact support, improves motion estimation accuracy. In addition, model parameters provide a concise description of global motion. Srinivasan and Chellapa have recently proposed to model global motion by a set of overlapped cosine windows [10]. Their approach is close to our algorithm. Our wor can be also compared to Szelisi s approach [11], which is based on a sparse velocity vector field estimation interpolated by spline functions. This paper is organized as follows: Section 2 outlines the problem of global motion model estimation from the image sequence brightness function and presents the motion model based on B-spline wavelets. Section 3 details the model parameter robust estimation based on a sparse system inversion. This procedure is then embedded in a coarse to fine scheme so as to handle a large range of motion magnitude. Section 4 presents experimental results on estimated motion model accuracy and a performance comparison with the Srinivasan and Svelisi s algorithm. This section also displays some preliminary experiments on video indexing based on motion wavelet coefficients.

2 2 Global Motion model 2.1 General remars on motion model estimation Let us consider an image sequence I(p i, t) with p i = (x i, y i ) Ω the location of each pixel in the image. The brightness constancy assumption states that the image brightness I(p i, t + 1) is a simple deformation of the image at time t: I(p i, t) = I(p i + V(p i, t), t + 1) (1) V(p i, t) = (u, v) is the optical flow (also named velocity field or global motion) between the two frames I(p i, t) and I(p i, t + 1). V(p i, t) is also defined over Ω. This velocity field can be globally modeled as a linear combination of basis functions: V(p i, t) = N c φ (p i ) with c = (c x, c y) T (2) =0 where φ (p i ) are basis functions. The motion parameter vector θ = [c 0... c N ] T is estimated by minimizing an objective function [8]: E = p i Ω ρ (I(p i + V(p i, t), t + 1) I(p i, t), σ) = p i Ω ρ (r(p i + V), σ) (3) where ρ(, σ) is a robust norm error (or M-estimator) applied to the difference of warped images r(p i + V) = I(p i + V(p i, t), t + 1) I(p i, t). A robust estimation is needed to reduce outlying data weight in the estimation process. Then, the solution for θ is given by: θ = argmin θ (E) (4) The success of the minimization stage depends on the model s ability to fit the real global motion between I(t) and I(t + 1). Wavelet basis functions are reputed to give the best approximation of piecewise smooth functions, such as optical flow, and are an effective solution to solve our problem. 2.2 Wavelet series expansion Any function f(x) L 2 (R) can be expanded as a weighted sum of basis functions: f(x) = c l, φ l, (x) + d j, ψ j, (x) (5) j l φ j, (x) = φ(2 j x ) and ψ j, (x) = ψ(2 j x ) are respectively the scaling and the wavelet functions, dilated and translated by at a level j. Be V j, with j Z, a set of closed subspaces of L 2 (R). The scaling functions {φ j, (x)} are a basis for V j. V j contains all functions of L 2 (R) at a resolution level j. The new details at level j are represented by the wavelet set {ψ j, (x)} which is a basis for W j. Then, the subspace V j+1 is defined as: V j+1 = V j W j V j V j+1 (6) Relation (6) leads to: c j+1, φ j+1, (x) = c j, φ j, (x) + d j, ψ j, (x) (7) Estimation of the {c j+1, } coefficients provides also scaling and wavelet coefficients for all lower resolution levels. Then the approximation of f(x) at level j could be expressed as a linear combination of scaling functions: f j (x) = c j, φ j, (x) (8) We want to model the velocity field V(x, y) by using two-dimensional scaling basis functions. A natural extension of one-dimensional to two-dimensional scaling basis function is: Φ j,1, 2 (x, y) = φ(2 j x 1 )φ(2 j y 2 ) (9) The subscripts j, 1, 2 represent respectively the resolution scale, the horizontal and the vertical translations. The global motion could then be approximated at a resolution scale j by: V j (p i, t) = 2 j 1 1, 2=0 2.3 B-spline basis functions c j,1, 2 Φ j,1, 2 (p i ) (10) So as to recover a smooth and regular optical flow, the scaling functions must be as smooth and symmetrical as possible. The B-spline wavelet has maximum regularity and symmetry, and is a good candidate to model motion. The B-spline function of degree N 1 is the convolution of N box functions: 0 (x) = (B B... B)(x) (11) with B = ( 1 2, 1 2). The B-spline scaling function at the resolution level j + 1 is defined by the dilatation equation: j+1 (x) = 21 N N =0 ( N ) j (2x ) (12)

0 0.5 1 1.5 2 2.5 3 3.5 4 0.12 error quantity: 0.1 0.08 0.06 0.04 Ẽ(V + δv) = ρ (r(p i + V) + δv I(p i + V, t + 1), σ) p i Ω (14) 0.

The M-estimation problem can be converted into an iterated reweighted least squares [7]: Ẽ(V + δv) = w(r(p i + V)) (r(p i + V) + δv I(p i + V, t + 1)) 2 p i Ω (15) (b) Φ 1 (x) (c) Φ 2 (x) (d) Φ 3 (x)

3 error quantity: Ẽ(V + δv) = ρ (r(p i + V) + δv I(p i + V, t + 1), σ) p i Ω (14) (a) φ 1,2,3 where I = [I x, I y ] T represents the spatial derivatives of I. The M-estimation problem can be converted into an iterated reweighted least squares [7]: Ẽ(V + δv) = w(r(p i + V)) (r(p i + V) + δv I(p i + V, t + 1)) 2 p i Ω (15) (b) Φ 1 (x) (c) Φ 2 (x) (d) Φ 3 (x) Figure 1. a) 1-dimensional B-spline scaling functions of degree 1, 2 and 3 and, b,c,d) the corresponding 2-dimensional scaling functions with w(x) = 1 x Tucey: { ρ(x) = σ 2 ρ(x) x. The ρ-function is the biweight of 6 (1 (1 ( x σ )2 ) 3 ) if x σ σ 2 6 if x > σ (16) Using global motion model defined in (10) for V and δv at a resolution level j, the robust error in (15) becomes with matrix notation: Ẽ(V + δv) = W (M j δθ j + B) T (M j δθ j + B) (17) 0 (x) is supported on [0, N]. Since we wor on a finite sampled signal, the scaling function φ0 N 1 (x) is mapped into the signal support. The two-dimensional basis functions are computed at different resolution levels using the product (9). Figure 1 represents the 1D B-spline functions of degree 1, 2 and 3 and the corresponding 2D scaling functions used to model motion. 3 Global Motion estimation 3.1 Incremental robust estimation To estimate global motion V, the robust error E(V) defined in (3), has to be minimized. This step is achieved by using an incremental scheme. E(V + δv) = p i Ω ρ (r(p i + V + δv), σ) (13) Given an estimate V (initially zero), the goal is to estimate δv that minimize equation (13). The first order Taylor series expansion of I(t+1) with respect to δv provides a new with, for 1 2 = [ j 1] 2 and p i Ω (N = (2 j ) 2 and M = card(ω)): M j = [I x (p i + V)Φ j,1, 2 (p i ), I y (p i + V)Φ j,1, 2 (p i )], M j R M 2N δθ j = [δc j,1, 2 ] T, δθ j R 2N W = [diag(w(p i + V))], W R M M B = r(p i + V), B R M (18) The minimum of (17) with respect to δθ j leads to the linear system: M T j W M j δθ j = M T j W B (19) and we obtain the solution: δθ j = ( M T j W M j ) 1 M T j W B (20) The matrix M T j W M j could be large when the resolution level j is high, and it is time and memory space expensive to compute its inverse with numerical techniques. Hopefully, because we use localized scaling functions (Fig. 1) as basis functions, system (19) is sparse. Figure 2 represents zero and nonzero entries of M T j W M j for the 3 first degree B-spline scaling functions at level j = 4. Note that sparsity depends on the spatial extent of the scaling function used.

(a) (b) (c) (a) Frame from Yosemite sequence (b) Real flow field (c) Estimated flow field Figure 2. M T j W M j matrix for B-spline scaling function of degree a) 1, b) 2 and c) 3 at level j = 4.

4 (a) (b) (c) (a) Frame from Yosemite sequence (b) Real flow field (c) Estimated flow field Figure 2. M T j W M j matrix for B-spline scaling function of degree a) 1, b) 2 and c) 3 at level j = 4. White and blac regions represent zero and non-zero entries, respectively. δθ j is then iteratively estimated in (20) using the generalized minimum residual method [9] which is effective and fast to solve such a sparse system. The first step of the reweighted least squares consist in obtaining a first estimate δθj 0 with the weight matrix W equal to the identity. New weights are next evaluated and a new estimate δθj 1 is obtained. This process is repeated until the incremental estimate δθj i is too small or after a predefined number of iterations. Then, the motion parameters vector at level j is: θ j = δθ j (21) Figure 3. Optical flow result. The optical flow estimated is modeled by the B-spline basis functions of degree 1 with a coarse to fine estimation from the level 2 to Performances on synthetic and real image sequences We test our algorithm on both synthetic and real sequences. The angular error [1] is used to evaluate our estimation when the true motion field (u r, v r ) is nown (in Yosemite sequence (Fig 3.a) downloaded from Barron et al. s FTP 1 ). This angular error is defined at each location by: ( ) uu r + vv r + 1 α e = arccos u2 + v (22) u 2 r + vr Coarse to fine estimation At coarse resolution level, the wavelet coefficients are estimated on large image regions, and can recover large but coarse motion. So as to provide an accurate global motion estimation for large and fine displacements, the incremental scheme is embedded in a coarse to fine refinement algorithm. A first estimate θ l is obtained at the coarsest level l. Then θ l is transmitted to a finer level where a new incremental estimation is done. This is repeated until the finest level L is reached. Then, the final motion parameter vector θ L is estimated, and contains all wavelet coefficients that describe global motion V L at resolution level L. 4 Results The global motion parameter vector estimated not only provides an optical flow estimation, but also describes the sequence activity by means of some wavelet coefficients. In this section, we present our results on motion estimation accuracy and some results on video indexing based on wavelet coefficients. This provides a degree value which taes care of velocity modulus and orientation errors. Table 1 compares the angular error average and standard deviation for B-spline scaling functions of degree 1, 2 and 3. The motion models estimated with the different B-splines are close to the real optical flow. The B-splines of degree 2 and 3 perform slightly better than the B-spline of degree 1 but for a larger computational time. Whatever the degree of the B-spline used, our algorithm outperforms the approach of Srinivasan and Chellappa [10], who use overlapped cosine windows to model global motion. With a lower density of velocity vector estimated, the Szelsii and Coughlan s algorithm performs slightly better. The next example, Baltrain sequence (Fig 4.a), is a real sequence where motion is more complex than Yosemite. Figures 4.b), c) and d) display the estimation scheme progress across the resolution levels. Estimated optical flows are close to perceptual view, whereas they are just defined by a small number of parameters (only wavelet coefficient vectors at level 4). To sum up, a motion model based on B-spline wavelets is efficient to approximate on a very few parameters the global optical flow. Actually, with only 16 basis functions (j = 2) 1 ftp.csd.uwo.ca

5 (a) Frame from baltrain sequence (b) Estimated flow field at j = 2 (c) Estimated flow field at j = 3 (d) Estimated flow field at j = 4 Figure 4. Frame from Baltrain sequence and global motion estimated at level a) 2, b) 3 and c) 4. B-spline of degree 3 were used to model motion. B-spline Level Avg. Std Density Φ 1 j = o 9.6 o 100% j = o 8.5 o 100% j = o 9.7 o 100% Φ 2 j = o 9.6 o 100% j = o 7.5 o 100% j = o 9.0 o 100% Φ 3 j = o 10.7 o 100% j = o 8.0 o 100% j = o 8.7 o 100% Srinivasan [10] 8.9 o 10.6 o 100% Szelisi [11] 3.1 o 7.6 o 39.6% Table 1. Motion estimation angular errors (average and standart deviation) for Yosemite sequence, using different B-spline scaling functions. The coarse to fine scheme was performed from the resolution level j = 2 to j = 4 it is possible to have a coarse motion estimation and then information about motion activities along a video sequence. 4.2 Video indexing based on wavelet coefficients In a motion based video indexing problem, the first stage is to define relevant motion features. These features have to contain enough information to distinguish activities, and to be coarse enough to be invariable to close motion activities. We will show in this section that a wavelet model estimated at coarse levels could provide relevant motion features. In the following, sequences are considered as elementary video shots Definition of the motion features Wavelet components extracted from different sequences provide for each frame i of sequence S a motion parameter vector θ i = [c 1... c N ] T R 2N. The motion-based feature vector associated to the sequence S of M frames is then defined as the center of gravity of all reduced motion parameters vectors of S. Θ S = 1 M M θi T σ 1 (23) i=1 where σ is a diagonal matrix which contains standard deviation of θ components over the whole sequence. Computing Θ S for K videos provides a motion-based feature space : Ω = {Θ Si }, i = 1... K (24) The dimension of Ω depends on the number of wavelets used to model global motion. The curvilinear componant analysis (CCA) [6] is an algorithm for dimensionality reduction and representation of multidimensional data sets. Applying CCA on the motion feature space Ω gives a revealing representation of it in low dimension, preparing a basis for further clustering and classification Experimental results We have built Ω with a video database representing six human activities (Figure 5.a): up, down, left, right, come and go. These sequences were acquired for activity recognition [5] (aquisition rate: 10Hz). Each sequence includes ten frames and there are five sequences for each activity with different persons. So, the video database contains 30 sequences. B-splines of degree 1 at resolution level 2 were used to model coarse motion. Figure 5.b) represents the 2D mapping of Ω. The projection shows that our motion features are relevant to cluster together the 6 activities. In addition to the six clusters,

The sequences up, right and come have a global motion to the right of the scene, whereas the other sequences to the left. The same division appears into the projection map.

6 up down left right come go (a) Figure 5. a) Typical video sequences in the database and b) 2D mapping of the motion feature space obtained by CCA (b) the activities are also grouped by their main global motion characteristics. The sequences up, right and come have a global motion to the right of the scene, whereas the other sequences to the left. The same division appears into the projection map. 5 Conclusion We have presented a new motion model based on B- spline wavelet. Using this framewor, it is possible to estimate accuratly motion in image sequence and also to extract global motion features from video sequences. It is based on the optical flow B-spline wavelet series expansion between two consecutive frames. This ind of expansion presents the advantage to model various types of motions and provides relevant information about motion structure. It is of importance to note that motion wavelet coefficients are estimated directly from the image derivatives and do not require prior information of dense image motion or image segmentation. We have illustrated the motion model relevance with regard to the motion estimation accuracy and showed that wavelet coeffients can be used for video indexing or activity recognition. Future wors will concern precisely video indexing problems. We have to study how to combine wavelet coefficients along video sequences and how to classify these motion features to obtain relevant video database representation. References [1] J. Barron, D. Fleet, and S. Beauchemin. Performance of optical flow techniques. International Journal of Computer Vision, 1(12):43 77, [2] M. Blac and A. Jepson. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(1):57 92, July [3] M. Blac, Y. Yacoob, A. Jepson, and D. Fleet. Learning parametrized models of image motion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 97, pages , Puerto Rico, June [4] E. Bruno and D. Pellerin. Global motion fourier series expansion for video indexing and retrieval. In Advances in Visual Information System, VISUAL, pages , Lyon, November [5] O. Chomat and J. Crowley. Probabilistic recognition of activity using local appearance. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 99, pages , June [6] P. Demartines and J. Herault. Curvilinear component analysis: A self-organising neural networ for non linear mapping of data sets. IEEE Transactions on Neural Networs, 8(1): , [7] P. Holland and R. Welsch. Robust regression using iteratively reweighted least squares. Communications in Statistique - Theor. Meth., A6: , [8] J.-M. Odobez and P. Bouthemy. Robust multiresolution estimation of parametric motion models. Journal of Visual Communication and Image Representation, 6(4): , December [9] W. H. Press, S. Teuolsy, W. Vetterling, and B. Flannery. Numerical Recipies in C, Second Edition. Cambridge University Press, [10] S. Srinivasan and R. Chellappa. Noise-resiliant estimation of optical flow by use of overlapped basis functions. Journal of the Optical Society of America A, 16(3): , March [11] R. Szelisi and J. Coughlan. Spline-based image registration. International Journal of Computer Vision, 22(3): , 1997.

Learning Parameterized Models of Image Motion

Learning Parameterized Models of Image Motion Michael J. Black Yaser Yacoob Allan D. Jepson David J. Fleet Xerox Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA 9434 Computer Vision Laboratory,