Video stabilization based on a 3D perspective camera model

Transcription

1 Vis Comput DOI /s z ORIGINAL ARTICLE Video stabilization based on a 3D perspetive amera model Guofeng Zhang Wei Hua Xueying Qin Yuanlong Shao Hujun Bao Springer-Verlag 2009 Abstrat This paper presents a novel approah to stabilize video sequenes based on a 3D perspetive amera model. Compared to previous methods whih are based on simplified models, our stabilization system an work in situations where signifiant depth variations exist in the senes and the amera undergoes large translational movement. We formulate the stabilization problem as a quadrati ost funtion on smoothness and similarity onstraints. This allows us to preisely ontrol the smoothness by solving a sparse linear system of equations. By taking advantage of the sparseness, our optimization proess is very effiient. Instead of reovering dense depths, we use approximate geometry representation and analyze the resulting warping errors. We show that by appropriately onstraining warping error, visually plausible results an be ahieved even using planar strutures. A variety of experiments have been implemented, whih demonstrates the robustness and effiieny of our approah. G. Zhang W. Hua X. Qin ) Y. Shao H. Bao State Lab of CAD&CG, Zhejiang University, Zhejiang, People s Republi of China xyqin@ad.zju.edu.n H. Bao ) bao@ad.zju.edu.n G. Zhang zhangguofeng@ad.zju.edu.n W. Hua huawei@ad.zju.edu.n Y. Shao shaoyuanlong@ad.zju.edu.n X. Qin Shool of Computer Siene & Tehnology, Shandong University, Jinan, People s Republi of China Keywords Video stabilization Struture from motion Optimization View warping Warping error 1 Introdution The goal of video stabilization is to remove annoying shaky motion from a video sequene. It plays an important role in many appliations, suh as video ompression, video editing, bakground estimation, and moving objets detetion. In general, video stabilization is made up of three stages: motion estimation, filtering and ompensation [15]. Motion estimation is to selet what kind of motion model and how to estimate the motion, while motion filtering is to eliminate undesirable image motion aused by jittering. Compensation is arried out by image re-sampling or view warping operations, usually amounting to 2D affine or homography transformations. Essentially, video stabilization is to eliminate the undesirable amera motion in the video aused by handheld or mehanial vibration. Traditional approahes employ simplified models e.g. affine model or homography model [8, 10, 13]) to estimate 2D global motion of images. Homography model is appropriate if the video sene undergoes planar perspetive transformation, e.g. planar senes with arbitrary amera motion, or arbitrary senes with fixed amera loation. Conditions of affine models are even more strit. However, these methods are too strit to hold on for general video sequenes beause they will result in large errors if signifiant depth variations exist in the senes and the amera undergoes large translational movement. Buehler et al. [4] handled this problem from image-based rendering standpoint based on projetive reonstrution of video sequenes and upgrading them to a quasi-affine model, and then smooth the sequenes via smoothing feature points.

2 G. Zhang et al. Some methods [5, 24] were proposed to smooth the video sequenes by ompensating 3D rotation, whih an be ompensated without the neessity of reovering depth information. Zhu et al. [27] analyzed possible motion patterns and proposed a 2.5D motion model that requires user interation to hoose the dominant harater of the motion in the video sequene being analyzed. However, this method annot deal with general movement. The estimation of general 3D perspetive amera motion is the lassial struture from motion SFM) problem [7, 9, 18, 25]. As an area of endeavor, SFM problem has reahed a degree of maturity with several ommerial offerings [1, 19]. In our implementation, we employ the method proposed in [25]. Several filtering methods [8, 10, 13, 14, 27] have been proposed to redue unintentional amera motion with respet to senes while preserving the dominant, intentional amera motion. As we know, missing image areas will appear after stabilization by motion ompensation. However, nearly all these previous methods did not disuss how to ontrol the tradeoff between missing image areas and smoothness. Reently, Pilu [17] studied this problem and used the Viterbi method to solve the most stable sequene allowed by the size of the output window. However, this method is reursive and does not give a losed-form solution. Matsushita et al. [13] proposed a motion in-painting approah to fill in the missing image areas. In ontrast, we formulate the stabilization problem as a quadrati ost funtion on smoothness and similarity onstraints. This allows us to preisely ontrol the smoothness by solving a sparse linear system of equations. In ompensation stage, original frames should be warped to obtain motion-ompensated frames. However, view warping without aurate geometry information will introdue warping error, whih may result in serious artifats. With depth maps, some methods suh as 3D warping [12] and layered-depth images [20] an render new views by projeting the pixels of the nearby points of view to their proper 3D loations and re-projeting them onto a new image. Reently, Bhat et al. [3] proposed to stabilize video by smoothing out the original amera path and re-rendering the sene as seen from the new amera path with reovered dense depth maps. Unfortunately, obtaining dense depth information from real images is hard even for the state-of-the-art vision algorithms. Snavely et al. [21], on the other hand, produed 3D meshes by triangulating sparse point louds, and rendered eah mesh with texture map to synthesize a new view. However, missing geometry and outlying points an sometimes ause distrating artifats. These limitations and pratial demands motivated us to develop an effetive method, whih does not involve omplex dense depths reovery, but an still obtain visually plausible stabilized results. In order to ahieve this goal, we make an in-depth analysis of the warping error aused by geometry approximation. Our experiments show that even using planar struture to approximate sene geometry, visually plausible results an be ahieved, by onstraining the warping error to ahieve optimal smoothness results and avoid the distortion artifats. In order to ahieve optimal smoothness, we formulate the stabilization problem as a quadrati ost funtion on smoothness and similarity onstraints. By exploiting the sparseness of the linear system, our optimization proess is very effiient and fast. The rest of this paper is organized as follows. Setion 2 gives an overview of our method, and introdues the proposed ost funtion for video stabilization as well as the optimization method. We elaborate our view warping method and make an analysis of warping error in Set. 3. Experimental results are given in Set. 4. Finally, we onlude the whole paper. 2 Our approah 2.1 Overview We begin the desription with the original video sequene V O ={Io i ) i = 1,...,n}, where o i is the ith original amera, Io i ) is its orresponding image, and O is the sequene of original ameras. We aim to obtain its motionompensated sequene V C ={I i ) i = 1,...,n}, where i is the ith ompensated amera, and C is its sequene. Our algorithm onsists of the following three major steps: Step 1 For eah original frame Io i ), reover the extrinsi and intrinsi parameters of the orresponding original amera o i in the set O ={o i i = 1,...,n}. Step 2 Solve the stabilized amera set C ={ i i = 1,...,n}. Step 3 For every i = 1,...,n, perform view warping operations to obtain motion-ompensated frames: Io i ) I i ), i = 1,...,n, where the output frames V C ={I i ) i = 1,...,n} form the resultant motion-ompensated video sequene. The first step is the SFM problem, whih outputs the reovered amera motion parameters with sparse 3D feature points. In the seond step, the stabilized amera motion parameters an be ahieved, by solving a quadrati ost funtion on smoothness and similarity onstraints. In the third step, we employ view warping to obtain the motionompensated video sequene. 2.2 Camera motion estimation Our automati feature traking algorithm is based upon SIFT algorithm [11] for its reliable performane even in wide-baseline mathing ase. We extrat SIFT features from eah frame of the input video sequene and math the features frame by frame. Corresponding features are onstrained aording to the epipolar geometry theory [26]. We

3 Video stabilization based on a 3D perspetive amera model use RANSAC algorithm [6] to find a set of inliers that have onsistent epipolar geometry. The mathed feature points onstitute the feature traks. Then we use the SFM method proposed in [25] to reover the amera motion parameters from the given video sequene. For ompleteness we briefly summarize the algorithm as follows. As we know, struture and motion estimation with longer traks is more reliable and robust than with short traks [7, 25]. Let N be the minimum trak length we require. Then we selet the traks not shorter than N as superior traks for reonstrution. And we use the interval N 1)/2 to selet the key frames to ensure that all superior traks stride over at least two key frames. Then we initialize the projetive reonstrution from the referene triple key frames aording to the riteria proposed in [25]. The projetive reonstrution is upgraded to a metri framework at an appropriate moment through self-alibration. For eah newly added frame, the new amera parameters and 3D points are initialized, and existing struture and motion are refined. Finally, the whole struture and motion are refined through bundle adjustment [23]. 2.3 The ost funtion for video stabilization Previous approahes usually employ lassial filter algorithms e.g. Gaussian filtering, Kalman filtering, time averaging, et.), whih is diffiult to preisely ontrol the tradeoff between missing image areas and smoothness. We adopt an optimization proess to determine C by minimizing a ost funtion on smoothness and similarity onstraints. Essentially, image jittering is aused by the shaky motion of a amera. Therefore, if we an smooth the amera motion, image jittering an be removed. At the same time, the motionompensated sequene should be similar to the original sequene, in order to make missing areas small. Therefore, eah term is separated into two parts, whih are smoothness ost and similarity ost. We use subsripts m and s to distinguish smoothness and similarity, and supersripts and o to indiate ompensated and original parameters, respetively. The amera motion an be deomposed into rotation, translation, and zooming omponents. To ensure the visual smoothness of the stabilized video, the rotational, translational and zooming aeleration should be minimized. Therefore, we respetively define the rotational smoothness term E Θm, zooming smoothness term E fm, and translational smoothness term E tm. The formulations are given as the following: E Θm = Θ i 2Θ i+1 + Θ 2 i+2, E fm = E tm = 1 i n 2 1 i n 2 1 i n 2 fi 2fi+1 + f i+2 t i 2t + i+1 t 2 i+2, 2, 1) where Θ denotes the rotational vetor expressed by Euler angles, f denotes the foal length, and t denotes the translational vetor. Similarity onstraints require that the warped views should look similar to the original ones and they share large ommon sene region. Therefore, the amera parameters of the original frames should be as lose to those of the warped frames as possible. Our similarity terms are simply given as follows: E Θs = Θ i Θi o 2, 1 i n E fs = fi fi o 2, 2) 1 i n E ts = 1 i n t i to 2 i. Finally, the ost funtions respetively orresponding to rotational, zooming and translational omponents are defined as: E Θ = ω 2 Θm E Θm + ω 2 Θs E Θs, E f = ω 2 fm E fm+ ω 2 fs E fs, 3) E t = ω 2 tm E tm + ω 2 ts E ts, where ω Θm, ω fm, ω tm, ω Θs, ω fs, ω ts are weights of the ost terms, and the square terms are for onveniene of linear equations in 4). We should minimize E Θ + E f + E t ) to obtain the target sequene. Sine the rotation, foal length and translation are independent of eah other, we an minimize E Θ, E f and E t respetively. It an speed up omputation with less memory requirement. Sequentially, without loss of generality, we an set ω Θs = 1, ω fs = 1, ω ts = 1. We mainly use ω Θm and ω fm to ontrol smoothness sine human observers are muh more sensitive to rotational vibrations, and only amera rotation and zooming an be ompensated without the neessity of reovering depth information. Usually ω Θm = ω fm = 100 an obtain extremely smooth results. Speifially, E f an be ignored if the foal length is onstant. For translation, sine we do not have dense depth information, any kind of warping methods for ompensating the translation may result in warping error, whih eventually auses image jittering. Therefore, the tradeoff between translational smoothness and warping error; i.e., the translational smoothness weight ω tm should be arefully set. We will disuss view warping and analyze warping error in the next setion. 2.4 Optimization The ost funtions in 4) are quadrati and independent. Therefore, optimizing them is equal to solving the following

4 linear equation arrays for rotation, foal length, and translation, respetively: { ωθm Θ i 2Θi+1 + ) Θ i+2 = 0, i = 1,...,n 2, ω Θs Θ i Θi o ) = 0, i = 1,...,n, { ωfm f i 2fi+1 + f i+2) = 0, i = 1,...,n 2, ω fs f i fi o ) 4) = 0, i = 1,...,n, { ωtm t i 2t i+1 + ) t i+2 = 0, i = 1,...,n 2, ω ts t i t o ) i = 0, i = 1,...,n. For n frames, the number of equations of eah array is 6n 6, 2n 2, and 6n 6 for the omponents of rotation, foal length, and translation, respetively. Without loss of generality, here we only onsider the optimization of the foal length omponent. If we denote X = [ f1 ],,f 2,...,f n b = [ 0,...,0,ω fs f o 1,...,ω fsf o n ], then the orresponding equation array an be expressed as AX = b. By applying the least square method, its solution is: X = A A ) 1 A b. If n is large, the omputation is very time-onsuming. However, we find that although A is a 2n 2) n matrix, it has only 5n 6 non-zero elements, whih means that both A and A A are highly sparse. Therefore, we an exploit the sparseness to speed up the omputation. Here, we propose a smart method to implement A A by taking advantage of sparseness effiiently. Generally, for any m n matrix A = a ij ),wehave: A A = i=1,...,m = i=1,...,m A i A i a i1 a i2... a i1 a i2... a in ). 5) a in Here A i denotes the ith row vetor of A. If we store only the non-zero elements of A i, then A i A i omputes only the multipliation of non-zero elements to eliminate all the redundany of the zero omputation. Therefore, we have hosen to represent sparse matries using Compressed Row Storage CRS) format [2]. The CRS format is a general format whih makes no assumptions about the sparsity struture of the matrix and does not store any unneessary elements. The additional ost is to searh index in sparse matrix A A due to the ompressed struture. We denote N A G. Zhang et al. Table 1 Effiieny examination of our optimization algorithm Frame number Parameter number Cal. time seonds) and N r as the average number of non-zero elements in a row of A and A A, respetively. We an employ a balaned binary searh tree for quik searhing. In our implementation, we diretly use C++ STL map lass, whose omplexity is Olog 2 N r ). Consequently, the total ost of A A operation is OmN 2 A log 2 N r). Then we an use a sparse linear equation solver to solve it. In our implementation, we adopt the TAUCS Library [22] to solve the sparse linear systems. Table 1 shows the running time of our optimization algorithm. The omputation ost is nearly linear to the number of frames being proessed. The total omputation time for 521 frames is less than one seond, whih proves the effiieny of our optimization method. 3 View warping and warping error As we know, it is still not robust to obtain dense depth maps or sparse point louds with existing algorithms. Therefore, we try other ways to sidestep this problem. Warping without aurate geometry information will introdue warping errors, whih may ause image jittering or other distrating artifats. In this setion, we make an in-depth analysis of the warping error aused by using planar approximation. We have experimented with two planar warping tehniques: slant planar impostors and optimal onstant depth i.e. fronto-parallel plane), onstraining warping error in two different ways. 3.1 Warping error Warping error an be measured by the differene between the warped image and the desired aurate image. Consider a 2D point x p on the original image, and its orresponding 3D loation X p. We define warping error as the distane between the warped point H x p and x p K R X p + t ) on the desired image: ep W = H x p x p, 6) where H is a 3 3 matrix i.e. homography ). K, R and t are the motion-ompensated amera parameters. For previous approahes using affine or homography models, onstraining warping error is diffiult, sine it is

5 Video stabilization based on a 3D perspetive amera model hard to formulate the aurate target images due to the lak of the information of aurate amera parameters and 3D geometry. 3.2 View warping with slant planar impostors If senes are roughly planar, we an estimate the planar transform or homography) H i by minimizing the following funtion: min H i x ij K i R i X j + t ) 2 i, 7) j where K i, R i and t i are the intrinsi matrix, rotation matrix and translation vetor of the ith warped frame, respetively, x ij is the 2D image position of the jth feature point on the ith original frame, and X j is its orresponding 3D loation. The goal of this method is to minimize the total warping error of all the feature points, and this method works well in planar senes. However, if the senes deviate from planar struture, it may ause visible image distortions, as demonstratedinfig.1and the supplementary video. 3.3 View warping with optimal onstant depth Suppose that the sene in one frame lies in a plane whih is perpendiular to the viewing diretion i.e. onstant depth). With the given amera parameters, we an projet eah pixel to its orresponding 3D loation and sequently re-projet it onto the target view. Similarly to the warping with slant planar impostor, this warping is also a planar transformation and an be represented by a homography. The optimal depth of this plane is z = 2z 1 min + z 1 max ) 1 see Appendix for the detail), where [z min,z max ] is the depth range. In step 1, we an obtain a set of spare 3D feature points. Aording to the depth distribution of these features, we an reliably estimate z min and z max. Although the total warping error is a little larger than the slant planar impostors, the resulting artifats are usually less objetionable, perhaps beause we are muh more sensitive to seeing unnatural image distortions and the optimal onstant depth method does not ause this annoying artifat. Therefore, we prefer to use it rather than planar impostors as default for view warping. 3.4 Warping error analysis Warping error also results in visual jittering and therefore should be ontained in the measurement of video jittering. By making an analysis of the warping error, we an get the solution of optimally setting the weights ω tm and ω ts. For the ith frame, its original translation is t o i, and its ompensated translation is t i. Therefore, the hanging translation vetor t = t x,t y,t z ) an be omputed by t = t i to i. For point p, its 3D homogeneous oordinate is denoted as x, y, 1, 1/z), where z is its depth value. If we use z to estimate its depth, the warping error will be see Appendix for the detail): 1 e W = f z 1 ) t x xt z,t y yt z ). 8) z Then we have: e W 1 = f z 1 z t x xt z ) 2 + t y yt z ) 2. 9) Beause the amera foal angle is usually small typially less than 35 degrees) and most interesting points are lose to the image enter i.e. x and y is small), so, for onvenient omputation, we approximate it as: e W f 1 z 1 z t =f 1 z 1 z t i to i. 10) Another part of video jittering results from the vibration of the ompensated translation. For a point p, its image position in the ompensated image i is x i, and its image position in the ompensated image i + 1isx i+1. Then the displaement d an be defined as d = x i+1 x i. If we denote t i = t i+1 t i,wehaved = f z t x xt z, t y yt z). In order to minimize jittering, the displaement needs to be onstant, i.e. d ) = d i = 0. Denoting a = t i i = ti 2,i.e. i 2 the seond derivative of translation, we have: d ) f = z a x xa z,a y ya z ), 11) where a x, a y and a z are the three omponents of the vetor a. Similarly to 10), for a = t i 1 2t i + t i+1,wehave: d ) 1 f z a =f 1 t z i 1 2t i + t i+1. 12) In order to onstrain the jittering, we should minimize both e W 2 and d ) 2. Then the ost funtion an be formulated as: ) 1 2 E jit = t i 1 z 2t i + t 2 1 i+1 + z 1 ) 2 t z i to i 2. 13) When weight ratio ω tm : ω ts = z 1 : z 1 z 1, the ost funtion E jit is equal to the ost funtion E t. In this ase, the solution not only minimizes E t but also E jit. That is, the best smoothness effet an be ahieved. If z is lose to z min or z max ), the orresponding optimal value is lose to z 1 min : z 1 min z 1 or zmax 1 : z 1 max z 1 ). Espeially under the optimal onstant depth z =

6 2z 1 min + z 1 max ) 1, they beome 2z min /z max z min ) and 2z max /z max z min ), respetively. Considering that z is distributed in [z min,z max ], the optimal ω tm : ω ts an be omputed by averaging them as follows: ω tm : ω ts ) opt = 2z min/z max z min ) + 2z max /z max z min ) 2 = z max + z min )/z max z min ), 14) whih beomes 1 if z max z min. Therefore, if z min and z max are unknown, we usually set ω tm : ω ts = 1 as the default in 4). 4 Results We have tested our approah with a variety of video sequenes taken by a hand-held video amera. All experiments are arried out on a PC with Intel Pentium IV 2.4 GHz CPU with 1 GB memory. Appealing results are obtained in our experiments. The urrent proessing speed of our SFM step is relative slow, whih typially takes about two minutes to proess 100 frames. There is a lot of room for improvement in our unoptimized researh ode sine we employ a standard SFM algorithm, whose running time was not our major onern in this paper. Reently, Nistér [16] proposed a fast SFM method for video with real-time performane. It would be espeially benefiial to our system. Our stabilization optimization step is quite quik, whih only takes about 0.14 seonds for 100 frames Table 1). Our view warping an perform near real-time about 0.1 seond per frame with resolution). The evaluation and omparison of video stabilization algorithms is a diffiult task, sine there is no ground truth available for real sequenes and the standard of evaluation is also diffiult to formulate. Espeially, our method employs a 3D perspetive amera model, and previous approahes employ affine/homography models. Perhaps, pereptual judgment of stabilization is the best option to evaluate video stabilization algorithms aimed at the human observer. For omparison, we also implement the stabilization method proposed in [13] with both affine and homography motion models. The radius of Gaussian filtering range is 10 frames i.e. k = 10 in [13]). The algorithms are shown in Table 2, Table 2 Stabilization algorithms Algorithm Model Filtering Compensation 1 affine Gaussian affine 2 homography Gaussian homography 3 3D amera linear opt. opt. onst. depth 4 3D amera linear opt. planar impostors G. Zhang et al. Table 3 Smoothness evaluation on the four examples shown in this paper. A small value indiates that the stabilized result is more smooth Smoothness Evaluation Algorithm Example A Example B Example C Example D where the algorithms 3 and 4 are our algorithms with different warping strategies, i.e. optimal onstant depth and slant planar impostors, respetively. We seleted four video sequenes whih we all A, B, C and D) and 7 persons to perform a user test. For eah example, there are four stabilization results obtained by four different algorithms shown in Table 2). Eah user independently evaluates the stabilized results. Exept the example C for whih four algorithms produe omparable results, all users onsider the effets with algorithms 3 and 4 signifiantly outperform those with other two algorithms. In order to quantitatively evaluate the stabilized results, we measured the smoothness of the traked feature points in the stabilized sequenes by omputing the average of the seondorder differentials of the trajetories of all feature points, i.e. 2x t x t 1 x t+1. Table 3 reports the detailed statistis. Again, the smoothness errors with algorithms 3 and 4 are muh smaller than those with their ounterparts, for all examples exept the example C. This is atually in aordane with the user study. The reason is that most video sequenes ontain large amera translations and signifiant depth variations in the senes. It should be noted that a 2D affine or homography model works well in many examples in whih the senes undergo planar perspetive transformations. In these ases, our stabilized results are omparable with results by previous methods. In order to illustrate obvious differenes of view warping with optimal onstant depth and slant planar impostors, we generate a simulation sequene. The reason is that real videos usually do not ontain exaggerated shaky translations. In addition, exaggerated shaky motion may ause serious motion blur, whih is diffiult for traking. We onstrut a sene whih ontains two planes. Figure 1 shows the different results employing slant planar impostors and optimal onstant depth with different ost weights, in whih the depth of the front one is 100, and that of the bak one is 300. The sequene of translation whih is reovered from a real video sequene is shaky, and is smoothed with two different ost weights ω tm = 2 and ω tm = 100 ω Θm = 100, the foal length is onstant), as shown in Fig. 1a). As an be seen, if w tm is set larger, the ompensated translational parameters get smoother the urves get smoother). However,

7 Video stabilization based on a 3D perspetive amera model Fig. 1 The omparison of view warping in two different ways with different ost weights. a) The original translational motion parameters and stabilized ones for the simulation sequene. b) The omparison of the stabilized results of view warping in two different ways optimal onstant depth and slant planar impostors), with different ost weights due to the warping error, it does not mean that the ompensated video will ertainly beome more stable. Figure 1b) shows the motion-ompensated frames, by optimal onstant depth and slant planar impostors with different ost weights. The optimal value of ω tm is 2 in this sequene aording to 14); larger values annot obtain smoother result. On the ontrary, they an ause larger warping errors and unnatural artifats far objets beome very shaky, et.), as demonstrated in the supplementary video. It should be noted that view warping with slant planar impostors auses obvious image distortions in the ase of large warping error, whereas the result of view warping with optimal onstant depth is muh more natural. However, if the sene an be dominated by a slant plane, view warping with slant planar impostors will obtain more aurate result. Figure 2 shows an example for a simulated world onsisting of a single inlined plane under similar motions. We set the ost weights ω tm = 100, ω Θm = 1000 foal length is onstant). Sine the sene is a single slant plane, it an be exatly represented by plane impostors. Therefore, for this example, view warping with planar impostors an obtain the best result, without introduing warping error Fig. 2)). In ontrast, view warping with optimal onstant depth will ause warping error as shown in Fig. 2, b) and d)), so that the jittering annot be fully eliminated. We examine four real sequenes with various senes and amera motion taken by a hand-held amera. All these sequenes ontain lots of vibrations and depth variation. In these examples, the weights are all set as ω Θm = ω fm = 100 and ω tm = 1, if without mentioning. Sine human beings are muh more sensitive to distrating effet in video sequenes than in still images, please refer to the aompanying videos for the detail effets. In order to demonstrate the sequene effets of smoothness in still images, we mark all the superior traks, where the traks are showed by white lines, their image positions in the urrent frame are indiated by Fig. 2 A simulated example onsisting of a single inlined plane. a) The original translational frame. b) The ompensated frame by warping with optimal onstant depth. ) The ompensated frame by warping with planar impostors. d) The differene image of b)and). Due to the warping error, b) has a signifiant displaement ompared with ) green rosses, and the positions in neighbor frames by red rosses. As shown in Fig. 3, sequene A onsists of omplex sene with tree leaves in the front and a building in the distane. From Fig. 3d), we notie that the building ompensated by affine and homography models is slanted in different ways within 0.5 seonds, whih results in obvious jittering in target video sequenes. Both our methods demonstrate very stable effets in this sequene. Sequene B was taken in our ampus, and is a roughly planar struture. Only the trak line and the urrent feature points are marked in Fig. 4.FromFig.4a), the routes of the

8 G. Zhang et al. Fig. 3 Stabilization of the sequene A. a) The reovered rotational parameters and the stabilized ones. b) One of the original frames with the reovered amera trajetory. )Three frames of stabilized sequenes with four algorithms refer to Table 2). d) The magnified snapshots of ) around the building area Fig. 4 The stabilization of the sequene B with different algorithms: a) shows the original frame; b), ) and d) show one stabilized frame of the algorithms 1, 2, and 4, respetively feature points in the original sequene are very wandering, whih results in very dense white line folding sine the amera motion is unstable. The stabilized feature traks based on affine model or homography model are still a little wander- ing, while our model method demonstrates a very smooth route. Carefully omparing the video sequenes of Fig. 4, b) and ), we an find that homography model is prior to affine model in planar senes. This is beause that homog-

9 Video stabilization based on a 3D perspetive amera model Fig. 5 The results of the sequene C: a) shows the reovered foal length. b) An original frame with feature traks. ) The stabilized image with feature traks applying algorithm 3 Fig. 6 The results of the sequene D: a) shows the reovered amera trajetory; b) shows one original frame; and ), d), e), and f) show the different stabilization results with algorithms 1, 2, 3, and 4, respetively raphy model an aurately desribe the image motion in planar senes. Our amera traking method an handle variable foal lengths in a zooming sequene. Our stabilization algorithm demonstrates robust effets in video sequene C, in whih the foal length varies in large range as shown in Fig. 5a), while an original and its motion ompensated frame are showninfig.5, b) and ), respetively, with the illustrative feature traks. Sequene D is a mountain area downloaded from iss.bu.edu/litvin/stabilization/). From the trajetory of video amera shown in Fig. 6a), we an find that the translation distanes are very uneven. Both affine and homography models are obviously distorted, espeially in the end of the sequene where some objets are very lose to the amera. Notie the left peak area in Fig. 6, ) and d), where the feature points are not proportionally spaed in the sequene, while our method is muh better as shown in Fig. 6, e) and f). However, for our methods, the result of applying optimal onstant depth is appreiably better than slant planar impostors. The reason is that the senes deviate from planar, espeially in the later part of the sequene. We ompare the results of different ω Θm in Fig. 7b), where the larger value produes smoother results but larger missing regions.

10 G. Zhang et al. Fig. 7 Results with different ost weights. a) The reovered and stabilized rotational motion parameters for sequene D with different ost weights. b) The original and motion-ompensated frames. The middle olumn: ω Θm = 5, ω tm = 1. The right olumn: ω Θm = 100, ω tm = 1 5 Conlusion and disussion We have proposed a novel method to stabilize video sequenes based on a 3D perspetive amera model without reovering dense depth maps. The video stability is optimized by balaning the smoothness and similarity, whih is related to the rotation, zooming, and translation omponents with suitable weights. Based on a 3D perspetive amera model, the depth relative motion, i.e. amera translation, and depth irrelative motion, i.e. amera rotation and zooming, are separated. Consequently, the unwanted motion, espeially amera rotation and zooming, an be smoothed effiiently without introduing warping error, and translation is also smoothed under ontrol by setting optimal weight. By taking advantage of the sparseness of the linear system, our optimization proess is very effiient. We have experimented with two warping methods with planar geometry approximation, i.e. slant planar impostors and optimal onstant depth method, whih have obvious advantages over traditional stabilization methods, suh as employing affine or homography models. Our stabilization method an produe high quality stabilized video sequenes, and is very useful for some high-end appliations, suh as film-making and TV. Our approah employs a 3D perspetive amera model, and thus highly relies on the auray of the Struture-from- Motion SFM) results. Till now obtaining aurate amera motion parameters for long video sequenes is still quite hallenging [7], our approah may not work well if the SFM algorithm fails to get preise results. We expet to address this problem in our future work. Aknowledgements We would like to thank the anonymous reviewers for their onstrutive omments to improve the paper, and also thank Prof. Xiaofei He and Wei Chen for their enormous help in revising this paper. This work is supported by NSF of China No ), 973 program of China No.2009CB320802), and 863 program of China No. 2007AA01Z326). Appendix For pixel p in the ith frame, its 3D loation is xz, yz, z) in the oordinate system of the original amera, where z is its depth value. So its homogeneous oordinate is x, y, 1, 1/z) in the oordinate system of the original amera. For the ith frame, its original translation is t o i, and its ompensated translation is t i. Therefore, the hanging translation vetor t = t x,t y,t z ) an be omputed: t = t i to i. From the original amera to the motion-ompensated amera, its 3D position beomes xz + t x,yz+ t y,z+ t z ),i.e. homogeneous oordinate beomes xz+t x z+t z, yz+t y z+t z, 1, 1/z + t z )). Then the displaement of p in the 2D image an be omputed as follows: d = f xz + t x,f yz + t ) y f x, fy) z + t z z + t z = f t x xt z,f t ) y yt z. z + t z z + t z For onveniene, we replae z + t z with z by simply offsetting the oordinate, hene d = f t x xt z,f t ) y yt z. 15) z z If we use the onstant depth z for eah pixel, instead of its true depth value, the estimated displaement beomes d W = f t x xt z,f t ) y yt z. 16) z z

11 Video stabilization based on a 3D perspetive amera model Therefore, the warping error is 1 e W = d d W = f z 1 ) t x xt z,t y yt z ). z We assume the depths of the sene are in the range of [z min,z max ],i.e.z min z z max. Therefore, when z = 2z 1 min + z 1 max ) 1, the upper bound of warping error is minimal: e W = f 1 z 1 z t x xt z ) 2 + t y yt z ) f 1 ) t x xt z ) z min z 2 + t y yt z ) 2. max Referenes 1. 2d3: 2. Barrett, R., Berry, M., Chan, T.F., Demmel, J., Donato, J., Dongarra, J., Eijkhout, V., Pozo, R., Romine, C., der Vorst, H.V.: Templates for the Solution of Linear Systems: Building Bloks for Iterative Methods, 2nd edn. SIAM, Philadelphia 1994) 3. Bhat, P., Zitnik, C.L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., Kang, S.B.: Using photographs to enhane videos of a stati sene. In: Kautz, J., Pattanaik, S. eds.) Rendering Tehniques 2007, Proeedings Eurographis Symposium on Rendering, pp Eurographis 2007) 4. Buehler, C., Bosse, M., MMillan, L.: Non-metri image-based rendering for video stabilization. In: CVPR, pp ) 5. Davis, L.S., Bajsy, R., Herman, M.: RSTA on the move. In: Proeedings of the ARPA Image Understanding Workshop, pp ) 6. Fishler, M.A., Bolles, R.C.: Random sample onsensus: A paradigm for model fitting with appliations to image analysis and automated artography. In: CommACM, vol. 24, p ) 7. Fitzgibbon, A., Zisserman, A.: Automati amera traking. In: Shah, M., Kumar, R. eds.) Video Registration, pp Kluwer, Dordreht 2003) 8. Hansen, M., Anadan, P., Dana, K., van de Wal, G., Burt, P.: Realtime sene stabilization and mosai onstrution. In: Pro. IEEE Image Understanding Workshop, pp ) 9. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge 2000) 10. Litvin, A., Konrad, J., Karl, W.: Probabilisti video stabilization using Kalman filtering and mosaiking. In: IS&T/SPIE Symposium on Eletroni Imaging, Image and Video Communiations, pp ) 11. Lowe, D.G.: Distintive image features from sale-invariant keypoints. Int. J. Comput. Vis. 602), ) 12. Mark, W.R., MMillan, L., Bishop, G.: Post-rendering 3D warping. In: SI3D, pp. 7 16, ) 13. Matsushita, Y., Ofek, E., Tang, X., Shum, H.Y.: Full-frame video stabilization. In: CVPR 1), pp ) 14. Morimoto, C., Chellappa, R.: Fast 3D stabilization and mosai onstrution. In: CVPR, pp ) 15. Morimoto, C., Chellappa, R.: Evaluation of image stabilization algorithms. In: International Conferene on Aoustis, Speeh and Signal Proessing, vol. 5, pp ) 16. Nistér, D.: Preemptive ransa for live struture and motion estimation. Mah. Vis. Appl. 165), ) 17. Pilu, M.: Video stabilization as a variational problem and numerial solution with the Viterbi method. In: CVPR 1), pp ) 18. Pollefeys, M., Gool, L.J.V., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koh, R.: Visual modeling with a hand-held amera. Int. J. Comput. Vis. 593), ) 19. REALVIZ: Shade, J., Gortler, S.J., wei He, L., Szeliski, R.: Layered depth images. In: SIGGRAPH, pp ) 21. Snavely, N., Seitz, S.M., Szeliski, R.: Photo-tourism: exploring photo-olletions in 3D. ACM Trans. Graph. 253), ) 22. TAUCS: Triggs, B., MLauhlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment a modern synthesis. In: Workshop on Vision Algorithms, pp ) 24. Yao, Y., Burlina, P., Chellappa, R., Wu, T.: Eletroni image stabilization using multiple visual ues. In: Proeedings of International Conferene on Image Proessing, pp ) 25. Zhang, G., Qin, X., Hua, W., Wong, T.T., Heng, P.A., Bao, H.: Robust metri reonstrution from hallenging video sequenes. In: CVPR 2007) 26. Zhang, Z.: Determining the epipolar geometry and its unertainty: A review. Int. J. Comput. Vis. 272), ) 27. Zhu, Z., Xu, G., Yang, Y., Jin, J.S.: Camera stabilization based on 2.5D motion model estimation and inertial motion filtering. In: IEEE Conferene on Intelligent Vehiles, pp ) Guofeng Zhang reeived his BS degree in Computer Siene from Zhejiang University, P.R. China, in Currently, he is a PhD andidate in omputer siene at State Key Laboratory of CAD&CG, Zhejiang University. His main researh interests inlude amera traking, 3D reonstrution, augmented reality and video enhanement. Wei Hua reeived the BS degree in Biomedial Engineering from Zhejiang University in 1996, and the PhD degree in Applied Mathematis from Zhejiang University in Currently, he is an Assoiate Professor of the State Key Laboratory of CAD&CG of Zhejiang University. His researh interests inlude realtime simulation and rendering, virtual reality and software engineering.

12 Xueying Qin reeived her PhD from Hiroshima University of Japan in 2001, and MS and BS from Zhejiang University and Peking University in 1991 and 1988, respetively. She was an Assoiate Professor of the State Key Laboratory of CAD&CG of Zhejiang University. Currently, she is a Professor of Shool of Computer Siene & Tehnology, Shandong University, P.R. China. Her main researh interests are augmented reality, videobased rendering, and photo-realisti rendering. G. Zhang et al. Hujun Bao reeived his Bahelor and PhD in Applied Mathematis from Zhejiang University in 1987 and He is urrently the diretor of State Key Laboratory of CAD&CG of Zhejiang University. He is also the prinipal investigator of the virtual reality projet sponsored by Ministry of Siene and Tehnology of China. His researh interests inlude realisti image synthesis, real-time rendering tehnique, digital geometry proessing, field-based surfae modeling, virtual reality and video proessing. Yuanlong Shao reeived his BS degree in Computer Siene from Zhejiang University, P.R. China, in Currently, he is a Master andidate in Computer Siene at State Key Laboratory of CAD&CG, Zhejiang University. His main researh interests inlude amera traking, feature mathing, augmented reality and video enhanement.