Optimal Multiple Sprite Generation based on Physical Camera Parameter Estimation

Size: px

Start display at page:

Download "Optimal Multiple Sprite Generation based on Physical Camera Parameter Estimation"

Timothy Moore
5 years ago
Views:

1 Otimal Multile Srite Generation based on Physical Camera Parameter Estimation Matthias Kunter* a, Andreas Krutz a, Mrinal Mandal b, Thomas Sikora a a Commun. Systems Grou, Technische Universität Berlin, Einsteinufer 17, 1587 Berlin, Germany b Det. of Electrical & Comuter Engineering, University of Alberta, Edmonton, Canada T6G 2V4 ABSTRACT We resent a robust and comutational low comlex method to estimate the hysical camera arameters, intrinsic and extrinsic, for scene shots catured by cameras alying an, tilt, rotation, and zoom. These arameters are then used to slit a sequence of frames into several subsequences in an otimal way to generate multile srites. Hereby, otimal means a minimal usage of memory while keeing or even imroving the reconstruction quality of the scene background. Since wide angles between two frames of a scene shot cause geometrical distortions using a ersective maing it is necessary to art the shot into several subsequences. In our aroach it is not mandatory that all frames of a subsequence are adjacent frames in the original scene. Furthermore the angle-based classification allows frame reordering and makes our aroach very owerful. Keywords: camera calibration, image mosaicing, multile srites, srite generation 1. INTRODUCTION The growing comutational ower of today s comuting systems allows modern video analysis and coding tools to aly more comlex methods. Therefore, object based aroaches are now attracting a great deal of attention. In current hybrid video coding algorithms like MPEG-4, object based schemes are standardized for a long time but reliable all-urose algorithms roviding those objects do emerge only sarsely. A owerful tool to minimize coding costs is the srite coding tool 6. A background srite, also known as video mosaic, is the summarization of all rigid background objects of a video shot into one oversized image 2,3. Unfortunately, a universal aroach to determine the limits of srite generation is not available.. In this work, we aroach the roblem of limiting srites by roviding an efficient algorithm dividing a single video shot into a number of sub-shots based on a robust estimation of intrinsic and extrinsic camera arameters. The artition of the shots and thus a construction of multile srites is necessary due to geometrical distortions caused by the ersective image transformation, which is the aroriate maing for two image lanes in 3D sace, assuming only very small lens distortions 2. The ersective maing is given by x = H ( m) x x h a1 a2 a x y h = b1 b2 b y, (1) h c 1 c2 1 1 where (x,y) T are the coordinates of the reference image and (x,y ) T the corresonding oints in the image to register. The vector m=[a..a 2,b..b 2,c 1,c 2 ] is the arameter vector. Note that (1) is written in homogeneous coordinates, thus k H H for all k ú\{}. If the maing angle for a oint corresondence aroaches 9, i.e. c 1 x+c 2 y -1, the srite size grows towards infinity. Hence it is obvious that the srite has to be arted. The maing H(m) in equation (1), also known as homograhy, can be decomosed into a matrix F of internal camera arameters and a rotation matrix R x h x f ox x y h = F R y = f oy Rϕ R R y z ϕx ϕy. (2) h * kunter@nue.tu-berlin.de; hone ; fax ;

2 Without loss of generality we can omit the coordinates of rincile oint (o x,o y ) T while defining the image middle as the oint of origin. Thus, F can be reduced to a diagonal matrix containing focal length f n. R is a rotation matrix comosed of three rotations about y-, x-, and z-axis using Euler angles. In our work the video shot is divided in an otimal sense, which means that we obtain a minimum memory cost for the multi-srites. In order to reach this goal, we determine the hysical camera arameters esecially the focal length f i and rotation angles ϕ y,i and ϕ x,i and derive a discrete criteria for the estimation of the number of sub-srites. Estimation of camera arameters is an imortant task in image and video rocessing. The objective of the estimation techniques is to comute the intrinsic and/or extrinsic arameters of the camera, which was used to shoot a articular video. The estimated arameters can be used in many alications such as content analysis and 3D video Multile Srites Farin 1 roosed a technique for construction of multile srites with minimal coding costs based on the estimation of the size of the surrounding rectangle. In order to find the best solution, he determines the bounding box size for every air of frames i, k, being first and last frame of a sub-srite while testing every reference frame F r (i r k). Tracing back the cost values from the last frame one obtains the otimal sequence artitioning for srite construction. However, the algorithm still works sequentially, i.e. all frames that comose a sub-srite are always adjacent frames of the original shot. Additionally, the algorithm itself remains structurally comlex even if the cost comutation can be simlified. In our aroach we rely on a very comrehensible hysical camera arameter based sequence artitioning. If the camera angle between two views of a scene becomes too wide, i.e. the ersective distortion causes huge srite image sizes, the overall angle (an and/or tilt) will be arted into M angles of equal size. Thus, the artition algorithm yields in the calculation of M ot minimizing a simle cost function deending on the camera arameter set. One advantage of our aroach is the ossible reordering of the frames. Therefore, unlike other methods, frames of different temoral ositions can comose one sub-srite. 1.2 Calibration based aroach Figure 1 shows the flowchart of our roosed multile srite generation technique. In a first state we calculate the ersective transformation to register every frame F i of a sequence with resect to the reference frame F while generating a reliminary srite as roosed in our earlier work 4. If it is imossible to create one reliminary srite for the whole sequence we generate several srites with the last ossible registered frame as new reference frame F r. Second, intrinsic and extrinsic camera arameters are estimated robustly. Then, according to the calibration results we art the video sequence into M subsequences and construct the multile srites S to S M-1. The remainder of the aer is organized as follows. A quick overview of the reliminary srite generation technique is given in section 2. The estimation of hysical camera arameters and the shot artition into several sequences is described in section 3. Section 4 exlains the final construction of the multile srites. In section 5 we show our exerimental results where the quality of our calibration technique is assessed and effectiveness of the multile srite generation scheme is measured. Section 6 concludes the aer. video shot Preliminary frame-tosriteregistration H H 1,,N 1 Physical camera arameter estimation ( ϕx ϕy ϕz) Partition of video shot according to focal length and rotation angles f i Construction of srites for every sub-sequence S SM 1 Fig. 1. Flowchart of the roosed calibration based multile srite generation aroach 2. IMAGE REGISTRATION AND PRELIMINARY SPRITE GENERATION As shown in Fig. 1, reliminary srite generation is the first ste in our roosed technique. In this section, we resent the details of the image registration and srite generation used in our aer. The flowchart of our srite generation aroach is shown in Fig. 2. The rocess can be divided into two arts, the image registration and the blending rocess. In the image registration rocess otimal transformation arameters for waring the actual frame I n into the reference

3 coordinate system of Srite S n-1 are calculated. Since the convergence of a high order transformation calculation is very critical we aly hierarchical arameter estimation starting with feature-based robust estimation of the affine transformation arameters (a a 2, b b 2, c 1 =c 2 =) between consecutive frames I n and I n-1. As robust estimation rocedure the Monte Carlo tye random samle consensus (RANSAC) 8 algorithm is used. These short-term arameters are then otimized using Levenberg-Marquardt gradient energy minimization. The energy function is reresented by 1 1 E( n ) = I ( x, y ) I ( x,y ) 2 N Ω (x,y) Ω ( ) 2 n 1 n (3) Srite S n-1 Frame I n-1 Frame I n Robust Feature based Estimation Affine Transf. Gradient based Estimation Affine Transf. Concatenation of Short- and Long- Term Parameter Gradient based Estimation Homograhy H F inal Parameter Set (Long-Term) Srite Udate of newly discovered areas Initial Parameter Set (Short-Term) F inal Parameter Set (Short-Term) Initial Parameter Set (Long-Term) Srite S n Fig. 2. Flowchart of the srite construction rocess (left) and art of reliminary srite of sequence Charlottenburg (right) see artifacts of the indeendently moving foreground object The concatenation of short-term transformation arameters m n,affine with reviously comuted long-term arameters m n-1,ers using H = H H (4),n n 1,n,n 1 is refined by a direct frame-to-srite estimation of the Homograhy H,n alying Levenberg-Marquardt again. Thus, an oen-loo based accumulation of registration errors is revented. In order to do the direct frame-to-srite estimation a reliminary srite is generated and udated for every registered frame. This is achieved by simly blending newly discovered content into the srite. It is clear that indeendently moving foreground objects will distort the visual image quality. However, due to robust registration the arameter estimation will not be affected. Result of this rocess is a set of homograhies {H,1 H,N-1 } that relate every frame I n with n=,,n-1 to the reference image I and can be written as 1 f n f H,n = Hn H = Fn R,n F = fn Rϕ R R f z ϕx ϕ, (5) y 1 1 where R,n defines the rotation matrix between view and view n.

4 x y F -1 Rotation ϕ y Rotation ϕ x F n x y Rotation ϕ z x y Fig. 3. Scheme of the decomosition of homograhy H(m) into several hysical based extrinsic and intrinsic arameter matrices. Note that vector x and x lie in the same image lane and have the same vector norm: 2x 2=2x 2. view x f x view n f 1 ϕ Fig. 4. On the derivation of ratio α,n between the focal lengths of two views of a scene catured with rotating and zooming camera 3. ESTIMATION OF CAMERA PARAMETERS AND SHOT PARTITIONING The use of Euler angles for the decomosition of homograhies can be difficult because of ossible singularities having imact on the numerical stability 7. On the other hand the order of the three indeendent rotations (y-axis, x-axis, and z- axis) can be chosen arbitrarily. Thus, we can write H,n as 1 1,n n ϕz ϕx ϕy ϕz n ϕx ϕy H = F R R R F = R F R R F. (6) Figure 3 shows the decomosed transformation scheme. The final transformation about the z-axis can be regarded as inlane rotation. Hence, for comutation based on vector norms 2 2 in 2D Euclidean sace this rotation can be neglected. The in-lane rotation has also no influence for the rincial oints =(,,1) T and =(,,1) T. 3.1 Camera Parameter Estimation Since for common cameras the rincial oint =(x,y,1) T =(,,1) lies in the image middle the focal ratio can be comuted as α f x tan ϕ,n = = = fn x tan ϕ x x, (7) where x is the image of the rincial oint of view n in view and x the image of the rincial oint of view in view n. See Figure 4 for derivation of equation (7). Having the ratio α of focal length for all image airs, n (1 n N-1) the value for the focal length f is comuted as median value of all real solutions alying Szelisky s linear method 2. Rewriting equation (5) we obtain

5 Fig. 5. Partition of one srite into two subsrites for constant focal length (constant FOV) along horizontal dimension. See that the overall size of the two subsrites is smaller than the original srite due to geometrical distortions. The reference frame is chosen as the middle frame with resect to the view s absolute angle. The field of view (FOV) is marked as gray. r r1 fr2 h h1 h2 1 H,n = r1 r11 fr12 h1 h11 h12 α =. (8),n r 2α,n f r21α,n f r22α,n h2 h21 h 22 Taking into account the orthogonality constraint for row vectors and column vectors of R as well as the vector norm constraint for rows and columns we can arrange 12 indeendent equations for the calculation of focal length f. Note that we obtain 12 equations for every homograhy H,n with n=1,, N-1. From all real solutions we choose the median as final value. As our exerimental results show, this method yields accurate results for a minimum number of frames in a scene, which lies between 15 and 25 frames deending on the sequences. The angles of the rotations are then estimated directly using the images of both rincial oints. Inverting equation (6) yields in which follows in sinϕycosϕy h x = FRϕ R F sin y ϕx n Rϕ = F R R F z ϕy ϕx n = F ϕx, (9) cosϕycosϕ y ftanϕ y tanϕx x = f cosϕ. (1) y 1 After calculating angles ϕ y and ϕ x we can determine the image of rincial oint in the reference image before and after the final z-rotation n ϕx H,n 1 h x = F R Rϕ F y. (11) h x = ϕ z is now calculated as difference of the absolute angles of vector x and x with the horizontal line. ϕz = ϕz ϕz with y y tan ϕz = and tan ϕz =. (12) x x

6 3.2 Shot Partitioning To slit a video shot into several sequences, we first comute angle division for the rotation angle (ϕ y or ϕ x ) with the maximum overall rotation ϕ max. We minimize cost function C with resect to the number of angles M in order to find the number of srites to be generated for one rotation lane. M 1 ϕmax FOV C( ϕmax,m ) = fmax,i 2tan + 2M 2 (13) i= Factor f max,i describes the maximum focal length within the set of frames classified by a certain angle range i ϕ max /N ϕ ϕ min (i+1) ϕ max. The field of view (FOV) is the average aerture angle for the same subset of frames. An examle for subdivision along one dimension is given in Figure 5. After subdividing the sequence in one rotation lane (an or tilt), we aly equation 3 for the erendicular lane as well. Thus, we handle the sequence division as a searable roblem, which is only an aroximation for the non-searable ersective maing. However, for small angles ϕ max /N the introduced error is negligible. Beside its low comutational comlexity this aroach has the advantage that views that are relative close in sace are classified into on subsrite even if they are temoral divided by views classified into another subsrite. 4. MULTIPLE SPRITE CONSTRUCTION In this section we describe the eventual generation of the multile srites for the different subsequences of the video shot. It can be achieved in two stes. First the exact registration into the articular reference frame coordinate system is achieved by direct estimation using the LM-energy minimization framework of section 2. As reference image always the frame I r with rotation angle ϕ closest to the medium angle of a subsequence is used. Since this frame is not the frame with the biggest zoom the reference frame coordinate system is scaled by i fmax s =, (14) f where f i max is the maximum focal length within a subsequence. Thus, we revent resolution degradation due to subsamling of images during the waring rocess. In a second ste we robustly blend the contents of all transformed and wared frames of a subsequence into the final srite. In order to remove artifacts of indeendently moving foreground objects temoral filtering is erformed. Under all ixel candidates we consider only those ixels which hold ( ) τ ( τ ) τ with m= median Iυ ( T r υ(x,y )). I T ( x, y ) m A median I T ( x, y ) m n r n r υ r ( ) T r n ( ) reresents the transformation from the reference frame to a frame at time n. In order to minimize the lowass effect as a result of ixel interolation, out of these candidates the ixel with smallest distance d to integer osition in the original frame is chosen. To enhance the background reconstruction quality additionally correlation analysis can be conducted, which unfortunately would also increase the algorithms comlexity. 5. EXPERIMENTAL RESULTS In this section we first demonstrate the exactness of our calibration technique (section 5.1) while comaring the estimated camera arameter for a synthetic scene with the ones used for rendering the scene. In section 5.2 we deict the results for srite artitioning for real scenes. (15) 5.1 Sequence with ground truth Calibration accuracy Figures 7 and 8 comare the curves of the estimated camera arameters with the original ones of the rendered scene TUB-room (Fig. 6). The reliminary srite is also shown in Figure 7. As the results show, focal length and rotation angles are estimated very exactly. The maximum relative error for the focal length is less than 2% and the maximum

reference frame) obtained in two different ways according to equation (11).

The diagram in Figure 7 deicts this distance differences 2x 2-2x 2 over all frames of the sequence.

Exemlary frames of rendered sequence TUB-room, frame, 5, 1, 15 (left to right 44% of original size). 5 x 1-14 4 diff.

Preliminary srite of synthetic sequence TUB-room for calibration accuracy assessment (left); Difference 2x 2-2x 2 for vector

7 angle deviation is about 2 degree. To strengthen our arameter estimation result, we comare the vector norms 2x 2and 2x 2 (images of rincial oint in the reference frame) obtained in two different ways according to equation (11). Smaller values for the difference between these two absolute values indicate a more exact estimation. The diagram in Figure 7 deicts this distance differences 2x 2-2x 2 over all frames of the sequence. The maximal deviation is only about ixels. Fig. 6. Exemlary frames of rendered sequence TUB-room, frame, 5, 1, 15 (left to right 44% of original size). 5 x diff. of vector magnitudes Fig. 7. Preliminary srite of synthetic sequence TUB-room for calibration accuracy assessment (left); Difference 2x 2-2x 2 for vector norm of images x and x of rincial oint (view ), sequence TUB-room (right) small values indicate an accurate estimation of rotation angles ϕ y and ϕ x and focal length f n with resect to homograhy H,n and its inverse H n, =(H,n ) -1 focal lenght in ixel angle in degree y-rot x-rot z-rot Fig. 8. Comarison of estimation (solid) with ground truth values (dashed) for focal length (left) and rotation angles (right)

8 5.2 Sequences without ground truth We calculated extrinsic and intrinsic camera arameters for the well-known Stefan sequence. Since the anning angle is too wide to construct only one reliminary srite a second one was generated using the last registered frame as reference frame. The left diagram of Figure 9 shows the obtained focal length for this sequence comared to the result calculated with a commercial alication (Boujou) 9. The relative difference is about 8.5%. Note that the commercial result is no real ground truth data. The rotation angles (Figure 9 right) are very similar to those estimated with nonlinear methods in Farin s aroach 7. As indication of the calibration accuracy of this method as well the vector length of the reference rincile oint images is comared for 2x 2and 2x 2 (Figure 12). The maximum difference is about ixels. The sequence was then slit into three subsequences according to equation (13) based on the horizontal rotation. Since only very small vertical rotation aears no further artitioning along the erendicular dimension was rocessed. In Figure 9 the artitioning borders and the assignment of frames to the three srites is deicted. The srite obtained by a one-srite-aroach and the generated multile srites are shown in Figures 1 and 11. From the size of the surrounding rectangles it can be seen that memory cost is much higher for the single srite even if not all 3 frames of the sequence are registered. Additionally, we comared the comressed file sizes for the one-srite-aroach (only 264 frames) and the multilesrite-aroach. The amount of data was reduced by 77.4% (Table 1) in the comressed domain at quality level 75 for JPEG comression. Thus we can arove the results for multile srite memory cost minimization found in focal lenght in ixel Proosed method Commercial alication y-rotation -1 x-rotation z-rotation Fig. 9. Focal length (left) and rotation angles with artitions (right) for sequence Stefan angle in degree Srite 1 Srite 2 Srite 3 Fig. 1. Single-srite-aroach - Stefan frame to 263 (9% of orig. size: 6921x2592el).

261 lower left; 679x261 el, frame 262 to 299 lower right). 2 x 1-13 1.5 diff. of vector magnitudes 1.5 -.

Difference of vector magnitudes of reference rincile oint images (sequence Stefan ) File sizes for JPEG

9 Fig. 11. Multile-srite-aroach Stefan (35% of original size: 11x317 el, frame to 244 uer; 799x317 el, frame 245 to 261 lower left; 679x261 el, frame 262 to 299 lower right). 2 x diff. of vector magnitudes Fig. 12. Difference of vector magnitudes of reference rincile oint images (sequence Stefan ) File sizes for JPEG comression Image Single-srite-aroach (Quality level 75) Srite 1 - multi-srite-aroach (QL 75) Srite 2 - multi-srite-aroach (QL 75) Srite 3 - multi-srite-aroach (QL 75) File Size 694 kbyte 63 kbyte 49 kbyte 45 kbyte Tab. 1. Comarison of file sizes for single and multi-srites

10 6. SUMMARY AND CONCLUSION We resented a geometry based aroach for generation of multile srites for scenes recorded with cameras alying an, tilt, roll, and zoom. In order to derive a rule for subdivision of the video shots, fast and effective intrinsic and extrinsic camera arameter estimation was conducted. Because it is only alied for angle based shot artitioning the calibration does not have to satisfy maximal accuracy constraints. More imortant for this aroach are robustness and low comlexity. However, results have shown that our estimation of camera arameters is very close to other calibration techniques. The construction of multile srites minimizes the memory cost for the background which is an imortant matter for srite based video comression algorithms. Moreover, the reconstruction quality can be reserved or even imroved in case of changing background objects. The main advantage over other multile srite techniques is the ossibility of clustering of temoral non-successive frames into one subsrite. ACKNOWLEDGMENT This work was develoed within 3DTV (FP6-PLT DTV), a Euroean Network of Excellence funded under the Euroean Commission IST FP6 rogramme. REFERENCES 1. D. Farin, P. H.N. de With, W. Effelsberg, Minimizing MPEG-4 Srite Coding-Cost Using Multile Srites, in SPIE Visual Comm. and Image Proc. (VCIP), vol. 538, Jan R. Szelisky, H.-Y. Shum, Creating full view anoramic image mosaics and environment mas, in Comuter Grahics, Annual Converence Series, vol. 31, A. Smolic, T. Sikora, and J.-R. Ohm, Long-Term Global Motion Estimation and Its Alication for Srite Coding, Content Descrition, and Segmentation, IEEE Trans. Circuits Syst. Video Technol., vol. 9(8)., , Dec M. Kunter, J. Kim, T. Sikora, Suer-resolution Mosaicing using Embedded Hybrid Recursive Flow-based Segmentation, Int. Conf. on Information, Communications and Signal Processing (ICICS '5), Dec A. Iketani, et al., Suer-resolved video mosaicing for documents based on extrinsic camera arameter estimation, in Proc. of the 7th Asian Conference on Comuter Vision, (ACCV), vol. 3852, T. Sikora, The MPEG-4 video standard verification model, IEEE Trans. Circuits Syst. Video Technol., vol. 15, , D. Farin, P. H. N. de With, Estimating Physical Camera Parameters based on Multi-Srite Motion Estimation, in SPIE Image and Video Communications and Processing, vol. 5685, January M. A. Fischler and R. C. Bolles, Random samle consensus: a aradigm for model fitting with alications to image analysis and auto-mated cartograhy, Commun. ACM, vol. 24, no. 6, , htt://

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University, China