SEMI-ONLINE VIDEO STABILIZATION USING PROBABILISTIC KEYFRAME UPDATE AND INTER-KEYFRAME MOTION SMOOTHING

Transcription

1 SEMI-ONLINE VIDEO STABILIZATION USING PROBABILISTIC KEYFRAME UPDATE AND INTER-KEYFRAME MOTION SMOOTHING Juhan Bae 1,2, Youngbae Hwang 1 and Jongwoo Lim 2 1 Multimedia IP Center, Korea Electronics Technology Institute 2 Department of Computer Science and Engineering, Hanyang University jhbae@keti.re.kr 1,2, ybhwang@keti.re.kr 1, jlim@hanyang.ac.kr 2 ABSTRACT In this paper, we propose a video stabilization method that takes advantages of both online and offline video stabilization methods in a semi-online framework. Our approach takes the fixed length incoming frames for having the advantages of offline methods with the same length of delayed results. We stabilize input frames by warping to the keyframe to increase the visual stability. For preserving user intent camera motion correctly, we determine the next keyframe update by measuring inconsistency between a current keyframe and incoming frames. Moreover, inter-frame motion smoothing by quadratic fitting bridges the keyframes smoothly for pleasant viewing experiences. Our algorithm not only handles rapid camera motion changes, but also stabilizes the input camera path smoothly in real-time. Experimental results show that the proposed algorithm is comparable to the state-of-the-art offline video stabilization methods with only fixed length of incoming frames. Index Terms Semi-online, Video Stabilization, Adaptive Keyframe Update, Inter-keyframe Smoothing 1. INTRODUCTION Video camera systems are widely installed in many daily and intelligent applications such as security systems and robotics. Especially in intelligent security systems, it is necessary to have a stable image sequence directly from the camera for prompt response. Accordingly, both the immediate stabilization result and pleasant visual quality are key factors for evaluating the quality of video stabilization. Video stabilization is categorized into online and offline approaches by whether to stabilize the current frame directly. In the previous offline methods, Matsushita et al.[1] stabilized a video by smoothing an estimated 2D affine camera path with motion inpainting technique. Grundmann et al. [2][3] synthesized a stabilized video by smoothing 2D camera motion based on cinematography rules using linear programming framework and mixture of homographies for rolling shutter removal. Offline approaches show visually robust due to the optimized estimation using the full motion chain. However, offline methods are not applicable for real-time applications since full motion chain should be computed prior to the moment of displaying stabilization results. For online methods, simultaneous motion estimation and smoothing are performed for immediate stabilization. Previously Kalman filter [4][5], low-pass filtering [6] and motion vector integration [7] are applied for smoothing the original motion trajectory. Regarding the direct stabilization results, they are suitable for real-time applications. However, online methods are not able to utilize further motion chain at the moment of displaying and users may experience the stabilization results not robust to severe motion and parallax changes. In this paper, we propose a semi-online video stabilization method that combines the advantages of both online and offline approaches. It stabilizes the input frames by warping to the keyframe for piecewise static motion. Our algorithm updates the keyframe at the maximum discrepancy between incoming frames and the current keyframe for a moving camera. This enables the proposed method to be robust to rapid motion changes with varying update frequency. We smoothly concatenate between the previous and current keyframe by quadratic fitting with the continuity constraint for the visually comfortable conversion. In the experimental results, the effectiveness of the proposed method is shown by the comparisons of the smoothed motion trajectory and Interframe Transform Fidelity. 2. SEMI-ONLINE VIDEO STABILIZATION The major difference between online and offline video stabilization is whether to use future information against the time of displaying a stabilized result t d. If the incoming camera path is not known, the large motion drift that is the gap between the original path and the smoothed path is easily appeared at online methods when the motion change is considerable. For the offline approaches, stabilized results are not live at t d, but visually robust against for the severe and unexpected camera motion. To combine the benefits from both online and offline methods, we propose a semi-online video stabilization method

2 Fig. 1. Semi-online video stabilization with inter-keyframe motion smoothing. as shown in the Fig.1. We take N incoming frames for the future motion at t d. An initial keyframe is set as the first frame and the proposed algorithm estimates the difference between the keyframe and incoming frames. If the maximum difference is lower than a certain threshold, we consider that the input motion is close to static and perform the keyframe based stabilization. Otherwise, we update the keyframe at the maximum difference and perform inter-keyframe smoothing by quadratic fitting for reducing large deviation compared to the input motion. If the proposed method is reached at the newly updated keyframe, it resumes the process of determining the next keyframe within N incoming frames, or remains static Inter-keyframe motion smoothing Video stabilization based on cinematographic principles [2] shows the most pleasant viewing experiences by modifying the input camera path into a static, constant velocity and acceleration path. Our algorithm approximates the cinematographic rules in a semi-online framework. Our algorithm utilizes N future motions at t d that cause N frame dealy of stabilization result. For trivial camera motion changes, our approach warps the current frame I(t) to the keyframe I k (t) for a static camera motion where k is the keyframe index. If the camera path is drifted apart from the keyframe, the algorithm determines the next keyframe at the maximum difference. Therefore, it is crucial to bridge between the previous and the current keyframe smoothly. Suppose A(t) is inter-frame motion from frame t 1 and t. C(t) is the input camera pose at frame t and it can be written as : C(t) = A(t)C(t 1), C(t) = A(t)A(t 1)... A(1) (1) Let M(t) be the smoothed camera path and it can be described as : M(t) = C(t) 1 B(t), (2) where B(t) is the update motion. To approximate the cinematographic rules, the algorithm estimates B(t) as the quadratic motion. Static camera motion can be covered in the keyframe based stabilization. If the camera motion is in constant velocity with jitters, quadratic estimation is considered as linear motion as shown in the Fig.1. The estimated motion is approximated as quadratic when the camera motion is in constant acceleration. Accordingly, the algorithm performs the least-squares quadratic fitting on each of the affine motion parameter chain (horizontal and vertical scale, translation, skew and rotation) with N incoming frames. For visual smoothness of the keyframe conversion, the algorithm satisfies the continuity constraint that the beginning slope of the newly smoothed chain is the same as the end slope of the previously smoothed chain Keyframe update Adequate keyframe update on our framework can prevent the propagation of accumulated errors and large inconsistency between the input and smoothed path. Consequentially, estimating the moment of proper keyframe update is essential for minimizing the path drift compared to the keyframe. If the frequency of keyframe update is too long, the smoothed path is far deviated from the original path when the motion changes are considerably large. When the update frequency is too short, the effect of stabilization is decreased since the discrepancy between the input and the smoothed path is reduced while the algorithm tracks the rapid camera motion well. To make an appropriate keyframe update, the inconsistency with the keyframe should be measured. Our algorithm aims to find the maximum change from the keyframe by measuring the difference of time, motion and photometric error. If we set r Ω k, where Ω k consists of N incoming frames of I k (t) and r is the index of incoming frames, the motion discontinuity w d (r) consists of two zero-mean Gaussian function: w d (r) = G m ( M(r) C(k) 2 ) G t ( r k ), (3) where G t () and G m () measures the difference of a motion of frame r and time compared to the keyframe respectively. However, the term of motion changes ( M(r) C(k) 2 ) mainly depends on the translational motion while the rest of components are negligible. To make a correct measurement of the deviation, we take into account the photometric difference w p (r) by the zero-mean Gaussian of photometric errors G p () against the keyframe and it is shown as below w p (r) = G p (R(r)), (4) where R(r) is the root mean square error between I(r) and the current keyframe I k (t). R(r) is defined as R(r) = j=r,g,b D j (r) 2, D j (r) = I j k 3 (t) Ij (r), (5) where j and D j are the color channel variations and channel color difference respectively. Adaptive keyframe selection

3 (a) Video 1 (slow and periodic motion) (b) Video 2 (rapid horizontal motion) (c) Video 4 (static camera with vertical motion) Fig. 2. Motion smoothing results decides the next keyframe at the maximum difference from the keyframe. The index of next keyframe is set as r = argmin wd (r) wp (r), (6) r and the next keyfrmae is set as Ik+1 (t) = I(r ). The algorithm sets the next keyframe when the wd (r ) wp (r ) exceeds a certain threshold Tp. In our implementation, we set N =30 as acceptable delay, the standard deviation of Gt () as 20 and the standard deviation of Gm () and Gp () as 10 for higher sensitivity on motion changes and photometric errors. We set Tp as for the balanced stabilization result. Table 1. Mean and standard deviation of absolute difference of affine parameters(horizontal and vertical scale, rotation, translation and skew) on stabilization results compared to the ground truth. We use Deshaker for slow and periodic camera motion and L1-optimal camera path for the rapid camera motion as the ground truth. Deshaker [8] sx sy 3. EXPERIMENTAL RESULT angle We run our algorithm on the Intel i7 3.4 GHZ Quad-Core machine with 8G RAM. We extract good features to track features [9] and use KLT tracker [10] with HomographyRANSAC based rejection for estimating inter-frame motion. The proposed algorithm is applied on 5 selected from video stabilization dataset [2] and our own CCTV videos. It runs in real-time with a single core on 640x480 resolution input video (average 34 fps). We use N =60 that 30 incoming frames are used as the future motion and previous 30 frames as the past motion for Offline Gaussian method [1]. Online Gaussian methods takes previous 30 frames. We use σ = 20 for both online and offline Gaussian methods for strong stabilization effect. Deshaker [8] and L1-optimal camera path [2] implemented in OpenCV that stabilize the video by fully optimized path with automatic trimming are the one of the state-of-the-art video stabilization algorithms in the offline framework and they are used as the ground truth without trimming for comparison. Fig.2 shows the stabilized translational camera path. For slow and periodic camera motion in Fig.2(a), the proposed tx ty sk Offline Online Gaussian[1] Gaussian (0.003) (0.003) (7.03) 5.89 (2.92) (0.007) (0.028) (0.031) (19.48) 7.26 (5.22) (0.006) L1 [2] (0.008) () (4.93) 5.88 (4.17) (0.008) Offline Online Gaussian[1] Gaussian (0.256) (0.24) (0.037) (67.31) 8.41 (5.99) (0.088) (0.114) 0.04 (0.036) () (163.9) (16.01) (0.022) 0.01 (0.006) (0.002) (31.16) 6.23 (6.26) (0.012) method shows less deviation from the ground truth and the stabilized path is passing between online and offline Gaussian methods. The algorithm is robust for rapid motion in Fig.2(b) by adaptive keyframe update that maintains more consistency between the input and smoothed paths than online and offline Gaussian methods as shown in the Fig.3. The effectiveness of the proposed method becomes better when the static camera with trembling motion is used. The algorithm effectively increases the stability by reducing the effect of periodic motion even compared to L1 and Deshaker in Fig.2(c). In Table.1, quantitative evaluation of the stabilization quality is shown compared to the ground truth. Preserving user intent camera motion is evaluated by mean difference and stability is measured by standard deviation with the ground

4 (a) Input (b) Offline Gaussian[1] (c) Online Gaussian (d) Deshaker[8] (e) L1 [2] (f) Fig. 3. Stabilzation results. From the video 1-3 (V1-V3 in row 1-3), videos from hand-held camera are used and CCTV videos 4-5 (V4-V5 in row 4-5) with vertical the unwanted motion are used for stabilization comparison. Table 2. ITF comparison (db) Input Deshaker L1[2] Offline Online Gaussian[1] Gaussian V V V V V better stability than Deshaker and L1-optimal camera path due to the keyframe warping with fewer updates. The effect of a window size on motion smoothing is shown in Fig.4. Online Gaussian method gains more stability as higher N is applied. The proposed method shows better stability by suppressing small bounces of the smoothed path effectively than the online Gaussian method with strong motion smoothing with larger N. 4. CONCLUSION Fig. 4. Stabilization results with varying N truth. For slow camera motion, translation is the most varying parameters and the proposed method preserves the user intent motion better than online and offline Gaussian cases. In the case of rapid motion, the proposed method shows better tracks and stability as well while online and offline Gaussian shows large deviation from the input path. In addition, quantitative results by Interframe Transformation Fidelity(IT F ) [11] that measures the inter-frame similarity shows that our approach are comparable with offline methods for both moving and static cameras in spite of only using N future frames information (Table.2). Especially for the static CCTV camera, the proposed methods obtains even We have presented a semi-online algorithm to stabilize the input video in real-time. The essence of our algorithm is to stabilize the video with only N future frame. Adaptive keyframe update and interframe motion smoothing enables the proposed method comparable to state-of-the-art offline methods as Deshaker and L1-optimal camera path. 5. ACKNOWLEDGMENT This work was partially supported by the IT R&D program of MKE/KEIT. [ , Development of Robot Vision SoC/Module for acquiring 3D depth information and recognizing objects/faces] This work was partially supported by the IT R&D program of MKE/KEIT (No ) and the Center for Integrated Smart Sensors as Global Frontier (CISS ). This research was partially supported by the Cross-Ministry Giga KOREA Project of the Ministry of Science, ICT and Future Planning, Republic of Korea(ROK). [GK130100, Development of Interactive and Realistic Massive Giga-Content Technology]

5 6. REFERENCES [1] Y. Matsushita, E. Ofek, Weina Ge, Xiaoou Tang, and Heung- Yeung Shum, Full-frame video stabilization with motion inpainting, PAMI, IEEE Transactions on, vol. 28, no. 7, pp , [2] M. Grundmann, V. Kwatra, and I. Essa, Auto-directed video stabilization with robust l1 optimal camera paths, in CVPR, 2011 IEEE Conference on, 2011, pp [3] M. Grundmann, V. Kwatra, D. Castro, and I. Essa, Calibration-free rolling shutter removal, in ICCP, 2012 IEEE International Conference on, 2012, pp [4] Chuntao Wang, Jin-Hyung Kim, Keun-Yung Byun, Jiangqun Ni, and Sung-Jea Ko, Robust digital image stabilization using the kalman filter, Consumer Electronics, IEEE Transactions on, vol. 55, no. 1, pp. 6 14, [5] Junlan Yang, D. Schonfeld, Chong Chen, and M. Mohamed, Online video stabilization based on particle filters, in Image Processing, 2006 IEEE International Conference on, 2006, pp [6] A. J. Crawford, H. Denman, F. Kelly, F. Pitie, and A.C. Kokaram, Gradient based dominant motion estimation with integral projections for real time video stabilisation, in Image Processing, ICIP International Conference on, 2004, vol. 5, pp Vol. 5. [7] S. Battiato, G. Gallo, G. Puglisi, and S. Scellato, Sift features tracking for video stabilization, in ICIAP th International Conference on, 2007, pp [8] Gunnar Thalin, Deshaker, se/video/deshaker.htm. [9] Jianbo Shi and Carlo Tomasi, Good features to track, in 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 94), 1994, pp [10] Carlo Tomasi and Takeo Kanade, Detection and tracking of point features, Tech. Rep., International Journal of Computer Vision, [11] C. Morimoto and R. Chellappa, Evaluation of image stabilization algorithms, in Acoustics, Speech and Signal Processing, Proceedings of the 1998 IEEE International Conference on, May 1998, vol. 5, pp vol.5.