Digital Image Stabilization by Adaptive Block Motion Vectors Filtering

Transcription

1 Digital Image Stabilization by Adaptive Block Motion Vectors Filtering Filippo Vella, Alfio Castorina, Massimo Mancuso, Giuseppe Messina Abstract This paper presents a robust algorithm for video sequences stabilization. Motion estimation is achieved using block motion vectors. In this way the same motion estimator of mpeg encoder can be used. The simple use of block motion vectors can give unreliable global motion vectors and so elaborations are done to make the algorithm robust. Index Terms Image Stabilization, Frame Motion Compensation, Tracking. A I. INTRODUCTION cquired video sequences can be affected by unwanted motions that create tiresome effects in the video sequence. The presented algorithm detects these movements in the video sequence and compensates them in order to obtain a more enjoyable and better compressed output. Digital Image Stabilization systems can be subdivided in three modules: motion estimation, detection of unwanted movements and compensation. Motion estimation is done evaluating the match in a search area of subparts of image. The size of areas and the matching function depends on the different approaches. Image subparts can cover the entire frame as in [6] or can cover only a subpart as in [2] and [3]. Besides, this matching can be done on a subset of original pixels (as in Representative Point Matching or RPM)[2][6] or considering a Xor operation on selected binary bit planes [1][3][4]. A matching based on edge pattern is used in [5]. F. Vella, A. Castorina, G. Messina are at Catania Advanced System Tecnology Lab of STMicroelectronics, Stradale Primosole 50, Bld. L7, Catania 95121, Italy( Filippo.Vella@st.com, Alfio.Castorina@st.com, Giuseppe.Messina@st.com ). M.Mancuso is at Agrate Advanced System Tecnology Lab of STMicroelectronics, via C. Olivetti 2, Agrate-Brianza, Italy ( Massimo.Mancuso@st.com ) In [2] a method using Block Motion Vectors (BMV) for the Global Motion Vector (GMV) evaluation is used although authors state that in some case this algorithm has worse performance than the method based on RPM. In section II a robust algorithm using block motion vectors is presented. Section III shows experiments results. Conclusions are given in Section IV. II. DIGITAL IMAGE STABILIZATION Digital Image Stabilization achieves the correction of unwanted movements in a video sequence through elaborations on digital data. The proposed algorithm uses as motion estimator a module that produces block motion vectors and from them evaluates a vector identifying unwanted movements. This vector summarizes the motion of the frame and is called Global Motion Vector (GMV). A. Motion Estimation Motion estimation allows detecting motion in the frame referring to previous frame. Motion Estimation evaluated on blocks gives a motion vector for each block considered. The advantages are that the same motion estimator of the mpeg encoder can be used and a big set of motion vectors can be evaluated for each frame. The drawback is that not all block motion vectors (BMV) are reliable. For example a homogeneous block in a homogeneous area can give a wrong motion vector as a poor matching has been found. The algorithm considers two zones and block motion estimation is done in these two areas. An area is associated to foreground and the other to background. The division has been done in an empirical way considering that the

2 subject is in the central zone of the frame while background is near the border of image. Figure 1 and Figure 2 show a frame where the areas for background and foreground have been highlighted. Figure 4, Input Frame with Background BMVs Figure 1, Foreground Area Figure 2, Background Area The choice of the background area can be done considering areas on top corners of image or using two strips along the frame sides. In each case the position is driven by the empirical consideration about which zone will grab background information. For each area of the frame, block motion estimation is done and a number of motion vectors are estimated. From them a global motion vector for foreground and one for background is evaluated. In Figure 3 and Figure 4, BMV are shown in the foreground and background areas. To calculate a Global Motion Vector for the areas a single Motion Vector must be evaluated from the set of BMVs. The most frequent vector present in the set is the best candidate to extract the Global Motion Vector. It can justified considering that the most part of image moved with that vertical and horizontal displacement. By this way a square histogram will work better than two linear histograms (one for horizontal and the other for vertical displacement). The square histogram considers the most frequent motion vector in the frame, two linear histograms extract the most frequent motion vector components and the result can be different. Considering simply all vectors detected in an area to build the histogram a number of unreliable motion vectors is used and final result can be wrong. Considering that homogeneous blocks can produce a wrong matching in a homogeneous search area and that blocks with high frequency components have a more reliable match, different weights for the histogram accumulation are considered. Homogeneous blocks will have a low weight while block with high frequency will have a bigger weight. In this way a most reliable block will be counted with more strength than a normal block. To estimate the high-frequency components present in a block the functions Activation or Mean Absolute Difference(MAD) are used. Activation is defined as the absolute difference of consecutive elements in rows and in columns of the block. A A hor A vert ( 1 ) Where A hor N 1 N 1 i 0 j 1 i, j i, j 1 Figure 3, Input frame with Foreground BMVs ( 2 )

3 31 S1 26 S S15 11 S and A vert N 1 N 1 j 0 i 1 In the equations ( 2) and ( 3) i, j i, j i 1, j ( 3 ) is the luminance of the pixel of the block indexed by i and j. N is the horizontal dimension of the considered block. MAD is the average of deviation in the block. MAD N N N 2 j 0 i 0 i, j MAD Weight MAD < < MAD < < MAD < < MAD < < MAD < < MAD 16 Table 1, Weights for BMVs The histogram, built in this way, contains the occurrence of each motion vector weighted with a value related to the evaluation function. Its maximum value will give the GMV estimation for the considered area (Figure 6). ( 4 ) Where is the luminance average in the block. The advantage of using the Activation function is that the block is scanned only once. On the other side MAD function is already present in the mpeg standard and so no further code is needed. In Figure 5 a frame is shown where blocks with different MAD are highlighted Figure 6, Square weighted histogram The resulting value for the Global Motion Vector will be the most reliable motion vector among the most frequent vectors. Two histograms are built: one for background and one for foreground. The maximum of each histogram will give a GMV for each region. B. Detection of unwanted movements Figure 5, Different MAD values for image blocks To give a weight to each motion vector, the values of the high frequency evaluation function are divided in ranges. Values are assigned considering that the higher is the value of the function the bigger must be the weight associated. An example of association of weight to MAD is shown in Table 1. If GMV for foreground and background are equal, the detected motion is considered as the global motion that affects the frame. Otherwise a decision must be taken whether to stabilize accordingly to the first or the second. Unwanted movements in a sequence can be due to a shaky position of the camera during acquisition or to difficult frame of a fast moving subject. In the first case jiggling of the camera affects the frames of the sequence. This kind of movement can be detected through the motion of the background.

4 In the second case user moves the camera following the subject but his motions are not as fast as the subject and unwanted motions are due to the inexact correction of framing. To decide whether to consider motion of foreground or background the number of blocks that produce the GMV is considered. If the background area GMV is produced by a higher number of blocks than the foreground area GMV, background is stabilized, otherwise foreground is stabilized. In the first case, unwanted camera motions are compensated and final sequence will be less affected by jiggling. In the second case, the subject in the central part of image is followed and it will always be in the center of the frame creating a tracking effect. C. Motion compensation When unwanted motion of the frame is detected, a motion compensation is done. Figure 7, Example frame with its GMV If a simple frame compensation according GMV is done, every frame would be stabilized only accordingly to previous frame. To have a stabilization of the whole sequence the correction must be done considering a common reference and so the Absolute GMV (AGMV) is used. The AGMV is the vector that accumulates the components of the GMV of each frame and it can be considered as the vector referred to the absolute coordinate system. Motion affecting the frame is corrected applying a traslation opposite to Absolute Global Motion Vector. To achieve the traslation a part of the initial image, corresponding to zones identified as vertical and horizontal margin, will be discarded and only the central part of image will remain stable. Final stabilized image will be a subpart of the initial frame. Figure 8, GMV compensated frame An important point in motion correction is the distinction between jiggling and panning. Jiggling is the oscillatory movement that has to be stabilized. It hasn t a constant direction on consecutive frames and its amplitude is generally little. Panning is the wanted motion that user does to capture a wide area of the scene. It is directed in the same direction and displacements are bigger than jiggling. To discriminate if a movement can be classified as jiggling or as panning a spatial threshold has been set for the components of the Absolute Global Motion Vector (AGMV). When a jiggling affects the sequence the AGMV components will have an oscillatory value around the central point and their values will not grow indefinitely because consecutive movements are aimed in opposite directions. In the case of a panning, for example in horizontal direction, the x component will increase its value because the motion is constant in the same direction. A threshold for vertical component and one for horizontal component is fixed. When the component overcomes the threshold a large and continuative movement has been accomplished in a direction and so no further compensation is done. Stabilization will be again allowed when the component becomes lower than threshold. The usage of a spatial criterion for the discrimination avoids to consider the value of the sequence frame rate. AGMV.x AbsGMV.x > T Panning No correction AbsGMV.x < T Jiggling Stabilization Figure 9,Horizontal Panning/Jiggling discrimination

5 A consideration must be done about the AGMV. Considering this vector, stabilization is done referring to the first frame that becomes the reference frame. When a panning occurs this reference frame must be changed. In [2] it is achieved integrating the motion vector found with the previous step integrated vector. In this paper the problem is solved maintaining the value of the AGMV constant when panning is detected. In this way the reference frame follows the panning and when panning ends, stabilization will be done again with a new context. III.RESULTS Tests have been done with real and artificial sequences. Artificial sequences have been created capturing a sequence with a still grabber and adding a traslational jiggling to the sequence. With this class of sequences the algorithm gives very good performances and the result is a perfectly stabilized sequence. Real sequences have been acquired with a hand-held grabber. In this case results are good and final sequence is characterized by a reduced jiggling. It must be underlined that when motion among frame is not purely traslational and a rotation or warping is present, the algorithm stabilizes the most similar traslational motion. When motion is purely traslational it is compensated and the stabilization has a very good effect. To evaluate the stabilization in real sequences the number of blocks with null motion vectors is considered. In Figure 10 is shown the chart of zero vector occurrences in a sequence before (lighter color) and after elaboration (darker color). It can be noticed that after stabilization the number of zero vector in the frame grows and so a wider part of image remains still. Figure 10, Zero vector occurrence in test sequence before (lighter color) and after stabilization (darker color) IV.CONCLUSIONS Video sequence stabilization allows to record more enjoyable videos and better compressed information. For these reasons it will be an important module in each hand held device (camcorder but also mobile phones, PDA, etc.) equipped with a sensor and able to acquire videos. The algorithm proposed stabilizes a video sequence evaluating unwanted motions starting from BMV. In this way the same motion evaluation module of the mpeg encoder is usable and besides a big set of sample can be used to evaluate the global motion. To have a reliable motion estimation each BMV is considered in relation to a function that considers the high-frequency components in the block. Stabilization is done considering foreground or background accordingly to the sequence so a camera stabilization or a tracking is achieved. Discrimination between jiggling and panning has been considered to avoid to inhibit wanted motions. V.REFERENCES [1] C. K. Cheung, Fast motion estimation techniques for video compression, Ph.D. Thesis, City University of Hong Kong, July 1998; [2] A.Engelsberg, G. Schmidt, A comparative review of digital image stabilising algorithms for mobile video communications, IEEE Transactions on Consumer Electronics, Vol. 45, No. 3, August 1999; [3] S.-J. KO, S.-H. Lee, K.-H. Lee, Digital image stabilization algorithms based on bit-plane matching, IEEE Transaction on Consumer Electronics, Vol. 44, No. 3, pp , August 1998; [4] S.-J. KO, S.-H. Lee, S.-W. Jeon, E.-S. Kang, Fast digital image stabilizer based on gray-coded bit-plane matching, IEEE Transaction on Consumer Electronics, Vol. 45, No. 3, pp , August 1999; [5] J. K. Paik,. C. Park, D.W. Kim, An adaptive motion decision system for digital image stabilizer based on edge pattern matching, IEEE Transaction on Consumer Electronics, Vol. 38, No. 3, August 1992; [6] K. Uomori, A. Morimura, H. Ishii,. Kitamura, Automatic image stabilization system by full-digital signal processing, IEEE Transaction on Consumer Electronics, Vol. 36, No. 3, August 1990.

6 Filippo Vella received his degree in Electronic Engineer (summa cum laude) in 2000 from the University of Palermo doing a thesis about hierarchical neural networks for the description of complex scenes. Since October 2000 he has been working in STMicroelectonics in the AST Digital Still Camera & Multimedia Mobile Group as System Engineer. His current activities are related to Panorama Technology and Digital Image Stabilization. Alfio Castorina received his degree in Computer Science in 2000 at the University of Catania doing a thesis about watermarking algorithms for digital images. Since middle of September 2000 he has been working in STMicroelectonics in the AST Digital Still Camera & Multimedia Mobile Group as System Engineer. His current activities include exposure compensation and high dynamic range imaging. M. Mancuso received the degree in Electronics Engineering (summa cum laude) from the University of Palermo (Italy) in He cooperated with the University of Genova for the development of CAL system for DSP. After a research period at the University of Palermo in the Image Processing field, he joined the Co.Ri.M.Me (a consortium between SGS-Thomson and the University of Catania) and he was involved in image processing using Fuzzy Logic based techniques. At the present he works as Digital Still Camera & Multimedia Mobile Program Manager at AST (Advanced System Technology) Catania Lab of STMicroelectronics. He is author of several papers and patents in Image Processing field. Giuseppe Messina received his degree in Computer Science at Catania University in 2000 doing a thesis about statistical methods for textures discrimination. Since March 2001 he has been working in STMicroelectonics in the AST Digital Still Camera & Multimedia Mobile Group as System Engineer. His current activities include research for resolution enhancement with the collaboration of Catania University.