Focused Video Estimation from Defocused Video Sequences

Transcription

1 Focued Video Etimation from Defocued Video Sequence Junlan Yang a, Dan Schonfeld a and Magdi Mohamed b a Multimedia Communication Lab, ECE Dept., Univerity of Illinoi, Chicago, IL b Phyical Realization Reearch Center of Excellence, Motorola Lab, Schaumburg, IL ABSTRACT Thi paper propoe a novel technique for etimating focued video frame captured by an out-of-focu moving camera. It relie on the idea of Depth from Defocu (DFD), however overcome the hortage of DFD by reforming the problem in a computer viion framework. It introduce a moving-camera cenario and explore the relationhip between the camera motion and the reulting blur characteritic in captured image. Thi knowledge lead to a ucceful blur etimation and focued image etimation. The performance of thi algorithm i demontrated through error analyi and computer imulated experiment. Keyword: Depth from Defocu (DFD), Focuing, Camera motion, Image retoration 1. INTRODUCTION Focuing i one of the peritent concern in camera deign. Current auto-focu olution in commercial digital camera are mainly baed on focu meaure. A digital camera equipped with auto-focu function move it adjutable len to take multiple picture and then find the bet picture by minimizing certain out-of-focu penaltie. The hortage of auto-focu olution i that it require a focal-length changing len and an accurate len engine to move the len with a particular tep ize. From a different point of view however, image proceing olution model the out-of-focu len a a linear filter whoe impule repone i known a the Point Spread Function (PSF). The focued image can be recovered through a deconvolution proce. The overall philoophy of etimating PSF and it Fourier tranform, alo known a Optical Tranfer Function (OTF), i baed on a fundamental obervation that the blur characteritic relate only to the object depth and the camera phyical parameter. Depite that the relationhip i uually implified uing firt order optic, thi obervation verifie itelf through the ucce of depth-from-defocu (DFD) algorithm. Claic DFD technique 1 applie two etting of camera parameter for acquiring two differently blurred image. Auming a Gauian PSF, a cloe form etimation of the blur parameter can be achieved. 1 A olution i provided in 2 when the PSF ha a cylindrical form. More generally, OTF can be approximated by a parametric polynomial 3 and the parameter can be etimated uing a leat-quare criteria. In a more recent work, 4 the technique of DFD i combined with tereo pair and the etimation i performed with Markov random field to improve the accuracy. DFD for etimating the PSF ha a olid and elegant theoretical foundation, however it poe a high requirement on the hardware. Due to the fact that changing camera etting uch a camera aperture and focal length cannot be done without ophiticated len ytem, it limit the application in practice. The algorithm propoed in thi paper, on the other hand, i deigned for a rigid camera whoe phyical parameter are all fixed. Therefore it can be applied to imple digital camera epecially webcam and mobile-phone camera. Another novelty of our algorithm i to take advantage of multiple image available in the video taken by moving camera. Image taken in variou poition not only provide multiple differently blurred image for etimation but alo help to improve the etimation accuracy. We alo provide in thi paper a novel idea on the etimation of Phae Tranfer Function (PTF). While it i a popular aumption that the PSF being patially ymmetric and thu OTF being a real function with only it modulu component known a Modulu Tranfer Function (MTF), it i in general not an accurate model for real camera. Camera len with variou propertie and manufacturing imperfection introduce inevitably the phae component to OTF referred a the PTF. The modelling and etimation of PTF ha alway been a challenge for image recontruction, where everal technique have been propoed including the technique of Phae from jyang24,dan@uic.edu, Magdi.Mohamed@motorola.com

2 Magnitude baed on projection onto convex et. 5 Common diadvantage for thee approache are their iterative cheme and no guarantee of convergence to the true phae. In thi paper, we propoe a non-iterative approach for the etimation of PTF baed on our framework. The ret of thi paper i organized a follow. In Section 2, we begin with the problem definition. In Section 3, we explain the main idea of blur etimation through two example of PSF and the idea of multiple-image etimation. Section 4 analyze the noie performance for the ytem. Section 5 dicue the etimation of PTF. Section 6 provide the imulation reult and Section 7 draw the concluion. 2. CAMERA AND IMAGING MODEL We begin by conidering a cenario in which there i a moving camera taking a video of a tatic object. The camera i a rigid camera, meaning that it ha a fixed len aperture, focal length and image plane-to-len ditance. One point in the object project onto different image coordinate when the camera move. In time t and time t, the camera take two image, frame k and frame k. The pixel location (x 0, y 0 ) in image frame k and (x 1, y 1 ) in frame k are related by a 2D affine tranform: 6 [ x1 y 1 ] = z [ ] [ ] [ ] 0 f r11 r 12 x0 tx + z 1 f r 21 r 22 y 0 t y where r 11, r 12, r 21, r 22 are rotation parameter and t x, t y are tranlation parameter of the camera motion. z 0 and z 1 are ditance between the object and the camera len, in time t and time t repectively, which are commonly referred a depth of the object. For the implicity of derivation, here we aume that the camera capture only planar object. In the cae of 3D object and 3D cene, one can apply directly the whole algorithm to a neighborhood of each pixel in the frame, with the neighborhood ize mall enough to enure a uniform depth. f i the focal length of the camera. Define z0 f z 1 f. Denote the Fourier tranform of frame k and frame k a F 0 (u, v) and F 1 (u, v). According to the affine theorem for 2D Fourier tranform, 7 F 0 (u, v) and F 1 (u, v) ha the following relationhip: F 1 (u, v) = 1 F 0( r 22u r 21 v, r 12u + r 11 v )exp{ j2π [(r 22t x r 12 t y )u + (r 11 t y r 21 t x )v]}, (2) where 2 (r 11 r 22 r 12 r 21 ). Uing the motion etimation and tabilization technique, 6 we can compenate for rotation before we further proce the image. Therefore we only dicu in thi paper when the rotation matrix i identity. Then (2) can be implified a F 1 (u, v) = 1 2 F 0( u, v )exp{j2π(t u x + t v y )}. (3) We model in thi ection the PSF to be a ymmetric function and thu the OTF being a real function. Therefore we conider only the magnitude component of (3): F 1 (u, v) = 1 2 F 0( u, v ). (4) When a camera i out of focu, the reulting image i blurred by a pecific PSF, whoe parameter are uniquely determined by the blur radiu R. In frequency domain, the pectrum of blurred image Y (u, v) will be the original pectrum time the OTF H(u, v, R): Y i (u, v) = F i (u, v)h(u, v, R i ), i = 0, 1. (5) (1) With (4) and (5), we have 2 Y 1 (u, v) = Y 0 ( u, v ) H(u, v, R 1 ) H(u/, v/, R 0 ). (6)

3 To proceed, we need to incorporate the knowledge from optic geometry. The blur radiue are given a a function of depth and camera parameter: 1 R i = vl( 1 f 1 1 ), i = 0, 1. (7) z i v where v i the image plane-to-len ditance, L i the radiu of len aperture, and z i the depth of the object. It can be een that the blur radiu i affected only by the depth of the object for one particular camera. From the definition of we can continue to arrive at = z 0 f z 1 f = R 0 + L R 1 + L R 1 + L vl/f R 0 + L vl/f. (8) To etimate the focued image from the blurred image, we need to etimate the OTF, which equal identifying the blur radiue. With v, L, f being known camera parameter, it i able to olve for, R 0, R 1, thu H(u, v, R 0 ) and H(u, v, R 1 ), baed on (6) and (8). 3. BLUR PARAMETER ESTIMATION AND VIDEO RECONSTRUCTION In thi ection, we will preent our algorithm baed on two type of PSF. In both cae, we begin with auming the energy conervation contraint, namely H(0, 0, R) = 1. Thu, can be olved by noticing the DC component in (6) yield = Y 0 (0, 0)/Y 1 (0, 0). (9) 3.1 Gauian Blur Model When PSF take the form of a Gauian function, 1 we have: Subtituting (10) to (6), we can obtain H(u, v, R) = exp{ 1 4 (u2 + v 2 )R 2 }. (10) 2 Y 1(u, v) Y 0 ( u, v ) = exp{ 1 4 (u2 + v 2 )(R 2 1 R 2 0/ 2 )}. (11) Equation (11) i true for all pair of (u, v), o an averaged olution 1 i given a follow: c 1 M 1 (u,v) I 1 4 Y 1(u, v) u 2 + v 2 ln(2 Y 0 ( u, v (12) ) ), R 2 1 R 2 0/ 2 = c, (13) where I 1 i the region within which the ummation i well-defined and M 1 i the number of (u, v) pair in I 1. With (8), (9) and (13), we can olve R 0 and R 1 uniquely. An approximated olution can be achieved baed on the fact that v f and L R. (8) can be then implified a R 1 = R 0, o that R1 2 R0/ 2 2 = R0( 2 2 1/ 2 ) = c. Hence, R 0 = c Thi approximation avoid meauring v, L, f and i found to be accurate enough in experiment. 3.2 Geometric Blur Model According to geometric optic, the firt order approximation of the PSF take the form of a cylindrical function in the cae of a circular aperture. 2 Therefore we have H(u, v, R) = 2 J 1(R u 2 + v 2 ) R u 2 + v 2. (14)

4 Adopting the polynomial expanion 8 of a beel function, Equation (6) become J 1 (x) = x 2 x x x , (15) 8 2 Y 1(u, v) Y 0 ( u, v ) = 1 + a 1(u 2 + v 2 ) + a 2 (u 2 + v 2 ) , a 1 = 1 8 (R2 1 R 2 0/ 2 ); a 2 = (R4 1 R 4 0/ 4 ) + R a 1;... (16) Once we identify a n, n = 1...N, we can olve for R 0 and R 1 with (8) and (9). N i the number of coefficient. Theoretically, identifying only a 1 i enough. However, more coefficient are deired for a reliable olution. The identification problem equal olving the following matrix equation: Y 1(u,v) Z(u 0, v 0 ) Z(u 0, v 1 )... = 1 u2 0 + v 2 0 (u v 2 0) u v 2 1 (u v 2 1) a 1... a N (17) where Z(u, v) 2 Y 0(u/,v/). The LHS i a K 1 vector, where K i the number of non-zero frequency component being ued. The matrix in RHS i of ize K N, and the vector in RHS i the unknown vector of ize N 1. An leat-quare olution of [a 1,...a N ] can be obtained from (17) then we will have an over-determined equation array (16) for olving R 0 and R 1. Once we get the etimation of the tranfer function, we can proce the degraded image with an invere filter or a Wiener filter to recover the focued image. And we apply thi etimation technique to ucceive video frame until the entire focued video equence ha been recovered. 3.3 Multiple Image Etimation When three or more frame available, we can ue them together for etimation in order to improve the accuracy. For intance, with Gauian PSF, if we have a third image Y 2 (u, v), we can form another et of equation uing the implified verion of (8): 2 Y 2(u, v) Y 0 ( u, v ) = exp{ 1 4 (u2 + v 2 )(R2 2 R0/ 2 2 )}; (18) R 2 /R 0 = = Y 0 (0, 0)/Y 2 (0, 0). Along with (11), (9) and R 1 = R 0, we have ix equation for five unknown. It i overdetermined, which enable u to ue information from three frame to form one etimation: w 1 M 2 (u,v) I 2 4 Y 1(u, v) u 2 + v 2 ln(2 2 Y 2(u, v) Y 0 ( u, v Y ) 0 ( u, v (19) ) ); R 0 = w 2 ( 4 1) + 2 ( 4 1). (20) where I 2 i the region within which the ummation i well-defined and M 2 i the number of (u, v) pair in I 2. A we can ee in the imulation reult, the etimation baed on multiple image improve the performance of our algorithm.

5 4. ERROR ANALYSIS The performance of our etimation algorithm can be evaluated by introducing an additive noie in the model: Y i (u, v) = H(u, v, R i )F i (u, v) + N i (u, v); i = 0, 1. (21) Combining (10), (11), (21) and the implified verion of (8) a in Section 3.1, we can give the etimation of OTF for firt frame with the preence of noie a: Ĥ(u, v, R 0 ) = [ 2 Y 1 (u, v) N 1 (u, v) Y 0 (u/, v/) N 0 (u/, v/) ] 2 The etimation of the focued image i given by ˆF 0 (u, v) = Y 0 (u, v)/ĥ(u, v, R 0), while the noie free etimation i F 0 (u, v) = Y 0 (u, v)/h(u, v, R 0 ). Thi make u arrive at the noiy verion of focued image etimate a: ˆF 0 (u, v) = F 0 (u, v)[ Y 1(u, v) N 1 (u, v) Y 1 (u, v) 4 1. Y 0 ( u, v ) Y 0 ( u, v ) N 0( u, v (22) )] Notice that the original additive noie become multiplicative noie in the final etimation. The tatitical characteritic of the noie alo change. The random variable inide the quare bracket i the ratio of two non-zero mean normal random variable. It ditribution ha been tudied, 9 baed on which we can tudy the ditribution and expectation of the noie. More importantly, by making a realitic aumption that ignal-tonoie ratio (SNR) of the two blur image are identical, we notice that the term inide the quare bracket ha value cloe to one. Thi ugget that SNR of the etimation in (22) will be high, which claim the uperiority of our algorithm in term of uppreing noie. 5. ESTIMATION OF PHASE TRANSFER FUNCTION Section 3 preent our algorithm for etimating the OTF when it ha no phae component. However the real camera ytem doe introduce a diturbance on the phae of the original image. Unfortunately, no pecific knowledge on PTF i available in linear optic and it alo depend ignificantly on phyical pecification of variou len, uch a material, hape and ize. Here we provide an approach for etimating the PTF without the knowledge of phyical characteritic of the len. Recall from (3) and (5) that the two blurred image ha the following relationhip in frequency domain: 2 Y 1(u, v) Y 0 ( u, v ) = H(u, v, R 1) H(u/, v/, R 0 ) exp{j2π(ut x + v t y )}. (23) Let u aume now the OTF conit of MTF H(u, v, R i ) and PTF θ i (u, v): H(u, v, R i ) = H(u, v, R i ) exp{j θ i (u, v)}, i = 0, 1. (24) We regard the PTF a a mooth function mainly determined by the camera ytem and thu conitent within thee two frame, i.e., θ 1 (u, v) = θ 0 (u, v) θ(u, v). Therefore, a relationhip on phae between the two frame can be derived from Eq. (23) a the following: Y 1 (u, v) Y 0 ( u, v ) 2π(ut x + v t y ) = θ(u, v) θ(u, v ), (25) where Y i (u, v) denote the phae of the image in frequency domain. Note that the LHS of (25) i conit of known variable. Without auming any pecific form of θ, one idea for olving the above equation i to ue Taylor expanion. We apply the 2D Taylor expanion of the firt order to θ(u, v) at the point (u/, v/), and denote the LHS of (25) a C(u, v) to obtain: C(u, v) = ( 1) u θ u( u, v ) + ( 1)v θ v( u, v ), (26)

6 where θ u ( u, v ) denote the partial derivative of θ with repect to u evaluated at the point of ( u, v ). Similar interpretation applie to θ v ( u, v ). Denote D( u, v ) = C(u,v) 1 and perform a change of variable, ξ = u and η = v, we arrive at a nonlinear partial differential equation (PDE) a follow: θ(ξ, η) θ(ξ, η) D(ξ, η) = ξ + η. (27) ξ η A general olution 10 can be obtained in the following form: 1 θ(ξ, µ) = D(ξ, ξµ)dξ + Φ(µ), ξ µ = η/ξ. (28) In the above integral, µ i conidered a a fixed parameter. Φ i an arbitrary function that depend on the boundary condition of the above PDE. For implicity, we aume Φ = 0. The olution in (28) i in the form of integral, and it i approximately equivalent to ummation when the variable are dicrete. (In our cae, we have dicrete Fourier tranform). Equation (28) i thu equivalent to the following explicit equation: θ(ξ, µ) = ξ i=0 1 D(i, iµ). (29) i After the ummation,we ubtitute µ by µ = η/ξ to get θ(ξ, η). And we can further obtain θ(u, v) by ubtituting ξ = u/ and η = v/. 6. SIMULATION RESULTS We tet the effectivene of our algorithm in video equence captured by a digital camcorder. The equence are captured in a frame rate of 10fp and with a reolution of In all equence, the camcorder move toward the object and along the optical axi, i.e., no rotation are preented. The camcorder ued here ha a large focu, namely the focu depth i far from the len. In thi cae, when depth decreae the blur radiu will increae. Fig.1 (a) how the frame 1, 3, 5, 7 and 9 of the video equence BOOK, in which the planar object i a book poitioning parallel to the camera. Fig.1 (b) how the equence blurred by a imulated Geometric blur a in (14) with blur radiu for frame 1 a R 1 = 8 (in pixel). Fig.1 (c) i the recontruction for frame 1 uing increaing number of frame, i.e., the econd image i the etimated focued frame 1 computed uing frame 1, frame 2 (not hown) and frame 3 in the blurred equence (b). It i eay to notice that multiple-frame etimation perform much better than two-frame etimation. The etimated frame become very cloe to the original frame when the number of frame ued exceed 5. Thi i further demontrated in Fig. 4 (a), which how the blur radiu etimation veru groundtruth. The black olid line repreent the true blur radiu for frame 1 R 1 = 8, with two black dahed line repreenting ± 5%R 1 within which the recontruction will have reaonable quality. The blue line repreent the etimation of blur radiu uing two frame: only frame 1 and the current frame; while the red line i the reult uing current frame and all the previou frame, namely, it i an accumulative average of the blur line. A we can ee, although the two-frame etimation varie from -25%R 1 to 25%R 1 among different frame, multiple-frame etimation give teady reult and lie within ± 5%R 1 after frame number exceed 6. Fig.2 (a) how the frame 6, 56, 106, 156, 206 and 256 of a long video equence SOCCER, in which the object occer i an approximately planar object. Fig.2 (b) how the equence blurred by a imulated Gauian blur a in (10) with a linearly increaing blur radiu from R 6 = 4 to R 206 = 12. The increaing blur radiu model the effect of decreaing depth caued by forward camera motion. Fig.2 (c) i the recontructed equence where 5 frame are ued for etimating each frame. A can be een that the etimation of focued equence give contantly good performance over a large number of frame. Fig.3 (a) how the frame 1, 31, 61, 91 and 121 of a video equence DESK, where multiple object form a background-foreground cene. Gauian ynthetic blur are added according to the depth of the object a hown

7 (a) Original image frame (b) Blurred image frame (c) Recontructed image frame Figure 1. Etimation reult for frame 1 in video BOOK, in the cae of a geometric blur PSF. (a) Original image frame (b) Blurred image frame (c) Recontructed image frame Figure 2. Etimation reult for frame 6, 56, 106, 156, 206 and 256 in video SOCCER, in the cae of a Gauian blur PSF. in Fig.3 (b). A mentioned in Section 2, our algorithm can be applied to maller region of the image to enure each region ha the ame depth. In a rather imple cae a in DESK, we can divide the frame into background

8 region and foreground region. Fig.3 (c) i the recontructed equence where we ue 8 frame for etimating each frame. It can be een that our algorithm give high-quality recontruction for both the foreground object and the background. Fig. 4(b) i a plot with the groundtruth blur radiu for background (olid black line) and the one for foreground (black line with dot). Similarly a in SOCCER, the imulated blur increae when the depth decreae. The blue line and the red line repreent the blur etimation for background and foreground, repectively. They are both cloe to the groundtruth, which verifie the robutne of our algorithm in dealing with 3D cene. (a)original image frame (b)blurred image frame (c)recontructed image frame Figure 3. Etimation reult for frame 1, 31, 61, 91 and 121 in video DESK, in the cae of a Gauian blur PSF. (a) Etimated Blur radiu for equence BOOK veru groundtruth, in the cae of two-frame etimation and multiple-frame etimation (b) Etimated Blur radiu for equence DESK veru groundtruth, in the cae of different blur radiue for background and foreground. Figure 4. Plot for etimated blur radiue veru groundtruth for video equence BOOK and DESK.

9 7. CONCLUSIONS In thi paper, we introduced a novel method for focued image equence etimation. The propoed algorithm relie on the difference in blur characteritic of multiple image reulting from camera motion in video equence. Noie analyi ugget the etimation improve SNR while computer imulation prove the competence of our approach. Our technique could potentially be ued to replace the expenive apparatu required for auto-focu adjutment by miniature engine or enor in many camera device. ACKNOWLEDGMENTS The author would like to thank the Phyical Realization Reearch Center of Excellence, Motorola Lab, for upporting thi work. REFERENCES 1. M. Subbarao, Efficient depth recovery through invere optic, Machine Viion for Inpection and Meaurement (H. Freeman, Ed.), New York: Academic, J. En and P. Lawrence, An invertigation of method for determining depth from focu, IEEE Tran. on Pattern Analyi and Machine Intelligence 15, pp , Feb J. Rayala, S. Gupta, and S. Mullick, Etimation of depth from defocu a polynomial ytem identification, IEE Proceeding, pp , A. Rajagopalan, S. Chaudhuri, and U. Mudenagudi, Depth etimation and image retoration uing defocued tereo pair, IEEE Tran. on Pattern Analyi and Machine Intelligence 26, pp , Nov A. Levi and H. Stark, Image retoration by the method of generalized projection with application to retoration from magnitude, Journal of the Optical Society of America A 1, pp , Sep J. Yang, D. Schonfeld, C. Chen, and M. Mohamed, Online video tabilization baed on particle filter, in IEEE International Conference on Image Proceing, (Atlanta, GA), Nov R. Bracewell, K. Chang, A. Jha, and Y. Wang, Affine theorem for two dimenional fourier tranform, Electronic Letter, Feb F. Bowman, Introduction To Beel Function, Dover Publication Inc., New York, D. Hinckley, On the ratio of two correlated normal random variable, Biometrika, A.D.Polyanin, V.F.Zaitev, and A. Mouiaux, Handbook of firt order partial differential equation, Taylar and Franci, New York, 2002.