arxiv: v1 [cs.cv] 18 Apr 2017

Transcription

1 Ligh Field Blind Moion Deblurring Praul P. Srinivasan 1, Ren Ng 1, Ravi Ramamoorhi 2 1 Universiy of California, Berkeley 2 Universiy of California, San Diego 1 {praul,ren}@eecs.berkeley.edu, 2 ravir@cs.ucsd.edu arxiv: v1 [cs.cv] 18 Apr 2017 Absrac We sudy he problem of deblurring ligh fields of general 3D scenes capured under 3D camera moion and presen boh heoreical and pracical conribuions. By analyzing he moion-blurred ligh field in he primal and Fourier domains, we develop inuiion ino he effecs of camera moion on he ligh field, show he advanages of capuring a 4D ligh field insead of a convenional 2D image for moion deblurring, and derive simple mehods of moion deblurring in cerain cases. We hen presen an algorihm o blindly deblur ligh fields of general scenes wihou any esimaion of scene geomery, and demonsrae ha we can recover boh he sharp ligh field and he 3D camera moion pah of real and synheically-blurred ligh fields. 1. Inroducion Moion blur is he resul of relaive moion beween he scene and camera, where phoons from a single incoming ray of ligh are spread over muliple sensor pixels during he exposure. In his work, we make boh heoreical and pracical conribuions by sudying he effecs of camera moion on ligh fields and presening a mehod o resore moionblurred ligh fields. Ligh field cameras are ypically used in siuaions wih opically significan scene deph ranges and ou-of-plane camera moion, so i is imporan o consider how moion blur varies boh spaially wihin each subaperure image and angularly beween sub-aperure images. Theory We derive a forward model ha describes a moion-blurred ligh field as an inegraion over ransformaions of he sharp ligh field along he camera moion pah. By analyzing he moion-blurred ligh field in he primal and Fourier domains (Sec. 3 and Figs. 3, 4, 5), we show ha capuring a ligh field enables novel mehods of moion deblurring ha are no possible wih jus a convenional image. Firs, we show ha a ligh field blurred wih in-plane camera moion is a simple convoluion of he sharp ligh field wih he camera moion pah kernel, regardless of he deph conens of he scene (Sec ). This allows us o use simple deconvoluion o resore he sharp ligh field, which canno be done wih convenional images because he magniude of he moion blur is deph-dependen (Figs. 4, 6). Addi- Figure 1. We heoreically sudy he effecs of moion blur on a capured ligh field and presen a pracical algorihm o deblur ligh fields of general scenes capured wih 3D camera moion. Lef: a 4D ligh field (visualized as a 2D sub-aperure image and a 2D epipolar slice) is blurred by he synheic camera moion shown in he inse. Righ: absen knowledge of he synheic moion pah, our algorihm is able o accuraely recover he sharp ligh field and he moion pah. See Fig. 9 for examples wih real handheld camera moion. ionally, we show ha a ligh field blurred wih ou-of-plane camera moion is an inegral over shears of he sharp ligh field (Sec ). Therefore, we can blindly deblur a ligh field of a exured plane capured wih ou-of-plane camera moion by modulaing a slice of he Fourier specrum of he moion-blurred ligh field (Figs. 4, 7). This is no possible for convenional images due o he spaially-varying blur caused by ou-of-plane camera moion. Pracical Algorihm General ligh fields of 3D scenes capured wih 3D camera moion are inegrals over composiions of shears and shifs of he sharp ligh field. The general ligh field blind moion deblurring problem lacks a simple analyic approach and is severely ill-posed because here is an infinie se of pairs of sharp ligh fields and moion pahs ha explain any observed moion-blurred ligh field. We propose a pracical ligh field blind moion deblurring algorihm o correc he complex blurring ha occurs in siuaions where ligh field cameras are useful (Sec. 4). Our forward model is differeniable wih respec o he camera moion pah parameerizaion and he esimaed ligh field,

2 allowing us o simulaneously solve for boh using firsorder opimizaion mehods. Furhermore, by reaing moion blur as an inegraion of ransformaions of he sharp ligh field, we can simplify he problem by bypassing any esimaion of scene geomery. Insead of solving for a dense marix ha represens spaially and angularly varying moion blurs or separaely deblurring each sub-aperure image by solving for a 2D blur kernel and 2D deph map, we direcly solve for a parameerizaion of he coninuous camera moion curve in R 3. This is a much lower-dimensional opimizaion problem, and i allows us o uilize he srucure of he ligh field o efficienly recover he moion curve and sharp ligh field. Finally, we demonsrae he performance of our algorihm on real (Fig. 9) and synheically-blurred (Figs. 1, 8) ligh fields. 2. Relaed Work Ligh Fields The 4D ligh field [14, 22, 24] is he oal spaio-angular disribuion of ligh rays passing hrough free space, and ligh field cameras capure he ligh field ha exiss inside he camera body [27]. A convenional 2D fullaperure image is produced by inegraing he rays enering he enire aperure for each spaial locaion. Therefore, a capured ligh field will be ineresing and more useful han a convenional image when he equivalen full-aperure image conains significan deph-of-field effecs, because his indicaes ha rays from differen regions of he aperure have differen values. Common phoography siuaions where capuring a 4D ligh field would be useful include macro and porrai phoography. Previous work has demonsraed he benefis of lifing problems in compuer vision, compuer graphics, and compuaional phoography ino he 4D ligh field space. These include rendering 2D pinhole images as slices of he 4D ligh field [22], sereo reconsrucion from a single capure [2], changing he focus and deph-of-field of phoographs afer capure [27], correcing lens aberraions [26], passive deph esimaion [33], glare arifac reducion [31], and scene flow esimaion [32]. Previous works have also examined he Fourier specrum of ligh fields for various purposes. Chai e al. [6] analyzed he specral suppor of ligh fields for sampling in ligh field rendering and showed ha Lamberian objecs a specific dephs correspond o angles in he Fourier domain. Durand e al. [11] analyzed he effecs of shading, occlusion, and propagaion on he ligh field specrum. Ng [25] showed ha refocusing a 2D full-aperure image is equivalen o aking 2D slices of he 4D ligh field specrum and analyzed he performance of ligh field refocusing. Liang and Ramamoorhi [23] developed a ligh ranspor framework o invesigae he fundamenal limis of ligh field camera resoluion. Dansereau e al. [9] derived he 4D specral suppor of ligh fields for rendering, denoising, and refocusing. Addiionally, Egan e al. [12] analyzed he specrum of moionblurred 3D space-ime images o derive filers for efficien rendering of moion-blurred images. In his work, we analyze he Fourier specrum of moion-blurred ligh fields o provide inuiion for he effecs of camera moion on he capured ligh field and mehods o deblur ligh fields. Moion Deblurring Blind moion deblurring, removing he moion blur given jus a noisy blurred image, is a very challenging problem ha has been exensively sudied (see [19] for a recen review and comparison of various algorihms). Represenaive mehods for single image blind deblurring include he variaional Bayes approaches of Fergus e al. [13] and Levin e al. [21], and algorihms using novel image priors such as normalized sparsiy [18], an evolving approximaion o he L 0 norm [34], and L 0 norms on boh image gradiens and inensiies [28]. Previous muli-image blind deblurring works have also presened algorihms ha recover a single 2D image, given muliple observaions ha have been blurred differenly [10, 35, 36]. Jin e al. [15] presen a mehod ha uses a moionblurred ligh field of a scene wih wo deph layers o recover a 2D image and bilayer deph map. Our mehod also akes a moion-blurred ligh field as inpu, bu we recover a full 4D deblurred ligh field as opposed o a 2D exure. Moreover, our mehod does no need o esimae a deph map. Many compuaional phoography works have modified he imaging process o make moion deblurring easier. Raskar e al. [30] used coded exposures o preserve high frequency deails ha would be aenuaed due o objec moion. Anoher line of work focused on modified imaging mehods o engineer poin spread funcions ha would be invarian o objec moion. This includes focal sweeps [4, 17], parabolic camera moions [7, 20], and circular sensor moions [3]. In conras, we focus on he problem of deblurring ligh fields ha have already been capured, and we do no modify he imaging process. Concurrenly wih our work, Dansereau e al. [8] inroduced a non-blind algorihm o deblur ligh fields capured wih known camera moion. 3. A Theory of Ligh Field Moion Blur In our analysis below, we perform a flaland analysis of moion-blurred ligh fields wih a single angular dimension u and a single spaial dimension x, and noe ha i is sraighforward o exend his o he full 4D ligh field wih spaial dimensions (x, y) and angular dimensions (u, v). We focus on 3D as opposed o 6D camera moion, so he camera moion pah is a general 3D curve and he opical axis does no roae Forward Model The observed blurred ligh field is he inegraion over he ligh fields capured a each ime during he exposure: f(x, u) = l (x, u )d, (1)

3 Figure 2. Lef: we use a 2-plane parameerizaion for ligh fields, where each ray (x, u) is defined by is inercep wih he u and x planes separaed by disance s. Noe ha he x coordinae is relaive o he u coordinae, which is convenien for laer derivaions. Righ: consider a camera ranslaing along a pah p() = (p x(), p y(), p z()) during is exposure (in flaland we consider x and z only). The local camera coordinae frame for each ime has is origin locaed a he cener of he camera aperure. The ligh field l (x, u ) is he sharp ligh field ha would have been recorded by he camera a ime, in he local camera coordinaes a ime. The diagram shows ha ray (x, u ) in he local coordinae frame a ime is equal o ray (x, u + p x() x s p z()) in he local coordinae frame a ime = 0. where f is he observed ligh field and l (x, u ) is he sharp ligh field a ime during he exposure. Figure 2 illusraes ha he ligh field a ime is a ransformaion of he sharp ligh field a ime = 0, l(x, u), based on he camera moion pah p() = (p x (), p z ()) (p y () is no included in he flaland analysis bu is included in he full 3D model). Our ligh field parameerizaion is equivalen o considering he ligh field as a collecion of pinhole cameras wih ceners of projecion u and sensor pixels x, and we se he separaion beween he x and u planes s = 1 so x is a ray s spaial inercep 1 uni above u in he z direcion. The observed moion-blurred ligh field is hen f(x, u) = l(x, u + p x () xp z ())d. (2) Since he ligh field conains all rays ha inersec he wo parameerizaion planes, his forward model accouns for occluded poins, as long as he parameerizaion planes lie ouside he convex hull of he visible scene geomery. Cerain rare scenarios, such as a macro phoography sho where he camera moves beween blades of grass during he exposure, may violae his assumpion, bu i generally holds for ypical phoography siuaions. This model also assumes ha he ligh field parameerizaion planes are infinie, because camera moion can cause he sharp ligh field a ime o conain rays ouside he field-of-view of he ligh field a a previous ime Space-Angle and Fourier Analysis We examine he moion-blurred ligh field in he primal space-angle and Fourier domains o beer undersand he effecs of camera moion on he capured ligh field. We denoe signals in he Fourier domain wih capial leers, and use Ω x and Ω u o denoe spaial and angular frequencies. I is useful o uilize he Affine Theorem for Fourier ransforms [5, 29]: if h(a) = g(mb + c), where M is a marix, a, b, and c are vecors, and h and g are funcions, he relevan Fourier ransforms are relaed as follows: H(Ω) = de(m) 1 G(M T Ω) exp(2πiω T M 1 c), (3) where de(m) is he deerminan of M and i = 1. We use his o ake he Fourier ransform of he observed moion-blurred ligh field in Eq. 2, wih ransformaion marices M = ( ) ( 1 0 p z() 1 and c = 0 ) p x() : F (Ω x, Ω u) = L (Ω x + p z()ω u, Ω u) exp [2πiΩ up x()] d. As visualized in Fig. 5, he Fourier specrum is an inegraion over shears based on he ou-of-plane moion p z () and here is also a phase in he complex exponenial corresponding o in-plane moion. This complex exponenial is he Fourier ransform of δ(x)δ(u+p x ()), so we can rewrie he flaland primal domain moion-blurred ligh field as f(x, u) = [l(x, u xp z()) δ(x)δ(u + p x())]d. (5) The spaial and frequency domain expressions now separae in-plane moion, which is a convoluion wih a kernel corresponding o he in-plane camera moion pah, and ouof-plane moion, which is an inegraion over shears in boh he spaial and frequency domains. Noe ha his convoluion kernel is resriced o a subspace of he ligh field space (1D subspace of 2D for flaland ligh fields, and 2D subspace of 4D for full ligh fields). To gain greaer insigh ino hese expressions, we consider wo special cases for purely in-plane camera moion, and purely ou-of-plane camera moion, wih general moion being an inegral over composiions of hese wo cases In-Plane Camera Moion For camera moion pahs ha are parallel o he x and u parameerizaion planes, p z () = 0, and he expression for he primal domain moion-blurred ligh field simplifies o f(x, u) = l(x, u) = l(x, u) δ(x)k(u), δ(x)δ(u + p x ())d (4) (6)

4 Figure 3. In-plane camera moion is equivalen o a convoluion of he ligh field and he corresponding muliplicaion of he Fourier specrum. We are able o easily recover a ligh field blurred wih known in-plane camera moion using 4D deconvoluion. Noe ha in-plane camera moion causes spaially-varying (wih x) blur due o varying scene dephs, as shown by he whie brackes, while he blur magniude does no vary angularly (wih u), as shown by he yellow verical arrows. Figure 4. Ou-of-plane camera moion is equivalen o an inegraion over shears in boh he primal and Fourier domains. Noe ha ou-ofplane camera moion causes boh spaially and angularly varying blur. Given a ligh field of a single frono-parallel exured plane (Vincen van Gogh s Whea Field wih Cypresses ) wih ou-of-plane camera moion, we can blindly recover he exure, wih sligh arifacs due o finie aperure and edge effecs, by modulaing a 2D slice of he 4D Fourier specrum. Figure 5. General 3D camera moion is an inegraion over shears and shifs of he ligh field and an inegraion over shears and phase muliplicaions of he Fourier specrum. Blindly deblurring ligh fields capured wih general camera moion lacks a simple analyic approach and is severely ill-posed, so we solve his as a regularized inverse problem. where k(u) = R δ(u + px ())d is he inegraed in-plane where K(Ωu ) = R exp[2πiωu px ()]d is he inegraed in- camera moion pah. In he Fourier domain, plane blur kernel specrum. Z F (Ωx, Ωu ) =L(Ωx, Ωu ) exp[2πiωu px ()]d =L(Ωx, Ωu )K(Ωu ), (7) An imporan insigh is ha for in-plane camera moion, i is possible o ake he original ligh field ou of he inegral. This clearly idenifies he moion-blurred ligh field as a simple convoluion of he sharp ligh field wih he inplane blur kernel, regardless of he conen and range of dephs presen in he scene, as illusraed in Fig. 3. No such

5 Figure 6. Ligh fields of general 3D scenes blurred wih known in-plane camera moion can be recovered by simple 4D deconvoluion. This is no possible wih convenional 2D images because he moion blur magniude is deph-dependen. We synheically blur a ligh field wih increasing linear in-plane moion, and noe ha he roo mean square error (RMSE) of he cenral sub-aperure image obained by 2D deconvoluion consisenly increases, while he RMSE of he cenral sub-aperure image obained by 4D deconvoluion of he full ligh field says relaively consan. simple resul holds for convenional 2D images, as quanified in Fig. 6, because he moion blur magniude is dephdependen. Inuiively, in-plane moion is a convoluion of he sharp ligh field because ligh field cameras a poins along he moion pah observe he same se of rays shifed, while convenional cameras a poins along he moion pah observe disjoin ses of rays. If we know he blur kernel, we can recover he sharp ligh field by simple deconvoluion, as shown in Figs. 3, 6. However, if boh he blur kernel and ligh field are unknown, we need o use priors o esimae he blur kernel and sharp ligh field, as discussed in Sec Ou-of-Plane Camera Moion For purely ou-of-plane camera moion, p x () = 0, and he expression for he primal domain moion-blurred ligh field simplifies o f(x, u) = l(x, u xp z ())d (8) In he Fourier domain, F (Ω x, Ω u ) = L(Ω x + p z ()Ω u, Ω u )d. (9) These are simply inegraions over differen shears of he ligh field, as illusraed in Fig. 4. I is paricularly ineresing o consider he ligh field of a exured frono-parallel plane w(x) a deph z. The geomery of our ligh field parameerizaion indicaes ha l(x, u) = w(xz + u). In he primal domain, he ou-of-plane moion-blurred ligh field of his exured plane is f(x, u) = w(x(z p z()) + u)d = w(xz() + u)d, where we define z() = z p z (). (10) Using he Affine Theorem for Fourier ransforms wih ransformaion marices M = ( ) z() and c = ( 0 0 ), afer noing ha he original Fourier ransform of he exured plane is W (Ω x )δ(ω u ), he Fourier ransform of he ou-ofplane moion-blurred ligh field is ( ) ( 1 F (Ω x, Ω u ) = z() W Ωx δ Ω u Ω ) x d. z() z() (11) This is also an inegraion over various shears, each a line wih slope given by Ω u = Ω x /z(). The moion-blurred ligh field akes he original exure frequencies (in a line) and shears hem o lines of differen slopes, followed by inegraion. Using he sifing propery of he dela funcion, we can simplify he above expression, zmax ( F (Ω x, Ω u ) =W (Ω u ) δ Ω u Ω ) x γ(z) dz, z min z (12) where we have swiched o inegraion over z direcly (effecively subsiuing z for ), and for simpliciy, we assume z() monoonically increases wih ime. The erm γ(z) = ( z dz/d) 1 accouns for he 1/ z facor and change of variables. Inuiively, he moion-blurred ligh-field specrum is a double wedge [6, 9, 11], bounded by slopes z min and z max and conaining an infinie number of lines in he frequencydomain. The magniudes along each line are he same, deermined by he original exure W (Ω u ), bu every value is uniformly scaled by a facor γ(z), based on he amoun of ime he camera lingered a ha deph (oher han W (0), which is consan for all lines). The dela funcion in Eq. 12 can hen be evaluaed, seing z = Ω x /Ω u, leading o he simple expression, ( ) Ωx F (Ω x, Ω u ) = W (Ω u )β, (13) where he funcion β includes γ, as well as he change of variables from he dela funcion, and is given by β(z) = z Ω x dz/d Ω u ( β(ω x /Ω u ) = Ω u dz 1 d. Ωx/Ω u) (14) Texure Recovery The srucure of he ou-of-plane moion-blurred ligh field enables blind deblurring by a very simple facorizaion (essenially a rank-1 decomposiion of he 2D ligh field marix ino 1D facors for W and γ or β). One simple approach is o esimae W from any line, hen fix he scaling by comparing he overall magniude of W across lines o esimae he moion blur kernel (β or γ), and finally divide W (0) by he oal exposure ime.

6 Figure 7. Visualizaion of process o blindly recover a exured plane from a ligh field capured wih ou-of-plane camera moion. Taking a slice of he ligh field in he Fourier domain can be implemened in he primal domain by a sheared inegral projecion, and his is equivalen o refocusing he full-aperure image o a specific deph [25]. Inuiively, his means ha blind deblurring of he exure can be performed in he primal domain by compuing he full-aperure image refocused o a single deph during he exposure. In summary, we can separaely esimae he blur kernel and he original exure for ou-of-plane moion of a ligh field camera, assuming a single frono-parallel exured plane, by exracing a slice in he Fourier domain or equivalenly refocusing he full-aperure image in he primal domain. Figure 4 shows an example of a ligh field of a exured plane blurred by linear ou-of-plane moion. We are able o blindly recover he exure, wih sligh arifacs due o finie aperure and edge effecs, by compuing a sheared inegral projecion (equivalen o aking a 2D slice of he 4D Fourier specrum). As a pracical noe, when compuing his in he discree case, we mus linearly scale frequencies in he exraced slice by ζω x + 1 o correc for he value of each discree frequency being spread across he shear lengh during he exposure, where ζ is a consan corresponding o he relaive ime he camera lingers a he deph corresponding o ha slice. ζ can be auomaically deermined by sampling Fourier slices and comparing heir magniudes o calculae he relaive ime spen a each deph along he moion pah. This process is visualized in Fig. 7. As deailed in [25], he resoluion of he recovered Fourier slice is limied by he angular resoluion of he ligh field camera. Comparison wih a Convenional Image I is also insighful o compare his o informaion available from a convenional 2D image (1D in flaland), corresponding o he view from he cenral pinhole of a ligh field camera. In his case, we se u = 0 in Eq. 10, defining l(x) = w(xz ). Since we are now working in 1D, from he Fourier scale heorem, ( ) 1 zmax ( ) F (Ω x) = z() W Ωx Ωx d = W γ(z)dz. z() z z min (15) This is similar o he ligh field case, excep ha we no longer have he dela funcion for muliple sheared lines in 2D; indeed we only have a single 1D line, wih a frequency specrum scaled according o z. I is clear ha from he perspecive of analysis and recovery, he convenional image case provides far less insigh han in he ligh field case. We canno separae ou he exure W, and he mehods for recovery discussed in he ligh field case do no apply, since here are no muliple lines in a 2D specrum we can sudy. In fac, i is no even sraighforward o recover exure and moion blur kernel even when one of he facors is known General 3D Moion and Scenes General 3D camera moion is an inegral over composiions of shears and shifs of he ligh field, as shown in Fig. 5. Blindly deblurring a ligh field of a general scene capured wih 3D camera moion lacks a simple analyic approach and is a severely ill-posed problem because here is an infinie se of pairs of ligh fields and moion pahs ha explain any observed moion-blurred ligh field. Below, we presen an algorihm o esimae he sharp ligh field and camera moion pah by solving a regularized inverse problem. 4. Blind Ligh Field Deblurring Algorihm For blind ligh field moion deblurring, we esimae boh he camera moion curve p() and he sharp ligh field l. We uilize our forward model derived in Eq. 2 o formulae a regularized inverse problem, and our approach is paricularly efficien due o our direc represenaion of he camera moion curve, as discussed below. We solve a discree opimizaion problem, since ligh field cameras record samples and no coninuous funcions: min l,p() ˆf(l, p()) f λψ(l), (16) where he firs erm minimizes he L 2 norm of he difference beween he observed moion-blurred ligh field f and ha prediced by he forward model ˆf, and he second erm, ψ(l), is a prior on he sharp ligh field. To address finie aperure and sensor planes, we assume replicaing boundaries for he sharp ligh field. We use bilinear inerpolaion o ransform he sharp ligh field along he camera moion pah, so our forward model is differeniable wih respec o he camera pah and he sharp ligh field. Camera Moion Pah Represenaion We model he camera moion pah p() as a Bézier curve made up of n conrol poins in R 3. This approach is much more efficien han he alernaive approaches of solving for a dense marix o represen spaially and angularly varying blurs, or separaely deblurring each sub-aperure image. A dense moion blur ray ransfer marix would have size r r, where r is he number of rays sampled by he ligh field camera (his marix would have size for he ligh fields used in his work). Separaely deblurring each sub-aperure image involves esimaing a 2D deph map and a 2D convoluion kernel, each of size s s, where s is he number of samples along each spaial dimension (his equaes o solving for wo marices of size for he ligh fields used in his work). Insead, we solve for a much lowerdimensional vecor of conrol poins wih 3n elemens. In pracice, we find ha ypical camera moion pahs can be represened by n = 3 or n = 4 conrol poins.

7 Figure 8. Blind deblurring resuls on synheically moion-blurred ligh fields. Our algorihm is able o correcly recover he sharp ligh field and esimae he 3D camera moion pah, while alernaive mehods perform poorly due o he large spaial variance in he blur. Addiionally, as demonsraed by he epipolar images, oher algorihms do no recover a ligh field ha is consisen across angular dimensions. The roo mean square error (RMSE) of our deblurred resuls are consisenly lower han hose of he alernaive mehods. Ligh Field Prior To regularize he inverse problem above, we use a 4D version of he sparse gradien prior proposed in [34]: ( 1 2 X if l, 2 l ψ(l) = (17) 1 oherwise. x,y,u,v This funcion gradually approximaes he L0 norm of gradiens by hresholding a quadraic penaly funcion parameerized by, and approaches he L0 norm as 0. Implemenaion Deails We uilize he auomaic differeniaion of Tensorflow [1] o differeniae he loss of he blind deblurring problem in Eq. 16 wih respec o he camera moion pah conrol poins and sharp ligh field, and use he firs-order Adam solver [16] for opimizaion. While he prior in Eq. 17 is effecive for esimaing he camera moion pah, he sharp ligh fields esimaed using his prior ypically appear unnaural and over-regularized. We hold he camera moion pah consan and solve Eq. 16 for he sharp ligh field using a 4D oal variaion (L1 norm of gradiens) prior o obain he final sharp ligh field Resuls We validae our algorihm using ligh fields capured wih he Lyro Illum camera, ha have been blurred by boh synheic camera moion wihin a 2 mm cube using our forward model in Eq. 2 and real handheld camera moion using a shuer speed of 1/20 second. We compare our resuls o he alernaive of applying sae-of-he-ar blind image moion deblurring algorihms o each sub-aperure image. As shown in a recen review and comparison paper [19], he algorihms of Krishnan e al. [18] and Pan e al. [28] are wo of he op performers for blind deblurring of boh real and synheic images wih spaially-varying blur, so we compare our algorihm o hese wo mehods. As demonsraed by boh he synheically moion-blurred resuls in Fig. 8 and he real moion-blurred resuls in Fig. 9, our algorihm is able o accuraely esimae boh he sharp ligh field and he camera moion pah. The sae-of-he-ar blind image moion deblurring algorihms are no as successful due o he significan spaial variance of he blur. Furhermore, hey are no designed o ake advanage of he ligh field srucure and do no esimae a 3D camera moion pah, so heir resuls are inconsisen beween sub-aperure images, as demonsraed by he epipolar image resuls. In he synheically-blurred examples in Fig. 8, noe ha our algorihm correcly esimaes he ground ruh complex camera moion pahs and correcs he large spaiallyvarying blurs in he flowers and leaves. In he real handheld blurred examples in Fig. 9, noe ha our algorihm correcs he blur in he speculariies and edges of he circui componen, he leaves and flower of he rosemary plan, and he hair, eyebrows, eeh, and background plans.

8 Figure 9. Blind deblurring resuls on real handheld moion-blurred ligh fields. Our algorihm is able o correcly recover he sharp ligh field and esimae he 3D camera moion pah. Noe ha we correc he moion seen in he specular reflecions and objec edges in he circui componen example, he moion seen in he leaves and flower in he rosemary plan example, and he moion seen in he hair, eyebrows, eeh, and background plans of he porrai example. Furhermore, our mehod produces angularly-consisen resuls, as demonsraed by he epipolar slices of all 3 examples. Please view our supplemenary video and projec webpage for animaed visualizaions of our resuls. 5. Conclusion In his work, we sudied he problem of deblurring ligh fields of general scenes capured wih 3D camera moion. We analyzed he effecs of moion blur on he ligh field in he primal and Fourier domains, derived simple mehods o deblur ligh fields in specific cases, and presened an algorihm o infer he sharp ligh field and camera moion pah from real and synheically-blurred ligh fields. I would be ineresing o exend our forward model o accoun for 3D roaions of he opical axis, and heoreically analyze he effecs of camera roaion on he moion-blurred ligh field. Since he forward model would be differeniable wih respec o he roaion parameers, our blind deblurring opimizaion algorihm can easily generalize o accoun for camera roaion. We hink ha he insighs of his work enable fuure invesigaions of ligh field priors ha more explicily consider he effecs of moion blur on he ligh field, as well as novel inerpreaions of single and muli-image moion deblurring as subses of he general ligh field moion deblurring problem. Acknowledgmens This work was suppored in par by ONR gran N , NSF gran , NSF graduae research fellowship DGE , a Google Research Award, he UC San Diego Cener for Visual Compuing, and a GPU donaion from NVIDIA.

9 References [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Ciro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawa, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuser, J. Shlens, B. Seiner, I. Suskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Waenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor- Flow: Large-scale machine learning on heerogeneous sysems, [2] E. H. Adelson and J. Y. A. Wang. Single lens sereo wih a plenopic camera. IEEE Transacions on Paern Analysis and Machine Inelligence, [3] Y. Bando, B.-Y. Chen, and T. Nishia. Moion deblurring from a single image using circular sensor moion. In Compuer Graphics Forum, [4] Y. Bando, H. Holzman, and R. Raskar. Near-invarian blur for deph and 2D moion via ime-varying ligh field analysis. In ACM Transacions on Graphics, [5] R. N. Bracewell, K.-Y. Chang, A. K. Jha, and Y. Wang. Affine heorem for wo-dimensional fourier ransform. Elecronics Leers, [6] J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum. Plenopic sampling. In SIGGRAPH, , 5 [7] T. S. Cho, A. Levin, F. Durand, and W. T. Freeman. Moion blur removal wih orhogonal parabolic exposures. In ICCP, [8] D. G. Dansereau, A. Eriksson, and J. Leiner. Moion deblurring for ligh fields. In arxiv: , [9] D. G. Dansereau, O. Pizarro, and S. Williams. Linear volumeric focus for ligh field cameras. In ACM Transacions on Graphics, , 5 [10] M. Delbracio and G. Sapiro. Burs deblurring: removing camera shake hrough fourier burs accumulaion. In CVPR, [11] F. Durand, N. Holzschuch, C. Soler, E. Chan, and F. X. Sillion. A frequency analysis of ligh ranspor. In ACM Transacions on Graphics, , 5 [12] K. Egan, Y.-T. Tseng, N. Holzschuch, F. Durand, and R. Ramamoorhi. Frequency analysis and sheared reconsrucion for rendering moion blur. In ACM Transacions on Graphics, [13] R. Fergus, B. Singh, A. Herzmann, S. T. Roweis, and W. T. Freeman. Removing camera shake from a single image. In ACM Transacions on Graphics, [14] S. J. Gorler, R. Grzeszczuk, R. Szeliski, and M. Cohen. The lumigraph. In SIGGRAPH, [15] M. Jin, P. Chandramouli, and P. Favaro. Bilayer blind deconvoluion wih he ligh field camera. In ICCV Workshops, [16] D. Kingma and J. Ba. Adam: a mehod for sochasic opimizaion. In ICLR, [17] T. Kobayashi, F. Sakaue, and J. Sao. Deph and arbirary moion deblurring using inegraed PSF. ECCV Workshops, [18] D. Krishnan, T. Tay, and R. Fergus. Blind deconvoluion using a normalized sparsiy measure. CVPR, , 7 [19] W.-S. Lai, J.-B. Huang, Z. Hu, N. Ahuja, and M.-H. Yang. A comparaive sudy for single image blind deblurring. CVPR, , 7 [20] A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman. Moion-invarian phoography. In ACM Transacions on Graphics, [21] A. Levin, Y. Wiess, F. Durand, and W. T. Freeman. Efficien marginal likelihood opimizaion in blind deconvoluion. In CVPR, [22] M. Levoy and P. Hanrahan. Ligh field rendering. In SIG- GRAPH, [23] C.-K. Liang and R. Ramamoorhi. A ligh ranspor framework for lensle ligh field cameras. In ACM Transacions on Graphics, [24] G. Lippmann. La phoographie inégrale. In Compes- Rendus, Académie des Sciences, [25] R. Ng. Fourier slice phoography. In ACM Transacions on Graphics, , 6 [26] R. Ng and P. Hanrahan. Digial correcion of lens aberraions in ligh field phoography. In SPIE Inernaional Opical Design, [27] R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowiz, and P. Hanrahan. Ligh field phoographhy wih a hand-held plenopic camera. CSTR , [28] J. Pan, Z. Hu, and M.-H. Su, Z. Yang. Deblurring ex images via L0-regularized inensiy and gradien prior. CVPR, , 7 [29] R. Ramamoorhi, D. Mahajan, and P. Belhumeur. A firsorder analysis of lighing, shading, and shadows. ACM Transacions on Graphics, [30] R. Raskar, A. Agrawal, and J. Tumblin. Coded exposure phoography: Moion deblurring using fluered shuer. In ACM Transacions on Graphics, [31] R. Raskar, A. Agrawal, C. Wilson, and A. Veeraraghavan. Glare aware phoography: 4D ray sampling for reducing glare effecs of camera lenses. In ACM Transacions on Graphics, [32] P. P. Srinivasan, M. W. Tao, R. Ng, and R. Ramamoorhi. Oriened ligh-field windows for scene flow. In ICCV, [33] M. W. Tao, P. P. Srinivasan, J. Malik, S. Rusinkiewicz, and R. Ramamoorhi. Deph from shading, defocus, and correspondence using ligh-field angular coherence. In CVPR, [34] L. Xu, S. Zheng, and J. Jia. Unnaural L0 sparse represenaion for naural image deblurring. CVPR, , 7 [35] H. Zhang, D. Wipf, and Y. Zhang. Muli-image blind deblurring using a coupled adapive sparse prior. In CVPR, [36] X. Zhu, F. Sroubek, and P. Milanfar. Deconvolving PSFs for a beer moion deblurring using muliple images. In ECCV,