THE micro-lens array (MLA) based light field cameras,

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., A Generic Muli-Projecion-Cener Model and Calibraion Mehod for Ligh Field Cameras Qi hang, Chunping hang, Jinbo Ling, Qing Wang, Member, IEEE and Jingyi Yu Absrac Ligh field cameras can capure boh spaial and angular informaion of ligh rays, enabling D reconsrucion by a single exposure. The geomery of D reconsrucion is affeced by inrinsic parameers of a ligh field camera significanly. In he paper, we propose a muli-projecion-cener (MPC) model wih inrinsic parameers o characerize ligh field cameras based on radiional wo-parallel-plane (TPP) represenaion. The MPC model can generally parameerize ligh field in differen imaging formaions, including convenional and focused ligh field cameras. By he consrains of D ray and D geomery, a D projecive ransformaion is deduced o describe he relaionship beween geomeric srucure and he MPC coordinaes. Based on he MPC model and projecive ransformaion, we propose a calibraion algorihm o verify our ligh field camera model. Our calibraion mehod includes a close-form soluion and a non-linear opimizaion by minimizing re-projecion errors. Experimenal resuls on boh simulaed and real scene daa have verified he performance of our algorihm. Index Terms muli-projecion-cener (MPC) model; ligh field cameras; wo-parallel-plane (TPP) represenaion; calibraion INTRODUCTION THE micro-lens array (MLA) based ligh field cameras, including convenional ligh field camera [] and focused ligh field camera [], can capure radiance informaion of ligh rays in boh spaial and angular dimensions, i.e., D ligh field [], []. The daa from ligh field camera is equivalen o narrow baseline images of radiional cameras wih coplanar projecion ceners. The measuremen of same poin in muliple direcions allows or srenghens he applicaions in compuaional phoography and compuer vision, such as digial refocusing [], deph esimaion [], segmenaion [] and so on. Recen work also proposed he mehods on ligh field regisraion [] and siching [], [] o expand he field of view (FOV). To suppor hese applicaions, i is crucial o accuraely calibrae ligh field cameras and esablish exac relaionship beween he ray space and D scene. I plays an imporan role o build a model for describing he ray sampling paern of ligh field cameras. Previous approaches have deal wih imaging models on ligh field cameras in differen opical designs [], [], [], [], []. The common poins are based on he fac ha he microlens is regarded as a pinhole model and he main-lens is described as a hin-lens model. However, some of open issues sill remain in he models and mehods. Firsly, he proposed models focus on angular and spaial informaion of rays, bu he relaionship beween ligh field and D scene geomery is no explored. Secondly, very lile work has considered a generic model before o describe ligh field cameras wih differen image formaions [], []. Thirdly, Qi hang, Chunping hang, Jinbo Ling, and Qing Wang (corresponding auhor) are wih he School of Compuer Science, Norhwesern Polyechnical Universiy, i an, China ( qwang@nwpu.edu.cn). Jingyi Yu is wih he ShanghaiTech Universiy, Shanghai, China, ( jingyi.udel@gmail.com) The work was suppored by NSFC under Gran. Manuscrip received December, ; revised June,. exising inrinsic parameers of ligh field camera models are eiher redundan or incomplee such ha corresponding soluions are neiher effecive nor efficien. In he paper, we firs propose a muli-projecion-cener (MPC) model based on wo-parallel-plane (TPP) represenaion [], []. Then we deduce he ransformaions beween D scene geomery and D ligh rays. Based on geomery ransformaions in he MPC model, we characerize various ligh field cameras in a generic -inrinsic-parameer model and presen an effecive inrinsic parameer esimaion algorihm. Experimenal resuls on boh virual (simulaed daa) and physical (Lyro, Illum and a self-assembly focused) ligh field cameras have verified he effeciveness and efficiency of our model. Our main conribuions have hree aspecs, including () We deduce he ransformaions o describe he relaionship beween ligh field and scene srucure. () We describe ligh field cameras wih differen image formaions as a generic -parameer model wihou redundancy. () We propose an effecive inrinsic parameer esimaion algorihm for ligh field cameras, including a closedform linear soluion and a nonlinear opimizaion. The remainder of he paper is organized as follows. Secion summarizes relaed work on he models of ligh field cameras and calibraion mehods. Secion inroduces our MPC model and he ransformaions beween he D srucure and D ligh field. Based on he heory of ligh field parameerizaion, a generic -inrinsic-parameer ligh field camera model is proposed. Secion provides he deails of our calibraion mehod and analyzes compuaional complexiy of he closed-form soluion. In Secion, we presen exensive resuls on he simulaed and real scene ligh fields, demonsraing more accurae inrinsic parameer esimaion han previous work [], [].

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., RELATED WORK To acquire D ligh field, here are various imaging sysems developed from radiional camera. Wilburn e al. [] presen a camera array o obain ligh field wih high spaial and angular resoluions. Classic calibraion approach is employed for he camera array []. More general, in radiional muli-view geomery framework, muliple cameras in differen poses are defined as a se of unconsrained rays, which is known as Generalized Camera Model (GCM) []. The ambiguiy of he reconsruced scene is discussed in radiional opics []. However, such applicaions on he camera array are limied by is high cos and complex conrol. In conras, he MLA enables a single camera o record D ligh field more convenienly and efficienly, hough he baseline and spaial resoluion are relaively smaller han camera array. Compared o he camera array, muliple projecion ceners of MLA-based ligh field camera are aligned on a plane sricly due o physical design. Recen work devoes o inrinsic parameer calibraion of ligh field cameras in wo designs [], [], which are quie differen according o he image paern of micro-lenses. The main difference of ligh field cameras is he relaive posiion of main lens s imaging plane and he MLA plane []. I deermines rays disribuion from he same poin, which affecs he way o exrac sub-aperures from raw image, i.e., he micro-lens images [], []. However, he measuremens of he same poin in muliple direcions are obained in differen ypes of ligh field cameras, equivalen o he daa of GCM. Therefore, he ligh field camera model can use classic muli-view geomery heory for reference. Recenly, some sae-of-he-ar mehods have proposed models on convenional ligh field camera, where muliple viewpoins or sub-aperures are convenien o be synhesized. Dansereau e al. [] presen a model o decode pixels ino rays for a Lyro camera, where a -free-parameer ransformaion marix is relaed o reference plane ouside he camera (in nonlinear opimizaion, inrinsic parameers and disorion coefficiens are finally esimaed). However, he calibraion mehod using radiional camera calibraion algorihm is no effecive, also here are redundan parameers in he decoding marix. Bok e al. [] formulae a geomeric projecion model consising of a main lens and a MLA (heir exended work has been published in IEEE TPAMI []). Inrinsic parameers are esimaed by conducing raw images direcly and an analyical soluion is deduced. Moreover, Thomason e al. [] ry o deal wih he misalignmen of he MLA and esimaed is posiion and orienaion. Apar from his, oher researchers have explored models on he focused ligh field camera, where muliple projecions of he same poin are convenien o be recognized. Johannsen e al. [] propose o calibrae inrinsic parameers of he focused ligh field camera. By reconsrucing D poins from he parallax in adjacen micro-lens images, he parameers (including deph disorion) are esimaed. However, he geomery cener of micro image is on is micro-lens s opical axis in he camera model. This assumpion causes inaccuracy on reconsruced poins and esimaed resuls are finally compensaed by he coefficiens of deph disorion. Hahne e al. [] furher discuss he influence of above-menioned assumpion, i.e., he deviaion of micro-lens and is image. Heinze e al. [] apply a similar model wih Johannsen e al. [] and deduce a linear iniializaion for inrinsic parameers. In a word, previous ligh field camera models are eiher redundan or complex, which leads o a non-unique soluion of inrinsic parameer esimaion or inaccuracy of decoding ligh field. An unreliable camera model is also a boleneck ha migh impede ligh field applicaions for compuer vision and compuaional phoography, especially on ligh field regisraion, siching and enhancemen. To suppor furher applicaions, a general ligh field camera model capable of represening rays and scene geomery more concisely is in urgen need. MULTI-PROJECTION-CENTER MODEL In his secion, we firs propose he MPC model based on he TPP represenaion of ligh field. Then we deduce he ransformaion marix o relae D scene geomery and D rays. Finally, we uilize he MPC model o describe he image formaion of ligh field cameras and define generic inrinsic parameers, including convenional and focused ligh field cameras. Table gives he noaion of symbols used in he following secions. Term TABLE Noaion of symbols in he paper Definiion L(i, j, u, v) Indexed pixel of raw image inside he camera L(I, J, U, V ) Virual (conjugae) ligh field ouside he camera L(s,, x, y) Decoded physical ligh field (k i, k j, k u, k v, u, v ) Inrinsic parameers w D poin in he world coordinaes d D poin reconsruced by L(i, j, u, v) c D poin reconsruced by L(s,, x, y) R = [r r r ] Roaion marix of exrinsic parameer = ( x, y, z) Translaion vecor of exrinsic parameer M n Measuremen marix of n rays P Homogenous projecion marix A Non-homogenous projecion marix pariioned from P H Homography marix decided by inrinsic and exrinsic parameers only d=(k, k, k, k ) Disorion vecor. The Coordinaes of MPC Model As shown in Fig., here are hree coordinaes in he MPC model, i.e., D world coordinaes O w w Y w w, D camera coordinaes OY, D TPP coordinaes o s s o xy xy (o s s for he view plane and o xy xy for he image plane). In general, he ransformaion beween world and camera coordinaes is relaed by exrinsic parameers [R ]. The spacing beween wo parallel planes of radiional TPP represenaion is normalized as o describe a se of rays [], []. Alhough i is complee and concise, o derive he ransformaion beween D srucure and D rays in ligh field cameras, we prefer a model consising of wo parallel planes wih he spacing f.

3 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., Y o os Two-parallel-plane coordinaes f s y oxy camera coordinaes x r R, w o w world coordinaes Fig.. An illusraion of hree coordinaes in he MPC model. Le L f (s,, x, y) denoe ligh field in he MPC model wih he spacing f. Then he ray is parameerized by wo planes, i.e., = and = f. Le = denoe he view plane o s s and = f denoe he image plane o xy xy. In he MPC model, r = (s,, x, y) defines a ray passing (s,, ) and (x, y, f), where (s,, ) is he projecion cener and (x, y, f) is he corresponding projecion. Given a projecion cener (s,, ) (i.e., he (s, )-h view or sub-aperure) and he D poin =(, Y,, ), we can ge he image projecion x=(x, y, ) in he local coordinae of he (s, )-h view, λ x y = f fs f f Y Y w. () Since here are muliple projecion ceners (s,, ), s, =,,..., N, he D poin can be observed for N N imes. Obviously, when he spacing f changes o and here is only one projecion cener (,, ) on he view plane, he image formaion degeneraes ino radiional cenralprojecive camera model [].. Transformaion beween Geomery and Rays I is known ha differen direcional rays from one poin enable D reconsrucion. Le he ray r inersec a he poin in he D space, we can ge he relaionship beween he ray and D poin by he riangulaion, f x fs f y f {z } M Y =, () where M is a n marix consising of n rays and he MPC parameer f. If wo rays r i = (s i, i, x i, y i ) and r j = (s j, j, x j, y j ) are from one D poin, hey can be represened by he following wo equivalen forms, and Y = Y = y i y j s j x i s i x j i (x i x j ) y i (s i s j ) () x i x j f(s j s i ) s i(y i y j ) x i ( i j ) j y i i y j. () f( j i ). D Projecive Transformaion In fac, a linear ransformaion on he coordinaes of r causes D projecive disorion on he reconsruced poin [], deduced from Eqs.() and (). As shown in Fig., we show hree examples of linear ransformaions, including he changing of f, scaling in he image plane k xy (k xy =k x =k y ) (in general here are scaling facors k s, k, k x, k y, wo in he view plane and wo in he image plane respecively), and ranslaion in he image plane of specific view (,, x, y ) (generally (s,, x, y ) in boh planes). The deails are derived as follows. () If we change f ino f, he imaging poin (x, y, f) passed by r becomes (x, y, f ) and he inersecion of rays becomes. Subsiuing i ino Eqs.() and (), we have = P (f ) = f /f Y, () where and are in he homogeneous coordinaes. () Le r become r = (s+s, +, x+x, y+y ), hus here is a ransformaion on he rays caused by he offse m = (s,, x, y ). Subsiuing i ino Eqs.() and (), we can ge he ransformaion marix beween and, = P (m) = x /f s y /f Y. () () Le r become r = (k s s, k, k x x, k y y), hus here is a ransformaion caused by he scaling vecor k = (k s, k, k x, k y ). Then he ransformaion marix beween and is, and =P (k)= =P (k)= k s k k s /k x k s k k /k y Y, () Y. () In paricular, Eqs.() and () hold when k s /k =k x /k y. As shown in he lef-mos of Fig., here is a scene wih a Lamberian cube recorded by a MPC model. The observaion of he cube in muliple direcions is D ligh field. If he coordinaes are linearly ransformed and he ligh inensiy keeps consan, he inersecions of rays will be ransformed by a D projecion marix. Therefore, he cube will be projeced by ransformaion parameers (he righ hree of Fig. ).. The MPC Model in Ligh Field Cameras Ligh field cameras are improved from radiional cameras. They record real world scene in differen bu similar ways. In radiional cameras, he cenral projecion process of a D

4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., z z (x,y) f f ' f f s o s o Fig.. Examples of ransformaions beween D srucure and D ligh field of a Lamberian cube. The lefmos is an original cube and ohers are he projeced ones wih he changing of parameer f, scaling in he image plane k xy (=k x=k y) and ranslaion (,, x, y ) respecively. image is a dimension reducion of D space []. In ligh field camera, D srucure projeced by he main lens is arranged by he design of ligh pah on he image sensor. The processes of muliple cener projecions are analyzed as follows. On he one hand, as for a convenional ligh field camera, he sampling paern of ligh field is shown in Fig.. The pixel (u, v) of N N sub-aperure images is exraced from he micro-lens of (u, v). The sub-aperure image of he (i, j) view is exraced from he pixels (i, j) in he local micro-lens image coordinaes, as shown in Fig.. Obviously, here are wo ligh fields, i.e., L f (i, j, u, v) inside he camera and L F (I, J, U, V ) in he ouer world. Considering he projecion of main lens, here is a D projecive disorion beween he D poins reconsruced from L f and L F. On he oher hand, as for he focused ligh field cameras, wo sampling paerns of ligh field in wo differen opical pahs are shown in Fig.. The micro-lenses projec he disored D scene inside he camera on he image sensor, where he image range is conrolled by he aperure of main lens and he disance of componens. The ligh field inside he camera can be decoded by he pixels of image sensor (u, v, f) and heir corresponding opical ceners of microlens (i, j, ), i.e., L f (i, j, u, v). In addiion, (i, j) is deermined by he layou of MLA, as shown in Fig. b. By he ransformaion on he coordinae of L f (i, j, u, v) we have discussed in Sec., he ouside ligh field L F (I, J, U, V ) is obained, which is he conjugae MPC coordinae ouside Image Sensor v u i j z j i I s o Main Lens J z o s U Virual MLA Fig.. Opical pah of a convenional ligh field camera []. There are wo MPC coordinaes inside he camera and in he ouer world wih linear ransformaion, i.e., L f (i, j, u, v) and L F (I, J, U, V ) respecively. V (a) Telescopic case (b) Binocular case Fig.. Opical pahs of focused ligh field cameras wih differen designs []. There is a conjugae MPC coordinae L F (I, J, U, V ) in he ouer world wih he inner one L f (i, j, u, v). he camera. The real world scene can be reconsruced by he ligh field L F (I, J, U, V ) wihou projecive disorion. Le L(i, j, u, v) denoe indexed pixels of ligh field cameras wih f =. Moreover, L(i, j, u, v) is a se of indexed pixels and no a physical ligh field. In convenional ligh field camera, L(i, j, u, v) are he sub-aperure images indexed by he (i, j) view. In he focused ligh field cameras, L(i, j, u, v) are micro-lens images indexed by heir relaive posiions on he raw image. Obviously, by a linear ransformaion on he L(i, j, u, v), we can conduc L F (I, J, U, V ) and eliminae D projecive disorion caused by he main lens. However, o parameerize D ligh field wihou redundancy, he spacing of wo parallel planes should be. Le L(s,, x, y) denoe he normalized ligh field. According o Eqs.() o (), he normalizaion is a linear operaion on he coordinaes, and ransformaion marices P, P and P are all ideniy marices. I means ha indexed pixels L(i, j, u, v) can be ransformed o physical rays L(s,, x, y) in real world scene by linear ransformaions as we discussed before. The indexed pixels L(i, j, u, v) and decoded physical ligh field L(s,, x, y) of ligh field cameras in wo differen designs are shown in Fig., where pixels and physical rays are relaed by inrinsic parameers. In summary, we can ransform an indexed pixel of raw image L(i, j, u, v) ino a normalized physical ligh field

5 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., (a) Convenional ligh field camera (b) The focused ligh field camera Fig.. The indexed pixels L(i, j, u, v) and decoded physical ligh field L(s,, x, y) of ligh field cameras in wo designs. L(s,, x, y) by a decoding marix D ha is consising of inrinsic parameers (k i, k j, k u, k v, u, v ). s x y = k i k j k u u k v v {z } =: D i j u v. () Le d =[ d, Y d, d, ] and c =[ c, Y c, c, ] denoe wo D poins reconsruced by L(i, j, u, v) and L(s,, x, y) respecively. According o Eq.(), he relaionship beween d and c is /k i u /k i /k j v /k j k u /k i {z } =:P c Y c c = d Y d d, () where P = P ()P (k)p (m) is deermined by inrinsic parameers in he decoding marix D. Here, m = (,, u, v ) and k = (/k i, /k j, /k u, /k v ), which are oally decided by he mapping from indexed pixels o real world ligh rays. In addiion, he ligh field inside a convenional ligh field camera (in Fig. ) can also be parameerized by he MPC model ha is consising of image sensor and he MLA. However, considering he convenience of exracing subaperure images and he difficuly on deecing poins on raw image in a convenional ligh field camera, we prefer o discuss he daa as a se of sub-aperure images. Conversely, for he focused one, we model he parameerizaion plane by he raw image plane and discuss he raw image direcly. LIGHT FIELD CAMERA CALIBRATION We verify our ligh field camera model by inrinsic parameer calibraion. We will provide he deails of how o solve inrinsic parameers, including a linear closed-form soluion and a nonlinear opimizaion o minimize he re-projecion error. In our mehod, he prior scene poins are suppored by a planar calibraion board in differen poses.. Linear Iniializaion Afer necessary preprocessing, he micro-lens images are recognized [], [], [], i.e., L(i, j, u, v). We assume ha he prior D poin w in he world coordinaes is relaed o he D poin c in he MPC coordinaes by a rigid moion, c =R w +, wih he roaion R SO() and ranslaion =( x, y, z ) R. Le r i denoe i-h column vecor of R. The relaionship among R,, w and inrinsic parameers P =(k i, k j, k u, k v, u, v ) is obained by Eqs.() and (). MP r r r w =. () where M is a n measuremen marix of n rays and n. These rays are derived from he indexed pixels L(i, j, u, v) as menioned in Eq.(). Suppose ha he calibraion board is on he plane of = in he world coordinaes, hus w =. To solve he unknown parameers, we simplify Eq.() as, M [ w Y w ] H =, () where H is a marix sreched on row from H. is a direc produc operaor. H is a marix only consising of inrinsic and exrinsic parameers, defined as H = P r r. () In addiion, M is a marix conaining a leas rays from ligh field L(i, j, u, v), according o Eq.(). By sacking measuremens from a leas non-collinear poins w, he homography H can be esimaed by Eq.(). In order o derive inrinsic parameers from H, we can pariion P o exrac a upper riangle marix A. Le h ij denoe he elemen on he i-h row and j-h column of H, we rewrie Eq.() as follows, h G h h = A r r, () where G is a marix, i.e., op-lef of H. Le g j = (g j, g j, g j ), j =, denoe he j-h column vecor of G. Uilizing he orhogonaliy and ideniy of R, we have g A A g =, g A A g = g A A g. where A = k i u k i /k u k j v k j /k v. k i /k u ()

6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., Le a symmeric marix B denoe A A. The analyical form of B is B= ki ki u /k u kj kj v /k v ki u /k u kj v /k v ki /k u +u +v. () Noe ha here are only disinc non-zero elemens in B, denoed by b := (b, b, b, b, b ). To solve B, we rewrie Eq.() as follows, g g g g +g g g g (g g g g ) g g g g g g +g g (g g g g ) g g g g b=. () By sacking a leas wo such equaions (from wo poses) as Eq.(), we can obain a unique general non-zeros soluion for b, which is defined up o an unknown scale facor. Once B = A A is deermined, i is an easy maer o solve A using Cholesky facorizaion []. Le  denoe he esimaion of A, i.e., λâ = A. Le â ij denoe he elemen on he i-h row and j-h column of Â, inrinsic parameers excep k i and k j are esimaed by he raio of elemens k u = â /â, k v = â /â, u = â /â, v = â /â. () Apar from inrinsic parameers, exrinsic parameers in differen poses can be exraced as follows, λ = α  g +  g /, r =  g /λ, r =  g /λ, =  [h h h ] /λ, r = r r, () where denoes L norm. α values or - and i is decided by image formaion. In convenional ligh field camera and he focused one wih shorer ligh pah (as shown in Fig. and b), α makes z >. Oherwise, in he focused ligh field camera wih longer ligh pah (see Fig. a), α makes z <. To obain oher wo inrinsic parameers k i and k j, we subsiue he resuls in Eq.() for Eq.() and obain c = R w + using he esimaed exrinsic parameers. Then, Eq.() is rewrien as, i ki c x = c. () j k j Y c y c Sacking he measuremens in differen poses, we can obain a unique non-zeros soluion for k i and k j.. Nonlinear opimizaion The mos common disorion of radiional camera is radial disorion. The opical propery of main lens and physical machining error of he MLA migh lead o he disorion of rays in ligh field camera. Theoreically, due o wo level imaging design wih main lens and micro-lens array, here should exis radial disorion on he image plane (x, y) and sampling disorion on he view plane (s, ) simulaneously. In he paper, we only consider he disorion on he image plane and omi sampling disorion on he view plane (i.e., angular sampling grid is ideal wihou disorion). xu = ( + k rxy + k rxy)x + k s y u = ( + k rxy + k rxy)y, () + k where rxy = x + y and r = (s,, x, y) is he ray ransformed from he measuremen L(i, j, u, v) by inrinsic parameer P according o Eq.(). d = (k, k, k, k ) denoes disorion vecor and x u = (x u, y u ) is undisored projecion from he disored one x = (x, y) in he local image coordinaes under he (s, ) view. In he disorion vecor d, k and k regulae radial disorion on he image plane. k and k represen he disorion of image plane affeced by he sampling view (s, ), which is caused by non-paraxial rays of he main lens. We minimize he following cos funcion wih he iniializaion solved in Secion. o refine he parameers, including inrinsic parameer P, disorion vecor d, and exrinsic parameers R p and p, p =,..., P, P is he number of poses. #pose p= #poin n= #view i= x u i (P, d) ˆx i (R p, p, w,n ), () where x u is he image poin from L(i, j, u, v) according o Eq.() and followed by disorion recificaion according o Eq.(). ˆx is he projecion of D poin w,n in he world coordinaes according o Eq.(). In Eq.(), R is parameerized by Rodrigues formula []. In addiion, he Jacobian marix of cos funcion is simple and sparse. This nonlinear minimizaion problem can be solved wih he Levenberg-Marquard algorihm based on rus region mehod []. We adop MATLAB s lsqnonlin funcion o complee he opimizaion.. Compuaional Complexiy The calibraion algorihm of ligh field camera is summarized in Alg.. Le (S, T ) denoe sampling number on he view plane, (M, N) be he number of prior poins on he calibraion board, and P be he number of poses, respecively. For he measuremen of each pose, here are ( S T ) linear equaions o solve H. Then ( P ) linear equaions and ( P M N) equaions are solved o obain inrinsic parameers. The main complexiy is spen on he soluion of H from differen poses, i.e., O(P ). By conras, he algorihm in Dansereau e al. [] calculaes a homography for every sub-aperure image in differen view, i.e., O(P S T ). I suffers from a higher complexiy and a lower accuracy on parameer iniializaion. The algorihm in Bok e al. [] solves a linear equaion on every pose and is compuaional complexiy is O(P ). However, here are inrinsic and exrinsic parameers in he equaions, which causes inaccuracy on he soluion.

7 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., Algorihm Ligh Field Camera Calibraion Algorihm. Inpu: D prior poins w and corresponding rays L(i, j, u, v). Oupu: Inrinsic parameers P = (k i, k j, k u, k v, u, v ); Exrinsic parameers R p, p( p P ); Disorion vecor d = (k, k, k, k ). : for p = o P do : for each D poin w do : Generae he measuremen marix M from indexed pixel L(i, j, u, v) Eq.() : end for : Calculae he homography marix H p according o w and M Eq.() : end for : Calculae he marix ˆB Eq.() : Calculae projecion marix  from ˆB using Cholesky facorizaion : Obain four inrinsic parameers (k u, k v, u, v ) Eq.() : for p = o P do : Ge exrinsic parameers R p and p Eq.() : end for : Obain oher wo inrinsic parameers (k i, k j) Eq.() : Iniialize disorion coefficien d = (,,, ) : Creae he cos funcion according o inrinsic parameers, exrinsic parameers and disorion coefficien Eq.() : Obain opimized resuls using nonlinear LM algorihm TABLE Inrinsic parameer configuraion of he simulaed ligh field camera. k i k j k u k v u v.e-.e-.e-.e TABLE Min and Max relaive errors of inrinsic parameers (uni: %) on he simulaed daa when he number of poses is grea han. k i k j k u k v u v Min Max (a) k i (b) k j EPERIMENTAL RESULTS In his secion, we verify our ligh field camera model by he calibraion of inrinsic parameers. We presen various experimenal resuls boh on simulaed and real daases. The performance is analyzed by comparing wih he ground ruh or baseline algorihms [] and [].. Simulaed daa In his subsecion we verify our calibraion mehod on simulaed daa. The simulaed ligh field camera has he following propery referred o Eq.(), as shown in Table. These parameers are close o he seing of Lyro camera so ha we obain plausible inpu close o real-world scenarios. The checkerboard is a paern wih poins wih.mm.mm cells. (c) k u (d) k v.. Performance w.r. he number of poses and views Firsly, we es he performance wih respec o he number of poses and he number of views. We vary he number of poses from o and he number of views from o. For each combinaion of pose and view, rails of independen calibraion board poses are generaed. The roaion angles are randomly generaed from o, and he measuremens are all added wih Gaussian noise wih zero mean and sandard deviaion. pixels. The calibraion resuls wih increasing measuremens are shown in Fig.. We find ha he relaive errors decrease wih he increase in he number of poses. When he number of pose is greaer han, all he relaive errors are wihin an accepable level, as summarized in Table. Meanwhile, he errors reduce as he number of views grows once he number of poses is fixed. In paricular, when #pose and #view, all he relaive errors are less han.%. Furhermore, he sandard deviaions of relaive errors of Fig. are shown in Fig., from which we can see ha sandard deviaions decrease significanly when he number (e) u (f) v Fig.. Relaive errors of inrinsic parameers on he simulaed daa wih differen number of poses and views. of pose is greaer han. Paricularly, when #pose and #view, sandard deviaions keep a a low level sably. The resuls in Fig. and Fig. have verified he effeciveness of he proposed calibraion algorihm... Performance w.r. he measuremen noise Secondly, we employ he measuremens of poses and views o verify he robusness of calibraion algorihm. The roaion angles of poses are (,, ), (,, ) and (,, ) respecively. Gaussian noise wih zero mean and a sandard deviaion σ is added o he projeced image poins. We vary σ from. o. pixels wih a. sep. For each noise level, we performed

8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., k i (a) k i (b) k j k j k u (a) k i and k j (b) k u and k v k v u v -u /k u -v /k v (c) k u (d) k v (c) u and v (d) u /k u and v /k v (e) u (f) v Fig.. Sandard deviaions of relaive errors of inrinsic parameers on he simulaed daa wih differen number of poses and views. independen rials. The mean resuls compared wih ground ruh are shown in Fig.. I demonsraes ha he errors increase almos linearly wih he noise level. For σ =. pixels which is larger han normal noise in pracical calibraion, he errors of (k i, k j ) and (k u, k v ) are less han.%. Alhough he relaive error of (u, v ) is.%, he absolue error of ( u /k u, v /k v ) is less han. pixel (In Eq.(), u = (x u )/k u and v = (y v )/k v, where ( u /k u, v /k v ) is he principal poin of subaperure imaging), which furher exhibis ha he proposed algorihm is robus o higher noise level.. Physical camera We also verify he calibraion mehod on real scene ligh fields capured by convenional and focused ligh filed cameras. For he convenional ligh field camera, we use Lyro and Illum o obain measuremens. For he focused one, we use a self-assembly camera according o opical design in Fig. a... Convenional Ligh Field Camera The sub-aperure images are obained by he mehod of Dansereau e al. []. We compare he proposed mehod in ray re-projecion error wih sae-of-he-ars, including DPW by Dansereau e al. [] and BJW by Bok e al. []. Fig.. Relaive errors of inrinsic parameers on he simulaed daa wih differen noise levels from. o. pixels. Firsly, we carry ou calibraion on he daases colleced wih []. For every differen pose, he middle subaperures are uilized similar o DPW. Table summarizes he roo mean square (RMS) ray re-projecion errors of our mehod and DPW []. In Table, he errors of DPW []- are aking from he paper direcly. The errors of DPW []- are obained by running heir laes released code. On he iem of iniial, he proposed mehod provides a smaller ray re-projecion error han DPW excep on daases A and B. The resul on daase A performs worse because of bad corner exracion from several poses (i.e., h, h, h and h ligh field). On he iem of opimized, compared wih DPW [] which employs inrinsic parameers, he proposed MPC model only employs a half of parameers bu achieves similar performance on ray re-projecion errors (he resuls on daases A, B and D are beer bu he resuls on daases C and E are worse). Ligh fields wihin each daase are aken over a range of dephs and orienaions, as shown in Fig.. The ranges of daases A, B are.m whils he ranges of daases C and D do no exceed.m. Meanwhile, he ranges do no exceed m in daase E. Large ranges are reasonable in all daases only deducing he accuracy in ligh of disorion model considering he shifed view. This is he main reason why he performance of he proposed mehod is worse han ha of DPW on daase E. From he daase A, we selec ligh fields from which he corners are exacly exraced for he proposed mehod. The ray re-projecion error decreases obviously in Table. Considering he fac ha he errors exhibied in DPW are minimized in is own opimizaion (i.e., ray re-projecion error), we addiionally evaluae he performance in mean re-projecion error of DPW and BJW. As exhibied in Table, he errors of he proposed mehod are obviously smaller han hose of DPW and BJW. In addiion, he calibraion wih fewer number of poses on he daases [] is conduced. For daase D, we randomly selec ligh fields, and for daases B, C

9 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., TABLE RMS ray re-projecion errors of iniial parameer esimaion and opimizaion wih disorion recificaion (uni: mm). The daases are from []. The (N) indicaes he number of ligh fields used for calibraion among ligh fields of daase A. The errors of DPW[]- are provided by he paper direcly, and he errors of DPW[]- are obained by running he laes released code. A A() B C D E DPW[] Iniial DPW[] Ours DPW[] Opimized DPW[] Ours TABLE Mean re-projecion errors of opimizaion wih disorion recificaion (uni: pixel). The resuls of DPW[] are obained by running heir laes released code. The resuls of BJW[] are from heir laes paper. A A() B C D E DPW[] BJW[] Ours TABLE RMS errors of opimizaion wih disorion recificaion using fewer poses. The (N) indicaes he number of ligh fields used for calibraion. TABLE Inrinsic parameer esimaion resuls of daases capured by []. A B C D E k i.e-.e-.e-.e-.e- k j.e-.e-.e-.e-.e- k u.e-.e-.e-.e-.e- k v.e-.e-.e-.e-.e- u v k..... k k k TABLE RMS re-projecion error of sub-aperures in daase B (uin: pixel). i j Ray re-projecion error uni: mm Re-projecion error uni: pixel DPW[] Ours DPW[] Ours B().... C().... D().... E() (a) A (b) A() and E, ligh fields are randomly seleced for calibraion. Table summarizes RMS ray re-projecion errors and RMS re-projecion errors of he proposed mehod and DPW [] respecively. In Table, he proposed mehod achieves smaller errors han DPW. Besides, he calibraion resuls on daases D and E are obviously improved by reducing he number of poses. We find ha smaller range of poses conribues o a performance improvemen on daases D and E, which is shown in Fig.. Table liss inrinsic parameer esimaion resuls. The re-projecion errors of sub-aperure images of B are summarized in Table. The disribuion of errors is almos homogeneous. All resuls have verified he effeciveness of he proposed mehod. Unlike he core idea of DPW, BJW direcly uilizes raw daa insead of sub-aperures. However i has a sricer requiremen on he acquisiion of he calibraion board. The daa for calibraion mus be unfocused in order o make he measuremens deecable, hus some daases provided by DPW are incalculable for BJW, jus as shown in Table (i.e. daases C and D). In order o direcly compare wih DPW and BJW, we collec oher daases using Lyro and Illum cameras. The daase Illum- shoos corners wih.mm.mm cells, including poses. The daase Lyro- shoos corners wih.mm.mm cells,. hp:// (c) B. (e) D (d) C (f) E Fig.. Pose esimaion resuls of daases capured by []. Ligh fields used for calibraion in Table are indicaed wih bold red indexes of corresponding camera poses in Figs. (c-f). including poses. The daases Illum- and Lyro- shoo corners wih.mm.mm cells, including

10 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., TABLE RMS ray re-projecion errors of iniial parameer esimaion, opimizaions wihou and wih disorion recificaion (uni: mm). Illum- Illum- Lyro- Lyro- DPW[].... Iniial BJW[].... Ours.... Opimized DPW[].... wihou BJW[] Recificaion Ours.... Opimized DPW[].... wih BJW[].... Recificaion Ours.... TABLE Inrinsic parameer esimaion resuls of our colleced daases. Illum- Illum- Lyro- Lyro- k i.e-.e-.e-.e- k j.e-.e-.e-.e- k u.e-.e-.e-.e- k v.e-.e-.e-.e- u v k k.... k k (a) Illum (c) Lyro (b) Illum (d) Lyro- Fig.. Pose esimaion resuls of our colleced daases TABLE RMS re-projecion error of sub-aperures in daase Illum- (uin: pixel). i j poses. For Illum- and Illum- daases, he middle views are used ( views in oal). For Lyro- and Lyro-, he middle views are used ( views in oal). Table summarizes he RMS ray re-projecion errors compared wih DPW and BJW a hree calibraion sages. As exhibied in Table, he proposed mehod obains smaller ray re-projecion errors on he iem of iniial soluion which verified he effeciveness of linear iniial soluion for boh inrinsic and exrinsic parameers. Besides, he proposed mehod provides similar or even smaller ray re-projecion errors on he iem of opimizaion wihou recificaion compared wih DPW. I is noiced ha he resul on daase Lyro- is relaively larger han ha of DPW. The main reason is ha disorion coefficiens k and k in our model are similar o he elemens of he decoding marix in []. Considering he fac ha MPC model employs less parameers (i.e. -parameer) han DPW (i.e. -parameer), he proposed mehod is compeiive wih accepable calibraion performance. Furher, i is more imporan ha we achieve smaller ray re-projecion errors if disorion recificaion is inroduced in opimizaion. The ray re-projecion errors are encouraging ha he proposed mehod ouperforms DPW (a) Illum- (b) Illum- Fig.. The siching resuls of Illum- and Illum- daases (he firs pose is regarded as he reference view). and BJW. Consequenly, he -parameer MPC model and -parameer disorion model are effecive o represen ligh field cameras. The reason why we compare ray re-projecion errors here is o eliminae differences in camera models. The decoding marix in [] is similar o Eq.(), excep for he non-diagonal elemens. The non-zero elemens H, and H, indicae ha pixels on he same sub-aperure image have specific relaionships among differen views. If we calculae he esimaed rays by c for he measuremen (s,, x, y), he views may be differen. I indicaes ha here are errors boh on he view plane and image plane in []. As a resul, i is no reasonable o compare re-projecion error only. Moreover, he resuls of inrinsic parameer esimaion and pose esimaion on our daases are demonsraed in Table and Fig. respecively. Afer he calibraion process, we measure he RMS re-projecion errors of sub-aperure images by uilizing esimaed parameers, as shown in Table. In order o furher verify he accuracy of inrinsic and exrinsic parameer esimaion, we sich all oher ligh fields on he firs pose, as shown in Fig., from which we can see all view ligh fields are regisered and siched very well. Evenually, i is worhy noing ha here exiss

11 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., (a) Original cenral view image (b) Proposed (.) (c) DPW [] (.) (d) BJW [] (.) Fig.. The cenral view sub-aperure and disorion recificaion resuls of firs pose ligh field in Illum- daase. The re-projecion error (uni: pixel) of cenral view sub-aperure image is represened in parenheses. TABLE Quaniaive comparison of differen calibraion mehods (uni: mm). The relaive error is indicaed in parenheses. P A M I Ruler.... DPW[]. (.%). (.%). (.%). (.%) BJW[]. (.%). (.%). (.%). (.%) Ours. (.%). (.%). (.%). (.%) disinc disorion in he Illum camera. In Fig., we show original cenral view sub-aperure image and recificaion resuls using disorion models of he proposed mehod, DPW [] and BJW [] respecively. Since he re-projecion error indicaes he image disance beween a projeced poin and a recified one, i can be used o quanify he error of disorion recificaion resuls. In Fig., we lis he RMS re-projecion error of cenral view sub-aperure image using differen mehods in parenheses, which furher verifies ha he recificaion resuls of he proposed mehod are beer han hose of baseline algorihms. High-precision calibraion is essenial in early sages of ligh field processing pipeline. In order o verify he accuracy of geomeric reconsrucion of he proposed mehod compared wih baseline mehods, we capure wo real scene ligh fields, hen reconsruc several ypical corner poins and esimae he disances beween hem. Fig. shows reconsrucion resuls on he cenral view sub-aperure images. As exhibied in Fig., he esimaed disances beween poins reconsruced by he proposed mehod are nearly equal o hose measured lenghs from real objecs by rulers. In addiion, Table liss he comparisons of reconsrucion resuls wih sae-of-he-ar mehods. The relaive errors of reconsrucion resuls demonsrae he performance of our mehod. As menioned above, since BJW has a sricer requiremen on he image of calibraion board, some daases (i.e. C and D) [] are incalculable for BJW. In order o direc- Fig.. The evaluaions of ligh field measuremens. (a) shows disances beween D poins measured by rulers. (b), (c) and (d) demonsrae he esimaed disances beween he reconsruced poins afer ligh filed camera calibraion wih differen mehods. TABLE The running ime of iniial parameer esimaion (uni: s). Illum- Illum- Lyro- Lyro- DPW[].... BJW[].... Ours.... ly compare wih DPW and BJW, we uilize our colleced daases o analyze he running ime of iniial parameer esimaion, as illusraed in Table. All algorihms are execued in MATLAB on a deskop compuer (.GHz CPU and G RAM). The proposed mehod is mos efficien comparing o oher mehods. Since he compuing of SVD in iniial linear soluion esimaion of BJW is ime-consuming (i.e. he running ime of SVD on four daases is.,.,. and.s respecively), he running ime of BJW is shorer han ha of DPW if he execuion ime of SVD is excluded. The resuls of running ime conform well o our complexiy analysis in Sec...

12 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., (a) poses (b) poses (c) poses Fig.. Pose esimaion resuls of our self-assembly focused ligh field camera wih differen number of poses. TABLE Calibraion resuls of inrinsic parameers of our physical camera wih differen number of poses. #Pose k i k j k u k v u v Ini..e-.e-.e-.e Op w/o.e-.e-.e-.e Op w.e-.e-.e-.e Fig.. The self-assembly focused ligh field camera and he MLA inside he camera. The ligh field design paern is shown in Fig. a... Focused Ligh Field Camera We capure daa using a self-assembly focused ligh field camera. The camera and he MLA are shown in Fig.. The camera consiss of a GigE camera wih a CCD image sensor whose resoluion is wih µm pixel widh, a Nikon AF Nikkor f/.d F-moun lens wih mm focal lengh, and a MLA wih µm diameer and.mm focal lengh in hexagon layou. The ligh field design paern is shown in Fig.. The calibraion board is poins wih mm mm cells. We shoo raw images wih differen poses of calibraion board. Besides, we also collec raw images wih real scene and calibraion board. All he images are siched ogeher o generae an enhanced ligh field wih wider FOV. We carry ou calibraion using differen numbers of ligh fields from poses (i.e., poses, poses and poses). The esimaed inrinsic parameers of our physical camera are lised in Table. The op row shows he esimaed resuls by direc linear iniializaion. The middle and boom rows show he opimized resuls wihou and wih disorion recificaion. As shown in Table, he RMS re-projecion errors are less han. pixels afer he opimizaion wih disorion recificaion. Fig. shows he esimaed poses of our physical camera. Afer he calibraion, we render he images by ray racing using raw images wih real scene and calibraion board in differen poses. The refocus rendering pipeline of a focused ligh field camera is shown in Fig., which includes four key seps. The esimaed poses and rendered resuls are shown in Fig.. On he siched image in Fig. a, he doarray calibraion board on he background is focused and he blocks on he foreground are unfocused. The smooh boundary from differen poses indicaes ha our calibraion algorihm can esimae exac parameers. Ini..e-.e-.e-.e Op w/o.e-.e-.e-.e Op w.e-.e-.e-.e Ini..e-.e-.e-.e Op w/o.e-.e-.e-.e Op w.e-.e-.e-.e TABLE The disorion vecors and RMS re-projecion errors (uni: pixel) of our physical camera wih differen number of poses. #Pose k k k k Error.e-.e- -.e- -.e-. -.e- -.e- -.e-.e-. -.e- -.e- -.e-.e-. CONCLUSION In he paper, we presen a muli-projecion-cener model o parameerize ligh field and describe ligh field imaging formaion. We deduce he ransformaions o describe he relaionship beween D rays and D scene srucure by a projecive ransformaion. Then we verify our ligh field camera model by inrinsic parameer calibraion. We firs derive a closed-form soluion for iniial esimaion of inrinsic and exrinsic parameers and hen propose a parameer refinemen by minimizing he re-projecion error. Experimens on convenional ligh field camera Lyro and Illum and a self-assembly focused ligh field camera are performed and analyzed exensively. The comparisons wih ground ruh and sae-of-he-ar calibraion mehods have verified he robusness and validiy of he proposed model and our calibraion mehod, especially he iniializaion and opimizaion wih disorion recificaion. In fuure, we end o focus on sampling disorion model on he view plane, ligh field regisraion and enhancemen from un-calibraed cameras for arbirary scenes, and reparameerizaion of D ligh field from arbirary poses.

13 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., Focus Plane Raw Image Virual MLA Ă y ݕݔ ݐݏ x s z Micro Image (a) Recify raw image. (c) Transform he micro images hrough he projecion ceners o he focus plane (adjacen ransformed images overlap each oher). (b) Calculae projecion ceners (opical ceners of virual MLA) for every micro image by. (d) Sum he projeced images. Fig.. The refocus rendering pipeline of a focused ligh field camera. x Y x x.... x Y (a)... x (b) Fig.. Experimenal resuls of ligh field siching. (a) shows he rendered images. The op row shows lefmos and righmos ligh fields capured by our focused ligh field camera. The boom row is refocused resul of he siched ligh field from ligh fields. (b) shows he esimaed poses of ligh fields. ACKNOWLEDGMENTS [] The auhors would like o hank he Ji, Dr. Guoqing hou and Dr. haolin iao for heir helpful suggesions. We also hank anonymous reviewers for heir valuable feedback. [] [] R EFERENCES [] [] [] [] [] [] R. Ng, Digial ligh field phoography, Ph.D. disseraion, Sanford Universiy,. A. Lumsdaine and T. Georgiev, The focused plenopic camera, in Proc. IEEE In. Conf. Compu. Phoography,, pp.. M. Levoy and P. Hanrahan, Ligh field rendering, in Proc. ACM rd Annu. Conf. Compu. Graph. Ineracive Techn.,, pp.. S. J. Gorler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, The lumigraph, in Proc. ACM rd Annu. Conf. Compu. Graph. Ineracive Techn.,, pp.. R. Ng, Fourier slice phoography, ACM Trans. Graph., vol., no., pp.,. H.-G. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y.-W. Tai, and I. S. Kweon, Accurae deph map esimaion from a lensle ligh field camera, in Proc. IEEE Conf. Compu. Vis. Paern Recogni.,, pp.. [] [] [] [] [] [] S. Wanner, C. Sraehle, and B. Goldluecke, Globally consisen muli-label assignmen on he ray space of d ligh fields, in Proc. IEEE Conf. Compu. Vis. Paern Recogni.,, pp.. O. Johannsen, A. Sulc, and B. Goldluecke, On linear srucure from moion for ligh field cameras, in Proc. IEEE In. Conf. Compu. Vis.,, pp.. C. Birklbauer and O. Bimber, Panorama ligh-field imaging, Compu. Graph. Forum, vol., no., pp.,.. Guo,. Yu, S. B. Kang, H. Lin, and J. Yu, Enhancing ligh fields hrough ray-space siching, IEEE Trans. Vis. Compu. Graphics, vol., no., pp.,. D. G. Dansereau, O. Pizarro, and S. B. Williams, Decoding, calibraion and recificaion for lensele-based plenopic cameras, in Proc. IEEE Conf. Compu. Vis. Paern Recogni.,, pp.. O. Johannsen, C. Heinze, B. Goldluecke, and C. Perwaß, On he calibraion of focused plenopic cameras, in Proc. Seminar TimeFligh Deph Imag. Sens., Algorihms, Appl.,, pp.. Y. Bok, H.-G. Jeon, and I. S. Kweon, Geomeric calibraion of micro-lens-based ligh-field cameras using line feaures, in Proc. h Eur. Conf. Compu. Vis.,, pp.. C. Hahne, A. Aggoun, S. Haxha, V. Velisavljevic, and J. C. J. Ferna ndez, Ligh field geomery of a sandard plenopic camera, Opics Express, vol., no., pp.,. C. Thomason, B. Thurow, and T. Fahringer, Calibraion of a

14 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO., microlens array for a plenopic camera, in Proc. nd Aerospace Sci. Meeing: Amer. Ins. Aeronauics Asronauics,, pp.. [] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Anunez, A. Barh, A. Adams, M. Horowiz, and M. Levoy, High performance imaging using large camera arrays, ACM Trans. Graph., vol., no., pp.,. [] V. Vaish, B. Wilburn, N. Joshi, and M. Levoy, Using plane+ parallax for calibraing dense camera arrays, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., vol.,, pp.. [] R. Pless, Using many cameras as one, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., vol.,, pp. II. [] R. Harley and A. isserman, Muliple view geomery in compuer vision. Cambridge Universiy Press,. [] T. E. Bishop and P. Favaro, The ligh field camera: exended deph of field, aliasing, and superresoluion, IEEE Trans. Paern Anal. Mach. Inell., vol., no., pp.,. [] D. Cho, M. Lee, S. Kim, and Y.-W. Tai, Modeling he calibraion pipeline of he lyro camera for high qualiy ligh-field image reconsrucion, in Proc. IEEE In. Conf. Compu. Vis.,, pp.. [] S. Wanner, J. Fehr, and B. Jähne, Generaing epi represenaions of d ligh fields wih a single lens focused plenopic camera, in Proc. Sevenh In l Conf. Advances in Visual Compuing,, pp.. [] Y. Bok, H.-G. Jeon, and I. S. Kweon, Geomeric calibraion of micro-lens-based ligh field cameras using line feaures, IEEE Trans. Paern Anal. Mach. Inell., vol., no., pp.,. [] C. Hahne, A. Aggoun, and V. Velisavljevic, The refocusing disance of a sandard plenopic phoograph, in Proc. Conf. DTV,, pp.. [] C. Heinze, S. Spyropoulos, S. Hussmann, and C. Perwaß, Auomaed robus meric calibraion of muli-focus plenopic cameras, in Proc. IEEE In. Insrum. Meas. Technol. Conf.,, pp.. [] C. hang,. Ji, and Q. Wang, Decoding and calibraion mehod on focused plenopic camera, Compuaional Visual Media, vol., no., pp.,. [] R. Harley, Self-calibraion of saionary cameras, In. J. Compu. Vis., vol., no., pp.,. [] O. Faugeras, Three-dimensional compuer vision: a geomeric viewpoin. MIT Press,. [] K. Madsen, H. B. Nielsen, and O. Tingleff, Mehods for non-linear leas squares problems, nd Ediion. Informaics and Mahemaical Modelling, Technical Universiy of Denmark, April. Jinbo Ling received he B.E. degree in Sofware Engineering from School of Informaion Engineering, Chang an Universiy, in. He is now a posgraduae a School of Compuer Science, Norhwesern Polyechnical Universiy. His research ineress include D reconsrucion, muliview ligh field compuing and processing. Qing Wang (M ) is now a Professor in he School of Compuer Science, Norhwesern Polyechnical Universiy. He received he B.S. degree from Peking Universiy in. He hen joined Norhwesern Polyechnical Universiy. In and he obained Maser and Ph.D. degrees from Norhwesern Polyechnical Universiy. In, he was awarded as ousanding alen program of new cenury by Minisry of Educaion, China. He is now a Member of IEEE and ACM. He is also a Senior Member of China Compuer Federaion (CCF). He worked as research assisan and research scienis in he Deparmen of Elecronic and Informaion Engineering, he Hong Kong Polyechnic Universiy from o. He also worked as a visiing scholar in he School of Informaion Engineering, The Universiy of Sydney, Ausralia, in and. In and, he visied Human Compuer Ineracion Insiue, Carnegie Mellon Universiy, for six monhs and Deparmen of Compuer Science, Universiy of Delaware, for one monh. Prof. Wang s research ineress include compuer vision and compuaional phoography, such as D reconsrucion, objec deecion, racking and recogniion in dynamic environmen, ligh field imaging and processing. He has published more han papers in he inernaional journals and conferences. Qi hang received he B.E. degree in Elecronic and Informaion Engineering from i an Universiy of Archiecure and Technology in, and Maser degree in Elecrical Engineering from Norhwesern Polyechnical Universiy in. He is now a Ph.D. candidae a School of Compuer Science, Norhwesern Polyechnical Universiy. His research ineress include compuaional phoography, ligh field imaging and processing, muliview geomery and applicaions. Chunping hang received he B.E. degree and Maser degree in Compuer Science from School of Compuer Science, Norhwesern Polyechnical Universiy in and respecively. Her research ineress include compuaional phoography, ligh field camera model and calibraion, muliview sereo reconsrucion. Jingyi Yu is Direcor of Virual Realiy and Visual Compuing Cener in he School of Informaion Science and Technology a ShanghaiTech Universiy. He received B.S. from Calech in and Ph.D. from MIT in. He is also a full professor a he Universiy of Delaware. His research ineress span a range of opics in compuer vision and compuer graphics, especially on compuaional phoography and nonconvenional opics and camera designs. He has published over papers a highly refereed conferences and journals including over papers a he premiere conferences and journals CVPR/ICCV/ECCV/TPAMI. He has been graned US paens. His research has been generously suppored by he Naional Science Foundaion (NSF), he Naional Insiue of Healh (NIH), he Army Research Office (ARO), and he Air Force Office of Scienific Research (AFOSR). He is a recipien of he NSF CAREER Award, he AFOSR YIP Award, and he Ousanding Junior Faculy Award a he Universiy of Delaware. He has previously served as general chair, program chair, and area chair of many inernaional conferences such as ICCV, ICCP and NIPS. He is currenly an Associae Edior of IEEE TPAMI, Elsevier CVIU, Springer TVCJ and Springer MVA.