An Anlysis of Color Demosicing in Plenoptic Cmers Zhn Yu Jingyi Yu University of Delwre Newrk, DE 19716, USA {zyu,yu}@cis.udel.edu Andrew Lumsdine Indin University Bloomington, IN 47405, USA lums@osl.iu.edu Todor Georgiev Qulcomm Sn Diego, CA 92121, USA todor@tgeorgiev.net Abstrct A plenoptic cmer cptures the 4D rdince bout scene. Recent prcticl solutions mount microlens rry on top of commodity SLR to directly cquire these rys. However, they suffer from low resolution s hundreds of thousnds of views need to be cptured in single shot. In this pper, we develop simple but effective technique for improving the imge resolution of the plenoptic cmer by mneuvering the demosicing process. We first show tht the trditionl solution by demosicing ech individul microlens imge nd then blending them for view synthesis is suboptiml. In prticulr, this demosicing process often suffers from lising rtifcts, nd it dmges high frequency informtion recorded by ech microlens imge hence degrdes the imge qulity. We insted propose to demosic the synthesized view t the rendering stge. Specificlly, we first trnsform the rdince to the desired focl plne nd then pply frequency domin plenoptic resmpling. A full resolution color filtered imge is then creted by performing 2D integrl projection from the reprmeterized rdince. Finlly, we conduct demoscing to obtin the color result. We show tht our solution cn chieve visible resolution enhncement on dynmic refocusing nd depth-ssisted deep focus rendering. 1. Introduction Recent dvnces in computtionl photogrphy hve given rise to previously unexplored effects in imging. A notble exmple is the reliztion of the plenoptic (or light field ) cmer [25, 21, 2, 8], cmer tht uses microlens rry to cpture 4D rdince bout scene. The cquired rdince informtion cn be post-processed for either synthesizing dynmic depth of field effects or for recovering the 3D scene. There re numerous pplictions for this emerging cmer technology, rnging from entertinment (Lytro [22]) to depth recovery for industril nd scientific pplictions (Rytrix [26]). The plenoptic cmer, in essence, is single-shot, multiview cquisition device. In order to overcome the sptiongulr trdeoff, n ultr-high resolution sensor is commonly used. The resulting imges, however, re still t disppointingly low resolution. The Adobe light field cmer [8] cptures 20 different views of scene using 10 megpixel sensor, resulting in rendered imges with visible rtifcts t resolution of 700 700. Ng[25] proposed different design with 296 296 microlens rry covering 16 megpixel sensor. The dense ngulr resolution gretly suppressed rtifcts with higher refocusing power. Nevertheless, the imge resolution is low, equl to the number of microlenses in the cmer (296 296). The recently relesed Lytro light field cmer uses 11 megpixel sensor for cpturing the rdince. Pictures published on Lytro s website still suffer from low resolution of 0.7 megpixel, with some visible rtifcts round thin objects nd shrp edges. In this pper, we develop simple but effective technique for improving the imge resolution of the plenoptic cmer by using more pproprite demosicing process. A plenoptic cmer, like trditionl color cmers, cptures color informtion with Color Filter Arry (CFA) msking the sensor pixels. We first show tht the trditionl solution [25, 21, 2, 8] tht demosics ech individul microlens imge nd then blends them for rendering is suboptiml. In prticulr, this demosicing process dmges high frequency informtion recorded by ech microlens imge, hence gretly degrding the chievble resolution of the finl photogrph. We insted perform demosicing on the synthesized color photogrph t ech refocusing plne Π. Specificlly, we first reprmeterize the light field to the desired focl plne nd then pply frequency-domin plenoptic resmpling. A full resolution color filtered imge is then creted by performing 2D integrl projection from the reprmeterized light field. Demosicing is performed s lst step to obtin the finl color result. Experiments on synthetic nd nturl scenes show tht our pproch genertes imges with higher resolution nd fewer rtifcts compred with clssicl plenoptic rendering. We demonstrte the qulity enhncements of our results in 1
Min lens Smpled imge plne Refocus plne Smpled imge plne Refocus plne Refocus plne Smpled imge plne Microlens rry Sensor Microlens Arry Sensor Min lens principle plne( plne) Min lens imge plne( plne) Microlens focl plne () Figure 1. Plenotpic Cmer Designs. () Ng. Lumsdine et l. severl pplictions such s dynmic refocusing nd depthssisted deep focus rendering. Without loss of generlity, in this pper, 2D simplifiction is used for visuliztion. However, forml mthemticl nlysis nd lgorithms re ll deling with 4D spce. 2. Bckground Integrl or light field photogrphy hs its roots in the methods introduced by Lippmnn [ 19] nd Ives [14] over 100 yers go. Recently, it hs re-emerged with the introduction of plenoptic cmers. In this section, we briefly review previous work in this field. 2.1. Plenoptic Cmer Design Over the lst twenty yers, numerous integrl cmers hve been built [2, 11, 13, 16]. However, it ws not until recently tht Ng [25] improved the trditionl plenoptic cmer design nd introduced new methods for computtionl refocusing. This plenoptic cmer plces the microlens rry on the imge plne Π of the min lens to seprte the converging rys onto the sensor behind it (Figure 1()). The sensor is locted t the focl plne of ech microlens so tht ech microlens is focusing t its opticl infinity (min lens principl plne). The F-numbers of the min lens nd ech microlens re mtched to void Cross-Tlk mong microlens imges. A version of this cmer is vilble from Lytro [22]. Lumsdine et l. [21] introduced nother design by focusing the microlens rry on Π nd correspondingly djusting the position of the microlens rry nd the sensor (Figure 1). In this cse ech microlens imge will hve smples with more sptil resolution nd less ngulr resolution on Π. Therefore this design is cpble of producing () (c) Figure 2. Projection of the rdince corresponding to focusing t different focl depth with the prmeteriztion of the smpled plne. higher resolution results when focusing ner the smpled imge plne. However, the lower ngulr resolution my cuse ringing rtifcts in out of focus regions of the rendered imge. 2.2. Refocusing s Rdince Trnsform In clssicl plenoptic or light field rendering, two prllel plne prmeteriztion is commonly used to represent rys, where ech ry is prmeterized with the coordintes of the minlens principl plne Π uv nd the minlens focl plne Π st. For simplicity, we use two dimensionl vectors s nd u to represent positions (s, t) nd (u, v) on Π st nd Π uv respectively. The irrdince I t positions cn then be computed s: I(s) = 1 R 2 r(s,u)cos 4 Φdu, (1) where R is the distnce between the Π st nd Π uv, nd cos 4 Φ is opticl vignetting term. Sme s [25], we combine cos 4 Φ into r(s,u) for further simplifiction. Eqution 1 shows tht ny photogrph produced by conventionl cmer is the result of 2D integrl projection of the ngulr dimensions of the rdince onto the sptil dimensions, where the slope of the projection relies on the focl depth of the cmer. The reprmeteriztion pproch of [13] llows us to refocus t different scene depths by trnsforming the rdince, s shown in Figure 2: I (s) = 1 R 2 r (s,u)du, (2) where r (s,u) =r(u +(s u) R,u), (3) R I is the irrdince t new focl depth, nd R is the distnce between Π uv nd the trget focl plne Π s t.
2.3. Imge Demosicing While significnt mount of work on plenoptic cmers hs been focusing on improving the imge resolution [27, 5, 10], demosicing remins s n understudied problem. Demosicing, in essence, converts single-ccd color representtions of one color chnnel per-pixel into full per-pixel RGB. The most populr type of CFA in current use is the Byer filter [4]. Demosicing rw Byer imge requires n underlying imge model to guide decisions for reconstructing the missing color chnnels: t every pixel only one color chnnel is smpled nd therefore we need to use its nerby smples to reconstruct the other two chnnels. Mny sequentil methods [17, 15, 1, 12, 24] hve been introduced bsed on the ssumption tht green chnnel is less lised thn the other two due to higher smpling frequency. More sophisticted methods impose locl grdients [ 20] or frequency sttistics [18, 3, 6, 23] s constrints to improve the performnce. However, by fr nerly ll demosicing techniques im to process imges cptured by commodity digitl cmers nd very little work hs been focused on developing solutions specificlly for plenoptic cmers. Existing plenoptic cmers typiclly demosic ech individul microlens imge nd tret the cptured plenoptic function s cptured RGB imge. One exception is the pper by Georgiev et l. [9] tht pplies demosicing fter plenoptic rendering to improve plenoptic superresolution. The pproch presented in [9] used strightforwrd demosicing tht did not resmple the lightfield, resulting in significnt color rtifcts in out-of-focus regions of the rendered imges. Other relted work is the sptil domin multi-frme demosicing nd super-resolution technique reported in [ 7]. However, their focus is to combine multiple low resolution imges wheres we im to mnipulte demosicing to improve refocused imges produced by plenoptic rendering. 3. Imge Demosicing in Plenoptic Cmer Before proceeding with our nlysis, we introduce our nottion. Let I(s) represent the irrdince of pixel s on the imge plne Π s t nd r i represent the RGB rdince of smple ry in microlens m i. I i is the idel opticl RGB imge t m i. In relity, since color filter is used to seprte the colors, we insted consider color filtered imge I fi. For ech color chnnel, I fi cn be viewed s n undersmpled version of I i in tht chnnel. The demosicing opertor D upsmples I fi to recover I i. 3.1. Clssicl Rendering The clssicl plenoptic rendering pproch first pplies demosicing to ech individul microlens imge nd then pplies integrl projection s given in Eqution 2 for focused imge formtion. Let b denote the distnce from the () (c) Figure 3. Artifcts on the cptured rdince introduced by clssicl demosicing. () Ground Truth. Rw microlens imge nd its frequency spectrum. (c) Demosiced microlens imge nd its frequency spectrum. sensor to the microlens rry nd s i denote the loction of the opticl center of m i. In the discrete cse, if we focus t Π s t with distnce to the microlens rry, we cn rewrite the irrdince of Eqution 2 s: I (s) i D(I fi ((s i s) b +s i), (4) Let ω i denote the highest frequency of I i nd ω denote the smpling frequency of I fi. In the trivil cse ( i)[2ω i ω], we cn completely recover the full frequency microlens imges I i nd hence the refocused imge I. In the generl cse when ( j)[2ω j > ω], the spectrum of I fj exhibits lising due to undersmpling s shown in Figure 3. In this cse, the demosic opertor D is used to eliminte undersmpling rtifcts. However, D generlly behves s low pss filter, indiscrimintely removing high frequencies, thereby degrding the imge shrpness of the finl refocused imge. Finlly, if I fi is severely undersmpled, demosicing (such s tht performed by Adobe Photoshop Cmer Rw) cn introduce inconsistent color interpoltion nd cuse color blending in the refocused imge s shown in Figure 3(c) (blck nd white to colorful). 3.2. Resolution on the Refocus Plne Unlike the clssicl pproch, which directly pplies demosicing to the microlens imges, we first project I fi onto the focused plne Π s t nd then perform demosicing. In this section, we provide theoreticl nlysis to show tht the projected imge I f on plne Π s t hs higher smpling frequency thn ny of the microlens imges, hence performing demosicing on I f could gretly improve the imge resolution. For simplicity, we model ech microlens s pinhole cmer nd only nlyze rys pssing through ech opticl center. Also for simplicity, nd without loss of
B C A Refocus plne (h) Δ(h) () s it will increse the resolution between AB. We cn then compute p C = +b h nd the distnce β between A nd C on the focl plne s: m1 h () m2 Microlens rry ˊ ˊ Sensor Figure 4. () Possible resolution enhncement on the refocus plne by projecting multiple microlens imges. Plots of function Δ d (h), β(h), nd γ(k). generlity, we show only one sptil dimension s. Consider two djcent pixels p A nd p B (p A <p B ) in specific microlens m 1 tht mp to two points A nd B on the trget focl plne Π s t. Assume the distnce between p A nd p B is 1, the distnce between two djcent microlenses is d, Π s t lies t distnce to the microlens rry, the sensor lies t distnce b to the microlens rry, nd the spcing between m 1 nd m 2 is h, s shown in Figure 4(). Note tht since the pixel distnce is vnishingly smll compred with, b, nd d, we simply tret these ltter quntities s integers. Our gol is to study how mny rys (pixels) from other microlenses would fll between A nd B on Π s t. This number pproximtes the fctor of resolution enhncement compred with the clssicl demosicing followed by rendering pproch. In order to pproximte this number, we first introduce function γ which mps the index of given microlens to its smpling point between A nd B. Since ll the microlenses out of the minimum period T of γ re duplictions of smples within T, we find out T of γ nd use it s the upper bound of the resolution enhncement. Note tht for ech microlens m 2 different from m 1,we cn hve t most 1 point between AB tht mps to pixel to m 2 s the length AB is preserved in ll microlenses. Assume A nd B mp to points p A nd p B in m 2, s shown in Figure 4(). Note tht p A nd p B my not be pixels. In the first cse, p A nd p B fll exctly on the pixels position. In tht cse, no dditionl rys (pixels) from m 2 would intersect the segment AB on plne Π s t. Therefore, m 2 would not contribute to enhncing the resolution between AB. Under similitude reltionship, the conclusion holds for ny pir of djcent pixels in m 1 nd m 2, i.e., m 2 would not contribute to enhncing the resolution to m 1 s imge. In the second cse, A nd B do not coincide with pixels in m k nd there is exctly one point C between A nd B tht mps to pixel in p C in m k. We cll C super-pixel 1 (+) h β(h) =( + b h + b h ) b. (5) Note tht function β(h) is periodic function with mini- +b mum period of < 1. For ech microlens, we cn substitute its distnce h into m 1 nd compute the loction of this super-pixel. If the super-pixels in some N microlenses hve identicl β vlues, then these microlenses only contribute 1 rther N super-pixels for enhncing the resolution between AB. To finlly compute the exct resolution enhncement, recll tht in the microlens rry setting, h = kd from m 1, where k is some positive integer nd d > 1. We cn then conctente the microlens smpling function ( Dirc comb) Δ d (h) with the distnce function β(h) s: Δ d (h) β(h). To further simplify, we cn fctor d into γ(h) d(+b) so tht γ(k) =Δ(k) β (k). where β (k) hs period nd Δ(k) hs period 1. Clerly γ(k) hs minimum integer period equl to the lest common integer multiple of d(+b) nd 1. We rewrite d(+b) s n irreducible frction two integers m n. Thus, S (k) hs minimum integer period m = gcd(,d(+b)), where gcd denotes the gretest common divisor opertor (Figure 4). Note tht the number of microlenses shring field of view lso constrins the number of distinct smples between p A nd p B. Since the shift from one microlens imge to nother for ny point p on Π s t is Δ=db, we cn compute the number of microlens covering p s: n p = d Δ = f. (6) f Combining with Eqution 5 we obtin tht the resolution enhncement fctor from microlens imge m i to Π s t is equl to min(m, n p ). Since b = f f, this fctor is controlled only by nd f, nmely, the depth of the scene nd the cmer optics. 3.3. Our Approch Projecting smples of ech microlens to the refocus plne Π s t gives us higher resolution imge I f. However, s shown in Figure 5(), when the cptured rdince is trnsformed to Π s t for projection, s proposed by Georgiev et l. [9], the spcing of ech color component is not uniform on I f, resulting in rndom RGB ptterns (Figure 6()). This issue cretes trouble for demosicing I f. Therefore crucil step of our pproch is to resmple the rdince with the prmeteriztion of Π s t to chieve constnt spcing on ech dimension (Figure 5).
Opticl phse spce () Color filtered imge Figure 5. Opticl phse spce illustrtion of resmpling the cptured rdince. () Directly projecting the cptured rdince onto the refocus plne. Projecting the resmpled rdince onto the refocus plne. () Resmpling We dopt similr pproch to tht in [28], which ws originlly developed for multi-frme single chnnel imge restortion. We use frequency-domin pproch to resmple the 4D color filtered rdince. This simplifies to reconstructing higher resolution color imge by perfect registrtion with n rry of low resolution color imges tken t the sme time in 2D imge restortion cse. Here we only consider the green rys. The other two chnnels cn be computed in similr mnner. Suppose we hve q microlenses. Ech microlens cptures low resolution rdince with N s nd N u smples on ech dimension. Let r o (s,u) be the originl green rys prmeterized by Π st nd Π uv. Given the distnce from Π st to the microlens rry, the registrtion of recorded sub-rdince r i cn be computed ccurtely s offsets σ s,σ u on ech dimension respectively. Therefore, the smpled rys by microlens m i is r i (s,u) =r o (s + σ si,u + σ ui ). In frequency domin, this yields: R i (S,U)=e j2π(σsis+σuiu) R o (S,U) (7) where R o (S,U) nd R i (S,U) re CFT of r o (s,u) nd r i (s,u) respectively. Let pixels under m i cpture r i with uniform spcing (T s,t u ), nd R di (Ω) be the discrete Fourier trnsform (DFT) of the rys recorded by i th microlens t frequency Ω=(ω s,ω u ). From the lising reltionship between CFT nd DFT, R di (Ω) stisfies the following eqution: R di (Ω)=K m s (R i ( ω s + m s f s, N m s T s u where K = 1 T st u, nd f s, f u re smpling frequencies on ech dimension of ll micro imges. All opertors rnge from to nd m s,m u re integers. Substituting R i from Eqution 7 to Eqution 8 yields: V Ω = M Ω R Ω, (9) where V Ω is q dimensionl column vector with i th element equl to R di (Ω); Let B S,B U be periodic boundries of R o such tht R o (S,U) =0for ny condition of Figure 6. Rendered results using () the pproch proposed by Georgiev et l. [9] nd our pproch. The out of focus foreground objects exhibit RGB ptterns in () due to non-uniform spcing of color components fter integrl projection. S >B S f s, U >B U f u stisfies; R Ω is 4B S B U dimensionl column vector with the k th element R o ( ωs N st s + ω γ s f s, u N ut u + γ u f u ), nd γ s = kmod(2b S ) B S, γ u = k 2B S B U, nd M Ω is q 4B S B U mtrix with (i, k) th element 1 exp{j2π[σ si ( ω s + γ s f s )+σ ui ( + γ u f u )]}. T s T u N s T s N u T u Since we know the loctions of Π st nd of ech microlens m i, σ si nd σ ui cn be ccurtely computed. R di (Ω) cn be cquired by performing the 4D DFT on the smpled rdince by ech microlens. Therefore Eqution 9 is solvble for unknown R Ω, which contins 2B S nd 2B U frequency smples of R o (Ω) on ech dimension respectively. Combining ll R Ω provides n estimte of R o with 2N s B S, 2N u B U smples rnging from ( B S f s, B U f u ) 1 1 to (B S f s nd B U f u ) with spcing ( N st s, N ut u ) on ech dimension respectively. We then use it to estimte r o (s,u) ( Ts 2B S, from (0, 0) to ((N s 1)T s, (N u 1)T u ), with spcing T u 2B U ). Hence the resolution of the resmpled rdince is incresed by 2B S, 2B U on s nd u compred with tht of ech originl microlens imge. An optimized process of solving for R Ω is presented in Appendix A. ω u + m u f u )), (8) N u T u Integrl Projection nd Demosicing As shown in Figure 5, with the previous resmpling process, we cn chieve n evenly-smpled rdince on the trget focl plne Π s t. The integrl projection is immeditely pplied to get I f. An exmple of the green chnnel of I f is shown by Figure 7(c). However, due to the higher smpling rte of the green chnnel, demosicing process is still needed for red nd blue chnnels of I f to render full RGB imge with the resolution of the green chnnel. ω u
Cptured Rdince Rendered Imge Resmpling Integrl Projection Demosicing () (c) Figure 7. From ()-(c), we compre the ground truth, the result using clssicl pproch, nd the result using our pproch. The frequency spectrums re shown in the bottom row. Trditionl sequentil demosicing frmeworks first recover full resolution green chnnel nd subsequently use tht green chnnel to fcilitte the recovery of red nd blue chnnels. In our cse, the full resolution green chnnel is lredy known fter the integrl projection. Bsed on this green chnnel, the red nd blue chnnels re reconstructed by pplying the stte-of-the-rt nisotropic dptive filtering [18] in the frequency domin. Figure 6 shows tht by employing the resmpling scheme, the demosicing cn be performed on the integrl projection result nd the finl imge is free of RGB ptterns. Suppose the resmpled rdince hs highest frequency ω. The most common sitution is ( i)[ω > 2ω i >ω]. In this cse the new demosicing process preserves more high frequency informtion of the rdince, hence producing higher resolution imge (Figure 7). In other cses such s ( i)[ω >ω>2ω i ] (very smooth regions such s plces with constnt color), both processes recover the full rdince nd the resolution of the resultnt imges re the sme. If ( i)[2ω i >ω >ω] (texture rich regions or shrp edges), the finl imges re both over-smoothed. As illustrted by column () nd of Figure 7, with the clssicl pproch, significnt losses in high frequency components occur in texture-rich regions nd the rendered result suffers from over-smoothing compred with the ground truth. Column (c) shows our method preserves much more high frequency informtion of the ground truth, therefore cpble of producing higher resolution imge. 4. Implementtion nd Applictions Figure 8 shows the pipeline for implementing our proposed plenoptic demosicing nd rendering scheme. We first resmple the rdince, then integrl project it onto the sptil domin, nd finlly demosic the color filtered result. Our experimentl dt is cptured by plenoptic cmer Figure 8. Our plenoptic demosicing nd rending pipeline. similr to tht described in [21]. We use 39-megpixel sensor with pixel size 6.8 μm. The min lens is mounted on the cmer with 13mm extension tube, which provides the needed spcing to estblish n pproprite distnce from the min lens focl plne to the microlens rry. The focl length of the min lens nd of ech microlens re 80mm nd 1500μm respectively. The microlens pitch is 500 μm, which mkes it work with the F-number of the min lens. The distnces between microlenses re 74 pixels. 4.1. Enhnced Dynmic Refocusing We first test our resolution enhncement performnce by synthesizing photogrphs with shllow depth of field. Figure 9 shows the comprison of our pproch nd clssicl rendering () on resolution chrt scene. The bottom rows of () nd compre the demosiced nd rw microlens imges of three highlighted regions. Note tht severe lising effects pper on ech rw microlens imge nd the structure of the resolution chrt is not visible. If demosicing is performed directly on ech microlens imge, colorful rtifcts re introduced, dmging the high frequency informtion nd over-smoothing microlens imges. As result, these regions could not be successfully reconstructed in the finl imge, s shown in (). On the contrry, our pproch utilizes ech lised microlens imge to resmple high resolution rdince before demosicing is performed. Thus preserving lrger portion of high frequency informtion nd producing higher resolution imge, s shown in. Also note tht low frequency regions such s the left bottom prt of the chrt re eqully cler in both cses, nd very high frequency regions such s the bottom of the red highlighted region re both blurry. The top row of Figure 10 shows n outdoor scene. Apprently, the numbers on the licence plte in re not visible but redble in (c). Another visible rtifct of the clssicl frmework here is tht smll regions of speculr highlight pper less shiny due to over-smoothing on ech microlens imge. In nother rel scene shown in the second row of Figure 10. In column, the first line of chrcters re brely
() Figure 9. Comprison of rendered imge employing clssicl pproch nd our pproch. () Clssicl pproch. Top row: Rendered imge. Bottom Row: Demosiced microlens imge. Our pproch. Top row: Rendered imge. Bottom row: Rw microlens imge. redble using the clssicl rendering. Nevertheless, they re clerly rendered with our pproch. Note tht colorful rtifcts introduced by demosicing ech microlens imge remin on positions of nf nd ffi in nd ringing rtifcts lso pper round the edges of the chrcters. Furthermore, the lower chrcters re totlly blurry in while still redble in (c). 4.2. Extended Depth of Field Another populr ppliction of our method is the extended depth of field photogrphy. Our pproch precomputes the depth of the smpled rdince nd renders ech pixel by choosing its own depth mong smples utomticlly. We present our depth estimtion lgorithm designed for the cptured rdince in Appendix B. The third row of Figure 10 shows our extended depth of field ppliction on the sme dt s the second row. Note tht the originl out of focus regions such s the fce nd hir of the person re brought into focus, s if the photogrph is cptured by pinhole perture cmer. However, with our frmework, shown in (c), the rendered result preserves more high frequency informtion thn the clssicl pproch shown in, therefore produces much more detiled look. 5. Discussions nd Limittions We hve presented well-principled plenoptic demosicing nd rendering frmework, which preserves more high frequency informtion from the cptured rdince nd generte less lising rtifcts compred with the clssicl pproch. Our frmework does not pply demosicing directly to the imge cptured by the plenoptic cmer. Insted, with resmpling scheme which helps chieve constnt spcing on ech dimension, it dynmiclly performs demosicing fter integrl projection. Extensive experiments show tht this frmework could produce photogrphs with commercilly cceptble resolution. As nlyzed in Section 3.2, the resolution enhncement of ech plne in the scene chieved by our lgorithm vries ccording to the depth of the plne. This could cuse unplesnt results if the resolution enhncements re low on plnes of interests. In the extreme cse, the resolution could be s low s the clssicl frmework. Like clssicl plenoptic photogrphy, our pproch ssumes the cptured rdince re thin rys in order to reconstruct refocused imge. This is lso our ssumption for theoreticl resolution enhncement nlysis. 6. Acknowledgments Z. Yu nd J. Yu were supported by the Ntionl Science Foundtion under grnts IIS-CAREER-0845268 nd IIS-RI-1016395, nd by the Air Force Office of Science Reserch under the YIP Awrd. Imgery used to crete the bottom two rows of Figure 10 re from WSCG 2010 (http://www.wscg.eu), courtesy of Vclv Scl. References [1] J. Adms nd J. H. Jr. Adptive color pln interpoltion in single sensor color electronic cmer, ptent us 5506619, 1996. 3 [2] E. Adelson nd J. Wng. Single lens stereo with plenoptic cmer. IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence, 14:99 106, 1992. 1, 2 [3] D. Alleysson, S. Susstrunk, nd J. Herult. Liner demosicing inspired by the humn visul system. IEEE Trnsctions on Imge Processing, 14(4):439 449, pr. 2005. 3 [4] B. E. Byer. Color imging rry. US Ptent 3,971,065, 1976. 3 [5] T. E. Bishop, S. Znetti, nd P. Fvro. Light field superresolution. In IEEE ICCP, 2009. 3 [6] E. Dubois. Filter design for dptive frequency-domin byer demosicking. In IEEE Interntionl Conference on Imge Processing, oct. 2006. 3 [7] S. Frsiu, M. Eld, nd P. Milnfr. Multifrme demosicing nd super-resolution of color imges. IEEE Trnsctions on Imge Processing, 15(1), jn. 2006. 3 [8] T. Georgeiv, K. C. Zheng, B. Curless, D. Slesin, S. Nyr, nd C. Intwl. Sptio-ngulr resolution trdeoff in integrl photogrphy. In Proceedings of Eurogrphics Symposium on Rendering, pges 263 272, 2006. 1 [9] T. Georgiev, G. Chunev, nd A. Lumsdine. Superresolution with the focused plenoptic cmer. In Proc. SPIE 7873, 2011. doi:10.1117/12.872666. 3, 4, 5 [10] T. Georgiev nd A. Lumsdine. Focused plenoptic cmer nd rendering. Journl of Electronic Imging, 19, 2010. 3 [11] S. Gortler, R. Grzeszczuk, R. Szeliski, nd M. Cohen. The lumigrph. In Proceedings of ACM SIGGRAPH, pges 43 54, 1996. 2
() (c) Figure 10. Comprison of three results with clssicl pproch nd our pproch. First nd second row show shllow depth of field rendering. The third row shows extended depth of field rendering. () Our rendered result. nd (c) re enlrged highlighted regions in () with clssicl pproch nd our pproch respectively. [12] R. Hibbrd. Apprtus nd method for dptively interpolting full color imge utilizing luminnce grdients, ptent us 5506619, 1995. 3 [13] A. Isksen, L. McMilln, nd S. Gortler. Dynmiclly reprmeterized light fields. In Proceedings of ACM SIGGRAPH, pges 297 306, 2000. 2 [14] F. Ives. Prllx stereogrm nd process of mking sme, ptent us 725567, 1903. 2 [15] R. Kkrl nd Z. Bhrv. Adptive demosicing with the principl vector method. IEEE Trnsctions on Consumer Electronics, 48(4):932 937, nov. 2002. 3 [16] M. Levoy nd P. Hnrhn. Light field rendering. In Proceedings of ACM SIGGRAPH, pges 31 42, 1996. 2 [17] X. Li nd M. Orchrd. New edge-directed interpoltion. IEEE Trnsctions on Imge Processing, 10:1521 1527, 2001. 3 [18] N. Lin, L. Chng, Y. Tn, nd V. Zgorodnov. Adptive filtering for color filter rry demosicking. IEEE Trnsctions on Imge Processing, 16(10):2515 2525, oct. 2007. 3, 6 [19] G. Lippmnn. L photogrphie intgrle. Comptes-Rendus, Acdmie des Sciences, 146:446 451, 1908. 2 [20] W. Lu nd Y. Tn. Color filter rry demosicking: new method nd performnce mesures. IEEE T-IP, 12:1194 1210, oct. 2003. 3 [21] A. Lumsdine nd T. Georgiev. The focused plenoptic cmer. In In Proc. IEEE ICCP, 2009. 1, 2, 6 [22] Lytro. www.lytro.com. 1, 2 [23] D. Menon nd G. Clvgno. Demosicing bsed on wvelet nlysis of the luminnce component. In IEEE Interntionl Conference on Imge Processing, volume 2, pges 181 184, 2007. 3 [24] D. Muresn nd T. Prks. Demosicing using optiml recovery. IEEE Trnsctions on Imge Processing, 14(2), feb. 2005. 3 [25] R. Ng, M. Levoy, M. Brdif, G. Duvl, M. Horowitz, nd P. Hnrhn. Light field photogrphy with hnd-held plenoptic cmer. Stnford University Computer Science Tech Report, 2(2005-02):1 11. 1, 2 [26] Rytrix. www.rytrix.com. 1 [27] J. Stewrt, J. Yu, S. J. Gortler, nd L. McMilln. A new reconstruction filter for undersmpled light fields. EGSR 03, pges 150 156, 2003. 3 [28] R. Tsi nd T. Hung. Multi-frme imge restortion nd registrtion. Advnces in Computer Vision nd Imge Processing, 1, 1984. 5
Appendices A. Solving R Ω If we seprte M Ω into components relted nd unrelted to Ω, it cn be further decomposed into D Ω M v, where the D Ω is digonl mtrix with i th digonl element equls to: 1 exp{j2π[σ si ( ω s ω u B S f s )+σ tu ( T s T u N s T s nd M v is p 4B S B U mtrix with [M v ] ik equls to: exp{j2π[σ si kmod(2b S )+σ ui k ]}. B S therefore we cn rewrite Eqution 9 s: M v R Ω = D 1 Ω V Ω. (10) Since M v is independent of Ω, therefore we only need to solve it once for ll smpled frequencies. The rows of M v re linerly independent if the shifts on dimensions stisfies σ si σ sj + n s T s, σ ui σ uj + n u T u, where i j nd n s,n u re some integers. If q 4B S B U, Eqution 10 is consistent. Otherwise, M v is singulr nd we obtin the minimum-norm, lest-squre-error estimte of R Ω by first computing the generlized inverse of M v. This yields: R Ω =(M T v M v ) 1 (M T v M v )D 1 Ω V Ω. (11) Note tht for ech micro imge, the smpling frequency on t, v dimensions out ofs,u is doubled for the green chnnel, therefore the resmpled rdince hs two times higher resolution on t, v dimensions of the green chnnel. B. Depth Estimtion on the Rdince Recll tht ech microlens imge cptures prt of the scene from different view point. Therefore, multi-view depth estimtion is fesible to retrieve the depth informtion from the entire cptured rdince. Out lgorithm itertively computes the depth of the pixel bsed on Grph Cuts. Given certin depth, we utilize the vrince of the corresponding pixels mong microlens imges s the dt term. Since our frmework does not perform demoscing on the rw dt. Therefore, when computing the dt term for ech pixel, only the pixels with the sme color chnnel in different microlens imges re used. One common problem with depth estimtion is tht even though the correct depth is ssigned to one pixel, the dt term could still be lrge due to occlusion. This problem is especilly severe in the rdince cse since mny views re involved in the computtion. To resolve the this issue, grdully incresing confidence threshold is given in ech itertion, so tht the pixels with lower depth will converge first nd the pixels with lrger depth my still generte dt terms bigger thn the threshold due to occlusion. Once the depth of one pixel is decided, it will not be involved in the computtion nymore. Thus, the pixels with lrge depth will void these low depth pixels nd produce low dt term if ssigned the correct depth in the lter itertions. The complete lgorithm is given in Algorithm 1. B U f u )]}, Algorithm 1 Clculte depth of ech pixel d i N u T u Require: Cptured rdince r for j =1 M do for i =1 N do if p i hs been ssigned with ny depth vlue then Continue; end if Compute dt term e d of pixel p i ; if e d >k j then {Dt term is bigger thn current threshold} Continue; else Add p i to grph end if end for Perform grph cuts; Assign depths to pixels in the grph; end for