Artifacts Reduction in Image Rendering of the Focused Plenoptic Camera Peng Liu 1, Rumin Zhang 2

Size: px

Start display at page:

Download "Artifacts Reduction in Image Rendering of the Focused Plenoptic Camera Peng Liu 1, Rumin Zhang 2"

Ashlyn Bond
6 years ago
Views:

1 International Conference on Intelligent Systes Research an Mechatronics Engineering (ISRME 2015) Artifacts Reuction in Iage Renering of the Focuse Plenoptic Caera Peng Liu 1, Ruin Zhang 2 School of Electronic an Inforation Engineering, Beijing University of Aeronautics & Astronautics, No.37 Xueyuan Roa, Haiian Distract,Beijing,China State Key Laboratory of Wireless Mobile Counications, China Acaey of Telecounications Technology (CATT),Beijing,China @qq.co, liuijun@leacoretech.co Keywors: plenoptic caera;icro lens ; artifacts reucing; igital refocusing; epth of fiel Abstract. Plenoptic caera provies us with the capability of refocusing photograph an extening the epth of fiel(dof) after exposure. But when we rener iage patch by patch, the output iage is renere with visible artifacts because of isatch of the ajacent patch bounaries. The paper introuce a novel iage renering algorith base on the focuse plenoptic caera. The algorith extracts the epth inforation fro the raw light fiel ata, an then rener the result iage patch by patch base on the epth ap. The algorith can export an iage setting in a esirable focus. It can also exten the epth of fiel to the case that the whole scene in the iage is in focus. An fro the siulation, it can be learnt that coparing with the traitional algorith, the artifacts in the renering result are suppresse significantly to a sightless level. Introuction Plenoptic caera cae out in recent years. It ipleents a high efficient, portable an low cost evice to recor both spatial an irectional inforation of the scene. With the capture light fiel ata, it is to set the focus an exten the epth of fiel when the exposure is over. Fourier slice is propose to rener refocus result in high efficiency. But it rener only one pixel per sub iage. So the resolution of the iage renere fro plenoptic caera is as the nuber of the icro lenses in the. A novel structure of plenoptic caera was presente later by A.Lusaine an T.Geogiev. The icro lenses are carefully set to focus on the iage fore by the ain lens. So it is calle focuse plenoptic caera or plenoptic caera 2.0. It traes irectional inforation for the spatial resolution of the light fiel inforation. Thus, the spatial resolution of focuse plenoptic caera increases in the refocus result. However, it ay lea to visible artifact ue to lack of irectional inforation. Base on the oel of focuse plenoptic caera, the paper illustrates the relationship between the light fiel saples an the target renering iage, an introuces a novel algorith to buil an igital refocus an extene DOF iage. The epth ap of the scene is neee in the algorith an it will be extracte by fining the corresponence between the sub iages of ajacent icro lenses. With the assistant of new algorith in the paper, when we rener an refocus iage fro patch to patch corresponing to the each sub iage, we will get a continuous transition at the joint of ajacent renere patches. The spatial resolution of the final output picture is ecie by the epth ap of the scene. Renering Iage with a focuse Plenoptic Caera A. Focuse Plenoptic Caera The structure of focuse plenoptic caera propose by A.Lusaine an T.Geogiev is as escribe in Fig. 1. It consists of a ain lens an a icro lens positione close to the sensor. Each icro lens serves as a icro caera that saples the iage of the ain lens The authors - Publishe by Atlantis Press 1292

2 Micro lenses ipleent ray separators. Their iage circles ivie the sensor into sub iages. The coorinate of one sub iage (, n) cobining with the pixel ( s, t) in this sub iage give us the raiance corresponing to a efinite irection an position. For all the sub iages an pixels, the light fiel in the caera can be establishe as t,, n). In a practical plenoptic syste, the icro lens is square. For siplicity, the following analysis only takes the ( s, iension into consieration. For the ( n, t) iension it has a siilar conclusion. The light fiel ata is rea fro the sensor ata I s (x). Thus it has: = I s ( * Nx + s) (1) where Nx is the resolution of each sub iage. It has Nx = δ / δ. Here is the size of icro lens an δ is the iaeter of the pixel on the sensor. Micro lens Iage plane Main lens Sensor plane v Figure 1. z v z u Structure of focuse plenoptic caera It is well known that the connection lines between the optic center of the ain lens an the ege of the icro lenses ivie the iage of the ain lens into NX patches. It is the sae as the nuber of icro lenses. An our algorith will rener fro patch to patch on the ifferent iage planes. To rener an artifacts suppresse result, the epth ap corresponing to each iage plane z ( 0 < NX ) is necessary. A approach which extracts epth ap fro ulti-views is propose as follow. B. Depth Map Estiation Depth estiation can be consiere as a iniization of the atching error of the iages fore by neighbor icro lenses. For an arbitrary sub iage an its neighborhoo, it has: 1 Φ( ) = I( + i, I s ( i, + 1) (2) W i is the isparity of pixels between ajacent sub iages as in the Fig. 2. An W is nuber of atching pixels. We nee to search for the optial when the atching error Φ( ) gets its iniu. An the ap of epth can be calculate as: z = v, 0 < N x (3) 1293

3 Micro lens v z iage Sensor plane Figure 2. Depth ap is extracte fro the relation of neighbor sub iages. C. Renering Algorith The propose algorith will rener an iage at specifie F-nuber N F an refocus plane that has a istance of α v'. We will euce the relationship between an arbitrary patch an the light fiel. For an pixel Z(s) on the renering patch, the light projection of an arbitrary point Z(s) to the icro lens overlays a certain nuber of icro lenses, whose sub iages offer effective inforation to the renering patch. An it can be known that the iaeter of this projection is ecie by the specifie photographing aperture size N F an epth of the patch z. The iaeter(nuber of icro lenses) can be calculate: z F / N F L = (4) z [ ] is the rouning sybol. As it is shown in Fig.3, sub iages of the icro lenses that lay within this range have effective inforation of Z(s). We consier the as the effective sub iages of patch. An the raiance of Z(s) is the weighte average of all the corresponing re pixels of Z(s) on the sensor. Uner the Labertian assuption that the object prouces reunant raiance in ifferent angle, we can figure that all corresponing pixels of Z(s) on the sensor contribute the sae to the raiance of Z(s). Thus all weights are equal. Fro Fig.4, we can see the pixels unerline by re on the sensor contribute inforation to the patch, while those unerline by blue contribute to the patch +1. In each effective sub iage, the nuber of effective pixels is: v z Pz ( = z Pz ( ) (5) is the function of the epth of the patch. Thus ifferent patch has ifferent renere resolution on its epth. To get the final output, it is necessary to project all the patches fro their epth to the reference plane. The reference plane is selecte as the refocus plane, which is ecie by the refocus factor α. We scale the patch by the resolution of P( α v') on the reference plane as shown in Fig. 4. An the renering result is the stitching of all the scaling patches. 1294

Sensor Micro lens Subiage L zs () Renering patch, f F / N f, F Figure 3. v Repetitions of the renering pixel on the sensor z Sensor Micro lens Digital refocusing plane Pz ( ) z a z : Figure 4.

4 Sensor Micro lens Subiage L zs () Renering patch, f F / N f, F Figure 3. v Repetitions of the renering pixel on the sensor z Sensor Micro lens Digital refocusing plane Pz ( ) z a z : Figure 4. v, f z + 1 F / N f, F Renering algorith base on the epth ap Experiental Results D. Ipleentation Our Algorith can perfor as following 4 steps: The 4D light fiel of the scene is establishe fro the sensor ata. The epth ap for all the patches to be renere is estiate fro the light fiel ata. Every patches is renere accoring to the epth ap. They have ifferent resolution ue to ifferent epth. Aperture factor N is specifie to ecie the repetition of a renering pixel on F the sensor. It controls the DOF of the result. Project all the patches to the reference plane. The reference plane is selecte as the specifie refocus plane by the factor α. The final refocusing result is the stitching of all the projections of renere patches on the reference plane. E. Siulation Exaple Our siulation is one with the pre-processe light fiel ata capture with a prototype of focuse plenoptic caera by T. Georgiev. Fig. 5 is the raw light fiel iage, which is copose of an of sub iages(96*72). Each icro lens aps part of the iage of the ain lens onto the sensor plane. Sub iages are fore behin the icro lenses an have a resolution of 75*75. If we zoo in an exaine the ifference between the ajacent sub iages, we can clearly fin a parallax shift. Figure 5. Raw plenoptic ata capture by focuse plenoptic caera 1295

Fig. 6 inicates the epth ap of the scene. We extract it by fining the parallax of the sub iages. The light part on the picture eans shallow epth where the ark part eans eep epth.

5 Fig. 6 inicates the epth ap of the scene. We extract it by fining the parallax of the sub iages. The light part on the picture eans shallow epth where the ark part eans eep epth. The resolution of epth ap is as uch as the nuber of nuber of renering patches. Figure 6. Depth ap extracte fro the plenoptic ata Figure 7. Siulation exaple (a) Reconstruction of refocus iage with the propose algrotih (b) without epth ap Figure 8. Siulation exaple (a) Reconstruction of all-in-focus iage with the propose algrotih (b) without epth ap Fig. 7(a) shows the renere iage with the algorith in the text. The refocus factor α is carefully set to refocus the iage at the face of the person in the front. Fig. 7(b) shows the result renere by algorith without epth estiation. Fro the coparison we can learn that the result in Fig. 7(b) has visible artifacts especial in the area out of focus because of sparse. An in Fig. 7(a), the backgroun is soothly blurring an artifacts are suppresse obviously. Fig 10(a) shows renere iage that the F-nuber N F is set to its axiu. An the epth of fiel is extene to axiu. The iage is all-in-focus fro the woo in the backgroun to the people in the foregroun. Fig 10(b) is the all-focus result renere by algorith without epth estiation. It is well known that the artifacts are sightless after using the propose algorith when we rener an iage that is all focus. 1296

6 Conclusions Reconstruction resolution is one of the ain liitations put on the application of plenoptic caera. When we use focuse plenoptic caera to trae irectional resolution for high renering resolution, it ay bring about artifacts in the final output ue to lack of irectional inforation. The paper first iscusse the structure of focuse plenoptic caera propose by Georgiev, T. an Lusaine, A. Then it evelope a new refocus algorith to rener picture base on the epth ap extracte fro the light fiel ata. Coparison of renering results between this two techniques was also ae both for igital refocusing iage an all-in-focus iage. It can be seen that new algorith reconstructs result as high resolution as the ol algorith, with artifacts to an unconspicuous level, at the cost of the calculation of epth ap estiation. The renere iage has sooth transition at the ege of stitching patches. Acknowlegent The research work was supporte by State Key Laboratory of Wireless Mobile Counications, China Acaey of Telecounications Technology (CATT). References [1] Aelson,T. an Wang. Single lens stereo with a plenoptic caera. IEEE Trans Pattern Anal Machine Intell, 14:99-106, [2] Ng, Levoy M. et al. Light fiel photography with a han-hel plenoptic caera. CTSR02. California: Stanfor University Coputer Science [3] Lusaine, A. an Georgiev, T. The focuse plenoptic caera. International Conference on Coputational Photography, April [4] Chihchieh Chen, Yichang Lu, Mingshing Su. Light Fiel Base Digital Refocusing Using a DSLR Caera with a Pinhole Array Mask. ICASSP2010 [5] A. Lusaine. T. Georgiev. Full Resolution Light-fiel Renering. Teach.rep, Aobe Systes, January [6] Lusaine, A. an Georgiev, T., The focuse plenoptic caera, in International Conference on Coputational Photography, April [7] Georgiev, T. an Lusaine, A., Reucing plenoptic caera artifacts, Coput. Graph. Foru 29(6), [8] Liu Peng, Liu Dijun. All-in-focus iage reconstruction in plenoptic caeras ICIG 2013 [9] Liu Peng, Liu Dijun. High resolution igital refocusing base on focuse plenoptic caera. ICCSA 2014 [10] R. Ng, M. Levoy, M. Brif, G. Duval, M. Horowitz, an P. Hanrahan. Light fiel photography with a han-hel plenoptic caera. Tech. Rep., [11] R. Ng. Digital light fiel photography. PhD thesis, Stanfor,CA, USA, Aviser- Patrick Hanrahan. [12] Anrew Lusaine, LiliLin, Jereiah Willcock, Yuuo Zhou, Fourier Analysis of the Focuse Plenoptic Caera, SPIE-IS&T/ Vol. 8667,

Evaluation of a multi-frame blind deconvolution algorithm using Cramér-Rao bounds

Evaluation of a multi-frame blind deconvolution algorithm using Cramér-Rao bounds Evaluation of a ulti-frae blind deconvolution algorith using Craér-Rao bounds Charles C. Beckner, Jr. Air Force Research Laboratory, 3550 Aberdeen Ave SE, Kirtland AFB, New Mexico, USA 87117-5776 Charles