Constrained Up-Scaling for Direct and Global Image Components

Transcription

1 Constrained Up-Saling for Diret and Global Image Components Julian Bader Martin Pätzold Andreas Kolb Computer Graphis Group, Institute for Vision and Graphis, University of Siegen Hölderlinstraße 3, Siegen, Germany { julian.bader,martin.paetzold,andreas.kolb }@uni-siegen.de Abstrat The separation of diret and global illumination omponents is interesting for many appliations in Computer Graphis and Computer Vision, suh as BRDF estimation or material lassifiation. However, for full-resolution images, a large number of oded images have to be aquired. For many interative appliations, suh as the aquisition of dynami senes or video apturing, this is not feasible. In this paper, a new onstrained up-saling tehnique for separated diret and global illumination images is proposed whih requires two to three oded input images, only. Our approah imposes the boundary ondition that the sum of the diret and global omponents equals the fully illuminated image. We work in a preditive-orretive manner where we first use a single-image up-saling method in order to predit the higher resolution images. Afterwards, the missing higher frequenies are determined using a fully illuminated image. As the distribution of the higher frequenies differs among the various frequeny bands, we apply our approah in an iterative way for small up-saling steps distributing the missing information by minimizing the overall frequenies. We evaluate the up-saling sheme and demonstrate the improvement ompared to single-image approahed. As our method aims at minimizing the strutured light patterns needed for aquisition, we additionally disuss the performane of existing pattern sets in terms of appliability for dynami senes. Keywords Computational Photography, Image Proessing, Global Illumination 1 INTRODUCTION The separation of diret and indiret illumination in a sene is an interesting task for both, Computer Graphis and Computer Vision domains. The diret omponent helps obtaining the distribution of refleted light in a sene. This an be used to derive a model for the bidiretional refletion distribution funtion (BRDF) [GKGN11]. Also, 3D reonstrution methods are more robust when working only on the diret omponent [GKGN11, NG12]. The global omponent gives insight into the sattering behaviour between one or more objets in a sene. This is useful to ahieve more photorealisti renderings of a sene by observing the omplex light flow [MYR10]. Also, objet reognition benefits from suh approahes as the sattering of an objet highly depends on the material [GL12]. In 2006, Nayar et al. [NKGR06] presented an approah for separating the diret and global illumination using a strutured- Permission to make digital or hard opies of all or part of this work for personal or lassroom use is granted without fee provided that opies are not made or distributed for profit or ommerial advantage and that opies bear this notie and the full itation on the first page. To opy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speifi permission and/or a fee. light approah. However, for a high resolution separation, a large number of strutured-light patterns is needed. Thus, this approah is not feasible for dynami senes, suh as apturing biometri information of human faes or mobile navigation, or for video apturing. In their work, an alternative method was proposed where only one strutured light pattern is suffiient. The disadvantage of this method is that the image resolution dereases dramatially. As a onsequene, small sale details of fine textured regions are lost. In this paper, we propose a new onstrained image upsaling tehnique for diret and global omponent images. We employ an iterative predition-orretion approah whih uses a single-image up-saling method in order to predit the higher resolution representation for eah omponent. These up-saled images are then orreted aording to a high resolution fully illuminated image whih has been additionally aquired. Fine strutures in images, suh as textiles or fur, an be reovered although they are not apparent in the low resolution input image. Our ontributions are the following: A general onstrained up-saling sheme for the transfer of high frequeny information. The appliation of the proposed sheme for diret and global omponent images. Volume 21, ISSN

2 An evaluation of various low-resolution separation approahes whih shows that the resulting image quality is not suffiient for the proposed up-saling sheme. To the best of our knowledge, we are the first proposing a onstrained up-saling approah for diret and global omponent images. The rest of the paper is strutured in the following way: Se. 2 gives an overview of existing single-image and multi-modal up-saling tehniques. In Se. 3, the priniples of the separation approah of [NKGR06] is reviewed and the impat of the different strutured light patterns is disussed. Our onstrained up-saling method is desribed in Se. 4. In Se. 5, the results are disussed. Furthermore, the improvement of our approah ompared to the single-image predition is shown. Finally, Se. 6 onludes the paper. 2 RELATED WORK In the past, the problem of image up-saling has been addressed many times during the past 20 years. Here, a brief overview of the major approahes is given whereas we distinguish between the single-image and multi-modal approahes. Single-image methods. The single-image up-saling methods an be divided in to four groups: regionbased interpolation, edge-direted interpolation, edgedireted reonstrution and example-based reonstrution. Region-based interpolation methods apply different interpolation algorithms depending on the region in the image. Thurnhofer and Mitra [TM96] proposed an image interpolation approah whih lassifies image pixels into three different regions (onstant, irregular and oriented). Other methods distinguish only between two regions (oriented, homogeneous) for different interpolation sheme appliation [ABA01, CHL05]. Zi et al. [ZDLL12] proposed a pereptual motivated segmentation of regions. For the general and transition regions, different kinds of interpolation algorithms are used. The attention region is omputed aording to an energy formulation optimizing urvature ontinuity, urvature enhanement and isolevel urve smoothing. Edge-direted interpolation shemes adjust the interpolation aording to the edges present in the image. Li and Orhard [LO01] use the loal ovariane for an edge-direted interpolation. Alternatively, sharp luminane variations and disontinuities are onsidered [BGS02]. Su and Willis [SW04] proposed a pixel triangulation sheme in order to use only pixels for interpolation not separated by an edge. Other methods propose the use of the seond order derivative in order to determine the interpolation diretion [GA08, GA11] and iterative refinement. Edge-direted reonstrution algorithms try to estimate a high-resolution edge image whih is then used to reonstrut the up-saled image. Tai et al. [TTT06] proposed to use tensor voting for high resolution edge reonstrution. In alternative approahes, edge statistis are learned in order to predit enhaned edge images [Fat07, SXS08, YXYC12]. Another kind of approah [YXXY12, WXM + 13] omputes sharp high resolution gradients diretly from the low resolution images. Example-based approahes try to build up a ditionary whih relate low resolution image pathes to missing higher frequenies [YWL + 12]. Freeman et al. propose a nearest neighbour searh in the training set to find the high frequenies for image pathes [FJP02]. Kim and Kwon [KK08] formulated the relationship between low and high frequenies as a linear regression model in order to derive a suitable image path. Alternatively, a single-hidden-layer feed-forward neural network is used for learning the relation between low and high images frequenies [AB12]. A variety of methods are based on self similarity [GADTH97, EV07, SSU08, GBI09, FF11]. Here, the assumption is that an image ontains similar strutures at different sale levels. Multi-modal methods. In [LLK08] and [STDT08], methods are proposed for the ombination of low resolution time-of-flight depths images with high resolution RGB images. However, instead of using the RGB information for a onstrained up-saling, both approahes first do a single-image up-saling of the depth image before fusing with RGB data. Langmann et al. [LHL11, LHL12] desribe a fusion approah of timeof-flight and RGB data where several ross bilateral filtering strategies are ompared. Additionally, a framework for joint segmentation and super-resolution is proposed. They assume the smooth hange of the depth field in homogeneous regions. Fine strutures, suh as in fur or textiles, an therefore not be reovered. Diret-Global Separation methods. In 2006, Nayar et al. [NKGR06] proposed a method for separating the diret and global image omponents using high-frequeny strutured-light patterns. Gu et al. [GKGN11] extended this approah in order to use multiple light soures. In 2012, two approahes were presented whih estimate the light transport in a sene [ORK12, RRC12]. Both are also apable of diret-global separation. However, O Toole et al. use a ompliated setup and long aquisition times. The method of Reddy et al. needs a high number of strutured-light patterns. As we fous on the aquisition of dynami senes, the latter approahes are not suitable. The aim of our approah is the reonstrution of small detail whih get lost during single-oded-image separation proess. As the above mentioned single-image up-saling methods an handle singularities, suh as Volume 21, ISSN

3 Here L + (i) refers to the intensity value of a sene element at position i = (x,y) T whih is diretly illuminated, whereas L (i) represent the intensity when the element is not diretly illuminated. L d and L g refer to the diret and global part of the illumination and α is the fration of projetor soure elements whih are lit. Sine digital projetors annot be tuned ompletely dark, b aounts for the fration of a white soure element emitted by a "blak" projeted pixel with 0 b 1. For a more detailed desription of the separation method, the reader is referred to the paper of Nayar et al. [NKGR06]. Figure 1: Light an reah the amera in different ways. Light being diretly refleted to the amera is alled diret illumination. Global illumination denotes light that is sattered in the sene and travels an indiret path to the amera. edges or joints, quite well, fine homogeneous strutures, suh as fur or threads in textiles, annot be reonstruted. Due to the smoothness assumption of depth data, the multi-modal super-resolution methods for fusion of time-of-flight data also annot over these kind of fine details. In ontrast, our proposed method uses a single-image up-saling predition for the diret and global omponent images and enfores the sum of these two to be equal to a fully illuminated image. 3 SEPARATION OF DIRECT AND GLOBAL ILLUMINATION The illumination in a sene an be deomposed into two parts. The first part is the so alled diret illumination. This overs all the light whih is emitted by a light soure and is refleted diretly to the sene observer (usually a amera or the human eye). The seond part addresses all the light whih is first sattered or otherwise redireted and reahes the observer in an indiret path. This is onsidered as global illumination (f. Fig. 1). 3.1 Separation Sheme In 2006, Nayar et. al [NKGR06] proposed a method to separate the diret and global illumination in a sene using strutured light. Here, the underlying assumption is that the sattering of light, whih is responsible for the global illumination effets, ats as a kind of low-pass filter, i.e. there are no high frequenies present in the global omponent. Thus, by projeting different high frequeny patterns onto the sene, the following relations an be formulated depending on whether a part of the sene is hit by a light or dark soure element: L + (i) = L d (i) + αl g (i) + b(1 α)l g (i), (1) L (i) = bl d (i) + (1 α)l g (i) + αbl g (i). (2) 3.2 Strutured-Light Patterns Nayar et al. [NKGR06] proposed several highfrequeny strutured-light patterns in order to aquire L + (i) and L (i) for a sene. Cheker-board. The sene is illuminated by a sequene of shifted heker-board patterns. For eah pixel in a sene the brightest and the darkest intensity value among all the aquisitions is hosen. With this kind of patterns, one an ompute the diret and global omponent diretly at the amera s resolution. However, due to the aquisition of multiple soure images, this method is not feasible for the fast aquisition of senes. Phase shifted sinusoidal. Eah pixel of the pattern an be expressed as p 0 = 0.5(1 + sinφ) for φ [0,2π]. Analogue to the first pattern, two more are generated where φ is phase shifted by 2π 3 and 4π 3. In general, the pixels of the first pattern an be hosen arbitrarily, but in order to ensure high frequeny, a sinusoidal funtion whih varies in both, x- and y-diretion, should be hosen. Using these patterns, only three aquisitions are neessary. However, Gu et al. [GKGN11]stated that this method suffers from serious image artefats. Therefore, it is not suitable if the apturing of fine details is required. Stripes. By using a horizontal or vertial stripe pattern, only one aquisition is neessary. Here, the brightest and darkest pixels are determined in small retangular pathes whih are aligned perpendiular to the stripes. These extrema are interpolated and averaged in order to generate the minimum L + (i) and maximum L (i). Then, the diret and global omponents are omputed. As only one pattern needs to be aquired, this method is suitable for apturing dynami senes. However, due to the averaging proess, the resolution of the resulting images is dereased by a fration of four [NKGR06]. 4 A CONSTRAINED UP-SCALING METHOD The aim of our approah is the onstrained up-saling of diret and loal omponent images aording to one Volume 21, ISSN

4 Figure 2: Overview of the omplete system. The left side (green) shows the separation of global and diret omponents aording to [NKGR06]. The right side (blue) desribes our onstrained up-saling tehnique. high-resolution fully illuminated image whereas we fous on the reonstrution on fine strutures in homogeneous regions. Before the details are disussed in the following two setions, a rough system overview is given. As input images, we take the two low resolution global and diret omponent images retrieved by the stripe pattern separation approah proposed by Nayar et al. [NKGR06]. Additionally we use a high resolution image whih is aquired with full illumination. The proposed up-saling sheme onsists of two steps whih are applied iteratively until we reah the target resolution (f. Fig. 2): 1. In the predition step, we estimate the higher resolution image using a single-image up-saling tehnique. The first iteration takes the low-resolution diret and global omponent images as input. For all following iterations, the orreted images from the previous iteration are used. 2. The orretion step orrets the predited images aording to the main ondition that the sum of diret and global images results in the fully illuminated image. In order to derease the noise level in the up-saled image, the error to the full image is distributed in suh a way that the overall intensity of higher frequenies is minimized. 4.1 Predition In general, there are no limitations regarding the predition step in our up-saling sheme. As the orretion works on the predited image diretly, every singleimage up-saling method ould be used. In our experiments, we used the so-alled Loal Self-Examples approah proposed by Freedman and Fattal [FF11]. This method has two advantages. On the one hand, it an predit the objet boundaries quite well. This improves the performane of the orretion step, as there are lower intensity errors. On the other hand, the approah inreases the resolution of the input images iteratively by nondyadi sales (eg. 5:4 or 3:2). Therefore, in eah iteration only a small higher frequeny band is added to the image, leading to a more robust orretion in terms of frequeny distribution. 4.2 Corretion After the predition step, the image is orreted aording to the onstraint L f (i) = L p d (i) + Lp g(i) (3) where L f (i) is the pixel at position i under full illumination. L p d (i) and Lp g(i) orrespond to the (unonstrained) predited diret and global pixels respetively. Taking the differene of the sum of the diret and global images and the fully illuminated image D(i) = L f (i) (L p d (i) + Lp g(i)), (4) we get the error introdued by the predition step. Please note that, similar to the Differene of Gaussians, D(i) only ontains the missing higher frequenies whih ould not be reovered by the single-image up-saling approah. In order to orret the predited images, we distribute the error via two funtions f d (i,l p d,d), f g(i,l p g,d) so that D i = ( f d (i,l p d,d) + Lp d (i)) + ( f g(i,l p g,d) + L p g(i)) (5) holds. These funtions should be designed in suh a way that they selet these higher frequenies whih are missing in the orresponding predited image. If we now assume f d and f g to be linear with respet to D, we express our orreted images as follows L d (i) = Lp d (i) + α d(i) D(i), (6) L g(i) = L p g(i) + α g (i) D(i) (7) where L d and L g are the orreted diret and global omponent images, respetively. Here, α d, α g represent the weights of distributing the predition error D Volume 21, ISSN

5 Journal of WSCG Sene 1 orr. unorr RMSE (d) RMSE (g) `1 -Norm (d) `1 -Norm (g) `2 -Norm (d) `2 -Norm (g) Sene 2 orr. unorr Sene 3 orr. unorr Sene 4 orr. unorr Sene 5 orr. unorr Table 1: Comparison of five test senes with different error metris (root mean square error (RMSE), `1 - and `2 norm): Low resolution images with separated diret(d) and global(g) illumination were saled up by a fator of 2 with the approah of [FF11]. The values of the orreted image an be ompared with the unorreted image for eah sene. a single-lens reflex amera (Nikon D300) was used. Strutured-Light patterns are projeted into the sene that is imaged by the amera. The diret and global illumination for the given sene are separated by the approah of Nayar et al. [NKGR06]. In order to get a reliable ground truth, 25 shifted heker-board patterns were used. to both images on a per-pixel level. Now, we need a ondition how to set the α-values. As the wrong distribution tends to inrease the noise level, a reasonable ondition is to hoose the weights in suh a way that the present high frequenies are minimized. Thus, we an formulate a minimization problem where the sum of the squared Laplaian ( ) of both images is minimized on a per-pixel level: 2 2 (αd (i), αd (i)) = arg min {( Ld ) + ( Lg ) }. αd,αg In Se. 5.1, we evaluate our up-saling sheme. Therefore, we sampled down the ground truth and used these low-resolution images as input. Five test senes were reated for evaluating the influenes of different sene properties, the impat of our orretion for several preditors and the influene of the up-saling fator. Se. 5.2 shows how our up-saling sheme performs on low-resolution diret and global omponent images produed diretly using different strutured-light patterns. We foussed on patterns whih require up to three aquisitions, only. (8) Furthermore, we satisfy the onstrains 0 αd, αg 1 and αd + αg = 1 in order to eliminate the predition error D. Solving for αf = 0 and αfg = 0 with F = d ( Ld )2 + ( Lg )2 yields αd (i) = Ldp (i) Lgp (i), αg (i) =. D(i) D(i) (9) The onstraints an easily be satisfied by normalizing eah αd, αg with the sum αd + αg. Thus, the resulting orretion terms for diret and global omponent images an be expressed as Ld (i) = Ld (i) = Ldp (i) p Ld (i) + Lgp (i) Lgp (i) Ldp (i) + Lgp (i) D(i) + Ldp (i), (10) D(i) + Lgp (i). (11) 5.1 Tab. 1 shows different error values for the test senes and the separated illumination. The error of the orreted image an be ompared with the error of the unorreted image for eah test ase. The error values in sene 1 look similar while the error values in sene 3 are redued by half. Depending on the properties of the sene, the error varies. It is observable that a more detailed sene an be orreted better than a sene with large homogeneous areas. The diret and global omponent images Ld, Lg are then given as input for the predition step of the next iteration until the target resolution is reahed. 5 Influene of the Sene Properties RESULTS In the Figs. 3, 4 and 5, we show the influene of different kinds of luttered regions on our up-saling sheme. We used the approah of Freedman and Fattal [FF11] as the preditor and ompare the predited image with In order to evaluate our up-saling sheme and the lowresolution separation methods, a measurement setup onsisting of a projetor (Panasoni PT-LC76E) and Volume 21, 2013 Evaluation of the Up-Saling Sheme 73 ISSN

6 (a) Ground Truth (b) Predition () Corretion (a) Ground Truth (b) Predition () Corretion (d) Zoom out (e) RMSE (f) RMSE Figure 3: Diret omponent of a stuffed toy in front of a homogeneous bakground. The high frequenies of the fur are orreted more than the low frequenies of the bakground. the results of the proposed orretion sheme. Fig. 3 shows a lose-up of a stuffed toy in front of a homogeneous bakground. As an be seen, our sheme reprodues the fine fur struture more detailed while in the predited image this region looks smeared out. The homogeneous bakground is reprodued well with both methods. In Fig. 4, two piees of halk wrapped in paper in front of a printed box are shown. Here, in ontrast to the stuffed toy, the fine details are not due to fine 3- dimensional strutures. The letters on the box and the printed patterns of the paper wrapping the halk are reonstruted in more detail with our orretion. Fig. 5 shows small wooden figures with small sale painting and thin ornamentations. The edges an be reprodued more preisely with the orretion sheme ompared to the predition. These three examples show that the quality of the orretion is independent of the sene objets. Only the frequenies in the resulting image of the sene objets are responsible for the impat of the improvement. Fine strutures, i.e. high frequenies, in the image lead to a large improvement in quality while areas with low frequenies annot be orreted muh more. Improvement of the Corretion The up-saling of the five exemplary senes was done with three different methods to evaluate the influene of the predition to the orretion. Nearest neighbour interpolation, biubi interpolation and loal selfexamples from [FF11] were used as preditors for the up-saling sheme. Fig. 6 shows a omparison between the RMSE for the different predition methods and the afterwards orreted images. It an be seen that the orretion always improves the predited image, but depending on the quality of the predition the orretion (d) Zoom out (e) RMSE (f) RMSE Figure 4: Global omponent of two piees of halk wrapped in paper in front of a printed box. Independent of the 3D objet struture, the high frequenies are orreted muh better. (a) Ground Truth (b) Predition () Corretion (d) Zoom out (e) RMSE (f) RMSE Figure 5: Diret omponent of small wooden figures in front of a printed box. The orretion between different objets is orreted very good. High frequenies are orreted independently of the objets. annot ompensate the larger error. For example, the orretion of the nearest neighbour interpolation, whih has the largest error in all senes, improves the error but still has a larger absolute error than a better predition without a orretion for the global illumination in sene 3 and 5. An additional onlusion is that our proposed orretion improves prospetive preditions that will have less errors. Please note that the loal self-examples approah produes a higher predition error than biubi interpolation for senes 2 to 5. In ontrast to sene 1, these senes ontain luttered regions. As stated by Freedman and Fattal, their approah is not able of handling these kind of image region and the algorithm spuriously reonstruts edges there [FF11]. This behaviour auses larger error than biubi interpolation with its smoothing ability. Volume 21, ISSN

7 a b S. 1 a b b a a b Diret Illumination a b a orreted predition unorreted predition S. 2 S. 3 S. 4 S. 5 S. 1 S. 2 S. 3 S. 4 S. 5 b a b b a a b Global Illumination Figure 6: The sorted RMSE for (a) biubi interpolation, (b) Loal Self-Examples [FF11] and () nearest neighbour interpolation as preditors shows that the quality of the predition influenes the quality of the orretion. The orretion improves the quality of the predition for diret and global illumination in all five senes. a b x 3x 2x 4x 3x 2x 4x 3x 2x 4x 3x 2x error differene orretion error 4x 3x 2x Sene 1 Sene 2 Sene 3 Sene 4 Sene 5 Figure 7: The dependeny of the up-saling fators is shown. For eah sene the fators 4x, 3x, 2x are ompared (left to right bar). A higher saling fator leads to a higher error differene between predition and orretion. The global illumination images of the five test senes are upsaled using [FF11]. Influene of the Up-Saling Fator The up-saling fator is another parameter of influene. Fig. 7 ompares the RMSE for the saling fators of 4x, 3x and 2x from left to right for eah global illumination image of the five senes. The single-image up-saling method of [FF11] is used as preditor for this omparison. The absolute error introdued by the predition is higher for larger saling. But, it an be seen that the improvement of the reonstrution error by our orretion sheme gets better for higher saling fators in all senes. As our algorithm works in an iterative manner, for higher up-saling fators, more refinement steps are done. As in eah step the predited image is orreted we are able to better guide the up-saling proess. 5.2 Evaluation of Patterns We used different strutured-light patterns for the separation of the illumination omponent in order to evaluate their appliability regarding our up-saling sheme. As the fous is on low aquisition time, only patterns were hosen where up to three aquisition (inluding the fully illuminated image) are needed. Thus, we tested vertial stripes, horizontal stripes, a ombination of both stripe patterns and a high-frequeny hekerboard pattern where eah square had the same dimensions as the width of the stripes. The single-image separation approah proposed by Nayar et al. [NKGR06] was used in order to determine the low-resolution diret and global image omponents (f. left part of Fig. 2). As for the stripes patterns a blurring of edges within the diretion of the stripes our, we ombined the separation results of both patterns in order to ompensate for this effet. However, for this proedure an additional aquisition is required. Therefore, we introdued the high-frequeny heker-board pattern. Here, the same separation methods were applied as for horizontal Volume 21, ISSN

8 Journal of WSCG (a) Global Cheker Board (ground truth) (b) Global Stripes (up-saled) () Global Stripes (low resolution) Figure 8: Comparison of the separation methods using heker-board (a) and stripes strutured-light patterns (b). As an be seen, the stripes separation () suffers severe artefats. Our up-saling sheme is not apable of properly reonstruting the high-resolution images. vert. hor. omb. vert. hor. omb. down-sampled stripes stripes stripes heker heker heker ground truth RMSE (LSE) `1 -Norm (LSE) `2 -Norm (LSE) RMSE (BC) `1 -Norm (BC) `2 -Norm (BC) Table 2: Different kind of patterns were used for a fast separation. The resulting images were fed into our upsaling sheme with loal self-examples (LSE) and biubi interpolation (BC) as preditor. 6 and vertial stripes. In order to improve blurring artefats, the separation results were ombined, also. Tab. 2 shows the reonstrution error of our up-saling sheme for eah of these separation methods ompared to the down-sample ground truth. As preditors we used biubi interpolation and loal self-examples [FF11]. Our experiments show that the reonstrution results differ depending on whether the down-sampled ground truth or the low-resolution images resulting from the stripes or heker-board patterns are used. As an be seen in Fig. 8, the stripes-pattern separation suffers severe reonstrution artefats. Besides the wrong olor reprodution in homogeneous regions, at diagonal edges stair-ase artefats are present. Furthermore, luttered regions are subjeted to noise. Although our up-saling sheme is apable of reduing the artefats in brighter image regions, it annot reover the fine details. In general, the up-saling error for the stripes patterns is smaller as for the heker-board. The hoie of the preditor does not influene the up-saling result signifiantly. Also, the ombination of horizontal and vertial stripes ould not improve the quality of the input images. By monitoring the weights for the predition error distribution αd, αg, we realized that they were lose to 0.5 for all pixels of the diret and global images. This is no surprise, as the noise in both images annot be redued signifiantly by distributing the predition error. Volume 21, 2013 CONCLUSIONS We presented a new onstrained up-saling approah for diret an global omponent images. Preditions based on single-image up-saling tehniques are orreted using a high resolution fully illuminated image. It an be seen that our method an visually improve luttered regions of an image suh as fur or fine strutured ornamentations. The experiments showed that our algorithm sales with the quality of the predition as even well predited images an be improved further. The iterative manner of the method is also apable of improving single-image up-saling approahes for large up-saling fators. Our method is appliable when the aquisition full resolution images using many strutured-light patterns is not possible due to limited time-onstraints. This is the ase for apturing dynami senes in Computer Vision tasks, suh as biometry of human faes or mobile autonomous navigation. Here, only two images are required. However, by aquiring more than one image our method is prone to motion artefats and misalignment of the two images. Thus, it might be neessary to apply optial flow or motion ompensation. Also, our up-saling sheme poses additional omputational omplexity whih might not be suitable for real-time analysis of the aquired data. 76 ISSN

9 Furthermore, when using strutured-light patterns or variations known in literature for fast aquisition, the quality of the resulting low-resolution images is not suffiient. Obviously, these separation methods are strongly influened by wrong olor reprodution and stair-ase artefats. Thus, these kind of patterns produe fundamentally different diret and global omponent images than methods whih work diretly at the resolution of the aquired images. In future work, we study the impat of different light patterns on the separation results. This will give us useful insight about how to solve the above mentioned disrepanies between low-resolution images and groundtruth data. 7 ACKNOWLEDGMENTS This work was funded by the German Researh Foundation (DFG) as part of the researh training group GRK 1564 Imaging New Modalities. 8 REFERENCES [AB12] L. An and B. Bhanu. Image superresolution by extreme learning mahine. In Pro. IEEE Int. Conf. Image Proessing (ICIP), pages , [ABA01] C. Atkins, C. Bouman, and J. Allebah. Optimal image saling using pixel lassifiation. In Pro. Int. Conf. Image Proessing, volume 3, pages , [BGS02] S. Battiato, G. Gallo, and F. Stano. A loally adaptive zooming algorithm for digital images. Image and Vision Computing, 20(11): , [CHL05] M.-J. Chen, C.-H. Huang, and W.-L. Lee. A fast edge-oriented algorithm for image interpolation. Image and Vision Computing, 23(9): , [EV07] M. Ebrahimi and E. Vrsay. Solving the inverse problem of image zooming using self-examples. In M. Kamel and A. Campilho, editors, Image Analysis and Reognition, volume 4633 of Leture Notes in Computer Siene, pages Springer Berlin Heidelberg, [Fat07] R. Fattal. Image upsampling via imposed edge statistis. ACM Trans. Graph., 26(3), [FF11] G. Freedman and R. Fattal. Image and video upsaling from loal self-examples. ACM Trans. Graph., 30(2):12:1 12:11, [FJP02] W. Freeman, T. Jones, and E. Pasztor. Example-based super-resolution. IEEE Computer Graphis and Appliations, 22(2):56 65, [GA08] A. Giahetti and N. Asuni. Fast artifat free image interpolation. In Pro. BMVC 2008, [GA11] A. Giahetti and N. Asuni. Real-time artifat-free image upsaling. IEEE Trans. Image Proessing, 20(10): , [GADTH97] M. Gharavi-Alkhansari, R. DeNardo, Y. Tenda, and T. S. Huang. Resolution enhanement of images using fratal oding. Pro. SPIE 3024, Visual Communiations and Image Proessing, 3024: , [GBI09] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Pro. IEEE Int. Conf. Computer Vision, pages , [GKGN11] J. Gu, T. Kobayashi, M. Gupta, and S. K. Nayar. Multiplexed illumination for sene reovery in the presene of global illumination. In Pro. IEEE Int. Conf. Computer Vision (ICCV), pages 1 8, [GL12] J. Gu and C. Liu. Disriminative illumination: Per-pixel lassifiation of raw materials based on optimal projetions of spetral brdf. In Pro. IEEE Conf. Computer Vision and Pattern Reognition (CVPR), pages , [KK08] K. Kim and Y. Kwon. Examplebased learning for single-image superresolution. In G. Rigoll, editor, Pattern Reognition, volume 5096 of Leture Notes in Computer Siene, pages Springer Berlin Heidelberg, [LHL11] B. Langmann, K. Hartmann, and O. Lof- [LHL12] [LLK08] feld. Comparison of depth superresolution methods for 2d/3d images. Int. Journal of Computer Information Systems and Industrial Management Appliations, 3: , B. Langmann, K. Hartmann, and O. Loffeld. A modular framework for 2d/3d and multi-modal segmentation with joint super-resolution. In A. Fusiello, V. Murino, and R. Cuhiara, editors, Computer Vision - ECCV Workshops and Demonstrations, volume 7584 of Leture Notes in Computer Siene, pages Springer Berlin Heidelberg, M. Lindner, M. Lambers, and A. Kolb. Volume 21, ISSN

10 Data fusion and edge-enhaned distane refinement for 2d rgb and 3d range images. Int. J. on Intell. Systems and Tehn. and App. (IJISTA), Issue on Dynami 3D Imaging, 5(1): , [LO01] X. Li and M. Orhard. New edgedireted interpolation. IEEE Trans. Image Proessing, 10(10): , [MYR10] Y. Mukaigawa, Y. Yagi, and R. Raskar. Analysis of light transport in sattering media. In Pro. IEEE Conf. Computer Vision and Pattern Reognition (CVPR), pages , [NG12] S. K. Nayar and M. Gupta. Diffuse strutured light. In Pro. IEEE Int. Conf. Computational Photography, pages 1 8, [NKGR06] S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar. Fast separation of diret and global omponents of a sene using high frequeny illumination. ACM Trans. Graph., 25(3): , [ORK12] M. O Toole, R. Raskar, and K. N. Kutulakos. Primal-dual oding to probe light transport. ACM Trans. Graph., 31:39:1 39:11, [RRC12] D. Reddy, R. Ramamoorthi, and B. Curless. Frequeny-spae deomposition and aquisition of light transport under spatially varying illumination. In Computer Vision - ECCV Springer Berlin Heidelberg, [SSU08] N. Suetake, M. Sakano, and E. Uhino. Image super-resolution based on loal self-similarity. Optial Review, 15:26 30, [STDT08] S. Shuon, C. Theobalt, J. Davis, and S. Thrun. High-quality sanning using time-of-flight depth superresolution. In Pro. IEEE Conf. on Comp. Vis. & Patt. Reogn.; Workshops (CVPRW), pages 1 7, [SW04] D. Su and P. Willis. Image interpolation by pixel-level data-dependent triangulation. Computer Graphis Forum, 23(2): , [SXS08] J. Sun, Z. Xu, and H.-Y. Shum. Image super-resolution using gradient profile prior. In Pro. IEEE Conf. Computer Vision and Pattern Reognition (CVPR), pages 1 8, [TM96] S. Thurnhofer and S. K. Mitra. Edgeenhaned image zooming. Optial Engi- [TTT06] [WXM + 13] [YWL + 12] [YXXY12] [YXYC12] [ZDLL12] neering, 35(7): , Y.-W. Tai, W.-S. Tong, and C.-K. Tang. Pereptually-inspired and edge-direted olor image super-resolution. In Pro. IEEE Conf. Computer Vision and Pattern Reognition (CVPR), volume 2, pages , L. Wang, S. Xiang, G. Meng, H.-y. Wu, and C. Pan. Edge-direted single image super-resolution via adaptive gradient magnitude self-interpolation. IEEE Trans. Ciruits and Systems for Video Tehnology, PP(99):1, J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang. Coupled ditionary training for image super-resolution. IEEE Trans. Image Proessing, 21(8): , L. Yu, H. Xu, Y. Xu, and X. Yang. Robust single image super-resolution based on gradient enhanement. In Pro. Signal Information Proessing Assoiation Annual Summit and Conferene (APSIPA ASC), pages 1 6, Q. Yan, Y. Xu, X. Yang, and K. Chen. Image super-resolution based on a novel edge sharpness prior. In Pro. Int. Conf. Pattern Reognition (ICPR), pages , L. Zi, J. Du, M. Liang, and J. Lee. A pereption-motivated image interpolation algorithm. In Pro. World Congress on Intelligent Control and Automation (WCICA), pages , Volume 21, ISSN