Consistent Illumination within Optical See-Through Augmented Environments

Transcription

1 Consstent Illumnaton wthn Optcal See-Through Augmented Envronments Olver Bmber, Anselm Grundhöfer, Gordon Wetzsten and Sebastan Knödel Bauhaus Unversty Bauhausstraße 11, Wemar, Germany, {olver.bmber, anselm.grundhoefer, gordon.wetzsten, Abstract We present technques whch create a consstent llumnaton between real and vrtual objects nsde an applcaton specfc optcal see-through dsplay: the Vrtual Showcase. We use projectors and cameras to capture reflectance nformaton from dffuse real objects and to llumnate them under new synthetc lghtng condtons. Matchng drect and ndrect lghtng effects, such as shadng, shadows, reflectons and color bleedng can be approxmated at nteractve rates n such a controlled mxed envronment. llumnaton of real and vrtual components easer to acheve than n real world envronments. 1. Introducton To acheve a consstent lghtng stuaton between real and vrtual envronments s mportant for convncng augmented realty (AR) applcatons. A rch pallet of algorthms and technques have been developed that match llumnaton for vdeo- or magebased augmented realty. However, very lttle work has been done n ths area for optcal see-through AR. For the reasons that are dscussed n [1], we beleve that the optcal see-through concept s currently the most advanced technologcal approach to provde an acceptable level of realsm and nteractvty. The Vrtual Showcase [2] s an applcaton specfc optcal see-through dsplay. Fgure 1 llustrates our latest prototype. It conssts of up to four tlted CRT screens that are reflected by a pyramd-shaped mrror beam spltter. Wreless nfrared trackng determnes the observers perspectves to render hgh-resoluton 1 stereoscopc graphcs onto the screens. Vdeo projectors are mounted under ts roof and allow a pxel-precse llumnaton of the real content [3]. Between two and three networked offthe-shelf PCs that are ntegrated nto the Vrtual Showcase s frame are used to drve the dsplay. Besde ts hgh resoluton, the Vrtual Showcase also provdes a dark and well controllable envronment that makes a consstent Fgure 1: Vrtual Showcase prototype wth cameras and projectors. The contrbuton of ths paper s the ntroducton of methods whch create a consstent llumnaton between real and vrtual components wthn an optcal see-through envronment such as the Vrtual Showcase. Combnatons of vdeo projectors and cameras are appled to capture reflectance nformaton from dffuse real objects and to llumnate them under new synthetc lghtng condtons. For dffuse objects, the capturng process can also beneft from hardware acceleraton supportng dynamc update rates. To handle ndrect lghtng effects (lke color bleedng) an off-lne radosty procedure s outlned that conssts of multple renderng passes. For drect lghtng effects (such as smple shadng, shadows and reflectons) hardware accelerated technques are descrbed whch allow to acheve nteractve frame rates. The reflectance nformaton s used n addton to solve a man problem of a prevously ntroduced technque whch creates consstent occluson effects for multple users wthn such envronments [3]. 1 Currently UXGA per user.

2 2. Related Work Inspred by the poneerng work of Nakamae et al. [16] and later Fourner et al. [9], many researchers have approached to create consstent llumnaton effects whle ntegratng synthetc objects nto a real envronment. To our knowledge, all of these approaches represent the real envronment n form of mages or vdeos. Consequently, manly mage processng, nverse renderng, nverse global llumnaton, mage-based and photo-realstc renderng technques are appled to solve ths problem. Due to the lack of real-tme processng, these approaches are only applcable n combnaton wth desktop screens and an unresponsve 2 user nteracton. Devces that requre nteractve frame-rates, such as head-tracked personal or spatal dsplays, cannot be supported. Representatve for the large body of lterature that exsts n ths area, we want to dscuss several more recent achevements: Bovn et al. [5] present an nteractve and herarchcal algorthm for reflectance recovery from a sngle mage. They assume that the geometrc model of the scenery and the lghtng condtons wthn the mage are known. Makng assumptons about the scene s photometrc model, a vrtual mage s generated wth global llumnaton technques (.e., ray-tracng and radost. Ths synthetc mage s then compared to the photograph and a photometrc error s estmated. If ths error s too large, ther algorthm wll use a more complex BRDF model (step by step usng dffuse, specular, sotropc, and fnally ansotropc terms) n the followng teratons, untl the devaton between synthetc mage and photograph s satsfactory. Once the reflectance of the real scenery s recovered, vrtual objects can be ntegrated and the scene must be re-rendered. They report that the analyss and re-renderng of the sample mages takes between 30 mnutes to several hours dependng on the qualty requred and the scene s complexty. Yu et al. [22] present a robust teratve approach that uses global llumnaton and nverse global llumnaton technques. They estmate dffuse and specular reflectance, as well as radance and rradance from a sparse set of photographs and the gven geometry model of the real scenery. Ther method s appled to the nserton of vrtual objects, the modfcaton of llumnaton condtons and to the re-renderng of the scenery from novel vewponts. As for Bovn s approach, BRDF recovery and re-renderng are not supported at nteractve frame-rates. Loscos et al. [13] estmate only the dffuse reflectance from a set of photographs wth dfferent but controlled real world llumnaton condtons. They are able to nsert and remove real and vrtual objects and shadows, and to 2 Not real-tme. modfy the lghtng condtons. To provde an nteractve manpulaton of the scenery, they separate the calculaton of the drect and ndrect llumnaton. Whle the drect llumnaton s computed on a per-pxel bass, ndrect llumnaton s generated wth a herarchcal radosty system that s optmzed for dynamc updates [8]. Whle the reflectance analyss s done durng an offlne preprocessng step, nteractve frame-rates can be acheved durng re-renderng. Dependng on the performed task and the complexty of the scenery, they report re-renderng tmes for ther examples between 1-3 seconds on a SGI R Although these results are qute remarkable, the update rates are stll too low to satsfy the hgh response requrements of stereoscopc dsplays that support head-trackng (and possbly multple users). Gbson and Murta [10] present another nteractve mage composton method to merge synthetc objects nto a sngle background photograph of a real envronment. A geometrc model of the real scenery s also assumed to be known. In contrast to the technques descrbed above, ther approach does not consder global llumnaton effects to beneft from hardware accelerated mult-pass renderng. Consequently, a reflectance analyss of the real surfaces s not requred, ndrect llumnaton effects are gnored, and a modfcaton of the lghtng condtons s not possble. The llumnaton of the real envronment s frst captured n form of an omndrectonal mage. Then a seres of hgh dynamc bass radance maps are pre-computed. They are used durng runtme to smulate a matchng drect llumnaton of the synthetc objects usng sphere mappng. Shadow castng between real and vrtual objects s approxmated wth standard shadow mappng. Wth ther method, convncng mages can be rendered at frame rates up to 10fps on an SGI Onyx 2. However, t s restrcted to a statc vewpont. 3. Dffuse Reflectance Analyss We want to assume that the geometry of both object types real and vrtual has been modeled or scanned n advance. Whle the materal propertes of vrtual objects are also defned durng ther modelng process, the dffuse reflectance of physcal objects s captured nsde the Vrtual Showcase wth a set of vdeo projectors and cameras. Ths sort of analyss s standard practce for many range scanner setups. But snce we consder only dffuse real objects (a projector-based llumnaton wll generally fal for any specular surface), our method can beneft from hardware accelerated renderng technques. In contrast to conventonal scannng approaches, ths leads to dynamc update rates. Fgure 2 llustrates an example.

3 3.1. Calbraton The ntrnsc and extrnsc parameters of projectors and cameras wthn the world coordnate system have to be estmated frst. We calbrate each devce separately. As descrbed n [3], we nteractvely mark the twodmensonal projectons of known three-dmensonal fducals on a projector s/camera s mage plane. Usng these mappngs, we apply Powell s drecton set method [18] to solve a perspectve n-pont problem for each devce. Ths results n the perspectve projecton matrces P, C of a projector and a camera. Both matrces ncorporate the correct model-vew transformatons wth respect to the orgn of our world coordnate system. Some types of vdeo projectors (such as dgtal lght projectors, DLPs) dsplay a sngle mage wthn sequental, tme-multplexed lght ntervals to acheve dfferent ntensty levels per color. If such projectors are used, a sngle snapshot of the llumnated scene would capture only a slce of the entre dsplay perod. Consequently, ths mage would contan ncomplete color fragments nstead of a full-color mage. The wdth of ths slce depends on the exposure tme of the camera. To overcome ths problem, and to be ndependent of the camera s exposure capabltes, we capture a sequence of mages over a predefned perod of tme. These mages are then combned and averaged to create the fnal dffuse radance map I rad (cf. fgure 2a) Creatng Intensty Images To extract the dffuse materal reflectance out of I rad the lghtng condtons that have been created by the projector have to be neutralzed. OpenGL s dffuse lghtng component s gven by [17]: 1 (3.1) I = 2 cos(θ )(Dl Dm ) r where I s the fnal ntensty (color) of a vertex, Dl s the dffuse color of the lght, Dm s the dffuse materal property, the angle θ s spanned by the vertex s normal and the drecton vector to the lght source, and the factor 1 / r j2 represents a square dstance attenuaton. Fgure 2: (a) Captured radance map of a fosslzed dnosaur footprnt; (b) Intensty mage rendered for calbrated projector from (a); (c) Computed reflectance map; (d) Novel llumnaton stuaton; (e) Reflectance map under novel llumnaton from (d); (f) Reflectance map under vrtual llumnaton from (b) Capturng Radance Maps A vdeo projector s used to send structured lght samples to the dffuse physcal object and llumnate t wth a predefned color C p and an estmated ntensty η. Synchronzed to the llumnaton, a vdeo camera captures an nput mage. Snce ths mage contans the dffuse reflectance of the object s surface under known lghtng condtons t represents a radance map. Whte-balancng and other dynamc correcton functons have been dsabled n advance. The parameters of the camera s response functon are adjusted manually n such a way that the recorded mages approxmate the real world stuaton as close as possble. Smlar as n [19], an ntensty mage I nt that contans only the dffuse llumnaton can be created by renderng the object s geometry (wth Dm = 1 ) from the perspectve of the vdeo camera, llumnated by a pont lght source (wth Dl = C pη ) that s vrtually located at the poston of the projector (cf. fgure 2b). In addton, hard shadows are added to the ntensty mage by applyng standard shadow mappng technques. Consequently, the background pxels of I nt, as well as pxels of regons that are occluded from the perspectve of the lght source are blanked out ( I nt y ) = 0 ), whle all other pxels are shaded under consderaton of equaton Extractng Reflectance and Re-Renderng Dffuse Gven the captured radance map I rad and the rendered ntensty mage I nt, the dffuse reflectance for each surface that s vsble to the camera can be computed by:

4 I rad I ref = for I nt > 0, I nt I ref = 0 for I nt = 0 (3.2) We store the reflectance mage I ref, together wth the matrx C and the real object s world-transformaton O c that s actve durng the capturng process wthn the same data-structure. We call ths data structure reflectance map (cf. fgure 2c). The captured reflectance map can be re-rendered together wth the real object s geometrc representaton from any perspectve wth an arbtrary worldtransformaton O. Thereby, a I ref s appled as projectve texture map wth the texture matrx 3 1 set to Oa OcC. Enablng texture modulaton, t can then be re-lt vrtually under novel llumnaton condtons (cf. fgures 2d, 2e and 2f) Shortcomngs and Solutons The basc reflectance analyss method as descrbed above faces the followng problems: (1) due to under-samplng, surfaces whch span a large angle φ between ther normal vectors and the drecton vectors to the camera can cause texture artfacts f I s re-mapped from a dfferent ref perspectve; (2) a sngle reflectance map covers only the surface porton that s vsble from the perspectve of the camera; (3) the radance map can contan ndrect llumnaton effects caused by lght fractons that are dffused off other surfaces (so-called secondary scatterng). The ntensty mage I nt, however, does not contan secondary scatterng effects snce a global llumnaton soluton s not computed. Consequently, the extracted reflectance s ncorrect at those areas that are ndrectly llumnated by secondary scatterng; (4) the projector ntensty η has to be estmated correctly; To overcome the under-samplng problem, we defne that only surfaces wth φ φ are analyzed. All other max surfaces wll be blanked-out n I ref (.e., I ref = 0 ). We found that φ = 60 s approprate. Ths corresponds max 3 Includng the correspondng mappng transformaton from normalzed devce coordnates to normalzed texture coordnates. to the fndngs n [19], descrbng a maxmum angle between projector and projecton surface. Multple reflectance maps that cover dfferent surface portons can be captured under varyng transformatons O or C. They are merged and alpha blended durng remappng them va mult-texturng onto the object s c geometrc representaton. Ths ensures that regons whch are blanked out n one reflectance map can be covered by other reflectance maps. To generate seamless transtons between the dfferent texture maps, b- or tr-lnear texture flterng can be enabled. Illumnatng the entre scene can cause an extreme secondary scatterng of the lght. To mnmze the appearance of secondary scatterng n I rad, we dvde the scene nto dscrete peces and capture ther reflectance one after the other. For ths, we can apply the same algorthm as descrbed above. The dfference, however, s that we llumnate and render only one pece at a tme whch then appears n I rad and I nt. By evaluatng the blanked out background nformaton provded n I nt, we can effectvely segment the selected pece n I nt and compute ts reflectance. Ths s repeated for each frontfacng pece, untl I s complete. ref We estmate the projector s ntensty η as follows: Frst, we generate a reflectance map wth an ntal guess of η. Ths reflectance map s then re-mapped onto the object s geometrc representaton, whch s rendered from the perspectve of the camera and llumnated by a vrtual pont lght source wth η located at the projector. The rendered radance map I radv s then compared to the captured radance map I rad by determnng the average square dstance error among all correspondng pxels. Fnally, we fnd an approxmaton for η by mnmzng the error functon f. For ths we apply Brent s nverse parabolc mnmzaton method wth bracketng [7]. By estmatng η, we can also ncorporate the constant blacklevel of the projector. 4. Augmented Radosty In computer graphcs, the radosty method [11] s used to approxmate a global llumnaton soluton by solvng an energy-flow equaton. Indrect llumnaton effects, such as secondary scatterng can be smulated wth radosty. The general radosty equaton for n surface patches s gven by: n B = E + ρ B F (4.1) j = 1 where B s the radance of surface, E s the emtted energy per unt area of surface, ρ s the reflectance of j j

5 surface, and F represents the fracton of energy that s j exchanged between surface and surface j (the formfactor). The smulaton of radosty effects wthn an optcal see-through envronment that conssts of dffuse physcal and vrtual objects, s facng the followng challenges and problems: (1) lght energy has to flow between all surfaces real ones and vrtual ones; (2) physcal objects are llumnated wth physcal lght sources (.e., vdeo projectors n our case) whch do not share the geometrc and radometrc propertes of the vrtual lght sources; (3) no physcal lght energy flows from vrtual objects to real ones (and vce versa). Consequently, the llumnated physcal envronment causes (due to the absence of the vrtual objects) a dfferent radometrc behavor than the entre envronment (.e., real and vrtual objects together). 0 envronment the radance values B for all surfaces have been computed 6. Color-bleedng and shadow-castng are clearly vsble. 4.1 Vrtual Objects For vrtual objects, the computed radance values are already correct (cf. fgure 3d). The rendered mage represents a radance map that s generated from one specfc perspectve. Renderng the vrtual objects from multple perspectves results n multple radance maps that can be merged and alpha blended durng re-mappng them va mult-texturng onto the vrtual geometry (as descrbed for reflectance maps n secton 3.5). In ths case, our radance maps are equvalent to lght maps that are often beng appled durng pre-lghtng steps to speed up the onlne renderng process. The pre-lt vrtual objects can smply be rendered together wth ther lght maps and can be optcally overlad over the physcal envronment. 4.2 Physcal Objects Fgure 3: Mult-Pass Radosty. An example s llustrated n fgure 3a 4. The entre envronment conssts of three walls, a floor, two boxes and a surface lght source on the celng. We want to assume that the walls and the floor are the geometrc representatons of the physcal envronment, and the boxes as well as the lght source belong to the vrtual envronment. Whle the dffuse reflectance ρ of the physcal envronment can be automatcally captured (as descrbed n secton 3), t has to be defned for the vrtual envronment. After a radosty smulaton 5 of the entre 4 We have chosen a physcal mock-up of the Cornell room snce t s used n many other publcatons as a reference to evaluate radosty technques. 5 We appled a hem-cube-based radosty mplementaton wth progressve refnement, adaptve subdvson and nterpolated renderng for our smulatons. The physcal surfaces, however, have to emt the energy that was computed n B 0 (cf. fgure 3b). To approxmate ths, we frst assume that every physcal 0 surface patch drectly emts an energy E that s equvalent to B 0. If ths s the case, fractons of ths energy wll radate to other surfaces and llumnate them n addton. Ths can be smulated by a second radostypass (cf. fgure 3c), whch computes new reflectance 1 values B for all the physcal surfaces, by assumng that 0 0 E = B, and not consderng the drect nfluence of the vrtual lght source. If we subtract the radance values that have been computed n both passes we receve the scattered lght only. That s, the lght energy radated between the 1 0 physcal surfaces B B (cf. fgure 3h). Consequently, B = B ( B B ) (4.2) approxmates the energy that has to be created physcally on every real surface patch. To prove ths we can apply a thrd radosty pass to smulate the energy flow between the patches (cf. fgure 3f). We can see that the remanng 1 0 energy B wll be nearly added, and we receve: B ( B B ) B 3 2 B = B + (4.3) By removng the vrtual objects from the envronment and smulatng the second radosty pass, lght energy wll also be radated onto surfaces whch were orgnally blocked or covered by the vrtual objects (ether 6 Note, that the upper ndex represents the radosty pass.

6 completely or partall. Examples are the shadow areas that have been cast by the vrtual objects. Ths can be observed n fgure 3h and fgure 3. Consequently, negatve radance values are possble for such areas after applyng equaton 4.2. To avod ths, the resultng values have to be clpped to a vald range. 0 1 The average devatons between B and B, as well as 0 3 between B and B, wthn the three spectral samples red (R), green (G), and blue (B) are presented below fgures 3h and 3, respectvely. Treatng a vdeo projector as a pont lght source B 2 can be expressed as a smplfed verson of equaton 4.1: B 2 = ρl F (4.4) where L s the rradance that has to be projected onto surface by the projector, and F s the form-factor for surface, whch s gven by: cos(θ ) F = h (4.5) 2 r where θ s the angle between a surface patch s normal and the drecton vector to the projector, r s the dstance between a surface patch and the projector, and h s the vsblty term of the surface patch, seen from the projector s perspectve. Extendng and solvng equaton 4.4 for L, we receve (cf. fgure 3g): 2 B L = η (4.6) ρ F To cope wth the ndvdual brghtness of a vdeo projector, we add the ntensty factor η. How to estmate η for a specfc projector was descrbed n secton 3.5. To be consstent wth our prevously used termnology, we call L the rradance map. 4.3 Lmtatons The computed radance and rradance values are vewndependent. Consequently, rradance maps for the real objects and radance maps for the vrtual objects can be pre-computed offlne. The real objects are llumnated wth projected lght durng runtme by renderng the generated rradance map from the vewpont of the projector (e.g., as llustrated n fgure 3g). Vrtual objects are rendered wth the computed lght maps (e.g., as llustrated n fgure 3d) and are then optcally overlad over the real envronment. Due to the vew-ndependence of the method, the augmented scene can be observed from any perspectve (.e., head-trackng and stereoscopc renderng are possble). However, the scene has to reman statc, snce any modfcaton would requre to re-compute new radance and rradance maps throughout multple radosty passes. Ths s not yet possble at nteractve rates. Fgure 4 shows a photograph of (a) the physcal object under room llumnaton, (b) a screen-shot of captured reflectance maps that have been re-rendered under novel lghtng condtons, (c) a screen-shot of the smulated 0 radance stuaton B, and (d) a photograph of a physcal object that has been llumnated wth L. Note, that small devatons between the mages can be contrbuted to the responds of the dgtal camera that was used to take the photograph, as well as to the hgh black-level of the projector that, for nstance, makes t mpossble to create completely black shadow areas. (a) (b) (c) (d) Fgure 4: (a) Photograph of orgnal object under room llumnaton; (b) Screen-shot of captured reflectance re-lt wth vrtual pont lght source and Phong shadng; (c) Screen-shot of smulated radosty soluton wth captured reflectance, vrtual surface lght source (shown n fgure 3), and two vrtual objects (show n fgure 3); (d) Photograph of orgnal object llumnated wth the computed rradance. 5. Interactve Approxmatons In the followng secton we descrbe several nteractve renderng methods that make use of hardware acceleraton. In partcular we dscuss how to create matchng shadng, shadow and reflecton effects on real and vrtual objects. Indrect lghtng effects such as colorbleedng, however, cannot be created wth these technques. Yet, they create acceptable results at nteractve frame rates for multple head-tracked users and stereoscopc vewng on conventonal PCs.

7 5.1 Shadng The generaton of drect llumnaton effects on vrtual surfaces caused by vrtual lght sources s a standard task of today s hardware accelerated computer graphcs technology. Real-tme algorthms, such as Gouraud shadng or Phong shadng are often mplemented on graphcs boards. Consstent and matchng shadng effects on real surfaces from vrtual lght sources can be acheved by usng vdeo projectors that project approprate rradance maps onto the real objects. Raskar et al. [19] show how to compute an rradance map to lft the radance propertes of neutral dffuse objects wth unform whte surfaces nto a pre-computed radance map of a vrtual scene llumnated by vrtual lght sources. An rradance map that creates vrtual llumnaton effects on dffuse real objects wth arbtrary reflectance propertes (color and texture) can be computed as follows: Frst, the real objects captured reflectance map ( I ref ) s rendered from the vewpont of the projector and s shaded wth all vrtual lght sources n the scene. Ths results n the radance map I. Then rad _ I 1 ref s rendered agan from the vewpont of the projector. Ths tme, however, t s llumnated by a sngle pont lght source (wth D l = 1 η ) whch s located at the poston of the projector. Ths results n the radance map I. Fnally, the correct rradance map s computed by: I rad _1 L = (5.1) I Note that equaton 5.1 correlates to equaton 4.6. The dfference s the appled llumnaton model. Whle equaton 4.6 s dscussed wth respect to an ndrect global llumnaton model (radost, equaton 5.1 apples hardware accelerated drect models (such as Phong or Gouraud shadng). It s easy to see that I s the rad _ 1 2 opponent to B and that I corresponds to ρ Fη. Note also that ths method s actually completely ndependent of the real objects reflectance. Ths can be shown by equalzng I wth rad _ I through equaton In ths case the dffuse materal property D (.e., the m reflectance) s canceled out. Consequently, I and rad _ 1 I can be rendered wth a constant (but equal) reflectance ( D ). If we choose m D m = 1 then the rradance map s smply the quotent between the two ntensty mages I and nt_ 1 I that result from the two dfferent nt_ 2 lghtng condtons the vrtual one and the real one. The rradance map L should also contan consstent shadow nformaton. How to acheve ths s outlned below. Fgure 5 llustrates examples wth matchng shadng effects Shadows We can dentfy sx types of shadows wthn an optcal see-through envronment: (1) shadows on real objects created by real objects and real lght sources; (2) shadows on vrtual objects created by vrtual objects and vrtual lght sources; (3) shadows on vrtual objects created by real objects and vrtual lght sources; (4) shadows on real objects created by real objects and vrtual lght sources; (5) shadows on real objects created by vrtual objects and vrtual lght sources; (6) occluson shadows; (a) (b) (c) (d) Fgure 5: (a) Unrealstc unform llumnaton wth shadow type 6 (the wooden plate s real, the dragon and the dnosaur skull are vrtual); (b)-(d) Realstc llumnaton under varyng vrtual lghtng condtons wth matchng shadng and shadows (types 2,3,5, and 6). The frst type of shadow s the result of occlusons and self-occlusons of the physcal envronment that s llumnated by a physcal lght source (e.g., a vdeo projector). Snce we focus on controllng the llumnaton condtons wthn the entre envronment va vrtual lght sources we have to remove these shadows. Ths can be acheved by usng multple synchronzed projectors that are able to llumnate all vsble real surfaces. Several technques have been descrbed whch compute a correct color and ntensty blendng for mult-projector dsplays [14, 19, 21]. 7 Note that we have chosen a smple wooden plate to demonstrate and to compare the dfferent effects. However, all technques that are explaned n ths paper can be appled to arbtrary object shapes.

8 The second and thrd shadow types can be created wth standard shadow mappng or shadow bufferng technques. To cast shadows from real objects onto vrtual ones, the regstered geometrc representatons of the real objects have to be rendered together wth the vrtual objects when the shadow map s created (.e., durng the frst shadow pass). Such geometrc real world representatons (sometmes called phantoms [6]) are often rendered contnuously to generate a realstc occluson of vrtual objects by real ones. Note that these hardware accelerated technques create hard shadows whle global llumnaton methods (such as radost can create soft shadows. Texture blendng, however, allows ambent lght to be added to the shadow regons. Ths results n dark shadow regons that are blended wth the underlyng surface texture, nstead of creatng unrealstc black shadows. Shadow types number 4 and 5 can also be created va shadow mappng. However, they are projected on the surface of the real object together wth the rradance map L, as dscussed n secton 5.1. Therefore, I has to rad _ 1 contan the black (non-blended) shadows of the vrtual and the real objects. Ths s acheved by renderng all vrtual objects and all phantoms durng the frst shadow pass to create a shadow map. Durng the second pass the shaded reflectance texture and the generated shadow texture are blended and mapped onto the objects phantoms. A dvson of the black shadow regons by I preserves these regons. Note that a blendng of the projected shadows wth the texture of the real objects occurs physcally f the correspondng surface portons are llumnated (e.g., by a relatvely small amount of projected ambent lght). Occluson shadows [3] are specal vew-dependent shadows created by the projectors on the real objects surfaces. We have ntroduced them to acheve a realstc occluson of real objects by vrtual ones. They are normally not vsble from the perspectve of the observer, snce they are dsplayed exactly underneath the graphcal overlays. Occluson shadow-maps, however, have also to be blended to the rradance map L before t s projected. 5.3 Reflectons Usng hardware accelerated cube mappng technques, the vrtual representaton of the real envronment (.e., the objects geometry together wth the correctly llumnated reflectance map) can be reflected by vrtual objects (cf. fgure 6). Therefore, only the regstered vrtual representaton of the real envronment has to be rendered durng the generaton step of the cube map. Vrtual objects are then smply rendered wth cube mappng enabled. Note, that for conventonal cube mappng, reflecton effects on a vrtual object are physcally correct for only a sngle pont the center of the cube map frusta. To create convncng approxmatons ths center has to be matched wth the vrtual object s center of gravty, and the cube map has to be updated every tme the scene changes. (a) (b) Fgure 6: (a) A vrtual sphere and (b) a vrtual torus reflectng and occludng the real object (wooden plate). 5.4 Occludng Occluson Shadows The occluson shadow method [3] s currently one of two functonng solutons that can create consstent occlusons effects for optcal see-through dsplays. A man drawback of ths approach s ts lmted support for multple users: If the same real surfaces are smultaneously vsble from multple ponts of vew (as t s the case for dfferent observers), ndvdual occluson shadows that project onto these surfaces are also vsble from dfferent vewponts at the same tme (cf. fgure 7a). (a) (b) Fgure 7: (a) Occluson shadow of second observer s clearly vsble; (b) Wrongly vsble occluson shadow s covered by optcally overlayng the correspondng part of the reflectance map. Although two dfferent approaches have been presented n [3] that reduce the effects of ths problem, t s not completely solved for any type of surface. Knowng the reflectance nformaton of the real surfaces, however, leads to an effectve and general soluton: As descrbed n sectons 5.1 and 5.2, the real objects are llumnated by projected lght (also contanng occluson shadows for all observers) and the vrtual objects are shaded, rendered and optcally overlad over the real scene (on top of each observer s occluson shadows). In addton to the vrtual scene, we render the portons of the real scene (.e., ts regstered reflectance

9 map) that are covered by the occluson shadows of all other observers. Remember that these reflectance-map portons are llumnated and shaded under the same lghtng condtons as ther real counterparts (outlned n sectons 5.1 and 5.2). Ths creates seamless transtons between the real and the vrtual parts. For each observer the occluson shadows of all other observers are rendered nto the stencl buffer frst. Ths s done by renderng the real scene s geometry from each observer s perspectve and addng the correspondng occluson shadows va projectve texture mappng, as descrbed n [3]. The stencl buffer has to be flled n such a way that the area surroundng the occluson shadows wll be blanked out n the fnal mage. Then the real scene s reflectance map s rendered nto the frame buffer (also from the perspectve of the observer) and s shaded under the vrtual lghtng stuaton. After stenclng has been dsabled, the vrtual objects can be added to the observer s vew (cf. fgure 7b). 6. Summary and Future Work The overall goal of ths work s to enhance realsm for optcal see-through AR envronments. To acheve ths we have develop technques whch allow creatng consstent llumnaton effects between real and vrtual objects. We have mplemented and demonstrated these technques based on the Vrtual Showcase, snce ths dsplay provdes a well controllable envronment. We use vdeo projectors and cameras as essental components of the Vrtual Showcase. They allow to retreve nformaton out of the Vrtual Showcase s nsde at dynamc update rates, and to llumnate real objects on a per-pxel bass n real tme. Currently only reflectance data s scanned from real objects. In the future, other surface nformaton, such as geometry and external emsson can be measured as well. The scanned geometry nformaton wll lead to the development of an automatc regstraton procedure for real objects. Usng the reflectance nformaton, Augmented Radosty has been descrbed as a global llumnaton technque for optcal see-through devces that s able to create a hgh level of realsm. Only statc augmented envronments can be created wth ths method. However, due to ts vew-ndependency, head-trackng and stereoscopc renderng s possble. To evaluate nteractve global llumnaton methods n combnaton wth ths technque belongs to our lst of future tasks. To reach nteractve renderng frame rates, hardware accelerated methods have been used to generate convncng approxmatons of a consstently and realstcally llumnated augmented envronment. Throughout several renderng passes shadng, shadow mappng and cube mappng technques have been appled n combnaton wth the captured reflectance nformaton to acheve ths. Instead of capturng the reflectance map of a real object and computng a radance map, ts physcal radance can be captured drectly after the synthetc llumnaton nformaton (.e., shadng and shadows, as descrbed n secton 5.1) have been created on ts surface. Ths radance map can then be appled n combnaton wth the reflecton and occluson shadow technques descrbed n sectons 5.3 and 5.4. The advantage of ths approach s that the llumnaton nformaton on the real object has to be computed only once before t s projected onto ts surface. The resultng radance map can smply be captured wth the vdeo camera and contans all mportant nformaton of the real envronment, such as reflectance, shadng and shadows. The occludng occluson shadow technque allows to solve the mult-user lmtaton that s lnked to the orgnal occluson shadow dea. In addton, t allows to make seamless transtons on the mxed realty contnuum [15], that are mportant for applcatons of the Vrtual Showcase, such as dgtal storytellng [4]. The bggest problem of ths extenson, however, s a slght color nconsstency between the real object and the vrtual overlay. Ths results from photometrc devatons between the CRT screens and the vdeo cameras wth the observers vsual percepton of the real object. Currently we adjust ths manually by modfyng the physcal and the synthetc llumnaton untl the real object and the vrtual overlay appear to concde vsually. Automatc calbraton technques need to be developed n the future. To reduce geometrc msalgnments caused by small regstraton errors, the edges of the vrtual overlay can be blurred n addton. A next mportant step towards realsm wll be to vsually enhance the vrtual components whle retanng nteractve frame rates. Advanced renderng technques, such as lght felds [12, 20] and hardware-accelerated procedural shadng technology mght allow blurrng the boundares between real and vrtual even further. Acknowledgements We thank Werner Purgathofer for frutful dscussons on the occludng occluson shadows topc. The Vrtual Showcase project s supported by the European Unon, IST References [1] Azuma, R., Ballot, Y., Behrnger, R., Fener, S., Juler, S., MacIntyre, B. Recent Advances n Augmented Realty. IEEE Computer Graphcs & Applcatons, vol. 21, no.6, pp , 2001.

10 [2] Bmber, O., Fröhlch, B., Schmalsteg, D., and Encarnação, L.M. The Vrtual Showcase. IEEE Computer Graphcs & Applcatons, vol. 21, no.6, pp , [3] Bmber, O, and Fröhlch, B. Occluson Shadows: Usng Projected Lght to Generate Realstc Occluson Effects for Vew-Dependent Optcal See-Through Dsplays. In proceedngs of ACM/IEEE Internatonal Symposum on Mxed and Augmented Realty (ISMAR 02), pp , [4] Bmber, O., Encarnação, L.M, and Schmalsteg, D. The Vrtual Showcase as a new Platform for Augmented Realty Dgtal Storytellng. In proceedngs of IPT/EGVE 2003 workshop, pp , [5] Bovn, S. and Gagalowcz, A. Image-based renderng of dffuse, specular and glossy surfaces from a sngle mage. In proceedngs of ACM Sggraph, pp , [6] Breen, D.E., Whtaker, R. T., Rose, E. and Tuceryan, M. Interactve Occluson and Automatc Object Placement for Augmented Realty. Computer and Graphcs Forum (proceedngs of EUROGRAPHICS'96), vol. 15, no. 3, pp. C11-C22, [7] Brent, R.P. Algorthms for mnmzaton wthout dervatves. Prentce-Hall, Engelwood Clffs, NJ, [8] Drettaks, G. and Sllon, F. Interactve update of global llumnaton usng a lne-space herarchy. In proceedngs of ACM Sggraph 97, pp , [9] Fourner, A. Gunawan, A.S., and Romanzn, C. Common llumnaton between real and computer generated scenes. In proceedngs of Graphcs Interface 93, pp , [10] Gbson, S. and Murta, A. Interactve renderng wth real-world llumnaton. In proceedngs of 11th Eurographcs Workshop on Renderng, pp , [11] Goral, C.M., Torrance, K.E., Greenberg, D.P., and Battale, B. Modelng the nteracton of lght between dffuse surfaces. Computer In proceedngs of ACM Sggraph 84, vol. 18, no. 3, pp , [13] Loscos, C., Drettaks, G., and Robert, L. Interactve vrtual relghtng of real scenes. IEEE Transactons on Vsualzaton and Computer Graphcs, vol, 6, no. 3, pp , [14] Majumder, A. He, Z., Towles, H., and Welch, G. Achevng color unformty across mult-projector dsplays. In proceedngs of IEEE Vsualzaton, pp , [15] Mlgram, P. and Kshno, F. A Taxonomy of Mxed Realty Vsual Dsplays. IEICE Transactons on Informaton Systems E77-D, vol.12, pp , [16] Nakamae, E., Harada, K., Ishzak, T., and Nshta, T.A montage method: The overlayng of computer generated mages onto background photographs. In proceedngs of ACM Sggraph 86, pp , [17] Neder, J., Davs, T., and Woo, M. OpenGL Programmng Gude. Release 1, Addson Wesley, ISBN , pp , [18] Press, W.H., Teukolsky, S.A., Vetterlng, W.T. and Flannery, B.P. Numercal Recpes n C - The Art of Scentfc Computng (2nd edton), Cambrdge Unversty Press, ISBN , pp ,1992. [19] Raskar, R. Welch, G., Low, K.L., and Bandyopadhyay, D. Shader Lamps: Anmatng real objects wth mage-based llumnaton. In Proc. of Eurographcs Renderng Workshop (EGRW 01), [20] Wood, D.N., Azuma, D.I., Aldnger, K., Curless, B., Duchamp, T., Salesn, D.H., and Stuetzle, W. Surface Lght Felds for 3D Photography, In proceedngs of ACM Sggraph 00, pp , [21] Yang, R., Gotz, D., Hensley, J., Towles, H., and Brown, M.S. PxelFlex: A reconfgurable mult-projector dsplay system. In proceedngs of IEEE Vsualzaton, pp , [22] Yu, Y., Debevec, P., Malk, J., and Hawkns, T. Inverse global llumnaton: Recoverng reflectance models of real scenes from photographs. In proceedngs of ACM Sggraph 99, pp , [12] Levoy, M. and Hanraham, P. Lght feld renderng. In proceedngs of ACM Sggraph 96, pp , 1996.