Interactive Virtual Relighting of Real Scenes

Transcription

1 Frst submtted: October 1998 (#846). Edtor/revewers please consult accompanyng document wth detaled responses to revewer comments. Interactve Vrtual Relghtng of Real Scenes Célne Loscos, George Drettaks, Luc Robert Abstract Computer augmented realty (CAR) s a rapdly emergng feld whch enables users to mx real and vrtual worlds. Our goal s to provde nteractve tools to perform common llumnaton,.e., lght nteractons between real and vrtual objects, ncludng shadows and relghtng (real and vrtual lght source modfcaton). In partcular, we concentrate on vrtually modfyng real lght source ntenstes and nsertng vrtual lghts and objects nto a real scene; such changes can be very useful for vrtual lghtng desgn and prototypng. To acheve ths, we present a three-step method. We frst reconstruct a smplfed representaton of real scene geometry usng sem-automatc vson-based technques. Wth the smplfed geometry, and by adaptng recent herarchcal radosty algorthms, we construct an approxmaton of real scene lght exchanges. We next perform a preprocessng step, based on the radosty system, to create unoccluded llumnaton textures. These replace the orgnal scene textures whch contaned real lght effects such as shadows from real lghts. Ths texture s then modulated by a rato of the radosty (whch can be changed) over a dsplay factor whch corresponds to the radosty for whch occluson has been gnored. Snce our goal s to acheve a convncng relghtng effect, rather than an accurate soluton, we present a heurstc correcton process whch results n vsually plausble renderngs. Fnally, we perform an nteractve process to compute new llumnaton wth modfed real and vrtual lght ntenstes. Our results show that we are able to vrtually relght real scenes nteractvely, ncludng modfcatons and addtons of vrtual lght sources and objects. 1

2 1 Introducton Computer augmented realty (CAR) s a rapdly emergng feld whch enables users to mx real and vrtual worlds. Many applcatons of augmented realty already exst, for example n entertanment, flm and televson or help-systems for repar and manufacturng. Usng real scenes or mages of real scenes has many advantages over tradtonal, manually modeled vrtual envronments: the great effort requred to model detaled objects s avoded, the resultng complexty n geometrc prmtves s greatly reduced, and the user s confronted wth famlar, real-world references, allowng mmedate and ntutve mmerson. The focus of prevous research n the doman has manly been on regstraton and calbraton [Azu97]. Our nterest however focuses on common llumnaton between real and vrtual objects,.e. the nteracton of lght (shadows, reflectons etc.) between real lghts and objects and vrtual lghts and objects. Prevous work n common llumnaton [SHC + 96, NHIN86, YM98, JNP + 95, Deb98, FGR93, DRB97] has provded solutons whch handle certan cases of lght nteractons between real and vrtual objects, and especally the case of vrtual objects castng shadows onto real objects. Most of them focus on the producton of hgh-qualty, off-lne generaton of flms, or are lmted n the effects they produce. In ths paper, we present a system whch allows nteractve modfcaton of real or vrtual lghts, based on a smple model of the real scene and recent developments n nteractve herarchcal radosty. Modfcaton of real lghts s a dffcult problem because real shadows are already present n textures representng the real world, whch are mapped onto the reconstructed models of real objects. Prevous solutons [NHIN86, YM98] allow the vrtual modfcaton of the sun poston n outdoors real envronments, but are based on nherently non-nteractve algorthms. We buld our method on a prevous nteractve common llumnaton approach [DRB97], whch was restrcted n the effects t could treat to vrtual objects castng shadows on reconstructed real surfaces. Concurrently wth ths work, two new solutons have been proposed [YDMH99, LFD + 99]. We dscuss how ths work relates to these soluton n Secton

3 Our soluton has three man steps. We frst reconstruct a 3D representaton of a real scene, usng advanced vson-based technques. A herarchcal radosty system s then ntalzed n a preprocess step, to represent real world llumnaton. The preprocess then automatcally creates new unoccluded llumnaton textures whch represent llumnaton wthout takng shadows nto account, and fnally reprojects real shadows usng the modfed radosty estmaton. The thrd step s an algorthm allowng nteractve modfcaton of real and vrtual lght source ntensty. For a lghtng desgner, ths system provdes realstc tools to experment wth the llumnaton of an enhanced real envronment. All that s requred s a few photographs of the real scene, the reconstructon of a small number of objects, and the system preprocess as wll be descrbed n ths paper; the desgner can then nteractvely manpulate real lght ntenstes, or nsert and manpulate vrtual lghts and objects. In Fgure 1, an example of a modeled real scene s shown n (a). In (b), ts real llumnaton was modfed by swtchng off two lghts. Moreover, a vrtual lght source was nserted nto the real scene, modfyng real object shadows. In (c), a vrtual object was also nserted nto the real scene, castng shadows onto real objects. Ths vrtual object can be moved nteractvely at 3 frames per second (see accompanyng web move). (a) (b) (c) Fgure 1: (a) Orgnal real scene. (b) Vrtual modfcaton of the llumnaton of the real scene. (c) Real scene enhanced by a vrtual object that moves at 3 frames per second. In the followng sectons, we frst present prevous work n the several domans related to ths work: augmented realty, 3D reconstructon of real scenes and herarchcal radosty. Prevous common llumnaton approaches are then dscussed n more detal. We proceed to explan how we buld a 3D geometrc model representng the real scene, and present an overvew of the 3

4 algorthm for nteractve re-lghtng. The preprocess phase s presented n detal, followed by a descrpton of the nteractve relghtng process. We then descrbe results of relghtng and nteractve modfcaton of real lght ntenstes and the nserton of vrtual lghts and conclude wth at dscusson and future work. 2 Prevous Work Our work draws on multple felds; n partcular augmented realty, vson based reconstructon and global llumnaton. In the followng, we wll frst gve a rapd overvew of augmented realty whch concentrates, n general, on regstraton and calbraton aspects. We next brefly dscuss the 3D reconstructon method we use to buld a smplfed model of the real scene. We wll use herarchcal radosty to create a representaton of real-world llumnaton, and also to permt nteractve updates when movng vrtual objects or modfyng llumnaton; we thus ntroduce the basc concepts of ths approach whch are central to the understandng of our algorthm. We fnally detal prevous work on global common llumnaton, nsstng n partcular on the most closely related approaches whch use radosty methods. 2.1 Introducton to augmented realty There are two man approaches to augmented realty: vrtual and real envronments can be combned by supermposng vrtual objects on the real world vewed from sem-transparent glasses; alternatvely, vrtual and real envronments can be merged wth vdeo mages, and the result reprojected onto a screen. These two approaches are presented n detal n the survey of Azuma [Azu97] whch also provdes extensve references to related lterature. The frst approach nvolves the calbraton, regstraton and dsplay of vrtual objects n real tme to avod delays between projected mages and the perceved real world. The second approach allows more nteracton between real and vrtual objects, because a geometrc representaton of the real scene s created from the mages. We can therefore handle occluson 4

5 between real and vrtual objects, as well as vsual effects such as common llumnaton, whch s the nteracton of lght between vrtual and synthetc objects. Nevertheless, achevng real tme or nteractve dsplay of these effects remans a challengng problem. 2.2 Reconstructon of Real Models The smulaton of common llumnaton effects requres a geometrc representaton of the real world. Much research on the subject exsts n the feld of Computer Vson; we have chosen to use an advanced vson-based technque, whch allows sem-automatc reconstructon based on multple vews. The approach we use s presented n [FLR + 97]. In order to buld a representaton of a real scene, several vson technques are combned: automatc calbraton of the camera, mosacng, computaton of the eppolar geometry whch results n a polygonal reconstructon of the scene, and the projecton of textures. The frst step s the calbraton of the camera whch conssts n retrevng the ntrnsc parameters from a non-planar calbraton pattern mage usng an automatc algorthm [Rob95]. The user provdes approxmate postons of 6 reference ponts. From ths, the system retreves ntrnsc and extrnsc parameters of the camera. Then, four sets of three photographs each are taken, and a mosac s bult automatcally for each set as presented n [ZFD97]. From the four mosacs, a 3D model s defned usng the TotalCalb system [Tot] developed at the ROBOTVIS group. Ths system, shown n Fgure 2, combnes several technques. Pont correspondences are provded by a user, who clcks on one mage to create a reference pont. The matched ponts on the 3 other mosacs are gven automatcally by the system. From about 30 pont correspondences, fundamental matrces are computed usng a non-lnear method [ZDFL95]. Polygonal regons are next manually selected by a user from pont correspondences, and the system provdes 3D coordnates of these polygons from the projecton equatons. Fnally, textures are projected to allow correct perspectve effects for a fxed vewpont [FLR + 97]. For each reconstructed polygon, a texture mage s computed by 5

6 de-warpng the orgnal mage (from a gven vewpont), and mappng t to the plane of the polygon. Fgure 2: The TotalCalb system to buld 3D models of real scenes, usng automatc calbraton, and eppolar geometry. The man advantage of such a system s that user nterventon s restrcted to the choce of reference matches and polygon vertex selecton. Ths system s however not wthout lmtatons: the resultng model of the real scene s approxmate and may contan artfacts, snce there s no guarantee that geometrc propertes such as parallel edges or orthogonal angles wll be preserved. Ths drawback can be removed by takng nto account addtonal user nput, as presented n the work of Debevec et al. [DTM96] or Pouln et al. [POF98b]. In the work by Debevec et al. [DTM96], reconstructon s based on a herarchy of blocks. The man dea s to buld polyhedra whch nclude geometrc constrants, such as parallelsm, orthogonalty, and sze aspects. Polyhedra provde good approxmatons of many objects of the real world, especally for outdoor archtectural scenes. Ths also allows the reconstructon of vertces whch are nvsble n the orgnal mages, but correspond to hdden vertces of the polyhedra. Another approach s descrbed n [POF98b] n whch the prmtves are ponts, lnes and polygons, and constrants such as parallelsm, orthogonalty, or co-planarty are determned 6

7 by the user. 2.3 Herarchcal radosty To acheve nteractve relghtng, we need an effcent descrpton of lght exchanges n the scene, ncludng shadow nformaton. We have chosen to use the herarchcal radosty approach wth clusterng [HSA91, Sl95] wth the extensons to dynamc envronments [DS97]. We next ntroduce certan basc concepts of radosty methods. The radosty method s based on energy exchanges, and has been used n computer graphcs to smulate lght nteractons n synthetc envronments [SP94], ncludng ndrect llumnaton. Snce the radosty method s a fnte-element approach, a mesh representaton of the scene s requred, whch s usually constructed wth quadtrees. Herarchcal radosty [HSA91] uses a mult-resoluton representaton of lght, by creatng a herarchy of patches on each surface. Lght exchanges are establshed at the approprate levels at the patch herarchy va a lnk data structure, resultng n an effcent soluton. A generalzaton of ths approach can be acheved usng clusters [SAG94, SDS95], whch represent groups of objects. The entre scene s contaned n a sngle, root cluster. Clusters and patches can be lnked at the approprate level, dependng on the refnement crteron, whch decdes whether the lnk represents the lght transfer at a sutable, user defned, level of accuracy. If the lght exchange s not suffcently well-represented, the lnk s refned and the patches or clusters are then subdvded. When lnks are establshed, the ncomng rradance s gathered at each patch, followed by a push-pull step performed to mantan a coherent mult-resoluton representaton of radant exchanges [HSA91]. The cluster-based herarchcal radosty starts wth the root cluster lnked to tself. The algorthm descrbed by Sllon [Sl95] performs a refnement step, establshng lnks at approprate levels followed by the gather and push-pull steps. Irradance s pushed down to the leaves of the patch herarchy, and radosty s pulled up by averagng [Sl95]. Ths 7

8 s repeated untl convergence. Vsblty nformaton and form factors are stored wth lnks. The vsblty nformaton can be of three types: VISIBLE, INVISIBLE or PARTIAL. When computng radosty exchanges between two patches, the ncomng rradance s multpled by the form factor and an attenuaton factor, whch vares from zero when the patches are mutually completely occluded, to one when the patches are entrely mutually vsble. The attenuaton factor represents the degree of occluson between two patches. It s typcally estmated by shootng rays between the two patches, and countng the percentage of rays blocked by occluders. The herarchcal representaton wth lnks can be adapted to allow fast radosty modfcaton [DS97], by augmentng the lnks wth a shaft data structure [HW91]. In addton, prevously subdvded lnks, called passve lnks are mantaned. The passve lnks contan all the necessary nformaton allowng them to be reactvated at no cost, f t s requred by a geometry change. SeeFgure3foranexample. (a) (b) (c) Fgure 3: (a) Orgnal subdvson and lnks n purple. (b) Addng a dynamc object, and updatng the herarchy of elements and lnks. Eght lnks shown n blue were created. (c) The passve lnks wth ther shafts are mantaned n the herarchy, allowng fast dentfcaton of the dynamc object movement. In ths case, two passve lnks shown n green were mantaned. The correspondng shaft s outlned n grey. 2.4 Common llumnaton n augmented realty The retreval and smulaton of common llumnaton between vrtual and real objects has been treated by several researchers n prevous work [SHC + 96, NHIN86, YM98, JNP + 95, Deb98, 8

9 FGR93, DRB97, YDMH99]. All use some form of a 3D representaton of the real scene. State et al. [SHC + 96] use a composton of vson-based and magnetc trackng methods for accurate regstraton of the real envronment. Vrtual objects are nserted nto a real scene and common llumnaton s performed, wth a movng (real) pont lght source. Shadow maps are used allowng updates n real tme, but only for drect llumnaton and sharp shadows from pont sources. Nakamae et al. [NHIN86] developed a soluton for mergng vrtual objects nto background photographs, and estmated the sun locaton to smulate common llumnaton effects n outdoors envronments. More recently Yu [YM98] proposed a soluton to vrtually modfy the llumnaton wth dfferent vrtual postons of the sun n outdoors scenes. A pseudo-brdf s frst estmated, whch s a functon of the ncdent radance on the reflected dfferental radance. Dffuse and specular reflectances are retreved usng multple mages from multple vewponts. From varous vrtual postons of the sun and from modfed sky and envronment llumnaton, modfed outdoors llumnaton s performed pxel by pxel for each reconstructed trangle. However, for certan applcatons, an approxmaton of only the dffuse reflectance s suffcent. For ndoors envronments, Jancène et al. [JNP + 95] used vson-based technques to retreve the geometry of the real scene from a vdeo sequence. Common llumnaton between vrtual and real objects s smulated. Ths allows the creaton of vdeo sequences, wth anmated vrtual objects such as a cloth, and the modfcaton of the reflectve propertes of real objects. The fnal renderng s performed usng a ray-tracng system, and mages are merged usng a maskng algorthm. Debevec [Deb98] also smulates common llumnaton effects usng RADIANCE [War94], a ray tracng based global llumnaton system. In ths work, the real envronment s decomposed nto the dstant scene and the local scene. The dstant scene s used to evaluate the global radance, and the source emttance [DM97]. An approxmate geometrc model of the local scene s bult usng the methods prevously developed by the same author [DTM96]. Snce 9

10 radance s accurately retreved from mages, renderng wth mxed mages s done by usng the dfference of the wanted effects and the orgnal mage value. Ths method can be adapted for ndoors or outdoors envronments. The common llumnaton methods presented above are geared towards hgh-qualty mage generaton, requrng n the order of mnutes per frame. Those whch allows relghtng need several mages under dfferent lghtng condtons, or several vewponts. Our approach s complementary: we want to use smple data, that s a sngle mage of a sngle vewpont under orgnal lghtng condtons, and from ths we want to provde nteractve common llumnaton effects, whch wll allow a desgner to modfy and experment wth dfferent lghtng condtons. Dgtal prototypng or mock-ups requre ths type of nteractve capablty; for a fnal hgh-qualty anmaton, one of the prevous methods can always be used. Radosty-based systems for common llumnaton The most closely related prevous work s that of Fourner et al. [FGR93] and ts nteractve extenson [DRB97]. The system presented permts the retreval of radosty parameters from the textures extracted from the real scene mages. In our approach, we use Fourner et al. s basc dervatons for the extracton of the quanttes requred to ntalze the radosty equatons. We thus descrbe ths work n more detal. Frst the real scene s modeled manually, usng a smplfed representaton. Gven ths model whch s subdvded nto patches, the reflectance can be extracted from the mage textures. The reflectance of each patch s chosen to be: ρ = ˆB ˆB A ˆρ (1) where ˆB s the average ntensty of the pxels n an mage correspondng to the projected patch, ˆB A the average ntensty of the real mage, and ˆρ s the average reflectance of the scene (gven by the user). Ths estmaton of the reflectance depends on the color of the texture (.e., the 10

11 photograph of the real scene), whch wll be darker for patches n shadow. The emttance E of each source s estmated from the followng equaton: E A =(1,ˆρ)ˆB A A (2) wth A beng the area of patch. Ths approxmaton s based on the estmated ambent term n the progressve radosty algorthm [CCWG88]. To smplfy, and as t s approxmately the case for our scenes, we consder that all the sources have the same ntensty. However a system of equatons could be solved for non-homogeneous ntenstes. Once the reflectance(s) ρ and the emttance(s) E have been estmated, a progressve radosty soluton s appled. The result of ths smulaton s the radosty B of each patch. The dsplay s done usng a dsplay correcton factor D of a patch, whch s frst ntalzed to the radosty B. When the scene s modfed, the current radosty B s updated to reflect the change. For example, f a vrtual object s nserted, the patches on whch a shadow s cast wll have B < D. Modfcatons to the scene (notably the addton of vrtual lghts and objects), are performed by modulatng the texture T of a pxel as follows: B D T (3) It s mportant to note here that the accuracy of the radosty estmaton B s rrelevant. Snce a rato s beng used (whch s 1 f there s no change), the only requrement s that the modfcatons to B have to be consstent. Note that ray-castng s used for renderng n [FGR93]. Ths approach was adapted n [DRB97], n the context of a herarchcal radosty system, whch allows common llumnaton between a dynamc vrtual object and a real scene. The nteractve update of the llumnaton when the vrtual object moves uses the dynamc herarchcal radosty soluton descrbed n [DS97]. An example of the results obtaned by the method of [DRB97] s shown n Fgure 4, where a red dynamc vrtual object was nserted nto a real scene, on the top of the desk. The shadows are vrtually projected onto the table, usng the dsplay rato descrbed above (Eq. (3)). 11

12 (a) (b) (c) Fgure 4: (a) A vrtual object, floatng above the table, was nserted nto a real scene usng [DRB97], n 5.65 seconds. Shadows are projected onto the table usng the dsplay rato of Eq. (3). (b) and (c) The dynamc object moves above the table. Lnks and radosty are updated nteractvely n 3 frames per seconds. 2.5 Shortcomngs of Prevous Approaches If we use the method of [FGR93, DRB97] to change the ntensty of a real lght source, the result s unsatsfactory. Ths can clearly be seen n Fgure 5(b). (a) Fgure 5: (a) The orgnal llumnaton of the real scene. Two sources (left and rght) llumnate the wall, causng the shadow of the table to be cast on the wall. (b) Usng the method of [DRB97], we turn off the left hand lght; pre-exstng shadows are not removed and the qualty of the relghtng s unsatsfactory: the shadow due to the left hand lght s stll clearly vsble. (b) To see why, recall that the dsplay s performed usng the ntal real world photograph [FGR93] or textures [DRB97]. The mage or textures are then modulated by the rato of the current radosty value B (changed for example by turnng off a lght) over the orgnally com- 12

13 puted radosty value D. Snce the texture beng modulated s a snapshot of the real global llumnaton n the scene, real shadows are already represented. Consder the n Fgure 5(b) for whch the left hand lght has been turned off. Observe the regon n the blue square: t contans a regon of the wall whch was n shadow wth respect to the left hand lght, and a regon whch was not. Snce the left hand lght s turned off, the current radosty value B wll be reduced for both regons, by amounts whch are very close n value snce they vary only by the correspondng form factors. The textures of both regons are modulated by the rato of ths current radosty B over the orgnal radosty value before the lght was swtched off. Snce the texture (photo) correspondng to the regon orgnally n shadow s much darker to begn wth, the shadow wll stll be vsble after the change. For the correct mage to be dsplayed, we would need a way to make the texture n both shadowed and unshadowed have smlar values. Ths reveals the lmtaton of prevous texture modulaton approaches, whch can only treat modfcatons to vrtual objects or sources, snce modfcaton of real lghtng condtons requres the modfcaton of the orgnal mages or textures of the scene. 3 The Common Illumnaton System The goal of our approach s to allow nteractve modfcaton of real source ntenstes, the nserton (and modfcaton) of vrtual sources, and the nserton and nteractve manpulaton of other vrtual objects. All nteractve updates wll be performed wth consstent update of shadows of real and vrtual objects. Our system conssts of 3 steps: 3D reconstructon of the real scene, a preprocessng ntalzaton stage, and an nteractve modfcaton stage, durng whch the user can modfy and enhance the real scene. The entre algorthm s summarzed n Fgure 6. 13

14 3D reconstructon Buld a smplfed 3D model of the real scene Preprocess Herarchcal radosty system set up Refnement for shadow boundares Creaton of the unoccluded llumnaton textures System re ntalzaton and shadow reprojecton Addtonal preprocess for the nserton of vrtual objects Interactve modfcaton Modfcaton of real and vrtual lghts. Update when a vrtual object moves. Fgure 6: Complete algorthm. Representaton of the real scene The real scene s represented n our system wth an approxmaton of ts geometry and wth projected textures. The model of the scene s bult sem-automatcally, usng advanced vson technques [FLR + 97, DRB97] as descrbed n Secton 2.2. Ths process allows the reconstructon of the basc 3D model vsble n the captured mages (for example the mosacs shown n Fgure 7). The rest of the scene, whch s not vsble n the mages, s bult wth ponts measured manually. Approxmate textures are used to map the polygons of ths part. The postons of the lght sources are also measured manually and nserted nto the 3D model. The model s then an approxmaton of the captured room, wth a more precse model for the vsble part of the scene, and a coarse model for the rest. An example of the resultng reconstructon s shown n Fgure 8. A lmtaton of ths approach s that the projecton of the textures s done only for a sngle pont of vew. We are therefore restrcted to vewng the scene from a statc vewpont. In Fgure 8(a), the model s vewed from our system, and n (b) the complete model s shown, ncludng the non-vsble part. 14

15 (a) (b) (c) Fgure 7: The four mosacs from four dfferent ponts of vew. (d) Preprocess to enable nteractve re-lghtng The man contrbuton of our approach s the preprocessng algorthm whch results n the generaton of modfed orgnal textures, approxmatng unoccluded radosty n the scene. We call these the unoccluded llumnaton textures. The orgnal values of the textures, taken from the ntal scene photograph are thus modfed to represent llumnaton as f shadows of real objects where not taken nto account. These shadows can be due to real lght sources, or other secondary reflector objects. Once we have created ths unoccluded llumnaton texture, we can perform rapd relghtng by modulatng the texture wth a rato, correspondng to the ncrease or decrease n llumnaton due to lghtng changes. Ths s acheved usng a mesh of elements created by the radosty algorthm. These elements are fner n the regons of shadow, and suffcent to capture other changes n llumnaton (for example due to ndrect lght). The preprocess begns by settng up all necessary parameters for the estmaton of the real scene llumnaton as n [DRB97]. A sutably subdvded mesh s essental for the approprate modulaton of texture; to acheve ths a texture-based refnement s appled. To create the 15

16 (a) Fgure 8: (a) The real scene vewed from our system. (b) The complete model ncludng four lghts (A, B, C, D). (b) unoccluded lght textures, the nformaton contaned n the radosty soluton s used. Due to naccuraces of the capture and reconstructon process, we use a heurstc correcton to ths process, based on shadow boundares whch are approprately nferred from the radosty soluton. The result of ths process are the unoccluded radosty textures, whch can then be modulated by the rato of the fnal radosty to unoccluded radosty to reproject shadows and other llumnaton effects. Ths pre-process s explaned n detal n the next secton. Vrtual objects and vrtual lght sources can then be nserted f desred. The nserton of dynamc objects s performed usng the method of [DRB97]. The algorthm used to nsert vrtual lght sources s descrbed n secton 4.5. Interactve Relghtng When the entre preprocessng step s completed, we can nteractvely modfy the llumnaton of the lght sources. The algorthm used s presented n secton 5. Our nterface, shown n Fgure 9, allows the user to choose a new emttance n each RGB channel for real and vrtual lght sources. A smlar nterface also exsts for the nserton of real and vrtual lghts or objects (see web vdeo for examples). 16

17 Fgure 9: A screen snapshot of the nteractve system. The user can manually select new lght ntenstes for real or vrtual lght sources usng the slders shown n the nset. 4 Preprocessng for Vrtual Interactve Relghtng Recall that our approach uses texture modulaton as descrbed n [FGR93, DRB97], and n Eq. (3) for rapd dsplay, allowng the use of the graphcs hardware. As n [DRB97], we start by ntalzng the herarchcal radosty system based on textures extracted from the orgnal photographs, as presented n detal n Secton 2.4. To acheve the modfcaton of real lghtng we need to construct the unoccluded llumnaton textures, whch are textures capturng an approxmaton of the llumnaton n the envronment as f there were no occluson from the lght sources and secondary sources.. The creaton of these textures has two steps: frst we add n blocked lght, usng the nformaton contaned n the radosty soluton. Snce ths gves mperfect results due to the numerous approxmatons performed, a heurstc correcton s appled by fndng an approprate reference patch whch wll gve us a strong ndcaton of the desred fnal color. For both steps, t s mportant to have an approprate mesh subdvson for radosty, notably for the shadows on objects whch are vsble for our gven vewpont. We begn by descrbng our texture-based refnement, and then proceed to descrbe the two steps of the unoccluded llumnaton texture generaton. 17

18 4.1 Texture-based refnement for shadow boundares If we use standard refnement crtera, such as BF refnement [HSA91] or error-drven refnement [GH96] we do not obtan sutable radosty mesh subdvson. The man problem s that these approaches do not always guarantee good shadow boundares (even when usng the vsblty factor of [HSA91]). In addton, the problem s more apparent n our case, snce the geometry reconstructon and the vsblty computaton va ray-castng are not completely accurate. Dscontnuty meshng [LTG93] s unsutable for the same reasons, snce dscontnuty lnes would be geometrcally naccurate. As a consequence, we use quadtree subdvson, wth new, texture-based refnement. The man dea s to use color nformaton contaned n orgnal textures (.e. the photos of the real scene reprojected as texture onto the reconstructed polygons), combned wth the vsblty nformaton provded by the radosty system as ntalzed above. Real shadows already exst n the textures, and correspond to regons whch are darker. By usng the vsblty type (VISIBLE, PARTIAL, OCCLUDED see Secton 2.3) contaned n the lnks to patches n penumbra, and the color dfferences between neghborng patches, we can force refnement n regons correspondng to real shadow. Ths refnement occurs after the frst approxmaton of the radosty soluton of the real scene usng the approach of [FGR93, DRB97]. The frst radosty soluton s used to ntalze several parameters such as reflectances and lght source ntenstes. As shown n Fgure 10(a), the ntal subdvson, obtaned usng BF refnement, s coarse. Lnks have been attached to the leaves of the herarchy of patches as n Fgure 10(b), to provde accurately vsblty nformaton wth respect to the source patches. The texture-based refnement algorthm compares the vsblty and the color of two neghborng leaves of the patch herarchy. The vsblty must be consstent wth the color dfferences. We consder two cases, for a patch and each of ts neghbors (the meanng of smlar for color and vsblty s defned below): 18

19 (a) Fgure 10: (a) Coarse mesh before refnement. (b) All lnks are at leaves. (b) 1. If the two patches have smlar colors, they should also have the same vsblty type wth respect to all the real lght sources. If t s not the case, then we subdvde the patch. 2. If the two patches have dfferent colors, they should also have dfferent vsblty types. If not, we subdvde the patch. If the patch has been subdvded, we examne the chldren created; f there s no change n vsblty, the patch subdvson s cancelled and the patch s agan a leaf of the patch herarchy. Case 1 occurs at the lmts of shadow boundares, and helps n producng fner elements n these regons. The process wll stop when we reach the maxmum subdvson level or when the patches are separated nto vsble and n shadow. Case 2 occurs when ray-castng has faled to dentfy the correct vsblty type. The patch may be unsubdvded however when the color dfference s not due to a vsblty change, but to a dfferent texture. Ths s the case for the orange poster on the back wall n Fgure 11(a). Fgure 12 shows how the refnement algorthm recursvely traverses the herarchy of elements and compares each par of neghborng herarchy leaves. We consder that the vsblty s smlar f the dfference of the attenuaton factor s less than a vsblty threshold fxed by the user. Smlarly, we consder two patches to have the same color f the dstance n color s less than a color threshold also fxed by the user. To compute ths dstance, we frst convert RGB 19

20 (a) Fgure 11: (a) A patch n pnk wth a dfferent color than neghbors but the same vsblty. The patch was not subdvded. (b) Mesh after texture-based refnement wth mproved shadow boundares. Compare to Fgure 10. values nto LAB values. The dstance between the two colors s smply computed by: (b) Dstance Color (;n)= q (L,L n ) 2 +(A,A n ) 2 +(B,B n ) 2 (4) Refnement for shadow boundares for each leaf, compare wth ts neghbor leaves n f has a smlar color to n and a dfferent lght source vsblty then subdvde else f has a dfferent color to n and smlar lght source vsblty then subdvde else do nothng f has been subdvded then f has no lght source vsblty dfferences wth ts chldren then remove the subdvson of ( s a leaf agan) else redo the process for each new chld of. Fgure 12: Texture-based refnement for shadow boundares. At the end of the refnement process, we set up all the necessary parameters: the reflectance, the dsplay correcton factor D org, whch s equal to the orgnal radosty B org,andthetexture T org, whch s the orgnal texture before any correcton:.e., extracted drectly from the orgnal photographs. 20

21 Lnks from real lght sources are fxed at the leaves of the patch herarchy. A radosty step (gather/push-pull) s then computed, correspondng to ths new subdvson. The texture-based refnement results n well-defned shadow boundares, whch s very mportant for the subsequent texture modfcaton step. The resultng refnement s shown n Fgure 11(b). 4.2 Creatng the Unoccluded Illumnaton Texture, Step 1: Addng n Blocked Lght Once an approprate refnement has been performed for a radosty soluton, we can proceed wth the creaton of the unoccluded llumnaton textures. As mentoned above, the frst step conssts n addng n lght blocked. The result of ths step wll be the generaton of a modfed texture, n whch the blocked lght has been ncorporated. We defne Ē s to be the rradance from an source s blocked from patch due to occluson. A source s ether a prmary lght source or a secondary source (.e., a reflectng patch). Ths addtonal rradance s the sum of the radosty of each source tmes the form factor F s and the complement of the attenuaton factor equal to (1,V s ) for each prmary or secondary source s. Consderng each real source, we have the addtonal rradance Ē for patch : Ē = F s (1,V s )E s (5) s The fact that all lnks are at the patch herarchy leaves allows satsfactory estmaton of Ē, snce the form-factor and vsblty nformaton are relatvely accurate. For more accuracy, we take nto account the occluded ndrect llumnaton. However, snce we have not reconstructed every object of the scene, and snce the geometrc model s approxmate, the occluded rradance due to ndrect llumnaton s less accurate. In our tests the effect of addng n ndrect lght at ths step has not been decsve. To generate a new texture wth the blocked lght added, the orgnal texture s modulated by a correcton factor computed at the vertces of the leaf radosty patches. Modulatng the texture at patch vertces results n smooth modfed textures. 21

22 The correcton factor s based on the addtonal rradance descrbed above n Eq. (5). To nclude the blocked radosty, we modulate the orgnal texture T org as follows: T nter = ρ E + ρ Ē D org T org (6) In ths equaton, Ē s the potentally blocked rradance (drect plus ndrect), and B org = ρ E.However,Ē s computed wth the approxmate values F s, V s and E s, and thus the D org modulaton of Eq. (6) s not suffcently accurate. = The ntermedate texture T nter s generated by renderng the leaves of the radosty herarchy wth approprate modulaton values (Eq. (6)). If the modulaton factor s greater than one, a mult-pass approach s used, as descrbed n Appendx A. In Fgure 13 (b), we show an example of the texture generated after the addton of the blocked lght, on the floor s orgnal texture shown n (a). As can be clearly seen, the values computed are far too brght n the regons of shadow, due to the naccuraces of the dfferent processes used. (a) (b) (c) Fgure 13: (a) Orgnal texture. (b) The resultng texture T nter, (c) The resultng texture T f nal. The texture obtaned after ths frst step s used to update new reflectance values ρ nter, extracted n the same manner as for the orgnal photographs (Secton 2.4). Radosty values B nter are updated usng these new reflectance values, as well as the dsplay correcton factor 22

23 (a) (b) (c) Fgure 14: (a) Orgnal texture. (b) The resultng texture T nter, wth real occluded llumnaton removed, mapped onto the geometry of the real scene. (c) The fnal texture T f nal after the texture-based correcton. D nter whch s set equal to the newly computed radosty plus the blocked lght. As was demonstratng n the example (Fgure 14)(b), the resultng textures cannot be used as s. To compensate for the naccuraces of the ntal step, a subsequent heurstc correcton factor s appled, based on texture color. 4.3 Creatng the Unoccluded Illumnaton Texture, Step 2: Texture color based correcton The ntuton behnd the heurstc correcton step s to estmate the desred color of the unoccluded texture for a gven element n shadow, based on a truly unoccluded element elsewhere on the same surface. We would lke the color of each pxel of the occluded part of the texture to have a color smlar to that of an unoccluded pxel. The smlarty s modulated by the form factors snce we want to keep unoccluded llumnaton effects n the fnal texture. Consder a patch n shadow, and a patch r chosen approprately whch s unoccluded. We want the correspondng texture values T and T r to be equal, modulated by the form-factor, snce the poston of and r wth respect to the lght sources s dfferent. For the lght sources S, the resultng desred value for the texture for s as follows: T = T r s F s s F rs (7) 23

24 Snce we operate n the context of polygon-based hardware renderng, we perform ths correcton on a per-patch bass. We modulate the texture of each patch usng a correcton factor. Instead of usng the color of the texture, we use reflectance values whch are stored wth the radosty system, and whch are computed drectly from textures. We assocate to each occluded mesh radosty element, a reference patch whch wll serve to correct the texture. For each radosty mesh element n shadow, we thus attempt to fnd a correspondng unoccluded mesh element. We attempt to fnd a patch whch s close, and whch has smlar reflectance values. To do ths, we frst determne the fronter between occluded and unoccluded patches accordng to all lght sources. Havng all lnks at leaves ensures the classfcaton VISIBLE, INVISIBLE or PARTIAL for a gven patch wth respect to a gven source. We are therefore able to defne a fronter composed of completely unoccluded patches that have occluded neghbors wth respect to real lght sources. Ths fronter usually encloses the regons where we need to modfy the texture. However the algorthm does not depend on the creaton of a closed regon. The fronter elements wll be used as references as explaned below. From these selected elements, we keep only those whch are vsble from the vewpont. Ths restrcton s due to the vew-dependent property of the textures we use. An example of such fronter patches s shown n Fgure 15(a). For each occluded patch, we defne a reference patch chosen n the fronter of unoccluded patches. The reference patch r s chosen to have a smlar color as the occluded patch and to be at a mnmum dstance from. For the occluded red patch, shown n Fgure 15(b), the algorthm chooses the black patch as a reference patch from the fronter lst of unoccluded elements shown n Fgure 15(a). The black fronter patch s the closest patch that has a smlar color to the red occluded patch (see the algorthm n Fgure 16). We defne colors to be smlar f the dstance between them s less than a threshold defned by the user. As for the refnement, reflectances are converted nto LAB values, and the dstance Dstance Color s computed usng Eq. (4). Once the reference patch has been chosen, we use Eq. (7), to determne the correcton factor to be appled to the texture of the patch. Snce the reference patch s at a certan dstance from 24

25 (a) Fgure 15: (a) Fronter n green composed of unoccluded patches, whch enclosed shadow regons. (b) Black patch chosen n the fronter as a reference for the red selected patch n shadow. the occluded patch, we modulate the reflectance of the reference patch by the rato of the form factors F s of patch and F rs of patch r wth respect to the source s. Frst, a corrected reflectance ρ corr s computed: ρ corr (b) = ρ r s F s s F rs (8) If no patch n the fronter of unoccluded elements s found for a certan patch, then the reference patch s a default reference patch prevously selected by the user on the polygon before the texture correcton process. Usng the corrected reflectance, we generate the fnal unoccluded llumnaton texture. To generate ths texture, we render the textured leaf patches of the patch herarchy wth an approprate modulaton factor, as when addng n blocker lght. ρ corr For occluded patches only, the texture T nter of patch over the ntermedate reflectance ρ nter textures T nter : If ρ corr as for Step 1. T f nal s modulated by the rato of the correcton factor computed drectly from the ntermedate = ρcorr ρ nter T nter (9) s greater than ρ nter, we use a mult-pass dsplay method descrbed n the Appendx A, 25

26 for each leaf mn dstance = mn color = ε Re f erence = for each patch n fronter lst n f Dstance(, n) < mn dstance and Dstance Color (n, ) < mn color then Re f erence = mn dstance = Dstance(, n) mn color = Dstance Color (, n) Fgure 16: Algorthm to choose reference patches. ρ f nal From ths fnal unoccluded llumnaton texture T f nal, we recompute new reflectance values for occluded patches and perform a radosty step, resultng n new radosty values B f nal based on the new reflectance. We then compute a new dsplay correcton factor D f nal, equal to the new reflectance tmes the sum of the occluded rradance E f nal and the addtonal rradance Ē (see Eq. (5)). Note that ths dsplay factor does not take nto account shadow calculatons. An llustraton of D f nal s gven n Fgure 17(a), and B f nal s shown n Fgure 17(b). The result of the fnal textures s shown n Fgure 17(c). Note that shadows have been mostly removed, and the texture does effectvely represent llumnaton as f shadows had not been computed. 4.4 Shadow reprojecton After the steps prevously descrbed, we have a texture representng unoccluded llumnaton; we now need a way to () reproject orgnal shadows and () modfy the ntensty and add vrtual objects and lght sources. Ths s acheved by modulatng the unoccluded llumnaton texture T f nal by the rato B fnal whch ntutvely s the rato of radosty ncludng shadow calculatons over radosty wthout shadows. Snce B f nal has a smaller value than D f nal D fnal n regons of shadow, these areas are, 26

27 (a) (b) (c) (d) Fgure 17: (a) Dsplay correcton D f nal correspondng to the new texture T f nal. (b) Radosty B f nal correspondng to the new texture T f nal. (c) The resultng fnal texture wth shadows removed. (d) The resultng reprojecton usng these fnal values. darker. As a result, shadows are approprately reprojected, resultng n an mage whch s close to the orgnal photograph. The result of ths process s shown n Fgure 17(d). It s usually unnecessary to mantan the same subdvson used to modfy the textures durng preprocess, snce t needs to very fne. For the nteractve updates, ths can be wasteful. Nonetheless, n some cases the mesh used for the preprocess s satsfactory. To generate a coarser mesh, we clear everythng prevously computed, that s reflectances, radosty, and the dsplay correcton factor. We also clear the subdvson and the lnk herarchy. We then re-compute a soluton based on a smple BFV refnement [HSA91], whch results n a coarser mesh. To compute both B f nal and D two radosty solutons are actually performed. 27

28 At the end of the frst soluton, the dsplay correcton factor D s computed wth all lnks at leaves of the herarchy of mesh to allow accurate blocked lght computaton. A second radosty soluton s then computed, but keepng the same mesh; ths permts the ntalzaton of B f nal, usng less lnks. The resultng mesh s shown n Fgure 18. Fgure 18: After the texture modfcaton, a radosty soluton may be computed, usng a BFV refnement. The resultng mesh s coarser n shadow regons than the one used to correct the texture. 4.5 Modfed refnement to nsert vrtual sources To treat the nserton of a vrtual lght source, we adapt the method of [DRB97], n whch a vrtual object can be nserted nto the real scene and nteractvely manpulated [DS97]. Ths results n the projecton of the shadows due to the vrtual source on the real objects. The nfluence of a vrtual lght s often sgnfcant, and thus we force addtonal refnement by establshng all lnks to vrtual sources on the polygons as opposed to allowng lnks from the vrtual sources to the clusters. Ths s done on the polygons vsble n the captured mages; the polygons correspondng to the hdden parts of the scene are not affected by ths forced refnement. The addtonal lght sources brghten the scene; agan the mult-pass method of Appendx A s used to acheve ths effect. Vrtual lght source nserton s llustrated n Fgures 26, 21, 22,

29 5 Fnal relghtng At ths stage, we have completed the preprocess, and modfcatons are based on changes to the radosty system. Lnks between patches and clusters n the radosty herarchy have already been establshed, ncludng the form factor computaton and the vsblty determnaton. In order to acheve fast updates, the subdvson and the lnks are mantaned durng relghtng. Snce we only modfy the ntensty of the lght sources, the subdvson and lnks stll ft to the llumnaton even after modfcaton. Keepng the same herarchy may result n overly fne mesh subdvson f lghts are swtched off; snce the user may swtch them on agan later however, we prefer to mantan the mesh subdvson. The modfcaton process conssts n recursvely removng radosty stored at each level of the herarchy. We then perform a complete radosty step: wthout performng addtonal refnement, we gather radosty across the lnks, and perform the push-pull step to mantan a coherent representaton of radosty n the herarchy. The teratve process s stopped when the global llumnaton s stable. Ths process s nteractve snce the costly refnement step (whch ncludes vsblty and form-factor computaton) s avoded. The update tme depends on the ntal level of subdvson. Note however that the nserton of a vrtual object may result n addtonal subdvson. The update rate s the same f we modfy one or several lghts. Example update rates are shown n Fgure 19, and dscussed n the followng secton n detal. 6 Results We have tested the algorthm for two dfferent real scenes. For one of them, we use radance mages obtaned usng an adapted verson of the algorthm of Debevec et al. [DM97] (see Appendx B). For each scene, we present results of relghtng and addng vrtual lght and vrtual objects, all performed nteractvely. All tmngs are reported on an SGI Onyx2 Infnte Realty workstaton, R10000 runnng at 195Mhz. The frst scene s shown n Fgure 20(a), under the orgnal llumnaton. We frst swtch off 29

30 Refnement Decrease lght ntensty Tmes for modfcaton 0.2 sec. 0.3 sec. 0.7 sec. Tmes for dsplay 0.2 sec. 0.2 sec. 0.6 sec. Number of leaves/lnks 3486/ / /50787 Fgure 19: Interactve modfcaton of a vrtual lght source ntensty. The tme rates depends on the level of the subdvson, and the number of establshed actve lnks. The leaves at the bottom elements of the subdvson herarchy. the two back lghts (C, D) shown n Fgure 8. In the resultng mage Fgure 20(b), the scene s darker than the orgnal llumnaton shown n Fgure 20(a) but wth no change n shadows. We then swtch off the front left lght (A) and double the ntensty of the rght lght (B) (see Fgure 20(c)). The resultng shadow of the table s homogeneous n the umbra regons. As expected, the shadow due to the left lght has dsappeared, snce the part of the scene whch was llumnated by ths lght source s darker. Compare the new result wth that of the method of [DRB97] prevously shown n Fgure 5(c), whch was nexact, snce real shadows were not removed from textures. We now swtch on the left lght wth double the orgnal ntensty and swtch off the rght lght (see Fgure 20(d)). Agan, shadows are as expected (.e. the shadow boundary of the rght lght s no longer vsble). For each lght modfcaton, the whole process (radosty step and 30

31 dsplay) takes 0.8 seconds. The accompanyng vdeo 1 shows these lght modfcatons, recorded n real tme on an SGI Onyx2 Infnte Realty workstaton. (a) (b) (c) Fgure 20: (a) Orgnal scene lt wth shadow reprojecton. (b) Back lghts are swtched off. (c) Left lght s swtched off, and rght lght has double ntensty. (d) Rght lght s swtched off, and left lght has double ntensty. Note that n ths case, the mesh used for the texture correcton was suffcent. (d) We can also nsert a vrtual source, and modfy ts ntensty as descrbed above. The nserton takes 7.8 seconds. An nterestng test s to swtch off all the real lghts, and to llumnate the real scene only by a vrtual source (see Fgure 21(a) and (b)). Notce that real shadows from real lght sources can no longer be seen. However, real objects such as the table cast new shadows on the floor and the walls, due only to the vrtual lght. Wth ths new llumnaton, we are stll able to nteractvely move a dynamc vrtual object, such as the orange box on the floor, prevously nserted n 1.42 seconds, n Fgure 22. Updates

32 (a) (b) Fgure 21: (a) Insert a vrtual lght. Swtch off all real lghts. The real scene s lt only by the vrtual lght. (b) Decrease the ntensty of the vrtual lght. take 0.3 sec. per frame, when movng the vrtual object, wth the subdvson shown n Fgure 22(a). Wth both real and vrtual llumnaton, ths vrtual object casts shadows onto the real scene. (a) (b) (c) (d) Fgure 22: (a) Inserton of vrtual object and the consequent subdvson. (b), (c), (d) The red vrtual object s movng at nteractve rates. We have also tested our method on another real scene 2, shown n Fgure 23(a). In (b), we 2 Ths real scene was modeled usng the Rekon system developed at Montreal [POF98a] 32