Light Factorization for Mixed-Frequency Shadows in Augmented Reality

Transcription

1 Lght Factorzaton for Mxed-Frequency Shadows n Augmented Realty Dere Nowrouzezahra 1 Stefan Geger 2 Kenny Mtchell 3 Robert Sumner 1 Wojcech Jarosz 1 Marus Gross 1,2 1 Dsney Research Zurch 2 ETH Zurch 3 Blac Roc Studos ABSTRACT Integratng anmated vrtual objects wth ther surroundngs for hgh-qualty augmented realty requres both geometrc and radometrc consstency. We focus on the latter of these problems and present an approach that captures and factorzes external lghtng n a manner that allows for realstc relghtng of both anmated and statc vrtual objects. Our factorzaton facltates a combnaton of hard and soft shadows, wth hgh-performance, n a manner that s consstent wth the surroundng scene lghtng. Index Terms: H.5.1 Multmeda Informaton Systems]: Artfcal, augmented, and vrtual realtes ; I.3.7 Three-Dmensonal Graphcs and Realsm]: Color, shadng, shadowng, and texture 1 INTRODUCTION Shadows provde mportant perceptual cues about the shape and relatve depth of objects n a scene, as well as the surroundng lghtng. Incorporatng realstc shadows n a manner that s consstent wth a scene s lghtng s an mportant problem n augmented realty. We address the problem of shadowng anmated vrtual characters, as well as statc objects, n augmented realty wth lghtng captured from the real-world. Typcally, real-world llumnaton causes both hard and soft shadows, the latter due to lght reflectng off surroundng objects as well as from broad area lghts, and the former due to smaller lght sources such as the sun (Fgure 1). We factorze lght n a well-founded manner, allowng hard and soft shadows to be consstently computed n real-tme wth a combnaton of shadow-mappng and bass-space relghtng approaches. After a bref summary of prevous wor (Secton 2), we overvew basc concepts and notaton (Secton 3), dscuss geometry and lght calbraton (Sectons 3.2 and 4), detal our lght factorzaton (Secton 5) and shadng/shadowng models (Secton 6), and dscuss conclusons and future wor (Secton 9). Sectons 4 and 6 focus on the mathematcal dervatons of our factorzaton and shadng models; however, despte an nvolved exposton, our run-tme mplementaton s qute straghtforward and readers not nterested n these detals can sp ahead to Secton 7 for mplementaton detals. 2 PREVIOUS WORK The semnal wor by Sloan et al. 11] on precomputed radance transfer proposed a technque for compactly storng precomputed reflectance and shadowng functons, for statc scenes, n a manner that allows for hgh-performance relghtng under novel llumnaton at run-tme. In follow-up wor, Sloan et al. use a dfferent representaton for smlar reflectance/shadowng functons that allowed for approxmate shadng of deformable objects 12]. Ren et al. 7] extend ths lne of PRT wor to fully dynamc scenes, allowng for soft shadows to be computed on anmatng geometry from dynamc envronmental lghtng n real-tme. We combne deas from several of these approaches and elaborate on techncal specfcs n more detal n Sectons 3, 4 and 6. Debevec and colleagues 1, 2] poneered the area of mage-based lghtng, detalng an approach for capturng envronmental lghtng Fgure 1: Hard and soft shadows computed n real-tme (> 70 FPS), from real-world llumnaton, usng our lght factorzaton. from the real-world and usng ths to shade vrtual objects. More advanced lghtng capture technques exst, combnng nowledge of the surroundng geometry wth more detalng drectonal capture (e.g., from omn-drectonal cameras) 8], however we buld on the smplcty and effcency of Debevec and Mal s approach 2]. In augmented realty, wor on shadng vrtual objects can be roughly dvded nto three groups: tradtonal computer graphcs models, dscretzed lghtng models, and bass-space relghtng. The frst set of wor smply apples smple pont/drectonal lghtng models to compute shadng (and possbly shadows) from vrtual object onto the real scene. Wth such smple models, the shadng of real and vrtual objects s nconsstent, causng an unacceptable perceptual gap. Dscretzed lghtng models use more advanced radosty and nstant-radosty based approaches 6] to compute realstc shadng on the vrtual objects (at a hgher performance cost); however, ntegratng these technques n a manner that s consstent wth the shadng of the real-world objects s an open problem. Thus, the ncreased realsm of the vrtual object shadng s stll overshadowed by the dscrepancy between vrtual and real shadng. Bass-space relghtng approaches capture lghtng from the real-world and use t to lght a vrtual object 4]. By constructon, the shadng on the vrtual object wll be consstent (to varyng degrees) wth the real-world shadng. However, the couplng of shadng between vrtual and real objects s a dffcult radometrc problem where even slght errors can cause objects to appear to float or stand-out from the real-world objects. We address a core component of ths problem, computng consstent shadng/shadowng on vrtual objects and onto perceptually mportant regons of the realworld. Furthermore, we support anmated objects, whereas pror bass-space approaches only handle statc geometry. 3 MATHEMATICAL OVERVIEW AND NOTATION We adopt the followng notaton: talcs for scalars and 3D ponts/vectors (e.g., ω), boldface for vectors and vector-valued functons (e.g., y), and sans serf for matrces/tensors (e.g., M). 3.1 Sphercal Harmoncs Many lght transport sgnals, such as the ncdent radance at a pont, are naturally expressed as sphercal functons. The sphercal harmonc (SH) bass s the sphercal analogue of the Fourer bass and can be used to compactly represent such functons. We summarze some ey SH propertes (that we wll explot durng shadng) below, and leave renderng-specfc propertes to Secton 6.

2 Defnton. The SH bass functons are defned as follows: 2 P m y m l (ω) = l (cosθ) sn( mφ), m < 0 Km l 2 P m l (cosθ) cos(mφ), m > 0, (1) Pl 0(cosθ), m = 0 where ω = (θ,φ) = (x,y,z) are drectons on the sphere, Pl m are assocated Legendre polynomals, Kl m s a normalzaton constant, l s a band ndex, and l m l ndexes bass functons n band-l. Bass functons n band-l are degree l polynomals n (x,y,z). SH s an orthonormal bass, satsfyng ym l (ω) ym l (ω)dω = δ lm,l m, where the Kroenecer delta δ x,y s 1 f x = y. Projectons and Reconstructon. A sphercal functon f (ω) can be projected onto the SH bass, yeldng a coeffcent vector n 2 f = f (ω) y(ω) dω, and: f (ω) f (ω) = f y (ω), (2) =0 where y s a vector of SH bass functons, the order n of the SH expanson denotes a reconstructon up to band l = n 1 wth n 2 bass coeffcents, and we use a sngle ndexng scheme for the bass coeffcents/functons where = l(l + 1) + m. Zonal Harmoncs. The m = 0 subset of SH, called the zonal harmoncs (ZH), exhbt crcular symmetry about the z-axs and can be effcently rotated to algn along an arbtrary axs ω a. Gven an order-n ZH coeffcent vector, z, wth only n elements (at the m = 0 projecton coeffcents), we can compute the SH coeffcents correspondng to ths functon rotated from z to ω a as n 12] g m l = 4π/(2l + 1) z l y m l (ω a). (3) In Secton 6, we explot ths fast rotaton expresson for effcent renderng wth a varety of dfferent surface BRDF models. SH Products. Gven functons f and g, wth SH coeffcent vectors f and g, the SH projecton of the product h = f g s ] ] h = h(ω) y (ω) dω f j y j (ω) g y (ω) y (ω) dω j = j f j g y j (ω) y (ω) y (ω) dω = f j g Γ j, (4) j where Γ s the SH trple-product tensor. Computng these general products s expensve, despte Γ s sparsty, but by eepng one of the functons constant, a specalzed product matrx can be used: M f] = f Γ j such that h = M f g. (5) j In Secton 6, we wll use specalzed product matrces for shadng. 3.2 Placement and Geometrc Calbraton An essental component of any relable AR system s the computaton of a consstent coordnate frame relatve to the camera. Two common approaches are marer-based and marerless tracng. Marer-based approaches compute a coordnate frame that remans consstent and stable under sgnfcant camera/scene moton. Marerless based tracng nstead reles on computer vson algorthms to establsh ths coordnate frame, but these technques cannot provde scale nformaton and can become unstable durng camera/scene moton. The strength of marerless tracng les n ts generalty: no marers are requred whereas, for marer-based tracng, f a marer s occluded, unexpected results may be computed. Our wor focusses on consstent lghtng and shadng gven a pre-calbrated AR coordnate frame, and so we buld on prevous technques and combne marer-based and marerless approaches to obtan robust correspondence, even under camera/scene moton. We place a marer on the planar surface we wsh to place our vrtual objects on, detect the poston and orentaton of the marer usng the ARToolKtPlus lbrary 13], and fnd a mappng between ths coordnate frame and the coordnate frame computed wth the PTAM marerless tracng lbrary 5]. To do so, PTAM computes a homography from correspondng feature ponts of the two captured mages. The domnant plane of the feature ponts s placed at the z = 0 plane, formng our ntal coordnate frame. The ARToolKtPlus has a coordnate system centered on the marer (see nset) and, gven the two mages, can compute the postons (p 1, p 2 ) and rotatons (R 1,R 2 ) relatve to the marer. PTAM can smlarly compute postons ( ˆp 1, ˆp 2 ) and rotatons ( R 1, R 2 ) wth respect to ts coordnate frame. In a global coordnate frame (R 1,R 2 ) and ( R 1, R 2 ) specfy the same set of rotatons, and so the rotaton that maps from ARToolKtPlus frame to PTAM s frame s R = R 1 R 1 = R 2 R All that remans s to determne the relatve scale and translaton between the two frames. Frst we compute the ntersecton pont o of two rays wth orgns ( ˆp 1, ˆp 2 ) and drectons ( dˆ 1, dˆ 2 ) = (R ( p 1 ),R ( p 2 )). These rays are not only guaranteed to ntersect, but wll do so at the orgn of the marer (relatve to the PTAM frame). The parametrc ntersecton dstance and relatve scale are t = ( ] dˆ 1 ) x ( ˆp2 ) y ( ˆp 1 ) y ( dˆ 1 ) y ( ˆp 2 ) x ( ˆp 1 ) x ] ( dˆ 1 ) y ( dˆ 2 ) x ( dˆ 1 ) x ( dˆ 2 ) y 4 REAL-WORLD LIGHTING and s = t ˆ d 2 p 2. In order to shade vrtual characters, and statc geometry, n a manner that s consstent wth the real-world, t s mportant to capture and apply the lghtng from the surroundng envronment to these vrtual elements. We combne the two most common vrtual lghtng models n mxed realty, tradtonal pont/drectonal lghts and envronmental lght probes, n a novel manner. Pont and drectonal lghts are a convenent for lghtng vrtual objects, supportng many surface shadng (BRDF) and shadowng models. However, these lghts rarely match lghtng dstrbutons present n real-world scenes, even wth many such lghts as n e.g. nstant radosty based approaches 6]. Moreover, the hard shadows that result from usng these approaches can appear unrealstc, especally when vewed next to the soft shadows of real objects. On the other hand, envronmental lght probes can be used to shade vrtual objects wth lghtng from the real scene, ncreasng the lelhood of consstent appearance between real and vrtual objects (see Fgure 2). One drawbac s that, whle shadow functons can be precomputed for statc vrtual objects, t s dffcult to effcently compute soft shadows (from the envronmental lght) from vrtual objects onto the real world and anmated vrtual objects. We frst dscuss two technques for capturng real-world lghtng (Secton 4.1), followed by a factorzaton of the ths lghtng (Secton 5) that allows us to both shade and shadow vrtual objects n a manner that s consstent wth the real scene (Secton 6). 4.1 Capturng Envronmental Lghtng In Secton 6 we show how to compute the shade at a pont x n the drecton towards the eye ω o, L out (x,ω o ). Ths requres (among other thngs) the ncdent lghtng dstrbuton at x, L n (x,ω). In our wor, we assume that the spatal varaton of lghtng n the scene can be aggregated nto the drectonal dstrbuton, represented as an envronment map of ncdent lght, so that L n (x,ω) = L env (ω). We now outlne the two approaches we use to capture L env. Mrror Ball Capture. We place a mrror sphere at a nown poston relatve to a marer and use the ARTooltPlus marer-based feature tracng lbrary 13] to detect the camera poston relatve

3 Domnant Lght Drecton. Startng wth the smpler case of monochromatc ncdent lght L(ω) (wth SH coeffcents L), the lnear (l = 1) SH coeffcents are scaled lnear monomals n y, z and x respectvely, and thus encode the 1 st -moment vector drecton (gnorng coordnate permutatons and scale factors) of the sphercal functon they represent (n ths case, the monochromatc lght): ] T d = ωx,ω y,ω z L(ω) dω, (8) where (ω x,ω y,ω z ) are the Cartesan coordnates of ω. In ths case, d s the prncpal lght vector and, from Equatons 1 and 8, we can solve for the normalzed prncpal lght drecton n terms of L as ( L 3, L 1,L 2 ) = 3 =1(L ) 2 ] 1 2 ( L 3, L 1,L 2 ), (9) Fgure 2: Our lght factorzaton allows for consstent lghtng between vrtual and real objects: note the consstent shadng, from envronmental lght bounces, and shadows from captured lght sources. to the sphere. We requre a parameterzaton of the sphere mage we capture n order to project t nto SH (see below). If we normalze the mage coordnates of the (cropped) sphere mage to (u,v) = 1,1] 1,1], we can map sphercal coordnates as ω uv = (θ,φ) = (arctan(v/u),π u 2 + v 2 ). Furthermore, when dscretzng Equaton 2, we compute f u,v f (ω uv ) y(ω uv ) dω uv, (6) where dω uv = (2π/w) 2 snc(θ) uses the wdth w n pxels of the (square) cropped mrror sphere mage 2]. Free Roamng Capture. In the case where usng a marerbased system s not feasble, we can capture an approxmaton of the envronment lghtng usng the PTAM lbrary 5]. We capture mages of the surroundng envronment and, for each mage, place a vrtual omndrectonal camera at the mean dstance of all feature ponts computed by PTAM for that mage. The mage s then projected to the 6 faces of a vrtual cube (placed n a canoncal orentaton upon system ntalzaton). We smlarly requre a dscrete projected sold angle measure when computng Equaton 6 usng ths cube map parameterzaton. Wth the normalzaed mage coordnates (u,v) on a cube s face and the wdth w n pxels for a sde of the cube, we have (as n 9]) t = 1 + u 2 + v 2 and dω uv = 4 t (3/2)] /w 2. (7) In Secton 7 we dscuss how we mplement hgh-performance capture and dscretzed SH projecton completely on the GPU. 5 LIGHTING FACTORIZATION We propose a two-term factorzaton of the envronmental lghtng nto a drectonal term L d (ω) and a resdual global lghtng L g (ω), and we wll enforce that L env (ω) = L d (ω) + L g (ω). Gven the SH projecton coeffcents for each color channel of L env, {L r],l g],l b] }, our factorzaton sees to compute a domnant lght drecton/color, treat t as a separate ncdent lghtng sgnal, and leave a resdual lghtng sgnal whch corresponds to nondomnant lghtng (e.g., from broad area lght sources and smooth ndrect lght bouncng off of surroundng surfaces). where we use SH sngle ndexng here for compactness. Whle we could use more complex approaches (e.g., 9]) for the trchromatc lghtng case, we nstead choose a smpler, more effcent technque: we convert trchromatc lghtng coeffcents nto monochromatc coeffcents usng the standard color-to-grayscale converson. Thus, the domnant lght drecton can be extracted from L env as ω d = L r] 3 L g] 3 L b] 3 L r] 1 L g] 1 L b] 1 L r] 2 L g] 2 L b] (10) Domnant Lght Color. Gven the domnant lghtng drecton ω d, we now determne the domnant lght color n ths drecton. To determne ths color, we place a planar reflector perpendcular to a unt ntensty SH drectonal lght at ω d, and compute ts albedo so that t reflects unt radance. Gven a planar dffuse reflector wth a normal ω d, the SH coeffcents of a drectonal lght at ω d that yelds unt outgong radance on the plane are αy m l (ω d), where α = 16π/17 for order-3 and order-4 SH, and 32π/31 for order-5 and order-6 SH 1 9]. Smlar scalng factors can be derved for non-dffuse reflectance, but we have found that usng ths factor yelds plausble results, regardless of the underlyng surface BRDF used at run-tme (see Secton 6). We can now analytcally solve for the color as 1 ] 1 ] c ] = α y m 1 (ω d) L ] 2+m / (α y m 1 (ω d)) 2. (11) m= 1 m= 1 Fnal Factorzaton. Gven ω d and RGB color (c r],c g],c b] ), the drectonal term of our factorzaton and ts SH projecton are L d (ω) = c ] δ(ω d ) and L ] d = α c ] y m l (ω d). (12) The global term and ts SH projecton are L g (ω) = L env (ω) L d (ω) and L ] g = L ] L ] d. (13) Note that the factorzaton s perfect, n the sense that the envronmental lght can be perfectly reconstructed from the drectonal and global terms (both n the prmal and SH spaces; see Fgure 3). In Secton 6, we use ths factorzaton to combne several shadng/shadowng models, resultng n realstc shadng wth hard and soft shadows that s consstent wth shadng on real-world objects. 1 The α values are shared across two orders because the SH projecton of the clamped cosne ernel vanshes for all even bands above l = 2.

4 Lghtng SH projecton Drectonal Global L env (ω) L env (ω) L d (ω) L g (ω) Synthetc Test Data Captured Data Fgure 3: Factorzaton nto drectonal and global lghtng terms. 6 SHADING WITH FACTORIZED REAL-WORLD LIGHTING Drect lghtng at a pont x towards the eye from the envronment s L out (x,ω o ) = L env (ω) V (x,ω) f + (x,ω o,ω) dω (14) where V (x,ω) s the bnary vsblty functon that s 1 for drectons that are not occluded by geometry, and 0 otherwse, and f + (x,ω o,ω) = f (x,ω o,ω) (n x ω) s a combnaton of the vewevaluated BRDF f (x,ω o,ω) and a cosne foreshortenng term. There are several challenges to accurately solvng Equaton 14. Frstly, we must ntegrate over all lghtng drectons. Secondly, durng ntegraton, we need to evaluate the bnary vsblty functon whch, n the most general case, requres tracng rays through the scene for each lghtng drecton (and at each x). Prevous wor has ether approxmated L env (or, more generally, L n ) wth many pont lghts, or assumed statc geometry where V can be precomputed and stored at a dscrete set of x s. In the frst case, a pont lght approxmaton allows for vsblty to be computed usng e.g. shadow maps, however many such pont lghts may be requred and the cost of computng and shadng wth many shadow maps qucly becomes the bottlenec of these approaches. In the latter case, f statc scene geometry s assumed, SH based shadng approaches (whch we wll dscuss below) can be used to qucly ntegrate over the many lghtng drectons (wthout explctly dscretzng them nto e.g. ndvdual pont lghts) and, along wth precomputed vsblty data, can compute soft shadows that respond to the dynamc lghtng envronment. Unfortunately, these approaches cannot handle anmatng or deformng geometry, snce the vsblty changes at run-tme n these nstances. We nstead explot our factorzaton, usng a combnaton of approaches to solve the two problems of lght ntegraton and dynamc vsblty computaton. Substtutng Equaton 13 nto 14 yelds L out (x,ω o ) = L d (ω) V (x,ω) f + (x,ω o,ω) dω + S 2 L g (ω) V (x,ω) f + (x,ω o,ω) dω = Lout(x,ω d o ) + L g out (x,ω o) (15) and we wll dscuss solutons to each of these terms ndependently. 6.1 Effcent Computaton of Lout d Substtutng Equaton 12 nto the defnton of Lout d yelds Lout(x,ω d o ) = c ] δ(ω d ) V (x,ω) f + (x,ω o,ω) dω = c ] V (x,ω d ) f + (x,ω o,ω d ). (16) Model f + (x,ω o,ω) ZH coeffcents z l Lambertan (n x ω) 0.282,0.326,0.158,0] ] Phong (ω r ω) s 0.282(s+2) s+1,0.489, 0.631s(s+2) s 2 +4s+3, 0.746(s 1) (s+4) Mrror δ(ω = ω r ) 0.282,0.489,0.631,0.746] ω r s the reflecton of ω o about n x and s s the Phong exponent. Table 1: Analytc form of the BRDFs we use and ther ZH coeffcents. In ths form, ntegraton over lght drectons s replaced by a sngle evaluaton and vsblty can be computed usng shadow maps. The BRDF models we use when evaluatng f + (as well as ther ZH representatons; see Secton 6.2) are summarzed n Table 1. Although Equaton 16 requres a smple applcaton of shadowed pont lghtng, the parameters of ths model are carefully derved n our factorzaton to mantan consstency wth the global shadng term, whch we evaluate wthout explctly samplng any lghtng drectons (Secton 6.2). The combnaton of Lout d and Lg out, and the manner n whch L d and L g are derved and appled, whch maes the use of pont lghtng acceptable for our soluton. 6.2 Effcent Computaton of L g out Unle Lout d n Equaton 16, Lg out cannot reduce nto a sngle samplng operaton snce L g s composed of lghtng from all drectons. Samplng and evaluatng the three terms n the ntegrand, for all drectons at run-tme, s not feasble. We explot the smoothness of L g and perform ths computaton effcently n the SH doman. As dscussed earler, the two man challenges when computng Equaton 14 are ntegraton over all lght drectons and evaluaton of the vsblty functon (at all drectons and all x). The expresson below for Lout d also exhbts these problems, L g out (x,ω o) = L g (ω)v (x,ω) f + (x,ω o,ω)dω, (17) and we wll frst dscuss the ntegraton problem (assumng we have a soluton to the vsblty problem), and then dscuss several approaches we employ for solvng the vsblty problem 2. Integraton wth SH. As we readly have the SH projecton of L g, L g (from Secton 4.1), suppose we can express V and f + wth SH projectons V and f, then the most general soluton to Equaton 17 usng SH nvolves summng over the trple-product tensor, ] ] ] L g out ] Lg y (ω) V j y j (ω) f y (ω) dω j ] = Lg V j f Γ j, (18) j whch s a computatonally expensve procedure, partcularly when executed at every x. Alternatvely, n the case where one of the three terms n the ntegrand s nown beforehand, we can precompute the SH product matrx of ths functon to accelerate the computaton. For example, we could precompute a product matrx for L g as ] M L g = j Lg ] Γ j such that L g out ] M Lg f V. (19) Equaton 19 avods the per-pont evaluaton of the trple-product tensor n Equaton 18, offloadng ths computaton to a one-tme (per lghtng update) evaluaton of the trple-product when computng M L g; the run-tme now nvolves a smpler matrx-vector product followng by a dot-product. Note that we construct the product matrx for the lghtng, nstead of the BRDF, snce lghtng wll change at most once per-frame whereas the vew-evaluated 2 We drop the ] superscrpt for color channel ndexng, and assume that all equatons are appled to each color channel ndependently.

5 BRDF(s) changes at least once per frame (and potentally once per pont f we do not assume a dstant vewer model). We can further smplfy run-tme evaluaton f two of the three ntegrand terms are nown apror. For example, n the case of statc geometry and dffuse reflecton, the product T (x,ω) = V (x,ω) f + (x,ω) can be projected nto SH durng precomputaton, yeldng the followng run-tme computaton 3 T x ] = T (x,ω)y (ω)dω such that L g out T x ] Lg ], (20) whch corresponds to the standard PRT double product 11] and can be easly derved usng the orthonormalty property of SH. One detal we have not dscussed s how to compute the SH projecton of the vew-evaluated BRDF. We currently support the three common BRDF models n Table 1. Each of these BRDFs are crcularly symmetrc about an axs: the Lambertan clamped cosne s symmetrc about the normal n x, and the Phong and Mrror BRDFs are symmetrc about the reflecton vector ω r. At run-tme, the SH coeffcents of the BRDFs s computed usng Equaton 3 and the order-4 ZH coeffcents lsted n Table 1. We note that, n the case of perfect mrror reflecton, nstead of nducng a blur on the (very sharp) mrror reflecton lobe, we explot the fact that we have captured L env (ω) and can readly sample L g (ω). In ths case, we sample the SH-projected vsblty when reconstructng the global lghtng shade as L g out L g(ω r ) V y (ω r ). (21) Fgure 4 llustrates the dfferent BRDF components for an object shaded wth only the global lghtng. Dffuse Glossy Mrror Fgure 4: Shadng, ncludng soft shadows, from global/ambent lght (drectonal lght omtted) due to each BRDF component (> 70 FPS). Whle we have dscussed shadowng for drectonal lghtng, we have so far assumed that the SH projecton of bnary vsblty at x, V, was readly avalable for use n shadowng the global lghtng. Below we dscuss the dfferent ways we compute V. SH Vsblty. We dscuss dfferent shadowng models for global lghtng, dependng on the type of vrtual object we are shadng. For statc objects, we use ether standard PRT (Equaton 20) for dffuse objects or precomputed SH vsblty product matrces and Equaton 19 (replacng M L g wth a product matrx for vsblty). For anmatng or deformable objects, we segment shadowng nto two components: cast shadows from the object onto the envronment, and self shadows from the object onto tself. For cast shadows, we are motvated by prevous wor on sphercal blocer approxmatons 7]. We ft a small number (10 to 20) of spheres to our anmatng geometry (e.g., n ts rest pose for artculated characters) and sn ther postons durng anmaton. Shadowng s due to the sphercal proxy geometry, as opposed to the actual underlyng geometry. The use of sphercal blocer approxmaton s justfed n two ways. Frstly, snce the global lghtng component s a smooth sphercal functon, equvalent to large and broad area lght sources, fne scale geometry detals wll be lost n 3 Note that f + (x,ω o,ω) = f + (x,ω) n the case of dffuse reflectance. the smooth shadows that are produced by these types of area lghts. Secondly, SH vsblty can be effcently computed usng sphere blocers, as we wll now dscuss. Consderng only a sngle sphere blocer, the vsblty functon due to that blocer s a crcularly symmetrc functon, and so f we algn the vector from x to the center of the blocer wth the z axs, the sngle sphere vsblty s defned as n 7], { 0, f arccos(ω z) arcsn( r V s (ω) = d ) (22) 1, otherwse, and has ZH coeffcents V s = V s (ω) y(ω)dω (23) S 2 ( = ) 1 R, 1.5R, 2R 1 R, 0.6R( 4d 2 + 5r 2) ] d 2, where R = (r/d) 2. If we approxmate our anmated object wth a sngle sphere, applyng Equaton 3 to Equaton 23 wll yeld V, whch can be used n Equaton 19 to compute the fnal shade. However, t s rarely the case that a sngle blocer sphere s suffcent. In the case of multple spheres, SH products of the vsblty from ndvdual spheres must be taen to compute the total vsblty V. Instead of computng SH vsblty for a sphere at a tme and applyng Equaton 4 (whch requres expensve summaton of Γ, for each sphere), we compute an analytc product matrx for a sngle sphere blocer orented about an arbtrary axs by combnng Equatons 23, 3 and 5. In the case of only a few blocer spheres (e.g., f we only model cast shadows from the feet of a character onto the ground, usng two spheres), applyng the precomputed sphere blocng product matrx to L g (or f) and shadng wth Equaton 19 s an effcent soluton. However, as we ncrease the number of sphercal blocers (and thus, the number of product matrx multplcatons), performance degrades rapdly. In ths case, we accelerate ths SH mult-product by performng computaton n the logarthmc SH doman 7]. The technque of Ren et al. 7] computes the SH projecton of log(v s (ω)) (whch, n the canoncal orentaton, s stll a ZH) and tabulates these coeffcents as a functon of (r/d). Rotated log SH coeffcents are computed for each sphere blocer, usng Equaton 3, and summed together yeldng a net log SH vsblty vector, V log. We use the tabulated values provded by Ren et al. 7] and drectly apply the optmal lnear SH exponentaton operator to convert the net log SH vsblty to the fnal SH vsblty vector as ( Vlog ] ) ) V exp 0 / 4π (a V log 1 + b V log V log, (24) where V log s V log wth ts DC term set to 0, and 1 = ( 4π,0,...,0) s the SH projecton of the constant functon one(ω) = 1. For self-shadows, Ren et al. 7] propose sphere replacement rules for handlng shade ponts that le on the surface of the mesh (potentally wthn the volume of several blocer spheres), however we use a smpler soluton that yelds sutable results. We precompute and store the DC projecton of vsblty, whch s related to ambent occluson, at vertces of the dynamc object. At run-tme, we need only scale the unshadowed lghtng by ths occluson factor. Ths s equvalent to usng Equaton 19 wth only V 0 0; n the case of trple product ntegraton, f one of the terms has only a non-zero DC component, ths s equvalent to scalng the double product ntegraton by the DC component of the thrd (DC-only) term n ntegrand 9]. Unshadowed lght s computed by rotatng the approprate ZH vector from Table 1 usng Equaton 3, and then computng the double product ntegral (as n Equaton 20) of L g wth these rotated BRDF coeffcents. Fgure 5 llustrates the contrbuton of the dfferent shadowng components we use.

6 Unshadowed Cast Shadows Self-Shadows set of drectonal and envronment lghts), or by usng loopng constructs n a renderng sngle-pass. All of our results run at hgher than 70 FPS, ncludng all geometrc calbraton, lghtng and shadng computatons. We dd not focus much effort on optmzng our mplementaton: we use standard mplementatons of shadow mappng wth PCF, and we perform all SH shadng usng straghtforward mplementaton of the equatons ncluded n the exposton. We nterface drectly wth PTAM and ARToolKtPlus mplementatons provded onlne. Drect Shadow Sphere Proxes Fnal Image (wth both shadows) Fgure 5: Top row: smooth cast shadows (mddle) and self-shadows (rght) due to global lghtng gve subtle depth and lghtng cues compared to no shadows (left). Bottom row: drectonal shadows (left), vsualzng blocer spheres wth drectonal and cast shadows (mddle), and the fnal composted mage (rght; rendered at > 70 FPS). 7 IMPLEMENTATION AND PERFORMANCE We benchmar our system on an Intel Core2 Duo 2.8 GHz Laptop wth an NVda Quadro FX 770M GPU. We use the PlayStaton Eye camera for capture. Our end-to-end algorthm performs the followng computatons, mplementng the equatons drectly n GLSL: At ntalzaton, compute geometrc calbraton (Secton 3.2), Capture lghtng from mrror sphere or free roamng (Secton 4), Compute SH projecton of L env (see detals below; Equaton 6), Compute L d and L g usng lghtng factorzaton (Secton 5), Compute Lout d (see detals below; Secton 6.1 and Equaton 16), Compute L g out (Secton 6.2), and Composte drect and global shadng components. We compute the SH projecton of the captured lghtng by precomputng y(ω uv ) dω uv values n a texture and use graphcs hardware to qucly multply ths precomputed texture wth the captured lghtng mage, L env (ω uv ). Dependng on the sphercal parameterzaton (sphere vs. cube), we use the approprate defnton of dω uv as well as the approprate texture mage layout (.e., sx textures are requred for the cube map case). The summaton n Equaton 6 s computed wth a mult-pass sum on the GPU, manually mplementng a mp-map reducton where each pass averages 4 4 pxel blocs and results n an output texture of half the sze (per dmenson). Table 2 summarzes the lght projecton performance. Mrror Sphere Capture and Projecton Fnd Marers LDR to HDR SH Project Total 4 ms 0.6 ms 1.7 ms 6.3 ms Free Roamng Capture and Projecton Fnd Homography LDR to HDR Map to Cube SH Project Total 2 ms 0.8 ms 1.1 ms 1.8 ms 5.7 ms Table 2: Performance breadown for capturng and projectng realworld llumnaton nto SH usng our two capture methods. When computng Lout d wth Equaton 16, we use a bt shadow map, and 4 4 percentage-closer flterng 3]. All shadng computatons are performed on the GPU usng GLSL shaders wth lghtng coeffcents computed every frame, whch allows for dynamc lghtng response. We usually use a sngle drectonal lght and envronment lght, however the performance of our approach degrades lnearly wth addtonal lghts. In ths manner, we can support an arbtrary number of drectonal and envronment lghts usng ether mult-pass deferred renderng (one pass for each 8 DISCUSSION In all cases, we factor out a sngle drectonal lght from the captured envronment. Snce factorzaton taes place n the SH doman, the extracted drectonal lghtng nformaton s robust to hgh-frequency nose or moton artfacts n the raw captured lghtng sgnal. Such hgh-frequency ssues may only affect the qualty of mrror reflectons, where the captured lghtng s used wthout flterng, but no such artfacts arose n our examples. Prevous wor n AR relghtng has used subsets of the technques we mplement, wthout lghtng factorzaton, however an alternatve approach based on nstant radosty also targets smlar smooth and sharp shadng effects n AR applcatons 6]. Our approach shares the followng smlartes to ths prevous wor: both compute lghtng from the real envronment pror to compostng the shadng from vrtual objects, and both mae smlar assumptons about the BRDFs of the real world objects (nether technque properly handles speculartes from real objects). Whle more general (.e., color bleedng from vrtual objects can be approxmated), ths nstant radosty based approach requres hgher end hardware and stll acheves lower performance. In the case of statc geometry, we could ntegrate real objects nto the PRT preprocess, but ths would ncur smlar performance penaltes as the nstant radosty procedure. Our approach combnes lghtng factorzaton wth several dfferent renderng approaches, each targetng a specfc shadng component, n order to acheve a fnal consstent (albet approxmate) shadng result. The nstant radosty approach nstead apples a sngle renderng algorthm that models many dfferent lghtng paths. Both approaches have ther merts, and we opted for a lghter weght soluton based on a novel lghtng factorzaton. Explotng the nherent smoothness of SH projecton also allows us to reduce temporal artfacts that are commonly present n nstant radosty approaches. The qualty of our results are sutable for nteractve applcatons (e.g. gamng), where response rate s more mportant than physcal accuracy. We are able to handle dynamc changes n the lghtng and camera (see our vdeo), mantanng consstent shadng between real and vrtual objects. In a few scenaros, the extracted domnant lghtng drecton exhbts low-frequency nose durng anmaton; we are nvestgatng wndowng and flterng approaches to mantan a more robust temporal estmate of the domnant lghtng drecton. Lmtatons. Our factorzaton currently only supports extracton of a sngle drectonal lght. Whle ths can perform well n many ndoor (e.g. room wth a domnant lght source) and outdoor (e.g. forest envronment on a sunny day) scenaros, many realworld lghtng stuatons have several domnant lght sources. In these cases, our approaches folds all secondary domnant lghts nto the global lghtng term; combnng deas from Sloan s multlght extracton approach 9] wth our renderng algorthm s one approach to solve ths problem, however handlng temporal flcerng ssues may become a larger problem n these cases. Moreover, as mentoned earler, we do not support bounced lght from vrtual objects onto real objects. An approxmate soluton based on the sphere proxes 10] can be mplemented, however a more accurate proxy of the real-world geometry s necessary. Lastly, we have only appled rudmentary tone mappng to our nput

7 mages, and developng a more complete, nteractve hgh-dynamc range capture and renderng soluton s left to future wor. 9 CONCLUSIONS AND FUTURE WORK We presented a technque to factor envronmental lght emttng from, and bouncng off, the real-world. Our factorzaton s desgned to drectly support both hard and soft shadowng technques. Usng bass-space relghtng technques and GPU acceleraton, we can effcently compute shadows from both statc and anmated vrtual objects. The manner n whch we factor and combne dfferent llumnaton contrbutons s novel, and generates more consstent shadng than prevously possble (e.g., wth only the ndvdual applcaton of any one of the technques we ncorporate). Tradtonal cast shadows n AR are sharp and colored n a manner that does not respond to the surroundng lghtng, whereas our hard shadows are shaded based on dynamc envronmental ambent lght. Moreover, soft shadows due to resdual global lghtng add physcally-based smooth shadng, ncreasng perceptual consstency n a manner smlar to dffuse nterreflecton. Our factorzaton roughly dentfes a drect lghtng and ntellgent bounced lghtng terms, however we stll apply drectllumnaton ntegraton to these components, gnorng the effects of ndrect lght bounces from the vrtual objects onto the real-world geometry. In the future we plan on ncorporatng such effects by, for example, modelng lght bouncng off of the sphere proxes 10]. 13] D. Wagner and D. Schmalsteg. ARToolKtPlus for pose tracng on moble devces artoolt. Proceedngs of 12th Computer Vson Wnter Worshop (CVWW 07), ACKNOWLEDGEMENTS We than Paul Debevec for the lght probes, and Zhong Ren for the humanod mesh and tabulated log SH sphere blocer data. REFERENCES 1] P. Debevec. Renderng synthetc objects nto real scenes: brdgng tradtonal and mage-based graphcs wth global llumnaton and hgh dynamc range photography. In SIGGRAPH, NY, USA, ACM. 2] P. E. Debevec and J. Mal. Recoverng hgh dynamc range radance maps from photographs. In SIGGRAPH, NY, USA, ACM Press/Addson-Wesley Publshng Co. 3] R. Fernando. Percentage-closer soft shadows. In SIGGRAPH Setches, NY, USA, ACM. 4] S. Heymann, A. Smolc, K. Müller, and B. Froehlch. Illumnaton reconstructon from real-tme vdeo for nteractve augmented realty. Internatonal Worshop on Image Analyss for Multmeda Interactve Servces (WIAMIS 05), ] G. Klen and D. Murray. Parallel tracng and mappng for small AR worspaces. In Proc. 6th IEEE and ACM Internatonal Symposum on Mxed and Augmented Realty (ISMAR 07), Nara, Japan, Nov ] M. Knecht, C. Traxler, O. Mattausch, W. Purgathofer, and M. Wmmer. Dfferental nstant radosty for mxed realty. ISMAR 10: Proceedngs of the th IEEE Internatonal Symposum on Mxed and Augmented Realty, Oct ] Z. Ren, R. Wang, J. Snyder, K. Zhou, X. Lu, B. Sun, P.-P. Sloan, H. Bao, Q. Peng, and B. Guo. Real-tme soft shadows n dynamc scenes usng sphercal harmonc exponentaton. In SIGGRAPH, NY, USA, ACM. 8] I. Sato, Y. Sato, and K. Ieuch. Acqurng a radance dstrbuton to supermpose vrtual objects onto a real scene. IEEE Transactons on Vsualzaton and Computer Graphcs, ] P.-P. Sloan. Stupd sphercal harmoncs (SH) trcs. Game Developers Conference, ] P.-P. Sloan, N. K. Govndaraju, D. Nowrouzezahra, and J. Snyder. Image-based proxy accumulaton for real-tme soft global llumnaton. In Pacfc Graphcs, USA, IEEE Computer Socety. 11] P.-P. Sloan, J. Kautz, and J. Snyder. Precomputed radance transfer for real-tme renderng n dynamc, low-frequency lghtng envronments. In SIGGRAPH, NY, USA, ACM. 12] P.-P. Sloan, B. Luna, and J. Snyder. Local, deformable precomputed radance transfer. In SIGGRAPH, NY, USA, ACM.