Short Papers. Toward Accurate Recovery of Shape from Shading Under Diffuse Lighting 1 INTRODUCTION 2 PROBLEM FORMULATION

Transcription

1 1020 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER 1997 Short Papers Toward Accurate Recovery of Shape from Shadng Under Dffuse Lghtng A. James Stewart and Mchael S. Langer Abstract A new surface radance model for dffuse lghtng s presented whch ncorporates shadows, nterreflectons, and surface orentaton. An algorthm s presented that uses ths model to compute shape-from-shadng under dffuse lghtng. The algorthm s tested on both synthetc and real mages, and s found to perform more accurately than the only prevous algorthm for ths problem. Index Terms Shape-from-shadng, dffuse lghtng, nterreflectons, shadows, vsual events, horon, skylne. 1 INTRODUCTION THE classcal formulaton of the shape-from-shadng problem has been to assume that surface radance s determned entrely by the surface orentaton relatve to a pont lght source at nfnty [5], [7]. Shadows and nterreflectons are usually gnored. When the lght source s dffuse, however, ths model s not applcable [9], [10]. To see ths, consder the followng two examples: The frst s a scene consstng of a convex Lambertan object restng on a ground plane, llumnated from above by a unform hemspherc source. It s argued by Horn and Sjoberg [6] that, because of self-shadowng, the drect component of radance on the object s determned by the surface normal. Whle ths s true for the object, t s untrue for the ground plane. The drect component of radance on the ground plane vares contnuously because of cast shadows, wthout any change n surface normal. The second example s a sphercal concavty excavated from a ground plane. A surprsng result s that when shadowng, surface orentaton, and nterreflecton effects are all modeled, surface radance s constant wthn the concavty [11]. In partcular, the nteror of the concavty has the same local appearance under dffuse lghtng as would a plane under pontsource-at-nfnty lghtng. It s clear from these examples that, under dffuse lghtng, surface normal varatons are nether necessary nor suffcent for radance varatons. 2 PROBLEM FORMULATION We address the followng scenaro: Suppose a Lambertan surface wth albedo, r, s llumnated by a unform hemspherc source of radance, R src. We assume that the source does not tself reflect lght, nor s lght attenuated between the source and the surface. The surface radance, R(x), s then determned by the radosty equaton [13], [2]: r r Rbg x = RsrcNbg x u dw + Rc bx ughnbg x u d bg x P, W (1) p 9 p + bg x \ 9bg x where: x s a surface pont; N(x) s the surface normal; +(x) = {u : A.J. Stewart s wth the Department of Computer Scence, Unversty of Toronto, 10 Kngs College Road, Room SF4306, Toronto, Ontaro, Canada M5S 3G4. E-mal: jstewart@dgp.toronto.edu. M.S. Langer s wth NECI, 4 Independence Way, Prnceton, NJ E-mal: langer@research.nj.nec.com. Manuscrpt receved 1 May Recommended for acceptance by S.K. Nayar. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: tpam@computer.org, and reference IEEECS Log Number N(x) u > 0} s the hemsphere of outgong unt vectors; 9(x) s the set of unt drectons n whch the dffuse source s vsble from x; dw s an nfntesmal sold angle; P(x, u) s the surface pont vsble from x n drecton u (P denotes projecton ). 2.1 Interreflecton Model The frst contrbuton of ths paper s to approxmate (1) by removng the dependence on P(x, u). To ntutvely motvate the approxmaton, consder a sphere restng on a plane. Observe that the lower half of the sphere tends to be dark for two reasons: because of self-shadowng and because t s llumnated by darker ponts mmedately below t on the ground plane. Smlarly, ponts on the upper half of the sphere tend to be brght both because lttle self-shadowng occurs and because the nterreflectons that do occur are from dstant unshadowed ponts on the ground plane. A smlar observaton holds for a smooth depth map. From a hlltop, one mostly sees other hlltops, whle, wthn a valley, one mostly sees the valley. These observatons suggest the followng heurstc to smplfy (1): Under dffuse lghtng and constant albedo, a pont on a surface tends to be llumnated by other ponts havng smlar radance. Ths heurstc can be formaled as follows: Approxmate the rght hand sde of (1) by replacng the ncomng radance, R(P(x, u)), from other surfaces by the outgong radance, R(x), from x tself. An algebrac manpulaton yelds the nterreflecton model: 1 r Rsrc Nx bg udw p 9 x R1bg bg x. (2) F Nx bg I r udw p HG 9bg x The model s exact for the case of the sphercal concavty, snce surface radance s constant wthn such a concavty. It s also exact for a surface of arbtrary geometry n the lmt of r Æ 0, snce the nterreflecton component vanshes, as well as n the lmt of r Æ 1, snce R(x) Æ R src. For ntermedate albedos and more general surface shapes, we emprcally test the nterreflecton model by renderng depth maps wth a varant of the computer graphcs radosty algorthm [2]. Fg. 1 shows a rendered mage of a smooth depth map wth r = 0.5, along wth a scatter plot of R(x) versus R 1 (x). Note the hgh correlaton between the two. Smlar plots were observed for the other smooth depth maps tested. The R 1 model breaks down s some stuatons. For example, Fg. 2 shows a scatter plot of R 1 (x) versus R(x) for a scene consstng of a sphere sttng on a ground plane. (The scene was rendered usng the RADIANCE lghtng system [15].) A scatter plot s also shown. Two clusters occur, correspondng to ponts on the ground plane and ponts on the sphere. The R 1 model overestmates the radance of the ground plane but underestmates the radance of the sphere. 2.2 Image Formaton Model We assume that mages are formed as follows: Image ntenstes are lnearly related to surface radance (we gnore sensor calbraton ssues). The surface seen n the mage s Lambertan wth known albedo, r. The mage s formed under orthographc projecton. The depth map (x, y) s a contnuous, sngle-valued functon of (x, y). The source s a unform hemsphere centered about the lne of sght wth ts equator at depth ero. Surfaces outsde the mage doman do not cast shadows on surfaces wthn the mage doman. From these sx assumptons, t follows that the hghest mage ntensty, I max, corresponds to a surface pont that sees the KJ /97/$ IEEE

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER Fg. 1. A synthetc smooth depth map of a drapery was rendered usng the radosty equaton R(x) wth r = 0.5. The scatter plot compares R 1 (x) to R(x). entre hemspherc source [10]. Moreover, from (2), we have the followng mage formaton model: c h Ixy, HG 9 bg x bg 1 Imax Nx udw p 9bg x =. (3) F 1 1- r 1- Nx bg udw p The problem we address s how to compute a depth map (x, y) whch s consstent wth a gven nput mage, I(x, y), and wth the mage formaton model. 3 NEW ALGORITHM The only prevous algorthm for solvng the shape-from-shadng problem under dffuse lghtng s due to Langer and Zucker (LZ). The reader s referred to ther papers for detals [9], [10]. The algorthm we ntroduce s smlar n sprt to the LZ algorthm. Each pxel s assocated wth a node 1(x, y) n space, such that the depth value (x, y) of that node ncreases monotoncally from ero as the algorthm progresses. Over the course of the algorthm, each node descends untl t reaches the surface depth that satsfes (3). The key mprovement of the new algorthm over the LZ algorthm s that the new algorthm accounts for surface normal effects. It does so by usng a more accurate, contnuous representaton of the I KJ Fg. 2. A rendered mage of a sphere restng on a ground plane, and a scatter plot comparng R 1 (x) to R(x) n ths scene. depth varable (whereas the LZ algorthm uses dscrete depths wth a fxed nterval), a contnuous representaton of the vsble sky varable (whereas the LZ algorthm samples a hemcube), and a more sophstcated set of routnes for computng the depths and vsble sky. 3.1 Vsual Events The vsble sky, 9(x), of a node at poston x s represented by ts one-dmensonal boundary curve, whch we call the skylne. From the vewng assumptons that led to (3), t follows that a skylne s a sngle-valued functon of the amuth angle. The algorthm dvdes the skylne of a node by amuth angle nto a number of equal-se sectors. There are typcally 16 sectors, each spannng p radans. The algorthm approxmates the skylne 8 n each sector wth a constant elevaton angle, whch s the hghest elevaton angle of the surface nodes vsble n that sector. We mplctly assume that each surface node spans the entre sector. Accuracy can be traded for speed by ncreasng the number of sectors. See Fg. 3. As the depth of N ncreases, we must contnually mantan the skylne so as to determne, at any depth, whether (3) s satsfed. Rewrtng (3) to solate the vsble sky, 9(x), we get

3 1022 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER 1997 Fg. 3. The skylne around node 1 s dvded nto sectors. The skylne wthn each sector s approxmated by a constant elevaton angle, whch s the hghest elevaton angle of all surface nodes (gray n the fgure) n that sector. Fg. 5. Each sector of 1 stores a convex chan of nodes that can potentally appear on the skylne. Nodes below the chan cannot appear on the skylne and are not stored, whle nodes on the chan appear on the skylne n order of decreasng horontal dstance as 1 descends. In the fgure, the black nodes are on the convex chan. Fg. 4. As the depth of a node 1 ncreases, the surface defnng the skylne of 1 changes. A vsual event occurs at the pont of change (v n the fgure). The hgher node appears on the skylne when 1 s above v and the lower node appears when 1 s below. b gc h p 1- r Ixy, Ncxy,, h u dw =. (4) 9bxy,, g Imax - ricx, yh Consder one sector of the skylne of a node 1. As the depth of 1 ncreases, closer surface nodes n the sector obscure more dstant surface nodes, snce surface nodes reman fxed whle 1 descends and snce each surface node s assumed to span the whole sector. When the surface node of hghest elevaton wthn a sector becomes obscured, a vsual event occurs and the skylne of 1 undergoes a qualtatve change and must be updated (see Fg. 4). Vsual events were ntroduced by Kœndernk and van Doorn [8] and developed further by Plantnga and Dyer [12] and Ggus and Malk [4]. To facltate ths update, each sector of 1 stores the surface nodes that can potentally appear on the skylne, sorted by ncreasng horontal dstance. The nodes n ths lst form a convex chan, as shown n Fg. 5. Clearly, no node below the convex chan can appear on the skylne, snce t wll be obscured by a closer node. Also, the most dstant node n a convex chan has maxmum elevaton and appears on the skylne of the sector. As the depth of 1 ncreases, each node on the convex chan appears at the skylne n turn, n order of decreasng horontal dstance to 1. If a sector contans at least two surface nodes on ts convex chan, the hghest vsual event s computed n a straghtforward manner: Let 1 be the depth of the most dstant node on the chan. Let d 1 be ts horontal dstance from 1. Defne 2 and d 2 smlarly for the next-most-dstant node on the chan. Then, the vsual event occurs at depth 2 + d 2 ( 1-2 )/(d 2 - d 1 ). Each sector stores the depth of ts hghest vsual event. 3.2 Surface Events and Normal Estmaton A surface event occurs when a node 1(x, y) reaches a depth that satsfes (4). In ths secton, we dscuss how to determne, gven the skylne around 1(x, y). The rght sde of (4) s computed once for each (x, y). Assume, for the moment, that we can compute the ntegral on the left sde for any partcular value of. Then, the soluton of the equaton can be determned by bnary search, as follows: Let mn be a lower bound on and let max be an upper bound. That s, the ntegral s greater than (less than) the rght sde, when evaluated at mn ( max ). Let md = ( mn + max )/2. Evaluate the ntegral n (4) at md. If ths value s greater than the value of the rght sde, assgn md to mn ; otherwse, assgn md to max. Contnue untl the dfference between mn and max s below some threshold (we use 10-4 ). We next descrbe how to compute the ntegral on the left sde, gven a depth and a node 1(x, y) for whch the skylne s known. Frst, the surface normal N(x, y, ) s estmated by consderng the nodes n the eght-neghborhood of (x, y). For each par of such nodes whose fnal depths are already known, we determne the plane that passes through the two nodes and the pont (x, y, ). The normals of all such planes are averaged to yeld the estmate for N(x, y, ). If only one adjacent surface node exsts, we use the most upward pontng normal to the lne passng through (x, y, ) and the node (that s, the normal of most negatve component). If no surface nodes exst, we use (0, 0, -1) as the normal. Gven the normal, the ntegral s computed as follows. The skylne s dvded nto k sectors 0, 1, º, k - 1, such that sector spans the amuth angle 2 p 2 p, ( + 1) k k. Let f be the elevaton angle, measured from the vertcal, of the most dstant node 1 n the convex chan of sector as seen from the pont (x, y, ). Then, Ncxy,, h u dw ª 9bxy,, g k-1 kb + 1g f Ncxy,, h  sn cos, sn sn, cos sn d d c f q f q fh f f q = 0 = 0 k k-1 1 sn sn Ncxy,, h FF 2fI F 2fI I 2  f- DSIN, f - DCOS, sn f k, 2 = 0HGHG 2 KJ HG 2 KJ KJ where DSIN = sn 2 p + - ( 1) sn k k and DCOS = cos - cos ( + 1) k k are computed once and stored. 3.3 Prorty Queue of Events The algorthm mantans a prorty queue of vsual and surface events, where the prorty of an event s the depth at whch t occurs.

4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER For each pxel (x, y) of the mage, only the hghest event nvolvng the node 1(x, y) s stored n the queue. The algorthm proceeds by removng and processng events from the queue n order of ncreasng depth. When the queue becomes empty, all nodes wll have a depth that satsfes (4) and the algorthm termnates. Intally, nodes 1(x, y) correspondng to pxels wth maxmum ntensty are nserted nto the queue as surface events of prorty ero (recall that these nodes are assumed to have depth ero). As descrbed n Secton 3.5, removal of a surface event from the queue may cause more events to be nserted, so the se of the queue vares over tme. 3.4 Processng a Vsual Event Consder a vsual event that occurs at depth event n sector of node 1. When 1 falls below ths depth, the hghest surface node n sector becomes obscured by a closer surface node and 1 s skylne must be updated. The vsual event s processed as follows: 1) Delete the most dstant surface node n the convex chan of sector. 2) Calculate and store the depth of the next vsual event n sector. Ths depth depends on the depths and dstances of the two most dstant nodes on the convex chan. 3) Determne the sector j of node 1 wth the hghest vsual event. Ths may or may not be sector. Let vs be the depth of ths event. 4) Compute the ntegral on the left sde of (4) at vs. If ths ntegral s greater than the rght sde, nsert nto the prorty queue a new vsual event for sector j of node 1. Otherwse, a surface event must occur above vs ; n ths case, compute the depth of the surface event (as descrbed n Secton 3.2) and nsert nto the prorty queue a new surface event for Processng a Surface Event A surface event occurs to node 1 when (4) s satsfed. Node 1 wll reman statonary whle the remanng free nodes (those whose depth s not yet known) wll contnue to descend. Thus, node 1 may appear on the skylnes of remanng free nodes. For each remanng free node 1 free, perform the followng: 1) Determne the sector of 1 free n whch 1 appears. 2) If 1 s more dstant than the closest surface node on the convex chan of sector, then nothng needs to be done, snce 1 wll always be below the convex chan and so wll never appear on the skylne of 1 free. Otherwse, perform the followng steps: a) Add 1 to the front of the convex chan of secton. b) Addton of 1 may cause the chan to become nonconvex. Remove those nodes of the chan that are not vertces n the new convex hull. c) If the second-to-last node on the chan has changed, the hghest vsual event n the sector wll have changed. In ths case, remove from the prorty queue the event assocated wth node 1 free and compute a new event for that node (by performng on 1 free Steps 2 through 4 n Secton 3.4). Insert the new event nto the prorty queue. Related work n computatonal geometry has been reported by Cole and Sharr [3] and Bern et al. [1], who answer ray shootng queres from ponts on vertcal flghtpaths. However, we are nterested only n detectng those ponts on vertcal flghtpaths at whch the horon changes. Unlke the stuaton that they address, we do not have the depth map provded ahead of tme, and so cannot buld a complete data structure beforehand (as they do). Furthermore, whle they treat a sngle vertcal flghtpath, we are concerned wth one flghtpath per pxel. 4 RESULTS We now compare the performance of the LZ and the new algorthm on several test mages, each of se pxels. For each mage, we show a cross-secton of the real depth map and of the depth maps computed by the LZ and the new algorthms. Fg. 6 shows a gray-level mage of a hemsphercal concavty excavated from the ground (recall the example n Secton 1). Note the smooth, accurate depth map that results from the new algorthm, whch represents depth as a contnuum. Fg. 7 shows the results of the algorthm on the smooth depth map of Fg. 1. Observe that both algorthms underestmate the depth of central hlltop. Ths s due to a subtle ll-condtonng property of the shape-from-shadng-under-dffuse-lghtng problem. Small dfferences n mage ntensty of the brghtest ponts n the mage can correspond to relatvely large dfferences n depth. For example, over the entre surface, the depths range from three to 39 and the ntenstes range from 32 to 250. Pxel (x, y) = (15, 4), whch s a local ntensty maxmum, has depth four and ntensty 249, whle pxel (x, y) = (90, 71), also a local ntensty maxmum, has depth 13 and ntensty 247. In ths case, a 0.9 percent dfference n ntensty corresponds to a 25 percent dfference n depth. Such a small ntensty dfference s typcally lost n the mage nose. Observe that both algorthms overestmate the depth at the rght boundary of the mage. Because there s a local ntensty maxmum on the rght boundary, both algorthms assume the surface s horontal. (Snce both algorthms construct the depth map n order of ncreasng depth, the darker pxels below the local maxmum are not processed untl the local maxmum s fxed. In partcular, n the new algorthm, the deeper pxels cannot contrbute to the normal estmaton.) From the cross-secton, t s clear that the normal should be pontng leftward. Because of ths error, the surface receves less lght than the algorthms expect, so the algorthms nfer a smaller sold angle of the vsble source and, hence, a greater depth. Fnally, we compare the LZ and new algorthms on a real mage. Fg. 8 shows two eggs restng on a ground plane, vewed from above by a Sony CCD vdeo camera. The eggs and ground were panted gray and matte wth albedo 0.7. The scene and camera were surrounded by a large whte bed sheet, whch acted as a hemspherc dffuse lght source. Snce the dstance to the bed sheet (one meter) was much larger than the se of the scene, the nterreflectons between the bed sheet and surfaces were neglgble. Calbrated depth maps were obtaned drectly from the mages, usng the fact that eggs have one degree of rotatonal symmetry. Pxel nose was reduced to one gray-level out of 255 by averagng over multple mages. A range of camera apertures was used to verfy the lnearty of the sensor response. Vgnettng effects near the mage boundary were removed by dvdng the mage ntenstes by those of a second mage contanng only the gray ground. In Fg. 8, observe that the new algorthm was able to recover the heght of the ground plane more accurately than the LZ algorthm. Table 1 summares the mean and root-mean-square (RMS) errors for each of the mages dscussed above. Error s measured n pxel-wdth unts. The true depth maps were between 35 and 50 unts deep at ther deepest pont. The data show that the new algorthm performs better than the LZ algorthm. Further dscusson appears n [14]. TABLE 1 ERROR MEASUREMENTS (IN PIXEL-WIDTH UNITS) Scene Algorthm Mean Error RMS Error Sphercal New Concavty LZ Smooth New LZ Two Eggs New LZ

5 1024 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER 1997 Fg. 6. An analytcally computed mage of a sphercal concavty (r = 0.5) and a cross-secton of the depth map on row 50. Fg. 7. Fg. 1 (repeated) and a cross-secton of the depth map on row 50. (c) Fg. 8. A real mage of two eggs restng on a ground plane, vewed from above. Both eggs and ground were panted gray wth r = 0.7. Horontal slces through the actual and computed depth maps are shown for rows 36 and 75.

6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 9, SEPTEMBER The new algorthm took between 20 and 36 mnutes to process the 100 by 100 mages descrbed n ths secton. It was between 1.5 and three tmes slower than the LZ algorthm, dependng upon the mage. In the new algorthm, about 15 percent of the tme was spent processng vsual events, whle surface events took the remanng 85 percent. Each surface event s expensve because t nvolves nsertng a new surface node nto the skylne of every other node. In all, the algorthm performs about n 2 such nsertons for an mage of n pxels. Snce many of the nsertons do not result n a new node on the convex chan of the skylne (Step 2 n Secton 3.5), an nterestng problem s to develop an algorthm whch only performs the necessary nsertons. [12] H. Plantnga and C.R. Dyer, Vsblty, Occluson, and the Aspect Graph, Int l J. Computer Vson, vol. 5, no. 2, pp , [13] R. Segel and J.R. Howell, Thermal Radaton Heat Transfer. Hemsphere Publshng, [14] A.J. Stewart and M.S. Langer, Towards Accurate Recovery of Shape from Shadng Under Dffuse Lghtng, Proc. IEEE Conf. Computer Vson and Pattern Recognton, pp , June [15] G. Ward, The RADIANCE Lghtng Smulaton and Renderng System, Computer Graphcs (SIGGRAPH 94), pp , July CONCLUSION The mprovements of our algorthm over the LZ algorthm n the drapery and real egg scenes are modest. Ths suggests that the three types of errors we have dscussed namely, approxmatons n the mage formaton model, poor local constrants at the mage boundary, and ll-condtonng of the problem tself provde an ultmate bound on the algorthm s performance. Moreover, we dd not consder the case that the albedo was estmated ncorrectly or that the hemspherc source was not algned wth the vewng drecton. We expect these to ntroduce further errors n the computed results. Those warnngs asde, we should keep n mnd that we have restrcted ourselves to shadng nformaton alone. We expect that by combnng our algorthm wth one based on stereo, moton, or other cues, t would be possble to obtan more accurate results than what we have acheved. Ths remans a subject for future work. ACKNOWLEDGMENTS James Stewart s work s supported by the Informaton Technology Research Centre of Ontaro, the Natural Scences and Engneerng Research Councl of Canada, and the Unversty of Toronto. Mchael Langer s work was partally supported by grants from NSERC and AFOSR, whle he was wth the Center for Intellgent Machnes, McGll Unversty. The authors would lke to thank Davd Walt and the revewers for helpful comments. REFERENCES [1] M. Bern, D. Dobkn, D. Eppsten, and R. Grossman, Vsblty wth a Movng Pont of Vew, Algorthmca, vol. 11, pp , [2] M. Cohen and D. Greenberg, The Hem-Cube: A Radosty Soluton for Complex Envronments, Computer Graphcs SIGGRAPH 85), B.A. Barsky, ed., vol. 19, pp , Aug [3] R. Cole and M. Sharr, Vsblty Problems for Polyhedral Terrans, J. Symbolc Computng, vol. 7, pp , [4] Z. Ggus and J. Malk, Computng the Aspect Graph for the Lne Drawngs of Polyhedral Objects, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 12, no. 2, Feb [5] B.K.P. Horn, Obtanng Shape from Shadng Informaton, The Psychology of Computer Vson. New York: McGraw-Hll, [6] B.K.P. Horn and R.W. Sjoberg, Calculatng the Reflectance Map, Appled Optcs, vol. 18, no. 11, pp. 1,770-1,779, [7] Shape from Shadng, B.K.P. Horn and M.J. Brooks, eds. Cambrdge, Mass.: MIT Press, [8] J.J. Koendernk and A.J. van Doorn, The Sngulartes of the Vsual Mappng, Bologcal Cybernetcs, vol. 24, pp , [9] M.S. Langer and S.W. Zucker, Dffuse Shadng, Vsblty Felds, and the Geometry of Ambent Lght, Proc. Fourth Int l Conf. Computer Vson, pp , Berln, [10] M.S. Langer and S.W. Zucker, Shape-from-Shadng on a Cloudy Day, J. Optcal Soc. Am. A, vol. 11, no. 2, pp , [11] P.H. Moon and D.E. Spencer, The Photc Feld. Cambrdge, Mass.: MIT Press, 1981.