An Introduction to Image-based 3D Surface Reconstruction and a Survey of Photometric Stereo Methods

Transcription

1 3D Res. 2, 03(2011) /3DRes.03(2011)4 3DR REVIEW w An Intoducton to Image-based 3D Suface Reconstucton and a Suvey of Photometc Steeo Methods Steffen Hebot Chstan Wöhle Receved: 21Febuay 2011 / Revsed: 20 Mach 2011 / Accepted: 11 May D Reseach Cente, Kwangwoon Unvesty and Spnge 2011 Abstact 1 Ths pape povdes an ntoducton to photometc methods fo mage-based 3D shape econstucton and a suvey of photometc steeo technques. We begn wth taxonomy of actve and passve shape acquston technques. Then we descbe the methodcal backgound of photometc 3D econstucton, defne the canoncal settng of photometc steeo (Lambetan suface eflectance, paallel ncdent lght, known llumnaton decton, known suface albedo, absence of cast shadows), dscuss the 3D econstucton of sufaces fom local gadents, summaze the concept of the bdectonal eflectance dstbuton functon (BRDF), and outlne seveal mpotant empcally and physcally motvated eflectance models. We povde a detaled teatment of seveal genealzatons of the canoncal settng of photometc steeo, namely non-dstant lght souces, unknown llumnaton dectons, and, n some detal, non- Lambetan suface eflectance functons. An mpotant specal case s puely specula eflectons, whee an extended lght souce allows captung a suface that conssts of pefectly specula suface patches. Lnea combnatons of puely Lambetan and puely specula eflectance components ae favoably used fo econstuctng smooth sufaces and also human skn. Nonunfom suface eflectance popetes ae estmated based on a smultaneous 3D econstucton and detemnaton of the locally vaable paametes of the eflectance functon based on a multtude of mages. Assumng faceted sufaces, the effectve esoluton of the 3D econstucton esult can be nceased to some extent beyond that of the undelyng mages. Othe appoaches sepaate specula and dffuse eflectance components based on polazaton data o colo nfomaton. The specula eflectons can be used addtonally to estmate the decton fom whch the suface s llumnated. Fnally, we descbe methods to S. Hebot 1 C. Wöhle 1 ( ) 1 Image Analyss Goup, TU Dotmund Unvesty Otto-Hahn-St. 4, Dotmund, Gemany emal: webmaste@chstan-woehle.de combne photometc 3D econstucton technques wth actve and passve tangulaton-based appoaches. Keywods 3D suface econstucton, shape analyss, suvey, Photometc Steeo 1. Intoducton The poblem of depth data econstucton fom one o seveal two-dmensonal mages has extensvely been examned snce Hon s ntoducton of Shape fom Shadng n The poblem as such s vey complex due to the dvesty of each of the nvolved pats. These concen the object tself, ts envonment, and the senso that s used fo data acquston. Hstocally, Hon consdeed the stongly specalzed case of an object that emanates stctly dffuse eflectons, a fully calbated dstant sngle pont lght souce fo llumnaton, and a monochome camea wth known poston and oentaton. Ths settng needs genealzatons of most of ts components, f eal sufaces ae consdeed. Genealzatons thus egad (1) the object, whch may eflect dffusely, speculaly, unde occuence of subsuface scatteng, show nte-eflectons, move, and/o change ts fom, etc. (2) The llumnaton may consst of one o seveal known o unknown pont lght souce(s), dstant and non-dstant lght souce(s), extended lght souce(s), lase(s), global llumnaton, etc. (3) The senso type fnally may captue chomatc, monochome, polazed, etc. data unde senso-specfc esponse cuves, and may be sngle, multple, fxed, o movng. Ths huge amount of vaatons gves an nsght nto the possbltes and dffcultes of depth econstucton, and explans the wde ange of eseach that has been caed out n that feld. Solvng the depth data econstucton task povdes computes wth the ablty to undestand thee-dmensonal scenes and objects on the bass of two-dmensonal data wth wdespead applcatons angng fom obotcs and ndustal qualty nspecton ove human-machne nteacton though e.g. gestue and face ecognton to

2 2 3D Res. 2, 03(2011)4 moves and achtectual applcatons. Ths pape ntally evews basc pncples,.e. photometcally and geometcally motvated appoaches, and then examnes genealzed appoaches that am fo a econstucton of abtay sufaces. Detaled attenton s pad to ecent publcatons, and patculaly photo-metc steeo algothms that deal wth unknown lght souces and non- Lambetan sufaces ae examned. Geometc notatons have been used accodng to the ecommendatons by Ncodemus et al. 67 o have been adopted fom the coespondng contbutons n the lteatue. These ae shown fo efeence n Appendx A Taxonomy of 3D Shape Acquston Technques The man delneaton between popula 3D shape acquston technques egads the actve o passve chaacte. Culess 13 descbed that actve appoaches nteact wth the object by physcal contact, o pojecton of some knd of enegy. Although passve technques as well poject enegy upon the suface,.e. n the fom of the pojected lght fom (dstant) pont lght souces, these methods ultmately ely on the optcal appeaance of an object unde non-tghtly focused llumnaton. In contast, actve technques examne the nteacton (e.g. pesence, absence, defomaton) of the suface wth an enegy sgnal. A summay and taxonomy s shown n Fg. 1. The systematc ovevew s gatheed fom the elaboate evew by Culess 13 and Lanman and Taubn 51, and supplemented accodng to Setz 75 and Debevec 15. Fo a detaled ovevew and futhe nfomaton egadng the ndvdual technques, please efe to the cted papes. Fo ths pape, the taxonomy seves as an ovevew of exstng technques and ntends a categozaton of the technques that ae dscussed n the subsequent sectons. Fg. 1 Taxonomy of actve and passve 3D shape acquston technques. The soldly undelned technques ae dscussed boadly n ths pape, the technques undelned wth a boken lne ae only dscussed wth espect to the pncples. The pape stats wth a descpton of the semnal woks Shape fom Shadng (SfS) and Photometc Steeo (PS) n Sectons 2.1 and 2.2, espectvely. Secton 2.3 then befly dscusses appoaches fo depth econstucton fom local gadents that ae obtaned by photometc technques. Secton 3 egads common altenatves to photometc

3 3D Res. 2, 03(2011)4 3 appoaches, namely actve and passve tangulaton methods (Secton 3.1), stuctued lght, and tme of flght technques (Secton 3.2). The secton closes wth a compason of the advantages and dsadvantages of photometc methods n contast to passve and actve tangulaton appoaches (Secton 3.3). A geneal teatment of eflectance phenomena s gven n Secton 4, whle Secton 5 focuses on genealzatons of photometc appoaches egadng non-dstant lght souces (Secton 5.1), unknown llumnaton dectons (Secton 5.2), and non-lambetan sufaces (Secton 5.4). The case of non- Lambetan sufaces s examned n geate detal egadng puely specula eflectons (Secton 5.4.1), combnatons of eflecton components (Secton 5.4.2), and sepaaton of eflecton types (Secton 5.4.3). Secton 6 evews some algothms that combne absolute depth data fom tangulaton-based appoaches wth photometc data. Fnally, the pape s concluded n Secton Photometc Methods fo 3D Suface Reconstucton Photometc appoaches fo 3D suface econstucton geneally nhet the pncple of appeaance analyss of a thee-dmensonal object on ts two-dmensonal mage. Based on the ntensty nfomaton, these appoaches attempt to nfe the shape of the depcted object. The appeaance/ntensty nfomaton tself emeges fom the eflectance popetes of the object s suface n nteacton wth the lght souce(s) n the object s envonment. Snce a two-dmensonal mage geneally lacks the complete nfomaton that s needed fo the depth econstucton pocess to be unambguous 4, the poblem as such s llposed, and any obtaned esult fom any algothm theefoe depends on the appoach-specfc combnaton of (egulazaton) constants and po knowledge. In the followng, the undelyng pncples of the fundamental photometc methods Shape fom Shadng (Secton 2.1) and Photometc Steeo (Secton 2.2) ae explaned and dscussed. Whle both examne ntensty values, the man dstncton les n the numbe of avalable mages (one fo SfS, seveal fo PS) and the equed constants. The followng ntoducton to SfS does not attempt to summaze the whole banch of eseach of that feld, but athe seves as an ntoducton to the topc and a foundaton fo the explanaton of Photometc Steeo (Secton 2.2). 2.1 Shape fom Shadng (SfS) The fst examnaton of the poblem of shape econstucton has been addessed by Hon 37. He consdeed the case of a lagely known envonment wth dstant pont lght souce llumnaton of known stength, poston, and decton. The SfS statement as such s llposed and theefoe, addtonal constants ae equed to solve t unquely Backgound The theoetcal backgound of SfS s centeed on the descpton of dffusely eflectng sufaces by Lambet s law 50, whch connects the obseved ntensty I wth llumnaton decton l, llumnaton stength, suface oentaton n and suface albedo α : I = α l n (1) wth obseved ntensty I R, suface albedo α R, llumnaton ntensty R, llumnaton decton T 3 1 l = (lx,ly,lz ) R, and suface nomal v T 3 1 n = (nx,ny,nz ) R. Clealy, the peceved lght ntensty eflected fom an llumnated suface element s popotonal to the effectve aea of that patch vsble fom the souce and ndependent of the obsevaton decton v. Hstocally 44, Hon appled hs chaactestc stp expanson method fo solvng the (nonlnea fst-ode patal dffeental) mage ntensty equaton 41. The appoach has been efned 38-40, 42 though e.g. the ntoducton of a eflectance map fo efnng the esults of the mage ntensty equaton. The maps ae used to geneate a look-up table that s ndexed by bghtness values and etuns suface oentaton values The Canoncal Settng Fg. 2 Shape fom Shadng pncple: Detemnaton of suface gadents (suface nomals) fom the appeaance on an mage. Applyng Lambet s Law mposes seveal estctons upon the expemental setup. The most smplfed fom s the canoncal settng of SfS: (1) The object suface s stctly Lambetan. (2) Dstant lght souces,.e. object extents << lght souce dstance. 3 1 (3) The llumnaton decton l R and ntensty s known. (4) Othogaphc mage pojecton (dstant camea poston). (5) The suface albedo α ( x, y) s known. (6) Nethe cast not attached shadows fall upon the object suface. (7) Suface shape does not poduce nte-eflectons.

4 4 3D Res. 2, 03(2011)4 The ctcal step that fnally detemnes the suface oentaton fo each ntensty value conssts of solvng the mage ntensty equaton fo the suface nomal v T decton n = (n,n,n. Howeve, even wth the x y z ) llumnaton decton l, the suface albedo α ( x, y), and the ntensty value I beng known, the equaton cannot be solved unquely due to beng undedetemned. Ths s a consequence snce only one ntensty value I(x, y) s avalable pe pxel, but two ndependent nomal dectons z z p =, q = have to be detemned. Theefoe, x y addtonal constants need to be ntoduced. These ae e.g. the assumpton of a globally constant albedo α ( x, y) = α, the nomalzaton of the z-component of n v such v T that n = ( p, q,l), and/o constants egadng the gadent space such as smoothness constants o a selfshadow bounday as appled by Ikeuch and Hon 45. Ths ncopoaton of addtonal (egulazaton) constants poses the fundamental dffeence between SfS and PS, whch s usually ndependent of addtonal constants due to a suffcently lage numbe of ntensty measuements fo each pxel. Detals egadng possble constants and egadng the pefomance of dffeent SfS algothms can be obtaned fom the Shape fom Shadng suveys by Zhang et al. 106 and, moe ecently, by Duoua et al. 22. Snce SfS analyzes the amount of eflected lght fom an object s suface, the appoach s geneally vulneable aganst all phenomena that nfluence the ntensty values n the object s mage. These ae the object s eflecton-, the lght souces llumnaton and the senso s senstvty popetes. Reflectons that have been examned n the context of SfS egad the smplest case of dffuse eflectons at Lambetan sufaces, specula eflectons at glang sufaces, nte-eflectons, and/o eflectons caused by subsuface scatteng. The ctcal pont s that each eflecton mechansm causes dffeent amounts of lght to be obseved fom a gven vewpont unde a gven llumnaton decton and llumnaton stength. Appoaches that specfcally attempt to deal wth nondffuse eflectons ae examned n the followng sectons. Futhe nfluence of the ntensty values ases fom how the object s llumnated,.e. the numbe, lght decton, and adaton chaactestcs of the lght souces. Ultmately, the epesentaton of the depth nfomaton of the shape s gven ethe by actual depth values z(x, y), v T 3 1 suface nomals n = (nx,ny,nz ) R, suface gadents T 2 1 g(x,y) = (p,q) R, o the suface gadent angles slant ϕ and tlt θ. Snce depth data always needs a efeence pont fom whch ts values ae measued, ethe the camea poston o some abtaly defned xy-plane seves as the ogn 106. It s mpotant to note that SfS detemnes suface oentatons nstead of absolute depth values as t s the case fo tangulaton-based appoaches (see Secton 3). 2.2 Photometc Steeo The canoncal SfS settng has been fundamentally changed by Woodham n hs semnal wok 102 though the acquston of N mages I wth { 0... N 1} of an object unde the 3 1 same vewpont v R wth vayng llumnaton 3 1 dectons l R wth llumnaton stengths R. Ths appoach s commonly known as Photometc Steeo (PS). In contast to SfS, ths esolves the poblem of an undedetemned equaton system fo the elatonshp between suface oentaton and bghtness: I 0 l 0 0 I1 1 l1.. = α n = α L n (2).... I N 1 N 1 l N 1 Note, that fo L to be nvetble, ts ows (.e. llumnaton dectons) need to be lnealy ndependent. In the case of moe than thee acqued mages, L s ove detemned, but the equaton can then be solved n the least mean squaes sense usng the pseudo nvese of L (e.g. 46, 70 ). The acquston of mages s tadtonally acheved sequentally n tme multplex, but fequency multplexed captung by applcaton of coloed lghts as appled by e.g. Henández et al. 34 s possble as well and benefcal fo movng objects. Once suface gadents ae obtaned, these need to be ntegated to fom actual object shapes. Ths poses the last yet mpotant step fo the suface econstucton based on photometc cues. Some appoaches ae dscussed n the followng secton. 2.3 Reconstucton of the Object Shape fom Local Gadents Once suface nomals ae obtaned fom e.g. photometc appoaches such as SfS o PS, the fnal shape of the object needs to be detemned based on the suface gadents. Apat fom the classcal chaactestc stp expanson appoach by Hon 37, Zhang et al. 106 evewed dffeent appoaches towads suface shape detemnaton va SfS and categozed these nto mnmzaton, popagaton, local, and lnea appoaches. Agawal et al. 1 then examned, that a whole contnuum fo the soluton of non-ntegable gadent felds becomes avalable. They pont out that the degee of vaaton n the possble solutons esults fom ansotopc weghts appled to the gadents dung the ntegaton pocess. On those bases, they analyze the set of possble solutons usng α-sufaces, M-estmatos and edgepesevng egulazaton, and dffuson. The expemental esults show that the applcaton of ansotopc weghts, as t s the case fo dffuson, mpoves the econstucton detal whle suppessng nose. Snce the pesent suvey manly focuses on methods fo suface gadent eteval, the descpton of the popetes of the poposed solutons s omtted, but detals egadng these and elated appoaches can be obtaned fom e.g. 1 and 106.

5 3D Res. 2, 03(2011) Altenatve Shape Acquston Technques The followng llustaton seves as a efeence fo compason and atng of the stengths and weaknesses of the photometc shape acquston appoaches dscussed thoughout ths pape. Fo a geneal undestandng of elevant and conceptually nteestng technques, the followng dscusson focuses on actve tangulaton, passve tangulaton, and (actve) tme-of-flght technques. The man conceptual and pefomance-elated dffeences between these appoaches and the above mentoned photometc appoaches ae emphaszed thoughout the llustaton. 3.1 Actve and Passve Tangulaton Methods Actve tangulaton pojects a lase sgnal upon the object suface and then detects t wth a sutable senso (mostly a dgtal camea, see Fg. 3 fo an llustaton of the setup). The technque as such s called actve tangulaton, snce the lase beam ogn, the camea, and the pont whee the beam meets the object fom tangle fom whch the depth data s obtaned usng geometcal elatons. The exactness of tangulaton-based ange scannng methods s lmted snce the depth accuacy only eaches a fxed multple of the sample spacng and usually a sgnfcant amount of hgh fequency nose appeas Stuctued Lght and Tme-of-Flght Apat fom actve tangulaton, a class of lase-based systems has emeged that explots the suface shape dependent defomaton of lght pattens. These systems use stuctued lghtng to poject a known lght-patten (stpes, ccles, etc.) upon an object s suface and then detemne the suface shape fom the defomaton of these pattens 69. Ths (actve) pojecton of lght pattens upon a suface s closely elated to a suface beng (passvely) textued wth a known patten, snce n both cases; pattens ae pojectvely defomed accodng to the shape of the suface. Fo the latte case, Gadng 25 examned Shape fom Textue by analyss of peodc ccle pattens on object sufaces, whch explots the same geneal pncples as stuctued lght analyss. Geneally, stuctued lght technques ae found to lack the hgh fequency detal of photometc appoaches. Howeve, ecent publcatons show stong mpovements 23, 69. Tme-of-flght technques ae based on the fnte speed of lght, whch then detemnes the nonzeo ound-tp tme t t of a lase pulse fom the lase to the object and back. The dstance between the lase emtte and the object s t t then computed by s = c, wth c beng the speed of 2 lght. Geneal dffcultes of tme-of-flght technques esult e.g. fom the pesence of moton snce the measuements ae acqued n tme multplex (see the followng secton fo detals). Fg. 3 Pncple of tangulaton technques: Geometcal elatons ae used to obtan the depth nfomaton. Actve tangulaton (see mage) uses a lase and a camea wth known postons fo the tangulaton, passve tangulaton employs a second camea nstead of the lase. Image souce: Wkpeda, LasePncple.png, publc doman lcense. Passve tangulaton apples the same pncple of geometc tangles, but the actve component (lase scanne) s exchanged wth a second camea (passve component). The man challenge of these algothms les n the dentfcaton of coespondng ponts n both mages. The establshment of coespondences s especally challengng n the absence of suface textue, whee the dentfcaton of featues becomes dffcult. Smla to actve tangulaton, passve tangulaton suffes fom hgh fequency nose n the depth measuements, but fnds obust absolute depth values wth lttle low-fequency nose. Fg. 4 Genealzed Bas-Relef Ambguty: Wth unknown lghtnng condtons, t s always possble to fnd an llumnaton condton that makes dffeent suface pofles appea dentcal n the espectve mages. 3.3 Geneal Compason of Photometc Methods vs. Tangulaton Technques The advantages of lase-based systems ove photometc appoaches nclude wde ndependence fom ambent llumnaton, and the acquston of absolute depth values. A geneal dsadvantage esults fom the tme multplex unde whch the depth data s acqued, snce ths causes poblems fo movng objects. Moton compensaton as a post-pocessng fo objects that have moved dung the acquston poves to be dffcult, snce t causes much of the hgh fequency detal to get lost 95. The man dffeence between photometc and tangulaton appoaches n geneal les n the accuacy of hgh and low fequency depth data. Whle the photometc

6 6 3D Res. 2, 03(2011)4 appoaches pefom well fo hgh-fequency detals, low fequency bases may coupt the appeaance of the object. Tangulaton technques on the othe hand show obust low fequency data, but suffe fom hgh fequency nose. Some of the combnatons of both types that explot the complementay stengths ae dscussed n Secton 6. Passve tangulaton appoaches show futhe dffcultes egadng the calbaton of the system. Snce photometc appoaches usually apply a sngle camea, only one camea needs to be calbated. Passve tangulaton technques ae geneally vulneable aganst mutual dsplacements of the cameas, snce a dslocaton of the ep-pola lnes poduces eos n the econstucton pocess. Futhe dffcultes ae found n aeas whee the object s un-textued. Fo these egons, passve algothms ae not capable of detemnng pecsely located coespondences between the mages fom both cameas. In contast to passve tangulaton technques, actve tangulaton technques and photometc technques do not suffe fom such poblems egadng un-textued egons. Seveal challenges emege fom the object s suface type, whch concens all of the mentoned appoaches. Non- Lambetan sufaces mpact the suface nomal detemnaton fo photometc appoaches due to potentally stong changes n the appeaance unde small vewng o llumnaton decton changes. Passve tangulaton methods ae faced wth moe complcated mechansms fo coespondence establshment on non- Lambetan sufaces, snce the appeaance of the object potentally changes sgnfcantly between the mages fom the espectve cameas. Actve tangulaton methods fnally need to deal wth lase beams that ae eflected away fom athe than nto the camea, whch lacks the avalablty of a measuement sgnal. Theefoe, each of the mentoned appoaches faces poblems wth non-lambetan sufaces, but dffeent undelyng mechansms ae esponsble fo that behavo. Table 1 Summay: Compason of the stengths and weaknesses of photometc, passve and actve tangulaton technques. The man dffeences egad econstucton of hgh fequency detals, calbaton expenses, and object suface types. Although non-lambetan sufaces ae challengng fo each class of technques, the actual dffcultes ase fom dffeent mechansms as dscussed n Secton 3.3. cteon photometc passve tangulaton actve tangulaton Object shape data suface nomals absolute depth data absolute depth data Hgh fequency detals yes no scanne-dependent Hgh fequency nose vey low hgh scanne-dependent Low fequency bas lkely vey low vey low Camea 1 fxed 2+ fxed o 1+ movng 1 Un-textued aeas OK ambguous esults OK Calbaton 1 camea 2+ cameas 1+ cameas, 1 lase Illumnaton mpotant mno mpotance mpotant Object movement OK OK poblematc non-lambetan sufaces possble challengng challengng Manly due to the advantage egadng hgh fequency detals, photometc appoaches show wde applcaton and ntated a wde feld of eseach. Howeve, the analyss of non-lambetan sufaces s challengng and a majo lack n the pefomance of photometc appoaches. Recent and past attempts fo the applcablty to abtay (non- Lambetan) sufaces ae dscussed n detal n the followng sectons. A summay of the man advantages and dsadvantages s gven n Table Reflectance Phenomena The amount of lght that s eflected fom the suface depends on the suface mateal as well as the llumnaton and vewng condtons. Fo dffeent classes of suface mateals, a vaety of empcally o physcally motvated eflectance models have been ntoduced. Howeve, snce the feld of eflectance modelng s vey boad and nsped a wde aea of eseach, t s not possble to summaze that feld wthn ths pape. Nevetheless, snce suface econstuctons of non-lambetan sufaces possbly eque eflectance modelng (see e.g. Secton 5.4.2), a geneal ntoducton and an ovevew of basc yet cucal concepts s gven n the followng. Fo a bette ovevew of dffeent eflectance functons, the eflectance and scatteng functons dscussed by Weych et al. 96 ae shown n Fg The Bdectonal Reflectance Dstbuton Functon (BRDF) Fg. 5 Taxonomy of eflectance functon types accodng to Weych et al. 96. Notatons: BRDF - Bdectonal Reflectance Dstbuton Functon, SS - Subsuface Scatteng, SV - Spatally Vayng. A commonly appled eflectance functon s the case of a Bdectonal Reflectance Dstbuton Functon (BRDF), whee the amount of ncomng lght (adance E ) s elated to the amount of outgong lght (adance L 0 ) wth espect to ncdent angles (azmuth ϕ, elevaton/ nclnaton/pola angle θ ), and emsson angles ( ϕ, θ ). BRDFs ae thus 4D-functons

7 3D Res. 2, 03(2011)4 7 Lo BDRF = = f (θ, ϕ,θ, ϕ ) (3) E Although BRDFs aleady ae a specal case of the geneal eflectance case, whch ncludes seveal addtonal dependences, the geneal pncple of eflectance descpton dependng on cetan paametes becomes clea. Ealy appoaches to descbe a BRDF appled lowpaametc appoxmaton models fo dffeent eflectance phenomena 3. These can be classfed nto empcally motvated models (e.g. Lambetan model, Phong s model 71 ) and physcally motvated models (e.g. Wad 94, Toance-Spaow 92, Cook-Toance 12 ). These appoaches assume that the eflectons ae descbed by the sum of one o two specula and a dffuse component. A summay of the chaactestcs of the thee eflecton components specula spke, specula lobe, and Lambetan component s gven by Naya et al. 64. Futhe smplfcatons have been employed by Noth Coleman, J. and Jan 68 and Basky and Petou 6, who assume a naow angula extent of the specula lobe, whch allows teatng specula eflectons as outles, but affects the coectness of the BRDFs to an even geate extent. (a) (b) Fg. 6 (a) Appeaance acheved by Phong s model 71. The model conssts of a sum of an ambent component (left), a dffuse component (second fom the left), and a specula component (second fom the ght) and yelds an empcally enhanced esult (ght). (b) Commonly consdeed eflectance phenomena n photometc algothms. Naya et al. 64 publshed a systematc examnaton of the dffeences between models based on ay optcs (Toance- Spaow model 92 ) and computatonally moe expensve models based on Maxwell s theoy of electomagnetc waves (Beckmann-Spzzchno model 8 ). The latte one s usually consdeed f the wavelength s not anymoe small compaed to the sze of the suface mcostuctues. Fo the Toance-Spaow model, they fnd that the specula lobe s a good appoxmaton to the specula lobe descbed by the Beckmann-Spzzchno model. Howeve, the Toance- Spaow model s found to be nexact n the descpton of the specula spke component 62. The man poblem of BRDF descpton apat fom choosng an appopate model les n the detemnaton of the functon paametes. Methods fo the extacton of dffuse and specula eflecton popetes fom multple mages unde dect llumnaton ae e.g. Maschne s PhD-Thess 57 and the wok by Sato et al. 74. The latte appoach manly focuses on sotopc dffuse and specula eflectons. An extended appoach fo modelng and measung ansotopc eflectance has been poposed by Wad 94. An appoach fo the extacton of eflecton coeffcents fom a macoscopc scene has been pesented by Yu et al The algothm employs a (spase) set of calbated HDR mages of a scene wth known dect llumnaton, and addtonally known geomety of the scene. A specalty of ths appoach s that dffuse and specula nte-eflectons ae consdeed and not teated as outles as t s the case fo most appoaches. Yu et al. 103 apply an teatve scheme to compute the dffuse and specula eflecton paametes of a low-paametc model fo each egon n the mage. Ths nfomaton s then used to obtan a hgh esoluton dffuse albedo map fo the whole scene. The evaluatons on mages fom a synthetc scene show quanttatvely small eos n the estmaton of the eflecton coeffcents. A second evaluaton on a eal wold example wth no gound tuth fo a quanttatve eo analyss beng avalable showed qualtatvely good esults, e.g. f the scene s endeed wth alteed lghtng condtons of addtonal vtual objects. Pevous wok on nteeflectons between dffusely eflectng sufaces can be obtaned fom Naya et al. 63. Moe ecently, measung eflectance functons has ganed populaty snce ecent techncal advances now povde the possblty of such appoaches 59. Weych et al. 96 ague that the concept of measued eflectance data compses a geate vaety of optcal phenomena than modeled data. The followng secton thus coves some of the ecent appoaches towads measued eflectance popetes. As mentoned by Szelsk 81 and suveyed by Weych et al. 96, a lagely genealzed case fo eflectance descpton s the Bdectonal Subsuface-Scatteng Reflectance-Dstbuton Functon (BSSRDF). The BSSRDF descbes the amount of lght that s obseved

8 8 3D Res. 2, 03(2011)4 dependng on vewpont (θo, ϕ o ), llumnaton decton (θ, ϕ ), outgong adance pont (x o, y o ) and ncdent adance pont (x, y ) : BSSRDF = S(x, y,θ, ϕ, xo, y o,θo, ϕo ) (4) Although ths models a complex envonment, the appoach s fa fom geneal and futhe effects can be consdeed, whch nclude wavelength-dependent eflectance, wavelength dependent scatteng, fluoescence 2, o phosphoescence Hstocally, eflectance functon measuement evolved wth consdeably less degees of feedom as the BSSRDF: Whte et al. 97 measued the eflectance popetes usng a gonoeflectomete dependng on the angles of ncdence (θ, ϕ ) and the angles of eflectance (θ, ϕ ).e. a BRDF (see Secton 4). Wth the consdeaton of the azmuth ϕ and ϕ, the appoach allows modelng ansotopc mateals, whch oftentmes eman unconsdeed when applyng the above-mentoned low-paametc models. As a sub-categoy of ansotopc mateals, eflectance popetes of sotopc mateals (mateals wthout suface gan ) only depend on the absolute dffeence n the azmuth angles ϕ = ϕ ϕo. A geneal dawback of gonoeflectometes les n the tme that s needed to fully captue the BRDF of some mateal, snce only one sample at a tme s obtaned 96. Wth the avalablty of hgh esoluton dgtal camea sensos, the eflecton popety sample ate gew mmensely snce the acquston of BRDF data compses a whole set of angles ove a sngle mage. Wad 94 used one of such settngs fo the acquston of ansotopc BRDFs. He used plana samples fo the acquston, whch yelds an easy geomety n the estmaton pocess. Late, Maschne et al. 55 captued the BRDF fom cuved objects wth known shape. The appoach has been extended to abtay shapes 56, whch then allows captung the BRDF of human skn. Howeve, the latte pocedue eques the po knowledge of the shape befoe the BRDF can be detemned. Snce such a shape detemnaton algothm (e.g. (actve) tangulaton steeo) nduces addtonal naccuaces, the nceased genealty comes at the cost of a loss n pecson compason to objects wth known shape. Fo sotopc mateals, the object can be exchanged wth a metallc sphee (agan unde contolled lghtng and a camea as a captung devce), as pesented by Matusk et al. 58. In a sees of publcatons, Do et al pusued the goal of suface mateal dentfcaton fom sngle object mages. In contast to the above mentoned appoaches on BRDF measuement, they focused on the dentfcaton of the mateal that has been captued athe than ts exact eflectance functon. They showed that coase eflectance popetes aleady povde the equed nfomaton to classfy a mateal based on a set of pevously detemned popetes. Alldn 3 publshed esults on BRDF estmaton unde natual llumnaton by applcaton of sphecal hamoncs, whch possbly yeld bette appoxmatons than low-paametc models. To account fo the uncontolled lghtng, a metallc sphee that seves as a lght pobe s used to ecove the lghtng envonment. Aftewads, seveal mages of an object wth known shape ae taken as t has aleady been pefomed n pevously descbed methods. The sphecal hamonc appoxmaton to the BRDF s obtaned by solvng a lnea equaton system based on known lghtng, known geomety, and known adance. The po knowledge of the examned objects concens shape (pefectly sphecal), mateal (completely homogeneous), sotopy, and neglgble subsuface scatteng. Alldn s esults show that hs sphecal hamoncs appoxmaton poduces undesed atfacts and lacks n the epesentaton of speculates. 4.2 Non-paametc Reflectance and BRDF Independence An ealy BRDF-ndependent appoach 11 has aleady been dscussed n ths pape along wth the case of non-dstant llumnaton (Secton 5.1). An altenatve appoach that featues BRDF ndependence has been ntoduced by Zckle et al. 108 by explotaton of the Helmholtz ecpocty 93. Alldn et al. 2 ntoduce an appoach that smultaneously ecoves shape and spatally vayng eflectance - an explct appeaance model - fom a set of mages. They apply the Photometc Steeo pncple (sngle camea and multple mages unde vayng but known llumnaton) to econstuct sufaces wth spatally vayng sotopc eflectance wthout the equement of a paametc model. Intally, the suface nomal econstucton poblem s educed to a sngle degee of feedom as ntoduced by Alldn and Kegman 4 by the assumpton of an sotopc mateal. Then, the BRDF s educed egadng ts doman fom 4D to two dmensons by applcaton of a nonpaametc b-vaate appoxmaton smla to Stak et al. 80. The b-vaate BRDF s epesented as a lnea combnaton of known bass mateal BRDFs smla to Lawence et al. 52 wth constants egadng smoothness and monotoncty ove the BRDF s doman. The pactcal examnaton equed a qute lage set of mages fo the econstucton of the evaluated objects (> 100), whch s not always avalable fo ealstc poblems. Howeve, the esults show that the macostuctue and the meso-stuctue of the object ae ecoveed accuately, and that new vews of the object can be endeed wth nea photogaphc qualty. In ode to account fo almost abtay eflecton popetes of examned mateals, the placement of a efeence object of known shape n the feld of vew of the camea along wth the examned unknown object has been poposed by Slve 78. The appoach has been eexamned by Hetzmann and Setz 35-36, who appled metallc sphees as efeence objects. The appoach eques that the efeence object mateal and the mateal of the unknown object ae dentcal o at least show the same eflectance popetes. Once ths condton s fulflled, the efeence object seves as a efeence to detemne the connecton between suface oentatons and appeaance. 2 Wavelength of ncdent lght and emtted lght may dffe 3 Possble tme delay between lght ncdence and emsson

9 3D Res. 2, 03(2011) Genealzatons of the Canoncal Settng The canoncal settng mposes estctons upon the llumnaton and the photometc popetes of the object suface. In the followng, appoaches ae dscussed that genealze that settng to at least a cetan extent. Statng wth the case of non-dstant o geneally unknown lght souces, a boad dscusson of appoaches that handle non- Lambetan sufaces s pesented fom Secton 5.4 onwads. 5.1 Genealzaton 1: Non-Dstant Lght Souces Clak 11 ntoduced a method temed Actve Photometc Steeo, whch estmates absolute depth values fom mages that have been acqued unde contolled movement of a pont lght souce located elatvely close to the object suface. In contast to the appoach by Iwaho et al. 47, who as well appled pont lght souces located nea the object suface, Clak s method only eques solvng a lnea equaton (see below) and s futhemoe not estcted to objects wth a Lambetan suface. Instead of just usng absolute mage adance values E I, the poposed algothm apples the adance changes oe I that esult fom the lght souce movement. The deved equaton fo the computaton of the absolute depth Z then esults n T ( oei ) o + 2E Z = T (5) ( oei ) p wth o beng the tanslaton vecto of the pont lght souce n the camea-lens-centeed wold coodnate system XYZ, x y T p = (,, l) ndcates the poston of the mage plane f f T pont p I = (x, y) wth espect to the wold coodnate system, and E E E (,, E o I = ) denotes the adance o x o y o z change ate wth espect to the lght souce poston changes. Note that Eq. 5 s ndependent of the eflectance popetes of the suface, whch s a stong advantage ove othe appoaches, and t mplctly contans the decay of the lght souce adaton stength wth the ecpocal squaed lght tavel dstance. Clak s evaluaton of the algothm showed a stong dependence on the mage nose, whch can be educed to a cetan extent by acquston of multple mages and applcaton of a weghted least squaes appoxmaton to the esultng depth values, o, n the based case, a medan flte. Futhemoe, the exactness of the algothm suffes fom satuated bghtness values n the mage. In the followng secton, the case of non-dstant lght souces s evoked. Instead, the genealzaton of the canoncal SfS settng (see Secton 2.1.2) egadng the known llumnaton decton s dscussed. 5.2 Genealzaton 2: Unknown Illumnaton Decton(s) The assumpton of unknown llumnaton dectons l wth < l R has been analyzed by Hayakawa 31, who examned the applcaton of Photometc Steeo to a setup wth abtaly movng llumnaton. The man nsght of Hayakawa s wok s, that the detemnaton of the llumnaton decton and the llumnaton ntensty fom a set of mages s not unque and thus eques the ntoducton of constants. Fo that poblem, Hayakawa poposes two possble constants that ae capable of esolvng the ambgutes: Both constants eque sx ponts wth known o constant suface eflectance o known o constant lght souce ntensty. Wth these constants, the lght decton can be detemned unquely and theefoe allows suface econstucton algothms to wok wthout that po knowledge. Addtonally, Hayakawa ntoduces a scheme fo dealng wth cast and attached shadows on the object s sufaces by successvely emovng the affected mage ows and columns, whch ae then flled wth sutable estmates. The geneal ambguty poblem has been examned n geate detal by Belhumeu et al. 9, who concluded that wthout knowng the lght decton, one can only solve up to the so-called Genealzed Bas-Relef (GBR) ambguty: An object s vsble suface f(x, y) s theefoe ndstngushable fom an mage unde a genealzed baselef tansfomaton f(x, y) = λ f(x, y) + µ x + νy,.e. dentcal to a scaled, e.g. flattened, veson λ f(x, y) of the suface tself wth an added plane µ x + νy see Fg. 4 fo a >0 geneal explanaton, λ R, µ, ν R. Solutons fo the poblem that do not eque sx known ponts have been poposed by Yulle et al. 104 though ntoducng and enfocng Feeman s genec vewpont constant 24. Futhe appoaches have been ntoduced by Dbohlav and Saa 17 though explotaton of speculates and thus enfocng the consstent vewpont constant. In the wok, the assumpton of a Lambetan suface wth supemposed mo-lke eflectons s appled and speculates ae analyzed as addtonal geometc constants that solve the poblem of unknown lght souces. The suface econstucton then teats specula eflectons as outles and pefoms classc Lambetan PS. Ultmately, fou mages obtaned unde vayng lght postons suffce to esolve the ambguty and compute the shape of the object at hand. A conceptually smla appoach usng the Toance- Spaow eflectance model 92 has been ntoduced by Geoghades 26, but allows solvng the GBR ambguty only up to the bnay convex/concave ambguty. Late, Dbohlav and Chantle 16 educed the numbe of equed specula pxels (.e. numbe of mages captued wth dffeent lght postons) fom fou as equed by Dbohlav and Saa 17 to two. Fo the case of sngle lght souces n each mage, non-unfom albedo and only mo-lke specula eflectons, they ultmately popose a lnea scheme to solve the GBR based on only two specula hghlghts. Tan et al. 85 genealzed the concluson by Dbohlav and Chantle 16 egadng the fom of the

10 10 3D Res. 2, 03(2011)4 eflectance functon and showed that the GBR ambguty s esolvable usng an abtay sotopc and spatally nvaant non-lambetan eflectance functon and explotng the Helmholtz ecpocty 93. The appoach s evaluated on eal and synthetc mages and compaed wth a gound tuth obtaned fom a calbated PS appoach. The econstucted sufaces show a low aveage nomal eo, whle the angula eo of the ecoveed llumnaton dectons anges fo one pesented dataset fom Futhe nsght s gven by Tan and Zckle 82 along wth a pojectve famewok based on the eal pojectve plane that allows the analyss of eflectance symmetes usng sotopy and ecpocty, o half-vecto symmety. Fo uncalbated PS, they show that constants esultng fom the analyss of sotopy and ecpocty n a sngle mage allow esolvng the GBR ambguty. The appoach s based upon Tan et al. 85 and mpoves that technque by educng the numbe of mages used by fom two to one. Bas et al. 7 show that Photometc Steeo can be pefomed unde unknown llumnaton condtons wth the sole constant of dstant lght souces. The llumnaton of each mage may even nclude an abtay combnaton of dffuse souces, pont souces, and extended souces. Unde these llumnaton types and Lambetan sufaces, they apply sphecal hamoncs to model the lghtng condtons. The esults show low eos n the estmated suface nomals aganst a efeence shape obtaned wth a lase scanne. The numbe of equed mages ses fom fou mages fo a fst ode sphecal hamonc appoxmaton to nne mages fo a second ode sphecal hamonc appoxmaton wth small effect on the suface estmaton eo. Howeve, the appoxmaton only apples low-ode sphecal hamoncs, whch ae not sutable fo the epesentaton of specula eflectance. Detemnaton of envonmental lghtng has futhe been studed by Shen and Tan 76, who examned feely avalable ntenet mages of a scene wth vayng vewpont, nose, and unknown global llumnaton based on Bas et al. 7 and Ramamooth 73. Fo these mages, they compute the global llumnaton and ultmately classfy the mage-specfc weathe condtons based on the llumnaton chaactestcs. An appoach usng nteeflectons on non-lambetan sufaces wth non-tanslatonal symmety has been examned by Chandake et al. 10. They pove mathematcally that nteeflectons contan the equed nfomaton to esolve the GBR ambguty and then show expemental esults fo nose-fee endeed and eal mages. The esults show low eos n compason to (1) esults wth neglected nteeflectons, (2) to the algothm by Naya et al. 63 that teatvely compensates fo nteeflectons, and (3) to a gound tuth. Alldn et al. 5 have shown, that a mnmzaton of the entopy of the suface albedo dstbuton as well esolves the ambguty, even wthout geometcal consdeatons. Zhou and Tan 107 ecently ntoduced a method fo Lambetan sufaces that apples a ng lght souce,.e. lght souces that ae dstbuted unfomly ove a ng placed aound the camea. Ths povdes addtonal po knowledge s.th. the GBR s solvable up to the ng lght ambguty,.e. a scalng, a vetcal mong, and a hypebolc o ccula otaton. Fo the emanng tansfomatons, Zhou and Tan deve and dscuss combnatons of constants, whch solve the ambgutes successfully. In compason to a gound tuth establshed usng metallc sphee calbaton, the esults fom the nglght PS appoach appea qualtatvely vey good. Ultmately, the appoach allows eleasng dffeent scene constants such as ntegablty and thus extends the applcablty to moe geneal scenes. Anothe ecent algothm by Sh et al. 77 ntally analyze (colo) mages of Lambetan sufaces to dentfy ponts wth equal nomals and albedoes, whch then allows to esolve the GBR ambguty based on the nfomaton fom at least fou pxels wth the same albedo but dffeent nomals. The expements povde esults fo the successful detemnaton of suface nomal goups, albedo goups and, addtonally, the adometc esponse functon of the camea. The angula eo of the detemned nomals s epoted to be sgnfcantly lowe n compason to Alldn et al. 5 as a efeence method fo GBR dsambguaton. Henandez and Vogatzs 32 ntoduced a self-calbatng appoach fo a monocula facal captung system. The actual shape eteval s based on mult-spectal Photometc Steeo (MSPS), whch needs to be calbated po to ts applcaton to a human face. The calbaton stage apples an ntal stuctue-fom-moton algothm to obtan the moton n the scene and then uses a multvewsteeo algothm to estmate the coase shape of the face. The coase 3D shape povdes suffcent nfomaton egadng the nomals and ntenstes to estmate the lghtng paametes and thus calbate the system. Snce the lghtng paametes vay between dffeent faces, the calbaton pocedue needs to be epeated befoe captung an unknown face. Afte calbaton, fequency-multplexed mages ae captued that povde the shape eteval data equed fo the algothm descbed by Henández et al. 34. Fequency-multplexed mages do not suffe fom uncompensated moton lke tadtonal tme-multplexed Photometc Steeo mages. Afte shadow pocessng accodng to Henandez et al. 33, the obtaned face econstucton esults show good hgh-fequency detal but suffe fom the low fequency nose that s typcal fo photometc appoaches. 5.3 Genealzaton 3: Pespectve Pojecton Pados and Faugeas 72 developed a new fomulaton of the SfS poblem unde consdeaton of the pnhole camea model and thus account fo pespectve pojectons. Consequentally, a new patal dffeental equaton emeges, whose vscosty soluton s found to be unque, exact, and convegng usng the poposed appoxmaton scheme and numecal algothm. The expemental evaluaton shows stong advantages ove the othogonal appoxmaton and obustness aganst ntensty nose and aganst eos n the llumnaton decton. Tankus et al. 89 ntally dscussed the sgnfcance of the econstucton eo that occus f a pojectvely obtaned mage s econstucted unde the othogaphc assumpton. Futhemoe, they deve the mage adance equaton smla to 72 based on the natual logathm of the depth ln (z) The soluton of the depth z based on the pojectve famewok s found to be nvaant to scale changes

11 3D Res. 2, 03(2011)4 11 z = c z wth z R, whch s moe plausble than the othogaphc famewok s ndependence fom tanslatons z = z + c. They popose a scheme to solve fo the depth z usng a Fast Machng algothm (FM) based on 49. A compason between the solutons obtaned by pojectve FM, othogaphc FM, and vscosty solutons shows advantages of the pespectve algothms ove the othogaphc algothm as epoted by Pados and Faugeas 72. Although the pojectve FM soluton and the vscosty soluton pefom smla egadng suface econstucton exactness, FM shows pactcal advantages e.g. egadng speed of convegence and ndependence of specal bounday condtons. 5.4 Genealzaton 4: Non-Lambetan Sufaces In the canoncal settng (see Secton 2.1.2), the case of stctly dffuse sufaces (Lambetan sufaces) among known lghtng dectons and dstant lght souces s assumed and has been appled n that setup by the ponees Hon 38 and Woodham 102. The genealzaton of that case eques coect handlng of vaous eflecton effects that occu addtonally to dffuse eflectons. In the followng, appoaches that consde the smultaneous appeaance of dffuse and specula eflecton components ae dscussed. In the begnnng, the focus les upon methods that cove the case of puely specula eflectance and combnatons of specula and dffuse eflectance. Then, methods fo the sepaaton of both eflecton components ae dscussed. Aftewads, the descpton of eflectance by modelng o measung the eflectance popetes s descbed. The case of abtay sotopc eflectance s gven addtonal mpotance due to the genealty of that case Puely Specula Reflectons An appoach fo dealng wth puely specula sufaces has been ntoduced by Ikeuch 44. It altes the llumnaton settng of PS fom pont souces as appled by Hon 38 and Woodham 102, and used an extended lght souce,.e. lght that s eflected fom a plana Lambetan suface, whch n tun s llumnated by one of thee dstnct lnea lamps. In contast to pont lght souces, ths allows captung a suface that conssts of pefectly specula suface patches wth vayng oentatons. Unde the assumpton that the ncdent lght s exclusvely eflected speculaly, Ikeuch deves a model fo the eflectons at the object s suface. The connecton between the ncdent adance L and the eflected adance L e fom a sngle pont on the Lambetan adato s gven by L e (θe, ϕ e ) = L ( θ, ϕ + π ) (6) Afte mage captung, the ntenstes ae elated to the ntegated adance ove the whole adato aea. Fo the detemnaton of suface oentatons, Ikeuch appled the Photometc Steeo technque by Woodham 102, whch employs a numbe of eflectance maps equal to the numbe of llumnaton souces. Fo the elaxaton pocess, two appoaches have been appled, whee the fst one aveages the esults fom two dstnct suface oentaton lookups and the second one apples a suface smoothness constant n combnaton wth an adance constant 43. Snce Ikeuch s techncal equpment yelded coase 128px 128px data, the esults ae athe qualtatve, but t s shown that the technques allows the extacton of easonable suface oentatons fom the examned objects. Late, Moel et al. 60 poposed a econstucton method based on polazaton data. In the expements, a dome lght s employed to povde (almost) omndectonal llumnaton, whch allows explotng Fesnel s fomulae fo suface gadent eteval. The esults have been compaed to the data obtaned by a ange scannng system, fo whch the object had to be coveed wth a dffuse coatng. The poposed algothm has been found to povde a qualtatvely bette esult, especally n the coss-sectonal pofle of the object unde examnaton Lnea Combnaton of Dffeent Reflecton Components The pevously dscussed Lambetan assumpton (Secton 2) and the pue speculaty assumpton (Secton 5.4.1) ae both estcted n the applcaton, snce they ae specalzatons of a geneal case. Futhe genealzaton of eflectance phenomena compses a combnaton of a Lambetan and a specula component, wth addtonal ncluson of a specula lobe component fo ough sufaces 44. In the followng, the genealzaton egads the smultaneous descpton of two effects at each pont of a suface a puely specula spke component and a Lambetan component. Although Naya et al. 62 ntally dscusses sufaces wth thee eflecton components, they fnally examne the case of smooth sufaces, fo whch the eflectons ae modeled as a lnea combnaton of a Lambetan and a specula spke component. Fo the acquston of the ntensty mages, multple extended lght souces wth unfom dstbuton aound the object ae used, snce they have aleady poven well-suted fo captung sufaces wth puely specula eflectance (see 44 and Secton 5.4.1). An extended lght souce boadens the angula extent of the specula spke and addtonally balances the elatve ntenstes of specula and dffuse eflectons. Both of these effects ae advantageous fo the detecton pocess snce the specula spke s vsble unde a lage ange of angles and allows measung both eflectons n the same mage. Naya et al. 61 then pesent an algothm fo the smultaneous extacton of suface oentatons and eflectance paametes, based on a samplng constant and a unque oentaton constant. The esults fom examnatons of plastc and metal objects of known shape show that the mean suface oentaton eo amounts to Futhe detals and tests on futhe sufaces can be obtaned fom the authos techncal epot Geoghades modeled the specula (lobe) eflectance behavo usng the Toance-Spaow eflectance model 92 n addton to a Lambetan tem. The model complexty s educed unde the assumpton of small dectonal dffeences between vewng decton v and llumnaton decton l, whch allows neglectng the bstatc shadowng facto and the Fesnel eflectvty. The

12 12 3D Res. 2, 03(2011)4 econstucton as such s an extenson of the eale wok 27, whch teatvely detemnes suface nomals, Lambetan albedo, lght souces, and paametes of the Toance-Spaow eflectance model n the least squae sense. They demonstate the pefomance of the appoach wth lacqueed sufaces and human faces, whch both beneft fom accountng fo specula lobes. The ecoveed paametes fo human skn ae compaable to the measued paametes obtaned by Maschne et al. 56 and thus egaded plausble. Goldman et al ncluded the sotopc Wad model 94 nto a PS famewok to estmate spatally vayng BRDFs whle smultaneously ecoveng the object shape. They assume that the examned mateal s composed of a small set of fundamental mateals, whose combnaton allows modelng the suface at hand. The econstucton algothm then mnmzes an eo measue fo nomals, mateal paametes, and mateal weght maps. The ecoveed endeed mages show low RMS eos and plausble mateal weghts and paametes but show some ambgutes at specula hghlghts due to ove ftted mateal eflecton models. Although the appoach stll suffes fom some uncetantes, t s an mpotant step towads the econstucton of abtay mateals and objects. Tan et al. 86 poposed an appoach that ecoves the shape of non-lambetan sufaces wth a esoluton geate than the undelyng mage esoluton. As n the ntal wok on that topc 84, they extend the descpton of suface eflectvty such that moe than a sngle suface facet pe pxel may contbute to the obseved bghtness. When then solvng fo an optmal fne esoluton suface facet dstbuton wth addtonal suface constants, a esoluton enhanced suface s successfully obtaned. The pocess as such eques a qute lage numbe of mages ( ) unde vayng llumnaton dectons n ode to solve fo the suface. The authos pesent esults fo synthetc and eal sufaces at 2px 2px and 4px 4px esoluton enhancement. These show educed angula nomal eos n compason to an a po avalable gound tuth and exhbt the expected ncease n hgh fequency suface detal, but possbly contan addtonal nose Sepaaton of Specula and Dffuse Components Fo a sepaaton of the specula lobe and the dffuse component (the specula spke s assumed to be neglgble), an examnaton of the chaactestc popetes of each eflecton type s needed fo the explotaton. The thee pncpal dffeences ae Tan and Ikeuch 88 : (1) Specula eflectons have a lage degee of polazaton than dffuse eflectons. (2) The ntensty dstbutons of dffuse eflectons appoxmately follow Lambet s Law, the ntensty dstbutons of specula (lobe) eflectons follow the Toance-Spaow model 92 o the Beckmann-Spzzchno model 8. (3) In the vsual lght spectum, specula eflectons have been found to be lagely ndependent of the object suface s spectal eflectance,.e. t shows spectal powe dstbuton (.e. colo) ndependence. In contast, the spectal powe dstbuton of dffuse eflectons depends on the object s body spectal eflectance, whch shows stonge wavelength-dependence egadng the vsual spectum. A method that uses the fst popety (polazaton) fo the dentfcaton of specula eflectons has been pesented by e.g. Wolff 100 and Wolff and Boult 101, who explot that specula eflectons show a hghe degee of polazaton D than dffuse eflectons. D can be obtaned fom e.g. the hghest obseved ntensty value I max (x,y) = I(x,y,ϕ max ) and the lowest obseved ntensty value I mn (x,y) = I(x,y,ϕ mn ) of a pxel (x, y) n an mage wth espect to the polazaton flte oentaton ϕ : Imax Imn DOP =, 0 D 1 (7) Imax + Imn The pncple has been extended usng polazaton and colo by Naya et al. 65. Algothms that use colo nfomaton allow a educton of the equed mages fo eflecton component sepaaton to a sngle mage. A method fo the sepaaton of specula and dffuse eflecton components on the bass of one sngle colo mage has been ntoduced by Tan and Ikeuch The assumptons eque a chomatc suface colo ( R G B ),.e. not a black, whte, o gay colo. In the specula-to-dffuse mechansm, the llumnaton colo of the nput mage s ntally nomalzed to obtan pue whte specula components. Aftewads, the mage pxels ae examned n a maxmum chomatcty ove mage ntensty space, whee the specula and dffuse components can be sepaated by a scala theshold value. Although the authos clam that the specula fee mage has an exactly dentcal geometcal pofle as the dffuse component, the evaluated esults show eos of up to 35 ntensty steps, whch ae 13.7% fo 8 bt mages. Tan et al. 83 poposed a sngle-colo-mage hghlght emoval algothm that estmates the undelyng dffuse colo by llumnaton-constaned n-pantng. Constants ae obtaned fom the patal dffuse colo nfomaton contaned n specula hghlghts and the llumnaton colo unfomty, whch can be deved patally fom a chomatcty analyss. The expements show an mpovement of tadtonal vecto fttng based o total vaaton based algothms due to an mpoved ecovey of obscued textues. Both, mages wth lage scale textue and mages wth detaled textue ae pocessed successfully. Mallck et al. 54 popose anothe sepaaton method fo dchomatc sufaces and extend the examnaton by pefomng PS on the extacted dffuse pat. In contast to the appoaches descbed above, thee mages ae acqued as necessay fo PS and analyzed fo specula hghlghts. These ae dentfed and sepaated fom the dffuse component by a data-dependent otaton of the RGB colospace. The mpotant popety of that SUV - colospace s the pesevaton of shadng nfomaton, whch ultmately allows applyng PS. Late, the appoach of Tan and Ikeuch has been extended by Thomas and Sugmoto 90 though elmnaton of Tan and Ikeuch s llumnaton constant and thus allow the numbe and espectve decton of the lght souces to be unknown. Whle the algothm ams fo the egstaton of ange mages, the numbe of acqued mages can be ased uncondtonally to two and thus suface nomals can