3D Sene Reonstruton Sstem from Multple Snhronzed Vdeo Images aewoo Han 1, Juho Lee 2, Hung S. Yang 3 AIM Lab., EE/CS Dept., KAIS 1,2,3 373-1, Guseong-dong, Yuseong-gu, Daejon, Republ of Korea { bluebrd 1, jhlee 2, hsang 3 }@paradse.kast.a.kr Abstrat One of the goals of three-dmensonal (3D) omputer graphs s to reate vrtual realst mages of dnamall hangng senes. In ths paper, we desgned and mplemented a sstem that proesses multple snhronzed vdeo sequenes and generates 3D renderng of dnam objets n real tme. he vdeomage-based vrtual volumetr sene reonstruton sstem aqures snhronzed multple vdeo mages and renders dnam real-world senes. It nludes an effent mage-based reonstruton sheme that omputes and shades 3D objets from slhouette mages, as well as t nludes a slhouette extraton sheme that s robust to llumnaton hange. he proposed sstem s relatvel low-ost and does not requre foregong an speal hardware or spef envronment. Ke words: sene reonstruton, vsual hull, magebased renderng 1. Introduton In omputer graphs and omputer vson, volumetr sene reonstruton of a 3D model from multple twodmensonal (2D) photograph mages s alread an old and one of the most mportant problems. It remans one of the most dffult problems. But, man researhers have been workng on the reaton of vrtual senes from mages n man applatons, suh as vrtual realt, games, multmeda, robot navgaton, and speal effets for movng ptures [1]. Generall, the volumetr struture of a sene an be reonstruted f ts materal haratersts, llumnaton, and geometr onstrants are arefull onsdered. he methods used to aqure 3D nformaton on dnamall hangng senes are lassfed nto two approahes, e.g., the atve and the passve method aordng to the tpes of magng sensors. Atve approahes use strutured lghts or laser sanners to dretl aqure 3D nformaton about the subjet [2]. he produe hgh qualt data sne the emtted lasers or lghts dretl obtan the range to the parts of the subjet. However, the equpment used n atve approahes s almost onsderabl expensve and have phsal restrtons suh as partular lght and speal pantng. As well, the are not adequate to apture dnamall hangng senes n real-tme as t takes a long tme to get data for one sene. Compared to atve approahes, passve approahes extrat 3D data ndretl wthout ontat wth the objet. Utlzng the pture mages taken from ameras s representatve of passve approahes [1][3][4]. radtonall extratng 3D data usng passve magng s less aurate than usng expensve atve sensors. Also t needs dramat omputng tme. he most deal sstem must be able to onstrut hghqualt mages n a short tme usng low-ost equpment. Moreover, t should not be restrted to an envronment and should have a broad feld of applaton that enompasses even suh areas as sports, dane, and remote vdeo-onferenng. In ths paper, we desgned and mplemented a sstem that aqures snhronzed multple vdeo mages and reonstruts vrtual senes ost-effetvel usng the slhouette mages. he sstem should be apable of realtme snhronous apturng of amera vdeo mages, amera albraton, and slhouette extraton that s nvarant to llumnaton hange. It omputes and shades 3D objets usng the mage-based vsual hull. It ontrbutes to the speed-up and qualt mprovement of the prevous reonstruton methods. In the next hapter, we roughl desrbe tehnques to reonstrut senes from photographs. hen we explan the sstem s algorthm. In Chapter 4, we show the desgned sstem struture and the expermental results. And we onlude ths paper n Chapter 5. 2. Vrtual Sene Reonstruton ehnques here are man methods for 3D sene reonstruton based on passve magng. he atve methods requre speal hardware or a spef envronment. But passve methods do not need the speal hardware exept some amera and apture board. he passve methods are dvded nto voxel olorng, stereo vson, mage-based renderng, and vsual hull. 2-1. Voxel olorng Voxel olorng depends on olor onssten [5-8]. If
the olors from the dfferent ameras are the same at a vsble pont n 3D spae, that pont exsts. Otherwse, that pont does not exst. he amera, llumnaton, and other external ondtons ma affet olor onssten aordngl, thereb eldng norret results. 2-2. Stereo vson Stereo mathng s used to obtan the range nformaton from a par of 2D mages. In ths method, the robust searh for the same pont n two mages s ver dffult [9-10]. As suh, the amera vews are often arranged along the baselne, and n most ases, assumng a lmted dspart range. Another lmtaton of the stereo mathng method s the oluson problem. 2-3. Image-based renderng Image-based renderng s another method of modelng and renderng [11-13]. he ke advantage of ths tehnolog s ts realst results. B sntheszng the resultng mage dretl from mages wthout the tradtonal modellng proess, we an obtan the resultng mage regardless of the geometr omplext of the objet and the omplext of the mage. However the depth s almost flat. 2-4. Vsual hull he vsual hull s defned as a maxmal volumetr shape that makes the same slhouette from all vews of the real objet [14]. It s smlar to the onvex hull, although t an have holes. When man ameras are used, the nferred vsual hull approxmates the orgnal shape of the objet. However the obtaned vsual hull does not orrespond to the orgnal shape of the objet beause of ts onave regons [15-16]. would not be requred to proess the reonstruton. Speal mahnes suh as dgtal sgnal proessors (DSPs) or a dstrbuted omputng envronment usng multple proessors or heav workstatons should not be assumed. hrd, mplementaton should produe the result ontnuousl wthn a low laten tme after aqurng the nput mages. In the end, the sstem should be real-tme. Fourth, the resultng sene should be as realst as possble. For ths paper we seleted the passve aquston method to satsf the frst ondton, and the vsual hull method to satsf the seond and the thrd ondtons. Furthermore, we used the mage-based renderng tehnque to satsf the fourth ondton. In addton, we used the mage-based vsual hull to exeute mage-based renderng, wth the nterseton of ras rather than n the voxel spae to aheve the osteffetve results. 3-2. Vsual hull samplng In ths paper, a vrtual mage reonstruton sheme based on the slhouette mage from eah amera s used to reonstrut the objet based on the vsual hull of 3D vrtual spae (Fgure 1). After polhedra are generated from the enter of projeton (COP) of the amera and from the slhouette mage orrespondng to that amera, the vsual hull of the objet s obtaned b ntersetng these polhedra. In addton, the range mage s obtaned b projetng the ras from the COP towards the objet at equal spang after assumng that a vrtual amera exsts. hs proess s alled vsual hull samplng. 3. Sstem Algorthms 3.1 Algorthm desgn objetve he frst sstem developed n ths paper was a tral of the method, usng the stereo vson that s smlar to Kanade s method, to determne the evaluaton fator of sstem desgn and algorthm seleton. We made an effort to reonstrut the objet n 3D vrtual spae b ombnng several depth maps alulated from feature dspartes n aptured real senes. Wth ths sstem, we found out that dspart s not robust to texture, llumnaton, and bakground. Moreover stereo-based 3D reonstruton s expensve, and t s also dffult to ombne the resultng depth maps suh as zpperng. hrough developng ths sstem, we defned the desgn rtera for mplementaton. Frst, the sstem should not requre an speal hardware or spef bakground to aqure the real sene, e.g., speal llumnaton equpment suh as that whh emts the grd lghts or the laser range sanner should be unused. Seond, the algorthm should be so hand that speal hardware Fgure 1 Vsual hull: Vsual hull s defned b nterseton of one-shaped polhedra from extruson of mage slhouettes. When the albraton nformaton of the ameras to be used n apturng senes as well as the aptured mages are gven, we an alulate the range mage of the vrtual amera. In ths alulaton, the projetve geometr s ver mportant. We onl onsder the pnhole amera model. Let ( p x, p ) be a enter of mage oordnates, f be a foal length, dpx and dp be x, dretonal lengths
orrespondng to one pxel sze, then projetve transform of a pont X = [ x,, z,1] n 3D homogeneous spae s wrtten as x u f s / dpx 0 px 0 U = v = 0 f / dp p 0 = K[ I 0] X (1) z 1 0 0 1 0 1 In the above equaton K s alled the amera matrx. Sne X s a pont n the amera oordnate sstem, needs to be transformed to a pont n the world X oordnate sstem. When the rotaton matrx of a amera s R and a vetor C represents the enter of the projeton (COP) of the amera, ths transform s expressed as followngs: x R RC R RC X = = X z (2) 0 1 0 1 1 Substtutng X n the equaton (2) for X n the equaton (1) we an get the followng equaton. R RC [ I ] X = K[ IR RC] X = KR[ I C]X U = K 0 1 (3) If we express X n the non-homogeneous oordnate form, that s X ˆ = [ x,, z], the equaton (3) an be wrtten as ˆ 1 U = ( KR)( X C) = P ( Xˆ C) (4) where P s the transformaton from mage oordnates U = [ u, v,1] to a lne n 3D spae. So, the mage oordnates orrespondng to the 3D pont an be omputed b the equaton (4). And a lne through two ponts, the COP of a amera and an mage pont U, s expressed b the projeton of the COP of one amera to the mage spae of the other amera. Usng the fundamental matrx we an ompute the oeffents, a, b, and of the lne, ax + b + = 0 We ompute the slhouette mage for eah mage, and subsequentl ntalze the vrtual range set mage to the nterval from 0 to nfnt. A pxel of the vrtual range set mage represents the nterval wheren the objet ma exst. After that nterval s determned for eah mage, the vrtual range set mage s redued va the set nterseton operaton. he proess for omputng the vrtual range set mage follows. Frst, we alulate the 3D ra from the albraton nformaton of the vrtual amera. We then projet that ra onto the mage spae of one real amera to get the projeted lne. Next we fnd the nterseton ponts of the lne wth the slhouette edges. We alulate the reovered 3D-lne ntervals b reprojetng the nterseton ponts nto the 3D spae. Fnall, we fnd the nterseton ntervals of the vrtual range set mage wth the reovered 3D lne ntervals. 3-3. Performane mprovement of the vsual hull samplng algorthm When the vrtual sene s reonstruted usng the vsual hull method, there are varous tehnques to mprove the speed and orretness of the reonstruton algorthm. Frst, we an utlze the lne ahng to alulate the ntersetng ponts of the slhouette edges and the projeted lne. Seond, the ode optmzaton strateg an be used to speed up the sstem. X ( t) = C + tpu (5) where t s a parameterzaton varable for the lne and s used to desrbe a partular 3D poston on the lne. If we arefull onsder the relatonshp between ras of one amera and an mage spae of the other amera, we wll realze that man 3D ras from one amera are projeted to a sngle lne n mage spae of the other amera. Fgure 2 shows an example of ths relatonshp. he sngle lne s alled an eppolar lne. Fundamental matrx depted n the equaton (6) relates a pxel of one amera mage to an eppolar lne. 0 e' z e' 1 F = e' z 0 e' x P P (6) e' e' x 0 where oordnate [ e ', e', e' ] s alled the eppole, x z Fgure 2 Multple ras of one amera are projeted to the same lne n the mage plane of another amera. If we arefull onsder the relatonshp of the amera ras and another mage spae, we wll realze that man 3D ras from the amera are projeted to the same lne n another mage spae. hs projeted lne s alled the eppolar lne. Fgure 2 shows an example of ths relatonshp.
Beause of ths fat, the proess of fndng the nterseton ntervals of the projeted lne and the slhouette edges s redundant. In ths proess, lne ahng an be used. When ahng s utlzed, we frst ompute the ahng ndex to hek f the nterseton ntervals have alread been alulated. In ase the ahe buket s alread full, we smpl use the ntervals n t. In ase the ahe buket s empt, we fnd the nterseton ntervals to save those n the ahe buket. For the ahe ndex, we use the further pont from the eppole among the nterseton ponts of the projeted lne and the boundar of the segmented objet (Fg. 3). Fgure 3 For redung the omputaton ost, nterseton ponts of the eppolar lne wth the retangular boundar surroundng the objet are used as a ahng ndex. As a seond method to speed up the sstem, ode optmzaton tehnques are emploed. When the nterseton ponts of the projeted lne and the slhouette edges are omputed, the dgtal dfferental analzer s utlzed to redue the addton and multplaton operaton of the real numbers. In ths sstem, we used the floatng-pont verson of the Brensenham s lne drawng algorthm beause the end ponts of the projeted lne are gven n real numbers. Whle astng the real number to an nteger number an speed up the sstem, t ma ompromse ts aura. Stat memor an also be alloated nstead of dnam memor to speed up the sstem. However, dnam memor s preferred sne the nterseton ntervals n the 2D or 3D spae var n relaton to the slhouette mage and the vrtual amera. Another optmzaton tehnque nvolves the reduton of the floatng-pont operaton. he dstane from the COP of the amera to the objet s represented as a real number, n the same wa that the poston of the COP of the amera and the projeton matrx are represented as a real number. B onvertng these real numbers to nteger numbers the sstem wll beome to be fast. hus, the sstem beame two tmes faster than before through ode optmzatons. 4. Sstem Implementaton and Results 4-1. Sstem overvew Four sets of general olor-mage, NSC-output CCD amera (Samsung SC340, nterlaed san, 1CCD, Baerpatterned) and 5mm~15mm anon lenses wth 4 framegrabbers are used to apture movng mages, whle 60Hz AC power s used for snhronzaton. Wth onl one PC, the sstem aptured 320x240 24-bt mages n real tme (30Hz) from 4 ameras muh more easl than Kanade s sstem and other sstems, wth omparable or better performane. 4-2. Slhouette extraton Fgure 4 Indexng ponts are alulated separatel for nne ases aordng to the relatve poston of the eppole to the retangular boundar of an objet. Instead of alulatng the dstanes to the two nterseton ponts, we dretl fnd the pont that wll be used as the ahe ndex through the postonal nformaton of the eppole, whh wll more sgnfantl redue the omputng tme. he postonal relaton of the eppole and the retangle boundar of the objet an be lassfed nto 9 ases, whh are shown n Fgure 4. In ths fgure, we use the nterseton pont of the projeted lne and the thk boundar as a ahe ndex. Generall, the extraton of the slhouette emplos bakground subtraton. Illumnaton hanges and shadows make extraton of the orret slhouette dffult. In ths paper, we mplemented a mathematal model between the llumnaton ntenst, refleton ndex of objets, and pxel values of the mages [17]. Assumng that the dstrbuton of llumnaton ntenst n a ver small regon s onstant, we defne the varables as follows: r: refleton ndex of one pont of the objet α: llumnaton ntenst of one pont of the objet β: bas value of the apturng equpment
he pxel value of the mage s defned as: q= αr + β. Here, the hange n the llumnaton ntenst affets the pxel value lnearl. hus, Fgures 6 show the sample of the nput mage, and Fgures 7 and 8 show the resultng mage and the segmented mage. p = q-β = αr. herefore, the pxel values of the referene mage and the nput mage as follows: p p ( ) = p = p ( ) ( ) = α = α r ( ) r he rato of llumnaton s wrtten as: ( )., α t = α ( ). Assumng α s not hanged, the value t s also onstant at the same value r. Usng ths relatonshp, the varaton of the llumnaton ntenst n the same poston aordng to the llumnaton hange s alulated as shown n Fgure 5. Fgure 6 Input mage Fgure 7 Resultng mage Fgure 5 Varaton urve of the pxel value (bottom left) and the standard devaton (botom rght) aordng to 7 dfferent llumnaton ondtons of the same poston In the mage, the pxel value of the x and oordnates s wrtten as: (x, ) = (αxr + β, αr + β). herefore, we an alulate β usng the rossng pont of the graph as follows: Ftted = (-5.747, -4.334), β =(-5.75-4.33)/2= -5.04, σ^2 = 2.75 We use the standard devaton as the threshold value, when the wndow sze s 7x7 pxel. he boundar of the objet that s robust to the llumnaton ondton s onsequentl obtaned b determnng the hghl hangng poston. Fgure 8 Segmented mage 4-3. Vrtual sene reonstruton results From our mplementaton of the vrtual sene reonstruton sstem, we saw that the sstem ould reonstrut the vrtual mage at a speed of about 8 frames per seond usng the 320x320 24-bt olor mages from four ameras. Fgure 9 and 10 show the sample of reonstruted mages.
Ensurng the qualt of the mages for use on broadast and nema requres hgh-qualt segmentaton, more aurate amera albraton, and refnement of the vsual hull through stereo vson or voxel olorng. Enhaned mage-based renderng tehnques ma also be tred to mprove mage qualt. Aknowledgements he researh reported heren was supported n part b Dgtal Meda Lab. Fgure 9 he vrtual mage 1 Fgure 10 he vrtual mage 2 5. Conluson In ths paper, we mplemented a sstem that an reonstrut vrtual senes from real vdeo streams wthout the need for speal hardware, usng onl one PC wth general vdeo apturng omponents. he sstem was mplemented b omputng vsual hull range data sampled from a vrtual amera, usng the amera ra s projeton and nterseton wth the slhouette edges and texturng the omputed range mage through the mage of the amera that was most smlar to the ra dreton of the vrtual amera. he sstem regstered an almost real-tme performane from ontnuous vdeo streams usng onl an ordnar PC sstem. herefore, the sstem an be appled to varous felds gven an mproved qualt of the generated mage and the sstem s performane. Some Applatons of ths sstem nlude: the real-tme remote vrtual presene sstem; dgtzaton of sports, dane, martal arts, et.; and the vrtual amera walkng sstem for nema produton, nludng 3D objet modelng. Referenes 1. G. Slabaugh, W. Brue Culbertson, homas Malzbender, and Ron Shafer, A Surve of Methods for Volumetr Sene Reonstruton from Photographs, Internatonal Workshop on Volume Graphs 2001, pp. 81-100, Ston Brook, New York, June 21-22, 2001. 2. Demetr erzopoulos and Keth Waters, Modelng and Anmatng Faes usng Sanned Data, the Journal of Vsualzaton and Computer Anmaton, Vol. 2, pages 123-128, 1991. 3. Gerald Ekert, Automat Shape Reonstruton of Rgd 3-D Objets from Multple Calbrated Images, Proeedngs of Euspo 2000, ampere, Fnland, Sep. 4-8, 2000. 4. Rhard Hartle, Andrew Zsserman, Multple Vew Geometr n Computer Vson, Cambrdge Unverst Press, 2000. 5. S. Setz and C. Der, Photorealst Sene Reonstruton b Voxel Colorng, Proeedngs of the IEEE Conferene on Computer Vson and Pattern Reognton, pp. 1067-1073, June 1997. 6. K. N. Kutulakos and S. M. Setz, What Do N Photographs ell Us about 3D Shape?, R680, Computer Sene Dept., Unv. of Rohester, Januar 1998. 7. W. B. Culbertson,. Malzbender, and G. Slabaugh, Generalzed Voxel Colorng, Proeedngs of the ICCV Workshop, Vson Algorthms heor and Prate, Sprnger-Verlag Leture Notes n Computer Sene 1883, pp.100-115, September 1999. 8. P. Esert, E. Stenbah, and B. Grod, Mult- Hpothess, Volumetr Reonstruton of 3-D Objets from Multple Calbrated Camera Vews, Proeedngs of the Internatonal Conferene on Aousts, Speeh, and Sgnal Proessng, pp.3509-3512, 1999. 9. Y. Yang, A. Yulle, and J. Lu, Loal, Global, and Multlevel Stereo Mathng, Proeedngs of the IEEE Conferene on Computer Vson and Pattern Reognton, pp. 274-279, 1993. 10. Q. Chen and G. Medon, A Volumetr Stereo Mathng Method: Applaton to Image-Based Modelng, Proeedngs of the IEEE Conferene on
Computer Vson and Pattern Reognton, pp.29-34, June 1999. 11. S. E. Chen and L. Wllams, Vew Interpolaton for Image Snthess, pp. 279-288, SIGGRAPH 93. 12. P. Debeve, C. alor, and J. Malk, Modelng and Renderng Arhteture from Photographs, pp. 11-20, SIGGRAPH 96. 13. M. Levo and P. Hanrahan, Lght Feld Renderng,pp. 31-42, SIGGRAPH 96. 14. A. Laurentn, he Vsual Hull Conept for Slhouette-based Image Understandng, IEEE ransatons on Pattern Analss and Mahne Intellgene, Vol.16, No. 2, Februar. 1994. 15. J. Koendernk, Sold Shape, he MI Press, 1990. 16. E. Boer and M. Berger, 3D Surfae Reonstruton Usng Oludng Contours, IJCV 22, pp. 219-233, 1997. 17. Naoa Ohta, A Statstal Approah to Bakground Subtraton for Survellane Sstems, Proeedngs of ICCV, 2001.