A METHOD OF MODELING DEFORMATION OF AN OBJECT EMPLOYING SURROUNDING VIDEO CAMERAS

Transcription

1 A METHOD OF MODELING DEFORMATION OF AN OBJECT EMLOYING SURROUNDING IDEO CAMERAS Joo Kooi TAN, Seiji ISHIKAWA Deparmen of Mechanical and Conrol Engineering Kushu Insiue of Technolog, Japan KEY WORDS: Modeling, -D modeling, Shape recover, Deformaion, Moion, Non-rigid objecs, Facorizaion. ABSTRACT A novel echnique is presened for modeling deformaion of an objec, no onl is fron side bu also he rear side, emploing surrounding muliple video cameras. In order o make a -D model of an objec from is acual images, -D shape recovering echniques based on image capuring are usuall emploed. Unlike eisen echniques which make a model b muuall connecing pariall recovered shapes, he proposed echnique provides a model of an objec b recovering is enire shape simulaneousl emploing muliple fied video cameras surrounding he objec. In he echnique, F pairs of, herefore F, video cameras are placed around he objec concerned and ake he images of he objec s deformaion. A single measuremen mari is defined b eracing feaure poins locaions from he F video image sreams and facorizaion is applied o he mari once o derive he -D coordinaes of all he feaure poins on he objec during observaion. Thus he enire model of he objec is obained. Facorizaion based on orhographic projecion is emploed for -D shape recover. This causes approimael o % of recover errors in he echnique. Insead, he presen echnique necessiaes neiher camera calibraion ecep for making he ligh aes of a facing pair of video cameras parallel, nor regisraion of recovered shape. The echnique is presened and eperimenal resuls are shown. INTRODUCTION Three-dimensional objec modeling has drawn much aenion mainl in compuer vision communi. arious -D shape recovering echniques for rigid objecs have been sudied Horn,98 and some of hem are now commerciall available such as a sereo vision ssem and a srucured lighs projecion ssem. When deformable objecs are aken ino accoun, opical mehods represened b sereo vision seem promising compared wih non-opical mehods such as goniomeers and magneic sensors which one has o pu on his bod. Eisen opical mehods including various moion capuring mehods, however, necessiae sric camera calibraion, which is no ver convenien especiall for oudoor use. A echnique called facorizaion has been proposed b Tomasi & Kanade99 for recovering -D shape of a rigid objec and camera moion from a video image sream wihou using camera parameers. This echnique has been eended o a deformable objec s shape recover b Tan e al.998 and Tan & Ishikawa999 emploing uncalibraed muliple video cameras. The echnique is, however, insufficien for making an enire -D model of an objec irrespecive of rigid or non-rigid, since onl he objec par commonl visible from he muliple video cameras is recovered. Normall an enire model of an objec is, if i canno be modeled b a CAD ssem because of is complicaed shape, made b muuall connecing pariall recovered shape of he objec emploing eisen echniques. In he presen paper, a echnique is described for modeling deformaion of an objec, no onl is fron side bu also he rear side, emploing surrounding muliple video cameras. In he echnique, F pairs of, herefore F, video cameras are placed around he objec concerned and ake he images of he objec s deformaion. In hese F video image sreams, feaure poins on he objec are racked during observaion ime and heir image coordinaes are all sored ino a single measuremen mari. This is he novel of he presen paper. Once he single measuremen mari is obained, facorizaion is applied o he mari, resuling in he recover of he -D coordinaes of all he chosen feaure poins. Thus he enire model of he objec is obained. If he objec is rigid, he enire shape of he objec is modeled. If he objec is deformable, he enire shape a each sampled ime and he enire deformaion during observaion ime is modeled. The echnique is presened and eperimenal resuls are shown. Since facorizaion based on orhographic projecion is emploed in he echnique, some recover errors are ineviable. This is also discussed. 80 Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam 000.

2 SHAE RECOERY OF A DEFORMABLE OBJECT An ouline of -D shape recover for deformable objecs based on facorizaion is presened according o Tan & Ishikawa999. F video cameras are fied around an objec. During discree observaion ime =,,,T, video camera f akes images of he objec, producing image sreams I f =,,,T; f=,,,f. The feaure poins defined on he objec observed b all he video cameras a ime are denoed b s p p=,,,. Feaure poin s p is a column vecor and is projeced a fp, fp on he image plane of video camera f a ime. A measuremen mari a ime denoed b W is defined as a F mari whose fh componen of he ph column conains fp, whereas f+fh componen of he ph column conains fp. The measuremen marices W=,,,T are unified ino a single mari called an eended measuremen mari in he following form; W W W W T =. The mean value of each row of mari W is subraced from he componens of he corresponding row o ield a mari W ~ as ~ W = W W Q E, T where Q =, i.e., he number of all he feaure poins, and E is a Q Q mari whose enries are all uni. Le he = world origin be specified a he cenroid of all he Q feaure poins and le he feaure poins be newl denoed b s p p=,,, wih respec o he world origin. Then, since orhographic projecion is assumed in he presen echnique, he fp componen of mari W ~ can be described b i f, s p and he f+fp componen b j f, s p, where i f and j f are column vecors and he se of vecors i f, j f, and i f j f defines he orhonormal ssem of he lens coordinae ssem of video camera f. This leads o he following decomposiion of he mari W ~ Tomasi & Kanade, 99; ~ W = M S, where marices M and S have he following form; M i i,, i j, j, j T =,, F, F S = s, s,, s s, s,, s s T, s T,, s T T. Here T signifies ranspose of a mari or a vecor. Equaions,, and indicae ha -D locaions of all he Q feaure poins during he observaion ime =,,,T are calculaed simulaneousl from mari W ~ along wih he orienaions of he fied F video cameras. I should be noed ha he feaure poins commonl visible from he F video cameras a each observaion ime are recovered heir -D locaions. Those which are visible from less han F video cameras mus herefore be eliminaed from mari W. Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam

3 TRANSFORMATION BETWEEN REAR AND FRONT IMAGE LANES The deformaion recover echnique described in he former secion recovers parial -D shape of an objec commonl visible from all he F video cameras. In order o recover he enire shape, F pairs of cameras are placed around he objec in he proposed echnique. Each pair of video cameras called a fron video camera and a rear video camera face wih each oher beond he objec and heir ligh aes are muuall se parallel. The feaure poins on he objec projeced ono he image plane of he rear video camera are ransformed ino he projeced feaure poins on he image plane of he fron video camera in order o make a single measuremen mari. Le us ake he fh pair of video cameras See Figure. ideo camera C f akes he image of he fron side of objec O, whereas video camera D f akes he image of he rear side of objec O. Le us denoe image planes of video cameras C f and D f b I f and J f, respecivel. Suppose m feaure poins s i i=a,a,,a m are projeced ono image plane I f and n feaure poins s j j=b,b,,b n are projeced ono image plane J f. Locaions of he projeced feaure poins are denoed b i = i, i T and j = j, j T, respecivel. Their ses are denoed b S If ={ i i=,,,m} and S Jf ={ j j=,,,n}, respecivel. Generall, S If S Jf. Assuming ha a leas hree feaure poins are commonl observable on image planes I f and J f f=,,,f, affine ransform A is defined emploing hese feaure poins beween he wo image planes. The feaure poin j in S Jf are hen ransformed ino he poins on image plane I f b j Aξ j =. Thus he projeced fron feaure poins in S If and he projeced rear feaure poins in S Jf are unified on he fron image plane newl denoed b. f I RODUCING A SINGLE MEASUREMENT MATRIX In he echniquetan & Ishikawa, 999, he images of a deformable objec obained from F video cameras define a single measuremen mari, which receives singular value decomposiion o ield recover of -D deformaion process. In his paricular echnique, F successive video cameras, f k, f k, f k,, f k,f- k=,,,f are chosen among F video cameras surrounding he objec. Here kl=k+l mod F l=,,,f-. Wih respec o each se of he F video cameras, he feaure poins visible from all he F video cameras a ime =,,,T are aken correspondence o ield a mari W of he form given b Eq.7 Case of F=. = W 7 Tan, Joo Kooi Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam

4 O s i s j I f i C f J f ξ j ξ η D f Î j i Figure. Inegraing wo images acquired b muuall facing cameras Then, b he procedure saed in, he projeced feaure poins on he image of video camera F+m m=,,,f are ransformed ino he poins on he image of video camera m o ield he mari of he form given b Eq.8 Case of F=. = 8 The above mari herefore conains he informaion on he enire shape of he objec a ime. B merging he marices =,,,T, we have T =. 9 Tan, Joo Kooi Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam

5 This mari conains he informaion on deformaion process over he enire shape of he objec during observaion ime T. Mari is ransformed ino mari ~ in he same wa as shown b Eqs. and. Singular value decomposiion is applied o mari ~ ielding decomposiion of mari ~ ino wo marices, i.e., ~ = M S. 0 Shape mari S gives he model of he enire -D shape of he objec a each ime and he enire -D deformaion of he objec during observaion ime T. EXERIMENTAL RESULTS A o balloon is recovered is inflaion and deflaion process hree-dimensionall emploing he proposed echnique. In his eperimen, F=, i.e., video cameras surround he balloon, he ligh aes of each pair of video cameras being parallel wih each oher. Observaion ime is 7seconds: Sampling is done ever 0. second: Hence T=. As feaure poins, small colored papers are affied on he surface of he balloon. Correspondence of he feaure poins is found among ever hree images a each ime semi-auomaicall b calculaing normalized correlaion o ield mari W given b Eq.7. Wih respec o ever video camera pairs, affine ransform is defined and he projeced feaure poins on he rear image are ransformed ino he poins on he fron image b Eq. o obain mari of Eq.8. This procedure is ieraed for =,,,T and mari of Eq.9 is finall obained. The oal number of feaure poins chosen during he observaion is,99=q. iews of he deformaion of he o balloon from one of he video cameras are shown in Figure a some sampled imes. Resuls of he -D recover are illusraed in Figure. Time flows in he direcion of he arrows. Compuaion ime from feeding of Eq.9 ino he program o obaining he shape mari S in Eq.0 is. seconds b C eniumii:0mhz. DISCUSSION The proposed echnique is an eension of he facorizaion mehod Tomasi & Kanade, 99. The idea of soring feaure poins a differen imes ino a single measuremen mari has eended facorizaion o shape recover of deformable objecs Tan e al., 998, Tan & Ishikawa, 999. Furher eension has been done in he presen paper b inegraing he fron and he rear feaure poins on he respecive video images and ielding a single measuremen mari. This measuremen mari, also called an eended measuremen mari, is a novel idea conained in he presen echnique. The recover error defined b Tan e al.998 was approimael.% in average in he performed eperimen. The errors mainl resul from he assumpion of orhographic projecion and posiional displacemens when finding correspondence of feaure poins among he video images. A number of performed eperimens emploing various rigid objecs and human moions alwas provide us wih o % of recover errors. In his sense, he eperimen gives a saisfacor resul. Alhough he recover errors are larger in he proposed echnique han in eisen echniques, one is almos able o escape from he laborious camera calibraion in use of he presen echnique. I is anoher advanage ha, unlike eisen echniques, he proposed echnique is a regisraion-free echnique. The echnique ma have applicaions o modeling human realisic moions for R ssems, T games, or for manmachine inerfaces. I ma as well be applied o analzing no onl ahleic or dancing moions bu also analzing hose moions of he aged or he handicapped. 80 Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam 000.

6 Figure. iews of a o balloon becoming inflaed and deflaed as ime flows in he direcion of arrows Figure. Recovered deformaion of he balloon: Time proceeds as indicaed b arrows Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam

7 7 CONCLUSIONS A echnique was proposed for recovering -D enire shape and deformaion of an objec emploing surrounding video cameras. More han hree pairs of video cameras ake video images of he objec concerned. The feaure poins in he fron image sream and hose in he rear image sream are inegraed o ield a single measuremen mari o which facorizaion is applied. Advanages of he echnique include ha i is a calibraion-free echnique ecep for aligning he ligh aes of he facing fron and rear video cameras and ha i is a regisraion-free echnique. The proposed echnique ma find various applicaion fields such as modeling human realisic moions or analzing moions of he aged or he handicapped as well as ahleic or dancing moions. REFERENCES Horn, B. K.., 98. Robo vision, MIT ress, Cambridge, MA. Tan, J. K., Kawabaa, S., Ishikawa, S., 998. An efficien echnique for moion recover based on muliple views, roc. IAR Workshop on Machine ision Applicaions, pp Tan, J. K., Ishikawa, S., 999. Eracing -D moions of individuals a work b uncalibraed muliple video cameras, roc. 999 IEEE In. Conf. on Ssems, Man and Cberneics, pp.iii Tomasi, C., Kanade, T., 99. Shape and moion from image sreams under orhograph: A facorizaion mehod, Inernaional Journal of Compuer ision, 9, pp Inernaional Archives of hoogrammer and Remoe Sensing. ol. XXXIII, ar B. Amserdam 000.