A VR-BASED HYPER INTERACTION PLATFORM. Rong-Chi Chang

Size: px

Start display at page:

Download "A VR-BASED HYPER INTERACTION PLATFORM. Rong-Chi Chang"

Lewis Norris
5 years ago
Views:

1 Proceedngs of the 2006 Wnter Smulaton Conference L. F. Perrone, F. P. Weland, J. Lu, B. G. Lawson, D. M. Ncol, and R. M. Fujmoto, eds. A VR-BASED HYPER INTERACTION PLATFORM Chun-Hong Huang Hu-Huang Hsu Tmothy K. Shh Dept. of Computer Scence and Informaton Engneerng Tamang Unversty, Tape County, 25137, R.O.C. Rong-Ch Chang Department of Informaton and Desgn Asa Unversty, Tachung, Tawan, 41354, R.O.C. ABSTRACT We present a hyper-nteracton platform whch ntegrates several newly developed technques, ncludng moton classfcaton, a moton reacton control mechansm and a hgh precson 3-D model of the Internatonal Space Staton. The goal s to buld an nteracton platform for traners to navgate n a vrtual realty. The platform can be appled to customzed msson tranng. The system ntegrated vsualzaton envronment, underlyng 3-D model, and a human body tracng mechansm. The tracng system wll be extended to 3-D coordnaton reconstructon and thus behavors of users can be precsely dentfed. These navgaton parameters are used n controllng navgaton n the 3-D vrtual envronment. Another mportant ssue of ths system s to defne a model for moton reacton. The defnton of reacton can be appled to avatars or to the vrtual envronment. These ntegrated technologes can also be used n other vrtual realty envronments. 1 INTRODUCTION Human moton tracng was developed n the past few years. The MARG sensors were used to develop a system whch can embed seleton nto a vrtual envronment (Barchmann et al. 2001). Instead of usng sensors, vdeobased tracng strategy was proposed (Luo et al. 2002, Sgal et al., 2004). Typcally, the tracng nvolves matchng objects n consecutve frames usng pxels, ponts, lnes, and blobs, based on ther moton, shape, and other vsual nformaton (Leung et al. 1995, Sho et al. 1991, Polana et al. 1994). The dffculty of vdeo-based approach s on how to deal wth ncomplete moton such as occluson or bad vewng angle. Another dffculty s due to varatons of lght sources. Snce sensors are expensve n general, vdeo-based approach s usually used n commercal applcatons. We have developed a tracng system that can reconstruct 3-D coordnates of marers on a human seleton. Snce we got the seral 3-D coordnates, the moton types can be classfed. We propose an approach for unt moton classfcaton whch s based on Neural Networs. As long as the type of unt moton s dentfed, these types would be the tranng parameters of Bac-propagaton Neural Networs. The Bac-propagaton Neural Networs should understand human moton pattern accordng to the combnaton of the unt motons. As long as the human moton s dentfed, a set of pre-defned reactons s appled to the moton on a 3-D envronment, and they are used as the nput parameter of a smulated model. The smulatve specfcaton can be developed based on partcular mssons, such as movng objects or performng tass n the 3-D space. The partcular platform that we use s a hgh precson model of Russa Space Staton. We hope that, from moton detecton, tracng to understandng, moton reacton can be consdered as an nterestng research topc. We bult a studo that has a surroundng bacground (270 degrees) wth blac color. Lght sources are placed on the floor and on the celng to stablze lghtng parameters for vdeo tracng. Two cameras are used for calbraton of tracng ponts on a specal desgned sut. A bacprojected screen and a projector are equpped such that the 3-D model can be rendered on the screen wthout nterferng (.e., shadowng) by the actors n the studo. 2 MOTION TRACKING A tracng system that can reconstruct 3-D coordnates of marers on a human seleton. The tracng process nvolves four steps of computaton, whch are dscussed n turn n the remander of the paper: 1. Intalzaton and vdeo camera synchronzaton. 2. Bacground and trac ponts separaton. 3. Trac ponts dentfcaton. 4. Seleton reconstructon on bacground vdeo. 2.1 Intalzaton and vdeo camera synchronzaton The bacground need to be separated from trac ponts n order to dentfy trac ponts. An ntal vdeo bacground /06/$ IEEE 576

2 can be obtaned from a vdeo segment of 30 to 60 frames, where movng objects are removed and holes are npanted. We use a fast mage npantng algorthm whch reles on a dffuson ernel. In addton, snce we use two vdeo cameras for 3D coordnate reconstructon, the two cameras needs to be synchronzed. Our strategy requres the player to stand steady for at least 2 seconds. A varaton threshold for vdeo changes s set to tell the steady stuaton. Thus, n the frst step, the player need to move around for a few seconds, stand steady for 2 seconds, then our system start to update bacground varaton for tracng the seleton. 2.2 Bacground and trac ponts separaton Snce lght sources to trac ponts are qute dffcult to stay stable, t s necessary to update the bacground dynamcally n order to obtan a better separaton between bacground and trac ponts. Assumng that B (x, and B +1 (x, are values of pont (x, on the th and the +1 th frames, respectvely. and that, I (x, s the value of pont (x, on the comng th frame, we use: B ( + 1 [ B I ] B + α f I object pont x, = B + β f I object pont [ B I ], α = 1, β = 0.01 Ths strategy wors f small varaton of lght source occurs on bacground. However, f the varaton s too large, the bacground should be reconstructed. After that, we use medan flter to exclude solated ponts. Also, we use two morphology operators (eroson and dlaton) to perform the close operaton. The close operaton s able to elmnate small blocs. 2.3 The dentfcaton of trac ponts The most challengng ssue of multple object tracng s to dentfy each and all objects. We mantan a trac pont array to store propertes (locaton and color nformaton) of all trac ponts. To get TP property, we use seed fllng algorthm on the orgnal vdeo for basc color segmentaton. Boundares of trac ponts are roughly computed, to obtan the center of each trac pont. Each record n the trac pont array contans a locaton of the center and the HSI color nformaton. To precsely trac each pont, two ssues need to be solved. Trac ponts moves n a drecton whch s hard to predct; and, trac ponts can be cloaed (occluson). To solve the frst problem, predcton and searchng mechansms are used. The second problem can be partally solved wth multple cameras. However, t s possble to solve the second problem usng a unque color for each trac pont and searchng n a range of movement. A sample predcaton mechansm whch computes the average movement vector of each trac pont n the latest 30 frames s used. If the predcton of trac pont matches (wthn a small threshold of center locaton), the correspondng trac pont s updated n the array. Otherwse, the system searches the trac ponts n a small boundary (eght drectons to speed up computaton). Trac pont propertes are updated f there s a match. Otherwse, we count the trac pont as mss detecton. When a mss detecton occurs, we eep the mss detected trac pont n a trac pont queue. In the next teraton, trac ponts are searched aganst the queue wth a hgher prorty. The search focuses on the color of trac ponts. Our prelmnary experence shows that, wth the setup of our envronment, the mss detecton rate s tolerable. 2.4 Seleton reconstructon on bacground vdeo After the trac ponts are dentfed, we need to map them to a seleton whch represents a human body. The segments of seleton ncludes torso, upper arms and legs, and lower arms and legs. The mappng process s performed as soon as all TPs are dentfed n the frst run. Snce TPs are traced wth unque IDs, remappng to seleton s thus not necessary. The mappng strategy taes a smple heurstc rule assumng that the player s not upsde down or dong a handsprng (.e., flp): 1. Compute the fulcrum of 12 trac ponts based on the ntal posture, set fulcrum to be the pelvs. 2. Follow the vertcal lne up to fnd spne, nec, and head. A horzontal threshold of a few pxels s used n case that the 4 trac ponts of human body are not algned vertcally. 3. Splt the rest 8 TPs n the seleton to left and rght, accordng to the 4 trac ponts mapped n the above step. Followng the pelvs to fnd lower arms and legs, and followng the nec to fnd upper arms and legs. Spatal relatons of TPs are used as the heurstc. 4. Do the same for the rght hand sde trac ponts. Sde Camera Front Camera Fgure 1: Coordnates and Front and Sde Cameras Only the trac ponts that be captured by the front camera are used to dentfy the trac ponts. However, to reconstruct 3D coordnates for all trac ponts, t s necessary to use the sde camera (along the x-axs to the left of 577

3 the box n fgure 1). To restore 3D coordnates, we use the followng strategy: The vectors of S can be expressed as follows: 1. For each of the trac ponts on both cameras, fnd the dfferences of coordnates on y-axs. 2. Mnmzng the dfferences to algn TPs on y-axs usng dynamc programmng. 3. Tae the x-coordnates from the front camera and the z-coordnates from the sde camera. After the 3D coordnates are computed, t s necessary to perform a projecton of the coordnates to 2D space. One may argue that t s not necessary for 3D coordnate reconstructon and projecton, and usng 2D coordnates s enough. However, to mae the nteracton realstc, 3D nformaton s necessary. For nstance, the player must fell that a punch s ndeed located on an object n the vdeo by movng forward hs arm. In addton, f vrtual realty avatar s used, 3D coordnate s requred. After the projected 2D coordnates are obtaned, we map the seleton onto a scenaro vdeo. The scenaro vdeo follows MPEG-2, wth an mportant extenson allows a hyper jump among vdeo segments. Hyper jump tags are embedded n the user defned data secton of standard MPEG-2 vdeo clps. The scenaro vdeo can be a pre-recorded vdeo game or a tranng vdeo for customers. For performance consderaton, only a small secton of vdeo s stored n memory to speed up accessng tme. 3 MOTION CLASSIFICATION Next wor s moton analyss, snce we got 3-D coordnates of tracng ponts that were obtaned from 2 or more cameras. In order to dentfy what moton the actor(s) perform, we use approxmaton strategy based on Dynamc Programmng. A human moton specfcaton can be represented as 9 moton vectors wth regards to the 9 tracng ponts on a seleton. For each tracng pont, multple vectors are used. We segment the moton of the human body nto unt motons, and each unt moton has 200 frames. For example, f a human moton has 820 frames, then t would be dvded nto 5 unt motons. Suppose that each human unt moton data s a tme successve data S consstng n elements, the seral data can be presented as follows: S = S 1, S 2, S 3,,S. S n Here, S expresses a moton data of a certan segment of a human body. The dfference between S and S +1 s consdered as the value of vector v. Snce human moton data s 3D tme seral data, each element S can be express as space coordnates S=(x, y, z). Then the vector v s gven as follows: v = (( x + 1 x ), ( y + 1 y ), ( z + 1 z )) = S S + 1 V S = n ( v S 1, v S 2,... v S, v S ( + 1 ),... v S ( 1 ) ) where V s the set of vectors, all the feature trajectores S wth each segment of human body can be represented. Each moton vector s represented by a par of angles as shown as Fgure 2. Angle α s n between the projecton of moton vector v on the XZ-plane and the X-axs. Angle β S1 s between the moton vector and ts projecton. An angle has a value between 0 and 360 (nclusve). Y t 0 v S1 Fgure 2: Defnton of Moton Vectors Instead of usng precse coordnates n the moton dentfcaton computaton, we use an approxmaton approach whch only computes the rough locaton of each moton vector. For each angle (α or β), we defne an Angle Code (AC) as: AC = floor(angle / n) where Angle can be α or β and floor s the floor functon. The parameter n can be set to 45. As such, Angle Codes are between 0 and 7. It s possble to set n = Thus, Angle Codes are values between 0 and 15. In our tracng procedure, snce the length of a moton vector s short n general, coordnates of tracng ponts may not reflect the actons precsely. Usng an approxmaton approach, on the other hand, can reduce the computatonal cost n our wor. For the th moton vector, α and β are represented by a par of Angle Codes C = (AC αj, AC βj ). We can classfy the moton types accordng the seral Angle Codes. We should reconstruct the feature space whch stores the seral data of Angle Codes be parsed and translated from the dfferent nds of standard motons. Assume that each unt moton has ts own seral Angle Codes C j = (AC αj, AC βj ), whch stores n the feature space, thus we can compare the seral Angle Codes Csj of the new unt moton wth the Cs va the frst level classfer of neural networ. After the frst level classfer, we can fnd the smlar Unt Moton Types(UMT) for each unt moton as n the followng equaton: β α t 1 X Z 578

4 UMT =f 1 (Angle Codes Cs), (1 m) Snce the smlar types of each unt motons can be recognzed va f 1, a moton representaton can be Moton = { (UMT 1 ), (UMT 2 ),, (UMT m ) }. m s the amount of unt motons. The representaton of motons can be used n the second level of the neural networ model. A traned neural networ can be used as the mappng functon from a moton representaton to a moton classfcaton code (MCC) MCC j = f 2 (Moton x ) where j s between 1 and the maxmum number of moton code to be classfed, and x s assumed to be the maxmum number of possble motons. The mappng, f 2, decdes the MCC va the second level classfer. We propose a threelevel classfcaton mechansm usng a neural networ. As soon as a moton classfcaton code, MCC j, s classfed by the second level classfcaton mechansm, a seres of MCCs may represent a contnuous moton. However, smlar contnuous motons may have dfferent duratons. Therefore, a moton wndow s defned as an ordered lst of moton control codes, wth a varable number of MCCs. Fgure 3 llustrates the two-level archtecture. Each moton wndow starts at a moton control code. Thus, the number of motons wndows (.e., n) s equal to the number of motons n an acton. A thrd level classfer, f 3, taes as nput a moton wndow, and returns a moton sgnature (.e., MS ) as n the followng equaton: MS = f 3 (MW ), where 1 n Moton sgnatures are be used to trgger reactons, whch we dscuss n the next secton. 4 MOTION REACTION AND CONTROL Moton reacton can be defned n two types. The frst s changng the camera vew of a vrtual realty browser, by renderng the 3-D model (represented n VRML or the le) from dfferent vewng angles. Ths type of reacton depends on the movement of a traner. Thus, the defnton of motons should deal wth the movement of a traner n the studo. Yet, the movement specfcaton should consder the lmted space n the studo. Specal posture of the traner s recognzed as a partcular movement. The second type of reacton ncludes changng objects n the 3-D model, such as movng an object or assemblng an object n the space staton. In addton, the second type of reacton may nclude the reacton of other avatars n the 3-D scene, f the system s mplemented on a shared Web-based VR envronment. From moton detecton, moton tracng, moton understandng, to moton reacton, the poneer perspectve of ths research project s to nvestgate how a vrtual envronment (ncludng ts avatars) should react wth respect to a partcular moton dentfed by the system. For the two types of reactons controls (.e., changng camera vew and changng avatars), we mplement them n the Vrtools envronment as the followng: Control Camera Vew (MRA Camera ) 1. Create a target camera 2. Set camera poston and vewng angle 3. Attach camera poston to an avatar Control Avatars (MRA Avatar) 1. Create a database record by mportng the avatar and 3-D model resources. 2. Load the avatar and 3-D model nto the Vrtools envronment. 3. Add anmatons to the avatar. 4. Intate eyboard controller (usng moton control) and connect character control to anmatons After moton sgnatures are dentfed by the bacpropagaton neural networ, each moton sgnature s assocated wth a lst of moton reacton anmatons (.e., MRA Camera and MRA Avatar ). The anmatons rely on the above two types of controls n the Vrtools. In addton, t may be necessary to have addtonal object transformatons (or to add addtonal objects) n the 3-D scene, f the msson specfcatons have such a requrement. MW n MW 1 MW 2 MW 3 MW 4 MW 5 MCC j1 MCC j2 MCC j3 MCC j4 MCC j5 MCC jn Fgure 3: Moton Wndows of Varable Tme Slots 579

5 4.1 Reacton Resoluton To dentfy what reactons should be appled to each moton sgnature, a resoluton mechansm s used. Snce each moton sgnature contans a varable number of moton control codes, t s possble to have two or more sgnatures that share a lst of same control codes. In addton, snce each moton sgnature s assocated wth a lst of moton reacton anmatons, dfferent sgnatures should share reacton anmatons n overlapped tme slots. Thus, a mechansm to resolve the dfference of reacton anmatons wth respect to dfferent moton sgnatures s necessary. Fgure 4 llustrates a lst of moton reacton anmatons for each moton sgnature. Each MS x at tme slot x has a lst of moton reacton anmatons MRA T x, t. Assumng that a moton reacton anmaton, MRA T x, t, has a control type T, where T represents Avatar or Camera. The two subscrpts, x and t, represents the ndces of moton sgnatures and the ndex of tme slots. For the two types of controls, dfferent strateges are used to deduce the reactons. The frst equaton n Fgure 4 ndcates that for the MRA wth type Camera, the deduced moton reacton anmatons (DMRA) taes the average camera movements. That s, a DMRA refers to all camera motons up to a current tme slot t (.e., x x t), by tang the average camera coordnates n a 3D space. Ths strategy allows a contnuous moton to consder snapshots taen n current tme slot, as well as a few slots before. The deduced moton reacton anmatons results n a smooth camera movement. The second equaton s for the Avatar type of anmatons. It s necessary to consder multple avatars whch response to a moton sgnature. For new avatar, a new anmaton assocated wth the avatar s started. However, the anmaton of avatars started n prevous tme slots contnues. The computaton of DMRA s dynamc. That s, at each tme slot t, due to each moton sgnature, a lst of reacton behavor s sent to the reacton controller (to be dscussed). Thus, the dynamc reacton results n a smooth moton. 4.2 Rendezvous Communcaton Control A rendezvous control s a mechansm whch allows two processors to meet at a certan tme pont, where the two processors complete certan tass by ther own before the rendezvous happen. We use two servers. The tracng server and the VR renderng server are synchronzed n real-tme. Fgure 5 llustrates dfferent enttes of the rendezvous communcaton control. As soon as a moton sgnature (MS) s dentfed, the MS s ept n a moton sgnature queue, whch s controlled by the tracng server. An acton controller n the tracng server synchronzes wth a reacton controller n the VR renderng server, by the rendezvous communcaton control mechansm. The mechansm decdes where the two controllers wll meet. MS 1 MS 2 MRA Camera1,1 MRA Camera1,2 MRA Avatar1,3 MRA Camera2,2 MRA Camera2,3 MRA Camera2,4 MS 3 MS 4 MS n MRA Camera3,3 MRA Avatar3,4 MRA Avatar4,4 MRA Avatar4,5 MRA Cameran,n MS x MRA Tx,1 MRA Tx,2 MRA Tx,3 MRA Tx,4 MRA Tx,5 MRA Tx,n DM RA Camera t ( M RA C a m e ra x,t ) / (t-x'+1) x ' x t = M R A Avatar D M R A Avatar t = { M R A Avatar x ', f A v atar x ' s n o t co m p lete t, f A v a ta r t s n e w Fgure 4: Resoluton of Moton Reacton Anmatons 580

Reacton Controller Rendezvous Acton Controller Moton Sgnature Queue Control Message Queue Moton Reacton Classfer MS 1 MS 2 Fgure 5: Rendezvous Communcaton Control As soon as the controllers meet, the

The actvaton of avatars and renderng of 3D scenes consume heavy computaton. Thus, the reacton controller needs to wat tll the computaton s complete n order to move to the next step.

5 MISSION SPECIFICATION AND CONTROL A msson specfcaton s defned as a seres of actons that a traner needs to accomplsh by usng the system.

6 Reacton Controller Rendezvous Acton Controller Moton Sgnature Queue Control Message Queue Moton Reacton Classfer MS 1 MS 2 Fgure 5: Rendezvous Communcaton Control As soon as the controllers meet, the acton controller sends a moton message to the reacton controller, whch nteracts wth a moton reacton classfer to dentfy how to trgger avatars n the 3D VR envronment va control messages (CMs). The actvaton of avatars and renderng of 3D scenes consume heavy computaton. Thus, the reacton controller needs to wat tll the computaton s complete n order to move to the next step. Smlarly, the acton controller needs to wat tll the next moton sgnature s dentfed. The two controllers rendezvous. Thus, reactons n the 3D envronment are synchronzed wth the motons by the actor. 5 MISSION SPECIFICATION AND CONTROL A msson specfcaton s defned as a seres of actons that a traner needs to accomplsh by usng the system. The specfcaton may nclude some chec ponts, whch s used as the base of an assessment model. Msson assessment can be mantaned n a database for the analyss of ndvdual tranng performance. The specfcaton and smulaton model are desgned by doman experts. A msson specfcaton can be defned as a lst of events. Each event s assocated wth a locaton n the 3D space and a set of actons: CM Msson m = (Event 1, Event 2, Event 3,, Event x ) = ((Locaton 1, {A 11, A 12,, A 1 }), (Locaton 2, {A 21, A 22,, A 2j }),, (Locaton x, {A x1, A x2,, A x }) The order of actons wthn a msson can be defned n any topology. Thus, each acton s assocated wth a predecessor and a successor, except the last and the frst acton. Actons can be performed n sequental or parallel, as suggested by the topology, whch s desgned by usng a graphcal user nterface. The assessment of a msson ncludes an evaluaton functon whch taes two parameters as nput: an event n the msson and an event performed by the traner. Comparson of coordnates can be asserted nto the evaluaton functon as a chec pont (crtcal msson-based) or quanttatve outcome (overall performance). Comparson of actons n each locaton can be accomplshed by two factors: a detaled movement of the traner, and the reacton from the avatar. Msson specfcatons as well as the assessments for each traner are mantaned n a msson database. An analytcal model based on the outcome of assessment can be used by a hgh ranng offcer to evaluate ndvdual traners. We mplement a vdeo studo and ts software systems. The software systems run on two computers respectvely n a LAN. Fgure 6 shows the studo, the nteractve module, and example of the 3D model of Russa/Internatonal Space Staton. 6 CONCLUSION Ths paper nvestgates the fundamental technques and applcaton of buldng a 3-D vrtual envronment, as an nteracton platform. It proposes the platform and ts underlyng technology for cosmonaut msson tranng n the Vrtual Internatonal Space Staton. The system shows research collaboraton results that we have accomplshed n the past few years. (a) The Vdeo Studo wth an Actor (b) Interacton wth a 3D Stc Fgure (c) Snapshot of Space Staton Fgure 6: Implementaton of the System and Studo 581

7 The system use exstng tracng technques, wth a new nterestng ssue n moton reacton control. The software s mplemented on two computers wth a rendezvous control mechansm to adjust speed of reactons. Our experence encourages us to further enhance the collaboraton to nclude the nteror model of the space staton. We hope that, moton reacton wll be an mportant ssue n addton to detecton, tracng, and understandng n vdeo technology. ACKNOWLEDGMENTS Ths wor was supported n part by the Natonal Scence Councl of the Republc of Chna under contract NSC E REFERENCES Barchmann, E. R., R. B. McGhee, X. Yun, and M. J. Zyda Inertal and magnetc posture tracng for nsertng humans nto networed vrtual envronments. In Proceedngs of the ACM Symposum on Vrtual realty software and technology. pp Luo, Z., Y. Zhuang, F. Lu, and Y. Pan Incomplete Moton Feature Tracng Algorthm n Vdeo Sequences. In Proceedngs of the IEEE 2002 Internatonal Conference on Image Processng. pp. III/617- III/620. Sgal, L., S. Bhata, S. Roth, M. J. Blac, and M. Isard Tracng loose-lmbed people. In Proceedngs of the 2004 Computer Socety Conference on Computer Vson and Pattern Recognton. pp. I421-I428. Leung, M. K., and Y. H. Yang Frst sght: a human body outlne labelng system. IEEE Trans. Pattern Anal. Mach. Intell. 17 (4), pp Sho, A., and J. Slansy Segmentaton of people n moton. Proceedngs of the IEEE Worshop on Vsual Moton. pp Polana, R., and R. Nelson Low level recognton of human moton. In Proceedngs of the IEEE CS Worshop on Moton of Non-Rgd and Artculated Objects. pp

Parallelism for Nested Loops with Non-uniform and Flow Dependences

Parallelism for Nested Loops with Non-uniform and Flow Dependences Parallelsm for Nested Loops wth Non-unform and Flow Dependences Sam-Jn Jeong Dept. of Informaton & Communcaton Engneerng, Cheonan Unversty, 5, Anseo-dong, Cheonan, Chungnam, 330-80, Korea. seong@cheonan.ac.kr