Real-time Motion Capture System Using One Video Camera Based on Color and Edge Distribution

Transcription

1 Real-tme Moton Capture System Usng One Vdeo Camera Based on Color and Edge Dstrbuton YOSHIAKI AKAZAWA, YOSHIHIRO OKADA, AND KOICHI NIIJIMA Graduate School of Informaton Scence and Electrcal Engneerng, Kyushu Unversty 6- Kasuga-koen, Kasuga, Fukuoka, JAPAN Abstract: - Ths paper proposes a real-tme, vdeo based moton capture system usng only one vdeo camera. Snce conventonal vdeo based moton capture systems need many cameras and take a long tme to deal wth many vdeo mages, they cannot generate moton data n real tme. Therefore they cannot be used as a real-tme nput devce for a standard PC. On the other hand, the prototype system proposed n ths paper uses only one vdeo camera, t takes vdeo mages of the upper body of the person, e.g., x, y, z poston of the hands, a face rotaton, a body rotaton, etc., and t employs a very smple moton-trackng method to generate such upper body moton data n real tme. Ths paper manly descrbes ts aspects as a hand and face moton-capturng devce for a standard PC showng ts applcaton examples. Key-Words: -Image understandng, Moton capture, Moton recognton, Interface, Vrtual realty Introducton Ths paper treats a vdeo-based moton capture system usng one vdeo camera. Many researches on the moton capture system have been made so far snce moton data has become n great demand for CG anmaton productons and 3D game productons. Conventonal vdeo based moton capture systems use many vdeo cameras to obtan accurate, desred moton data so that they cannot generate moton data n real tme snce t takes a long tme to deal wth many vdeo mages. Consequently t s mpossble to use them as a real-tme nput devce of human moton. Moreover such systems take much cost and requre a huge space so that they are not sutable for an nput devce of a standard PC. In ths paper, we propose a real-tme, vdeo based moton capture system usng only one vdeo camera for a standard PC. We employ a very smple moton-trackng method based on color and edge dstrbuton. Usng ths method, our current system s capable to track the upper body, hands, a face, etc, and generates ther moton data n real tme. Ths paper manly descrbes ts aspects as a hand and face-moton capturng devce for a standard PC. The system takes vdeo mage of the hand from one vdeo camera and extracts ts x, y, z poston data per frame. Ths nformaton can be used as an nput data lke the mouse-devce moton. Furthermore the system recognzes specfed shapes of the hand, e.g., a paper or a stone shape. Ths nformaton can be used as an nput data lke the mouse-devce button press or release. Moreover our system also has a network communcaton faclty and t works as an nput devce dedcated for an nteractve 3D graphcs applcaton runs on another computer. If there are two systems connected wth each other through the network, they can work collaboratvely. Many works on the vdeo based moton capture system have been made so far [3]. Ther standard method for trackng the human moton s based on constructon of 3D shape as voxel data from several slhouette mages [][2]. However, ths process needs huge computaton tme. Some partcular technques and other constrants are necessary to reduce ths computaton tme. Wek and Ledtke[4] proposed a herarchcal method for 3D pose estmaton. Luck et al [5] proposed a real-tme algorthm wth reducng jonts and ther degrees of freedom of a human body. These systems use four vdeo cameras at least and need a huge performance space. Our system uses only one vdeo camera. Already some methods that use one vdeo camera are proposed [6][7] but our method s smpler than them. The remander of ths paper s organzed as follows. Secton 2 explans system overvew. Secton 3 explans trackng algorthm. Secton 4 shows applcaton examples. Fnally Secton 5 concludes ths paper. 2 System overvew Frst of all, as an overvew of the system, ths secton brefly descrbes ts hardware archtecture and software archtecture separately. 2. Hardware archtecture

2 The system hardware conssts of a standard PC, a vdeo capturng board, and a vdeo camera. If there are two systems connected wth each other through the network as shown n Fgure, they communcate wth each other and work collaboratvely. Ths hardware generates moton data by extractng person's mage from each frame of vdeo camera mages and by computng the dfference between two adjonng person's mages. Ths moton data s used as an nput data for other applcaton. Usng the network communcaton faclty (network thread n Fgure 2), ths moton data s sent to other applcaton runnng on another computer through the network. 2.2 Software archtecture The software archtecture has two man thread,.e., trackng thread and applcaton thread, as shown n Fgure 2. Trackng thread tracks the person's moton, generates moton data and sends t to a 3D graphcs applcaton,.e., applcaton thread. Vsualzaton thread dsplay a vrtual person as anmaton accordng to the moton data on a dsplay screen. Ths s used for checkng the moton trackng. Fnally network thread s a network communcaton faclty tself. Trackng thread sends moton data to other 3D graphcs applcatons runnng on a dfferent computer through ths trackng thread. 3 Trackng Ths secton explans how to track the person's moton. Before trackng, the system requests an ntalzng process. And then the system starts the trackng process. 3. Intalzng process As prevously mentoned, the system tracks the person's moton by extractng person's mage from each frame of vdeo camera mages and by computng the dfference between two adjonng person's mages. Frst of all the system needs to store a background mage excludng a person as an ntal treatment. After storng the background, the system starts to track the moton. For each vdeo frame n the trackng process, the system extracts the slhouette of a person by subtractng the stored background mage from the current vdeo mage, and extracts a person mage usng ths slhouette as shown n Fgure 3. As explaned n next subsecton, the moton trackng s based on the color nformaton, the system needs to store an ntal state of the person's mage color Informaton. The system requests the user to perform hs/her ntal pose n order to obtan the color nformaton of each trackng area of the user body as shown n Fgure 4. Fg.. Hardware archtecture Fg. 3. Person mage extracton Fg. 2. Software archtecture Fg. 4. Intal pose settng

3 3.2 Trackng The moton trackng s done based on the color nformaton of each specfc area of the body. Strctly speakng, the medan pont of the color nformaton s used as the center of the correspondng focus area. It s calculated usng the equaton (). m ( x y ) c, = C( x, y) = () m where C s the centrod of the color dstrbuton. c s -th color pont, m s the number of color ponts However practcally the color nformaton s nsuffcent for trackng the moton robustly. For example, the color of the skn s unformly dstrbuted over the arm as shown n Fgure 5. So f one wants to track the hand, ts color center s nfluenced by the arm color and t moves to the center of the arm area gradually. Consequently the system wll loose the focus area. To compensate ths weakness, we employ new measure concernng the edge dstrbuton besdes the color nformaton. Smlarly to the color nformaton, the medan pont of the edges, whch are the contour pxels of a focus area, s used as the center of the area. It s calculated usng the equaton (2). n ( x y ) e, = E( x, y) = (2) n where E s the centrod of the edge dstrbuton. e s -th edge pont, n s the number of edge ponts. The edge centrod s always located on the upper part of the hand. So the system does not loose the focus area. However, the edge centrod s strongly nfluenced by the hand shape change. Therefore, we use weght values for both the color centrod and the edge centrod. As a result, the focus area becomes stable. The centrod of the focus area s calculated usng the equaton (3). wee ( ) ( x, y) + wcc( x, y) P X, Y = (3) we + wc where P s the centrod of the focus area. w e s the weght of the edge and w c s the weght of the color. Above s the essental mechansm for trackng the moton, especally the hand moton. Next part s concerned the face trackng. For the face trackng, we use an ellpsod model, the contour of the user face as shown n Fgure 6. Ths modelng process s also performed n the ntalzng process. Then the system tracks the face moton by measurng three factors,.e., the centrod poston, the length of long and short radus of an ellpsod model. For example, when the user face turns up/down, ts centrod moves up/down vertcally and ts long radus becomes shorter as shown by the chart () n Fgure 7. Furthermore, when the user face turns left/rght, ts centrod moves left/rght horzontally and ts short radus becomes shorter as shown by the chart (2) n Fgure 7. In fact, these two charts are theoretcal cases. Practcally these two charts depend on the user s face shape and ts behavor. Therefore, t s necessary to actually measure ths nformaton and to buld such charts as shown n Fgure 7 before the trackng. 3.3 Moton data As descrbed n prevous subsecton, our system generates x, y locaton data for each trackng area. Ths s enough for most cases. However, for some cases t s not enough. For example, n a vrtual realty applcaton, usually we need 3D poston data for manpulatng a 3D object. Therefore, we employ another measure concernng the depth. Fg. 5. Computng focus pont Fg.6. Gettng the radus of face

4 The depth value s determned by the sze of a focus area as shown n Fgure 8. Ths reason s easy to understand because the sze of an object far from the camera poston s smaller than near. 3.4 Shape recognton Furthermore the system recognzes some shapes of a specfc object besdes generatng moton data. Currently t can recognze the hand shapes, e.g., a stone, a paper and so on. To recognze a requested hand shape, the system calculates the dfference between a current hand mage and a canddate hand shape mage. Actually the system compares two hstograms of ther edge dstrbutons. A hstogram s generated usng equatons 4 and 5. As shown n Fgure 9, equatons (4) and (5) mean how each pont of the edge s dstrbuted from the centrod of the hand mage. D = { D, D2, K, D, K, D n, D n } (4) where D s a set of all edge dstances from ther centrod. D s -th edge dstance calculated by the followng equaton. up and down left and rght D ( x X ) 2 + ( y Y ) 2 = (5) where X and Y are coordnates of the center calculated by equaton 3. x and y are the coordnates of -th edge. Fgure 0 shows two typcal hstograms of a stone shape mage and a paper shape mage. Snce a dfferent hand shape mage has a dfferent hstogram, to compare two hstograms of two hand shape mages dstngushes them. Our system currently recognzes the rough shapes, e.g., a stone shape, a paper shape and so on. We wll mplement more effcent technque to make our system able to recognze wder range of hand sgns [8]. As explaned n ths secton, consequently, our system outputs 3D moton data of a hand and rotaton data of a head, and furthermore a symbol value correspondng to a hand shape. 4 Applcaton examples Ths secton ntroduces an applcaton example. 4. Vrtual realty applcaton Ths s an nteractve 3D graphcs applcaton example. Ths applcaton s developed usng IntellgentBox[9, 0.8 r/r rotaton(rad) Fg. 7. Radus changes whle head turnng: These charts mean the changes of the rad when movng up/down, and movng left/rght. count/edge dstrbuton count Fg. 9. Shape recognton by edge dstrbuton stone paper dstance/radus of focus area Fg. 8. Depth changes and focus area szes Fg. 0. Edge dstrbuton: two typcal hstograms of a stone shape and a paper shape.

5 0], a constructve vsual 3D software development system. There s an avatar n a 3D vrtual space. The user controls ths avatar usng our moton capture system. Ths avatar manly conssts of a body and a head wth a camera attached. The camera s used to dsplay the avatar s vew on a computer screen. The avatar has fve degrees of freedom,.e., go forward/backward, go left/rght, and turn left/rght for the body, and turn left/rght and turn up/down for the head as shown n Fgure. Therefore, ths applcaton needs three values to control the body moton and two values to control the head moton of the avatar. So the user uses hs/her one hand and hs/her head to control the avatar as shown n Fgure 2. Frst the user tred to control the locaton of the avatar by hs left hand moton. However ths requests the user to keep hs/her left hand up durng operatons because the movement of hs/her left hand always nfluences the movement of the avatar. Ths s not convenent, so we added another operaton usng a symbol value correspondng to a hand shape. That s, the stone shape of a hand allows the avatar to move, and the paper shape of a hand forbds t. As for controlng the avatar s head, the user rotates hs/her head up/down or left/rght. Ths does not have any problem. Then the avatar moves freely n the 3D vrtual space by the user manpulatons as shown n fgure 3. Fgure 4 shows an mage seen from the vewpont of the avatar. Ths vew mages are sent from the camera attached to the avatar head. Actually ths mage s dsplayed on a computer screen. Then by lookng at the vew mages, the user can feel as f he s n the 3D vrtual space and can operate the avatar effcently. Furthermore, as shown n Fgure5. If the user wants to feel more mmerson, he can use a head-mounted dsplay nstead of a standard dsplay montor. However, n ths case, t becomes mpossble to track the face because the head-mounted dsplay dstrubs t. Then n ths case, the trackng s appled to the head-mounted dsplay nstead of the face. Fg. 3. Walkthrough n 3D vrtual space Fg.. Avatar Fg. 2. User control Fg. 4. Avatar s vew

6 of the proposed algorthm wth the help of an applcaton example. As future work, we wll develop more applcaton examples and evaluate ther performance to mprove our algorthm. Then we wll present ts new fndngs. Fg. 5. Head mounted dsplay Fg. 6. Communcaton wth other user Moreover, as prevously mentoned, our system provdes a network communcaton faclty. Usng ths faclty, multple users can work collaboratvely. For example, as shown n Fgure 6, when some users use ths same system on dfferent computers, each of them can control hs/her own avatar smultaneously. 5 Concluson Ths paper proposed the real-tme, vdeo based moton capture system usng only one vdeo camera. Snce conventonal vdeo based moton capture systems use many vdeo cameras and take a long tme to deal wth many vdeo mages, they cannot generate moton data n real tme. Therefore they cannot be used as a real-tme nput devce for a standard PC. On the other hand, our proposed system uses only one vdeo camera and generates moton data n real tme snce our system employ a very smple trackng mechansm based on color and edge dstrbuton of trackng focus areas. So our system can be used as an nput devce for a standard-pc. In ths paper, we clarfy usefulness Acknowledgements The work descrbed n ths paper s partally supported by Mnstry of Educaton, Culture, Sports, Scence and Technology of Japan. References: [] D. Snow, P. Vola, and R. Zabh, Exact voxel occupancy wth graph cuts, n Proc. IEEE CVPR, 2000 [2] K. M. Cheung, T. Kanade, J. Y. Bouguet, and M. Holler, A real tme system for robust 3D voxel reconstructon of human motons, n Proc. IEEE CVPR, 2000 [3] D. M. Gravrla, The vsual analyss of human movement: A survey, CVPR, vol. 73, pp ,999 [4] S. Wek, and C.-E. Ledtke, Herarchcal 3D pose estmaton for artculated human body models from a sequence of volume data, Robot Vson 200, LNCS 998, pp , 200 [5] Jason Luck, Dan Small, and Charles Q.Lttle, Real-tme trackng of artculated human models usng a 3D shape-from-slhouette method, Robot Vson 200, LNCS 998, pp.9-26, 200 [6] C. Wren, A. Azarbayejan, T. Darrel, and A. Pentland, Pfnder: Real-tme trackng of the human body, IEEE Trans. Pattern Anal. and Machne Intell., vol. 9, no. 7, pp , 997 [7] Y. Wu and T.S Huang, Color trackng by transductve learnng, n Proc. IEEE Conf. Computer Vson and Pattern Recognton, vol., pp , 2000 [8] Y. Cu and J. Weng, Hand sgn recognton from ntensty mage sequences wth complex background, n Proc. IEEE Conf. Computer Vson and Pattern Recognton, pp , 996 [9] Okada, Y. and Tanaka, Y.: IntellgentBox: A Constructve Vsual Software Development System for Interactve 3D Graphc Applcatons, Proc. of Computer Anmaton '95, IEEE Computer Socety Press, pp.4-25,995. [0] Okada, Y. and Tanaka, Y.: Collaboratve Envronments of IntellgentBox for Dstrbuted 3D Graphcs Applcatons, The Vsual Computer, Vol. 4, No. 4, pp , 998.