4th International Conference on Computer Integrated Manufacturing CIP 2007

Transcription

1 4th Internatonal Conference on Computer Integrated Manufacturng CIP 27 Desgn of a gestual servong nterface n telerobotcs N.E. Berrached and H. Abdelmoumene Intellgent Systems Research Laboratory, Electronc Departement, Faculty of Electrc Engneerng, USTO- BP. 55 Oran El M naouer 3 ORAN Emal : nasrberrached@yahoo.fr, nfvrf6@gmal.com Abstract Ths paper descrbes the realsaton of a real tme gestual servong of the manpulator arm «Mentor», whch s a 5 dof robot, n telerobotcs. Our goal s to ntroduce a natural communcaton between man and robot, through the machne, whle usng the rchness of the expressons provded by hand gestures. The command of ths system s based on the nterpretaton of 9 hand gesture confguratons recognzed by usng the prncpal components analyss method; each gesture s represented by two egenvectors. The realzaton of a such system of gestual servong generates a partcular study of two mportant parts, as well as the establshment of the relaton between them. The two parts are the gesture recognton and the control of the robot. I. INTRODUCTION Teleoperaton or telerobotcs s to control remotely a system remotely by a Clent ste connected to a Server va a Man- Machne nterface. Current progress s made for wdenng the doman of ths nteracton and makng t rcher and more natural by ntroducng the concept of gesture, and more precsely the hand gesture, because ths can express ether one or many smultaneous orders thanks to the number of degrees of freedom whch t ncludes. Ths last ntutve, natural and very sgnfcant faculty allows a very apprecable evoluton n the world of robotcs, ths new system of control s called gestual servong. Nevertheless, the process of gestual nterpretaton needs a tool for nformaton acquston, whch mples the provson of a vson sensor and thus computer vson technques. There were efforts n the teleoperaton doman to allow human nterventon by usng an average naturalness n order to permt the drect and natural robots nteracton command n a closed loop. Applcatons n ths doman are varous and extend from day to day. Ths varaton s due prmarly to the varous methods suggested for gesture recognton; because t s an mportant part whch allowed progress n Man-Machne nteracton. In [], hdden Markov model was appled to the recognton actvtes usng probablstc sensors. The method used by [2] for the gesture recognton s based on the fuzzy descrpton of the hand confguraton. [3] used the centre of the fst (CP) as a reference mark for the gesture recognton. [5] and [6] used characterstcs of Freeman codng for gestural nterpretaton, such as the gravty center and the hand surface. The applcaton of gesture recognton s a renewal n the telerobotc doman but consderng the dffculty of the realzaton of a such system; whch needs a control of several research domans; few works were done. Among them [4] whch used the Fuzzy C-means clusterng for the gesture recognton n order to command an arm manpulator va Internet n real tme. In our applcaton, we chose the Prncpal Components Analyss (PCA) to dentfy the tracked gesture. II. OBJECTIVE Our work s ntegrated wthn the feld of network based telerobotcs, whch s to control a robot n a closed loop, wthn a local area network, and n real tme. Ths control wll be carred out by a gestural nterface of 9 hand gesture confguratons, usng computer vson. To be done, we have to dentfy a vocabulary whch determnes the gesture sgnfcaton. See Fg. Hand gesture Camera Image processng Camera "Mentor" recognton Fg.. Dataflow of gestual servong system n telerobotcs. The realzaton of such nterfaces meets many obstacles: - The nature of the gesture: the problem les n the number of mages to process n the case of dynamc hand gesture. - The dynamcs of lghtng and the complexty of the scene.

2 4th Internatonal Conference on Computer Integrated Manufacturng CIP 27 - Important delay for the gesture detecton and dentfcaton. To solve those problems, we used a black glove, to be able to effectvely extract the hand from the totalty of the scene, based on the colour n order to reduce the executon tme as much as possble. III. IMPLEMENTATION The mplementaton of such gestural servong system ncludes two parts: - recognton, - The robot control. A. recognton: Ths stage represents the second part of the system, because startng from a hand gesture; a robot command can be confgured. It then conssts n nterpretng the confguraton of the tracked hand. Processng wll be appled to the acqured mage n order to extract nformaton whch the gesture wll carry, whle takng nto account the choce of these operatons n such way that they can probably gve the maxmum of nformaton n a reasonable tme. Ths s mportant snce the effectveness of our system s descrbed by ts reacton n real tme. The process of recognton s represented by the dagram below: Fg. 2 Frame grabbng Interpretaton Fg. 2. recognton process. The processngs, appled to all acqured mages n order to elmnate any nose generated from segmentaton, are: - Converson nto grey levels, - Bnarsaton, - Average flterng: It s based on the calculaton of the average of the pxels closed to the central pxel. Ths mask s defned by the followng matrx. masque = Pre-processng Extracton of charaterstcs Classfcaton Data base () - Medan flterng: The prncple of ths flter s that the new value of the pxel wll be the medan of the neghbourhood values. - Extracton of the square wrappng only the shape of the hand. - Redmensonng of the mages n the face 32 32, After the extracton stage comes the gesture classfcaton stage. As mentoned prevously, the method adopted n ths realzaton s the prncpal components analyss (PCA) for the gestures classfcaton, because t allows: - to gve nterestng results n a reasonable tme, - to reduce n a very mportant way the unt of ntal varables. A.. Phase of tranng: A... Constructon of the mage data base: We took for each hand confguraton, specfed n the gestural vocabulary consdered for the command of the robot, bnarzed mages, standardzed, normalzed and redmensonned n Images n the same class have the same confguraton wth dfferent appearances A..2. Covarance matrx: Ths matrx s an N vectors l, of N M dmensons, of M mages of lght ntensty, represented n columns form. Each vector l s standardzed by the subtracton of the average mage l such as: l = = = m l m The standardzed mages form the new matrx B such as: B [ ˆ, ˆ, ˆ,..., ˆ ] 2 3 m (2) = (3) The N N covarance matrx s defned by: Cov. m t = B B (4) After computng the covarance matrx, t s necessary to calculate the egenvalues and the egenvectors by solvng the followng lnear system: Cov. U = U.λ (5) where U represents the N egenvectors matrx and λ s the dagonal matrx of egenvalues. A.2. Classfcaton : Classfcaton conssts n projectng the new confguraton n all spaces and calculatng the dstance to these spaces. Whle seekng the ndex of the class whch mnmzes the error of reconstructon for all the class C :

3 4th Internatonal Conference on Computer Integrated Manufacturng CIP 27 Knowng that: e = arg mn e * = T ( T ( l)) l t φ = T( l) = E ( l l) (8) (8) * T ( φ ) = E. φ + l Transformatons T and T * denote projecton and partal reconstructon of an mage. B. of robot Mentor: Once the tracked gesture s recognzed, we used t to command the Mentor robot. However, t s necessary to model the robot for command t and execute the desred tasks. The mean crtera needed n order to choose a structure for a gven applcaton are: - Geometrcal performances: form and dmenson of ts work volume. - Characterstcs of the task to be executed. Consderng the nature of the task to be realzed whch s the sezure of an object wthout takng nto account speed and acceleraton for ts realzaton, and consderng the mechancal structure of the Mentor robot, the model chosen for the command of the arm manpulator Mentor s the geometrcal model. To provde geometrcal modelng, we show the process of the tradtonal modelng whch conssts n confgurng the robot geometrcally. To do that, two hypothess were consdered: - The segments of robot are nfntely dependent (rgd), they are ndeformable solds. - All the artculatons are perfect; wthout plays or frcton. The geometc model s based on the Denavt Hartenberg conventon [7]. B.. The drect geometrcal model (DGM): The MGD allows to obtan the coordnates (x, y, z) poston of the effector compared to the robot base, accordng to the parameters of translatons and/or rotatons gven by the system. ( )( ) T Cos θ Snθ Cos α = Snθ Snα Snθ Cos θ Cos α Cos θ Snα Snα Cos α α r Snα r Cos α (6) (7) (9) () And the poston of the fnal organ of robot manpulator s descrbed by the followng equaton: X = f ( q ) () wth x : operatonal varables (poston and orentaton). q : artcular varables B.2.The nverse geometrcal model (IGM): The nverse geometrcal model conssts on calculatng the artcular coordnates whch brng the fnal organ n a wshed stuaton, specfed by ts operatonal coordnates. The nverse geometrcal model robot manpulator s wrtten as: X Wth: f -: the nverse functon of f. IV. = ( q (2) The prncple of the Mentor robot control n real tme conssts n fndng the space correspondng to the confguraton whch gves the mnmal dstance, durng t projecton on 9 spaces and takng nto account the precson. We fxed a threshold of error n order to avod any confuson between gestures. The followng step s to translate the gesture nto an order accordng to the assocated gesture for each artculaton. See Fg.4. f IMPLEMENTATION OF THE SYSTEM We nstalled our applcaton on a Clent staton, and a Server applcaton on a staton consdered as Server whch s connected to the Mentor robot. The prncple s to send the angles of each artculaton startng from the Clent applcaton to the Server staton to send t n ts turn to the Mentor robot. The return wll be transmtted by the Server to the Clent staton whch comprses the new poston of the robot expressed by the angles of each artculaton. The same processng s done for the next gesture untl puttng an end to the applcaton, because our system s a gestural servong n real tme. The followng loop represents the sequence of the varous stages whch consttute ths system: See Fg. 3 Consgn Gestur recognton Fg.3. Gestual servong loop. V. RESULTS We desgned a gestual servong system, developed n Bulder C++, for the Mentor robot n teleoperaton and n real tme. Intally, as shown below, we tested our system n a ) s

4 4th Internatonal Conference on Computer Integrated Manufacturng CIP 27 vrtual model of the robot desgned n OpenGl, and then we made control n realty. trackng Projecton of the confguraton of the hand G on 9 spaces P. dfference s n the type of transmsson used for the data transfer to the robot. Ths data transfer s drect as t s the case n the vrtual control on the same staton, and t s made through a Server n the real case. Fg. 5 and 6 The followng dagram (Fg. 7) ncludes all the modules used n the realzaton of such an nterface, as well as the applcatons used such as the software Webcam32, for the vsual return and a FTP Server, TYPSoft ftp server for ts transmsson. No To seek the space P whch has the mnmal error of projecton of G P < Threshold Yes Movement of artculaton I concerned wth gesture I Hand gesture FTP Server Clent Webcam recognton s Fg. 4. Flow dagram of the vrtual robot control. Webcam32 Return Mentor Feed-back Server Fg. 7. System of robot Mentor. The followng fgure 8 represents the learnng stage usng PCA method wth the 9 hand gestures command and whch contans a data base module. Ths base comprses the 8 egenvectors representng the hand gesture base. Fg. 5 Interface of command of the vrtual robot. Fg. 8. PCA based learnng Interface. Fg. 6. I Interface of command of the real robot.. The results of our applcaton concernng both vrtual and real models of the robot, were almost smlar, the only The dentfcaton of the gesture s done by projecton on nne spaces, whch makes the executon expensve n term of tme (ncludng the varous mage processng tme). See Fgure 9. The second dffculty s that mage compresson leads to confuson between some gestures. As a soluton to ths

5 4th Internatonal Conference on Computer Integrated Manufacturng CIP 27 problem, we had to reduce the threshold used n the recognton process. The problem wth ths soluton s that t s necessary [5] A. Ahmed Blaha et L.Bouhalassa, «e gestuelle d un bras manpulateur», USTO-LARESI, Algére, 26. [6] M. Zalegh et A. Kerroum, «Asservssement gestuel d'un robot vrtuel», USTO-LARESI, Algére, 26 [7] E. Dombre et W. Khall, «Modélsaton et commande des robots», Ed. Hermes, 988. Projecton t n- t t n+ Processng Processng Processng Projecton Projecton T Iteraton n- Fg. 9. The tme passed durng the mage processng captured for each teraton of the robot. to express a good gesture whle tryng to approach the hand gesture data base as much as possble. See Fgures and Fg.. Example of two gestures whch resemble each other Fg.. Example of two dstnct gestures. VI. CONCLUSION We desgned a gestual servong system of the robot Mentor. The results are encouragng f we do not nclude the delay related to each teraton because of the mage processng. The prncpal components analyss was used for the gestual dentfcaton. The results of dentfcaton were satsfactory n term of rate of recognton. REFERENCES [] J. Martn, «Reconnassance de Gestes en Vson par Ordnateur», Thèse de doctorat, INP Grenoble, France, 2. [2] T. Allevard, «Représentaton symbolque de la confguraton de la man - applcaton à la reconnassance de sgnes et au contrôle d un robot moble», Thèse de doctorat, Unv. de Savoe, France, 25. [3] N. Kawarazak, T.Yoshdme, and K. Nshhara, «An Assstve System Usng and Voce Instructons», 2 nd Cambrdge Workshop on Unversal Access and Assstve Technology, CWUAAT 2, 22. [4] W. Juan and K. Ur, «Hand Telerobotc Systems usng Fuzzy clusterng algorthms», Ben Guron Unversty, Israel, 2.