3D Hand Trajectory Segmentation by Curvatures and Hand Orientation for Classification through a Probabilistic Approach

Transcription

1 3D Hand Tajectoy Segmentation by Cuvatues and Hand Oientation fo Classification though a Pobabilistic Appoach Diego R. Faia and Joge Dias Abstact In this wok we pesent the segmentation and classification of 3D hand tajectoy. Cuvatue featues ae acuied by computing (, θ, h), and the hand oientation is acuied by appoximating the hand plane in 3D space. The 3D positions of the hand movement ae acuied by makes of a magnetic tacking system [6]. By obseving human movements we pefom a leaning phase using histogam techniues. Based on the leaning phase, we classify each-to-gasp movements applying Bayes ule to ecognize the way that a human gasps an object by continuous classification based on multiplicative updates of beliefs. We ae classifying the hand tajectoy by its cuvatues and by hand oientation along the tajectoy (individuall. Both esults ae compaed afte some tials to veify the best classification between these two types of featues extaction. Using entopy as confidence level, weights fo each classification model ae assigned, allowing the combination of both types of featues in a mixtue model, acuiing a new classification model fo esults compaison. Using these techniues we developed an application to estimate and classify two possible types of human movements (eaching fo top and side gasps). These epoted steps ae impotant to undestand some human behavios befoe the object manipulation and can be used to endow a obot with autonomous capabilities (e.g. eaching objects fo handling). R I. INTRODUCTION obotics is moving towads the eseach and development of technologies that pemit the intoduction of the obots in ou daily lives. To ceate such applications some poblems need to be solved, including gasp stategies. Applications of sevice obots will euie advanced capabilities of gasping and skills that allow a obot to gasp diffeent types of objects in diffeent ways. Some of the most pefomed actions by humans in thei daily activities involve the handling of objects fo a specific task. The study of human each-to-gasp movements is impotant fo eseaches of diffeent aeas. In compute science field, hand tajectoies segmentation and classification ae useful fo humanmachine inteaction using gestues to inteact with machines, e.g., the hand can be used as compute mouse. Many appoaches have been poposed fo pedicting hand tajectoies. Hand tajectoy segmentation and classification ae useful also in obotics field fo imitation leaning fo human-obot inteaction. Typically, the global hand s tajectoy duing a manipulation task can be segmented into This wok is patially suppoted by the Euopean poject HANDLE ICT Diego Faia is suppoted by Potuguese Foundation fo Science and Technology. Diego Faia and Joge Dias ae with Institute of Systems and Robotics Univesity of Coimba Polo II, Coimba, Potugal (diego, joge)@is.uc.pt. diffeent stages: each, lift, tanspot and elease []. We focus ou attention in the each stage (each-to-gasp movement). Ou intention is developing an automated system fo tajectoies segmentation and classification by a pobabilistic appoach. In this wok, classification of eachto-gasp movements when someone is pefoming the gasping is pesented. Analyzing these movements we can be able to undestand some human behavios duing the hand jouney to each and gasp an object. This infomation can be used to endow obots using the movements befoe the object manipulation, i.e. using it as capability of a obot ecognizing how a human gasp an object to imitate his action. This appoach can also be applied fo gestue ecognition tasks. II. RELATED WORK Hand tajectoies have been studied in diffeent aeas such as neuoscience, obotics, egonomics, etc. In [], a modelling appoach fo 3D hand tajectoies in eaching movements is descibed. The authos use Bézie cuves fo geometical intepetation. Thei pupose is to descibe a modelling appoach to show how the tajectoies depend on some pedictos and how they vay fom epetition of the tajectoies. Bayesian models have been used in [3] to classify gestues fom images seuences. Tacking of human hands and face ae used based on skin-colo featues towads human-obot inteaction. The human actions ae intepeted and mapped to the obot actions. The authos have contibuted also with Laban Movement Analysis that assist identifying useful low-level featues to develop a classifie of expessive actions. Images seuence wee used in [4] fo hand tacking and hand shape epesentation when a peson is gipping a mug. They poposed a method fo hand shape epesentation that chaacteizes the finge-only topology of the hand using cepstal coefficients. Techniues of speech signal pocessing wee used fo that. This wok shows hand shape ecognition classified as top-gab, side-gab, flat-hand and handle-gab when the hand is close to object. In ou pevious wok [5], we developed an application to segment a tajectoy to find featues like up, down and line fo its classification. We have used second ode deivative to analyze the evolution of the tajectoy finding featues using just the x and y axis of a 3D tajectoy ignoing othe featues like diagonal, fowad and backwad diections. The classification esults wee satisfactoy, but we obtained undesied esults as false negative, and classification of the tajectoy with low pobability.

2 III. EXPERIMENTAL SETUP AND CONTEXT Polhemus Libety tacke [6] is used to tack the human hand tajectoies. Five sensos wee attached to a glove in ode to gab 3D hand tajectoies. One senso was also attached to the object to have a pio knowledge of the object position. The setup fo the expeiments is compised of a wooden table, without any metallic pats, since the magnetic tacke is sensitive to neaby feomagnetic mateials. The expeiments ae executed by a subject standing in font of the table fo the eaching tasks. The tabletop is 50cm by75cm and is placed at a height of 00cm. The object is placed on the cente of the tabletop in a maked egion fo all expeiments having the object in the same position. The magnetic tacke emitte unit that detemines the fame of efeence fo the motion tacking system is placed on anothe table nea to the object table. Thee is no any specific aea fo a subject stats the tajectoy to the taget. Usually the subject is positioned close to the table vaying the distance until one mete fa fom the object. Fig. shows the expeimental aea setup. Two each-to-gasp movements wee defined fo this wok: Top-Gasp and Side-Gasp (Fig.). The side-gasp happens when a peson wants to gasp the object by its side o by its handle. The top-gasp usually happens when someone wants to gip the object by its top just to displace it type of movement with diffeent distances. The subjects can stat the tajectoies in diffeent places eaching diffeent sizes of tajectoies esulting diffeent scales which can ham the esults. To solve this poblem, we ae nomalizing all tajectoies to have the size. To extact the featues we ae splitting the tajectoy in 8 simila pats (each one epesenting a hand displacement) to detect the featues and then fo each pat we can chaacteize the movement by these phases. The division of the tajectoy in 8 pats was chosen empiically. Howeve a moe sophisticated appoach can be applied, e.g., segmenting the tajectoies by events that chaacteize a manipulation tasks (e.g., eaching, lift, tanspot, and elease phases). Thus, the movements can be initialized fom diffeent positions with diffeent velocities without influencing the esults. To obtain the nomalization of a tajectoy, fo all points of each axis (x, y, z), the following euation is applied to escale it: X max min R cu min () whee R is the escaled point; X is the new size of the tajectoy (in ou case the size of the tajectoy is ); max epesents maximum value of the aw data found in the cuent axis, min is the minimum value found; and cu is the cuent value of the tajectoy that is being nomalized. A tajectoy smoothing is also necessay. Fo each point of each axis is calculated the mean value among its pevious fou neighbous and its fou fowad neighbous. Fig.3 shows an example of smoothing. Fig.. Expeimental setup used fo this wok. Fig3. Smoothed tajectoy: Blue colo aw data; Red colo esult. Fig.. (a) Side-gasp; (b) Top-gasp. IV. SEGMENTATION AND FEATURES EXTRACTION A. Pe-Pocessing step We ae not consideing tempoal analysis of the tajectoy to avoid some poblems. Fo example, if a movement was leaned with tajectoies pefomed in 0 seconds, and when a movement of same type is pefomed slowly, moe than 0 seconds, then this movement will not be consideed as the same of the leaned one, because the featues will not coespond to the leaned ones. We ae consideing the spatial infomation instead. Even consideing the spatial infomation we can find some difficulties to classify the same B. Tajectoy Cuvatue Featues As long as the tajectoy is in 3D space, fo bette cuvatue detection we can wok in cylindical (, θ, h) o spheical coodinate system (, θ, φ). Using two points of the tajectoy we have the vectos epesentation in 3D space. The angle fomed between these two vectos by the pojection on (x, plane we achieve the θ angle, which give us the pan infomation, if the angle is inceasing, we have the cuvatue left, o if it is deceasing we have the cuvatue ight. The same vectos and thei fomed angles by the pojection on (z, plane, we obtain the φ angle fo tilt infomation. In 3D space we can make some combinations of the possible diections, e.g., we have up and down obtained by h, left and ight obtained by θ and futhe and close obtained by. In this wok we detect the following discetized featues: up, down, left, ight, up-left, up-ight, down-left, down-ight and no-movement, esticting othe infomation, such as close

3 and futhe. We can obtain the height infomation (h) in a simple way using the cylindical coodinate system, calculating the diffeence between the z axis values using two points. In spheical coodinate system just the φ angle cannot infom us the height o diagonal movements, being also necessay veify the adius changes (). To know up o down, φ and change and θ emains the same. In cylindical coodinate system we need to combine, θ and h to know discetized featues like up-ight, up-left, down-ight and down-left. The cuvatue extaction is pefomed at each two points obseved fom the tajectoy. The next steps show us how to obtain (, θ, φ) in spheical coodinate system. Given two 3D points, fo the fist point we compute: x y z, () x y sin, (3) z cos, (4) actan (sin,cos ), (5) x cos, (6) x y y sin, (7) x y actan (sin,cos ). (8) Then with the second vecto acuied (i.e., the second 3D point) we follow the same steps pesented by euations ()-(8), obtaining then, φ and θ. Afte that, we obtain the θ angle and tilt infomation (height) given by φ angles as follows: h cos cos, (9). (0) In the cylindical coodinate system, to find the height infomation, we can simplify ignoing the euations () to (6), and we can ewite euation (9) as follows: h z z () To find a featue c we use the following ules: whee (x, is the adius in cylindical coodinate system epesented in (x, plane. It is obtined as follows:, (3) ( x, ( x, ( x, whee and ae given by: ( x, x y, ( x, x y (4). (5) If h, θ, and ae eual to zeo, then thee is no movement. Splitting the tajectoy we can chaacteize the tajectoy so that each pat could distinguish a type of gasp. Afte cuvatues detection, the pobability distibution of these featues in each pat of the tajectoy is computed. Fo each featue is computed the pobability distibution as follows: ci, k P ( ci ) k, (6) c whee c i,k epesents the occuence of a specific cuvatue in a specific hand displacement (tajectoy pat) and the denominato in (6) is the total of occuences of all cuvatues types found in each segment of the tajectoy. i, C. Hand Oientation Featues Using thee sensos on thee fingetips we can appoximate the hand plane, allowing computing its oientation to find out if the hand is in top o side oientation (Fig.4). We have used the thee finges (index, middle and ing finges) that usually emain paallel in the most pat of hand configuation fo gasping. These thee 3D points fom the hand plane and afte computing the nomal of the hand plane we compae it with the z axis of the Polhemus fame of efeence to know the hand oientation. At each 3 points in each pat of the tajectoy, the hand oientation is computed. In each pat of the tajectoy is found the occuences of each type of hand oientation. Thus, pobability distibution is given by: oi, k P ( oi ) k, (7) o whee o i,k epesents the occuences of hand oientation (side o top gasp) in a specific tajectoy pat and the denominato in (7) sums the total of occuences of all hand oientation featues found in a specific tajectoy pat. i, c = height > 0 and θ ~ 0 and (x, ~ 0 Up height < 0 and θ ~ 0 and (x, ~ 0 Down height =0 and θ > 0 and (x, = 0 Right height = 0 and θ < 0 and (x, = 0 Left height > 0 and θ > 0 and (x, ~ 0 UR () height > 0 and θ < 0 and (x, ~ 0 UL height < 0 and θ > 0 and (x, ~ 0 DR height < 0 and θ < 0 and (x, ~ 0 DL Fig.4. (a) Possible hands oientation fo side-gasp; (b) Possible hands oientation fo top-gasp.

4 D. Results of the Segmentation Step Fo each obsevation of ou dataset xml files that stoes the chaacteization each the tajectoy wee ceated, i.e. segmentation infomation: featues amount and thei pobability distibution. Two xml files fo each tajectoy was geneated, one with cuvatues and anothe with hand oientation infomation. This infomation is useful to fo leaning using histogam techniues that will be used in the classification step as likelihoods. Fig. 5 shows an example of top-gasp tajectoy. Table shows the esult of tajectoy segmentation by hand oientation acuied fom the tajectoy shown in Fig. 5. The same pocess of table is done fo the segmentation by cuvatues. ou dataset. Due to the leaning being achieved though histogam techniues, some featues might have zeo pobability, because they neve have been obseved. Wheneve these featues with zeo pobability occu in a classification step, the coesponding hypothesis will also eceive a zeo pobability. Ou classifie is continuous, based on multiplicative update of beliefs and this situation leads to definite out-ule of the hypothesis. To avoid this poblem we ae using the Laplace Succession Law, i.e., poducing a minimum pobability fo non-obseved evidences: ni P( F i), (8) N F whee F epesents types of the featues (e.g. cuvatues = 9, oientation = ); n i epesents total of occuence a specific featue type; N epesents the total of all featue occuences. Fig.5. Top-gasp tajectoy afte smoothing and nomalization. Tab.. Tajectoy Segmentation by Hand Oientation: Result of ou appoach fo the tajectoy shown in fig.5. The second column is the numbe of featues found in each pat; the thid column is the coesponding pobability distibution.. Tajectoy Pats Hand Oientation Side - Top Hand Oientation. Pobab. Side - Top V. LEARNING AND CLASSIFICATION Computational models fo human peception and action has been exploed by eseaches. Some studies about human bain epots that Bayesian methods have achieved success in ceating computational theoies fo peception [7]. Based on these studies, we follow Bayesian techniues to classify hand tajectoies. The leaning phase is based on histogam techniues given segmented featues. A. Gasping Leaning Table Duing the leaning phase, all tajectoies of ou dataset ae analyzed. Given a set of obsevations to epesent a type of Gasping G, at some displacement D, we have the pobability of each type of cuvatue C in each pat of a tajectoy epesented as P(C G D). The same ule is used fo hand oientation leaning, so that we have P(O G D) whee O epesent all possible hand oientation. Since each tajectoy geneates a histogam with pobability distibutions fo each class of featue, then we built leaned tables computing an aveaged histogam fo top and side gasp tajectoies. Fig.6 shows examples of the Gasping Leaning Tables obtained afte analysing all tajectoies of Fig.6. Left image epesents the top-gasp cuvatues leaning table and ight image top-gasp hand oientation leaning table. B. Classification Model using Bayesian Techniue Bayesian classification models have aleady poven thei usability in gestue ecognition systems as we can see in [3]. Based on this study we pesent a Bayesian classification of gasp types analyzing each-to-gasp movements. We adopt a simple dynamic Bayesian netwok, using a Naïve Bayes classifie with pobability tansition (i.e., last posteio becomes the cuent pio pobabilistic loop). Assuming the tajectoy that is being pefomed has size (i.e., we know the tajectoy size a pioi since we have the initial hand position and the mug position given by the sensos, then at each hand displacement coesponding by /8 of the tajectoy the posteio is updated. To undestand the geneal gasping classification model, some definitions ae done as follows: g is a known gasp fom all possible G (Gasp types); c is a cetain value of featue C (Cuvatue types); i is a given index fom all possible hand displacement composed of a distance D ( /8 of a tajecto of the leaned table. The pobability P(c g i) that a featue C has cetain value c can be defined by leaning the pobability distibution P(C G D) as explained in section IV and V. Knowing P(c G i) and the pio P(G) infomation of given tajectoy epesent a top o side gasp, we ae able to apply Bayes ule and compute the pobability distibution fo G given the hand displacement i of the leaned table and the featue c. Initially, the gasp vaiables (pios) G ae a unifom distibution and duing the classification thei values is updated applying Bayes ule as follows:

5 P(G c i) P(c G i) P(G). (9) We assume the same model of classification fo hand oientation featues, whee o is a cetain value of featue O (hand oientation fo side and top gasp). Knowing P(o, i) and the pio P(G) we apply Bayes ule as follows. P(G o i) P(o G i) P(G) (0) We fomulate the euation in a ecusive way. The posteio pobability of a pevious instant (tajectoy pat) becomes the pio fo the next instant (next hand displacement). The ule fo classification is based on the highest pobability value, taking into consideation a cetain confidence (e.g., pobability of 0.7). We expect that a eachto-gasp movement that is being pefomed by a subject to gasp the mug by top o side gasp will poduce a gasp hypothesis with a significant pobability. C. Entopy as Confidence Level fo Classifies Fusion The Shannon entopy H [8] as a measue of the uncetainty associated with a andom vaiable is used in seveal woks; we can see examples in [9] and [0]. In this wok entopy is used as confidence level to ty to impove and obtain a bette classification based on esults of pevious classification. Afte analyzing the classifications esults of tajectoies by hand oientation and by cuvatues, we can apply entopy to veify the best classification between both models in a leaning stage. Fo that, a confidence vaiable will be used as weight w i = {w,, w N } fo each classification model. The weight w will be used fo a mixtue model. Fo each model of classification we can compute the entopy of the posteio pobabilities (outputs of the classifies) as follows: H( P( G F D)) P( Gi F D) log( P( Gi F D)), () i whee P(G F D) epesents the posteio pobability of each classification model. The vaiable i epesents the index of each classification esults, i.e., afte n outputs of one model, we can apply (). Though the entopy H we can achieve the pobability distibution of the weights of each classification (e.g. by cuvatues and hand oientation). The weights ae computed as follows: Hc w, n () H i i0 whee w is the weight esult; H C is the cuent value of entopy that is being tansfomed in a weight; i epesents the index fo each entopy value. Given the confidence of classification, we can fuse the classification belief using the weights obtained by the entopy - this is known as a mixtue model. Then the new model of classification is given by: n P( G F D) w P( g f i) (3) j j j whee P(g j f i), epesents the posteio of each Bayesian model (9) and (0). Each posteio is multiplied fo its coespondent weight, thus, the new classifie is a weighted sum of both models, having this way a mixtue model as an ensemble of classifies. D. Leaning and Classification Results Fig. 7 shows a side gasp tajectoy pefomed by a subject and table shows the answe of ou appoach along this tajectoy, classifying it by using cuvatue and hand oientation featues individually. The pobability of the tajectoy being top o side gasp is updated by Bayes ule in each pat of the tajectoy. The pobability shown in table epesents the confidence of classification fo each pat of the tajectoy. Compaing this case of Fig.7, we can see that both classifications obtained suitable pefomance, coectly classifying the tajectoy. In this expeiment the classification confidence at tajectoy pat by using cuvatues was bette than by hand oientation. Fig.7. Side-gasp tajectoy (afte smoothing and escale). Tab.. Classification using Cuvatues (C) and Hand Oientation (O) fo the tajectoy shown in figue 7. It was classified as side gasp with 98.3% using cuvatues and 9% using hand oientation (O). Tajectoy Pat Top% (C) Side% (C) Top% (O) Side% (O) Following the potocol (section III), two subjects have pefomed each-to-gasp movements to test ou appoach. Table 3 shows the esults of the classification of 0 tials of side gasp using cuvatues featues, using hand oientation featues and combining them though the mixtue model using entopy as confidence level. The false negative values in the classification using cuvatues featues happened due to the fact that the side-gasp tajectoies ae simila to the top-gasp. The classification using cuvatues featues when positive obtained highe confidence than the classification using hand oientation featues, howeve, using hand oientation featues, we did not obtain false negative values. Using the entopy H as an uncetainty measue to assign weights fo each classification model, we obtained the following weights: w cuv = 0.6 and w h_o = Fig. 8 shows a plot compaing these 3 models along 0 tials. The esult obtained by the mixtue model using entopy belief

6 countebalances the esults of both pevious models. We implemented ou appoach using the pogamming language C++. We un the tests in a laptop HP Pavilion dv5000, AMD Tuion 64,.0Ghz, Gb of RAM. The pocessing time fo the segmentation pocess and classification ae on-the-fly. Tab.3. Result of 0 tials of side-gasp tajectoies. Two false negative (belief less than 50%) on tials 3 and 5 using cuvatues. The tials 4, 6 and 0 using hand oientations wee consideed as side-gasp but with low pobability, belief less than 70%. Just one false negative (tial 3) was obtained using entopy to combining both classification models. The tials 5 and 0 wee consideed side-gasp with low pobability. Tial Classification using Cuvatues - Classification using Hand Oientation 3 Mixtue model using entopy to combine both featues 98.3 % 9.00 % 95.85% % % 8.86% 3.67 % 9.53 % 48.8% % 6. % 75.5% % 8.53 % 67.4% % 5. % 80.63% % % 96.08% % 9.53 % 96.68% % % 9.58% % % 69.85% Fig.8. Compaison plot. Results of 0 tials fo two classification models using two diffeent types of featues, and a thid one using weights obtained by entopy to combine the pevious model outputs. To test the efficiency of the poposed method, we pefomed some gestues fo ecognition, i.e., we have leaned moe movements (bye-bye and cicula movements), with 30 obsevations fo both. Table 4 shows in the two fist ows the classification of two cicula movements and the last two ows show the classification of bye-bye movements. The esults ae simila to the top and side-gasp classification. In 0 tials we obtained false negative fo both movements. Tab.4. Classification of Cicula movements ( fist ows) and bye-bye ( last ows) movements. Tial Classification using Cuvatues - Classification using Hand Oientation 3 Mixtue model using entopy to combine both featues % % % % % % % 76.5 % 8.90 % % 8.9 % 86.9 % VI. CONCLUSION We poposed a pobabilistic appoach fo segmentation and classification of each-to-gasp movements. Two diffeent segmented featues in 3D space wee used, by cuvatues (change in diections) and hand oientation. A dataset of each-to-gasp movements wee ceated to be used in a leaning phase based on histogam techniues. Applying these two methods of segmentation we wee able to classify the tajectoies using Bayesian techniues. Entopy was adopted as uncetainty measue to obtain a confidence level assigning weights fo both classification models (one using only cuvatue featues and anothe one using hand oientation featues) fo thei fusion though a mixtue model (weighted sum of posteio). The esults show that using the weights obtained fom entopy fo a joint classification impoved some classification esults obtained fom single classifies when thei confidence pobability is too low, avoiding, thus, false negatives. The poposed appoach can also be used fo gestue ecognition (e.g. fo human-obot inteaction), eaching simila esults of each-to-gasp movements classification. REFERENCES [] J. R. Flanagan, M. C. Bowman, and R. S. Johansson, Contol stategies in object manipulation tasks. Cuent Opinion in Neuobiology, 6(6), 006, pp [] J. Faaway, M. Reed and J. Wang, "Modelling 3D tajectoies using Bézie cuves with application to hand motion". Jounal of the Royal Statistical Society: Applied Statistics 56, 007, pp [3] J. Rett and J. Dias, Human-obot inteface with anticipatoy chaacteistics based on Laban Movement Analysis and Bayesian models. Poceedings of the IEEE 0th Intenational Confeence on Rehabilitation Robotics (ICORR), 007, pp [4] E.-J Holden and R. Owens, Repesenting the finge-only topology fo hand shape ecognition, In Machine Gaphics & Vision Intenational Jounal, Volume, Issue, 003, pp [5] D.R. Faia and J. Dias, "Hand Tajectoy Segmentation and Classification Using Bayesian Techniues", Wokshop on Gasp and Task Leaning by Imitation 008 IEEE/RSJ Intenational Confeence on Intelligent Robots and Systems, Nice, Fance - Sept, -6, 008, pp [6] Polhemus Libety Electomagnetic Motion Tacking System. Available: [7] D. C. Knill and A. Pouget, The Bayesian bain: the ole of uncetainty in neual coding and computation, TRENDS in Neuosciences, vol. 7, 004, pp [8] T. M. Cove and J. A. Thomas, Elements of Infomation.Theoy, Wiley & Sons, 99. [9] T. Abel and F. P. Feie, "Entopy-based Gaze Planning", Image and Vision Computing, Elsevie Science, vol. 9, issue, Sept. 00, pp [0] O. Ludwig J., U. Nunes, L. Schnitman and H. Lepikson, Applications of infomation theoy, genetic algoithms, and neual models to pedict oil flow, Comm. in Nonlinea Science & Numeical Simulation, v. 4, p. 5-67, 009.