BEHAVIOR recognition for human subjects moving in an

Transcription

1 294 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 Behavior Recogniion Using Muliple Deph Cameras Based on a Time-Varian Skeleon Vecor Projecion Chien-Hao Kuo, Pao-Chi Chang, and Shih-Wei Sun, Member, IEEE Absrac User behavior recogniion in a smar office environmen is a challenging research ask. Wearable sensors can be used o recognize behaviors, bu such sensors could go unworn, making he recogniion ask unreliable. Cameras are also used o recognize behaviors, bu occlusions and unsable lighing condiions reduce such mehods recogniion accuracy. To address hese problems, we propose a ime-varian skeleon vecor projecion scheme using muliple infrared-based deph cameras for behavior recogniion. The conribuion of his paper is hreefold: 1) The proposed mehod can exrac reliable projeced skeleon vecor feaures by compensaing occluded daa using nonoccluded daa; 2) he proposed occlusion-based weighing elemen generaion can be employed o rain suppor-vecor-machine-based classifiers o recognize behaviors in a muliple-view environmen; and 3) he proposed mehod achieves superior behavior recogniion accuracy and involves less compuaional complexiy compared wih oher sae-of-he-ar mehods for pracical esing environmens. Index Terms Behavior recogniion, skeleon, join, muliple cameras, deph camera, Kinec. I. INTRODUCTION BEHAVIOR recogniion for human subjecs moving in an indoor environmen such as an office has become increasingly crucial in recen years. For example, Zenonos e al. [1] proposed he use of wearable sensors and smarphones o recognize human behaviors and hereby monior he healh of office workers, which could ulimaely reduce he cos o governmen of work-relaed sress, anxiey, and depression. Once he behavior or mood of a human subjec can be recognized in a smar office environmen, a proper response from a server can be suggesed o a user. For example, Sun e al. [2] proposed a smar living space ha allows users o ener using an RF-ID card, recognizes facial expressions using a saic camera, and gauges users mood using a hear-rae sensor on heir smar wach, as illusraed in Fig. 1. Once he expression and exernal behavior of a user can be recognized, corresponding audiovisual responses can be made accordingly. For example, when a user eners he Manuscrip received November 14, 2016; revised January 29, 2017; acceped February 21, Dae of curren version Augus 7, This work was suppored in par by he Minisry of Science and Technology, Taiwan, under Gran MOST E C.-H. Kuo and P.-C. Chang are wih he Deparmen of Communicaion Engineering, Naional Cenral Universiy, Jong-Li 320, Taiwan ( chkuo@vaplab.ce.ncu.edu.w; pcchang@ce.ncu.edu.w). S.-W. Sun is wih he Deparmen of New Media Ar, Taipei Naional Universiy of he Ars, Taipei 112, Taiwan ( swsun@newmedia.nua.edu.w). Digial Objec Idenifier /TETCI Fig. 1. Smar office scenario proposed by Sun e al. [2]: (a) facial expression recogniion, (b) hear rae measuremen using smar wach wih buil-in app, (c) audiovisual response from server, and (d) open gesure is recognized by he sysem and he drawer is opened by a servo moor conrolled by he smar office server. smar living space for he firs ime, he sysem can auomaically display a map and highligh he user s physical locaion using an Arduino conrol board wihin he Inerne of Things (IoT) environmen. During he developmen of a smar living space [2], cameras are one of he mos criical sensors for human behavior recogniion, and heir use may obviae he necessiy of users wearing or bringing sensor devices, hus leading o a naural and undisurbed user experience. Wearable sensors can precisely measure he inernal sae of a user, bu once he smar wach or mobile device conaining he sensors has been se aside by he user, hey are unable o correcly recognize behaviors. Behavior recogniion using camera devices also has drawbacks, wih recogniion becoming exremely challenging under changing lighing condiions or occlusions o he field of view. In his paper, we propose a ime-varian skeleon vecor projecion scheme using muliple deph cameras for behavior recogniion in a smar office environmen; he proposed scheme avoids common recogniion sysem drawbacks such as users forgeing o wear heir wearable sensors and objec occlusion wihin camera-based approaches. The conribuion of he proposed mehod is hreefold: 1) The proposed ime-varian skeleon vecor projecion involving muliple cameras and a classifier raining process can recognize human behaviors by using informaion from non-occluded X 2017 IEEE. Personal use is permied, bu republicaion/redisribuion requires IEEE permission. See hp:// sandards/publicaions/righs/index.hml for more informaion.

2 KUO e al.: BEHAVIOR RECOGNITION USING MULTIPLE DEPTH CAMERAS BASED ON A TIME-VARIANT SKELETON VECTOR PROJECTION 295 views o compensae for he occluded views; 2) he proposed occlusion-based weighing elemen generaion process improves behavior recogniion accuracy by analyzing he number of joins whose posiion is precisely known because no occlusion occurred; and 3) compared wih sae-of-he-ar mehods, he proposed mehod achieves higher recogniion accuracy a a lower compuaional cos. The remainder of his paper is organized as follows. In Secion II, relaed sudies on behavior recogniion are reviewed. The proposed mehod is described in deail in Secion III, and he experimenal resuls are presened in Secion IV. Finally, Secion V provides he conclusion. II. RELATED WORK Exising mehods for recognizing behavior in a smar office environmen can be classified ino four caegories: wearable sensor, join-based camera, parly camera-based, and muliplecamera approaches. Wearable sensor echniques recognize behaviors by measuring he inerial signals of users. Zenonos e al. [1] used wearable sensors and smarphones o exrac hear rae, acceleraion, emperaure, and pulse rae and rain mood classificaion models o recognize he mood of users, gahering he informaion in a healhy office app ha can improve an office s working efficiency. Sprin e al. [3] proposed he collecion of sensor daa from indoor environmens for use in aciviy recogniion by deecing behavior change. To recognize human aciviies, Shen e al. [4] proposed he use of smarphone moion sensors and Lee e al. [5] used a user s smarwach and smar bel for behavior recogniion. To undersand a user s movemen, Sun e al. [6] buil a pressure sensing sysem using fiber-opic sensors mouned on he floor and a space encoding process, saisical modeling, and mixure learning. Human aciviies (walking, working, resing, and alking) can be recognized in an indoor office environmen wihin an 8 8 block area. However, one of he mouned sensors can only respond wih a binary resul for a user s occupancy wih limied spaial precision. Pham e al. [7] used an inerial measuremen uni for moion daa collecion and disribued passive infrared sensors for human aciviy recogniion, localizaion, and racking. Tao e al. [8] employed wireless sensor neworks from acceleraion sensor signals for human behavior recogniion. Tan and Yang [9] proposed he use of Wi-Fi signals o recognize user gesures. However, pressure sensors, fiber-opic sensors, passive infrared sensors, and Wi-Fi signals, as used in he aforemenioned sudies, only provide a rough esimaion of a user s locaion and behavior recogniion requires much higher spaial precision. Addiionally, if a user forges o wear heir sensing device, any behavior recogniion resuls will be incorrec. Join-based camera approaches uilize he geomerical relaionships among he analyical joins of he body deermined using moion capure (MoCap) or deph cameras. Lv and Nevaia [10] used he 3D posiions of muliple joins as feaures and applied hidden Markov model (HMM) [11] weak classifiers in AdaBoos [12] o boos he classifiers in a single-mocap-device environmen. To uilize emporal informaion, Sheikh e al. [13] proposed he use of 4D space (wih ime as he fourh dimension) o recognize acions using join angles in a MoCap environmen. Employing he angular relaionships among join vecors, Hussein e al. [14] proposed a 3D join covariance descripor wih he linear suppor vecor machine (SVM) classifier for recognizing acions using a Kinec deph camera. By adoping MoCap and a Kinec deph camera, Wang e al. [15] exraced 3D join feaures and used local occupancy paerns o generae spaial hisograms for behavior recogniion, using LIBSVM [16], a library for SVMs, o rain he classifiers. By modeling he observed spherical coordinaes of joins, Xia e al. [17] used linear discriminan analysis, vecor quanizaion, and a discree HMM o recognize behaviors. Yang and Tian [18] proposed he EigenJoins mehod based on a principal componen analysis for behavior recogniion. Furhermore, Zhu e al. [19] adoped muliple spaioemporal feaures [20] [23] and skeleon join feaures [18] for feaure quanizaion, and a random fores algorihm [24] was used for feaure fusion and acion classificaion in a Kinec camera environmen. Parly camera-based approaches use he relaionships beween skeleon daa and muliple join ses for behavior recogniion. Chaudhry e al. [25] uilized conneciviy relaionships among muliple join locaions o generae shape conex feaures [26] and opimal muliple kernel learning weighs [27] o generae SVM classifiers in an environmen wih a single Mo- Cap and Kinec deph camera. To observe how 3D join locaions change wih ime, Ofli e al. [28] proposed a mehod of measuring he Levenshein disance [29] beween joins, which can hen be used o rain classifiers. To analyze deph informaion around he joins, Ohn-Bar and Trivedi [30] proposed a hisogram of oriened gradien-based [31] mehod for raining classifiers. Using four joins for behavior recogniion, Evangelidis e al. [32] proposed a skeleal quads mehod using a Gaussian mixure model and Fisher score [33] o generae feaure vecors. For modeling emporal informaion for behavior recogniion, Vemulapalli e al. [34] adoped Lie algebra for mapping joins in he spaioemporal vecor space and generaing SVM classifiers. In all he aforemenioned join-based or parly camera-based approaches, however, parial and self-occlusion reduces behavior recogniion accuracy. Recenly, muliple-camera environmen approaches have begun o be used for behavior recogniion. Azis e al. [35] used fused skeleon daa from wo cameras, and a neares-neighbor dynamic ime warping mehod was adoped for behavior recogniion in a wo-kinec-camera environmen. Furhermore, he same research group proposed a weighed averaging fusion algorihm [36] for generaing a muliview skeleon including 3D join posiions, pairwise join disances, and hisogram of cubes [35], which can be used as feaures o generae behavior classifiers. The mehod proposed in his paper avoids issues of inconvenience regarding wearable sensors and also overcomes problems regarding occlusions ha occur in join-based and parly camera-based approaches. The mehod proposed herein achieves behavior recogniion using a muliple-camera environmen, compensaing for occluded views using he oher camera s non-occluded view. The raining daa obained from he muliple views are properly analyzed o achieve high behavior recogniion accuracy wih low compuaional complexiy.

3 296 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 Fig. 3. Process for exracing feaures from skeleon daa obained from muliple-view deph cameras and he feaure vecor modeling process. Fig. 2. Example joins and skeleons deeced using Kinec SDK [37] and he corresponding measuremens: (a) joins on human skeleon; (b) heigh measuremen (red arrow) and he posiion of he head relaive o he hip cener join (purple dashed arrow); (c) joins of p 1, (d) joins of p 2. III. TIME-VARIANT SKELETON PROJECTION Before behavior recogniion can be aemped, skeleon join daa are obained from he official Microsof Kinec SDK 1.8 [37]. Each join is represened by is real 3D posiion. For example, he op of he head j h =(x h,y h,z h ) (he op join displayed in Fig. 2), obained from he Kinec SDK, represens he physical posiion deeced in he deph camera s field of view of he head poin j h a ime. When a behavior is performed by a user, he skeleon daa are exraced from muliple views - for example, he cener, righ, and lef views - which are recorded from consecuive frames, as illusraed by Behavior 1onhe lef-hand side of Fig. 3 (wherein joins and skeleon are obained from he hree views menioned). The skeleon daa are analyzed based on he proposed relaive join posiions using a normalizaion process, basis vecor generaion, projecion from a join vecor ono he basis vecors, and behavior classifier raining, all of which are described in deail in he following secions and are represened by blocks in he cenral par of Fig. 3. In addiion, he behavior feaures (righ-hand side of Fig. 3) obained from muliple views are furher used for classifier raining and behavior recogniion. A. Relaive Join Posiion Wih a Normalizaion Process To deal wih siuaions involving human subjecs wih differen heighs, which are ineviable in real applicaions of he mehod, a body heigh normalizaion process is necessary. As shown in Fig. 2(b), a ime, given he posiions of he righ foo join j fr =(x fr,y fr,z fr ) and lef foo join j fl =(x fl,y fl,z fl ), a middle foo join can be defined by aking heir average: j fm fr (j +j = fl ) 2. Furhermore, given he posiion of he head poin j h =(x h,y h,z h ), he heigh of a subjec can be measured by calculaing he Euclidean disance from he head poin j h o he middle foo join j fm : ( ) h = d j h,j fm ( ) 2 ( ) 2 ( ) 2, = x h x fm + y h y fm + z h z fm (1) which is depiced by he red line segmen in Fig. 2(b). Taking he hip cener join j hc [he purple circle in Fig. 2(b)] as he origin poin of a relaive coordinae sysem, he relaive posiion of he head poin normalized by he body heigh h : ( ) ( ) j j h h = j hc x h = x hc,y h y hc,z h z hc, (2) h h can be calculaed and is illusraed by he purple dashed line in Fig. 2(b). The relaive posiions of he res of he joins can be similarly obained. Hereafer, a prime symbol on a join posiion is used o represen a normalized relaive posiion; represens he relaive posiion of he head wih respec o he hip cener j hc [he purple dashed vecor in Fig. 2(b)], normalized by h [he red line segmen in Fig. 2(b)]. Even if users have differen heighs, as illusraed in Fig. 2(c) for he joins belonging o person p 1 and Fig. 2(d) for person for example, j h poin j h p 2, he calculaed (j h ) p1 (j h ) p2. Therefore, he proposed relaive join posiions obained hrough normalizaion can be reaed as a user-invarian (regardless of users heigh) feaure. B. Basis Vecors Generaion Given he relaive posiions of he righ shoulder join j sr and lef shoulder join j sl [Fig. 2(a)], a shoulder vecor S = j sr j sl can be obained, as illusraed by he green arrow in Fig. 4(a). Given he relaive posiion of he foo middle join j fm, a foo vecor F = j sr j fm can also be obained

4 KUO e al.: BEHAVIOR RECOGNITION USING MULTIPLE DEPTH CAMERAS BASED ON A TIME-VARIANT SKELETON VECTOR PROJECTION 297 Fig. 5. Join vecor projecions: (a) he indexes used for joins and he basis vecors; (b) he geomeric relaionships among he hand join vecors and he basis vecors. respecively. The derived expression f N = N H cos α =< N, H >, (4) Fig. 4. Basis vecor generaion: (a) shoulder, foo, and normal vecors; (b) he obained represenaive basis vecors; (c) he basis vecors a ime 1 ; (d) basis vecors a ime 2 ; (e) basis vecors in a real es a ime 3 ; and (f) basis vecors in a real es a ime 4. [red arrow in Fig. 4(a)]. A normal vecor N : N = S F, (3) can subsequenly be calculaed from he ouer produc of S and F. Noably, N is orhogonal o S and F. Therefore, he vecors N, S, and F are reaed as he basis vecors [Fig. 4(b)]. A ime = 1, a se of basis vecors { N 1, S 1, F 1 } is obained [Fig. 4(c)], whereas a ime = 2, anoher se of basis vecors { N 2, S 2, F 2 } is obained [Fig. 4(d)]. Thus, he obained basis vecors are ime dependen. For example, he basis vecors are differen a imes 3 and 4 because of he differen posures assumed by he user. Hence, he proposed se of ime-varian basis vecors N, S, and F can be applied for furher feaure descripion. C. Projecion of Join Vecor Ono he Basis Vecors Given a righ hand join j hr and righ shoulder join j sr, a hand join vecor H = j sr j hr can be obained [blue arrow in Fig. 5(a)]. The geomeric relaionship beween he hand join vecor H and basis vecors N, F, and S is depiced in Fig. 5(b). The projecions of H ono he basis vecors N, F, and S are defined as H cos α, H cos β, and H cos γ, respecively. The norms N, F, and S of he basis vecors can be subsiued ino hese expressions o give N H cos α, F H cos β, and S H cos γ, f F = F H cos β =< F, H >, (5) f S = S H cos γ =< S, H >. (6) Therefore, for a given join, he feaure vecor a ime is defined as f =[f N,f F,f S ]. (7) The index i is used o represen he differen joins of a human subjec, and f i is used o represen he feaure vecors belonging o he ih join. Thus, he complee se of feaures is represened as φ = f i=1,n f i=2,n f i=1,f f i=2,f f i=1,s f i=2,s f i=20,n f i=20,f f i=20,s, (8) where he indexes i =1,...20 are depiced in Fig. 5(a). Over a specific period, he 3D posiion of each join can be racked. Fig. 6 illusraes he spaioemporal domain informaion for differen joins, wherein he z-axis (deph) is ignored for display convenience. The righ hand join (green circles, i = 12) is seen o move in large moions, bu he righ elbow (pink circles) and lef ankle (red circles) move lile. The 3D posiions of he joins are defined by he proposed basis vecor projecions. An example of he complee se of feaures [(8)] is depiced in Fig. 7. The orange circles depiced a =1are he projecion vecors of all he join vecors (i =1,...i=20) ono he basis vecor N =1, which are defined as f i=1,n =1...f i=20,n =1, [i.e., he firs column vecor of (8)]. Similarly, he red and green circles represen he projecion vecors ono he basis vecors F =1 and S =1, and hey are he second and hird column vecors in (8), respecively.

5 298 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 Fig. 8. Training of behavior classifiers using he feaures exraced from muliple views. Fig. 6. Posiions of he seleced joins as deeced by one of he muliple cameras in he spaioemporal domain. Fig. 7. Illusraive example of (8) for =1,...T. Given an obained feaure φ,c a camera c in a muliple-view environmen, a feaure se φ T,c =[φ =1,c,φ =2,c,,φ =T,c ] can be obained over a period T. Moreover, he feaures obained from muliple views are concaenaed, a cross-view feaure can be obained: Φ T =[φ T,c=1,φ T,c=2, φ T,c=C ], (9) where C is he number of cameras. The feaure vecors found using (9) are depiced on he righ-hand side of Fig. 3. D. Behavior Classifier Training Once he feaure vecors from muliple views are obained (examples of which are illusraed in Fig. 7), LIBSVM [16] is adoped for generaing he classifiers for differen behaviors in a muliple-view environmen. For example (Fig. 8), he spaioemporal feaures of Behavior 1 can be exraced from he muliple views of he deph cameras (in his example, he cener, righ, and lef views), and he salien spaioemporal feaures can hen be seleced from he pool of rajecories (se of curves in Fig. 9). The variance of each rajecory along he ime axis is calculaed and rajecories wih variances larger han a hreshold v T (such as he bolder curves in Fig. 9) are kep as he salien spaioemporal rajecories. Noably, he rajecories presened in Fig. 9 operae in he feaure domain [i.e., (8)], no in he physical spaioemporal domain. Nex, he adapive weighing facors are calculaed according o he occlusion siuaion analyzed from muliple views. Fig. 10 Fig. 9. Proposed rajecory filering process, wih plos of 20 joins projeced on one of he basis vecors in he spaioemporal feaure domain; he hree filered feaure rajecories are presened in bold. Fig. 10. Capured deph camera daa: (a) color frames; (b) deph frames wih subjecs deeced; and (c) he resulan skeleons and basis vecors. illusraes an example gesure (raised righ hand) ha can be observed from views c1 and c2 bu no c3, in which i is occluded by he user s body. In he behavior classifier raining, herefore, he views in which occlusion does and does no occur have differen weighs. In his paper, we propose an occlusionbased weighing elemen generaion process ha assesses he

6 KUO e al.: BEHAVIOR RECOGNITION USING MULTIPLE DEPTH CAMERAS BASED ON A TIME-VARIANT SKELETON VECTOR PROJECTION 299 Fig. 11. Example of he proposed occlusion-based weighing elemen generaion. occlusion siuaion and generaes suiable SVM-based behavior classifier (middle of Fig. 8). 1) Occlusion-Based Weighing Elemen Generaion: During he deecion of skeleons and joins wih muliple deph cameras in pracice, joins may be occluded by oher pars of he user s body when a behavior is being performed. In Kinec SDK, he deeced joins are classified ino hree caegories: racked, indicaing a racked join ha he camera can see; inferred, indicaing a racked join ha he camera canno see bu he posiion of which can be esimaed by surrounding join daa; and no racked, indicaing he lack of join daa. The inferred and no racked joins are possibly occluded (by he user or oher objecs) in he field of view. In Kinec SDK1.8 [37], 20 joins are deeced in each frame, and more inferred and no racked joins indicae ha here are less racked joins. Thus, he reliabiliy of a represenaion is proporional o he number of racked joins τ,c a camera c. Views wih less occlusion (higher τ,c ) should have a greaer effec on he feaure generaion process and he following behavior recogniion process. Therefore, hrough he consideraion of he occlusion effecs for differen views a differen ime poins, a ime-varian weighing funcion can be independenly defined in erms of he feaure vecor: Φ w T =[w c=1 φ T,c=1,w c=2 φ T,c=2, w c=c φ T,c=C ]. (10) Equaion (10) reveals he effec of occlusions a differen ime poins. By considering he compuaional complexiy and accuracy in he feaure generaion and recogniion processes, we propose he designaion of he weighing elemens as follows: w c = τ c C c=1 τ c, (11) where τ,c=1 is he number of racked joins in each view. A hree-view example of he proposed weighing elemens is illusraed in Fig. 11. A differen ime insances, he weighs of he muliple cameras (i.e., c1, c2, and c3,) are deermined by he number of racked joins in he hree views. E. Behavior Recogniion LIBSVM [16] is adoped for classifier raining and esing for behavior recogniion (Fig. 8, boom; Fig. 12, righ-hand side). The proposed feaure exracion process using muliple views Fig. 12. Process of behavior recogniion from unknown skeleon daa exraced from muliple deph cameras. (Fig. 12, upper-lef) and he feaure rajecory filering process (e.g., Fig. 9) are operaed for unknown skeleon daa. The calculaed weigh marix (e.g., Fig. 11) is sen as side informaion ogeher wih he SVM classifiers ( Fig. 12, righ-hand side) so ha he behaviors of a user can be recognized. IV. EXPERIMENTAL RESULTS In he experimen, hree Kinec v1 deph cameras were used o capure a user s behavior wih he official Kinec SDK 1.8 [37], and his served as he raw skeleon daa. The Kinec cameras (wo marked by red recangles in Fig. 10) were mouned a a heigh of approximaely 1.6 m from he floor. One of he deph cameras was posiioned o capure he fron view, and he ohers were mouned o capure he wo side views. Color frames, deph frames wherein he subjec has been deeced, and skeleons (blue line segmens) wih joins (blue circles) are presened in Fig. 10(a) (c), respecively. Through he proposed mehod, he hree basis vecors N, F and S were calculaed using (3) and are depiced by he orange, red, and green arrows in Fig. 10(c). Furhermore, he calculaed vecor projecions [hrough (4), (5), and (6)] for he hree views over a specific period are ploed in Fig. 13(a) and (b) for Person1 and Person2, respecively. These wo ses of vecor projecions have similar disribuions in he environmen wih muliple deph cameras. Therefore, he obained feaure vecors were used o rain he classifiers for behavior recogniion. The rajecory hreshold, defined in Secion III-D, was empirically deermined as v T =0.02, and his value was used in all subsequen ess. Ten behaviors were recorded in he environmen for evaluaion, wih a oal of 15,169 frames recorded in s, and 10 voluneer users were asked o perform he behaviors (examples in Fig. 14). The 10 users performed each behavior hree imes; herefore, = 300 video clips were generaed, wih a manual ime synchronizaion process used. Half of he uniformly sampled video clips were used as he raining daase for behavior recogniion, and he oher half were used as he esing daase. The proposed mehod was compared wih oher sae-of-he-ar mehods using he muliple-view es video sequences provided by Azis e al. [36]. The performance level of he proposed mehod was compared wih hose of he mehods proposed by Azis e al. [36] (muliple-view approach) and Vemulapalli e al. [34] (parly camera-based single-view approach).

7 300 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 TABLE I COMPARISON RESULTS OF MEAN AVERAGE PRECISION (MAP) proposed daase [36] daase Vemulapalli s mehod [34], cener view 100 ± 0.00% ± 10.94% Vemulapalli s mehod [34], righ view ± 14.74% Vemulapalli s mehod [34], lef view ± 14.56% Vemulapalli s mehod [34], side view ± 17.63% Azis s mehod [36] ± 9.34% ± 13.12% Proposed Mehod ± 2.11% ± 19.65% Fig. 13. Feaure vecors among hree views for he same behavior as performed by (a) person 1 and (b) person 2. A. Quaniaive Evaluaion The mean average precision (map) resuls associaed wih he proposed mehod (muliple-view) and he mehods from [36] (muliple-view) and [34] (parly camera-based single-view) are compared in Table I. The single-view mehod [34] was applied o he frames obained from each view. Because he cener view is less frequenly occluded, he single-view [34] resuls derived for he cener view were more accurae han he side-view resuls. As revealed by he second and hird rows of Table I, occlusions ha occurred in he side views resuled in decreased accuracies of 93.33% (righ view) and 87.25% (lef view) for he mehod of [34], whereas he proposed mehod achieved he bes map of 99.33% when all views were considered. Neverheless, a cener view canno always be obained in pracical applicaions. Of he wo muliple-view approaches, he proposed mehod had superior overall performance o he mehod of Axis e al. [36] for he proposed daase. For he more challenging daase provided by [36] (Table I), he proposed mehod also had he bes behavior recogniion capabiliy when compared wih he wo oher mehods. The deailed behavior recogniion resuls for differen behaviors are presened hrough confusion marices in Fig. 15. The label of each row in he confusion marices is he acual behavior label (i.e., he behavior ha was performed), and he labels a he foo of each column are he prediced behavior labels. The corresponding elemen of he confusion marix is he number of imes he prediced label was idenified by he mehod divided by he oal number of imes he acual behavior was performed in he daase (such ha he sum of he elemens in each row mus equal 1). The las row of Fig. 15(f) corresponds o he side kick, which was incorrecly recognized as oher behaviors because only he fron view and one side view could be obained in he daase of [36]. Similar performance degradaion was observed when he mehod of Axis e al. [36] (muliple-view approach) was used, as shown by he las row of Fig. 15(d). Therefore, if informaion from less-occluded views could be obained, he behavior recogniion capabiliy of he proposed mehod would be improved because he occluded daa could be compensaed. Fig. 14. Par of he behavior daabase recorded wih voluneer users. Behaviors included were he one hand wave (OHW), wo hand wave (THW), forward punch (FP), side punch (SP), forward kick (FK), side kick (SK), bend (BD), hand clap (HC), high hrow (HT), and jog (JG). (a) Ten differen behaviors behaved by hree voluneers. (b) Color and deph frames from he hree views. B. Qualiaive Evaluaion Fig. 16(a) and (c) presen deph frames, wherein he subjec has been deeced (he firs hree rows), and skeleons (blue line segmens in he subsequen hree rows) obained using Kinec SDK; from hese, he basis vecors N (orange arrows), F

8 KUO e al.: BEHAVIOR RECOGNITION USING MULTIPLE DEPTH CAMERAS BASED ON A TIME-VARIANT SKELETON VECTOR PROJECTION 301 Fig. 15. Confusion marix for he behavior recogniion accuracy of Vemulapalli s mehod [34] (single-view), Azis s mehod [36] (muliple-view), and he proposed mehod (muliple-view). The behavior labels are as lised in he capion of Fig. 14, and he abbreviaions used in he [36] daase are: hand clapping (HC), waving boh hands (WBH), single hand waving (SHW), punching (PC), bend forward (BF), forward kick (FK), answering a phone call (AP), siing down (SD), and side kick (SK). (a) Vemulapalli s mehod [34] from he cener view; (b) Vemulapalli s mehod [34] wih he fron view of he [36] daase; (c) Azis s mehod [36] using hree views; (d) Azis s mehod [36] using wo views of he [36] daase; (e) proposed mehod using hree views; (f) proposed mehod using wo views of he [36] daase. (red arrows), and S (green arrows) could be obained using he proposed mehod. For example, he movemen of a foo could be idenified from deph frames in view c1. However, he foo is occluded by he subjec in view c2. The foo joins marked by he green dashed circles in view c1 in Fig. 16(b) could be used o reliably deermine F (red arrow) in view c1 according o he definiion given in Secion III-B. Neverheless, because of he occlusion of he foo in views c2 and c3, he obained F (red arrows) could be erroneous (wih he magena circles being he inferred joins). In he proposed mehod, he weighs of he joins reduce he effec of he inferred joins. Thus, view c1 could have had higher weighing and relaive angles, as illusraed by he skeleon idenified in view c1 in Fig. 16(a). Similarly, he raised hand in frame 2341 in Fig. 16(d) is shown o be more sably racked in view c1 bu severely occluded in Fig. 16. Time-varian skeleon vecor projecion for a foo/hand movemen behavior. (a) Deph frames (foo movemen) from views c1, c2, and c3, and he corresponding skeleons; (b) skeleons obained from frame 1517: blue and magena circles are he racked and inferred joins, respecively. (c) Deph frames (hand movemen) from views c1, c2, and c3, and he corresponding skeleons.

9 302 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 TABLE II TIME COMPLEXITY WHEN VEMULAPALLI S METHOD [34] (SINGLE-VIEW), AZIS S METHOD [36] (MULTIPLE-VIEW), AND THE PROPOSED METHOD (MULTIPLE-VIEW)WERE USED ime (s) proposed daase [36] daase Modeling Classificaion Modeling Classificaion [34] [36] Proposed TABLE III RESULTS OF THE PROPOSED METHOD WITH DIFFERENT NUMBERS OF VIEWS USED FOR THE PROPOSED DATASET;MEAN AVERAGE PRECISION (MAP), MODEL TIME (MT), AND CLASSIFICATION TIME (CT) used views map MT (s) CT (s) cener + righ + lef (3 views) ± 2.11% cener + righ (2 views) ± 2.11% cener + lef (2 views) ± 7.17% lef + righ (2 views) ± 10.06% cener (1 view) ± 2.81% righ (1 view) ± 12.09% lef (1 view) ± 16.35% views c2 and c3. The basis vecor S is correc in view c1 bu erroneously deeced in view c2 because of occlusion. In view c2, he erroneous basis vecor N (orange arrow) is due o he erroneous S (green arrow). Similar resuls can be observed in view c2 for frames However, he proposed mehod could reduce he effec of he erroneous basis vecors because view c1 had a larger weighing facor. C. Complexiy Comparison The proposed mehod (muliple-view), he mehod of Axis e al. [36] (muliple-view), and he mehod of Vemulapalli e al. [34] (single-view) were execued on he same compuer, which had an Inel Core i7, a 2.67-GHz CPU, and 12Gb of RAM. The oal compuaional ime required for each mehod wih each daase is presened in Table II. The previously proposed mehods in [36] and [34] involve dynamic ime warping operaions bu he proposed mehod does no and hus requires he leas compuaional ime for is modeling and classificaion operaions. The proposed mehod only needs o projec he obained skeleon vecor ono he proposed basis vecors, afer which he weighing facors are deermined by any occlusion deeced by he Kinec SDK, which occurs in real ime. Therefore, he proposed mehod has he poenial o be applied in real-ime applicaions. In he proposed mehod, he number of deph cameras used for he behavior recogniion can affec he overall recogniion accuracy. As revealed in he firs row of Table III, he map was 99.33% when all hree views were used in he behavior recogniion process. When only wo views were used, he map was reduced o 90.00% 99.33%, and he modeling and classificaion imes were also reduced. Furhermore, when only one view was Fig. 17. Confusion marix for he behavior recogniion accuracy of he proposed mehod when wo views and a single view were used. (a) cener and righ views used; (b) cener and lef views used; (c) lef and righ views used; (d) cener view used; (e) righ view used; (f) lef view used. used, he map was furher reduced o 78.00% 98.67%, wih furher savings in modeling and classificaion imes. However, he map resuls were highly correlaed wih he view seleced. In our ess, he lef view included severe occlusions; herefore, when he lef view was seleced for behavior recogniion using a single view or wo views, he map was reduced. Neverheless, occlusions can be considered by deermining he weighing facors proposed in (10) in he proposed mehod. The deailed map resuls in Table III are also illusraed by he confusion marices in Fig. 17. V. CONCLUSION In conclusion, he proposed behavior recogniion scheme in a muliple-deph-camera environmen recognizes behaviors wih higher accuracy and lower compuaional complexiy han he sae-of-he-ar mehods of [34] and [36]. The conribuion of his paper is hus hreefold: 1) The proposed ime-varian skeleon projecion using muliple views can compensae for occluded views and idenify feaures reliably; 2) he proposed SVM-based classifier accuraely recognizes behaviors; and 3) he execuion of he proposed mehod has low compuaional complexiy compared wih he oher mehods. As revealed in Table III, he addiion of more cameras increases he accuracy of he mehod, bu using fewer cameras saves compuaional power, which could make he mehod suiable for real-ime applicaions. Therefore, he proposed mehod can be adoped in he fuure for smar office applicaions, recognizing user behaviors and hen riggering corresponding IoT-based applicaions.

10 KUO e al.: BEHAVIOR RECOGNITION USING MULTIPLE DEPTH CAMERAS BASED ON A TIME-VARIANT SKELETON VECTOR PROJECTION 303 As illusraed by he example in Fig. 1(d), a smar office may have differen IoT-conneced devices such as lighs or a servo moor conrolled by an Arduino conroller. Once a user s forward punch gesure (which does no involve physical ouch) is recognized in a muliple-deph-camera environmen according o he mehod proposed in his paper, he server can auomaically urn on he ligh using a swich and rigger he servo moor o open a drawer, all conrolled by an Arduino conroller. In he fuure, he proposed behavior recogniion scheme using muliple deph cameras can be exended o muliple skeleon-based devices such as he Kinec v2 and Leap Moion and moion capure devices. REFERENCES [1] A. Zenonos, A. Khan, G. Kalogridis, S. Vasikas, T. Lewis, and M. Sooriyabandara, Healhyoffice: Mood recogniion a work using smarphones and wearable sensors, in Proc. IEEE In. Conf. Pervasive Compu. Commun. Workshops, Mar. 2016, pp [2] Moving o 2026, he Innovaion, Creaion, and Enrepreneurial Cones, Minisry of Educaion, Taiwan, (he Bes Innovaion Award, he Bes Technology Inegraion Award, Advisor: Prof. S.W. Sun) [Online]. Available: hps://youu.be/yijjj8yxva [3] G. Sprin, D. Cook, R. Friz, and M. Schmier-Edgecombe, Deecing healh and behavior change by analyzing smar home sensor daa, in Proc. IEEE In. Conf. Smar Compu., May 2016, pp [4] C. Shen, Y. Chen, and G. Yang, On moion-sensor behavior analysis for human-aciviy recogniion via smarphones, in Proc. IEEE In. Conf. Ideniy, Securiy Behavior Anal., Feb. 2016, pp [5] J. G. Lee, M. S. Kim, T. M. Hwang, and S. J. Kang, A mobile robo which can follow and lead human by deecing user locaion and behavior wih wearable devices, in Proc. IEEE In. Conf. Consum. Elecron., Jan. 2016, pp [6] Q. Sun, F. Hu, and Q. Hao, Human movemen modeling and aciviy percepion based on fiber-opic sensing sysem, IEEE Trans. Human- Mach. Sys., vol. 44, no. 6, pp , Dec [7] M. Pham, D. Yang, W. Sheng, and M. Liu, Human localizaion and racking using disribued moion sensors and an inerial measuremen uni, in Proc. IEEE In. Conf. Robo. Biomimeics, Dec. 2015, pp [8] D. Tao, L. Jin, Y. Wang, and X. Li, Rank preserving discriminan analysis for human behavior recogniion on wireless sensor neworks, IEEE Trans. Ind. Informa., vol. 10, no. 1, pp , Feb [9] S. Tan and J. Yang, Fine-grained gesure recogniion using wifi, in Proc. IEEE Conf. Compu. Commun. Workshops, Apr. 2016, pp [10] F. Lv and R. Nevaia, Recogniion and segmenaion of 3-D human acion using HMM and muli-class adaboos, in Proc. Eur. Conf. Compu. Vis., 2006, pp [11] L. R. Rabiner, A uorial on hidden Markov models and seleced applicaions in speech recogniion, Proc. IEEE, vol. 77, no. 2, pp , Feb [12] Y. Freund and R. Schapire, A decision heoreic generalizaion of on-line learning and applicaion o boosing, J. Compu. Sys. Sci., vol.55, no.1, pp , [13] Y. Sheikh, M. Sheikh, and M. Shah, Exploring he space of a human acion, in Proc. IEEE In. Conf. Compu. Vis., Oc. 2005, vol. 1, pp [14] M. Hussein, M. Torki, M. Gowayyed, and M. El-Saban, Human acion recogniion using a emporal hierarchy of covariance descripors on 3d join locaions, in Proc. In. Join Conf. Arif. Inell., 2013, pp [15] J. Wang, Z. Liu, Y. Wu, and J. Yuan, Mining acionle ensemble for acion recogniion wih deph cameras, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., Jun. 2012, pp [16] C. Chang and C. Lin, LIBSVM: A library for suppor vecor machines, ACM Trans. Inell. Sys. Technol., vol. 2, pp. 27:1 27:27, [Online]. Available: hp:// cjlin/libsvm [17] L. Xia, C. C. Chen, and J. K. Aggarwal, View invarian human acion recogniion using hisograms of 3D joins, in Proc. IEEE Compu. Soc. Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2012, pp [18] X. Yang and Y. L. Tian, Eigenjoins-based acion recogniion using navebayes-neares-neighbor, in Proc. IEEE Compu. Soc. Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2012, pp [19] Y. Zhu, W. Chen, and G. Guo, Fusing spaioemporal feaures and joins for 3D acion recogniion, in Proc. IEEE Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2013, pp [20] I. Lapev and T. Lindeberg, Velociy adapaion of space-ime ineres poins, in Proc. In. Conf. Paern Recogni., Aug. 2004, vol. 1, pp [21] G. Willems, T. Tuyelaars, and L. V. Gool, An efficien dense and scaleinvarian spaio-emporal ineres poin deecor, in Proc. Eur. Conf. Compu. Vis., Par II, 2008, pp [22] I. Lapev, M. Marszalek, C. Schmid, and B. Rozenfeld, Learning realisic human acions from movies, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., Jun. 2008, pp [23] A. Klaser, M. Marszalek, and C. Schmid, A spaio-emporal descripor based on 3d-gradiens, in Proc. Bri. Mach. Vis. Conf., 2008, pp. 275:1 10. [24] L. Breiman, Random foress, Mach. Learn., vol. 45, no. 1, pp. 5 32, [25] R. Chaudhry, F. Ofli, G. Kurillo, R. Bajcsy, and R. Vidal, Bio-inspired dynamic 3d discriminaive skeleal feaures for human acion recogniion, in Proc. IEEE Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2013, pp [26] S. Belongie, J. Malik, and J. Puzicha, Shape maching and objec recogniion using shape conexs, IEEE Trans. Paern Anal. Mach. Inell., vol. 24, no. 4, pp , Apr [27] A. Rakoomamonjy, F. Bach, S. Canu, and Y. Grandvale, Simplemkl, J. Mach. Learn. Res., vol. 9, pp , [28] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, Sequence of he mos informaive joins (SMIJ): A new represenaion for human skeleal acion recogniion, in Proc. IEEE Compu. Soc. Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2012, pp [29] V. I. Levenshein, Binary codes capable of correcing deleions, inserions and reversals, Sovie Phys. Doklady, vol. 10, 1966, Ar. no [30] E. Ohn-Bar and M. M. Trivedi, Join angles similariies and hog2 for acion recogniion, in Proc. IEEE Conf. Compu. Vis. Paern Recogni. Workshops, Jun. 2013, pp [31] N. Dalal and B. Triggs, Hisograms of oriened gradiens for human deecion, in Proc. IEEE Compu. Soc. Conf. Compu. Vis. Paern Recogni., Jun. 2005, vol. 1, pp [32] G. Evangelidis, G. Singh, and R. Horaud, Skeleal quads: Human acion recogniion using join quadruples, in Proc. In. Conf. Paern Recogni., Aug. 2014, pp [33] T. Jaakola and D. Haussler, Exploiing generaive models in discriminaive classifiers, in Proc. Conf. Adv. Neural Inf. Process. Sys. II, 1999, pp [34] R. Vemulapalli, F. Arrae, and R. Chellappa, Human acion recogniion by represening 3d skeleons as poins in a lie group, in Proc. IEEE Conf. Compu. Vis. Paern Recogni., Jun. 2014, pp [35] N. A. Azis, H. J. Choi, and Y. Iraqi, Subsiuive skeleon fusion for human acion recogniion, in Proc. In. Conf. Big Daa Smar Compu., Feb. 2015, pp [36] N. A. Azis, Y. S. Jeong, H. J. Choi, and Y. Iraqi, Weighed averaging fusion for muli-view skeleal daa and is applicaion in acion recogniion, IET Compu. Vis., vol. 10, no. 2, pp , [37] Kinec for windows sdk 1.8, [Online]. Available: hps://www. microsof.com/en-us/download/deails.aspx?id=40278 Chien-Hao Kuo received he B.S. degree in communicaion engineering from Naional Cenral Universiy (NCU), Taiwan, in He is currenly working oward he Ph.D. degree in he Video-Audio Processing Laboraory, Deparmen of Communicaion Engineering, NCU, Taiwan. His research ineress include video/image processing and video compression.

11 304 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. 1, NO. 4, AUGUST 2017 Pao-Chi Chang received he B.S. and M.S. degrees from Naional Chiao Tung Universiy, Taiwan, and he Ph.D. degree from Sanford Universiy, Sanford, CA, USA, 1986, all in elecrical engineering. From 1986 o 1993, he was in research saff a IBM T.J. Wason Research Cener, New York, NY, USA. In 1993, he joined he faculy of NCU, Taiwan, where he is presenly a Professor in he Deparmen of Communicaion Engineering. His main research ineress include speech/audio coding, video/image compression, and mulimedia rerieval. Shih-Wei Sun (M 12) received he B.S. degree from Yuan-Ze Universiy and he Ph.D. degree from Naional Cenral Universiy, Taiwan, in 2001 and 2007, respecively, boh in elecrical engineering. From 2007 o 2011, he was a Pos Docoral Research Fellow in he insiue of informaion science, Academia Sinica. In 2012, he joined he Deparmen of New Media Ar, Taipei Naional Universiy of he Ars, Taiwan, as an Assisan Professor. Since 2016, he is an Associae Professor. He is he founding leader wih he Ulra-Communicaion Vision Laboraory (ucvision Lab). His research ineres includes visual conen analysis, compuer vision and he applicaion for ineracive echnologies, and sensor applicaions for mobile devices. He received he Research Award from he Prize Award of Mulimedia Grand Challenge from he 2014 ACM Mulimedia Conference. He published more han 40 inernaional journal papers and conference papers. He serves as he Reviewers and echnical program commiee Members for many journals and conferences.