A 3D Human Skeletonization Algorithm for a Single Monocular Camera Based on Spatial Temporal Discrete Shadow Integration

Transcription

1 appled scences Artcle A 3D Human Skeletonzaton Algorthm for a Sngle Monocular Camera Based on Spatal Temporal Dscrete Shadow Integraton Je Hou *, Baolong Guo, Wangpeng He and Jnfu Wu School of Aerospace Scence and Technology, Xdan Unversty, No. 2 Taba Rd, X an , Chna; blguo@xdan.edu.cn (B.G.); hewp@xdan.edu.cn (W.H.); wjf505@gmal.com (J.W.) * Correspondence: je.hou.xdu@gmal.com; Tel.: Academc Edtor: Takayosh Kobayash Receved: 8 May 2017; Accepted: 29 June 2017; Publshed: 3 July 2017 Abstract: Three-dmensonal (3D) human skeleton extracton s a powerful tool for actvty acqurement and analyses, spawnng a varety of applcatons on somatosensory control, vrtual realty and many prosperng felds. However, the 3D human skeletonzaton reles heavly on RGB-Depth (RGB-D) cameras, expensve wearable sensors and specfc lghtenng condtons, resultng n great lmtaton of ts outdoor applcatons. Ths paper presents a novel 3D human skeleton extracton method desgned for the monocular camera large scale outdoor scenaros. The proposed algorthm aggregates spatal temporal dscrete jont postons extracted from human shadow on the ground. Frstly, the projected slhouette nformaton s recovered from human shadow on the ground for each frame, followed by the extracton of two-dmensonal (2D) jont projected postons. Then extracted 2D jont postons are categorzed nto dfferent sets accordng to actvty slhouette categores. Fnally, spatal temporal ntegraton of same-category 2D jont postons s carred out to generate 3D human skeletons. The proposed method proves accurate and effcent n outdoor human skeletonzaton applcaton based on several comparsons wth the tradtonal RGB-D method. Fnally, the applcaton of the proposed method to RGB-D skeletonzaton enhancement s dscussed. Keywords: 3D model; shadow nformaton; skeletonzaton; spatal temporal ntegraton 1. Introducton The development of three-dmensonal (3D) human skeleton extracton contrbutes enormously to prosperng felds lke vrtual realty and somatosensory human computer nteracton. However, current 3D human skeletonzaton algorthms requre specfed acquston equpments ncludng RGB-Depth (RGB-D) cameras and wearable sensors, or a specfc expermental setup lke rng llumnator array. RGB-D cameras lke Mcrosoft Knect are desgned to perform human skeletonzaton n a short range [1,2]. Wearable sensors only perform effectve skeletonzaton on human subjects wearng expermental tags. Rng llumnator array requres precse subject poston and llumnator array setup durng the 3D modellng and skeletonzaton procedures. These setup restrctons of tradtonal human skeletonzaton methods brng great lmtaton on the outdoor applcatons. Instead of deployng algorthms on tradtonal specfed platforms, ths work pays attenton to the commonest projecton of the human body on the ground. Shadow s the projecton of a opaque object on a certan surface, contanng sngle-vew slhouette nformaton of the object. Multple methods have been developed to extract nformaton from shadow. Current methods manly focus on the recovery of mesh model [3 5] or pont clouds [6,7] of statc objects based on partal shadow nformaton [8]. In ths paper, a slhouetted shadow-based skeleton extracton (SSSE) method s proposed. The proposed SSSE method deploys shadow nformaton extracton algorthm to the feld of human skeletonzaton [9 11]. Appl. Sc. 2017, 7, 685; do: /app

2 Appl. Sc. 2017, 7, of 25 Based on the proposed SSSE method, sx 3D jont postons n the human skeleton can be precsely extracted n outdoor scenaros wth a normal monocular camera. Compared wth current ndoor 3D human skeleton extracton methods based on RGB-D cameras lke Knect, the proposed SSSE method reduces constrants on nput devce choce and applcaton envronment setup. Ths work s motvated by the procedure of takng a slhouette photo. Durng ths procedure, the human body blocks a part of lght from reachng flm or sensor, leavng a body sketch on the slhouette photo. Human shadow on the ground, from the aspect of slhouette magng, can be regarded as a slhouette photo of the human body on a specal gant flm. The ground surface plays the role of flm. For captured frames contanng human shadows, each shadow on the ground can provde extra human contour nformaton from a unque observaton angle vew other than the camera vew. Ths paper manly focuses on the extracton and aggregaton of the extra slhouette nformaton from spatal temporal dscrete human shadows on the ground, amng to perform 3D human skeletonzaton wth a monocular camera n outdoor scenaros. Based on the aggregaton of multple shadows from dscrete spatal temporal coordnates, SSSE s capable of launchng 3D human skeletonzaton even n outdoor scenes where the scale s too large for tradtonal methods [12 14] to handle [6,15,16]. The man contrbutons of ths paper are related to three aspects: (1) The 3D human skeletonzaton s realzed wth a normal monocular camera based on the proposed SSSE method. (2) The proposed SSSE method acheves 3D human skeletonzaton n a large-scale outdoor scene. (3) The proposed SSSE method deploys the aggregaton of temporal spatal dscrete two-dmensonal (2D) shadow nformaton n a 3D human skeletonzaton procedure The remanng sectons of ths paper are organzed as follows: In Secton 2, the basc theory for shadow-based sngle frame human skeletonzaton s ntroduced frst, followed by the advanced SSSE method aggregatng temporal spatal dscrete shadow nformaton to recover complete skeleton sequences. In Secton 3, a fve-step method s ntroduced to deploy the proposed SSSE method n large-scale outdoor scenaros wth a monocular camera. In Secton 4, the effectve range and precson of the SSSE skeletonzaton results are evaluated n comparson wth the skeletonzaton result of tradtonal RGB-D method. Addtonally, a fuson applcaton of the SSSE and RGB-D skeletonzaton method s acheved n Secton 5, provdng much wder effectve range n outdoor scenaros. Eventually, the advantages and potental applcatons of the proposed SSSE method are llustrated n Secton Basc Theory Ths secton presents the basc theory of the slhouetted shadow-based 3D human skeletonzaton method. To llustrate our method clearly, the basc theory under multple lght source scenaros s ntroduced frst. Then the advanced theory desgned to aggregate temporal spatal dscrete shadow nformaton s ntroduced to acheve skeleton recovery under sngle lght source scenaros Skeleton Smulaton n Mult-Lght-Source Scenaros In a multple lght source scenaro, contour of each human shadow on the ground s decded by two factors: (1) human contour shape. (2) postonal relatonshp between the lght source and the human. Snce each sngle shadow on the ground s restrcted n a 2D plate, t s mpossble to reproduce 3D nformaton from any sngle shadow mage. However, multple shadows generated by dfferent lght sources can carry contour nformaton from multple 3D vew angles, allowng the reproducton of 3D nformaton.

3 Appl. Sc. 2017, 7, of 25 A 3D voxel model of an object can be smulated from shadows generated by a annular set of lght sources [6]. However, out of the laboratory envronment, accurate manual arrangement of lght source postons s elusve. Thus SSSE s desgned to be adaptve to the posteror combnaton of random lght source postons. Wth two or more shadows generated by dfferent lght sources, our method s capable of smulatng 3D human skeleton nformaton. In a multple lght source scenaro shown n Fgure 1a, Sh a and Shb are shadows of human body M, generated by lght sources Sa and Sb respectvely. The scenaro s captured by camera C, and the human body M and shadows Sh a, Shb are captured as P, Shc a, Shcb n the frame, respectvely. Wth the captured frame, 3D poston of the certan jont part M p M can be extracted based on the followng three steps. (a) (b) Fgure 1. Demo of slhouetted shadow-based skeleton extracton (SSSE) n a mult lght source scenaro: (a) A smulated dual lght source scenaro; (b) Scenaro reconstructon D Scenaro Reproducton The slhouettes of human shadow are projected on the ground. To locate 2D shadow areas correspondng to dfferent human jonts, 3D scenaro reproducton s launched to extract orgnal 2D slhouette nformaton Sh from the correspondng mages Shc captured by camera C. The extracton s launched through the a two step perspectve transformaton between the ground surface plane S(u, v) and the camera coordnate plane C ( x 0, y0 ). Due to the progressve road engneerng and partal patchng, the heght levels between dfferent road parts are normally dscontnuous. Instead of deployng global perspectve transformaton between the ground surface plane S(u, v) and mage coordnate plane Im( x, y), ths work proposes a block matrx-based projecton transformaton optmzed for uneven road surfaces. Before the 3D scenaro reproducton procedure, the projecton transformaton parameter matrces are calculated once. Then frame-by-frame block matrx-based projecton transformatons are launched to extract orgnal slhouette nformaton. In order to llustrate the block matrx-based projecton transformaton clearly, tradtonal plane-to-plane projecton transformaton s presented frst. Then the block matrx-based projecton transformaton s ntroduced along wth the optmzed extracton soluton for parameter matrces. Based on the extracted parameter matrces, the smplfed Equaton (15) for frame-by-frame projecton transformaton s presented. Plane-to-Plane Projecton Transformaton Durng the magng process of a monocular camera, the projecton transformaton from ground surface plane to mage coordnate plane s carred out n two steps. Frstly, each pont (u, v) on ground surface plane s projected to correspondng coordnates ( x 0, y0 ) on the camera coordnate plane.

4 Appl. Sc. 2017, 7, of 25 Secondly, a lnear transformaton happens nsde the camera, transformng coordnates (x, y ) to pxel coordnates (x, y) on the mage coordnate plane. The projecton transformaton between the pont coordnates (u, v) on the ground surface plane and the correspondng coordnates (x, y ) on the camera coordnate plane s presented as below: [x, y, w ] = [u, v, 1] A (1) w s a fxed camera nternal parameter that affects the lnear transformaton from the camera coordnate plane to the mage coordnate plane. Notceably, A s the projecton transformaton calbraton matrx that defnes the relatonshp between ground surface plane and camera coordnate plane. Multple transformatons are taken nto consderaton n archtectng the projecton transformaton calbraton matrx A. Rotaton transformaton. Most survellance cameras are not precsely set up at the horzontal angle whch s parallel wth the ground surface. The non-horzontal nstallaton atttude brngs a rotated feld of vew. The rotaton transformaton s ntroduced to calbrate the rotated feld of vew, ensurng the calbrated feld of vew parallel wth the ground surface. Scale transformaton. The coordnate system of the ground surface plane s measured n centmeters. However, pxel s the basc unt of measurement n the mage coordnate plane. Thus the scale transformaton s ntroduced to brdge two dfferent unts of measurement, extractng ground surface plane coordnates from the pxel coordnates. Both rotaton transformaton and scale transformaton are lnear transformatons. The coordnates of both transformatons are combned nto the lnear parameter matrx L. Translaton transformaton. For the mage coordnate plane, the orgn of the coordnate system s fxed at the bottom left corner. For each captured frame, the orgn of the coordnate system on the ground surface plane does not necessarly concde wth the orgn of mage coordnate plane. The translaton transformaton s ntroduced to calbrate the translaton between two coordnate systems. The detaled parameters for translaton transformaton are gven n parameter matrx T. Perspectve transformaton. Instead of the flat vew, a perspectve vew s captured by each monocular survellance camera n each frame. Thus, the perspectve transformaton s ntroduced to recover the flat ground surface plane from the captured perspectve vew. The detaled perspectve transformaton parameters are gven n parameter matrx P. For lnear transformaton parameter matrx L, the scale transformaton parameters c x and c y and rotaton angle θ are ncluded. [ ] c L = x cos θ c x sn θ (2) c y snθ c y cos θ The translaton transformaton parameter matrx T s made up of translate values t x and t y n dfferent axs drectons. [ ] T T = t x t y (3) The perspectve transformaton parameter matrx P s made up of perspectve values p x and p y n dfferent axs drectons. [ ] T P = p x p y (4)

5 Appl. Sc. 2017, 7, of 25 Based on the detaled transformaton parameter matrces T, L and P, the projecton transformaton matrx A can be presented as: [ ] L P A = T T = 1 c x cos θ c x sn θ p x c y snθ c y cos θ p y t x t y 1 = a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 1 (5) Based on the Equatons (1) and (5), coordnates x,y and w on the camera coordnate plane can be presented by coordnates u, v on the ground surface plane and sub-parameters of matrx A. x = a 11 u + a 21 v + a 31 y = a 12 u + a 22 v + a 32 w = a 13 u + a 23 v + 1 (6a) (6b) (6c) Then, a lnear transformaton carred out to calculate pxel coordnates x and y n mage coordnate plane. The transformaton s controlled by the camera nternal parameter w. (x, y) T = ( x w, y w ) T (7) Eventually, the pxel coordnates x and y can be presented by ground surface plane coordnates u, v and sub-parameters of projecton transformaton calbraton matrx A. x = a 11u + a 21 v + a 31 a 13 u + a 23 v + 1 y = a 12u + a 22 v + a 32 a 13 u + a 23 v + 1 (8a) (8b) Addtonally, f the human shadow pxel coordnates x and y and projecton calbraton matrx A s acknowledged, the real-world coordnates u and v of the human shadow can be extracted based on solvng the Equatons (8a) and (8b). The procedure of solvng real-world coordnates u and v s smplfed n Equaton (9). (u, v) T = f ((x, y) T, A) (9) Block Matrx-Based Projecton Transformaton Parameter Calculaton The tradtonal plane-to-plane projecton transformaton s desgned for deal scenaros wth contnuous flat ground surface. Nevertheless, the realstc scenaros contan uneven ground surfaces wth dscontnuous pavement levels. Thus, the sngle projecton transformaton calbraton matrx A s not capable of ensurng precse projecton transformaton for all sub-blocks of the uneven ground surface. In order to deploy the projecton transformaton on realstc scenaros wth hgh precson, a block matrx-based projecton transformaton s proposed n ths part. Instead of deployng mprecse plane-to-plane global transformaton, the proposed method launches a set of precse sub-transformatons. Each sngle sub-transformaton covers only one partally flat sub-block on the ground surface, ensurng the precse projecton transformaton between a surface sub-block and the correspondng mage subset. For each sub-block, the unque projecton transformaton calbraton matrx A sub s non dentcal wth the parameter matrces belongng to other sub-blocks. The parameter matrces A sub of dfferent sub-blocks are calculated separately based on Equatons (8a) and (8b). To solve the unque calbraton matrx of each sub-block, four pars of marked pont coordnates on ground surface plane and ther correspondng pxel coordnates on

6 Appl. Sc. 2017, 7, of 25 mage coordnate plane are requred. However, manpulatng massve markers to calculate parameter matrces of all sub-blocks wll brng a heavy workload. In order to smplfy the setup, the optmzed block matrx based parameter calculaton procedure s desgned to be marker coordnates multplexable and parallel computng frendly. From the top-vew angle, the ground surface s dvded nto a matrx consstng of multple ntensve square sub-blocks as shown n Fgure 1b. Each sub-block s a unt square area Sq sub defned by the four corner markers, occupyng one meter square area on the ground surface as shown n Fgure 1b. The coordnate set of four markers on ground surface s defned as Msub S = {(u, v ), = 1, 2, 3, 4}, ther correspondng mage coordnate set s Msub Im = {(x, y ), = 1, 2, 3, 4}. For each sub-block Sq sub, a set of auxlares s ntroduced to smplfy the calculaton of parameter matrces A sub based on Equaton (10). The scale auxlary parameters set ncludes x 1, x 2, y 1, and y 2. x 1 = x 2 x 3 x 2 = x 4 x 3 y 1 = y 2 y 3 (10) y 2 = y 4 y 3 x 3 = x 1 x 2 + x 3 x 4 y 3 = y 1 y 2 + y 3 y 4 Addtonally, the parallel auxlary parameters x 3 and y 3 are ntroduced as Equaton (10) as well. If both auxlary parameters x 3 and y 3 approach zero, the feld of camera vew s regarded as parallel wth the sub-block. The translaton parameter T sub, perspectve parameter P sub and lnear parameter L sub n each calbraton matrx A sub can be solved as: [ ] T [ ] T T sub = a 31 a 32 = x 1 y 1 (11a) L sub = [ P sub = [ a 11 a 12 a 21 a 22 a 13 a 23 ] T = [ x 3 y 2 x 2 y 3 x 1 y 2 x 2 y 1, x 1 y 3 x 3 y 1 x 1 y 2 x 2 y 1 ] T ] = [ (x 2 x 1 + a 12 x 2 ) (y 2 y 1 + a 13 y 2 ) (x 4 x 1 + a 12 x 3 ) (y 4 y 1 + a 23 y 4 ) ] (11b) (11c) The extracton procedure of the block matrx based projecton transformaton calbraton matrx can be smplfed as: [ ] A sub = P(Msub S, MIm sub ) = L sub Psub T (12) T sub 1 Block-Matrx Based Projecton Transformaton Deployment Based on Equaton (9) and the calculated parameters n matrx A sub, real-world coordnates (u, v) of each pont n one sub-block area can be calculated from the correspondng pxel coordnates (x, y). The presentaton of extracton procedure can be smplfed as: (u, v) T = f ((x, y) T, A sub ) (13) Notceably, dfferent from the orgnal global calbraton matrx A, each sub-block calbraton matrx A sub s only deployed on the restrcted regonal transformaton between the sub-block area on the ground and the correspondng pxel range n the mage. Once all parameter matrces A sub for dfferent sub-blocks are extracted through the block matrx-based parameter calculaton procedure, coordnates (x, y) of pxels belongng to dfferent

7 Appl. Sc. 2017, 7, of 25 sub-blocks can be transformed to correspondng real-world coordnates (u, v) nsde the sub-block Sq sub. Block matrx-based projecton transformaton s deployed based on the parallel computaton of sub-transformatons llustrated n Equaton (13). The deployment algorthm of a sub-transformaton s llustrated n Algorthm 1. Algorthm 1: Block matrx based projecton transformaton deployment Algorthm Input: Msub S = {(u, v ), = 1, 2, 3, 4}: coordnates set of marker postons for sub-block Sq sub ; Msub Im = {(x, y ), = 1, 2, 3, 4} :correspondng pxel coordnates set of Msub S on mage coordnate plane Im(x, y); (x, y): mage coordnates of captured pxel n human shadow slhouette Output: (u, v): correspondng real-world coordnates of (x, y) 1 foreach Sq sub do 2 [ x 1, x 2, y 1, y 2 ] = [x 2 x 3, x 4 x 3, y 2 y 3, y 4 y 3 ] 3 [ x 3, y 3 ] = [x 1 x 2 + x 3 x 4, y 1 y 2 + y 3 y 4 ] 4 T sub = [ a 31 a 32 ] T = [ x 1 y 1 ] T 5 P sub = [ a 13 a 23 ] T = [ x 3 y 2 x 2 y 3 x 1 y 2 x 2 y, x 1 y 3 x 3 y 1 1 [ ] [ 6 L sub = a11 a T 12 (x2 x a 21 a 22 = 1 + a 12 x 2 ) (y 2 y 1 + a 13 y 2 ) (x 4 x 1 + a 12 x 3 ) (y 4 y 1 + a 23 y 4 ) [ ] Lsub P 7 A sub = sub Tsub T 1 ; 8 end 9 (u, v) T = f ((x, y) T, A sub ) x 1 y 2 x 2 y 1 ] T ] T For each sub-block area Sq sub, a dstnctve sub-transformaton thread s launched based on the specfc calbraton matrx A sub. The parallel computaton of block matrx-based projecton transformaton contans multple sub-transformaton threads. For the smplcty of the parallel computaton presentaton, A mat s ntroduced as the collecton of all calbraton sub-matrces {A sub } for dfferent sub-blocks. The overall transformaton s smplfed as Equaton (14). (u, v) T = F((x, y) T, A mat ) (14) Based on Equaton (14), the real-world coordnates (u, v) of human shadow slhouette Sh can be extracted from correspondng pxel coordnates (x, y ) Sh c captured by a monocular camera. The block matrx-based projecton transformaton between the captured human shadow slhouette Sh c and the correspondng real-world shadow slhouette Sh s llustrated n Equaton (15). Sh = F(Sh c, A mat ) (15) The benefts of the block matrx based projecton transformaton are obvous: The postons of markers can be reused to smplfy the scenaro set up. For a scenaro contanng a m n square meter area, the number of markers s reduced from (4 m n) to (m + 1) (n + 1). Parallel sub-transformatons on dfferent sub-blocks can be processed synchronously to accelerate the overall projecton transformaton procedure. Only when the poston of camera s moved or the ground surfaced s repaved, wll partal recalbraton work be necessary for the affected sub-block Sq sub. Overall, all parameter matrces A sub for dfferent sub-blocks only need to be calculated once. Then all pxel coordnates n vdeo frames can be transformed nto the real-world coordnates on the ground surface plane. The block matrx-based structure also smplfes the parameter mantenance procedure when changes occur n the scenaro.

8 Appl. Sc. 2017, 7, of Slhouette Informaton Extracton For the extracted human shadow contour Sh on the ground surface, jont postons are extracted through an optmzed method based on the slhouette contour extreme pont seekng method. Comparng wth tradtonal human segmentaton methods, only slhouette nformaton s avalable for shadow contour segmentaton n our work. In order to perform an effcent jont poston extracton based on precse slhouette contour segmentaton [17], a two-step algorthm s presented n ths secton. Human Shadow Slhouette Contour Preprocess Frstly, a survey for global peak ponts on the shadow contour s launched to locate most obvous jont postons on the human shadow contour. In ths step, the gravty center coordnate (u, v) of human shadow contour Sh s calculated frst. For human shadow contour Sh contanng N contour ponts (u m, v m ), the gravty center (u, v) can be extracted based on the Equaton (16). ( (u, v) = 1 N ) N N 1 u m, N v m m=1 m=1 (16) Then, the the dstance curve D between contour ponts (u m, v m ) Sh and the gravty center (u, v) s calculated for the localzaton of global peak ponts. The value of each pont on the dstance curve D s calculated based on Equaton (17). The Cartesan dstance s appled n the Equaton (17) as a lnearzed approxmaton for the value of each pont on the dstance curve. D (m) = (u m u) 2 + (v m v) 2 (17) In order to reduce the nterference of grany ground surface n the jont poston extracton procedure, the dstance curve D s denosed based on Equaton (18). The smooth length unt η s set as 10 n our experment. In the next step, the localzaton procedure of major jont postons s based on the denosed dstance curve D. η D (m) = η D (u m+l, v m+l ) (18) l= η 2 The global peak ponts ncludng head and two feet appear at the maxmum pont on the dstance curve. Based on the denosed dstance curve D, the major jont postons can be located through seekng peak ponts. The normalzed dstance curve extracton procedure s llustrated from Equaton (16) to Equaton (18) and smplfed n the stage Equaton (19). In order to smplfy the subsequent presentatons, functon Pre s ntroduced to cover the extracton procedure for the normalzed dstance curve D based on the human shadow contour Sh. D = Pre(Sh) (19) Localzaton of Major Jont Postons on Human Shadow Slhouette Contour In the second step, a quck localzaton of global maxmum peaks s launched frst to locate the postons of head and both feet, then elaborate local search for major jonts ncludng hands, shoulders and knees s carred out. (1) Localzaton of Global Convex Areas Three global maxmum peaks of denosed curve D s marked n correspondng postons on Fgure 2a wth square symbols. The marked postons ndcate precse global convex area on the

9 Appl. Sc. 2017, 7, of 25 human shadow slhouette contour, ncludng head Sp head, left foot Sp f oot le f t and rght foot Sp f oot rght. As shown n Fgure 2b, the area contanng the head are marked n red, and areas contanng the feet are marked n green. (2) Localzaton of Auxlary Anchor Ponts Based on the acknowledged major jont postons ncludng head and feet, the postons of rest jonts are calculated through locatng the local peak and nadr ponts. Based on the three major jont postons, the shadow contour s dvded nto three sub-curves. Each sub-curve contans one auxlary anchor pont at the correspondng local nadr poston on curve D. The auxlary anchor ponts are markered wth star symbols n Fgure 2a. The sub-curve between two feet jonts contans the poston of hp center Sp hp at the local nadr poston. The sub-curves between the head poston and two feet postons contan postons of two oxters at local nadr postons, respectvely C 3 sub C 4 sub C 2 sub C 5 sub C 1 sub Global Convex Local Convex Global Concave C 6 sub (a) foot left keen foot left rght keen rght hand left hand rght elbow elbowleft rght shoulder rght neck shoulder left head (b) Fgure 2. Slhouette nformaton analyses and jont poston extracton. (a) Sub-curve segmentaton; (b) two-dmensonal (2D) jont poston extracton on the shadow area. (3) Localzaton of Remanng Major Jont Postons Based on the three global peaks and three auxlary anchor ponts, the shadow contour s subdvded nto sx new sub-curves. To llustrate the extracton of remanng jont postons clearly, sx sub-curves are marked as Csub. The ndcated ranges from 1 to 6 as shown n Fgure 2a. C1 sub to Csub 6 cover the shadow contour n a clockwse drecton, ntatng from the head poston. Sub-curves Csub 1 and C6 sub cover the contour ranges of left arm and rght arm. Thus the local peak postons of these two sub-curves are hand postons. Ther local nadr postons are located between the cervcal vertebra poston Sp neck and two shoulders. The local peak postons of C 2 sub and C4 sub ndcate the postons of two keens Spkeen le f t and Sp keen rght n the shadow area. Smlarly, the nadr postons of C 3 sub and C5 sub can assst the postonng of both keens Spkeen le f t and Sp keen rght. The major jont poston localzaton procedure s llustrated n the three steps above and smplfed n stage Equaton (20). In order to smplfy the subsequent presentatons, functon Loc s ntroduced to cover the localzaton procedure for major jont poston set { Sp jont} based on the human shadow contour Sh and the correspondng dstance curve D. { Sp jont } = Loc ( Sh, D ) (20)

10 Appl. Sc. 2017, 7, of 25 Notceably, the jont poston localzaton procedure can also be adopted n the jont poston extracton from a normal human pose contour. The human pose classfcaton llustrated n Secton s based on the jont poston extracton procedure llustrated n Equaton (20) D Jont Poston Estmaton and Skeleton Synthess In a multple lght source scenaro, more than one human shadow s projected on the ground surface at the same tme. In order to dentfy shadow areas generated by dfferent lght sources, 2D human shadow contour Sh and jont poston regon Sp jont are footnoted wth correspondng lght source dentfer as shown n Equaton (21). Addtonally, the pont coordnates (u, v) nsde the each regon Sp jont are footnoted as (u jont, v jont ). Sp jont Sh (21) In order to estmate the 3D jont poston Mp jont of each major jont, the lght beams L jont from dfferent lght sources S blocked by Mp jont are reconstructed frst. Then, the 3D poston of Mp jont s calculated based on allocatng the shared voxel area between multple reconstructed lght beams L jont. Fnally, the human skeleton s syntheszed based on the calculated 3D jont poston set {Mp jont }. For the frst step, each lght beam L jont s generated as a 3D cone wth ts vertex on the lght source poston S = (u, v, h ). The undersde of each cone s the jont area Sp jont Whle any lght beam L jont human body M, the jont shadow area Sp jont the drecton of blocked lght beam L jont n the shadow. from lght source S s blocked by the certan jont part Mp jont of = {(u jont leads to shadow area Sp jont, v jont )} s produced on the ground. Thus, Sh, gong through human body part Mp jont. If w [0, h ] s ntroduced as the heght component n the cone expresson of L jont, the 3D space caused by L jont can be presented as Equaton (22). L jont = ( (h w)u jont h, (h w)v jont h, w) (22) The 3D lght beam shape extracton procedure presented by Secton s smplfed n the stage Equaton (23). For the smplcty of the subsequent presentatons, functon Occ s ntroduced to cover the 3D lght beam shape extracton procedure for the occuped 3D cone shape L jont correspondng jont poston Sp jont and the lght source poston S. L jont based on the = Occ(Sp jont, S ) (23) Then, for multple lght beams L jont generated by dfferent lght sources S, Mp jont s the shared subset for all reconstructed L jont. Thus, the 3D poston Mp jont can be generated based on the ntersecton of all reconstructed L jont as shown n Fgure 3a. sum l s the sum of lght sources n the scenaro. Sp neck a Mp jont = sum S =1 L jont (24) Fgure 3a demonstrates the recovery of neck jont area Mp neck based on two related shadow areas and Sp neck b generated by lght sources S a and S b. Fnally, 3D human skeleton Sk wth multple jont postons s syntheszed by calculatng Mp jont jont by jont. Fgure 3b presents the skeleton synthess procedure of a human beng based on human shadow nformaton under a mult lght source scenaro. The llustrated human skeleton synthess procedure s presented n Algorthm 2 and smplfed n Equaton (25).

11 Appl. Sc. 2017, 7, of 25 Sk = Syn({Sh }, {S }) (25) However, there are two restrctons for the deployment of the basc theory: Condton (1) Two or more lght sources are requred n the scene. Condton (2) Relatve angular postons between human body and dfferent lght sources should be dfferent (a) (b) Fgure 3. Demo of SSSE n a multple lght source scenaro: (a) A smulated dual lght source scenaro. Sp neck a and Sp neck a are the jont areas of the neck poston n the shadows projected by lght source a and b, respectvely. Smlarly, lght beams L neck a and L neck b are generated by lght sources a and b, respectvely. The enclosure Mp neck s the ntersecton area of L neck a and L neck b. (b) Scenaro reconstructon. Algorthm 2: Human skeleton synthess procedure under multple lght source scenaro Input: {Sh }: 2D human shadow contour on the ground surface. {S } :3D postons of multple lght sources {(u, v, h )}. Output: Sk: 3D human skeleton synthess based on seven major jont postons. 1 foreach S = (u, v, h ) do 2 { D = Pre(Sh ) Sp jont } = Loc ( ) Sh, D 3 4 L jont = ( (h w)u jont h 5 end 6 foreach jont do 7 Mp jont = sum l 8 end 9 Sk = {Mp jont } =1 L jont, (h w)v jont h, w), w [0, h ] 2.2. Skeleton Smulaton n Sngle-Lght-Source Scenaro The basc theory ntroduced n Secton 2.1 s only effectve n scenes contanng two or more shadows generated by multple lght sources. For sngle lght source scenaros, only one shadow s generated n each captured frame. In order to extend the proposed basc theory n sngle lght source scenaros, a vdeo sequence nstead of a sngle frame s taken nto consderaton. Human shadow contours are footnoted wth tme coordnate t n ths part. The extenson soluton s ntroduced below.

12 Appl. Sc. 2017, 7, of Theoretc Proof of the Extenson Soluton n a Sngle Lght Source Scenaro For every vdeo sequence, the extenson soluton s based on two facts: Temporal Dstngushed Relatve Poston between Lght Source and Human Body In a sequence, the relatve poston between a movng human and a fxed lght source keeps changng. In other words, temporal dscrete shadows Sht are generated by the lght sources from dfferent relatve postons θt towards the human. The temporal neghborng human shadows Sht and Sht+1 are dstngushed from each other because of dfferent relatve postons between the lght source S and the human body. For neghborng frames at tme coordnates t and t + 1, t s clear that θt 6= θt+1 and Sht 6= Sht+1. Temporal Dscrete Shadows for Same Human Pose In order to categorze dfferent frames based on the human poses, the 2D contour of human body captured by a monocular camera s regarded as the human pose Pt at the tme coordnate t. As shown n Fgure 4b, same human pose P0 appears repeatedly durng an actvty sequence. Snce each relatve poston between the lght source and human body s dfferent frame by frame, multple frames sharng the same human pose P0 can be found. Each frame owns dfferent shadows Sht and projecton angles sθt. In an actvty sequence, all the human shadows Sht sharng the same human pose P0 are categorzed nto the set {Sht }. For dfferent human shadows Sht {Sht }, ther correspondng relatve poston angles θt are dstngushed from each other. If multple human shadows n {Sht } wth dfferent projecton angles θt are ntegrated n one sngle frame as shown n Fgure 4b, condton (2) of launchng the basc theory proposed n Secton 2.1 s satsfed. Through applyng translaton transformatons on each ntegrated frames to make the all human poses Pt spatally concde wth the central pose Pjc, an artfcal multple lght source scenaro satsfyng condtons (1) and (2) s establshed as shown n Fgure 4c. (a) (b) (c) (d) Fgure 4. Demo of SSSE procedure n a sngle lght source scenaro. (a) Pose classfcaton based on major jont postons; (b) Spatal temporal dscrete human poses belongng to same class; (c) Temporal spatal aggregaton; (d) Three-dmensonal (3D) human skeletonzaton.

13 Appl. Sc. 2017, 7, of 25 The smulated scenaro makes t feasble to recover the skeleton of the shared pose P o n a sngle lght source scenaro based on the basc theory proposed n Secton Temporal Spatal Aggregaton Method Before the deployment of human skeletonzaton, t s necessary to fnd shadows that share the same human pose P 0, yet have dstnctve projecton angles θ t. Human pose classfcaton and temporal spatal shadow aggregaton are deployed to ft spatal coordnates of shadows n {Sh t } wth the chosen central pose poston P tc. Human Pose Classfcaton In order to analyses the human pose P t at each tme coordnate t, the denosed dstance curve D t between the human pose contour and human pose gravty center s extracted based on the same method llustrated n Equaton (19). Smlarly, the stage Equaton (26) covers the normalzed dstance curve extracton procedure llustrated from Equaton (16) to Equaton (18). The functon Pre presents the extracton procedure for the normalzed dstance curve D t based on the contour curve P t. D t = Pre(P t ) (26) Based on the dstance curve D t, major peak pont set {Jp k t k = 1, 2, 3} ncludng head and two feet are extracted from the captured human contour based on the same procedure presented n Equaton (20). Smlar to the stage Equaton (20), stage Equaton (27) covers the major jont poston localzaton procedure llustrated n the Secton The functon Loc presents the human jont poston extracton procedure for major jont poston set {Jp k t } based on the human contour Sh t and the correspondng dstance curve D t. {Jp k t } = Loc(Sh t, D t ) (27) The postons of three peak ponts of the human pose contour are combned nto a star feature to descrbe the human pose n each frame [10,18]. Then unsupervsed classfcaton s adopted to assort each frame wth correspondng pose category label based on the star feature [19]. C t = Lab({Jp k t }) = j arg mn j Temporal Spatal Shadow Aggregaton ( 3 k=1 Jp k j P j Jp k t Jp k j 2 ) (28) Durng the temporal-spatal shadow aggregaton procedure shown n Fgure 4c, temporal dscrete lght sources are aggregated n a sngle frame. Normally, for multple human poses, the human pose P t wth medan tme coordnate t s chosen as the central pose P jc. For each human pose P t {P j }, the translaton transformaton parameter T t j c s defned by the vector between correspondng jont ponts n P t and P jc, satsfyng the spatal transformaton from P t to P jc. Notceably, the jont postons Jp k t and Jp k j c are captured n the mage coordnate plane. Before calculatng the translaton transformaton T t j n real-world coordnates, t s necessary to transform the jont coordnates nto the real- world coordnates Sp k t and Sp k j c based on the stage projecton transformaton F((x, y), A mat ) presented n Equaton (14). Sp k t = F(Jp k t, A mat ) (29a) Sp k j c = F(Jp k j c, A mat ) (29b)

14 Appl. Sc. 2017, 7, of 25 T (t,j c ) = k=2 A t j c = Sp k t Sp k j c = u (t,j c ) α + v (t,j c ) β u (t,j c ) v (t,j c ) 1 (29c) (29d) As shown n Equaton (29a), Sp k t s the extracted real-world human jont coordnates at the orgnal captured poston. In Equaton (29b), Sp k j c s the human jont coordnates at the destnaton poston. Then the translaton vector T t j c s calculated based on the averaged horzontal translaton vectors from the orgnal poston to the destnaton poston. As shown n Equaton (29c), α and β are two vertcal unt vectors of the real-world coordnate system on the ground surface. The translaton vector T t j c s presented as a combnaton of translaton components u (t,j c ) and v (t,j c ) n two vertcal drectons. Based on the translaton components u (t,j c ) and v (t,j c ), a translaton transformaton matrx A t j c can be establshed for the translaton calculaton as shown n Equaton (29d). For the convenence of further llustraton, the extracton procedure of matrx A t j c s smplfed n Equaton (30). The functon Par s ntroduced to present translaton matrx extracton procedure llustrated from Equaton (29a) to Equaton (29d). A t j c = Par({Jp k t }, {Jp k j c }) (30) In order to mantan a consstent expresson system, the translaton transformaton s presented n the same format wth Equatons (14) and (15). When the translaton transformaton n Equatons (31) and (32) s deployed synchronously on the lght source S t and human shadow contour Sh t for each frame, all the transformed human shadows Sh t ft the spatal coordnates of the central human pose P tc n each smulated scenaro. Sh t = F(Sh t, A t j c ) (31) The poston of lght source S t apples the same transformaton T t j c along wth the related shadow Sh t, smulatng multple lght sources S t n the sngle frame. S t = f (S, A t j c ) (32) The temporal spatal aggregaton procedure llustrated above s presented n Algorthm 3. Wth more than two postonal dstnctve lght sources smulated n the same frame, the skeleton synthess procedure presented n Equaton (24) can be appled on the smulated human shadow set {Sh t } and the correspondng lght source set {S t }. Based on Equaton (33), the skeleton Sk jc of pose P jc can be syntheszed under a sngle lght source scenaro as shown n Fgure 4d. The detaled human skeleton synthess procedure under a sngle lght source scenaro s llustrated n Secton 3. Sk jc = Syn({Sh t }, {S t }) (33)

15 Appl. Sc. 2017, 7, of 25 Algorthm 3: Temporal spatal aggregaton procedure Input: t : tme coordnate for each frame; Sh t :human shadow on the ground surface n frame t ; P t : human pose n frame t ; S: lght source poston; {Jp k j c }: jont poston set of the central pose on the aggregaton destnaton ; Output: Sh t : ntegrated human shadow Sh t n the smulated scenaro. S t : ntegrated lght source poston n correspondence wth Sk t. 1 foreach tme coordnate t do 2 D t = Pre(P t ) 3 {Jp k t } = Loc(Sh t, D t ) 4 C t = Lab({Jp k t }) 5 A t j c = Par({Jp k t }, {Jp k j c }) 6 Sh t = F(Sh t, A t j c ) 7 S t = f (S, A t j c ) 8 end 3. Proposed Method Based on the basc theory and ts extenson ntroduced n Secton 2, a normal sngle lght source scenaro can support 3D human skeletonzaton. In ths secton, a fve-step algorthm s proposed accordng to the llustrated theory as shown n Fgure 5. The procedure of the proposed method s shown n Algorthm 4. Pose Classfcaton Temporal-spatal Aggregaton Block-matrx based 3D Reconstructon Projecton Transformaton Shadow Slhouette Analyses 2D Jont Localzaton 3D Jonts Extracton Skeleton Synthess Fgure 5. The flow chart of skeleton synthess procedure based on SSSE.

16 Appl. Sc. 2017, 7, of 25 Algorthm 4: Skeleton synthess procedure Input: t : tme coordnate for each frame; Shc t :captured human shadow n frame t ; P t : human pose n frame t ; S: lght source poston; {Sq sub }: the set of sub-blocks on the ground surface plane S {Msub S }: the marker poston coordnate sets for sub-block Sq sub on S; {Msub Im }: the pxel coordnates set of {MS sub } on the mage coordnate plane. Output: Sk t :3D human skeleton correspondng to Sh t at tme coordnate t 1 foreach Sq sub do 2 A sub = Par(Msub S, MIm sub ) 3 end 4 A mat = {A sub }. 5 foreach P t at t do 6 D t = Pre(P t ) 7 {Jp k t } = Loc(Sh t, D t ) 8 C t = Lab({Jp k t }) 9 A t j c = Par({Jp k t }, {Jp k j c }) 10 end 11 foreach Shc t at t do 12 Sh t = F(Shc t, A mat ) 13 Sh t = F(Sh t, A t j c ) 14 S t = f (S, A t j c ) 15 end 16 foreach Pose Category C do 17 Sk jc = Syn({Sh t }, {S t }) 18 end 19 foreach t do 20 Sk t = T jc t (Sk jc ) 21 end Pose Classfcaton In a human actvty sequence captured under a sngle lght source scenaro, frames at dfferent tme coordnates are classfed based on human poses [8] on the captured frames. For each captured human pose P t at tme coordnate t, the denosed dstance curve between contour ponts and the gravty center of P t can be extracted based on the method presented n Equaton (19). The deployment of the extracton method on the human pose P t s presented n Equaton (34). D t = Pre(P t ) (34) Based on the method presented n Equaton (20), a major peak jont poston set {Jp k t } s extracted from the human contour P t, ncludng the head poston Jp 1 t, the left foot poston Jp 2 t and rght foot poston Jp 3 t. {Jp k t } = Loc(Sh t, D t ) (35) Based on the normalzed peak jont postons, raw frames contanng same class human poses P t s aggregated to the human pose category P j based on the automatc unsupervsed clusterng llustrated n Equaton (28). C t s the category label of human pose P t as shown n Equaton (36). C t = Lab({Jp k t }) (36)

17 Appl. Sc. 2017, 7, of 25 Preprocess The preprocess procedure transforms the captured shadow contour pxel coordnates Shc t nto the real-world coordnates Sh t. Before the preprocess of the frst shadow contour Shc t, all the A sub A mat are calculated and saved for further preprocess procedures. For each square unt area Sq sub, the related projecton parameter matrces A sub are calculated based on four real-world coordnates {Msub S } and ther correspondng magng coordnates {Msub Im } based on Equaton (12). Based on the calbraton matrx set A mat = {A sub }, the global projecton transformaton F(Shc t, A mat ) can be fgured out. Based on the projecton transformaton presented n Equaton (37), captured human shadow contour pxel coordnates Shc t can be extracted from each of the raw frames and transformed nto the real-world coordnates Sh t. Temporal Spatal Aggregaton Sh t = F(Shc t, A) (37) Preprocessed shadow contours Sh t are aggregated accordng to category P j of correspondng human pose P t. Nevertheless, the real-world coordnates of P t P j are spatally dspersed due to the human movement as shown n Fgure 4a. Thus t s necessary to aggregate shadow contours Sh t of the same central human pose P jc to deploy precse jont poston estmaton. For each pose category, one central human pose P jc s set up as the aggregatng destnaton for other human shadow Sh t related wth P t P j. The translaton of each human shadow Sh t s based on the translaton transformaton calbraton matrx A t j c. The translaton transformaton matrx s calculated based on the Equaton (30). Snce the major peak jont poston sets {Jp k t } and {Jp k j c } are obtaned n the pose classfcaton step, the translaton transformaton calbraton matrx A t j c can be extracted as shown n Equaton (38). A t j c = Par({Jp k t }, {Jp k j c }) (38) Along wth the 2D translaton of each Sh t, the correspondng 3D lght source poston S s moved wth the dentcal translaton as shown n Equaton (39b). The aggregated human shadow Sh t and lght source S t offer the deal multple lght source stuaton for 3D jont poston estmaton. As shown n Fgure 4b, when P t2 s setup as the aggregatng destnaton, other Sh t are aggregated to the aggregatng destnaton through the 2D translaton as shown n Equaton (39a). Sh t = F(Sh t, A t j c ) S t = f (S, A t j c ) (39a) (39b) Jont Poston Estmaton and Skeleton Synthess For each aggregated human shadow contour Sh t, jont area estmaton s launched based on the algorthm ntroduced n the basc theory secton. Frst of all, the gravty center G t of curve Sh t s calculated. Then, the denosed dstance curve D t between each pont (x t, y t ) Sh t and G t s avalable based on the preprocess procedure llustrated n Secton D t = Pre(Sh t ) (40) The 2D postons of major jont areas ncludng head Sp 1, neck Sp 2, hp center Sp 3, left keen Sp 4, rght keen Sp 5, left foot Sp 6 and rght foot Sp 7 can be obtaned through locatng the peak and nadr ponts n D t.

18 Appl. Sc. 2017, 7, of 25 {Sp k t } = Loc(Sh t, D t ) (41) In each smulated scenaro, slhouette nformaton extracton s appled to each jont area Sh _k. In order to estmate the 3D jont poston based on Sp t, the ray set L k t connectng lght source S t and jont shadow area Sp t s smulated. L k t = Occ(Sp k t, S t ) (42) Snce more than two smulated lght sources S t exst n the scenaro, slhouette nformaton of sngle jont area s extracted separately for each lght sources. Based on all ray sets L t _k targetng at the same jont, 3D jont poston M k p j can be calculated based on Equaton (43). Mp k j = sum l =1 L k t (43) Repeatng steps above for each major jonts, 3D jont poston set {Mp k j } contanng all jont postons can be fgured out. Then jont postons can be syntheszed based on the combnaton Equaton (44). Sk jc = {Mp k j } (44) In order to smplfy the presentaton n Algorthm 4, the llustrated jont poston estmaton and skeleton synthess procedure s smplfed nto Equaton (45). Sk jc = Syn({Sh t }, {S t }) (45) Frame Integraton Repeatng the above steps, syntheszed 3D human skeletons Sk jc can be generated for all human poses category by category. The knematc model of skeleton Sk pj contans seven major jonts, ncludng head, neck, hp, both keens and both feet. The bones connectng partcular jonts are regarded as rgd objects. Based on the pose classfcaton result n the step (1), the tme coordnate t of each P t P j can be tracked. Then, reassgn syntheszed human skeleton Sk jc to frame t as Sk t based on the reverse translaton transformaton. Sk t = F 1 (Sk jc, A t j c ) (46) 4. Expermental Valdaton In ths secton, the expermental data source and settngs are llustrated frst. Then the effectve range and precson of the proposed method are valdated n comparson wth the RGB-D based method Data Source Descrpton and Expermental Settngs The experments are launched based on data captured by a Knect RGB-D camera, contanng daly human actvtes. Captured sequences nclude both RGB frames and normal depth frames captured by Knect. Knect extracts human skeleton automatcally based on the combned nformaton of RGB frames and depth frames [15]. However, SSSE s deployed only on RGB frames captured by the monocular RGB camera on Knect.

19 Appl. Sc. 2017, 7, of 25 In each sequence captured for effectve range valdaton, Knect s set up at a statc dstance from the human subject. The photographc dstance ncreases from 1 m to 20 m wth a fxed step of 1 m. Sampled skeletonzaton results based on both methods at dfferent dstances are presented n Fgure 6a. (a) Knect RGB wth SSSE (b) Fgure 6. Expermental results. (a) A comparson of trackng results; (b) Effectve ranges of RGB-D-based results and SSSE-based results Effectve Range and Precson Analyses In order to valdate the effectveness of the proposed SSSE method and tradtonal RGB-D method, two aspects ncludng effectve range and precson are evaluated. In the followng, the effectve dstance range s marked frst. Then, the precson of sx major 3D jont postons extracted by SSSE s evaluated. Effectve Range Effectve range s defned as the dstance between the sensor and human, whch allows effectve human skeleton extracton. Effectve human skeleton extracton n the effectve range generates vald human jont postons. For the RGB-D based method, each extracted jont poston comes wth a

20 Appl. Sc. 2017, 7, of 25 confdence ndex. Vald jonts are jonts wth confdence above 0.7. For the SSSE method, vald jonts are extracted jonts not affected by shelterng. In the followng experments, frames wth all vald smulated human jonts are defned as effectve frames. In order to obtan the effectveness dstance relatonshp of both methods, the shares of effectve frames at dfferent dstance levels are measured. In addton, 1000 to 1200 frames contanng 3D human skeletons sampled at each photographc dstance from 1 to 20 m are evaluated for each method. For the RGB-D-based skeletonzaton procedure, effectve frames are automatcally labeled based on the correspondng jont confdence. For an SSSE-based procedure, effectve frames are chosen based on the number of vald jonts n each skeleton. The effectveness of both methods at the same dstance can be represented by the shares of effectve frames among all frames. For each method, effectve range covers photographc dstances whose effectveness exceed a specfed threshold. The offcal parameter of Knect [1,15] ndcates the effectve range of state-of-art Knect result s from 0.8 m to 3.5 m. Thus, the range of dstance where effectveness s above 0.8 s regarded as the effectve range. As shown n Fgure 6b, the effectve range of SSSE s 7 10 m. Note that the effectveness of SSSE decreases when photographc dstance exceeds 10 m because of the lmtaton of camera resoluton. The expermental result n Fgure 6a shows that SSSE can provde relable 3D human skeletonzaton at an effectve range of 7 10 m, whle Knect s unable to extract human skeleton nformaton when the photographc dstance exceeds 5 m. Precson Evaluaton As wth the effectveness evaluaton result mentoned above, the RGB-D-based method and SSSE provde effectve skeleton extracton results at dfferent dstance ranges. Precsons of all extracted jonts by SSSE are determned by the devaton values relatve to correspondng ground truth jont postons. In the precson evaluaton procedure, two Knects are setup for dfferent purposes. Knect No.1 s set up 9 m way from human object, capturng RGB frames for human skeletonzaton based on SSSE. Knect No.2 s setup 3 m away from human object, capturng RGB-D frames along wth 3D human skeletons smultaneously. Snce 3 m s nsde the effectve range of the RGB-D-based 3D skleletonzaton, the 3D jont postons captured by Knect No.2 are vald jonts, provdng ground truth for the devaton calculaton. Based on the expermental scenaro setup, 1546 frames are captured smultaneously for both methods, of whch 1345 effectve frames are evaluated. For each skeleton extracted from a effectve frame, jont postons are normalzed relatve to the hp center, avodng devaton ntroduced by dfferent shot dstances. Sx major jonts are consdered n evaluaton, ncludng head, spne, both keens and both feet. Fgure 7 depcts the averaged precson evaluaton result. As presented n Fgure 7, due to the larger scale of upper body shadow on the ground, relatve hgh devatons appear at jonts of the head and spne, where averaged devatons reach 14.5 cm and 12.1 cm, respectvely. For the remanng jonts, the averaged devatons are around 4 cm and the hghest devaton remans below 8 cm. In summary, SSSE extracts jont postons n a reasonable precson at 9 m away from the target human, compared wth the ground truth Knect skeletonzaton result obtaned at a poston 6 m closer to the subject human.

21 Appl. Sc. 2017, 7, of 25 Fgure 7. Devaton between SSSE-extracted jonts and ground truth. 5. Dscusson Based on the expermental results n Secton 4.2, an nterestng phenomenon can be observed n that the effectve ranges of the proposed SSSE and tradtonal RGB-D method are hghly complementary. Thus, the fuson applcaton of SSSE and tradtonal RGB-D method can provde wde range human skeletonzaton for ndoor and outdoor scenaros. In the fuson method, the tradtonal RGB-D method and SSSE are deployed under dfferent scenaros. For humans nsde the effectve range of RGB-D cameras, the tradtonal RGB-D based skeletonzaton method can provde sold human skeleton extracton method. For humans outsde the effectve range of RGB-D cameras, SSSE method can redress the unrelable 3D jont postons appears n RGB-D skeletonzaton result. In order to evaluate the fuson applcaton effectveness, a comparson between the relable jont percentage of orgnal skeletons extracted by Knect and redressed skeletons processed by SSSE s carred out n ths secton. Relable jonts are defned as jonts generated by SSSE not affected by shelterng, and jonts generated by Knect wth a confdence ndex above 0.7. On the contrary, unrelable jonts are unavalable jonts affected by shelterng n SSSE methods, or jonts generated by Knect wth confdence ndex under 0.7. For better evaluaton of the fuson applcaton, Knect s set up to skeletonze a human subject outsde ts effectve range. The unrelable 3D jont postons n Knect skeletonzaton result s redressed by SSSE smultaneously. In total, 20 sets of experments have been launched to evaluate the relable jont percentage enhancement Relable Jont Percentage Enhancement The enhancement of the relable jont percentage s evaluated by determnng the precsely recovered jont rate J R and precsely recovered frame rate F R. As shown n Equaton (6a) (6c), N Ej and N E f are the unrelable jont number and relevant affected frame number, respectvely. N rj s the number of total recovered unrelable jont postons after deployng the SSSE procedure, whle E rj s the number of naccurately recovered jonts. From the aspect of frame statstcs, N r f s the total number of recovered frames and E r f s the number of frames contanng naccurately recovered jonts. J R = N rj E rj N Ej F R = N r f E r f N E f (47)

22 Appl. Sc. 2017, 7, of 25 The 20 test sets presented n Table 1 ndcate that more than four-ffths of all unrelable jonts are successfully redressed based on the proposed SSSE method, and more than three-quarters of all frames contanng unrelable jont skeletonzaton results are accurately fxed. Based on the expermental results above, the fuson applcaton of SSSE and tradtonal RGB-D method proved effectve n relable jont percentage enhancement. Table 1. Result of unrelable jont poston redress. J R s the precsely recovered jont rate. F R s the precsely recovered frame rate. N Ej s the number of unrelable jonts. N E f s the number of frames affected by unrelable jonts. N rj s the number of total recovered unrelable jont. E rj s the number of naccurately recovered jonts. N r f s the total number of recovered frames. E r f s the number of frames contanng naccurately recovered jonts. Test Set Jonts Frames Set No. J R N Ej N rj E rj F R N E f N r f E r f Total Computatonal Cost Evaluaton The smultaneous collaboraton between the RGB-D skeletonzaton method and proposed SSSE method s crucal for the real-tme deployment of the fuson applcaton. Thus, lmtng the computatonal cost s essental for the effectveness of the fuson method. The test platform s a manstream personal laptop connected wth the frst generaton Knect, equpped wth one Intel Core 7 central processng unt (CPU) and 16 Ggabyte of random access memory (RAM). Two ndcators,.e., maxmum process capablty per second and sngle frame delay are concerned n order to evaluate the computatonal cost. Ths evaluaton test ams to process as many frames as the computatonal capablty allows based on the proposed method. The computaton cost effcency of the fuson applcaton s determned by the number of frames processed per second. As shown n Fgure 8, the stable maxmum process capablty remans around 25 frames per second after the ntal stage where less than 10 frames are processed per second. The expermental result ndcates that the fuson applcaton s feasble for real-tme deployment based on ts stable maxmum process capablty.

23 Frames Processed Per Second Total Processed Frames Appl. Sc. 2017, 7, of Tmestamp(Second) Frames Processed Total Processed Frames Average Performance Fgure 8. Maxmum process capablty test. 6. Conclusons In ths paper, we proposed a shadow slhouette-based skeleton extracton (SSSE) method. SSSE extracts three-dmensonal human skeleton based on the human shadow nformaton on the ground. Specfcally, the proposed SSSE method comprses the followng: (1) A block matrx-based projecton transformaton s proposed, allowng the reconstructon of precse shadow slhouette nformaton from human shadow captured by monocular camera. (2) A slhouette shadow-based human skeleton extracton method s proposed. The proposed SSSE method extracts 3D postons of seven major jonts n the human skeleton based on the reconstructed human shadow slhouette nformaton and lght source poston. (3) A temporal spatal ntegraton algorthm for dscrete shadow slhouette nformaton s proposed, empowerng the SSSE-based human skeletonzaton n sngle lght source scenaro. As shown n Table 2, compared wth the tradtonal RGB-D human skeletonzaton method and other mono-rgb method, the proposed SSSE method has the followng advantages: (1) The SSSE method can be deployed n large-scale outdoor scenaros where tradtonal 3D human skeletonzaton algorthms are not effectve. (2) the SSSE method s capable of extractng human skeleton from frames shot by any normal monocular camera. (3) The SSSE method can be deployed n stretchng the effectve range of tradtonal RGB-D skeletonzaton method n the fuson applcaton. Table 2. Comparson between external sensor nformaton-based quadcopter montorng methods. Methods Devce Effectve Output Jont Requrement Range R e Format Numbers SSSE Sngle RGB Camera 7.0 m <R e < 10 m Human Skeleton 7 Tradtonal RGB-D Method [20] RGB-D Camera 0.8 m < R e < 3.5 m Human Skeleton 20 SSSE and RGB-D Fuson RGB-D Camera 0.8 m <R e < 10 m Human Skeleton 7 to 20 Jafar s RGB-D method [16] RGB-D Camera Not Avalable (N/A) Human Voxel 0 Yang s mono-rgb method [6] Multple RGB Cameras N/A Partal Voxels 0 For tradtonal outdoor survellance systems, the lmted 8-Bt color depth n the analogy transmsson system restrcts the precson of depth nformaton. Based on the proposed SSSE method, precse 3D human skeleton actvtes can be extracted at any montorng termnal. The extracted 3D human skeleton actvtes wll enrch the nformaton for survellance vdeo analyses, empowerng convenent 3D scenaro reproducton. Because of the smplcty n devce requrement and the compatblty wth the tradtonal survellance network, the proposed SSSE s an deal upgrade soluton for a tradtonal survellance system wthout extra hardware expendture.

24 Appl. Sc. 2017, 7, of 25 In concluson, SSSE offers an extra choce for 3D human skeletonzaton other than depth camera, wearable sensors, or llumnator array, layng down a mlestone to deploy n-lab human skeleton-related methods [6,16,20] n outdoor scenaros wth normal photographc devces. Based on the unque outdoor merts provded by SSSE, we wll focus our future research on applcatons of SSSE on outdoor survellance and unmanned aeral vehcle navgaton. Acknowledgments: We apprecate the detaled comments and knd suggestons provded by revewers. Ths research was supported by the Natonal Natural Scence Foundaton of Chna under Grants No and Author Contrbutons: All authors contrbuted to the research work. Je Hou conceved the new SSSE method and desgned the experments. Baolong Guo and Wangpeng He revewed the research work. Jnfu Wu partcpated n the experments. Conflcts of Interest: The authors declare no conflct of nterest. References 1. Zhang, Z. Mcrosoft knect sensor and ts effect. IEEE Multmeda 2012, 19, Xa, L.; Aggarwal, J. Spato temporal depth cubod smlarty feature for actvty recognton usng depth camera. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, Portland, OR, USA, June 2013; pp Zhang, X.; Gao, Y. Heterogeneous specular and dffuse 3-D surface approxmaton for face recognton across pose. IEEE Trans. Inf. Forenscs Secur. 2012, 7, Zhang, Y.; Mu, Z.; Yuan, L.; Zeng, H.; Chen, L. 3D Ear Normalzaton and Recognton Based on Local Surface Varaton. Appl. Sc. 2017, 7, Lay, Y.L.; Yang, H.J.; Ln, C.S.; Chen, W.Y. 3D face recognton by shadow moré. Opt. Laser Technol. 2012, 44, Yang, T.; Zhang, Y.; L, M.; Shao, D.; Zhang, X. A mult-camera network system for markerless 3d human body voxel reconstructon. In Proceedngs of the 2009 IEEE Ffth Internatonal Conference on Image and Graphcs, X an, Chna, September 2009; pp Chen, L.C.; Hoang, D.C.; Ln, H.I.; Nguyen, T.H. Innovatve methodology for mult-vew pont cloud regstraton n robotc 3D object scannng and reconstructon. Appl. Sc. 2016, 6, Gouaa, R.; Meuner, J. 3D reconstructon by fusonng shadow and slhouette nformaton. In Proceedngs of the IEEE 2014 Canadan Conference on Computer and Robot Vson (CRV), Montreal, QC, Canada, 6 9 May 2014; pp Wang, Y.; Huang, K.; Tan, T. Human actvty recognton based on r transform. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, Mnneapols, MN, USA, June 2007; pp Chen, C.C.; Aggarwal, J. Recognzng human acton from a far feld of vew. In Proceedngs of the 2009 IEEE Workshop on Moton and Vdeo Computng, Snowbrd, UT, USA, 8 9 December 2009; pp Jn, X.; Km, J. A 3D Skeletonzaton Algorthm for 3D Mesh Models Usng a Partal Parallel 3D Thnnng Algorthm and 3D Skeleton Correctng Algorthm. Appl. Sc. 2017, 7, Shotton, J.; Grshck, R.; Ftzgbbon, A.; Sharp, T.; Cook, M.; Fnoccho, M.; Moore, R.; Kohl, P.; Crmns, A.; Kpman, A.; et al. Effcent human pose estmaton from sngle depth mages. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, Shotton, J.; Sharp, T.; Kpman, A.; Ftzgbbon, A.; Fnoccho, M.; Blake, A.; Cook, M.; Moore, R. Real-tme human pose recognton n parts from sngle depth mages. Commun. ACM 2013, 56, Song, Y.; Lu, S.; Tang, J. Descrbng trajectory of surface patch for human acton recognton on RGB and depth vdeos. IEEE Sgnal Proc. Lett. 2015, 22, Han, J.; Shao, L.; Xu, D.; Shotton, J. Enhanced computer vson wth mcrosoft knect sensor: A revew. IEEE Trans. Cybern. 2013, 43, Jafar, O.H.; Mtzel, D.; Lebe, B. Real-tme RGB-D based people detecton and trackng for moble robots and head-worn cameras. In Proceedngs of the 2014 IEEE Internatonal Conference on Robotcs and Automaton (ICRA), Hong Kong, Chna, 31 May 7 June 2014; pp Juang, C.F.; Chang, C.M.; Wu, J.R.; Lee, D. Computer vson-based human body segmentaton and posture estmaton. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 2009, 39,

25 Appl. Sc. 2017, 7, of Hseh, J.W.; Hsu, Y.T.; Lao, H.Y.M.; Chen, C.C. Vdeo-based human movement analyss and ts applcaton to survellance systems. IEEE Trans. Multmeda 2008, 10, Yuan, X.; Yang, X. A robust human acton recognton system usng sngle camera. In Proceedngs of the 2009 Internatonal Conference on Computatonal Intellgence and Software Engneerng, Wuhan, Chna, December 2009; pp Hu, M.C.; Chen, C.W.; Cheng, W.H.; Chang, C.H.; La, J.H.; Wu, J.L. Real-tme human movement retreval and assessment wth Knect sensor. IEEE Trans. Cybern. 2015, 45, c 2017 by the authors. Lcensee MDPI, Basel, Swtzerland. Ths artcle s an open access artcle dstrbuted under the terms and condtons of the Creatve Commons Attrbuton (CC BY) lcense (