Improved TLD Algorithm for Face Tracking

Transcription

1 Absrac Improved TLD Algorihm for Face Tracking Huimin Li a, Chaojing Yu b and Jing Chen c Chongqing Universiy of Poss and Telecommunicaions, Chongqing , China a li.huimin666@163.com, b @163.com, c @qq.com For he Tracking-Learning-Deecion (TLD) for dynamic face racking has he problem of racking drif when he face is occluded seriously, his paper proposed an improved mehod based on TLD. Afer locaing he face successfully, inroducing he mehod of occlusion window o deermine he degree of face occlusion. When he face is occluded slighly, using he original learner o solve i. If he face is occluded seriously, adding he direcion predicor of Markov model in TLD deecor o reduce he deecion range and enhance he discriminaion abiliy for similar faces. A he same ime, adding Kalman filer in TLD racker o esimae he area in curren frame where he face may exis, which could reduce he number of scanning grids and improve he processing speed. The experimens show ha he improved TLD algorihm has a srong robusness in differen environmens, can rack he dynamic face quickly and accuraely, and also improve he problem of racking drif when he face is occluded seriously Keywords Tracking-Learning-Deecion (TLD), Markov model, Kalman filer, face racking. 1. Inroducion The problems of he increasing aging populaions and children safey cusody have become serious, so he uelar home-service robos have received more and more aenion hese years [1]. Adding he echnology of face racking in home-service robo o realize securiy monioring and human-compuer ineracion is an imporan research direcion in service robo field. Tracking-Learning-Deecion (TLD) is firs proposed by Zdenek kalal in 2010 as single arge long erm racking algorihm [2]. TLD has good performance in real-ime and sabiliy under normal circumsances so ha i araced exensive aenion in relaed fields. There were some improved algorihms in he basis of TLD. Dong Yongkun [3] had made some improvemens o he TLD deecor, using Hisogram of Oriened Gridien (HOG) and incremenal classifier based on Normalized Cross Correlaion mehod (NCC) replaced he TLD deecor o deec he arge. Cheng e al. [4] has added Random Sample Consensus (RANSAC) in he TLD racker o esimae he global moion model and improved he successful rae of racking. This paper proposed an improved face-racking algorihm based on TLD o solve he racking drif problem which caused by serious occlusion. A lo of experimens show ha he improved algorihm proposed in his paper can solve he above problems and rack he face accuraely and quickly. 2. TLD algorihm TLD is a racking algorihm which is made up primarily of racker, deecor and learner. Fig. 1 is he framework of TLD. According o he principle of TLD, supposing he arge is visible beween wo adjacen frames so we can esimae he arge s moion sae by he informaion of moion. The learner updaes he racker and he deecor consanly hrough P-N learning mechanism, and assesses he deecion error and updae deecor s arge model according o he racking resul [5]. The deecor searches each frame in oal graph o locae he arge, which is obained according he previous learning and deecion. Significanly, he deecor and racker run a he same ime o deermine he 35

2 locaion of he arge. To verify he accuracy of he arge informaion, he daa obained from deecor and racker is fed back o he learner [6-7]. Posiion of objec Tracking Video sequences Updae Learning Updae Combinaion Posiion of objec Deecion Fig.1 The framework of original TLD algorihm 3. Tracking face based on improved TLD When he face is occluded, i is easy o have racking drif problem for original TLD algorihm. So an improved mehod based on TLD would be inroduced o solve his problem. To deermine he degree of occlusion of he face, Occlusion Window is proposed. And hen Markov predicion model [8-9] and Kalman filer [10] are added in TLD deecor and racker o deal wih differen occlusion siuaions. The operaion procedure of he improved TLD racking algorihm is exhibied in Fig. 2. Sar Deecing face in he firs frame by Adaboos Iniializing TLD, Kalman filer and Markov predicion model Tracking he face by TLD racker; deecing he face by TLD deecor Yes Tracking failed? Yes Occlusion serious? Yes No Occlusion? Predicing he direcion of he moving face by Markov model No No Updae he learner of TLD Esimaing he face locaion by Kalman filer Displaying he face locaion Video is over? No Yes Qui Fig.2 he flowchar of racking face based on improved TLD 36

3 3.1 The deerminaion of he occlusion degree of he face The seing mehod of Occlusion Window is showed in he Fig. 3, which represen he siuaions ha objecs occluding face ener he window form horizonal and verical direcion. In he wo picures, yellow bold recangles are racking windows, and blue serif recangles are Occlusion Windows inroduced in his paper. The red lines in he middle of Occlusion Windows divide windows ino wo symmerical pars of he A and A`. Fig.3 The seings of he occlusion window While he occluded objecs enering from differen direcions and occluding he faces, color hisograms will be calculaed corresponding he upper-down pars. And he normalizaion resuls are marked, which represens he hisograms of racking windows and Occlusion Windows. We suppose ha he occluded objecs ener Occlusion Window from he A side in he ener racking window in he h frame o sar o occlude he face. The similariy of he hisograms of A and A` s racking window and Occlusion Windows can be calculaed a he by using Bhaacharyya coefficien. The calculaion formulas are showed as (1), (2), (3). TR (, ) R(, ) 1 ( k, T k k (1) ) T( k, ) 37 1h frame and 1h and h frame TR 2 ( k, ) T( k, ) R( k, 1) (2) TR 3 ( k, ) T( k, ) T( k, 1) (3) As shown above, i corresponds A or A` par when k equals -1 or 1. When k equals -1, TR 1 ( k, ) denoes he similariy of he hisogram of A s racking window and occlusion window a ; denoes TR 2 ( k, ) he similariy of he hisogram of A s racking window and occlusion window a and denoes he similariy of he hisograms of A s racking window a ime and A s racking window a TR k 1. 3 (, ) 1. Correspondingly, he racking window and Occlusion Window represen he par of A when k equals 1. (4) TR1( k, ) TR2( k, ) TR 3 ( k, ) ( ) (5) We define when Eq. (4) is saisfied only, he face is occluded lighly; when Eq. (4) and Eq. (5) are boh saisfied, he face is occluded badly. In order o improve he robusness of he occlusion judgmen, we will repea a coninuous 10 frames o deermine wheher i mea hese formulas. 3.2 The predicion of face-moion direcion by Markov model I is possible o appear some similar faces in he arge area and overlapped each oher on he move. In his siuaion, i is hard for he original TLD o rack he correc face because of is poor resoluion wih similar faces. To solve his problem, we add Markov model in he TLD deecor o esimae he moion direcion of he face. The face moion is decomposed ino verical direcion and horizonal direcion. Take verical direcion for example, we define 1 represens lef movemen and -1 represen righ movemen.

4 According o he direcion predicor of Markov model, we suppose he moion sae of he face in curren frame only have relaion wih he sae ransiion marix and he moion sae in he previous frame. The sae ransiion marix a ime can be obained from he Eq. (6) s p( s 1 1 s 1) p( s 1 1 s 1) P p( s 1 1 s 1) p( s 1 1 s 1) denoes he moion sae of he face a ime. By means of he sae ransiion marix and moion sae of he face a ime, he moion sae of he face a ime +1 can be esimaed by Eq. (7): p( s 1) p( s 1) P 1 p( s 1 1) p( s 1) In he formula, represens he probabiliy of he face moving lef a ime and represens he probabiliy of he face moving righ a ime. A ime, if he face move lef, ps ( 1) 1. Else he resul equals 0. A ime, ps ( 1) 0.5 if he face dose no move. The sae ransiion marix can be calculaed by he Eq. (8) according o he face moion sae. Where, n 1, 1 ( n 1 ( ps ( 1) n n p( s -1 s -1) p( s -1 s 1) P n n p( s 1 s -1) p( s 1 s 1) -1,-1-1,1 1 1 n-1 n1 1,-1 1,1 1 1 n-1 n1 (6) (7) ps ( 1) n 1 ) denoes he number of he frames while he face moving lef (righ) during 0~ ime. n 1,1 ) represens he number of he frames while he face moving lef (righ) boh in he curren and he previous frame during 0~ ime. n 1, 1 ( (8) n 1,1 ) represen he number of he frames while he face moves lef (righ) in curren frame bu moving righ (lef) in previous frame. In his paper, he difference of he cener posiion of he face in he verical beween adjacen frames is calculaed o esimae he face s moving direcion. On he basis of he Markov model, we can esimae he direcion of he face moving in he curren frame. According o he acual posiion in he las frame, he deecion area of he face in he curren frame can be obained. And hose faces which appear in his deeced area mee he condiions of Markov model. So we can exclude he inerference and overlapping problem of oher faces. Fig. 4 represens ha Markov prediced he movemen direcion of face. In his picure, he posiion of he face in he previous frame is represened by he yellow recangle. And by he use of he Markov model, we can forecas he face has a endency o move o he righ in he curren frame. So on he righ side of he red hread is he par which is sen o he TLD deecor. Conversely he lef par is no longer sen o he TLD deecor. So we can narrow he deecion area. Fig.4 Markov prediced he movemen direcion of face 38

5 3.3 The esimaion o he face locaion by Kalman filer When he sae of he face saisfies Eq. (4), we deermine i as sligh occlusion and send i o he TLD learner. When i saisfies boh Eq. (4) and Eq. (5), we deermine i as serious occlusion and add Kalman filer in TLD racker o predic he posiion of he face in he curren frame. Wih hese operaions, he search area can be narrowed, and he processing ime can be depressed. The cenral posiion of he face in each frame is obained o consruc he Kalman filer's observaion marix and sae marix, which are showed in Eq. (9) and Eq. (10): In hese formulas, c x, c y k x y T O c c (9) k x y x y T S c c s s (10) denoe he coordinae componens of he face cener in horizonal and verical posiions. s x, suppose he face moving a a consan velociy beween wo adjacen frames because he adjacen frames are very closed. And is sae ransiion marix and observaion marix are showed as Eq. (11): s y denoe he speed of he moving face in horizonal and verical direcion. We A k,k-1, Hk Where, denoes inerval ime beween wo adjacen frames. The original TLD racker needs o scan all he sub-windows in he image which may conain human faces o deermine he posiion of he face. I is no efficien for he original TLD because i has no limi he search scope, which caused he sub scanning grids oo much o reduce efficiency. To improve his problem, he Kalman filer is added ino TLD racker o predic he possible posiion of he face in curren frame, reduce he number of scanning grids. Seps of he improved TLD proposed in his paper are elaboraed as follows: Sep 1 Predic he cenral posiion of he face in curren frame by using Kalman filer; Sep 2 A big recangle, cenering on his posiion, whose raio of lengh o widh is same wih he face window in previous frame and is area is 6 imes as much as he face window in previous frame, is defined as Search Scope(S-Scope). And S-Scope is he area where he face may appear in curren frame. Sep 3 Send he sub-windows o TLD racker, which are inerseced wih S-Scope, o deermine if he face exiss. Fig. 5 represens he scanning grids are reduced by he use of Kalman filer. In his picure, he blue hick recangular frame is he area which is obained by Kalman filer and he face may appear in. We will send he recangle 1, 3 inerseced or conained by he blue recangle o TLD deecor. As opposed o recangle 2, 4, 5. So we can reduce he number of scanning grids of TLD. (11) Fig.5 Kalman reduces he number of child windows 39

6 4. The experimen resuls and analysis 4.1 Robusness es In order o verify he robusness and effeciveness of he proposed algorihm in differen experimenal environmen, 3 mehods are used o compare he proposed algorihm, including he original TLD algorihm, he TLD algorihm joining he Markov direcion predicor only(m-tld), he TLD algorihm joining he Kalman filer only(k-tld), he improve TLD algorihm proposed in his paper (Imp-TLD). The experimens are finished in 4 differen environmens as shown as Fig. 6. Label 1 represens normal illuminaion wih simple background. Label 2 shows poor illuminaion wih simple background. Label 3 shows normal illuminaion wih complex background. And label 4 shows poor illuminaion wih complex background. As shown below, able 1 denoes he number of frames conaining faces in 4 cases and able 2 denoes he experimenal by using 4 differen algorihms in he four differen environmens. Fig.6 Experimenal environmens Table 1. The frames of he video images (frames) Environmen Toal frames Face is occluded or disappear Normal face Table 2. Tes resuls of he video Number The frames racking Correcly Frame rae (frames/s) TLD M-TLD K-TLD Imp-TLD TLD K-TLD M-TLD Imp-TLD From able 2 we can see ha he resuls of he 4 algorihms are similar in he normal illuminaion and simple background environmen wihou occlusion. They all can rack he face accuraely. Comparaively, he original TLD has he problem of racking drif in he normal or poor illuminaion environmen wih complex background. Under he same environmen, M-TLD and K-TLD s successful racking frames are less han he original TLD because of he reducion of search sub-windows. As for he frame rae, M-TLD and K-TLD have a higher processing speed han he original TLD. Imp-TLD proposed in his paper no only enhances he judgmen o similar faces bu also narrows he search area by inroducing Occlusion Window, Markov predicion model and Kalman filer. So Imp-TLD has beer performance han he oher hree algorihms synheically. Through he analysis and comparison of he experimenal daa, we can prove ha Imp-TLD is srongly robus o differen environmens, and i can rack o face accuraely and fas in he same ime. 40

7 4.2 Occlusion es In his paper, we have conduced occlusion es. Imp-TLD and he original TLD are performed under he complex environmen wih objec inerference o. Fig. 7 denoes he es resul of he original TLD, and Fig. 8 denoes he resul of Imp-TLD. In hese figures, (a),(b),(c),(d) represen he face in he siuaions of no occlusion, sligh occlusion, serious occlusion and he face reappearance, respecively. In he repeaed 40 experimens, he resuls are basically consisen wih Fig. 7 and Fig. 8. As he resuls indicae, when he face is slighly occluded, boh he original TLD and Imp-TLD can work successfully because he slighly occluded par is added o original TLD learner as a posiive sample. However when he face is occluded seriously, he original TLD is very likely o rack he face failed due o he model drifing. On he conrary, because of Markov predicion model added in he TLD deecor and Kalman filer added o he TLD racker. Imp-TLD has beer performance in he siuaion ha he face is occluded seriously. (a) No occlusion (b) Sligh occlusion (c) Serious occlusion (d) he face reappearance Fig.7 Tracking resuls of original TLD algorihm (a) No occlusion (b) Sligh occlusion (c) Serious occlusion (d) he face reappearance Fig.8 Tracking resuls of improved TLD algorihm 5. Conclusion To solve he problem ha i is likely o rack he face failed when he face is occluded seriously for original TLD, a improved TLD algorihm is proposed for racking face in his paper. Afer locaed he face, he degree of occlusion is evaluaed by he Occlusion Window which is inroduced in his paper. If he face is slighly occluded, he original TLD processes sill. Bu if he face is occluded seriously, he direcion predicor of Markov model is added o narrow search area and srenghen he resoluion o similar faces. In he same ime, Kalman filer is added in he TLD racker o esimae he face s posiion in he nex frame and reduce he running ime. Through a lo of experimens, i is proved ha Imp-TLD can rack he face accuraely and quickly. A he same ime, Imp-TLD has good performance o deal wih he racking drif problem when he face is occluded seriously. Acknowledgemens This work was suppored by he Chongqing Science and Technology Commission (CSTC) (No. csc2015jcyjbx0066). References [1] Xu F, Zhang X, Du Z. Home service robo indusry developmen presen siuaion invesigaion repor of our counry [J]. Robo Technique and Applicaion, 2009, 02:14-19(in Chinese) 41

8 [2] Kalal Z, Maas J, Mikolajczyk K. P-N learning: boosrapping binary classifiers by srucural consrains[c]//proceedings of IEEE Conference on Compuer Vision and Paern Recogniion. New York: IEEE Press, 2010: [3] Dong Y, Wang Chun-Xi, Xue L, e al. Pedesrian deecion and racking based on TLD framework [J]. Journal of Huazhong Universiy of Science and Technology(Naural Science Ediion), 2013,S1: (in Chinese) [4] Cheng S, Liu G W, Sun J X. Robus and fas Tracking-Learning-Deecion. In: Proceedings of Inernaional Conference On Compuer Science and Inelligen Communicaion (CSIC). Zhengzhou, china: ACSR, [5] Zhou X, Qian Q, Ye Y, Wang Cong-Qing, Improved TLD visual arge racking algorihm [J]. Journal of Image and Graphics, 2013,09: [6] Chen L Y, Zhang D, Zhao S Y, e al. An Algorihm of Visual Tracking Based on Trackinglearning-deecion [J]. Science Technology and Engineering, 2013, 9: (in Chinese) [7] Gopalan R, Hong T, Shneier M, e al. A learning approach owards deecion and racking of lane markings [J]. IEEE ransacions on inelligen ransporaion sysems, 2012, 13(3): [8] Lamberi R, Sepier F, Salman N, e al. Sequenial Markov Chain Mone Carlo for muli-arge racking wih correlaed RSS measuremens [C], 2015 IEEE 10h Inernaional Conference on Inelligen Sensors, April 7-9, 2015, Singapore, 2015:1-6 [9] Liu Y L, Yu Y X. An Effecive Mehod o Formulae Sae Transiion Probabiliy Marix of Markov Model of Large-Scale Sysem[J]. Journal of Tianjin Universiy: Naural Science and Engineering Technology, 2013, 9: (in Chinese) [10] Jiang C, Zhang Y. Reduced-order Kalman filering for sae consrained linear sysems [J]. Journal of Sysems Engineering and Elecronics, 2013, 4: