Convolutional Neural Network- based Human Recognition for Vision Occupancy Sensors

Transcription

1 10 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV'18 Convolutonal Neural Network- based Human Recognton for Vson Occupancy Sensors Seung Soo Lee and Manbae Km * Dept. of Computer and Communcatons Engneerng, Kangwon Natonal Unversty Chunchon, Republc of Korea E-mal: {rugh, * manbae}@kangwon.ac.kr Abstract In buldng and households, occupancy moton sensors are nstalled where the lghts are turned on/off accordng to the detecton of movng objects by the occupancy sensors. One of dsadvantages s that PIR sensor cannot detect the statonary person due to ts functonalty of detectng the varaton of thermal temperature. To solve ths problem, the utlzaton of camera vson sensors has ganed nterests, where object trackng s used for detectng the statonary persons. However, the object trackng has an nherent problem such as trackng drft. Therefore, the recognton of humans for statc trackers s an mportant task. In ths paper, we propose a CNN-based human recognton to determne whether a statc tracker contans humans. Expermental results valdate that human(s) and non-humans (background) are classfed wth accuracy of about 85% and that the proposed method can be ncorporated nto practcal vson occupancy sensors. varatons. Even people can also be statonary for short-term and long-term duratons. Therefore, the classfcaton of drftng trackers becomes an mportant ssue. Fg. 1 shows the boundng boxes of statc trackers. Statc trackers are marked n a blue square. A boundng box n Fg. 1(a) ndcates a statc tracker wthout humans. Therefore, the box tracker needs to be removed. Other real example s shown n Fg 1(b). We observe four statc trackers, here a tracker n the bottom-rght contans a human, but other three trackers have background mage. Therefore, the latter are subject to the elmnaton from trackng process. Keywords: Vson sensor, occupancy sensor, convolutonal neural network, human classfcaton 1 Introducton Most occupancy sensors nstalled n buldngs, households and so forth are pyroelectrc nfra-red (PIR) sensors [1-6]. PIRs operate by the detecton of thermal temperature of humans. They detect the change of thermal temperatures. Snce PIRs detect the varaton of the temperature, they have three demerts. 1) It s mpossble to detect statonary people, 2) hot arflow mght cause the malfuncton of the sensors, and 3) the classfcaton of humans, cats, dogs s not easy a task. Recently, on-gong works utlzng vson sensors have been ntroduced to replace PIRs wth camera sensors [3, 10]. The use of the camera sensors could not only overcome the nherent problems of the PIRs, but also addtonal nformaton such as people countng, trackng, human actvtes, and ntellgent survellance can be obtaned. Generally, the vson sensors carry out object trackng for the statonary people detecton, whch can be accomplshed only by the camera sensors. In other words, trackng can detect statc humans. However, a trackng drft s a man problem that happens due to mage color smlarty, the llumnaton changes and so forth. If the drft occurs, most drftng trackers wll move to any fxed locatons because background has no color (a) Fg. 1. The detected statonary boundng boxes marked n blue. The movng boxes are marked n red. Ths paper presents CNN-based statonary people detecton. Usng ths neural network, we ntend to determne whether a statc tracker contans human or background, thereby mprovng the performance of occupancy sensors. A convolutonal neural network (CNN) s comprsed of one or more convolutonal layers (often wth a subsamplng step) and then followed by one or more fully connected layers as n a standard multlayer neural network. The archtecture of a CNN s desgned to take advantage of the 2D structure of an nput mage. Ths s acheved wth local connectons and ted weghts followed by some form of poolng whch results n translaton nvarant features. Another beneft of CNNs s that they are easer to tran and have many fewer parameters than fully connected networks wth the same number of hdden unts. Applyng CNN classfcaton to all the trackers requres hgh computatonal complexty. Movng trackers have hgh probablty of contanng humans. Therefore, we lmt the applcaton of CNN classfcaton only to statc trackers, thereby reducng the processng tme. The trackng performance s beyond the scope of ths paper. The purpose s (b)

2 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV'18 11 the recognton of human and non-human or background for each statc tracker. Ths paper s organzed as follows; Next secton presents the overall methodology of ths papers. In Secton 3, object trackng method s ntroduced. CNN-based human classfcaton s presented n Secton 4. Expermental results are descrbed n Secton 5 followed by concluson and future works. 2 Proposed Method t T MHI (2) MAX (, MHI ) otherwse where denotes the number of frames n whch a person s acton contnues. s a postve decay parameter. If < T, then MHI approaches, otherwse t s decreased by. MHI ncreases rapdly at a large and drastcally drops at a large. As well, the MHI of a pxel whose moton has recently occurred wll quckly approach. MHI 0 s 0. Fg. 3 show the subsequent mages wth thers MHIs. Fg. 3. Subsequent mages and ther MHIs Fg. 2. Flow dagram of the proposed method. The overall structure of the proposed method s shown n Fg. 2. Object trackng s needed for examnng occupancy for statonary people. The basc trackng method s based on moton hstory mage (MHI) for real-tme. Snce trackers repeat dynamc and statc motons, CNN-based classfcaton s carred out only for statc trackers. If the CNN classfer decdes any trackers as a background tracker, the tracker s automatcally removed. Pxel-wse search requres hgh computatonal tme. In order to reduce the tme, we decompose a trackng search wndow nto 8x8 blocks. (Fg. 4(b)). Then MHI energy E k s computed for each block. k s the block number. Fg. 4(c) shows the close-up of MHI energy n Fg. 4(d). E = ( ) (3) 3 MHI based Trackng Ths secton ntroduces a trackng method used n ths paper. Even though the am of our work s the decson of human occupancy, a trackng s needed to acqure statonary trackers that could contan human or background. For each occupant, a search wndow s put on a tracker. An overhead camera s nstalled on the top of the space. The heght of the celng s 3m. Snce humans could move fast, ths can result n trackng drft. In other words, the tracker n the next frame has hgh probablty of beng located outsde the search wndow. The MHI proposed by Bovck [7] s manly used n the recognton of human actvtes as well as depth generaton [8]. A basc defnton of MHI s as follows: A dfferental mage s derved from the current and prevous mages by = (1) where s a pxel ndex and t s a frame number. Then, MHI s computed by Fg. 4. Savng the sum of all pxel MHIs nto a 8x8 block after decomposton of a search wndow We search for a block wth a maxmum energy of one current block and eght neghborng blocks, whose locaton becomes a trackng pont TP n of a tracker n. The determnaton of a statonary tracker s easly made from the energy varaton. The CNN network that classfes the human or background type of the objects takes as nput wth ths statonary tracker. Fg. 5 show the resultng trackng mages n the upper two rows and ther assocated MHIs n the bottom two rows. Two trackers accompany a sngle person. Red box s a moton tracker and blue box ndcates a statonary tracker.

3 12 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV'18 occupancy sensors. Fg. 8 shows the baselne CNN model Fg. 5. Resultng trackng mages obtaned by the MHI energy-based trackng method 4 Proposed Human Recognton based on CNN Model The am of the CNN model s object classfcaton. We explan the procedure of the classfcaton. The number of object classes are fxed to be 5 and 2. The labels and ther assocated objects n 5-label classfcaton (L5-CNN) are descrbed n Table 1. L 1 contans floor and wall that have lttle changes of ntenstes. Chars are categorzed nto L 2. L 3 contans desk, whteboard wth vertcal boundary. Bookshelf, computer, boxes are L 4. L 1 L 4 are background. Humans belong to L 5. In 2-label classfcaton CNN (L2-CNN), bnary labels (0,1) are used where L 1 L 4 classfcaton are grouped nto L 1 and humans become L 2 (Fg. 7). The human and background objects are dvded nto only two classes, whch reduces the tranng tme. Further, the usage of fve labels can delver more nterestng and dverse nformaton. Fgs. 6 and 7 show examples of label objects. Table 1. Labels and objects for 5-label classfcaton CNN (L5-CNN) Label Object L 1 floor, wall L 2 char L 3 desk, whteboard L 4 bookshelf, computer, box human L 5 For practcal applcatons, the complexty of a vson occupancy sensor needs to be compatble or less compared wth PIR sensors. Therefore, to satsfy ths requrement, we frstly desgn the most smple neural network model whch s composed of a sngle convolutonal layer and a sngle fullyconnected layer. Then we wll compare the performance wth the well-known SVM (support vector machne). The performance needs to satsfy a mnmum requrement of Fg. 6. Tranng mages belongng to [L 1, L 5] n L5-CNN Fg. 7. Tranng mages belongng to [L 1, L 2] n L2-CNN Fg. 8. Network structure of 5-class classfcaton (L5-CNN). The nput to the network s a 64x64 grayscale mage, then the number of nput nodes s 4,096. The number of convolutonal layers s at [1, 4] and the number of fullyconnected layers vares at [1, 2]. The actvaton functon s ReLU [9] and 2x2 max poolng s used. The fnal classfcaton s carred out by Softmax. Xaver weght ntalzaton [10] s used and stochastc gradent descent (SGD) s employed for network backpropagaton.. Table 2. Parameter values used n neural networks Layer Actvaton functon nput layer 64 x 64 node convoluton layer 3 x 3 x 15 ReLU poolng layer 2 x 2 max poolng hdden layer 50 ReLU output layer 5 Softmax

4 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV' Expermental Results convolutonal and hdden layers. The number of flters starts at 16 and ncreases n double. Zero paddng s used to mantan the same sze of nput and output. The number of hdden nodes n the fully-connected layers s 50 for one hdden layer and for two hdden layers. In the experment, we used 80% tranng data and 20% test data. The data was randomly shuffled. Table 3 show the accuracy for L5-CNN. 4-1 CNN shows the accuracy of 62.8%. The smplest model, 1-1 CNN has the accuracy of 50%. Snce 5-class classfcaton s not satsfactory, we tested a bnary classfcaton, L2-CNN that determnes only human or background. Table 3. Accuracy measured by no. of convoluton flters and hdden layers n L5-CNN. n s the number of convoluton layers and l s the number of hdden layers n n-l CNN. Fg. 9. Images acqured n the lab from the overhead camera The test mages were acqured n our lab. Fg. 8 shows mages captured n the lab. The overhead camera s vertcally located on the top of the celng. Image resoluton s 720x480 wth RGB channels. The dstance between the ground and the celng s 2.7m. The codes are wrtten n Matlab and C/C++. Human classfcaton requres the labellng process. For nonhuman mages, 64x64 mage was captured at no-human occupancy. 500 mages per each label were produced and 5,000 test mages are avalable. To acqure human mages, multple persons repeated ext/entrance n the lab. Then, human mages were manually captured. As well, whle our camera sensor s operatng, mages are captured when any trackers become statonary. Snce ths mage mght contan human or background, we manually separated them nto ts assocated label category. The purpose of the classfcaton s to decde whether the mage patch contans human(s). The mages assgned a label s used for tranng. At no-occupancy states, non-human mages were captured. Several persons are walkng around the lab and the human magers were captured by the vson sensor. The archtecture of the network s summarzed n Table 3. Our network s composed of l convoluton layers, n fully connected layers. Each network s denoted by l-n CNN throughout ths paper, where l {1, 2, 3, 4} represents the number of convoluton layers and n {1, 2} denotes the number of hdden layers. The frst convolutonal layer takes 64x4 nput mages wth 16 kernels of sze 5x5x3. Rectfed Lnear Unt (RELU) neuron s used as an actvaton functon for each convolutonal layer. y max( 0, x) (4) where x and y are the nput and output values, respectvely. Ths functon can reduce the vanshng gradent problem [13] that mght occur when a sgmod or hyperbolc tangent functon s adopted n back-propagaton tran and has a faster processng speed than a non-lnear actvaton functon. Ths s sutable to our vson occupancy sensor. The learnng rate s We changed the number of CNN type conv layer hd layer Accuracy 1-1 CNN % CNN % 2-1 CNN % CNN % 3-1 CNN % CNN % 4-1 CNN % CNN % Table 4. Accuracy measured by no. of convoluton flters and hdden layers n L2-CNN. n s the number of convoluton layers and l s the number of hdden layers for n-l CNN. CNN type conv layer hdden layer Accuracy 1-1 CNN % CNN % 2-1 CNN % CNN % 3-1 CNN % CNN % 4-1 CNN % CNN % For comparatve performance valdaton, SVM [12] (Support Vector Machne) s compared wth our CNN models. The pxels of the estmated regons of nterest (ROIs) are made nto feature vectors, and human classfcaton s performed va a support vector machne (SVM). In [14], hstograms of orented gradents (HOGs) from the ROIs are

5 14 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV'18 acqured from each mage. The humans and background were dfferentated va the SVM. To make a far comparson, we used the same tranng data set. SVM has shown excellent performance n object recognton. SVM s mplemented n L2-SVM for two classes and L5-SVM for fve classes. The performance s shown n Table 5. The accuracy of L5- SVM s 51.6% and slghtly better than 1-1 CNN and 1-3 CNN by 1.6, 8.0%. 2-1 CNN ~4-2 CNN outperform the SVN by a maxmum of 9.0%. Therefore, for 5-class classfcaton, we need to use at least two convolutonal layers to outperform the SVM. Ths observaton ndcates that any CNN model can be substantally better on the human recognton than conventonal machne learnng methods. In the comparatve classfcaton accuracy of Table 5, n bnary classfcaton, L2-SVM shows the accuracy of 60.5%. In L2-CNN, even the smplest model, 1-1 CNN shows the accuracy of 83.17%. As the number of convolutonal layers ncrements, a maxmum of 88.17% s acheved. The expermental results shows that 1-1 CNN can outperform SVM. It was mentoned n the ntroducton that the camera vson senor needs to be compatble wth PIRs to have compettve cost n terms of practcal applcatons. The expermental results valdated ths asserton. Table 5. Classfcaton accuracy of the proposed CNN wth SVM for L2 and L5. 5 Labels 2 Labels Classfer Accuracy Classfer Accuracy L5-SVM 51.60% L2-SVM 60.50% 1-1 CNN 50.00% 1-1 CNN 83.17% 1-2 CNN 43.60% 1-2 CNN 84.67% 2-1 CNN 52.80% 2-1 CNN 83.00% 2-2 CNN 56.80% 2-2 CNN 83.00% 3-1 CNN 60.00% 3-1 CNN 87.17% 3-2 CNN 50.40% 3-2 CNN 88.17% 4-1 CNN 62.80% 4-1 CNN 86.00% 4-2 CNN 55.60% 4-2 CNN 86.67% Fnally, the msclassfed mage are shown n Fg. 10. In L5-CNN, bookshelf s msclassfed as floor, computer as a char, human as a char. In L2-CNN, three dfferent humans are classfed as other background objects. Ths msclassfcaton needs a further research. In practce, such background objects can be elmnated usng the stop duraton tme [11]. Then, combnng ths emprcal decson wth our CNN network s expected to acheve satsfactory vson occupancy sensor. (b) Fg. 10. Images wth classfcaton error and ther labels. (a) L5-CNN and (b) L2-CNN 6 Concluson In ths paper, we proposed a vson occupancy sensor that s expected to replace current PIR moton sensors. Trakng s one of mporant functonaltes that can track statonary people. PIRs cannot detect statc objects. The trackng drft s unavodable n the system and thus requres the removal decson of drftng trackers. To solve ths, CNN-based people recognton has been presented n ths paper. In L5-CNN, at least 2 convolutonal layers are needed to outperform SVM. In L2-CNN, a sngle convolutonal layer and one hdden layer outperforms SVM by a large margn. The expermental results valdate that vson occupancy sensor could replace conventonal PIRs n terms of nstallaton and operatng cost. 7 Acknowledgement Ths research was supported by Basc Scence Research Program through the Natonal Research Foundaton of Korea (NRF) funded by the Mnstry of Educaton (No. 2017R1D1A3B ). 8 References [1] P. Lu et al. Occupancy nference usng pyroelectrc nfrared sensors through hdden Markov model, IEEE Sensors Journal, 16(4), Feb [2] F. Wahl, M. Mlenkovc, and O. Amft, A dstrbuted PIRbased approach for estmatng people count n offce envronments, IEEE Conf. on Computatonal Scence and Engneerng, [3] Y. Benezeth et al. Towards a sensor for detectng human presence and characterzng actvty, Energy and Buldngs, 43, [4] J. Han and B. Bhanu, Fuson of color and nfrared vdeo for movng human detecton, Pattern Recognton, 40, [5] S. Nakashma, Y. KItazono, L. Zhang, and S. Serkawa. Development of prvacy-preservng sensor for person

6 Int'l Conf. IP, Comp. Vson, and Pattern Recognton IPCV'18 15 detecton, Proceda, 2, [6] I. Amn, A. Taylor, F. Junejom, A. Al-Hababeh, and R. Parkn, Automated people-countng by usng lowresoluton nfrared and vsual cameras, Measurement, 41, [7] A. Bobck and J. Davs, "The recognton of human movement usng temporal templates," IEEE Trans. Pattern Recognton and Pattern Analyss, Vol 23, No. 3, Mar [8] J. Gl and M. Km, Moton depth generaton usng MHI for 2D-to-3D converson, Electroncs Letters, Vol. 53, No. 23, pp , Nov [9] A. L. Maas, A. Y. Hannun, A. Y. Ng, Rectfer Nonlneartes Improve Neural Network Acoustc Models, Proc. of the 30th Int. Conf. on Machne Learnng, Atlanta, USA, June [10] X. Glorot and Y. Bengo, Understandng the dffculty of tranng deep forward neural networks, Int Conf. Artfcal Intellgence and Statstcs, Socety for Artfcal Intellgence and Statstcs, [11] J. Gl and M. Km, Real-tme People Occupancy Detecton by Camera Vson Sensor, Journal of Broadcast Engneerng, Vol. 22, No. 6, Nov [12] C-C. Chang and C-J. Ln, LIBSVM: A lbrary for support vector machnes, ACM Tran. Intellgent Systems and Technology, Vol. 2, No. 3, pp. 27:1-27:27, 2011 [13] X. Glorot, A. Bodes, and Y. Bengo, Deep sparse rectfer neural networks, Int. Conf. Artfcal Intellgence and Statstcs, Apr