Adaptive Silhouette Extraction In Dynamic Environments Using Fuzzy Logic. Xi Chen, Zhihai He, James M. Keller, Derek Anderson, and Marjorie Skubic

Transcription

1 2006 IEEE Internatonal Conference on Fuzzy Systems Sheraton Vancouver Wall Centre Hotel, Vancouver, BC, Canada July 16-21, 2006 Adaptve Slhouette Extracton In Dynamc Envronments Usng Fuzzy Logc X Chen, Zhha He, James M. Keller, Derek Anderson, and Marjore Skubc Department of Electrcal and Computer Engneerng Unversty of Mssour, Columba, MO 65203, USA Abstract - Extractng a human slhouette from an mage s the enablng step for many hgh-level vson processng tasks, such as human trackng and actvty analyss. Although there are a number of slhouette extracton algorthms proposed n the lterature, most approaches work effcently only n constraned envronments where the background s relatvely smple and statc. In a prevous paper, we addressed some of the challenges n slhouette extracton and human trackng n a real-world unconstraned envronment where the background s complex and dynamc. We extracted features from mage regons, accumulated the feature nformaton over tme, fused hgh-level knowledge wth low-level features, and bult a tme-varyng background model. A problem wth our system s that by adaptng the background model, objects moved by a human are dffcult to handle. In order to rensert them nto the background, we run the rsk of cuttng off part of the human slhouette, such as n a quck arm movement. In ths paper, we develop a fuzzy logc nference system to detach the slhouette of a movng object from the human body. Our expermental results demonstrate that the fuzzy nference system s very effcent and robust. 1. INTRODUCTION Slhouette extracton, namely, segmentng the human body from the background, s often the frst step and also the enablng step for many hgh-level vson analyss tasks, such as human trackng and actvty recognton. One central task n slhouette extracton s background modelng. Once a background model s establshed, those mage regons wth sgnfcantly dfferent characterstcs from the background are consdered foreground objects. In [1], an LMeds (Least Medan of Squares) method s used to construct the background model. The dfferencng functon proposed n [2] s used to extract movng human slhouettes. In [3], a non-parametrc kernel densty estmaton method s employed to model and subtract the background. The adaptve background updatng n that approach s able to deal wth small moton n the background scene, such as movng tree branches. In [4], a slhouette extracton approach s developed n YUV color space that s able to detect shadows and segment the human body from the background. Although there are a number of algorthms developed n the lterature for slhouette extracton and human trackng, most of them have targeted and work effcently only n controlled envronments, where the lghtng condton s good, and the background s smple and nearly statc. In ths work, as a part of our project to develop automated vdeobased actvty analyss for eldercare, we propose an effcent and robust slhouette extracton and human trackng algorthm n a real-world, unconstraned envronment. More specfcally, we wll use vdeo cameras to collect nformaton about elderly resdents daly actvty at home. From these vdeos, we extract mportant actvty nformaton to perform automated functonal assessment and detect abnormal events, such as people fallng on the floor. To desgn an effcent vdeo actvty montorng system for eldercare, there are two major ssues we need to consder. The frst one s prvacy protecton. Our approach s to extract the human body from the background and fll the correspondng mage regon wth whte pxels so as to block the dentfyng features. The second ssue s actvty analyss. Our approach s to extract feature nformaton from the human slhouette and develop statstcal models, such as Hdden Markov Models, to model and recognze actvtes. In [5], we proposed our baselne archtecture to generate human slhouettes n dynamc temporal envronments. Whle the results are excellent for general human moton, nteracton of humans wth objects n the scene stll pose problems. In ths paper, we address those problems through the use of fuzzy logc. 1.1 Major Challenges In real-world envronments, such as n-home eldercare montorng, an effcent and robust slhouette extracton and human trackng algorthm has to deal wth the followng challenges: (1) the lghtng condton s tme-varyng. The change n lghtng could be caused by changng sunlght, passng vehcles outsde the apartment, ndoor lghts turned on or off, adjustment of blnds, etc. (2) the background s dynamc. Objects n the background, such as furntures and applances are beng moved by resdents. (3) slhouette extracton should be accurate and robust. Ths s because the /06/$20.00/ 2006 IEEE 236 1

2 subsequent vson analyss tasks, such as actvty modelng and abnormal event detecton, are totally based on the feature nformaton extracted from the slhouette. Ths mples that the extracted slhouette should be closely conformed to the human body. For example, a smple foreground segmentaton algorthm may nclude a char beng moved by the person as part of the slhouette. Ths wll create a potental problem because the actvty analyss module wll detect a sudden change n the slhouette, from a vertcal slm human body to horzontally stretched slhouette (human + char), and may report ths change as a abnormal event. Accordng to our study, ths s one of the major sources of false alarms n vdeo-based actvty montorng. 1.2 The Baselne Approach We formulate the slhouette extracton as an adaptve classfcaton problem. Hgh-level knowledge s fused wth low-level feature-based classfcaton to handle a tmevaryng background, and a decson process based on a fuzzy logcal nference system s used to detach the movng objects from the human slhouette. Our expermental results demonstrate that the proposed slhouette extracton and human trackng scheme works effcently and robustly. The rest of the paper s organzed as follows. Secton 2 gves an overvew of the baselne system descrbed n [5]. Secton 3 descrbes feature extracton and foregroundbackground segmentaton. In Secton 4, we develop an algorthm for adaptve background update. The decson process based on fuzzy logc nference system for detachng movng objects s presented n Secton 5. Secton 6 concludes the paper. 2. OVERVIEW OF THE ALGORITHM We assume that the camera s fxed. We formulate slhouette extracton as an adaptve classfcaton problem. The nput vdeo frame s parttoned nto small blocks, for example, 4x4 blocks. For each block, a classfcaton decson s made: the block belongs to the human slhouette or not. To ths end, we extract nvarant features from the mage blocks, as llustrated n Fg. 1. Based on these features, we buld a statstcal model for the background and classfy the mage blocks nto two categores: foreground and background. Because the background s tme-varyng, the background model and the classfer should be adaptve. However, background adaptve s also rsky. For example, f a person sts stll or sleeps n the couch for a long tme, say hours, the adaptve background model wll consder the person part of the background. Ths s not acceptable because of unprotected prvacy. To solve ths problem, we fuse the hgh-level knowledge obtaned from human trackng wth the low-level feature-based classfcaton so as to gude the background update. At ths stage, the foreground may stll contan objects other than the human slhouette, such as objects beng moved by or attached to the person. To address ths ssue, we develop a decson process based on a fuzzy logc nference system to detach these objects from the human slhouette. In the followng sectons, we explan the proposed slhouette extracton scheme n detal. 3. FEATURE EXTRACTION We extract features n the RGB color space. Two feature varables: chromatcty and brghtness dstortons are used to classfy the foreground and background [6]. The color model used here separates the brghtness from the chromatcty components. Let E be the color vector for pxel n the RGB space and let I be the observed color vector. Brghtness dstorton α s defned as follows: 2 α = arg mn I α E (1) α The chromatcty dstorton s defned by CD = I α E (2) The foreground and background classfcaton s based on the followng observaton: mage blocks n the background should have lttle change n ther color dstorton. The brghtness dstorton s very helpful n detectng shadows. In the proposed adaptve background modelng and classfcaton scheme, we use the mage data n the past frames to compute the jont dstrbuton of theses to buld a background model. Based on these two features and background model, we classfy the mage block nto foreground or background. Fg. 2 shows the results of segmentaton usng ths feature extracton and classfcaton scheme. Fg.1: Block dagram of the proposed system for slhouette extracton and human trackng 4. KNOWLEDGE-BASED ADAPTIVE BACKGROUND UPDATE It should be noted that the value of the sze of the shftng wndow should be approprately chosen. If t s too small, the background s updated very fast; the human body could be easly updated as background. Ths mples that the tme duraton of the shftng wndow should be qute large

3 However, f the tme duraton s too large, the background s updated very slowly. If a new object s ntroduced nto the background or a background object s moved, before the background s updated, ths object wll be classfed nto the foreground and hence become part of the slhouette. To solve ths problem, we utlze hgh-level knowledge about human moton to gude the adaptve update of the background model. slhouette n frame n separately. We use the followng lnear predcton model V L n+ = = 0 1 av (3) n to predct Vn+ 1 for the human body n frame n + 1. Vn+1 then defnes a boundng box, whch should contan the human body f the predcton s accurate. Fg. 3 shows the human slhouette extracton wth and wthout human movement trackng and adaptve background update. (a) (a) (b) (b) Fg.2: (a) Orgnal mage, (b) Slhouette wthout shadow suppresson and Result wth shadow suppresson Our basc dea s to track the human body and predct ts regon n the scene. The mage blocks whch contan the human body should be updated very slowly such that the human body would not be updated as background. Those blocks outsde the body regon can be updated much faster to make sure that the new objects are quckly absorbed nto the background. The predcton of human poston s denoted by V n = ( xn, yn, wn, hn ) whch defne a bondng box of the human body n frame n. x, y ) and ( n n ( w, h n n ) denote the centrod and dmenson of human body Fg.3: Background-foreground segmentaton of orgnal mage (a) wthout (b) and wth human movement predcton and background update. 5. DETACHING THE MOVING OBJECTS USING FUZZY LOGIC INFERENCE SYSTEM The man contrbuton of ths paper s a methodology to detach objects beng moved by the person from the human slhouette. Detachng objects from the human slhouette s a challengng task because of the lack of automated reasonng about what s, and s not, a human beng, as evdenced n a 238 3

4 vdeo sequence. Certanly, sophstcated human recognton and dentfcaton algorthms could be used. However, these algorthms are often computatonally ntensve and not robust n a dynamc envronment lke a home. Therefore, n ths work, we propose a smple yet effcent algorthm for a object detachment. Our scheme s based on a fuzzy logc nference system [7], whch fuses multple sources of nformaton together for decson makng. In effect, we wll create a fuzzy slhouette that we wll harden by dsplayng an approprate α-level cut. Suppose we are workng on frame n and the human slhouette n frame n 1 has been correctly extracted, denoted by O n 1. Let the foreground mage regon n frame n be O n, whch mght contan the human body and movng objects. Our fuzzy logc nference system s based on the followng observatons: (1) If an mage block no n belongs to the human body, t should have a hgh possblty of fndng a good match O n n 1. We use the SAD (sum of absolute dfference) to measure the goodness of matchng. (2) If a lot of blocks n ts neghborhood have good matches no n 1, t s a hgh possblty that ths block also belongs to the human body. (3) If ths block s far from the predcted poston of the human body centrod, as descrbed n Secton 4, t s a low possblty that ths block belongs to the human body. Based on these observatons, we extract the followng feature varables from each block no n : SAD (Sum of Absolute Dfference) n moton matchng. For each block n O n, we fnd ts best match n frame n 1. The SAD between ths block and ts best match form the frst feature varable. The number of neghborng blocks havng a good match n the human body. The dstance between the new block and the predcted human body centrod. To determne the membershp functons of these feature varables, we created some ground truth for human slhouettes usng several tranng vdeo sequences and found ther dstrbutons, shown n Fg. 4. Fg.4: Dstrbuton of three feature varables based on manually obtaned ground truth, (a), (b) and for correct classfcaton, (d) (e) and (f) for ms-classfcaton. A set of membershp functons were defned from these dstrbutons. The followng 12 rules are used n our fuzzy logc nference system, and s llustrated n Fg. 5 n a MATLAB mplementaton [8]. 1. If SAD s very low AND Neghborhood s huge AND Dstance s close, THEN Slhouette s hgh. 2. If SAD s large AND Neghborhood s small AND Dstance s very far, THEN Slhouette s low. 3. If SAD s low AND Neghborhood s large AND Dstance s medum, THEN Slhouette s hgh. 4. If SAD s medum AND Neghborhood s medum AND Dstance s medum, THEN Slhouette s medum. 5. If SAD s large AND Neghborhood s medum AND Dstance s medum, THEN Slhouette s medum. 6. If SAD s large AND Neghborhood s large AND Dstance s close, THEN Slhouette s hgh. 7. If SAD s medum AND Neghborhood s large AND Dstance s medum, THEN Slhouette s hgh. 8. If SAD s medum AND Neghborhood s large AND Dstance s close, THEN Slhouette s hgh. 9. If SAD s very large AND Neghborhood s small AND Dstance s far, THEN Slhouette s low. 10. If SAD s medum AND Neghborhood s small AND Dstance s very far, THEN Slhouette s low. 11. If SAD s low AND Neghborhood s huge AND Dstance s medum, THEN Slhouette s hgh. 12. If SAD s low AND Neghborhood s medum AND Dstance s medum, THEN Slhouette s hgh

5 detect that the book s not part of the human body. However, the fuzzy logc processng cleanly, clearly a superor result. (a) Fg.5: Membershp functons of 12 rules for the fuzzy logc nference system. We observe that usng the fuzzy nference system to detach movng objects wll erode the human body slhouette due to ms-classfcaton, as shown n Fg. 6. We use morphologcal operatons and neghborhood nformaton to repar these mssng parts Expermental Results The followng expermental results on several sequences show that the proposed fuzzy nference system for movng object detachment not only preserves the advantage of the low-level feature-based slhouette extracton algorthm, but also robustly detaches the movng object from the human body. As llustrated n Fgs. 7 to 10, column (a) contans the orgnal mages; column (b) shows the results of featurebased slhouette extracton,.e., wth no fuzzy logc processng, and column dsplays the results after movng object detachment usng the fuzzy logc nference system. Here, the whte pxels correspond to blocks n the 0.5 level cut of the fuzzy slhouette. Fg. 7 shows the results of the sequence Rasng arms. Ths sequence does not contan any movng object attached to human body. It s dsplayed to show that our temporal fuzzy logc processng stll mantans an accurate slhouette n the baselne scenaro. Columns (b) and are very smlar n ths sequence, as they should be. Fg. 8 contans the results for the sequence Rasng a book n whch a book s attached to and moved by the person n the scene. Ths s a complex problem snce the book s small. Sngle frame slhouette detecton n our standard system cannot (b) Fg.6: Degradaton of the slhouette due to msclassfcaton: (a) the orgnal frame; (b) the slhouette results before detachng algorthm; the fuzzy slhouette. The dark gray n the edge of the slhouette s the degradaton due to msclassfcaton. In Fg. 9, a sequence called Fallng down n whch the human has dramatc moton change s dsplayed. These results also verfy the robustness of the proposed detachng algorthm. The fuzzy system even has the added beneft of removng some of the dark shadows that our color-based 240 5

6 algorthm fals to detach. However, some of the hardened slhouettes appear slghtly eroded. Snce the goal s to perform actvty analyss, the eroded slhouettes should not cause serous problems, although we are nvestgatng better algorthms to fll out the fuzzy shape. Fg. 10 contans the results on one frame of the sequence Movng char n whch the human grabs a much bgger object. Whle the fuzzy logc system performs better than the correspondng crsp one, a good porton of the human slhouette s lost. As the sequence progresses, ths error s compounded. Hence, we need more research on technques to rebuld the dsplayed crsp slhouette from the fuzzy slhouette and methods to reacqure a good slhouette occasonally durng the sequence. Snce we wll process long sequences of actvty, a reasonng module should be able to detect when the human s statonary for a short tme. Then, we beleve that we can reset the golden standard slhouette. 6. CONCLUSIONS AND FUTURE WORK In ths paper, we proposed a accurate and robust slhouette extracton scheme for a dynamc envronment. We bult a background model and extract classfcaton features n RGB color space. To deal wth the challenges of slhouette extracton n dynamc envronment, we fused hgh-level knowledge and low-level features and developed a fuzzy logc nference system workng on three feature varables: SAD, Neghborhood and Dstance to the centrod of the human body to produce a fuzzy slhouette that was subsequently hardened by vewng an approprate level cut. The results on several sequences show that ths algorthm s effcent and robust for the dynamc envronment wth new objects n t. However, from results n Fg. 9 and Fg. 10, we can see that the smple morphologcal cleanng operaton s not suffcent to completely recover the degradaton for some sequences, especally those wth greater moton and much larger objects n the scene. We are currently workng on makng the predcton more accurate and creatng a scheme to recover mssng slhouette parts usng known results from feature-based classfcaton. Also, we ntend to study the mpact of the accuracy of slhouette results on the performance of future actvty modelng and analyss. ACKNOWLEDGEMENTS The authors are grateful for the support from NSF ITR grant IIS and the U.S. Admnstraton on Agng, under grant 90AM3013. REFERENCES [1] L. Wang, T. Tan, H. Nng and W. Hu, Slhouette Analyss-Based Gat Recognton for Human Identfcaton, IEEE Trans. Pattern Analyss and Machne Intellgence,, pp , December [2] Y. Kuno, T. Watanabe, Y. Shmosakoda, and S. Nakagawa, Automated Detecton of Human for Vsual Survellance System, Poc. Int l Conf. Pattern Recognton, pp , [3] A. Elgammal, D. Harwood, L. Davs, Non-parametrc Model for Background Subtracton, 6th European Conference on Computer Vson. Dubln, Ireland, June/July [4] O. Schreer, I. Feldmann, U. Golz and P. Kauff, Fast and Robust Shadow Detecton n Vdeoconference Applcatons, 4 th EURASIP-IEEE Regon 8 nternatonal Symposum on Vdeo/Image Processng and Multmeda Communcatons, pp , June [5] X. Chen, Z. He, D. Anderson, J. Keller, and M. Skubc, Adaptve Slhouette Extracton and Human Trackng n Complex and Dynamc Envronments, Proceedngs, Internatonal Conference on Image Processng, Atlanta, October, [6] T. Horprasert, D. Harwood, L. Davs, A Statstcal Approach for Real-tme Robust Background Subtracton and Shadow Detecton, Proc. IEEE ICCV'99 FRAME- RATE Workshop, Kerkyra, Greece, September [7] George J. Klr, Bo Yuan, Fuzzy Sets and Fuzzy Logc, Prentce Hall, [8] Fuzzy Logc Toolbox User s Gude, The Math Works, Inc

7 Fg. 7: The sequence Rasng arms ; (a) the orgnal color mages n the sequence, (b) the sngle frame slhouette extracton, hardened fuzzy slhouette extracton usng temporal nformaton. Fg. 8: The sequence Rasng a book ; (a) the orgnal color mages n the sequence, (b) the sngle frame slhouette extracton, hardened fuzzy slhouette extracton usng temporal nformaton

8 (a) Fg.10: One frame from the Movng char sequence; (a) the orgnal color mage, (b) the sngle frame slhouette extracton, hardened fuzzy slhouette extracton usng temporal nformaton. Fg. 9: The sequence fallng down ; (a) the orgnal color mages n the sequence, (b) the sngle frame slhouette extracton, hardened fuzzy slhouette extracton usng temporal nformaton