Classfer Swarms for Human Detecton n Infrared Imagery Yur Owechko, Swarup Medasan, and Narayan Srnvasa HRL Laboratores, LLC 3011 Malbu Canyon Road, Malbu, CA 90265 {owechko, smedasan, nsrnvasa}@hrl.com Abstract- In ths paper, we descrbe a new method for vsual recognton of objects n an mage that combnes feature-based object classfcaton wth effcent search mechansms based on swarm ntellgence. Our approach utlzes the partcle swarm optmzaton algorthm (PSO), a populaton based evolutonary algorthm, whch s effectve for optmzaton of a wde range of functons. PSO searches a mult-dmensonal soluton space for a global optmum usng a populaton of partcles n whch each partcle has ts own velocty vector. In our approach, we extend PSO usng sequental nchng methods to handle multple mnma. Also, n our approach, each partcle n the swarm s actually a self-contaned classfer that flys through the soluton space seekng the most object-lke regons. By performng ths optmzaton, the classfer swarm smultaneously fnds objects n the scene, determnes ther sze, and optmzes the classfer parameters. I. INTRODUCTION We descrbe a new method for vsual recognton of objects n a scene whch combnes feature-based object classfcaton wth effcent search mechansms based on swarm ntellgence. Objects n a vsual scene need to be located and classfed so they can be tracked effectvely for automotve safety, survellance, permeter protecton, and a varety of other government, mltary, and commercal applcatons. Typcally, classfcaton of objects n an mage s performed usng features extracted from an analyss wndow that s scanned across the mage. Ths brute force search can be very computatonally ntensve, especally f a small wndow s used snce a classfcaton must be performed at each wndow poston. Conventonal approaches to reducng the computatonal load are based on reducng the search space by usng another sensor such as a scannng radar to cue the vson system and measure the range of the object. Lmtatons of the radar approach nclude hgh cost, false alarms, the need to assocate radar tracks wth vsual objects, and overall system complexty. Alternatvely, prevous vson-only approaches have utlzed moton-based segmentaton usng background estmaton methods to reduce the search space by generatng areas of nterest (AOI) around movng objects and/or usng stereo vson to estmate range n order to reduce searchng n scale. These methods add cost and complexty by requrng addtonal cameras and computatons. Moton-based segmentaton s also problematc under challengng lghtng condtons or f background moton exsts as s the case for movng host platforms. We descrbe a novel search mechansm that can effcently fnd multple nstances of multple object classes n a scene wthout the need for cueng sensors or scan-based searchng. Our approach utlzes the partcle swarm optmzaton (PSO) algorthm [1,2], a populaton based evolutonary algorthm, whch s effectve for optmzaton of a wde range of functons. The algorthm models the exploraton of mult-dmensonal soluton space by a populaton of ndvduals where the success of each ndvdual has an nfluence on the dynamcs of other members of the swarm. Basc PSO has proved effectve n explorng complcated ftness landscapes and convergng populatons of partcles to a sngle global optmum, although t has been shown that the basc PSO s not guaranteed to converge to a local or global optmum. However, some optmzaton problems requre the dentfcaton of global as well as local mnma n a multmodal framework. We extend the PSO algorthm usng sequental nchng methods [3,4] to enable t to locate multple objects n the scene. Our approach s substantally dfferent from the prevous work n that each partcle from the populaton s a unque classfer. As the populaton swarms around, the classfers adjust parameters to best detect the objects n the scene. Our method also dffers from other vson algorthms that use swarm ntellgence n that the other methods use swarms to buld up features usng ant colony pheromonebased deas. In our method swarmng s done at the classfer level n a space consstng of object locaton, scale, and classfer parameter dmensons and where each partcle s a complete classfer. The partcles swarm n ths space n order to fnd the local optma that correspond to objects n the mage. The classfer detals are not vsble at the abstracton level of the swarm. To our knowledge, there are no exstng methods that usng partcle swarms to mplement object detecton systems. However, there have been attempts to use Genetc Algorthms (GAs) and Evolutonary Algorthms for object detecton [5]. In [5], the authors employ GA to detect and verfy faces from mages encodng only the poston of the face. Object scale s handled by scalng the nput mage. Dstances to egen spaces are used as the ftness functons. Genetc algorthms have been used before for decreasng the search space n vson systems [6]. These systems employ a populaton of ndvdual solutons that crossover and mutate n an effort to maxmze the ftness functon. Other efforts have used GAs for tranng and adaptng neural networks to recognze objects [7]. The paper s organzed as follows. In Secton II, we brefly ntroduce the PSO algorthm. Detals on usng classfer swarms for object detecton are presented n Secton III. The Sequental Nchng-based PSO (SNPSO) s ntroduced n detal n Secton IV. In Secton V, we
present a varety of results on the human detecton applcaton. Fnally, we summarze our conclusons n Secton VI. II. PARTICLE SWARM OPTIMIZATION PSO s a relatvely smple optmzaton method that has ts roots n artfcal lfe n general, and to brd flockng and swarmng theory n partcular [1,2]. Conceptually, t ncludes aspects of genetc algorthms and evolutonary programmng. Each potental soluton s assgned a randomzed velocty vector and the potental solutons called partcles then fly through the space n search of the functon optma. Each partcle keeps track of ts coordnates n mult-dmensonal space that are assocated wth the best soluton (pbest) t has observed so far. A global best parameter (gbest) s used to store the best locaton among all partcles. The velocty of each partcle s then changed towards pbest and gbest n a probablstc way accordng to v ( t) = wv + c 2 ( t 1) + c * rand() * ( gbest x 1 * rand() * ( pbest x ( t 1)) ( t 1)) (1) x ( t) = x ( t 1) + v ( t) (2) Where x (t) and (t) are the poston and velocty vectors at tme t of the -th partcle and c 1 and c 2 are parameters that weght the nfluence of ther respectve terms n the velocty update equaton. w s a decay constant whch allows the swarm to converge to a soluton more quckly. The rand() functon generates a random number between 0 and 1 wth a unform dstrbuton. The above dynamcs reflect a soco-psychologcal model where ndvdual partcles change ther belefs n accordance wth a combnaton of ther own experence and the best experence of the group. (Ths s n contrast to other models of cognton where an ndvdual changes hs belefs to become more consstent wth hs own experence only.) The random element ntroduces a source of nose, whch enables an ntal random search of the soluton space. The search then becomes more drected after a few teratons as the swarm starts to concentrate on more favorable regons. Ths type of search s much more effcent than scannng or gradent based search methods. It s smlar to genetc algorthms n that t can be used for dscontnuous and nosy soluton spaces snce t only requres an evaluaton of the functon to be optmzed at each partcle poston. No gradent nformaton s used. Unlke GAs, the PSO partcles are not modfed at each teraton, they just travel to a dfferent poston, calculate the soluton at that poston, and compare t wth ther own and global best value n order to update ther velocty vectors. PSO reles on the fact that n most practcal v problems the optmum soluton usually has better than average solutons resdng n a volume around t. These good solutons tend to attract the partcles to the regon where the optmum les. The swarm becomes more and more concentrated on lkely regons untl the optmum s found, e.g. gbest no longer changes. PSO has been appled to a wde varety of optmzaton problems. It has been found expermentally that the number of partcles and teratons requred scale weakly wth the dmensonalty of the soluton space. The total number of functon evaluatons s very small compared to the sze of the soluton space, as wll be seen below. Basc PSO searches only for a sngle optmum n the soluton space, but varous approaches have been descrbed for fndng multple local optma or nches [8,9]. We now ntroduce classfer swarms and descrbe them n detal. III. CLASSIFIER SWARMS Objects n a vsual scene need to be located and classfed so they can be tracked effectvely for automotve safety, survellance, permeter protecton, and a varety of other government, mltary, and commercal applcatons. Typcally, classfcaton of objects n an mage s performed usng features extracted from an analyss wndow that s scanned across the mage. We propose an approach where a swarm of classfers moves around n the search space lookng for selected class of objects. One of the novel aspects of our approach s that two of the dmensons are used to locate objects n the mage, whle the rest of the dmensons are used to optmze the classfer parameters. Our approach s a much more effcent method for fndng objects n an mage compared to searchng based on scannng the mage or usng gradent nformaton, especally f the scale of the object s not known beforehand. We have measured speedup factors of over 1000 relatve to sequental scannng when searchng n three dmensons. The number of false alarms per mage s also greatly reduced, whch s very mportant for practcal applcatons. The speedup and false alarm advantages over sequental scannng ncrease as the number of dmensons s ncreased whch makes t feasble to nclude object rotaton angle as one of the search space dmensons. Ths approach wll help ncrease the range of applcatons for vson systems by mprovng performance, reducng computatonal requrements dramatcally, elmnatng the need for cueng sensors such as radar, and reducng overall cost of practcal systems. The basc archtecture for our approach s shown n Fg. 1. The objectve s to fnd multple nstances of an object class n an nput mage. The PSO partcles fly n a soluton space where two of the dmensons represent the x and y coordnates n the nput mage. The key concept n our approach s that each partcle n the PSO swarm s a self-contaned object classfer whch outputs a value
representng the classfcaton confdence that the mage dstrbuton n the analyss wndow assocated wth that partcle s or s not a member of the object class. All partcles are nstances of the same classfer and only the classfer parameters vary as the partcle vsts dfferent postons n the soluton space. As mentoned before, two of the soluton space dmensons represent the locaton of the analyss wndow on the nput mage. A thrd dmenson represents the sze or scale of the analyss wndow n order to match the unknown sze of objects n the mage. Addtonal dmensons can be used to represent other classfer parameters such as, for example, the rotaton angle of the object. One can magne a multdmensonal surface of classfer confdence (or salency map) that can be generated f the classfer s scanned across all of the dmensons. The salency map for an mage can be dscontnuous and nosy, wth many solated false alarms where the classfer responds ncorrectly to patterns n the mage. Thus gradent-based methods cannot be used to fnd objects n the mage, whch s why brute force scannng s usually used. By generatng salency maps for many mages, we have found expermentally that objects n the scene tend to have large cores of hgh confdence values. Many false alarms tend to be solated wth small cores. Snce the probablty of a partcle passng near or through a larger core s greater for a larger core, the partcles are attracted more to larger cores and the number of false alarms n a mage are reduced usng SNPSO compared to smple scannng. In smple scannng, all of the false alarms n an mage wll be detected so the classfer must be based towards very low false alarm rates n order to keep the overall false alarm rate low, whch also has the sde effect of reducng the detecton rate. A comparson of the computatonal requrements of PSO classfcaton compared to brute force scannng s gven n Table. 2. The speedup factor for PSO s SM 2 /PK where t s assumed that M scan postons are searched along x and y and S postons are searched along the scale dmenson. P s the number of partcles and K s the number of teratons. We have found for realstc mages that P=80 and K=10 work well for 3 dmensons. If M=S=100, then the speedup factor s 1250. The speedup advantage of PSO ncreases further wth the number of dmensons. The number of false alarms s also reduced relatve to brute force scannng. In our case, we developed a classfer to detect humans [8]. Gven a 2:1 aspect rato wndow n an mage, pre-selected Haar wavelet features were computed along wth a set of fuzzy edge-symmetry features. Ths combned 190D feature vector was then passed to a Non-lnear Dscrmnant Analyss (NDA) neural network for detectng the presence of a human n the selected wndow regon. Typcally, to fnd all humans n the scene, the wndow needs to be scanned across the entre mage. In conventonal approaches, an estmate of the object heght or depth from the camera needs to be known to determne the heght of the scannng wndow. In our approach, the heght of the classfer s also automatcally obtaned from the PPSO swarm dynamcs. The performance of the traned human detecton classfer on dfferent nfrared magery s presented n Fgures 2. We now descrbe the sequental nchng approach to classfer swarms that wll be used to fnd multple objects n the scene. III. SEQUENTIAL NICHING PARTICLE SWARM OPTIMIZATION (SNPSO) The flow chart for our sequental nchng method for fndng multple objects n a scene s shown n Fg. 1. After an nput mage s receved, a runnng lst of object postons n the scene s cleared as well as the boundary flag table descrbed below. The partcle swarm of classfers s then ntalzed n random postons n the soluton space. The boundares of the ntalzaton volume can be set usng varous crtera. For example, f the flat ground constrant s approprate, then we can utlze the fact that objects of a certan sze wll appear only n certan subregons of the mage to reduce the search volume. After ntalzaton, the swarm dynamcs are terated untl the global best (gbest) exceeds a preset threshold or the number of teratons reaches a preset maxmum value. If gbest does not exceed the threshold then t s assumed no targeted objects are present n the scene and the system wats for the next nput mage. If gbest does exceed the threshold, then a neghborhood check s performed around the gbest poston to see f a certan number of neghborng postons n the mage also exceed the threshold for that analyss wndow sze. If the gbest poston passes the neghborhood test, then gbest s added to the lst of object postons. The mage s then erased locally at the gbest poston wth a Gaussan whose wdth s proportonal to the analyss wndow sze. If gbest does not pass the neghborhood test, then the mage s erased at gbest wthout addng gbest to the object lst. The purpose of the local erasure step s to remove that object s nfluence on the swarm when the swarm s re-ntalzed to search for the next object. By erasng the mage locally, the nfluence of that regon on the swarm across all dmensons s elmnated. After the mage erasure step, a boundary flag table s updated wth the erasng Gaussan. Ths table has the same dmensons as the mage and s n one-to-one correspondence wth t. All table entres wthn the wdth of the erasng Gaussan are set ON. If a partcle lands on a locaton whose flag s ON, the partcle wll not run ts classfer snce t s already known that an object s present at that locaton. Instead, the partcle wll keep ts prevous pbest value. Ths elmnates unnecessary classfer evaluatons n regons where t has already been
determned that an object exsts whch speeds up the swarm, especally f large objects are present n the scene. We do not mplement any repellng forces at object locatons to avod solatng regons. After the Gaussan flag table s updated, the swarm s re-ntalzed to search for the next object. The boundary flag table s cleared when a new mage s acqured. Ths approach can be extended naturally to searchng for members of dfferent object classes n parallel. We smply run multple swarms for multple object classes. Each swarm mantans ts own gbest, partcle best (pbest), and object lsts. The swarms nteract ndrectly only through the erasng Gaussans and boundary table whch are common to all swarms. Thus f a swarm detects an object at a partcular locaton n the mage, the other swarms do not try to fnd other objects there. IV. RESULTS We have successfully mplemented the sequental nchng swarmng classfers method for detectng objects n a scene of unknown poston and sze. Each of the partcles represented a classfer based on Haar wavelet and fuzzy symmetry features and a backpropagaton neural network classfcaton engne. The demonstraton was done n Matlab wth the partcle classfers mplemented usng a C++ dynamc lnk lbrary called from Matlab. Due to the paucty of space we only show few relevant examples. The results are shown n Fgs. 3 to 5. The swarm was programmed to search for pedestrans n the x, y, and sze dmensons. In these examples we used nfrared mages, but the approach works equally well n the vsble. In fact, the classfer was traned usng vsble lght mages only. The frst test mage was a sngle person walkng outsde between a buldng and a fence wth hlls n the background. Fg. 3 shows the swarm after a sngle teraton. The 3D soluton space wth the current postons of the classfer partcles s shown n the upper left. At ths early stage the partcles are dstrbuted wdely n x, y, and scale. Partcles that exceed the classfcaton threshold but don t pass the neghborhood test are colored red, partcles that pass both crtera are colored green, and all other partcles are blue. The upper rght mage shows the partcle postons projected on the x-y plane of the nput mage, ncludng the effects of any erasng Gaussans. The analyss wndows correspondng to the green partcles are shown supermposed on the nput mage n the lower part of the fgure. After only a sngle teraton, the partcles are all blue and the pedestran has not yet been found. Fg. 4 shows the swarm after 10 teratons. The partcles are now centered on the pedestran wth a range of wndow szes. Only one partcle passes the classfer threshold and neghborhood tests. The correspondng erasng Gaussan and wndow are shown. It should be emphaszed that t s not necessary to wat untl the entre swarm condenses on an object before decdng an object s at that locaton. We label that locaton as an object as soon as a sngle partcle passes the classfcaton and neghborhood tests, at whch pont the mage s erased locally, the boundary table s updated, the swarm s re-ntalzed, and search starts for the next object n the scene. In Fgure 5, we show results for an nfrared mage wth multple pedestrans. After 13 teratons all 3 pedestrans were detected and t was determned that no more pedestrans were present. Note the 3 erasng Gaussans n the upper rght centered on the pedestrans. The same swarm parameters and classfer were used for both of these examples. The number of partcles was 80 and the maxmum number of teratons was lmted to 10. The total number of classfer evaluatons was 1280 for the sngle pedestran and 1840 for the mult-pedestran example. Ths s far less than the number of evaluatons necessary to search by scannng n x, y, and scale. Pedestrans are often detected wthn 2 to 3 teratons. Most of the processng s spent on determnng that no more pedestrans are present by propagatng the swarm untl the maxmum allowed number of teratons s reached. The effect of ths overhead can be reduced by usng the tme dmenson or f the number of objects s known a pror. When the tme dmenson s used, the swarm searches a vdeo cube consstng of a stack of mage snapshots taken at dscrete tme ntervals nstead of a sngle mage. Advantages of searchng n tme as well as space nclude automatcally fndng the space-tme paths of objects, further reducton n false alarms because consstent objects wll form extended salency tubes n space-tme, and the object sweep necessary for determnng f all objects have been found needs to be done only once for each mage stack rather than for each ndvdual mage. V. CONCLUSIONS In ths paper, we ntroduced the concept of classfer swarms for effectvely searchng an mage for multple objects of nterest. We frst extended the partcle swarm optmzaton algorthm usng sequental nchng methods to search for multple mnma. A human detecton classfer that uses Haar wavelet and edge-symmetry features was desgned. A human detecton classfer was used n place of the usually used partcle n the sequental nchng PSO. From the varety of results presented, we can see that the proposed approach s an effcent and effectve search mechansm. It s also shown to be very fast and can robustly detect multple objects n the scene. VI. REFERENCES [1] Kennedy, J., Eberhart, R. C., and Sh, Y., Swarm Intellgence San Francsco: Morgan Kaufmann Publshers, 2001. [2] R. C. Eberhart and Y. Sh, Partcle Swarm Optmzaton: Developments, Applcatons, and
Resources, 2001. [3] R. Brts, A. P. Engelbrecht, and F. van den Bergh, A Nchng Partcle Swarm Optmzer, 2002. [4] D. Beasley, D. R. Bull, and R. R. Martn, A Sequental Nchng Technque for Multmodal Functon Optmzaton, Evolutonary Computaton, 1(2), p. 101-125, 1993. [5] G. Bebs, S. Uthram, and M. Georgopoulos, Face Detecton and Verfcaton Usng Genetc Search, Internatonal Journal on Artfcal Intellgence Tools, vol. 9, no 2, pp. 225-246, 2000. [6] D. L. Swets, B. Punch, and J. Weng, Genetc Algorthms for Object Recognton n a complex scene, Proc. of Intl. Conference on Image Processng, vol. 2, Oct, pp 23-26, 1995. [7] V. Ceselsk and M. Zhang, Usng Genetc Algorthms to Improve the Accuracy of Object Detecton, In Proceedngs of the thrd Pacfc-Asa Knowledge Dscovery and Data Mnng Conference, Nng Zhong and Lzhu Zhou (Eds.), Knowledge Dscovery and Data Mnng -- Research and Practcal Experences. Tsnghua Unversty Press, page 19-24. Bejng, Chna, Aprl 26-31, 1999. [8] N. Srnvasa, S. Medasan, Y. Owechko, and D. Khosla, Fuzzy edge-symmetry features for mproved ntruder detecton, The 12th IEEE Internatonal Conference on Fuzzy Systems, Vol: 2, 920-925, 2003. Start Next nput mage Clear lst of object postons Intalze and run swarm untl gbest > threshold or no. of teratons = max allowed Object lst complete for current mage Fal gbest > threshold? Yes Neghborhood Test Pass Add gbest poston to object lst No Erase mage locally at gbest wth gaussan Update gaussan boundary flag table Fgure 1. Flow chart for sequental nchng PSO vson system.
Table. 1. Comparson of computatonal requrements of PSO vson system wth brute force scannng. Conventonal Scannng Partcle Swarm Wavelet Sze = NxN No. of Partcles = P Image Sze = MxM No. of Iteratons = K No. of Features =F No. of Features =F Operatons per Feature = 2N(N-2) Operatons per Feature = 2N(N-2) Total complexty per mage = M 2 *F*2*N(N-2) Total complexty per mage = P*K*F*2*N(N-2) (Assumes that complete mage s scanned to fnd (Note that P*K << M 2 ) objects of nterest, stepsze = 1 pxel) Fgure 2. Salency maps generated by human detecton classfer for the IR mages on the left. The salency maps show the classfer response at each spatal locaton n the IR mage. Black and whte ndcate the maxmum and mnmum classfer responses, respectvely.
Fgure. 3. Results for sequental nchng PSO vson system for sngle pedestran n the nfrared. Results are shown after a sngle teraton of the swarm. The object s not yet detected. Fg. 4. Results for sequental nchng PSO vson system for sngle pedestran n the nfrared. Results are shown after 10 teratons of the swarm. The object has been detected and t was determned that no other pedestrans are present n the mage. Fg. 5. Results for sequental nchng PSO vson system for multple pedestrans n the nfrared. Results are shown after 10 teratons of the swarm. All three objects have been detected and t was determned that no other pedestrans are present n the mage.