Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection

Transcription

1 Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection Rainer Lienart, Alexander Kuranov, Vadim Pisarevsky Microprocessor Researc Lab, Intel Labs Intel Corporation, Santa Clara, CA 95052, USA MRL Tecnical Report, May 2002, revised December 2002 ABSTRACT Recently Viola et al. ave introduced a rapid object detection sceme based on a boosted cascade of simple feature classifiers. In tis paper e introduce and empirically analysis to extensions to teir approac: Firstly, a novel set of rotated aar-like features is introduced. Tese novel features significantly enric te simple features of [6] and can also be calculated efficiently. Wit tese ne rotated features our sample face detector sos off on average a 10% loer false alarm rate at a given it rate. Secondly, e present a troug analysis of different boosting algoritms (namely Discrete, Real and Gentle Adaboost) and eak classifiers on te detection performance and computational complexity. We ill see tat Gentle Adaboost it small CART trees as base classifiers outperform Discrete Adaboost and stumps. Te complete object detection training and detection system as ell as a trained face detector are available in te Open Computer Vision Library at sourceforge.net [8]. 1 Introduction Recently Viola et al. ave proposed a multi-stage classification procedure tat reduces te processing time substantially ile acieving almost te same accuracy as compared to a muc sloer and more complex single stage classifier [6]. Tis paper extends teir rapid object detection frameork in to important ays: Firstly, teir basic and over-complete set of aar-like features is extended by an efficient set of 45 rotated features, ic add additional domain-knoledge to te learning frameork and ic is oterise ard to learn. Tese novel features can be computed rapidly at all scales in constant time. Secondly, e empirically so tat Gentle Adaboost outperforms (it respect to detection accuracy) Discrete and Real Adaboost for object detection tasks, ile aving a loer computational complexity, i.e., requiring a loer number of features for te same performance. Also, te usage of small decision trees instead of stumps as eak classifiers furter improves te detection performance at a comparable detection speed. Te complete training and detection system as ell as a trained face detector are available in te Open Computer Vision Library at ttp:/ sourceforge.net/projects/opencvlibrary/ [8]. 2 Features Te main purpose of using features instead of ra pixel values as te input to a learning algoritm is to reduce/increase te in-class/outof-class variability compared to te ra input data, and tus making classification easier. Features usually encode knoledge about te domain, ic is difficult to learn from a ra and finite set of input data. Te complexity of feature evaluation is also a very important aspect since almost all object detection algoritms slide a fixed-size indo at all scales over te input image. As e ill see, our features can be computed at any position and any scale in te same constant time. At most 8 table lookups are needed per feature. 2.1 Feature Pool Our feature pool as inspired by te over-complete aar-like features used by Papageorgiou et al. in [5,4] and teir very fast computation sceme proposed by Viola et al. in [6], and is a generalization of teir ork. Let us assume tat te basic unit for testing for te presence of an object is a indo of W H pixels. Also assume tat e ave a very fast ay of computing te sum of pixels of any uprigt and 45 rotated rectangle inside te indo. A rectangle is specified by te tuple r= ( xyα,,,, ) it 0 xx, W, 0 yy, H, xy, 0,, > 0, α { 0, 45 }, and its pixel sum is denoted by RecSum() r. To examples of suc rectangles are given in Figure 1. W Windo uprigt rectangle Figure 1: 45 rotated rectangle Example of an uprigt and 45 rotated rectangle. Our ra feature set is ten te set of all possible features of te form feature I = ω i RecSum( r i ), i I={ 1,, N} ere te eigts ω i R, te rectangles r i, and N are arbitrarily cosen. Tis ra feature set is (almost) infinitely large. For practical reasons, it is reduced as follos: 1. Only eigted combinations of pixel sums of to rectangles are considered (i.e., N= 2 ). 2. Te eigts ave opposite signs, and are used to compensate for te difference in area size beteen te to rectangles. Tus, for non-overlapping rectangles e ave H

2 0 Area( r 0 ) = 1 Area( r 1 ). Witout restrictions e can set 0 = 1 and get 1 = Area( r 0 ) Area( r 1 ). 3. Te features mimic aar-like features and early features of te uman visual patay suc as center-surround and directional responses. Tese restrictions lead us to te 14 feature prototypes son in Figure 2: Four edge features, Eigt line features, and To center-surround features. Tese prototypes are scaled independently in vertical and orizontal direction in order to generate a ric, over-complete set of features. Note tat te line features can be calculated by to rectangles only. Hereto it is assumed tat te first rectangle r 0 encompasses te black and ite rectangle and te second rectangle r 1 represents te black area. For instance, line feature (2a) it total eigt of 2 and idt of 6 at te top left corner (5,3) can be ritten as feature I = 1 RecSum( 53620,,,, ) 3 RecSum( 73220,,,, ). Only features (1a), (1b), (2a), (2c) and (4a) of Figure 2 ave been used by [4,5,6]. In our experiments te additional features significantly enanced te expressional poer of te learning system and consequently improved te performance of te object detection system. Tis is especially true if te object under detection exibit diagonal structures suc as it is te case for many brand logos. 1. Edge features XY W 1 z X H 1 z Y 1 2 it z=. Table 1 lists te number of features for a indo size of 24x24. Feature Type / X/Y # 1a ; 1b 2/1 ; 1/2 12/24 ; 24/12 43,200 1c ; 1d 2/1 ; 1/2 8/8 8,464 2a ; 2c 3/1 ; 1/3 8/24 ; 24/8 27,600 2b ; 2d 4/1 ; 1/4 6/24 ; 24/6 20,736 2e ; 2g 3/1 ; 1/3 6/6 4,356 2f ; 2 4/1 ; 1/4 4/4 3,600 3a 3/3 8/8 8,464 3b 3/3 3/3 1,521 Sum 117,941 Table 1: Number of features inside of a 24x24 indo for eac prototype. 2.2 Fast Feature Computation All our features can be computed very fast in constant time for any size by means of to auxiliary images. For uprigt rectangles te auxiliary image is te Summed Area Table SAT( x, y). SAT( x, y) is (a) (b) (a) (b) (c) (d) SAT(x,y) TSAT(x,y) 2. Line features (a) (b) (c) (d) (e) (f) (g) () 3. Center-surround features - (c) (a) (b) 4. Special diagonal line feature used in [3,4,5] Figure 2: Feature prototypes of simple aar-like and center-surround features. Black areas ave negative and ite areas positive eigts. NUMBER OF FEATURES. Te number of features derived from eac prototype is quite large and differs from prototype to prototype and can be calculated as follos. Let X= W and Y= H be te maximum scaling factors in x and y direction. An uprigt feature of size x ten generates XY W 1 X H Y 1 2 features for an image of size WxH, ile a 45 rotated feature generates defined as te sum of te pixels of te uprigt rectangle ranging from te top left corner at (0,0) to te bottom rigt corner at (x,y) (see Figure 3a) [6]: SAT( x, y) = Ix' (, y' ). x' x, y' y It can be calculated it one pass over all pixels from left to rigt and top to bottom by means of it Figure 3: (a) Uprigt Summed Area Table (SAT) and (b) Rotated Summed Area Table (RSAT); calculation sceme of te pixel sum of uprigt (c) and rotated (d) rectangles. SAT( x, y) = SAT( x, SAT( x 1, y) Ixy (, ) SAT( x 1, SAT( 1, y) = SAT( x, = SAT( 1, = 0 From tis te pixel sum of any uprigt rectangle r=( xy0,,,, ) can

3 be determined by four table lookups (see also Figure 3(c): Tis insigt as first publised in [6]. For 45 rotated rectangles te auxiliary image is te Rotated Summed Area Table RSAT( x, y). It is defined as te sum of te pixels of a 45 rotated rectangle it te bottom most corner at (x,y) and extending upards till te boundaries of te image (see Figure 3b): RSAT( x, y) = Ix' (, y' ). y' y, y' x ' x It can be calculated also in one pass from left to rigt and top to bottom over all pixels by it RecSum() r = SAT( x 1, SAT( x 1, SAT( x 1, SAT( x 1, RSAT( x, y) = RSAT( x 1, RSAT( x 1, RSAT( x, 2) Ixy (, ) Ixy (, RSAT( 1, y) = RSAT( x, = RSAT( x, 2) = 0 RSAT( 1, = RSAT( 1, 2) = 0 as son in Figure 4. From tis te pixel sum of any rotated rectangle r=( xy45,,,, ) can be determined by four table lookups (see Figure 5): RecSum() r = RSAT( x, RSAT( x,. RSAT( x, RSAT( x, -RSAT(x,y-2) RSAT(x-1,y- Figure 4: I(x,y)I(x,y- RSAT(x1,y- Calculation sceme for Rotated Summed Area Tables (RSAT). 2.3 Fast Ligting Correction Te special properties of te aar-like features also enable fast contrast stretcing of te form Ixy (, ) µ Ixy (, ) = , c R. cσ µ can easily be determined by means of SAT(x,y). Computing σ, oever, involves te sum of squared pixels. It can easily be derived by calculating a second set of SAT and RSAT auxiliary images for I 2 ( xy, ). Ten, calculating σ for any indo requires Figure 5: -RSAT(x-,- (x,y) RSAT(x-,- RSAT(x,y- -RSAT(x,- Calculation sceme for rotated areas. only 4 additional table lookups. In our experiments c as set to 2. 3 (Stage) Classifier We use boosting as our basic classifier. Boosting is a poerful learning concept. It combines te performance of many "eak" classifiers to produce a poerful 'committee' [1]. A eak classifier is only required to be better tan cance, and tus can be very simple and computationally inexpensive. Many of tem smartly combined, oever, result in a strong classifier, ic often outperforms most 'monolitic' strong classifiers suc as SVMs and Neural Netorks. Different variants of boosting are knon suc as Discrete Adaboost (see Figure 6), Real AdaBoost, and Gentle AdaBoost (see Figure 7)[1]. All of tem are identical it respect to computational complexity from a classification perspective, but differ in teir learning algoritm. All tree are investigated in our experimental results. Learning is based on N training examples ( x 1, y 1 ),,( x N, y N ) it x R k and y i { 1,. x i is a K-component vector. Eac component encodes a feature relevant for te learning task at and. Te desired to-class output is encoded as 1 and 1. In te case of object detection, te input component x i is one aar-like feature. An output of 1 and -1 indicates eter te input pattern does contain a complete instance of te object class of interest. 4 Cascade of Classifiers A cascade of classifiers is a degenerated decision tree ere at eac stage a classifier is trained to detect almost all objects of interest (frontal faces in our example) ile rejecting a certain fraction of te non-object patterns [6] (see Figure 8). For instance, in our case eac stage as trained to eliminated 50% of te non-face patterns ile falsely eliminating only 0.1% of te frontal face patterns; 20 stages ere trained. Assuming tat our test set is representative for te learning task, e can expect a false alarm rate about e 07 and a it rate about

4 Discrete AdaBoost (Freund & Scapire [1]) 1. Given N examples ( x 1, y 1 ),,( x N, y N ) it x R k, yi { 1, 2. Start it eigts i = 1/N, i = 1,..., N. 3. Repeat for m = 1,..., M (a) Fit te classifier f m () x { 1, using eigts i on te training data ( x 1, y 1 ),,( x N, y N ). (b) Compute err m = E [ 1( y fm () x ) ], c m =log( ( 1 err m ) err m ). (c) Set i i exp( c m 1( yi f m ( x i ))), i = 1,..., N, and renormalize eigts so tat i = 1. M i 4. Output te classifier sign c m f m ( x) m= 1 Figure 6:Discrete AdaBoost training algoritm [1]. Gentle AdaBoost 1. Given N examples ( x 1, y 1 ),,( x N, y N ) it x R k, yi { 1, 2. Start it eigts i = 1/N, i = 1,..., N. 3. Repeat for m = 1,..., M (a) Fit te regression function f m () x by eigted least-squares of y i to x i it eigts i (c) Set i i exp( y i f m ( x i )), i = 1,..., N, and renormalize eigts so tat i = Output te classifier sign M f m () x i m= 1 Figure 7:Gentle AdaBoost training algoritm [1] stage N itrate= N Figure 8: 1-f 1-f 1-f 1-f input pattern classified as a non-object falsealarms= fn Cascade of classifiers it N stages. At eac stage a classifier is trained to acieve a it rate of and a false alarm rate of f. Eac stage as trained using one out of te tree Boosting variants. Boosting can learn a strong classifier based on a (large) set of eak classifiers by re-eigting te training samples. Weak classifiers are only required to be sligtly better tan cance. Our set of eak classifiers are all classifiers ic use one feature from our feature pool in combination it a simple binary tresolding decision. At eac round of boosting, te feature-based classifier is added tat best classifies te eigted training samples. Wit increasing stage number te number of eak classifiers, ic are needed to acieve te desired false alarm rate at te given it rate, increases (for more detail see [6]). 5 Experimental Results All experiments ere performanced on te complete CMU Frontal Face Test Set of 130 grayscale pictures it 510 frontal faces [7]. A it as declared if and only if te Euclidian distance beteen te center of a detected and actual face as less tan 30% of te idt of te actual face as ell as te idt (i.e., size) of te detected face as itin ±50% of te actual face idt. Every detected face, ic as not a it, as counted as a false alarm. Hit rates are reported in percent, ile te false alarms are specified by teir absolute numbers in order to make te results comparable it related ork on te CMU Frontal Face Test set. Except oterise noted 5000 positive frontal face patterns and 3000 negative patterns filtered by stage 0 to n-1 ere used to train stage n of te cascade classifer. Te 5000 positive frontal face patterns ere derived from 1000 original face patterns by random rotation about ±10 degree, random scaling about ±10%, random mirroring and random sifting up to ±1 pixel. Eac stage as trained to reject about alf of te negative patterns, ile correctly accepting 99.9% of te face patterns. A fully trained cascade consisted of 20 stages. During detection, a sliding indo as moved pixel by pixel over te picture at eac scale. Starting it te original scale, te features ere enlarged by 10% and 20%, respectively (i.e., representing a rescale factor of 1.1 and 1.2, respectively) until exceeding te size of te picture in at least one dimension. Often multiple faces are detect at near by location and scale at an actual face location. Terefore, multiple nearby detection results ere merged. Receiver Operating Curves (ROCs) ere constructed by varing te required number of detected faces per actual face before merging into a single detection result. During experimentation only one parameter as canged at a time. Te best mode of a parameter found in an experiment as used for te subsequent experiments.

5 Figure 10: Performance comparison beteen identically trained cascades it tree different boosting algoritms. Only te basic feature set and stumps as eak classifiers (nsplit= ere used. 5.1 Feature Scaling Any multi-scale image searc requires eiter rescaling of te picture or te features. One of te advantage of te Haar-like features is tat tey can easily be rescaled. Independent of te scale eac feature requires only a fixed number of look-ups in te sum and squared sum auxilary images. Tese look-ups are performed relative to te top left corner and must be at integral positions. Obviously, by fractional rescaling te ne correct positions become fractional. A plain vanilla solution is to round all relative look-up positions to te nearest integer position. Hoever, performance may degrade significantly, since te ratio beteen te to areas of a feature may ave canged significantly compared to te area ratio at training due to rounding. One solution is to correct te eigts of te different rectangle sums so tat te original area ratio beteen tem for a given aar-like feature is te same as it as at te original size. Te impact of tis eigt adapation on te performance is amazing as can be seen in Figure 9. *-Rounding so te ROCs for simple rounding, ile *-AreaRatio sos te impact if also te eigt of te different rectangles is adjusted to reflect te eigts in te feature at te original scale. 5.2 Comparision Beteen Different Boosting Algoritms We compared tree different boosting algoritms: Discrete Adaboost, Real Adaboost, and Gentle Adaboost. Tree 20-stage cascade classifiers ere trained it te respective boosting algoritm using te basic feature set (i.e., features 1a, 1b, 2a, 2c, and 4a of Figure 2) and stumps as te eak classifiers. As can be seen from Figure 10, Gentle Adaboost outperformed te oter to boosting algoritm, despite te fact tat it needed on average feer features (see Table 2, second column). For instance, at a an absolute false alarm rate of 10 on te CMU test set, RAB deteted only 75.4% and DAB only 79.5% of all frontal faces, ile GAB acieved 82.7% at a rescale factor of 1.1. Also, te smaller rescaling factor of 1.1 as very beneficial if a very lo false alarm rate at ig detection performance ad to be acieved. At 10 false alarms on te CMU test set, GAB improved from 68.8% detection rate it rescaling factor of 1.2 to 82.7% at a rescaling factor of 1.1. Table 2 sos in te second column (nsplit = te average number false alarm rate Figure 9: BASIC14 -Rounding BASIC14 - AreaRatio BASIC19 - AreaRatio BASIC19 - Rounding it rate Performance comparision beteen different feature scaling approaces. *-Rounding rounds te fractional position to te nearest integer position, ile *-AreaRatio also restores te ratio beteen te different rectangles to its original value used during training. of features needed to be evaluted for background patterns by te different classifiers. As can be seen GAB is not only te best, but also te fastest classifier. Terefore, e only investigate a rescale scaling factor 1.1 and GAB in te subsequent experiments. NSPLIT DAB GAB RAB Table 2: Average number of features evaluated per background pattern at a pattern size of 20x Input Pattern Size Many different input pattern sizes ave been reported in related

6 ork on face detection ranging from 16x16 up to 32x32. Hoever, none of tem ave systematically investigated te effect of te input pattern size on detection performance. As our experiments so for faces an input pattern size of 20x20 acieves te igest it rate at an absolute false alarms beteen 5 and 100 on te CMU Frontal Face Test Set (see Figure 1. Only for less tan 5 false alarms, an input pattern size of 24x24 orked better. A similar observation as been made by [2]. Figure 12: Performance comparison it respect to te order of te eak CART classifiers. GAB as used togeter it te basic feature set and a pattern size of 18x18. Figure 11: Performance comparison beteen identically trained cascades, but it different input pattern sizes. GAB as used togeter it te basic feature set and stumps as eak classifiers (nsplit=. 5.4 Tree vs. Stumps Stumps as eak classifer do not allo learning dependencies beteen features. In general, N split nodes are needed to model dependency beteen N-1 variables. Terefore, e allo our eak classifier to be a CART tree it NSPLIT split nodes. Ten, NSPLIT=1 represents te stump case. As can be seen from Figure 12 and Figure 13 stumps are outperformed by eak tree classifiers it 2, 3 or 4 split nodes. For 18x18 four split nodes performed best, ile for 20x20 to nodes ere sliglty better. Te difference beteen eak tree classifiers it 2, 3 or 4 split nodes is smaller tan teir superiority it respect to stumps.te order of te computational complexity of te resulting detection classifier as unaffected by te coise of te value of NSPLIT (see Table. Te more poerful CARTs proportionally needed less eak classifiers to acieve te same performance at eac stage. 5.5 Basic vs. Extended Haar-like Features To face detection systems ere trained: One it te basic and one it te extended aar-like feature set. On average te false alarm rate as about 10% loer for te extended aar-like feature set at comparable it rates. Figure 14 sos te ROC for bot classifiers using 12 stages. At te same time te computational complexity as comparable. Te average number of features evaluation per patc as about 31 (see [3] for more details). Tese results suggest tat altoug te larger aar-like feature set usually complicates learning, it as more tan paid of by te added domain knoledge. In principle, te center surround feature ould ave been sufficient to approximate all oter features, oever, it is in general ard for any macine learning algoritm to learn joint Figure 13: Performance comparison it respect to te order of te eak CART classifiers. GAB as used togeter it te basic feature set and a pattern size of 20x20. beavior in a reliable ay. 5.6 Training Set Size So far, all trained cascades used 5000 positive and 3000 negative examples per stage to limit te computational complexity during training. We also trained one 18x18 classifiers it all positive face examples, in total and 5000 negative training examples. As can be seen from Figure 15, tere is little difference in te training results. Large training sets only sligtly improve performance indicating tat te cascade trained it 5000/3000 examples already came close to its representation poer. Conclusion Our experimental results suggest, tat 20x20 is te optimal input pattern size for frontal face detection. In addition, tey so tat

7 it rate Performance comparison beteen basic and extented feature set Using Basic Features Using Extended Features false alarms Figure 14: Basic versus extended feature set: On average te false alarm rate of te face detector exploiting te extended feature set as about 10% better at te same it rate (taken from [3]). Gentle Adaboost outperforms Discrete and Real Adaboost. Logitboot could not be used due to convergence problem on later stages in te cascade training. It is also beneficial not just to use te simplest of all tree classifiers, i.e., stumps, as te basis for te eak classifiers, but representationally more poerful classifiers suc as small CART trees, ic can model second and/or tird order dependencies. We also introduced an extended set of aar-like features. Altoug frontal faces exibit little diagonal structures, te 45 degree rotated features increased te accuracy. In practice, te ave observed tat te rotated features can boost detection performance if te object under detection exibit some diagonal structures suc as many brand logos. Te complete training and detection system as ell as a trained face detector are available in te Open Computer Vision Library at ttp:/ sourceforge.net/projects/opencvlibrary/ [8]. 6 REFERENCES [1] Y. Freund and R. E. Scapire. Experiments it a ne boosting algoritm. In Macine Learning: Proceedings of te Tirteent International Conference, Morgan Kauman, San Francisco, pp , [2] Stan Z. Li, Long Zu, ZenQiu Zang, Andre Blake, HongJiang Zang, and Harry Sum. Statistical Learning of Multi-Vie Face Detection. In Proceedings of Te 7t European Conference on Computer Vision. Copenagen, Denmark. May, [3] Rainer Lienart and Jocen Maydt. An Extended Set of Haarlike Features for Rapid Object Detection. IEEE ICIP 2002, Vol. 1, pp , Sep [4] A. Moan, C. Papageorgiou, T. Poggio. Example-based object detection in images by components. IEEE Transactions on Pattern Analysis and Macine Intelligence, Vol. 23, No. 4, pp , April [5] C. Papageorgiou, M. Oren, and T. Poggio. A general frameork for Object Detection. In International Conference on Computer Vision, [6] Paul Viola and Micael J. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE CVPR, [7] H. Roley, S. Baluja, and T. Kanade. Neural netork-based face detection. In IEEE Patt. Anal. Mac. Intell., Vol. 20, pp , [8] Open Computer Vision Library. ttp:/sourceforge.net/projects/ opencvlibrary/ Figure 15: Performance comparison it respect to te training set size. One 18x18 classifier as trained it face and 5000 non-face examples using GAB and te basic feature set.