Incremental Multiple Kernel Learning for Object Recognition

Transcription

1 Incremental Multple Kernel Learnng for Obect Recognton Anruddha Kembhav, Behat Sddque, Roland Mezano, Scott McClosey, Larry S. Davs Unversty of Maryland, College Par Honeywell Labs Abstract A good tranng dataset, representatve of the test mages expected n a gven applcaton, s crtcal for ensurng good performance of a vsual categorzaton system. Obtanng tas specfc datasets of vsual categores s, however, far more tedous than obtanng a generc dataset of the same classes. We propose an Incremental Multple Kernel Learnng (IMKL) approach to obect recognton that ntalzes on a generc tranng database and then tunes tself to the classfcaton tas at hand. Our system smultaneously updates the tranng dataset as well as the weghts used to combne multple nformaton sources. We demonstrate our system on a vehcle classfcaton problem n a vdeo stream overloong a traffc ntersecton. Our system updates tself wth mages of vehcles n poses more commonly observed n the scene, as well as wth mage patches of the bacground, leadng to an ncrease n performance. A consderable change n the ernel combnaton weghts s observed as the system gathers scene specfc tranng data over tme. The system s also seen to adapt tself to the llumnaton change n the scene as day transtons to nght.. Introducton The problem of vsual category recognton has receved consderable nterest over the past few years. The most common approach conssts of three maor components: nterest pont detecton, nterest regon descrpton and classfcaton. A recent focus has been on mprovng regon descrptors. Ths has led to a number of powerful descrptors beng proposed such as Hstograms of Orented Gradents [8], Geometrc Blur [3] and Pyramdal Hstogram of Vsual Words [6]. Whle each of these descrptors provdes good classfcaton accuraces for dfferent obect classfcaton tass, combnng nformaton from such multple sources has been shown to be more relable [5, 4, 9]. Varma et al. [9] proposed combnng multple descrptors usng Multple Kernel Learnng (MKL) and showed mpressve results on vared obect classfcaton tass. Usng such a set of powerful descrptors, along wth a nonlnear classfer such as a Support Vector Machne (SVM), can lead to a boost n classfcaton performance. Correspondng author: anem@umd.edu Detector IMKL Obect Detector Generc Obect Fgure. Sample result frames showng varyng llumnaton condtons. Our ncremental framewor (IMKL) tunes tself to the scene by updatng tself wth mages of obects n commonly observed poses and mages of the varyng bacground. Thus, t outperforms a statc detector bult on a generc tranng set. But t s equally mportant to have a good set of tranng mages, representatve of the test mages that are expected n the gven applcaton. Collectng large number of mages and formng a generc tranng dataset for commonly seen obects s relatvely easy usng an nternet search engne such as Google. Furthermore for many standard obects such as cars, tranng datasets are already avalable, such as the UIUC Car Database []. However, obtanng a representatve tranng database for a gven applcaton s not as straghtforward, as t requres a far amount of manual labor. Consder a camera at a traffc ntersecton detectng and classfyng vehcles such as shown n Fgure. Frst, the locaton of the camera n ths scene and typcal paths traversed by the vehcles, restrcts the observed poses. Second, the camera poston restrcts the mages representng the negatve class (n our case, the bacground mages) for ths classfcaton tas. Thrd, mages correspondng to vehcles as well as bacground also change over tme, due to factors such as llumnaton changes and shadows cast by the nearby buldngs. Obtanng such scene specfc exam-

2 ples of the obect classes and the bacground class would clearly beneft the vsual classfer, but would requre a tedous manual annotaton procedure. Our Incremental Multple Kernel Learnng (IMKL) approach uses an easly obtaned generc tranng database as nput, and then tunes tself to the classfcaton tas at hand. It smultaneously updates the tranng examples to talor them towards the obects n the scene. It also updates the weghts that determne the optmal combnaton of dfferent nformaton sources, whle allowng dfferent combnatons to be chosen for dfferent obect classes. Fnally, t tunes the classfer to the updated tranng dataset. As the scene changes over tme, a feedbac loop updates our tranng dataset wth detectons from all obect classes. The ncremental procedure s then nvoed to update the ernel combnaton weghts as well as the classfer. Our fnal system s obtaned by combnng the outputs of ths onlne classfer wth the hgh probablty outputs of the orgnal offlne classfer traned on the generc tranng database. Ths enables us to tune the classfer to the gven scene, whle reducng the number of msclassfcatons on rarely seen obects. We can also remove mages from our tranng database over tme. Ths s useful when dealng wth gradual llumnaton changes, for example. We frst descrbe the MKL formulaton of Bach et al. [5], nown as SmpleMKL, whch we use to obtan a classfer for the ntal tranng database. SmpleMKL carres out ths optmzaton n an SVM framewor to smultaneously learn the SVM model parameters as well as ernel combnaton weghts. Our ncremental procedure for MKL s an exact onlne soluton that allows us to update the Lagrangan multplers of the tranng ponts, as well as the ernel combnaton weghts, one new pont at a tme. The central dea s to add a new data pont to the soluton and update ts Lagrangan multpler whle mantanng the Karush-Kuhn-Tucer condtons on all the current data ponts n the soluton. We derve our IMKL procedure n Secton 3. We demonstrate our vsual categorzaton framewor on the tas of vehcle detecton and classfcaton. The dataset we use conssts of vdeo sequences collected from a camera overloong a traffc ntersecton. We ntalze our tranng database wth a set of mages collected from Google and update t ncrementally to mprove the classfcaton performance over tme. The dataset also shows a sgnfcant change n llumnaton condtons n the scene as day transtons nto nght. Our system s able to update tself over tme to handle ths transton. We compare our algorthm wth OPTIMOL [3], an ncremental model learnng approach, recently proposed for the tas of automatc obect dataset collecton.. Related Wor Early wors on obect recognton used global features such as color or texture hstograms [4]. However these In ths paper each nformaton source refers to a ernel matrx. features were not robust to vew-pont changes, clutter and occluson. Over the years, more sophstcated approaches such as part-based [9] and bag-of-features [6] methods have become more popular. Increased nterest n obect recognton has resulted n new feature descrptors and a multtude of classfers. Inspred by the pyramdal feature matchng approach of [], Bosch et al. proposed two new regon descrptors - the Pyramdal Hstogram of Orented Gradents (PHOG) and Pyramdal Hstogram of Vsual Words (PHOW) [6]. These features were then used wth Random Forests as a multway classfer [5]. Zhang et al. used the Geometrc Blur (GB) feature [3] and proposed usng a dscrmnatve nearest neghbor classfcaton for obect recognton [3]. Wu et al. [] used edgelet features to capture the local shape of obects and were able to smultaneously detect and segment obects of a nown category. Zhang et al. [4] combned multple descrptors and obtaned mproved results for texture classfcaton and obect recognton. They provded equal weghts to each descrptor. Smlarly, Bosch et al. [5] lnearly combned the PHOG and PHOW descrptors to obtan mproved performance. The lnear combnaton weghts were, however, obtaned by a brute force search usng a valdaton dataset. Snce the number of features was small, ther search space had few dmensons, thus mang the brute force computatonally feasble. Wu et al. [] combned multple heterogeneous features for obect detecton by usng cascade structured detectors n a boostng framewor. Features were combned usng ther classfcaton powers and computatonal cost. Lancret et al. [] ntroduced the MKL procedure to learn a set of lnear combnaton weghts, whle usng multple sources of nformaton wth a ernel method, such as an SVM. Ther problem formulaton, however, resulted n a convex but non-smooth mnmzaton problem. Bach et al. [] consdered a smoothed verson of the problem. Ther Sequental Mnmal Optmzaton (SMO) algorthm was sgnfcantly more effcent than the prevous formulaton n []. Sonnenburg et al. [7] reformulated the problem as a sem-nfnte lnear program and solved t effcently by recyclng the standard fast SVM mplementatons. Ther algorthm wored for hundreds of thousands of examples or hundreds of ernels. Raotomamony et al. [5] formulated the problem usng a -norm regularzaton formulaton to a smooth and convex optmzaton problem. Ther method provded the addtonal advantage of encouragng sparse ernel combnatons. Varma et al. [9] combned multple features usng MKL and showed a consderable ncrease n the performance of ther vsual classfer. A number of unsupervsed, onlne learnng algorthms have been used for computer vson applcatons. L et al. [3] used a non-parametrc graphcal model n an ncremental approach for automatc dataset collecton from the Internet (OPTIMOL). Ther teratve framewor smultaneously learns obect category models and collects obect category datasets. We compare our IMKL method wth OP-

3 TIMOL n Secton 5. Boostng technques for ncremental learnng have also been popular. Javed et al. [] used co-tranng to label ncomng data and used t to update a boosted classfer. Co-tranng [4] s a method for tranng a par of learners, gven that the two algorthms use dfferent vews of the data. The two classfers are used to provde addtonal nformatve labeled examples to one another, whch mproves the overall performance. Wu et al. [] extended the onlne boostng algorthm and proposed an onlne framewor for cascade structured detectors. An automatc labeler called the oracle, wth a hgh precson rate, provded samples to update the onlne obect detector. In order to prevent the boostng algorthm from overfttng nosy data (provded by the oracle), they employed two nose resstant strateges from varants of the Adaboost algorthm desgned to be robust to outlers. Our ntal obect classfer, bult from a generc tranng dataset, s tuned smlar to ths oracle. Our wor bulds on MKL and fts well nto the SVM framewor. It also provdes the useful property of beng able to adapt ernel weghts over tme n addton to updatng the tranng database. 3. An Incremental Soluton 3.. The Multple Kernel Learnng Problem Kernel based learnng methods have proven to be an extremely effectve dscrmnatve approach to classfcaton as well as regresson problems. Gven multple sources of nformaton, one mght calculate multple bass ernels, one for each source. In such cases, the resultant ernel s often computed as a convex combnaton of the bass ernels, K K Φ(x, x ) = d Φ (x, x ), d =, d () = = where x are the data ponts, Φ (x, x ) s the th ernel and d are the weghts gven to each nformaton source (ernel). Learnng the classfer model parameters and the ernel combnaton weghts n a sngle optmzaton problem s nown as the Multple Kernel Learnng problem []. There have been a number of formulatons for the MKL problem, as noted n Secton. Our ncremental approach bulds on the MKL formulaton of [5], nown as SmpleMKL. Ths formulaton enables the ernel combnaton weghts to be learnt wthn the SVM framewor. The optmzaton equaton s gven by, such that mn w w T + C ξ d y φ (x ) + y b ξ () ξ, d, d = where b s the bas, ξ s the slac afforded to each data pont and C s the regularzaton parameter. The soluton to the above MKL formulaton s based on a gradent descent on the SVM obectve value. An teratve method alternates between determnng the SVM model parameters usng a standard SVM solver and determnng the ernel combnaton weghts usng a proected gradent descent method. 3.. Karush-Kuhn-Tucer Condtons The support vectors returned by the tranng algorthm of an SVM generally represent a small fracton of all the tranng examples, but are able to summarze the decson boundary between the classes very well. Thus, one way to ncrement an SVM s to retan only the support vectors, to reduce the computatonal load requred at every successve tranng step [8]. The same approach could be used for the MKL problem. However, ths gves only approxmate results. The frst exact onlne approach to tran SVMs was proposed by Cauwenberghs et al. [7]. New data ponts are presented to the SVM one at a tme. The new data pont s added to the soluton whle ensurng that the Karush-Kuhn- Tucer (KKT) condtons are retaned on all the prevous data ponts. Our proposed approach to IMKL s nspred by ths wor. The ey dea behnd the Incremental SVM s that the SVM optmzaton problem s convex. Thus, the KKT condtons are not only necessary but also suffcent. Thus, mantanng the KKT condtons on all old ponts, as well as the new pont, ndcates that a new soluton has been obtaned. The optmzaton problem gven by the SmpleMKL framewor n Equaton s also convex, mang t sutable for our purposes. The KKT condtons for our problem are derved from the Lagrangan functon correspondng to Equaton, L = w w + C ξ ν ξ µ d d α (y w φ (x ) + y b + ξ ) λ( d ) (3) where α s the Lagrange multpler correspondng to the frst constrant n Equaton, ν and µ are the Lagrange multplers assocated wth the non-negatvty constrants on ξ and d respectvely, whle λ corresponds to the Lagrange multpler of the l -norm equalty constrant on d. The optmal soluton of the multple ernel system n Equaton occurs at the saddle pont of Equaton 3. The saddle pont s obtaned by dfferentatng the Lagrangan equaton wth respect to the prmal varables (w, d, ξ, b) and the dual varables (α, ν, µ ). A small amount of algebrac manpulatons yelds the KKT condtons gven below, g = d α Q + y b =, α y = α α Q + µ λ =, µ d =, d = (4) where Q = y Φ (x, x )y. Note that g = y f(x ), where f(x ) s the soluton of the multple ernel SVM gven by, f(x new) = d α y Φ (x, x new) + b (5)

4 3.3. Algorthm Consder a set of data nstances (x, x,..., x n ) wth correspondng class labels (y, y,..., y n ). Let Φ (x, x ) be the set of K ernels. The MKL soluton for the gven data s obtaned by SmpleMKL and t thus satsfes the KKT condtons n Equaton 4. The data ponts are dvded nto three dsont sets based on ther Lagrange multplers (α s): set L contanng the set of ponts lyng on the correct sde of the margn vectors (α = ), set S contanng the support vectors ( < α < C) and set E contanng the ponts lyng on the wrong sde of the margns (α = C). We also dvde the ernels nto two sets: set D + contanng ernels wth postve weghts and set D wth ernels havng zero weght. These sets are llustrated n Fgure. When a new pont x q s added to the soluton, we need to calculate ts Lagrange multpler α q ( α q C) such that the KKT condtons are satsfed once agan. We begn wth a value α q = and eep ncreasng t untl we reach the updated soluton. Every tme we ncrement α q, the remanng Lagrangan multplers, the ernel weghts and the bas must be changed to mantan the constrants n Equaton 4. These changes are gven by the dfferental form of the constrants, α d Q + α Q + d α Q + y b =, S, {S, E, L, q} α α Q + α α Q (6) + µ λ =, K α y = {S, E, L, q}, d = µ d + µ d + µ d =, K For a gven step sze α q, Equaton 6 s a set of (num S + K + ) equatons n (num S + K + ) unnowns. Here, num S s the number of ponts n set S and K s the number of ernels. The unnown varables are: { α..., α nums, d,..., d K, µ,..., µ K, b, λ}. These non-lnear equatons can be solved usng a standard non-lnear equaton solvng pacage. Snce an addton of a new pont may not alter the system sgnfcantly, a good ntal soluton for all the unnowns n Equaton 6 s. The above dfferental equatons only hold when α q s small enough to ensure that there s no change n set membershp for ether the ponts or the ernels. Thus, when set membershp changes, the dfferental equatons are updated and the process s repeated. The condtons for a change n the set membershp are descrbed n Fgure. The algorthm s termnated when any of the followng condtons occur. g q > at α q = : x q s a correctly classfed pont. Added to set L. g q = before α q = C: x q s a support vector. Added to set S. g > g = g < g (+,) g (,-) L S E α (+,) α (+,C) α = < α < C α = C d = D + d (+,) μ (+,) Fgure. Categorzaton of the data ponts and ernels. The mage on the left shows the values of the Lagrange multplers (α s) and the output of the system (g s) for each of the sets: L, S and E. It also shows the condtons that are checed to detect a set transton. (Notaton: g (+, ) denotes the value of g changng from a postve value to.) The mage on the rght shows the two ernel sets, the correspondng values of the weghts (d s) and ther Lagrange multplers (µ s) and the set change condtons. α q = C and g q < : x q s on the wrong sde of the margn. Added to set E. A smlar procedure can be used for removng data ponts from the classfer (decremental unlearnng). The number of computatons requred by the IMKL algorthm depends on the computatons to solve the nonlnear system and the number of steps taen to reach the fnal value of α q. In our experments, we have observed that settng the ntal soluton of the non-lnear solver to a zero vector, reduces the computatonal cost sgnfcantly. The number of steps taen to reach the fnal soluton s lower bounded by the number of set changes that are requred to arrve at the fnal soluton. We use a large step sze at every tme nstant and bactrac our soluton f we observe a set change for the gven step sze. The IMKL algorthm can also be sped up by gnorng the hgher order terms n Equaton 6 to obtan lnear equatons. However ths provdes only an approxmate soluton. Consder the two class classfcaton problem shown n Fgure 3. A new pont q, mared n red, s added to the system, and t ntally gets msclassfed. As the Lagrange multpler α q s ncremented upwards from a value of, the dstance between the new pont and the margn reduces, whle some of the other ponts change set membershp. At the same tme, the ernel combnaton weghts also change. 4. Obect Recognton Framewor A tranng database, representatve of the expected test ponts, s an essental component of any classfcaton system. In a practcal obect recognton framewor, a good tranng database s one that contans mages of the expected obects n ther more lely poses and llumnaton condtons. It must also contan a representatve set of mages n the negatve set, whch, n an obect recognton framewor, s usually the bacground. Obtanng such a set of good tranng examples can often be a tedous process. On the other hand, t s easer to obtan a generc tranng dataset of mages of the expected obect classes. Our obect detector d > D μ = μ >

5 Fgure 3. A -class classfcaton example. Ponts n class are shown n orange and ponts n class are shown n blue. Ponts n set S are mared wth a blac border. Ponts n set L are sold colored whle ponts n set E are not flled wth color. Kernel (weght shown by the brown bar) captures the smlarty between the y-coordnates of the ponts, whle Kernel (green bar) captures the smlarty between the x-coordnates. The left fgure shows the effect of addng a new pont (shown n red) on the orgnal ponts and the weghts. A change n set membershp s observed for some ponts. The fgure on the rght shows the fnal classfer after addng 7 new ponts close to the frst new pont. s ntalzed on a generc tranng dataset and tunes tself towards the obects and bacground n the scene. e Databas g T rann enerc Global Obect Detector Local Obect Detector Updatng Crtera Fgure 4. Obect recognton framewor. Fgure 4 provdes an overvew of our vsual categorzaton framewor. Tranng mages from a generc tranng dataset are used to tran an ntal obect detector whch we call the global detector. The global detector s not updated at any tme and serves as a generc obect classfer. The generc tranng dataset s also used to tran a local obect detector, whch runs n an onlne mode throughout the duraton of analyss. Incomng mages from a vdeo stream are scanned usng overlappng wndows and each wndow s classfed nto one of the classes by both the detectors. The classfcaton results returned by the global detector eep updatng the tranng mage sets of the local obect detector. The updatng crteron dffers for the foreground classes (buses and cars) and the bacground class. The mage wndows that are classfed by the global detector as belongng to one of the foreground classes are thresholded so as to retan only very hgh confdence detectons. Such wndows are consdered relable detectons and used to update the foreground tranng sets of the local obect detector. Snce the purpose of the local obect detector s to tran on typcally observed appearances and poses, updatng t wth hgh confdence samples wors well. The hgh precson of the global detector comes at the cost of a lower recall. Updatng Updatng crtera the local detector wth false postves can lead to a sgnfcant drop n the performance of the system, and the probablty threshold s set suffcently hgh to mnmze ths. On the other hand, for the bacground class, such an updatng crteron leads to the addton of a large number of mage patches from a sngle porton of the scene. Ths s because bacground patches wth very smlar appearances repeat over several frames. Thus f a patch gets classfed wth a very hgh probablty of belongng to the negatve set, several smlar mages also get added to the local tranng set. Ideally, one would le the entre scene to be well represented n the bacground class of the local detector. Thus, we frst threshold mage wndows classfed by the global detector as belongng to the bacground class. Then, for every mage patch passng ths ntal crteron, we evaluate ts postonal entropy wth respect to the dstrbuton of the postons of all mage patches currently n the local bacground tranng set. Ths s gven by, H(I) = w {BG local } p(w (x,y) I (x,y) ) log p(w (x,y) I (x,y) ) (7) where w represents an mage patch n the current bacground set, I represents the new mage patch and (x, y) represent the co-ordnates of an mage patch n the scene. Image patches passng the ntal bacground threshold, as well as havng a hgh entropy wth respect to the current local tranng set, form good canddates to mprove the dversty of the local bacground set and are added to t. Over tme, the obect classes get updated wth mages of obects n ther typcal observed appearances and poses and the bacground class gets updated wth mage patches from dfferent parts of the scene. Fgure 5 demonstrates the mage patches n the local bacground set whch has been updated usng both crtera. Usng the entropy crtera n addton to a probablty threshold, samples the entre scene well. L et al. [3] used a smlar crtera to update ther dataset. Whle ther entropy s calculated n the feature space, our measure s calculated n the mage co-ordnate space. The local detector fts tself towards mage patches observed n the recent past, mprovng ts performance. However, t also has the tendency of msclassfyng obects that are atypcal n the scene, due to overfttng on the observed data. The more genercally traned global detector helps classfy such atypcal obects. The outputs of both detectors are combned to obtan the fnal detectons. The resultant obect detectons are used to update the local detector. In order to ft the local detector towards a dynamcally changng scene, t s also mportant to dscard mage patches from the local tranng dataset. For every mage patch added to the local set, we retan a tmestamp ndcatng the frame t was obtaned from. We use ths to dscard tranng samples based on the length of ther stay n the tranng set. Thus the classfer adapts tself towards changng llumnaton condtons, partcularly when day transtons to nght. Our IMKL algorthm descrbed n Secton 3 s used to update the Local classfer wth new tranng mages. Ths

6 Cars Bacground Fgure 5. Representaton of the local negatve tranng set usng two samplng methods to update the tranng set. For ths dsplay, all mage patches n the set are added together at the approprate locatons n the scene. Thus brghter regons corresponds to more patches n that porton of the scene, blac regons ndcate that no mage patches represent that porton of the scene. (Left) Hgh probablty crtera - Only certan portons of the scene are represented. (Rght) Hgh probablty + hgh entropy crtera - Most portons of the scene are represented equally. also results n an update of the ernel combnaton weghts based on the tranng data. We use multple -Vs-All classfers for our purpose of mult-class classfcaton. Ths enables us to compute a separate set of ernel combnaton weghts, one for each obect class. In Secton 5 we show an example of the evoluton of these ernel weghts over tme. 5. Experments We test the performance of our system on the tas of obect detecton on vdeos taen from a traffc dataset. Ths dataset conssts of challengng vdeos (48 x 74 pxels at 5 frames/second), of a busy ntersecton, taen from a traffc survellance camera. The total number of frames s more than,. Our tas s to detect two classes of obects, cars and buses. We have ground truth mared for every tenth frame n ths dataset. Due to the camera locaton and traffc restrctons n the scene, cars n the vdeo typcally have a frontal vew, whle buses typcally appear n a profle vew. Other vews are also observed, but they are less common. The car category ncludes cars of varyng szes as well as SUV s and trucs. Wth a few exceptons, buses have a smlar appearance, snce most of them are publc transportaton buses. The dataset conssts of vdeos captured at dfferent tmes of the day, resultng n a varety of llumnaton condtons as shown n Fgure, ncludng street-lghts at nght. For vdeos captured durng the transton of day to nght, the appearances of the vehcles also change (most promnently, vehcles n the dar have ther headlghts turned on). 5.. Kernel Matrces We use 5 nds of features n our system, gvng rse to a total of 7 ernel matrces. The frst feature used s the Pyramdal Hstogram of Orented Gradents (PHOG-8) [6] to represent local shape. Ths conssts of HOG features calculated over ncreasngly fner spatal grds. The orentatons are calculated over the nterval [, 8].We set the number of levels of the pyramd to 4. HOG features calculated for grds wthn the same level of the pyramd are concatenated to form a long feature vector, but feature vectors calculated Sample mages added to Local Intal Tranng Set Buses Fgure 6. Snapshots of the tranng set at 4 tme nstants. Top row shows the ntal tranng set. The next 3 rows show sample mages added to local over tme. The llumnaton change s notceable at each tme nstant. The dataset gets updated wth many obects n smlar poses and representatve bacground patches. at dfferent levels are treated ndependently. Our IMKL algorthm automatcally weghts each level of the pyramd based on the tranng dataset. Hstogram ntersecton s used as the smlarty metrc for all features n ths paper. The frst feature gves rse to 4 ernels, one for each level of the pyramd. The second feature s the PHOG-36. It only dffers from PHOG-8 n that orentatons are calculated over the nterval [, 36]. Ths also gves rse to 4 ernels. The thrd feature, PHOW-Gray [6], encodes appearance. SIFT features are densely sampled at pxel ntervals n each drecton and quantzed to a 3 vsual words vocabulary. Hstograms of vsual words are calculated over an ncreasng number of grds at each pyramdal level. We use 3 levels. The fourth feature s PHOW-Color. The only dfference from PHOW-Gray s that t s calculated on the 3 channels of the HSV mage. These gve rse to 6 ernels. The ffth feature s Geometrc Blur (GB) [3], whch captures shape nformaton of the obects and also accounts for the geometrc dstorton between mages. The un-quantzed GB feature was used wth an expensve correspondence based dstance metrc n [3]. However, n order to speed-up computatons, we quantzed the GB feature to a set of 3 vsual words. We then calculated hstograms of GB words n the same pyramdal framewor to enforce some measure of spatal constrants. We used a 3 level pyramd. Thus we obtaned a total of 7 ernel matrces. 5.. Analyss Evaluaton of MKL. We frst evaluate the power of usng multple ernels and usng MKL to determne ernel weghts for the gven classfcaton tas. For ths purpose, we created a valdaton dataset consstng of mages of buses, cars and bacground extracted from the ground truth as well as the ntal tranng set (obtaned from Google). We then ndvdually evaluated each ernel as well as the combnaton of ernels usng a Sum of Kernels (SoK) approach (such as n []) and an MKL approach for both obect classes over the valdaton set. The SoK approach assgns equal weghts to all ernels. SoK has been nown to provde good results when ernels are carefully chosen for the gven data, but ts performance degrades n the presence

7 Precson Sum of Kernels PHOW Color Geometrc Blur Multple Kernel Learnng Recall Precson Generc Obect Detector Updated Ob Detector SoK Updated Ob Detector OPTIMOL Updated Ob Detector IMKL Recall Precson Generc Obect Detector Updated Ob Detector SoK Updated Ob Detector OPTIMOL Updated Ob Detector IMKL Recall Processng tme (secs) Retranng wth MKL Updatng wth IMKL Number of ncrements (a) Evaluaton of MKL (b) Precson-Recall for buses (c) Precson-Recall for cars (d) Effcency of IMKL Fgure 7. (a) shows the evaluaton of the ndvdual ernels, combnaton usng SoK and combnaton usng MKL. MKL outperforms all other schemes. The best performng ndvdual ernel s GB. (b) and (c) show the Precson-Recall curves for the bus and car classes respectvely. Usng our ncremental obect detector consstently ncreases performance n both cases. (d) compares the processng tme of our ncremental approach to retranng the MKL system at every step usng all avalable mages. of nosy ernels. In our experments, the MKL approach performs better than all other methods where as the SoK approach comes n second, outperformng both GB (the best performng ndvdual feature) and the popular SIFT feature. Fgure 7(a) shows the results for the Buses class. Local dataset snapshots. We now demonstrate results of our IMKL approach on the vdeo dataset. Startng from a generc tranng dataset, our IMKL algorthm smultaneously updates the tranng dataset as well as the ernel combnaton weghts. Fgure 6 shows snapshots of the tranng database at dfferent tme nstants for one vdeo. Kernel weghts over tme. Fgure 8 demonstrates the change of ernel combnaton weghts over tme. For ths experment, we chose a vdeo where the scene s brght n the begnnng but gets very dar by the end. We do not dsplay ernel weghts to 8, snce they do not show consderable change over tme. Tme refers to the ntal tranng dataset obtaned from Google. Between tmes and, we do not update the foreground classes to study the effect of updatng only the bacground tranng set. Ths also causes a non-trval change of weghts (Tme ). After tme, we update all obect classes. Between tmes and 3, the scene s brght. In ths perod, the detector tunes tself towards obects of specfc poses and bacground patches. Beyond tme 3, the scene gets darer. Here, PHOW-Color weghts show a consderable drop (ernels -4), snce color nformaton n the vdeo deterorates, whle PHOW-Gray ernels get hgher weghts. GB at fne spatal resoluton (ernel 7) gets hgh weghts wth decreasng llumnaton, ndcatng added mportance to postonal nformaton (such as mportance gven to the poston of vehcle headlghts). Performance evaluaton. Fgures 7(b) and 7(c) show the performance of our system for the bus and car classes respectvely, averaged over all vdeos n the dataset. We compare our IMKL obect detector wth 3 other detectors. Our baselne detector (whch we call the Generc detector), represents an obect detector bult offlne usng only the generc tranng dataset and s not updated over tme. It uses all 7 ernels and MKL to obtan the ernel weghts. Our second comparson s to an obect detector bult on the generc Tme Instants Illumnaton condtons - Brght scene 3 - Brght scene 4 - Darer scene 5 - Dar scene 3 Intal Set 4 Updatng bacground Updatng bacground +foreground 5 PHOG-Gray PHOW-Color Geom-Blur Features Fgure 8. Kernel combnaton weghts sampled at multple tme nstants. Results shown for Buses class. (see text for detals) Google dataset and updated over tme, but usng SoK (equal ernel weghts). Snce these ernel weghts are fxed over tme, an ncremental SVM approach suffces as the classfer. Our thrd comparson s to OPTIMOL [3], an ncremental model learnng approach, recently proposed for automatc obect dataset collecton. The OPTIMOL algorthm s run ndependently of the IMKL system wth a sngle change. In [3], L et. al use SIFT as ther feature descrptor. But gven the superor performance of GB n our valdaton set (Fgure 7(a)), we use hstograms of GB based vsual words as our feature descrptor for OPTIMOL. Our IMKL approach outperforms the other 3 methods, especally at hgh recalls. Fgure provdes some more nsght nto the results. Ths plot shows the performance of the varous methods over tme for one of the vdeos n our dataset for whch llumnaton changes. The mages at the bottom show a sample frame wthn the specfed tme nterval. OPTIMOL starts off slowly but as t gets updated, t catches up wth the rest of the obect detectors. As the scene gets darer, however, ts performance deterorates. OPTI- MOL uses GB, and even our IMKL approach begns to reduce the mportance gven to ths ernel when the scene be- We obtaned code for OPTIMOL from the authors Feature Weghts

8 Fgure 9. Sample results from a vdeo sequence showng the ablty of our system to adapt to gradual llumnaton changes. comes dar. We also notced a low overall performance of OPTIMOL (on a subset of the data) whle usng other ernels such as PHOW-Gray and PHOW-COLOR. Ths s because no sngle ernel has been able to provde consstently good results n all scene condtons. Usng multple ernels wth fxed weghts (SoK) was also sub-optmal. Our IMKL approach provded the best results because t was able to dynamcally change ernel weghts based on the current obect and scene characterstcs. IMKL s performance decreases at tmes 4 and 6 snce the scene changes, but recovers at nstants 5 and 7, once t updates tself suffcently. Fgure 9 shows sample results. Overall, we detect buses more relably than cars. We are unable to consstently detect cars smaller than 6x6 pxels, whch s the case for cars approachng from a dstance, gvng rse to a number of false negatves. Fnally, Fgure 7(d) llustrates the computatonal effcency of the IMKL algorthm as compared to retranng the entre system usng SmpleMKL. Our approach (IMKL) Generc obect detector Updated detector - SoK Updated detector - OPTIMOL Area under ROC curve Brght llumnaton Dar llumnaton Golden llumnaton 3 4 Tme nstants Fgure. Performance comparson of obect detectors over tme for a sngle vdeo for Buses class. (see text for detals) 6. Concluson We have proposed an Incremental Multple Kernel Learnng (IMKL) approach to obect recognton and demonstrated the performance gans on a vehcle classfcaton tas. Acnowledgements. Ths research was partally supported by the ONR survellance grant N4944. The authors also than Vlad Moraru for useful dscussons. References [] S. Agarwal, A. Awan, and D. Roth. Learnng to detect obects n mages va a sparse, part-based representaton. IEEE Transactons on PAMI, 4. [] F. Bach, G. Lancret, and M. Jordan. Multple ernel learnng, conc dualty, and the smo algorthm. ICML, 4. [3] A. Berg and J. Mal. Geometrc blur for template matchng. CVPR,. [4] A. Blum and T. Mtchell. Combnng labeled and unlabeled data wth co-tranng. COLT: Proceedngs of the Worshop on Computatonal Learnng Theory, 998. [5] A. Bosch, A. Zsserman, and X. Munoz. Image classfcaton usng random forests and ferns. ICCV, 7. [6] A. Bosch, A. Zsserman, and X. Munoz. Representng shape wth a spatal pyramd ernel. CIVR, 7. [7] G. Cauwenberghs and T. Poggo. Incremental and decremental support vector machne learnng. NIPS,. [8] N. Dalal and B. Trggs. Hstograms of orented gradents for human detecton. CVPR, 5. [9] R. Fergus, P. Perona, and A. Zsserman. Obect class recognton by unsupervsed scale-nvarant learnng. CVPR3. [] O. Javed, S. Al, and M. Shah. Onlne detecton and classfcaton of movng obects usng progressvely mprovng detectors. CVPR, 5. [] G. Lancret, N. Crstann, L. El Ghaou, P. Bartlett, and M. Jordan. Learnng the ernel matrx wth sem-defnte programmng. JMLR, 4. [] S. Lazebn, C. Schmd, and J. Ponce. Beyond bags of features: Spatal pyramd matchng for recognzng natural scene categores. CVPR, 6. [3] L. L, G. Wang, and L. Fe-Fe. Optmol: automatc obect pcture collecton va ncremental model learnng. CVPR, 7. [4] M. Pontl and A. Verr. Support vector machnes for 3d obect recognton. IEEE Transactons on PAMI, 998. [5] A. Raotomamony, F. R. Bach, S. Canu, and Y. Grandvalet. More effcency n multple ernel learnng. ICML, 7. [6] J. Svc, B. Russell, A. Efros, A. Zsserman, and W. Freeman. Dscoverng obects and locaton n mages. ICCV, 5. [7] S. Sonnenburg, G. Ra tsch, C. Scha fer, and B. Scho lopf. Large scale multple ernel learnng. JMLR, 6. [8] N. Syed, H. Lu, and K. Sung. Incremental learnng wth support vector machnes. IJCAI, 999. [9] M. Varma and D. Ray. Learnng the dscrmnatve powernvarance trade-off. ICCV, 7. [] B. Wu and R. Nevata. Improvng part based obect detecton by unsupervsed, onlne boostng. CVPR, 7. [] B. Wu and R. Nevata. Smultaneous obect detecton and segmentaton by boostng local shape feature based classfer. CVPR, 7. [] B. Wu and R. Nevata. Optmzng dscrmnaton-effcency tradeoff n ntegratng heterogeneous local features for obect detecton. CVPR, 8. [3] H. Zhang, A. Berg, M. Mare, and J. Mal. Svm-nn: Dscrmnatve nearest neghbor classfcaton for vsual category recognton. CVPR, 6. [4] J. Zhang, M. Marszale, S. Lazebn, and C. Schmd. Local features and ernels for classfcaton of texture and obect categores: A comprehensve study. IJCV, 7.