Object Trajectory Proposal via Hierarchical Volume Grouping

Transcription

1 Objec Trajecory Proposal via Hierarchical Volume Grouping Xu Sun 1, Yuanian Wang 1, Tongwei Ren 1,, Zhi Liu 2, Zheng-Jun Zha 3, and Gangshan Wu 1 1 Sae Key Laboraory for Novel Sofware Technology, Nanjing Universiy, Nanjing, China 2 School of Communicaion and Informaion Engineering, Shanghai Universiy, Shanghai, China 3 Universiy of Science and Technology of China, China sunx@smail.nju.edu.cn,wangy@smail.nju.edu.cn,renw@nju.edu.cn,liuzhi@saff.shu.edu.cn,zhazj@usc.edu.cn, gswu@nju.edu.cn ABSTRACT Objec rajecory proposal aims o locae caegory-independen objec candidaes in videos wih a limied number of rajecories, i.e., bounding box sequences. Mos exising mehods, which derive from combining objec proposal wih racking, canno handle objec rajecory proposal effecively due o he lack of comprehensive objecness measuremen hrough analyzing spaio-emporal characerisics over a whole video. In his paper, we propose a novel objec rajecory proposal mehod using hierarchical volume grouping. Specifically, we firs represen a given video wih hierarchical volumes by mapping hierarchical regions wih opical flow. Then, we filer he shor volumes and background volumes, and combinaorially group he reained volumes ino objec candidaes. Finally, we rank he objec candidaes using a muli-modal fusion scoring mechanism, which incorporaes boh appearance objecness and moion objecness, and generae he bounding boxes of he objec candidaes wih he highes scores as he rajecory proposals. We validaed he proposed mehod on a daase consising of 2 videos from ILSVRC216-VID. The experimenal resuls show ha our mehod is superior o he saeof-he-ar objec rajecory proposal mehods. CCS CONCEPTS Arificial inelligence Compuer vision; Hierarchical represenaions; KEYWORDS Objec rajecory proposal, hierarchical volume represenaion, volume combinaorial grouping, muli-modal fusion scoring ACM Reference Forma: Xu Sun 1, Yuanian Wang 1, Tongwei Ren 1,, Zhi Liu 2, Zheng-Jun Zha 3, and Gangshan Wu Objec Trajecory Proposal via Hierarchical Volume Grouping. In ICMR 18: 218 Inernaional Conference on Mulimedia Rerieval, June 11 14, 218, Yokohama, Japan. ACM, New York, NY, USA, 9 pages. hps://doi.org/1145/ Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. Copyrighs for componens of his work owned by ohers han ACM mus be honored. Absracing wih credi is permied. To copy oherwise, or republish, o pos on servers or o redisribue o liss, requires prior specific permission and/or a fee. Reques permissions from permissions@acm.org. ICMR 18, June 11 14, 218, Yokohama, Japan 218 Associaion for Compuing Machinery. ACM ISBN /18/6... $15. hps://doi.org/1145/ objec proposal + racking ours Figure 1: An example of objec rajecory proposal using objec proposal and racking combinaion and our grouping mehod. 1 INTRODUCTION Objec rajecory proposal aims o locae caegory-independen objec candidaes in videos wih a limied number of bounding box sequences named rajecory [29]. These generaed rajecory proposals provide spaio-emporal characerisics of objec candidaes, which can be exraced and analyzed in numerous mulimedia applicaions, such as objec deecion [17], acion recogniion [34], visual relaion deecion [28], objec segmenaion [35] and conenbased video rerieval [7]. A primary sraegy for objec rajecory proposal is o ake advanage of he advances in objec proposal, which is sudied o locae caegory-independen objec candidaes in images wih a limied number of bounding boxes. Mos exising objec rajecory proposal mehods apply objec proposal on one or several seleced video frames, and rack he generaed bounding boxes on oher frames o generae rajecory proposals [11, 21, 3, 39]. Neverheless, i is inracable o auomaically selec one or few video frames ha conain all he objecs appearing in a given video. As shown in he op row of Figure 1, wo objecs are omied because he objec proposal is only applied on he middle frame bu hese wo objecs do no appear in his frame. As he sae-of-hear objec proposal mehods like [9, 43] and objec deecion mehods like [15, 26] can process images in real-ime, some works apply objec proposal densely or even on all he frames o avoid objec omission. Differen o specific objec deecion, such as pedesrian[2, 15], his kind of rajecory proposal mehods can easily obain excessive candidaes, because he feaures of caegory-independen objecs are much more general han hose in paricular caegories. I will lead o high ime consumpion in objec racking and candidae merging. To make maers worse, hese mehods only measure objecness on video frames wih appearance characerisics, bu ignore he characerisics of objec

2 candidaes on oher modaliies, such as moion. Recenly, Shang e al. [29] improve his sraegy by raversing all he video frames wih bounded compuaional cos and parially incorporaing objec moion in objecness measuremen. However, hey sill conduc objecness measuremen independenly on each frame insead of generaing objec candidaes hrough analyzing spaio-emporal characerisics over he whole video. To overcome hese drawbacks, we learn from he advances in objec proposal on images. There are wo sraegies mainly used in objec proposal: window scoring and grouping. The former samples sufficien bounding boxes and selecs he ones wih high objecness scores as proposals, while he laer segmens images ino regions and merges hem ino proposals. Considering here exis enormous possible rajecories in a video which canno be sufficienly sampled, we choose grouping sraegy for conducing objec rajecory proposal in a whole video. Similar o groupingbased objec proposal mehods, here are hree key issues which should be addressed in grouping-based objec rajecory proposal mehods: Firs, how o define a ype of basic uni ha faciliaes grouping o represen various video conen? Second, how o group he basic unis ino objec candidaes even hey have complex composiions? Third, how o score he objec candidaes wih he spaio-emporal characerisics of he video? In his paper, we propose a novel objec rajecory proposal mehod based on hierarchical volume grouping. Figure 2 shows an overview of he proposed mehod. Given a video, we firs represen i wih hierarchical volumes by mapping hierarchical regions wih opical flow. Then, we filer he shor volumes and he background volumes, and group he reained volumes ino objec candidaes. Finally, we rank he objec candidaes by fusing appearance objecness and moion objecness, and generae he bounding boxes for he objec candidaes wih he highes scores as he rajecory proposals. A grouping-based mehod wih a similar framework was proposed in [12]. However, i has obvious limiaions in addressing he aforemenioned key issues of groupingbased objec rajecory proposal (refer o Secion 2.2) which leads o is unsaisfacory performance (refer o Secion 4.3) as compared o ha of our mehod. We validaed he proposed mehod on a daase consising of 2 videos from ILSVRC216-VID [27]. I shows ha our mehod ouperforms he sae-of-he-ar objec rajecory proposal mehods. Our conribuions mainly include: We propose a grouping-based objec rajecory proposal mehod, which generaes rajecory proposals by analyzing he spaio-emporal characerisics over a whole given video. We presen muliple key echniques, namely hierarchical volume represenaion, volume combinaorial grouping, and muli-modal fusion scoring, which can effecively address he key issues in grouping-based objec rajecory proposal. We consruc a daase wih 2 videos from ILSVRC216- VID o validae he performance of he proposed mehod. The experimenal resuls show ha our mehod is superior o he sae-of-he-ar mehods. 2 RELATED WORK 2.1 Objec proposal The exising objec proposal mehods can be roughly classified ino wo caegories: window scoring-based mehods and grouping-based mehods. Window scoring-based mehods measure he objecness of sufficienly sampled bounding boxes and selec he bounding boxes wih high scores as proposals on RGB images [1, 9, 4, 43] and RGB- D images [19]. Generally speaking, window scoring-based mehods are efficien in objec proposal, bu hey easily fail o generae he proposals wih high Inersecion of Union (IoU) o he groundruhs of objecs. Grouping-based mehods segmen images ino regions and merge hem ino proposals on RGB images [2, 23, 33] and RGB- D images [36]. In conras, grouping-based mehods can generae accurae proposals agains he groundruhs, bu hey usually have low efficiency because of he ime-consuming boom-up merging. 2.2 Objec rajecory proposal Mos exising objec rajecory mehods derive from combining objec proposal wih racking. Some mehods mainly focuses on salien objecs [3, 38] or moving objecs [14, 22], while oher mehods exrac objecs in a unified objecness sraegy [8, 11, 18, 21, 29, 39]. As menioned above, hese mehods canno handle objec rajecory proposal effecively due o he lack of comprehensive objecness measuremen hrough analyzing spaioemporal characerisics over he whole video. Dan e al. [12] firs propose a grouping-based objec rajecory proposal mehod using supervoxel clusering. Though Dan s mehod is similar in overview as compared o our mehod, i has some obvious limiaions in addressing he key issues of grouping-based objec rajecory proposal. Firs, Dan s mehod uses he supervoxels generaed by hierarchical clusering as he basic unis in grouping. These supervoxels on he coarses level may no conain he deailed objec porions because of he inaccuracy of clusering. In conras, our mehod represens he video wih hierarchical volume, which conains all he levels for he subsequen grouping. Second, Dan s mehod only merges he supervoxels wih high similariies in grouping, which canno group he objec pars wih differen appearances. In conras, our mehod uses combinaorial grouping, which can combine he neighboring volumes wih significan differences. Third, Dan s mehod does no provide an effecive scoring sraegy, which direcly reas he clusering resul as he proposals. Our mehod presens a mulimodal fusing scoring mechanism o measure appearance objecness and moion objecness ogeher. 2.3 Objec racking Objec racking aims o esimae he exisence and locaions of arge objecs in he subsequen frames. Numerous objec racking mehods have been proposed. For example, sparse represenaion based mehods uilize local sparse represenaions and collaboraive represenaions o deermine he arge locaions [6, 41]; color hisogram based mehods rack arge objecs via pixel level saisics [1, 25] and edge informaion [24, 32]; discriminaive

3 hierarchical volume generaion volume combinaorial grouping muli-modal fusion scoring L1 L L3 L4 video frame hierarchical volume objec candidae rajecory proposal Figure 2: An overview of he proposed mehod. model based mehods uilize learned binary classifiers o segmen he arge objecs from background [4, 5]. Alhough boh racking and rajecory proposal generae bounding box sequences as heir oupus, hey are significanly differen. Firs, objec racking requires one or several manually labeled bounding boxes on he firs frame for iniializaion while rajecory proposal searches for objec candidaes on all he frames auomaically. Second, racking only searches for similar regions hrough adjacen frames while rajecory proposal requires o measure objecness for rajecory candidaes and eliminaes hose wih low objecness. Third, he number of he rajecories exraced by rajecory proposal is much larger han he number of hose generaed by racking, which means ha rajecory proposal has sricer compuaional cos limiaion in rajecory generaion. 3 OUR METHOD 3.1 Hierarchical volume represenaion Hierarchical volume is he basic uni of combinaorial grouping in our mehod. A volume is a spaio-emporal componen wihin a given video, which is composed of a sequence of ordered regions. In effecive hierarchical volume represenaion, each volume covers he regions wih he same conen on consecuive frames wih coheren boundaries and accurae duraion on differen levels. Such hierarchical volume represenaion reains boh he principal componens and parial deails of objecs, and helps o generae objec candidaes. Hierarchical region represenaion of video frame. To obain he hierarchical volume represenaion of a video, we firs represen each video frame wih hierarchical regions referring o [23]. Specifically, we deec he conours in a given video frame and weigh hem in he value range of [, 1] using ulra-meric conour map [3], in which color, brighness and exure gradien are combined o provide indexed conours for hierarchical regions. Based on hese conours, we obain he fines region represenaion of he video frame. Then, we merge hese regions ieraively by removing he conours whose srenghs are lower han an incremenal hreshold from a se of rained hresholds. The regions surrounded wih he res of closed conours in each ieraion are consruced as he hierarchical regions. To each frame f, assume he number of leaf regions (i.e., he regions on he fines level), is N L and he number of he regions on all he levels is NR. We consruc a binary marix C in size of NR N L o denoe he coverage relaionships beween hierarchical regions and leaf regions. Here, Ci, j equals 1 if he ih hierarchical region covers he jh leaf region; oherwise, i equals. Volume generaion by region mapping. We map he hierarchical regions on adjacen video frames o generae hierarchical volumes. Raher han direcly searching he mos similar region for each region in he subsequen frame, which is very ime consuming, we uilize opical flow o esablish he mapping relaionships beween regions on adjacen frames, because opical flow esimaion is only required o conduc once for a whole frame and i can be used for he mapping of all he regions in he frame. We firs esimae bidirecional dense opical flow using Deepflow [37], which effecively handles large displacemens in videos via dense correspondences maching. To wo adjacen video frames f and f +1, we consruc wo marices O +1 in size of NL NL +1 and O +1 in size of NL +1 NL o denoe he mapping relaionships beween he leaf regions in f and f +1. Specifically, Oi, +1 j denoes he number of he pixels in he ih leaf region in f whose corresponding pixels mapped wih he opical flow from f o f +1 belong o he jh leaf region in f +1 : O +1 i, j = P +1 i P +1 j, (1) where Pi +1 is he se of he pixels in f +1 which are mapped wih opical flow from he pixels in he ih leaf region in f ; Pj +1 denoes he se of he pixels belonging o he jh leaf region in f +1 ; denoes he cardinaliy of a se, i.e., he number of pixels wihin a se. The elemens in O +1 are defined in a similar manner. Based on he region coverage marix and he leaf region mapping marix, we map he hierarchical regions beween he adjacen frames. To f and f +1, we consruc wo marices M +1 in size of NR N R +1 and M +1 in size of NR +1 NR o denoe he mapping relaionships beween he hierarchical regions in f and

4 f +1 : M +1 = C O +1 (C +1 ) T, (2) M +1 = C +1 O +1 (C ) T. (3) where each elemen M +1 i, j in M +1 denoes he number of he pixels in he ih region in f whose corresponding pixels mapped wih he opical flow from f o f +1 belong o he jh region in f +1. The elemens in M +1 are defined in a similar manner. We normalize he elemens in M +1 o consruc a marix Ψ +1 as follows: Ψi, +1 j = Mi, +1 j / Qi, (4) where Qi denoes he se of he pixels belong o he ih hierarchical region in f ; denoes he cardinaliy of a se. We can consruc a marix Ψ +1 from M +1 in a similar manner. To each hierarchical region in f, we search is mapping region in f +1 by checking he elemens in he ih row in Ψ +1 and he ih column in Ψ +1. If boh Ψi, +1 j and Ψ +1 j,i are larger han a predefined hreshold, which equals in our experimens, we consider he ih region in f is mapped o he jh region in f +1 successfully and merge hem ino a volume. If a region in f is mapped o more han one regions in f +1, we selec he one wih he highes mapping score, i.e., he sum of Ψi, +1 j and Ψ +1 j,i. We repea he above procedure from he firs frame o he las one, and generae hierarchical volumes by merging he mapped regions. Noe here, we map he hierarchical regions and reain he volumes on all he levels for he subsequen grouping, i.e., one volume generaed by our mehod may conain anoher volume or is porions. In conras, supervoxels in [12] are generaed by hierarchical clusering of single level superpixels, which only consis of he coarses regions and have no overlap beween each oher. There are wo advanages of our hierarchical volume represenaion as compared o hierarchical supervoxel in [12]. On he one hand, i is difficul o find appropriae relaionships when direcly mapping he regions in he adjacen frames on superpixel level. In fac, we canno generae he volumes wih long lenghs (he lengh of a volume denoes he number of he regions belonging o he volume) by mapping he leaf regions in he experimens unless we relax he mapping hreshold o a small value. I leads o he generaed supervoxels wih low cohesion and prevens he effeciveness of he supervoxel represenaion in [12]. On he oher hand, boh he generaion of hierarchical volume and supervoxel are only based on low-level feaures, which ineviably brings in inaccuracy. If only using he supervoxels composed of he coarses regions [12], he deailed objec porions los in clusering canno be obained in grouping. I will furher cause he inaccuracy of generaing rajecory proposals. As comparison, our mehod generaes he volumes in differen levels and reains mos of hem, which will provide more comprehensive candidaes in grouping. Broken volume connecion. Our hierarchical volume represenaion maps he regions based on opical flow, bu opical flow only represens he low-level similariy beween pixels in adjacen frames. I is easy o generae broken volumes because opical flow is sensiive o parial occlusion and illuminaion variaion. These broken volumes increase he complexiy of grouping hem ino he objec candidaes. Hence, we connec he volumes, which may represen he same conen, based on heir high-level appearance similariies. Assume v i and v j are wo volumes, and v i ends before v j sars. We rack he bounding box of he region, which belongs o v i in is end frame, using kernelized correlaion filer racker [16]. If he racked bounding box in he sar frame of v j has large enough IoU o he bounding box of he region belonging o v j in he same frame, we consider ha v i and v j can be conneced. In our implemenaion, we filer he volumes wih exremely shor lenghs for efficiency, which are less han 1 frames in our experimens, and sor all he reained volumes ino wo queues q s and q e based on heir sar frames and end frames. To each volumev in q e, we check wheher exis volumes saring in a shor emporal inerval afer v ends, which equals 4 frames in our experimens. If so, we greedily connec v o hese volumes by racking, i.e., we only connec a volume o he firs volume which can be conneced o i. The IoU hreshold for connecion in our experimens is se o. Because he number of volumes ha sar in a shor emporal inerval afer each volume can be reaed as a consan and only once raversal is required for q e, he ime complexiy of volume connec is O(N V log N V ), here N V is he number of volumes afer filering. 3.2 Volume combinaorial grouping Adapive shor volume filering. Considering mos objecs are visible for relaively long duraions in videos, he volumes wih shor lenghs are usually useless for generaing objec candidaes. Preliminary filering of such shor volumes can decrease he number of volumes and effecively reduce he compuaional complexiy of volume grouping. Considering he variaion of video lenghs, we use an adapive hreshold in shor volume filering. On he one hand, we consider ha he volumes conribuive o objec candidae generaion should be visible for more han a cerain duraion, for example, one second (2 frames) in our experimens. On he oher hand, o avoid oo sric filering for shor videos, we require ha he lengh of he filered volumes should be no larger han a cerain raio of he lengh of he whole video, such as 2% in our experimens. Hence, he final hreshold κ for shor volume filering is calculaed as: κ = min(2, V ), (5) where V denoes he lengh of he whole video. Background volume eliminaion. The volumes conaining background conen usually have long lenghs because he sabiliy in background appearance helps o region mapping in hierarchical volume generaion. These background volumes canno be removed in shor volume filering, bu hey hamper rajecory proposal because hey are easy o be grouped wih oher volumes in combinaorial grouping and generae many objec candidaes covering no objecs. To eliminae he background volumes, we calculae he boundary conneciviy of each volume based on ha of he regions belonging o his volume. Assume v i is a volume composed of a region sequence {r k i } k=1,, vi. Here, v i sars in frame f 1 and ends in frame f v i, and v i denoes he lengh of v i. To each region

5 r k i, we calculae is boundary conneciviy B(r k i ) by measuring he exen of is connecion o frame boundary [42] as follows: B(r k i ) = C(r k i ), (6) r k i where C(r k i ) denoes he se of pixels which are on boh he boundaries of r k i and frame f k ; denoes he cardinaliy of a se, i.e., he number of pixels wihin a se or a region. Based on he boundary conneciviy of all he regions belonging o v i, we calculae he boundary conneciviy of v i as follows: B(v i ) = {r k i B(r k i ) > θ, k = 1,, v i }, (7) v i where θ is a boundary conneciviy hreshold, which equals 1 in our experimens referring o [42]. To avoid eliminaing he volumes covering he objecs ha appear in frame boundaries, we use a high hreshold in background volume eliminaion. In our experimens, only he volumes whose boundary conneciviy is larger han.9 will be eliminaed as background. Candidae generaion by volume grouping. Based on he reained volumes, we generae he objec candidaes by grouping hese volumes. Mos exiing mehods conduc grouping based on volume similariy, such as [12]. However, such grouping sraegies usually fail in generaing objec candidaes effecively, because he pars of an objec usually have significan differences in appearance and/or moion. Inspired by [23], we adop a combinaorial grouping sraegy in volume grouping, which only measures he neighborhood relaionships beween volumes while aking no accoun of volume similariy. Considering he neighborhood relaionship beween wo volumes is more complex han ha beween wo regions, e.g., wo volumes may be overlapping, adjacen and disjoined in differen frames, we relax he combinaion condiion in volume grouping from adjacen in [23] o overlapping or adjacen, in order o generae sufficien objec candidaes. Specifically, we firs consruc a binary marix A in size of NL N L o represen he neighborhood relaionship beween he leaf regions in frame f, in which Ai, j equals 1 if he ih leaf region and he jh leaf region in f are adjacen and oherwise. Then, we represen he neighborhood relaionships beween he hierarchical regions in f by a marix Λ in size of NR N R : Λ = C A (C ) T, (8) where C is he coverage relaionship marix defined in Secion 3.1. Here, Λi, j is larger han if he ih region and he jh region in f are overlapping or adjacen, and Λi, j equals if hey are disjoined. Assuming ha wo volumes v i and v j appear in a video wih he common duraion from f 1 o f K, we measure he neighborhood relaionship beween hese wo volumes as follows: Θ(v i,v j ) = K =1 η(r i, r j ) min( v i, v j ), (9) where r i and r j are he regions belonging o v i and v j in f, respecively; η(, ) equals 1 if wo regions are overlapping or adjacen in f and oherwise, which can refer o Λ in Eq. (8); denoes he lengh of a volume. If Θ(v i,v j ) is larger han a hreshold, which equals in our experimens, v i and v j are allowed o group. Based on he consrain in Eq. (9), we group he volumes ino he candidaes conaining from one o four volumes. Similar o [23], we consrain he number of objec candidaes using Pareo fron opimizaion [13] o avoid he combinaorial explosion. In our experimens, volume grouping will erminae if all volume combinaions are exhaused or he number of objec candidaes reaches 1,. 3.3 Muli-modal fusion scoring We measure he objecness of he objec candidaes, and selec he ones wih high scores o generae he final rajecory proposal. In objecness measuremen, we analyze boh appearance objecness and moion objecness of each candidae. The former has been widely used in image proposal [9, 43], and he laer has been validaed is effeciveness in rajecory proposal [29]. As for appearance objecness, we adop he scoring model provided in [23], which is a regressor rained on muli-modal feaures including shape, edges, size and locaion. To each candidae c k, which appears from frame f 1 o f K, we measure he appearance objecness of he grouped regions belonging o c k on each frame, and calculae he appearance objecness score of c k as follows: s A (c k ) = K =1 s A (д k ), (1) c k where s A ( ) denoes appearance objecness score; д k denoes he grouped regions belonging o c k in frame f ; denoes he lengh of a candidae, i.e., he number of he frames ha he candidae appears in. As for moion objecness, we measure he moion conras of an objec candidae o background. We calculae he average moion srengh of all he regions belonging o background volumes (refer o Secion 3.2) on each frame as he moion srengh of background. If no background volume is deeced in some frame, we use he average moion srengh of he whole frame o insead. To each candidae c k, which appears from frame f 1 o f K, we calculae he disance beween he average moion srengh of he grouped regions belonging o c k and ha of background, and calculae he moion objecness score of c k as follows: s M (c k ) = K =1 m k m B, (11) c k where s M (.) denoes moion objecness score; m denoes he k average moion srengh of he grouped regions belonging o c k in frame f ; m B denoes he average moion srengh of background in frame f. We normalize he appearance objecness score and he moion objecness score of each candidae c k o he value range of [, 1], and fuse hem wih linear combinaion as he oal objecness score of c k : s(c k ) = λ s A (c k ) + (1 λ) s M (c k ), (12) where λ is a weigh parameer, which equals o in our experimens o handle he inaccuracy of moion conras in moion objecness measuremen.

6 We rank all he candidaes according o heir objecness scores, generae he bounding boxes of he op seleced candidaes as he rajecory proposals, and preferenially reurn he proposals composed wih less volumes. 4 EXPERIMENTS 4.1 Daase and experimen seings We validaed he performance of he proposed mehod on a daase consising of 2 videos, which are randomly seleced from ILSVRC216-VID [27]. These videos have various conen and conain 3.26 objecs on average. The average objec number of our daase is larger han he one of he whole ILSVRC216-VID daase, which is only 2.6. The manually labeled objec rajecories in ILSVRC216-VID were used as he groundruhs in our experimens. To he bes of our knowledge, i is he larges daase for objec rajecory proposal. We used he evaluaion crieria proposed in [29], namely mean Trajecory IoU () and recall, on he op 5 rajecory proposals for each video. These crieria are defined as follows: mt -IoU = 1 N V recall = 1 N V N V N V 1 N k G k=1 N G k j=1 1 N k G max {T -IoU i, j }, (13) i k=1 N G k δ ( max {T -IoU i, j } τ ), (14) i j=1 where T -IoU i, j denoes he rajecory IoU beween a rajecory proposal T i and a rajecory groundruh G j ; N V is he number of videos; NG k is he number of groundruhs in he kh video; δ is a funcion ha oupus 1 if he inpu is posiive, and oherwise; τ is a hreshold, which equals o in our experimens. All he experimens were conduced on a compuer wih i7 3.5GHz CPU and 32GB memory. For all he mehods in comparison, we used heir defaul seings suggesed by he auhors. 4.2 Componen analysis There are several key echniques incorporaed in our mehod: region mapping, volume represenaion, volume filering and candidae scoring. We validaed heir influences o he performance of our mehod. Region mapping. We use bidirecional opical flow in region mapping o consruc he mapping relaionship beween he regions in adjacen frames (refer o Secion 3.1), which has high compuaional cos. To evaluae he necessiy of bidirecional opical flow, we generae a baseline only using forward opical flow (FOF), which requires half compuaional cos in opical flow esimaion as compared o our mehod. Figure 3 shows he performance of our mehod using differen opical flow in region mapping. We can see ha he performance declines on boh and recall when replacing bidirecional opical flow wih forward opical flow. The reason is ha opical flow esimaion only focuses on pixel-level similariy measuremen even using global consisency consrain, which is sensiive o variable video conen. Once using bidirecional opical flow, we can diminish he influence of opical flow inaccuracy and map he regions more effecively..8 FOF FOF Figure 3: Evaluaion of our mehod using differen opical flow in region mapping on and recall..8 FLV CLV FLV CLV Figure 4: Evaluaion of our mehod using differen volume represenaion on and recall. Volume represenaion. We represen video conen wih hierarchical volumes on all he levels, i.e., one volume may conain anoher volume or is porions (refer o Secion 3.1). Such a volume represenaion can represen boh he principal componens and he deails of objecs. To illusrae he superioriy of our hierarchical volume represenaion, we generae wo baselines: using he fines level volumes (FLV), which are generaed by he fines level regions on each video frame, and using he coarses level volumes (CLV), which merges he regions on each video frame unil he number of regions is no more han 3% of he one of he fines regions and maps hese regions o generae he volumes. Figure 4 shows he performance of our mehod using differen volume represenaions. We can see ha boh FLV and CLV have dramaic drops on and recall. For example, he and recall values obained by CLV on 5 proposals are only half of he ones obained by our mehod. I shows ha only he coarses level volumes, which represen he principal componens of objecs, canno effecively suppor rajecory proposal because all he objec deails are los. Moreover, FLV has worse performance han CLV, which is caused by wo reasons: 1) The leaf regions are difficul o be mapped ino long-duraion fines level volumes due o he inaccuracy of opical flow, which increases he complexiy of volume grouping. 2) Our mehod only groups a mos four volumes ino a candidae o avoid combinaorial explosion, which leads o he candidae grouped by he fines level volumes are usually seriously incomplee agains objecs. I validaes he effeciveness of hierarchical volume for video conen represenaion.

7 .8 NoF SF BF.8 NoF SF BF.8 AO MO.8 AO MO Figure 5: Evaluaion of our mehod using differen volume filering sraegies on and recall. Figure 6: Evaluaion of our mehod using differen candidae scoring mechanisms on and recall. Volume filering. We consider he shor volumes and he background volumes are useless in objec represenaion, and filer hem o reduce he complexiy of volume grouping (refer o Secion 3.2). To illusrae he necessiy of volume filering, we generae hree baselines: wihou shor volume filering or background volume filering (NoF), only filering shor volumes (SF), and only filering background volumes (BF). Figure 5 shows he performance of our mehod using differen volume filering sraegies. We can see ha he performance of NoF is worse han ha using oher volume filering sraegies. Noe here, in broken volume connecion (refer o Secion 3.1), we have filered he volumes whose lenghs are less han 1 frames for efficiency. I means ha NoF acually uses he volume filering sraegy by removing he exremely shor volumes (less han 1 frames) raher han filers nohing. Once increasing he filering hreshold on volume lengh from 1 frames o 2 frames, he performance of SF is improved significanly as compared o NoF. I shows he effeciveness of shor volume filering. Moreover, we can see ha he performance will be improved when filering background volumes, no maer on NoF (i.e., BF) and on SF (i.e., ). I shows ha background volume eliminaion is beneficial o rajecory proposal. Candidae scoring mechanism. We use a muli-modal fusion scoring mechanism in ranking objec candidaes, which fuses boh appearance objecness and moion objecness (refer o Secion 3.3). To validae he effeciveness of our muli-modal fusion scoring mechanism, we generae wo baselines: scoring wih appearance objecness (AO), and scoring wih moion objecness (MO). Figure 6 shows he performance of our mehod using differen scoring sraegies. We can see ha he performance of boh AO and MO is significanly worse han ha of our mehod on and recall, which illusraes he effeciveness of fusing appearance objecness and moion objecness in scoring. Moreover, he performance of MO is only slighly worse han ha of AO. Because objecs usually have differen moions agains background, moion objecness conribues o rajecory proposal especially when objec appearance is complex. 4.3 Comparison wih he sae-of-he-ars To validae he performance of our mehod, we compared i wih hree sae-of-he-ar video objec proposal mehods, namely free objec discovery (FOD) [11], objec rajecory proposal (OTP) [29], and spaio-emporal objec deecion proposal (SODP) [12]. In he.8 FOD OTP SODP EB* MCG* FOD OTP SODP EB* MCG* Figure 7: Evaluaion of differen rajecory proposal mehods on and recall. same way as [29], we generae wo baselines EB* and MCG*, which generae objec proposals on he middle frame of each video and rack he proposals wih KCF racker [16] bidirecionally o generae rajecory proposals. Figure 7 shows he performance of our mehod and five compared mehods. We have: 1) Our mehod is superior o all he compared mehods on boh and recall, which illusraes he effeciveness of our mehod. 2) Though SODP has a similar framework o our mehod, is performance is dramaically worse han ha of our mehod and oher mehods. I shows he effeciveness of he key echniques in our mehods, namely hierarchical volume represenaion, volume combinaorial grouping, and muli-modal fusion scoring. 3) The performance of EB* and MCG* is worse han oher mehods excep SODP, hough EB and MCG are he bes objec proposal mehods on images. I is because hey canno propose he objec rajecories which do no appear on he middle frames of videos (or any oher seleced frames as he sar of rajecory generaion). I illusraes he drawbacks of objec rajecory proposal by combining image objec proposal and racking. 4) Though FOD and OTP aim o address he sar frame problem by raversing all he video frames, heir performance is sill worse han ours. I is because hey canno effecively measure objecness based on spaio-emporal characerisics, and propose objecs independenly on each frame. Figure 8 illusraes several examples of our resuls. In each video example, he rajecory proposal wih he highes o each rajecory groundruh from he op 5 proposals is shown. I

8 Figure 8: Qualiaive examples of objec rajecory proposal using our mehod. In each video example, he bounding boxes wih he same color indicae a rajecory proposal, and he rajecory proposal wih he highes o each rajecory groundruh from he op 5 proposals is shown. Table 1: Comparison wih he sae-of-he-ar mehods and he baselines on ime cos per frame. Mehod Language Time (s) FOD C++ & Pyhon 3.4 OTP C++ & Pyhon 3.4 SODP C++ & Malab & Pyhon 6.7 EB* C++ & Malab 5.5 MCG* C++ & Malab 3.5 C++ & Malab 7.5 shows ha our mehod can handle he videos wih muliple objecs in various conen. We also validaed he efficiency of our mehod. Table 1 shows he ime cos per frame of our mehod and he compared mehods. I shows ha our mehod is comparable o oher grouping-based objec rajecory proposal mehods, such as SODP. The mos ime consuming componen in our mehod is bidirecional opical flow esimaion. I can be acceleraed by GPU [31] o make our mehod more efficien. 4.4 Discussion In he experimens, we found some limiaions of our mehod. As shown in he op row of Figure 9, one goa is omied because i is occluded by wo oher goas wih similar appearances, which leads o serious inaccuracy in volume represenaion. Anoher failure example shown in he boom row of Figure 9. There are many moorcycles and cars wih small sizes are omied. I is because hese small objecs are composed of muliple small volumes, which are easily filered in volume grouping. Moreover, as shown in Table 1, he ime coss of our mehod and oher exising rajecory proposal mehods are sill high, which prevens heir applicaion in real-ime condiions. 5 CONCLUSION In his paper, we proposed an objec rajecory proposal mehod based on hierarchical volume grouping. Three echniques, namely hierarchical volume represenaion, volume combinaorial grouping and muli-modal fusion scoring, were presened o address he key Figure 9: Our failure examples influenced by objec occlusion and small objecs. issues in grouping-based objec rajecory proposal. Benefiing from measuring objecness hrough analyzing spaio-emporal characerisics over a whole video, our mehod can effecively generae he rajecory proposals on he videos wih muliple objecs in various conen. We consruced a daase consising of 2 videos from ILSVRC216-VID daase. The experimenal resuls on he daase show ha our mehod ouperforms he saeof-he-ar objec rajecory proposal mehods. 6 ACKNOWLEDGEMENTS This work is suppored by Naional Science Foundaion of China ( , , ), and Collaboraive Innovaion Cener of Novel Sofware Technology and Indusrializaion. REFERENCES [1] Bogdan Alexe, Thomas Deselaers, and Viorio Ferrari Measuring he Objecness of Image Windows. IEEE Transacions on Paern Analysis & Machine Inelligence 34 (212), [2] Mykhaylo Andriluka, Sefan Roh, and Bern Schiele. 28. People-rackingby-deecion and people-deecion-by-racking. In Compuer Vision and Paern Recogniion, 28. CVPR 28. IEEE Conference on [3] P Arbelaez. 26. Boundary Exracion in Naural Images Using Ulrameric Conour Maps. In Compuer Vision and Paern Recogniion Workshop, 26. CVPRW 6. Conference on [4] Shai Avidan. 27. Ensemble Tracking. IEEE Transacions on Paern Analysis & Machine Inelligence 29, 2 (27), [5] Boris Babenko, Ming Hsuan Yang, and Serge Belongie Robus Objec Tracking wih Online Muliple Insance Learning. IEEE Transacions on Paern Analysis & Machine Inelligence 33, 8 (211), 1619.

9 [6] Chenglong Bao, Yi Wu, Haibin Ling, and Hui Ji Real ime robus L1 racker using acceleraed proximal gradien approach. In IEEE Conference on Compuer Vision and Paern Recogniion [7] Rachid Benmokhar and Benoi Hue. 27. Muli-level Fusion for Semanic Video Conen Indexing and Rerieval.. In Inernaional Workshop on Adapive Mulimedia Rerieval [8] Michael Van Den Bergh, Gemma Roig, Xavier Boix, Saniago Manen, and Luc Van Gool Online Video SEEDS for Temporal Window Objecness. In IEEE Inernaional Conference on Compuer Vision [9] Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip H. S. Torr BING: Binarized Normed Gradiens for Objecness Esimaion a 3fps. In IEEE Conference on Compuer Vision and Paern Recogniion [1] Dorin Comaniciu, Visvanahan Ramesh, and Peer Meer. 23. Kernel-Based Objec Tracking. IEEE Transacions on Paern Analysis & Machine Inelligence 25, 5 (23), [11] Giovanni Cuffaro, Federico Becaini, Claudio Baecchi, Lorenzo Seidenari, and Albero Del Bimbo Segmenaion Free Objec Discovery in Video. In European Conference on Compuer Vision workshops [12] Oneaa Dan, Jerome Revaud, Jakob Verbeek, and Cordelia Schmid Spaioemporal Objec Deecion Proposals. In European Conference on Compuer Vision [13] Mahias Ehrgo. 27. Mulicrieria Opimizaion. Springer-Verlag New York, Inc pages. [14] Kaerina Fragkiadaki, Pablo Arbelaez, Panna Felsen, and Jiendra Malik Learning o segmen moving objecs in videos. In IEEE Conference on Compuer Vision and Paern Recogniion [15] L. Van Gool, M. Mahias, R. Timofe, and R. Benenson Pedesrian deecion a 1 frames per second. In Compuer Vision and Paern Recogniion [16] JoÃčo F. Henriques, Caseiro Rui, Pedro Marins, and Jorge Baisa High- Speed Tracking wih Kernelized Correlaion Filers. IEEE Transacions on Paern Analysis & Machine Inelligence 37, 3 (215), [17] Kai Kang, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang Objec Deecion from Video Tubeles wih Convoluional Neural Neworks. In IEEE Conference on Compuer Vision and Paern Recogniion [18] Jianwu Li, Tianfei Zhou, and Yao Lu Learning o generae video objec segmen proposals. In IEEE Inernaional Conference on Mulimedia and Expo [19] Jing Liu, Tongwei Ren, Yuanian Wang, Sheng Hua Zhong, Jia Bei, and Shengchao Chen Objec proposal on RGB-D images via elasic edge boxes. Neurocompuing 236 (216). [2] Saniago Manen, Mahieu Guillaumin, and Luc J. Van Gool Prime Objec Proposals wih Randomized Prim s Algorihm. In IEEE Inernaional Conference on Compuer Vision [21] Liang Peng and Xiaojun Qi Temporal objecness: Model-free learning of objec proposals in video. In IEEE Inernaional Conference on Image Processing [22] Federico Perazzi, Oliver Wang, Markus Gross, and Alexander Sorkine-Hornung Fully Conneced Objec Proposals for Video Segmenaion. In IEEE Inernaional Conference on Compuer Vision [23] Jordi Ponuse, Pablo Arbelaez, Jonahan Barron, Ferran Marques, and Jiendra Malik Muliscale Combinaorial Grouping for Image Segmenaion and Objec Proposal Generaion. IEEE Transacions on Paern Analysis & Machine Inelligence 39, 1 (216), [24] Faih Porikli. 25. Inegral Hisogram: A Fas Way To Exrac Hisograms in Caresian Spaces. In IEEE Conference on Compuer Vision and Paern Recogniion [25] P. PÃľrez, C. Hue, J. Vermaak, and M. Gangne. 22. Color-Based Probabilisic Tracking. In European Conference on Compuer Vision [26] Joseph Redmon and Ali Farhadi YOLO9: Beer, Faser, Sronger. (216), [27] Olga Russakovsky, Jia Deng, Hao Su, Jonahan Krause, Sanjeev Saheesh, Sean Ma, Zhiheng Huang, Andrej Karpahy, Adiya Khosla, and Michael Bernsein ImageNe Large Scale Visual Recogniion Challenge. Inernaional Journal of Compuer Vision 115, 3 (215), [28] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Ta-Seng Chua Video Visual Relaion Deecion. In ACM Inernaional Conference on Mulimedia. Mounain View, CA USA. [29] Xindi Shang, Tongwei Ren, Hanwang Zhang, Gangshan Wu, and Ta Seng Chua Objec rajecory proposal. In IEEE Inernaional Conference on Mulimedia and Expo [3] Gilad Sharir and Tinne Tuyelaars Video objec proposals. In IEEE Conference on Compuer Vision and Paern Recogniion Workshops [31] Narayanan Sundaram, Thomas Brox, and Kur Keuzer. 21. Dense poin rajecories by GPU-acceleraed large displacemen opical flow. In European Conference on Compuer Vision [32] Feng Tang, S Brennan, Qi Zhao, and Hai Tao. 27. Co-Tracking Using Semi-Supervised Suppor Vecor Machines. In IEEE Inernaional Conference on Compuer Vision [33] Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arnold W. M. Smeulders Selecive Search for Objec Recogniion. Inernaional Journal of Compuer Vision 14, 2 (213), [34] Heng Wang, A. Klaser, C. Schmid, and Cheng Lin Liu Acion recogniion by dense rajecories. In IEEE Conference on Compuer Vision and Paern Recogniion [35] Wenguan Wang, Jianbing Shen, and F Porikli Saliency-aware geodesic video objec segmenaion. In IEEE Conference on Compuer Vision and Paern Recogniion [36] Yuanian Wang, Lei Huang, Tongwei Ren, Sheng-Hua Zhong, Yan Liu, and Gangshan Wu Objec Proposal via Deph Conneciviy Consrained Grouping. In Pacific-Rim Conference on Mulimedia [37] Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid DeepFlow: Large Displacemen Opical Flow wih Deep Maching. In IEEE Inernaional Conference on Compuer Vision [38] Fanyi Xiao and Jae Lee Yong Track and Segmen: An Ieraive Unsupervised Approach for Video Objec Proposals. In IEEE Conference on Compuer Vision and Paern Recogniion [39] Jae Lee Yong, Jaechul Kim, and Krisen Grauman Key-segmens for video objec segmenaion. In Inernaional Conference on Compuer Vision [4] Ziming Zhang, Yun Liu, Xi Chen, Yanjun Zhu, Ming Ming Cheng, Venkaesh Saligrama, and Philip H. S. Torr Sequenial Opimizaion for Efficien High-Qualiy Objec Proposal Generaion. IEEE Transacions on Paern Analysis & Machine Inelligence PP, 99 (215), 1 1. [41] W. Zhong, H. Lu, and M. H. Yang Robus objec racking via sparse collaboraive appearance model. IEEE Transacions onimage Processing 23, 5 (214), [42] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun Saliency Opimizaion from Robus Background Deecion. In Compuer Vision and Paern Recogniion [43] C. Lawrence Zinick and Pior Dollár Edge Boxes: Locaing Objec Proposals from Edges. In European Conference on Compuer Vision