Reducing Frame Rate for Object Tracking

Transcription

1 Reducng Frame Rate for Object Trackng Pavel Korshunov 1 and We Tsang Oo 2 1 Natonal Unversty of Sngapore, Sngapore 11977, pavelkor@comp.nus.edu.sg 2 Natonal Unversty of Sngapore, Sngapore 11977, oowt@comp.nus.edu.sg Abstract. Object trackng s commonly used n vdeo survellance, but typcally vdeo wth full frame rate s sent. We prevously have shown that full frame rate s not needed, but t s unclear what the approprate frame rate to send or whether we can further reduce the frame rate. Ths paper answers these questons for two commonly used object trackng algorthms (frame-dfferencng-based blob trackng and CAMSHIFT trackng). The paper provdes () an analytcal framework to determne the crtcal frame rate to send a vdeo for these algorthms wthout them losng the tracked object, gven addtonal knowledge about the object and key desgn elements of the algorthms, and () answers the questons of how we can modfy the object trackng to further reduce the crtcal frame rate. Our results show that we can reduce the 3 fps rate by up to 7 tmes for blob trackng n the scenaro of a sngle car movng across the camera vew, and by up to 13 tmes for CAMSHIFT trackng n the scenaro of a face movng n dfferent drectons. 1 Introducton Object trackng s a common operaton n vdeo survellance systems. However, gven an object trackng algorthm, t s unclear what frame rate s necessary to send. Typcally, vdeo s sent at the rate of full vdeo camera capacty, whch may not be the best opton f network bandwdth s lmted. Prevously, we have shown n [1] that frame rate can be sgnfcantly reduced wthout object trackng losng the object. We found that the crtcal frame rate for a gven algorthm depends on the speed of tracked object. The smple way to determne the crtcal frame rate s to run algorthm on a partcular vdeo sequence, droppng frames and notcng whch rate causes the algorthm to lose the object. Such approach however s not practcal, because objects n real survellance vdeos move wth dfferent speeds, and the crtcal frame rate therefore should depend on ths parameter. We suggest fndng crtcal frame rate usng analyss based on the algorthm s key desgn elements (specfc object detecton and trackng mechansm) and measured speed and sze of the tracked object. In ths paper, we focus on two trackng algorthms, blob trackng algorthm that reles on frame dfferencng and foreground object detecton by L et al. [2] as well as Kalman flter for trackng; and CAMSHIFT algorthm [3], n whch objects are represented as color hstograms, and trackng s performed usng mean

2 j Fg. 1. Droppng out of + j frames. s the drop gap. shft algorthm. We present an analytcal framework formalzng the dependency between vdeo frame rate and algorthms accuracy. We estmate crtcal frame rate usng analyss wth assumpton of known speed and sze of the tracked object. Guded by the estmaton, we slghtly modfy these trackng algorthms makng them adaptve and more tolerant to the vdeos wth even lower frame rate. In Secton 2, we present analyss of the crtcal frame rate for object trackng. In Secton 3, we demonstrate how the dependency between frame rate and accuracy can be estmated specfcally for the blob trackng. We also specfy the crtcal frame rate for ths algorthm. In Secton 4, we present smlar analyss for CAMSHIFT trackng. In Secton, we show how, usng our estmatons and measurement of speed and sze of the tracked object, we can modfy these trackng algorthms adaptng them to the reduced frame rate. Secton 6 ends the paper wth concluson and future works. 2 General Analyss We degrade temporal vdeo qualty by applyng the droppng pattern drop frames out of +j frames, where s drop gap,andj s the number of consecutve remanng frames (see Fgure 1). Note that the same frame rate can correspond to two dfferent droppng patterns, for nstance, droppng 2 out of 3 frames results n the same frame rate as droppng 4 out of 6 frames. The reason for choosng such droppng pattern s because we found that drop gap s more mportant factor for the performance of the trackng than smply a frame rate. Therefore, nstead of crtcal frame rate, we focus on fndng crtcal drop gap, whch would determne the correspondng frame rate. Frst, we present an estmaton of the crtcal drop gap for an object trackng algorthm wthout takng nto account the specfc method of detecton and trackng. For smplcty, consder a vdeo contanng a sngle movng object, whch can be accurately tracked by the algorthm. We can notce that droppng frames affects the speed of object. Snce vdeo s a sequence of dscrete frames, the speed of object can be understood as a dstance between the centers of object postons n two consecutve frames, whch we call nter-frame speed denoted as Δd. Wthout loss of generalty, we can say that for every object trackng algorthm there exsts a Δ d such that, f object moves for a larger dstance than Δ d, the algorthm loses t.

3 Δx orgnal vdeo drop gap = 1 Fg. 2. The schema of the dfference between object foreground detecton for orgnal vdeo and for vdeo wth dropped frames. Let Δd be the maxmal nter-frame speed of the object n the orgnal vdeo, when no frame droppng s appled yet. If we drop frames wth drop gap =1, the new maxmum nter-frame speed can be approxmated as Δd 1 = 2Δd. Then, for general frame droppng pattern, Δd =( +1)Δd. Assume we know the orgnal speed of the object and the algorthm s threshold Δ d. Then, we can compute the maxmum number of consecutve frames that can be dropped,.e., crtcal drop gap ĩ, as ĩ = Δ d Δd 1. (1) 3 Blob Trackng Algorthm For blob trackng algorthm, due to frame dfferencng detecton, the value ĩ depends on the sze and the speed of tracked object. If too many consecutve frames are dropped, the object n the current frame appear so far away from ts locaton n the prevous frame that the frame dfferencng operaton results n detectng two separate blobs (see Fgure 3(b)). Such trackng falure occurs when the dstance between blob detected n the prevous frame and blob n the current frame s larger than the sze of the object tself. Therefore, ths dstance s the threshold dstance Δ d. To determne ts value, we need to estmate the coordnates of the blob center n the current frame, whch depend on ts locaton and sze n the prevous frame. In ths analyss, we assume a sngle object monotonously movng n one drecton. Although ths assumpton consders only a smplfed scenaro, many practcal survellance vdeos nclude objects movng n a sngle drecton towards or away from the camera vew. Also, such movements of the object n camera s vew as rotatng or only changng n sze (when object goes away/towards camera vew but does not move sdeways) do not have a sgnfcant effect on frame dfferencng object detecton. We also assume, wthout loss of generalty, that the object moves from left to rght wth ts sze ncreasng lnearly. The assumpton allows us to consder only changes n coordnate x, and wdth w.

4 (a) Detected foreground object wth drop gap 14 frames. PETS21 vdeo. (b) Bnary mask of the frame n 3(a). Effect of drop gap on frame dfferencng. Fg. 3. The foreground object detecton based on frame dfferencng. Increase/decrease n sze s mportant because when tracked objects approach or move away from the camera, ther sze changes. In practce, when object moves n both x and y coordnates, the overall crtcal drop gap would be the mnmum of the two values estmated for correspondng coordnates. Consder the orgnal vdeo when no frames are dropped. We assume the average dstance between fronts of the blob when t shfts from the prevous frame to the current frame s Δx. We consder the front of the object because t s more accurately detected by frame dfferencng. When frame dfferencng s used, the resulted detected blob s the unon of the object presented n the prevous and current frames (see Fgure 3(b)). Therefore, when we drop frames, the wdth of the blob n the frame followng after the drop gap wll be larger than that n the orgnal vdeo sequence (see Fgure 2 for llustraton). However, the front of the blob would be detected n the same way as n the orgnal vdeo. Snce frame droppng affects sze of the detected object, we consder average change n sze as Δw. The superscrpt ndcates the sze of the drop gap, whch s when frames are not dropped. Assume that x k s x-coordnate of blob s center n k-th frame, then, we can estmate ts coordnate n the frame k + +1 as followng, x k++1 = x k +( +1)Δx ( +1) Δw 2. (2) If frames are dropped after frame k, the detected blob n the k ++1 frame s the unon of actual object appearng n frames k and k (as Fgure 2 llustrates). Then, the wdth dfference (wk++1 /2 w k /2) can be approxmated as ( +1)Δx /2. Therefore, the blob s center n the k frame can be estmated as, x k++1 = x k +( +1)Δx ( +1) Δx 2 = x k +( +1) Δx 2, (3) snce x k = x k.

5 As was mentoned, Δ d = xĩ xĩk, where ĩ ndcates the crtcal drop k+ĩ+1 gap. The falure of the blob trackng mples that Δ d = wk, where value w k s the wdth of the blob detected n frame k. Therefore, from equaton (3), we obtan wk = Δ d =(ĩ +1) Δx 2, from whch we can fnd the crtcal drop gap to be ĩ = 2w k 1. (4) Δx In practce, values wk and Δx can be determned by ether keepng the hstory of speed and sze of tracked object or by estmatng ther average values for a partcular survellance ste. In addton to the estmaton of the crtcal drop gap for blob trackng, we can estmate the dependency functon between accuracy of the algorthm and vdeo frame rate. Such estmaton s possble because of the way drop gap affects the accuracy of the frame dfferencng object detecton algorthm used n blob trackng. We can defne blob detecton error for a partcular frame as the dstance between blob centers detected n ths frame for the degraded vdeo (wth dropped frames) and the orgnal vdeo. Then, the average error, denoted as ɛ j,sthe average blob trackng error for all frames n the vdeo. Ths ɛ j functon can be used as accuracy metrc for the blob trackng depctng the tradeoff between trackng accuracy and vdeo frame rate. Usng equatons (2) and (3) we can estmate the blob trackng error for k++1 frame as followng, ( x k++1 x Δx Δw ) k++1 =( +1) =( +1)C, () 2 where constant C depends on the sze and the speed of object n the orgnal vdeo. Snce we apply the droppng pattern drop frames out of + j frames, we need to estmate the blob trackng error for each of the remanng j frames n the vdeo. There s no error n detectng blob for j 1 frames that do not have drop gap n front of them,.e., for these frames, the result of the frame dfferencng would be the same as n orgnal vdeo wth no droppng. Therefore, the average error for all j frames s the error estmated for the frame, whch follows the drop gap (equaton ()) dvded by j: ɛ j = +1 j ( Δx Δw ) 2 = +1 C. (6) j Note the mportant property of ths functon that the average error s proportonal to and nversely proportonal to j. We performed experments to valdate the estmaton of the average blob trackng error ɛ j. We use several vdeos from VSOR vdeo database, PETS21 datasets, as well as vdeos we shot on campus wth a hand-held camera (example screenshots n Fgure 3(a), Fgure 4(c), and Fgure 4(d)). Vdeos nclude movng

6 (a) Fast movng face shot wth a webcam (CAMSHIFT face trackng). tory (CAMSHIFT face (b) From database by SEQAM labora- trackng). (c) Shot on campus wth hand-held camera (blob trackng). (d) From VISOR vdeo database (blob trackng). Fg. 4. Snapshot examples of vdeos used n our experments. cars, person on a bcycle and people walkng n a dstance. We ran blob trackng algorthm on these vdeos and appled dfferent droppng patterns. We plot the resulted average error aganst drop gap when value j s 1, 3, 6, and 12. The results are shown n Fgure (a) (orgnal vdeo s 18 frames of , 3 fps) and Fgure 6(a) (orgnal vdeo s 148 frames of 32 26, 3 fps). Fgure (a) shows the resulted average trackng error plotted aganst the drop gap when value j s 1, 3, 6, and 12. It can be noted from the Fgure (a) that for each fxed value j the average error s proportonal to. Also, average error s nversely proportonal to j, as ndcated by the angles of each lne n the graph (for nstance, angle of the lne marked as s three tmes larger than the angle of the lne ). Fgure 6(a) demonstrates smlar results. These expermental results strongly support our analytcal estmaton of the average error gven n the equaton (6). The fgures do not reflect the crtcal drop gap value because even for large drop gaps the blob trackng dd not lose the track of the car n ths test vdeo sequence.

7 Average Error Blob Trackng (a) Average Error Adaptve Blob Trackng (b) Fg.. Accuracy of orgnal and adaptve blob trackng algorthm for PETS21 vdeo (snapshot n Fgure 3(a)). 4 CAMSHIFT Algorthm CAMSHIFT object trackng [4] reles on color hstogram detecton and mean shft algorthm for trackng. The algorthm searches for a gven object s hstogram nsde a subwndow of the current frame of the vdeo, whch s computed as 1% of the object sze detected n the prevous frame. Therefore, f the object, moves between two frames from ts orgnal locaton for a dstance larger than half of ts sze, the algorthm wll lose the track of the object. Hence, assumng we drop frames before frame k + + 1, the threshold dstance Δ d = w k 2, where wk s the wdth of the blob detected n frame k. Snce CAMSHIFT does not use frame dfferencng, drop gap does not have an addtonal effect on object s sze. Therefore, we can estmate the center of the blob after drop gap usng the equaton (2) nstead of equaton (3). Hence, the crtcal drop gap can be derrved as wk ĩ = 2Δx 1. (7) Δw Estmatng the average trackng error loses ts meanng for CAMSHIFT trackng because t uses a smple threshold for detecton of the object n the current frame. If the drop gap of the gven frame droppng pattern s less than crtcal drop gap n equaton (7), the algorthm contnue trackng the object, otherwse t loses t. And the crtcal drop gap depends on the changes n speed and sze of the object.

8 Average Error Blob Trackng (a) Average Error Adaptve Blob Trackng (b) Fg. 6. Accuracy of orgnal and adaptve blob trackng algorthm for VISOR vdeo (snapshot n Fgure 4(d)). We performed experments wth CAMSHIFT trackng algorthm to verfy our analytcal estmaton of the crtcal drop gap (equaton (7)). We used several vdeos of a movng face shot wth a smple web-cam, vdeos of talkng heads by SEQAM laboratory and some move clps (example screenshots n Fgure 4(a) and Fgure 4(b)). Fgure 8(a) (orgnal vdeo s 6 frames of , 3 fps) and Fgure 7 (orgnal vdeo s 33 frames of 32 23, 3 fps) show average trackng error vs. drop gap for CAMSHIFT trackng and varous frame droppng patterns. Fgure 7, correspondng to the vdeo of a talkng head (see snapshot n Fgure 4(b)), demonstrates that trackng algorthm does not lose the face even when drop gap s 14 frames. The reason s because the face n the vdeo does not move around and s always present n the search subwndow of CAMSHIFT tracker. However, for the experments shown n Fgure 8(a), the vdeo wth fast movng head was used (see snapshot n Fgure 4(a)). It can be noted that the algorthm does not lose the face untl value of drop gap s 8, because for the smaller drop gaps, the face s stll wthn a search subwndow and can be detected by the hstogram matchng. The fluctuatons n the average error for the larger drop gaps appear because the face s ether lost by the tracker or, for some large enough gaps, t would move out of the subwndow and move back n, hence the tracker does not lose t. We conducted experments wth more vdeos and observed that the crtcal drop gap value s smaller for vdeos wth faster movng faces and larger for vdeos wth slower movng faces. These observatons agree wth equaton (7).

9 Average Error CAMSHIFT Face Trackng Fg. 7. Accuracy of orgnal and adaptve CAMSHIFT trackng algorthm for vdeo wth slow movng face (snapshot n Fgure 4(a)). Adaptve Trackng We propose to modfy blob trackng and CAMSHIFT algorthms and make them more tolerant to vdeo wth low frame rate. We have shown that average error and the crtcal frame rate of trackng algorthms depend on speed and sze of the object n the orgnal vdeo. Therefore, f we record these characterstcs for prevous frames, the locaton and the sze of object n the frame that follows a drop gap can be approxmated. Adjustng to frame droppng n such way allows us to reduce the average error for blob trackng algorthm and ncrease the crtcal drop gap for the CAMSHIFT algorthm. Blob trackng algorthm tracks the detected foreground object usng the smplfed verson of Kalman flter: x k =(1 α)x k 1 + αz k, where x k and x k 1 represent estmated coordnates of the object n the current and prevous frames, z k s the output of the object detector, and α 1 s some constant. When α =1, then the tracker trusts the measurement z k fully and ts average error can be estmated by equaton (6). In cases when α<1, the accuracy of the trackng aganst the frame droppng worsens, due to the larger shfts n blobs centers for vdeos wth hgh drop gap. We propose usng adaptve Kalman flter [] to make blob trackng more tolerant to the frame droppng. We apply the flter only to the wdth of the object, because the front s detected correctly by frame dfferencng (see Fgure 2). The flter can be defned as followng, w k = w k 1 + K k (w k 1 + u k ) Pk = P k + Q k P k =(1 K k ) P k K k = P k ( P k + R k ), where Q k and R k are the process and measurement nose covarances; w k s the new estmate of the blob s wdth n the current frame; w k 1 s blob s wdth n the last not dropped frame; u k s the wdth measurement provded by the frame-dfferencng based detector. Kalman flter depends on correct estmaton of the error parameters, Q k and R k. By lookng at Fgure 2, we can set Q k =(Δw ) 2, whch estmates how bg the tracked object should be at frame k compare to ts wdth before the

10 Average Error CAMSHIFT Face Trackng (a) Average Error Adaptve CAMSHIFT Face Trackng (b) Fg. 8. Accuracy of orgnal and adaptve CAMSHIFT trackng algorthm for vdeo wth fast movng face (snapshot n Fgure 4(a)). drop gap at frame k. R k s essentally the error of the measurement,.e., the output of the foreground object detector, therefore, R k =(w k++1 w k++1 )2. Snce w k++1 can be estmated as w k +( +1)Δx and w k++1 as w k +( + 1)Δw, we can approxmate R k =( +1) 2 (Δx Δw ) 2. We obtan the values of Δw and Δx by recordng the speed of the object and how fast t grows n sze usng last two avalable frames. To compare how adaptve Kalman flter mproves the accuracy of blob trackng, we performed the same experments varyng frame droppng pattern. The average error for blob trackng wth adaptve Kalman flter s plotted n Fgure (b) and Fgure 6(b), whch can be compared to results wth orgnal algorthm n Fgure (a) and Fgure 6(a) respectvely. We can note that the accuracy of the adaptve blob trackng algorthm s mproved for larger drop gaps (larger frame rate reducton). In both fgures, Fgure (b) and Fgure 6(b), the angles of the lnes n the graph are not nversely proportonal to j anymore, gvng fundamentally dfferent bound on the average error. All lnes wth j>1 are almost parallel to x-axs. It means that Kalman flter adapts very well to the drastc changes n speed and sze of the object that occur due to the frame droppng. The constant ncrease n the average error for j = 1, s because, for such droppng pattern, all remanng frames are separated by drop gaps. In ths scenaro, adaptve Kalman flter accumulates the approxmaton error of object s sze and speed. Therefore, the crtcal frame rate can be acheved wth j that s at least equalto2.ifwetake = 12, the orgnal frame rate s reduced by 7 tmes.

11 We also modfed the CAMSHIFT trackng algorthm, adjustng the sze of ts search subwndow to the frame droppng. We smply ncreased the subwndow sze n the current frame by Δx, where s the drop gap. The average error of ths adaptve CAMSHIFT algorthm for the vdeo wth fast movng face s shown n Fgure 8(b). Comparng wth the results of orgnal algorthm n Fgure 8(a), we can notce that the adaptve tracker performs sgnfcantly better for the larger drop gaps. The experments show that we can drop 13 frames out of 14 wth a tradeoff n small average error. It means that CAMSHIFT algorthm, for ths partcular vdeo sequence, can accurately track the face wth frame rate reduced by 13 tmes from the orgnal. For the news vdeos of talkng heads, where face does not move sgnfcantly around, adaptve algorthm performs wth exactly the same accuracy results as the orgnal algorthm. Therefore, Fgure 7 llustrates essentally both versons of the algorthm, orgnal and adaptve. These experments demonstrate that by usng analyss to modfy CAMSHIFT algorthm, we can mprove ts performance on vdeos wth fast movng faces, whle retanng the orgnal accuracy on vdeos wth slow movng faces. 6 Concluson In ths paper, we use analyss to estmate the tradeoff between accuracy of two common trackng algorthms and vdeo frame rate. Such estmaton depends on the speed and sze of the tracked object, and therefore, n practce, such measurements of the object need to be taken (for nstance, runnng average of these values durng the last few frames). We also show that slght modfcatons to exstng algorthms can sgnfcantly mprove ther accuracy for the vdeo wth larger reductons n frame rate. These fndngs motvate us to use reasonng for determnng crtcal frame rate (not just runnng many dfferent experments) for other vdeo analyss algorthms. The fndngs also encourage the development of the new object trackng algorthms robust to hghly degraded vdeo. References 1. Korshunov, P., Oo, W.T.: Crtcal vdeo qualty for dstrbuted automated vdeo survellance. In: Proceedngs of the ACM Internatonal Conference on Multmeda, ACMMM, Sngapore (November 2) L, L., Huang, W., Gu, I.Y., Tan, Q.: Foreground object detecton from vdeos contanng complex background. In: Proceedngs of the ACM Internatonal Conference on Multmeda, ACMMM 3, Berkeley, CA, USA (November 23) Bradsk, G.R.: Computer vson face trackng as a component of a perceptual user nterface. In: Proceedngs of the Forth IEEE Workshop on Applcatons of Computer Vson, WACV 98, Prnceton, NJ (January 1998) Boyle, M.: The effects of capture condtons on the CAMSHIFT face tracker. Techncal Report , Department of Computer Scence, Unversty of Calgary, Alberta, Canada (21). Welsh, G., Bshop, G.: An ntroducton to the kalman flter. In: Proceedngs of SIGGRAPH 21. Volume Course 8., Los Angeles, CA, USA (August 21)