CSE 237A: Final Project Report Object Tracking Willis Hoang & Shimona Carvalho November 27, 2006

Transcription

1 CSE 237A: Fnal Project Report Object Trackng Wlls Hoang & Shmona Carvalho November 27, Project Descrpton For ths project, we plan to mplement an object trackng applcaton usng the ntegrated vdeo camera on Intel's PXA27x embedded system platform. Our man goal s to track an object aganst a unform color backdrop. The relatve poston wll be dsplayed on the LCD. We also seek to optmze our applcaton to obtan the best possble rate and qualty of mage capture and analyss. If tme permts, we wll attempt to expand the scope to track more complex movng objects aganst a non-unform background. 2 Motvaton, Challenges and Lmtatons 2.1 Applcatons and challenges Moton trackng has many applcatons n today's dverse range of embedded systems. Ths type of trackng s frequently used for securty and survellance, medcal therapy, retal space nstrumentaton, traffc management and vdeo edtng. On cars, moton trackng could be used to montor other vehcles and alert the drver to dangers such as vehcles lurkng n the host vehcle's blnd spot. In the manufacturng ndustry, moton trackers are used for analyss of hgh-speed assembly lne producton. The man ssues n the development of such a system are the optmzng of the speed of capture and analyss of vdeo. Whle there are many pattern recognton algorthms avalable, care must be taken to select an optmal pattern recognton algorthm for each task. For example, n the manufacturng example, t would be easy to ensure that the robotc arm beng tracked was covered by an easly recognzable pattern. However, n the case of the blnd-spot trackng, the applcaton would need to dscern between the farly smlar mages of a car n the next lane a potental threat - and one two lanes over - not a threat [1]. The recognton algorthm also must be fast to provde useful real-tme response. The best car detecton algorthm would be worthless f the car was detected after t had gone by. Therefore, for a good embedded moton trackng system, an optmal balance between these varables should be mantaned. 2.2 State of the Art Current state of the art technques nvolve developng an unsupervsed, robust, computatonally effcent mult-object tracker. Key requrements of trackng algorthms are mnmum ntalzaton effort and fne-tunng for camera setups. MERL (Mtsubsh Electrc Research Laboratores) uses the methods descrbed below for most of ther survellance products [2]. One of the challenges presented by trackng algorthms s the desre for accurate mult-object trackng wth the constrant of low computatonal complexty. The two man algorthms use ether back-trackng or forward trackng. The back trackng based approach segments the foreground regons and attempts to fnd a correspondence of regons between the current and prevous mages. The forward trackng based approach estmates the postons n the current frame of the regons found n the prevous frame. To establsh correspondence, dfferent templates are needed to optmze for dfferent objects. For

2 example, trackng of humans wll requre a dfferent correspondence algorthm than that for factory equpment. 3 Implementng the Moton Tracker 3.1 Evaluaton of avalable camera code Our frst step was to evaluate the current c-capture code to decde whether t would be able to capture the vdeo we needed. Pror to capturng the stll mage, the program handled a few seconds of vdeo as a prevew. Ths was enough to base our code on. One ssue we ran nto at ths pont was that the vdeo s dsplayed only as 176 x 144. We researched on the web and attempted to make the program use 320 x 240 but found that the overlay2 devce returned fxed frame sze of 176 x 144. Any larger frame sze request resulted n the devce falng to open. We decded to stay wth 176 x 144. The vdeo captured by the camera s set to YCbCr planar format. Ths s because ths s the natve format of the LCD and therefore transfer from the camera to the screen s drect and needs no converson algorthm. Whle dealng wth RGB pxels mght have been more ntutve, we decded that the converson from YCbCr to RGB before the moton trackng algorthm would have hurt the effcency of the algorthm. Instead we researched the YCbCr format and used t drectly. 3.2 YUV format The YCbCr format separates the lumnance (or brghtness nformaton) from the chromnance (or color nformaton) of vdeo. Y s the luma component, and Cb and Cr the blue and red chroma components. YCbCr s the dgtal vdeo verson of the analog colorspace YUV. It more closely approxmates the human eye s response to lght, than RGB. Snce the human eye has farly small color senstvty, half the nformaton n the chroma components s dscarded. Whle there are as many Y values as there are pxels, there are half the amount of Cb and Cr values as there pxels. Ths s why the format s sometmes called YCbCr 4:2:2, the numbers representng the rato of Y values to Cb and Cr [3][4]. An example of an mage splt nto the three components s shown n Fgure 1. Fgure 1 YUV Image splt nto Lumnance, Red Chroma and Blue Chroma 3.3 Research of moton detecton algorthms Our ntal goal was to track a whte object aganst a unform black backdrop. We felt ths was farly easy to mplement so to challenge ourselves, we decded to mplement an algorthm that tracked a movng object regardless of the backdrop color. The algorthm mplemented conssts of three stages: dfference, threshold, and centrod [5]. Refer to Fgure 2 for an llustratve dagram.

3 In the dfference stage, we subtract the current vdeo frame wth the prevous vdeo frame. Let us denote I c as the current vdeo frame, I p as the prevous vdeo frame, and as the resultng dfference. Mathematcally, ths s gven as I = I I. By d c! dong so, we are able to detect movement between the current and prevous vdeo frames. Areas wthout sgnfcant movement wll be approxmately zero and areas wth sgnfcant movement wll be approxmate one. Effectvely, ths removes the background and statonary objects. p I d Fgure 2. Moton Detecton Algorthm To make processng easer, we threshold the pxel values n the second stage producng a bnary mage. Dong ths "exaggerates" the detected movements. Ths s mathematcally gven as I = { 1 f I d < t, 0 f I d! t where t s the threshold value. The last stage s centrod, ths can be thought of as calculatng the "center of mass" of I t. Center of mass s genercally gven as =! system, r s the poston, and sum of the pxels n I t, calculated twce, once for x and once for y. CM 1 m r where M s the total mass of the M m s the weghted mass [6]. For our applcaton, M s the total m s the pxel value, and r s the pxel locaton. The centrod s Ths process s calculated for each frame where the centrod denotes the moton tracked object. To clearly represent ths and denote relatve poston, we overlay a frame surroundng the centrod coordnates on the LCD of our embedded platform. 3.4 Problems and Solutons The ntegrated camera sensor was very senstve to slght changes and nose such as the flckerng of the ambent lght. Ths nose would show up as dfference after the subtracton stage. To avod ths, a threshold mass was used whch requred a certan mass of moved pxels to accumulate before regsterng as actual moton. Ths successfully fltered out the effects of sensor nose and flcker. Snce the FPS of the vdeo capture s very hgh, takng the dfference between two consecutve dd not result n much change. To extract more meanngful nformaton, we used a sldng wndow method wheren we compared the current buffer to one several frames before t. The dfference between these two frames was more sgnfcant as more tme had elapsed between them. Another ssue we found was that the centrod had no nerta and jumped around very quckly. Whle accurate, t reduced the qualty of the trackng movement as vewed by the human eye. To solve ths ssue we used a weghtng system to pull the new centrod closer to old centrod. By expermentng wth dfferent weghts, we found an deal weght whch stopped the tracker from appearng too jumpy wthout causng too much lag f the object moved quckly. t

4 3.5 Possble Extensons We found that multple movng objects cause the algorthm to pck a pont at the centrod of the dfferent movng objects. Ths s because the algorthm could not detect that there were multple movng objects and assumed one large movng object. Wth more tme, an algorthm could be developed to detect and separate multple movng objects, and track them ndvdually, but t was beyond our scope at ths tme. Algorthms are currently beng developed that employ occluson detecton methods such as merge-splt and straght-through [7] to track multple objects. 4 Optmzng the Moton Tracker 4.1 Box Frame Drawng One optmzaton we added n the mplementaton phase was the statc frame drawng. When we frst calculated the FPS before any trackng was turned on, we got an average FPS of We then tred our software wth the trackng turned on and had an FPS of Ths was a large drop n the FPS, but we suspected the code that overlad a box marker around the movng object. Our orgnal soluton was to determne the box pxels usng a dynamc algorthm that took the centrod locaton as an nput. Ths was too computaton ntensve as t needed to be done per frame, so we desgned a more effcent, alternate algorthm. The new algorthm used a predefned array of relatve postons to map out the box pxels. Ths statc array reduced our computaton tme to a fracton of the orgnal and resulted n an FPS rate of 43.79, very close to the orgnal FPS before trackng. 4.2 The Camera Capture Bottleneck Our frst attempts to measure performance led us to an nterestng result. Our bggest bottleneck was not the algorthm but the speed of camera capture. Reducng the amount of analyss we dd per frame was not havng a sgnfcant effect on our FPS, so we decded to measure the tme taken for the camera to capture 1000 frames, compared to the tme taken to capture and analyze 1000 frames. The results are tabulated n Table 1. Frequency (MHz) DVFM FPS Capture (sec) Capture + Analyze (sec) Vdeo Qualty good good poor Table 1. Comparson of Camera Capture tme and Capture + Analyss tme As you can see, at the hghest two frequences, the tme taken to capture + analyze s only slghtly greater than the tme taken to capture. Ths s because the software spends a sgnfcant amount of tme suspended, watng for the camera hardware to return a frame of vdeo data. The camera operates parallel to software except when the software s transferrng a frame of captured vdeo over. The algorthm proceeds as follows. On recept of a frame of vdeo from the camera, the processor analyzes t, transfers t onto the LCD and requests another frame of data. Then, t suspends untl t receves t. Thus, at the hgher frequences, where the trackng algorthm takes less tme than the capture tme of the camera, no optmzatons of the algorthm tself wll mprove the FPS. Also

5 note that the frequency of the processor has no effect on the speed of capture, whch appears to be entrely hardware-dependent. 4.3 Skp A Frame or Fve We then turned our attenton to the lower frequences. In the 208MHz example n Table 1, the tme taken for analyss has now exceeded the tme taken to capture. Ths means, by the tme the algorthm s ready to receve a new frame of vdeo data, t has already mssed one and needs to wat a whole new capture cycle to get the next one. As a result the tme taken s more than twce that of the capture tme, and the vdeo qualty s poor. Hence we realzed, t was possbly to optmze n the lower frequency cases. One technque would be to run the trackng algorthm only once every n frames nstead of each tme. In frames where no algorthm s run, the data s stll transferred from the camera to the screen. Ths would reduce the total tme spent runnng the moton detecton algorthm. We would probably stll mss a frame of vdeo data but only once every n tmes. 4.4 Results Whle the frame skppng method gave us large energy savngs, we also notced that the more frames, we skpped the more the trackng box would lag behnd the movng object, especally f the object moved at a hgh speed. By balancng out both these factors, we found that we could acheve 75% energy savngs runnng at 208MHz skppng 5 frames. Ths s a good result gven that pror to our optmzatons, we were unable to run at ths power mode wth acceptable vdeo qualty. Freq (MHz) DVFM Skp Frames FPS Qualty Total Energy (mj) % Energy Saved good % good % poor % poor % good % good % good % bad % good % bad % bad % bad % bad % bad % bad % bad % Table 2. Comparson of Energy Savngs usng Power Modes and Frame-Skppng

6 5 Concluson and Outlook On the whole, the algorthm mplementaton was farly straghtforward. From the very begnnng, we attempted to wrte the code mndful of effcency and number of cycles used per teraton. Ths helped us mantan a hgh frames-per-second rate, and opened up possbltes for the power consumpton optmzatons. At the same tme, we attempted to keep the vdeo qualty at an acceptably smooth level as perceved by the human eye. The result of the optmzatons was a 75% energy savngs on the orgnal measurement. Ths rather large dfference would mprove the potental of usng ths applcaton n more portable settngs, for example n stuatons where the camera communcates wrelessly wth a server. Further extensons of the applcaton would nvolve detectng several movng objects and dstngushng between them. Multple cameras could also be used to obtan 3D nformaton. Moton trackng s an exctng tool n the development of ncreasngly reactve technologes. Ths wll save humans from constantly havng to montor ther applances and electroncs, and allow some decsons to be made autonomously, such as montorng of manufacturng equpment. It can also be used to help humans make decsons by provdng them wth pre-measured useful nformaton prevously unavalable, such as n blnd spot detecton. Wth technques lke ths, embedded systems can move away from more needy nteractve models, toward more convenent and speeder reactve models.

7 References [1] Bourbaks, N.; Fndler, M., "Smart cars as autonomous ntellgent agents," Tools wth Artfcal Intellgence, Proceedngs of the 13th Internatonal Conference on, vol., no.pp.25-32, 7-9 Nov 2001 [2] MERL, "Object Trackng and Understandng," Mtsubsh Electrc Research Laboratores, July 15, 2004 < [3] "Ycbcr." Wkpeda, The Free Encyclopeda. 28 Oct 2006, 07:29 UTC. Wkmeda Foundaton, Inc. 14 Nov 2006 < [4] Kovacevc, Mjo. "Color n Image and Vdeo." Nov < [5] Han, Xu; Sun, Yng, "Algorthm for the Actve Image Moton Seekng (AIMS) Camera System." Unversty of Rhode Island. 13 Nov < [6] Center of mass." Wkpeda, The Free Encyclopeda. 10 Nov 2006, 10:04 UTC. Wkmeda Foundaton, Inc. 14 Nov 2006 < [7] P. F. Gabrel, J. G. Verly, J. H. Pater, and A. Genon. "The state of the art n multple object trackng under occluson n vdeo sequences." In Advanced Concepts for Intellgent Vson Systems (ACIVS), 2003