Joint Tracking of Features and Edges

Transcription

1 Jont Trackng of Features and Edges Stanley T. Brchfeld Shrnvas J. Pundlk Electrcal and Computer Engneerng Department Clemson Unversty, Clemson, SC {stb, Abstract Sparse features have tradtonally been tracked from frame to frame ndependently of one another. We propose a framework n whch features are tracked jontly. Combnng deas from Lucas-Kanade and Horn-Schunck, the estmated moton of a feature s nfluenced by the estmated moton of neghborng features. The approach also handles the problem of trackng edges n a unfed way by estmatng moton perpendcular to the edge, usng the moton of neghborng features to resolve the aperture problem. Results are shown on several mage sequences to demonstrate the mproved results obtaned by the approach. 1. Introducton Trackng features between consecutve mage frames s a fundamental problem n computer vson. By establshng correspondence between sparse ponts, feature trackng captures the essence of the moton of a scene n a compact descrpton, makng t useful for a wde varety of applcatons, such as structure from moton [19, 21, 5, 9], moton segmentaton [5, 13, 16], object trackng [11], mage mosackng [23], and face trackng [6]. The classc approach to feature trackng s the dfferental technque of Lucas and Kanade [14], whch estmates the moton of a small patch of mage ntenstes (features). The basc dea s smple: A lnear system s repeatedly solved to fnd the best algnment for the mage patch n the other mage. Over the years, researchers have proposed a number of mprovements to the basc algorthm. Sh and Tomas [17] determne when a feature has been lost by computng the best affne warp between the frst frame and the current frame. They noted that, although the translaton model s acceptable between consecutve mage frames, a rcher model such as affne s necessary when consderng frames separated wdely n tme. Ther work was later extended by Tommassn et al. [20] to automatcally reject spurous features. More recently, Šegvć et al. [22] model the feature support adaptvely to mprove long-range performance of a feature n the presence of domnant forward moton. When non-causal processng s approprate, Svc et al. [18] descrbe a technque for reparng the trajectores of features usng the start and end locatons n a set of frames. To handle lghtng and exposure changes, varous methods have been proposed [12, 10]. Other researchers have focused ther efforts upon mprovng the performance of Lucas-Kanade when appled to a sngle large mage patch to be tracked [7, 15, 1]. One problem that remans largely unaddressed n the lterature, however, s that of trackng features together. Most prevous approaches to feature trackng consder each feature ndependently of the other features, thus neglectng mportant nformaton that s avalable n determnng the moton of a feature. Ths bottom-up approach s at odds wth the Gestalt emphass upon the mportance of moton coherence, namely that nearby pxels wll often have smlar motons. Even though ndependent trackng oftentmes succeeds, errors persst because the neghborng nformaton s gnored. In ths paper we propose a framework n whch features are tracked jontly. Inspred by the work of Bruhn et al. [4], we use global optc flow methods (Horn-Schunck) to mprove the results obtaned by local optc flow (Lucas- Kanade). A smoothness term s added to the formulaton to penalze the devaton of the dsplacement of a feature from ts expected value, whch s computed by fttng a moton model to the dsplacements of the neghbors. The approach bears a resemblance to that of Km et al. [12] n whch features are also tracked jontly, but n our work the features nfluence one another geometrcally rather than photometrcally. One advantage of the proposed framework s that t facltates the unform treatment of features and edges for trackng. Edges have often been avoded n trackng due to ther underconstraned nature (the aperture problem). Here, however, the gradent nformaton that enables an edge to be tracked n the drecton perpendcular to the edge s combned naturally wth the moton of neghborng features. The moton of the edge s determned by mnmzng a 2D energy functonal that takes nto account both /08/$ IEEE

2 the mage data and the smoothness of the nearby estmated motons. 2. Lucas-Kanade and Horn-Schunck As explaned by Bruhn et al. [4], dfferental methods for both dense optcal flow as well as sparse feature trackng are based on the assumpton that the ntensty values of the projecton of scene ponts do not change over tme: I(x + u, y + v, t +1)=I(x, y, t), (1) where I(x, y, t) s the ntensty of pxel x = (x, y) T n frame t, and u =(u, v) T s the dsplacement of the pxel between consecutve frames t and t +1. For small dsplacements, a lnearzed Taylor seres expanson yelds the wellknown optc flow constrant equaton: f(u, v; I) =I x u + I y v + I t =0, (2) where the subscrpts denote partal dervatves. The wellknown aperture problem arses because ths sngle equaton s nsuffcent to recover the two unknowns u and v. The Lucas-Kanade [14] approach to overcomng the aperture problem assumes that the unknown dsplacement u of a pxel s constant wthn some neghborhood. As a result, the dsplacement can be computed by mnmzng ( E LK (u, v) =K ρ (f(u, v; I)) 2), (3) where K ρ ( ) denotes convoluton wth an ntegraton wndow of sze ρ. Dfferentatng wth respect to u and v, and settng the partal dervatves to zero, yelds the lnear system [ ][ ] [ ] Kρ (Ix) 2 K ρ (I x I y ) u Kρ (I K ρ (I x I y ) K ρ (Iy) 2 = x I t ) v K ρ (I y I t ) (4) whch s solved teratvely to mnmze E LK. Alternatvely, the Horn-Schunck [8] approach regularzes the underconstraned optc flow constrant equaton by mposng a global smoothness term. Whle Lucas-Kanade fnds the dsplacement of a small wndow around a sngle pxel, Horn-Schunck computes the global dsplacement functons u(x, y) and v(x, y) by mnmzng E HS (u, v) = (f(u, v; I)) 2 + λ ( u 2 + v 2) dx dy, Ω (5) where λ s the regularzaton parameter and Ω s the doman of the mage. The mnmum of ths functonal s found by solvng the correspondng Euler-Lagrange equatons, leadng to [ I 2 x I x I y I x I y I 2 y ][ ] u = v [ λ 2 u I x I t λ 2 v I y I t ], (6) where 2 u = 2 u x + 2 u 2 y and 2 v = 2 v 2 x + 2 v 2 y are the 2 Laplacan of u and v, respectvely. Solvng ths equaton for u and v and usng the approxmaton that 2 u h(ū u), where ū s the average of the values of u among the neghbors of the pxel, and h s a constant scale factor, we get [ ] [ ] u ū = I [ ] xū + I y v + I t Ix v v hλ + Ix 2 + Iy 2. (7) I y Thus, the sparse lnear system can be solved usng the Jacob method wth teratons for pxel (, j) T of the form: u (k+1) j = ū (k) j γi x (8) v (k+1) j = v (k) j γi y, where γ = I xū (k) j + I y v (k) j + I t hλ + Ix 2 + Iy 2. (9) Faster convergence s obtaned by performng computatons n place (Gauss-Sedel) and usng successve overrelaxaton (SOR). 3. Jont Feature Trackng It s mportant to note that, although the dervaton of Eq. (6) assumes a contnuous formulaton, the fnal result n Eqs. (8) (9) corresponds to a dscrete energy functonal, due to the dscrete approxmaton of the Laplacan. Ths observaton motvates us to combne the Lucas-Kanade and Horn-Schunck approaches n Eqs. (3) and (5) nto the followng functonal to be mnmzed: E JLK = N (E D ()+λ E S ()), (10) =1 where N s the number of feature ponts, and the data and smoothness terms are gven by ( E D () = K ρ (f(u,v ; I)) 2) (11) E S () = ( (u û ) 2 +(v ˆv ) 2). (12) In these equatons, the energy of feature s determned by how well ts dsplacement (u,v ) T matches the local mage data, as well as how far the dsplacement devates from the expected dsplacement (û, ˆv ) T. Note that the expected dsplacement can be computed n any desred manner and s not necessarly requred to be the average of the neghborng dsplacements. Dfferentatng E JLK wth respect to the dsplacements (u,v ) T, =1,...,N, and settng the dervatves to zero, yelds a large 2N 2N sparse matrx equaton, whose (2 1)th and (2)th rows are gven by Z u = e, (13)

3 where Z = e = [ ] λ + K ρ (I x I x ) K ρ (I x I y ) K ρ (I x I y ) λ + K ρ (I y I y ) [ ] λ û K ρ (I x I t ). λ ˆv K ρ (I y I t ) Ths sparse system of equatons can be solved usng Jacob teratons of the form ũ (k+1) = û (k) ṽ (k+1) = ˆv (k) J xxû (k) J xyû (k) + J xyˆv (k) + J xt (14) λ + J xx + J yy + J yyˆv (k) + J yt, (15) λ + J xx + J yy where J xx = K ρ (Ix), 2 J xy = K ρ (I x I y ), J xt = K ρ (I x I t ), J yy = K ρ (Iy), 2 and J yt = K ρ (I y I t ). As before, convergence s greatly ncreased by performng Gauss-Sedel teratons so that û (k) and ˆv (k) are actually computed usng a mxture of values from the kth and (k +1)th teratons (dependng upon the order n whch the values are updated), and by performng a weghted average of the most recent estmate and the new estmate (successve overrelaxaton). Wth ths modfcaton, the update =(1 ω)u (k) + ωũ (k+1), s the estmate expressed n Eqs. (14 15), and equatons are gven by u (k+1) where ũ (k+1) ω (0, 2) s the relaxaton parameter. For fast convergence, ω s usually set to a value between 1.9 and Note that for ω =1the approach reduces to Gauss-Sedel. 4. Pyramdal Implementaton Both the standard Lucas-Kanade method and the proposed jont Lucas-Kanade method nvolve teratvely solvng a sparse 2N 2N lnear system to fnd the mnmum of a quadratc cost functonal. In the former, the matrx s blockdagonal, leadng to a smple and effcent mplementaton va a set of 2 2 lnear systems, whle n the latter, the offdagonal terms requre the approach presented n the prevous secton. The dfference between the approaches becomes apparent when consderng a pyramdal mplementaton, whch s usually necessary to overcome the defcences n the lnearzaton approxmaton n the formulaton of the problem n Eq. (2). The two algorthms are shown. Standard Lucas-Kanade terates through each pyramd level for each feature, whle jont Lucas-Kanade terates through each feature for each pyramd level. Note that f λ =0, =1,...,N, then the two algorthms are exactly the same (except for mnor dfferences n the termnaton crteron), and that the computaton requred s the same. Both algorthms are O(Nnm), where N s the number of features, n s the number of pyramd levels, and m s the average number of teratons. However, because t consders all the features at a tme, the jont Algorthm: Standard Lucas-Kanade For each feature, 1. Intalze u (0, 0) T 2. Set λ 0 3. For pyramd level n 1 to 0 step 1, (a) Compute Z (b) Repeat untl convergence:. Compute the dfference I t between the frst mage and the shfted second mage: I t (x, y) =I 1 (x, y) I 2 (x + u,y+ v ). Compute e. Solve Z u = e for ncremental moton u v. Add ncremental moton to overall estmate: u u + u (c) Expand to the next level: u αu, where α s the pyramd scale factor Algorthm: Jont Lucas-Kanade For each feature, 1. Intalze u (0, 0) T 2. Intalze λ For pyramd level n 1 to 0 step 1, 1. For each feature, compute Z 2. Repeat untl convergence: (a) For each feature,. Determne û. Compute the dfference I t between the frst mage and the shfted second mage: I t (x, y) =I 1 (x, y) I 2 (x + u,y+ v ). Compute e v. Solve Z u = e for ncremental moton u v. Add ncremental moton to overall estmate: u u + u 3. Expand to the next level: u αu, where α s the pyramd scale factor algorthm nvolves dfferent memory requrements: Instead of precomputng all the pyramdal mages, t must precompute the Z matrces for all the features.

4 Several mplementaton ssues reman. Frst, how should the regularzaton parameters λ be chosen? Snce a large number of features can often be tracked accurately wthout any assstance from ther neghbors, one could magne weghtng some features more than others, e.g., usng one of the standard measures for detectng features n the frst place [17]. For example, snce large egenvalues of the gradent covarance matrx ndcate suffcent mage ntensty nformaton for trackng, such features could receve smaller smoothng weghts (regularzaton parameter values) than those wth nsuffcent nformaton. However, ths scheme s frustrated by the fact that the egenvalues do not take nto account mportant ssues such as occlusons, moton dscontnutes, and lghtng changes, makng t dffcult to determne beforehand whch features wll actually be tracked relably. As a result, we smply set all of the regularzaton parameters to a constant value n ths work: λ =50. Another ssue s how to determne the expected values (û, ˆv ) T of the dsplacements. Because the features are sparse, a sgnfcant dfference n moton between neghborng features s not uncommon, even when the features are on the same rgd surface n the world. As a result, we cannot smply average the values of the neghbors as s commonly done [8, 4]. Instead, we predct the moton dsplacement of a pxel by fttng an affne moton model to the dsplacements of the surroundng features, whch are nversely weghted accordng to ther dstance to the pxel. We use a Gaussan weghtng functon on the dstance, wth σ =10pxels. Fnally, because the algorthm enforces smoothness, t s able to overcome the aperture problem by determnng the moton of underconstraned pxels that le along ntensty edges. We modfy the feature detecton algorthm accordngly. To detect features, we use the two egenvalues e mn and e max, e mn e max of the orgnal Lucas-Kanade gradent covarance matrx (.e., Z wth λ =0). Rather than selectng the mnmum egenvalue e mn, as s often done [17], we select features usng max(e mn,ηe max ), where η<1 s a scalng factor. The ratonale behnd ths choce s that along an ntensty edge e max wll be large whle e mn wll be arbtrarly small. Instead of treatng an edge lke an untextured regon, the proposed measure rewards the feature for the nformaton that t does have. For pxels havng two comparable egenvalues, the proposed measure reduces to the more common mnmum egenvalue. In ths work we set η = Expermental Results The proposed algorthm has been compared aganst a state-of-the-art mplementaton of pyramdal Lucas-Kanade [3]. We ran both algorthms wth dentcal parameters on four mage sequences (Rubber Whale, Hydrangea, Venus, and Dmetrodon) recently proposed for comparng optcal flow algorthms [2]. These sequences nclude statc scenes as well as ndependently movng rgd and non-rgd objects. For all mages we detected 1000 features and tracked usng a 7 7 wndow, wth ten maxmum teratons and three pyramd levels (.e., the orgnal mage plus two mages obtaned by downsamplng the orgnal by a factor of two and four, respectvely). No color nformaton was used by ether algorthm. The results of the experment are shown n Table 1. For both algorthms, we computed the average angular error (AE) and the average endpont error (EP) [2]. In all cases, the jont trackng algorthm consderably outperforms the tradtonal approach, oftentmes reducng the error by nearly half. More recent technques, such as [22, 12], would perform smlarly to the tradtonal approach on these mages snce they also do not enforce spatal contnuty. It s also worth notng that the errors of the jont trackng algorthm are sgnfcantly less than those of the leadng dense optcal flow algorthms, whch acheve 9.26 (AE) and 0.35 (EP) on Dmetrodon and 7.64 (AE) and 0.51 (EP) on Venus. 1 Fgure 1 dsplays the results of the two algorthms on three of the mage sequences. In general, the jont trackng algorthm exhbts smoother flows and s thus better equpped to handle features wthout suffcent local nformaton. In partcular, repettve textures that cause ndvdual features to be dstracted by smlar nearby patterns usng the tradtonal algorthm do not pose a problem for the proposed algorthm. A close-up showng ths behavor s n Fgure 2. The dfference between the two algorthms s even more pronounced when the scene does not contan much texture, as s often the case n ndoor man-made envronments. Fgure 3 shows one such scene, along wth the results computed by the two algorthms. In ths sequence the camera s movng down and to the rght wth a slght counterclockwse rotaton. The camera gan control causes a severe ntensty change n the wndow of the door, causng those features to be lost. The jont algorthm s able to compute accurate flow vectors for features that do not contan suffcent local nformaton to be accurately tracked, whle the tradtonal algorthm fals n these locatons Concluson In ths paper we have combned the deas of Lucas- Kanade and Horn-Schunck n the opposte manner as that of Bruhn et al. [4]. Instead of aggregatng local nformaton to mprove global flow, we aggregate global nformaton to mprove the trackng of sparse feature ponts. Because of ther For more results and vdeos of ths sequence, see stb/research/jonttrackng.

5 Rubber Whale Hydrangea Venus Dmetrodon Algorthm AE EP AE EP AE EP AE EP Standard LK [3] Jont LK (ths paper) Table 1. Quanttatve comparson of the two algorthms on mages wth ground truth, showng the average angular error (AE) n degrees and the average endpont error (EP) n pxels. Rubber Whale Hydrangea Venus Fgure 1. Results of standard Lucas-Kanade (top) and jont Lucas-Kanade (bottom) on mage pars from three sequences. Red dots ndcate tracked feature ponts, and red lnes show the dsplacements (scaled for dsplay). The latter algorthm exhbts fewer erroneous dsplacements n regons of repettve texture. sparsty, the moton dsplacements of neghborng features cannot smply be averaged as s commonly done. Rather, an affne moton model s ft to the neghborng features, and the resultng expected flow vector s used n performng the Newton-Raphson teratons for computng the dsplacement of a partcular feature. By ncorporatng off-dagonal elements nto the otherwse block-dagonal trackng matrx, sgnfcantly mproved results are obtaned, partcularly n areas of repettve texture, one-dmensonal texture (edges), or no texture. Many mprovements are possble wth ths work. Most notably, the smoothng of moton dsplacements across moton dscontnutes wll create artfacts n the resultng flow felds. To solve ths problem, robust penalty functons or segmentaton algorthms can be employed. In addton, explct modelng of occlusons would reduce the effects of drastc unpredctable changes wthn the feature wndow tself due to the appearance and dsappearance of surfaces. Fnally, the dsplacements of sparse feature ponts can be used as a startng pont for nterpolatng a dense flow feld. We are explorng these deas n our current research. References [1] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unfyng framework. Internatonal Journal of Computer Vson, 56(3): , [2] S. Baker, D. Scharsten, J. Lews, S. Roth, M. J. Black, and R. Szelsk. A database and evaluaton methodology for optcal flow. In Proceedngs of the Internatonal Conference on Computer Vson, [3] J.-Y. Bouguet. Pyramdal mplementaton of the Lucas Kanade feature tracker. OpenCV documentaton, Intel Corporaton, Mcroprocessor Research Labs, [4] A. Bruhn, J. Weckert, and C. Schnörr. Lucas/Kanade meets Horn/Schunck: Combnng local and global optc flow methods. Internatonal Journal of Computer Vson, 61(3): , [5] J. Costera and T. Kanade. A mult-body factorzaton method for moton analyss. In Proceedngs of the Internatonal Conference on Computer Vson, pages , [6] D. DeCarlo and D. Metaxas. Optcal flow constrants on deformable models wth applcatons to face trackng. Internatonal Journal of Computer Vson, 38(2):99 127, July [7] G. D. Hager and P. N. Belhumeur. Effcent regon trackng wth parametrc models of geometry and llumnaton. IEEE

6 Standard Lucas-Kanade Jont Lucas-Kanade Fgure 2. TOP: A close-up of the repettve texture from the toprght corner of the Rubber Whale mage. BOTTOM: The results of the two algorthms (moton vectors are scaled for dsplay). The actual moton s between 0.78 and 1.3 pxels to the left wth almost no vertcal moton (less than 0.1 pxels). Wth the standard algorthm (left), many pxels move ten or more pxels n the vertcal drecton, whle the jont algorthm accurately estmates the moton of all the pxels. mage gradent magntude Standard Lucas-Kanade Jont Lucas-Kanade Fgure 3. Comparson of the two algorthms on a relatvely untextured scene. The standard algorthm computes erroneous results for many features, whle the jont algorthm computes accurate flow vectors even for features on ntensty edges or n untextured regons. Transactons on Pattern Analyss and Machne Intellgence, 20(10): , Oct [8] B. K. P. Horn and B. G. Schunck. Determnng optcal flow. Artfcal Intellgence, 17(185): , [9] H. Jn, P. Favaro, and S. Soatto. Real-tme 3-D moton and structure of pont features: Front-end system for vsonbased control and nteracton. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, [10] H. Jn, P. Favaro, and S. Soatto. Real-tme feature trackng and outler rejecton wth changes n llumnaton. In Proceedngs of the Internatonal Conference on Computer Vson, [11] N. K. Kanhere, S. J. Pundlk, and S. T. Brchfeld. Vehcle segmentaton and trackng from a low-angle off-axs camera. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton (CVPR), pages , June [12] S. J. Km, J.-M. Frahm, and M. Pollefeys. Jont feature trackng and radometrc calbraton from auto-exposure vdeo. In Proceedngs of the Internatonal Conference on Computer Vson, [13] R. Lublnerman, M. Sznaer, and O. Camps. Dynamcs based robust moton segmentaton. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , [14] B. D. Lucas and T. Kanade. An teratve mage regstraton technque wth an applcaton to stereo vson. In Proceedngs of the 7th Internatonal Jont Conference on Artfcal Intellgence, pages , [15] I. Matthews, T. Ishkawa, and S. Baker. The template update problem. IEEE Transactons on Pattern Analyss and Machne Intellgence, 26(6): , June [16] F. Rothganger, S. Lazebnk, C. Schmd, and J. Ponce. Segmentng, modelng, and matchng vdeo clps contanng multple movng objects. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , [17] J. Sh and C. Tomas. Good features to track. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages , [18] J. Svc, F. Schaffaltzky, and A. Zsserman. Object level groupng for vdeo shots. In Proceedngs of the European Conference on Computer Vson, volume 2, pages 85 98, [19] C. Tomas and T. Kanade. Shape and moton from mage streams under orthography: A factorzaton method. Internatonal Journal of Computer Vson, 9(2): , [20] T. Tommasn, A. Fusello, E. Trucco, and V. Roberto. Makng good features track better. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, [21] L. Torresan, D. Yang, G. Alexander, and C. Bregler. Trackng and modellng non-rgd objects wth rank constrants. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, [22] S. Šegvć, A. Remazelles, and F. Chaumette. Enhancng the pont feature tracker by adaptve modellng of the feature support. In Proceedngs of the European Conference on Computer Vson, pages , [23] S. Zhlong and R. Quq. Image regstraton based on KLT feature tracker n mage mosacng applcaton. In Proceedngs of the 5th Internatonal Conference on Sgnal Processng (ICSP), volume 2, pages , 2000.