Computation of Fluid and Particle Motion from Time Sequenced Image Pair: A Global. Outlier Identification Approach. Nilanjan Ray 1, Member, IEEE

Transcription

1 Computaton of Flud and Partcle Moton from Tme Sequenced Image Par: A Global Outler Identfcaton Approach Nlanan Ray 1, Member, IEEE Abstract Flud moton estmaton from tme sequenced mages s a sgnfcant mage analyss task. Its applcaton s wdespread n expermental fludcs research and many related areas lke bomedcal engneerng and atmospherc scences. In ths paper, we present a novel flow computaton framework to estmate the flow velocty vectors from two consecutve mage frames. In an energy mnmzaton-based flow computaton, we propose a novel data fdelty term, whch 1 can accommodate varous measures, such as cross-correlaton or sum of absolute or squared dfferences of pxel ntenstes between mage patches, has a global mechansm to control the adverse effect of outlers arsng out of moton dscontnutes, proxmty of mage borders, and 3 can go hand-n-hand wth varous spatal smoothness terms. Further the proposed data term and related regularzaton schemes are both applcable to dense and sparse flow vector estmatons. We valdate these clams by numercal experments on benchmark flow data sets. Keywords: optcal flow, partcle mage velocmetry, outlers, moton estmaton, optmzaton. EDICS: ARS-IVA Copyrght c 010 IEEE. Personal use of ths materal s permtted. However, permsson to use ths materal for any other purposes must be obtaned from the IEEE by sendng a request to pubs-permssons@eee.org. 1 Emal: nray1@ualberta.ca. Afflaton: Department of Computng Scence, Unversty of Alberta, -1 Athabasca Hall, Edmonton, AB T6GE8, Canada. Phone: Fax:

2 1. Introducton Partcle Image Velocmetry PIV s a standard process to measure the nstantaneous veloctes of flud flow. PIV has been used by the flud mechancs communty snce early 1980 for macroscale flows. PIV has a range of applcatons n flud dynamcs related to flud flow n bomedcal, atmospherc scences, and other related felds [9]. Gven a sequence of mages of flud flow through flud channels or tunnels, the obectve s to estmate the dense velocty vectors for flud flow vsualzaton. Over the years, the velocty vectors for PIV are evaluated prmarly by computer vson algorthms see [3], [33]. For a recent survey on flow estmaton technques, we refer the reader to [17]. In general, the flow estmaton technques from a tme sequenced dgtal mage par can be dvded nto two classes: technques usng mage patch-based measures [5], [9] and those usng the prncples of optcal flow [10], [16], [3], [35]. Recently, the formal connecton between optcal flow and flud flow has been establshed by Lu and Shen [0]. In mage patchbased technques, the PIV mages are dvded nto a set of nterrogaton tles or patches based on Nyqust crtera. These tles can be overlappng or non-overlappng dependng on the nature of the applcaton. Then, maxmum cross-correlaton methods establsh correspondence between these tles n consecutve mage frames. Flow vectors are computed as the vectors onng the centers of correspondng tles. Instead of cross-correlaton, sum of absolute dfferences between mage patches can also be employed. A sgnfcant demert of ths smple patch-based approach s that t often yelds spatally non-coherent or non-smooth moton vectors. So, a post processng, such as medan flter, needs to be appled after moton vectors are computed by maxmum crosscorrelaton. Post processng s ad-hoc, and often, t fals to adequately handle moton dscontnutes and outlers.

3 The classcal optcal flow model assumes ntensty constancy between two tme sequenced mages [18]. It does not look at tles/mage patches; ts computaton s typcally dfferental equaton based. One advantage here s the use of regularzaton or smoothness wthn the optmzaton framework, unlke the ad-hoc post processng n the mage patch-based methods. The classcal frst order regularzaton by Horn and Schunck [18] drectly resembles Markov random feld [19] type spatal smoothness. Subsequently, many regularzaton/smoothness crtera have evolved over tme; among them notable ones are Black and Anandan s robust regularzers [3], Lukas-Kanade method [], and others. A dv-curl dvergence and curl regularzaton functonal has been used to deal wth moton dscontnutes and large moton felds [10], [16]. Whle these aforementoned optcal flow methods apply regularzaton wthn an unconstraned optmzaton framework, n a more recent work, a constraned optmzaton method s utlzed to deal wth regularzaton for turbulent flows [15]. For flud flow computaton, Corpett et al. [10] have modfed the classcal optcal flow data fdelty term by ntroducng dvergence of flud flow that accounts for the physcs of ntensty varatons. Dvergence-based optcal flow constrant has also been llustrated n the work of del Bmbo et al. before [1]. In addton to the dvergence-based data fdelty term, Corpett et al. [10] ntroduced regularzatons that penalze large dvergences and vortctes of flud flow. We refer to ths regularzaton as the dv-curl term n ths paper. Utlzaton of the physcs of the flow n the computaton has also been attempted before, where conservaton of mass s used [13], [35]. Mass conservaton prncple for flud flow wthn the optcal flow computaton s formally ustfed by Lu and Shen [0]. Lu and Shen also provde examples of physcally grounded data models for flud moton estmaton n varous flow vsualzaton

4 technques [0]. Parameterzed flud flow computaton has also been utlzed n the optcal flow framework [4]. Recently, Hetz et al. [16] have ntroduced an approach combnng both correlaton and optcal flow model assumng the flud dsplacement feld as a combnaton of large and small scales. In the work of Hetz et al. [16] both data fdelty term, whch encourages smoothness of mage ntenstes and ther gradents, and the regularzaton term, whch enforces smoothness of the dvergence and curl of ntensty varatons, are extended for large and small-scale dsplacements. The key dea of ther proposal s to replace the coarse velocty estmates of a mult-resoluton scheme by dense large scale dsplacement estmates derved from a collecton of correlaton-based vectors. Along the same drecton of research, Stenbrucker et al. [34] have expermentally establshed on a reduced computer vson benchmark, the superorty of data terms, such as, normalzed cross-correlaton or sum of absolute dfferences between small mage patches over conventonal pxel dfferences wth or wthout robust penalty. The survey artcle [17] also artculates the mportance of combnng correlaton-based data terms wth varatonal penalty terms n computng optcal flow. Apart from the data fdelty terms and regularzaton technques, another crucal component n the optcal flow estmaton s the ablty of a method to deal wth outlers. Black and Anandan have successfully establshed the use of robust penalty methods n optcal flow n the past [3]. In a recent enhancement to ther technque, Sun et al. emphaszes the use of medan flters n the ntermedate stages of flow computaton amed at reducng the adverse effect of outlers, moton dscontnutes, etc. [31].

5 Overall, gven that an mage s dvded nto a set of overlappng tles or mage patches, we pose the moton estmaton problem as a matchng between these tles and a correspondng set of tles n the next mage n the tme sequence. Ths part of the problem formulaton s smlar to the classcal correlaton/mage patch-based technque. For matchng costs, one can use correlaton or sum of squared ntensty dfferences. However, as mentoned before, one potental dffculty of ths matchng would be generaton of spatally non-smooth flow feld, whch s generally undesrable. To overcome ths dffculty, we propose a cost mnmzaton framework, whch wll smultaneously solve for the spatally coherent flow feld and tle matchng. The framework s general at several levels. Frst, dfferent photo-metrc cost functons can be ncorporated. Second, several optcal flow related flow smoothng/regularzaton terms can be ncorporated. Thrd, a pror nformaton about the flow, as for example, a parabolc, or a vortex type velocty, can be ncorporated nto the proposed framework. Fourth and we beleve the most mportant of all, va the proposed novel data fdelty term we can control the negatve nfluence of outlers. Varous sources of outlers exst sharp moton dscontnutes, moton vectors at the mage boundares, where the flud flow s gong out of the feld of vew, and so on. To mnmze our ntended cost functon, we propose to use an alternatng mnmzaton AM technque also known as block coordnate descent []. The outlers are dentfed wthn the course of the AM technque by globally rankng the pxels where moton vectors are to be computed. Our proposed computatonal scheme can be best vewed as a successful marrage between dfferental equaton-based optcal flow method and block matchng va a sound optmzaton framework. Because of the low order Taylor seres approxmaton, the classcal data fdelty term n optcal flow method must resort to a mult-resoluton MR computatonal scheme to deal wth large dsplacements [4]. MR computaton destroys thn mage structures and

6 may fal to compute ther dsplacements. If advanced knowledge s avalable about the magntude of the moton feld, then one can apply a combnaton of sngle and mult-resoluton approach [8]. MR computaton s not a theoretcal requrement n our method; t s more of a computatonal convenence. Thus, our method s more relable for computaton of large moton of thn structures. These motons often arse n the bomedcal mage analyss applcatons, as llustrated n the results secton. Addtonally, va the proposed framework, we can compute sparse moton felds n approprate applcatons. We llustrate one such example of computng cell veloctes n the results secton. All these cases are adequately accommodated n the alternatng mnmzaton framework of computaton. In summary, the sngle most mportant contrbuton of our work s a data fdelty term for flow computaton that s versatle on at least three grounds: 1 photometrc measures, regularzatons, 3 sparse or dense moton computatons. Addtonally, we beleve the most crucal strength and novelty of the proposed data term s ts ablty to control outlers wth a global rankng. To the best of our knowledge, no prevous work n flow computaton has had a way to control the effect of outlers globally. In the next secton, we lay down our proposed cost optmzaton framework for flow computaton. In Secton 3, we dscuss alternatng optmzaton technque for mnmzng our cost functon. Secton 4 dscusses regularzaton terms that can be used n our framework. Secton 5 provdes results and comparsons. Addtonally, Secton 5 also explans both vsually and numercally our global outler dentfcaton method and ts mportance n flow moton computaton. Secton 6 concludes our work.

7 . Proposed Flow Computaton Framework To descrbe the proposed framework, we begn wth some notatons. Ix, y gray scale mage ntensty at tme t on pxel locaton x, y Jx, y gray scale mage ntensty at tme t+1 on pxel locaton x, y ndex to denote a pxel locaton x, y on mage I ndex to denote a pxel locaton X, Y on mage J N locatons on mage J neghbourng to the pxel x, y on mage I a 0-1 assgnment matrx between locatons on I and on J c photometrc cost when pxel locaton on I s matched to a pxel locaton on J u, v D two dmensonal flow vector at pxel locaton x, y on mage I The proposed framework of flow computaton can be nterpreted as a spatally coherent block matchng technque. Ths s acheved by a novel data fdelty term we refer to as assgnment-based data ABD fdelty term. ABD fdelty term has two components: photometrc and geometrc. The photometrc component c behaves as a block matchng cost,.e., t s responsble for choosng a pxel locaton x, y on the frst mage I that matches best wth a locaton X, Y on the second mage J. The geometrc part of the data term assumes that there s a flow feld u, v, whch s n close agreement wth the dsplacement vector X x, Y y. The proposed ABD fdelty term s defned as: d a, u, v a [ ], N c X x u Y y v 1

8 where s a user-tuned non-negatve weghtng parameter that balances the sgnfcance of the photometrc and the geometrc components. The bnary 0/1 assgnment varables a ensure that for each pxel locaton, at the most a sngle matchng pxel locaton s selected. All the varables are compactly denoted by vector notatons a, u, v. The ABD term 1 can be combned wth a spatal smoothness/coherence/regularzaton term r as follows: f a, u, v d a, u, v r u, v, where s a user-tuned non-negatve weghtng parameter controllng the smoothness of the soluton. The cost functon s subect to the followng constrants: a { 0,1}, ; a 1, ; a K. N 3 The second set of constrants n 3 ensures that a pxel locaton s matched at most once. The last constrant equaton n 3 avods a trval soluton of all zeros for a by requrng the total number of matches to be K. Ths constrant s also responsble for controllng the effect of outlers explaned n the next secton. We note that we can mnmze the cost functon f n cascaded stages, frst creatng g: g a mn u, v f a, u, v, 4 then mnmzng g wth respect to the bnary varables a. However, at ths pont we also note that we can relax the bnary varables to a set of contnuous varables a polytope as follows: P { a :0 a 1, a 1, a K}. N N 5

9 It s noteworthy that the functon g a defned over the polytope P s concave. The vertces of P are bnary 0/1 vectors. Because a concave functon defned on a polytope acheves ts mnmum on a vertex of the polytope [1], the contnuous set 5 and the dscontnuous set 3 are equvalent constrants for mnmzaton of. Establshng the concavty of g s easy, because the data term 1 s lnear n a : consder two vectors: a, b P, and a real number [ 0,1], then, g a 1 b mn f a 1 b, u, v u, v mn f a, u, v 1 u, v mn f b, u, v u, v g a 1 g b. 3. Mnmzaton of Cost Functon and Computaton of Flow Vectors Ths secton descrbes technques for mnmzng the cost functon to compute moton vectors. Mnmzng a concave functon s NP-hard n general [1]. However, often local mnmzers, whch can be found effcently, suffce the needs n an applcaton. Here, we employ alternatng mnmzaton AM to fnd a local mnmum of. Alternatng Mnmzaton AM Step 1: Intalze Step : Repeat the followng two steps untl convergence: Step a: Mnmze wth respect to,.e., compute: Step b: Update a by solvng the lnear program: Alternatng mnmzaton AM s sometmes known as block coordnate descent and s a standard technque of optmzaton []. In the closest context, AM was used for fttng ellpses and was shown to converge to a local mnmum of a concave cost functon [30]. The advantage

10 of AM for a concave functon s ts fast convergence [30], whch s also llustrated n the results secton. The dsadvantage s that the result s dependent on the startng vector n Step 1. We use a smple strategy for computng the ntal vector that works qute well for the flow computaton problem: for each select the ndex, k arg mn { c }, and set a k = 1, ntally. Step a of the aforementoned AM algorthm depends on the choce of regularzaton functon r n. However, most regularzatons beng convex functons, Step a s a convex optmzaton problem. Step b, solves a lnear program over the polytope P, hence t s a convex optmzaton too. Thus, AM solves two convex optmzatons teratvely to compute a local mnmzer of the concave functon 4. However, we do not need to nvoke any lnear program routne here, snce the structure of the polytope P s smple. In step b, for each pxel locaton, the lowest cost s N computed as: s mn{ c X x u Y y v }. N Next, these mnmum costs {s } are sorted n the ascendng order and the leadng K ndces are chosen to set these a s to 1. In realty, f the total number of pxels n mage I s M, then we use the rato K/M as a user tuned parameter. Ths partcular stage of AM s responsble for outler dentfcaton by global rankng, because we are dscardng M-K pxel locatons from computatons that have the worst ABD cost components. Durng the course of AM, these outler pxel locatons usually vary, because the geometrc cost component part n ABD changes from one teraton to the next. However, because of the guaranteed convergence of AM, the set of outlers.e., worst M-K pxel locatons accordng to ABD also converges at the end of the AM algorthm. The Results secton provdes some nsght how the global outler dentfcaton mechansm works. Although, MR computaton s not an absolute requrement n the proposed method, t greatly helps to speed up the computaton. To complete our algorthmc descrptons, we provde

11 below how an MR pyramdal scheme can be mplemented n conuncton wth the proposed AM technque. Ths MR scheme s smlar to the one used to compute dense optcal flow [4]. In the followng MR algorthm, a user nput parameter s the number of pyramd levels L. In Step 1, the orgnal nput mage par s denoted as I L and J L. Whle buldng the pyramd, an antalasng flter s used before applyng subsamplng. The mage par at the topmost level of the pyramd s denoted by I 1 and J 1. Mult-Resoluton Flow Vector Computaton Step 1: For an nput level pyramd L, compute MR pyramds {I l, J l }, l=1,,, L. Step : For l = L down to 1 If l = L Compute photometrc cost c between the mage par I L and J L. Use AM to compute flow vector u, v. Else Up-sample u, v wth the same factor used to buld the pyramd. Warp the second mage J l usng the translaton vector u, v to compute J warp. Compute photometrc cost c between the mage par I l and J warp. Use AM to compute a correcton flow vector p, q. Add correcton vector: u, v u, v p, q. End End We now dscuss the photometrc cost computaton. The ABD fdelty term 1 or the overall cost functon does not enforce any restrcton on the photometrc cost c and we can choose a varety of cost functons. For example, a sum of absolute dfferences of mage ntenstes between mages I and J can be used: c I x m, y n J X m, Y m n n. 6 We have observed that wth the MR flow vector computaton algorthm, the wndow sze specfed by m and n n 6 can be as small as {-1, 0, 1} by {-1, 0, 1},.e., a 3 pxel by 3 pxel wndow, to produce very compettve results. The pxel locatons x, y on mage I are always

12 consdered nteger pxel coordnates n all the experments here, whle X, Y on mage J are consdered subpxel locatons allowng fractonal coordnate values. The Results secton specfes our choce of subpxel step lengths for varous experments. One can also use negatve of correlaton-based measures for the photometrc cost c. For computatonal speed, we keep the neghbourhood sze N small. Wth the MR computaton, at each pyramd level we only lmt N to be a 1.5 pxels by 1.5 pxels wndow. Thus, for each, c s never needed to be computed beyond dsplacements more than 1.5 pxels n ether drecton. 4. Regularzaton Functons In ths secton we dscuss the regularzaton functon r. One of the earlest works by Horn and Schunck [18] on the dense optcal flow has used a frst order smoothng functon: 1 u u v v r dxdy. 7 x y x y For the sake of notatonal convenence, we have defned r n contnuous functonal form 7. In recent developments, a dv-curl regularzaton has been proposed [10], [16]: r 1 u x v y x 1 v x dxdy y u y x dxdy, y 8 where spatally smooth correcton terms x, y and x, y have been added to the dvergence and the curl of the velocty feld ux, y, vx, y, respectvely, before penalzng ther magntudes. s a non-negatve user tunable parameter here. Both these regularzatons 7 and

13 8 can be used n the proposed cost functon. For the regularzaton 8, Step a of AM algorthm solves four coupled Posson s equatons of the followng dscrete form:, , , 4, 4 w e n s w e s n n s w e w e s n w e n s w e s n n s w e w e s n N N N N v v u u v v u u a y Y v v v v v a v a x X u u u u u a u 9 where the ndces n, s, e and w respectvely denote north, south, east and west pxel locatons of the th pxel. One can apply sem-mplct numercal scheme [6] to solve 9. For a more accurate mplementaton scheme of dv-curl regularzaton, the reader s referred to [36]. Besdes the aforementoned explct regularzaton functons, our formulaton can also use mplct parametrc regularzaton method. For example, f we know that a flow s of pure rotatonal vortex nature and can be parameterzed by -by, bx assumng the orgn to be the center of the vortex and b to be the unknown parameter, then the proposed cost functon can be defned as ]. [, N bx y Y by x X c a b a f 10 Once agan Alternatng Mnmzaton AM can be used for 10. Here, Step a n AM algorthm fnds the least squares soluton for b from 10 assumng a s to be fxed weghts. Step b remans unaltered.

14 Fg.1: Tranng mage par a Std07-1, b Std07-, c ground truth and flow vectors computed by dfferent methods: d ABD+DC, e OFC+CC+DC, f OFC+DC, g BA [3], [11], h correlaton [5]. The velocty vectors are scaled for better vsualzaton.

15 5. Results We have carred out flow vector computatons n varous settngs: lamnar flow, turbulent flow, and sparse moton computatons. Addtonally, we llustrate nose senstvty, convergence, performance senstvty to parameter values of our proposed method. We used two sets of test mages for PIV experments: one from PIV challenge ste [8] and the other from the cte [5]. The former data set consttutes D lamnar flows and the latter contans D turbulent flows. In the followng experments, we compare the proposed ABD-based flow computaton wth three other flow computaton methods: a Black-Anandan BA method [3] we have used publcly accessble code [11], b optcal flow constrant wth dv-curl regularzaton OFC+DC and c optcal flow constrant wth dv-curl regularzaton and cross correlaton OFC+CC+DC. The latter two methods are smplfed versons of Corpett et al. s [10] and Hetz et al. s [16] works, respectvely. The smplfcatons correspond manly to the use of a smpler optmzaton procedure: a centered fnte dfference scheme wthout any temporal consstency constrant. The same optmzaton procedure has been adopted for the proposed method to make the comparsons far. We refer to our proposed method as ABD+DC assgnment based data term wth dv-curl regularzaton. 5.1 PIV Experments on Lamnar Flow In ths secton we use eght standard partcle mage sequences of lamnar flow [8]. We refer to these sequences as Std01 through Std08. Each sequence conssts of four tme sequenced mages Std0*-1, Std0*-, Std0*-3 and Std0*-4. We use the frst mage par.e., Std0*-1 and Std0*- for tranng the methods,.e., tune the user nput parameters for ther best performances. We use root mean squared error RMSE as the performance measurement crteron. Thus, for every

16 method, we have eght dfferent parameter settngs for eght sequences obtaned by the tranng phase. For the testng phase, we use the tranng parameter settng for sequences of test pars,.e., Std0*- and Std0*-3 and Std0*-3 and Std0*-4. Fg.1 shows an example tranng mage sequence par Std07-1 and Std07-, ground truth vectors and computed flow vectors by dfferent methods. In Table 1, we llustrate RMSE s computed on the tranng mage pars. These RMSE values are the best values obtaned by exhaustvely searchng the user tunable parameters space. We use a coarse-to-fne grd search technque for the exhaustve search. The parameter space s a 4D space conssts of four scalar parameters: K/M,,, refer to equatons 1,, and 8 for the proposed method,.e., ABD+DC method. To llustrate an example, after tranng wth Std01-1 and Std01- mage par, we obtan =0.15, =5, =, and K/M = 0.5. Then, n the testng phase, we have used these parameter values for computng moton wth mage pars Std01-, Std01-3 and Std01-3, Std01-4. We keep the same MR pyramd level L=6 wth pyramd spacng 1.5 for all of the methods descrbed here. The pxel neghborhood N see ABD fdelty term 1 for all the experments s chosen such that f N, then x X 1. 5 and y Y 1.5. The subpxel locatons X, Y have an accuracy of half-pxel length, whle x, y has full pxel accuracy,.e., they have ntegral mage coordnates wth unty spacng. Table 1 also llustrates Barron s angular measurement [1] n degrees referred to as Angle n the all the Tables and Fgures reported here. A low angular measurement shows better algnment of computed moton vectors wth the ground truth vectors. Whle RMSE weghs n moton vector magntudes, Barron s angular measurement weghs all vectors unformly. From Table 1, t s clear that the method that came closest n performance to the proposed method s Black-Anandan s method. Thus, we carry out Wlcoxon sgned rank test [9], between ther RMSE values and ours, and

17 found that our proposed method s superor wth p-value Usually a p-value of 0.05 or less can be consdered sgnfcantly dfferent between two methods. For the angular measures Wlcoxon sgn rank test also yelded statstcally sgnfcant results n favor of the proposed method. The average tme taken on a Wndows XP Intel Core GHz desktop s as follows. ABD+DC Matlab mplementatons took 31 seconds, BA method Matlab wth C mplementaton took 16 seconds, whle OFC+DC Matlab mplementatons and OFC+CC+DC Matlab mplementatons took 55 seconds and 70 seconds respectvely. The computaton tme of the proposed method depends on subpxel accuracy for locatons X, Y, neghborhood sze N, number of levels L n the MR scheme. Sequence\Method OFC+DC OFC+CC+DC BA ABD+DC RMSE Angle RMSE Angle RMSE Angle RMSE Angle Std Std Std Std Std Std Std Std Table 1: RMSE values for dfferent methods on tranng pars Std0*-1 and Std0*-. Sequence\Method OFC+DC OFC+CC+DC BA ABD+DC RMSE Angle RMSE Angle RMSE Angle RMSE Angle Std Std Std Std Std Std Std Std Table : RMSE values on test pars Std0*- and Std0*-3 for dfferent methods. Next, we use the test partcle mage sequence pars and report the RMSE s of dfferent methods n Table and Table 3. We also compute Wlcoxon sgned test for the RMSE and values for Tables and 3 combned, and found that the proposed method s performng

18 statstcally sgnfcantly better than the Black and Anandan s method wth p-value Smlar sgn rank test for Barron s angular measurements also establshed superorty of the proposed method on ths PIV data set. Sequence\Method OFC+DC OFC+CC+DC BA ABD+DC RMSE Angle RMSE Angle RMSE Angle RMSE Angle Std Std Std Std Std Std Std Std Table 3: RMSE values for test pars Std0*-3 and Std0*-4 obtaned wth dfferent methods. 5. PIV Experments on Turbulent Flow In ths secton, we use the D turbulent data set [5] for PIV experments. Here for all the methods, we use only the frst mage par for tranng,.e., for parameter tunng. Tranng parameters for ABD+DC were found as follows: =0.35, =1.75, =1.75, and K/M = For the next 99 tme sequenced mages we keep these parameter values fxed and perform moton computaton. For MR computaton scheme, L s chosen as 6 and pyramd spacng s chosen as 1.5. These two parameters are kept fxed for all the competng methods. For ABD+DC, the subpxel locatons X, Y are allowed to have one tenth 1/10 pxel length accuracy for very accurate flow computaton. The neghborhood consdered here only N s only 1 pxel by 1 pxel n sze at each mult-resoluton level. So, cardnalty of N s 11x11 = 11. Fg. shows an mage par, ground truth flow vector feld, color-coded moton felds obtaned wth all competng methods. Notce that the proposed ABD+DC and BA yelded the closest approxmaton to the ground truth feld for ths data set. Numercal results are provded wth box plots n Fg. 3 showng medans and percentle ranges for both RMSEs and Barron s angular

19 measurements. Once agan at 5% statstcal sgnfcance level, ABD+DC outperformed other methods here. Fg. a and b a partcle mage par showng D turbulent flow [5]; c shows ground truth moton vector [5]; d shows color-coded dense moton feld for the ground truth; e Color-coded moton feld computed by ABD+DC; f Flow vector feld computed by ABD+DC; g, h and show color-coded moton feld computed by BA, OFC+DC and OFC+CC+DC, respectvely. Fg 3: Performance of all methods on D turbulence data set [5]. a RMSE values, b Barron s angular measurements n degrees.

20 Fg. 4: PSNR versus performance plots for D turbulent data [5]. a Mean RMSEs and b mean Barron s angular measurements degrees are plotted for all the methods. Fg.5: Images a Std01-1 and b Std01-. The mage A for c K/M=0.75 and d K/M=0.5. In our next experment on D turbulent data set [5], we llustrate the advantage of usng a patch-based term. Intutvely, a patch-based term should be more reslent to nose. Ths ntuton s supported n the followng experment. In all the D turbulent mages n the sequences, we added zero-mean Gaussan nose and repeated all the experments. The amount of added nose s measured by PSNR peak-sgnal-to-nose-rato. Fg. 4 shows mean RMSEs and mean Barron s

21 angular measurements degrees plots for four dfferent PSNR values. In Fg. 4, as the nose degrades the mages more, the performance gap between BA and ABD-based proposed method ncreases. The performance gaps between the proposed soluton and those of OFC+DC and OFC+CC+DC methods are observed to dmnsh as the mages degrade. At 0dB PNSR, both these methods performed better than BA method n our experments. 5.3 Algorthmc Analyss on PIV Experments In ths secton we provde analyss of our AM algorthm vs-à-vs the proposed ABD term. We frst llustrate both vsual and expermental evdences for the effectveness of the global rankngbased outler dentfcaton mechansm nherent to the proposed data fdelty term. Fg. 5a and 5b show a tranng mage par n lamnar flow: Std01-1 and Std01-. As mentoned n Secton 3, the rato K/M s a user nput parameter and s responsble for the outler control. So, we set two values of K/M here for the experment: 0.75 and 0.5. Fg. 5c and 5d show the mages A a. Brght pxel values n Fg. 5c and 5d mply A = 1, and the dark pxels mply A N = 0. Accordng to our proposed algorthm also evdent from equaton 9, A =1 mples that the data fdelty term s relable for the th pxel, whereas A =0, mples unrelable data or outlers. When A =0, the flow vector at th locaton s nterpolated from ts neghborng locatons k N wth A k = 1. The relable/unrelable nature of the data fdelty s very clearly depcted by Fg. 5c and 5d, especally at the rght border, where the flow vectors are movng the partcles out of the feld of vew also see the ground truth arrows n Fg. 1c. To the best of our knowledge, no dense optcal flow method has such an explct way of globally dentfyng and controllng the effect of outlers. Black and Anandan s method handles outlers locally wth robust statstcs and spatal medan flters. Spatal medan flter uses local ranks. However, as we have already

22 observed that the proposed method has outperformed Black and Anandan s method on both the test and the tranng sequences consdered here. The statstcal sgnfcance s partcularly pronounced for the test mage pars, revealed by the p-value. Image par\method Wthout Outler Identfcaton Wth Outler Identfcaton Std01-1 and Std Std0-1 and Std Std03-1 and Std Std04-1 and Std Std05-1 and Std Std06-1 and Std Std07-1 and Std Std08-1 and Std Table 4: RMSE values showng sgnfcance of K: tranng pars Std0*-1 and Std0*-. Image par\method Wthout Outler Identfcaton Wth Outler Identfcaton Std01- and Std Std0- and Std Std03- and Std Std04- and Std Std05- and Std Std06- and Std Std07- and Std Std08- and Std Table 5: RMSE values showng sgnfcance of K: test pars Std0*- and Std0*-3. Further, to establsh that our global outler dentfcaton mechansm s ndeed effectve, we carry out an experment on the eght lamnar flow mage sequences. For the frst set of experments, we fx K/M=100%,.e., there s no outler dentfcaton mechansm. In the second case, we use K/M values from our prevous experments. Table 4 reports RMSE s for the tranng mage pars. For the frst column n Table 4, three parameters:,,, are found out by coarse-tofne search. For the test mage pars, these parameters are used to produce RMSE s reported n Table 5 and 6. In all the three tables 4, 5 and 6, the second columns are coped from prevous Tables 1,, and 3 as they refer to the experments wth outler dentfcaton. These RMSE values demonstrate that the outler dentfcaton makes a sgnfcant dfference n performance.

23 Image par\method Wthout Outler Identfcaton Wth Outler Identfcaton Std01-3 and Std Std0-3 and Std Std03-3 and Std Std04-3 and Std Std05-3 and Std Std06-3 and Std Std07-3 and Std Std08-3 and Std Table 6: RMSE values showng sgnfcance of K: test pars Std0*-3 and Std0*-4. Parameter \ Pyramd Level Table 7: Number of AM teratons requred for convergence on D turbulent data. To better understand the nature of the proposed ABD-based data term and the convergence of AM algorthm, we performed an experment to see when AM acheves convergence. Table 7 shows values and dfferent levels n the MR computaton scheme. The entres n the Table 7 show the number of teratons AM had taken to converge. Note that because AM s mnmzng a concave functon over a smple polytope, we dd not have to use any tolerance value for ths convergence. We stopped the AM algorthm when the old bnary values of assgnment varables matched the new values once agan, bnary exactly. For these experments, we had kept =1.75, =1.75 fxed. Notce that when the nfluence of the geometrc component s sgnfcant n ABD, the algorthm requred more teratons to converge. Ths s expected, because the photometrc term s statc,.e., t does not change wth teratons; on the other hand, because the flow vectors are recomputed at every teraton, the geometrc term s dynamc t vares wth teratons. Thus, more teratons are requred for convergence when the weght s more on the dynamc component n ABD.

24 RMSE Angle Table 8: versus RMSE and Angle for D turbulent dataset RMSE Angle Table 9: versus RMSE and Angle for D turbulent dataset RMSE Angle Table 10: versus RMSE and Angle for D turbulent dataset We have carred out further experments to demonstrate the nature of parameter senstvty on the performance of our proposed ABD-based method. In tables 8, 9, and 10 we show RMSE and Barron s angular measurements at dfferent parameter values. For each table we have kept two parameter values fxed and vared the thrd parameter value. For producng Table 8, we only vared, whle kept =1.75, =1.75 fxed. For Table 9, we vared, but kept =0.35, =1.75 fxed. For Table 10, we kept =0.35, =1.75, and vared. From these tables, we observe that RMSE and Barron s angular measurements are qute stable at the optmum parameter values. 5.4 Sparse Moton Computaton In many applcatons, moton computaton s ether sparse, or dense wth concentraton n certan areas of the mage. In ths secton, we provde two such examples from bomedcal applcatons and compute these moton by the proposed ABD fdelty based framework. The purpose s to demonstrate extensve characterstcs of our method. One of the examples demonstrates the usefulness of moton computaton at the orgnal resoluton of the mage par. Computng moton for thn structures s challengng wth optcal flow methods that rely on optcal flow constrant equaton. The reason s the sub-samplng of the mages n the MR computaton scheme, whch s essental for such methods to deal wth even moderately large dsplacements e.g., to 5 pxels. In contrast, the ABD fdelty term does not requre sub-

25 samplng of mages and hence the MR scheme s not an absolute requrement for our method. In Fg. 6 we llustrate two tme sequenced mcroscopy mages of endothelal cells. As observed n the mages, the structures for whch moton needs to be computed are delcate and thn. Such thn structures would almost nvarably be destroyed by repeated sub-samplng. Because these thn structures appear brght on mcroscopy mages, one can detect them by applyng a gray level threshold. Ths can be acheved by Otsu s adaptve threshold [6]. Our ABD term 1 along wth dv-curl regularzaton can compute the moton of the delcate thn structures by ncorporatng the threshold nformaton n the photometrc cost as follows: c 0, f R and x I x m, y X y n J X m, Y m n Y 0; n, otherwse. 11 where R s the set of pxels of I that are below the Otsu grayscale threshold value. So, R represents the regons of mage I other than the delcate cell structures. Note that 11 encourages no moton zero velocty for pxel locatons off the thn structures. Fg. 6c and 6e shows the results of AM computaton at the orgnal resoluton wth data term 11 and dv-curl regularzaton 8. We used the followng parameter values: K/M = 0.95, = 0.1, = and = 5. In contrast, Fg. 6d and 6f show the results of Black and Anandan optcal flow computaton, where a maxmum pyramd level L= s used. Note that the computed moton vectors n Fg. 6d and 6f are not only dense, they also have lost ther dstnctveness on the thn cell structures.

26 Fg.6: a and b Mcroscopy mage par of endothelal cells; c Moton computaton wth ABD +dv-curl d Moton vector computed by Black and Anandan s method [3], [11]. e and f Zoomed n results. All arrows are scaled up for better vsualzaton. The proposed data term can also be utlzed n vsualzng velocty vectors that are nherently sparse n nature. Consder another bomedcal applcaton: trackng a number of cells flowng through an n vtro assay two such tme lapsed mcroscopy mage frames are shown n Fg. 7a and 7b. Ths s essentally an obect trackng applcaton where number of obects to be tracked s large where bpartte graph matchng proves to be more useful than conventonal partcle flter [7]. Here we llustrate that bpartte matchng can be further utlzed n a framework where spatal coherence n the cell veloctes can be mposed. Let us assume n ths case that N and M cells are detected respectvely on mage frame 1 and frame. Let us denote ther locatons

27 by x, y and X, Y. We need to establsh 1-1 correspondences among them. Also, note that some of the detected cells may be false postves for whch we need no matches. In ths case the ABD fdelty term 1 practcally remans the same as 1: N M d a, u, v a [ c X x u Y y v ], However, we mpose 1-1 matchng wth at the most K matchng pars wth K-assgnment polytope [14]: Q { a :0 a 1, a 1, a 1, a K}. 13 The regularzaton for the cell velocty feld s defned as a frst order spatal smoothness term: r u, v u w u v w v, 14 k N k k k N k k where, as before, N s the neghborhood of th detected cell on frame 1. N can be defned by those detected cells n frame 1 that are wthn a crcle of a user defned radus centered at x, y. In our experment, we have taken ths radus to be 100 pxels. We set the weghts w k s as nversely proportonal to the dstance between x, y and x k, y k. Once agan we can solve ths problem usng Algorthm AM. Ths tme, Step b of AM solves a bpartte matchng problem for whch we use the lnear program n Matlab. Also, note that we do not need any pyramdal MR mplementaton here. The user suppled parameter K once agan plays an mportant role that of suppressng the effect of false cell detectons on both the frames. In our experment, we have chosen K/M=0.85. We defne the photometrc cost c as follows. If F and G are two small mage patches around th and th detected cells n frame 1 and frame, respectvely, then: c SAD F, G, 15

28 where the functon SAD computes sum of absolute dfferences between two mage patches. Ths photometrc cost s smlar to the data term used n [34]. Fg. 7a overlays computed veloctes on detected cells. In comparson, Fg 7c overlays the flow vectors on the frst frame, f K 1-1 assgnments are made only wth the photometrc cost 15. From ths vsual comparson, we can realze the sgnfcance of the regularzaton 14 and our proposed AM computaton scheme. Fg. 7: Human monocytes movng n cell assay. a and b show two tme lapsed frames. a Overlad velocty vector computed by AM. c Cell moton found on frst frame by one-to-one assgnments wth photometrc cost only. The arrow lengths are scaled up for the purpose of vsualzaton. 6. Conclusons In ths paper we propose a moton computaton scheme usng tme sequenced mage par. The ABD fdelty term s the prncpal contrbuton n our work. We have demonstrated a number of advantages of ABD. Among those, outler dentfcaton s one of the most crucal functons acheved by ABD. In our proposed method, outler dentfcaton s global: the pxels are globally ranked and a bottom percentle s dscarded from ABD. At these outler pxels, moton vectors are computed takng nto account ts neghborng vectors. We have demonstrated n the results

29 secton that such a global rankng mechansm outperforms local outler detecton strateges. The nterestng characterstc of our proposed framework s that ths global rankng and outler dentfcaton are all part of the optmzaton no computaton s added n an ad-hoc manner as post- or pre-processng. Addtonally, the proposed ABD s generc n nature. It s able to compute dense flow vectors as well as sparse moton llustrated by our examples usng bomedcal mages. Acknowledgements Ths work has been supported by NSERC dscovery grant. References [1] J.L. Barron, D.J. Fleet, and S.S. Beauchemn, Performance of optcal flow technques, Internatonal Journal of Computer Vson, vol.1, no.1, pp.43-77, [] D.P. Bertsekas, Nonlnear programmng: nd Edton, Athena Scentfc, [3] M.J. Black and P. Anandan, The robust estmaton of multple motons: Parametrc and pecewse-smooth flow felds, Comp. Vs. Im. Understandng, vol. 63, pp , [4] A. Bruhn, J. Weckert, and C. Schn orr. Lucas/Kanade meets Horn/Schunck: combnng local and global optc flow methods, IJCV, vol.61, no.3, pp.11 31, 005. [5] J. Carler and B. Weneke, Report 1 on producton and dffuson of flud mechancs mages and data, Flud proect delverable [6] A. H. Charles and T. A. Porschng, Numercal analyss of partal dfferental equatons. Englewood Clffs, NJ: Prentce-Hall, [7] A.S. Chowdhury, R. Chatteree, M. Ghosh, N. Ray, Cell trackng n vdeo mcroscopy usng bpartte graph matchng, ICPR 010.

30 [8] I. Cohen and I Herln, Non unform multresoluton method for optcal flow and phase portrat models: Envronmental applcatons, IJCV, vol.33, pp.9-49, [9] G.W. Corder and D.I. Foreman, Nonparametrc statstcs for non-statstcans: A step-by-step approach, New Jersey: Wlley, 009. [10] T. Corpett, D. Hetz, G. Arroyo, E. Memn, A. Santa-Cruz, Flud expermental flow estmaton based on an optcal-flow scheme, Experments n Fluds, vol.40, pp.80 97, 006 [11] last accessed September 010. [1] A. del Bmbo, P. Nes, and J.L.C. Sanz, Optcal flow computaton usng extended constrants, IEEE Trans. Image Processng, vol.5, no.5, pp , [13] J.M. Ftzpatrck, Method for calculatng velocty n tme dependent mages based on the contnuty equaton, Int. Conf. Computer Vson Pattern Recognton, pp.78-81, [14] J. Gll, S. Lnusson, The k-assgnment polytope, Dsc. Opt., vol., pp , 009. [15] P. Heas, E. Memn, D. Hetz, and P.D. Mnnn, Bayesan selecton of scalng laws for moton modelng n mages, Internatonal Conference on Computer Vson, 009. [16] D. Hetz, P. Heas, E. Mémn, J. Carler "Dynamc consstent correlaton-varatonal approach for robust optcal flow estmaton," Experm. n Fluds, vol.45, pp , 008. [17] D. Hetz, E. Memn, C. Schnorr "Varatonal flud flow measurements from mage sequences: synopss and perspectves," Experments n Fluds, vol.48, pp , 010. [18] B.K.P. Horn and B.G. Schunck, "Determnng optcal flow." Artfcal Intellgence, vol.17, pp , [19] S.Z. L, Markov random feld modelng n mage analyss, Sprnger-Verlag 001. [0] T. Lu and L. Shen, Flud flow and optcal flow, Journal of Flud Mech., vol.614, pp.53-91, 008.

31 [1] R. Horst, P.M. Pardalos, N.V. Thoa, Introducton to global optmzaton, Sprnger, 000. [] B. D. Lucas and T. Kanade, An teratve mage regstraton technque wth an applcaton to stereo vson, Proc. of Imagng Understandng Workshop, pp , [3] R. Larsen, K. Conradsen, B.K. Ersboll, Estmaton of dense mage flow felds n fluds IEEE T. Geoscence and Remote Sensng 36: 56-64, [4] E. Memn and P. Perez, Flud flow recovery by couplng dense and parametrc vector felds, ICCV, pp.60-65, [5] accessed n June 009. [6] Nobuyuk Otsu "A threshold selecton method from gray-level hstograms," IEEE Trans. Sys., Man., Cyber. Vol.9, pp.6 66, [7] C.H. Papadmtrou and K. Stegltz, Combnatoral optmzaton: Algorthms and complexty, Dover Publcatons, [8] accessed n September 010. [9] M. Raffel, C. Wllert, S. Wereley, J. Kompenhans, Partcle Image Velocmetry: A Practcal Gude, Sprnger, 007. [30] N. Ray, A concave cost formulaton for parametrc curve fttng: Detecton of leukocytes from ntravtal mcroscopy mages, IEEE ICIP 010. [31] D. Sun, S. Roth, and M.J. Black, Secrets of optcal flow estmaton and ther prncples, IEEE Conf. on Computer Vson and Pattern Recog., June 010. [3] M. Stanslas, K. Okamoto, C. J. Kahler, J. Westerweel, Man results of the Second Internatonal PIV Challenge, Experments n Fluds, vol.39, pp , 005. [33] M. Stanslas, K. Okamoto, C. J. Kahler, J. Westerweel, and F. Scarano, Man results of the Thrd Internatonal PIV Challenge, Experments n Fluds, vol.45, pp.7 71, 008.

32 [34] F. Stenbrucker, T. Pock, D. Cremers "Advanced data terms for varatonal optc flow estmaton." 15th Int l Workshop on Vson, Modellng und Vsualzaton, pp , 009. [35] R.P. Wldes, M.J. Amable, A-M. Lanzllotto and T-S. Leu Recoverng estmates of flud flow from mage sequence data, CVIU, vol.80, pp.46-66, 000. [36] J. Yuan, C Schnörr, and E. Memn, Dscrete orthogonal decomposton and varatonal flud flow estmaton, J. of Mathematcal Imagng and Vson, vol.8, no.1, pp.67-80, 007. Author Bography Nlanan Ray receved hs bachelor s degree n mechancal engneerng from Jadavpur Unversty, Calcutta, Inda, n 1995; hs master s degree n computer scence from Indan Statstcal Insttute, Calcutta, n 1997; and hs Ph.D. n electrcal engneerng from Unversty of Vrgna, USA, n 003. After postdoctoral and ndustral work experence, he s now an assstant professor at the Department of Computng Scence, Unversty of Alberta, Canada. Hs research area s mage and vdeo analyss: mage segmentaton, obect trackng, and moton analyss. He s a recpent of the CIMPA-UNESCO fellowshp for mage processng n 1999; and the best student paper award from IEEE Sgnal Processng Socety at IEEE Internatonal Conference on Image Processng, Rochester, NY n 00. Nlanan has coauthored two monographs: Bomedcal mage analyss: Trackng and Bomedcal mage analyss: Segmentaton.