Yair Weiss. Dept. of Brain and Cognitive Sciences. Massachusetts Institute of Technology. Abstract

Transcription

1 n: M.C. Mozer, M.I. Jordan and T. Petsche, edtors, Advances n Neural Informaton Processng Systems (997). Interpretng mages by propagatng Bayesan belefs Yar Wess Dept. of Bran and Cogntve Scences Massachusetts Insttute of Technology E0-20, Cambrdge, MA 0239, USA ywess@psyche.mt.edu Abstract A central theme of computatonal vson research has been the realzaton that relable estmaton of local scene propertes requres propagatng measurements across the mage. Many authors have therefore suggested solvng vson problems usng archtectures of locally connected unts updatng ther actvty n parallel. Unfortunately, the convergence of tradtonal relaxaton methods on such archtectures has proven to be excrucatngly slow and n general they do not guarantee that the stable pont wll be a global mnmum. In ths paper we show that an archtecture n whch Bayesan Belefs about mage propertes are propagated between neghborng unts yelds convergence tmes whch are several orders of magntude faster than tradtonal methods and avods local mnma. In partcular our archtecture s non-teratve n the sense of Marr [5]: at every tme step, the local estmates at a gven locaton are optmal gven the nformaton whch has already been propagated to that locaton. We llustrate the algorthm's performance on real mages and compare t to several exstng methods. Theory The essence of our approach s shown n gure. Fgure a shows the prototypcal ll-posed problem: nterpolaton of a functon from sparse data. Fgure b shows a tradtonal relaxaton approach to the problem: a dense array of unts represents

2 Y* Y* Y µ,σ µα,σα µ β,σ β a b c Fgure : a. a prototypcal ll-posed problem b. Tradtonal relaxaton approach: dense array of unts represent the value of the nterpolated functon. Unts update ther actvty based on local nformaton and the actvty of neghborng unts. c. The Bayesan Belef Propagaton (BBP) approach. Unts transmt probabltes and combne them accordng to probablty calculus n two non-nteractng streams. the value of the nterpolated functon at dscretely sampled ponts. The actvty of a unt s updated based on the local data (n those ponts where data s avalable) and the actvty of the neghborng ponts. As dscussed below, the local update rule can be dened such that the networ converges to a state n whch the actvty ofeach unt corresponds to the value of the globally optmal nterpolatng functon. Fgure c shows the Bayesan Belef Propagaton (BBP) approach to the problem. As n the tradtonal approach the functon s represented by the actvty of a dense array of unts. However the unts transmt probabltes rather than sngle estmates to ther neghbors and combne the probabltes accordng to the probablty calculus. To formalze the above dscusson, let y represent the actvty of a unt at locaton, and let y be nosy samples from the true functon. Atypcal nterpolaton problem would be to mnmze: J(Y )= w (y y ) 2 + (y y + ) 2 () Where we have dened w = 0 for grd ponts wth no data, and w = for ponts wth data. Snce J s quadratc, any local update n the drecton of the gradent wll converge to the optmal estmate. Ths yelds updates of the sort: y y + (( y + y + 2 y )+w (y y )) (2) Relaxaton algorthms der n ther choce of : ==(+w ) corresponds to Gauss-Sedel relaxaton and =:9=(+w ) corresponds to successve over relaxaton (SOR) whch s the method of choce for such problems [0]. To derve a BBP update rule for ths problem, note that that mnmzng J(Y ) s equvalent to maxmzng the posteror probablty ofy gven Y assumng the followng generatve model: y + = y + (3) y = w y + (4) Where N(0; R ), N(0; D ). The rato of D to R plays a role smlar to that of n the orgnal cost functonal. The advantage of consderng the cost functonal as a posteror s that t enables us to use the methods of Hdden Marov Models, Bayesan Belef Nets and Optmal

3 Estmaton to derve local update rules (cf. [6, 7, ]). Denote the posteror by P (u) =P(Y =ujy ), the Marovan property allows us to factor P (u) nto three terms: one dependng on the local data, another dependng on data to the left of and a thrd dependng on data to the rght of.thus: P (u) =c (u)l (u) (u) (5) where (u) =P(Y =ujy ; ); (u)=p(y =ujy +;N );L (u)=p(y jy =u) and c denotes a normalzng constant. Now, denotng the condtonal C (u; v) = P(Y =ujy =v), (u) can be wrtten n terms of (v): (u) =c Z v (v)c (u; v)l (v) (6) where c denotes another normalzng constant. A symmetrc equaton can be wrtten for (u). Ths suggests a propagaton scheme where unts represent the probabltes gven n the left hand sde of equatons 5{6 and updates are based on the rght hand sde,.e. on the actvtes of neghborng unts. Speccally, for a Gaussan generatng process the probabltes can be represented by ther mean and varance. Thus denote P N( ; ), and smlarly N( ; ) and N( ; ). Performng the ntegraton n 6 gves a Kalman-Flter le update for the parameters: w D Y + + R +( w D w D Y + w (the update rules for the parameters of are analogous) (7) D (8) + w D ) (9) So far we have consdered contnuous estmaton problems but dentcal ssues arse n labelng problems, where the tas s to estmate a label L whch can tae onm dscrete values. We wll denote L (m) = f the label taes on value m and zero otherwse. Typcally one mnmzes functonals of the form: J(L) = m V (m)l (m) m L (m)l + (m) (0) Tradtonal relaxaton labelng algorthms mnmze ths cost functonal wth updates of the form: L f(v ;L ;L ;L + ) () Agan derent relaxaton labelng algorthms der n ther choce of f. A lnear sum followed by a threshold gves the dscrete Hopeld networ updates, a lnear sum followed by a \soft" threshold gves the contnuous or mean-eld Hopeld updates and yet another form gves the relaxaton labelng algorthm of Rosenfeld et al. (see [3] for a revew of relaxaton labelng methods ).

4 To derve a BBP algorthm for ths case one can agan rewrte J as the posteror of a Marov generatng process, and calculate P (L (m) = ) for ths process.. Ths gves the same expressons as n equatons 5{6 wth the ntegral replaced by a lnear sum. Snce the probabltes here are not Gaussan, the ; ;P wll not be represented by ther mean and varances, but rather by a vector of length M. Thus the update rule for wll be: () c l (l)c (; l)l (l) (2) (and smlarly for.). Convergence Equatons 5{6 are mathematcal denttes. Hence, t s possble to show [6] that after N teratons the actvty of unts P wll converge to the correct posterors, where N s the maxmal dstance between any two unts n the archtecture, and an teraton refers to one update of all unts. Furthermore, we have been able to show that after n<n teratons, the actvty of unt P s guaranteed to represent the probablty of the hdden state at locaton gven all data wthn dstance n. Ths guarantee s sgncant n the lght of a dstncton made by Marr (982) regardng local propagaton rules. In a scheme where unts only communcate wth ther neghbors, there s an obvous lmt on how fast the nformaton can reach a gven unt:.e. after n teratons the unt can only now about nformaton wthn dstance n. Thus there s a mnmal number of teratons requred for all data to reach all unts. Marr dstngushed between two types of teratons { those that are needed to allow the nformaton to reach the unts, versus those that are used to rene an estmate based on nformaton that has already arrved. The sgncance of the guarantee on P s that t shows that BBP only uses the rst type of teraton { teratons are used only to allow more nformaton to reach the unts. Once the nformaton has arrved, P represents the correct posteror gven that nformaton and no further teratons are needed to rene the estmate. Moreover, we have been able to show that propagatons schemes that do not propagate probabltes (such as those n equatons 2) wll n general not represent the optmal estmate gven nformaton that has already arrved. To summarze, both tradtonal relaxaton updates as n equaton 2 and BBP updates as n equatons 7{9 gve smple rules for updatng a unt's actvty based on local data and actvtes of neghborng unts. However, the fact that BBP updates are based on the probablty calculus guarantees that a unt's actvty wll be optmal gven nformaton that has already arrved and gves rse to a qualtatve derence between the convergence of these two types of schemes. In the next secton, we wll demonstrate ths derence n mage nterpretaton problems.

5 a. b. Fgure 2: a. the rst frame of a sequence. The hand s translated to the left. b. contour extracted usng standard methods 2 Results Fgure 2a shows the rst frame of a sequence n whch the hand s translated to the left. Fgure 2b shows the boundng contour of the hand extracted usng standard technques. 2. Moton propagaton along contours Local measurements along the contour are nsucent to determne the moton. Hldreth [2] suggested to overcome the local ambguty by mnmzng the followng cost functonal: J(V )= (dx t v + dt ) 2 + v + v 2 (3) where dx; dt denote the spatal and temporal mage dervatves and v denotes the velocty at pont along the contour. Ths functonal s analogous to the nterpolaton functonal (eq. ) and the dervaton of the relaxaton and BBP updates are also analogous. Fgure 3a shows the estmate of moton based solely on local nformaton. The estmates are wrong due to the aperture problem. Fgure 3b shows the performance of three propagaton schemes: gradent descent, SOR and BBP. Gradent descent converges so slowly that the mprovement n ts estmate can not be dscerned n the plot. SOR converges much faster than gradent descent but stll has sgncant error after 500 teratons. BBP gets the correct estmate after 3 teratons! (Here and n all subsequent plots an teraton refers to one update of all unts n the networ). Ths s due to the fact that after 3 teratons, the estmate at locaton s the optmal one gven data n the nterval [ 3;+ 3]. In ths case, there s enough data n every such nterval along the contour to correctly estmate the moton. Fgure 3c shows the estmate produced by SOR after 500 teratons. Even wth smple vsual nspecton t s evdent that the estmate s qute wrong. Fgure 3d shows the (correct) estmate produced by BBP after 3 teratons. 2.2 Drecton of gure propagaton The extracted contour n gure 2 bounds a dar and a lght regon. Drecton of gure (DOF) (e.g. [9]) refers to whch of these two regons s gure and whch s For certan specal cases, nowng P (L (m) = ) s not sucent for choosng the sequence of labels that mnmzes J. In those cases one should do belef revson rather than propagaton [6]

6 error BBP SOR Gradent a. b teratons c. d. Fgure 3: a. Local estmate of velocty along the contour. b. Performance of SOR, gradent descent and BBP as a functon of tme. BBP converges orders of magntude faster than SOR. c. Moton estmate of SOR after 500 teratons. d. Moton estmate of BBP after 3 teratons. ground. A local cue for DOF s convexty - gven three neghborng ponts along the contour we prefer the DOF that maes the angle dened by those ponts acute rather than obtuse. Fgure 4a shows the results of usng ths local cue on the hand contour. The local cue s not sucent. We can overcome the local ambguty by mnmzng a cost functonal that taes nto account the DOF at neghborng ponts n addton to the local convexty. Denote by L (m) the DOF at pont along the contour and dene J(L) = m V (m)l (m) wth V (m) determned by the acuteness of the angle at locaton. m L (m)l + (m) (4) Fgure 4b shows the performance of four propagaton algorthms on ths tas: three tradtonal relaxaton labelng algorthms (MF Hopeld, Rosenfeld et al, constraned gradent descent) and BBP. All three tradtonal algorthms converge to a local mnmum, whle the BBP converges to the global mnmum. Fgure 4c shows the local mnmum reached by the Hopeld networ and gure 4d shows the correct soluton reached by the BBP algorthm. Recall (secton.) that BBP s guaranteed to converge to the correct posteror gven all the data. 2.3 Extensons to 2D In the prevous two examples ambguty was reduced by combnng nformaton from other ponts on the same contour. There exst, however, cases when nformaton should be propagated to all ponts n the mage. Unfortunately, such propagaton problems correspond to Marov Random Feld (MRF) generatve models, for whch

7 error BBP MFA Relax label Relax Gradent a. b teratons c. d. Fgure 4: a. Local estmate of DOF along the contour. b. Performance of Hop- eld,gradent descent, relaxaton labelng and BBP as a functon of tme. BBP s the only method that converges to the global mnmum. c. DOF estmate of Hopeld net after convergence. d. DOF estmate of BBP after convergence. calculaton of the posteror cannot be done ecently. However, Wllsy and hs colleagues [4]have recently shown that MRFs can be approxmated wth herarchcal or mult-resoluton models. In current wor, we have been usng the mult-resoluton generatve model to derve local BBP rules. In ths case, the Bayesan belefs are propagated between neghborng unts n a pyramdal representaton of the mage. Although ths wor s stll n prelmnary stages, we nd encouragng results n comparson wth tradtonal 2D relaxaton schemes. 3 Dscusson The update rules n equatons 5{6 der slghtly from those derved by Pearl [6] n that the quanttes ; are condtonal probabltes and hence are constantly normalzed to sum to unty. Usng Pearl's orgnal algorthm for sequences as long as the ones we are consderng wll lead to messages that become vanshngly small. Lewse our update rules der slghtly from the forward-bacward algorthm for HMMs [7] n that ours are based on the assumpton that all states are equally lely aprore and hence the updates are symmetrc n and. Fnally, equaton 9 can be seen as a varant of a Rccat equaton []. In addton to these mnor notatonal derences, the context n whch we use the update rules s derent. Whle n HMMs and Kalman Flters, the updates are seen as nterm calculatons toward calculatng the posteror, we use these updates n a parallel networ of local unts and are nterested n how the estmates of unts n

8 ths networ mprove as a functon of teraton. We have shown that an archtecture that propagates Bayesan belefs accordng to the probablty calculus yelds orders of magntude mprovements n convergence over tradtonal schemes that do not propagate probabltes. Thus mage nterpretaton provdes an mportant example of a tas where t pays to be a Bayesan. Acnowledgments I than E. Adelson, P. Dayan, J. Tenenbaum and G. Galpern for comments on versons of ths manuscrpt; M.I. Jordan for stmulatng dscussons and for ntroducng me to Bayesan nets. Supported by a tranng grant from NIGMS. References [] Arthur Gelb, edtor. Appled Optmal Estmaton. MIT Press, 974. [2] E. C. Hldreth. The Measurement of Vsual Moton. MIT Press, 983. [3] S.Z. L. Marov Random Feld Modelng n Computer Vson. Sprnger-Verlag, 995. [4] Mar R. Luettgen, W. Clem Karl, and Allan S. Wllsy. Ecent multscale regularzaton wth applcaton to the computaton of optcal ow. IEEE Transactons on mage processng, 3():4{64, 994. [5] D. Marr. Vson. H. Freeman and Co., 982. [6] Judea Pearl. Probablstc Reasonng n Intellgent Systems: Networs of Plausble Inference. Morgan Kaufmann, 988. [7] Lawrence Rabner and Bng-Hwang Juang. Fundamentals of Speech recognton. PTR Prentce Hall, 993. [8] A. Rosenfeld, R. Hummel, and S. Zucer. Scene labelng by relaxaton operatons. IEEE Transactons on Systems, Man and Cybernetcs, 6:420{433, 976. [9] P. Sajda and L. H. Fnel. Intermedate-level vsual representatons and the constructon of surface percepton. Journal of Cogntve Neuroscence, 994. [0] Glbert Strang. Introducton to Appled Mathematcs. Wellesley-Cambrdge, 986.