Message-Passing Algorithms for Quadratic Programming Formulations of MAP Estimation

Transcription

1 Message-Passng Algorthms for Quadratc Programmng Formulatons of MAP Estmaton Akshat Kumar Department of Computer Scence Unversty of Massachusetts Amherst Shlomo Zlbersten Department of Computer Scence Unversty of Massachusetts Amherst Abstract Computng maxmum a posteror (MAP) estmaton n graphcal models s an mportant nference problem wth many applcatons. We present message-passng algorthms for quadratc programmng (QP) formulatons of MAP estmaton for parwse Markov random felds. In partcular, we use the concaveconvex procedure (CCCP) to obtan a locally optmal algorthm for the non-convex QP formulaton. A smlar technque s used to derve a globally convergent algorthm for the convex QP relaxaton of MAP. We also show that a recently developed expectatonmaxmzaton (EM) algorthm for the QP formulaton of MAP can be derved from the CCCP perspectve. Experments on synthetc and real-world problems confrm that our new approach s compettve wth maxproduct and ts varatons. Compared wth CPLEX, we acheve more than an order-ofmagntude speedup n solvng optmally the convex QP relaxaton. 1 INTRODUCTION Probablstc graphcal models provde an effectve framework for compactly representng probablty dstrbutons over hgh dmensonal spaces and performng complex nference usng smple local update procedures. In ths work, we focus on the class of undrected models called Markov random felds (MRFs) [Wanwrght and Jordan, 008]. A common nference problem n ths model s to compute the most probable assgnment to varables, also called the maxmum a posteror (MAP) assgnment. MAP estmaton s crucal for many practcal applcatons n computer vson and bonformatcs such as proten desgn [Yanover et al., 006; Sontag et al., 008] among others. Computng MAP exactly s NP-hard for general graphs. Thus approxmate nference technques are often used [Wanwrght and Jordan, 008; Sontag et al., 010]. Recently, several convergent algorthms have been developed for MAP estmaton such as tree-reweghted max-product [Wanwrght et al., 00; Kolmogorov, 006] and max-product LP [Globerson and Jaakkola, 007; Sontag et al., 008]. Many of these algorthms are based on the lnear programmng (LP) relaxaton of the MAP problem [Wanwrght and Jordan, 008]. A dfferent formulaton of MAP s based on quadratc programmng (QP) [Ravkumar and Lafferty, 006; Kumar et al., 009]. The QP formulaton s an attractve alternatve because t provdes a more compact representaton of MAP: In a MRF wth n varables, k values per varable, and E edges, the QP has O(nk) varables whereas the LP has O( E k ) varables. The large sze of the LP makes off-the-shelf LP solvers mpractcal for several real-world problems [Yanover et al., 006]. Another sgnfcant advantage of the QP formulaton s that t s exact. However, the QP formulaton s non-convex, makng global optmzaton hard. To remedy ths, Ravkumar and Lafferty [006] developed a convex QP relaxaton of the MAP problem. Our man contrbuton s the analyss of the QP formulatons of MAP as a dfference of convex functons (D.C.) problem, whch yelds effcent, graph-based message-passng algorthms for both the non-convex and convex QP formulatons. We use the concaveconvex procedure (CCCP) to develop the message passng algorthms [Yulle and Rangarajan, 003]. Motvated by geometrc programmng [Boyd et al., 007], we present another QP-based formulaton of MAP and solve t usng the CCCP technque. The resultng algorthm s shown to be equvalent to a recently developed expectaton-maxmzaton (EM) algorthm that provdes good performance for large MAP problems [Kumar and Zlbersten, 010]. The CCCP approach, however, s more flexble than EM and makes t easy to ncorporate addtonal constrants that can

2 tghten the convex QP [Kumar et al., 009]. All the developed CCCP algorthms are guaranteed to converge to a local optmum for non-convex QPs, and to the global optmum for convex QPs. All the algorthms also provde monotonc mprovement n the objectve. We experment on synthetc benchmarks and realworld proten-desgn problems [Yanover et al., 006]. Aganst max-product [Pearl, 1988], CCCP provdes sgnfcantly better soluton qualty, sometmes more than 45% for large Isng graphs. On the real-world proten desgn problems, CCCP acheves near-optmal soluton qualty for most nstances, and s sgnfcantly faster than the max-product LP method [Sontag et al., 008]. Ravkumar and Lafferty [006] proposed to solve the convex QP relaxaton usng standard QP solvers. Our message-passng algorthm for ths case provdes more than an order-of-magntude speedup aganst the state-of-the-art QP solver CPLEX. QP FORMULATION OF MAP A parwse Markov random feld (MRF) s descrbed usng an undrected graph G = (V, E). A dscrete random varable x wth a fnte doman s assocated wth each node V of the graph. Assocated wth each edge (, j) E s a potental functon θ j (x, ). The complete assgnment x has the probablty: ( ) p(x; θ) exp θ j (x, ) j E The MAP problem conssts of fndng the most probable assgnment to all the varables under p(x; θ). Ths s equvalent to fndng the assgnment x that maxmzes the functon f(x; θ) = j E θ j(x, ). We assume w.l.o.g. that each θ j s nonnegatve, otherwse a constant can be added to each θ j wthout changng the optmal soluton. Let p be the margnal probablty assocated wth each MRF node V. The MAP quadratc programmng (QP) formulaton [Ravkumar and Lafferty, 006] s gven by: max p j E subject to x x, p (x )p j ( )θ j (x, ) (1) p (x ) = 1, p (x ) 0 V The above QP s compact even for large graphcal models and has smple lnear constrants: O(nk) varables and n normalzaton constrants where n = V and k s the doman sze. Ravkumar and Lafferty [006] also show that ths formulaton s exact. That s, the global optmum of the above QP wll maxmze the functon f(x; θ) and an ntegral MAP assgnment can be extracted from t. However ths formulaton s non-convex, makng global optmzaton hard. Nonetheless, for several problems, a local optmum of ths QP provdes a good soluton as we wll show emprcally. Ths was also observed by Kumar and Zlbersten [010]..1 The Concave Convex Procedure The concave-convex procedure (CCCP) [Yulle and Rangarajan, 003] s a popular approach to optmze a general non-convex functon expressed as a dfference of two convex functons. We use ths method to obtan message-passng algorthms for QP formulatons of MAP. We descrbe t here brefly. Consder the optmzaton problem: mn{g(x) : x Ω} () where g(x) = u(x) v(x) s an arbtrary functon wth u, v beng real-valued convex functons and Ω beng a convex set. The CCCP method provdes an teratve procedure that generates a sequence of ponts x l by solvng the followng convex program: x l+1 = arg mn{u(x) x T v(x l ) : x Ω} (3) Each teraton of CCCP decreases the objectve g(x) and s guaranteed to converge to a local optmum [Srperumbudur and Lanckret, 009].. Solvng MAP QP Usng CCCP We frst show how the CCCP framework can be used to solve the QP n Eq. (1). We adopt the conventon that a MAP QP always refers to the QP n Eq. (1); the convex varant of ths QP shall be explctly dfferentated when addressed later. Consder the followng functons u, v: u(p) = θ j(x, )( p (x ) + p j( ) ) j x (4) v(p) = θ j(x, )( p (x ) + p j ( ) ) j x (5) The above functons are convex because the quadratc functons f(z) = z and f(y, z) = (y + z) are convex, and the nonnegatve weghted sum of convex functons s also convex [Boyd and Vandenberghe, 004, Ch. 3]. It can be easly verfed that the QP n Eq. (1) can be wrtten as mn p {u(p) v(p)} wth normalzaton and nonnegatvty constrants defnng the constrant set Ω. Intutvely, we used the smple dentty xy = (x + y ) (x + y). We also negated the objectve functon to convert maxmzaton to mnmzaton. For smplcty, we denote the gradent v/ p(x ) by x v. x v =p (x ) θ j (x, )+ θ j (x, )p j ( )

3 The frst part of the above equaton nvolves a local computaton assocated wth an MRF node and the second part defnes the messages δ j from neghbors j Ne() of node. It can be made explct as follows: ˆθ(x )= θ j (x, ); δ j (x )= θ j (x, )p j ( ) x v= p (x )ˆθ(x ) + δ j (x ) (6) CCCP teratons: Each teraton of CCCP nvolves solvng the convex program of Eq. (3). Frst we wrte the Lagrangan functon nvolvng only the normalzaton constrants, later we address the nonnegatvty nequalty constrants. l v denotes the gradent from the prevous teraton l. L(p, λ) = θ j(x, ){ p (x ) + p j( ) } j x p (x ) l x v + λ ( p (x ) 1) (7) x x Solvng for the frst order optmalty condtons p L(x, λ ) and λ L(x, λ ), we get the soluton: p l+1 (x ) = l x v λ ˆθ(x ) λ = 1 x 1 ˆθ(x ) ( x ) l x v ˆθ(x ) 1 (8) (9) Nonnegatvty constrants: Nonnegatvty constrants n the MAP QP are nequalty constrants whch are harder to handle as the Karush-Kuhn- Tucker (KKT) condtons nclude the nonlnear complementary slackness condton µ j p j () = 0 [Boyd and Vandenberghe, 004]. We can use nteror-pont methods, but they lose the effcency of graph based message passng. Fortunately, we show that for MAP QP, the KKT condtons are easly satsfed by ncorporatng an nner-loop n the CCCP teratons. Alg. 1 shows the complete message-passng procedure to solve MAP QPs. Each outer loop corresponds to solvng the CCCP teraton of Eq. (3) and s run untl the desred number of teratons s reached. The messages δ are used for computng the gradent v as n Eq. (6). The nner loop corresponds to satsfyng the KKT condtons ncludng the nonnegatvty nequalty constrants. Intutvely, the strategy to handle nequalty constrants s as hghlghted n [Bertsekas, 1999, Sec ] consderng all possble combnatons of nequalty constrants beng actve (p (x )=0) or nactve (p (x ) > 0) and solvng the resultng KKT condtons, whch s easer as they become lnear equatons. If the resultng soluton satsfes the KKT condtons of the orgnal problem, then we have a vald soluton for the orgnal optmzaton problem. Of course, 1: Graph-based message passng for MAP estmaton nput: Graph G = (V, E) and potentals θ j per edge //outer loop starts repeat foreach node V do δ j() P x p (x )θ j(x, ) Send message δ j to each neghbor j Ne() foreach node V do zeros φ //nner loop starts repeat Set p (x ) 0 x zeros Calculate p (x ) usng Eq. (8) x / zeros zeros zeros {x : p (x ) < 0} untl all belefs p (x ) 0 untl stoppng crteron s satsfed return: The decoded complete ntegral assgnment ths s hghly neffcent for the general case. But fortunately for the MAP QP, we show that the nner loop of Alg. 1 recovers the correct soluton and the Lagrange multplers are computed effcently for the convex program of Eq. (3). We descrbe t below. The nner loop ncludes local computaton to each MRF node and does not requre message passng. Intutvely, the set zeros tracks all the settngs x of the varable x for whch p (x ) was negatve n any prevous nner loop teraton. It then clamps all such belefs to 0 for all future teratons. Then the belefs for the rest of the settngs of x are computed usng Eq. (8). The new Lagrange multpler λ (whch corresponds to the condton λ L(x, λ ) = 0) s calculated usng the equaton x \x p (x ) = 1. Lemma 1. The nner loop of Alg. 1 termnates wth worst case complexty O(k ), and yelds a feasble pont for the convex program of Eq. (3). Proof. The sze of the set zeros ncreases wth each teraton, therefore the nner loop must termnate as each varable s doman s fnte. Wth the doman sze of a varable beng k, the nner loop can run for at most k teratons. Computng new belefs wthn each nner loop teraton also requres O(k) tme. Thus the worst case total complexty s O(k ). The nner loop can termnate only n two ways before teraton k or at teraton k. If t termnates before teraton k, then t mples that all the belefs p (x ) must be nonnegatve. The normalzaton constrants are always enforced by the Lagrange multplers λ s. If t termnates durng teraton k, then t mples that k 1 settngs of the varable x are clamped to zero as the sze of the set zeros wll be exactly k 1. The sze cannot be k because that would make all the belefs equal to zero, makng t mpossble to satsfy the normalzaton constrant; λ wll not allow ths. The sze

4 cannot be smaller than k 1 because the set zeros grows by at least one element durng each prevous teraton. Therefore the only soluton durng teraton k s to set the sngle remanng settng of the varable x to 1 to satsfy the normalzaton and nonnegatvty constrants smultaneously. Therefore the nner loop always yelds a feasble pont upon termnaton. Emprcally, we observed that even for large proten desgn problems wth k = 150, the number of requred nner loop teratons s below 0 far below the worst case complexty. For a fxed outer loop l, let the nner loop teratons be ndexed by r. Lemma. The Lagrange multpler correspondng to the normalzaton constrant for a MRF varable x always ncreases wth each nner loop teraton. Proof. Each nner loop teraton r computes a new Lagrange multpler λ r for the constrant p (x ) = 1 usng Eq. (8). We show that λ r+1 > λ r. For the nner loop teraton r, some of the computed belefs must be negatve, otherwse the nner loop must have termnated. Let x denote those settngs of varable x for whch p (x ) < 0 n teraton r. From the normalzaton constrant for teraton r, we get: x l x v λ r ˆθ(x ) = 1 (10) We used the explct representaton of p (x ) from Eq. (8). Snce p (x ) are negatve, we get: x \x l x v λ r ˆθ(x ) > 1 (11) The belef for all such x wll become zero for the next nner loop teraton r + 1. From the normalzaton constrant for teraton r + 1, we get: x \x l x v λ r+1 ˆθ(x ) = 1 (1) We used a slght smplfcaton n the above equatons as we gnored the effect of prevous teratons, before teraton r. However, t wll not change the concluson as all the belefs that were clamped to zero earler (before teraton r) shall reman so for all future teratons. Note that x and ˆθ do not depend on the nner loop teratons. Subtractng Eq. 1 from Eq. 11: (λ r+1 λ r ) x \x 1 ˆθ(x ) > 0 (13) Snce we assumed that all potental functons θ j are nonnegatve, we must have (λ r+1 λ r ) > 0. Hence > λ r and the lemma s proved. λ r+1 Theorem 3. The nner loop of Alg. 1 correctly recovers all the Lagrange multplers for the equalty and nequalty constrants for the convex program of Eq. (3), thus solvng t exactly. Proof. Lemma 1 shows that the nner loop provdes a feasble pont of the convex program. We now show that ths pont also satsfes the KKT condtons, thus s the optmal soluton. The KKT condtons for the normalzaton constrants are always satsfed durng the belef updates (see. Eq. (8)). The man task s to show that for the nequalty constrant p (x ) 0, the KKT condtons hold. That s, f p (x ) = 0, then the Lagrange multpler µ (x ) 0, and f p (x ) < 0, then µ (x ) = 0. By usng the KKT condton p(x )L(p, λ, µ ) = 0, we get: θ j (x, )p (x ) l x v + λ µ (x ) = 0 (14) The man focus of the proof s on the belefs for elements n the set zeros. Let us focus on the end of an nner loop teraton r, when a new element x s added to zeros because ts computed belef p (x ) < 0. Usng Eq. (8), we know that p (x ) = l x p (x ) < 0 we get: v λ r ˆθ(x ). Because λ r > l x v (15) For all future teratons of the nner loop, p (x ) wll be set to zero. Therefore the KKT condton for teraton r + 1 mandates that µ r+1 (x ) 0. Settng p (x ) = 0 n Eq. (14), we get: µ r+1 (x ) = λ r+1 l x v (16) We know from Lemma that λ r+1 > λ r. Combnng ths fact wth Eq. (15), we get µ r+1 (x ) > 0, thereby satsfyng the KKT condton. Note that the only component dependng on the nner loop n the above condton s λ r ; x s fxed durng each nner loop. Furthermore, for all future nner loop teratons, the KKT condtons for all elements x s n the set zeros wll be met due to the ncreasng nature of the multpler λ. Therefore, when the nner loop termnates, we shall have correct Lagrange multplers µ satsfyng µ 0 for all the elements of the set zeros. For the rest of the elements, the multpler µ = 0, satsfyng all the KKT condtons. As the frst order KKT condtons are both necessary and suffcent for optmalty n convex programs [Bertsekas, 1999, Sec ], the nner loop solves exactly the convex program n Eq. (3).

5 .3 Solvng Convex MAP QP Usng CCCP Because the prevous QP formulaton of MAP s nonconvex, global optmzaton s hard. To remedy ths, Ravkumar and Lafferty [006] developed a convex QP relaxaton for MAP, whch performed well on ther benchmarks. Recently, Kumar et al. [009] showed that the convex QP relaxaton s also equvalent to the second order cone programmng (SOCP) relaxaton. Ravkumar and Lafferty [006] proposed to solve such QP usng standard QP solvers. We show usng CCCP that ths QP relaxaton can be solved effcently usng graph-based message passng, and the resultng algorthm converges to the global optmum of the relaxed QP. Expermentally, we found the resultng message-passng algorthm to be hghly effcent even for large graphs, outperformng CPLEX by more than an order-of-magntude. The relaxed QP s descrbed as follows: max p (x )d (x )+ p (x )p j ( )θ j (x, ) p x j x, p (x )d (x ) (17) x The relaxaton s based on addng a dagonal term, d (x ), for each varable x. Note that under the ntegralty assumpton p (x ) = p (x ), thus the frst and last terms cancel out, resultng n the orgnal MAP QP. The dagonal term s gven by: d (x ) = θ j(x, ) Consder the convex functon u(p) represented as: θ j(x, )( p (x ) + p j( ) ) + p (x )d (x ) x,x j and the convex functon v(p) represented as: θ j(x, )( p (x )+p j ( ) ) + p (x )d (x ) j x,x The above two functons are the same as the orgnal QP formulaton, except for the added dagonal terms d (x ). It can be easly verfed that the relaxed QP objectve can be wrtten as mn p {u(p) v(p)} subject to normalzaton and nonnegatvty constrants. Note that the maxmzaton of the relaxed QP s converted to mnmzaton by negatng the objectve. The gradent requred by CCCP s gven by: x v = p (x )ˆθ(x ) + θ j (x, )p j ( ) + d (x ) Notce the close smlarty wth the MAP QP case n Eq. (6). The only addtonal term s d (x ), whch needs to be computed only once before message passng begns. The messages for the relaxed QP case are exactly the same as the δ messages for the MAP QP. The Lagrangan correspondng to the convex program of Eq. (3) s smlar to the MAP QP case (see Eq. (7)) wth an addtonal term,x p (x )d (x ). The constrant set Ω ncludes the normalzaton and nonnegatvty constrants as for the MAP QP case. Solvng for the optmalty condtons p L(p, λ ) and λ L(p, λ ), we get the new belefs as follows: p l+1 (x ) = l x v λ d (x ) + ˆθ(x ) (18) The Lagrange multpler λ for the normalzaton constrant can be calculated by usng the equaton x p (x ) = 1. The only dfference from the correspondng Eq. (8) for the MAP QP s the addtonal term d (x ) n the denomnator. Thanks to these strong smlartes, we can show that Alg. 1 also works for the convex MAP QP wth mnor modfcatons. Frst, we calculate the dagonal terms d (x ) once for each varable x of the MRF. The message-passng procedure for each outer loop teraton remans the same. The second dfference les n the nner loop that enforces the nonnegatvty constrants. The nner loop now uses Eq. (18) nstead of Eq. (8) to estmate new belefs p (x ). The proof s omtted beng very smlar to the MAP QP case. Theorem 4. The CCCP message-passng algorthm converges to a global optmum of the convex MAP QP. The result s based on the fact that CCCP converges to a statonary pont of the gven constraned optmzaton problem that satsfes the KKT condtons [Srperumbudur and Lanckret, 009]. Because the KKT condtons are both necessary and suffcent for convex optmzaton problems wth lnear constrants [Bertsekas, 1999], the result follows. We also hghlght that a global optmum of the convex QP may not solve the MAP problem exactly, as the convex QP s a varatonal approxmaton to MAP that may not be tght. Nonetheless, t has shown to perform well n practce [Ravkumar and Lafferty, 006]..4 GP Based Reformulaton of MAP QP We now present another formulaton of the MAP QP problem based on geometrc programmng (GP). A GP s a type of mathematcal program characterzed by objectves and constrants that have a specal form. For detals, we refer to [Boyd et al., 007]. Whle the QP formulaton of MAP n Eq. (1) s not exactly a GP, t bears a close resemblance. Ths allows us to transfer some deas from GP, whch we shall descrbe. We start wth some basc concepts of GP.

6 Defnton 1. Let x 1,..., x n denote real postve varables. A real valued monomal functon has the form f(x) = cx a1 1 xa xan n, where c > 0 and a R. A posynomal functon s a sum of one or more monomals: f(x) = K k=1 c kx a 1k 1 x a k x a nk n In a GP, the objectve functon and all nequalty constrants are posynomal functons. The MAP QP (see Eq. (1)) satsfes ths requrement the potental functon θ j corresponds to c k and s postve (the θ j = 0 case can be excluded for convenence); node margnals p (x ) are real postve varables. Snce we already assumed margnals p to be postve, nonnegatvty nequalty constrants are not requred. In a GP, the equalty constrants can only be monomal functons. Ths s not satsfed n MAP QP as normalzaton constrants are posynomals. Nonetheless, we proceed as n GP by convertng the orgnal problem usng certan optmalty-preservng transformatons. The frst change s to let p (x ) = e y(x), where y (x ) s an unconstraned varable. Ths s allowed as all margnals must be postve. The second change s to take the log of the objectve functon; because log s a monotonc functon, ths wll not change the optmal soluton. The reformulated MAP QP s shown below: { } mn: log j subject to: x x, exp ( y (x )+y j ( )+log θ j (x, ) ) e y(x) = 1 V Ths nonlnear program has the same optmal solutons as the orgnal MAP QP. As log-sum-exp s convex, the objectve functon of the above problem s concave. Note that we are mnmzng a concave functon that can have multple local optma. We agan solve t usng CCCP. Consder the functon u(y) = 0 and v(y) as the objectve of the above program, but wthout the negatve sgn. The gradent requred by CCCP s: θ j (x, ) exp ( y (x )+y j ( ) ) l y v = j x, θ j (x, ) exp ( y (x )+y j ( ) ) (19) The Lagrangan correspondng to Eq. (3) wth constrant set ncludng only normalzaton constrants s: L(y, λ) =,x y (x ) l y v + λ ( x e y(x) 1) Usng the frst order optmalty condton, we get: exp ( y (x ) ) = l y v λ (0) We note that the denomnator of Eq. (19) s a constant for each y (x ). Therefore we represent t usng c l. Resubsttutng p (x ) = e y(x) and l y v n Eq. (0): p x (x ) = j θ j (x, )p (x )p j ( ) c l λ where p (x ) s the new parameter for the current teraton, and parameters wthout astersk (on the R.H.S.) are from the prevous teraton. Snce c l λ s also a constant, we can replace them by a normalzaton constant C to get the fnal update equaton: p (x ) = p (x ) θ j (x, )p j ( ) C The message-passng process for ths verson of CCCP s exactly the same as that for MAP QP and convex QP. Ths verson does not requre an nner loop as all the node margnals reman postve usng such updates. Ths update process s also dentcal to the recently developed message-passng algorthm for MAP estmaton that s based on expectaton-maxmzaton (EM) rather than CCCP [Kumar and Zlbersten, 010]. However, CCCP provdes a more flexble framework n that t handled the nonconvex and convex QP n a smlar way as shown earler. Furthermore, the CCCP framework allows for addtonal constrants to be added to the convex QP to make t tghter [Srperumbudur and Lanckret, 009]. In sum, we have shown that the concave-convex procedure provdes a unfyng framework for the varous quadratc programmng formulatons of the MAP problem. Each teraton of CCCP can be easly mplemented usng graph-based message passng. Interestngly, the messages exchanged for all the QP formulatons we dscussed reman exactly the same; the dfferences le n how new node margnals are computed usng such messages. 3 EXPERIMENTS We now present an emprcal evaluaton of the CCCP algorthms. We frst report results on synthetc graphs generated usng the Isng model from statstcal physcs [Baxter, 198]. We compare max-product (MP) and the CCCP message-passng algorthm for the QP formulaton of MAP. We generated D nearest neghbor grd graphs for a number of grd szes (rangng between to 50 50) and varyng values of the couplng parameter. All the varables were bnary. The node potentals were generated by samplng from the unform dstrbuton U[ 0.05, 0.05]. The couplng strength, d coup, for each edge was sampled from U[ β, β] followng the mxed Isng model. The bnary edge potental θ j was defned as follows: { d coup x = θ j (x, ) = d coup x

7 CCCP CCCP CCCP MP MP MP CCCP CCCP CCCP MP MP MP (a) 1 Isng 3! (a) Isng 4 3(a)! 100 Isng 53 4! (b) 1 Isng 3! (b) Isng 4 3(b)! 400 Isng 53 4! (c) 1 Isng 3! (c) Isng 4 3! (c) 900 Isng 53 4! (a) (b) (c) (a) Isng (a)! 100 Isng (a)! 100 Isng! 100 (b) Isng (b)! 400 Isng (b)! 400 Isng! 400 (c) Isng!(c) 900 Isng (c)! 900 Isng! CCCP CCCP CCCP CCCP CCCP CCCP (d) 1 Isng!(d) Isng 4 (d)! Isng 53 4! (e) 1 Isng!(e) Isng 4 (e)! Isng 53 4! (f) Proten 0 5 0(f) desgn 40 Proten (f) nstances desgn 40 Proten nstances 40 desgn nstances (d) Isng (d)! 1600 Isng (d)! 1600 Isng! 1600 (e) Isng (e)! 500 Isng (e)! 500 Isng! 500 (f) Proten (f) desgn Proten (f) nstances Proten desgn nstances desgn nstances (d) (e) Fgure 1: (a) (e) show qualty comparson between max-product (MP) and CCCP for Isng graphs wth varyng number of nodes ( ). The x-axs denotes the couplng parameter β, y-axs shows soluton qualty. (f) shows the soluton qualty CCCP acheves as a percentage of the optmal value (y-axs) for dfferent proten desgn nstances (x-axs). (f) For every grd sze and each settng of the couplng strength parameter β, we generated 10 nstances by samplng d coup per edge. For each nstance, we consdered the best soluton qualty of 10 runs for both max-product and CCCP. We then report the average qualty of the 10 nstances acheved for each parameter β. Both max-product and CCCP were mplemented n JAVA and ran on a.4ghz CPU. Max-product was allowed 1000 teratons and often dd not converge, whereas CCCP converged wthn 500 teratons. Fg. 1(a e) show soluton qualty comparsons between MP and CCCP. For graphs (Fg. 1(a)), both CCCP and MP acheve smlar soluton qualty. The gan n qualty provded by CCCP ncreases wth the sze of the grd graph. For 0 0 grds, the average gan n soluton qualty, ((Q CCCP Q MP )/Q MP ), for each couplng strength parameter β s over 0%. For (Fg. 1(c)) grds, the gan s above 30% for each parameter β; for grds t s 36% and for grds t s 43%. Overall, CCCP provdes much better performance than max-product over these Isng graphs. And unlke max-product, CCCP monotoncally ncreases soluton qualty and s guaranteed to converge. A detaled performance evaluaton of the convex QP s provded [Ravkumar and Lafferty, 006]. As such Isng graphs have relatvely small QP representaton, the CCCP message passng method and CPLEX had smlar runtme for the convex QP. We also expermented on the proten desgn benchmark (total of 97 nstances) [Yanover et al., 006]. In these problems, the task s to fnd a sequence of amno acds that s as stable as possble for a gven backbone structure of proten. Ths problem can be modeled usng a parwse Markov random feld. These problems are partcularly hard and dense wth up to 170 varables, each wth a large doman sze of up to 150 values. Fg. 1(f) shows the % of the optmal value CCCP acheves aganst the best upper bound provded by the LP based approach MPLP [Sontag et al., 008]. MPLP has been shown to be very effectve n solvng exactly the MAP problem for several real-world problems. However for these proten desgn problems, due to the large varable domans, ts reported mean runnng tme s 9.7 hours [Sontag et al., 008]. As Fg. 1(f) shows, CCCP acheves nearoptmal solutons, on average wthn 97.7% of the optmal value. A sgnfcant advantage of CCCP s ts speed: t converges wthn 100 teratons for all these problems and requres 403 seconds for the largest nstance, much faster than MPLP. The mean runnng tme of CCCP was 170 seconds for ths dataset. Thus CCCP can prove to be qute effectve when fast, approxmate solutons are desred. The man reason for ths speedup s that CCCP s messages are easer to compute than MPLP s as also hghlghted n [Kumar and Zlbersten, 010]. Compared to the EM approach of [Kumar and Zlbersten, 010], CCCP provdes better soluton qualty: EM acheved 95% of the optmal value on average, whle CCCP acheves 97.7%. The overhead of the nner loop n CCCP s small aganst EM whch takes 35 seconds for the largest nstance, whle CCCP takes 403 seconds. We also tested CCCP on the proten predcton dataset [Yanover et al., 006]. The problems n ths dataset are much smaller than those n the proten desgn dataset, and both max-product and MPLP acheve good soluton qualty. CCCP s performance was worse n ths case, partly due to the local optma present n the nonconvex QP formulaton of MAP. The convex QP formulaton was not tght n ths case. Fg. (a) shows runtme comparson of CCCP aganst CPLEX for the convex QP for the 5 largest proten

8 15% 10% relaxaton, CCCP provded more than an order-ofmagntude speedup over the state-of-the-art QP solver CPLEX. These results offer a powerful new way for solvng effcently large MAP estmaton problems. 5% 0% (a) Tme comparson (b) Qualty comparson Fgure : (a) Tme comparson of CCCP for convex QP aganst CPLEX for the largest 5 proten desgn nstances (x-axs). The y-axs denotes T CCCP /T CP LEX as a percentage. (b) denotes the sgned qualty dfference Q CCCP Q CP LEX, a hgher value s better. desgn problems w.r.t. the number of graph edges. After tryng dfferent QP solver optons avalable n CPLEX, we chose the barrer method whch provded the best performance. As CPLEX was qute slow, we let CPLEX use 8 CPU cores wth 8GB RAM, whle CCCP only used a sngle CPU. As ths fgure shows, CCCP s more than an order-of-magntude faster than CPLEX even when t uses a sngle core. The longest CPLEX took was 3504 seconds, whereas CCCP only took 99 seconds for the same nstance. The mean runnng tme of CPLEX was 1914 seconds; for CCCP, t was 96 seconds. Surprsngly, CCCP converges n only 15 teratons to the optmal soluton for all 5 problems. Fg. (b) shows the sgned qualty dfference between CCCP and CPLEX for the convex QP objectve. CPLEX provdes the optmal soluton wthn some non-zero ɛ (we used the default settng). Ths fgure shows that even wthn 15 teratons, CCCP acheved a slghtly better soluton. The decoded soluton qualty provded by the convex QP was decent, wthn 80% of the optmal value, but not as hgh as the CCCP method for the nonconvex QP. 4 CONCLUSION We presented new message-passng algorthms for varous quadratc programmng formulatons of the MAP problem. We showed that the concave-convex procedure provdes a unfyng framework for dfferent QP formulatons of the MAP problem represented as a dfference of convex functons. The resultng algorthms were shown to be convergent to a local optmum for the nonconvex QP and to the global optmum of the convex QP. Emprcally, the CCCP algorthm was shown to work well on Isng graphs and real-world proten desgn problems. The CCCP approach provded much better soluton qualty than max-product for Isng graphs and converged sgnfcantly faster than max-product LP for proten desgn problems, whle provdng near optmal solutons. For the convex QP Acknowledgments Support for ths work was provded n part by the NSF Grant IIS and by the AFOSR Grant FA References Baxter, R. (198). Exactly Solved Models n Statstcal Mechancs. Academc Press, London. Bertsekas, D. P. (1999). Nonlnear Programmng. Athena Scentfc, nd edton. Boyd, S., Km, S.-J., Vandenberghe, L., and Hassb, A. (007). A tutoral on geometrc programmng. Optmzaton and Engneerng, 8: Boyd, S. and Vandenberghe, L. (004). Convex Optmzaton. Cambrdge Unversty Press, New York, NY, USA. Globerson, A. and Jaakkola, T. (007). Fxng Max- Product: Convergent message passng algorthms for MAP LP-relaxatons. In NIPS, pages Kolmogorov, V. (006). Convergent tree-reweghted message passng for energy mnmzaton. IEEE Trans. Pattern Anal. Mach. Intell., 8: Kumar, A. and Zlbersten, S. (010). MAP estmaton for graphcal models by lkelhood maxmzaton. In NIPS, pages Kumar, M. P., Kolmogorov, V., and Torr, H. S. P. (009). An analyss of convex relaxatons for map estmaton of dscrete mrfs. J. Mach. Learn. Res., 10: Pearl, J. (1988). Probablstc Reasonng n Intellgent Systems. Morgan Kaufmann Publshers Inc. Ravkumar, P. and Lafferty, J. (006). Quadratc programmng relaxatons for metrc labelng and Markov random feld MAP estmaton. In ICML, pages Sontag, D., Globerson, A., and Jaakkola, T. (010). Introducton to Dual Decomposton for Inference. Optmzaton for Machne Learnng. Sontag, D., Meltzer, T., Globerson, A., Jaakkola, T., and Wess, Y. (008). Tghtenng LP relaxatons for MAP usng message passng. In UAI, pages Srperumbudur, B. and Lanckret, G. (009). On the convergence of the concave-convex procedure. In NIPS, pages Wanwrght, M., Jaakkola, T., and Wllsky, A. (00). MAP estmaton va agreement on (hyper)trees: Message-passng and lnear programmng approaches. IEEE Transactons on Informaton Theory, 51: Wanwrght, M. J. and Jordan, M. I. (008). Graphcal models, exponental famles, and varatonal nference. Foundatons and Trends n Machne Learnng, 1: Yanover, C., Meltzer, T., and Wess, Y. (006). Lnear programmng relaxatons and belef propagaton an emprcal study. J. Mach. Learn. Res., 7:006. Yulle, A. L. and Rangarajan, A. (003). The concaveconvex procedure. Neural Comput., 15: