Dynamic Programming Bipartite Belief Propagation For Hyper Graph Matching

Transcription

1 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) Dynami Programming Bipartite Belief Propagation For Hyper Graph Mathing Zhen Zhang 1, Julian MAuley 2, Yong Li 1, Wei Wei 1, Yanning Zhang 1, Qinfeng Shi 3 1 Shool of Computer Siene & Engineering, Northwestern Polytehnial University, Xi an, China 2 Computer Siene and Engineering Department, University of California, San Diego, USA 3 Shool of Computer Siene, The University of Adelaide, Australia zhangzhen@mail.nwpu.edu.n Abstrat Hyper graph mathing problems have drawn attention reently due to their ability to embed higher order relations between nodes. In this paper, we formulate hyper graph mathing problems as onstrained MAP inferene problems in graphial models. Whereas previous disrete approahes introdue several global orrespondene vetors, we introdue only one global orrespondene vetor, but several loal orrespondene vetors. This allows us to deompose the problem into a (linear) bipartite mathing problem and several belief propagation sub-problems. Bipartite mathing an be solved by traditional approahes, while the belief propagation sub-problem is further deomposed as two sub-problems with optimal substruture. Then a newly proposed dynami programming proedure is used to solve the belief propagation sub-problem. Experiments show that the proposed methods outperform state-of-the-art tehniques for hyper graph mathing. 1 Introdution The feature mathing problem aims to establish onsistent orrespondenes between two feature sets, and is a fundamental subroutine in omputer vision and pattern reognition [Leordeanu and Hebert, 2; Torresani et al., 28; Li et al., 21]. When establishing orrespondenes between features, it is important to onsider not only loal similarity between features, but also strutured similarity between sets of features. Graphial models provide an elegant framework to enode feature mathing problems, as they are naturally able to express strutured relationships between sets of variables. However, graphial model based methods [Zhang et al., 216; Torresani et al., 28; Ahn et al., 21] are typially limited to pairwise mathing, whih an be used to express (for example) mathes between rigid strutures. Unfortunately, pairwise relations are insuffiient to enode rih geometrial strutures [Lee et al., 211]; thus higher-order information should be inluded and the problem formulated as one of hyper graph mathing [Yan et al., 21; Nguyen et al., 21; Lee et al., 211; Duhenne et al., 29]. Suh a formulation an be more robust to noise and outliers, though hyper graph mathing is less well studied than pairwise mathing, due to the inreased diffiulty of optimization. Among hyper graph mathing methods, most reent ones fall into one of two ategories. In the first ategory, the orrespondene vetor (whih enodes the mathing) is relaxed to be ontinuous, and then post-proessing (suh as the Munkres/Hungarian algorithm) is used to disretize the orrespondene vetor; Hyper Reweighted Random Walks Methods (HRRWM) [Lee et al., 211] belong to this ategory. The seond ategory, whih inludes Disrete Hyper Graph Mathing [Yan et al., 21] and Tensor Blok Coordinate Asent Graph Mathing (TBCAGM) [Nguyen et al., 21], relaxes the mathing objetive as a multi-linear or multi-quadrati problem, and then alternating optimization methods are used to optimize over the relaxed objetive. Due to the relaxation suh alternating optimization methods may return multiple inonsistent orrespondene vetors, though in pratie their performane is state-of-the-art. Regularizers an be added to the objetive to enourage the multiple orrespondene vetors to be onsistent, however this may result in a looser relaxation. Another issue for multi-linear or multi-quadrati relaxations is their non-onvexity; due to non-onvexity, alternating optimization methods may easily get stuk in loal optima. In this paper, we formulate the hyper graph mathing problem as a onstrained higher order MAP inferene problem in a graphial model. To solve this we propose a onvex linear programming (LP) relaxation. In the primal form of the relaxation, rather than introdue several global orrespondene vetors, we introdue only one global orrespondene, but several loal orrespondene vetors, whih represent orrespondenes only within a small sub-graph (e.g. an edge, a triplet or other small sub-graph). In the dual formulation, the hyper graph mathing problem is deomposed as a linear bipartite mathing problem (whih an be effiiently solved by the Munkres/Hungarian algorithm [Munkres, 197]), and several non-linear mathing problems, whih we term belief propagation sub-problems, with relatively small sale. These sub-problems are solved iteratively, and during optimization the global orrespondene vetor is enouraged to be onsistent with the loal orrespondenes. If the global orrespondene vetor is onsistent with all loal orrespondenes, our algorithm is guaranteed to provide an exat solution. 4662

2 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) Although some previous work has used similar formulations for pairwise mathing problems [Torresani et al., 28; Zhang et al., 216], the exhaustive searh proedures involved when solving belief propagation sub-problems are omputationally expensive when applied to hyper graph mathing. As a response, by exploiting the internal struture of hyper graph mathing problems, we further deompose the belief propagation sub-problems as several overlapping subproblems with optimal substruture. This allows effiient dynami programming methods to be developed to solve these problems. We title the proposed methods Dynami Programming Bipartite Belief Propagation. Experiments show that the proposed methods outperform the urrent state-of-the-art in hyper graph mathing problems. 2 Notation and Problem Formulation A hyper graph H = (V, C) onsists of nodes v V and hyper ars C, where a hyper ar is a subset of V. The order of a hyper ar is its ardinality and the order of a hyper graph is the largest order among all hyper ars. We seek a orrespondene between two hyper graphs: the model graph H P = (V P, C P ) and the data graph H Q = (V Q, C Q ), and their assoiated attributes A P and A Q. For any hyper ar from C P or C Q, there is a orresponding attribute, denoted by a P or a Q. Attributes ould enode the orientation of a keypoint, the length of an edge, the angle between two edges, et.. The goal of hyper graph mathing is to find the optimal orrespondene between V P and V Q. Here to simplify the derivation, we assume that V = V P = V Q = {1, 2,, n}, though this an easily be relaxed (e.g. to handle outliers) by adding dummy nodes in either V P or V Q. Given a orrespondene vetor y i = j (meaning that the i th node in V P orresponds to the j th node in V Q ), finding an optimal oneto-one mathing between V P and V Q an be formulated as a onstrained MAP inferene problem in a graphial model: max y θ i (y i )+ θ (y ), s.t. 1(y i = l) = 1, l V,(1) C P where 1(S) is an indiator funtion (i.e. 1 if S is true and otherwise). θ i (y i ) and θ (y ) are value funtions (or potentials ) of the form θ i (y i ) = ϕ(a P i, a Q y i ), θ (y ) = ϕ(a P, a Q y ), (2) where ϕ are real-valued funtions, whih measure the similarity between unary or higher order attributes. In pratie, the similarity funtion are often defined as Duhenne et al.; Nguyen et al. [29; 21] ϕ(a P, a Q y ) = γ 1, exp( a P a Q y /γ 2, ), (3) where γ 1, and γ 2, are problem-speifi parameters for order hyper ars. 3 LP Relaxation and Its Dual The problem (1) is NP-hard in general, thus relaxations are required. The approahes of Zhang et al. [216] and Torresani et al. [28] are restrited to pairwise problems. Thus Table 1: Introdued dual variables Constraints i V P, y i µ i(y i) = 1 l V Q, µi(l) = 1 C P, i, y \{i} µ (y ) = µ i(y i ) Dual Variables u i v l λ i(y i ) we proposed a linear programming (LP) relaxation for both pairwise and hyper graph mathing problems as follows: argmax µ i, θ i + µ, θ, (4a) µ C P s.t. C P, y µ (y ) = 1, y, µ (y ), (4b) C p, i, y \{i} µ (y ) = µ i (y i ), (4) i V, l V, µ i (l) = 1, (4d) where f, g = x f(x)g(x) denotes the inner produt of two funtions. In the LP relaxation, µ = [µ i (y i )],yi [n] is the global orrespondene vetor, and µ i (y i = j) = 1 means that the i th node in V P orresponds to the j th node in V Q. Eah µ (y ) is a loal orrespondene vetor, and µ (y = [a i ] i ) = 1 means that the i th node in V P orresponds to the (a i ) th node in V Q for all i. It is well-known that off-the-shelf solvers suh as CPLEX are slow for LP problems like (4) [Werner, 27]. Thus we use dual blok oordinate desent based belief propagation to solve suh problems effiiently. First we introdue dual variables (shown in Table 1), then by standard Lagrangian duality the dual objetive is as follows, min g(λ, u, v) = λ,u,v + max y i + P u i + C P max y Y [ θ (y ) λ i (y i ) i [ θ i (y i ) u i v yi + ], ] λ i (y i ) i C P l V Q v l. () In order to satisfy the one-to-one mathing onstraint, the feasible set Y is { } i, y Y = y i [n]; l [n], i 1(y. (6) i = l) 1 The dual variables λ i (y i ) are referred to as messages, and u, v are referred to as mathing variables as in previous work [Zhang et al., 216]. Similar to Zhang et al. [216] for pairwise problems, we alternately optimize over u, v and λ. When optimizing u and v, we fix all λ s. When optimizing λ, in eah step we pik a partiular from C P, and allow those λ i (y i ), i to vary. Then the dual problems () an be 4663

3 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) deomposed as the following two groups of sub-problems: Mathing Sub-problem (7) [ min max θi (y i ) u i v yi + ] λ u i,v i y i(y i ) i C P,i + u i + v l, P l V Q Belief Propagation Sub-problem (8) [ min max θ (y ) λ i (y i ) ] λ y Y i + [ max ϑi (y i ) + λ i(y i ) ], y i i :i, C where λ denotes [λ i (y i )] i, and the augmented potential ϑ i (y i ) is defined as ϑ i (y i ) = θ i (y i ) u i v yi. (9) The mathing sub-problem (7) an be solved by the Hungarian algorithm [Zhang et al., 216]. However, the belief propagation sub-problems are not easy to solve in general. For general higher order potentials, the time omplexity for solving (8) is O(n ), whih inreases very quikly as the order of hyper ars grows. However, for hyper-graph mathing problems the higher order potentials are usually sparse [Yan et al., 21; Nguyen et al., 21; Lee et al., 211]. Thus we will use the sparsity to derive an effiient solver. Remarks For sparse potentials whose non-zero entries are positive, Rother et al. [29] s and Potetz and Lee [28] s approahes an be applied to derive an effiient message passing proedure. However, these methods are not ompatible with one-to-one mathing onstraints whih must be satisfied in mathing problems. 4 Dynami Programming Approahes for the Belief Propagation Sub-problem It is intratable to ompute affinities between two hyper graphs by onsidering all possible mappings between hyper ars. Even for pairwise mathing problems, the number of terms required to evaluate all possible pairs of onnetions is already n 4, whih is quite expensive for large-sale problems. Thus we seek methods that an be used to sparsify the potentials. Here we derive dynami programming approahes for the belief propagation sub-problem that exploit the sparsity of higher-order potentials. 4.1 Deomposition of the Belief Propagation Sub-problem To obtain our dynami programming belief propagation sheme, we start at a losed-form solution of the belief propagation sub-problem. To simplify our derivation we define b (y ) = θ (y ) λ i (y i ), C P i b i (y i ) = ϑ i (y i ) + λ i (y i ), (1) :i, C P then we an give the losed-form solution of (8) as λ i(y i ) =λ i (y i ) b i (y i ) (11) + 1 [ max b (ŷ ŷ Y y i ) + b i (ŷ i ) ]. where the domain Y yi is defined as i Y yi = {ŷ Y \{i} ŷ i = y i }. (12) The losed-form solution (11) an be omputed by exhaustive searh. However, the time omplexity required would be O(n ), whih is not salable for hyper graph mathing problems. Thus we deompose the most expensive part (the max-marginal) into several overlapping problems with optimal substruture. Proposition 1. The max-marginal proedure in (11) an be be performed as follows max [b (ŷ ŷ Y y i ) + { b i (ŷ i )] = max max ζ i (ŷ i ), ŷ i Y y i i max 1(θ (ŷ ŷ Y y i ) > ) [ θ (ŷ ) + ζ i (ŷ i ) ]}, (13) i where ζ i (ŷ i ) = b i (ŷ i ) λ i (ŷ i ). The seond sub-problem (last row of (13)) only onsiders non-zero entries of θ (y ), whih an be done effiiently via exhaustive searh. The solution of the first term (seond last row of (13)) an be reformulated as a linear bipartite mathing problem as follows. Proposition 2. Let y (l) i denote the entries orresponding to the l th largest ζ i (y i ), and let Ỹ yi = {ŷ ŷ Y yi, i \ {i}, y i {y (1) then we have that ξ i (y i ) = max ŷ Y y i i ζ i (ŷ i ) = max ŷ Ỹy 1 i,... y( ) i }} ζ i (ŷ i ). i By Proposition 2, the first term in (13) an be effiiently omputed via partial sort operations, and solving a bipartite mathing problem whose ost matrix size is 2. While the size of is very small (usually 3 or 4) in most higher order mathing problems [Yan et al., 21; Nguyen et al., 21] the bipartite mathing problem an be solved via exhaustive searh with omplexity O(!). For large liques, we an use Hungarian methods to further aelerate the proedure. 4.2 The Partial Reparametrization Formulation The terms b i (y i ) and b (y ) are in fat reparametrizations of the augmented potentials, that is y b i (y i ) + b (y ) = ϑ i (y i ) + θ (y ). C P C P Thus in our implementation, we only store b i (y i ), θ (y ), all messages λ, and mathing variables u, v. Using this formulation, the augmented potentials ϑ i (y i ) and θ i (y i ) an be reovered easily. Another benefit of this formulation (rather 4664

4 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) Algorithm 1: The DPMU Proedure input : A lique, potential b i(y i), i, messages λ i(y i). output: b i (y i), λ i(y i), i 1 for i do 2 ζ i(y i) = b i(y i) λ i(y i); 3 Compute ξ i(y i), i by Proposition 2; 4 with ζ i(y i) as input; for i do b i (y i) ξ i(y i); 6 for y non-zero entries of θ (y ) do 7 f v = θ (y ) + i ζi(yi); 8 for i do 9 if f v > b i (y i) then b i (y i) f v; 1 for i do 11 b i (y i) 1 b i (y i); 12 λ i(y i) λ i(y i) b i(y i) + b i (y i); than storing all potentials) is that signifiant omputation an be avoided. Reall the definition of b i (y i ) in (1); if we only store messages, then several messages have to be added together every time we ompute b i (y i ). However the optimal b i (y i) an be omputed diretly from max-marginals by the following proposition. Proposition 3. The optimal b i (y i) an be omputed by b i (y i ) = θ i (y i ) + λ i(y i ) + λ i(y i ) = 1 i:i, max [b (y ) + b i (y i )]. (14) i ŷ Y y i By Proposition 3 and (11), the optimal messages an be omputed as λ i(y i ) = λ i (y i ) + b i (y i ) b i (y i ). The above results, Propositions 2 and 1 together result in our Dynami Programming Message Updating (DPMU) algorithm in Algorithm 1. Putting DPMU into a oordinate desent framework results in our Dynami Programming Bipartite Belief Propagation (DPBBP) algorithm in Algorithm 2. Deoding Our Dynami Programming Bipartite Belief Propagation methods are dual based methods, thus a feasible integer solution is required. We onsider two strategies to deode an integer solution. In the first strategy, integer solutions are deoded by the bipartite mathing. This strategy is natural sine in eah iteration of belief propagation, a bipartite mathing proedure is used to optimize over the dual variables u and v, and they provide a feasible primal integer solution at the same time. In the seond strategy, we use a multi-linear relaxation method to further refine our results. Although suh methods an easily get stuk at non-optimal points, our algorithm usually provides near optimal solutions in pratie, so that using multi-linear relaxations to refine the results usually yields satisfatory results. Multi-linear relaxations are more expensive than bipartite mathing, thus we only perform this refinement proedure at the last step. Algorithm 2: The Dynami Programming Bipartite BP input : Potentials θ i(y i), i V, θ (y ), C P ; MaxIter; threshold ɛ 1 and ɛ 2. output: y. 1 f max =, u =, v = ; 2 b i(y i) = θ i(y i), i V, y i [n]; 3 for k {1, 2,..., MaxIter} do 4 for C P do Compute optimal λ i(y i), b i (y i), i by Algorithm 1; 6 Updating λ i(y i), b i(y i), i to λ i(y i), b i (y i), i ; 7 Compute optimal u, v for sub-problem (7), and deoding a andidates ŷ by the Hungarian algorithm; 8 Update [u, v] to [u, v ]; 9 f k = θi(ŷi) + C θ (ŷ P ); 1 If f k > f max then f max = f k, y = ŷ; 11 g k urrent dual objetive of (); 12 if f max g k < ɛ 1 or g k g k 1 < ɛ 2 then break; 4.3 Analysis In this setion, we analyse the onvergene, time omplexity and auray of the proposed method. The onvergene of the proposed methods is guaranteed by the following proposition. Proposition 4. For arbitrary bounded input, the Dynami Programming Bipartite Belief Propagation produes a monotonially dereasing dual objetive sequene. Sine the MAP value is a natural lower bound of the dual objetive, the proposed methods must onverge to a fixed dual objetive. At the fixed point of the proposed algorithm, the loal and global orrespondene vetors have the following weak onsisteny properties: Proposition (Weak Consisteny). Assume that Dynami Programming Bipartite Belief Propagation onverges to a fixed point; let ŷ be the optimal assignment produed by linear bipartite mathing in (7), and C P let ȳ = argmax[b (y ) + b i (y i )], y Y i then we have that C P, b i (ȳ i ) = b i (ŷ i ) = max b i (y i ). (1) y i i i i By the weak onsisteny properties, we an see that the loal and global orrespondene vetors are enouraged to attain the same optimal objetive value on unary terms. By this property, we immediately obtain the ondition for global optimality. Proposition 6. Assume that Dynami Programming Bipartite Belief Propagation onverges to a fixed point; if i V, there some ŷ i s.t. ȳ i ŷ i, b(ŷ i ) > b(ȳ i ), then Dynami Programming Bipartite Belief Propagation provides a globally optimal solution of the problem (1). Time Complexity In eah iteration, we must solve C P belief propagation sub-problems and one linear bipartite 466

5 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) BCA BCA-MP BCA-IPFP HGM HRRWM TM BBP FGM DP-BBP DP-BBP-MLR Auray Auray Figure 1: Experimental results on the CMU House data set. Left: Typial mathing result of our algorithm (green lines: orret mathing; yellow lines: mathes between outliers; purple liens: wrong mathes); Middle: Average auray vs. image separation; Right: Average normalized objetive value. Top: The mathing problem inludes all landmarks. Bottom: For eah image pair, landmarks are randomly removed from eah image, and a rigid transformation is applied to the seond image with sale parameter 4 and rotation angle 3π/4. Due to the large sale and rotation transformation, pairwise methods failes to give satisfatory results. mathing problem. The time omplexity of solving a bipartite mathing problem is O(n 3 ). The time omplexity for solving belief propagation sub-problems by Algorithm 1 is O( n log +! + NNZ(θ (y )) ), where we use NNZ(θ (y )) to denote the set of non-zero entries of θ (y ). On the other hand the time omplexity for solving belief propagation sub-problems by exhaustive searh is O(n ). Although both expressions grow exponentially in, the proposed methods are still signifiantly faster than exhaustive searh for sparse potentials. This is beause in pratie the value is usually very small (often 2 or 3), so that and even! an be viewed as (small) onstants. Treating as a onstant, the time omplexity for eah iteration of our algorithm is O(n 3 + C P n + C NNZ(θ (y P )) ). Experiments In this setion, we apply Dynami Programming Bipartite Belief Propagation (DP-BBP) to several hyper graph mathing tasks. Our method is applied with two deoding shemes. In the first deoding sheme, we simply use solutions for bipartite mathing as andidates, while in the seond one, we use multiple linear relaxation based methods to further refine the deoding results. The first algorithm is referred to as DP-BBP, and the seond is referred to as DP-BBP-MLR. Our method is ompared with several existing popular and state-of-the-art hyper graph mathing algorithms, inluding: The Tensor Blok Coordinate Desent Method for multilinear relaxation ( BCA ) [Nguyen et al., 21]; The Tensor Blok Coordinate Desent Method for multiquadrati relaxation with IPFP [Leordeanu et al., 29] as the quadrati solver ( BCA-IPFP ); The Tensor Blok Coordinate Desent Method for multiquadrati relaxation with Max Pooling Mathing [Cho et al., 214] as the quadrati solver ( BCA-MP ); The Probabilisti Hyper Graph Mathing method ( HGM ) [Zass and Shashua, 28]; The Tensor Mathing method ( TM ) [Duhenne et al., 29]; The hyper graph mathing via reweighted random walking method ( HRRWM ) [Lee et al., 211]. Two state-of-the-art quadrati graph mathing algorithms, Fatorized Graph Mathing (FGM) [Zhou and De la Torre, 212] and Bipartite Belief Propagation (BBP) [Zhang et al., 216], are also inluded as baselines. We mainly ompare the auray and objetive value for different algorithms. Implementation details In our experiments, the proposed algorithms DP-BBP and DP-BBP-MLR are implemented in ++ and python. As in previous work [Zhang et al., 216], when there is a gap between the dual and deoded primal, a most-frational-first branh-and-bound strategy is used to tighten the gap. We run at most 1 branh-and-bound iterations, and in eah branh-and-bound, we run at most 1 iterations of DP-BBP or DP-BBP-MLR. All other methods are based on publily available implementations. Higher Order Potential Computation In our experiments, we mainly onsider third-order similarities, whih are invariant to sale and rotation. Given two sets of points P = [p i ] n i=1 and Q = [q i] n i=1, we use Delaunay triangulation to onstrut the set of hyper ars C P and C Q. The higher order potentials are omputed as θ (y ) = exp( 3 i=1 d(αi, αy i ))1(y C Q ) (16) where α(α i y i ) is the i th angle of the triangle (y ), and the distane between two angles d(α, i αy i ) is omputed as { α d(α, i αy i i ) = αy i, α i αy i π, 2π α i αy i (17), otherwise. The pairwise potentials for pairwise methods are as follows, θ ij (y i, y j ) = exp( dij dy i y j τ ij,yi )1({y y i, y j } C Q ) (18) j where d ij (d yiy j ) are Eulidean distanes between nodes i and j (y i and y j ), and τ ij,yiy j is a user-speified parameter. 4666

6 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) BCA-IPFP BCA-MP BCA BCA-IPFP BCA-MP..2 TM BBP DP-BBP-MLR FGM DP-BBP DP-BBP-MLR DP-BBP-MLR DP-BBP HRRWM Auray Auray HGM 1..7 DP-BBP RRWHM BCA-IPFP 1.. RRWHM BCA-MP Figure 2: Experimental results on the Car and Motorbike data set. Top two rows: Typial mathing results (The olor of lines have the same meaning as Figure 1). Bottom: The left two olumns are results on Car, and the right two olumns are results on Motorbike. As we applied large sale and rotation transformation to the original data, pairwise methods failes to give satisfatory results..1 CMU House Dataset 1% higher ompared to these methods. The CMU house dataset has been widely used in previous work to evaluate mathing algorithms [Nguyen et al., 21]. The house sequene onsists of 111 frames of a toy house aptured from different view points. In eah image there are 3 manually marked landmark points with known orrespondenes. We math all images spaed by 1, 2,..., 9 frames and ompute the average performane per separation gap. In this experiment, unary terms are set to zero [Nguyen et al., 21]. Again for eah image we use Delaunay triangulation to onnet landmark points. Eah triplet in the onneted graph beomes a hyper ar in the hyper graph (or eah edge in the ase of pairwise mathing methods). Higher order potentials are omputed as in (17) and pairwise potentials are omputed as in (18). For hyper graph mathing methods, we only use higher order potentials and we use pairwise potentials only for pairwise mathing methods, where we set the parameter τij,yi yj = 1/2 [Leordeanu et al., 29]. We onsider two different settings. In the first setting, all landmark points are used to onstrut the hyper graph. In the seond setting, for eah image pair, we randomly remove landmarks from eah image, whih results in outliers (i.e. landmarks that do not have any orrespondenes). Then for landmarks in the seond image in the pair, a sale transformation with sale parameter 4 and a rotation transformation with angle 3π/4 are applied. Results are shown in Figure 1. In the first setting, for small image separation all methods find the exat mathing, while the auray of BCA, BCA-MP, HGM and TM drops as the separation inreases. In the seond setting, pairwise mathing methods fail to obtain satisfatory results as they are sensitive to sale variation and rotation. For hyper graph mathing methods, for image separation less than 8 frames our algorithm attains similar (or slightly better) auray ompared to the best results among BCA-MP, BCA-IPFP and HRRWM. For the highest separation our methods auray is more than The Cars and Motorbikes Dataset The Cars and Motorbikes dataset onsists of 3 pairs of images of ars and 2 pairs of images of motorbikes from the Pasal 27 dataset [Everingham et al., 29]. Eah pair ontains a number of ground-truth orrespondenes and several outliers. In the dataset, the sale and viewing angle have only small variations. To investigate the performane of mathing methods under large sale and viewing angle variations, for eah image pair, we add a rigid transformation to the keypoints in the seond image with sale parameter 4 and rotation angle π/4. We set the parameter τij,yi yj = min(dij, dyi yj ) as in previous work [Zhou and De la Torre, 212; Zhang et al., 216]. We randomly added 1-2 outliers from the bakground to the mathing problem, with the result shown in Figure 2. From the figure we an see that our methods provide better average auray in all ases. 6 Conlusion In this paper, we formulate hyper graph mathing problems as onstrained MAP inferene problems in graphial models. The proposed method expliitly models only one global orrespondene vetor, but several loal orrespondene vetors, allowing us to deompose diffiult high order optimization problems into a linear bipartite mathing problem and several belief propagation sub-problems with small sale. We further propose a dynami programming proedure to effiiently solve the belief propagation sub-problems. Experiments demonstrate the proposed method outperforms stateof-the-art methods for hyper graph mathing. Aknowledgments This work was supported by NSFC Projet , and , China 863 Projet 21AA1642, ARC Projet DP and DP16173.

7 Proeedings of the Twenty-Sixth International Joint Conferene on Artifiial Intelligene (IJCAI-17) Referenes [Ahn et al., 21] Sung-Soo Ahn, Sejun Park, Mihael Chertkov, and Jinwoo Shin. Minimum weight perfet mathing via blossom belief propagation. In NIPS, 21. [Cho et al., 214] Minsu Cho, Jian Sun, Olivier Duhenne, and Jean Pone. Finding mathes in a haystak: A maxpooling strategy for graph mathing in the presene of outliers. In CVPR, 214. [Duhenne et al., 29] Olivier Duhenne, F Bah, In-So Kweon, and J Pone. A tensor-based algorithm for highorder graph mathing. In CVPR, 29. [Everingham et al., 29] Mark Everingham, Lu Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual objet lasses hallenge. In PASCAL Challenge Workshop, 29. [Lee et al., 211] Jungmin Lee, Minsu Cho, and Kyoung Mu Lee. Hyper-graph mathing via reweighted random walks. In CVPR, 211. [Leordeanu and Hebert, 2] Marius Leordeanu and Martial Hebert. A spetral tehnique for orrespondene problems using pairwise onstraints. In Computer Vision, 2. ICCV 2. Tenth IEEE International Conferene on, volume 2, pages IEEE, 2. [Leordeanu et al., 29] Marius Leordeanu, Martial Hebert, and Rahul Sukthankar. An integer projeted fixed point method for graph mathing and MAP inferene. In NIPS, 29. [Li et al., 21] Hongsheng Li, Edward Kim, Xiaolei Huang, and Lei He. Objet mathing with a loally affine-invariant onstraint. In Computer Vision and Pattern Reognition (CVPR), 21 IEEE Conferene on, pages IEEE, 21. [Munkres, 197] James Munkres. Algorithms for the assignment and transportation problems. Journal of the soiety for industrial and applied mathematis, (1):32 38, 197. [Nguyen et al., 21] Quynh Nguyen, Antoine Gautier, and Matthias Hein. A flexible tensor blok oordinate asent sheme for hypergraph mathing. In CVPR, 21. [Potetz and Lee, 28] Brian Potetz and Tai Sing Lee. Effiient belief propagation for higher-order liques using linear onstraint nodes. Computer Vision and Image Understanding, 112(1):39 4, 28. [Rother et al., 29] Carsten Rother, Pushmeet Kohli, Wei Feng, and Jiaya Jia. Minimizing sparse higher order energy funtions of disrete variables. In Computer Vision and Pattern Reognition, 29. CVPR 29. IEEE Conferene on, pages IEEE, 29. [Torresani et al., 28] Lorenzo Torresani, Vladimir Kolmogorov, and Carsten Rother. Feature orrespondene via graph mathing: Models and global optimization. In ECCV, 28. [Werner, 27] Tomas Werner. A linear programming approah to max-sum problem: A review. IEEE Transations on Pattern Analysis and Mahine Intelligene, 29(7): , 27. [Yan et al., 21] Junhi Yan, Chao Zhang, Hongyuan Zha, Wei Liu, Xiaokang Yang, and Stephen M Chu. Disrete hyper-graph mathing. In CVPR, 21. [Zass and Shashua, 28] Ron Zass and Amnon Shashua. Probabilisti graph and hypergraph mathing. In CVPR, 28. [Zhang et al., 216] Zhen Zhang, Qinfeng Shi, Julian MAuley, Wei Wei, Yanning Zhang, and Anton van den Hengel. Pairwise mathing through max-weight bipartite belief propagation. In CVPR, 216. [Zhou and De la Torre, 212] Feng Zhou and Fernando De la Torre. Fatorized graph mathing. In CVPR,