Two Efficient Algorithms for Outlier Removal in Multi-view Geometry Using L Norm

Transcription

1 Two Efficient Algorithms for Outlier Removal in Multi-view Geometry Using L Norm Yuchao Dai 1, Mingyi He 1 Hongdong Li 3, 1 School of Electronics and Information, Northwestern Polytechnical University Shaanxi Key Laboratory of Information Acquisition and Processing Xi an, 7119, China Research School of Information Sciences and Engineering, Australian National University Canberra, ACT 1, Australia 3 Canberra Research Lab, NICTA Abstract L norm has been recently introduced to multi-view geometry computation to achieve globally optimal computation. It however suffers from a serious sensitivity to outliers. A few remedies have been proposed but with high computational complexity. This paper presents two efficient algorithms to overcome these problems. Our first algorithm is based on a cheap and effective local descent method (as opposed to the conventional but expensive SOCP(Second Order Cone Programming)). The second algorithm further improves the first one by using a Depth-first search heuristics. Both algorithms retain the nice property of global optimality of the L scheme, while at cost only a small fraction of the original computation. Experiments on both synthetic data and real images have validated the proposed algorithms. 1. Introduction A recent trend in Multi-view geometry computation in computer vision is to seek truly globally optimal (hence unique) solution. The L -norm based methods is such a scheme with the desired property. Under the L norm, an optimal solution with geometrically meaningful merit is guaranteed. The L scheme has therefore been recently applied to a variety of geometric vision problems such as triangulation, camera resection(uncalibrated case), homography, structure and motion recovery with known camera orientation [5], camera motion recovery [1], outlier detection [7], camera pose(calibrated case), two-view relative orientation [], multi-camera system [] and etc. Outlier Removing. A major criticism of using L minimization is, however, its sensitiveness to outliers. In order to apply the L optimization framework, the data should be free of outliers or with as few outliers as possible. To deal with this issue, researchers have tried various approaches. RANSAC [] is the most popular method. But, as it relies on random sampling to find a set of measurements containing only inliers, it may easily miss a few outliers out. This is already fatal for the L scheme because in which case even a single remaining outlier may destroy the estimation. Sim and Hartley [13] proposed a method by throwing away bad points with the maximal residuals. However, as their experimental results showed, this remedy is generally not very reliable for discriminating true inliers from outliers as it will throw outliers and inliers at the same time. Li [7] proposed a theoretic algorithm for L triangulation with outliers. The algorithm aimed to remove small number of outliers after RANSAC polish. However, the algorithm is generally expensive, particularly when the ratio of outliers is large. Another reason is that it uses SOCP as its solver, which we argue that SOCP is relatively expensive. Olsson et al. [11] proposed a framework that first finding local maxima subset through RANSAC then verifying whether the model is global maxima and they claimed that the algorithm had a polynomial-time bound. Martinec et al. [9] proposed to use L to delete the outliers in relative motion measurements or epipolar geometry(eg) rather than image points. In this paper we present two algorithms to solve the outlier removal problem more efficiently. The first one is a unified framework for outlier removal based on the so-called LP-type problem theory along with fast localdescent primitive solver. The second is a heuristic algorithm using depth-first priority search. Both algorithms are

2 able to speed up the computation significantly even when the faction of outliers is large. The algorithms proposed in this paper can be employed after the RANSAC in order to further remove any remaining outliers.. L with bisection method For completeness, this section briefly reviews the standard bisection SOCP method for solving L problems. Many problems in geometric vision may be written in the following form: A i x + b i min max x i c T i x + d i subject to c T i x + d i (1) It is easily seen that the problem (1) can be transformed, by introducing an additional variable γ, into an equivalent problem of the following form: min γ subject to A ix+b i γ(c T i x+d i ) and γ > γ,x () For any fixed value of γ, the set of points satisfying the constraint A i x + b i γ(c T i x + d i) is a convex set. Thus () is an SOCP problem which we may easily solve via a bisection scheme as follows. There has been recent work for speeding up the computations using fractional programming theory [1]. Require: Initial interval [γ l, γ u ] known to contain the optimal value of γ and tolerance ɛ >. 1: repeat : γ := (γ l + γ u )/. 3: Solve the SOCP feasibility problem to get x. : if feasible then 5: x = x, γ u := max i A ix+b i c T i x+di. : else 7: γ l := γ : end if 9: until γ u γ l ɛ. Algorithm 1: Improved Bisection Algorithm 3. Solving L optimization as LP-type Problem It is easily to observe that L minimization is determined by only part of the constraints thus implies that we could solve the problem efficiently. In this section, we describe the work on solving the L optimization problem treated as an LP-type problem LP-type Problem Paper [7] introduced a very original perspective to revisit the L problems in multi-view geometry,which is based on the so-called LP-type problems theory originally due to [1, 3]. Consider optimization problems specified by pairs (H, w), where H is a finite set, and w : H W is a function with values in a linearly ordered set (W, ); we assume that W has a minimum value. The elements of H are called constraints, and for G H, w(g) is called the value of G. Definition An abstract optimization problem is called an LP-type problem if the monotonicity and locality axioms below are satisfied. 1. Monotonicity axiom: For any F, G with F G H, we have w(f ) w(g).. Locality axiom: For any F G H with w(f ) = w(g) and for any h H, w(g) < w(g {h}) implies w(f ) < w(f {h}). The goal of LP-type computation is to compute a minimal subset B H H with the same value as H that is ω(b H ) = ω(h) (from which, in general, the value of H is easy to determine). Here we refer some basic terms to describe the LP-type problem. A set B of constraints is called a basis if w(b ) < w(b) for every proper subset of B. Given a set G of constraints, a subset B G is called a basis of set of constraints if it is a basis and w(b) = w(g)(i.e., an inclusion-minimal subset of G with equal w-value). Combinatorial dimension: The maximum cardinality of any basis in LP-type problem (H, w), denoted by δ = δ(h, w). Violation test: Decides whether or not w(b) < w(b {h}), for a basis B and a constraint h. Basis computation: Delivers a basis of B {h}, for a basis B and a constraint h. Support set: Given a minmax problem as with a solution Θ opt, the support set I supp is the set I supp = {i I f i (Θ opt ) = δ opt } From the definition of basis and support set, we can easily draw the conclusion that the basis for a set of constraints is the subset of the support set. We say that a constraint h H violates a subset G H if we have ω(g h) > ω(g). The violation set V (G) is all the constraints that violate G. The cardinality k of the violation set V (G) of G is called the level of the set G, k = V (G). For more details about the application of the LP-type theory to L computation, the reader is referred to [7] and references therein.. Algorithm-1: an exact approach In this section, we present our first algorithm for outlier removal under L norm. The key motivation is to find the optimal solution in the presence of outliers.

3 We first formulate the outlier removal with the example of triangulation under L norm. In this case we are given camera projection matrices P i, i = 1,, n and correspondences x i across multiple views. The aim is to infer the 3D point X in 3D space. Under L norm, we seek to attain the smallest maximal residual that is δ opt = min X max i f i (X). The definition of outlier can be stated that some measurements (the inliers) fit a parameterized model within a given maximum residual δ in while the other measurements are outliers. Denote the index set I of all the constraints made up of two subsets I in and I out as the inlier and outlier sets, respectively where the inlier set satisfies min X max i Iin f i (X) < δ in. According to this definition, the inlier set may be taken to be the largest subset, or any maximal (or even non-maximal) subset of I satisfying the inlier condition. On the other side, when we seeking model for the constraints, we aim to find the largest possible subset with a preset tolerance δ in. Thus we have the following mathematical model as: min max f i (X) X i I in max I in (3) Subject to f i (X) < δ in This is a multiple objective optimization problem. To solve the problem, a direct way is to change one of the objective function to constraint thus obtain the following model as: max I in Subject to f i (X) < δ in () which aims at a model with maximum inliers, whose residuals are all under given threshold. One of the key innovations that we use to speed up the computation is to accelerate the LP-type basis computation via a fast local method applied to primitive (sub)problems (to be defined later). First, we define the operation of Basis Change. Given the basis B k for the constraint set H, for each constraint in the basis b i, compute the new basis for the set H\{b i }. Thank to the locality, starting from the basis B k \{b i }, we are easily arrive at a new basis in fewer iteration. Second, for the tree structure contains all the basis at different levels, along all the paths, the function value will decease. So when we need to find the basis with the smallest function value, we only need to access the bases at the leaves of the tree. Algorithm-1 1. Initial basis finding: Finding the basis B satisfying all the constraints H = H, B = B(H ) and k =, n = 1;. Generate new bases: Generate new bases by performing the following procedures. Choose the current basis B n in the queue with level k; For each b B n, generate a basis B i = B(H n \b), H n = H n \b; (c) Add the new basis to the front of the queue, n = n + 1; 3. Selection Choose from the bases at level k with least L residual..1. Some details of the algorithm Primitive problem reduction. In [] [11], a main reduction theorem is proven which also gives an upper bound of the size of LP-type bases. Theorem.1 (main reduction theorem) Consider L problem min x max i f i (x), where x R n, i = 1,..., N, N > (n + 1), each f i (x) is pseudoconvex. Denote f I (x) = max i I f i (x), where I is a subset of N = 1,..., N. Also denote fi as the minimum of f I (x). Then there must exists a proper subset B which is of size B n + 1 such that fb = f N and x B = x N are their(equal) optimizers. This theorem predicts that the solution of the big problem is dominated by a small basis set of at most (n + 1) constraints. For the multi-view triangulation problem, the combinational dimension is, so only distinct primitive problems exist: 1-view which is trivial, -view, 3-view and -view. The left three primitive problems can be solved using local gradient method applying the KKT conditions. After obtaining the KKT conditions, the problem reduces to the following very small primitive problems and can be solved effectively by any local gradient descent method or Newton method, as did in []. -view case { Ci (x) = A i x + b i γ(c T i x + d i) =, i = 1, λ C 1 (x) + (1 λ) C (x) =, λ > 3-view case { Ci (x) = A i x + b i γ(c T i x + d i) =, i = 1,, 3 i λ i C i (x) =, i λ i = 1, λ i > -view case C i (x) = A i x + b i γ(c T i x + d i ) =, i = 1,...,.. Basis Change and Finding Now, we detail the steps for basis change using as an example the triangulation problem. The complete algorithm is given as follows.

4 1. Initialization: Randomly choose a pair of constraints as the initial basis B, set the number of elements in the basis as k =.. Violation Test: Solve the basis B t to obtain the 3D point X t, compute and sort the L residual for each constraint. If all the constraints are met, return the basis as B t ; else, select the constraint b which violates basis B t with the largest residual. 3. Compute new basis for subset B t b If the current basis consists of constraints denoted as constraints 1 and, while the candidate constraint to add is 3, test the bases {1, 3},{, 3} and {1,, 3} to meet all the constraints 1, and 3, set the corresponding basis size, the new basis is B t+1 ; If the current basis consists of 3 constraints denoted as constraints 1, and 3, while the candidate constraint to add is, test the bases {1, },{, }, {3, }, {1,, },{1, 3, }, {, 3, } and {1,, 3, } to meet all the constraints 1,,3 and, set the corresponding basis size, the new basis is B t+1 ; (c) If the current basis consists of constraints denoted as constraints 1,,3 and, while the candidate constraint to add is 5, test the bases {1, 5},{, 5}, {3, 5}, {, 5}, {1,, 5},{1, 3, 5}, {, 3, 5}, {,, 5}, {1,, 3, 5}, {1,,, 5},{1, 3,, 5} and {, 3,, 5} to meet all the constraints 1,,3, and 5, set the corresponding basis size, the new basis is B t+1 ;. Iteration: Do the above step until a basis satisfies all the constraints is found. Convergence Analysis. During each iteration, we add the constraint violate the basis with the largest residual thus the size of inlier set for the current basis B t will increase and the size of violation set will decrease. So the algorithm will terminate in finite steps and output a basis for the whole set I. The number of iteration is upper bounded by N the size of the whole set I however in practice the iteration will generally be far smaller than this bound. 5. Algorithm-: a heuristic depth-first search algorithm The above algorithm can obtain all the k-level basis thus output the optimal result. The upper bound for basis number is k = k Thus the complexity is bound by O( k+1 ) in the worst case. When k is small, we could obtain the optimal solution efficiently. However with the increase of k, the complexity become incredible to implement. Thus an efficient algorithm is in need. Let us consider the following model: min max f i (X) < δ in X i I in I out k where N is number of measurements and k is the upper bound of outliers in the measurements. To speed up the implementation, one choice is to relax the constraint on the size of inlier set. In Thus the model change to: min max f i (X) < δ in X i I in I out k + ɛ Compared with model 5, the number of outlier has been increased by ɛ that means we could remove a little more inliers constraints as outliers but it does not matter as at the cost of a few inliers removed we can speed up the implementation thus a compromise between efficiency and quality. Our analysis is based on some heuristic observation. As the basic property of outlier shown, with the outlier added, the L residual will change dramatically. So with the removing of the outliers, the L residual will decrease dramatically, when all outliers have been removed, the residual will not change more by removing more constraints. Based on this observation, we could apply a local gradient like algorithm. Given a basis and the constraints, we know that there is at least one outlier in the basis, removing each constraint in the basis will decrease the residual to different extent. To determine which constraint is the outlier with the largest possibility, we choose the one which cause the residual decrease with the largest amplitude. Repeated this operation, we could obtain a sequence of bases B, B 1,, B k. If we are lucky enough, we will obtain the optimal solution this way. If not we could remove a few more constraint to make the residual under the prefix threshold. When we do not obtain the feasible solution after the above iterations, or with more constraints removed we still can not obtain the feasible solution with residual under threshold, then we could apply backtrace operation to revisit the tree. The complete algorithm description is given below: Algorithm-: Depth-first search 1. Obtain the basis B at level ; (5) (). For each constraint in the basis, remove b B k, generate the new basis originally from the basis B k. Find the new basis with the smallest L residual. 3. Do the above step until the following cases are met:

5 The residual is lower than a given threshold δ; The depth is the larger than the upper bound of outlier number; The implementation process is illustrated in Figure 1 where B is the basis for all the constraints also the basis at level. The path that finds the basis for k = 3 is B B 1 B B 3. We can conclude that by solving bases number have linear relationship with level we can obtain the solution k= k=5 k= k= k= k=5 k= k=7 Figure. Synthetic experiment result for view with views corrupted by outliers. Log of residual for the view triangulation problem after removing,,5,,7 outliers respectively; Sorted residual for removing,,5,,7 outliers respectively. Log of k= k= k= k= Figure 1. Illustration of the operation on tree of bases Complexity Analysis For the problem of outlier removing, when applying the method of exhaustive search, the total number of feasible test is k i= Ci N. In [7] the total number of feasible test has been decreased to O( k ) which is independent of size of all the constraints N thus efficient reduce the number. In our new algorithm, the number of feasible tests is (k + ɛ).. Experiments To evaluate the performance of the proposed algorithms, we conduct both synthetic and real image experiments..1. Synthetic Data First to illustrate the success of the algorithm, we conduct two groups of experiments on synthetic data set. In the first group, we generate view triangulation problem where 3D points is randomly generated and projected on the image plane with Gaussian noise added and to simulate the effect of outlier, random image is corrupted by Gaussian noise with variance times the image noise. Figure illustrate the result which is obtained on 1 experiments. From this figure, we see that when we have removed k = constraint, almost all of the L residual is under.1, with further removing of constraints, the residual does decrease but with small amplitude thus show almost all the outliers have been removed successfully. The second group of synthetic experiments are to illustrate the performance for large number of outliers. In this experiment, outliers occurs in the 1 views. The result is shown as Figure 3. The figure shows the algorithm works well on large number of views. Figure shows the number of primitive problems for outlier dealing with the 5th, th, 7th outlier for the view 1 Figure 3. Synthetic experiment result for 1 view with views corrupted by outliers. Log of residual for the 1 view triangulation problem after removing,,1,,3 outliers respectively; Sorted residual for removing,,1,,3 outliers respectively. outlier problem. From the histogram, we see the number of primitive problems almost constant for different outlier Remving the 5th outlier Removing the th outlier Removing the 7th outlier Figure. Histogram of number of primitive problems for outlier. Removing the 5th outlier; Removing the th outlier; (c) Removing the 7th outlier. Experiments are conducted times for view triangulation with views is set as outlier... Real Image Experiments In order to test the proposed algorithm, we have used two publicly available sequences. 1 The first sequence consists of 11 images(of size 51 51) in a corridor. There are 1 points correspondence visible in all images. The other image set is a turnable sequence of a dinosaur, containing 3 images(of size 57 7) and, in total, 3 images correspondences with lots of occlusions. For this real image multiple view geometry triangulation problem, views of image points are perturbed by 5 pixel Gaussian noise as a simulation of the outliers. The result is shown as Figure and Figure 7. From the figures, we see the residual change dramatically from outlier to 1 outlier 1 Available at vgg/data.html (c)

6 resection, structure and motion known rotation and etc. Acknowledgment This work is supported by National Natural Science Foundation of China under key project number 737. The first author would like to thank the China Scholarship Council. NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program. (c) (d) Figure 5. Real Image: Corridor and Dinosaur until outlier, then the residual does not change more and under 1 pixel which means we have successfully removed almost all of the outliers as there is outliers in fact. 1 1 k= k= 1 3 Figure. Experimental Results on Corridor. Log of residual for the 1 view triangulation problem; for the 1 view triangulation problem. 1 1 k= k= Figure 7. Experimental Results on Dinosaur. Log of residual for the 1 view triangulation problem; for the 1 view triangulation problem. 7. Conclusion In this paper, two efficient algorithms are proposed to remove outliers using L norm. The first algorithm is based on a cheap and effective local descent method (as opposed to the conventional but expensive SOCP. The second algorithm further improves the first one by using a Depth-first search heuristics. Experiments on both synthetic and real data have proven the performance of the algorithms. For future work, rigorous analysis should be discussed and the same strategy can be applied to other outlier removing problem under L norm such as homograph, camera References [1] S. Agarwal, N. Snavely, and S. Seitz. Fast algorithms for L problems in multiview geometry. In CVPR, pages 1, June. [] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, ():31 395, [3] J. E. Goodman and J. O Rourke, editors. Handbook of discrete and computational geometry. CRC Press, Inc., Boca Raton, FL, USA, [] R. I. Hartley and F. Kahl. Global optimization through rotation space search. IJCV, (1): 79, 9. 1 [5] F. Kahl and R. Hartley. Multiple-view geometry under the L -norm. PAMI, 3(9):13 117,. 1 [] J.-H. Kim, H. Li, and R. Hartley. Motion estimation for multi-camera systems using global optimization. In CVPR, pages 1,. 1 [7] H. Li. A practical algorithm for L triangulation with outliers. In CVPR, pages 1, 7. 1,, 5 [] H. Li. Efficient reduction for solving L-infinity problems in multiview geometry. In CVPR, pages 1, June 9. 3 [9] D. Martinec and T. Pajdla. Robust rotation and translation estimation in multiview reconstruction. In CVPR, volume 1, pages 1, June 7. 1 [1] J. Matousek, M. Sharir, and E. Welzl. A subexponential bound for linear programming. Algorithmica, 1(/5):9 51, 199. [11] C. Olsson, O. Enqvist, and F. Kahl. A polynomial-time bound for matching and registration with outliers. In CVPR, pages 1, June. 1, 3 [1] K. Sim and R. Hartley. Recovering camera motion using L minimization. In CVPR, pages ,. 1 [13] K. Sim and R. Hartley. Removing outliers using the L norm. In CVPR, pages 5 9,. 1