Dual Principal Component Pursuit: Improved Analysis and Efficient Algorithms

Transcription

1 Dual Principal Component Pursuit: Improve Analysis an Efficient Algorithms Zhihui Zhu MINDS Johns Hopkins University Yifan Wang SIST ShanghaiTech University Daniel Robinson AMS Johns Hopkins University Daniel Naiman AMS Johns Hopkins University Rene Vial MINDS Johns Hopkins University Manolis C. Tsakiris SIST ShanghaiTech University Abstract Recent methos for learning a linear subspace from ata corrupte by outliers are base on convex l an nuclear norm optimization an require the imension of the subspace an the number of outliers to be sufficiently small [27]. In sharp contrast, the recently propose Dual Principal Component Pursuit (DPCP) metho [22] can provably hanle subspaces of high imension by solving a non-convex l optimization problem on the sphere. However, its geometric analysis is base on quantities that are ifficult to interpret an are not amenable to statistical analysis. In this paper we provie a refine geometric analysis an a new statistical analysis that show that DPCP can tolerate as many outliers as the square of the number of inliers, thus improving upon other provably correct robust PCA methos. We also propose a scalable Projecte Sub-Graient Metho (DPCP-PSGM) for solving the DPCP problem an show that it achieves linear convergence even though the unerlying optimization problem is non-convex an non-smooth. Experiments on roa plane etection from 3D point clou ata emonstrate that DPCP-PSGM can be more efficient than the traitional RANSAC algorithm, which is one of the most popular methos for such computer vision applications. Introuction Fitting a linear subspace to a ataset corrupte by outliers is a funamental problem in machine learning an statistics, primarily known as (Robust) Principal Component Analysis (PCA) [0, 2]. The classical formulation of PCA, ating back to Carl F. Gauss, is base on minimizing the sum of squares of the istances of all points in the ataset to the estimate linear subspace. Although this problem is non-convex, it amits a close form solution given by the span of the top eigenvectors of the ata covariance matrix. Nevertheless, it is well-known that the presence of outliers can severely affect the quality of the compute solution because the Eucliean norm is not robust to outliers. The sensitivity of classical l 2 -base PCA to outliers has been aresse by using robust maximum likelihoo covariance estimators, such as the one in [25]. However, the associate optimization problems are non-convex an thus ifficult to provie global optimality guarantees. Another classical approach is the exhaustive-search metho of Ranom Sampling An Consensus (RANSAC) [5], which given a time buget, computes at each iteration a -imensional subspace as the span of ranomly chosen points, an outputs the subspace that agrees with the largest number of points. Even though RANSAC is currently one of the most popular methos in many computer vision applications such as 32n Conference on Neural Information Processing Systems (NIPS 208), Montréal, Canaa.

2 Table : Probabilistic upper bouns for the number M of tolerate outliers as a function of the number N of inliers, the subspace imension, an the ambient imension D, by ifferent methos uner a ranom Gaussian or ranom spherical moel. Metho GGD [6] Ranom Gaussian Moel M D(D ) REAPER [3] M D D N, 2 GMS [30] l 2,-RPCA [27] TME [29] TORP [3] M M M (D )D N N N max(, log(m+n ) M < D N N max(, log(m+n) ) 2 Metho FMS [] CoP [9] DPCP (this paper) Ranom Spherical Moel N/M 0, N, i.e., any ratio of outliers when N M D 2 N, < D M N 2 D log 2 D multiple view geometry [9], its performance is sensitive to the choice of a thresholing parameter. Moreover, the number of require samplings may become prohibitive in cases when the number of outliers is very large an/or the subspace imension is large an close to the imension D of the ambient space (i.e., the high relative imension case). As an alternative to traitional robust subspace learning methos, uring the last ecae ieas from compresse sensing have given rise to a new class of methos that are base on convex optimization, an amit elegant theoretical analyses an efficient algorithmic implementations. Prominent examples are base on ecomposing the ata matrix into low-rank an column-sparse parts [27], expressing each ata point as a sparse linear combination of other ata points [20, 28], an measuring the coherence of each point with every other point in the ataset [9]. The main limitation of these methos is that they are theoretically justifiable only for subspaces of low relative imension /D. However, for applications such as 3D point clou analysis, two/three-view geometry in computer vision, an system ientification, a subspace of imension D (high relative imension) is sought [2, 26]. A promising irection towars hanling subspaces of high relative imension is minimizing the sum of the istances of the points to the subspace, which is a non-convex problem that REAPER [3] relaxes to a Semi-Definite Program (SDP). Even though in practice REAPER outperforms low-rank methos [27, 20, 28, 9] for subspaces of high relative imension, its theoretical guarantees still require < (D )/2. This is improve upon by the recent work of [6], which stuies a graient escent algorithm on the Grassmannian, an establishes convergence with high-probability to the inlier subspace for any /D, as long as the number of outliers M scales as (D/)O(N). The focus of the present paper is the recently propose Dual Principal Component Pursuit (DPCP) metho [22, 24, 23], which seeks to learn recursively a basis for the orthogonal complement of the subspace by solving an l minimization problem on the sphere. In fact, this optimization problem is precisely the unerlying non-convex problem associate to REAPER an [6] for the special case = D. As shown in [22, 24], as long as the points are well istribute in a certain eterministic sense, any global minimizer of this non-convex problem is guarantee to be a vector orthogonal to the subspace, regarless of the outlier/inlier ratio an the subspace imension; a result that agrees with the earlier finings of [4]. Inee, for synthetic ata rawn from a hyperplane ( = D ), DPCP has been shown to be the only metho able to correctly recover the subspace with up to 70% outliers (D = 30). Nevertheless, the analysis of [22, 24] involves geometric quantities that are ifficult to analyze in a probabilistic setting, an consequently it has been unclear how the number M of outliers that can be tolerate scales as a function of the number N of inliers. Moreover, even though [22, 24] show that relaxing the non-convex problem to a sequence of linear programs (LPs) guarantees finite convergence to a vector orthogonal to the subspace, this approach is computationally expensive. Alternatively, while the Iteratively Reweighte Least Squares (IRLS) scheme propose in [24, 23] is more efficient than the linear programming approach, it comes with no theoretical guarantees an scales poorly for high-imensional ata, since it involves an SVD at each iteration. In this paper we make the following specific contributions: This asymptotic result assumes that an D are fixe, thus these two parameters are omitte. 2

3 Theory: An improve analysis of global optimality for DPCP that replaces the cumbersome geometric quantities of [22, 24] with new quantities that are both tighter an easier to boun in probability. Specifically, employing a spherical ranom moel suggests that DPCP can hanle M = O( D log 2 D N 2 ) outliers. This is in sharp contrast to existing provably correct state-of-the-art robust PCA methos, which as per Table can tolerate at best M = O(N) outliers. 2 Algorithms: A scalable Projecte Sub-Graient Metho algorithm with piecewise geometrically iminishing step sizes (DPCP-PSGM), which is proven to solve the non-convex DPCP problem with linear convergence an using only matrix-vector multiplications. This is in contrast to classic results in the literature on the PSGM, which usually requires the problem to be convex in orer to establish sub-linear convergence []. DPCP-PSGM is orers of magnitue faster than the LP-base an IRLS schemes propose in [24], which allows us to exten the size of the atasets that we can hanle from 0 3 to 0 6 ata points. Experiments: Experiments on roa plane etection from 3D point clou ata using the KITTI ataset [6], which is an important computer vision task in autonomous car riving systems, show that for the same computational buget DPCP-PSGM outperforms RANSAC, which is one of the most popular methos for such computer vision applications. 2 Global Optimality Analysis for Dual Principal Component Pursuit Review of DPCP Given a unit l 2 -norm ataset X = [X O]Γ R D L, where X R D N are inlier points spanning a -imensional subspace S of R D, O are outlier points having no linear structure, an Γ is an unknown permutation, the goal of robust PCA is to recover the inlier space S or equivalently to cluster the points into inliers an outliers. Towars that en, the main iea of Dual Principal Component Pursuit (DPCP) [22, 24] is to first compute a hyperplane H that contains all the inliers X. Such a hyperplane can be use to iscar a potentially very large number of outliers, after which a metho such as RANSAC may successfully be applie to the reuce ataset 3. Alternatively, if is known, then one may procee to recover S as the intersection of D orthogonal hyperplanes that contain X. In any case, DPCP computes a normal vector b to the first hyperplane H as follows: min X b 0 s. t. b 0. () b R D Notice that the function X b 0 being minimize simply counts how many points in the ataset are not containe in the hyperplane with normal vector b. Assuming that there are at least + inliers an at least D outliers (this is to avoi egenerate situations), an that all points are in general position 4, then every solution b to () must correspon to a hyperplane that contains X, an hence b is orthogonal to S. Since () is computationally intractable, it is reasonable to replace it by 5 min f(b) := X b s. t. b 2 =. (2) b R D Although problem (2) is non-convex (because of the constraint) an non-smooth (because of the l norm), the work of [22, 24] establishe conitions suggesting that if the outliers are well istribute on the unit sphere an the inliers are well istribute on the intersection of the unit sphere with the subspace S, then global minimizers of (2) are orthogonal to S. Nevertheless, these conitions are eterministic in nature an ifficult to interpret. In this section, we give improve global optimality conitions that are i) tighter, ii) easier to interpret an iii) amenable to a probabilistic analysis. Geometry of the critical points The heart of our analysis lies in a tight geometric characterization of the critical points of (2) (see Lemma below). Before stating the result, we nee to introuce some further notation an efinitions. Letting P S be the orthogonal projection onto S, we efine the principal angle of b from S as φ [0, π 2 ] such that cos(φ) = P S(b) 2 / b 2. Since we will 2 Table is an aaptation of Table I from [2]. We note that not all bouns are irectly comparable because ifferent methos might be analyze uner ifferent moels, e.g., for DPCP we use the ranom spherical moel, while for REAPER the ranom Gaussian moel is use. Nevertheless, the two moels are closely relate, since a ranom vector istribute accoring to the stanar normal istribution tens to concentrate aroun the sphere. 3 Note that if the outliers are in general position, then H will contain at most D outliers. 4 Every -tuple of inliers is linearly inepenent, an every D-tuple of outliers is linearly inepenent. 5 This optimization problem also appears in ifferent contexts (e.g., [8] an [2]). 3

4 consier the first-orer optimality conitions of (2), we naturally nee to compute the sub-ifferential of the objective function in (2). Towars that en, we enote the sign function by sign(a) = a/ a when a 0, an sign(a) = 0 when a = 0. We also require the sub-ifferential Sgn of the absolute value function a efine as Sgn(a) = sign(a) when a 0, an Sgn(a) = [, ] when a = 0. We use sign(a) to inicate that we apply the sign function element-wise to the vector a an similarly for Sgn. Next, global minimizers of (2) are critical points in the following sense: Definition. A vector b S D is calle a critical point of (2) if there exists f(b) such that the Riemannian graient := (I bb ) = 0, where f(b) = X Sgn( X b) is the sub-ifferential of f at b. We now illustrate the key iea behin characterizing the geometry of the critical points. Let b be a critical point that is not orthogonal to S. Then, uner general position assumptions on the ata, b can be orthogonal to K D columns of X, of which at most can be inliers (otherwise b S). It follows that any Riemannian sub-graient evaluate at b has the form = (I bb )O sign(o b) + (I bb )X sign(x b) + ξ, (3) where ξ = K i= α j i x ji with x j,..., x jk the columns of X orthogonal to b an α j,..., α jk [, ]. Note that ξ 2 < D. Since b is a critical point, Definition implies a choice of α ji so that = 0. Define b = cos(φ)s + sin(φ)n, where φ is the principal angle of b from S, an s = P S (b)/ P S (b) 2 an n = P S (b)/ P S (b) 2 are the orthonormal projections of b onto S an S, respectively. Defining g = sin(φ)s + cos(φ)n an noting that g b, it follows that which in particular implies that Thus, we obtain Lemma after efining η O := M max g,b S D,g b 0 = g O sign(o b) sin(φ) X s + g ξ, (4) sin(φ) ( g O sign(o b) + D ) / X s. (5) g O sign(o b) an c X,min := N min b S S D X b. (6) Lemma. Any critical point b of (2) must either be a normal vector of S, or have a principal angle φ from S smaller than or equal to arcsin (Mη O /Nc X,min ), where η O := η O + D/M. Towars interpreting Lemma, we first give some insight into the quantities η O an c X,min. First, we claim that η O reflects how well istribute the outliers are, with smaller values corresponing to more uniform istributions. This can be seen by noting that as M an assuming that O remains well istribute, the quantity M O sign ( O b ) tens to the quantity c D b, where c D is the average height of the unit hemi-sphere of R D [22, 24]. Since g b, in the limit η O 0. Secon, the quantity c X,min is the same as the permeance statistic efine in [3], an for well-istribute inliers is boune away from small values, since there is no single irection in S sufficiently orthogonal to X. We thus see that accoring to Lemma, any critical point of (2) is either orthogonal to the inlier subspace S, or very close to S, with its principal angle φ from S being smaller for well istribute points an smaller outlier to inlier ratios M/N. Interestingly, Lemma suggests that any algorithm can be utilize to fin a normal vector to S as long as the algorithm is guarantee to fin a critical point of (2) an this critical point is sufficiently far from the subspace S, i.e., it has principal angle larger than arcsin (Mη O /Nc X,min ). We will utilize this crucial observation in the next section to erive guarantees for convergence to the global optimum for a new scalable algorithm. Global optimality In orer to characterize the global solutions of (2), we efine quantities similar to c X,min but associate with the outliers, namely c O,min := M min O b an c O,max := b S D M max O b. (7) b S D The next theorem, whose proof relies on Lemma, provies new eterministic conitions uner which any global solution to (2) must be a normal vector to S. Theorem. Any global solution b to (2) must be orthogonal to the inlier subspace S as long as M N η 2 O + (c O,max c O,min ) 2 <. (8) c X,min 4

5 Towars interpreting Theorem, recall that for well istribute inliers an outliers η O is small, while the permeance statistics c O,max, c O,min are boune away from small values. Now, the quantity c O,max, thought of as a ual permeance statistic, is boune away from large values for the reason that there is not a single irection in R D that can sufficiently capture the istribution of O. In fact, as M increases the two quantities c O,max, c O,min ten to each other an their ifference goes to zero as M. With these insights, Theorem implies that regarless of the outlier/inlier ratio M/N, as we have more an more inliers an outliers while keeping D an M/N fixe, an assuming the points are well-istribute, conition (8) will eventually be satisfie an any global minimizer must be orthogonal to the inlier subspace S. A similar conition to (8) is given in [22, Theorem 2]. Although the proofs of the two theorems share some common elements, [22, Theorem 2] is erive by establishing iscrepancy bouns between (2) an a continuous analogue of (2), an involves quantities ifficult to hanle such as spherical cap iscrepancies an circumraii of zonotopes. In aition, as shown in Figure, a numerical comparison of the conitions of the two theorems reveals that conition (8) is much tighter. We attribute this to the quantities in our new analysis better representing the function X b being minimize, namely c X,min, c O,min, c O,max, an η O, when compare to the quantities use in the analysis of [22, 24]. Moreover, our quantities are easier to boun uner a probabilistic moel, thus leaing to the following characterization of the number of outliers that may be tolerate. Theorem 2. Consier a ranom spherical moel where the columns of O are rawn uniformly from the sphere S D an the columns of X are rawn uniformly from S D S, where S is a subspace of imension < D. Fix any t < (a) Check [22, (24)] (c) Check (8) (b) Check [22, (24)] () Check (8) Figure : Check whether the conition (8) an a similar conition in [22, Theorem 2] are satisfie (white) or not (black) for a fixe number N of inliers while varying the outlier ratio M/(M + N) an the subspace relative imension /D: (a)-(c), N = 500; (b)-(), N = (c N 2). Then with probability at least 6e t 2 /2, any global solution of (2) is orthogonal to S as long as ( ) 2 ( (4 + t) 2 2 ) 2, M + C 0 D log D + t M π N (2 + t/2) N (9) where C 0 is a universal constant that is inepenent of N, M, D, an t. Interestingly, Theorem 2 suggests that DPCP can roughly tolerate M = O( D log 2 D N 2 ) outliers. We believe this makes DPCP the first metho that is able to tolerate O(N 2 ) outliers when an D are fixe, since as per Table current provably correct state-of-the-art methos can hanle at best M = O(N). For example, REAPER [3] requires M O( D N). On the other han, our boun is a ecreasing function of D, which is an artifact of the proof technique use; we conjecture that this can be mene by a more sophisticate analysis of the term η O. Finally, our choice to use a spherical ranom moel as oppose to a Gaussian moel is a technical one: the analysis is more ifficult when the functions are both non-lipschitz an unboune. That being sai, we believe that this choice oes not impose any practical limitations, since one can always normalize the ata without changing the angles of the inliers/outliers to the linear subspace. 3 A Scalable Algorithm for Dual Principal Component Pursuit Note that the DPCP problem (2) involves a convex objective function an a non-convex feasible region, which nevertheless is easy to project onto. This structure was exploite in [8, 22], where in the secon case the authors propose an Alternating Linearization an Projection (ALP) metho that solves a sequence of linear programs (LP) with a linearization of the non-convex constraint an then projection onto the sphere. 6 Although efficient LP solvers (such as Gurobi [8]) may be use to solve each LP, these methos o not scale well with the problem size (i.e., D, N an 6 Details of the proceure can be foun in the supplementary material, where we are also able to provie an improve analysis for their ALP metho. 5

6 M). Inspire by Lemma, which states that any critical point that has principal angle larger than arcsin (Mη O /Nc X,min ) must be a normal vector of S, we now consier solving (2) with a first-orer metho, specifically Projecte Sub-Graient Metho (DPCP-PSGM), which is state in Algorithm. Algorithm (DPCP-PSGM) Projecte Sub-graient Metho for Solving (2) Input: ata X R D L an initial step size µ 0 ; Initialization: set b 0 = arg min b X b 2, s. t. b S D ; : for k =, 2,... o 2: upate the step size µ k accoring to a certain rule; 3: b k = b k µ k X sign( X bk ); b k = P S D (b k ) = b k / b k ; 4: en for Unlike projecte graient escent for smooth problems, the choice of step size for PSGM is more complicate since a constant step size in general can not guarantee the convergence of PSGM even to a critical point, though such a choice is often use in practice. For the purpose of illustration, consier a simple example h(x) = x without any constraint, an suppose that µ k = 0.08 for all k an that an initialization of x 0 = 0. is use. Then, the iterates {x k } will jump between two points 0.02 an 0.06 an never converge to the global minimum 0. Thus, a wiely aopte strategy is to use iminishing step sizes, incluing those that are not summable (such as µ k = O(/k) or µ k = O(/ k)) [], or geometrically iminishing (such as µ k = O(ρ k ), ρ < ) [7, 4, 5]. However, for such choices, most of the literature establishes convergence guarantees for PSGM in the context of convex feasible regions [, 7, 4], an thus can not be irectly applie to Algorithm. For the rest of this section, it is more convenient to use the principal angle θ [0, π 2 ] between b an the orthogonal subspace S ; thus b is a normal vector of S if an only if θ = 0. We also nee a quantity similar to η O that quantifies how well the inliers are istribute within the subspace S: η X := N max g,b S S D,g b g X sign(x b). Our next result provies performance guarantees for Algorithm for various choices of step sizes ranging from constant to geometrically iminishing step sizes, the latter one giving an R-linear convergence of the sequence of principal angles to zero. Theorem 3 (Convergence guarantee for PSGM). Let { b k } be the sequence generate by Algorithm with initialization b 0, whose principal angle θ 0 to S is assume to satisfy θ 0 < arctan ( (NcX,min ) / ( NηX + DMη O ) ). (0) Let µ := 4 max{nc X,min,Mc O,max }. Assuming that Nc X,min Nη X + DMη O, the angle θ k between b k an S satisfies the following properties in accorance with various choices of step sizes. (i) (constant step size) With µ k = µ µ, k 0, we have { max{θ0, θ θ k (µ)}, k < K (µ), θ (µ), k K (µ), tan(θ 0) where K (µ) := µ(nc X,min max{,tan(θ 0)}( Nη X + DMη O )) an θ (µ) := arctan (ii) (iminishing step size) With µ k µ, µ k 0, k= µ k =, we have θ k 0. (iii) (iminishing step size of O(/k)) With µ 0 µ, µ k = µ0 () ( ) µ 2µ. k, k, we have tan(θ k) = O( k ). (iv) (piecewise geometrically iminishing step size) With µ 0 µ an { µ µ k = 0, k < K 0, µ 0 β (k K0)/K + (2), k K 0, where β (0, ), is the floor function, an K 0, K N are chosen such that K 0 K (µ 0 ) an K ( 2βµ ( Nc X,min ( Nη X + DMη O ))) (3) 6

7 with K (µ) efine right after (), we have tan(θ k ) { µ max{tan(θ0 ), 2µ 0 k < K 0, 2µ µ 0 β (k K0)/K, k K 0. (4) First note that with the choice of constant step size µ, although PSGM is not guarantee to fin a normal vector, () ensures that after K (µ) iterations, b k is close to S in the sense that θ k θ (µ), which can be much smaller than θ 0 for a sufficiently small µ. The expressions for K (µ) an θ (µ) inicate that there is a traeoff in selecting the step size µ. By choosing a larger step size µ, we have a smaller K (µ) but a larger upper boun θ (µ). We can balance this traeoff accoring to the requirements of specific applications. For example, in applications where the accuracy of θ (to zero) is not as important as the convergence spee, it is appropriate to choose a larger step size. An alternative an more efficient way to balance this traeoff is to change the step size as the iterations procee. For the classical iminishing step sizes that are not summable, Theorem 3(ii) guarantees convergence of θ k to zero (i.e., all limit points of the sequence of iterates { b k } are normal vectors), though the convergence rate epens on the specific choice of step size. For example, Theorem 3(iii) guarantees a sub-linear convergence of tan(θ k ) for step size iminishing at the rate of /k. The approach of piecewise geometrically iminishing step size (see Theorem 3(iv)) takes avantage of the traeoff in Theorem 3(i) by first using a relatively large initial step size µ 0 so that K (µ 0 ) is small (although θ (µ 0 ) is large), an then ecreasing the step size in a piecewise fashion. As illustrate in Figure 2, with such a piecewise geometrically iminishing step size, (4) establishes a piecewise geometrically ecaying boun for the principal angles. Note that the curve tan(θ k ) is not monotone because, as note earlier, PSGM is not a escent metho. Perhaps the most surprising aspect in Theorem 3(iv) is that with the iminishing step size (2), we obtain a K-step R-linear convergence rate for tan(θ k ). This linear convergence rate relies on both the choice of the step size an certain beneficial geometric structure in the problem. As characterize by Lemma, one such structure is Figure 2: Illustration of Theorem 3(iv): θ k is the principal angle between b k an S generate by the PSGM Algorithm with piecewise geometrically iminishing step size. The re otte line represents the upper boun on tan(θ k ) given by (4), while the green ashe line inicates the choice of the step size (2). that all critical points in a neighborhoo of S are global solutions. Asie from this, other properties (e.g., the negative irection of the Riemannian subgraient points towar S ) are use to show the ecaying rate of the principal angle. This is ifferent from the recent work [4] in which linear convergence for PSGM is obtaine for sharp an weakly convex objective functions an convex constraint sets. Thus, we believe the choice of piecewise geometrically iminishing step size is of inepenent interest an can be useful for other nonsmooth problems. 7 4 Experiments on Synthetic Data an Real 3D Point Clou Roa Data Synthetic Data We first use synthetic ata to verify the propose PSGM algorithm. We fix D = 30, ranomly sample a subspace S of imension = 29, an uniformly at ranom sample N = 500 inliers an M = 67 outliers (so that the outlier ratio M/(M + N) = 0.7) with unit l 2 -norm. Inspire by the Piecewise Geometrically Diminishing (PGD) step size, we also use a moifie backtracking line search (MBLS) that always uses the previous step size as an initialization for fining the current one within a backtracking line search [7, Section 3.] strategy, which ramatically reuces the computational time compare with a stanar backtracking line search. The 7 While smoothing allows one to use graient-base algorithms with guarantee convergence, the obtaine solution is a perturbe version of the targete one an thus a rouning step (such as solving a linear program [8]) is require. However, as illustrate in Figure 3, solving one linear program is more expensive than the PSGM for (2) when the ata set is relatively large, thus inicating that using a smooth surrogate is not always beneficial. 7

8 PSGM-PGD PSGM-MBLS IRLS ALP PSGM-PGD PSGM-MBLS IRLS ALP iteration Figure 3: (L) Convergence of PSGM for ifferent step sizes. Comparison of PSGM with ALP an IRLS in [24] M in terms of (M) iterations an (R) computing time. Here D = 30 an = 29, N = 500, = 0.7. M+N corresponing algorithm is enote by PSGM-MBLS. (This variant oes not have any convergence guarantee for nonsmooth problems but performe well in practice.) We set K 0 = 30, K = 4 an β = /2 for the PGD step size with initial step size obtaine by one iteration of a backtracking line search an enote the corresponing algorithm by PSGM-PGD. We efine b 0 to be the bottom eigenvector of X X, which has been emonstrate to be effective in practice [24]. Figure 3(L) isplays the convergence of the PSGM (Algorithm ) with ifferent choices of step sizes. We observe linear convergence for both PSGM-PGD an PSGM-MBLS, which converge much faster than PSGM with constant step size or classical iminishing step size. In Figure 3(M)/(R) we compare PSGM algorithms with the ALP an IRLS algorithm (referre to as DPCP-LP an DPCP-IRLS, respectively, in [24]). First observe that, as expecte, although ALP fins a normal vector in few iterations, it has the highest time complexity because it solves an LP uring each iteration. Figure 3(R) inicates that one iteration of ALP consumes more time than the whole proceure for PSGM. We also note that asie from the theoretical guarantee for PSGM-PGD, it also converges faster than IRLS (in terms of computing time), the latter lacking a convergence guarantee. Finally, Figure 4 illustrates Theorem 2, using the same setup, by showing the principal angle from S of the solution to the DPCP problem compute by the PSGM-MBLS algorithm: the phase transition is inee quaratic, inicating that DPCP can tolerate as many as O(N 2 ) outliers as preicte by Theorem Fitte 0.00 Quaratic fit Figure 4: The principal angle θ between the solution to the DPCP problem (2) an S : black correspons to π an white correspons to 0. Here D = 30 an 2 = 29. For each M, we fin the smallest N (re ots) such that θ The blue quaratic curve inicates the least-squares fit to these points. Experiments on real 3D point clou roa ata We compare DPCP-PSGM (with a moifie backtracking line search) with RANSAC [5], l 2, -RPCA [27] an REAPER [3] on the roa etection challenge 8 of the KITTI ataset [6], recore from a moving platform while riving in an aroun Karlsruhe, Germany. This ataset consists of image ata together with corresponing 3D points collecte by a rotating 3D laser scanner. In this experiment we use only the 360 3D point clous with the objective of etermining the 3D points that lie on the roa plane (inliers) an those off that plane (outliers). Typically, each 3D point clou is on the orer of 00,000 points incluing about 50% outliers. Using homogeneous coorinates this can be cast as a robust hyperplane learning problem in R 4. Since the ataset is not annotate for that purpose, we manually annotate a few frames (e.g., see the left column of Fig. 5). Since DPCP-PSGM is the fastest metho (on average converging in about 00 millisecons for each frame on a 6 core 6 threa Intel (R) i machine), we set the time buget for all methos equal to the running time of DPCP-PSGM. For RANSAC we also compare with 0 an 00 times that time buget. Since l 2, -RPCA oes not irectly return a subspace moel, we extract the normal vector via SVD on the low-rank matrix returne by that metho. Table 2 reports the area uner the Receiver Operator Curve (ROC), the latter obtaine by thresholing the istances of the points to the hyperplane estimate by each metho, using a suitable range of ifferent threshols 9. As seen, even though a low-rank metho, l 2, -RPCA performs reasonably well but not on par with DPCP-PSGM an REAPER, which overall 8 Coherence Pursuit [9] is not applicable to this experiment because forming the require correlation matrix of the thousans of 3D points is prohibitively expensive. 9 For RANSAC, we also use each such threshol as its internal thresholing parameter. 8

9 Figure 5: Frame 2 of ataset KITTI-CITY-48: raw image, projection of annotate 3D point clou onto the image, an etecte inliers/outliers using a groun-truth threshol on the istance to the hyperplane for each metho. The corresponing F measures are DPCP-PSGM (0.933), REAPER (0.890), `2 -RPCA (0.248), RANSAC (0.023), 0xRANSAC (0.622), an 00xRANSAC (0.824). ten to be the most robust methos. On the contrary, for the same time buget, RANSAC, which is a popular choice in the computer vision community for such outlier etection tasks, is essentially failing ue to an insufficient number of iterations. Even allowing for a 00 times higher time buget still oes not make RANSAC the best metho, as it is outperforme by DPCP-PSGM on five out of the seven point clous (, 45, an 37 in KITTY-CITY-5, an 0 an 2 in KITTY-CITY-48). Table 2: Area uner ROC for annotate 3D point clous with inex, 45, 20, 37, 53 in KITTYCITY-5 an 0, 2 in KITTY-CITY-48. The number in parenthesis is the percentage of outliers. KITTY-CITY-5 KITTY-CITY-48 Methos (37%) 45(38%) 20(53%) 37(48%) 53(67%) 0(56%) 2(57%) DPCP-PSGM REAPER `2,-RPCA RANSAC xRANSAC xRANSAC Conclusions We provie an improve analysis for the global optimality of the DPCP metho that suggests that DPCP can hanle O((#inliers)2 ) outliers. We also presente a scalable first-orer metho for solving the DPCP problem that only uses matrix-vector multiplications, for which we establishe global convergence guarantees for various step size selection schemes, regarless of the non-convexity an non-smoothness of the DPCP problem. Finally, experiments on 3D point clou roa ata emonstrate that the propose metho is able to outperform RANSAC even when RANSAC is allowe to use 00 times the computational buget of the propose metho. Extensions to allow for corrupte ata an multiple subspaces, an further applications in computer vision are the subject of ongoing work. Acknowlegment The co-authors from JHU were supporte by NSF grant We thank Tianyu Ding of JHU for carefully proof-reaing the longer version of this manuscript an catching some mistakes, Yunchen Yang an Tianjiao Ding of ShanghaiTech for refining the 3D point clou experiment, Ziyu Liu of ShanghaiTech for his help in eriving the concentration inequality for ηo, an the three anonymous reviewers for their constructive comments. 9

10 References [] S. Boy, L. Xiao, an A. Mutapcic. Subgraient methos. Lecture Notes of EE392o, Stanfor University, Autumn Quarter, 2004: , [2] E. Canès, X. Li, Y. Ma, an J. Wright. Robust principal component analysis. Journal of the ACM, 58, 20. [3] Y. Cherapanamjeri, P. Jain, an P. Netrapalli. Thresholing base efficient outlier robust pca. arxiv preprint arxiv: , 207. [4] D. Davis, D. Drusvyatskiy, K. J. MacPhee, an C. Paquette. Subgraient methos for sharp weakly convex functions. arxiv preprint arxiv: , 208. [5] M. A. Fischler an R. C. Bolles. RANSAC ranom sample consensus: A paraigm for moel fitting with applications to image analysis an automate cartography. Communications of the ACM, 26:38 395, 98. [6] A. Geiger, P. Lenz, C. Stiller, an R. Urtasun. Vision meets robotics: The kitti ataset. The International Journal of Robotics Research, 32():23 237, 203. [7] J.-L. Goffin. On convergence rates of subgraient optimization methos. Mathematical Programming, 3(): , 977. [8] I. Gurobi Optimization. Gurobi optimizer reference manual, 205. [9] R. Hartley an A. Zisserman. Multiple View Geometry in Computer Vision. Cambrige, 2n eition, [0] I. Jolliffe. Principal Component Analysis. Springer-Verlag, New York, 986. [] G. Lerman an T. Maunu. Fast, robust an non-convex subspace recovery. Information an Inference: A Journal of the IMA, 7(2): , 207. [2] G. Lerman an T. Maunu. An overview of robust subspace recovery. arxiv: [cs.lg], 208. [3] G. Lerman, M. B. McCoy, J. A. Tropp, an T. Zhang. Robust computation of linear moels by convex relaxation. Founations of Computational Mathematics, 5(2):363 40, 205. [4] G. Lerman an T. Zhang. l p-recovery of the most significant subspace among multiple subspaces with outliers. Constructive Approximation, 40(3): , 204. [5] X. Li, Z. Zhu, A. M.-C. So, an R. Vial. Nonconvex robust low-rank matrix recovery. arxiv preprint arxiv: , 208. [6] T. Maunu an G. Lerman. A well-tempere lanscape for non-convex robust subspace recovery. arxiv: [cs.lg], 207. [7] J. Noceal an S. J. Wright. Numerical Optimization, secon eition. Worl Scientific, [8] Q. Qu, J. Sun, an J. Wright. Fining a sparse vector in a subspace: Linear sparsity using alternating irections. In Avances in Neural Information Processing Systems, pages , 204. [9] M. Rahmani an G. Atia. Coherence pursuit: Fast, simple, an robust principal component analysis. arxiv preprint arxiv: , 206. [20] M. Soltanolkotabi, E. Elhamifar, an E. J. Canès. Robust subspace clustering. Annals of Statistics, 42(2): , 204. [2] H. Späth an G. Watson. On orthogonal linear l approximation. Numerische Mathematik, 5(5):53 543, 987. [22] M. Tsakiris an R. Vial. Dual principal component pursuit. In ICCV Workshop on Robust Subspace Learning an Computer Vision, pages 0 8, 205. [23] M. C. Tsakiris an R. Vial. Hyperplane clustering via ual principal component pursuit. In International Conference on Machine Learning, 207. [24] M. C. Tsakiris an R. Vial. Dual principal component pursuit. Journal of Machine Learning Research, 9(8): 50,

11 [25] D. E. Tyler. A istribution-free m-estimator of multivariate scatter. The Annals of Statistics, 5():234 25, 987. [26] Y. Wang, C. Dicle, M. Sznaier, an O. Camps. Self scale regularize robust regression. In Proceeings of the IEEE Conference on Computer Vision an Pattern Recognition, pages , 205. [27] H. Xu, C. Caramanis, an S. Sanghavi. Robust pca via outlier pursuit. In Avances in Neural Information Processing Systems, pages , 200. [28] C. You, D. Robinson, an R. Vial. Provable self-representation base outlier etection in a union of subspaces. In IEEE Conference on Computer Vision an Pattern Recognition, 207. [29] T. Zhang. Robust subspace recovery by tyler s m-estimator. Information an Inference: A Journal of the IMA, 5(): 2, 206. [30] T. Zhang an G. Lerman. A novel m-estimator for robust pca. The Journal of Machine Learning Research, 5(): , 204.