Minimum norm points on polytope boundaries

Transcription

1 Minimum norm points on polytope boundaries David Bremner UNB December 20, 2013 Joint work with Yan Cui, Zhan Gao David Bremner (UNB) Minimum norm facet December 20, / 27

2 Outline 1 Notation 2 Maximum margin classifiers 3 Enumerative Algorithms 4 Cutting Plane Approaches David Bremner (UNB) Minimum norm facet December 20, / 27

3 Notation Support functionals Linear Programs with varying objectives x, y = x T y = y T x For K R d, w R d, sup(k; w) := sup w, k (upper) k K inf(k; w) := inf w, k (lower) k K = sup(k; w) Let polyhedron K = { Ax b } sup(k; c) = max c T x Ax b David Bremner (UNB) Minimum norm facet December 20, / 27

4 Notation Support functionals Linear Programs with varying objectives s u p ( K ; w ) in f ( K ; w ) x, y = x T y = y T x K For K R d, w R d, sup(k; w) := sup w, k (upper) k K inf(k; w) := inf w, k (lower) k K = sup(k; w) w Let polyhedron K = { Ax b } sup(k; c) = max c T x Ax b David Bremner (UNB) Minimum norm facet December 20, / 27

5 Notation Support functionals Linear Programs with varying objectives s u p ( K ; w ) in f ( K ; w ) x, y = x T y = y T x K For K R d, w R d, sup(k; w) := sup w, k (upper) k K inf(k; w) := inf w, k (lower) k K = sup(k; w) w Let polyhedron K = { Ax b } sup(k; c) = max c T x Ax b David Bremner (UNB) Minimum norm facet December 20, / 27

6 Notation Outer Normal Cone Geometric LP Duality For K R d, p K, the outer normal cone N(K; p) := { y y, p = sup(k; y) } K p N(K,p) N(K; p) is the set of all objective functions optimal at x David Bremner (UNB) Minimum norm facet December 20, / 27

7 Notation Outer Normal Cone Geometric LP Duality For K R d, p K, the outer normal cone N(K; p) := { y y, p = sup(k; y) } K p N(K,p) N(K; p) is the set of all objective functions optimal at x David Bremner (UNB) Minimum norm facet December 20, / 27

8 Notation Polarity P = conv { y x P, y, x 1 } David Bremner (UNB) Minimum norm facet December 20, / 27

9 Notation Minkowski Norms x B = inf { γ 0 x γb } (1) = sup(b ; x) (2) David Bremner (UNB) Minimum norm facet December 20, / 27

10 Notation Minkowski Norms x B = inf { γ 0 x γb } (1) = sup(b ; x) (2) B 2 B B 1 = B 2 = x1 2 + x 2 2 = 1 = x 1 + x 2 max( x 1, x 2 ) David Bremner (UNB) Minimum norm facet December 20, / 27

11 Notation Minkowski Norms x B = inf { γ 0 x γb } (1) = sup(b ; x) (2) B 2 B B 1 = B 2 = x1 2 + x 2 2 = 1 = x 1 + x 2 max( x 1, x 2 ) S = bd B S = bd(b ) David Bremner (UNB) Minimum norm facet December 20, / 27

12 Maximum margin classifiers Margin Separable P, Q R d are separable if z 0 such that sup(q; z) inf(p; z). P γz Q Margin max γ such that z S sup(q; z) + γ inf(p; z) For the Euclidean norm z S just means z is a unit vector. Both that constraint and z 0 are non-convex David Bremner (UNB) Minimum norm facet December 20, / 27

13 Maximum margin classifiers Margin Separable P, Q R d are separable if z 0 such that sup(q; z) inf(p; z). γz P Q Margin max γ such that z S sup(q; z) + γ inf(p; z) For the Euclidean norm z S just means z is a unit vector. Both that constraint and z 0 are non-convex David Bremner (UNB) Minimum norm facet December 20, / 27

14 Maximum margin classifiers Margin Separable P, Q R d are separable if z 0 such that sup(q; z) inf(p; z). γz P Q Margin max γ such that z S sup(q; z) + γ inf(p; z) For the Euclidean norm z S just means z is a unit vector. Both that constraint and z 0 are non-convex David Bremner (UNB) Minimum norm facet December 20, / 27

15 Maximum margin classifiers Margin Separable P, Q R d are separable if z 0 such that sup(q; z) inf(p; z). γz P Q Margin max γ such that z S sup(q; z) + γ inf(p; z) For the Euclidean norm z S just means z is a unit vector. Both that constraint and z 0 are non-convex David Bremner (UNB) Minimum norm facet December 20, / 27

16 Maximum margin classifiers Margin Separable P, Q R d are separable if z 0 such that sup(q; z) inf(p; z). γz P Q Margin max γ such that z S sup(q; z) + γ inf(p; z) For the Euclidean norm z S just means z is a unit vector. Both that constraint and z 0 are non-convex David Bremner (UNB) Minimum norm facet December 20, / 27

17 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

18 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

19 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

20 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

21 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

22 Maximum margin classifiers A simplified convex approximation min w 2 + Cγ s.t. w, p q 1 γ p P, q Q γ 0 Need C suff. large to force γ < 1 at opt. Given optimal (w, γ < 1), ρ > 1, define w = ρw γ = 1 ρ(1 γ) If γ < 1 γ γ = (ρ 1)(1 γ) > 0 Therefore w Cγ w + Cγ For large enough C, we can scale w up without making the objective worse. David Bremner (UNB) Minimum norm facet December 20, / 27

23 Maximum margin classifiers Translate until separable Minimize Translation Let P and Q convex bodies. Q + t q + t inf t sup(p; z) inf(q + t; z) z 0, q Q, q + t P q t P Q z 0, inf r z N(P Q; r) r P Q Equivalent Formulation shift(p, Q) := min { t : t bd(p Q) } David Bremner (UNB) Minimum norm facet December 20, / 27

24 Maximum margin classifiers Translate until separable Minimize Translation Let P and Q convex bodies. Q + t q + t inf t sup(p; z) inf(q + t; z) z 0, q Q, q + t P q t P Q z 0, inf r z N(P Q; r) r P Q Equivalent Formulation shift(p, Q) := min { t : t bd(p Q) } David Bremner (UNB) Minimum norm facet December 20, / 27

25 Maximum margin classifiers Duality N(P Q; k) Proposition If 0 P Q, N(B, w) w z w S shift(p, Q) = margin(p, Q) Choose w S, z N(B ; w) bd(p Q) 0 S sup(p Q, w) z, w = z inf k k bd P Q If k bd P Q, w N(P Q; k) S k w, k = sup(p Q; w) David Bremner (UNB) Minimum norm facet December 20, / 27

26 Maximum margin classifiers Duality N(P Q; k) Proposition If 0 P Q, N(B, w) w z w S shift(p, Q) = margin(p, Q) Choose w S, z N(B ; w) bd(p Q) 0 S sup(p Q, w) z, w = z inf k k bd P Q If k bd P Q, w N(P Q; k) S k w, k = sup(p Q; w) David Bremner (UNB) Minimum norm facet December 20, / 27

27 Maximum margin classifiers Duality N(P Q; k) Proposition If 0 P Q, N(B, w) w z w S shift(p, Q) = margin(p, Q) Choose w S, z N(B ; w) bd(p Q) 0 S sup(p Q, w) z, w = z inf k k bd P Q If k bd P Q, w N(P Q; k) S k w, k = sup(p Q; w) David Bremner (UNB) Minimum norm facet December 20, / 27

28 Maximum margin classifiers Convex Maximization For convex polyhedron K inf r = min min r r bd K F facet of K r F = min min r F facet of K r aff F x y z min { x a T x = 1 } = 1 a Assuming 0 int K inf r = max v r bd K v vertex of K = max x x K Theorem (Brieden 2002) For 2 p < max x p p Ax b cannot be computed in polynomial time within a factor of 1.090, unless P=NP. David Bremner (UNB) Minimum norm facet December 20, / 27

29 Maximum margin classifiers Convex Maximization For convex polyhedron K inf r = min min r r bd K F facet of K r F = min min r F facet of K r aff F x y z min { x a T x = 1 } = 1 a Assuming 0 int K inf r = max v r bd K v vertex of K = max x x K Theorem (Brieden 2002) For 2 p < max x p p Ax b cannot be computed in polynomial time within a factor of 1.090, unless P=NP. David Bremner (UNB) Minimum norm facet December 20, / 27

30 Maximum margin classifiers Convex Maximization For convex polyhedron K inf r = min min r r bd K F facet of K r F = min min r F facet of K r aff F x y z min { x a T x = 1 } = 1 a Assuming 0 int K inf r = max v r bd K v vertex of K = max x x K Theorem (Brieden 2002) For 2 p < max x p p Ax b cannot be computed in polynomial time within a factor of 1.090, unless P=NP. David Bremner (UNB) Minimum norm facet December 20, / 27

31 Maximum margin classifiers Convex Maximization For convex polyhedron K inf r = min min r r bd K F facet of K r F = min min r F facet of K r aff F x y z min { x a T x = 1 } = 1 a Assuming 0 int K inf r = max v r bd K v vertex of K = max x x K Theorem (Brieden 2002) For 2 p < max x p p Ax b cannot be computed in polynomial time within a factor of 1.090, unless P=NP. David Bremner (UNB) Minimum norm facet December 20, / 27

32 Maximum margin classifiers Hardness breadth w (X ) := sup w, x y = sup(x X ; w) x,y X width(x ) := inf w S breadth w (X ) Proposition width(x ) = margin(x, X ) Proposition (Gritzmann Klee 1992) Computing the width is NP-hard. David Bremner (UNB) Minimum norm facet December 20, / 27

33 Maximum margin classifiers Hardness breadth w (X ) := sup w, x y = sup(x X ; w) x,y X width(x ) := inf w S breadth w (X ) Proposition width(x ) = margin(x, X ) Proposition (Gritzmann Klee 1992) Computing the width is NP-hard. David Bremner (UNB) Minimum norm facet December 20, / 27

34 Enumerative Algorithms Brute Force Solution function MinNormFacet(P, Q R d ) λ = + R = ExtremePoints(P Q) for all facets F of conv R do λ = min(minnormaff(f ), λ) end for end function David Bremner (UNB) Minimum norm facet December 20, / 27

35 Enumerative Algorithms Brute Force Solution x y z function MinNormFacet(P, Q R d ) λ = + R = ExtremePoints(P Q) for all facets F of conv R do λ = min(minnormaff(f ), λ) end for end function David Bremner (UNB) Minimum norm facet December 20, / 27

36 Enumerative Algorithms Enumerating Facets with Reverse Search Facets of P Q are computed without extra memory the number of facets and bases grow quickly. Adj C Adj f() Adj X Y f(adj(x, j)) = X David Bremner (UNB) Minimum norm facet December 20, / 27

37 Enumerative Algorithms Enumerating Facets with Reverse Search bases per vertex Facets of P Q are computed without extra memory the number of facets and bases grow quickly dimension Adj Y X C Adj Adj f() f(adj(x, j)) = X David Bremner (UNB) Minimum norm facet December 20, / 27

38 Enumerative Algorithms Classification Accuracy David Bremner (UNB) Minimum norm facet December 20, / 27

39 Enumerative Algorithms Maximum Error Test Set David Bremner (UNB) Minimum norm facet December 20, / 27

40 Cutting Plane Approaches Eccentricity r Q R P P Q Most of the volume is far away from the interesting part Hopefully most of the combinatorial complexity is also far away. David Bremner (UNB) Minimum norm facet December 20, / 27

41 Cutting Plane Approaches Eccentricity r Q R P P Q Most of the volume is far away from the interesting part Hopefully most of the combinatorial complexity is also far away. David Bremner (UNB) Minimum norm facet December 20, / 27

42 Cutting Plane Approaches Cutting Planes Cut Let x = argmin x bd P Q x. For any λ x, H tangent to λb preserves bd(p Q) λb x, H x Double description style update (Motzkin) Let E H be the edges of conv V that properly intersect H. The vertex set of conv(v ) H is V H e E H (e H) David Bremner (UNB) Minimum norm facet December 20, / 27

43 Cutting Plane Approaches Cutting Planes Cut Let x = argmin x bd P Q x. For any λ x, H tangent to λb preserves bd(p Q) λb x, H x Double description style update (Motzkin) Let E H be the edges of conv V that properly intersect H. The vertex set of conv(v ) H is V H e E H (e H) David Bremner (UNB) Minimum norm facet December 20, / 27

44 Cutting Plane Approaches Heuristics Adding a cutting plane is (usually) only effective if it cuts off many more vertices than it creates. Experimentally, the farthest vertex seems to be a good choice of cutting plane normal, at least for random-ish data. x H David Bremner (UNB) Minimum norm facet December 20, / 27

45 Cutting Plane Approaches Heuristics Adding a cutting plane is (usually) only effective if it cuts off many more vertices than it creates. Experimentally, the farthest vertex seems to be a good choice of cutting plane normal, at least for random-ish data. x H David Bremner (UNB) Minimum norm facet December 20, / 27

46 Cutting Plane Approaches Cutting Polytopes In some sense what we want is polytopal approximation of the sphere. Find polytope with small H and V complexity containing sphere. Using (approximately) regular simplex round carefully doesn t seem to iterate that well. David Bremner (UNB) Minimum norm facet December 20, / 27

47 Cutting Plane Approaches Cutting Polytopes In some sense what we want is polytopal approximation of the sphere. Find polytope with small H and V complexity containing sphere. Using (approximately) regular simplex round carefully doesn t seem to iterate that well. David Bremner (UNB) Minimum norm facet December 20, / 27

48 Cutting Plane Approaches Cutting Polytopes In some sense what we want is polytopal approximation of the sphere. Find polytope with small H and V complexity containing sphere. Using (approximately) regular simplex round carefully doesn t seem to iterate that well. David Bremner (UNB) Minimum norm facet December 20, / 27

49 Cutting Plane Approaches Cutting Polytopes In some sense what we want is polytopal approximation of the sphere. Find polytope with small H and V complexity containing sphere. Using (approximately) regular simplex round carefully doesn t seem to iterate that well. David Bremner (UNB) Minimum norm facet December 20, / 27

50 Cutting Plane Approaches Cutting Polytopes In some sense what we want is polytopal approximation of the sphere. Find polytope with small H and V complexity containing sphere. Using (approximately) regular simplex round carefully doesn t seem to iterate that well. David Bremner (UNB) Minimum norm facet December 20, / 27

51 Cutting Plane Approaches Performance in R 9 David Bremner (UNB) Minimum norm facet December 20, / 27

52 Cutting Plane Approaches Average runtime versus dimension David Bremner (UNB) Minimum norm facet December 20, / 27

53 Cutting Plane Approaches Cutting polytope method ⁵ ⁵ time (s) d=10 d=8 d=6 6e-5 x^ e-4 x^ e-5 x^ vertices David Bremner (UNB) Minimum norm facet December 20, / 27

54 Cutting Plane Approaches Conclusions Minimum norm boundary point is dual to maximum margin classifying hyperplane NP-Hard Solvable via (partial) enumerative approaches. Cutting planes/polytopes yield some speedup, but plenty of room for improvement. David Bremner (UNB) Minimum norm facet December 20, / 27

55 Cutting Plane Approaches Acknowledgements Geometric background: with Peter Gritzmann Software: lrslib (Avis), cddlib (Fukuda), gmp, Minkowski sum (Weibel) David Bremner (UNB) Minimum norm facet December 20, / 27

56 Cutting Plane Approaches Filtering redundant points of P Q u 1 + s 2 s 1 t 1 s 2 u 1 Proposition (Fukuda) Let m be the max degree of a vertex of conv(x i ). The extreme points of X 1 X 2 can be computed with O(m f 0 (X 1 X 2 ) + f0 2(X 1) + f0 2(X 2)) LPs with m constraints. s 1 + t 1 David Bremner (UNB) Minimum norm facet December 20, / 27

57 Cutting Plane Approaches Vertices of the unit ball λ 0 v 0 function MinNormCut(P, Q, B R d ) λ = + for all vertices v of B do F SepFacet(P Q, λv) if F then λ = min(minnormaff(f, λ)) end if end for end function λ 1 v 0 λ 2 v 2 λ 1 v 1 David Bremner (UNB) Minimum norm facet December 20, / 27