Linear Programming and Clustering

Transcription

1 and Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

2 Outline of Talk 1 Introduction 2 Motivation 3 Our Approach 4 A possible counter-example 1 Introduction 2 Observations Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

3 Introduction Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

4 Introduction Linear programming (LP) is a mathematical method for determining a way to achieve the most suitable outcome (such as maximum profit or lowest cost) in a given mathematical model for some list of requirements represented as linear relationships Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

5 Introduction Linear programming (LP) is a mathematical method for determining a way to achieve the most suitable outcome (such as maximum profit or lowest cost) in a given mathematical model for some list of requirements represented as linear relationships Linear programs are problems that can be expressed in canonical form: maximize c.x subject to Ax b where x represents the vector of variables (to be determined), c R n and b R m are vectors of (known) coefficients and A R m n is a (known) matrix of coefficients. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

6 Introduction Strongly Polynomial Time Algorithm: An algorithm is strongly polynomial if:- Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

7 Introduction Strongly Polynomial Time Algorithm: An algorithm is strongly polynomial if:- 1 it consists of the elementary arithmetic operations: addition, comparison, multiplication and division Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

8 Introduction Strongly Polynomial Time Algorithm: An algorithm is strongly polynomial if:- 1 it consists of the elementary arithmetic operations: addition, comparison, multiplication and division 2 the number of such steps is polynomially bounded in the dimension of the input Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

9 Introduction Strongly Polynomial Time Algorithm: An algorithm is strongly polynomial if:- 1 it consists of the elementary arithmetic operations: addition, comparison, multiplication and division 2 the number of such steps is polynomially bounded in the dimension of the input 3 when the algorithm is applied to rational input, then the size of the numbers during the algorithm is polynomially bounded in the dimension of the input and the size of the input numbers. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

10 Introduction Can we solve a standard instance of LP using at most f (m, n) arithmetic operations with f being a bounded-degree polynomial with no dependence on the description of A, b, c? Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

11 Motivation Outline of Talk Simplex Approach and Interior Points Algorithms Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

12 Motivation Outline of Talk Simplex Approach and Interior Points Algorithms Existing algorithms work in polynomial time but not strongly polynomial. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

13 Motivation Outline of Talk Simplex Approach and Interior Points Algorithms Existing algorithms work in polynomial time but not strongly polynomial. It has been an open question for a long time. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

14 Motivation Outline of Talk Simplex Approach and Interior Points Algorithms Existing algorithms work in polynomial time but not strongly polynomial. It has been an open question for a long time. Recently, Vempala-Barasz proposed an approach to strongly polynomial linear programming. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

15 The Affine-Invariant Algorithm Input : Polyhedron P given by linear inequalities {a j.x b j : j = 1...m}, objective vector c and a vertex z. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

16 The Affine-Invariant Algorithm Input : Polyhedron P given by linear inequalities {a j.x b j : j = 1...m}, objective vector c and a vertex z. Output : A vertex maximizing the objective value, or unbounded if the LP is unbounded. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

17 The Affine-Invariant Algorithm while The current vertex z is not optimal do H= the set of indices of active inequalities at z. For every t H, compute a vector v t : a h.v t = 0 for h H \ t and a t.v t < 0. T = {t H : c.v t 0} and S = H \ T. while T φ do Perform step 2 Return the current vertex. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

18 Step 2 Outline of Talk (a) For every t T, compute a vector v t 0 : a h.v t = 0 for h H \ {t}, c.v t 0 and the length of v t is the largest value for which z + v t is feasible. (b) Compute a non-negative combination v of {v t : t T }. (c) Let λ be maximal for which z + λv P, if no such maximum exists, return unbounded. z := z + λv (d) Let s be the index of an inequality which becomes active. Let t T be any index such that {a h : h {s} S T \ {t}} is linearly independent. Set S := S {s}, T := T \ {t} and H := S T. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

19 The Affine-Invariant Algorithm Note : If we are able to show a polynomial bound on the Vempala-Barasz algorithm, it is a strongly polynomial algorithm. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

20 The Affine-Invariant Algorithm Note : If we are able to show a polynomial bound on the Vempala-Barasz algorithm, it is a strongly polynomial algorithm. We can prove that the method of choosing coefficients does not affect the analysis of the algorithm in Klee-Minty case. We can, in fact, fix the coefficients rather than using random or centroid method. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

21 The Affine-Invariant Algorithm Note : If we are able to show a polynomial bound on the Vempala-Barasz algorithm, it is a strongly polynomial algorithm. We can prove that the method of choosing coefficients does not affect the analysis of the algorithm in Klee-Minty case. We can, in fact, fix the coefficients rather than using random or centroid method. The only necessary condition is that all the coefficients should be non-negative. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

22 Possibilities Outline of Talk 1 The algorithm works in strongly polynomial time for all the LP cases, no matter how we combine the improving rays(of course using non-negative coefficients). Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

23 Possibilities Outline of Talk 1 The algorithm works in strongly polynomial time for all the LP cases, no matter how we combine the improving rays(of course using non-negative coefficients). 2 The algorithm works in strongly polynomial time for all LP cases, but only when we combine the improving rays in a particular fashion. So, if possible an adversary can come up with choices at each step to ensure the algorithm takes exponential many steps. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

24 Possibilities Outline of Talk 1 The algorithm works in strongly polynomial time for all the LP cases, no matter how we combine the improving rays(of course using non-negative coefficients). 2 The algorithm works in strongly polynomial time for all LP cases, but only when we combine the improving rays in a particular fashion. So, if possible an adversary can come up with choices at each step to ensure the algorithm takes exponential many steps. 3 There may exist an LP problem for which the algorithm does not work in polynomial time. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

25 A Possible Counter-example In Klee-Minty or Goldfarb-Sit cases, the algorithm visits a facet of the polytope only once making the algorithm run in strongly polynomial time. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

26 A Possible Counter-example In Klee-Minty or Goldfarb-Sit cases, the algorithm visits a facet of the polytope only once making the algorithm run in strongly polynomial time. We attempt to construct a counter-example in which the algorithm visits some of the facets more than once. The top (left figure) and side (right) views of a polytope are presented in the figures in the next slide. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

27 A Possible Counter-example In Klee-Minty or Goldfarb-Sit cases, the algorithm visits a facet of the polytope only once making the algorithm run in strongly polynomial time. We attempt to construct a counter-example in which the algorithm visits some of the facets more than once. The top (left figure) and side (right) views of a polytope are presented in the figures in the next slide. In each step of the algorithm, we choose to move along only one of the possible improving rays, rather than taking any combination of all of them. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

28 A Possible Counter-example f X Objective vector e W 7 d 6 5 c 1 Z 4 b 2 3 a Y 0 X e 7 d 1 Z 2 5 W f 6 c U 4 b a 0 Y 3 Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

29 A Possible Counter-example Let the starting point be a. We have H = {X, Y, Z}. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

30 A Possible Counter-example Let the starting point be a. We have H = {X, Y, Z}. Running step 1 of the algorithm, we get T = {X, Y }, S = {Z}. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

31 A Possible Counter-example Let the starting point be a. We have H = {X, Y, Z}. Running step 1 of the algorithm, we get T = {X, Y }, S = {Z}. In the first iteration of step 2, we deterministically choose to move along 2 to reach vertex b. Now, the modified sets are :- T = {Y }, S = {U, Z} and H = T S. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

32 A Possible Counter-example In the next iteration of step 2, we deterministically choose to move along 4 to reach c. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

33 A Possible Counter-example In the next iteration of step 2, we deterministically choose to move along 4 to reach c. Now, T becomes empty and we have reached the next vertex c. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

34 A Possible Counter-example In the next iteration of step 2, we deterministically choose to move along 4 to reach c. Now, T becomes empty and we have reached the next vertex c. We repeat similar steps from c also to move along 5 first and then 7 to reach e as the next vertex. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

35 A Possible Counter-example In the next iteration of step 2, we deterministically choose to move along 4 to reach c. Now, T becomes empty and we have reached the next vertex c. We repeat similar steps from c also to move along 5 first and then 7 to reach e as the next vertex. Thus, we visit front facet and back facet alternately (more than once) in the algorithm. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

36 Introduction In statistics and data mining, k-means clustering is a method of clustering, which aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

37 Introduction In statistics and data mining, k-means clustering is a method of clustering, which aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean. An optimum clustering is the one which has the minimum cost of clustering. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

38 Introduction In statistics and data mining, k-means clustering is a method of clustering, which aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean. An optimum clustering is the one which has the minimum cost of clustering. Cost of clustering is defined as Σ C S Σ x C x ctr(c) 2 where ctr(c) is the center of cluster C and S is the set of all cluster-centers. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

39 Our Approach Frequently used method to achieve this, is to start with some random k initial centers (called seeds) and use Lloyd s iterations over these clusters to move the centers to decrease the cost of clustering. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

40 Our Approach Frequently used method to achieve this, is to start with some random k initial centers (called seeds) and use Lloyd s iterations over these clusters to move the centers to decrease the cost of clustering. We review a paper The effectiveness of Lloyd-type Methods for the k-means Problem, to study a novel probabilistic seeding process for the starting configuration of a Lloyd-type iteration. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

41 Our Approach Frequently used method to achieve this, is to start with some random k initial centers (called seeds) and use Lloyd s iterations over these clusters to move the centers to decrease the cost of clustering. We review a paper The effectiveness of Lloyd-type Methods for the k-means Problem, to study a novel probabilistic seeding process for the starting configuration of a Lloyd-type iteration. We implement the seeding algorithm described in the paper and compare it with the standard seeding methods. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

42 Our Approach We use the builtin program in R (Project for Statistical Computing) as the standard to compare our result against. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

43 Our Approach We use the builtin program in R (Project for Statistical Computing) as the standard to compare our result against. We particularly focus on the data sets whose clustering is verifiable e.g. Iris data set. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

44 Our Approach We use the builtin program in R (Project for Statistical Computing) as the standard to compare our result against. We particularly focus on the data sets whose clustering is verifiable e.g. Iris data set. We compare the clustering obtained by us against the one already provided and observe that almost 95% of the data-items are classified correctly using just the seeds without any Lloyd s iterations. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

45 Observations Data set Iris data set Value of k 5 3 Cost with built-in program Cost with just the seeds as centers Cost with built-in-with-seeds Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

46 Observations Cloud-1 data set Value of k Cost with built-in program Cost with just the seeds as centers Cost with built-in-with-seeds Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

47 Observations Glass data set Value of k 6 Cost with built-in program Cost with just the seeds as centers Cost with built-in-with-seeds Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

48 Conclusion Outline of Talk We rule out the technique used by Vempala-Barasz to prove polynomial bound on their algorithm for a general LP problem. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

49 Conclusion Outline of Talk We rule out the technique used by Vempala-Barasz to prove polynomial bound on their algorithm for a general LP problem. Based on our study of clustering, we can conclude that Lloyd s iterations might not be needed if the initial seeds are good enough, which is the case in most of the examples we studied Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

50 Open Problems Strongly Polynomial Time algorithm for LP is still an open problem. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

51 Open Problems Strongly Polynomial Time algorithm for LP is still an open problem. If we use efficient seeding, can we get rid of Lloyd s iterations or can we do it with just one Lloyd s iteration? Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

52 Outline of Talk 1 A New Approach to Strongly Polynomial, Mihaly Barasz and Santosh Vempala 2 A strongly polynomial algorithm to solve combinatorial linear programs, Éva Tardos 3, Howard Karloff 4, Va sek Chvátal 5 Wiki-page for 6 The effectiveness of Lloyd-type Methods for the k-means Problem, Rafail Ostrovsky, Yuval Rabani, Leonard J. Schulman, Chaitanya Swamy. 7 Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and

53 I would like to thank Dr. Leonard Schulman for his valuable guidance. I would also like to thank Dr. Chris Umans for his guidance on the multivariate polynomial interpolation problem. I would like to thank SURF committee for giving me this opportunity to present my work. Advisor: Dr. Leonard Schulman, Caltech Aditya Huddedar IIT Kanpur and