# On the Complexity of the Policy Improvement Algorithm. for Markov Decision Processes

Save this PDF as:

Size: px
Start display at page:

Download "On the Complexity of the Policy Improvement Algorithm. for Markov Decision Processes"

## Transcription

1 On the Complexity of the Policy Improvement Algorithm for Markov Decision Processes Mary Melekopoglou Anne Condon Computer Sciences Department University of Wisconsin - Madison 0 West Dayton Street Madison, WI ABSTRACT We consider the complexity of the policy improvement algorithm for Markov decision processes. We show that four variants of the algorithm require exponential time in the worst case.. Introduction Finding an optimal policy in a Markov decision process is a classical problem in optimization theory. Although the problem is solvable in polynomial time using linear programming (Howard [4], Khachian [7]), in practice, the policy improvement algorithm is often used. We show that four natural variants of this algorithm require exponential time in the worst case. A stationary Markov decision process consists of a finite number of states, which includes an initial state s 0. At each time t=0,,..., the process is in some state s t. For each state s there is a finite set of decisions D s. If the process is in state s at time t and a decision d in D s is chosen, there is a fixed transition probability p (s,s,d) of being in a state s at time t +. The decision d incurs a cost c (s,d). A policy S is a mapping from states to decisions. The cost of a policy can be measured in different ways (see Howard [4]). The discounted cost with discount factor b is the expectation of the sum over all t of c (s t, S (s t ))b t. The total cost is the expectation of the sum over all t of c (s t, S (s t )). The average cost is the limit as t goes to infinity, of the total cost up to time t divided by t +. The discounted cost is always finite, whereas the other measures may be infinite. For a given cost measure, a policy is optimal if it minimizes the expected cost. The policy iteration algorithm is an iterative procedure that finds an optimal policy within a finite number of steps, which is at most exponential in the number of states. Assume for the moment that the cost Supported by National Science Foundation grant number CCR

2 - - measure is the discounted measure, since this is always finite. Roughly, the algorithm proceeds as follows. With respect to a given policy, we say that a state is switchable if the policy obtained by changing the decision at that state improves the policy, that is, lowers its cost. Initially, choose an arbitrary policy to be the current policy. Improve the current policy by switching the decision at one or more switchable states. Repeat this until no state is switchable. At that point, an optimal policy is reached. Since the number of policies is at most exponential in the number of states, the algorithm halts within exponential time. See Howard [4] for a proof of correctness of this algorithm. We describe this algorithm in more detail in the next section. There are many versions of this algorithm, depending on how the states to be switched are selected at a step of the algorithm. In this paper, we consider four versions and prove that they require exponential time in the worst case. To do this, we present for each algorithm an example of a Markov decision process on which the algorithm takes exponential time.. Exponential Time Examples for the Policy Improvement Algorithm Our examples are represented as graphs, where vertices represent states and edges represent transitions. The graphs have two types of vertices, called min and average vertices. Min vertices have two decisions, 0 and, and on each there is one transition with probability. Average vertices have one decision, with two transitions, each with probability /. There are two special sink vertices, each with no outgoing edges. All edges have cost 0, except edges to the sink, which have cost. Thus the total cost of a policy is the probability of reaching the -sink vertex. The cost of a vertex i with respect to a policy P is the probability of reaching the -sink from i, when the policy is P. (Note that this cost is the total cost of the process, as defined in the previous section). A min vertex is switchable with respect to a fixed policy if its cost is not the minimum of the costs of its children. On these graphs, the policy improvement algorithm works as follows. The algorithm starts from an initial policy. Then it repeatedly selects a switchable min vertex, and switches it. That is, it changes the policy so that the edge corresponding to the selected vertex is replaced by the other edge from the vertex. The algorithm halts when there is no switchable vertex, in which case an optimal policy has been found. For our graphs, the policy is optimal with respect to any of the three cost measures defined in the previous section. More than one vertex may be switchable at any iteration. Thus, to completely specify the algorithm, a select procedure must be defined, which returns the next vertex to be switched, if any. We now describe four select procedures and later we present exponential time examples for each of these procedures. The select procedure of the Simple Policy Improvement Algorithm selects arbitrarily one of the switchable min vertices. Specifically, it selects the switchable vertex with the largest number. The algorithm is simple in the sense that the select procedure does not select the vertex to be switched based on

3 - 3 - either the costs of the vertices or the structure of the graph. The select procedure of the Topological Policy Improvement Algorithm needs some preprocessing of the graph. The vertices are topologically sorted, that is, an integer order is assigned to each min vertex, such that if there is a path from vertex i to vertex j, then the order of i is greater or equal to the order of j. The graph is in this way divided into components of vertices with the same order. The select procedure selects the largest numbered switchable vertex in the component of lowest order that contains switchable vertices. In this way, when a vertex i is switched, there is a decrease in the cost of every vertex with larger order from which there is a path to i, according to the current policy. This algorithm can perform a lot better than the simple algorithm. If a directed graph is divided into k strongly connected components, and the numbers of min vertices in them are n, n,..., n k, then, in the worst case the algorithm takes n + n n k steps. If the maximum size of a component is bounded by a constant, the algorithm can find a best policy in a polynomial number of steps. In particular, if the graph is acyclic the algorithm makes at most linear number of switches. We now describe the select procedure of the Difference Policy Improvement Algorithm. As in the topological policy algorithm, the vertices are first topologically ordered. Consider the min vertices in the component with the lowest order that contains switchable vertices. For these vertices, the select procedure computes the difference between the costs of its two children, and selects the vertex with the largest difference. (If more than one vertex has this difference, the largest numbered one is selected). This is a natural approach to the problem of reducing the maximum number of switches required by the policy improvement algorithm, because when a vertex with a large difference is switched, the decreases of the costs of the vertices with larger order may be large too. Finally, we describe the Best Decrease Policy Improvement Algorithm. The select procedure of this algorithm first topologically sorts the vertices of the graph. Again, consider the switchable min vertices in the component of lowest order that contains switchable vertices. For these vertices, the select procedure computes the decrease of the cost of the vertex, if it is switched. The vertex that will get the largest decrease is selected (if more than one vertex can get this decrease, the largest numbered one is selected). We now present exponential time examples for these four algorithms. We use the following conventions in presenting our examples. A policy for a graph with n min vertices is represented as an n-bit vector, S = S n S n... S, where S i is the label of the edge in S that originates from i. For simplicity, when describing the execution of a policy improvement algorithm, we denote by V (i) the cost of vertex i with respect to the current policy. In our figures, min vertices are represented with circles numbered,,..., average vertices with circles numbered 0,,..., and the sink vertices with squares. The two edges for every min vertex are labeled with the decisions 0 and.

4 The Simple, Topological and Difference Algorithms We first consider the difference policy improvement algorithm. Our exponential time example for this algorithm is also an exponential time example for the simple and topological policy improvement algorithms. We first present a basic graph and then we add a gadget in appropriate places of this graph. The basic graph, given in Figure., has n min vertices (labeled,,..., n), n + average vertices (0,,..., n ), and the two sinks. n 0 n 0 n n (n )... 0 Figure.: Basic graph for general n. The structure can be described recursively; to construct the basic graph for n +, given the graph for n, we add a new min vertex, vertex n +, and a new average vertex, vertex (n +). The two edges for vertex n + lead to vertex (n +) and vertex n. Vertex (n +) is the average of n and n (or 0 if n is ). Also, there is an edge from 0 to min vertex n + instead of n. places. This basic graph of Figure. is augmented by adding the gadget g k of Figure. in appropriate j 0 k... m Figure.: Gadget g k The gadget g k is placed between a min vertex j and another, m, which can be a min, average or sink vertex. It consists of k average vertices,,,..., k. Vertex i ( i k) has one edge going to min vertex j and another going to the previous average vertex, (i ), or to vertex m if i is. It is easy to notice that this gadget has the following properties:

5 - 5 - If a policy contains the edge (j, k ), then the cost of j is equal to the cost of m. If a policy contains the other edge from j, then V (k ) = ( k ) V (j) + k V (m) => V (j) V (k ) = k Hence the difference between the costs of j s children is at most k. (V (j) V (m)) k k. The gadget g (n k) is added between vertex k and each of its two neighbors of the basic graph, for every k, < k < n, and g (n ) is added only between vertex and (because vertex is switched only once). Note that the number of new average vertices is polynomial ((n ) ) in the number of the min vertices n. Let the resulting graph be G n. Now suppose that the initial policy is the 0-vector (S = S n S n... S = ). With respect to this policy, every min vertex has cost. The best policy is edge 0 for every min vertex except vertex, and edge for that (S = ). With this policy, every min vertex has cost /. Although this policy differs by only one edge from the initial policy, the difference policy improvement algorithm takes n steps to find it. We next describe how the algorithm performs for the graph for n equal to. Initially both min vertices and are switchable. The difference between the values of the neighbors of vertex is /8, whereas the corresponding difference for vertex is /4, so it is first switched. Vertex is switched next, and then vertex. We have a total of = 3 switches. The values V (0 ), V (), V (), V (3) can be expressed by the following expressions (if n is greater than or equal to 3): V (0 ) = ( + V (n)) = + V (n), V () = S V ( ) + ( S )V (0 ) = S + ( S ) + V (n) = + V (n) S V (n) = V (0 ) S V (n), V ( ) = (V ( ) + V (0 )) = + + V (n) = + V (n), 4 V () = S V ( ) + ( S )V () = V () S V (n) + S S V (n) = V () S S 4 V (n). V (3)... =... V () S 3 S S V (n). This leads us to Lemma. which is the first step of the proof that the difference policy improvement algorithm makes an exponential number of switches to reach the best policy for this structure. In the main lemma of this proof (Lemma.7) we show that if the policy for the leftmost k vertices is , and these

6 - 6 - vertices are switchable, the next k switches of the algorithm are made on these vertices to reach the policy for these vertices. The most important step to that lemma is to express the cost of every vertex with a formula that depends on the current policy and on costs of other vertices (Lemma.). (After that point we do not need the graph.) The formulas give necessary and sufficient conditions (depending only on the current policy), for a vertex to be switchable (Corollary.4). The next two results (Corollary.5 and Lemma.6, which are also used in the proof of the main lemma) give relations between the switching of a vertex and other vertices becoming switchable. Definition.. a (k), for every positive k, is defined to be a (k +) = a (k) Sk and a () =. The next lemma is technical and it is only used in the proof of Lemma.. Lemma.. For every k n: V (k ) V (k ) = a (k) a (k ) V ((k ) ) V (k ), where V (0) should be substituted by V (0 ) in case k is. Proof: Note that addition of the gadgets does not affect the costs that the vertices get, but only hides the actual difference between the min vertices. V(k ) V (k ) = V ((k ) ) + V (k ) S k V ((k ) ) + ( S k )V (k ) (by the construction) = = a (k) a (k ) V ((k ) ) V (k ) (by Definition..) Lemma.. For every k, k n, the costs of the vertices, with respect to the current policy S, are given by the following formulas: V (k) = V (k ) + S k V (n)a (k) and V () = V (0 ) + S V (n)a (), V (k ) = V (k ) + V (n)a (k), V ( ) = V (0 ) + V (n)a (), and V (0 ) = + V (n), (I) (II) where a (k) is as in Definition.. Proof: The formulas can be proved by induction on k. The pair (I) is proved first: Basis Case: It has been proved that (I) holds for k equal to, and 3 by the relations presented before the lemma. Induction Hypothesis: We suppose that (I) holds for every k m. Induction Step: We prove that (I) holds for k equal to m +. V(m +) = S m + V((m +) ) + ( S m + )V (m) (by the construction) =

7 - 7 - = V (m) + S m + V ((m +) ) V (m ) S m V (n)a (m) (induction hypothesis) = = V (m) + S m + V (m ) + V (m ) V (m ) Sm V (n)a (m) (by the construction) = = V (m) + S m + a (m) (V ((m ) ) V (m )) Sm V (n)a (m) (by Lemma.) =... = a (m ) = V (m) + S m + a (m) (V ( ) V (0 )) Sm V (n)a (m) (by applying Lemma. m times) = a () = V (m) + S m + V (n)a (m) Sm (by the construction) = = V (m) + S m + V (n)a (m +) (by Definition.). Proof of the pair (II): Basis Case: The formulas obviously hold for k equal to 0 and. The basis case for k equal to is derived as follows. V ( ) = V () + V (n)a () = V (0 ) + S V (n)a () + V (n)a () S = = + V (n) + V (n)a () = + V (n). 4 Induction Hypothesis: We suppose that (II) holds for every k m. Induction Step: We prove that (II) holds for k equal to m +. V ((m +) ) = V (m ) + V (m ) (by the construction) = = V (m ) + V (n)a (m) (induction hypothesis) = = V (m) + Sm V (n)a (m) (by applying (I) for m) = = V (m) + V (n)a (m +) (by Definition.). From Definition., the following corollary is easily derived. Corollary.3. If a (k) is as Definition., then the following holds, for every positive k. ( ) S k a (k +) = a (k)

8 - 8 - Proof: The proof is based on the recursive formula for a (k +): a (k +) = a (k) Sk. If S k is equal to then a (k +) = ( ) S k a (k) = a (k). If Sk is equal to 0 then a (k +) = a (k) = ( ) S k a (k). Corollary.4. Vertex k is switchable if and only if either S k is 0 and a (k) is negative, or S k is and a (k) is positive, for every positive k. Proof: Proof that if k is switchable and S k is 0, a (k) is negative: If k is greater than then, according to Lemma., V (k) = V (k ) + S k V (n)a(k). Since S k is 0, V (k) = V (k ). Since the switch of the vertex to S k equal to gives a smaller cost, V (k ) > V (k ) + V (n)a(k) => a (k) < 0. (since V (n) is always positive). The rest can be proved by a very similar reasoning. Corollary.5. If vertex k (for every positive k) is switchable, and one or more larger numbered vertices are switched arbitrarily, vertex k is still switchable. Proof: The fact that k is switchable means that either S k is 0 and a (k) is negative, or S k is and a (k) is positive. Since the switches of larger numbered vertices do not have any effect on the cost of a (k), k is still switchable after these switches. Lemma.6. If vertex k (for every positive k less than n) is switched, and S n... Sk + S k + = 0...0, then all the vertices k +, k +,..., n are switchable. Proof: First, suppose that S k is switched from 0 to. From Corollary.4, it is deduced that a (k) is negative. By Corollary.3, a (k +) is positive, so by Corollary.4, vertex k + is switchable. By Corollary.3, a (k +) is negative, so by Corollary.4, vertex k + is switchable. It can be proved by induction that the other larger numbered vertices are switchable, too. If S k is switched from to 0, the proof that the lemma holds is the same. Lemma.7. The following two statements hold (for every positive k): If S n... Sk + S k = 0...0, and the vertices k, k +,.., n are switchable, the next n k + switches of the difference policy improvement algorithm are made on these vertices, to reach the policy where S n... Sk + S k = If S n... Sk + S k = , and the vertices k, k +,.., n are switchable, the next n k + switches of the difference policy improvement algorithm are made on these vertices, to reach the policy where S n... Sk + S k =

9 - 9 - Proof: The lemma will be proved by induction on k. Before that, an important notice on the difference between the costs of the vertices: According to Lemma., the cost of vertex k (k n), according to the policy S = S n S n...s, is V (k) = V (k ) + S k V (n)a (k) and V () = V (0 ) + S V (n)a (). From these formulas it follows that the difference between the costs of the two neighbors of vertex k is V (n)a (k), or V (n) (by Corollary.3), for the basic structure (without the gadgets). Note that the differ- k ence is constant for every vertex and does not depend on the policy. With the gadgets added to it, the difference for k becomes V (n) n k, which is an increasing function of k, so the algorithm performs like the simple policy algorithm. Basis Case: It is obvious that the statements hold for the case k = n. Induction Hypothesis: The statements hold for every k m. Induction Step: We will prove that the statements hold for k equal to m. We prove the first one first: The difference policy improvement algorithm works first on the vertices numbered n, n,..., m, since the select procedure always chooses the highest numbered switchable vertex. From the second statement of the induction hypothesis, we deduce that it performs n m + switches to reach the policy where S n... Sk + S m = Vertex m is still switchable (Corollary.5), so it is switched (from S m = to 0). By Lemma.6, the vertices n, n,..., m are again all switchable. From the first statement of the induction hypothesis, we deduce that the difference policy improvement algorithm performs n m + switches on these vertices to reach the policy where S n... Sk + S m = The total number of the switches is n m +. The second statement is proved following the same reasoning. We can now present the main result of the section. Theorem.8. The difference policy improvement algorithm requires exponential time in the worst case. Proof: Consider the graph G n. For every positive n, the algorithm makes n switches to find the best policy (vector ) if the initial policy is the 0-vector. This follows from Lemma.7, if initially every vertex is switchable. Since the initial policy is the 0-vector and a () is negative, every other a (k) is negative, too; so, by Corollary.4, every vertex is switchable. As corollaries of this theorem, it also follows that the simple and topological improvement algorithms require exponential time in the worst case. This is because on the graph G n, the simple and topological policy improvement algorithms perform just like the difference algorithm. Corollary.9. The topological policy improvement algorithm requires exponential time in the worst case.

10 - 0 - Corollary.0. The simple policy improvement algorithm requires exponential time in the worst case. The proof of Theorem.8 showed that the difference between between the costs of the two neighbors of a vertex is not proportional to the decrease that will actually result by making the switch. As a result, the difference algorithm can require exponential time. A better approach might be to compute for each switchable vertex the decrease that its switch will give, and decide which vertex to select based on that. This observation leads us to the best decrease policy improvement algorithm... The Best Decrease Algorithm The select procedure of this algorithm first topologically sorts the vertices of the graph. For every switchable min vertex in the component with the lowest order that contains switchable vertices, the decrease of the cost of the vertex if it is switched is computed. The vertex that will get the largest decrease is selected (if more than one vertex can get this decrease, the largest numbered one is selected). The algorithm can find the best policy of the graph G n in one step, if the initial policy is the 0-vector. However, we can construct another graph on which the best decrease algorithm requires exponential time. For the new structure we will use another gadget; the gadget g m,l is presented in Figure.4. l... a k c i b m... j Figure.4: Gadget g m,l The gadget g m,l is placed between a min vertex k and another vertex j, that can be a min, average or sink vertex. The gadget introduces a new min vertex i, and l +m new average vertices. Each vertex p of the set of m average vertices, has one edge going to min vertex k and another going to the previous average vertex (p ) or to vertex j if p is. Each vertex p of the set of l average vertices (double-primed vertices), has one edge going to min vertex k and another going to the next average vertex (p +) or to the - sink if p is l. Finally, there are edges from the new min vertex i to the average vertices and m, and an edge from min vertex k to min vertex i. The letters a, b and c denote the labels (0 or ) of the edges (i, ), (i, m ) and (k, i), respectively.

11 - - Roughly, the key property of this gadget is as follows. If S k c, S i b and V (j) is much smaller than V (k), then k and i can decrease in cost by a large amount if both are switched. However, if only one of k or i is switched, the decrease in cost of k or i is small. It can be easily checked that the gadget has the following properties: If S k = c and S i = b, then V (k) = V (j). If S i = a, then V (i) = ( l )V (k) + l. If S i = b, then V (i) = ( m )V (k) + m V (j). If S k c, then the improvement of V (i) when it is switched from a to b is: V (k) + l ( V (k)) V (k) m (V (j) V (k)) = m (V (k) V (j)) + l ( V (k)). If S k c, then the improvement of V (i) when it is switched from b to a is: V (k) + m (V (j) V (k)) V (k) l ( V (k)) = m (V (j) V (k)) l ( V (k)). The last two properties make sense only if the cost of vertices j and k are the same before and after the switch of vertex i, because in this case V (j) and V (k) are the same with respect to the current policy before and after the switch. This is true for the structure that will be presented because no edge (other than the edge labeled c in the gadget) will connect any vertex with vertex i. 3. Conclusion and Open Problems We have studied the complexity of the policy improvement algorithm for Markov decision processes and have shown that many natural variations of this algorithm require exponential time in the worst case. Apparently, no probabilistic analysis of the policy improvement algorithm has been undertaken, motivating the following questions. Suppose we define a new algorithm, the Randomized Policy Improvement Algorithm, by defining the select procedure to choose a switchable vertex randomly and uniformly from the set of switchable vertices. Does the randomized policy improvement algorithm run in expected polynomial time on all inputs? For all of the constructions in this paper, the randomized policy improvement algorithm requires only polynomial expected time (we leave it to the reader to verify that this is the case). It would also be interesting to obtain results on the average case performance of the policy improvement algorithm, on reasonable distributions of inputs. The work of Tovey [], may be a useful start in this direction. A select procedure widely used in practice is one in which all switchable states are switched at each iteration. Does this algorithm require exponential time in the worst case?

12 - - References [] A. Condon. The Complexity of Stochastic Games. Information and Computation, to appear, 990. Also available as Technical Report Number 863, Computer Sciences Department, University of Wisconsin-Madison. [] C. Derman. Finite State Markov Decision Processes. Academic Press, 97. [3] J. Gill. The Computational Complexity of Probabilistic Turing Machines. SIAM Journal on Computing, 6: , 977. [4] Howard. Dynamic Programming and Markov Processes. M.I.T. Press, 960. [5] R. J. Jeroslow. The Simplex Algorithm with the Pivot Rule of Maximizing Criterion Improvement. Discrete Math., 4: , 973. [6] D. S. Johnson, C. H. Papadimitriou and M. Yannakakis. How Easy is Local Search? Journal on Computer and System Sciences, 37:79-00, 988. [7] L. G. Khachiyan. A Polynomial algorithm in linear programming. Soviet Math Dokl., 0:9-94, 979. [8] V. Klee and G. Minty. How Good is the Simplex Algorithm? Inequalities III, O. Shisha, Academic Press, New York, 59-75, 979. [9] C. H. Papadimitriou, A. A. Sch"affer and M. Yannakakis. On the Complexity of Local Search. Proceedings of the nd Annual Symposium on the Theory of Computing (STOC), , 990. [0] H. J. M. Peters and O. J. Vrieze. Surveys in game theory and related topics, CWI Tract 39. Centrum voor Wiskunde en Informatica, Amsterdam, 987. [] L. S. Shapley. Stochastic Games. Proceedings of the National Academy of Sciences, U.S.A, 39: , 953. [] C. A. Tovey. Low Order Polynomial Bounds on the Expected Performance of Local Improvement Algorithms. Mathematical Programming, 35(): 93-4, 986.

### The Complexity of Stochastic Games. Anne Condon 1. University of Wisconsin-Madison West Dayton Street. Madison, WI

The Complexity of Stochastic Games Anne Condon 1 Computer Sciences Department University of WisconsinMadison 1210 West Dayton Street Madison, WI 5370 Abstract We consider the complexity of stochastic games

### The simplex method and the diameter of a 0-1 polytope

The simplex method and the diameter of a 0-1 polytope Tomonari Kitahara and Shinji Mizuno May 2012 Abstract We will derive two main results related to the primal simplex method for an LP on a 0-1 polytope.

### New Results on Simple Stochastic Games

New Results on Simple Stochastic Games Decheng Dai 1 and Rong Ge 2 1 Tsinghua University, ddc02@mails.tsinghua.edu.cn 2 Princeton University, rongge@cs.princeton.edu Abstract. We study the problem of solving

### Subexponential Lower Bounds for the Simplex Algorithm

Subexponential Lower Bounds for the Simplex Algorithm Oliver Friedmann Department of Computer Science, Ludwig-Maximilians-Universität Munich, Germany. January 0, 011 Oliver Friedmann (LMU) Subexponential

### Root Cover Pebbling on Graphs

University of Dayton ecommons Honors Theses University Honors Program Spring 4-2015 Root Cover Pebbling on Graphs Claire A. Sonneborn Follow this and additional works at: https://ecommons.udayton.edu/uhp_theses

### Topic: Local Search: Max-Cut, Facility Location Date: 2/13/2007

CS880: Approximations Algorithms Scribe: Chi Man Liu Lecturer: Shuchi Chawla Topic: Local Search: Max-Cut, Facility Location Date: 2/3/2007 In previous lectures we saw how dynamic programming could be

### SAT-CNF Is N P-complete

SAT-CNF Is N P-complete Rod Howell Kansas State University November 9, 2000 The purpose of this paper is to give a detailed presentation of an N P- completeness proof using the definition of N P given

### Subexponential lower bounds for randomized pivoting rules for the simplex algorithm

Subexponential lower bounds for randomized pivoting rules for the simplex algorithm Oliver Friedmann 1 Thomas Dueholm Hansen 2 Uri Zwick 3 1 Department of Computer Science, University of Munich, Germany.

### The strong chromatic number of a graph

The strong chromatic number of a graph Noga Alon Abstract It is shown that there is an absolute constant c with the following property: For any two graphs G 1 = (V, E 1 ) and G 2 = (V, E 2 ) on the same

### Introduction to Automata Theory. BİL405 - Automata Theory and Formal Languages 1

Introduction to Automata Theory BİL405 - Automata Theory and Formal Languages 1 Automata, Computability and Complexity Automata, Computability and Complexity are linked by the question: What are the fundamental

### Parameterized graph separation problems

Parameterized graph separation problems Dániel Marx Department of Computer Science and Information Theory, Budapest University of Technology and Economics Budapest, H-1521, Hungary, dmarx@cs.bme.hu Abstract.

### IS BINARY ENCODING APPROPRIATE FOR THE PROBLEM-LANGUAGE RELATIONSHIP?

Theoretical Computer Science 19 (1982) 337-341 North-Holland Publishing Company NOTE IS BINARY ENCODING APPROPRIATE FOR THE PROBLEM-LANGUAGE RELATIONSHIP? Nimrod MEGIDDO Statistics Department, Tel Aviv

### A Fast Algorithm for Optimal Alignment between Similar Ordered Trees

Fundamenta Informaticae 56 (2003) 105 120 105 IOS Press A Fast Algorithm for Optimal Alignment between Similar Ordered Trees Jesper Jansson Department of Computer Science Lund University, Box 118 SE-221

### Lecture notes on the simplex method September We will present an algorithm to solve linear programs of the form. maximize.

Cornell University, Fall 2017 CS 6820: Algorithms Lecture notes on the simplex method September 2017 1 The Simplex Method We will present an algorithm to solve linear programs of the form maximize subject

### CS521 \ Notes for the Final Exam

CS521 \ Notes for final exam 1 Ariel Stolerman Asymptotic Notations: CS521 \ Notes for the Final Exam Notation Definition Limit Big-O ( ) Small-o ( ) Big- ( ) Small- ( ) Big- ( ) Notes: ( ) ( ) ( ) ( )

### A Network Coloring Game

A Network Coloring Game Kamalika Chaudhuri, Fan Chung 2, and Mohammad Shoaib Jamall 2 Information Theory and Applications Center, UC San Diego kamalika@soe.ucsd.edu 2 Department of Mathematics, UC San

### A Simplied NP-complete MAXSAT Problem. Abstract. It is shown that the MAX2SAT problem is NP-complete even if every variable

A Simplied NP-complete MAXSAT Problem Venkatesh Raman 1, B. Ravikumar 2 and S. Srinivasa Rao 1 1 The Institute of Mathematical Sciences, C. I. T. Campus, Chennai 600 113. India 2 Department of Computer

### CSE 20 DISCRETE MATH WINTER

CSE 20 DISCRETE MATH WINTER 2016 http://cseweb.ucsd.edu/classes/wi16/cse20-ab/ Today's learning goals Explain the steps in a proof by (strong) mathematical induction Use (strong) mathematical induction

### Mathematical and Algorithmic Foundations Linear Programming and Matchings

Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis

### THEORY OF LINEAR AND INTEGER PROGRAMMING

THEORY OF LINEAR AND INTEGER PROGRAMMING ALEXANDER SCHRIJVER Centrum voor Wiskunde en Informatica, Amsterdam A Wiley-Inter science Publication JOHN WILEY & SONS^ Chichester New York Weinheim Brisbane Singapore

### Random Walks and Universal Sequences

Random Walks and Universal Sequences Xiaochen Qi February 28, 2013 Abstract A random walk is a chance process studied in probability, which plays an important role in probability theory and its applications.

### Module 7. Independent sets, coverings. and matchings. Contents

Module 7 Independent sets, coverings Contents and matchings 7.1 Introduction.......................... 152 7.2 Independent sets and coverings: basic equations..... 152 7.3 Matchings in bipartite graphs................

### Bottleneck Steiner Tree with Bounded Number of Steiner Vertices

Bottleneck Steiner Tree with Bounded Number of Steiner Vertices A. Karim Abu-Affash Paz Carmi Matthew J. Katz June 18, 2011 Abstract Given a complete graph G = (V, E), where each vertex is labeled either

### From Static to Dynamic Routing: Efficient Transformations of Store-and-Forward Protocols

SIAM Journal on Computing to appear From Static to Dynamic Routing: Efficient Transformations of StoreandForward Protocols Christian Scheideler Berthold Vöcking Abstract We investigate how static storeandforward

### Discrete mathematics , Fall Instructor: prof. János Pach

Discrete mathematics 2016-2017, Fall Instructor: prof. János Pach - covered material - Lecture 1. Counting problems To read: [Lov]: 1.2. Sets, 1.3. Number of subsets, 1.5. Sequences, 1.6. Permutations,

### Algorithmic Game Theory and Applications. Lecture 6: The Simplex Algorithm

Algorithmic Game Theory and Applications Lecture 6: The Simplex Algorithm Kousha Etessami Recall our example 1 x + y

### Discrete Optimization. Lecture Notes 2

Discrete Optimization. Lecture Notes 2 Disjunctive Constraints Defining variables and formulating linear constraints can be straightforward or more sophisticated, depending on the problem structure. The

### Finding a -regular Supergraph of Minimum Order

Finding a -regular Supergraph of Minimum Order Hans L. Bodlaender a, Richard B. Tan a,b and Jan van Leeuwen a a Department of Computer Science Utrecht University Padualaan 14, 3584 CH Utrecht The Netherlands

### A SIMPLE APPROXIMATION ALGORITHM FOR NONOVERLAPPING LOCAL ALIGNMENTS (WEIGHTED INDEPENDENT SETS OF AXIS PARALLEL RECTANGLES)

Chapter 1 A SIMPLE APPROXIMATION ALGORITHM FOR NONOVERLAPPING LOCAL ALIGNMENTS (WEIGHTED INDEPENDENT SETS OF AXIS PARALLEL RECTANGLES) Piotr Berman Department of Computer Science & Engineering Pennsylvania

### Copyright 2000, Kevin Wayne 1

Guessing Game: NP-Complete? 1. LONGEST-PATH: Given a graph G = (V, E), does there exists a simple path of length at least k edges? YES. SHORTEST-PATH: Given a graph G = (V, E), does there exists a simple

### On Covering a Graph Optimally with Induced Subgraphs

On Covering a Graph Optimally with Induced Subgraphs Shripad Thite April 1, 006 Abstract We consider the problem of covering a graph with a given number of induced subgraphs so that the maximum number

### Distributed minimum spanning tree problem

Distributed minimum spanning tree problem Juho-Kustaa Kangas 24th November 2012 Abstract Given a connected weighted undirected graph, the minimum spanning tree problem asks for a spanning subtree with

### arxiv: v1 [cs.cc] 30 Jun 2017

On the Complexity of Polytopes in LI( Komei Fuuda May Szedlá July, 018 arxiv:170610114v1 [cscc] 30 Jun 017 Abstract In this paper we consider polytopes given by systems of n inequalities in d variables,

### These notes present some properties of chordal graphs, a set of undirected graphs that are important for undirected graphical models.

Undirected Graphical Models: Chordal Graphs, Decomposable Graphs, Junction Trees, and Factorizations Peter Bartlett. October 2003. These notes present some properties of chordal graphs, a set of undirected

### Eulerian disjoint paths problem in grid graphs is NP-complete

Discrete Applied Mathematics 143 (2004) 336 341 Notes Eulerian disjoint paths problem in grid graphs is NP-complete Daniel Marx www.elsevier.com/locate/dam Department of Computer Science and Information

### Definition For vertices u, v V (G), the distance from u to v, denoted d(u, v), in G is the length of a shortest u, v-path. 1

Graph fundamentals Bipartite graph characterization Lemma. If a graph contains an odd closed walk, then it contains an odd cycle. Proof strategy: Consider a shortest closed odd walk W. If W is not a cycle,

### Randomized Algorithms 2017A - Lecture 10 Metric Embeddings into Random Trees

Randomized Algorithms 2017A - Lecture 10 Metric Embeddings into Random Trees Lior Kamma 1 Introduction Embeddings and Distortion An embedding of a metric space (X, d X ) into a metric space (Y, d Y ) is

### Minimum Cost Edge Disjoint Paths

Minimum Cost Edge Disjoint Paths Theodor Mader 15.4.2008 1 Introduction Finding paths in networks and graphs constitutes an area of theoretical computer science which has been highly researched during

### Rainbow game domination subdivision number of a graph

Rainbow game domination subdivision number of a graph J. Amjadi Department of Mathematics Azarbaijan Shahid Madani University Tabriz, I.R. Iran j-amjadi@azaruniv.edu Abstract The rainbow game domination

### 1. (15 points) Solve the decanting problem for containers of sizes 199 and 179; that is find integers x and y satisfying.

May 9, 2003 Show all work Name There are 260 points available on this test 1 (15 points) Solve the decanting problem for containers of sizes 199 and 179; that is find integers x and y satisfying where

### Clustering: Centroid-Based Partitioning

Clustering: Centroid-Based Partitioning Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong 1 / 29 Y Tao Clustering: Centroid-Based Partitioning In this lecture, we

### Solution to Graded Problem Set 4

Graph Theory Applications EPFL, Spring 2014 Solution to Graded Problem Set 4 Date: 13.03.2014 Due by 18:00 20.03.2014 Problem 1. Let V be the set of vertices, x be the number of leaves in the tree and

### Acyclic Edge Colorings of Graphs

Acyclic Edge Colorings of Graphs Noga Alon Benny Sudaov Ayal Zas Abstract A proper coloring of the edges of a graph G is called acyclic if there is no 2-colored cycle in G. The acyclic edge chromatic number

### Chordal deletion is fixed-parameter tractable

Chordal deletion is fixed-parameter tractable Dániel Marx Institut für Informatik, Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany. dmarx@informatik.hu-berlin.de Abstract. It

### Graph Algorithms. Chromatic Polynomials. Graph Algorithms

Graph Algorithms Chromatic Polynomials Graph Algorithms Chromatic Polynomials Definition G a simple labelled graph with n vertices and m edges. k a positive integer. P G (k) number of different ways of

### The Full Survey on The Euclidean Steiner Tree Problem

The Full Survey on The Euclidean Steiner Tree Problem Shikun Liu Abstract The Steiner Tree Problem is a famous and long-studied problem in combinatorial optimization. However, the best heuristics algorithm

### The Simplex Algorithm for LP, and an Open Problem

The Simplex Algorithm for LP, and an Open Problem Linear Programming: General Formulation Inputs: real-valued m x n matrix A, and vectors c in R n and b in R m Output: n-dimensional vector x There is one

### Complexity Results on Graphs with Few Cliques

Discrete Mathematics and Theoretical Computer Science DMTCS vol. 9, 2007, 127 136 Complexity Results on Graphs with Few Cliques Bill Rosgen 1 and Lorna Stewart 2 1 Institute for Quantum Computing and School

### Theorem 2.9: nearest addition algorithm

There are severe limits on our ability to compute near-optimal tours It is NP-complete to decide whether a given undirected =(,)has a Hamiltonian cycle An approximation algorithm for the TSP can be used

### Dirac-type characterizations of graphs without long chordless cycles

Dirac-type characterizations of graphs without long chordless cycles Vašek Chvátal Department of Computer Science Rutgers University chvatal@cs.rutgers.edu Irena Rusu LIFO Université de Orléans irusu@lifo.univ-orleans.fr

### CSE 417 Branch & Bound (pt 4) Branch & Bound

CSE 417 Branch & Bound (pt 4) Branch & Bound Reminders > HW8 due today > HW9 will be posted tomorrow start early program will be slow, so debugging will be slow... Review of previous lectures > Complexity

### A more efficient algorithm for perfect sorting by reversals

A more efficient algorithm for perfect sorting by reversals Sèverine Bérard 1,2, Cedric Chauve 3,4, and Christophe Paul 5 1 Département de Mathématiques et d Informatique Appliquée, INRA, Toulouse, France.

### II (Sorting and) Order Statistics

II (Sorting and) Order Statistics Heapsort Quicksort Sorting in Linear Time Medians and Order Statistics 8 Sorting in Linear Time The sorting algorithms introduced thus far are comparison sorts Any comparison

### On the Relationships between Zero Forcing Numbers and Certain Graph Coverings

On the Relationships between Zero Forcing Numbers and Certain Graph Coverings Fatemeh Alinaghipour Taklimi, Shaun Fallat 1,, Karen Meagher 2 Department of Mathematics and Statistics, University of Regina,

### SCHEDULING WITH RELEASE TIMES AND DEADLINES ON A MINIMUM NUMBER OF MACHINES *

SCHEDULING WITH RELEASE TIMES AND DEADLINES ON A MINIMUM NUMBER OF MACHINES * Mark Cieliebak 1, Thomas Erlebach 2, Fabian Hennecke 1, Birgitta Weber 1, and Peter Widmayer 1 1 Institute of Theoretical Computer

### 6th Bay Area Mathematical Olympiad

6th Bay Area Mathematical Olympiad February 4, 004 Problems and Solutions 1 A tiling of the plane with polygons consists of placing the polygons in the plane so that interiors of polygons do not overlap,

### AXIOMS FOR THE INTEGERS

AXIOMS FOR THE INTEGERS BRIAN OSSERMAN We describe the set of axioms for the integers which we will use in the class. The axioms are almost the same as what is presented in Appendix A of the textbook,

### NP-Completeness. Algorithms

NP-Completeness Algorithms The NP-Completeness Theory Objective: Identify a class of problems that are hard to solve. Exponential time is hard. Polynomial time is easy. Why: Do not try to find efficient

### Solution for Homework set 3

TTIC 300 and CMSC 37000 Algorithms Winter 07 Solution for Homework set 3 Question (0 points) We are given a directed graph G = (V, E), with two special vertices s and t, and non-negative integral capacities

### FINAL EXAM SOLUTIONS

COMP/MATH 3804 Design and Analysis of Algorithms I Fall 2015 FINAL EXAM SOLUTIONS Question 1 (12%). Modify Euclid s algorithm as follows. function Newclid(a,b) if a

### To illustrate what is intended the following are three write ups by students. Diagonalization

General guidelines: You may work with other people, as long as you write up your solution in your own words and understand everything you turn in. Make sure to justify your answers they should be clear

### Pebbling on Directed Graphs

Pebbling on Directed Graphs Gayatri Gunda E-mail: gundagay@notes.udayton.edu Dr. Aparna Higgins E-mail: Aparna.Higgins@notes.udayton.edu University of Dayton Dayton, OH 45469 Submitted January 25 th, 2004

### Modular Representations of Graphs

Modular Representations of Graphs Crystal Altamirano, Stephanie Angus, Lauren Brown, Joseph Crawford, and Laura Gioco July 2011 Abstract A graph G has a representation modulo r if there exists an injective

### Extremal Graph Theory: Turán s Theorem

Bridgewater State University Virtual Commons - Bridgewater State University Honors Program Theses and Projects Undergraduate Honors Program 5-9-07 Extremal Graph Theory: Turán s Theorem Vincent Vascimini

### Lecture 22 Tuesday, April 10

CIS 160 - Spring 2018 (instructor Val Tannen) Lecture 22 Tuesday, April 10 GRAPH THEORY Directed Graphs Directed graphs (a.k.a. digraphs) are an important mathematical modeling tool in Computer Science,

### Algorithms for finding the minimum cycle mean in the weighted directed graph

Computer Science Journal of Moldova, vol.6, no.1(16), 1998 Algorithms for finding the minimum cycle mean in the weighted directed graph D. Lozovanu C. Petic Abstract In this paper we study the problem

### Edge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles

Theory and Applications of Graphs Volume 4 Issue 2 Article 2 November 2017 Edge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles Peter Johnson johnspd@auburn.edu Andrew Owens Auburn

### Greedy Algorithms 1 {K(S) K(S) C} For large values of d, brute force search is not feasible because there are 2 d {1,..., d}.

Greedy Algorithms 1 Simple Knapsack Problem Greedy Algorithms form an important class of algorithmic techniques. We illustrate the idea by applying it to a simplified version of the Knapsack Problem. Informally,

### AN ITERATIVE APPROACH TO THE IRREGULARITY STRENGTH OF TREES

AN ITERATIVE APPROACH TO THE IRREGULARITY STRENGTH OF TREES MICHAEL FERRARA, RONALD J. GOULD, MICHA L KAROŃSKI, AND FLORIAN PFENDER Abstract. An assignment of positive integer weights to the edges of a

### st-orientations September 29, 2005

st-orientations September 29, 2005 Introduction Let G = (V, E) be an undirected biconnected graph of n nodes and m edges. The main problem this chapter deals with is different algorithms for orienting

### Disjoint directed cycles

Disjoint directed cycles Noga Alon Abstract It is shown that there exists a positive ɛ so that for any integer k, every directed graph with minimum outdegree at least k contains at least ɛk vertex disjoint

### Chapter 9 Graph Algorithms

Introduction graph theory useful in practice represent many real-life problems can be if not careful with data structures Chapter 9 Graph s 2 Definitions Definitions an undirected graph is a finite set

### Local Search Approximation Algorithms for the Complement of the Min-k-Cut Problems

Local Search Approximation Algorithms for the Complement of the Min-k-Cut Problems Wenxing Zhu, Chuanyin Guo Center for Discrete Mathematics and Theoretical Computer Science, Fuzhou University, Fuzhou

### Are Hopfield Networks Faster Than Conventional Computers?

Are Hopfield Networks Faster Than Conventional Computers? Ian Parberry* and Hung-Li Tsengt Department of Computer Sciences University of North Texas P.O. Box 13886 Denton, TX 76203-6886 Abstract It is

### Forced orientation of graphs

Forced orientation of graphs Babak Farzad Mohammad Mahdian Ebad S. Mahmoodian Amin Saberi Bardia Sadri Abstract The concept of forced orientation of graphs was introduced by G. Chartrand et al. in 1994.

### A Note on Polyhedral Relaxations for the Maximum Cut Problem

A Note on Polyhedral Relaxations for the Maximum Cut Problem Alantha Newman Abstract We consider three well-studied polyhedral relaxations for the maximum cut problem: the metric polytope of the complete

### Number Theory and Graph Theory

1 Number Theory and Graph Theory Chapter 6 Basic concepts and definitions of graph theory By A. Satyanarayana Reddy Department of Mathematics Shiv Nadar University Uttar Pradesh, India E-mail: satya8118@gmail.com

### Discrete mathematics

Discrete mathematics Petr Kovář petr.kovar@vsb.cz VŠB Technical University of Ostrava DiM 470-2301/02, Winter term 2018/2019 About this file This file is meant to be a guideline for the lecturer. Many

### Trees. 3. (Minimally Connected) G is connected and deleting any of its edges gives rise to a disconnected graph.

Trees 1 Introduction Trees are very special kind of (undirected) graphs. Formally speaking, a tree is a connected graph that is acyclic. 1 This definition has some drawbacks: given a graph it is not trivial

### Trail Making Game. Hyun Sung Jun Jaehoon Kim Sang-il Oum Department of Mathematical Sciences KAIST, Daejeon, , Republic of Korea.

Trail Making Game Hyun Sung Jun Jaehoon Kim Sang-il Oum Department of Mathematical Sciences KAIST, Daejeon, 305-701, Republic of Korea. May 7, 2009 Abstract Trail Making is a game played on a graph with

### 31.6 Powers of an element

31.6 Powers of an element Just as we often consider the multiples of a given element, modulo, we consider the sequence of powers of, modulo, where :,,,,. modulo Indexing from 0, the 0th value in this sequence

### Mathematics of Optimization Viewpoint of applied mathematics Algorithm

Global Focus on Knowledge Lecture (The World of Mathematics #8) 2007 06 14 Mathematics of Optimization Viewpoint of applied mathematics Algorithm Kazuo Murota Department of Mathematical Engineering and

### Searching a Sorted Set of Strings

Department of Mathematics and Computer Science January 24, 2017 University of Southern Denmark RF Searching a Sorted Set of Strings Assume we have a set of n strings in RAM, and know their sorted order

### arxiv: v1 [math.co] 3 Apr 2016

A note on extremal results on directed acyclic graphs arxiv:1604.0061v1 [math.co] 3 Apr 016 A. Martínez-Pérez, L. Montejano and D. Oliveros April 5, 016 Abstract The family of Directed Acyclic Graphs as

### Lecture 9. Semidefinite programming is linear programming where variables are entries in a positive semidefinite matrix.

CSE525: Randomized Algorithms and Probabilistic Analysis Lecture 9 Lecturer: Anna Karlin Scribe: Sonya Alexandrova and Keith Jia 1 Introduction to semidefinite programming Semidefinite programming is linear

### 3 No-Wait Job Shops with Variable Processing Times

3 No-Wait Job Shops with Variable Processing Times In this chapter we assume that, on top of the classical no-wait job shop setting, we are given a set of processing times for each operation. We may select

### An Optimal Algorithm for the Euclidean Bottleneck Full Steiner Tree Problem

An Optimal Algorithm for the Euclidean Bottleneck Full Steiner Tree Problem Ahmad Biniaz Anil Maheshwari Michiel Smid September 30, 2013 Abstract Let P and S be two disjoint sets of n and m points in the

### New Upper Bounds for MAX-2-SAT and MAX-2-CSP w.r.t. the Average Variable Degree

New Upper Bounds for MAX-2-SAT and MAX-2-CSP w.r.t. the Average Variable Degree Alexander Golovnev St. Petersburg University of the Russian Academy of Sciences, St. Petersburg, Russia alex.golovnev@gmail.com

### Coloring edges and vertices of graphs without short or long cycles

Coloring edges and vertices of graphs without short or long cycles Marcin Kamiński and Vadim Lozin Abstract Vertex and edge colorability are two graph problems that are NPhard in general. We show that

### Figure : Example Precedence Graph

CS787: Advanced Algorithms Topic: Scheduling with Precedence Constraints Presenter(s): James Jolly, Pratima Kolan 17.5.1 Motivation 17.5.1.1 Objective Consider the problem of scheduling a collection of

### Lecture Notes on Karger s Min-Cut Algorithm. Eric Vigoda Georgia Institute of Technology Last updated for Advanced Algorithms, Fall 2013.

Lecture Notes on Karger s Min-Cut Algorithm. Eric Vigoda Georgia Institute of Technology Last updated for 4540 - Advanced Algorithms, Fall 2013. Today s topic: Karger s min-cut algorithm [3]. Problem Definition

### Shannon capacity and related problems in Information Theory and Ramsey Theory

Shannon capacity and related problems in Information Theory and Ramsey Theory Eyal Lubetzky Based on Joint work with Noga Alon and Uri Stav May 2007 1 Outline of talk Shannon Capacity of of a graph: graph:

### CS675: Convex and Combinatorial Optimization Spring 2018 The Simplex Algorithm. Instructor: Shaddin Dughmi

CS675: Convex and Combinatorial Optimization Spring 2018 The Simplex Algorithm Instructor: Shaddin Dughmi Algorithms for Convex Optimization We will look at 2 algorithms in detail: Simplex and Ellipsoid.

### Chapter 9 Graph Algorithms

Chapter 9 Graph Algorithms 2 Introduction graph theory useful in practice represent many real-life problems can be if not careful with data structures 3 Definitions an undirected graph G = (V, E) is a

### W[1]-hardness. Dániel Marx. Recent Advances in Parameterized Complexity Tel Aviv, Israel, December 3, 2017

1 W[1]-hardness Dániel Marx Recent Advances in Parameterized Complexity Tel Aviv, Israel, December 3, 2017 2 Lower bounds So far we have seen positive results: basic algorithmic techniques for fixed-parameter

### Distributed Planning in Stochastic Games with Communication

Distributed Planning in Stochastic Games with Communication Andriy Burkov and Brahim Chaib-draa DAMAS Laboratory Laval University G1K 7P4, Quebec, Canada {burkov,chaib}@damas.ift.ulaval.ca Abstract This

### 11.1 Facility Location

CS787: Advanced Algorithms Scribe: Amanda Burton, Leah Kluegel Lecturer: Shuchi Chawla Topic: Facility Location ctd., Linear Programming Date: October 8, 2007 Today we conclude the discussion of local