Approximate Linear Programming for Average-Cost Dynamic Programming

Save this PDF as:

Size: px
Start display at page:

Transcription

1 Approximate Linear Programming for Average-Cost Dynamic Programming Daniela Pucci de Farias IBM Almaden Research Center 65 Harry Road, San Jose, CA 51 Benjamin Van Roy Department of Management Science and Engineering Stanford University Stanford, CA 435 Abstract This paper extends our earlier analysis on approximate linear programming as an approach to approximating the cost-to-go function in a discounted-cost dynamic program [6] In this paper, we consider the average-cost criterion and a version of approximate linear programming that generates approximations to the optimal average cost and differential cost function We demonstrate that a naive version of approximate linear programming prioritizes approximation of the optimal average cost and that this may not be well-aligned with the objective of deriving a policy with low average cost For that, the algorithm should aim at producing a good approximation of the differential cost function We propose a twophase variant of approximate linear programming that allows for external control of the relative accuracy of the approximation of the differential cost function over different portions of the state space via state-relevance weights Performance bounds suggest that the new algorithm is compatible with the objective of optimizing performance and provide guidance on appropriate choices for state-relevance weights 1 Introduction The curse of dimensionality prevents application of dynamic programming to most problems of practical interest Approximate linear programming (ALP) aims to alleviate the curse of dimensionality by approximation of the dynamic programming solution In [6], we develop a variant of approximate linear programming for the discounted-cost case which is shown to scale well with problem size In this paper, we extend that analysis to the average-cost criterion Originally introduced by Schweitzer and Seidmann [11], approximate linear programming combines the linear programming approach to exact dynamic programming [] to ap-

3 # # + + # M When the current state is and action %'& is taken, a cost is incurred State transition probabilities represent, for each pair of states and each action % &, the probability that the next state will be given that the current state is and the current action is % & A policy is a mapping from states to actions Given a policy, the dynamics of the system follow a Markov chain with transition probabilities & For each policy, we define a transition matrix whose th entry is &, and a cost vector whose th entry is & We make the following assumption on the transition probabilities: Assumption 1 (Irreducibility) For each pair of states and and each policy, there is such that In stochastic control problems, we want to select a policy optimizing a given criterion In this paper, we will employ as an optimality criterion the average cost "!\$# &%(' *),+ -) / Irreducibility implies that, for each policy, this limit exists and 1 for all the average cost is independent of the initial state in the system * 345 We denote the minimal average cost by For any policy, we define the associated dynamic programming operator 6 by 6 8 : 8 Note that 6 operates on vectors #;<=>< corresponding to functions on the state space We also define the dynamic programming operator 6 by 6?45 6 A policy is called greedy with respect to if it attains the minimum in the definition of 6 An optimal policy minimizing the average cost can be derived from the solution of Bellman s equation A where is the vector of ones We denote solutions to Bellman s equation by pairs * * The scalar is unique and equal to the the optimal average cost The vector is called a differential cost function The differential cost function is unique up to a constant factor; if solves Bellman s equation, then CB is also a solution for all B, and all other solutions can be shown to be of this form We can ensure uniqueness by imposing for an arbitrary state Any policy that is greedy with respect to the differential cost function is optimal Solving Bellman s equation involves computing and storing the differential cost function for all states in the system This is computationally infeasible in most problems of practical interest due to the explosion on the number of states as the number of state variables grows We try to combat the curse of dimensionality by settling for the more modest goal of finding an approximation to the differential cost function The underlying assumption is that, in many problems of practical interest, the differential cost function will exhibit some regularity, or structure, allowing for reasonable approximations to be stored compactly &D We consider a linear approximation architecture: given a set of functions FEG ; H, we generate approximations of the form #O;< =><QP &IKJ 3R TSUSUS We define a matrix N L by N, ie, each of the basis functions is stored as a column of N, and each row corresponds tolcv a vector of the basis functions evaluated at a distinct state J S We represent in matrix notation as N In the remainder of the paper, we assume that (a manageable number of) basis functions are prespecified, and address the problem of choosing a suitable parameter vector For simplicity, W we choose an arbitrary state henceforth called state for X which we set ; accordingly, we assume that the basis functions are such that,y (1)

4 Y 3 Approximate Linear Programming Approximate linear programming [11, 6] is inspired by the traditional linear programming approach to dynamic programming, introduced by [] Bellman s equation can be solved by the average-cost exact 6 Note that the constraints 3 6 can be replaced by therefore we can think of problem () as an LP In approximate linear programming, we reduce the generally intractable dimensions of the average-cost ELP by constraining to be of the form N This yields the first-phase approximate LP N 6 N Problem (3) can be expressed as an LP by the same argument used for the exact LP We denote its solution by The following result is immediate Lemma 1 The solution of the first-phase ALP minimizes region *F () (3) over the feasible >C Proof: Maximizing in (3) is equivalent to maimizing Since the first-phase ALP corresponds to the exact LP () with extra constraints N, we have > for all feasible Hence " ", and the claim follows Lemma 1 implies that the first-phase ALP can be seen as an algorithm for approximating the optimal average cost Using this algorithm for generating a policy for the averagecost problem is based on the hope that approximation of the optimal average cost should also implicitly imply approximation of the differential cost function Note that it is not unreasonable to expect that some* approximation of the differential cost function should be involved in the minimization of ; for instance, we know that iff N The ALP has as many variables as the number of basis functions plus one, which will usually amount to a dramatically smaller number of variables than what we had in the ELP However, the ALP still has as many constraints as the number of state-action pairs This problem is also found in the discounted-cost formulation and there are several approaches in the literature for dealing with it, including constraint sampling [] and exploitation of problem-specific structures for efficient elimination of redundant constraints [8, 1] Our first step in the analysis of average-cost ALP is to demonstrate through a counterexample that it can produce arbitrarily bad policies, even if the approximation to the average cost is very accurate 4 Performance of the first-phase ALP: a counterexample We consider a Markov process with states, each representing a possible number of jobs in a queue with buffer of size The system state evolves according to "! # %\$'&(%)+* */, -/ "! # %\$'&(%)+* */, - H ) \$13(! 1

5 Y V From state, transitions to states H and occurs with probabilities and H, respectively From state, transitions to states and occur with probabilities / / and H H, respectively The arrival probability is the same for all states and we let The action to be chosen in each state is the departure probability or service rate /, which takes values the set The cost incurred at state if action / %/ is taken is given by / ) ) We use basis functions, For, the first-phase ALP yields an approximation * for the optimal average cost, which is within % of the true value However, the average cost yielded by the greedy policy with respect to N is 84 for, and goes to infinity as we increase the buffer size Figure 1 explains this behavior Note that N J is a very good approximation for over states, and becomes progressive worse as increases States correspond I to virtually all of the stationary probability under the optimal policy ( ), hence * it is not surprising that the first-phase ALP yields a very accurate approximation for, as other states contribute very little to the optimal average cost However, fitting the optimal average cost and the differential cost function over states visited often under the optimal policy is not sufficient for getting a good policy Indeed, N severely underestimates costs in large states, and the greedy policy drives the system to those states, yielding a very large average cost and ultimately making the system unstable, when the buffer size goes to infinity It is also troublesome to note that our choice of basis function actually has the potential to, the greedy policy associated with N has an average cost approximately equal to, regardless of the buffer size, which is only about larger than the optimal average cost Hence even though the first-phase ALP is being given a relatively good set of basis functions, it is producing a bad approximate differential cost function, which cannot be improved unless different basis functions are selected lead to a reasonably good policy indeed, for R# 5 Two-phase average-cost ALP A striking difference between the first-phase average-cost ALP and discounted-cost ALP is the presence in the latter of state relevance weights These are algorithm parameters that can be used to control the accuracy of the approximation to the cost-to-go function (the discounted-cost counterpart of the differential cost function) over different portions of the state space and have been shown in [6] to have a first-order impact on the performance of the policy being generated For instance, in the example described in the previous section, in the discounted-cost formulation one might be able to improve the policy yielded by ALP by choosing state-relevance weights that put more emphasis on states Inspired by this observation, we propose a two-phase algorithm with the characteristic that staterelevance weights are present and can be used to control the quality of the differential cost function approximation The first phase is simply the first-phase ALP introduced in Section 3, and is used for generating an approximation to the optimal average cost The second phase consists of solving the second-phase ALP for finding approximations to the differential cost function: N (4) 6 N N 1 and are algorithm parameters to be specified by the The state-relevance weights user and denotes the transpose of We denote the optimal solution of the second-phase ALP by We now demonstrate how the state-relevance weights and can be used for controlling the quality of the approximation to the differential cost function We first define, for any

6 Y S given, the function, given by the unique solution to [3] 6 W If is our estimate for the optimal average cost, then the differential cost function * the difference between and, when can be seen as an estimate to Our first result " links the difference between and to For simplicity of notation, we implicitly drop from all vectors and matrices rows and columns corresponding to state, so that, for instance, corresponds to the original vector without the row corresponding to state, and corresponds to the original matrix without rows and columns corresponding to state Lemma For all, we have Proof: Equation (5), satisfied by, corresponds to Bellman s equation forthe problem of finding the stochastic shortest path to state, when costs are given by [3] Hence corresponds to the vector of smallest expected lengths of paths until state It Note that if " ", we also have, In the following theorem, we show that the second-phase ALP minimizes N over the feasible region The weighted norm, which will be used in the remainder of the paper, is defined as &, for any Theorem 1 Let be the optimal solution to the second-phase ALP Then it minimizes N over the feasible region of the second-phase ALP Proof: Maximizing N is equivalent to minimizing N It is a well-known result that, for all such that we have It follows that N over the feasible region of the second-phase ALP, and N minimizes N N N For any fixed choice of satisfying N N, we hence the second-phase ALP minimizes an upper bound on the weighted norm N of the error in the differential cost function approximation Note that state-relevance weights determine how errors over different portions of the state space are weighted in the decision of which approximate differential cost function to select, and can be used for balancing accuracy of the approximation over different states In the next section, we will provide performance bounds that tie a certain norm of the difference between and N to the expect increase in cost incurred by using the greedy policy with respect to N This demonstrates that the objective optimized by the second-phase ALP is compatible with the objective of optimizing performance of the policy being obtained, and it also provides some insight about appropriate choices of state-relevance weights We have not yet specified how to choose An obvious choice is F estimate " for the optimal average cost yielded by the first-phase ALP and it (5) (6), since is the, so that bound (6) holds In practice, it may be advantageous to perform a line search

7 over to optimize performance of the ultimate policy being generated An important issue is the feasibility of the second-phase ALP will be feasible for a given choice of ; for, this will always be the case It can also be shown that, under certain conditions on the basis functions N, the second-phase ALP possesses multiple feasible solutions regardless of the choice of 6 A performance bound In this section, we present a bound on the performance of greedy policies associated with approximate differential cost functions This bound provide some guidance on appropriate choices for state-relevance weights Theorem Let Assumption 1 hold For all, let and denote the average cost and stationary state distribution of the greedy policy associated with Then, for all such that, we have Proof: We have 6, where and denote the costs and transition matrix associated with the greedy policy with respect to, and we have used in the first equality Now if, we have 6 6 " ( Theorem suggests that one approach to selecting state-relevance weights may be to run the second-phase ALP adaptively, using in each iteration weights corresponding to the stationary state distribution associated with the policy generated by the previous iteration Alternatively, in some cases it may suffice to use rough guesses about the stationary state distribution of the MDP as choices for the state-relevance weights We revisit the example from Section 4 to illustrate this idea Example 1 Consider applying the second-phase ALP to the controlled queue described in Section 4 We use weights of the form & This is similar to what is done in [6] and is motivated by the fact that, if the system runs under a stabilizing policy, there are exponential lower and upper bounds to the stationary state distribution [5] S Hence is a reasonable guess for the shape of the stationary distribution We also let Figure 1 demonstrates the evolution of N as we increase Note that there is significant improvement in the shape of N relative to N The best policy is obtained for, and incurs an average cost of approximately, regardless of the buffer size This cost is only about higher than the optimal average cost Conclusions We have extended the analysis of ALP to the case of minimization of average costs We have shown how the ALP version commonly found in the literature may lead to arbitrarily bad policies even if the choice of basis functions is relatively good; the main problem is that this version of the algorithm the first-phase ALP prioritizes approximation of the optimal average cost, but does not necessarily yield a good approximation for the differential cost function We propose a variant of approximate linear programming the two-phase approximate linear programming method that explicitly approximates the differential cost function The main attractive of the algorithm is the presence of staterelevance weights, which can be used for controlling the relative accuracy of the differential cost function approximation over different portions of the state space Many open issues must still be addressed Perhaps most important of all is whether there is an automatic way of choosing state-relevance weights The performance bound suggest in Theorem suggests an iterative scheme, where the second-phase ALP is run multiple

8 1 5 x 1 6 h Φr (ρ=) Φr (ρ=8) Φr (ρ=) Φr Figure 1: Controlled queue example: Differential cost function approximations as a function of From top to bottom, differential cost function, approximations N (with ), and approximation N times state-relevance weights are updated in each iteration according to the stationary state distribution obtained with the policy generated by the algorithm in the previous iteration It remains to be shown whether such a scheme converges It is also important to note that, in principle, Theorem holds only for If (, this condition cannot be verified for N, and the appropriateness of minimizing N is only speculative References [1] D Adelman A price-directed approach to stochastic inventory/routing Preprint, [] D Adelman Price-directed replenishment of subsets: Methodology and its application to inventory routing Preprint, [3] D Bertsekas Dynamic Programming and Optimal Control Athena Scientific, 15 [4] D Bertsekas and JN Tsitsiklis Neuro-Dynamic Programming Athena Scientific, 16 [5] D Bertsimas, D Gamarnik, and JN Tsitsiklis Performance of multiclass Markovian queueing networks via piecewise linear Lyapunov functions Annals of Applied Probability, 11 [6] DP de Farias and B Van Roy The linear programming approach to approximate dynamic programming To appear in Operations Research, 1 [] DP de Farias and B Van Roy On constraint sampling in the linear programming approach to approximate dynamic programming Conditionally accepted to Mathematics of Operations Research, 1 [8] C Guestrin, D Koller, and R Parr Efficient solution algorithms for factored MDPs Submitted to Journal of Artificial Intelligence Research, 1 [] AS Manne Linear programming and sequential decisions Management Science, 6(3):5 6, 16 [1] JR Morrison and PR Kumar New linear program performance bounds for queueing networks Journal of Optimization Theory and Applications, 1(3):55 5, 1 [11] P Schweitzer and A Seidmann Generalized polynomial approximations in Markovian decision processes Journal of Mathematical Analysis and Applications, 11:568 58, 185

15-780: MarkovDecisionProcesses

15-780: MarkovDecisionProcesses J. Zico Kolter Feburary 29, 2016 1 Outline Introduction Formal definition Value iteration Policy iteration Linear programming for MDPs 2 1988 Judea Pearl publishes Probabilistic

Reinforcement Learning: A brief introduction. Mihaela van der Schaar

Reinforcement Learning: A brief introduction Mihaela van der Schaar Outline Optimal Decisions & Optimal Forecasts Markov Decision Processes (MDPs) States, actions, rewards and value functions Dynamic Programming

Solving Factored POMDPs with Linear Value Functions

IJCAI-01 workshop on Planning under Uncertainty and Incomplete Information (PRO-2), pp. 67-75, Seattle, Washington, August 2001. Solving Factored POMDPs with Linear Value Functions Carlos Guestrin Computer

3 No-Wait Job Shops with Variable Processing Times

3 No-Wait Job Shops with Variable Processing Times In this chapter we assume that, on top of the classical no-wait job shop setting, we are given a set of processing times for each operation. We may select

NP-Hardness. We start by defining types of problem, and then move on to defining the polynomial-time reductions.

CS 787: Advanced Algorithms NP-Hardness Instructor: Dieter van Melkebeek We review the concept of polynomial-time reductions, define various classes of problems including NP-complete, and show that 3-SAT

The Encoding Complexity of Network Coding

The Encoding Complexity of Network Coding Michael Langberg Alexander Sprintson Jehoshua Bruck California Institute of Technology Email: mikel,spalex,bruck @caltech.edu Abstract In the multicast network

Bilinear Programming

Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu

Wireless Multicast: Theory and Approaches

University of Pennsylvania ScholarlyCommons Departmental Papers (ESE) Department of Electrical & Systems Engineering June 2005 Wireless Multicast: Theory Approaches Prasanna Chaporkar University of Pennsylvania

On the Complexity of the Policy Improvement Algorithm. for Markov Decision Processes

On the Complexity of the Policy Improvement Algorithm for Markov Decision Processes Mary Melekopoglou Anne Condon Computer Sciences Department University of Wisconsin - Madison 0 West Dayton Street Madison,

Rollout Algorithms for Discrete Optimization: A Survey

Rollout Algorithms for Discrete Optimization: A Survey by Dimitri P. Bertsekas Massachusetts Institute of Technology Cambridge, MA 02139 dimitrib@mit.edu August 2010 Abstract This chapter discusses rollout

5 MST and Greedy Algorithms

5 MST and Greedy Algorithms One of the traditional and practically motivated problems of discrete optimization asks for a minimal interconnection of a given set of terminals (meaning that every pair will

CSE151 Assignment 2 Markov Decision Processes in the Grid World

CSE5 Assignment Markov Decision Processes in the Grid World Grace Lin A484 gclin@ucsd.edu Tom Maddock A55645 tmaddock@ucsd.edu Abstract Markov decision processes exemplify sequential problems, which are

Iterative Improvement Algorithm design technique for solving optimization problems Start with a feasible solution Repeat the following step until no improvement can be found: change the current feasible

Delay-minimal Transmission for Energy Constrained Wireless Communications

Delay-minimal Transmission for Energy Constrained Wireless Communications Jing Yang Sennur Ulukus Department of Electrical and Computer Engineering University of Maryland, College Park, M0742 yangjing@umd.edu

Module 1 Lecture Notes 2. Optimization Problem and Model Formulation

Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization

Integer Programming ISE 418. Lecture 1. Dr. Ted Ralphs

Integer Programming ISE 418 Lecture 1 Dr. Ted Ralphs ISE 418 Lecture 1 1 Reading for This Lecture N&W Sections I.1.1-I.1.4 Wolsey Chapter 1 CCZ Chapter 2 ISE 418 Lecture 1 2 Mathematical Optimization Problems

5 MST and Greedy Algorithms

5 MST and Greedy Algorithms One of the traditional and practically motivated problems of discrete optimization asks for a minimal interconnection of a given set of terminals (meaning that every pair will

Medium Access Control Protocols With Memory Jaeok Park, Member, IEEE, and Mihaela van der Schaar, Fellow, IEEE

IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 18, NO. 6, DECEMBER 2010 1921 Medium Access Control Protocols With Memory Jaeok Park, Member, IEEE, and Mihaela van der Schaar, Fellow, IEEE Abstract Many existing

6. Lecture notes on matroid intersection

Massachusetts Institute of Technology 18.453: Combinatorial Optimization Michel X. Goemans May 2, 2017 6. Lecture notes on matroid intersection One nice feature about matroids is that a simple greedy algorithm

The complement of PATH is in NL

340 The complement of PATH is in NL Let c be the number of nodes in graph G that are reachable from s We assume that c is provided as an input to M Given G, s, t, and c the machine M operates as follows:

Formal Model. Figure 1: The target concept T is a subset of the concept S = [0, 1]. The search agent needs to search S for a point in T.

Although this paper analyzes shaping with respect to its benefits on search problems, the reader should recognize that shaping is often intimately related to reinforcement learning. The objective in reinforcement

Queuing Networks. Renato Lo Cigno. Simulation and Performance Evaluation Queuing Networks - Renato Lo Cigno 1

Queuing Networks Renato Lo Cigno Simulation and Performance Evaluation 2014-15 Queuing Networks - Renato Lo Cigno 1 Moving between Queues Queuing Networks - Renato Lo Cigno - Interconnecting Queues 2 Moving

Recent Developments in Model-based Derivative-free Optimization

Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints:

Discrete Optimization. Lecture Notes 2

Discrete Optimization. Lecture Notes 2 Disjunctive Constraints Defining variables and formulating linear constraints can be straightforward or more sophisticated, depending on the problem structure. The

Rollout Algorithms for Stochastic Scheduling Problems

Journal of Heuristics, 5, 89 108 (1999) c 1999 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands. Rollout Algorithms for Stochastic Scheduling Problems DIMITRI P. BERTSEKAS* Department

Topology and Topological Spaces

Topology and Topological Spaces Mathematical spaces such as vector spaces, normed vector spaces (Banach spaces), and metric spaces are generalizations of ideas that are familiar in R or in R n. For example,

Asynchronous Distributed Optimization With Event-Driven Communication Minyi Zhong, Student Member, IEEE, and Christos G. Cassandras, Fellow, IEEE

IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 55, NO. 12, DECEMBER 2010 2735 Asynchronous Distributed Optimization With Event-Driven Communication Minyi Zhong, Student Member, IEEE, Christos G. Cassras,

arxiv: v1 [math.co] 25 Sep 2015

A BASIS FOR SLICING BIRKHOFF POLYTOPES TREVOR GLYNN arxiv:1509.07597v1 [math.co] 25 Sep 2015 Abstract. We present a change of basis that may allow more efficient calculation of the volumes of Birkhoff

One-mode Additive Clustering of Multiway Data

One-mode Additive Clustering of Multiway Data Dirk Depril and Iven Van Mechelen KULeuven Tiensestraat 103 3000 Leuven, Belgium (e-mail: dirk.depril@psy.kuleuven.ac.be iven.vanmechelen@psy.kuleuven.ac.be)

AC : USING A SCRIPTING LANGUAGE FOR DYNAMIC PROGRAMMING

AC 2008-2623: USING A SCRIPTING LANGUAGE FOR DYNAMIC PROGRAMMING Louis Plebani, Lehigh University American Society for Engineering Education, 2008 Page 13.1325.1 Using a Scripting Language for Dynamic

HARNESSING CERTAINTY TO SPEED TASK-ALLOCATION ALGORITHMS FOR MULTI-ROBOT SYSTEMS

HARNESSING CERTAINTY TO SPEED TASK-ALLOCATION ALGORITHMS FOR MULTI-ROBOT SYSTEMS An Undergraduate Research Scholars Thesis by DENISE IRVIN Submitted to the Undergraduate Research Scholars program at Texas

The Encoding Complexity of Network Coding

The Encoding Complexity of Network Coding Michael Langberg Alexander Sprintson Jehoshua Bruck California Institute of Technology Email mikel,spalex,bruck @caltech.edu Abstract In the multicast network

MA4254: Discrete Optimization. Defeng Sun. Department of Mathematics National University of Singapore Office: S Telephone:

MA4254: Discrete Optimization Defeng Sun Department of Mathematics National University of Singapore Office: S14-04-25 Telephone: 6516 3343 Aims/Objectives: Discrete optimization deals with problems of

15-451/651: Design & Analysis of Algorithms November 4, 2015 Lecture #18 last changed: November 22, 2015

15-451/651: Design & Analysis of Algorithms November 4, 2015 Lecture #18 last changed: November 22, 2015 While we have good algorithms for many optimization problems, the previous lecture showed that many

DEGENERACY AND THE FUNDAMENTAL THEOREM

DEGENERACY AND THE FUNDAMENTAL THEOREM The Standard Simplex Method in Matrix Notation: we start with the standard form of the linear program in matrix notation: (SLP) m n we assume (SLP) is feasible, and

Math 5593 Linear Programming Lecture Notes

Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................

Rollout Algorithms for Constrained Dynamic Programming 1

April 2005 Report LIDS 2646 Rollout Algorithms for Constrained Dynamic Programming 1 by Dimitri P. Bertsekas 2 Abstract The rollout algorithm is a suboptimal control method for deterministic and stochastic

Approximation Algorithms

Approximation Algorithms Group Members: 1. Geng Xue (A0095628R) 2. Cai Jingli (A0095623B) 3. Xing Zhe (A0095644W) 4. Zhu Xiaolu (A0109657W) 5. Wang Zixiao (A0095670X) 6. Jiao Qing (A0095637R) 7. Zhang

10. Network dimensioning

Partly based on slide material by Samuli Aalto and Jorma Virtamo ELEC-C7210 Modeling and analysis of communication networks 1 Contents Introduction Parameters: topology, routing and traffic Dimensioning

Lecture notes on Transportation and Assignment Problem (BBE (H) QTM paper of Delhi University)

Transportation and Assignment Problems The transportation model is a special class of linear programs. It received this name because many of its applications involve determining how to optimally transport

1 Random Walks on Graphs

Lecture 7 Com S 633: Randomness in Computation Scribe: Ankit Agrawal In the last lecture, we looked at random walks on line and used them to devise randomized algorithms for 2-SAT and 3-SAT For 2-SAT we

Page rank computation HPC course project a.y Compute efficient and scalable Pagerank

Page rank computation HPC course project a.y. 2012-13 Compute efficient and scalable Pagerank 1 PageRank PageRank is a link analysis algorithm, named after Brin & Page [1], and used by the Google Internet

Throughput Maximization for Energy Efficient Multi-Node Communications using Actor-Critic Approach

Throughput Maximization for Energy Efficient Multi-Node Communications using Actor-Critic Approach Charles Pandana and K. J. Ray Liu Department of Electrical and Computer Engineering University of Maryland,

Stochastic Processing Networks: What, Why and How? Ruth J. Williams University of California, San Diego

Stochastic Processing Networks: What, Why and How? Ruth J. Williams University of California, San Diego http://www.math.ucsd.edu/~williams 1 OUTLINE! What is a Stochastic Processing Network?! Applications!

CSc 545 Lecture topic: The Criss-Cross method of Linear Programming

CSc 545 Lecture topic: The Criss-Cross method of Linear Programming Wanda B. Boyer University of Victoria November 21, 2012 Presentation Outline 1 Outline 2 3 4 Please note: I would be extremely grateful

In this chapter we introduce some of the basic concepts that will be useful for the study of integer programming problems.

2 Basics In this chapter we introduce some of the basic concepts that will be useful for the study of integer programming problems. 2.1 Notation Let A R m n be a matrix with row index set M = {1,...,m}

Online Facility Location

Online Facility Location Adam Meyerson Abstract We consider the online variant of facility location, in which demand points arrive one at a time and we must maintain a set of facilities to service these

Multi-Cluster Interleaving on Paths and Cycles

Multi-Cluster Interleaving on Paths and Cycles Anxiao (Andrew) Jiang, Member, IEEE, Jehoshua Bruck, Fellow, IEEE Abstract Interleaving codewords is an important method not only for combatting burst-errors,

A BRIEF INTRODUCTION TO DYNAMIC PROGRAMMING (DP) by Amarnath Kasibhatla Nanocad Lab University of California, Los Angeles 04/21/2010

A BRIEF INTRODUCTION TO DYNAMIC PROGRAMMING (DP) by Amarnath Kasibhatla Nanocad Lab University of California, Los Angeles 04/21/2010 Overview What is DP? Characteristics of DP Formulation Examples Disadvantages

The Geometry of Carpentry and Joinery

The Geometry of Carpentry and Joinery Pat Morin and Jason Morrison School of Computer Science, Carleton University, 115 Colonel By Drive Ottawa, Ontario, CANADA K1S 5B6 Abstract In this paper we propose

Chapter 2. Related Work

Chapter 2 Related Work There are three areas of research highly related to our exploration in this dissertation, namely sequential pattern mining, multiple alignment, and approximate frequent pattern mining.

An Eternal Domination Problem in Grids

Theory and Applications of Graphs Volume Issue 1 Article 2 2017 An Eternal Domination Problem in Grids William Klostermeyer University of North Florida, klostermeyer@hotmail.com Margaret-Ellen Messinger

On the Max Coloring Problem

On the Max Coloring Problem Leah Epstein Asaf Levin May 22, 2010 Abstract We consider max coloring on hereditary graph classes. The problem is defined as follows. Given a graph G = (V, E) and positive

Distributed Alternating Direction Method of Multipliers

Distributed Alternating Direction Method of Multipliers Ermin Wei and Asuman Ozdaglar Abstract We consider a network of agents that are cooperatively solving a global unconstrained optimization problem,

Steiner Trees and Forests

Massachusetts Institute of Technology Lecturer: Adriana Lopez 18.434: Seminar in Theoretical Computer Science March 7, 2006 Steiner Trees and Forests 1 Steiner Tree Problem Given an undirected graph G

The critical node problem in stochastic networks with discrete-time Markov chain

Croatian Operational Research Review 33 CRORR 7(206), 33 46 The critical node problem in stochastic networks with discrete-time Markov chain Gholam Hassan Shirdel,* and Mohsen Abdolhosseinzadeh Department

Chapter 18 out of 37 from Discrete Mathematics for Neophytes: Number Theory, Probability, Algorithms, and Other Stuff by J. M. Cargal.

Chapter 8 out of 7 from Discrete Mathematics for Neophytes: Number Theory, Probability, Algorithms, and Other Stuff by J. M. Cargal 8 Matrices Definitions and Basic Operations Matrix algebra is also known

Evolutionary Computation Algorithms for Cryptanalysis: A Study

Evolutionary Computation Algorithms for Cryptanalysis: A Study Poonam Garg Information Technology and Management Dept. Institute of Management Technology Ghaziabad, India pgarg@imt.edu Abstract The cryptanalysis

CS261: Problem Set #2

CS261: Problem Set #2 Due by 11:59 PM on Tuesday, February 9, 2016 Instructions: (1) Form a group of 1-3 students. You should turn in only one write-up for your entire group. (2) Submission instructions:

Polynomial-Time Approximation Algorithms

6.854 Advanced Algorithms Lecture 20: 10/27/2006 Lecturer: David Karger Scribes: Matt Doherty, John Nham, Sergiy Sidenko, David Schultz Polynomial-Time Approximation Algorithms NP-hard problems are a vast

Near-optimal switching strategies for a tandem queue

Near-optimal switching strategies for a tandem queue Daphne van Leeuwen, Rudesindo Núñez-Queija December 23, 2015 Abstract Motivated by various applications in logistics, road traffic and production management,

Apprenticeship Learning for Reinforcement Learning. with application to RC helicopter flight Ritwik Anand, Nick Haliday, Audrey Huang

Apprenticeship Learning for Reinforcement Learning with application to RC helicopter flight Ritwik Anand, Nick Haliday, Audrey Huang Table of Contents Introduction Theory Autonomous helicopter control

The simplex method and the diameter of a 0-1 polytope

The simplex method and the diameter of a 0-1 polytope Tomonari Kitahara and Shinji Mizuno May 2012 Abstract We will derive two main results related to the primal simplex method for an LP on a 0-1 polytope.

Convexity. 1 X i is convex. = b is a hyperplane in R n, and is denoted H(p, b) i.e.,

Convexity We ll assume throughout, without always saying so, that we re in the finite-dimensional Euclidean vector space R n, although sometimes, for statements that hold in any vector space, we ll say

Stability of Networks and Protocols in the Adversarial Queueing. Model for Packet Routing. Ashish Goel. December 1, Abstract

Stability of Networks and Protocols in the Adversarial Queueing Model for Packet Routing Ashish Goel University of Southern California December 1, 2000 Abstract The adversarial queueing theory model for

Heuristic Search in Cyclic AND/OR Graphs

From: AAAI-98 Proceedings. Copyright 1998, AAAI (www.aaai.org). All rights reserved. Heuristic Search in Cyclic AND/OR Graphs Eric A. Hansen and Shlomo Zilberstein Computer Science Department University

Introduction to Mathematical Programming IE406. Lecture 20. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406 Lecture 20 Dr. Ted Ralphs IE406 Lecture 20 1 Reading for This Lecture Bertsimas Sections 10.1, 11.4 IE406 Lecture 20 2 Integer Linear Programming An integer

An Efficient Approximation for the Generalized Assignment Problem

An Efficient Approximation for the Generalized Assignment Problem Reuven Cohen Liran Katzir Danny Raz Department of Computer Science Technion Haifa 32000, Israel Abstract We present a simple family of

Single Machine Scheduling with Interfering Job Sets. Arizona State University PO Box Tempe, AZ

Single Machine Scheduling with Interfering Job Sets Ketan Khowala 1,3, John Fowler 1,3, Ahmet Keha 1* and Hari Balasubramanian 2 1 Department of Industrial Engineering Arizona State University PO Box 875906

L2: Algorithms: Knapsack Problem & BnB

L2: Algorithms: Knapsack Problem & BnB This tutorial covers the basic topics on creating a forms application, common form controls and the user interface for the optimization models, algorithms and heuristics,

9.5 Equivalence Relations

9.5 Equivalence Relations You know from your early study of fractions that each fraction has many equivalent forms. For example, 2, 2 4, 3 6, 2, 3 6, 5 30,... are all different ways to represent the same

Markov Decision Processes and Reinforcement Learning

Lecture 14 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Stuart Russell and Peter Norvig Course Overview Introduction Artificial Intelligence

Topic: Local Search: Max-Cut, Facility Location Date: 2/13/2007

CS880: Approximations Algorithms Scribe: Chi Man Liu Lecturer: Shuchi Chawla Topic: Local Search: Max-Cut, Facility Location Date: 2/3/2007 In previous lectures we saw how dynamic programming could be

11.1 Facility Location

CS787: Advanced Algorithms Scribe: Amanda Burton, Leah Kluegel Lecturer: Shuchi Chawla Topic: Facility Location ctd., Linear Programming Date: October 8, 2007 Today we conclude the discussion of local

CHAPTER 3 A TIME-DEPENDENT k-shortest PATH ALGORITHM FOR ATIS APPLICATIONS

CHAPTER 3 A TIME-DEPENDENT k-shortest PATH ALGORITHM FOR ATIS APPLICATIONS 3.1. Extension of a Static k-sp Algorithm to the Time-Dependent Case Kaufman and Smith [1993] showed that under the consistency

Worst-case Ethernet Network Latency for Shaped Sources

Worst-case Ethernet Network Latency for Shaped Sources Max Azarov, SMSC 7th October 2005 Contents For 802.3 ResE study group 1 Worst-case latency theorem 1 1.1 Assumptions.............................

Optimality certificates for convex minimization and Helly numbers

Optimality certificates for convex minimization and Helly numbers Amitabh Basu Michele Conforti Gérard Cornuéjols Robert Weismantel Stefan Weltge October 20, 2016 Abstract We consider the problem of minimizing

10703 Deep Reinforcement Learning and Control

10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient I Used Materials Disclaimer: Much of the material and slides for this lecture

A Deterministic Dynamic Programming Approach for Optimization Problem with Quadratic Objective Function and Linear Constraints

A Deterministic Dynamic Programming Approach for Optimization Problem with Quadratic Objective Function and Linear Constraints S. Kavitha, Nirmala P. Ratchagar International Science Index, Mathematical

High Dimensional Indexing by Clustering

Yufei Tao ITEE University of Queensland Recall that, our discussion so far has assumed that the dimensionality d is moderately high, such that it can be regarded as a constant. This means that d should

Optimal Path Finding for Direction, Location and Time Dependent Costs, with Application to Vessel Routing

1 Optimal Path Finding for Direction, Location and Time Dependent Costs, with Application to Vessel Routing Irina S. Dolinskaya Department of Industrial Engineering and Management Sciences Northwestern

Figure : Example Precedence Graph

CS787: Advanced Algorithms Topic: Scheduling with Precedence Constraints Presenter(s): James Jolly, Pratima Kolan 17.5.1 Motivation 17.5.1.1 Objective Consider the problem of scheduling a collection of

Rigidity, connectivity and graph decompositions

First Prev Next Last Rigidity, connectivity and graph decompositions Brigitte Servatius Herman Servatius Worcester Polytechnic Institute Page 1 of 100 First Prev Next Last Page 2 of 100 We say that a framework

Introduction to Modeling. Lecture Overview

Lecture Overview What is a Model? Uses of Modeling The Modeling Process Pose the Question Define the Abstractions Create the Model Analyze the Data Model Representations * Queuing Models * Petri Nets *

THE preceding chapters were all devoted to the analysis of images and signals which

Chapter 5 Segmentation of Color, Texture, and Orientation Images THE preceding chapters were all devoted to the analysis of images and signals which take values in IR. It is often necessary, however, to

FMA901F: Machine Learning Lecture 6: Graphical Models. Cristian Sminchisescu

FMA901F: Machine Learning Lecture 6: Graphical Models Cristian Sminchisescu Graphical Models Provide a simple way to visualize the structure of a probabilistic model and can be used to design and motivate

Math 340 Fall 2014, Victor Matveev. Binary system, round-off errors, loss of significance, and double precision accuracy.

Math 340 Fall 2014, Victor Matveev Binary system, round-off errors, loss of significance, and double precision accuracy. 1. Bits and the binary number system A bit is one digit in a binary representation

Linear programming and duality theory

Linear programming and duality theory Complements of Operations Research Giovanni Righini Linear Programming (LP) A linear program is defined by linear constraints, a linear objective function. Its variables

OPTICAL NETWORKS. Virtual Topology Design. A. Gençata İTÜ, Dept. Computer Engineering 2005

OPTICAL NETWORKS Virtual Topology Design A. Gençata İTÜ, Dept. Computer Engineering 2005 Virtual Topology A lightpath provides single-hop communication between any two nodes, which could be far apart in

Optimal Control of a Production-Inventory System with both Backorders and Lost Sales

Optimal Control of a Production-Inventory System with both Backorders and Lost Sales Saif Benjaafar Mohsen ElHafsi 2 Tingliang Huang 3 Industrial & Systems Engineering, Department of Mechanical Engineering,

Empirical analysis of procedures that schedule unit length jobs subject to precedence constraints forming in- and out-stars

Empirical analysis of procedures that schedule unit length jobs subject to precedence constraints forming in- and out-stars Samuel Tigistu Feder * Abstract This paper addresses the problem of scheduling

Anytime Coordination Using Separable Bilinear Programs

Anytime Coordination Using Separable Bilinear Programs Marek Petrik and Shlomo Zilberstein Department of Computer Science University of Massachusetts Amherst, MA 01003 {petrik, shlomo}@cs.umass.edu Abstract

The Running Time of Programs

The Running Time of Programs The 90 10 Rule Many programs exhibit the property that most of their running time is spent in a small fraction of the source code. There is an informal rule that states 90%

10. EXTENDING TRACTABILITY

0. EXTENDING TRACTABILITY finding small vertex covers solving NP-hard problems on trees circular arc coverings vertex cover in bipartite graphs Lecture slides by Kevin Wayne Copyright 005 Pearson-Addison

1344 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 42, NO. 10, OCTOBER 1997

1344 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 42, NO. 10, OCTOBER 1997 Polynomial-Complexity Deadlock Avoidance Policies for Sequential Resource Allocation Systems Spiridon A. Reveliotis, Member, IEEE,

1.1 What is Microeconomics?

1.1 What is Microeconomics? Economics is the study of allocating limited resources to satisfy unlimited wants. Such a tension implies tradeoffs among competing goals. The analysis can be carried out at

Implementation Techniques

V Implementation Techniques 34 Efficient Evaluation of the Valid-Time Natural Join 35 Efficient Differential Timeslice Computation 36 R-Tree Based Indexing of Now-Relative Bitemporal Data 37 Light-Weight

Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity