Gradient Reinforcement Learning of POMDP Policy Graphs
|
|
|
- Joan Crawford
- 5 years ago
- Views:
Transcription
1 1 Gradient Reinforcement Learning of POMDP Policy Graphs Douglas Aberdeen Research School of Information Science and Engineering Australian National University Jonathan Baxter WhizBang! Labs July 23, 2001 CMU-ML Talk, 23 July 2001
2 2 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
3 3 Goals Develop scalable RL algorithms that learn near optimal controls for POMDPs without prior knowledge of the model. This is hard! Demonstrate these algorithms on a large scale, real world problems: speech processing; robot navigation.
4 4 A POMDP Environment y1 y2 y3 P(u=0) P(u=1) Agent
5 5 Historical Perspective I Bellman s Equation Richard Bellman (1957) Describes equations with unknowns ( = states). Model must be known. This formulation is for MDPs only. Intractable for more than a few tens of states.
6 6 Historical Perspective II Policy Iteration Bellman (1957) and Howard (1960) Finds a solution to the Bellman equation via dynamic programming. Practical for much larger state spaces. Related method: value iteration. Function approximation for RL in use by 1965 (Waltz and Fu 1965).
7 7 Historical Perspective III Simulated Methods Do not require the environment model. They learn from experience. Q-learning (Watkin s 1989). Eligibility traces: TD( ) (Sutton 1988).
8 8 Historical Perspective IV Policy Gradient Methods Learns the policy directly. Nice convergence properties, even for function approximators. Variance in the gradient estimates is a problem. REINFORCE (Williams 1992). GPOMDP (Baxter & Bartlett 1999). Hybrids: VAPS (Baird & Moore 1999).
9 9 Historical Perspective V Exact POMDP methods Aström (1965), Sondik (1971) Re-introduces the environment model. Modified Bellman equation computes the value of belief states. At least PSpace-complete so approximate methods are needed. Controlling POMDPs sans model, with infinite state and action spaces, is about as general as it gets.
10 10 Failings of current methods The drawbacks of current approximate POMDP methods include: Assumption of a model of the environment. Only recalling events finitely far into the past. Use of an independent internal state model that does not aim to maximise the long term reward. Do not easily generalize to continuous observations and actions. Applications to toy problems only.
11 11 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
12 12 The GPOMDP algorithm GPOMDP is a policy gradient approach to reinforcement learning. GPOMDP is an algorithm for of estimating the gradient of with respect to the parameters of the policy. Estimates the infinite horizon average reward gradient by using a parameter which is equivalent to discounting. Computes.!#"/$'&)(0*,".-!#"%$'&)(+*,".-.1 Works for POMDP environments if observations are belief states. Similar to REINFORCE (Williams 1992) and VAPS (Baird & Moore 1999), (Marbach & Tsitsiklis 1999).
13 13 Environment y1 y2 y3 P(u=0) P(u=1) Agent
14 14 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
15 15 GPOMDP with memory GPOMDP implements a memoryless controller, which is not always sufficient r = U L (Peshkin, Meuleau, Kaebling 1999) GPOMDP has been extended with I-states. 1 1 gives the next I-state probabilities. gives action probabilities. GPOMDP computes the gradient w.r.t and independently.
16 16 & Environment % "#$ ('! Agent
17 17 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
18 18 r = U L NULL U NULL Left Right L L U Policy graph learnt for the Load/Unload problem.
19 Average reward vs iterations std.dev Reward Iterations Convergence of the Load/Unload problem using 4 I-States. (Averaged over 100 runs.)
20 20 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
21 21 Related Work Use HMMs to learn the model (Chrisman 1992). Recurrent Neural Networks (Lin & Mitchell 1992). Differentiable approx. to piecewise function (Parr & Russell 1995). U-Tree s: Dynamic finite history windows (McCallum 1996). External memory setting actions (Peshkin, Meuleau, Kaebling 1999).
22 22 Pros of GPOMDP with I-states Converges to the optimal policy that can be learnt with I-states. Does not require a model of the POMDP. I-states can remember occurrences infinitely far into the past. Works with continuous state and action spaces. Theoretically scales to large problems.
23 23 Cons of GPOMDP with I-states 1. GPOMDP has a large variance as. 2. I-states increase the mixing time of the overall system. Importance Sampling; replace with an MDP alg. that works on the I-states; eligibility trace filtering to incorporate prior knowledge; deterministic Internal states are initially undifferentiated, resulting in. Define a sparse internal finite state machine.
24 24 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP The Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
25 25 Undifferentiated I-states I [(A,0), (B,1)] 1 [(B,0), (B,1)] r=1 Internal states 2 r= 1 Figure 1: Possible I-state trajectories for observation/action trajectories.
26 26 Undifferentiated I-states II [(A,0), (B,1)] 1 [(B,0), (B,1)] r=1 Internal states 2 r= 1 Figure 2: Alternate, equally likely, I-state trajectories.
27 27 Sparse transitions for I-states [(A,0), (B,1)] 1 [(B,0), (B,1)] r=1 2 r= 1 Figure 3: Reduced number of possible I-state trajectories.
28 28 Signpost problem description r= 1 r=1 r=1 r= 1 Figure 4: Simplified Heaven-Hell problem. Agent must visit lower state to determine which way to move at the top of the T (Thrun 2000), (Geffner & Bonet 1998).
29 29 I-states Trajectory Probabilities I state trajectory probabilities: uniform fully connected FSM 1500 I state trajectory probabilities: random fully connected FSM Trajectories sampled under path A Same trajectories under path B max theta = Trajectory count Trajectory count Trajectory probability Trajectory log probability (a) (b) Figure 5: Histogram of probabilities of 10,000 I-state trajectories sampled from the signpost problem. Shown for 2 sets of observations.
30 30 Sparse transitions for I-states 1400 I state trajectory probabilities: random degree 5 FSM Trajectories sampled under path A Same trajectories under path B max weight = 1 Trajectory count non 0, B Trajectory log probability Figure 6: I-state trajectory histograms for sparse I-state transitions.
31 31 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP Heaven-Hell problem (?) Using prior knowledge to reduce gradient variance
32 32 Outline Motivation GPOMDP, a policy gradient RL algorithm GPOMDP with I-state The Load-Unload problem Related Work Pros and Cons of GPOMDP with I-state Repairing I-state GPOMDP Heaven-Hell problem Using prior knowledge to reduce gradient variance
33 33 A simple POMDP? 0 p r=10 p r=11 p r=12 Figure 7: A UMDP which requires optimally. for GPOMDP to learn to act
34 .1 34 GPOMDP Eligibility Trace Update 1 1.1
35 35 Standard discounting 1 Impulse response of standard discounting β = Credit assigned τ
36 36 0 p r=10 p r=11 p r=12 We know the minimum delays from key action until rewards are issued.
37 37 Alternative filter I 1 Impulse Response of optimal filter for p= Output τ Figure 8: A bias optimal FIR filter for.
38 38 Alternative filter II 1.6 Impulse response for IIR filter, p= output τ Figure 9: A good IIR filter for.
39 39 Arbitrary IIR Trace Filter 1.1 $ $.1 $ $
40 40 Results Trace type Test I Test II Bias var Bias var FIR 7.72 IIR Table 1: Results of eligibility trace filtering tests. Note reduced variance of the filtered traces.
41 41 Key Conclusions I It is possible to perform a search for the optimal policy graph directly. II RL algorithms can be extended with I-states to perform this search. III A tough problem has been solved, using the sparse initialization trick to avoid the problem of low initial gradients. IV We can use eligibility trace filtering to add prior knowledge and hence reduce the gradient estimate variance.
42 42 Future Work I-state GPOMDP for larger problems from the literature. I-state GPOMDP for speech processing. I-state trained using EM like algorithm. Bounds on policy error introduced by too few I-states. Automatic selection of.
43 43 Acknowledgments Peter Bartlett Sebastian Thrun Questions?
A (Revised) Survey of Approximate Methods for Solving Partially Observable Markov Decision Processes
A (Revised) Survey of Approximate Methods for Solving Partially Observable Markov Decision Processes Douglas Aberdeen National ICT Australia, Canberra, Australia. December 8, 2003 Abstract Partially observable
Markov Decision Processes. (Slides from Mausam)
Markov Decision Processes (Slides from Mausam) Machine Learning Operations Research Graph Theory Control Theory Markov Decision Process Economics Robotics Artificial Intelligence Neuroscience /Psychology
Planning and Control: Markov Decision Processes
CSE-571 AI-based Mobile Robotics Planning and Control: Markov Decision Processes Planning Static vs. Dynamic Predictable vs. Unpredictable Fully vs. Partially Observable Perfect vs. Noisy Environment What
Reinforcement Learning (INF11010) Lecture 6: Dynamic Programming for Reinforcement Learning (extended)
Reinforcement Learning (INF11010) Lecture 6: Dynamic Programming for Reinforcement Learning (extended) Pavlos Andreadis, February 2 nd 2018 1 Markov Decision Processes A finite Markov Decision Process
Markov Decision Processes (MDPs) (cont.)
Markov Decision Processes (MDPs) (cont.) Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University November 29 th, 2007 Markov Decision Process (MDP) Representation State space: Joint state x
Forward Search Value Iteration For POMDPs
Forward Search Value Iteration For POMDPs Guy Shani and Ronen I. Brafman and Solomon E. Shimony Department of Computer Science, Ben-Gurion University, Beer-Sheva, Israel Abstract Recent scaling up of POMDP
Partially Observable Markov Decision Processes. Silvia Cruciani João Carvalho
Partially Observable Markov Decision Processes Silvia Cruciani João Carvalho MDP A reminder: is a set of states is a set of actions is the state transition function. is the probability of ending in state
Q-learning with linear function approximation
Q-learning with linear function approximation Francisco S. Melo and M. Isabel Ribeiro Institute for Systems and Robotics [fmelo,mir]@isr.ist.utl.pt Conference on Learning Theory, COLT 2007 June 14th, 2007
Value Iteration. Reinforcement Learning: Introduction to Machine Learning. Matt Gormley Lecture 23 Apr. 10, 2019
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Reinforcement Learning: Value Iteration Matt Gormley Lecture 23 Apr. 10, 2019 1
Introduction to Reinforcement Learning. J. Zico Kolter Carnegie Mellon University
Introduction to Reinforcement Learning J. Zico Kolter Carnegie Mellon University 1 Agent interaction with environment Agent State s Reward r Action a Environment 2 Of course, an oversimplification 3 Review:
Research Project. Reinforcement Learning Policy Gradient Algorithms for Robot Learning
University of Girona Department of Electronics, Informatics and Automation Research Project Reinforcement Learning Policy Gradient Algorithms for Robot Learning Work presented by Andres El-Fakdi Sencianes
Topics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound. Lecture 12: Deep Reinforcement Learning
Topics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound Lecture 12: Deep Reinforcement Learning Types of Learning Supervised training Learning from the teacher Training data includes
Solving POMDPs by Searching the Space of Finite Policies
417 Solving POMDPs by Searching the Space of Finite Policies Nicolas Meuleau, Kee-Eung Kim, Leslie Pack Kaelbling and Anthony R. Cassandra Computer Science Dept, Box 1910, Brown University, Providence,
10703 Deep Reinforcement Learning and Control
10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient I Used Materials Disclaimer: Much of the material and slides for this lecture
On Scalability Issues in Reinforcement Learning for Modular Robots. Paulina Varshavskaya, Leslie Pack Kaelbling and Daniela Rus
On Scalability Issues in Reinforcement Learning for Modular Robots Paulina Varshavskaya, Leslie Pack Kaelbling and Daniela Rus motivate reinforcement learning (RL) larger modular system challenges a specific
Perseus: Randomized Point-based Value Iteration for POMDPs
Journal of Artificial Intelligence Research 24 (2005) 195-220 Submitted 11/04; published 08/05 Perseus: Randomized Point-based Value Iteration for POMDPs Matthijs T. J. Spaan Nikos Vlassis Informatics
A Brief Introduction to Reinforcement Learning
A Brief Introduction to Reinforcement Learning Minlie Huang ( ) Dept. of Computer Science, Tsinghua University aihuang@tsinghua.edu.cn 1 http://coai.cs.tsinghua.edu.cn/hml Reinforcement Learning Agent
Value-Function Approximations for Partially Observable Markov Decision Processes
Journal of Artificial Intelligence Research 13 (2000) 33 94 Submitted 9/99; published 8/00 Value-Function Approximations for Partially Observable Markov Decision Processes Milos Hauskrecht Computer Science
Deep Reinforcement Learning
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic
CS885 Reinforcement Learning Lecture 9: May 30, Model-based RL [SutBar] Chap 8
![CS885 Reinforcement Learning Lecture 9: May 30, Model-based RL [SutBar] Chap 8](/thumbs/90/103880210.jpg "CS885 Reinforcement Learning Lecture 9: May 30, Model-based RL [SutBar] Chap 8")
CS885 Reinforcement Learning Lecture 9: May 30, 2018 Model-based RL [SutBar] Chap 8 CS885 Spring 2018 Pascal Poupart 1 Outline Model-based RL Dyna Monte-Carlo Tree Search CS885 Spring 2018 Pascal Poupart
A fast point-based algorithm for POMDPs
A fast point-based algorithm for POMDPs Nikos lassis Matthijs T. J. Spaan Informatics Institute, Faculty of Science, University of Amsterdam Kruislaan 43, 198 SJ Amsterdam, The Netherlands {vlassis,mtjspaan}@science.uva.nl
Incremental methods for computing bounds in partially observable Markov decision processes
Incremental methods for computing bounds in partially observable Markov decision processes Milos Hauskrecht MIT Laboratory for Computer Science, NE43-421 545 Technology Square Cambridge, MA 02139 milos@medg.lcs.mit.edu
Programming Reinforcement Learning in Jason
Programming Reinforcement Learning in Jason Amelia Bădică 1, Costin Bădică 1, Mirjana Ivanović 2 1 University of Craiova, Romania 2 University of Novi Sad, Serbia Talk Outline Introduction, Motivation
Experiments with Infinite-Horizon, Policy-Gradient Estimation
Journal of Artificial Intelligence Research 15 (2001) 351-381 Submitted 9/00; published 11/01 Experiments with Infinite-Horizon, Policy-Gradient Estimation Jonathan Baxter WhizBang! Labs. 4616 Henry Street
Markov Decision Processes and Reinforcement Learning
Lecture 14 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Stuart Russell and Peter Norvig Course Overview Introduction Artificial Intelligence
Apprenticeship Learning for Reinforcement Learning. with application to RC helicopter flight Ritwik Anand, Nick Haliday, Audrey Huang
Apprenticeship Learning for Reinforcement Learning with application to RC helicopter flight Ritwik Anand, Nick Haliday, Audrey Huang Table of Contents Introduction Theory Autonomous helicopter control
Reinforcement Learning (2)
Reinforcement Learning (2) Bruno Bouzy 1 october 2013 This document is the second part of the «Reinforcement Learning» chapter of the «Agent oriented learning» teaching unit of the Master MI computer course.
Learning and Solving Partially Observable Markov Decision Processes
Ben-Gurion University of the Negev Department of Computer Science Learning and Solving Partially Observable Markov Decision Processes Dissertation submitted in partial fulfillment of the requirements for
Slides credited from Dr. David Silver & Hung-Yi Lee
Slides credited from Dr. David Silver & Hung-Yi Lee Review Reinforcement Learning 2 Reinforcement Learning RL is a general purpose framework for decision making RL is for an agent with the capacity to
Hierarchical Average Reward Reinforcement Learning Mohammad Ghavamzadeh Sridhar Mahadevan CMPSCI Technical Report
Hierarchical Average Reward Reinforcement Learning Mohammad Ghavamzadeh Sridhar Mahadevan CMPSCI Technical Report 03-19 June 25, 2003 Department of Computer Science 140 Governors Drive University of Massachusetts
An Improved Policy Iteratioll Algorithm for Partially Observable MDPs
An Improved Policy Iteratioll Algorithm for Partially Observable MDPs Eric A. Hansen Computer Science Department University of Massachusetts Amherst, MA 01003 hansen@cs.umass.edu Abstract A new policy
When Network Embedding meets Reinforcement Learning?
When Network Embedding meets Reinforcement Learning? ---Learning Combinatorial Optimization Problems over Graphs Changjun Fan 1 1. An Introduction to (Deep) Reinforcement Learning 2. How to combine NE
ADAPTIVE TILE CODING METHODS FOR THE GENERALIZATION OF VALUE FUNCTIONS IN THE RL STATE SPACE A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL
ADAPTIVE TILE CODING METHODS FOR THE GENERALIZATION OF VALUE FUNCTIONS IN THE RL STATE SPACE A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY BHARAT SIGINAM IN
Reinforcement Learning and Optimal Control. ASU, CSE 691, Winter 2019
Reinforcement Learning and Optimal Control ASU, CSE 691, Winter 2019 Dimitri P. Bertsekas dimitrib@mit.edu Lecture 1 Bertsekas Reinforcement Learning 1 / 21 Outline 1 Introduction, History, General Concepts
CSE151 Assignment 2 Markov Decision Processes in the Grid World
CSE5 Assignment Markov Decision Processes in the Grid World Grace Lin A484 gclin@ucsd.edu Tom Maddock A55645 tmaddock@ucsd.edu Abstract Markov decision processes exemplify sequential problems, which are
Approximating a Policy Can be Easier Than Approximating a Value Function
Computer Science Technical Report Approximating a Policy Can be Easier Than Approximating a Value Function Charles W. Anderson www.cs.colo.edu/ anderson February, 2 Technical Report CS-- Computer Science
Reinforcement Learning: A brief introduction. Mihaela van der Schaar
Reinforcement Learning: A brief introduction Mihaela van der Schaar Outline Optimal Decisions & Optimal Forecasts Markov Decision Processes (MDPs) States, actions, rewards and value functions Dynamic Programming
0.1. alpha(n): learning rate
Least-Squares Temporal Dierence Learning Justin A. Boyan NASA Ames Research Center Moett Field, CA 94035 jboyan@arc.nasa.gov Abstract TD() is a popular family of algorithms for approximate policy evaluation
Artificial Intelligence
Artificial Intelligence Other models of interactive domains Marc Toussaint University of Stuttgart Winter 2018/19 Basic Taxonomy of domain models Other models of interactive domains Basic Taxonomy of domain
Neuro-Dynamic Programming An Overview
1 Neuro-Dynamic Programming An Overview Dimitri Bertsekas Dept. of Electrical Engineering and Computer Science M.I.T. May 2006 2 BELLMAN AND THE DUAL CURSES Dynamic Programming (DP) is very broadly applicable,
Using Linear Programming for Bayesian Exploration in Markov Decision Processes
Using Linear Programming for Bayesian Exploration in Markov Decision Processes Pablo Samuel Castro and Doina Precup McGill University School of Computer Science {pcastr,dprecup}@cs.mcgill.ca Abstract A
Solving Factored POMDPs with Linear Value Functions
IJCAI-01 workshop on Planning under Uncertainty and Incomplete Information (PRO-2), pp. 67-75, Seattle, Washington, August 2001. Solving Factored POMDPs with Linear Value Functions Carlos Guestrin Computer
15-780: MarkovDecisionProcesses
15-780: MarkovDecisionProcesses J. Zico Kolter Feburary 29, 2016 1 Outline Introduction Formal definition Value iteration Policy iteration Linear programming for MDPs 2 1988 Judea Pearl publishes Probabilistic
Prioritizing Point-Based POMDP Solvers
Prioritizing Point-Based POMDP Solvers Guy Shani, Ronen I. Brafman, and Solomon E. Shimony Department of Computer Science, Ben-Gurion University, Beer-Sheva, Israel Abstract. Recent scaling up of POMDP
Temporal Abstraction in Reinforcement Learning
Temporal Abstraction in Reinforcement Learning Pierre-Luc Bacon and Doina Precup February 17, 2017 Multi-steps Boostrapping 1 / 26 Multi-steps Boostrapping [...] they free you from the tyranny of the time
Hidden Markov Model Multiarm Bandits: A Methodology for Beam Scheduling in Multitarget Tracking
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 12, DECEMBER 2001 2893 Hidden Markov Model Multiarm Bandits: A Methodology for Beam Scheduling in Multitarget Tracking Vikram Krishnamurthy, Senior
servable Incremental methods for computing Markov decision
From: AAAI-97 Proceedings. Copyright 1997, AAAI (www.aaai.org). All rights reserved. Incremental methods for computing Markov decision servable Milos Hauskrecht MIT Laboratory for Computer Science, NE43-421
Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning
Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning Jan Peters 1, Stefan Schaal 1 University of Southern California, Los Angeles CA 90089, USA Abstract. In this paper, we
Tree Based Discretization for Continuous State Space Reinforcement Learning
From: AAAI-98 Proceedings. Copyright 1998, AAAI (www.aaai.org). All rights reserved. Tree Based Discretization for Continuous State Space Reinforcement Learning William T. B. Uther and Manuela M. Veloso
Reinforcement Learning in Discrete and Continuous Domains Applied to Ship Trajectory Generation
POLISH MARITIME RESEARCH Special Issue S1 (74) 2012 Vol 19; pp. 31-36 10.2478/v10012-012-0020-8 Reinforcement Learning in Discrete and Continuous Domains Applied to Ship Trajectory Generation Andrzej Rak,
Marco Wiering Intelligent Systems Group Utrecht University
Reinforcement Learning for Robot Control Marco Wiering Intelligent Systems Group Utrecht University marco@cs.uu.nl 22-11-2004 Introduction Robots move in the physical environment to perform tasks The environment
Point-based value iteration: An anytime algorithm for POMDPs
Point-based value iteration: An anytime algorithm for POMDPs Joelle Pineau, Geoff Gordon and Sebastian Thrun Carnegie Mellon University Robotics Institute 5 Forbes Avenue Pittsburgh, PA 15213 jpineau,ggordon,thrun@cs.cmu.edu
Hierarchical Reinforcement Learning for Robot Navigation
Hierarchical Reinforcement Learning for Robot Navigation B. Bischoff 1, D. Nguyen-Tuong 1,I-H.Lee 1, F. Streichert 1 and A. Knoll 2 1- Robert Bosch GmbH - Corporate Research Robert-Bosch-Str. 2, 71701
Reinforcement Learning of Walking Behavior for a Four-Legged Robot
TuM01-3 Reinforcement Learning of Walking Behavior for a Four-Legged Robot Hajime Kimura GEN~FE.DIS.TITECH.AC.JP Toru Yamashita YAMA~FE.DIS.TITECH.AC.JP Shigenobu Kobayashi KOBAYASI~DIS.TITECH.AC.JP Tokyo
Generalized Inverse Reinforcement Learning
Generalized Inverse Reinforcement Learning James MacGlashan Cogitai, Inc. james@cogitai.com Michael L. Littman mlittman@cs.brown.edu Nakul Gopalan ngopalan@cs.brown.edu Amy Greenwald amy@cs.brown.edu Abstract
PROBLEM SOLVING WITH
PROBLEM SOLVING WITH REINFORCEMENT LEARNING Gavin Adrian Rummery A Cambridge University Engineering Department Trumpington Street Cambridge CB2 1PZ England This dissertation is submitted for consideration
Rewind to Track: Parallelized Apprenticeship Learning with Backward Tracklets
Rewind to Track: Parallelized Apprenticeship Learning with Backward Tracklets Jiang Liu 1,2, Jia Chen 2, De Cheng 2, Chenqiang Gao 1, Alexander G. Hauptmann 2 1 Chongqing University of Posts and Telecommunications
10703 Deep Reinforcement Learning and Control
10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient II Used Materials Disclaimer: Much of the material and slides for this lecture
Diagnose and Decide: An Optimal Bayesian Approach
Diagnose and Decide: An Optimal Bayesian Approach Christopher Amato CSAIL MIT camato@csail.mit.edu Emma Brunskill Computer Science Department Carnegie Mellon University ebrun@cs.cmu.edu Abstract Many real-world
Lecture 13: Learning from Demonstration
CS 294-5 Algorithmic Human-Robot Interaction Fall 206 Lecture 3: Learning from Demonstration Scribes: Samee Ibraheem and Malayandi Palaniappan - Adapted from Notes by Avi Singh and Sammy Staszak 3. Introduction
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
Advances in Neural Information Processing Systems 8, pp. 1038-1044, MIT Press, 1996. Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding Richard S. Sutton University
Planning with Continuous Actions in Partially Observable Environments
Planning with ontinuous Actions in Partially Observable Environments Matthijs T. J. Spaan and Nikos lassis Informatics Institute, University of Amsterdam Kruislaan 403, 098 SJ Amsterdam, The Netherlands
Probabilistic Robotics
Probabilistic Robotics Sebastian Thrun Wolfram Burgard Dieter Fox The MIT Press Cambridge, Massachusetts London, England Preface xvii Acknowledgments xix I Basics 1 1 Introduction 3 1.1 Uncertainty in
Iterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms
Iterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms By:- Nitin Kamra Indian Institute of Technology, Delhi Advisor:- Prof. Ulrich Reude 1. Introduction to Linear
Planning for Markov Decision Processes with Sparse Stochasticity
Planning for Markov Decision Processes with Sparse Stochasticity Maxim Likhachev Geoff Gordon Sebastian Thrun School of Computer Science School of Computer Science Dept. of Computer Science Carnegie Mellon
Approximate Linear Programming for Average-Cost Dynamic Programming
Approximate Linear Programming for Average-Cost Dynamic Programming Daniela Pucci de Farias IBM Almaden Research Center 65 Harry Road, San Jose, CA 51 pucci@mitedu Benjamin Van Roy Department of Management
Targeting Specific Distributions of Trajectories in MDPs
Targeting Specific Distributions of Trajectories in MDPs David L. Roberts 1, Mark J. Nelson 1, Charles L. Isbell, Jr. 1, Michael Mateas 1, Michael L. Littman 2 1 College of Computing, Georgia Institute
Generalized and Bounded Policy Iteration for Interactive POMDPs
Generalized and Bounded Policy Iteration for Interactive POMDPs Ekhlas Sonu and Prashant Doshi THINC Lab, Dept. of Computer Science University Of Georgia Athens, GA 30602 esonu@uga.edu, pdoshi@cs.uga.edu
Residual Advantage Learning Applied to a Differential Game
Presented at the International Conference on Neural Networks (ICNN 96), Washington DC, 2-6 June 1996. Residual Advantage Learning Applied to a Differential Game Mance E. Harmon Wright Laboratory WL/AAAT
An Approach to State Aggregation for POMDPs
An Approach to State Aggregation for POMDPs Zhengzhu Feng Computer Science Department University of Massachusetts Amherst, MA 01003 fengzz@cs.umass.edu Eric A. Hansen Dept. of Computer Science and Engineering
Hierarchically Optimal Average Reward Reinforcement Learning
Hierarchically Optimal Average Reward Reinforcement Learning Mohammad Ghavamzadeh mgh@cs.umass.edu Sridhar Mahadevan mahadeva@cs.umass.edu Department of Computer Science, University of Massachusetts Amherst,
PPCP: Efficient Probabilistic Planning with Clear Preferences in Partially-Known Environments
PPCP: Efficient Probabilistic Planning with Clear Preferences in Partially-Known Environments Maxim Likhachev The Robotics Institute Carnegie Mellon University Pittsburgh, PA 15213 maxim+@cs.cmu.edu Anthony
Localization and Map Building
Localization and Map Building Noise and aliasing; odometric position estimation To localize or not to localize Belief representation Map representation Probabilistic map-based localization Other examples
Knowledge Transfer for Deep Reinforcement Learning with Hierarchical Experience Replay
Knowledge Transfer for Deep Reinforcement Learning with Hierarchical Experience Replay Haiyan (Helena) Yin, Sinno Jialin Pan School of Computer Science and Engineering Nanyang Technological University
Distributed and Asynchronous Policy Iteration for Bounded Parameter Markov Decision Processes
Distributed and Asynchronous Policy Iteration for Bounded Parameter Markov Decision Processes Willy Arthur Silva Reis 1, Karina Valdivia Delgado 2, Leliane Nunes de Barros 1 1 Departamento de Ciência da
Numerical schemes for Hamilton-Jacobi equations, control problems and games
Numerical schemes for Hamilton-Jacobi equations, control problems and games M. Falcone H. Zidani SADCO Spring School Applied and Numerical Optimal Control April 23-27, 2012, Paris Lecture 2/3 M. Falcone
Deep Learning. Architecture Design for. Sargur N. Srihari
Architecture Design for Deep Learning Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation
Task-Oriented Planning for Manipulating Articulated Mechanisms under Model Uncertainty. Venkatraman Narayanan and Maxim Likhachev
Task-Oriented Planning for Manipulating Articulated Mechanisms under Model Uncertainty Venkatraman Narayanan and Maxim Likhachev Motivation Motivation A lot of household objects are ARTICULATED A lot of
Switch Packet Arbitration via Queue-Learning
Switch Packet Arbitration via QueueLearning Timothy X Brown Electrical and Computer Engineering University of Colorado Boulder, CO 803090530 timxb@coloradoedu Accepted in Advances in Neural Information
ICRA 2012 Tutorial on Reinforcement Learning I. Introduction
ICRA 2012 Tutorial on Reinforcement Learning I. Introduction Pieter Abbeel UC Berkeley Jan Peters TU Darmstadt Motivational Example: Helicopter Control Unstable Nonlinear Complicated dynamics Air flow
CSE-571. Deterministic Path Planning in Robotics. Maxim Likhachev 1. Motion/Path Planning. Courtesy of Maxim Likhachev University of Pennsylvania
Motion/Path Planning CSE-57 Deterministic Path Planning in Robotics Courtesy of Maxim Likhachev University of Pennsylvania Tas k : find a feasible (and cost-minimal) path/motion from the current configuration
Decision Making under Uncertainty
Decision Making under Uncertainty MDPs and POMDPs Mykel J. Kochenderfer 27 January 214 Recommended reference Markov Decision Processes in Artificial Intelligence edited by Sigaud and Buffet Surveys a broad
WestminsterResearch
WestminsterResearch http://www.westminster.ac.uk/research/westminsterresearch Reinforcement learning in continuous state- and action-space Barry D. Nichols Faculty of Science and Technology This is an
Robotic Search & Rescue via Online Multi-task Reinforcement Learning
Lisa Lee Department of Mathematics, Princeton University, Princeton, NJ 08544, USA Advisor: Dr. Eric Eaton Mentors: Dr. Haitham Bou Ammar, Christopher Clingerman GRASP Laboratory, University of Pennsylvania,
Probabilistic Planning for Behavior-Based Robots
Probabilistic Planning for Behavior-Based Robots Amin Atrash and Sven Koenig College of Computing Georgia Institute of Technology Atlanta, Georgia 30332-0280 {amin, skoenig}@cc.gatech.edu Abstract Partially
Unsupervised Learning. Clustering and the EM Algorithm. Unsupervised Learning is Model Learning
Unsupervised Learning Clustering and the EM Algorithm Susanna Ricco Supervised Learning Given data in the form < x, y >, y is the target to learn. Good news: Easy to tell if our algorithm is giving the
euristic Varia Introduction Abstract
From: AAAI-97 Proceedings. Copyright 1997, AAAI (www.aaai.org). All rights reserved. euristic Varia s Ronen I. Brafman Department of Computer Science University of British Columbia Vancouver, B.C. V6T
A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning
Foundations and Trends R in Machine Learning Vol. 6, No. 4 (2013) 375 XXX c 2013 A. Geramifard, T. J. Walsh, S. Tellex, G. Chowdhary, N. Roy, and J. P. How DOI: 10.1561/2200000042 A Tutorial on Linear
Hierarchical Average Reward Reinforcement Learning
Journal of Machine Learning Research 8 (2007) 2629-2669 Submitted 6/03; Revised 8/07; Published 11/07 Hierarchical Average Reward Reinforcement Learning Mohammad Ghavamzadeh Department of Computing Science
On-line search for solving Markov Decision Processes via heuristic sampling
On-line search for solving Markov Decision Processes via heuristic sampling Laurent Péret and Frédérick Garcia Abstract. In the past, Markov Decision Processes (MDPs) have become a standard for solving
Rollout Sampling Policy Iteration for Decentralized POMDPs
Rollout Sampling Policy Iteration for Decentralized POMDPs Feng Wu School of Computer Science University of Sci. & Tech. of China Hefei, Anhui 2327 China wufeng@mail.ustc.edu.cn Shlomo Zilberstein Department
Problem Solving as Search. CMPSCI 383 September 15, 2011
Problem Solving as Search CMPSCI 383 September 15, 2011 1 Today s lecture Problem-solving as search Uninformed search methods Problem abstraction Bold Claim: Many problems faced by intelligent agents,
Space-Progressive Value Iteration: An Anytime Algorithm for a Class of POMDPs
Space-Progressive Value Iteration: An Anytime Algorithm for a Class of POMDPs Nevin L. Zhang and Weihong Zhang lzhang,wzhang @cs.ust.hk Department of Computer Science Hong Kong University of Science &
Dynamic Preferences in Multi-Criteria Reinforcement Learning
Sriraam Natarajan Prasad Tadepalli School of Electrical Engineering and Computer Science, Oregon State University, USA NATARASR@EECS.ORST.EDU TADEPALL@EECS.ORST.EDU Abstract The current framework of reinforcement
Reinforcement learning algorithms for non-stationary environments. Devika Subramanian Rice University
Reinforcement learning algorithms for non-stationary environments Devika Subramanian Rice University Joint work with Peter Druschel and Johnny Chen of Rice University. Supported by a grant from Southwestern
Perseus: randomized point-based value iteration for POMDPs
Universiteit van Amsterdam IAS technical report IAS-UA-04-02 Perseus: randomized point-based value iteration for POMDPs Matthijs T. J. Spaan and Nikos lassis Informatics Institute Faculty of Science University
COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS18. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions
COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS18 Lecture 2: Linear Regression Gradient Descent Non-linear basis functions LINEAR REGRESSION MOTIVATION Why Linear Regression? Simplest
Clustering with Reinforcement Learning
Clustering with Reinforcement Learning Wesam Barbakh and Colin Fyfe, The University of Paisley, Scotland. email:wesam.barbakh,colin.fyfe@paisley.ac.uk Abstract We show how a previously derived method of
Partially Observable Markov Decision Processes. Mausam (slides by Dieter Fox)
Partially Observable Markov Decision Processes Mausam (slides by Dieter Fox) Stochastic Planning: MDPs Static Environment Fully Observable Perfect What action next? Stochastic Instantaneous Percepts Actions
Cognitive Robotics Robot Motion Planning Matteo Matteucci
Cognitive Robotics Robot Motion Planning Robot Motion Planning eminently necessary since, by definition, a robot accomplishes tasks by moving in the real world. J.-C. Latombe (1991) Robot Motion Planning