Skill. Robot/ Controller

Transcription

1 Skill Acquisition from Human Demonstration Using a Hidden Markov Model G. E. Hovland, P. Sikka and B. J. McCarragher Department of Engineering Faculty of Engineering and Information Technology The Australian National University Canberra, Australia Abstract A new approach to skill acquisition in assembly is proposed. An assembly skill is represented by a hybrid dynamic system where a discrete event controller models the skill at the task level. The output of the discrete event controller provides the reference commands for the underlying robot controller. This structure is naturally encoded by a hidden Markov model (HMM). The HMM parameters are obtained by training on sensory data from human demonstrations of the skill. Currently, assembly tasks have to be performed by human operators or by robots using expensive xtures. Our approach transfers the assembly skill from an expert human operator to the robot, thus making it possible for a robot to perform assembly tasks without the use of expensive xtures. 1 Introduction Manipulation tasks such as assembly are easily performed by human operators. However, these tasks are still dicult for robots and require the use of precise and expensive xtures. Furthermore, human operators are able to acquire new skills corresponding to new tasks by explanation and practice. For robots to eectively acquire these skills, we need good models of the skills that are able to capture two aspects of human skill acquisition - the ability to represent and reason about tasks, and the ability to improve performance on the task by repeated practice. With these models of human skill, robots can acquire a skill from human demonstration of the task. Our approach to skill acquisition makes it easier to program robots for dierent assembly tasks. Another advantage is that, like a human operator, the robot can handle uncertainties in the location of parts. This allows the robot to perform assembly tasks without the use of precise and expensive xtures. In this paper, we present a new approach for representing and acquiring human assembly skills for applications in robotic assembly tasks. Following the approach presented by McCarragher and Asada in [1, ], an assembly task is modeled as a discrete event system in terms of contact congurations involving the object and the environment. The events correspond to changes in these contact congurations and are detected by a process monitor. The discrete event controller provides the reference commands and parameters to the robot controller depending on the desired event. We consider a skill to consist of both the discrete event controller and the process monitor. This paper focuses on the discrete event controller. We represent the discrete event controller for a skill as a Hidden Markov Model (HMM) [, 4]. The states of the hidden Markov model correspond to the states of the discrete event controller. The skill is acquired from human demonstration of the task, and so the observation symbols of the HMM are taken from the variables recorded while the task is performed by a human operator. The parameters of the HMM are learned using the structure of the HMM and the data obtained from human demonstration. In this way, the HMM is used to model the data generated by a human operator in performing the task. The HMM learned above is used to control a robot in performing the task. The observation symbols of the HMM now correspond to the reference commands for the robot controller. An advantage of this representation is that the HMM parameters can be interpreted in physically meaningful ways. For example, the transition probabilities of the hidden Markov process correspond to the probability of events occurring in the discrete event controller, while the probability distributions associated with each observation symbol indicate the probability of using a particular reference command in a given discrete state. Thus, this representation provides a preferred transition and a corresponding preferred reference command for each discrete state. Related Work Although HMMs have been used extensively in research on automatic speech recognition, it is only recently that they have been used in robotics for modeling skills. Hannaford and Lee [5] used an HMM to characterize the force signals generated by an operator performing

2 a tele-manipulation task. Xu and Yang [6] proposed HMM-based models for two types of human skill. An action skill, corresponding to tasks not requiring sensory feedback, is modeled as a left-to-right HMM, where the Markov states correspond to human mental states, while the observable output symbols correspond to measurable signals such as position and force [9]. A reaction skill, corresponding to tasks requiring sensory feedback, is modeled as a set of left-to-right HMMs where each HMM encodes the mapping between a sensory data stream and a control action. Both the approaches described above assume that the hidden Markov process corresponds to the mental states of the human operator. No attempt is made to explain what these mental states might be. In most assembly problems, the tasks can be structured to a large extent by geometrical considerations, and the human states correspond to this structure to some extent. A further limitation of the above approaches is that they use a left-toright stochastic automaton for the hidden process. This restricts their application to tasks that can be described by such automata, and many assembly tasks require a richer structure. Robot programming by human demonstration has been considered by several authors. Delson and West [8] present a method for robot programming by human demonstration. A robot trajectory for a pick-and-place task is learned from several human demonstrations of the task. Their method is limited in that it deals only with learning motion trajectories. Liu and Asada [1] represent a skill as a map from the process parameter space of the given task to the control actions used to accomplish the task. A major shortcoming of the above method is that it provides little insight into what the task parameters represent. Another limitation is the inability of the learned skill to handle situations not encountered in the training set. A hierarchical approach to skill representation is presented by Morrow and Khosla [7]. An assembly skill is represented as a nite-state automaton, and each transition of the automaton is associated with an intermediate layer of primitives, called sensori-motor primitives (SMPs), which integrate sensing and motion for a given task. Their approach does not specify how a small, optimal set of SMPs can be identied. Another limitation is that the SMPs may be very task-specic. Modeling Assembly Tasks To overcome some of the limitations in the previous approaches described above, we present a new approach to skill acquisition in robotic assembly. In our approach, a skill corresponding to an assembly task is explicitly represented as a hybrid dynamic system [1, ]. This framework is illustrated in Figure 1. Skill Discrete Event Controller ud Robot/ Controller u γ Monitor x Plant/ Figure 1: A model of robotic assembly tasks using the hybrid dynamic system framework. The task-level behavior of an assembly task is modeled as a discrete event system, characterized by the equations: k+1 = ( k ; e k ); u d k = ( k ); k is used to denote the discrete state of the system at instant k, e k is used to represent the discrete event occurring at instant k, and is a function that computes the next state of the system based on the current state of the system and the current event. The function computes the output of the discrete event system, u d k, based on the current state of the system. The output u d k is used as the reference command for the robot controller at instant k. An assembly task is characterized at the discrete level by the allowed geometrical contact congurations of the objects involved in the task. These contact congurations correspond to the discrete states of the discrete event system. For example, consider a planar peg-in-the-hole assembly process where we have only one type of contact, the edge-surface contact. The edge or the surface involved can be a part of either the workpiece or the environment. Examples of this contact type are shown in Figure. Denition 1 (Discrete State) A discrete state is de- ned as a contact formation consisting of one or two edge-surface contacts between the workpiece and the environment. The set of discrete states is then the set of all possible contact formations. Denition (Event) An event in robotic assembly is dened as the change of discrete state. The events are discrete in time and describe the gain or a loss of contact between the workpiece and the environment. The 1 possible events for the peg-in-hole task considered above are as follows: e 1 = 1! e =! 1 e = 1!

3 1 4 5 Figure : Planar peg-in-the-hole assembly with 5 contact states and 1 possible discrete events. e 4 =! 1 e 5 = 1! 5 e 6 = 5! 1 e 7 =! e 8 =! e 9 =! 4 e 1 = 4! e 11 = 4! 5 e 1 = 5! 4 An assembly task is conveniently represented in the form of a network, called the contact-state network, where the nodes correspond to the discrete states and the arcs correspond to the events. The contact-state network corresponding to the planar peg-insertion task is shown in Figure. 1 4 Figure : Contact state network used for skill acquisition. The physical task is modeled as a continuous system described by the equations: _x(t) = f(x(t); u(t)) These equations describe the behavior of the system in state-space. x denotes the state vector of the system and u denotes the input vector. In robotic manipulation, each discrete state imposes a set of constraints on the system. These constraints are represented as: 5 = g i (x(t); u(t)); i = 1 : : : m; where m refers to the number of constraints corresponding to a given contact state. The process monitor is responsible for monitoring the continuous time system to detect discrete events, and is characterized by the equation: e k = (x(t)): 4 Representing Assembly Skills Using Hidden Markov Models The hybrid dynamic framework used to model an assembly task highlights two important aspects of an assembly skill. The rst is the discrete event system that is used to represent the sequence of discrete states required to achieve the given task. The output of the discrete event controller is used as the reference command for the robot controller. The second aspect of a skill is the process monitor,, which monitors the plant variables and provides the discrete events at the discrete level. Any model of skill must take these two aspects into account. An HMM-based process monitor using the observed force signal is described in [11]. Each event is modeled as an HMM. The sensory data corresponding to an event is scored by all the HMMs and the recognized event corresponds to the HMM that predicts the sensory data with maximum probability. In this paper, we use the process monitor described in [11], and focus on the representation of the discrete event controller using an HMM. An HMM can be used naturally to represent a discrete event controller for an assembly skill. An HMM consists of two stochastic processes. The underlying stochastic process is a Markov process represented by the transition probability matrix A and the initial state distribution. The output of the HMM is a sequence of observation symbols that is obtained probabilistically from the underlying Markov process. The output corresponding to each state of the Markov process is a probability distribution dened over the set of observation symbols, instead of an observation symbol. b j (O k ) refers to the probability of observing symbol O k in state j. The set of all the probability distributions is denoted by B. Thus, an HMM is dened by the structure = fa; B; g. The network in Figure is the basis for the HMM we use in this paper. An advantage of using the contact state network as the basis for the hidden Markov process is that the Markov states correspond directly to the contact-states. The HMM transition probabilities correspond to the probabilities of discrete events. Thus, the transition probability matrix corresponds roughly to the function of the discrete event system. Since the contact state network has cycles, the task can not easily be represented by a left-to-right model. For the planar peg-insertion task considered in this paper, there are 5 contact-states and 1 possible events.

4 a a Skill a q Hidden Markov Model γ Monitor a 1 a 1 a55 a a 1 a 15 q q 1 q 5 a 1 a 4 a 51 a 11 a a 45 4 a 54 u d Robot/ Controller u x Plant/ q 4 a 44 Figure 4: A stochastic automaton representing the discrete event controller for the peg-insertion task. The Markov states correspond to the contact-states, leading to a 5 5 transition probability matrix, shown below: A = 64 a 11 a 1 a 1 a 15 a 1 a a a 1 a a a 4 a 4 a 44 a 45 a 51 a 54 a 55 Figure 4 shows the stochastic automaton representing the hidden Markov process of the HMM modeling the peg-insertion skill. In our approach, the assembly skill is acquired from human demonstration. We measure the position and force during the human demonstration of the task. The position data is used to compute the velocities of the peg. The discrete event controller is modeled as an HMM. Since human intentions cannot be known, we associate the observed positions and velocities with the observation symbols of the HMM. Figure 5 illustrates our model of an assembly skill using an HMM. We use a discrete HMM to model the discrete event controller. Hence, the positions and velocities are discretized into a nite set of observation symbols for the HMM. The velocities are discretized based on the direction, while the positions are discretized into 4 values based on a reference frame located at the center of the hole, as shown in Figure 6. Let V = fv ; ; v 7 g denote the set of discretized velocities and P = fp ; ; p g denote the set of discretized positions. Then, the observation symbols of the HMM belong to the set P V. The number of discrete values for the position and velocity, equal to 4 and 8 respectively, depend on the task. 75 Figure 5: A model of robotic assembly tasks using the hybrid dynamic system framework and an HMM. A robot skill corresponds to the parts included within the dashed box. The parts included within the dotted box corresponds to the ideas discussed in this paper. v 6 v 7 v 5 v v 4 v 1 Figure 6: The discretized positions and velocities used to generate the discrete observation symbols of the HMM. The output of the HMM is a sequence of the observation symbols described above. Each Markov state, j, is associated with a probability distribution b j over the set of observation symbols, as shown below: v b j (O k ) = P rfo k = p l :v m j = j g; l = : : : ; m = : : : 7: In summary, the discrete event part of the skill is represented as the hidden Markov process of an HMM. The corresponding behavior of the robot is represented by the observable symbols of the HMM, consisting of the discretized position and velocity of the peg during the task. Thus, for any given skill, an HMM represents the mapping from the contact states in the task space to the sensory output of the underlying robotic system. 5 Skill Acquisition and Execution The key step in the HMM approach to skill acquisition is the training of the HMM based on a training set. This training set is obtained from a human performing the v y p p x p 1 p

5 p assembly task several times. The training set consists of the sequence of observation symbols fo k g obtained from the observed positions and velocities from human demonstration. We use the Baum-Welch re-estimation method [] for obtaining the HMM parameters based on the training data. Once the HMM has been trained, the skill is represented by the transition probability matrix A and the probability distributions over the observation symbols for each state. From these parameters, we are able to extract meaningful information. An entry in the state transition matrix corresponds to the probability that the corresponding event will occur. This probability can also be interpreted as the desirability of the event for the acquired skill. Similarly, the probability distribution for the observation symbols can be interpreted as the velocity commands most likely to achieve the corresponding transition. The learned HMM can be used to control a robot to perform the task corresponding to the acquired skill. The transition probability matrix of the learned HMM is used to obtain the best sequence of states, corresponding to the sequence with the maximum probability, as the desired sequence of states for the assembly task. At any given instant, the next desired event is chosen from this sequence based on the information provided by the process monitor. Once the transition has been chosen, the probability distribution of the observation symbols corresponding to the current state, b j, is chosen. The observation symbols consist of a velocity and a position. Since the reference command to the robot controller, u d, is a velocity command, the probability distribution is modied to obtain the probability distribution d j over the set of velocities V : d j (v k ) = 4X m=1 b j (p m :v k ): The velocity v k with the maximum probability d j (v k ) is then chosed as the desired velocity command u d. 6 Experiments The experimental setup consists of an Eshed Scorbot 5- DOF manipulator and a six-axis Polhemus position sensor. Since we consider a planar assembly task, only three of the manipulator joints are used. The peg-insertion task described earlier is used to experimentally demonstrate our approach to robotic assembly skill acquisition. A human operator performs the desired task. During the task, the position and orientation of the peg is recorded at 1 Hz using a six-axis Polhemus position sensor. This data is used to derive a sequence of discretized observation symbols for the HMM. For example, Figure 7 shows the position data and the discretized observation symbol p, while Figure 8 shows the velocity data and the corresponding discretized observation symbol v. The human demonstration data (the training set) is obtained as follows. We dene 4 in Figure as the nal state in the assembly process. The human expert is then told to successfully move the peg to this nal contact state from each of the other contact states 1,, and 5. The Polhemus position sensor is used to derive the discrete observation symbols. This procedure is repeated N T times, where N T is the size of our training set. The training set obtained from human demonstration is used to train the HMM using the standard Baum-Welch re-estimation algorithm. The parameters of the trained HMM are then analyzed in terms of the contact-state network to ensure that the skill has indeed been transferred eectively to the robot. This is then demonstrated by using the HMM model to drive the robot in performing the peg-insertion task. Px [cm] v Py [cm] p p p Figure 7: Typical positions obtained from human demonstration data. Vx [cm/s] Vy [cm/s] v 4 v v v Figure 8: Typical velocity commands obtained from human demonstration data. The nal values for the HMM parameters a ij were as

6 follows. A = 64 :9 :4 :4 : :4 :94 : :9 : :85 : : :95 : :9 :5 :86 For each contact state i we get the probabilities of the commanded velocities from the observation symbol probability matrix B. v v1 v v v4 v5 v6 v From these parameters we can easily nd the most likely contact state sequences from 1 to 4 by multiplying the corresponding HMM state transition probabilities a ij Sequence Velocities HMM Probability 1!! 4 v4! v 1: 1? 75 1!!! 4 v4! v! v :4 1?5 1! 5! 4 v4! v4 1: 1? From this we see that the human demonstration data gives 1!! 4 as the most likely assembly sequence with the discretized velocities v 4! v. This information is transferred to the robot as a discrete event controller as described in a previous section, where the initial desired event trajectory will be 1! followed by! 4. However, since the position of the hole is unknown in the robotic assembly, the desired event 1! is very likely to cause the event 1!. When this situation occurs, the desired event trajectory is changed on-line by nding the new trajectory from to 4 with the highest HMM probability. In this particular case, the new desired event trajectory will be! followed by! 4. 7 Conclusion We have presented a new approach to assembly skill representation and acquisition from human demonstration, using the theory of hidden Markov models. This approach has been veried experimentally for a planar peginsertion task. Our approach has several advantages. The rst advantage is that the acquired skill is represented in terms of the contact-state network and so the HMM parameters can be examined to identify how a particular task is performed. The acquired skill can be described in physically meaningful terms based on the HMM parameters and the contact-state network. This compares favorably with other parametric models, such as neural networks, where it is dicult to interpret the parameters in physically meaningful terms. A limitation of our method derives from the limitation of any method based on a training set. The system may be unable to handle situations not encountered in the training set. This problem can be handled by generating large training sets. However, larger training sets take more time to generate and require more computing resources. References [1] B.J. McCarragher and H. Asada, The Discrete Event Control of Robotic Assembly. ASME Journal of Dynamic Systems, Measurements and Control, To be published. [] B.J. McCarragher and H. Asada, The Discrete Event Modeling and Trajectory Planning of Robotic Assembly. ASME Journal of Dynamic Systems, Measurements and Control, To be published. [] X.D. Huang, Y. Ariki, M.A. Jack, Hidden Markov Models for Speech Recognition. Edinburgh University Press, 199. [4] L.R. Rabiner and B.H. Juang, An Introduction to Hidden Markov Models. IEEE ASSP Magazine, January 1986, pp.4{16. [5] B. Hannaford and P. Lee, Hidden Markov Model Analysis of Force/Torque Information in Telemanipulation. The International Journal of Robotics Research, Oct. 1991, Vol. 1, No. 5, pp.58{59. [6] Y. Xu and J. Yang, Towards Human-Robot Coordination: Skill Modeling and Transferring via Hidden Markov Model. In Proceedings of IEEE International Conference on Robotics and Automation, 1995, pp.196{1911. [7] J. D. Morrow and P. K. Khosla, Sensorimotor Primitives for Robotic Assembly Skills. In Proceedings of IEEE International Conference on Robotics and Automation, 1995, pp.196{1911. [8] N. Delson and H. West, Robot Programming by Human Demonstration: The Use of Human Inconsistency in Improving D Robot Trajectories. In Proceedings of IEEE/RSJ/GI International Conference on Intelligent Robots and Systems, 1994, pp.148{155. [9] J. Yang, Y. Xu, C.S. Chen, Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics. IEEE Transactions on Robotics and Automation, Oct. 1994, Vol. 1, No. 5, pp.61{61. [1] S. Liu and H. Asada, Transfer of Human Skills to Robots: Learning from Human Demonstrations for Building an Adaptive Control System. American Control Conference, 199. [11] G. Hovland and B. J. McCarragher, Frequency-Domain Force Measurements for Discrete Event Contact Recognition, submitted to the 1996 IEEE International Conference on Robotics and Automation.