Robotic Control via Stimulus Response Learning. Andreas Birk. Universitat des Saarlandes, c/o Lehrstuhl Prof. W.J. Paul

Transcription

1 Robotic Control via Stimulus Response Learning Andreas Birk Universitat des Saarlandes, c/o Lehrstuhl Prof. W.J. Paul Postfach , Saarbrucken, Germany Abstract Stimulus response learning is a new approach to enable robots to learn an adaptive and robust control-mechanism. Learning in this paradigm is an evolutionary process, leading to a behavioral model of any given environment. The model is build up in a special data-structure as collection of regularities between sensor-data and activation of eectors. This process is guided by two simple and universal i.e., independent of the robot and environment, heuristics for rating the usefulness of basic elements of the model. We present a system learning from scratch to control a robot-gripper in a blocks-world via a camera. The system learns movements in the plane and to manipulate building-blocks. The most challenging experiments were done in noisy real world set-ups, using unprocessed high-resolution images. Keywords: Evolutionary Algorithm, Robotics, Adaptive Behavior, Machine Learning 1 Introduction The ability to learn is one of the most important features of intelligence. It enables cognitive systems to adapt to un-foreseen circumstances and to improve with practise. Though these tasks appear to be very easy to humans, automatic adaption and improvement of robots is still a very open research eld. Therefore, we developed stimulus response learning as a new machine learning approach suited for robotic control. We believe, that cognition can only emerge in systems interacting with a suciently complex environment. Or abusing an ancient Roman slogan \mens sana in corpere sano". Though our interests focus on understanding cognition and building a humanoid articial intelligence in long term, stimulus response learning is an interesting approach in regard to industrial applications. Because a learning control-mechanism allows noisy and changing environments, the execution of more various tasks and low-priced imprecise hardware.

2 2 Learning of Control We wish robots to be able to learn to do any given task in any given environment. So, we must supply our robot with a mechanism for investigating his environment, a suitable storage for his discoveries and a mechanism for applying his knowledge learned. 2.1 How to store Discoveries Our robot makes a discovery when he nds useful regularities between sensor-data and activation of eectors. The meaning of \useful" depends on the task(s) the robot is expected to do and will be discussed later. Discoveries are the basic drive for the evolution of the robot's model of the world, which tells him what to do in a certain situation to achieve a certain result. The world-model is built up in a special data-structure, a dynamic directed graph. The nodes are so-called stimulus response rules (SRR), simple behavioral rules made of tests on sensor-data and actions i.e., activation of eectors. In its most general form a SRR is a TOTE (Fig.1), an abbreviation for test 1, operate, test 2, exit. If test 1 the so-called condition holds, the TOTE can be executed. This means, the operate or action is repeatedly activated until test 2 the so-called feedback holds. After this, the exit or result should be fullled. This test represents the state of the environment after execution of the TOTE and is used for planning. false?? true true Condition Action Feedback Result? T est if the SRR is executable O peration of effectors T est if the action has to be repeated E xit to a goal state Figure 1: The TOTE as the most general form of a SRR In the graph representing the world-model, a directed edge (s 1 ; s 2 ) stands for a possible consecutive execution of s 1 and s 2. This means, the system witnessed frequently that the condition of s 2 holds after execution of s 1. An important point is, that this kind of implication is purely based on statistics. 2.2 The Evolution of the World-Model The robot evolves the directed graph representing his world-model in a step-wise manner. In a time-step t the robot chooses randomly between creation of a new SRR or edge and training of the current world-model G t = (V t ; E t ) with V t is the set of SRRs at time-step t and E t V t V t is the set of edges. In doing so, a SRR-pointer, the so-called standpoint is used to mark the SRR that was executed last. It represents the system's present position in the world.

3 Learning as operations on the graph takes mainly place in direct graph-theoretic neighborhood of the standpoint i.e., on M sp = fs 2 V t j(standpoint; s) 2 E t g, the set of SRRs with an edge leading to from the standpoint. This set is so to say the population of our evolutionary algorithm. Therefore, it is not necessary to process the whole world-model learned so far, but a very small subset to breed new knowledge. A new SRR s is created by using parts of SRRs from Msp. In doing so, the systems focuses on SRRs from Msp with good records according to the soon dened quality-measures. Creation of a new edge is done by searching a SRR s with holding condition in the current world-model G t. If the search is successful, an edge from the standpoint to s is included in G t+1 and s is executed. A training-step is done as follows. A SRR s with holding condition is randomly chosen among M sp and executed. Training is necessary to get the statistical records for the quality-measures. After every system-step, no matter if training or creation, all bad with respect to the quality measures SRRs and edges are deleted. 2.3 The Quality-Measures The quality-measures are designed to rate the usefulness of the robot's discoveries. They are a kind of universal tness-function in the evolution of the world-model. They use two simple concepts reliability and applicability and are based on the fact that a SRR or an edge represents an assumption: The execution of a SRR is successful if afterwards 1.) the condition is not holding i.e., the action has changed the state of the environment, and 2.) the feedback became true before the action was repeated more than constant rep max times, and 3.) the result is holding. An edge e = (s 1 ; s 2 ) is successful if the execution of its target-srr s 2 is successful. The reliability of a SRR s, respectively edge e is dened as number of successful executions of s (or e) per number of executions of s (or e) The applicability of a SRR s, respectively edge e is dened as number of executions of s (or e) per lifetime of s (or e) where lifetime is the number of time-steps of the system since creation of s (or e). The quality of a SRR s, respectively edge e is reliability times applicability: number of successful executions of s (or e) per lifetime of s (or e ) 2.4 Using the World-Model Using a graph as world-model planning becomes very easy. As mentioned before, a SRR-pointer standpoint models the system's current position in the world. It is always set on the last executed SRR. Given a desirable state g of the environment the system can search its world-model for a SRR s where the result of s holds on g. To achieve the goal g the system can execute the SRRs along the shortest path from the standpoint to s.

4 3 Learning Eye-Hand-Coordination 3.1 The Task According to the Swiss psychologist Piaget [3] child development proceeds in stages. The rst one is the sensorimotor stage, characterized by eye-hand-coordination. This can be compared to the task of learning to control a robot-arm via a camera (Fig.2): Camerapicture Robotgripper (red) Robotarm (black) Buildingblock (yellow) Background (black) Figure 2: The set-up A camera looks perpendicularly on a black background with some colored buildingblocks on it. A red robot-gripper, the so-called hand, can be moved by one of the actions north, south, west, or east in one of the four directions of the plane. Furthermore, the hand can grip the building block if its right under the hand and carry it around. It can release the block with the action ungrip. 3.2 The World-Model The world-model for moving the hand is a kind of grid. It consists of SRRs of the following form: \If I see the hand at position P, I can take (hand moving) action A and the hand will be visible at position P 0." Four edges are leading from every SRR except those on the edges of the grid to its neighbors in the four directions of the plane. If the system is told to get the hand to a position P in the current video image, the system searches a SRR s with a result representing P and executes the actions of the SRRs of the shortest path from the standpoint to s. The complete world-model for hand-movements and manipulating building-blocks consists of two such grids. The rst one is the above described \eye-hand-grid". The second one describes how building-blocks can be carried around. The two grid are interconnected by SRRs for proper grasping and un-grasping building-blocks i.e., grip is only used if and only if the block is under the hand, ungrip is only used if and only if the block is held. The hardness of the task to learn this world-model is twofold. First, the system has to learn appropriate representations of its hand and blocks dependent on the kind of tests used. We will present shortly results from experiments with two dierent kind

5 of tests. Second, the system has to nd out what is important to look at. For example the system has to discover that hand-movements are independent of block-positions, but grasping is not. 3.3 Teststrings and Gary L. Drescher In one class of experiments the camera picture is sectioned as a grid. In each eld of the grid the most frequent color in it is determined and written to a vector of so-called input-nerves. As tests we use strings as follows. Every position in the string corresponds to a position in the vector of input-nerves. A character in the string is either a color or a so-called joker. A teststring is fullled if and only if every character is either equal to the current value of the corresponding input-nerve or a joker. Test-strings are created with three universal operators: Adaption produces a test-string S as \snapshot" of the current environment i.e., the entries of S are set equal to the entries in the input-nerves. Mutation produces a new test-string S from an existing one S 0 by replacing an entry in S 0 by a joker at a random position. Crossover produces a new test-string S from two existing ones S 0 and S 00 by copying the head of S 0 and the tail of S 00 with respect to a random position j. The operators work on a population of tests from SRRs from M sp, the set of SRRs with an edge leading to from the standpoint. In doing so, they focus on SRRs with high tness according to the quality-measures. Gary L. Drescher presents in [2] an own approach to learn eye-hand-coordination. His best run on a Thinking Machines CM2 (16K processors, 512 Mbyte main memory) ended after one day with memory overow. His system learned approximately 70% of the desired world-model. A corresponding world-model is found with Stimulus Response Learning in 25 seconds on a SUN Sparc 10 completely. The total amount of memory used is less than 250 Kbyte. Furthermore, Drescher's experiments are done in simulations only. All experiments with Stimulus Response Learning were done in real world set-ups as well. In doing so, the system was very successful in dealing with noise and errors. 3.4 Unprocessed Real-World Images In the most challenging class of experiments a real-world set-up is used with video images with resolutions up to pixel under changing illumination conditions (weather). Furthermore, no vision processing is used. So, pictures of the hand and the building-blocks are disturbed by reexes, shadows, changing brightness etc. In this class of experiments, tests are represented as programs in a simple turtlegraphics language with commands to move the turtle, to draw lines, and to do a for-loop. A test is fullled if and only if the output of the program is contained in the current video image. Instead of adaption, mutation, and crossover, following operators are used to create turtle-tests: Conc : Takes two programs and concatenates them to one.

6 Split : Splits a program into two new ones at a randomly chosen place. Hill-climbing on constants : Performs a hill-climbing-step on a randomly chosen constant in a program to minimize a picture-distance-function. The picture-distance-function is a heuristic to measure the similarity of two images. Details can be found in [1]. In simulations the system learned very fast hand-movements and grasping. In doing so, it represented the hand as red triangle and building-blocks as rectangles. Due to this promising results we thought adaption to a real world set-up to be easy. We expected usage of vision lters and some fuzziness of tests to be sucient. But a series of experiments showed, that in the real world the camera input contains no triangles or rectangles. The hand and the building-blocks have no simple geometric representation. Noise due to reexes and shadows is always present. An unexpected solution of this problem was found by the system itself. We started a run using unprocessed real world images as input to see what would happen. After few hours the system started to move and grip successfully. It invented a kind of edge-detection. The system represented the hand and building-blocks via programs testing parts of the contour. Following analysis of the camera input revealed that these lines are hardly disturbed in our set-up. Therefore, they can be used by the system to determine the position of the hand and building-blocks. This is realized by the system with the help of the quality-measures. This result was achieved not only once. In every run done so far hand movements and grasping were learned using this edge-detection. The complete world-model for these tasks is learned in approximately 50 hours on average. In doing so, the run-time is dominated by the speed of the robot arm. 4 Conclusion We presented stimulus response learning as a new approach to enable robots to learn an adaptive and robust control-mechanism. This control-mechanism is based on a behavioral model of the world, that is learned with an evolutionary algorithm. The basic algorithm is neither dependent on the robot nor its environment. We presented experiments to learn eye-hand-coordination i.e., to control a robot-arm via a camera. Our results are not from simulations only, but as well from real-world set-ups. References [1] Andreas Birk. Learning geometric concepts with an evolutionary algorithm. In Proc. of The Fifth Annual Conference on Evolutionary Programming. The MIT Press, Cambridge, [2] Gary L. Drescher. Made-up minds, A constructivist approach to articial intelligence. The MIT Press, Cambridge, [3] Jean Piaget. Gesammelte Werke. Klett-Cotta, Stuttgart, 1991.