A TWO-DIMENSIONAL CONTINUOUS ACTION LEARNING AUTOMATON

Transcription

1 Control 4, Universit of Bath, UK, September 4 ID-34 A TWO-DIMENSIONAL CONTINUOUS ACTION LEARNING AUTOMATON K. Spurgeon, Q. H. Wu, Z. Richardson, J. Fitch Department of Electrical Eng. and Electronics, The Universit of Liverpool, Liverpool, L69 3GJ, U.K. National Grid Transco plc, Warwick Technolog Park, Gallows Hill, Warwick, CV34 6DA, U.K. Kewords: Continuous Action, Reinforcement Learning, Automata, CARLA. Abstract This paper presents an epansion of the Continuous Action Reinforcement Learning Automaton to incorporate two-dimensional (D) actions. The most significant change to the original CARLA methodolog is the introduction of a matri J that is used to store the last known reward value for each action. Inference rules based on comparisons between the values held in J, the current reward value and the probabilit distribution function of the automata are then used to directl guide the non-linear Reward-Penalt scheme. The automaton is tested firstl in an environment where the optimum is subject to noise and secondl tracking a time varing optimal. In both cases the learning algorithm performs well, providing significant reinforcement to the current optimum whilst simultaneousl avoiding saturation even after long periods. Introduction Continuous Action Reinforcement Learning Automata, CARLA, were first introduced b Howell, Frost, Gordon and Wu [] who used a single dimension CARLA to learn the optimum values of 3 independent control variables in a vehicle suspension problem to great success and then for control of engine idle-speed []. However, cases eist where the optimum combination of all variables involved must be found, not simpl the optimum case for each variable found independentl. B their ver nature, discrete learning algorithms are limited in their accurac b the finite number of possibilities that the can choose from, whereas the CARLA method s accurac can in fact be set b the engineer as required in a trade off against computational requirements. The basic concept of RL is that good performance, i.e. actions that produce favourable responses from the environment, are rewarded, and poor performance is either ignored or penalised. For an in-depth discussion regarding the fundamentals of The corresponding author: Professor Q. H. Wu. Tel: ; Fa: ; qhwu@liv.ac.uk RL, the authors recommend Madahaven [3],Simha and Kurose [4] and Tillotson[5] for some alternative reward schemes for automata and discussions on reinforcement learning. Traditionall, automata using such schemes randoml select an action i from the finite set {,,..., r } X with probabilit p i to be tested on the environment. If the response β(n) to the action is favourable, that action is rewarded b having its probabilit of selection at the net iteration increased. The CARLA sstem overcomes the problem of discrete RL b replacing the set of action probabilities with a normalised continuous probabilit distribution function, valid over the range of the action and therefore allowing an value of the action within the range to be chosen, the onl limit being as mentioned before the accurac set b the designer and computational considerations. Design and Implementation of D-CARLA In this section, the theor and issues in practical implementation of the D CARLA sstem are presented. Here we will consider a D action to be optimised b a single CARLA. Let and be two bounded continuous random variables defined over the intervals X = [ min, ma ] R and Y = [ min, ma ] R respectivel. Hence, if we let w i = ( i, i ) be the action variable, then w is also a bounded continuous random variable, but is defined over the plane region W = X Y = [( min, min ), ( min, ma ), ( ma, min ), ( ma, ma )] R. The set of discrete action probabilities, P = {p, p,..., p r } is now replaced with a continuous probabilit distribution function f(,, n) or f(w, n). It should be remembered that this function is in fact a 3D surface that subtends a volume which must satisf the following conditions before each iteration. f(,, n)dd = f(w, n)dw = () X Y W f(w, n), Initiall the distribution should be uniform across the region and the unit constraint maintained: { f(w, ) = ( ma min)( ma min) w W () otherwise

2 Control 4, Universit of Bath, UK, September 4 ID-34 With the initial distribution function in place the iterative process used to update f(w, n) is as follows:. Randoml select an action using f(w, n). Use action on the environment 3. Measure performance of action 4. Update f(w, n) based on performance measure β(n) Action selection in the D CARLA sstem uses the same principles as those used in the original CARLA, i.e. to randoml select (n) the following equalit is used: (n) min f(w, n)d = z (n) (3) and similarl to find (n) (n) min f(w, n)d = z (n) (4) where z (n) and z (n) are bounded uniform random numbers in the interval [,], i.e. z (n), z (n) U[, ]. What these equations mean phsicall is that to find the action (n), the volume under the surface f(w, n) is to be integrated along the ais until equal to z (n). The value of that satisfies this condition is then chosen as the value of (n) to be used at the net iteration b the automaton on the environment. Similarl, to find (n) the volume under the surface f(w, n) is integrated along the ais until the condition in equation (4) is satisfied. In practice the surface f(w, n) is stored as a matri F containing intermittent values of the function and linearit assumed between these point, hence the greater the resolution of the matri the greater the accurac. If we assume that columns hold cross sections of f(w, n) in the -direction, then rows hold cross sections in the -direction. Hence, to integrate the volume under f(w, n) in the -direction we firstl calculate the integral under the first two cross sections. This is done numericall using the intermittent values and the trapezium rule, N.B. for simplicit it is assumed that F is a square matri. The integral of the first cross section is: Int, = f(w, n)d min = d h N, + j= h,j + h,n (5) where N N are the dimensions of matri F, d = N is the step size, and h,j the height of the surface at row (since = min ) and column j of matri F. Similarl the integral of the second cross section is: Int, = d h N, + h,j + h,n (6) j= Let us now represent the two area integrals b two rectangular planes of unit width, with heights equal to the respective integrals calculated. If we also assume a linear change in cross-sectional area of f(w, n) as we move in the -direction, then the problem of volumetric integration is once again reduced to the trapezium rule: V = Y min +d min f(w, n)d = d (Int, + Int, ) (7) The above equation can now be epressed in general terms: i+d V = f(w, n)d = d Y i (Int,i + Int,i+ ) (8) where i =,,..., N and i = min + (i )d. Using this equation we can now formulate an iterative process to find the value of (n) which will satisf the condition stated in equation (3). for i =,,..., N. Initiall set T otal V olume Integral (T V I i ) = and i =. Calculate cross sectional integrals Int,i and Int,i+ 3. Using these values calculate the increase in volume integral V using trapezium rule (equation (7)) 4. Add the increase to T V I i+ = T V I i + V 5. Increment i = i + 6. If T V I i+ > z (n) then solve for (n) using equation (9) (n) f(w, n) = (Int,i + Int δ ) δ = z (n) T V I i i (9) Where Int δ is the vertical height of a plane of unit width representing the cross-sectional integral of f(w, n) at the point i + δ. Since we are assuming linear behaviour between the stored values of f(w, n), then at a distance δ from i the vertical height of the plane can be found b using the equation of a straight line, i.e. Int δ = Int,i + g δ, where g = (Int,i+ Int,i )/d. If we let δ V =

3 Control 4, Universit of Bath, UK, September 4 ID-34 z (n) T V I i, then we can now calculate the integral in equation (9) to be (n) f(w, n) = (Int,i + Int,i + g δ ) i δ = δ V () Rewriting the equation so that it is equal to zero gives us, gδ + Int,i δ δ V = () This is now simpl a quadratic equation which can be solved using the quadratic formula to give (Int,i ) ± (4Int,i ) + 8gδ V δ = () g The solution we require will alwas be the root that uses the positive square root in the numerator. Hence (n) = i + δ +. The same process is used to find (n) ecept that we now consider cross-section in the perpendicular direction. Now we are able to find (n) and (n) at each iteration, the net stage of implementation we must consider is that of updating the probabilit distribution function f(w, n) based on the response of the environment. Initiall the RL scheme used to update f(w, n) was a Linear Reward Inaction (L R I ) scheme as used for the original CARLA sstem as it had been proven to work effectivel. The scheme is of the form shown in the following equation: f(w, n + ) = { α[f(w, n) + β(n)h(r, w)] w W otherwise (3) ) H(R, w) = λep ( R σ (4) where R is the radial distance from the current action point w(n) = ((n), (n)), i.e. R = ( (n)) + ( (n)). H(R, w) is a Gaussian Bell Function about the vertical ais, centered on w. λ, σ [, ] are variables set b the designer to control the height and width of the bell function respectivel. The height determining the rate of convergence and the width determining the accurac of the solution. The values of the parameters chosen for these eperiments where, λ =.75 and σ =.5. These were chosen to provide a relativel tall et sharp peak for a balance between rapid response and accurac. Since the Gaussian function is valid from to +, the volume increase in the region W is less than the total volume under the bell, found to be β(n)πλσ. To find how much less using mathematics is a ver complicated task, so instead the total volume under the surface [f(w, n) + β(n)h(r, w)] is calculated numericall using the trapezium rule as before. Alpha is then simpl the reciprocal of this value. ( α = [f(w, n) + β(n)h(r, w)]dw) (5) W Practicall what happens is that ever value of F is multiplied b alpha therefore reducing the height of ever point on the surface and consequentl the volume integral. The point of this normalisation is to satisf the condition set in equation (). β(n) is calculated b using a set of logical rules that help determine what changes in the response of the environment impl and therefore guide the learning process in a more direct approach. What this achieves, as is shown in section 3., is that the automaton has the abilit to detect when a previous optimum ma have become redundant and promote possible new optimums when the are detected. This allows for fast tracking of a time varing optimum using relativel few iterations i.e. onl to 3. The logical rules are based on comparisons between the previous rewards returned b the environment, the current reward value and the mean and maimum value of all measured responses. As with f(w, n) the values of the measured responses are stored in a matri J which has the same dimensions as F, each element representing the last measured response to the region of actions that it represents (N.B. each element in J is initiall set to zero). The rules are given as a series of IF-THEN conditions, where j(w, n) is the current reward for the action w, chosen at iteration n, j(w, n ) is the previous value of the reward for that action, j δ (w, n) is the difference between j(w, n) and j(w, n ) and J is the mean of all the values stored in matri J:. IF j δ (w, n) < AND j(w, n ) J AND j(w, n) J THEN we have a point that ma be a previous optimum or have been close to an optimum and is still close to a possible optimum. IF j δ (w, n) < AND j(w, n ) > J AND j(w, n) < J THEN we have a point that ma be a previous optimum or have been close to an optimum and is now not significantl close to an optimum 3. IF j δ (w, n) < AND j(w, n ) < J THEN we have a point that was not significantl close to an optimum and is still not close to an optimum 4. IF j δ (w, n) > THEN we have a point that is now closer to an optimum than it was before 5. IF j δ (w, n)) = THEN the significance of the point has not changed

4 Control 4, Universit of Bath, UK, September 4 ID-34 The ke points in this set of rule are Cases and 4 for it is these that tell us whether a point mabe either a potential new optimum or close to a new optimum, or whether a previous optimum s significance ma be subsiding. These rules are used to set a parameter ξ which controls the sign and magnitude of β(n): β(n) = j(w, n) k ξ (6) where F (w, n) l Iff Case is True ξ = j(w, n)/j(w, n ) Iff Case 4 is True otherwise (7) (8) The parameter k is an integer chosen b the designer to suppress actions with lesser rewards and promote onl those with relativel high values of reward. Also, the actual value of F (w, n) is incorporated into the response to Case being True to act as a self scaling mechanism, where again l is an integer chosen b the designer to decide the rate of suppression of previous optimums. If a previous optimum has been heavil reinforced, then F (w, n) will be ver high and the action w will occup a large percentage of the total volume under the surface. This means that if Case arises and this point is a previous optimum, then the previous optimum can be reduced quickl, therefore allowing for possible new optimums to be promoted. For this eperiment k = and l = 5. Because of the scale of J and the need for a quick response from the CARLA a proviso was made to the updating mechanism of J. Tpicall a grid of is used to represent the action range, which means that there are 44 elements to be filled in J, meaning that man elements ma still be zero after even a thousand iterations. To speed up the process, if an elements contain a zero, then not onl will the be updated but so will an immediate neighbors also containing zeros. This means that a possible nine elements can be filled at one time, therefore greatl reducing the amount of time it takes to fill J. This now full describes the implementation of the D CARLA sstem. 3 Testing and Results Firstl the automaton was required to find a static optimum in an environment that was subject to noise, and secondl the automaton was required to track a time varing optimum during several discrete changes. All the tests were carried out with unit ranges of both and. 3. Nois Optimum An arbitrar static optimum w = (.5,.5) was chosen and the reward/cost field of the environment defined as follows. J(w) = [ λ J ep ( R σ J ) ] +. ma(j(w)) (9) where λ J =, σ J =. and R is the radial distance from w (n), i.e. the nois optimum. The static field is shown in the upper left part of Figure. Noise is added to the optimum b letting w (n) = w γ(n), where γ(n) = [γ (n), γ (n)] and γ (n), γ (n) N[.85,.5]. It must also be noted that γ (n) and γ (n) are chosen independentl. The upper right part of Figure shows the trace of the moving optimum, the blue trace tracking the dimension and the green trace tracking the direction. The automaton was given 3 iterations to find the optimum solution, the lower left part of Figure shows the probabilit distribution field of the automaton after 3 iterations. J(w,n) R, R *, * Iteration Iteration Figure : Results of D-CARLA Nois Optimal Test After 3 From the traces of and in the lower right part of Figure we can see that the D- CARLA correctl finds the optimum point to be (.5,.5). The test was then repeated and after 6 iterations the optimum point had remained constant and the average reward function had increased from AvgReward(w, 3) =.66 to AvgReward(w, 6) =.7 showing two things, i) that the D-CARLA will further reinforce the optimum with time, and ii) that within just 3 iterations the D-CARLA can provide an accurate optimum with a comparabl high reward. 3. Time Varing Optimum The net test for the D-CARLA was to track a time varing optimum which will take on four dif-

5 Control 4, Universit of Bath, UK, September 4 ID-34 ferent values over the course of the learning and the D-CARLA is given 3 iterations to learn each optimum before it is changed to the net point. The four optimum points are given in equation () where, n is the number of iterations. The environment reward/cost fields where formed using equation (9). (.5,.5) n < 3 w (.5,.75) 3 n < 6 (n) = (.75,.5) 6 n < 9 (.75,.75) 9 n < () *, * *, * *, * *, * Figures and 3 show the results of the moving optimum test. The top left of each figure showing the state of the CARLA after 3 iterations, the top right after after 6 iterations, the bottom left after 9 iterations and the bottom right after iterations. Figure shows the D-CARLA s probabilit distribution functions after each stage of learning. The optimum traces shown in Figure 3 show that the optimum in each case was found accuratel b the D-CARLA. The green traces of Figure 3 showing and the blue traces showing. The values of the average rewards for each probabilit distribution function are as follows, AvgReward(w, 3) =.395, AvgReward(w, 6) =.65, AvgReward(w, 9) =.693 and AvgReward(w, ) =.684. All four of the average rewards are greater than that of the nois optimum test, which is as to be epected and again show that a high degree of accurac in the optimum selection can be achieved b the D-CARLA within onl a short time. 3 3 Figure 3: Traces showing optimal point in action field at each iteration tri J) and reinforcement learning, provided a quick and accurate response to changes in the environment and are robust against pollution of the optimum b noise. The value of the average reward functions ma seem low, however for future work the authors see the possibilit of using such a sstem as part of a team of hierarchical automata [6] [7]. The job of the original D-CARLA being to find possible optimum regions quickl, whilst then bringing further automata into action with much narrower action ranges to provide more accurate action selection and therefore improve the average reward of the sstem. The original D-CARLA then being used as a pseudo supervisor control to watch for an significant global changes. 4 5 References Figure : D - CARLA Probabilit Distributions For Each Optimal 4 Conclusions and Future Work The results clearl demonstrate that the combination of inference rules (based on the past performance ma [] Howell, M.N., Frost, G.P., Gordon, T.J., Wu, Q.H.: Continuous Action Renforcemnt Learning Applied to Vehicle Suspension Control, Mechatronics, 997, Vol 7, No 3, pp [] Howell, M.N., Best M.C.: On-line PID tuning for engine idle-speed control using continuous action reinforcement leraning automata, J. Control Eng. Prac., 999, Vol 8, pp [3] Mahadevan, S.: Average Reward Reinforcement Learning : Foundations, Algorithms and Empirical Results, Machine Learning, 996, Vol, pp [4] Simha, R., Kurose, J.F.: Relative Reward Strength Algorithms for Learning Automata, Department of Computer and Information Science, Universit of Massachusetts, Amherst, Mass. 3 [5] Tillotson, P.R.J., Wu, Q.H., Hughes, P.M., Multi-agent learning for routing control within

6 Control 4, Universit of Bath, UK, September 4 ID-34 an internet environment, Engineering Applications of Artificial Intelligence, Special Issue Intelligent Control and Signal Processing - Vol.7, No., 4, pp [6] Wu, Q.H.: Reinforcement learning control using interconnected learning automata, International Journal of Control, 995, Vol.6, No., pp.-6. [7] Mikk, E.: HA - format, Institut für Informatik und Praktische Mathematik, Preußerstr. -9, D45 Kiel, German, Nov. 998