Department of applied mathematics. Mat Individual Research Projects in Applied Mathematics course

Transcription

1 Department of applied mathematics Mat Individual Research Projects in Applied Mathematics course Use of neural networks for imitating and improving heuristic system performance Jens Wilke 41946R Tf - N

2 Table of contents 1 Introduction Overview Setup overview Study phases Creation of heuristic system Creation of imitating neural network system Fine tuning of neural network weights Neural networks overview 5 3 Study setup Simulation environment Seeker inputs and outputs Seeker inputs Seeker outputs Heuristic seeker Turning Acceleration and deceleration Performance of the heuristic seeker Selected neural network architecture Network performance evaluation Tuning of network weights Searching the variable to modify Modifying the selected variable 18 4 Results Performance of the initial neural network Performance after iterative training Quantitative performance Qualitative performance 22 5 Conclusions References Literary references WWW-references 24 2

3 1 Introduction This document has been made to document the work done of Mat Individual Research Projects. The aim of this work is to study the use of neural networks when seeking solutions for improving the performance of heuristic rule based systems. This work evolves from a problem where a seeker system has to track a target object. A rule based heuristic system is created as a reference system. Then a neural network is trained to imitate that system. After successful neural network training, the network weights of neural network are altered in iterative manner in order to improve the performance of the neural network system. The objective is to create a better performing neural network based system than the original heuristic system. This work has been inspired by a system with pursuing and evading units whose performance was gradually improved. In the mentioned work, from an initial population of random network designs, successful designs in each generation were selected for reproduction with recombination, mutation, and gene duplication (Cliff et al. 1996). The underlying solution of this work and the referenced system are fundamentally different. This work uses neural networks for imitating the behavior of heuristic systems. This was inspired by a master s thesis done at HUT TML laboratory (Teirilä 1999). 2 Overview This section takes a look at the study setup and underlying fundamentals of neural networks. 3

4 2.1 Setup overview The setup of this study consists of a system, called seeker, trying to trace another item called target. Seeker has several attempts in different scenarios. After going through all the scenarios, the performance of the given seeker is evaluated. Seeker and target have both some general characteristics. They move on twodimensional plane. Target has constant heading and speed. Seeker can turn and accelerate or decelerate. The turning is limited by maximum turning angle per iteration. Seeker has also a limiting top speed. During the study different seeker control solutions are tested, i.e., heuristic and neural network based systems. The fundamental aim of this work is to test, if the performance of a neural network can be improved by gradually modifying the neural network parameters in order to improve its performance. This can be a useful feature if it is difficult to understand the functionality of the system, but it is easy to evaluate the performance of individual slightly altered systems. The tracing simulations have been done using Mathworks Matlab (Mathworks 2002). The Matlab files created are available for download and testing from author s homepage (Wilke 2002). 2.2 Study phases Following three sections list the activities in different phases of this work Creation of heuristic system First, a heuristic system was generated. Its tracing performance was tracked and the inputs and outputs of the heuristic system were stored. In other words, during the simulated evaluation of the heuristic system, in every situation the inputs and outputs of the system were stored into a file for later retrieval. 4

5 2.2.2 Creation of imitating neural network system After assessing the performance of the heuristic system, a neural network based system was created. The neural network based system was trained to imitate the actions of the heuristic system. During the training, the system was trained using the stored heuristic data, which was generated as the heuristic system s performance was assessed Fine tuning of neural network weights After creating the initial neural network and assessing its performance, finetuning of its performance started. Fine-tuning was done by modifying the weights and threshold values of the neural network. The work consisted of iterative rounds gradually improving the performance of the neural network based system. 2.3 Neural networks overview A neural network is a usually highly interconnected network of multi-input nonlinear processing units called neurons. The name neural network becomes from the fact that the neurons used in the artificial neural networks resemble to certain extent the human neural cells. Basically, three entities characterize artificial neural networks: the interconnection of neurons, the characteristics of individual neurons and the strategy for pattern training. Many network architectures exist including multilayer perceptron, radial basis, recurrent, selforganizing networks to name but a few. The manner in which the neurons of a neural network are structured is intimately linked with the learning algorithm used in the training of the network (Haykin 1994,1-22). Schalkoff discusses the use of neural networks in pattern recognition and pattern association, using multilayer feedforward networks (Schalkoff 1992, 237). Accordingly we chose to use multilayered feedforward networks in this work. The architecture of such a network consists of several layers of neurons. Each neuron has a number of inputs x i and respective weights w i. If the weighted sum of the inputs exceeds a certain threshold limit θ, an output is produced. An individual neuron is depicted in the figure 1. 5

6 X 1 W 1 X 2 X 3 W 2 W 3 Output Y k X p W p Threshold θ Figure 1 And individual perceptron The neurons are then arranged into layers where each layer s outputs function as the inputs of the following layer. A two-layered network is pictured in the figure 2. The behavior of the network is altered by changing the parameters of individual neurons. Input layer Hidden layer Output layer Figure 2 A network of interconnected perceptrons Before its use as a black box element the neural network has to be trained to its task. The design procedure for neural network pattern associator involves the following phases (Schalkoff 1992,221): 6

7 1) Defining suitable inputs/outputs and network structure 2) Choosing training method 3) Training the network 4) Performance assessing 3 Study setup 3.1 Simulation environment The testing scenario should be rich in that sense that it would have a large set of tests so that system behavior would be consistent in other somewhat different scenarios. In the test scenarios, seeker starts always from the origin of the system with the same initial speed as the target. Altogether 24 test cases are executed in order to assess the performance of each individual system. The target starts from 8 different initial locations. From each initial location the target starts traversing to 3 different initial directions. In figure 3, the 24 test cases are visualized: the seeker is marked with triangle with its initial speed vector and the 24 target setups are marked with 8 initial target locations and 3 initial speed vectors for each initial target location. Figure 3 Initial scenarios 7

8 The motivation of identical initial setups when comparing the system performances is the fact that performance differences between the systems are often minimal. If the start setup would vary, it would be probable that in some case not the best performing system would be selected for later iterations. A more advantageous setup could allow a weaker system to outperform a superior system. When the setup remains the same, it is assured that the selected system is the best performing one for the given environment. 3.2 Seeker inputs and outputs The inputs and outputs of the heuristic and neural network systems were selected to be identical. Consequently, direct system performance comparison and evaluation is possible. Following sections define the inputs and outputs of the seeker system Seeker inputs Seeker should receive information about the location and movement of the target. The number of inputs has to be limited but sufficient so that the seeker has all needed information for directing itself efficiently towards the target. Intuitive selections for the seeker inputs are the relative location and the relative heading of the target. The figure 4 below represents the relative positions and headings of seeker and target. The seeker is located in the origin and is heading in the direction of the Y-axis, and the location and heading of the target are given relative to the seeker. On the right side of the figure 4, the velocity vectors of the seeker and target are broken into components. 8

9 V see ker h see ker h perpendicu lar V x T arg et V seeker V T arg et V y T arg et V t arg et Figure 4 Relative locations and speeds of the seeker and the target If the location of the target is marked with X t arg et and the location of the seeker is marked with X see ker, the difference vector and heading vector are: d = X t arg et - see ker X h seeker = v v seeker seeker d directional = d h see ker d perpendicular = d h perpendicular The two values d directional and d perpendicular give information about the location of the target relative to the seeker. The first value d directional is the dot product of seeker s heading and the vector defining the location of the target relative to the seeker. This scalar value ranges from minus infinite to positive infinite. d directional is positive, if the target is located ahead of the seeker and vice versa. The other value d perpendicular is formed similarly, but measures the deviation perpendicularly to the seeker. d directional is positive if the target is located on the right side of the seeker and vice versa. 9

10 v directional = v see ker v t arg et v perpendicular = v perpendicular v t arg et The values v directional and v perpendicular give information about the speed of the target relative to the seeker. The first value v directional is dot product of seeker s speed and the target s speed vectors. It gives information about their relative speeds along the axis of seeker s heading. The second value v perpendicular on the other hand gives information about the speed of the target perpendicular to the seeker s heading. v perpendicular is positive, if the target is heading rightwards relative to the seeker and vice versa. The input should be scaled so that later on the input of the neural network is limited to some specific range. This is done in order to improve the performance of the neural network. Sigmoid function is used to scale the value of the inputs. The equations below define the hyperbolic tangent sigmoid scaling of the inputs of the neural network. d dirscaled 1+ e 2 = 2 d directiona l 1 d perpscaled 1+ e 2 = 2 d perpendicu lar 1 d dirscaled 2 1+ e = 2 V directiona l 1 The figure 5 below shows how the sigmoid function scales the input space to be between the values -1 and 1. It should be noted that the scaling preserves the sign of the input. 10

11 Figure 5 Sigmoid function Seeker outputs The amount of outputs should be kept as low as possible due computational reasons. When one thinks about a car, it has basically two controls: 1) Turning 2) Acceleration and deceleration Similarly, the number of seeker system outputs was set to two. 3.3 Heuristic seeker Heuristic seeker is guided by rigid rules when tracing the target. As defined in the section Seeker outputs, the heuristic logistic has to make following decisions when tracing the target: 1) How much to turn its course 2) How much to accelerate or decelerate Turning Turning is based on the location of the target relative to seeker. Figure 6 visualizes the distribution of relative locations to four distinct zones. 11

12 X X Zone 1 Zone 2 Zone 4 Zone 3 Figure 6 Heuristic turning logic The actions of the seeker are summarized in the table below. In the figure 6, the seeker is presented as a triangle in the origin. Target is represented as a circle, which is located more than X to the right of the seeker. In the represented situation, the seeker would turn right. When the heuristic seeker turns, the absolute magnitude of the turn is always constant. Table 1 Zones and corresponding activities Target location Zone 1 and Zone 3 Zone 2 Zone 4 Action Turn Left Don t turn Turn right Acceleration and deceleration As with turning, the acceleration and deceleration of the seeker is based on the location of the target relative to seeker. Additionally, the heading of the target is also observed. The target can be either in front of the seeker, i.e., in zone 1, or behind the seeker, i.e., in zone 2. Secondly the target can be heading in the 12

13 same or opposite direction as the seeker. The situations are visualized in figure 7. Zone 1 Target 1 Target 2 Target 3 Zone 2 Target 4 Figure 7 Heuristic turning logic 4 decision rules are constructed. They summarized in the table below. Table 2 Acceleration and deceleration Target location Target heading relative to seeker Seeker action Example in figure Zone 1 Same Accelerate Target 1 Opposite Accelerate Target 2 Zone 2 Same Decelerate Target 3 Opposite Decelerate Target Performance of the heuristic seeker After creating the heuristic seeker, its performance was analyzed qualitatively. The seeker managed to catch the target always, if enough time was given. Generally, the heuristic seeker was very efficient. The images in figure 8 13

14 illustrate seeker performance. In the figure, seeker starts tracing from the origin. Seeker s tracing path is curved and marked with darker blue color. The target starts from varied locations. Its path is straight and marked with brighter green color. Figure 8 a, b Heuristic seeker behavior Figure 8 c, d Heuristic seeker behavior 3.4 Selected neural network architecture Due to computational efficiency it was decided, that the neural network should be as simple as possible. The evaluation of neural network architecture started with multilayer perceptron network having 4 inputs and 2 outputs similarly as the seeker. Testing started with two perceptron layers each having 2 neurons. Consequently, the first perceptron layer has 4 weighted inputs and the second layer has 2 weighted inputs. Each of the 4 perceptrons has also threshold values as described in section 2.3 Neural networks overview. Consequently the 14

15 network had 4*2 + 2*2 * 4 = 16 parameters that could be varied. After initial tests, it became apparent that the neural network could imitate the performance of the heuristic seeker very well. The neural network training results vary on case-by-case, as the initial values of the network are somewhat randomly set. After several rounds of training and testing, the best performing neural network based seeker was only less than 1% weaker in performance than the heuristic seeker. The performance was measured using evaluation measure described in section 3.5 Network performance evaluation. Visually, no difference in tracing performance between the systems could be noted. 3.5 Network performance evaluation In order to evaluate the performance of each system, their performance is evaluated quantitatively. In each scenario, a scoring measure for the network performance was calculated. The aim of the seeker is to catch the target. Consequently, the score is based on two factors: 1) Steps needed to reach the target 2) Penalty for not catching the target If target was reached, the score is defined by the number of iterative steps needed in order to reach the target: Score = Steps needed If target was not reached, a penalty is added. The penalty is selected so that the penalty for missing the target is 10 times higher as the penalty for additional steps needed for reaching the target. The high penalty for target missing was selected because it was seen that target reaching is the primary goal of the seeker. Score = Steps needed + 10 * distance from the target After all 24 test cases, the individual scores are added in order to form a cumulated score for the tested system. The selection of the best performing 15

16 system is then purely based on the cumulated score. Shortly put, smaller score is better and the system with smallest cumulated score is selected for later iterative rounds. 3.6 Tuning of network weights Multilayer perceptron network is defined by the transfer functions of each layer, network weights and biases. Sigmoid function, described in section Seeker inputs was used as the transfer function in both layers, and it wasn t modified during the iterations. Network weights and threshold values were the only varied values. In the beginning of this work, different approaches to network tuning were tried. The first brute force tuning attempts were tried by using random network weight tuning. After initial attempts, no significant results were achieved and more sophisticated techniques had to be developed. Our goal is to minimize the score of the system, i.e., to maximize the efficiency of the seeker. The score of each system is evaluated after each round of simulations through all test scenarios. As the system is otherwise defined, the system performance depends purely on the network weights and perceptron thresholds. Tuning can be divided to two phases: finding the variable to modify, and modifying the given variable Searching the variable to modify Altogether there are 16 parameters to vary as noted in the section 3.4 Selected neural network architecture. Consequently we can view the system performance to be a function whose value depends on those 16 values: f ( x1, x2,..., x16) When searching for multidimensional optimum, the method of steepest descent is a fundamental procedure for minimizing a differentiable function of several variables. f ( x) / f ( x) is the direction of steepest descent minimizing the 16

17 value of the function (Bazaraa et al. 1993, 300). Unfortunately in this case, the neural network system is not differentiable. Consequently, the system has to be differentiated numerically. The values of the network variables are varied a small amount x i, and the simulation is conducted in order to define the difference to the previously best performing system. The differentiation is conducted using all 16 variables. f ( x 1, x2,..., x16 ) x 1 f ( x 1, x2,..., x16 ) x 16 It was decided conservatively to vary only one variable by experimenting with different amounts of variable variation. The variable, which gives deepest descent, i.e., largest improvement, is selected as the variable to be modified. The modification of this variable is described in the next section. After each simulation, it comes clear to which direction (if any) each network variable should be modified in order to decrease the performance measure. Unfortunately, it is not clear how much those values should be varied. On the first round of simulations, the value x i was defined to be 1% of each value x i. Due to somewhat long simulation times, on the second round of simulations, this value was increased so that network variables could be considerably altered already during the first phase when seeking for the network variable to modify. On this second simulation round, the value x i was defined to be 50% of each value x i. This could lead to situations where none of the systems created using varied weight and threshold would perform better than the original system. This would be due to too heavy variable modifications taking the system further from the optimum than in the first place. In such cases, the problem was addressed by halving the relative variation, e.g., decreasing the relative variation from 50% to 25%. Significant savings in simulation time were reached as seen in section Quantitative performance. 17

18 3.6.2 Modifying the selected variable As the variable to be varied had been selected according to previous section, different amounts of variation were tested in order to improve the system performance. Taking into account the scope of this work, no overly sophisticated system was generated for this purpose. The variations generated can be divided to relative and absolute. The relative variations are relative amounts of the variable value, and absolute variations are multiples of the value x used to differentiate the system in section Searching the variable to modify. The motivation of relative changes is to allow large changes when variables have large absolute values. The motivation of absolute changes is to allow the variables to change sign, when the variable value is close to zero. For example, if the value x would be 1, relative variations would have been defined to be 10 % and 20%, and the direction of variations would be positive, the results of varying value 100 could be following: * x = * x = * 100 = * 100 = 120 After modifying the given variable, each system (differing from each other only by one variable) is evaluated through simulation. The selected system is then defined to be the best performing system. Then the iterations continue by searching again for the variable giving the deepest descent as defined in section Searching the variable to modify. 18

19 Search for the variable to modify using small differentiations x i Decrease the differentiations x i A better performing system was found No Yes Modify selected variable by larger amounts Select new best performing system Figure 9 Iterative change of network variables If the minimal network weight value changes won't produce any improvement in the system performance, it can be assumed that the system is close to the local optimum of its performance. Further simulating using similar small changes won't improve the situation. This state is hypothetical, as it would take a very long to reach this state computationally. 4 Results This section takes a look at the results of this study. The quantitative estimates are based on the scoring algorithm. Qualitative analysis was based on visual analysis of the system performance. 4.1 Performance of the initial neural network As the initial neural network was trained to imitate the actions of the heuristic algorithm, it was assumed that the performance of the initial neural network would be lower than the performance of the heuristic algorithm. This was due to the fact that during the neural network training the weights of the neural network were adjusted so that sum square error would be minimized. As such 19

20 imitator is created, it is possible that the performance might be close the initial system performance, but unlikely that the imitating system s performance would exceed the performance of the initial system. This initial assumption proved to be true in all executed test cases. The heuristic system got performance score: The imitating neural networks were trained using the trainlm algorithm and the training data generated by the heuristic system. trainlm is a network training function that updates weight and bias values according to Levenberg-Marquardt optimization (Demuth 1994). The training method was readily available in Matlab s Neural Network Toolbox. The performance of the trained network varied due to randomized network initialization. During this work, in the beginning of simulations, each time a neural network was trained to imitate the heuristic system, its performance was compared with the previously trained networks. If the newly trained network s performance exceeded the performance of the previous networks, it was selected as the new best performing network. After some 20 attempts, the best imitating neural network system had a score: In practice the performance of the heuristic system and the imitating neural network system were identical as the performance score of the heuristic system was only % smaller than the performance score of the imitating neural network system. 4.2 Performance after iterative training This section takes a look at the quantitative and qualitative performance of the system that emerged after consecutive rounds of network variable adjustments Quantitative performance The neural network based system was initially weaker in performance than the heuristic algorithm that it imitated. The initial neural network based system 20

21 outperformed the heuristic system in 3 out of 24 test cases where it was set against the seeker based on the heuristic algorithm. The cumulated total score of the neural network based system was as opposed to the slightly better score of of the heuristic system. The simulations were done twice. On the first round, a differentation of 1% was used during the first round when searching for the variable to modify. On the second round, in order to accelerate the search, a differentation of 50% was used during the first round when searching for the variable to modify. The procedure is more precisely described in section Searching the variable to modify. First, a smaller differentiation was used of 1 % used. After 50 evolutionary rounds, the neural network based system could outperform the heuristic system in 14 out of 24 test scenarios where it was set against the seeker based on the heuristic algorithm. The final cumulated total score of the neural network based system after iterative training was with which it outperformed the score of the heuristic system. Figure 10 illustrates the development of system performance. Figure 10 System performance progress On the second simulation round, a larger differentiation was used in order to accelerate the search. After 50 evolutionary rounds, the neural network based system could outperform the heuristic system in 15 out of 24 test scenarios where it was set against the seeker based on the heuristic algorithm. The final 21

22 cumulated total score of the neural network based system after iterative training was with which it outperformed the score of the heuristic system. Thus the performance increase of the initial neural network based system was 6,2 %, i.e., decrease in system performance measure. Figure 11 illustrates the development of system performance measure. Figure 11 System performance progress Qualitative performance The behavior of the seeker didn't become dramatically different, but it became refined and somewhat smoother than the behavior of the heuristic seeker. As the neural network based system evolved, it started to anticipate the movement of the target more smoothly; incrementally improving its performance. 5 Conclusions This work demonstrated that the evolutionary variation of neural network variables can be used for improving the performance of heuristic rule based systems. The performance increase was somewhat slight as indicated in section Quantitative performance. On the other hand, the scope of this work is very limited and further study would be needed to determine the feasibility of this approach. Possible applications could be found in situations where one has a conception of the needed system functionality, but profound knowledge about the needed 22

23 system functionality is missing. In this case the system performance could be imitated and refined using the approach described in this work. Problems would be probably faced, as the complexity of the systems modeled would increase. More complex neural network architecture would be needed to model the functionality of the system. This would in turn increase the number of network parameters considerably. The increased number of network weights would certainly increase the time needed for incrementally seeking better performing systems. On the other hand, when training more complex networks, more sophisticated algorithms could be used to accelerate the iterative search for better performing systems. Another limitation is the fact that this approach is not capable of introducing major changes in system behavior. It would be difficult for the seeker to radically change its actions when the weights of the network are evolved slowly. Bigger changes of several variables could be needed when looking for radical changes in the seeker behavior. An example could be a situation when the seeker should anticipate the movement of target as the target is passing seeker from behind. Instead of turning left all the way and starting to track the target, it would be better if the seeker could anticipate the target's movement and turn right and cut in front of the target. This situation is visualized in figure 12. Early catch Late catch Seeker Initial target location Figure 12 Major change in seeker behavior 23

24 6 References 6.1 Literary references Bazaraa, Mokhtar and Sherali, Hanif and Shetty, C.M Nonlinear Programming. Theory and Algorithms. John Wiley & Sons, Inc. New York. Cliff, Dave and Miller, Geoffrey F Co-evolution of Pursuit and Evasion II: Simulation Methods and Results. From animals to animats 4: Proceedings of the Fourth International Conference on Simulation and Adaptive Behavior Demuth, Howard and Beale, Mark Neural Network Toolbox User s Guide. Third printing. The MathWorks, Inc. Massachusets. Haykin Simon Neural Networks a Comprehensive Foundation. Macmillan College Publishing Company, Inc. New York. Schalkoff, Robert J Pattern recognition: statistical, structural and neural approaches. John Wiley & Sons, Inc. New York. Teirilä, Juha Ihmishahmon käänteiskinemaattisen ongelman ratkaiseminen reaaliajassa. Diplomityö. Teknillinen korkeakoulu. 6.2 WWW-references The MathWorks Inc Matlab overview. Wilke Jens Homepage of Jens Wilke. 24