A motivated definition of exploitation and exploration

Transcription

1 A motivated definition of exploitation and exploration Bart Naudts and Adriaan Schippers Technical report at the University of Antwerp, Belgium. 1 INTRODUCTION The terms exploration and exploitation are as old as evolutionary algorithms themselves, yet few papers are truly devoted to this subject (e.g., [1, 2, 3, 4, 5]). There is even less agreement on the meaning of the terms. This paper can be seen as an exercise in formal reasoning to provide a well-understood, unambiguous definition of the terms exploitation and exploration. It is not our aim to distill a common denominator from the current usage of the two terms in the literature we refer to [3] for a detailed overview. Instead, we start from a simple algorithm framework and try to give a precise definition of exploitation and exploration in this context. Then we discuss these definitions in great depth. Population-based algorithms and algorithms with any sort of memory remain out of the scope of this paper, and will be considered in further contributions. The structure of the paper is as follows. In the second section, we set our context: a framework of simple evolutionary algorithms without population or memory. Then, in the third section, we define exploitation and exploration in the context of this framework. We proceed with examples (section 4) and an attempt at quantifying the different forms of exploitation and exploration encountered (section 5). The sixth section discusses some of the differences of our approach and the definitions found in the literature. We conclude this paper with a summary and some words on further work. 2 THE ALGORITHM FRAMEWORK In this section we formally define an algorithm framework which will be used in the next section as a context for defining the terms exploitation and exploration. Examples of algorithm instances of this framework are given in section 4. Note that since we only consider algorithms without memory and population, it is artificial to fit genetic algorithms in the framework by interpreting them as single-element algorithms in population space. Assume we are given a non-empty, finite search space S, whose elements we denote by s S. Implicitly associated with each element is a unique representation of this element. A neighborhood N(s) is defined in each element as a probability distribution on the set S. We require a representational relationship between an element and its neighborhood in terms of likelihood, i.e., elements with higher probability should be more related; that not all mass of the distribution is given to the element itself; that the neighborhoods are static, in the sense that they do not change during the run of the algorithm. The collection of neighborhoods is called the neighborhood structure. With each element s S we also associate an objective value f (s) R. To give a non-trivial meaning to the terms exploration and exploitation, the fundamental assumption of black-box optimization has to be satisfied: there must exist a (possibly unknown yet exploitable 1 ) relationship between the neighborhood structure and the objective function. The goal of an algorithm instance of the framework is to find an element satisfying a given stopping criterion which is dependent on objective values only. The algorithm framework is then defined as follows: 1. Choose an initial element s 0 S, and evaluate it to obtain f (s 0 ). Initialize the step counter i to If f(s i ) satisfies the stopping criterion, stop the algorithm. 1 Haha.

2 3. Using s i, generate the neighborhood N(s i ). 4. Construct a set T S by drawing elements from the distribution N(s i ) and compute the objective values of the elements of T. 5. Given only f (s i ) and f (t) for all t T, select one element t from T {s i }. 6. Accept t as s i+1, the next element of the algorithm. 7. Increase i, and go to step 2. Note that the step counter i is not used as a memory. One could as well use a 0/1 flipper as a counter. We could easily have done without step 3, since the neighborhood N(s i ) is defined for s i anyway. Instead of this mathematical approach we favor the constructive approach, which stresses the fact that the element is used to obtain the neighborhood. This approach will also be more suitable for the definitions in the next section. Of course, in an instance of this framework, not all elements of the neighborhood have to be generated or constructed physically. The construction of those of T would suffice. The selection of the subset T in step 4 can be deterministic or stochastic, but it cannot be biased by the representation of the elements of the neighborhood. Biasing the selection on the basis of the representation makes no sense because the neighborhoods are static and the algorithm has no memory. It would have the same effect as choosing different neighborhoods. By construction of the framework, any problem specific knowledge can only be put in either the neighborhood structure or the objective function. At the risk of being too restrictive, we have chosen to make the selection of the successor in step 5 fully representation independent. Given that step 3 is the only other step where information is used, we thus achieve a clear separation between the use of representational (step 3) information and objective information (step 5). 3 DEFINITION OF EXPLOITATION AND EXPLORATION It is clear that the important steps of the algorithm are steps 3 up to Step 3 Step 3 is the explicit restriction of candidate successors to the (purely representational) neighborhood of the current element. Relying on the fundamental assumption of blackbox optimization, we postulate that the action of step 3 is exploitative: the representation of the current element is used to generate candidate successors. This form of exploitation will be called representational exploitation. The probability distributions can be uniform over the whole space (e.g., random search, section 4.5), uniform over the Hamming cube (e.g., the single-bit-flip hill-climber, section 4.1), Gaussian (e.g., the (1+1)-EA on a real space, section 4.4), etc. 3.2 Step 4 Step 4 contains two actions: the selection of a subset for evaluation and the actual evaluation of the elements in the subset. As explained in the previous section, the selection of the set T is a process which is independent of the representation of the elements in the neighborhood. Examples are: always select all elements with non-zero probability (e.g., the single-bit-flip hill-climber, section 4.1), pick one element according to the distribution (e.g., the Metropolis algorithm, section 4.2 or random search, section 4.5), etc. We postulate that the actions of step 4 are explorative: new elements are selected for evaluation and then evaluated. Since this exploration is restricted to only a neighborhood by representational exploitation, we call it neighborhood exploration. 3.3 Step 5 In step 5, the information obtained by the neighborhood exploration (step 4) is used to select a successor from the candidate elements. We postulate this to be exploitative, and we call this form of exploitation objective exploitation, since only objective information is used here. 3.4 Step 6 Step 6 is the acceptance of the selected element as the successor. It is postulated to be explorative, since it moves the algorithm from one known element to a (known element) with an unknown neighborhood. This action is called generational exploration. When the current element s i is selected to become the next element, we say that there was no generational exploration. 3.5 Comments Note that we do not explicitly define exploitation and exploration in terms of operators like mutation and selection, although steps 3 and 4 together could be interpreted as a mutation and steps 5 as a selection operation. Neither do we relate exploitation and exploration to entities at sub-element level, because this can easily lead to problem dependent definitions and statements. For example,

3 we do not relate exploration to the recombination or generation of building blocks. When restricted to specific problem classes, the sub-element approach can give promising results (see [3]). 4 EXAMPLES In this section we consider some examples which fit the framework of section 2. Recall that the genetic algorithms only fit artificially in the framework. For this reason, they are omitted here. 4.1 A single-bit-flip hill-climber Consider a single-bit-flip hill-climber on the space of binary strings of length l. Clearly, S = {0, 1} l. We define the neighborhood of a string s S to consists of all strings at Hamming distance 1 from s (the Hamming cube of s). This is done by setting the probability of all elements of the Hamming cube to 1/l, and the probability of all other elements of S to 0. Representational exploitation is thus the restriction of the action radius of the hill-climber to the Hamming cube. The objective function f is the fitness function to be minimized or maximized. We define this hill-climber to perform full neighborhood exploration, i.e., all elements of the neighborhood with non-zero probability are evaluated before step 5 is started. Objective exploitation is only concerned with objective values: the element (or one of the elements) with the highest objective value is chosen to become the successor. When a single-bit-flip hill-climber is stuck in a local optimum, either a string with a worse fitness value is accepted, or the algorithm is restarted. The first option simply requires a different type of objective exploitation. The second option requires the representational exploitation (step 3) to be performed using distribution which is uniform on the whole space S. To summarize, the single-bit-flip hill-climber performs: representational exploitation: restriction of the search direction to the Hamming cube; neighborhood exploration: all elements are selected and evaluated; objective exploitation: select the best element from the cube; generational exploration: always accept the element. 4.2 Metropolis algorithm The Metropolis algorithm [6] is the core algorithm of simulated annealing. It contains everything but the annealing scheme: the temperature (or inverse temperature β) is fixed to a given value. Again we choose the space of binary strings of length l (S = {0, 1} l ). We retain the Hamming cube as the neighborhood of an element as in the previous section and take a fitness function f as objective function. As opposed to our single-bit-flip hill-climber, the Metropolis algorithm does not select its whole neighborhood for evaluation, but it simply draws one element from the uniform distribution. Hence, while representational exploitation remains the same, neighborhood exploration contains a stochastic component which selects one element from the neighborhood. (Note that this corresponds to the arbitrary selection of one bit to be flipped.) On the objective side, we choose to let exploitation make the acceptance decision, with generational exploration in case of acceptance and no generational exploration in case of rejection. To summarize, the Metropolis algorithm performs: representational exploitation: restriction of the search direction to the Hamming cube; neighborhood exploration: selection of one element out of the neighborhood and evaluation of this element; objective exploitation: acceptance decision based on the difference in objective value of the current and the new element; no representational selection; generational exploration: accept the new element in case of a positive acceptance decision; otherwise reject: no generational exploration. 4.3 A 1/l-mutation hill-climber A 1/l-mutation hill-climber can be constructed in many variants, but the new aspect is its neighborhood structure. As an operator, this mutation loops over the bits and flips them with probability 1/l, with l the length of the strings. In the neighborhood of s, the probability of an element t thus equals P(t) = ( ) 1 d ( 1 1 l d, (1) l l) with d the Hamming distance between s and t. The set T can then be composed in step 4 by drawing one or more elements from this probability distribution. 4.4 A strictly elitist (1+1)-EA on the reals Let us now give an example of a search algorithm on an n-dimensional Euclidean space. Let S = R n, and let

4 f : R n R be the objective function. The neighborhood of a point s S is given by a n-dimensional Gaussian distribution with mean s and variance σ R n +. Only one element is drawn from the neighborhood in step 4, and step 5 performs a strict elitist selection: only if the objective value of the new element is strictly greater than the objective value of the current element, the new element is accepted. To summarize, our (1+1)-EA performs: representational exploitation: restriction of the search direction to a Gaussian distribution around the current point; neighborhood exploration: selection of one element out of the neighborhood and evaluation of this element; objective exploitation: acceptance decision based on a strictly elitist scheme; generational exploration: accept the new element in case of a positive acceptance decision; otherwise reject: no generational exploration. Note that we avoid ESs where the mutation rate (and hence the variance of the Gaussian) is a part of the element, because we would have to give up the requirement of a static neighborhood structure. Implicitly, we would also add the capacity of a memory to the algorithm. 4.5 Random search We have seen in section 4.1 that performing representational exploitation using a uniform distribution over the whole space can be used to perform a restart. Permanently doing this results in a random search algorithm. Note that this type of representational exploitation violates our requirement of having a relation between the representation of an element and the probable elements of its neighborhood. We say that representational exploitation is disabled or inactive. A consequence of having representational exploitation permanently disabled, is that objective exploitation becomes meaningless, since there is no motivation for preferring one element over any of the other generated elements. This does not hold the other way round: if objective exploitation is permanently disabled (i.e., an element is chosen to be the successor without using representational or objective information), one could still implement a random walker or an enumerative algorithm (see section 4.6). To summarize, pure random search performs: no representational exploitation: elements are chosen stochastically from the whole search space; neighborhood exploration: evaluation of the selected elements; no objective exploitation; no grounds for making decisions; generational exploration without meaning: it is unimportant where the algorithm is; 4.6 Random walkers and enumerative algorithms A random walker can be constructed most easily by taking the single-bit-flip hill-climber, picking one element in step 4, and always selecting this new element in step 5. The same effect is achieved by taking the single-bit-flip hillclimber again and performing a stochastic selection in step 5 which is independent of the objective values. Note that in the latter case, more function evaluations are performed. The only effect, of course, is that the algorithm is slowed down. When one replaces the word stochastically by deterministically throughout the previous paragraph, one can easily construct a simple enumerative algorithm (a brute-force one that searches the whole space in a deterministic way). Let us summarize the actions performed by the random walker or the enumerative algorithm: representational exploitation: restriction of the search direction to the Hamming cube (in the case of a random walker) or some very restricted neighborhood (in the case of an enumerative technique); neighborhood exploration: stochastic or deterministic selection of one element from the neighborhood, and evaluation of this element; inactive objective exploitation: simply select the new element; generational exploration: always accept the new element. Note that the use of the terminology of spaces of bit-strings in this example is done without loss of generality. 5 A QUANTIFICATION In this section we give some hints related to the quantification of the different forms of exploitation and exploration. In the case of exploration, we can give formal definitions, but as long as none are given for exploitation, they have to be used with extreme care.

5 5.1 Representational exploitation As the name indicates, a quantification of representational exploitation plays entirely at the representational level: no objective values are involved. To start the discussion, consider the following two extreme examples: the Hamming cube as a neighborhood structure, and the unbiased (uniform) generation of arbitrary elements (i.e., inactive representational exploitation). In the first case, there is a strong representational relation between the current element and the elements of its neighborhood, while in the second case the elements are by definition representationally unrelated. These examples indicate that the correlation between the representation of the current element and that of the elements of its neighborhood could well be a first measure of neighborhood exploitation. Put differently: there is more representational exploitation when the neighborhood plays a more restrictive role. In this way, the single-bit-flip hill-climber (section 4.1) would perform more representational exploitation than the 1/l-mutation hill-climber (section 4.3). An alternative direction one might take is related to the intuitive use of the current element to generate the elements of the neighborhood. In this respect, both types of hillclimbers would perform an equal amount of representational exploitation, since the elements of T are generated in exactly the same way. Finally, note that to obtain a formal quantification, one has to realize that the only metric (in the topological sense) defined on the search space S is the one induced by the neighborhood structure. In the case of the 1/l-mutation hillclimber, this metric is clearly the Hamming distance. The neighborhood structure of the single-bit-flip hill-climber can less immediately be extended to a metric, and it is not clear if the result should also be the Hamming distance. 5.2 Neighborhood exploration The amount of neighborhood exploration can be summarized by counting the number of distinct elements which are evaluated. This is well-defined, easily computable and intuitively correct. It also corresponds to the definition in the literature that the explorative power of an operator is given by the number of new elements it can generate [2]. Note that neighborhood exploration is entirely independent of success. The amount of exploration is not influenced by the objective values associated with the explored elements. Of course, the quality of exploration (e.g., does it lead to an interesting area?) is much harder to quantify (if possible at all), and can only be done on a problem-specific basis. In the current algorithm framework, the amount of neighborhood exploration is strongly influenced by the neighborhood structure, which is static. In population-based algorithms, on the other hand, more interesting phenomena can occur, since the amount of neighborhood exploration can be directly related to the diversity of the population. 5.3 Objective exploitation Next, we try to quantify objective exploitation. In this phase we only deal with objective information (the objective values of the current element and those of the elements of a subset of its neighborhood). Let us first look at the case of the Metropolis algorithm. In the Metropolis algorithm, the element u selected from the neighborhood is accepted in step 5 of the algorithm with a probability of ( ) exp β(f(s i ) f (u)), (2) where β is the fixed inverse temperature, suitably scaled. In the case of β = 0, this probability is always 1; this is typical for inactive objective exploitation. The other extreme, β =, also corresponds to an extreme form of objective exploitation: the element with the best objective value is always selected. Conclusion: in the context of the Metropolis algorithm, the selection pressure β is a candidate quantifier of objective exploitation. This quantification holds trivially for the single-bit-flip hillclimber. Either objective exploitation is temporarily disabled to allow the hill-climber to escape from a local optimum, or it is at full strength to take the best element of the neighborhood. But selection pressure on its own might not be enough. It is easy to see that objective exploitation is restricted by the amount of information which is known at the moment of selection. The situation β = 0, inactive exploitation, is observed for any algorithm when all the objective values in the neighborhood are equal, i.e., when the algorithm is traversing a flat landscape. The quantification of objective exploitation we propose is selection pressure, influenced by the amount of objective information present. A more formal definition, though, will be deferred to a later contribution. 5.4 Generational exploration We cannot quantify generational exploration: either it happens, or it does not. One could keep a statistic of generational exploration over the run of an algorithm, and look at the percentage of acceptance of new elements. In the case of the Metropolis algorithm, this average (called the acceptance rate) is a useful metric. A interesting question now is: how independent is the acceptance rate from the selection pressure which is the proposed quantification of objective exploitation in the context of the Metropolis algorithm?

6 6 A COMPARISON In this section we compare our approach to some of the definitions found in the literature. Eiben and Schippers [3] list a number of questions and hypotheses concerning exploitation and exploration: 1. Using existing information equals exploitation. This would imply that even the use of bad material, or bad use of good information is seen as exploitation. [... ] 2. Selection is the source of exploitation. This is more or less a corollary of 1. The differences with our approach are clear. Selection is one of the sources of exploitation, not the only one. Representational exploitation is also a use of information. Moreover, we do not relate exploitation to success. 3. The operators mutation and recombination are purely explorative. [... ] In our approach, the mutation operator (which can be seen as steps 3 and 4) is also exploitative. 5. What information (on alleles, schemata, individuals) is available and which part of this information is actually exploited? [... ] This is highly problem dependent. [... ] We have chosen for problem independence and assume the fundamental assumption of black-box optimization to hold: the representation is chosen in such a way that it can guide the algorithm in its search for the optimum. Now consider Beyer [1]: Such a decomposition is of special interest if we want to quantify the relation between local exploitation and exploration. If we interpret exploitation as the ability to step into the local gradient direction and exploration as the ability to leave the gradient path then we might have a way to understand the evolutionary search behavior. Note that his definition is operator oriented (mutation on a real space), and also goal oriented, since the gradient is used to find local optima. This gradient is the only source of information used. A full comparison with our approach is difficult since a memory is needed to maintain an idea of the gradient over more than one generation. We found Beyer the first to propose a quantification of exploitation and exploration. It is also interesting to note that in his setup, new elements can be visited without exploration. This occurs when the new elements lie in the direction of the gradient. De Jong and Spears [2] take the schemata/hyper-plane approach of defining exploration, and consider mainly recombination instead of only mutation. They do not explicitly define exploitation. Eshelman, Caruana and Schaffer [4] also follow the recombination and schema/hyper-plane approach, and are most explicit in the exploitation of sub-individuals (subelements). Two quotes to situate their definitions: If a schema is present in a parent (and not its mate) and survives in the offspring, it is said to have been exploited. Otherwise the trial that the offspring represents is accounted an exploratory one (with respect to that schema). and An exploitative trial is assumed to be testing the given schema in a new setting, i.e., it is assumed that some loci outside of those that define the schema have been altered between parent and child. They do not mention objective exploitation. 7 CONCLUSIONS AND FURTHER WORK We have set a context in which we have defined exploitation and exploration rigorously. The context is simple: element-based black-box algorithms without memory and population. It contains many mutation-based hill-climbers, such as the single-bit-flip hill-climber, the Metropolis algorithm, random walkers and random search algorithms. More complex algorithms which are also more interesting in terms of exploitation and exploration will be studied in further contributions. Our algorithm framework clearly separates the use of representational information and objective information. This allows for a split definition of exploitation: exploitation of the representation and exploitation of the objective values. Exploration is also split in two: we distinguish between exploration of the neighborhood and exploration due to the evolutionary aspect of the search. Consequences of this model and its restrictions are: Exploitation is entirely identified with the use of information. The amounts of exploitation and exploration can hardly change during the run of the algorithm due to the static nature of the neighborhood structure. The restriction of static neighborhoods will be removed as soon as possible. Our setup does not allow for a definition of exploitation and exploration in terms of success, i.e., it is not goal-oriented. Neither does this framework allow for a sub-element definition of exploitation and exploration. Objective exploitation is meaningless without representational exploitation, or, put differently: fitness values can only be used in a context.

7 An important point which we have barely touched is that of quantifications of exploitation and exploration. We have given hints and some intuitive definitions, which are far from definitive. ACKNOWLEDGMENTS The first author is a research assistant of the Fund for Scientific Research Flanders (Belgium) (F.W.O.).

8 REFERENCES [1] H.-G. Beyer. On the explorative power of ES/EP-like algorithms. In V. W. Porto, N. Saravanan, D. Waagen, and A. E. Eiben, editors, Proceedings of the 7th Annual Conference on Evolutionary Programming. Springer, LNCS 1447, [2] K. A. DeJong and W. M. Spears. A formal analysis of the role of multi-point crossover in genetic algorithms. Annals of Mathematics and Artificial Intelligence, 5:1 26, [3] A.E. Eiben and C.A. Schippers. On evolutionary exploration and exploitation. Fundamentae Informaticae, 35:35 50, [4] L. Eshelman, R. A. Caruana, and J. D. Schaffer. Biases in the crossover landscape. In J. D. Schaffer, editor, Proceedings of the 3rd International Conference on Genetic Algorithms, pages Morgan Kaufmann Publishers, [5] L. J. Eshelman and J. D. Schaffer. Crossover s niche. In S. Forrest, editor, Proceedings of the 5th International Conference on Genetic Algorithms, pages Morgan Kaufmann Publishers, [6] N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller. Equations of state calculations by fast computing machines. Journal of Chemical Physics, 21: , 1953.