Introduction to Scientific Modeling CS 365, Fall Semester, 2011 Genetic Algorithms

Transcription

1 Introduction to Scientific Modeling CS 365, Fall Semester, 2011 Genetic Algorithms Stephanie Forrest

2 Genetic Algorithms" Principles of natural selection applied to computation:" Variation " Selection " Inheritance" Evolution in a computer:" Individuals (genotypes) stored in computerʼs memory " Evaluation of individuals (artificial selection)" Differential reproduction through copying and deletion" Variation introduced by analogy with mutation and crossover" Simple algorithm captures much of the richness seen in naturally evolving populations."

3 A Simple Genetic Algorithm" Population at T n " Population at T n+1 " 00111" 11100" Mutation" 01100" " " 11010" 01010"..." Selection" 01010"..." Crossover" 01100"..." F(00111) = 0.1 F(11100) = 0.9 F(01010) = 0.5 "

4 Example Performance Curve" mean fitness max fitness Fitness Generation

5 Where did these ideas originate?" Genetic algorithms (Holland, 1962)" Original idea was to create an algorithm that captured the richness of natural adaptive systems." Emphasized the adaptive properties of entire populations and the importance of recombination mechanisms such as crossover." Application to function optimization introduced by DeJong (1975)." Evolutionstrategie (Rechenberg, 1965)" Emphasized the importance of selection and mutation, as" Mechanisms for solving difficult real-valued optimization problems." Evolutionary programming (Fogel et al., 1966)" Emphasis on evolving finite state machines." Genetic programming (Koza, 1992) ==> Evolutionary Computation "

6 References" J. H. Holland Adaptation in Natural and Artificial Systems. Univ. of Michigan Press (1975). Second Edition published by MIT Press (1992)." D. E. Goldberg Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley (1989)." M. Mitchell An Introduction to Genetic Algorithms. MIT Press (1996)." S. Forrest Genetic Algorithms: Principles of natural selection applied to computation. Science 261: (1993)." J. Koza Genetic Programming. MIT Press (1992)."

7 Multi-parameter Function Optimization" F(x, y) = yx 2 - x Base Base 10 F(001101) = F(1,5) = = 4 Decimal Binary

8 Multi-parameter Function Optimization" F(x, y) = yx 2 - x 4 Degray Bit string (Gray coded) Base Base 10 F(001111) = F(1,5) = = 4 Decimal Binary Gray code

9 Example Applications of Genetic Algorithms Engineering applications:" Multi-parameter function optimization (e.g., spectroscopic applications, turbine engine design)." Sequencing problems (e.g., circuit design, factory scheduling, TSP)." Machine and robot learning." Complex data analysis." Automatic programming (e.g., genetic programming)." Modeling:" Rule discovery in cognitive systems." Learning strategies for games." Affinity maturation in immune systems." Ecosystem modeling."

10 Implementation Issues" Implementation issues" Genetic programming:" Example: Trigonometric functions " Software repair" Permutation problems and special operators:" Example: TSP" Modeling applications:" Example: Prisonerʼs Dilemma" Example: Classifier Systems " Example: Echo"

11 Implementation Issues" Data structures:" Packed arrays of bits" Byte arrays" Vectors of real numbers" Lists and trees" Representation:" Feature lists" Binary encodings, gray codes" Real numbers" Permutations" Trees " Selection" On next slide" Scaling" On next slide" Crossover:" 1-point, 2-point, n-point" Uniform" Special operators" Mutation:" Bit flips" Creep (Gaussian noise)" Elitism" Parameters (rough guidelines):" Bitstring length (32-10,000)" Population size ( )" Length of run (50-10,000)" Crossover rate (0.6 per pair)" Mutation rate (0.005 per bit)"

12 Selection Methods" Fitness-proportionate (used in theoretical studies): Expected value of individual i :" Implement as roulette wheel: f exp( i ) = f f i Rank-based: (SKIP)" Intended to prevent premature convergence (slow down evolution)." Each individual ranked according to fitness." Expected value depends on rank." Min, Max are constants." f rank ( i) 1 ( i) = Min + ( Max Min) exp N 1

13 Selection Methods cont." Tournament:" Computationally efficient." Choose T = size of tournament (2 is a common value)." Pick subsets of size T from the population randomly (with replacement)." Compare fitnesses within the subset and choose the winner (either deterministically or stochastically)." Iterate." Steady state." Implicit (e.g., Echo, immune models, etc.)" Reproduction rate proportional to resources obtained."

14 How do Genetic Algorithms Work? The Central Dogma of Genetic Algorithms (Holland, 1975)" Schema processing" Schema Theorem" Implicit parallelism" Building block hypothesis " K-armed bandit analogy"

15 Genetic Algorithms and Search" High-dimensional search spaces:" All binary strings of length l.! All possible strategies for playing the game of chess." All possible tours in the Travelling Salesman Problem." Genetic algorithms use biased sampling to search high-dimensional spaces:" Independent sampling." Selection biases search towards high-fitness regions." Crossover combines partial solutions from different strings." Partial solution formalized as schema. " *1**" 1***" *0**" 0***"

16 Schemas" Schemas capture important regularities in the search space: * * * * 1 1 * * 0 * * * * * 0 * * 1 * * 0 * 1 1 Implicit Parallelism: 1 individual samples many schemas simultaneously. Schema Theorem: Reproduction and crossover guarantee exponentially increasing samples of the observed best schemas. Order of a schema O(s) = number of defined bits. Defining length of a schema D(s) = distance between outermost bits.

17 Schema Theorem (Holland, 1975)" Let: s be a schema in population at time t," N(s,t) be the number of instances of s at time t." Question: What is the expected N(s,t+1)?" Assume: Fitness-proportionate selection." Expected number of offspring(x) =" Ignoring crossover and mutation," µ ˆ( s, t) N ( s, t + 1) = N( s, t) F( t) Note: If ˆµ( s, t) F( t) = c, then" t N( s, t) = c N( s,0) F( x) F( t) Crossover and mutation handled as loss terms:" N(s,t +1) ˆ µ (s,t) F(t) N(s,t)(1 p c D(s) l 1 )[(1 p m )O(s) ]

18 Royal Road Schemas" schema 1 schema 2 schema 3 Fitness Generation

19 Study Question" Given a population, consisting of" N individuals," Each individual is L bits long," How many schemas are sampled by the population (in one generation)?" Hint: " Minimum value is: 2 L " Maximum value is Nx2 L "

20 Questions" When will genetic algorithms work well and when they will not?" Not appropriate for problems where it is important to find the exact global optimum." GA domains are typically those about which we have little analytical knowledge (complex, noisy, dynamic, poorly specified, etc.)." Would like a mathematical characterization that is predictive." What makes a problem hard for genetic algorithms?" Deception, multi-modality, conflicting schema information, noise,..." What makes a problem easy for genetic algorithms?" What distinguishes genetic algorithms from other optimization methods, such as hill climbing?" What does it mean for a genetic algorithm to perform well?"

21 Building Blocks Hypothesis (Holland, 1975)" 1. GA initially detects biases in low-order schemas:" GA obtains good estimates of schema average fitness by sampling strings." 2. Over time, information from low-order schemas is combined through crossover, and" 3. GA detects biases in high-order schemas," 4. Eventually converging on the most fit region of the space. Implies that crossover is central to the success of the GA." Claim: GA allocates samples to schemas in a near optimal way:" K-armed bandit argument. "

22 Genetic Programming" Evolve populations of computer programs:" Typically use the language Lisp." Select a set of primitive functions for each problem." Represent program as a syntax tree." Function approximation vs. function optimization." Crossover operator:" Exchange subtrees between individual program trees." Schema Theorem?" Many applications:" Optimal control (e.g., the pole balancing problem)" Circuit design" Symbolic regression (data fitting)"

23 Genetic Programming" Expression X 2 + 3xy + y 2 LISP (+ (* x x) (*3 x y) (* y y)) + * * * x x 3 x y y y

24 Genetic Programming cont." Consider evolving a program to compute: cos 2x" A human-designed program: 1-2sin 2 x In Lisp: (- 1 (* 2 (* (sin x)(sin x))))" A genetic programming solution: (sin ( - ( - 2 (* x 2)) (sin (sin (sin (sin (sin (sin ( * (sin (sin 1)) (sin (sin 1)))))))))))" Junk DNA?"

25 Example Genetic Programming Application Automatically Repairing Software Bugs" University of New Mexico Stephanie Forrest ThanhVu Nguyen Eric Schulte Neal Holtschulte University of Virginia Westley Weimer Claire Le Goue Zak Fry

26 Assume: " Access to C source code" Summary of Approach" Negative test case (input = ; output = infinite loop)" Positive test cases (encode required program functionality)" Construct Abstract Syntax Tree (AST) using CIL" Evolve repair that avoids negative test case and passes positive test case" Minimize repair using program analysis methods (addresses code bloat)" Structural differencing and delta debugging"

27

28 Mutation operators" Swap" Delete" Insert"

29 Example: Microsoft Zune Players" Downloaded from (Jan. 2009). " Dec. 31, Microsoft Zune players mysteriously freeze up." Bug: Infinite loop when input is last day of a leap year." Negative test case: 10593, which corresponds to Dec 31, " Repair is not trivial. Microsoftʼs recommendation was to let Zune drain its battery and then reset."

30 Abstract Syntax Tree Representation"

31 Weighted path" Nodes visited by negative test case have weight 1.0" Nodes visited by negative and positive test cases have weight 0.01" All other nodes have weight 0.0"

32 The final evolved repair"

33 Evolutionary computation Innovations" Start with a working program" Focus on execution path through AST" Restrict mutation and crossover to execution path" Represent AST to level of statements" Leaves out expressions, variable declarations" Genetic operators" Donʼt invent any new code, crossback, macromutation operators" Minimize repair size using structural differencing"

34 Evolutionary computation details" Fitness: Weighted sum of test cases that the program passes:! F(Programs that donʼt compile) = 0" 5 positive test cases (weight = 1), 1 or 2 negative test cases (weight = 10)! Mutation operations: " Delete a statement, Insert a statement, Swap a stmt along the weighted path with a stmt from another part of the program," Crossover: Exchange subtrees" Population size is 40. Standard run is 10 gens + 10 gens"

35 Summary of repairs to date" Twenty distinct programs totaling 186,603 LOC (180kLOC)." Scientific Computing: 1" Scripting Languages: 3" Games, Graphics, Sound: 4" Servers (web, ftp, authentication): 4" Operating system utilities: 8" Buffer overflow attacks: 3" Segfaults: 7" Infinite loops: 4" Incorrect output: 2" Integer overflow: 2" Nonoverflow DOS: 1" Format string vulnerability: 1"

36 Benchmark programs "GECCO 2009, ICSE 2009, TSE (in press)" Program Lines of Code (LOC) Functionality Fault Time to Repair zune 28 media player infinite loop 42s gcd 22 handcrafted example infinite loop 153s uniq 1146 duplicate text processing segfault 34s look-u 1169 dictionary lookup segfault 45s look-s 1363 dictionary lookup infinite loop 55s units 1504 metric conversion segfault 109s deroff 2236 document processing segfault 131s nullhttp 5575 webserver heap overflow 578s indent 9906 source code processing infinite loop 546s flex lexical analyzer generator segfault 230s atris graphical tetris game stack overflow 80s openldap io.c 6519 directory protocol nonoverflow DOS 665s lighttp fastcgi.c webserver heap overflow 49s php string.c scripting language integer overflow 6s wu-ftpd FTP server format string 2256s

37 Time to discover repair" Average time to repair: approx. 3 minutes " Time includes: " GP algorithm (selection, mutation, calculating fitness, etc.)" Running test cases" Pretty printing and memoizing ASTs" gcc (compiling ASTs into executable code)"

38 Search time scales with weighted path length" m = 1.26

39 Why it works" Powerful intermediate representation" Weighted path greatly reduces search space" Minimization eliminates unnecessary fixes" Most bugs can be fixed with a few local modifications" 667 average atomic genetic operations to discover a repair; Repair discovered on average in 3.6 generations; 2.9 genetic operations per fitness evaluation" About 1/2 the time, Random Search does as well as GP " Will this scale to more complex problems? "

40 References" W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest Automatically finding patches using genetic programming. 31st International Conference on Software Engineering (ICSE) (2009). " S. Forrest, W. Weimer, T. Nguyen, and C. Le Goes A Genetic Programming Approach to Automated Software Repair. Genetic and Evolutionary Computation Conference (in press)." C. Le Goues, T. Nguyen, W. Weimer, and S. Forrest Closed -Loop Repair of Security Vulnerabilities. ETAPS(Submitted Oct. 2009). "

41 Permutation Problems and Special Operators" Problem: Find an optimal ordering for a sequence of N items. " Examples:" Traveling Salesman Problem (TSP)" Bin packing problems" Scheduling" DNA fragment assembly" Traveling Salesman Problem:"

42 Using Genetic Algorithms to Find Good Tours for TSP" Natural representation: Permutations in which each city is labeled by a unique integer (bitstring): " Problem: Mutation and crossover do not produce legal tours: " Solutions:" Other representations." Other operators." Penalize illegal solutions through fitness function."

43 Specialized Operators" What information (schemas) should be preserved?" Absolute position in the sequence" Relative ordering in the sequence (precedence)" Adjacency relations" How much randomness is introduced?" Order crossover (Davis, 1985)" Partially-mapped crossover (PMX) (Goldberg, et al. 1985)" Cycle crossover (Oliver, et al. 1987)" Edge-recombination:" Try to preserve adjacencies in parents" Favor adjacencies common to both parents" When 1,2 fail, make a random selection."

44 Edge Recombination Operator" Algorithm:" 1. Select one parent at random and assign the first element in its permutation to be the first one in the child." 2. Select the second element for the child, as follows:" If there is an adjacency common to both parents, then choose that." If there is an unused adjacency available from one parent, choose it." If (1) and (2) fail, then pick an adjacency randomly." 3. Select the remaining elements in order by repeating step 2."

45 Edge Recombination Example" Original Individuals New Individual Key Adjacent Keys 1 2,2,3,4, 2 1,1,5,6, 3 1,6,6, 4 1,5,6 5 2,4 6 2,3,3,4

46 Using Genetic Algorithms to Model Natural and Artificial Systems" Modeling social systems" Evolution as a model of imitation (Prisonerʼs Dilemma)" Economies (stock market)" Cognitive systems" Induction and learning (Classifier Systems)" Genetic evolution" Natural ecological systems (Echo)" Immune systems" Somatic evolution in CancerSim" Ecosystem modeling" Artificial life models" In each of these systems, adaptation is central" What can we learn from modeling with genetic algorithms?"

47 Using Complex Systems Approaches to Model Social Systems" Societies and political systems" Sugarscape models (related to Cellular Automata)" George Gummermanʼs agent-based simulation of Anasazi settlements." Economies" Stock market models" Econophysics approaches to understanding financial markets." Game theory / non-zero sum games " Ant colony models (social insects). "

48 Evolution of Cooperation (Axelrod, 1984)" What is the Prisonerʼs Dilemma? Player B Player A Cooperate Defect Cooperate 3,3 0,5 Defect 5,0 1,1 Non-zero sum game:" Total number of points is not constant for each configuration." What is the best move for Player 1? " Player 2?"

49 Prisonerʼs Dilemma cont." Iterated Prisonerʼs Dilemma:" Play game for an indeterminate number of steps." Raises possibility of strategies based on the behavior of the other player." What are the possible strategies?"

50 Prisonerʼs Dilemma cont." Iterated Prisonerʼs Dilemma:" Play game for an indeterminate number of steps." Raises possibility of strategies based on the behavior of the other player." What are the possible strategies?" RANDOM" Always cooperate, always defect" Tit-For-Tat" Model other player and try to maximize against that model." etc."

51 Evolution of Cooperation" ~1980 Two tournaments:" Each strategy (entry) encoded as a computer program." Round robin (everyone plays everyone)." Tit-For-Tat (TFT) won both times." Learning algorithms:" Can TFT evolve in a competitive environment? How could a TFT strategy evolve in nature?" Is there a better strategy?" How do strategies develop?" Use genetic algorithm to study these questions," In environment of tournament." In a changing environment (co-evolution)." In a 3-person game." In an n-person game." In a Norms game."

52 Prisonerʼs Dilemma Using Genetic Algorithm" Population consists of individual strategies:" 1 strategy per chromosome." Encoding?" Fitness function?"

53 Prisonerʼs Dilemma Using Genetic Algorithm" Population consists of individual strategies:" 1 strategy per chromosome." Each chromosome is a 64 (70) bit string." Each bit specifies the strategyʼs next move (cooperate or defect) for one particular history of 3 moves." Encoding:" Need to remember each playerʼs move for 3 time steps ==> 6 pieces of information." At each point, either player could cooperate or defect (binary decision)." 2 6 = 64 possible histories." Value of bit at a given position tells strategy what to do (0 = coop, 1 = defect) in the context of that history." Additional 6 bits encodes assumption about previous interactions before the start of the game."

54 Prisonerʼs Dilemma Encoding cont." Let A be the sum formed by," Counting the otherʼs defection as 2" Counting my own defection as 1," And giving weights of," 16 to the most recent moves," 4 to the move two time steps in the past, and" 1 to the move three time steps in the past." E.g.," History of mutual cooperation for 3 time steps ==> 0." History of mutual defection for 3 times steps ==> 63."

55 Prisonerʼs Dilemma Encoding cont." Let A be the sum formed by," Counting the otherʼs defection as 2" Counting my own defection as 1," And giving weights of," 16 to the most recent moves," 4 to the move two time steps in the past, and" 1 to the move three time steps in the past." E.g.," History of mutual cooperation for 3 time steps ==> 0." History of mutual defection for 3 times steps ==> 63." What history should we assume for first 3 moves?" Mutual cooperation." Genetically determined (need 6 more bits)."

56 Prisonerʼs Dilemma Using GA Fitness Function" Environment of tournament: " Select 8 strategies from ~160 tournament entries that predict performance, and play each candidate strategy against those 8 representatives." Co-evolutionary environment:" Strategies play against other strategies in the population."

57 Results for Fixed Env." From a random start, the GA evolved populations whose median member was just as successful as TFT." Most of the evolved strategies resemble TFT." In some rules, strategies were discovered that did significantly better than TFT (in tournament env.):" Discriminates between different opponents (based only on its behavior)." Adjusts its own behavior to exploit an exploitable opponent." Does this without getting into (too) much trouble with other opponents." Achieved this by defecting on first move, and then apologizing when necessary."

58 Results in Evolving Environment (From Axelrod, 1997)"

59 Social Norms" A norm exists in a given social setting to the extent that individuals usually act in a certain way and are often punished when seen not to be acting in this way" Powerful mechanism for regulating conflict in groups" Even with more than two agents and no central authority" How do norms evolve in the first place?" What makes them stable? Why do they change so quickly?" Examples:" Dueling: Alexander Hamilton v. Aaron Burr" Smoking in public" Gender-laden language" Prohibition against chemical warfare" Democratic movements in the Arab world?"

60 Norms Game"

61 Studying the Evolution of Norms with Genetic Algorithms" Each individual represents a strategy for playing the Norms game: <B,V>" 3 bits per dimension (6 bits total)" 8 different values for B; 8 different values for V" Pop size = 20" Fitness:" Each individual receives 4 opportunities to defect" Chance of being seen drawn from uniform distribution and compared to B" If individual observes a defection, it Enforces the norm with probability determined by V" Run experiments for 100 generations"

62 Norms Game Dynamics"

63 Comments" Relying on individuals to punish defectors may not be enough to enforce the norm" Possible mechanisms for enforcing norms" Metanorms (e.g., honor codes)" Dominance of one group over another " e.g., whites lynching blacks in South but not vice versa" E.g., Libya trying to extend the international norm of a 12-mile limit of territorial waters." Internalization (psychological pain for violating norm)" Deterrence"

64 Meta Norms"

65 Metanorms Dynamics"