Instructor: Dr. Benjamin Thompson Lecture 15: 3 March 2009
|
|
- Vernon Kelly
- 5 years ago
- Views:
Transcription
1 Instructor: Dr. Benjamin Thompson Lecture 15: 3 March 2009
2 Announcements Homework 5 due right now No homework assignment over Spring Break, because your instructor is such an awesome guy. In other words, try to spend sometime over the break thinking about your project! I will be available all next week for questions/office hours/coffee and a nice scone, perhaps. Maybe pumpkin, maybe blueberry. Black drip, please; none of that froo-froo latte stuff.
3 Characterized Analogously As a Constant Tone With Apparent Decreasing Frequency Decision Boundaries in Classification Problems Matlab Demonstration: Two Moons revisited Matlab Demonstration: Five-class problem Super-Awesome Really Fun and Amazing Term Project Assignment! Yay!
4 Characterized Analogously As a Constant Tone With Apparent Increasing Frequency Neural Network Training as an Optimization Problem Drawbacks of Gradient Descent Other Optimization Approaches Random Search Particle Swarm Optimization Now with minty-fresh MATLAB Demo! Genetic Algorithms Simulated Annealing
5 Because I know you re all craving more abstraction!
6 To Put It Another Way What are some other parameter estimation problems we ve seen? The Multi-Layered Perceptron Neural Network can be viewed as a parameter estimation problem: That is, the MLP may be viewed as a function o=f nn (x,w), where x is some vector of inputs, o is the output of the function, and W is the set of all the weights that parameterize the input/output relationship Then, given some (possibly unknown) function d=f(x), we seek to estimate the set of parameters W that makes F nn (x,w) come as closely as possible to equaling F(x) In technical terms, we seek to find the W that minimizes the approximation error ( ) (, ) 2 F x Fnn x W dx x
7 Still Puttin It Of course, since we don t necessarily know the underlying function d=f(x), we must rely on the training sample to tell us how we re doing So, for a set of inputs X={x p } and corresponding desired outputs D={d p }, we seek the set of parameters Wthat minimizes* the training error: P 1 F F x 2 p= 1 ( ) (, ) 2 p nn xp W which we ve seen in its more familiar form: 1 E = 2 P M ( d ) 2 pi, opi, p= 1 i= 1 *This isn t technically the same thing as on the previous slide. Remember our discussion on Generalization?
8 Long Story Slightly Longer In other words, we want to optimize the approximation error by finding the best Wfor the task! Backpropagation was just one proposed method for doing this, and was based on the gradient descent technique for iterative optimization
9 What s Wrong With Gradient Descent? Gradient descent is only guaranteed to work when the error surface is more-or-less bowl-like That is, all downward paths lead to the global minimum Gradient descent is highly prone to getting stuck in local minima In fact, if there s a local minimum between its initialization point and the global minimum, it s pretty much guaranteed to get stuck That being said, the performance of Gradient Descent is highly dependent on its initial position in weightspace
10 You know, like sneaking up from behind
11 Motivation For some error surfaces, Gradient Descent may not be up to the task of finding the best set of weights for a neural network Some other approaches may be warrantedea page from We once again take a page from nature in developing several alternatives
12 Goal The goal of all these approaches is, again, optimization: Specifically for the case of neural networks, we seek to optimize (minimize) the error or cost function defined as the error equation on the previous slide, as a function of the parameter set W In general, we seek to optimize some fitness function, which is a measure of how good a particular guess at the parameter set is Optimization may be thought of as a searchthrough weight-spaceto find the point that provides the optimal cost/error/fitness
13 As bad as it gets
14 Naïve Approach We ve already mentioned brute force search: That is, test every possible set of weights and pick the best one Practically impossible for almost all real problems Another approach: random search For each epoch k, do: Randomly select a guess at the weights, W[k] Calculate the fitness/cost/error of this guess, E k If E k is better than E best, save W[k] as W best set E best = E k Continue until E best reaches some acceptable level
15 More Naiveté This approach is memoryless That is, the guess on one iteration is completely unrelated to the guess on any other iteration Given enough time, this method will find the best answer Of course, enough might be infinite, or at least long enough for computers to evolve sentience and take over the entire world, thus rendering whatever optimization problem you were trying to solve completely moot Nobody ever, ever, ever uses this approach.
16 Better Options Particle Swarm Optimization Genetic Algorithms Simulated Annealing
17 Ever see Alfred Hitchcock s The Birds? It s like that, with less screaming.
18 Motivation Scientists and observers long noticed that a flock of birds behaved in an apparently choreographedmanner as it converged on a food source R. Eberhartand J. Kennedy (1995) proposed a simple set of rules that individual birds may follow to converge on a food source in such a manner Birds : a particular solution of some optimization problem i.e., a guess at W Food source : the location of the global optimum in the search space alternately, the food source may be interpreted the valueof the fitness/error/cost function when it is at its best
19 Particles? In order to abstract things, we call each bird a particle We have many, simultaneous guesses or particles, so it is a swarmof particles --hence, Particle Swarm Optimization(hereafter, PSO) Each particle moves about the search space according to some velocity That is, some directionand speed
20 Particles Illustrated w 2 Remember, each of these points represents one possible solution of W w 1
21 What defines the velocity? Each bird has a memory of two things: The best solution it has found so far personal best or p best The best solution any bird has found global best, or g best After each iteration, a bird adjusts its velocity using three components: Its current velocity (the momentum component) Its distance from its personal best (the cognitive component) Its distance from the global best (the social component)
22 Picture This! Personal best, so far Personal best, so far Current Velocity Current Velocity Global best, so far
23 So How Does This Work? Only two equations we need to know: position (weight) update: p = p + η v new old old velocity update: v = ω v + c r p p + c r g p ( ) ( ) new old 1 1 best new 2 2 best new Momentum term Cognitive term Social term ω, c1, and c 2 are tunable weights to adjust the emphasis of each of the three terms ωis typically chosen to be slightly less than one c1 and c 2 are typically chosen to be approx. 2 r1 and r2 are random numbers, typically taken from a U[0,1] distribution The addition of these terms enable a random component to the search
24 The PSO Algorithm Defined for minimization problems; how would you define this for maximization problems? Initialize N birds positions and velocities randomly Randomly initialize p best and g best Set E gbest to Inf Set E pbest to Inf FOR each iteration, do: FOR each bird, do: Update the bird s position using p = p + η v Evaluate the error of p new, E pnew If E pnew < E pbest Set E pbest = E pnew Set p best = p new If E pnew < E gbest Set E gbest = E pnew Set g best = p new Update velocities using End FOR End FOR new old old For a neural net, this means using pnew as the input, calculating the nnoutput, and comparing that to the desiredoutput. This also implies that you must have an oracleto determine the desired response. ( ) c r ( ) v = ω v + c r p p + g p new old 1 1 best new 2 2 best new
25 Some PSO Points Notice that nothing in the inner loop really depends on previous/following iterations of that loop That is, notice that, apart from the global best information, one bird doesn t care about what any other bird is doing at the time Thus, if the calculation of the error is computationally intensive, you could parallelize PSO by running each bird (or a subset of birds) on a separate computer, and have them communicate global-best information after each epoch
26 More PSO Points Extremely efficient, computationally speaking I ve coded it up in 6 lines of code in MATLAB, for example Only requires simple (scalar) multiplies and (vector) addition Computational footprint scales linearlywith the number of birds Easily extensible I ll mention some heuristic improvements in later slides Best of all: it works! (Demo after Spring Break!)
27 PSO Extensions Fully Informed PSO: Additional component in velocity update: That is, each bird also moves toward the weighted average of each other bird s personal best Repulsive PSO: Normal PSO tends to converge into a clump To counteract this, add an another component to the velocity N 1 update α r i= 1 In other words, move away from the centroid of all the other birds Random Jitter (, ) i i bird i bird Add an additive random vector to the velocity update And many, many, many more p p N 1 i= 1 α r ( p, p ) i i best i bird
28 PSO Demo My cost function is simply the familiar Rosenbrock function:
29 Instead of a corny quip, I present to you, reprinted with permission from xkcd.org:
30 Motivation Attempts to model the actual process of evolution on a small scale to evolvebetter solutions to some optimization problem given a previous generation of solutions In reality: Two parents in a given generation contribute chromosomes to an overall genetic code of a child via genetic crossover random mutation of the genetic code occurs with some small probability Each generation reproduces with some probability based on the fitness of the parents in that generation
31 The Genome In humans, of course, our genetic code determines our physical makeup Similarly, the genome in a Genetic Algorithm (GA) fully determines a single potential solution to the optimization problem For neural networks, the genome is just all the weights! Traditionally, GAs use bit-string genomes: That is, the genetic pattern is simply a string of ones and zeros So the continuously-valued weight matrices must be converted into binary! Non-binary methods also work
32 The Genome Each of these cells is a single gene Each block, then, may be thought of as a chromosome
33 A Short Primer on How To Convert To Binary 1. Pick a dynamic range (w min, w max ) for each parameter and number of bits N b to represent each parameter e.g., +/-10.0, and 10 bits Dynamic range impacts the min/max searchable values Number of bits impacts the refinement of your solution These can be different for each parameter, or the same. 2. Normalize your parameter wk by knorm, w w w w k min = 3. In MATLAB, use dec2bin(wk) to convert this to a binary (char) string w max min N 2 b
34 A Short Primer on How to Convert From Binary Given a binary string bcorresponding to a single number: 1) First, convert the binary string to an integer using, in MATLAB, w k,norm =bin2dec(b) 2) Un-normalize this integer by performing: w = w 2 Nb w w + w k ( ) knorm, max min min
35 The Algorithm, Conceptually Initialize a population of possible solutions Evaluate the fitness of each solution Each member of the population has a probability to reproduce based on its fitness More fit solutions are more likely to reproduce and pass on their fit genes Simple approach: normalize the fitness so that the most fit member will breed with probability 1, and the least fit with probability 0 Select pair-wise mates for each breeding member That is, the top two breeders mate, the third and fourth-best mate, and so on
36 More Conceptual Algorithm For a given mating pair: Ma Baby crossoveroccurs: each chromosome is randomly (coin flip) selected from one or the other parent to form a new offspring Pa
37 Still More Conceptual Algorithm Each gene of this new child has a very small probability to flip its bit this is mutation The child patterns form the guesses for the parameter vector on the following iteration That s it!
38 Genetic Algorithm Details Initialize a set of Nrandom guesses at the solution For each iteration, do: Evaluate the fitness of each member of the population For each member, do: If that member s fitness is in the top 50% of fitness, mark that member for reproduction end FOR From the set of marked for reproduction members, create Npairs of mates (cont. on next slide) I m not going into detail here because there are many, many ways to accomplish this
39 GA Details (cont.) For each mating pair, do: For each chromosome in a genome, do: generate a uniform[0,1] random number If that number is greater than or equal to 0.5, copy the chromosome from the father, else, copy the chromosome from the mother End FOR For each gene in the resulting child genome, do: Generate a uniform[0,1] number If this number is less than µ, the mutation rate parameter (a very small number), flip that gene s bit value (o->1 or 1->0) End FOR End FOR Repeat until a solution whose fitness is sufficiently low exists
40 GA Remarks A high mutation rate makes this approach random search A low mutation rate causes this to converge very slowly and be very prone to local minima GAs generally take a long time to converge The mutation of a single bit means that a single guess can jump a lot (for an MSB mutation) or very little (for an LSB mutation) Just like with PSO, many, many heuristics exist for improving the algorithm
Chapter 14 Global Search Algorithms
Chapter 14 Global Search Algorithms An Introduction to Optimization Spring, 2015 Wei-Ta Chu 1 Introduction We discuss various search methods that attempts to search throughout the entire feasible set.
More informationCHAPTER 6 HYBRID AI BASED IMAGE CLASSIFICATION TECHNIQUES
CHAPTER 6 HYBRID AI BASED IMAGE CLASSIFICATION TECHNIQUES 6.1 INTRODUCTION The exploration of applications of ANN for image classification has yielded satisfactory results. But, the scope for improving
More informationARTIFICIAL INTELLIGENCE (CSCU9YE ) LECTURE 5: EVOLUTIONARY ALGORITHMS
ARTIFICIAL INTELLIGENCE (CSCU9YE ) LECTURE 5: EVOLUTIONARY ALGORITHMS Gabriela Ochoa http://www.cs.stir.ac.uk/~goc/ OUTLINE Optimisation problems Optimisation & search Two Examples The knapsack problem
More informationIntroduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.
Chapter 7: Derivative-Free Optimization Introduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.5) Jyh-Shing Roger Jang et al.,
More informationIntroduction to Genetic Algorithms
Advanced Topics in Image Analysis and Machine Learning Introduction to Genetic Algorithms Week 3 Faculty of Information Science and Engineering Ritsumeikan University Today s class outline Genetic Algorithms
More informationIntroduction to Genetic Algorithms. Based on Chapter 10 of Marsland Chapter 9 of Mitchell
Introduction to Genetic Algorithms Based on Chapter 10 of Marsland Chapter 9 of Mitchell Genetic Algorithms - History Pioneered by John Holland in the 1970s Became popular in the late 1980s Based on ideas
More informationKyrre Glette INF3490 Evolvable Hardware Cartesian Genetic Programming
Kyrre Glette kyrrehg@ifi INF3490 Evolvable Hardware Cartesian Genetic Programming Overview Introduction to Evolvable Hardware (EHW) Cartesian Genetic Programming Applications of EHW 3 Evolvable Hardware
More informationCHAPTER 2 CONVENTIONAL AND NON-CONVENTIONAL TECHNIQUES TO SOLVE ORPD PROBLEM
20 CHAPTER 2 CONVENTIONAL AND NON-CONVENTIONAL TECHNIQUES TO SOLVE ORPD PROBLEM 2.1 CLASSIFICATION OF CONVENTIONAL TECHNIQUES Classical optimization methods can be classified into two distinct groups:
More informationGENETIC ALGORITHM VERSUS PARTICLE SWARM OPTIMIZATION IN N-QUEEN PROBLEM
Journal of Al-Nahrain University Vol.10(2), December, 2007, pp.172-177 Science GENETIC ALGORITHM VERSUS PARTICLE SWARM OPTIMIZATION IN N-QUEEN PROBLEM * Azhar W. Hammad, ** Dr. Ban N. Thannoon Al-Nahrain
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 Outline Local search techniques and optimization Hill-climbing
More informationDETERMINING MAXIMUM/MINIMUM VALUES FOR TWO- DIMENTIONAL MATHMATICLE FUNCTIONS USING RANDOM CREOSSOVER TECHNIQUES
DETERMINING MAXIMUM/MINIMUM VALUES FOR TWO- DIMENTIONAL MATHMATICLE FUNCTIONS USING RANDOM CREOSSOVER TECHNIQUES SHIHADEH ALQRAINY. Department of Software Engineering, Albalqa Applied University. E-mail:
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationSuppose you have a problem You don t know how to solve it What can you do? Can you use a computer to somehow find a solution for you?
Gurjit Randhawa Suppose you have a problem You don t know how to solve it What can you do? Can you use a computer to somehow find a solution for you? This would be nice! Can it be done? A blind generate
More informationFall 09, Homework 5
5-38 Fall 09, Homework 5 Due: Wednesday, November 8th, beginning of the class You can work in a group of up to two people. This group does not need to be the same group as for the other homeworks. You
More informationTraffic Signal Control Based On Fuzzy Artificial Neural Networks With Particle Swarm Optimization
Traffic Signal Control Based On Fuzzy Artificial Neural Networks With Particle Swarm Optimization J.Venkatesh 1, B.Chiranjeevulu 2 1 PG Student, Dept. of ECE, Viswanadha Institute of Technology And Management,
More informationModified Particle Swarm Optimization
Modified Particle Swarm Optimization Swati Agrawal 1, R.P. Shimpi 2 1 Aerospace Engineering Department, IIT Bombay, Mumbai, India, swati.agrawal@iitb.ac.in 2 Aerospace Engineering Department, IIT Bombay,
More informationAlgorithm Design (4) Metaheuristics
Algorithm Design (4) Metaheuristics Takashi Chikayama School of Engineering The University of Tokyo Formalization of Constraint Optimization Minimize (or maximize) the objective function f(x 0,, x n )
More informationArgha Roy* Dept. of CSE Netaji Subhash Engg. College West Bengal, India.
Volume 3, Issue 3, March 2013 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Training Artificial
More informationMETAHEURISTIC. Jacques A. Ferland Department of Informatique and Recherche Opérationnelle Université de Montréal.
METAHEURISTIC Jacques A. Ferland Department of Informatique and Recherche Opérationnelle Université de Montréal ferland@iro.umontreal.ca March 2015 Overview Heuristic Constructive Techniques: Generate
More informationGENETIC ALGORITHM with Hands-On exercise
GENETIC ALGORITHM with Hands-On exercise Adopted From Lecture by Michael Negnevitsky, Electrical Engineering & Computer Science University of Tasmania 1 Objective To understand the processes ie. GAs Basic
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 2 Outline Local search techniques and optimization Hill-climbing
More informationInternational Journal of Digital Application & Contemporary research Website: (Volume 1, Issue 7, February 2013)
Performance Analysis of GA and PSO over Economic Load Dispatch Problem Sakshi Rajpoot sakshirajpoot1988@gmail.com Dr. Sandeep Bhongade sandeepbhongade@rediffmail.com Abstract Economic Load dispatch problem
More informationParticle Swarm Optimization
Dario Schor, M.Sc., EIT schor@ieee.org Space Systems Department Magellan Aerospace Winnipeg Winnipeg, Manitoba 1 of 34 Optimization Techniques Motivation Optimization: Where, min x F(x), subject to g(x)
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationParticle Swarm Optimization applied to Pattern Recognition
Particle Swarm Optimization applied to Pattern Recognition by Abel Mengistu Advisor: Dr. Raheel Ahmad CS Senior Research 2011 Manchester College May, 2011-1 - Table of Contents Introduction... - 3 - Objectives...
More informationLECTURE 16: SWARM INTELLIGENCE 2 / PARTICLE SWARM OPTIMIZATION 2
15-382 COLLECTIVE INTELLIGENCE - S18 LECTURE 16: SWARM INTELLIGENCE 2 / PARTICLE SWARM OPTIMIZATION 2 INSTRUCTOR: GIANNI A. DI CARO BACKGROUND: REYNOLDS BOIDS Reynolds created a model of coordinated animal
More informationLocal Search (Greedy Descent): Maintain an assignment of a value to each variable. Repeat:
Local Search Local Search (Greedy Descent): Maintain an assignment of a value to each variable. Repeat: Select a variable to change Select a new value for that variable Until a satisfying assignment is
More informationArtificial Intelligence
Artificial Intelligence Informed Search and Exploration Chapter 4 (4.3 4.6) Searching: So Far We ve discussed how to build goal-based and utility-based agents that search to solve problems We ve also presented
More informationGeneration of Ultra Side lobe levels in Circular Array Antennas using Evolutionary Algorithms
Generation of Ultra Side lobe levels in Circular Array Antennas using Evolutionary Algorithms D. Prabhakar Associate Professor, Dept of ECE DVR & Dr. HS MIC College of Technology Kanchikacherla, AP, India.
More informationThe Binary Genetic Algorithm. Universidad de los Andes-CODENSA
The Binary Genetic Algorithm Universidad de los Andes-CODENSA 1. Genetic Algorithms: Natural Selection on a Computer Figure 1 shows the analogy between biological i l evolution and a binary GA. Both start
More informationTopological Machining Fixture Layout Synthesis Using Genetic Algorithms
Topological Machining Fixture Layout Synthesis Using Genetic Algorithms Necmettin Kaya Uludag University, Mechanical Eng. Department, Bursa, Turkey Ferruh Öztürk Uludag University, Mechanical Eng. Department,
More informationWelfare Navigation Using Genetic Algorithm
Welfare Navigation Using Genetic Algorithm David Erukhimovich and Yoel Zeldes Hebrew University of Jerusalem AI course final project Abstract Using standard navigation algorithms and applications (such
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 2 Outline Local search techniques and optimization Hill-climbing
More informationCost Functions in Machine Learning
Cost Functions in Machine Learning Kevin Swingler Motivation Given some data that reflects measurements from the environment We want to build a model that reflects certain statistics about that data Something
More informationParticle Swarm Optimization
Particle Swarm Optimization Gonçalo Pereira INESC-ID and Instituto Superior Técnico Porto Salvo, Portugal gpereira@gaips.inesc-id.pt April 15, 2011 1 What is it? Particle Swarm Optimization is an algorithm
More informationLecture 3: Linear Classification
Lecture 3: Linear Classification Roger Grosse 1 Introduction Last week, we saw an example of a learning task called regression. There, the goal was to predict a scalar-valued target from a set of features.
More informationAnother Case Study: Genetic Algorithms
Chapter 4 Another Case Study: Genetic Algorithms Genetic Algorithms The section on Genetic Algorithms (GA) appears here because it is closely related to the problem of unsupervised learning. Much of what
More informationMore on Neural Networks. Read Chapter 5 in the text by Bishop, except omit Sections 5.3.3, 5.3.4, 5.4, 5.5.4, 5.5.5, 5.5.6, 5.5.7, and 5.
More on Neural Networks Read Chapter 5 in the text by Bishop, except omit Sections 5.3.3, 5.3.4, 5.4, 5.5.4, 5.5.5, 5.5.6, 5.5.7, and 5.6 Recall the MLP Training Example From Last Lecture log likelihood
More informationReducing Graphic Conflict In Scale Reduced Maps Using A Genetic Algorithm
Reducing Graphic Conflict In Scale Reduced Maps Using A Genetic Algorithm Dr. Ian D. Wilson School of Technology, University of Glamorgan, Pontypridd CF37 1DL, UK Dr. J. Mark Ware School of Computing,
More informationModule 1 Lecture Notes 2. Optimization Problem and Model Formulation
Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization
More information1 Lab 5: Particle Swarm Optimization
1 Lab 5: Particle Swarm Optimization This laboratory requires the following: (The development tools are installed in GR B0 01 already): C development tools (gcc, make, etc.) Webots simulation software
More informationEvolutionary Algorithms. CS Evolutionary Algorithms 1
Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms Genetic Algorithms l Simulate natural evolution of structures via selection and reproduction, based on performance
More informationA *69>H>N6 #DJGC6A DG C<>C::G>C<,8>:C8:H /DA 'D 2:6G, ()-"&"3 -"(' ( +-" " " % '.+ % ' -0(+$,
The structure is a very important aspect in neural network design, it is not only impossible to determine an optimal structure for a given problem, it is even impossible to prove that a given structure
More informationLECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS
LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS Neural Networks Classifier Introduction INPUT: classification data, i.e. it contains an classification (class) attribute. WE also say that the class
More informationGenetic Algorithms. Genetic Algorithms
A biological analogy for optimization problems Bit encoding, models as strings Reproduction and mutation -> natural selection Pseudo-code for a simple genetic algorithm The goal of genetic algorithms (GA):
More informationLecture 4. Convexity Robust cost functions Optimizing non-convex functions. 3B1B Optimization Michaelmas 2017 A. Zisserman
Lecture 4 3B1B Optimization Michaelmas 2017 A. Zisserman Convexity Robust cost functions Optimizing non-convex functions grid search branch and bound simulated annealing evolutionary optimization The Optimization
More informationLecture 6: Genetic Algorithm. An Introduction to Meta-Heuristics, Produced by Qiangfu Zhao (Since 2012), All rights reserved
Lecture 6: Genetic Algorithm An Introduction to Meta-Heuristics, Produced by Qiangfu Zhao (Since 2012), All rights reserved Lec06/1 Search and optimization again Given a problem, the set of all possible
More information1 Lab + Hwk 5: Particle Swarm Optimization
1 Lab + Hwk 5: Particle Swarm Optimization This laboratory requires the following equipment: C programming tools (gcc, make), already installed in GR B001 Webots simulation software Webots User Guide Webots
More information1 Lab + Hwk 5: Particle Swarm Optimization
1 Lab + Hwk 5: Particle Swarm Optimization This laboratory requires the following equipment: C programming tools (gcc, make). Webots simulation software. Webots User Guide Webots Reference Manual. The
More informationMETAHEURISTICS Genetic Algorithm
METAHEURISTICS Genetic Algorithm Jacques A. Ferland Department of Informatique and Recherche Opérationnelle Université de Montréal ferland@iro.umontreal.ca Genetic Algorithm (GA) Population based algorithm
More informationCHAPTER 1 INTRODUCTION
1 CHAPTER 1 INTRODUCTION 1.1 OPTIMIZATION OF MACHINING PROCESS AND MACHINING ECONOMICS In a manufacturing industry, machining process is to shape the metal parts by removing unwanted material. During the
More informationHeuristic Optimisation
Heuristic Optimisation Part 10: Genetic Algorithm Basics Sándor Zoltán Németh http://web.mat.bham.ac.uk/s.z.nemeth s.nemeth@bham.ac.uk University of Birmingham S Z Németh (s.nemeth@bham.ac.uk) Heuristic
More informationHybrid PSO-SA algorithm for training a Neural Network for Classification
Hybrid PSO-SA algorithm for training a Neural Network for Classification Sriram G. Sanjeevi 1, A. Naga Nikhila 2,Thaseem Khan 3 and G. Sumathi 4 1 Associate Professor, Dept. of CSE, National Institute
More informationCHAPTER 4 GENETIC ALGORITHM
69 CHAPTER 4 GENETIC ALGORITHM 4.1 INTRODUCTION Genetic Algorithms (GAs) were first proposed by John Holland (Holland 1975) whose ideas were applied and expanded on by Goldberg (Goldberg 1989). GAs is
More informationMarch 19, Heuristics for Optimization. Outline. Problem formulation. Genetic algorithms
Olga Galinina olga.galinina@tut.fi ELT-53656 Network Analysis and Dimensioning II Department of Electronics and Communications Engineering Tampere University of Technology, Tampere, Finland March 19, 2014
More informationOptimization and least squares. Prof. Noah Snavely CS1114
Optimization and least squares Prof. Noah Snavely CS1114 http://cs1114.cs.cornell.edu Administrivia A5 Part 1 due tomorrow by 5pm (please sign up for a demo slot) Part 2 will be due in two weeks (4/17)
More informationInformation Fusion Dr. B. K. Panigrahi
Information Fusion By Dr. B. K. Panigrahi Asst. Professor Department of Electrical Engineering IIT Delhi, New Delhi-110016 01/12/2007 1 Introduction Classification OUTLINE K-fold cross Validation Feature
More informationMeta- Heuristic based Optimization Algorithms: A Comparative Study of Genetic Algorithm and Particle Swarm Optimization
2017 2 nd International Electrical Engineering Conference (IEEC 2017) May. 19 th -20 th, 2017 at IEP Centre, Karachi, Pakistan Meta- Heuristic based Optimization Algorithms: A Comparative Study of Genetic
More informationCHAPTER 6 REAL-VALUED GENETIC ALGORITHMS
CHAPTER 6 REAL-VALUED GENETIC ALGORITHMS 6.1 Introduction Gradient-based algorithms have some weaknesses relative to engineering optimization. Specifically, it is difficult to use gradient-based algorithms
More information1. Introduction. 2. Motivation and Problem Definition. Volume 8 Issue 2, February Susmita Mohapatra
Pattern Recall Analysis of the Hopfield Neural Network with a Genetic Algorithm Susmita Mohapatra Department of Computer Science, Utkal University, India Abstract: This paper is focused on the implementation
More informationmywbut.com Informed Search Strategies-II
Informed Search Strategies-II 1 3.3 Iterative-Deepening A* 3.3.1 IDA* Algorithm Iterative deepening A* or IDA* is similar to iterative-deepening depth-first, but with the following modifications: The depth
More informationArtificial Intelligence
Artificial Intelligence Local Search Vibhav Gogate The University of Texas at Dallas Some material courtesy of Luke Zettlemoyer, Dan Klein, Dan Weld, Alex Ihler, Stuart Russell, Mausam Systematic Search:
More informationThe Continuous Genetic Algorithm. Universidad de los Andes-CODENSA
The Continuous Genetic Algorithm Universidad de los Andes-CODENSA 1. Components of a Continuous Genetic Algorithm The flowchart in figure1 provides a big picture overview of a continuous GA.. Figure 1.
More informationPARTICLE SWARM OPTIMIZATION (PSO)
PARTICLE SWARM OPTIMIZATION (PSO) J. Kennedy and R. Eberhart, Particle Swarm Optimization. Proceedings of the Fourth IEEE Int. Conference on Neural Networks, 1995. A population based optimization technique
More informationATI Material Do Not Duplicate ATI Material. www. ATIcourses.com. www. ATIcourses.com
ATI Material Material Do Not Duplicate ATI Material Boost Your Skills with On-Site Courses Tailored to Your Needs www.aticourses.com The Applied Technology Institute specializes in training programs for
More informationIntroduction to Computer Science and Programming for Astronomers
Introduction to Computer Science and Programming for Astronomers Lecture 9. István Szapudi Institute for Astronomy University of Hawaii March 21, 2018 Outline Reminder 1 Reminder 2 3 Reminder We have demonstrated
More informationGenetic Programming. Charles Chilaka. Department of Computational Science Memorial University of Newfoundland
Genetic Programming Charles Chilaka Department of Computational Science Memorial University of Newfoundland Class Project for Bio 4241 March 27, 2014 Charles Chilaka (MUN) Genetic algorithms and programming
More informationSoftware Vulnerability
Software Vulnerability Refers to a weakness in a system allowing an attacker to violate the integrity, confidentiality, access control, availability, consistency or audit mechanism of the system or the
More informationBayesian Methods in Vision: MAP Estimation, MRFs, Optimization
Bayesian Methods in Vision: MAP Estimation, MRFs, Optimization CS 650: Computer Vision Bryan S. Morse Optimization Approaches to Vision / Image Processing Recurring theme: Cast vision problem as an optimization
More informationGenetic Algorithms. Kang Zheng Karl Schober
Genetic Algorithms Kang Zheng Karl Schober Genetic algorithm What is Genetic algorithm? A genetic algorithm (or GA) is a search technique used in computing to find true or approximate solutions to optimization
More informationGenetic Algorithms. PHY 604: Computational Methods in Physics and Astrophysics II
Genetic Algorithms Genetic Algorithms Iterative method for doing optimization Inspiration from biology General idea (see Pang or Wikipedia for more details): Create a collection of organisms/individuals
More informationA Study on Optimization Algorithms for Clustering Gene Expression Data
A Study on Optimization Algorithms for Clustering Gene Expression Data Athul Jose 1, Chandrasekar P 2 PG scholor Dept of CSE, Bannari Amman Institute of technology, Sathyamangalam, Tamilnadu, India 1,
More informationEvolutionary Computation Algorithms for Cryptanalysis: A Study
Evolutionary Computation Algorithms for Cryptanalysis: A Study Poonam Garg Information Technology and Management Dept. Institute of Management Technology Ghaziabad, India pgarg@imt.edu Abstract The cryptanalysis
More informationDERIVATIVE-FREE OPTIMIZATION
DERIVATIVE-FREE OPTIMIZATION Main bibliography J.-S. Jang, C.-T. Sun and E. Mizutani. Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence. Prentice Hall, New Jersey,
More informationInstructor: Dr. Benjamin Thompson Lecture 20: 31 March 2009
Instructor: Dr. Benjamin hompson Lecture 20: 31 March 2009 Announcements MIDERMS ARE GRADED! Yay! I will return them AFER the lecture, so that you pay attention to the lecture. I m sadistic like that.
More information(Refer Slide Time 3:31)
Digital Circuits and Systems Prof. S. Srinivasan Department of Electrical Engineering Indian Institute of Technology Madras Lecture - 5 Logic Simplification In the last lecture we talked about logic functions
More information1.7 Limit of a Function
1.7 Limit of a Function We will discuss the following in this section: 1. Limit Notation 2. Finding a it numerically 3. Right and Left Hand Limits 4. Infinite Limits Consider the following graph Notation:
More informationComparison of TSP Algorithms
Comparison of TSP Algorithms Project for Models in Facilities Planning and Materials Handling December 1998 Participants: Byung-In Kim Jae-Ik Shim Min Zhang Executive Summary Our purpose in this term project
More informationEvolutionary algorithms in communications
Telecommunications seminar Evolutionary algorithms in Communications and systems Introduction lecture II: More about EAs Timo Mantere Professor Communications and systems engineering University of Vaasa
More informationHyperparameter optimization. CS6787 Lecture 6 Fall 2017
Hyperparameter optimization CS6787 Lecture 6 Fall 2017 Review We ve covered many methods Stochastic gradient descent Step size/learning rate, how long to run Mini-batching Batch size Momentum Momentum
More informationArtificial Intelligence Application (Genetic Algorithm)
Babylon University College of Information Technology Software Department Artificial Intelligence Application (Genetic Algorithm) By Dr. Asaad Sabah Hadi 2014-2015 EVOLUTIONARY ALGORITHM The main idea about
More informationHybrid Particle Swarm and Neural Network Approach for Streamflow Forecasting
Math. Model. Nat. Phenom. Vol. 5, No. 7, 010, pp. 13-138 DOI: 10.1051/mmnp/01057 Hybrid Particle Swarm and Neural Network Approach for Streamflow Forecasting A. Sedki and D. Ouazar Department of Civil
More informationMidterm Examination CS540-2: Introduction to Artificial Intelligence
Midterm Examination CS540-2: Introduction to Artificial Intelligence March 15, 2018 LAST NAME: FIRST NAME: Problem Score Max Score 1 12 2 13 3 9 4 11 5 8 6 13 7 9 8 16 9 9 Total 100 Question 1. [12] Search
More informationNon-deterministic Search techniques. Emma Hart
Non-deterministic Search techniques Emma Hart Why do local search? Many real problems are too hard to solve with exact (deterministic) techniques Modern, non-deterministic techniques offer ways of getting
More informationLab 4: Evolutionary Computation and Swarm Intelligence
Lab 4: Evolutionary Computation and Swarm Intelligence Introduction Goals The goal of this assignment is to learn about two evolutionary computing methods: genetic algorithms, and particle swarm optimization.
More informationHYBRID GENETIC ALGORITHM WITH GREAT DELUGE TO SOLVE CONSTRAINED OPTIMIZATION PROBLEMS
HYBRID GENETIC ALGORITHM WITH GREAT DELUGE TO SOLVE CONSTRAINED OPTIMIZATION PROBLEMS NABEEL AL-MILLI Financial and Business Administration and Computer Science Department Zarqa University College Al-Balqa'
More informationGenetic Algorithm for Finding Shortest Path in a Network
Intern. J. Fuzzy Mathematical Archive Vol. 2, 2013, 43-48 ISSN: 2320 3242 (P), 2320 3250 (online) Published on 26 August 2013 www.researchmathsci.org International Journal of Genetic Algorithm for Finding
More informationThe Genetic Algorithm for finding the maxima of single-variable functions
Research Inventy: International Journal Of Engineering And Science Vol.4, Issue 3(March 2014), PP 46-54 Issn (e): 2278-4721, Issn (p):2319-6483, www.researchinventy.com The Genetic Algorithm for finding
More informationSimple Model Selection Cross Validation Regularization Neural Networks
Neural Nets: Many possible refs e.g., Mitchell Chapter 4 Simple Model Selection Cross Validation Regularization Neural Networks Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University February
More informationSolving Optimization Problems with MATLAB Loren Shure
Solving Optimization Problems with MATLAB Loren Shure 6 The MathWorks, Inc. Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization Optimization
More informationGradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz
Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint
More informationPerceptron: This is convolution!
Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image
More informationInertia Weight. v i = ωv i +φ 1 R(0,1)(p i x i )+φ 2 R(0,1)(p g x i ) The new velocity update equation:
Convergence of PSO The velocity update equation: v i = v i +φ 1 R(0,1)(p i x i )+φ 2 R(0,1)(p g x i ) for some values of φ 1 and φ 2 the velocity grows without bound can bound velocity to range [ V max,v
More information6.001 Notes: Section 6.1
6.001 Notes: Section 6.1 Slide 6.1.1 When we first starting talking about Scheme expressions, you may recall we said that (almost) every Scheme expression had three components, a syntax (legal ways of
More informationThe k-means Algorithm and Genetic Algorithm
The k-means Algorithm and Genetic Algorithm k-means algorithm Genetic algorithm Rough set approach Fuzzy set approaches Chapter 8 2 The K-Means Algorithm The K-Means algorithm is a simple yet effective
More informationAPPLICATIONS OF INTELLIGENT HYBRID SYSTEMS IN MATLAB
APPLICATIONS OF INTELLIGENT HYBRID SYSTEMS IN MATLAB Z. Dideková, S. Kajan Institute of Control and Industrial Informatics, Faculty of Electrical Engineering and Information Technology, Slovak University
More informationData Structures and Algorithms Dr. Naveen Garg Department of Computer Science and Engineering Indian Institute of Technology, Delhi.
Data Structures and Algorithms Dr. Naveen Garg Department of Computer Science and Engineering Indian Institute of Technology, Delhi Lecture 18 Tries Today we are going to be talking about another data
More informationParticle Swarm Optimization Artificial Bee Colony Chain (PSOABCC): A Hybrid Meteahuristic Algorithm
Particle Swarm Optimization Artificial Bee Colony Chain (PSOABCC): A Hybrid Meteahuristic Algorithm Oğuz Altun Department of Computer Engineering Yildiz Technical University Istanbul, Turkey oaltun@yildiz.edu.tr
More informationEnsemble methods in machine learning. Example. Neural networks. Neural networks
Ensemble methods in machine learning Bootstrap aggregating (bagging) train an ensemble of models based on randomly resampled versions of the training set, then take a majority vote Example What if you
More informationInverse Kinematics (part 1) CSE169: Computer Animation Instructor: Steve Rotenberg UCSD, Winter 2018
Inverse Kinematics (part 1) CSE169: Computer Animation Instructor: Steve Rotenberg UCSD, Winter 2018 Welman, 1993 Inverse Kinematics and Geometric Constraints for Articulated Figure Manipulation, Chris
More information