# Lecture 4. Convexity Robust cost functions Optimizing non-convex functions. 3B1B Optimization Michaelmas 2017 A. Zisserman

Save this PDF as:

Size: px
Start display at page:

Download "Lecture 4. Convexity Robust cost functions Optimizing non-convex functions. 3B1B Optimization Michaelmas 2017 A. Zisserman"

## Transcription

1 Lecture 4 3B1B Optimization Michaelmas 2017 A. Zisserman Convexity Robust cost functions Optimizing non-convex functions grid search branch and bound simulated annealing evolutionary optimization

2 The Optimization Tree

3 Unconstrained optimization function of one variable local minimum global minimum down-hill search (gradient descent) algorithms can find local minima which of the minima is found depends on the starting point such minima often occur in real applications

4 Cost functions in 2D Global maximum Multiple optima

5 How can you tell if an optimization has a single optimum? The answer is: see if the optimization problem is convex. If it is, then a local optimum is the global optimum. First, we need to introduce Convex Sets, and Convex Functions Note sketch introduction only

6 convex Not convex

7 Convex functions

8 Convex function examples convex Not convex A non-negative sum of convex functions is convex

9 Convex Optimization Problem Minimize: a convex function over a convex set Then locally optimal points are globally optimal Also, such problems can be solved both in theory and practice

10 Why do we need the domain to be convex? local optimum domain (not a convex set) can t get from here to here by downhill search

11 Examples of convex optimization problems Many more useful examples, see Boyd & Vandenberghe

12

13 Second order condition

14 Strictly convex strictly convex one global optimum Not strictly convex multiple local optima (but all are global)

15 Robust Cost Functions In formulating an optimization problem there is often some room for design and choice The cost function can be chosen to be: convex robust to noise (outliers) in the data/measurements

16 Motivation Fitting a 2D line to a set of 2D points Suppose you fit a straight line to data containing outliers points that are not properly modelled by the assumed measurement noise distribution The usual method of least squares estimation is hopelessly corrupted

17 Least squares: Robustness to noise Least squares fit to the red points:

18 Least squares: Robustness to noise Least squares fit with an outlier: Problem: squared error heavily penalizes outliers

19 Consider minimizing the cost function { a i } the data { a i } may be thought of as repeated measurements of a fixed value (at 0), subject to Gaussian noise and some outliers it has 10% of outliers biased towards the right of the true value the minimum of f(x) does not correspond to the true value

20 Examine the behaviour of various cost functions quadratic truncated quadratic L 1 huber

21 Quadratic cost function squared error the usual default cost function arises in Maximum Likelihood Estimation for Gaussian noise convex

22 Truncated Quadratic cost function for inliers behaves as a quadratic truncated so that outliers only incur a fixed cost non-convex

23 L 1 cost function absolute error called total variation convex non-differentiable at origin finds the median of { a i }

24 Huber cost function hybrid between quadratic and L 1 continuous first derivative for small values is quadratic for larger values becomes linear thus has the outlier stability of L 1 convex

25 Example 1: measurements with outliers { a i } quadratic truncated quadratic huber zoom

26 Example 2: bimodal measurements { a i } 70% in principal mode 30% in outlier mode quadratic truncated quadratic huber

27 Summary Squared cost function very susceptible to outliers Truncated quadratic has a stable minimum, but is non-convex and also has other local minima. Also basin of attraction of global minimum limited Huber has stable minimum and is convex

28 Application 1: Robust line fitting Huber cost Least squares (See also RANSAC algorithm) Boyd & Vandenberghe

29 Application 2: Signal restoration x Compare: i Quadratic smoothing Total variation

30 Quadratic smoothing Quadratic smoothing smooths out noise and steps in the signal Boyd & Vandenberghe

31 Total variation Total variation smoothing preserves steps in the signal Boyd & Vandenberghe

32 Optimizing non-convex functions function of one variable Sketch four methods: 1. grid search: uniform grid space covering 2. branch and bound 3. simulated annealing: stochastic optimization 4. evolutionary optimization local minimum global minimum

33 2. Branch and bound LB UB Key idea: A B C D Split region into sub-regions and compute bounds Consider two regions A and C If lower bound of A is greater than upper bound of C then A can be discarded divide (branch) regions and repeat

34 3. Simulated Annealing 1 2 local minimum global minimum The algorithm has a mechanism to jump out of local minima It is a stochastic search method, i.e. it uses randomness in the search

35 Simulated annealing algorithm At each iteration propose a move in the parameter space If the move decreases the cost, then accept it If the move increases the cost by E, then accept it with a probability exp(- E/T), Otherwise, don t move Note probability depends on temperature T Decrease the temperature according to a schedule so that at the start cost increases are likely to be accepted, and at the end they are not

36 Boltzmann distribution and the cooling schedule start with T high, then exp(- E/T) is approx. 1, and all moves are accepted many cooling schedules are possible, but the simplest is Boltzmann distribution exp(- E/T) exp(-x) exp(-x/10) exp(-x/100) 0.6 where k is the iteration number The algorithm can be very slow to converge

37 Simulated annealing The name and inspiration come from annealing in metallurgy, a technique involving heating and controlled cooling of a material to increase the size of its crystals and reduce their defects. The heat causes the atoms to become unstuck from their initial positions (a local minimum of the internal energy) and wander randomly through states of higher energy; the slow cooling gives them more chances of finding configurations with lower internal energy than the initial one. Algorithms due to: Kirkpatrick et al. 1982; Metropolis et al.1953.

38 Example: Convergence of simulated annealing AT INIT_TEMP unconditional Acceptance HILL CLIMBING move accepted with probability exp(- E/T) COST FUNCTION, C HILL CLIMBING HILL CLIMBING AT FINAL_TEMP NUMBER OF ITERATIONS

39 Steepest descent on a graph Slide from Adam Kalai and Santosh Vempala

40 Random Search on a graph

41 Simulated Annealing on a graph Phase 1: Hot (Random) Phase 2: Warm (Bias down) Phase 3: Cold (Descend) (Descent)

42 Simulated annealing for the Rosenbrock function Figures from

43 4. Evolutionary optimization Uses mechanisms inspired by biological evolution, such as reproduction, mutation, recombination, and selection. Candidate solutions to the optimization problem play the role of individuals in a population, and the cost function determines the quality of the solutions. Evolution of the population then takes place after repeated application of the above operators. Algorithm: 1. Generate the initial population of individuals randomly. 2. Evaluate the fitness of each individual in that population (cost function) 3. Repeat the following regenerational steps until termination: Select the best-fit individuals for reproduction. (Parents) Breed new individuals through crossover and mutation operations to give birth to offspring. Evaluate the individual fitness of new individuals. Replace least-fit population with new individuals. (from Wikipedia)

44 Crossover (genetic algorithm) One-point crossover: Two-point crossover: By R0oland - Own work, CC BY-SA 3.0,

45 Population operators Crossover: Parent 1 Parent Mutation: Child 1 Child Slide credit: Fred van Keulen, Matthijs Langelaar

46 Example local minimum global minimum 1 2 * * * * * * * * * *

47 Genetic Algorithm for the Rosenbrock function Figures from

48 Example: Evolving virtual creatures Virtual block creatures optimized to perform a given task, such as the ability to swim in a simulated water environment, or walk across a plane swimming hopping Karl Sims, 1994

49 There is more There are many other classes of optimization problem, and also many efficient optimization algorithms developed for problems with special structure. Examples include: Combinatorial and discrete optimization Dynamic programming Max-flow/Min-cut graph cuts See the links on the web page and come to the C Optimization lectures next year

### Introduction to Design Optimization: Search Methods

Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape

### Evolutionary Computation Algorithms for Cryptanalysis: A Study

Evolutionary Computation Algorithms for Cryptanalysis: A Study Poonam Garg Information Technology and Management Dept. Institute of Management Technology Ghaziabad, India pgarg@imt.edu Abstract The cryptanalysis

### Introduction to Genetic Algorithms

Advanced Topics in Image Analysis and Machine Learning Introduction to Genetic Algorithms Week 3 Faculty of Information Science and Engineering Ritsumeikan University Today s class outline Genetic Algorithms

### Simulated annealing/metropolis and genetic optimization

Simulated annealing/metropolis and genetic optimization Eugeniy E. Mikhailov The College of William & Mary Lecture 18 Eugeniy Mikhailov (W&M) Practical Computing Lecture 18 1 / 8 Nature s way to find a

### Simulated Annealing. Slides based on lecture by Van Larhoven

Simulated Annealing Slides based on lecture by Van Larhoven Iterative Improvement 1 General method to solve combinatorial optimization problems Principle: Start with initial configuration Repeatedly search

### An evolutionary annealing-simplex algorithm for global optimisation of water resource systems

FIFTH INTERNATIONAL CONFERENCE ON HYDROINFORMATICS 1-5 July 2002, Cardiff, UK C05 - Evolutionary algorithms in hydroinformatics An evolutionary annealing-simplex algorithm for global optimisation of water

### Introduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.

Chapter 7: Derivative-Free Optimization Introduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.5) Jyh-Shing Roger Jang et al.,

### SIMULATED ANNEALING TECHNIQUES AND OVERVIEW. Daniel Kitchener Young Scholars Program Florida State University Tallahassee, Florida, USA

SIMULATED ANNEALING TECHNIQUES AND OVERVIEW Daniel Kitchener Young Scholars Program Florida State University Tallahassee, Florida, USA 1. INTRODUCTION Simulated annealing is a global optimization algorithm

### Theoretical Concepts of Machine Learning

Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5

### Artificial Intelligence

Artificial Intelligence Local Search Vibhav Gogate The University of Texas at Dallas Some material courtesy of Luke Zettlemoyer, Dan Klein, Dan Weld, Alex Ihler, Stuart Russell, Mausam Systematic Search:

### Outline. Best-first search. Greedy best-first search A* search Heuristics Local search algorithms

Outline Best-first search Greedy best-first search A* search Heuristics Local search algorithms Hill-climbing search Beam search Simulated annealing search Genetic algorithms Constraint Satisfaction Problems

### Automated Test Data Generation and Optimization Scheme Using Genetic Algorithm

2011 International Conference on Software and Computer Applications IPCSIT vol.9 (2011) (2011) IACSIT Press, Singapore Automated Test Data Generation and Optimization Scheme Using Genetic Algorithm Roshni

### Dr. Stephan Steigele Vorlesung im SS 2008

Dr. Vorlesung im SS 2008 Bioinf / Universität Leipzig Recombination issues Genetic algorithms Holland s original GA is now known as the simple genetic algorithm (SGA) Other GAs use different: Representations

### MATLAB Based Optimization Techniques and Parallel Computing

MATLAB Based Optimization Techniques and Parallel Computing Bratislava June 4, 2009 2009 The MathWorks, Inc. Jörg-M. Sautter Application Engineer The MathWorks Agenda Introduction Local and Smooth Optimization

### Module 1 Lecture Notes 2. Optimization Problem and Model Formulation

Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization

### Hill Climbing. Assume a heuristic value for each assignment of values to all variables. Maintain an assignment of a value to each variable.

Hill Climbing Many search spaces are too big for systematic search. A useful method in practice for some consistency and optimization problems is hill climbing: Assume a heuristic value for each assignment

### Heuristic Optimization Introduction and Simple Heuristics

Heuristic Optimization Introduction and Simple Heuristics José M PEÑA (jmpena@fi.upm.es) (Universidad Politécnica de Madrid) 1 Outline 1. What are optimization problems? 2. Exhaustive vs. Heuristic approaches

### Perceptron: This is convolution!

Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image

### GENETIC ALGORITHM with Hands-On exercise

GENETIC ALGORITHM with Hands-On exercise Adopted From Lecture by Michael Negnevitsky, Electrical Engineering & Computer Science University of Tasmania 1 Objective To understand the processes ie. GAs Basic

### LINEAR AND NONLINEAR IMAGE RESTORATION METHODS FOR EDDY

LINEAR AND NONLINEAR IMAGE RESTORATION METHODS FOR EDDY CURRENT NONDESTRUCTIVE EVALUATION Bing Wang:12 John P. Basart,12 and John C. Moulderl lcenter for NDE 2Department of Electrical and Computer Engineering

### Outline. Motivation. Introduction of GAs. Genetic Algorithm 9/7/2017. Motivation Genetic algorithms An illustrative example Hypothesis space search

Outline Genetic Algorithm Motivation Genetic algorithms An illustrative example Hypothesis space search Motivation Evolution is known to be a successful, robust method for adaptation within biological

### Hardware-Software Codesign

Hardware-Software Codesign 4. System Partitioning Lothar Thiele 4-1 System Design specification system synthesis estimation SW-compilation intellectual prop. code instruction set HW-synthesis intellectual

### Genetic Algorithms. Kang Zheng Karl Schober

Genetic Algorithms Kang Zheng Karl Schober Genetic algorithm What is Genetic algorithm? A genetic algorithm (or GA) is a search technique used in computing to find true or approximate solutions to optimization

### Multi-Modal Metropolis Nested Sampling For Inspiralling Binaries

Multi-Modal Metropolis Nested Sampling For Inspiralling Binaries Ed Porter (AEI) & Jon Gair (IOA) 2W@AEI Workshop AEI September 2008 (We acknowledge useful converstions with F. Feroz and M. Hobson (Cavendish

### Fast oriented bounding box optimization on the rotation group SO(3, R)

Fast oriented bounding box optimization on the rotation group SO(3, R) Chia-Tche Chang 1, Bastien Gorissen 2,3 and Samuel Melchior 1,2 chia-tche.chang@uclouvain.be bastien.gorissen@cenaero.be samuel.melchior@uclouvain.be

### CS 331: Artificial Intelligence Local Search 1. Tough real-world problems

CS 331: Artificial Intelligence Local Search 1 1 Tough real-world problems Suppose you had to solve VLSI layout problems (minimize distance between components, unused space, etc.) Or schedule airlines

### A Steady-State Genetic Algorithm for Traveling Salesman Problem with Pickup and Delivery

A Steady-State Genetic Algorithm for Traveling Salesman Problem with Pickup and Delivery Monika Sharma 1, Deepak Sharma 2 1 Research Scholar Department of Computer Science and Engineering, NNSS SGI Samalkha,

### Evolutionary Algorithms. CS Evolutionary Algorithms 1

Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms Genetic Algorithms l Simulate natural evolution of structures via selection and reproduction, based on performance

### Sparse Matrices Reordering using Evolutionary Algorithms: A Seeded Approach

1 Sparse Matrices Reordering using Evolutionary Algorithms: A Seeded Approach David Greiner, Gustavo Montero, Gabriel Winter Institute of Intelligent Systems and Numerical Applications in Engineering (IUSIANI)

### Random Search Report An objective look at random search performance for 4 problem sets

Random Search Report An objective look at random search performance for 4 problem sets Dudon Wai Georgia Institute of Technology CS 7641: Machine Learning Atlanta, GA dwai3@gatech.edu Abstract: This report

### An Evolutionary Algorithm for Minimizing Multimodal Functions

An Evolutionary Algorithm for Minimizing Multimodal Functions D.G. Sotiropoulos, V.P. Plagianakos and M.N. Vrahatis University of Patras, Department of Mamatics, Division of Computational Mamatics & Informatics,

### CHAPTER 4 GENETIC ALGORITHM

69 CHAPTER 4 GENETIC ALGORITHM 4.1 INTRODUCTION Genetic Algorithms (GAs) were first proposed by John Holland (Holland 1975) whose ideas were applied and expanded on by Goldberg (Goldberg 1989). GAs is

### CS5401 FS2015 Exam 1 Key

CS5401 FS2015 Exam 1 Key This is a closed-book, closed-notes exam. The only items you are allowed to use are writing implements. Mark each sheet of paper you use with your name and the string cs5401fs2015

### Uninformed Search Methods. Informed Search Methods. Midterm Exam 3/13/18. Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall

Midterm Exam Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall Covers topics through Decision Trees and Random Forests (does not include constraint satisfaction) Closed book 8.5 x 11 sheet with notes

### Reducing Graphic Conflict In Scale Reduced Maps Using A Genetic Algorithm

Reducing Graphic Conflict In Scale Reduced Maps Using A Genetic Algorithm Dr. Ian D. Wilson School of Technology, University of Glamorgan, Pontypridd CF37 1DL, UK Dr. J. Mark Ware School of Computing,

### Using Genetic Algorithms to Solve the Box Stacking Problem

Using Genetic Algorithms to Solve the Box Stacking Problem Jenniffer Estrada, Kris Lee, Ryan Edgar October 7th, 2010 Abstract The box stacking or strip stacking problem is exceedingly difficult to solve

### Informed search algorithms. (Based on slides by Oren Etzioni, Stuart Russell)

Informed search algorithms (Based on slides by Oren Etzioni, Stuart Russell) The problem # Unique board configurations in search space 8-puzzle 9! = 362880 15-puzzle 16! = 20922789888000 10 13 24-puzzle

### Genetic.io. Genetic Algorithms in all their shapes and forms! Genetic.io Make something of your big data

Genetic Algorithms in all their shapes and forms! Julien Sebrien Self-taught, passion for development. Java, Cassandra, Spark, JPPF. @jsebrien, julien.sebrien@genetic.io Distribution of IT solutions (SaaS,

### CONCEPT FORMATION AND DECISION TREE INDUCTION USING THE GENETIC PROGRAMMING PARADIGM

1 CONCEPT FORMATION AND DECISION TREE INDUCTION USING THE GENETIC PROGRAMMING PARADIGM John R. Koza Computer Science Department Stanford University Stanford, California 94305 USA E-MAIL: Koza@Sunburn.Stanford.Edu

### A Genetic Algorithm for Graph Matching using Graph Node Characteristics 1 2

Chapter 5 A Genetic Algorithm for Graph Matching using Graph Node Characteristics 1 2 Graph Matching has attracted the exploration of applying new computing paradigms because of the large number of applications

### Simulated Annealing. G5BAIM: Artificial Intelligence Methods. Graham Kendall. 15 Feb 09 1

G5BAIM: Artificial Intelligence Methods Graham Kendall 15 Feb 09 1 G5BAIM Artificial Intelligence Methods Graham Kendall Simulated Annealing Simulated Annealing Motivated by the physical annealing process

### Neural Network Weight Selection Using Genetic Algorithms

Neural Network Weight Selection Using Genetic Algorithms David Montana presented by: Carl Fink, Hongyi Chen, Jack Cheng, Xinglong Li, Bruce Lin, Chongjie Zhang April 12, 2005 1 Neural Networks Neural networks

### Computer Vision I - Basics of Image Processing Part 1

Computer Vision I - Basics of Image Processing Part 1 Carsten Rother 28/10/2014 Computer Vision I: Basics of Image Processing Link to lectures Computer Vision I: Basics of Image Processing 28/10/2014 2

### Neural Network Learning. Today s Lecture. Continuation of Neural Networks. Artificial Neural Networks. Lecture 24: Learning 3. Victor R.

Lecture 24: Learning 3 Victor R. Lesser CMPSCI 683 Fall 2010 Today s Lecture Continuation of Neural Networks Artificial Neural Networks Compose of nodes/units connected by links Each link has a numeric

### Artificial Intelligence Application (Genetic Algorithm)

Babylon University College of Information Technology Software Department Artificial Intelligence Application (Genetic Algorithm) By Dr. Asaad Sabah Hadi 2014-2015 EVOLUTIONARY ALGORITHM The main idea about

### A TOOLSET FOR SUPPORTING CONTINUOUS DECISION MAKING CASE: GRADE CHANGE OPTIMIZATION

A TOOLSET FOR SUPPORTING CONTINUOUS DECISION MAKING CASE: GRADE CHANGE OPTIMIZATION Petteri Pulkkinen 1, Toni Ahonen, Risto Ritala Institute of Measurement and Information Technology, Tampere University

### Cost Functions in Machine Learning

Cost Functions in Machine Learning Kevin Swingler Motivation Given some data that reflects measurements from the environment We want to build a model that reflects certain statistics about that data Something

### Classification: Linear Discriminant Functions

Classification: Linear Discriminant Functions CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Discriminant functions Linear Discriminant functions

### Lecture Plan. Best-first search Greedy search A* search Designing heuristics. Hill-climbing. 1 Informed search strategies. Informed strategies

Lecture Plan 1 Informed search strategies (KA AGH) 1 czerwca 2010 1 / 28 Blind vs. informed search strategies Blind search methods You already know them: BFS, DFS, UCS et al. They don t analyse the nodes

### A Genetic Algorithm Framework

Fast, good, cheap. Pick any two. The Project Triangle 3 A Genetic Algorithm Framework In this chapter, we develop a genetic algorithm based framework to address the problem of designing optimal networks

### 2. Linear Regression and Gradient Descent

Pattern Recognition And Machine Learning - EPFL - Fall 2015 Emtiyaz Khan, Timur Bagautdinov, Carlos Becker, Ilija Bogunovic & Ksenia Konyushkova 2. Linear Regression and Gradient Descent 2.1 Goals The

### Outline of Lecture. Scope of Optimization in Practice. Scope of Optimization (cont.)

Scope of Optimization in Practice and Niche of Evolutionary Methodologies Kalyanmoy Deb* Department of Business Technology Helsinki School of Economics Kalyanmoy.deb@hse.fi http://www.iitk.ac.in/kangal/deb.htm

### Introduction to optimization methods and line search

Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi How to find optimal solutions? Trial and error widely used in practice, not efficient and

### Feature Matching and Robust Fitting

Feature Matching and Robust Fitting Computer Vision CS 143, Brown Read Szeliski 4.1 James Hays Acknowledgment: Many slides from Derek Hoiem and Grauman&Leibe 2008 AAAI Tutorial Project 2 questions? This

### Automata Construct with Genetic Algorithm

Automata Construct with Genetic Algorithm Vít Fábera Department of Informatics and Telecommunication, Faculty of Transportation Sciences, Czech Technical University, Konviktská 2, Praha, Czech Republic,

### Gradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz

Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint

### 10/03/11. Model Fitting. Computer Vision CS 143, Brown. James Hays. Slides from Silvio Savarese, Svetlana Lazebnik, and Derek Hoiem

10/03/11 Model Fitting Computer Vision CS 143, Brown James Hays Slides from Silvio Savarese, Svetlana Lazebnik, and Derek Hoiem Fitting: find the parameters of a model that best fit the data Alignment:

### What is GOSET? GOSET stands for Genetic Optimization System Engineering Tool

Lecture 5: GOSET 1 What is GOSET? GOSET stands for Genetic Optimization System Engineering Tool GOSET is a MATLAB based genetic algorithm toolbox for solving optimization problems 2 GOSET Features Wide

### Optimal Wireless Sensor Network Layout with Metaheuristics: Solving a Large Scale Instance

Optimal Wireless Sensor Network Layout with Metaheuristics: Solving a Large Scale Instance Enrique Alba, Guillermo Molina Departamento de Lenguajes y Ciencias de la Computación University of Málaga, 29071

### A More Stable Approach To LISP Tree GP

A More Stable Approach To LISP Tree GP Joseph Doliner August 15, 2008 Abstract In this paper we begin by familiarising ourselves with the basic concepts of Evolutionary Computing and how it can be used

### Simplex of Nelder & Mead Algorithm

Simplex of N & M Simplex of Nelder & Mead Algorithm AKA the Amoeba algorithm In the class of direct search methods Unconstrained (although constraints can be added as part of error function) nonlinear

### An Introduction to Evolutionary Algorithms

An Introduction to Evolutionary Algorithms Karthik Sindhya, PhD Postdoctoral Researcher Industrial Optimization Group Department of Mathematical Information Technology Karthik.sindhya@jyu.fi http://users.jyu.fi/~kasindhy/

### Introduction to Optimization

Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,

### Comparison of TSP Algorithms

Comparison of TSP Algorithms Project for Models in Facilities Planning and Materials Handling December 1998 Participants: Byung-In Kim Jae-Ik Shim Min Zhang Executive Summary Our purpose in this term project

### COMPARISON OF DIFFERENT HEURISTIC, METAHEURISTIC, NATURE BASED OPTIMIZATION ALGORITHMS FOR TRAVELLING SALESMAN PROBLEM SOLUTION

COMPARISON OF DIFFERENT HEURISTIC, METAHEURISTIC, NATURE BASED OPTIMIZATION ALGORITHMS FOR TRAVELLING SALESMAN PROBLEM SOLUTION 1 KIRTI PANDEY, 2 PALLAVI JAIN 1 Shri Vaishnav Institute of Technology &

### Image Processing. Filtering. Slide 1

Image Processing Filtering Slide 1 Preliminary Image generation Original Noise Image restoration Result Slide 2 Preliminary Classic application: denoising However: Denoising is much more than a simple

### Robust Regression. Robust Data Mining Techniques By Boonyakorn Jantaranuson

Robust Regression Robust Data Mining Techniques By Boonyakorn Jantaranuson Outline Introduction OLS and important terminology Least Median of Squares (LMedS) M-estimator Penalized least squares What is

### UNBIASED ESTIMATION OF DESTINATION CHOICE MODELS WITH ATTRACTION CONSTRAINTS

UNBIASED ESTIMATION OF DESTINATION CHOICE MODELS WITH ATTRACTION CONSTRAINTS Vince Bernardin, PhD Steven Trevino John Gliebe, PhD APRIL 14, 2014 WHAT S WRONG WITH ESTIMATING DOUBLY CONSTRAINED DESTINATION

### DIT411/TIN175, Artificial Intelligence. Peter Ljunglöf. 23 January, 2018

DIT411/TIN175, Artificial Intelligence Chapters 3 4: More search algorithms CHAPTERS 3 4: MORE SEARCH ALGORITHMS DIT411/TIN175, Artificial Intelligence Peter Ljunglöf 23 January, 2018 1 TABLE OF CONTENTS

### CT79 SOFT COMPUTING ALCCS-FEB 2014

Q.1 a. Define Union, Intersection and complement operations of Fuzzy sets. For fuzzy sets A and B Figure Fuzzy sets A & B The union of two fuzzy sets A and B is a fuzzy set C, written as C=AUB or C=A OR

### 10703 Deep Reinforcement Learning and Control

10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient I Used Materials Disclaimer: Much of the material and slides for this lecture

### Lecture

Lecture.. 7 Constrained problems & optimization Brief introduction differential evolution Brief eample of hybridization of EAs Multiobjective problems & optimization Pareto optimization This slides mainly

### Genetic Algorithm for FPGA Placement

Genetic Algorithm for FPGA Placement Zoltan Baruch, Octavian Creţ, and Horia Giurgiu Computer Science Department, Technical University of Cluj-Napoca, 26, Bariţiu St., 3400 Cluj-Napoca, Romania {Zoltan.Baruch,

### Simulated Annealing for Placement Problem to minimise the wire length

Simulated Annealing for Placement Problem to minimise the wire length Ajay Sharma 2nd May 05 Contents 1 Algorithm 3 1.1 Definition of Algorithm...................... 3 1.1.1 Algorithm.........................

### Introduction to unconstrained optimization - derivative-free methods

Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of

### THE preceding chapters were all devoted to the analysis of images and signals which

Chapter 5 Segmentation of Color, Texture, and Orientation Images THE preceding chapters were all devoted to the analysis of images and signals which take values in IR. It is often necessary, however, to

### Midterm Examination CS 540-2: Introduction to Artificial Intelligence

Midterm Examination CS 54-2: Introduction to Artificial Intelligence March 9, 217 LAST NAME: FIRST NAME: Problem Score Max Score 1 15 2 17 3 12 4 6 5 12 6 14 7 15 8 9 Total 1 1 of 1 Question 1. [15] State

### Derivative-Free Optimization

Derivative-Free Optimization Chapter 7 from Jang Outline Simulated Annealing (SA) Downhill simplex search Random search Genetic algorithms (GA) 2 The Big Picture Model space Adaptive networks Neural networks

### Origins of Operations Research: World War II

ESD.83 Historical Roots Assignment METHODOLOGICAL LINKS BETWEEN OPERATIONS RESEARCH AND STOCHASTIC OPTIMIZATION Chaiwoo Lee Jennifer Morris 11/10/2010 Origins of Operations Research: World War II Need

### A grid representation for Distributed Virtual Environments

A grid representation for Distributed Virtual Environments Pedro Morillo, Marcos Fernández, Nuria Pelechano Instituto de Robótica, Universidad de Valencia. Polígono Coma S/N. Aptdo.Correos 22085, CP: 46071

### What is machine learning?

Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship

### Optimization of Benchmark Functions Using Genetic Algorithm

Optimization of Benchmark s Using Genetic Algorithm Vinod Goyal GJUS&T, Hisar Sakshi Dhingra GJUS&T, Hisar Jyoti Goyat GJUS&T, Hisar Dr Sanjay Singla IET Bhaddal Technical Campus, Ropar, Punjab Abstrat

### Constrained Functions of N Variables: Non-Gradient Based Methods

onstrained Functions of N Variables: Non-Gradient Based Methods Gerhard Venter Stellenbosch University Outline Outline onstrained Optimization Non-gradient based methods Genetic Algorithms (GA) Particle

### Correspondence. CS 468 Geometry Processing Algorithms. Maks Ovsjanikov

Shape Matching & Correspondence CS 468 Geometry Processing Algorithms Maks Ovsjanikov Wednesday, October 27 th 2010 Overall Goal Given two shapes, find correspondences between them. Overall Goal Given

### Genetic programming. Lecture Genetic Programming. LISP as a GP language. LISP structure. S-expressions

Genetic programming Lecture Genetic Programming CIS 412 Artificial Intelligence Umass, Dartmouth One of the central problems in computer science is how to make computers solve problems without being explicitly

### Artificial Intelligence

Artificial Intelligence CS482, CS682, MW 1 2:15, SEM 201, MS 227 Prerequisites: 302, 365 Instructor: Sushil Louis, sushil@cse.unr.edu, http://www.cse.unr.edu/~sushil Informed Search Best First Search A*

### Optimization of heterogeneous Bin packing using adaptive genetic algorithm

IOP Conference Series: Materials Science and Engineering PAPER OPEN ACCESS Optimization of heterogeneous Bin packing using adaptive genetic algorithm To cite this article: R Sridhar et al 2017 IOP Conf.

OPTIMIZATION METHODS modefrontier is a registered product of ESTECO srl Copyright ESTECO srl 1999-2007 For more information visit: www.esteco.com or send an e-mail to: modefrontier@esteco.com NEOS Optimization

### Regularized Energy Minimization Models in Image Processing. Tibor Lukić. Faculty of Technical Sciences, University of Novi Sad, Serbia

20150715 NOVI SAD Regularized Energy Minimization Models in Image Processing Tibor Lukić Faculty of Technical Sciences, University of Novi Sad, Serbia SSIP Szeged, 2015 NOVI SAD NOVI SAD UNIVERSITY OF

### An Evolutionary Algorithm for the Multi-objective Shortest Path Problem

An Evolutionary Algorithm for the Multi-objective Shortest Path Problem Fangguo He Huan Qi Qiong Fan Institute of Systems Engineering, Huazhong University of Science & Technology, Wuhan 430074, P. R. China

### First-improvement vs. Best-improvement Local Optima Networks of NK Landscapes

First-improvement vs. Best-improvement Local Optima Networks of NK Landscapes Gabriela Ochoa 1, Sébastien Verel 2, and Marco Tomassini 3 1 School of Computer Science, University of Nottingham, Nottingham,

### Introduction to Machine Learning CMU-10701

Introduction to Machine Learning CMU-10701 Clustering and EM Barnabás Póczos & Aarti Singh Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K-

### PROJECT REPORT. Parallel Optimization in Matlab. Joakim Agnarsson, Mikael Sunde, Inna Ermilova Project in Computational Science: Report January 2013

Parallel Optimization in Matlab Joakim Agnarsson, Mikael Sunde, Inna Ermilova Project in Computational Science: Report January 2013 PROJECT REPORT Department of Information Technology Contents 1 Introduction

### COMS 4771 Support Vector Machines. Nakul Verma

COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron

### MAXIMUM LIKELIHOOD ESTIMATION USING ACCELERATED GENETIC ALGORITHMS

In: Journal of Applied Statistical Science Volume 18, Number 3, pp. 1 7 ISSN: 1067-5817 c 2011 Nova Science Publishers, Inc. MAXIMUM LIKELIHOOD ESTIMATION USING ACCELERATED GENETIC ALGORITHMS Füsun Akman

### GENETIC ALGORITHM VERSUS PARTICLE SWARM OPTIMIZATION IN N-QUEEN PROBLEM

Journal of Al-Nahrain University Vol.10(2), December, 2007, pp.172-177 Science GENETIC ALGORITHM VERSUS PARTICLE SWARM OPTIMIZATION IN N-QUEEN PROBLEM * Azhar W. Hammad, ** Dr. Ban N. Thannoon Al-Nahrain

### Segmentation of Noisy Binary Images Containing Circular and Elliptical Objects using Genetic Algorithms

Segmentation of Noisy Binary Images Containing Circular and Elliptical Objects using Genetic Algorithms B. D. Phulpagar Computer Engg. Dept. P. E. S. M. C. O. E., Pune, India. R. S. Bichkar Prof. ( Dept.

CutLeader Technology algorithm is the soul of nesting software. For example knapsack algorithm, Pair technology, are able to get a better nesting result. The former is the approximate optimization algorithm;