Data Mining Chapter 8: Search and Optimization Methods Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
|
|
- Dorothy Watts
- 5 years ago
- Views:
Transcription
1 Data Mining Chapter 8: Search and Optimization Methods Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
2 Search & Optimization Search and Optimization method deals with how to find the best model(s) that makes the score function producing the minimum (maximum) value Search for the best model structure from a set of candidates Optimize the model parameters within a given model Major problem: Both the number of the possible model structures and parameter space are very huge! How to conduct efficient search and optimization?
3 Simply search strategy for models Exhaustive search For every model in the candidate set, find the best parameters w.r.t. the score function, and then compare the scores of all the models to find the best. Pros: Cons: Guarantee to find the best model w.r.t. the score function May be implemented in a parallel fashion Re-optimize the parameter for each new model structure Face to potentially combinatorial explosion Highly inefficient, and most of the time infeasible!
4 Being smart with compromise (I) Making use of the decomposable score function The score function for a new structure will be an additive function of the score function for the previous structure as well as a term accounting for the change in structure Pros: Cons: Easy to obtain the score function value of the current model based on the previous model Limited to particular score function
5 Being smart with compromise (II) Use approximation for the best parameter ( incremental ) Leave the existing parameters in the model fixed to the previous values and only optimize the parameters added to the model Pros: Cons: Reduces the number of parameters to be estimated when changing the model structure for a little bit Can save up time for searching more candidate model structures Proven to be suboptimal Error accumulation problem
6 Being smart with compromise (III) Heuristic search Apply some heuristic to narrow down the search space of the model structures Pros: Efficient under combinatorial explosions Intuitive, and easy to implement Cons: Lack of mathematically validity Might be ineffective under certain unknown situation.
7 State-space formulation for model search Model search problem can be viewed as one of moving through a discrete set of states State space representation Each state corresponds to a particular model in the candidate set Each state can be represented as a vertex in a graph Search operators Search operators corresponds to legal moves in our search space Can be represented as edges between the (state) vertices in a graph
8 Simple greedy search algorithm 1. Initialize: 2. Iterate: Choose a initial state M 0, corresponding to a particular model structure M k evaluate the score function at all possible adjacent states (as defined by the operators) and move to the best one 3. Stopping criterion: Repeat step 2 until no further improvement can be attained in the local score function 4. Multiple restart: Repeat steps 1 through 3 from different initial starting points and choose the best solution found. Suboptimal
9 Systematic search Search tree Instead of following the single path to search the best at every step, we keep track of multiple models simultaneously. Traverse on search tree Blind search Breadth-first search Consume huge memories Depth-first search Memory-efficient
10 Systematic search Traverse on search tree with heuristics Beam search Keep track of the b best models at any point in the search! Suboptimal. A trade-off for efficiency! Branch-and-bound Keep track of the best model structure so far Calculate analytically a lower bound on the best possible score function from a particular branch of the search tree If the bound is greater than the best score function so far, prune this branch. Difficult to find a tight bound. Scalability is limited!
11 Parameter optimization For a given model structure, we can link the parameter for this model structure to the score function, and directly optimize over the score function. Optimization can be down by calculating the minimum (or maximum) value directly Close form solution: Let, solving d linear equations Iterative Optimization
12 Greedy method for optimizing smooth functions 1. Initialize (Randomly) Choose an initial value for the parameter vector 2. Iterate: Determined by Local information 3. Convergence How much to change the value Repeat 2 until S appears to have attained a local minimum 4. Multiple restart Repeat steps 1 through 3 from different initial starting points and choose the best solution found
13 Univariate Optimization (I) The Newton-Raphson Method Based on Taylor series expansion Since then, Second order method So the update rule is Pros: Cons: Convergence rate is quadratic if close to the solution May not converge at all if far from the solution
14 Univariate Optimization (II) The Gradient descent Method The update rule is: Momentum-Based methods: Accelerate the convergence of gradient descent by adding momentum term. Considering the history of the path information Bracketing Method Finding a bracket that provably contains the extremum of the function Accelerate the convergence in low curvature region.
15 From 1-D to n-d (I) Two key questions: In which direction should we move from θ i? How far should we step in that direction? Multivariate methods: Multivariate gradient descent If the learning rate is sufficiently small, the gradient descent is guaranteed to converge to local minimum of S
16 From 1-D to n-d (II) Multivariate methods Newton s method If S is quadratic, the step taken by Newton s method directly points to the minimum of S S can be regarded as locally quadratic near the minimum pint This method involved a inverse of a matrix, which might be time consuming
17 From 1-D to n-d (III) Other Multivariate methods Coordinate descent For each axis of the original space, iteratively conduct univariate gradient descent Using conjugate directions Use principal axes to transform data and then find the direction in the transformed space. Simplex search Use simplex reflection to find search direction and the size of simplex as type of step size
18 Constrained optimization What is constrained optimization? The parameters take value from the feasible region, instead of the whole space. How to solve? s.t. Introduces Lagrange multipliers Let gradient equal to 0 The optimum is obtained under the KKT condiction.
19 Maximizing likelihood with missing parameter Problem settings: Given: Data set Hidden variables associated to each data Goal: Optimize the parameters θ w.r.t. the log likelihood function Challenge: Both parameters and the hidden variables are unknown! Aha, we can fix one and determine the other, and then fix the other one and determine this one!
20 The EM Algorithm The Expectation-Maximization (EM) algorithm is used for finding MLE of parameters in probabilistic models that depends on unobserved under variables. It is an iterative procedure altering between Expectation step and Maximization step E step: computes an expectation of the log likelihood w.r.t. the current estimate of the conditional distribution for the hidden variables M step: computes the parameters which maximize the expected log likelihood found on the E step
21 Why EM works? Idea behind EM Find a lower bound of log likelihood function At each step, optimize the lower bound w.r.t. one set of unknown parameters by fixing other set of parameters at their current values Iterates until the process converges Maximizing the lower bound function will find the parameters that increases the log likelihood function.
22 Encodes the basic idea of EM with some mathematics The lower bound: Jessen s inequality Let s check it out in detail Iterative optimization: From k-th rounds to the (k+1)-th rounds Convergence: F value will be increased during the any successive rounds F is bounded from the top by L which has an maximum value
23 The EM cookbook To use EM, the following issues should be determined: What is the log likelihood function. (EM can apply to any likelihood function) What is the model parameters and what is the hidden variables What is the expectation in the E step, and how to compute it. What is supposed to be maximized based on the expectation, and how to compute it.
24 Optimizing parameters with single scan In many online applications Data come in stream Limited space for storing training data Need quick response for a given input. We can only receive one data point, make use of it and then discard it. Stochastic approximation would be applied Reform the batch score function to instantaneous score function which only depends on the current example Optimize the instantaneous score function (e.g., take a gradient descent) The average instantaneous score function should asymptotically approach to the batch score function to guarantee the approximation is good.
25 Heuristic search and optimization (I) Genetic search Genetic algorithms are general set of heuristic search techniques based on ideas from evolutionary biology A framework of GA Represent models as chromosomes (binary string) Evolve a population of such chromosomes by selectively pairing (according to their fitness defined by a score function) and mutate chromosomes to create off-springs Essential idea of GA: Maintaining a set of candidate models instead of one to allow simultaneously exploring of the state space New states are created based on combination of the current state, allowing jump to different part of the state-space to avoid being stuck in local minimum
26 Heuristic search and optimization (II) Simulated annealing Simulated annealing is a heuristic search technique based on ideas from physics The framework Allow the moves in state-space that decrease the score function to be minimized Allow some moves (with some probability) that can increase the score function, which is controlled by the temperature that gradually decreases. Key idea Higher temperature enables large moves in the parameter space to explore many possible place at beginning, in the hope that the large moves may lead to the deepest basin Lower temperature decreases the energy for large moves such that the search is stabilized into local search, to avoid escaping from the deep basin
27 Let s move to Chapter 9
Theoretical Concepts of Machine Learning
Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5
More informationModule 1 Lecture Notes 2. Optimization Problem and Model Formulation
Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization
More informationAPPLIED OPTIMIZATION WITH MATLAB PROGRAMMING
APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING Second Edition P. Venkataraman Rochester Institute of Technology WILEY JOHN WILEY & SONS, INC. CONTENTS PREFACE xiii 1 Introduction 1 1.1. Optimization Fundamentals
More informationToday. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient
Optimization Last time Root finding: definition, motivation Algorithms: Bisection, false position, secant, Newton-Raphson Convergence & tradeoffs Example applications of Newton s method Root finding in
More informationData Mining Chapter 9: Descriptive Modeling Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
Data Mining Chapter 9: Descriptive Modeling Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Descriptive model A descriptive model presents the main features of the data
More information3.6.2 Generating admissible heuristics from relaxed problems
3.6.2 Generating admissible heuristics from relaxed problems To come up with heuristic functions one can study relaxed problems from which some restrictions of the original problem have been removed The
More informationCost Functions in Machine Learning
Cost Functions in Machine Learning Kevin Swingler Motivation Given some data that reflects measurements from the environment We want to build a model that reflects certain statistics about that data Something
More informationCHAPTER 2 CONVENTIONAL AND NON-CONVENTIONAL TECHNIQUES TO SOLVE ORPD PROBLEM
20 CHAPTER 2 CONVENTIONAL AND NON-CONVENTIONAL TECHNIQUES TO SOLVE ORPD PROBLEM 2.1 CLASSIFICATION OF CONVENTIONAL TECHNIQUES Classical optimization methods can be classified into two distinct groups:
More information25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard
More informationArtificial Intelligence
Artificial Intelligence Informed Search and Exploration Chapter 4 (4.3 4.6) Searching: So Far We ve discussed how to build goal-based and utility-based agents that search to solve problems We ve also presented
More informationEscaping Local Optima: Genetic Algorithm
Artificial Intelligence Escaping Local Optima: Genetic Algorithm Dae-Won Kim School of Computer Science & Engineering Chung-Ang University We re trying to escape local optima To achieve this, we have learned
More information10703 Deep Reinforcement Learning and Control
10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient I Used Materials Disclaimer: Much of the material and slides for this lecture
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 Outline Local search techniques and optimization Hill-climbing
More informationConstrained Optimization COS 323
Constrained Optimization COS 323 Last time Introduction to optimization objective function, variables, [constraints] 1-dimensional methods Golden section, discussion of error Newton s method Multi-dimensional
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 2 Outline Local search techniques and optimization Hill-climbing
More informationLocal Search and Optimization Chapter 4. Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld )
Local Search and Optimization Chapter 4 Mausam (Based on slides of Padhraic Smyth, Stuart Russell, Rao Kambhampati, Raj Rao, Dan Weld ) 1 2 Outline Local search techniques and optimization Hill-climbing
More informationHeuristic Optimisation
Heuristic Optimisation Revision Lecture Sándor Zoltán Németh http://web.mat.bham.ac.uk/s.z.nemeth s.nemeth@bham.ac.uk University of Birmingham S Z Németh (s.nemeth@bham.ac.uk) Heuristic Optimisation University
More informationFinding optimal configurations Adversarial search
CS 171 Introduction to AI Lecture 10 Finding optimal configurations Adversarial search Milos Hauskrecht milos@cs.pitt.edu 39 Sennott Square Announcements Homework assignment is out Due on Thursday next
More informationLecture 4. Convexity Robust cost functions Optimizing non-convex functions. 3B1B Optimization Michaelmas 2017 A. Zisserman
Lecture 4 3B1B Optimization Michaelmas 2017 A. Zisserman Convexity Robust cost functions Optimizing non-convex functions grid search branch and bound simulated annealing evolutionary optimization The Optimization
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationNumerical Optimization: Introduction and gradient-based methods
Numerical Optimization: Introduction and gradient-based methods Master 2 Recherche LRI Apprentissage Statistique et Optimisation Anne Auger Inria Saclay-Ile-de-France November 2011 http://tao.lri.fr/tiki-index.php?page=courses
More informationBeyond Classical Search: Local Search. CMPSCI 383 September 23, 2011
Beyond Classical Search: Local Search CMPSCI 383 September 23, 2011 1 Today s lecture Local Search Hill-climbing Simulated annealing Local beam search Genetic algorithms Genetic programming Local search
More informationAn evolutionary annealing-simplex algorithm for global optimisation of water resource systems
FIFTH INTERNATIONAL CONFERENCE ON HYDROINFORMATICS 1-5 July 2002, Cardiff, UK C05 - Evolutionary algorithms in hydroinformatics An evolutionary annealing-simplex algorithm for global optimisation of water
More information5 Machine Learning Abstractions and Numerical Optimization
Machine Learning Abstractions and Numerical Optimization 25 5 Machine Learning Abstractions and Numerical Optimization ML ABSTRACTIONS [some meta comments on machine learning] [When you write a large computer
More informationGradient Descent. 1) S! initial state 2) Repeat: Similar to: - hill climbing with h - gradient descent over continuous space
Local Search 1 Local Search Light-memory search method No search tree; only the current state is represented! Only applicable to problems where the path is irrelevant (e.g., 8-queen), unless the path is
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationMETAHEURISTICS. Introduction. Introduction. Nature of metaheuristics. Local improvement procedure. Example: objective function
Introduction METAHEURISTICS Some problems are so complicated that are not possible to solve for an optimal solution. In these problems, it is still important to find a good feasible solution close to the
More informationMidterm Examination CS540-2: Introduction to Artificial Intelligence
Midterm Examination CS540-2: Introduction to Artificial Intelligence March 15, 2018 LAST NAME: FIRST NAME: Problem Score Max Score 1 12 2 13 3 9 4 11 5 8 6 13 7 9 8 16 9 9 Total 100 Question 1. [12] Search
More informationChapter 14 Global Search Algorithms
Chapter 14 Global Search Algorithms An Introduction to Optimization Spring, 2015 Wei-Ta Chu 1 Introduction We discuss various search methods that attempts to search throughout the entire feasible set.
More informationTDDC17. Intuitions behind heuristic search. Recall Uniform-Cost Search. Best-First Search. f(n) =... + h(n) g(n) = cost of path from root node to n
Intuitions behind heuristic search The separation property of GRAPH-SEARCH TDDC17 Seminar III Search II Informed or Heuristic Search Beyond Classical Search Find a heuristic measure h(n) which estimates
More informationTopological Machining Fixture Layout Synthesis Using Genetic Algorithms
Topological Machining Fixture Layout Synthesis Using Genetic Algorithms Necmettin Kaya Uludag University, Mechanical Eng. Department, Bursa, Turkey Ferruh Öztürk Uludag University, Mechanical Eng. Department,
More information4 INFORMED SEARCH AND EXPLORATION. 4.1 Heuristic Search Strategies
55 4 INFORMED SEARCH AND EXPLORATION We now consider informed search that uses problem-specific knowledge beyond the definition of the problem itself This information helps to find solutions more efficiently
More informationLocal Search (Ch )
Local Search (Ch. 4-4.1) Local search Before we tried to find a path from the start state to a goal state using a fringe set Now we will look at algorithms that do not care about a fringe, but just neighbors
More informationHeuristic Optimization Introduction and Simple Heuristics
Heuristic Optimization Introduction and Simple Heuristics José M PEÑA (jmpena@fi.upm.es) (Universidad Politécnica de Madrid) 1 Outline 1. What are optimization problems? 2. Exhaustive vs. Heuristic approaches
More informationIntroduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.
Chapter 7: Derivative-Free Optimization Introduction (7.1) Genetic Algorithms (GA) (7.2) Simulated Annealing (SA) (7.3) Random Search (7.4) Downhill Simplex Search (DSS) (7.5) Jyh-Shing Roger Jang et al.,
More informationCS 331: Artificial Intelligence Local Search 1. Tough real-world problems
CS 331: Artificial Intelligence Local Search 1 1 Tough real-world problems Suppose you had to solve VLSI layout problems (minimize distance between components, unused space, etc.) Or schedule airlines
More informationOptimization in Brachytherapy. Gary A. Ezzell, Ph.D. Mayo Clinic Scottsdale
Optimization in Brachytherapy Gary A. Ezzell, Ph.D. Mayo Clinic Scottsdale Outline General concepts of optimization Classes of optimization techniques Concepts underlying some commonly available methods
More informationEnergy Minimization -Non-Derivative Methods -First Derivative Methods. Background Image Courtesy: 3dciencia.com visual life sciences
Energy Minimization -Non-Derivative Methods -First Derivative Methods Background Image Courtesy: 3dciencia.com visual life sciences Introduction Contents Criteria to start minimization Energy Minimization
More informationSPATIAL OPTIMIZATION METHODS
DELMELLE E. (2010). SPATIAL OPTIMIZATION METHODS. IN: B. WHARF (ED). ENCYCLOPEDIA OF HUMAN GEOGRAPHY: 2657-2659. SPATIAL OPTIMIZATION METHODS Spatial optimization is concerned with maximizing or minimizing
More informationUsing Genetic Algorithms in Integer Programming for Decision Support
Doi:10.5901/ajis.2014.v3n6p11 Abstract Using Genetic Algorithms in Integer Programming for Decision Support Dr. Youcef Souar Omar Mouffok Taher Moulay University Saida, Algeria Email:Syoucef12@yahoo.fr
More informationSimplicial Global Optimization
Simplicial Global Optimization Julius Žilinskas Vilnius University, Lithuania September, 7 http://web.vu.lt/mii/j.zilinskas Global optimization Find f = min x A f (x) and x A, f (x ) = f, where A R n.
More informationAlgorithm Design (4) Metaheuristics
Algorithm Design (4) Metaheuristics Takashi Chikayama School of Engineering The University of Tokyo Formalization of Constraint Optimization Minimize (or maximize) the objective function f(x 0,, x n )
More informationTDDC17. Intuitions behind heuristic search. Best-First Search. Recall Uniform-Cost Search. f(n) =... + h(n) g(n) = cost of path from root node to n
Intuitions behind heuristic search The separation property of GRAPH-SEARCH TDDC17 Seminar III Search II Informed or Heuristic Search Beyond Classical Search Find a heuristic measure h(n) which estimates
More informationMarch 19, Heuristics for Optimization. Outline. Problem formulation. Genetic algorithms
Olga Galinina olga.galinina@tut.fi ELT-53656 Network Analysis and Dimensioning II Department of Electronics and Communications Engineering Tampere University of Technology, Tampere, Finland March 19, 2014
More informationCS 188: Artificial Intelligence Spring Today
CS 188: Artificial Intelligence Spring 2006 Lecture 7: CSPs II 2/7/2006 Dan Klein UC Berkeley Many slides from either Stuart Russell or Andrew Moore Today More CSPs Applications Tree Algorithms Cutset
More informationB553 Lecture 12: Global Optimization
B553 Lecture 12: Global Optimization Kris Hauser February 20, 2012 Most of the techniques we have examined in prior lectures only deal with local optimization, so that we can only guarantee convergence
More informationAdministrative. Local Search!
Administrative Local Search! CS311 David Kauchak Spring 2013 Assignment 2 due Tuesday before class Written problems 2 posted Class participation http://www.youtube.com/watch? v=irhfvdphfzq&list=uucdoqrpqlqkvctckzqa
More informationN-Queens problem. Administrative. Local Search
Local Search CS151 David Kauchak Fall 2010 http://www.youtube.com/watch?v=4pcl6-mjrnk Some material borrowed from: Sara Owsley Sood and others Administrative N-Queens problem Assign 1 grading Assign 2
More informationmywbut.com Informed Search Strategies-II
Informed Search Strategies-II 1 3.3 Iterative-Deepening A* 3.3.1 IDA* Algorithm Iterative deepening A* or IDA* is similar to iterative-deepening depth-first, but with the following modifications: The depth
More informationArtificial Intelligence
Artificial Intelligence Local Search Vibhav Gogate The University of Texas at Dallas Some material courtesy of Luke Zettlemoyer, Dan Klein, Dan Weld, Alex Ihler, Stuart Russell, Mausam Systematic Search:
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationArtificial Intelligence
Artificial Intelligence Information Systems and Machine Learning Lab (ISMLL) Tomáš Horváth 16 rd November, 2011 Informed Search and Exploration Example (again) Informed strategy we use a problem-specific
More informationHYBRID GENETIC ALGORITHM WITH GREAT DELUGE TO SOLVE CONSTRAINED OPTIMIZATION PROBLEMS
HYBRID GENETIC ALGORITHM WITH GREAT DELUGE TO SOLVE CONSTRAINED OPTIMIZATION PROBLEMS NABEEL AL-MILLI Financial and Business Administration and Computer Science Department Zarqa University College Al-Balqa'
More informationA Systematic Overview of Data Mining Algorithms
A Systematic Overview of Data Mining Algorithms 1 Data Mining Algorithm A well-defined procedure that takes data as input and produces output as models or patterns well-defined: precisely encoded as a
More informationCS 188: Artificial Intelligence
CS 188: Artificial Intelligence CSPs II + Local Search Prof. Scott Niekum The University of Texas at Austin [These slides based on those of Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley.
More information3 INTEGER LINEAR PROGRAMMING
3 INTEGER LINEAR PROGRAMMING PROBLEM DEFINITION Integer linear programming problem (ILP) of the decision variables x 1,..,x n : (ILP) subject to minimize c x j j n j= 1 a ij x j x j 0 x j integer n j=
More informationCrash-Starting the Simplex Method
Crash-Starting the Simplex Method Ivet Galabova Julian Hall School of Mathematics, University of Edinburgh Optimization Methods and Software December 2017 Ivet Galabova, Julian Hall Crash-Starting Simplex
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationInternational Journal of Current Research and Modern Education (IJCRME) ISSN (Online): & Impact Factor: Special Issue, NCFTCCPS -
TO SOLVE ECONOMIC DISPATCH PROBLEM USING SFLA P. Sowmya* & Dr. S. P. Umayal** * PG Scholar, Department Electrical and Electronics Engineering, Muthayammal Engineering College, Rasipuram, Tamilnadu ** Dean
More informationOutline of the module
Evolutionary and Heuristic Optimisation (ITNPD8) Lecture 2: Heuristics and Metaheuristics Gabriela Ochoa http://www.cs.stir.ac.uk/~goc/ Computing Science and Mathematics, School of Natural Sciences University
More informationIntroduction to ANSYS DesignXplorer
Lecture 5 Goal Driven Optimization 14. 5 Release Introduction to ANSYS DesignXplorer 1 2013 ANSYS, Inc. September 27, 2013 Goal Driven Optimization (GDO) Goal Driven Optimization (GDO) is a multi objective
More informationMetaheuristic Development Methodology. Fall 2009 Instructor: Dr. Masoud Yaghini
Metaheuristic Development Methodology Fall 2009 Instructor: Dr. Masoud Yaghini Phases and Steps Phases and Steps Phase 1: Understanding Problem Step 1: State the Problem Step 2: Review of Existing Solution
More information: Principles of Automated Reasoning and Decision Making Midterm
16.410-13: Principles of Automated Reasoning and Decision Making Midterm October 20 th, 2003 Name E-mail Note: Budget your time wisely. Some parts of this quiz could take you much longer than others. Move
More informationToday. CS 188: Artificial Intelligence Fall Example: Boolean Satisfiability. Reminder: CSPs. Example: 3-SAT. CSPs: Queries.
CS 188: Artificial Intelligence Fall 2007 Lecture 5: CSPs II 9/11/2007 More CSPs Applications Tree Algorithms Cutset Conditioning Today Dan Klein UC Berkeley Many slides over the course adapted from either
More informationCHAPTER 6 ORTHOGONAL PARTICLE SWARM OPTIMIZATION
131 CHAPTER 6 ORTHOGONAL PARTICLE SWARM OPTIMIZATION 6.1 INTRODUCTION The Orthogonal arrays are helpful in guiding the heuristic algorithms to obtain a good solution when applied to NP-hard problems. This
More informationCS:4420 Artificial Intelligence
CS:4420 Artificial Intelligence Spring 2018 Beyond Classical Search Cesare Tinelli The University of Iowa Copyright 2004 18, Cesare Tinelli and Stuart Russell a a These notes were originally developed
More informationArtificial Intelligence
Artificial Intelligence Information Systems and Machine Learning Lab (ISMLL) Tomáš Horváth 10 rd November, 2010 Informed Search and Exploration Example (again) Informed strategy we use a problem-specific
More informationData Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Exploratory data analysis tasks Examine the data, in search of structures
More informationARTIFICIAL INTELLIGENCE (CSCU9YE ) LECTURE 5: EVOLUTIONARY ALGORITHMS
ARTIFICIAL INTELLIGENCE (CSCU9YE ) LECTURE 5: EVOLUTIONARY ALGORITHMS Gabriela Ochoa http://www.cs.stir.ac.uk/~goc/ OUTLINE Optimisation problems Optimisation & search Two Examples The knapsack problem
More informationIntroduction to Optimization
Introduction to Optimization Approximation Algorithms and Heuristics November 21, 2016 École Centrale Paris, Châtenay-Malabry, France Dimo Brockhoff Inria Saclay Ile-de-France 2 Exercise: The Knapsack
More informationLecture 3, Review of Algorithms. What is Algorithm?
BINF 336, Introduction to Computational Biology Lecture 3, Review of Algorithms Young-Rae Cho Associate Professor Department of Computer Science Baylor University What is Algorithm? Definition A process
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationA Steady-State Genetic Algorithm for Traveling Salesman Problem with Pickup and Delivery
A Steady-State Genetic Algorithm for Traveling Salesman Problem with Pickup and Delivery Monika Sharma 1, Deepak Sharma 2 1 Research Scholar Department of Computer Science and Engineering, NNSS SGI Samalkha,
More informationHeuristic (Informed) Search
Heuristic (Informed) Search (Where we try to choose smartly) R&N: Chap., Sect..1 3 1 Search Algorithm #2 SEARCH#2 1. INSERT(initial-node,Open-List) 2. Repeat: a. If empty(open-list) then return failure
More informationOutline. Best-first search. Greedy best-first search A* search Heuristics Local search algorithms
Outline Best-first search Greedy best-first search A* search Heuristics Local search algorithms Hill-climbing search Beam search Simulated annealing search Genetic algorithms Constraint Satisfaction Problems
More informationPre-requisite Material for Course Heuristics and Approximation Algorithms
Pre-requisite Material for Course Heuristics and Approximation Algorithms This document contains an overview of the basic concepts that are needed in preparation to participate in the course. In addition,
More informationx n+1 = x n f(x n) f (x n ), (1)
1 Optimization The field of optimization is large and vastly important, with a deep history in computer science (among other places). Generally, an optimization problem is defined by having a score function
More informationLecture Plan. Best-first search Greedy search A* search Designing heuristics. Hill-climbing. 1 Informed search strategies. Informed strategies
Lecture Plan 1 Informed search strategies (KA AGH) 1 czerwca 2010 1 / 28 Blind vs. informed search strategies Blind search methods You already know them: BFS, DFS, UCS et al. They don t analyse the nodes
More informationClassical Gradient Methods
Classical Gradient Methods Note simultaneous course at AMSI (math) summer school: Nonlin. Optimization Methods (see http://wwwmaths.anu.edu.au/events/amsiss05/) Recommended textbook (Springer Verlag, 1999):
More informationLocal Search (Greedy Descent): Maintain an assignment of a value to each variable. Repeat:
Local Search Local Search (Greedy Descent): Maintain an assignment of a value to each variable. Repeat: Select a variable to change Select a new value for that variable Until a satisfying assignment is
More informationInformed search algorithms. Chapter 4
Informed search algorithms Chapter 4 Outline Best-first search Greedy best-first search A * search Heuristics Memory Bounded A* Search Best-first search Idea: use an evaluation function f(n) for each node
More informationLocal and Global Minimum
Local and Global Minimum Stationary Point. From elementary calculus, a single variable function has a stationary point at if the derivative vanishes at, i.e., 0. Graphically, the slope of the function
More informationSparse Matrices Reordering using Evolutionary Algorithms: A Seeded Approach
1 Sparse Matrices Reordering using Evolutionary Algorithms: A Seeded Approach David Greiner, Gustavo Montero, Gabriel Winter Institute of Intelligent Systems and Numerical Applications in Engineering (IUSIANI)
More informationSolving Traveling Salesman Problem Using Parallel Genetic. Algorithm and Simulated Annealing
Solving Traveling Salesman Problem Using Parallel Genetic Algorithm and Simulated Annealing Fan Yang May 18, 2010 Abstract The traveling salesman problem (TSP) is to find a tour of a given number of cities
More informationLocal Search. CS 486/686: Introduction to Artificial Intelligence Winter 2016
Local Search CS 486/686: Introduction to Artificial Intelligence Winter 2016 1 Overview Uninformed Search Very general: assumes no knowledge about the problem BFS, DFS, IDS Informed Search Heuristics A*
More informationFast-Lipschitz Optimization
Fast-Lipschitz Optimization DREAM Seminar Series University of California at Berkeley September 11, 2012 Carlo Fischione ACCESS Linnaeus Center, Electrical Engineering KTH Royal Institute of Technology
More informationAn Evolutionary Algorithm for Minimizing Multimodal Functions
An Evolutionary Algorithm for Minimizing Multimodal Functions D.G. Sotiropoulos, V.P. Plagianakos and M.N. Vrahatis University of Patras, Department of Mamatics, Division of Computational Mamatics & Informatics,
More informationOptimization Techniques for Design Space Exploration
0-0-7 Optimization Techniques for Design Space Exploration Zebo Peng Embedded Systems Laboratory (ESLAB) Linköping University Outline Optimization problems in ERT system design Heuristic techniques Simulated
More informationMulti-objective Optimization
Some introductory figures from : Deb Kalyanmoy, Multi-Objective Optimization using Evolutionary Algorithms, Wiley 2001 Multi-objective Optimization Implementation of Constrained GA Based on NSGA-II Optimization
More informationArtificial Intelligence, CS, Nanjing University Spring, 2016, Yang Yu. Lecture 5: Search 4.
Artificial Intelligence, CS, Nanjing University Spring, 2016, Yang Yu Lecture 5: Search 4 http://cs.nju.edu.cn/yuy/course_ai16.ashx Previously... Path-based search Uninformed search Depth-first, breadth
More information56:272 Integer Programming & Network Flows Final Exam -- December 16, 1997
56:272 Integer Programming & Network Flows Final Exam -- December 16, 1997 Answer #1 and any five of the remaining six problems! possible score 1. Multiple Choice 25 2. Traveling Salesman Problem 15 3.
More informationIntroduction to Optimization
Introduction to Optimization Approximation Algorithms and Heuristics November 6, 2015 École Centrale Paris, Châtenay-Malabry, France Dimo Brockhoff INRIA Lille Nord Europe 2 Exercise: The Knapsack Problem
More informationTwo approaches. Local Search TSP. Examples of algorithms using local search. Local search heuristics - To do list
Unless P=NP, there is no polynomial time algorithm for SAT, MAXSAT, MIN NODE COVER, MAX INDEPENDENT SET, MAX CLIQUE, MIN SET COVER, TSP,. But we have to solve (instances of) these problems anyway what
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Eric Xing Lecture 14, February 29, 2016 Reading: W & J Book Chapters Eric Xing @
More informationLecture 4: Local and Randomized/Stochastic Search
Lecture 4: Local and Randomized/Stochastic Search CS 580 (001) - Spring 2018 Amarda Shehu Department of Computer Science George Mason University, Fairfax, VA, USA February 14, 2018 Amarda Shehu (580) 1
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationDavid G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer
David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms
More informationEvolutionary Computation Algorithms for Cryptanalysis: A Study
Evolutionary Computation Algorithms for Cryptanalysis: A Study Poonam Garg Information Technology and Management Dept. Institute of Management Technology Ghaziabad, India pgarg@imt.edu Abstract The cryptanalysis
More information