A Learning Algorithm for Piecewise Linear Regression
|
|
- Prudence George
- 5 years ago
- Views:
Transcription
1 A Learning Algorithm for Piecewise Linear Regression Giancarlo Ferrari-Trecate 1, arco uselli 2, Diego Liberati 3, anfred orari 1 1 nstitute für Automatik, ETHZ - ETL CH 8092 Zürich, Switzerland 2 stituto per i Circuiti Elettronici - CNR via De arini, Genova, taly 3 Ce.S.T..A. - CNR c/o Politecnico di ilano Piazza Leonardo da Vinci, ilano, taly Abstract A new learning algorithm for solving piecewise linear regression problems is proposed. t is able to train a proper multilayer feedforward neural network so as to reconstruct a target function assuming a different linear behavior on each set of a polyhedral partition of the input domain. The proposed method combine local estimation, clustering in weight space, classification and regression in order to achieve the desired result. A simulation on a benchmark problem shows the good properties of this new learning algorithm. 1 ntroduction Real-world problems to be solved by artificial neural networks are normally subdivided in two groups according to the range of values assumed by the output. f it is Boolean or nominal, we speak of classification problems; otherwise, when the output is coded by a continuous variable, we are facing with a regression problem. n most cases, the techniques employed to train a connectionist model depend on the kind of problem we are dealing with. However, applications can be found, which lie on the borderline between classification and regression; these occur when the input space can be subdivided into disjoint regions X i characterized by different behaviors of the function f to be reconstructed. The target of the learning problem is consequently twofold: by analyzing a set of samples of f, possibly affected by noise, it has to generate both the collection of regions X i and the behavior of the unknown function f in each of them. f the region X i corresponding to each sample in the training set were known, we could add the index i of the region as an output, thus obtaining a classification problem which has the target of finding the effective form of each X i. On the other side, if the actual partition X i were known, we could solve several regression problems to find the behavior of the function f within each X i.
2 Because of this mixed nature, classical techniques for neural network training cannot be directly applied, but specific methods are necessary to deal with this kind of problems. Perhaps, the simplest situation one can think of is piecewise linear regression: in this case the regions X i are polyhedra and the behavior of the function f in each X i can be modeled by a linear expression. Several authors have treated this kind of problem [2, 3, 4, 8], providing algorithms for reaching the desired result. Unfortunately, most of them are difficult to extend beyond two dimensions [2], whereas others consider only local approximations [3, 4], thus missing the effective extension of regions X i. n this contribution a new training algorithm for neural networks solving piecewise linear regression problems is proposed. t combines clustering and supervised learning to obtain the correct values for the weights of a proper multilayer feedforward architecture. 2 The piecewise linear regression problem Let X be a polyhedron in the n-dimensional space R n and X i, i = 1,..., s, a polyhedral partition of X, i.e. X i X j = for every i, j = 1,..., s and s i=1 X i = X. The target of a Piecewise Linear Regression (PLR) problem is to reconstruct an unknown function f : X R having a linear behavior in each region X i f(x) = z i = w i0 + n w ij x j j=1 when only a training set S containing m samples (x k, y k ), k = 1,..., m, is available. The output y k gives an evaluation of f(x k ) subject to noise, being x k X; the region X i to which x k belongs is not known in advance. Scalars w i0, w i1,..., w in, for i = 1,..., s, characterize univocally the function f and their estimate is a target of the PLR problem; for notational purposes they will be included in a vector w i. Since regions X i are polyhedral, they can be defined by a set of l i linear inequalities of the following kind: a ij0 + n a ijk x k 0 (1) k=1 Scalar a ijk, for j = 1,..., l i and k = 0, 1,..., n, can be included in a matrix A i, whose estimate is still a target of the reconstruction process for every i = 1,..., s. Discontinuities may be present in the function f at the boundaries between two regions X i. Following the general idea presented in [8], a neural network realizing a piecewise linear function f of this kind can be modeled as in Fig. 1. t contains a gate layer that verifies inequalities (1) and decides which of the terms z i must be used as the output y of the whole network. Thus, the i-th unit in the gate
3 N 5 O K J F K J = O A H ) ) ) / = J A = O A H A = O A H N N 1 F K J = O A H Figure 1: General neural network realizing a piecewise linear function. layer has output equal to its input z i, if all the constraints (1) are satisfied for j = 1,..., l i, and equal to 0 in the opposite case. All the other units perform a weighted sum of their inputs; the weights of the output neuron, having no bias, are always set to 1. 3 The proposed learning algorithm As previously noted, the solution of a PLR problem requires a technique that combine classification and regression: the first has the aim of finding matrices A i to be inserted in the gate layer of the neural network (Fig. 1), whereas the latter provides weight vectors w i for the input to hidden layer connections. A method of this kind is reported in Fig. 2; it is composed of four steps, each of which is devoted to a specific task. The first of them (Step 1) has the aim of obtaining a first estimate of the weight vectors w i by performing local linear regressions based on small subsets of the whole training set S. n fact, points x k that are close to each other are likely to belong to the same region X i. Then, for each sample (x k, y k ), with k = 1,..., m, we build a set C k containing (x k, y k ) and the c 1 distinct pairs (x, y) S that score the lowest values of the distance x k x. The parameter c can be freely chosen, though the inequality c n must be respected to perform the linear regression. t can be easily seen that some sets C k, called mixed, will contain input patterns belonging to different regions X i. They lead to wrong estimates for w i and consequently their number must be kept minimum; this can be obtained by lowering the value of c. However,
4 ALGORTH FOR PECEWSE LNEAR REGRESSON 1. (Local regression) For every k = 1,..., m do 1a. Form the set C k containing the pair (x k, y k ) and the samples (x, y) S associated with the c 1 nearest neighbors x to x k. 1b. Perform a linear regression to obtain the weight vector v k of a linear unit fitting the samples in C k. 2. (Clustering) Perform a clustering process in the space R n+1 to subdivide the set of weight vectors v k into s groups V i. 3. (Classification) Build a new training set S containing the m pairs (x k, i k ), being V ik the cluster including v k. Train a multicategory classification method to produce the matrices A i for the regions X i. 4. (Regression) For every i = 1,..., s perform a linear regression on the samples (x, y) S with x X i to obtain the weight vector w i for the i-th unit in the hidden layer. Figure 2: Proposed learning method for piecewise linear regression. the quality of the estimate improves when the size c of the sets C k increases; a tradeoff must therefore be attained in selecting a reasonable value for c. Denote with v k the weight vector of the linear unit produced through the linear regression on the samples in C k. f the generation of the samples in the training set is not affected by noise, most of the v k coincide with the desired weight vectors w i. Only mixed sets C k yield spurious vectors v k, which can be considered as outliers. Nevertheless, even in presence of noise, a clustering algorithm (Step 2) can be used to determine the sets V i of vectors v k associated with the same w i. A proper version of the K-means algorithm [6] can be adopted to this aim if the number s of regions is fixed beforehand; otherwise, adaptive techniques, such as the Growing Neural Gas [7], can be employed to find at the same time the value of s. The sets V i generated by the clustering process induce a classification on the input patterns x k belonging to the training set S. As a matter of fact, if v k V i for a given i, the set C k is fitted by the linear neuron with weight vector w i and consequently x k is located into the region X i. The effective extension of this region can be determined by solving a linear multicategory classification problem (Step 3), whose training set S is built by adding as output to each input pattern x k the index i k of the set V ik to which the corresponding vector v k belongs. To avoid the presence of multiply classified points or of unclassified patterns in the input space, proper techniques [1] based on linear and quadratic programming can be employed. n this way the s matrices A i for the gate layer are generated; they can include redundant rows that are not necessary in the determination of the polyhedral regions X i. These rows can be removed by applying standard linear programming techniques.
5 y y x a) b) x Figure 3: Simulation results for a benchmark problem: a) unknown piecewise linear function f and training set S, b) function realized by the trained neural network (dashed line). Finally, weight vectors w i for the neural network in Fig. 1 can be directly obtained by solve s linear regression problems (Step 4) having as training sets the samples (x, y) S with x X i, where X 1,... X s are the regions built by the classification process. 4 Simulation results The proposed algorithm for piecewise linear regression has been tested on a one-dimensional benchmark problem, in order to analyze the quality of the resulting neural network. The unknown function to be reconstructed is the following x if 4 x 0 f(x) = x if 0 < x < 2 (2) 2 + 3x if 2 x 4 with X = [ 4, 4] and s = 3. A training set S containing m = 100 samples (x, y) has been generated, where y = f(x)+ε and ε is a normal random variable with zero mean and variance σ 2 = The behavior of f(x) together with the elements of S are depicted in Fig. 3a. The method described in Fig. 2 has been applied by choosing at Step 1 the value c = 6. At Step 2 the number s of regions has been supposed to be known, thus allowing the application of the K-means clustering algorithm [5]; a proper definition of norm has been employed to improve the convergence of the clustering process [6]. ulticategory classification (Step 3) has then been performed by using the method described in [1], which can be easily extended to realize nonlinear boundaries among the X i when treating a multidimensional
6 problem. Finally, least square estimation is adopted to generate vectors w i for piecewise linear regression. The resulting neural network realizes the following function, represented as a dashed line in Fig. 3b: f(x) = x if 4 x x if 0.24 < x < x if 2.12 x 4 As one can note, this is a good approximation to the unknown function (2). Errors can only be detected at the boundaries between two adjacent regions X i ; they are mainly due to the effect of mixed sets C k on the classification process. References [1] E. J. Bredensteiner and K. P. Bennett, ulticategory classification by support vector machines. Computational Optimizations and Applications, 12 (1999) [2] V. Cherkassky and H. Lari-Najafi, Constrained topological mapping for nonparametric regression analysis. Neural Networks, 4 (1991) [3] C.-H. Choi and J. Y. Choi, Constructive neural networks with piecewise interpolation capabilities for function approximation. EEE Transactions on Neural Networks, 5 (1994) [4] J. Y. Choi and J. A. Farrell, Nonlinear adaptive control using networks of piecewise linear approximators. EEE Transactions on Neural Networks, 11 (2000) [5] R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. (1973) New York: John Wiley and Sons. [6] G. Ferrari-Trecate,. uselli, D. Liberati, and. orari, A Clustering Technique for the dentification of Piecewise Affine Systems. Accepted at the Fourth nternational Workshop on Hybrid Systems: Computation and Control, Roma, taly, arch 28-30, [7] B. Fritzke, A growing neural gas network learns topologies. n Advances in Neural nformation Processing Systems 7 (1995) Cambridge, A: T Press, [8] K. Nakayama, A. Hirano, and A. Kanbe, A structure trainable neural network with embedded gating units and its learning algorithm. n Proceedings of the nternational Joint Conference on Neural Networks (2000) Como, taly,
A New Learning Method for Piecewise Linear Regression
A New Learning ethod for Piecewise Linear Regression Giancarlo Ferrari-Trecate and arco uselli 2 INRIA, Domaine de Voluceau Rocquencourt - B.P.5, 7853 Le Chesnay Cedex, France Giancarlo.Ferrari-Trecate@inria.fr
More informationTraining Digital Circuits with Hamming Clustering
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 47, NO. 4, APRIL 2000 513 Training Digital Circuits with Hamming Clustering Marco Muselli, Member, IEEE, and Diego
More informationA Clustering Technique for the Identification of Piecewise Affine Systems
A Clustering Technique for the Identification of Piecewise Affine Systems Giancarlo Ferrari-Trecate 1, Marco Muselli 2, Diego Liberati 2, and Manfred Morari 1 1 Institut für Automatik, ETH - Swiss Federal
More information4.12 Generalization. In back-propagation learning, as many training examples as possible are typically used.
1 4.12 Generalization In back-propagation learning, as many training examples as possible are typically used. It is hoped that the network so designed generalizes well. A network generalizes well when
More informationFigure (5) Kohonen Self-Organized Map
2- KOHONEN SELF-ORGANIZING MAPS (SOM) - The self-organizing neural networks assume a topological structure among the cluster units. - There are m cluster units, arranged in a one- or two-dimensional array;
More informationPATTERN CLASSIFICATION AND SCENE ANALYSIS
PATTERN CLASSIFICATION AND SCENE ANALYSIS RICHARD O. DUDA PETER E. HART Stanford Research Institute, Menlo Park, California A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS New York Chichester Brisbane
More informationIEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 6, NOVEMBER Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 6, NOVEMBER 1999 1271 Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming Bao-Liang Lu, Member, IEEE, Hajime Kita, and Yoshikazu
More informationNatural Language Processing CS 6320 Lecture 6 Neural Language Models. Instructor: Sanda Harabagiu
Natural Language Processing CS 6320 Lecture 6 Neural Language Models Instructor: Sanda Harabagiu In this lecture We shall cover: Deep Neural Models for Natural Language Processing Introduce Feed Forward
More informationLearning to Learn: additional notes
MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science 6.034 Artificial Intelligence, Fall 2008 Recitation October 23 Learning to Learn: additional notes Bob Berwick
More informationAssignment 2. Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions
ENEE 739Q: STATISTICAL AND NEURAL PATTERN RECOGNITION Spring 2002 Assignment 2 Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions Aravind Sundaresan
More information3 Nonlinear Regression
CSC 4 / CSC D / CSC C 3 Sometimes linear models are not sufficient to capture the real-world phenomena, and thus nonlinear models are necessary. In regression, all such models will have the same basic
More informationAn EM-based Ensemble Learning Algorithm on Piecewise Surface Regression Problem
An EM-based Ensemble Learning Algorithm on Piecewise Surface Regression Problem Juan Luo, Alexander Brodsky and Yuan Li Department of Computer Science George Mason University Fairfax, VA 22030, U.S.A jluo2@gmu.edu,
More informationCHAPTER IX Radial Basis Function Networks
CHAPTER IX Radial Basis Function Networks Radial basis function (RBF) networks are feed-forward networks trained using a supervised training algorithm. They are typically configured with a single hidden
More information4. Feedforward neural networks. 4.1 Feedforward neural network structure
4. Feedforward neural networks 4.1 Feedforward neural network structure Feedforward neural network is one of the most common network architectures. Its structure and some basic preprocessing issues required
More informationData Mining: Concepts and Techniques. Chapter 9 Classification: Support Vector Machines. Support Vector Machines (SVMs)
Data Mining: Concepts and Techniques Chapter 9 Classification: Support Vector Machines 1 Support Vector Machines (SVMs) SVMs are a set of related supervised learning methods used for classification Based
More informationIMPLEMENTATION OF RBF TYPE NETWORKS BY SIGMOIDAL FEEDFORWARD NEURAL NETWORKS
IMPLEMENTATION OF RBF TYPE NETWORKS BY SIGMOIDAL FEEDFORWARD NEURAL NETWORKS BOGDAN M.WILAMOWSKI University of Wyoming RICHARD C. JAEGER Auburn University ABSTRACT: It is shown that by introducing special
More informationCLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of
More informationCluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1
Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods
More informationArtificial Neural Networks Lecture Notes Part 5. Stephen Lucci, PhD. Part 5
Artificial Neural Networks Lecture Notes Part 5 About this file: If you have trouble reading the contents of this file, or in case of transcription errors, email gi0062@bcmail.brooklyn.cuny.edu Acknowledgments:
More informationThis leads to our algorithm which is outlined in Section III, along with a tabular summary of it's performance on several benchmarks. The last section
An Algorithm for Incremental Construction of Feedforward Networks of Threshold Units with Real Valued Inputs Dhananjay S. Phatak Electrical Engineering Department State University of New York, Binghamton,
More informationData mining with Support Vector Machine
Data mining with Support Vector Machine Ms. Arti Patle IES, IPS Academy Indore (M.P.) artipatle@gmail.com Mr. Deepak Singh Chouhan IES, IPS Academy Indore (M.P.) deepak.schouhan@yahoo.com Abstract: Machine
More informationLecture 3: Linear Classification
Lecture 3: Linear Classification Roger Grosse 1 Introduction Last week, we saw an example of a learning task called regression. There, the goal was to predict a scalar-valued target from a set of features.
More informationSupervised vs unsupervised clustering
Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful
More informationPattern Classification Algorithms for Face Recognition
Chapter 7 Pattern Classification Algorithms for Face Recognition 7.1 Introduction The best pattern recognizers in most instances are human beings. Yet we do not completely understand how the brain recognize
More informationAnalytical model A structure and process for analyzing a dataset. For example, a decision tree is a model for the classification of a dataset.
Glossary of data mining terms: Accuracy Accuracy is an important factor in assessing the success of data mining. When applied to data, accuracy refers to the rate of correct values in the data. When applied
More informationLinear Models. Lecture Outline: Numeric Prediction: Linear Regression. Linear Classification. The Perceptron. Support Vector Machines
Linear Models Lecture Outline: Numeric Prediction: Linear Regression Linear Classification The Perceptron Support Vector Machines Reading: Chapter 4.6 Witten and Frank, 2nd ed. Chapter 4 of Mitchell Solving
More information3 Nonlinear Regression
3 Linear models are often insufficient to capture the real-world phenomena. That is, the relation between the inputs and the outputs we want to be able to predict are not linear. As a consequence, nonlinear
More informationVisual object classification by sparse convolutional neural networks
Visual object classification by sparse convolutional neural networks Alexander Gepperth 1 1- Ruhr-Universität Bochum - Institute for Neural Dynamics Universitätsstraße 150, 44801 Bochum - Germany Abstract.
More informationClassifier C-Net. 2D Projected Images of 3D Objects. 2D Projected Images of 3D Objects. Model I. Model II
Advances in Neural Information Processing Systems 7. (99) The MIT Press, Cambridge, MA. pp.949-96 Unsupervised Classication of 3D Objects from D Views Satoshi Suzuki Hiroshi Ando ATR Human Information
More informationOperators-Based on Second Derivative double derivative Laplacian operator Laplacian Operator Laplacian Of Gaussian (LOG) Operator LOG
Operators-Based on Second Derivative The principle of edge detection based on double derivative is to detect only those points as edge points which possess local maxima in the gradient values. Laplacian
More informationOptimization Methods for Machine Learning (OMML)
Optimization Methods for Machine Learning (OMML) 2nd lecture Prof. L. Palagi References: 1. Bishop Pattern Recognition and Machine Learning, Springer, 2006 (Chap 1) 2. V. Cherlassky, F. Mulier - Learning
More informationLinear Machine: A Novel Approach to Point Location Problem
Preprints of the 10th IFAC International Symposium on Dynamics and Control of Process Systems The International Federation of Automatic Control Linear Machine: A Novel Approach to Point Location Problem
More information5 Learning hypothesis classes (16 points)
5 Learning hypothesis classes (16 points) Consider a classification problem with two real valued inputs. For each of the following algorithms, specify all of the separators below that it could have generated
More informationGene selection through Switched Neural Networks
Gene selection through Switched Neural Networks Marco Muselli Istituto di Elettronica e di Ingegneria dell Informazione e delle Telecomunicazioni Consiglio Nazionale delle Ricerche Email: Marco.Muselli@ieiit.cnr.it
More informationLecture on Modeling Tools for Clustering & Regression
Lecture on Modeling Tools for Clustering & Regression CS 590.21 Analysis and Modeling of Brain Networks Department of Computer Science University of Crete Data Clustering Overview Organizing data into
More information6. Dicretization methods 6.1 The purpose of discretization
6. Dicretization methods 6.1 The purpose of discretization Often data are given in the form of continuous values. If their number is huge, model building for such data can be difficult. Moreover, many
More informationSupport Vector Machines
Support Vector Machines RBF-networks Support Vector Machines Good Decision Boundary Optimization Problem Soft margin Hyperplane Non-linear Decision Boundary Kernel-Trick Approximation Accurancy Overtraining
More informationIdentification of the correct hard-scatter vertex at the Large Hadron Collider
Identification of the correct hard-scatter vertex at the Large Hadron Collider Pratik Kumar, Neel Mani Singh pratikk@stanford.edu, neelmani@stanford.edu Under the guidance of Prof. Ariel Schwartzman( sch@slac.stanford.edu
More informationModel learning for robot control: a survey
Model learning for robot control: a survey Duy Nguyen-Tuong, Jan Peters 2011 Presented by Evan Beachly 1 Motivation Robots that can learn how their motors move their body Complexity Unanticipated Environments
More informationPattern recognition methods for electronic tongue systems. Patrycja Ciosek
Pattern recognition methods for electronic tongue systems 1 Electronic tongue A SYSTEM FOR AUTOMATIC ANALYSIS AND CLASSIFICATION (RECOGNITION) OF LIQUID SAMPLES AN ARRAY OF CHEMICAL SENSORS PATTERN RECOGNITION
More informationCS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Image Data: Classification via Neural Networks Instructor: Yizhou Sun yzsun@ccs.neu.edu November 19, 2015 Methods to Learn Classification Clustering Frequent Pattern Mining
More informationMultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A
MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A. 205-206 Pietro Guccione, PhD DEI - DIPARTIMENTO DI INGEGNERIA ELETTRICA E DELL INFORMAZIONE POLITECNICO DI BARI
More informationCS6375: Machine Learning Gautam Kunapuli. Mid-Term Review
Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes
More informationChapter 15 Introduction to Linear Programming
Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2015 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of
More informationHybrid PSO-SA algorithm for training a Neural Network for Classification
Hybrid PSO-SA algorithm for training a Neural Network for Classification Sriram G. Sanjeevi 1, A. Naga Nikhila 2,Thaseem Khan 3 and G. Sumathi 4 1 Associate Professor, Dept. of CSE, National Institute
More informationUnsupervised learning in Vision
Chapter 7 Unsupervised learning in Vision The fields of Computer Vision and Machine Learning complement each other in a very natural way: the aim of the former is to extract useful information from visual
More informationNotes on Robust Estimation David J. Fleet Allan Jepson March 30, 005 Robust Estimataion. The field of robust statistics [3, 4] is concerned with estimation problems in which the data contains gross errors,
More informationIntroduction to machine learning, pattern recognition and statistical data modelling Coryn Bailer-Jones
Introduction to machine learning, pattern recognition and statistical data modelling Coryn Bailer-Jones What is machine learning? Data interpretation describing relationship between predictors and responses
More informationHomework. Gaussian, Bishop 2.3 Non-parametric, Bishop 2.5 Linear regression Pod-cast lecture on-line. Next lectures:
Homework Gaussian, Bishop 2.3 Non-parametric, Bishop 2.5 Linear regression 3.0-3.2 Pod-cast lecture on-line Next lectures: I posted a rough plan. It is flexible though so please come with suggestions Bayes
More informationBumptrees for Efficient Function, Constraint, and Classification Learning
umptrees for Efficient Function, Constraint, and Classification Learning Stephen M. Omohundro International Computer Science Institute 1947 Center Street, Suite 600 erkeley, California 94704 Abstract A
More information9.1. K-means Clustering
424 9. MIXTURE MODELS AND EM Section 9.2 Section 9.3 Section 9.4 view of mixture distributions in which the discrete latent variables can be interpreted as defining assignments of data points to specific
More informationLearning and Generalization in Single Layer Perceptrons
Learning and Generalization in Single Layer Perceptrons Neural Computation : Lecture 4 John A. Bullinaria, 2015 1. What Can Perceptrons do? 2. Decision Boundaries The Two Dimensional Case 3. Decision Boundaries
More informationMachine Learning in Biology
Università degli studi di Padova Machine Learning in Biology Luca Silvestrin (Dottorando, XXIII ciclo) Supervised learning Contents Class-conditional probability density Linear and quadratic discriminant
More informationMachine Learning Classifiers and Boosting
Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Naïve
More informationCS 229 Midterm Review
CS 229 Midterm Review Course Staff Fall 2018 11/2/2018 Outline Today: SVMs Kernels Tree Ensembles EM Algorithm / Mixture Models [ Focus on building intuition, less so on solving specific problems. Ask
More informationProcessing Missing Values with Self-Organized Maps
Processing Missing Values with Self-Organized Maps David Sommer, Tobias Grimm, Martin Golz University of Applied Sciences Schmalkalden Department of Computer Science D-98574 Schmalkalden, Germany Phone:
More informationInterpolation is a basic tool used extensively in tasks such as zooming, shrinking, rotating, and geometric corrections.
Image Interpolation 48 Interpolation is a basic tool used extensively in tasks such as zooming, shrinking, rotating, and geometric corrections. Fundamentally, interpolation is the process of using known
More informationFunction approximation using RBF network. 10 basis functions and 25 data points.
1 Function approximation using RBF network F (x j ) = m 1 w i ϕ( x j t i ) i=1 j = 1... N, m 1 = 10, N = 25 10 basis functions and 25 data points. Basis function centers are plotted with circles and data
More informationUnsupervised Learning : Clustering
Unsupervised Learning : Clustering Things to be Addressed Traditional Learning Models. Cluster Analysis K-means Clustering Algorithm Drawbacks of traditional clustering algorithms. Clustering as a complex
More informationUse of Multi-category Proximal SVM for Data Set Reduction
Use of Multi-category Proximal SVM for Data Set Reduction S.V.N Vishwanathan and M Narasimha Murty Department of Computer Science and Automation, Indian Institute of Science, Bangalore 560 012, India Abstract.
More informationThe Detection of Faces in Color Images: EE368 Project Report
The Detection of Faces in Color Images: EE368 Project Report Angela Chau, Ezinne Oji, Jeff Walters Dept. of Electrical Engineering Stanford University Stanford, CA 9435 angichau,ezinne,jwalt@stanford.edu
More informationSYDE Winter 2011 Introduction to Pattern Recognition. Clustering
SYDE 372 - Winter 2011 Introduction to Pattern Recognition Clustering Alexander Wong Department of Systems Design Engineering University of Waterloo Outline 1 2 3 4 5 All the approaches we have learned
More informationLocal Linear Approximation for Kernel Methods: The Railway Kernel
Local Linear Approximation for Kernel Methods: The Railway Kernel Alberto Muñoz 1,JavierGonzález 1, and Isaac Martín de Diego 1 University Carlos III de Madrid, c/ Madrid 16, 890 Getafe, Spain {alberto.munoz,
More information6. NEURAL NETWORK BASED PATH PLANNING ALGORITHM 6.1 INTRODUCTION
6 NEURAL NETWORK BASED PATH PLANNING ALGORITHM 61 INTRODUCTION In previous chapters path planning algorithms such as trigonometry based path planning algorithm and direction based path planning algorithm
More informationDistribution-free Predictive Approaches
Distribution-free Predictive Approaches The methods discussed in the previous sections are essentially model-based. Model-free approaches such as tree-based classification also exist and are popular for
More informationSPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES. Yuanqing Li, Andrzej Cichocki and Shun-ichi Amari
SPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES Yuanqing Li, Andrzej Cichocki and Shun-ichi Amari Laboratory for Advanced Brain Signal Processing Laboratory for Mathematical
More informationSegmentation of Images
Segmentation of Images SEGMENTATION If an image has been preprocessed appropriately to remove noise and artifacts, segmentation is often the key step in interpreting the image. Image segmentation is a
More informationCOMPUTATIONAL INTELLIGENCE SEW (INTRODUCTION TO MACHINE LEARNING) SS18. Lecture 6: k-nn Cross-validation Regularization
COMPUTATIONAL INTELLIGENCE SEW (INTRODUCTION TO MACHINE LEARNING) SS18 Lecture 6: k-nn Cross-validation Regularization LEARNING METHODS Lazy vs eager learning Eager learning generalizes training data before
More informationPractice EXAM: SPRING 2012 CS 6375 INSTRUCTOR: VIBHAV GOGATE
Practice EXAM: SPRING 0 CS 6375 INSTRUCTOR: VIBHAV GOGATE The exam is closed book. You are allowed four pages of double sided cheat sheets. Answer the questions in the spaces provided on the question sheets.
More informationMachine Learning Lecture 3
Many slides adapted from B. Schiele Machine Learning Lecture 3 Probability Density Estimation II 26.04.2016 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Course
More informationRadial Basis Function Networks: Algorithms
Radial Basis Function Networks: Algorithms Neural Computation : Lecture 14 John A. Bullinaria, 2015 1. The RBF Mapping 2. The RBF Network Architecture 3. Computational Power of RBF Networks 4. Training
More informationVoronoi Region. K-means method for Signal Compression: Vector Quantization. Compression Formula 11/20/2013
Voronoi Region K-means method for Signal Compression: Vector Quantization Blocks of signals: A sequence of audio. A block of image pixels. Formally: vector example: (0.2, 0.3, 0.5, 0.1) A vector quantizer
More informationMachine Learning Lecture 3
Course Outline Machine Learning Lecture 3 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Probability Density Estimation II 26.04.206 Discriminative Approaches (5 weeks) Linear
More informationSupport Vector Machines
Support Vector Machines RBF-networks Support Vector Machines Good Decision Boundary Optimization Problem Soft margin Hyperplane Non-linear Decision Boundary Kernel-Trick Approximation Accurancy Overtraining
More informationTime Series Prediction as a Problem of Missing Values: Application to ESTSP2007 and NN3 Competition Benchmarks
Series Prediction as a Problem of Missing Values: Application to ESTSP7 and NN3 Competition Benchmarks Antti Sorjamaa and Amaury Lendasse Abstract In this paper, time series prediction is considered as
More informationTopics in Machine Learning
Topics in Machine Learning Gilad Lerman School of Mathematics University of Minnesota Text/slides stolen from G. James, D. Witten, T. Hastie, R. Tibshirani and A. Ng Machine Learning - Motivation Arthur
More informationIntroduction to ANSYS DesignXplorer
Lecture 4 14. 5 Release Introduction to ANSYS DesignXplorer 1 2013 ANSYS, Inc. September 27, 2013 s are functions of different nature where the output parameters are described in terms of the input parameters
More informationGlobal Journal of Engineering Science and Research Management
A NOVEL HYBRID APPROACH FOR PREDICTION OF MISSING VALUES IN NUMERIC DATASET V.B.Kamble* 1, S.N.Deshmukh 2 * 1 Department of Computer Science and Engineering, P.E.S. College of Engineering, Aurangabad.
More informationRobust Signal-Structure Reconstruction
Robust Signal-Structure Reconstruction V. Chetty 1, D. Hayden 2, J. Gonçalves 2, and S. Warnick 1 1 Information and Decision Algorithms Laboratories, Brigham Young University 2 Control Group, Department
More informationWe use non-bold capital letters for all random variables in these notes, whether they are scalar-, vector-, matrix-, or whatever-valued.
The Bayes Classifier We have been starting to look at the supervised classification problem: we are given data (x i, y i ) for i = 1,..., n, where x i R d, and y i {1,..., K}. In this section, we suppose
More informationMATLAB representation of neural network Outline Neural network with single-layer of neurons. Neural network with multiple-layer of neurons.
MATLAB representation of neural network Outline Neural network with single-layer of neurons. Neural network with multiple-layer of neurons. Introduction: Neural Network topologies (Typical Architectures)
More informationExample Lecture 12: The Stiffness Method Prismatic Beams. Consider again the two span beam previously discussed and determine
Example 1.1 Consider again the two span beam previously discussed and determine The shearing force M1 at end B of member B. The bending moment M at end B of member B. The shearing force M3 at end B of
More informationNeural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer
More informationCharacter Recognition Using Convolutional Neural Networks
Character Recognition Using Convolutional Neural Networks David Bouchain Seminar Statistical Learning Theory University of Ulm, Germany Institute for Neural Information Processing Winter 2006/2007 Abstract
More informationApplying Supervised Learning
Applying Supervised Learning When to Consider Supervised Learning A supervised learning algorithm takes a known set of input data (the training set) and known responses to the data (output), and trains
More informationThe Facet-to-Facet Property of Solutions to Convex Parametric Quadratic Programs and a new Exploration Strategy
The Facet-to-Facet Property of Solutions to Convex Parametric Quadratic Programs and a new Exploration Strategy Jørgen Spjøtvold, Eric C. Kerrigan, Colin N. Jones, Petter Tøndel and Tor A. Johansen Abstract
More informationSupervised Learning with Neural Networks. We now look at how an agent might learn to solve a general problem by seeing examples.
Supervised Learning with Neural Networks We now look at how an agent might learn to solve a general problem by seeing examples. Aims: to present an outline of supervised learning as part of AI; to introduce
More informationSemi-Supervised Clustering with Partial Background Information
Semi-Supervised Clustering with Partial Background Information Jing Gao Pang-Ning Tan Haibin Cheng Abstract Incorporating background knowledge into unsupervised clustering algorithms has been the subject
More informationRecap: Gaussian (or Normal) Distribution. Recap: Minimizing the Expected Loss. Topics of This Lecture. Recap: Maximum Likelihood Approach
Truth Course Outline Machine Learning Lecture 3 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Probability Density Estimation II 2.04.205 Discriminative Approaches (5 weeks)
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationLeave-One-Out Support Vector Machines
Leave-One-Out Support Vector Machines Jason Weston Department of Computer Science Royal Holloway, University of London, Egham Hill, Egham, Surrey, TW20 OEX, UK. Abstract We present a new learning algorithm
More informationNeural Network based textural labeling of images in multimedia applications
Neural Network based textural labeling of images in multimedia applications S.A. Karkanis +, G.D. Magoulas +, and D.A. Karras ++ + University of Athens, Dept. of Informatics, Typa Build., Panepistimiopolis,
More informationClustering: Classic Methods and Modern Views
Clustering: Classic Methods and Modern Views Marina Meilă University of Washington mmp@stat.washington.edu June 22, 2015 Lorentz Center Workshop on Clusters, Games and Axioms Outline Paradigms for clustering
More informationUnit V. Neural Fuzzy System
Unit V Neural Fuzzy System 1 Fuzzy Set In the classical set, its characteristic function assigns a value of either 1 or 0 to each individual in the universal set, There by discriminating between members
More informationMachine Learning (CSE 446): Perceptron
Machine Learning (CSE 446): Perceptron Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 14 Announcements HW due this week. See detailed instructions in the hw. One pdf file.
More informationV4 Matrix algorithms and graph partitioning
V4 Matrix algorithms and graph partitioning - Community detection - Simple modularity maximization - Spectral modularity maximization - Division into more than two groups - Other algorithms for community
More informationClassification and Regression using Linear Networks, Multilayer Perceptrons and Radial Basis Functions
ENEE 739Q SPRING 2002 COURSE ASSIGNMENT 2 REPORT 1 Classification and Regression using Linear Networks, Multilayer Perceptrons and Radial Basis Functions Vikas Chandrakant Raykar Abstract The aim of the
More informationCIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, :59pm, PDF to Canvas [100 points]
CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, 2015. 11:59pm, PDF to Canvas [100 points] Instructions. Please write up your responses to the following problems clearly and concisely.
More informationLecture notes on the simplex method September We will present an algorithm to solve linear programs of the form. maximize.
Cornell University, Fall 2017 CS 6820: Algorithms Lecture notes on the simplex method September 2017 1 The Simplex Method We will present an algorithm to solve linear programs of the form maximize subject
More informationExtract an Essential Skeleton of a Character as a Graph from a Character Image
Extract an Essential Skeleton of a Character as a Graph from a Character Image Kazuhisa Fujita University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo, 182-8585 Japan k-z@nerve.pc.uec.ac.jp
More information