The Pre-Image Problem in Kernel Methods

Size: px
Start display at page:

Download "The Pre-Image Problem in Kernel Methods"

Transcription

1 The Pre-Image Problem in Kernel Methods James Kwok Ivor Tsang Department of Computer Science Hong Kong University of Science and Technology Hong Kong The Pre-Image Problem in Kernel Methods ICML

2 Outline Introduction Proposed Method Experimental Results Conclusion The Pre-Image Problem in Kernel Methods ICML

3 Kernel Principal Component Analysis Kernel PCA: linear PCA in the feature space Given {x 1,..., x N }, construct K = [K ij ] K = UΛU (assume that {ϕ(x 1 ),..., ϕ(x N )} has been centered) U = [α 1,..., α N ] with α i = [α i1,..., α in ] Λ = diag(λ 1,..., λ N ) kth orthonormal eigenvector: V k = N i=1 (α ki/ λ k )ϕ(x i ) Projection of ϕ(x) onto V k : β k = ϕ(x) V k Projection P ϕ(x) of ϕ(x): P ϕ(x) = K k=1 β kv k The Pre-Image Problem in Kernel Methods ICML

4 The Pre-Image Problem Given noisy x, can we recover an ˆx such that ϕ(ˆx) = P ϕ(x)? Non-trivial as ϕ is not invertible in general Moreover, the exact pre-image may not exist Only an approximate solution can be obtained The Pre-Image Problem in Kernel Methods ICML

5 Solution of Mika et al. (NIPS11) Minimize the feature-space distance ϕ(ˆx) P ϕ(x) nonlinear optimization iteration scheme for Gaussian kernels k(x, y) = exp( x y 2 /c): ˆx t+1 = N i=1 γ i exp( ˆx t x i 2 /c)x i N i=1 γ i exp( ˆx t x i 2 /c), γ i = K β k α ki k=1 Problems: numerical instability local minimum The Pre-Image Problem in Kernel Methods ICML

6 Basic Idea Neighbors are most important in determining the location find the distances between P ϕ(x) and its neighboring ϕ(x i ) s Obtain the corresponding input-space distances Use these distances to constrain the location of the pre-image The Pre-Image Problem in Kernel Methods ICML

7 Input- and Feature-Space Distances Feature-space distance between P ϕ(x) and its neighbor ϕ(x i ) d 2 (P ϕ(x), ϕ(x i )) = P ϕ(x) ϕ(x i ) 2 = P ϕ(x) 2 + ϕ(x i ) 2 2P ϕ(x) ϕ(x i ) recall that P ϕ(x) = K k=1 β kv k d 2 (P ϕ(x), ϕ(x i )) can be computed based on K Obtain the corresponding input-space distance d ij The Pre-Image Problem in Kernel Methods ICML

8 From d ij to d ij Isotropic kernels: k(x i, x j ) = κ( x i x j 2 ) d 2 ij = k(x i, x i ) + k(x j, x j ) 2k(x i, x j ) = K ii + K jj 2κ( x i x j 2 ) = K ii + K jj 2κ(d 2 ij) κ(d 2 ij) = 1 2 (K ii + K jj d 2 ij) typically, κ is invertible example: Gaussian kernel κ(z) = exp( βz) d 2 ij = 1 β log(1 2 (K ii + K jj d 2 ij)) The Pre-Image Problem in Kernel Methods ICML

9 Dot product kernels: k(x i, x j ) = κ(x i x j) K ij = k(x i, x j ) = κ(x ix j ) = κ(s ij ) κ is often invertible example: polynomial kernel κ(z) = z p s ij = K 1 p ij when p is odd corresponding input-space distance: d 2 ij = s2 ii + s2 jj 2s ij The Pre-Image Problem in Kernel Methods ICML

10 Back to Input Space Assume that the pre-image ˆx is in the span of the x i s Mika et al. also obtains the same assumption Maintain the distances between ˆx and its neighbors (in the leastsquare sense) similar to the ideas in classical scaling and metric multidimensional scaling (MDS) reduces to finding the least-square solution for a system of linear equations The Pre-Image Problem in Kernel Methods ICML

11 Details After kernel PCA, we obtain P ϕ(x) Identify the n neighbors (ϕ(x 1 ),..., ϕ(x n )) of P ϕ(x) X d n = [x 1, x 2,..., x n ] (input space) assume that they span a q-dimensional space (rank(x) = q) Obtain the desired input-space distances between the unknown pre-image and these neighbors d 2 = [d 2 (x, x 1 ), d 2 (x, x 2 ),..., d 2 (x, x n )] Obtain a basis for these centered neighbors center these neighbors: HX H is the centering matrix I 1 n 11 The Pre-Image Problem in Kernel Methods ICML

12 use SVD on (HX ) : XH = UΛV = UZ U d q = [e 1,..., e q ] (e i s orthonormal) Z q n = [z 1,..., z n ] (z i is the projections of x i onto e j s) distances of x i s to the centroid: d 2 0 = [ z 1 2,..., z n 2 ] embed the desired pre-image ˆx in the span of these neighbors d 2 (ˆx, x i ) d 2 i, i = 1,..., n. if exact pre-image exists, distances exactly preserved least-square solution ẑ = 1 2 Λ 1 V (d 2 d 2 0) expressed in terms of the coordinate system defined by the e j s transform back to the original coordinate system in the input space ˆx = Uẑ + x The Pre-Image Problem in Kernel Methods ICML

13 Experiment: Pre-Images in Kernel PCA images from USPS dataset, 100 test images ten neighbors are used in locating the pre-image Gaussian noise (σ 2 = 0.25, 0.3, 0.4, 0.5) similar results on salt-and-pepper noise The Pre-Image Problem in Kernel Methods ICML

14 Results (Gaussian kernel) noisy image; (300 training images) Mika et al. ; proposed method; (60 training images) Mika et al. ; proposed method number of σ 2 SNR training images noisy images our method Mika et al The Pre-Image Problem in Kernel Methods ICML

15 Experiment on Polynomial Kernel (Order=3) Mika et al. we derive a similar iteration scheme as for Gaussian kernels however, the iteration does not converge Proposed method Noisy image; 300 training images; 60 training images The Pre-Image Problem in Kernel Methods ICML

16 number of training images σ 2 SNR from our method The Pre-Image Problem in Kernel Methods ICML

17 Pre-Images in Kernel k-means Clustering Kernelized version of k-means clustering 3000 images randomly selected from the USPS data set k = 10 (number of digits) Gaussian kernel Use the proposed method to find pre-images of cluster centroids For comparison, also obtain cluster centroids by averaging in the input space over all patterns belonging to the same cluster The Pre-Image Problem in Kernel Methods ICML

18 Results Input space averaging Proposed method yields centroids that are visually more appealing The Pre-Image Problem in Kernel Methods ICML

19 Conclusion Finding the pre-image of a feature vector in the kernel-induced feature space Advantages: non-iterative involves only linear algebra does not suffer from numerical instabilities or the local minimum problem Can be applied equally well to both isotropic kernels and dot product kernels Experimental results show significant improvements over the traditional method The Pre-Image Problem in Kernel Methods ICML

20 Future Directions Can be applied to other kernel applications that also require computation of the pre-images Extension to other classes of kernels only addressed isotropic kernels and dot product kernels in this paper kernel dependency estimation: pre-images for structured objects The Pre-Image Problem in Kernel Methods ICML

SOLVING THE PRE-IMAGE PROBLEM IN KERNEL MACHINES: A DIRECT METHOD. Paul Honeine, Cédric Richard

SOLVING THE PRE-IMAGE PROBLEM IN KERNEL MACHINES: A DIRECT METHOD. Paul Honeine, Cédric Richard SOLVING THE PRE-IMAGE PROBLEM IN KERNEL MACHINES: A DIRECT METHOD Paul Honeine, Cédric Richard Institut Charles Delaunay (FRE CNRS 2848), LM2S, Université de technologie de Troyes, Troyes, France ABSTRACT

More information

The Pre-Image Problem and Kernel PCA for Speech Enhancement

The Pre-Image Problem and Kernel PCA for Speech Enhancement The Pre-Image Problem and Kernel PCA for Speech Enhancement Christina Leitner and Franz Pernkopf Signal Processing and Speech Communication Laboratory, Graz University of Technology, Inffeldgasse 6c, 8

More information

Large-Scale Face Manifold Learning

Large-Scale Face Manifold Learning Large-Scale Face Manifold Learning Sanjiv Kumar Google Research New York, NY * Joint work with A. Talwalkar, H. Rowley and M. Mohri 1 Face Manifold Learning 50 x 50 pixel faces R 2500 50 x 50 pixel random

More information

Unsupervised Learning

Unsupervised Learning Unsupervised Learning Learning without Class Labels (or correct outputs) Density Estimation Learn P(X) given training data for X Clustering Partition data into clusters Dimensionality Reduction Discover

More information

Iteratively Re-weighted Least Squares for Sums of Convex Functions

Iteratively Re-weighted Least Squares for Sums of Convex Functions Iteratively Re-weighted Least Squares for Sums of Convex Functions James Burke University of Washington Jiashan Wang LinkedIn Frank Curtis Lehigh University Hao Wang Shanghai Tech University Daiwei He

More information

Kernel Principal Component Analysis: Applications and Implementation

Kernel Principal Component Analysis: Applications and Implementation Kernel Principal Component Analysis: Applications and Daniel Olsson Royal Institute of Technology Stockholm, Sweden Examiner: Prof. Ulf Jönsson Supervisor: Prof. Pando Georgiev Master s Thesis Presentation

More information

Unsupervised Learning

Unsupervised Learning Networks for Pattern Recognition, 2014 Networks for Single Linkage K-Means Soft DBSCAN PCA Networks for Kohonen Maps Linear Vector Quantization Networks for Problems/Approaches in Machine Learning Supervised

More information

Divide and Conquer Kernel Ridge Regression

Divide and Conquer Kernel Ridge Regression Divide and Conquer Kernel Ridge Regression Yuchen Zhang John Duchi Martin Wainwright University of California, Berkeley COLT 2013 Yuchen Zhang (UC Berkeley) Divide and Conquer KRR COLT 2013 1 / 15 Problem

More information

Kernel Methods in Machine Learning

Kernel Methods in Machine Learning Outline Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong Joint work with Ivor Tsang, Pakming Cheung, Andras Kocsor, Jacek Zurada, Kimo Lai November

More information

Robust Kernel Methods in Clustering and Dimensionality Reduction Problems

Robust Kernel Methods in Clustering and Dimensionality Reduction Problems Robust Kernel Methods in Clustering and Dimensionality Reduction Problems Jian Guo, Debadyuti Roy, Jing Wang University of Michigan, Department of Statistics Introduction In this report we propose robust

More information

3D Surface Recovery via Deterministic Annealing based Piecewise Linear Surface Fitting Algorithm

3D Surface Recovery via Deterministic Annealing based Piecewise Linear Surface Fitting Algorithm 3D Surface Recovery via Deterministic Annealing based Piecewise Linear Surface Fitting Algorithm Bing Han, Chris Paulson, and Dapeng Wu Department of Electrical and Computer Engineering University of Florida

More information

General Instructions. Questions

General Instructions. Questions CS246: Mining Massive Data Sets Winter 2018 Problem Set 2 Due 11:59pm February 8, 2018 Only one late period is allowed for this homework (11:59pm 2/13). General Instructions Submission instructions: These

More information

Lecture 19: Convex Non-Smooth Optimization. April 2, 2007

Lecture 19: Convex Non-Smooth Optimization. April 2, 2007 : Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials

More information

Data Analysis 3. Support Vector Machines. Jan Platoš October 30, 2017

Data Analysis 3. Support Vector Machines. Jan Platoš October 30, 2017 Data Analysis 3 Support Vector Machines Jan Platoš October 30, 2017 Department of Computer Science Faculty of Electrical Engineering and Computer Science VŠB - Technical University of Ostrava Table of

More information

Graph Laplacian Kernels for Object Classification from a Single Example

Graph Laplacian Kernels for Object Classification from a Single Example Graph Laplacian Kernels for Object Classification from a Single Example Hong Chang & Dit-Yan Yeung Department of Computer Science, Hong Kong University of Science and Technology {hongch,dyyeung}@cs.ust.hk

More information

Chapter 7: Computation of the Camera Matrix P

Chapter 7: Computation of the Camera Matrix P Chapter 7: Computation of the Camera Matrix P Arco Nederveen Eagle Vision March 18, 2008 Arco Nederveen (Eagle Vision) The Camera Matrix P March 18, 2008 1 / 25 1 Chapter 7: Computation of the camera Matrix

More information

Unsupervised learning in Vision

Unsupervised learning in Vision Chapter 7 Unsupervised learning in Vision The fields of Computer Vision and Machine Learning complement each other in a very natural way: the aim of the former is to extract useful information from visual

More information

ELEG Compressive Sensing and Sparse Signal Representations

ELEG Compressive Sensing and Sparse Signal Representations ELEG 867 - Compressive Sensing and Sparse Signal Representations Gonzalo R. Arce Depart. of Electrical and Computer Engineering University of Delaware Fall 211 Compressive Sensing G. Arce Fall, 211 1 /

More information

COMP 558 lecture 19 Nov. 17, 2010

COMP 558 lecture 19 Nov. 17, 2010 COMP 558 lecture 9 Nov. 7, 2 Camera calibration To estimate the geometry of 3D scenes, it helps to know the camera parameters, both external and internal. The problem of finding all these parameters is

More information

Non-linear dimension reduction

Non-linear dimension reduction Sta306b May 23, 2011 Dimension Reduction: 1 Non-linear dimension reduction ISOMAP: Tenenbaum, de Silva & Langford (2000) Local linear embedding: Roweis & Saul (2000) Local MDS: Chen (2006) all three methods

More information

Diffusion Wavelets for Natural Image Analysis

Diffusion Wavelets for Natural Image Analysis Diffusion Wavelets for Natural Image Analysis Tyrus Berry December 16, 2011 Contents 1 Project Description 2 2 Introduction to Diffusion Wavelets 2 2.1 Diffusion Multiresolution............................

More information

Summarizing Inter-Query Learning in Content-Based Image Retrieval via Incremental Semantic Clustering

Summarizing Inter-Query Learning in Content-Based Image Retrieval via Incremental Semantic Clustering Summarizing Inter-Query Learning in Content-Based Image Retrieval via Incremental Semantic Clustering Iker Gondra, Douglas R. Heisterkamp Department of Computer Science Oklahoma State University Stillwater,

More information

Kernel spectral clustering: model representations, sparsity and out-of-sample extensions

Kernel spectral clustering: model representations, sparsity and out-of-sample extensions Kernel spectral clustering: model representations, sparsity and out-of-sample extensions Johan Suykens and Carlos Alzate K.U. Leuven, ESAT-SCD/SISTA Kasteelpark Arenberg B-3 Leuven (Heverlee), Belgium

More information

Multidimensional scaling Based in part on slides from textbook, slides of Susan Holmes. October 10, Statistics 202: Data Mining

Multidimensional scaling Based in part on slides from textbook, slides of Susan Holmes. October 10, Statistics 202: Data Mining Multidimensional scaling Based in part on slides from textbook, slides of Susan Holmes October 10, 2012 1 / 1 Multidimensional scaling A visual tool Recall the PCA scores were X V = U where X = HX S 1/2

More information

Laplacian Eigenmaps and Bayesian Clustering Based Layout Pattern Sampling and Its Applications to Hotspot Detection and OPC

Laplacian Eigenmaps and Bayesian Clustering Based Layout Pattern Sampling and Its Applications to Hotspot Detection and OPC Laplacian Eigenmaps and Bayesian Clustering Based Layout Pattern Sampling and Its Applications to Hotspot Detection and OPC Tetsuaki Matsunawa 1, Bei Yu 2 and David Z. Pan 3 1 Toshiba Corporation 2 The

More information

Filters. Advanced and Special Topics: Filters. Filters

Filters. Advanced and Special Topics: Filters. Filters Filters Advanced and Special Topics: Filters Dr. Edmund Lam Department of Electrical and Electronic Engineering The University of Hong Kong ELEC4245: Digital Image Processing (Second Semester, 2016 17)

More information

Machine Learning for Signal Processing Clustering. Bhiksha Raj Class Oct 2016

Machine Learning for Signal Processing Clustering. Bhiksha Raj Class Oct 2016 Machine Learning for Signal Processing Clustering Bhiksha Raj Class 11. 13 Oct 2016 1 Statistical Modelling and Latent Structure Much of statistical modelling attempts to identify latent structure in the

More information

Data-Dependent Kernels for High-Dimensional Data Classification

Data-Dependent Kernels for High-Dimensional Data Classification Proceedings of International Joint Conference on Neural Networks, Montreal, Canada, July 31 - August 4, 2005 Data-Dependent Kernels for High-Dimensional Data Classification Jingdong Wang James T. Kwok

More information

Alternative Statistical Methods for Bone Atlas Modelling

Alternative Statistical Methods for Bone Atlas Modelling Alternative Statistical Methods for Bone Atlas Modelling Sharmishtaa Seshamani, Gouthami Chintalapani, Russell Taylor Department of Computer Science, Johns Hopkins University, Baltimore, MD Traditional

More information

Clustering K-means. Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, Carlos Guestrin

Clustering K-means. Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, Carlos Guestrin Clustering K-means Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, 2014 Carlos Guestrin 2005-2014 1 Clustering images Set of Images [Goldberger et al.] Carlos Guestrin 2005-2014

More information

Support Vector Machines

Support Vector Machines Support Vector Machines RBF-networks Support Vector Machines Good Decision Boundary Optimization Problem Soft margin Hyperplane Non-linear Decision Boundary Kernel-Trick Approximation Accurancy Overtraining

More information

Feature scaling in support vector data description

Feature scaling in support vector data description Feature scaling in support vector data description P. Juszczak, D.M.J. Tax, R.P.W. Duin Pattern Recognition Group, Department of Applied Physics, Faculty of Applied Sciences, Delft University of Technology,

More information

Kernels and representation

Kernels and representation Kernels and representation Corso di AA, anno 2017/18, Padova Fabio Aiolli 20 Dicembre 2017 Fabio Aiolli Kernels and representation 20 Dicembre 2017 1 / 19 (Hierarchical) Representation Learning Hierarchical

More information

Clustering: Classic Methods and Modern Views

Clustering: Classic Methods and Modern Views Clustering: Classic Methods and Modern Views Marina Meilă University of Washington mmp@stat.washington.edu June 22, 2015 Lorentz Center Workshop on Clusters, Games and Axioms Outline Paradigms for clustering

More information

Advanced Machine Learning Practical 1: Manifold Learning (PCA and Kernel PCA)

Advanced Machine Learning Practical 1: Manifold Learning (PCA and Kernel PCA) Advanced Machine Learning Practical : Manifold Learning (PCA and Kernel PCA) Professor: Aude Billard Assistants: Nadia Figueroa, Ilaria Lauzana and Brice Platerrier E-mails: aude.billard@epfl.ch, nadia.figueroafernandez@epfl.ch

More information

Self-Organized Similarity based Kernel Fuzzy Clustering Model and Its Applications

Self-Organized Similarity based Kernel Fuzzy Clustering Model and Its Applications Fifth International Workshop on Computational Intelligence & Applications IEEE SMC Hiroshima Chapter, Hiroshima University, Japan, November 10, 11 & 12, 2009 Self-Organized Similarity based Kernel Fuzzy

More information

CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, :59pm, PDF to Canvas [100 points]

CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, :59pm, PDF to Canvas [100 points] CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, 2015. 11:59pm, PDF to Canvas [100 points] Instructions. Please write up your responses to the following problems clearly and concisely.

More information

Multiresponse Sparse Regression with Application to Multidimensional Scaling

Multiresponse Sparse Regression with Application to Multidimensional Scaling Multiresponse Sparse Regression with Application to Multidimensional Scaling Timo Similä and Jarkko Tikka Helsinki University of Technology, Laboratory of Computer and Information Science P.O. Box 54,

More information

SUPPORT VECTOR MACHINES

SUPPORT VECTOR MACHINES SUPPORT VECTOR MACHINES Today Reading AIMA 8.9 (SVMs) Goals Finish Backpropagation Support vector machines Backpropagation. Begin with randomly initialized weights 2. Apply the neural network to each training

More information

Kernels for Structured Data

Kernels for Structured Data T-122.102 Special Course in Information Science VI: Co-occurence methods in analysis of discrete data Kernels for Structured Data Based on article: A Survey of Kernels for Structured Data by Thomas Gärtner

More information

May 1, CODY, Error Backpropagation, Bischop 5.3, and Support Vector Machines (SVM) Bishop Ch 7. May 3, Class HW SVM, PCA, and K-means, Bishop Ch

May 1, CODY, Error Backpropagation, Bischop 5.3, and Support Vector Machines (SVM) Bishop Ch 7. May 3, Class HW SVM, PCA, and K-means, Bishop Ch May 1, CODY, Error Backpropagation, Bischop 5.3, and Support Vector Machines (SVM) Bishop Ch 7. May 3, Class HW SVM, PCA, and K-means, Bishop Ch 12.1, 9.1 May 8, CODY Machine Learning for finding oil,

More information

CS 195-5: Machine Learning Problem Set 5

CS 195-5: Machine Learning Problem Set 5 CS 195-5: Machine Learning Problem Set 5 Douglas Lanman dlanman@brown.edu 26 November 26 1 Clustering and Vector Quantization Problem 1 Part 1: In this problem we will apply Vector Quantization (VQ) to

More information

CSE 547: Machine Learning for Big Data Spring Problem Set 2. Please read the homework submission policies.

CSE 547: Machine Learning for Big Data Spring Problem Set 2. Please read the homework submission policies. CSE 547: Machine Learning for Big Data Spring 2019 Problem Set 2 Please read the homework submission policies. 1 Principal Component Analysis and Reconstruction (25 points) Let s do PCA and reconstruct

More information

A Weighted Kernel PCA Approach to Graph-Based Image Segmentation

A Weighted Kernel PCA Approach to Graph-Based Image Segmentation A Weighted Kernel PCA Approach to Graph-Based Image Segmentation Carlos Alzate Johan A. K. Suykens ESAT-SCD-SISTA Katholieke Universiteit Leuven Leuven, Belgium January 25, 2007 International Conference

More information

COSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor

COSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor COSC160: Detection and Classification Jeremy Bolton, PhD Assistant Teaching Professor Outline I. Problem I. Strategies II. Features for training III. Using spatial information? IV. Reducing dimensionality

More information

Chapter 7: Competitive learning, clustering, and self-organizing maps

Chapter 7: Competitive learning, clustering, and self-organizing maps Chapter 7: Competitive learning, clustering, and self-organizing maps António R. C. Paiva EEL 6814 Spring 2008 Outline Competitive learning Clustering Self-Organizing Maps What is competition in neural

More information

Data Preprocessing. Javier Béjar. URL - Spring 2018 CS - MAI 1/78 BY: $\

Data Preprocessing. Javier Béjar. URL - Spring 2018 CS - MAI 1/78 BY: $\ Data Preprocessing Javier Béjar BY: $\ URL - Spring 2018 C CS - MAI 1/78 Introduction Data representation Unstructured datasets: Examples described by a flat set of attributes: attribute-value matrix Structured

More information

Chap.12 Kernel methods [Book, Chap.7]

Chap.12 Kernel methods [Book, Chap.7] Chap.12 Kernel methods [Book, Chap.7] Neural network methods became popular in the mid to late 1980s, but by the mid to late 1990s, kernel methods have also become popular in machine learning. The first

More information

arxiv: v3 [cs.cv] 31 Aug 2014

arxiv: v3 [cs.cv] 31 Aug 2014 Kernel Principal Component Analysis and its Applications in Face Recognition and Active Shape Models Quan Wang Rensselaer Polytechnic Institute, Eighth Street, Troy, Y 28 USA wangq@rpi.edu arxiv:27.3538v3

More information

Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines

Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2007 c 2007,

More information

Sensor Network Localization Using Sensor Perturbation

Sensor Network Localization Using Sensor Perturbation Sensor Network Localization Using Sensor Perturbation YUANCHEN ZHU and STEVEN J. GORTLER Harvard University and DYLAN THURSTON Columbia University Sensor network localization is an instance of the NP-HARD

More information

Generalized Principal Component Analysis CVPR 2007

Generalized Principal Component Analysis CVPR 2007 Generalized Principal Component Analysis Tutorial @ CVPR 2007 Yi Ma ECE Department University of Illinois Urbana Champaign René Vidal Center for Imaging Science Institute for Computational Medicine Johns

More information

Network Traffic Measurements and Analysis

Network Traffic Measurements and Analysis DEIB - Politecnico di Milano Fall, 2017 Introduction Often, we have only a set of features x = x 1, x 2,, x n, but no associated response y. Therefore we are not interested in prediction nor classification,

More information

Homework 4: Clustering, Recommenders, Dim. Reduction, ML and Graph Mining (due November 19 th, 2014, 2:30pm, in class hard-copy please)

Homework 4: Clustering, Recommenders, Dim. Reduction, ML and Graph Mining (due November 19 th, 2014, 2:30pm, in class hard-copy please) Virginia Tech. Computer Science CS 5614 (Big) Data Management Systems Fall 2014, Prakash Homework 4: Clustering, Recommenders, Dim. Reduction, ML and Graph Mining (due November 19 th, 2014, 2:30pm, in

More information

Kernel Methods. Chapter 9 of A Course in Machine Learning by Hal Daumé III. Conversion to beamer by Fabrizio Riguzzi

Kernel Methods. Chapter 9 of A Course in Machine Learning by Hal Daumé III.   Conversion to beamer by Fabrizio Riguzzi Kernel Methods Chapter 9 of A Course in Machine Learning by Hal Daumé III http://ciml.info Conversion to beamer by Fabrizio Riguzzi Kernel Methods 1 / 66 Kernel Methods Linear models are great because

More information

Connecting the Out-of-Sample and Pre-Image Problems in Kernel Methods

Connecting the Out-of-Sample and Pre-Image Problems in Kernel Methods Connecting the Out-of-Sample and Pre-Image Problems in Kernel Methods Pablo Arias Universidad de la República parias@fing.edu.uy Gregory Randall Universidad de la República randall@fing.edu.uy Guillermo

More information

Local Isomorphism to Solve the Pre-image Problem in Kernel Methods

Local Isomorphism to Solve the Pre-image Problem in Kernel Methods Local Isomorphism to Solve the Pre-image Problem in Kernel Methods Dong Huang 1,2, Yuandong Tian 1 and Fernando De la Torre 1 1 Robotics Institute, Carnegie Mellon University, USA 2 University of Electronic

More information

Non-exhaustive, Overlapping k-means

Non-exhaustive, Overlapping k-means Non-exhaustive, Overlapping k-means J. J. Whang, I. S. Dhilon, and D. F. Gleich Teresa Lebair University of Maryland, Baltimore County October 29th, 2015 Teresa Lebair UMBC 1/38 Outline Introduction NEO-K-Means

More information

Basis Functions. Volker Tresp Summer 2017

Basis Functions. Volker Tresp Summer 2017 Basis Functions Volker Tresp Summer 2017 1 Nonlinear Mappings and Nonlinear Classifiers Regression: Linearity is often a good assumption when many inputs influence the output Some natural laws are (approximately)

More information

Module 4. Non-linear machine learning econometrics: Support Vector Machine

Module 4. Non-linear machine learning econometrics: Support Vector Machine Module 4. Non-linear machine learning econometrics: Support Vector Machine THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Introduction When the assumption of linearity

More information

Machine Learning / Jan 27, 2010

Machine Learning / Jan 27, 2010 Revisiting Logistic Regression & Naïve Bayes Aarti Singh Machine Learning 10-701/15-781 Jan 27, 2010 Generative and Discriminative Classifiers Training classifiers involves learning a mapping f: X -> Y,

More information

A survey of kernel and spectral methods for clustering

A survey of kernel and spectral methods for clustering A survey of kernel and spectral methods for clustering Maurizio Filippone a Francesco Camastra b Francesco Masulli a Stefano Rovetta a a Department of Computer and Information Science, University of Genova,

More information

CS Introduction to Data Mining Instructor: Abdullah Mueen

CS Introduction to Data Mining Instructor: Abdullah Mueen CS 591.03 Introduction to Data Mining Instructor: Abdullah Mueen LECTURE 8: ADVANCED CLUSTERING (FUZZY AND CO -CLUSTERING) Review: Basic Cluster Analysis Methods (Chap. 10) Cluster Analysis: Basic Concepts

More information

A Dendrogram. Bioinformatics (Lec 17)

A Dendrogram. Bioinformatics (Lec 17) A Dendrogram 3/15/05 1 Hierarchical Clustering [Johnson, SC, 1967] Given n points in R d, compute the distance between every pair of points While (not done) Pick closest pair of points s i and s j and

More information

Clustering. Department Biosysteme Karsten Borgwardt Data Mining Course Basel Fall Semester / 238

Clustering. Department Biosysteme Karsten Borgwardt Data Mining Course Basel Fall Semester / 238 Clustering Department Biosysteme Karsten Borgwardt Data Mining Course Basel Fall Semester 2015 163 / 238 What is Clustering? Department Biosysteme Karsten Borgwardt Data Mining Course Basel Fall Semester

More information

Second Order SMO Improves SVM Online and Active Learning

Second Order SMO Improves SVM Online and Active Learning Second Order SMO Improves SVM Online and Active Learning Tobias Glasmachers and Christian Igel Institut für Neuroinformatik, Ruhr-Universität Bochum 4478 Bochum, Germany Abstract Iterative learning algorithms

More information

Feature selection. Term 2011/2012 LSI - FIB. Javier Béjar cbea (LSI - FIB) Feature selection Term 2011/ / 22

Feature selection. Term 2011/2012 LSI - FIB. Javier Béjar cbea (LSI - FIB) Feature selection Term 2011/ / 22 Feature selection Javier Béjar cbea LSI - FIB Term 2011/2012 Javier Béjar cbea (LSI - FIB) Feature selection Term 2011/2012 1 / 22 Outline 1 Dimensionality reduction 2 Projections 3 Attribute selection

More information

Markov Networks in Computer Vision

Markov Networks in Computer Vision Markov Networks in Computer Vision Sargur Srihari srihari@cedar.buffalo.edu 1 Markov Networks for Computer Vision Some applications: 1. Image segmentation 2. Removal of blur/noise 3. Stereo reconstruction

More information

Face Recognition Using Wavelet Based Kernel Locally Discriminating Projection

Face Recognition Using Wavelet Based Kernel Locally Discriminating Projection Face Recognition Using Wavelet Based Kernel Locally Discriminating Projection Venkatrama Phani Kumar S 1, KVK Kishore 2 and K Hemantha Kumar 3 Abstract Locality Preserving Projection(LPP) aims to preserve

More information

Sparse multidimensional scaling for blind tracking in sensor networks

Sparse multidimensional scaling for blind tracking in sensor networks Sparse multidimensional scaling for blind tracking in sensor networks R. Rangarajan 1, R. Raich 2, and A. O. Hero III 3 1 University of Michigan rangaraj@eecs.umich.edu 2 University of Michigan ravivr@eecs.umich.edu

More information

Machine Learning for OR & FE

Machine Learning for OR & FE Machine Learning for OR & FE Unsupervised Learning: Clustering Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com (Some material

More information

Nearest Neighbor with KD Trees

Nearest Neighbor with KD Trees Case Study 2: Document Retrieval Finding Similar Documents Using Nearest Neighbors Machine Learning/Statistics for Big Data CSE599C1/STAT592, University of Washington Emily Fox January 22 nd, 2013 1 Nearest

More information

Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1

Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods

More information

Visualizing pairwise similarity via semidefinite programming

Visualizing pairwise similarity via semidefinite programming Visualizing pairwise similarity via semidefinite programming Amir Globerson Computer Science and Artificial Intelligence Laboratory MIT Cambridge, MA 02139 gamir@csail.mit.edu Sam Roweis Department of

More information

A Weighted Majority Voting based on Normalized Mutual Information for Cluster Analysis

A Weighted Majority Voting based on Normalized Mutual Information for Cluster Analysis A Weighted Majority Voting based on Normalized Mutual Information for Cluster Analysis Meshal Shutaywi and Nezamoddin N. Kachouie Department of Mathematical Sciences, Florida Institute of Technology Abstract

More information

Robust Kernel Principal Component Analysis

Robust Kernel Principal Component Analysis Robust Kernel Principal Component Analysis Minh Hoai Nguyen & Fernando De la Torre Carnegie Mellon University, Pittsburgh, PA 15213, USA. Abstract Kernel Principal Component Analysis (KPCA) is a popular

More information

Markov Networks in Computer Vision. Sargur Srihari

Markov Networks in Computer Vision. Sargur Srihari Markov Networks in Computer Vision Sargur srihari@cedar.buffalo.edu 1 Markov Networks for Computer Vision Important application area for MNs 1. Image segmentation 2. Removal of blur/noise 3. Stereo reconstruction

More information

Concept of Curve Fitting Difference with Interpolation

Concept of Curve Fitting Difference with Interpolation Curve Fitting Content Concept of Curve Fitting Difference with Interpolation Estimation of Linear Parameters by Least Squares Curve Fitting by Polynomial Least Squares Estimation of Non-linear Parameters

More information

Digital Image Processing Laboratory: MAP Image Restoration

Digital Image Processing Laboratory: MAP Image Restoration Purdue University: Digital Image Processing Laboratories 1 Digital Image Processing Laboratory: MAP Image Restoration October, 015 1 Introduction This laboratory explores the use of maximum a posteriori

More information

Adaptive Hash Retrieval with Kernel Based Similarity

Adaptive Hash Retrieval with Kernel Based Similarity Adaptive Hash Retrieval with Kernel Based Similarity Xiao Bai a, Cheng Yan a, Haichuan Yang a, Lu Bai b, Jun Zhou c, Edwin Robert Hancock d a School of Computer Science and Engineering, Beihang University,

More information

Using Machine Learning to Optimize Storage Systems

Using Machine Learning to Optimize Storage Systems Using Machine Learning to Optimize Storage Systems Dr. Kiran Gunnam 1 Outline 1. Overview 2. Building Flash Models using Logistic Regression. 3. Storage Object classification 4. Storage Allocation recommendation

More information

Nonlinear Noise Reduction

Nonlinear Noise Reduction Chapter 3 Nonlinear Noise Reduction Many natural and engineered systems generate nonlinear deterministic time series that are contaminated by random measurement or dynamic noise. Even clean time series

More information

Tracking system. Danica Kragic. Object Recognition & Model Based Tracking

Tracking system. Danica Kragic. Object Recognition & Model Based Tracking Tracking system Object Recognition & Model Based Tracking Motivation Manipulating objects in domestic environments Localization / Navigation Object Recognition Servoing Tracking Grasping Pose estimation

More information

Recent Developments in Model-based Derivative-free Optimization

Recent Developments in Model-based Derivative-free Optimization Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints:

More information

CPSC 340: Machine Learning and Data Mining. Principal Component Analysis Fall 2016

CPSC 340: Machine Learning and Data Mining. Principal Component Analysis Fall 2016 CPSC 340: Machine Learning and Data Mining Principal Component Analysis Fall 2016 A2/Midterm: Admin Grades/solutions will be posted after class. Assignment 4: Posted, due November 14. Extra office hours:

More information

Locally Weighted Least Squares Regression for Image Denoising, Reconstruction and Up-sampling

Locally Weighted Least Squares Regression for Image Denoising, Reconstruction and Up-sampling Locally Weighted Least Squares Regression for Image Denoising, Reconstruction and Up-sampling Moritz Baecher May 15, 29 1 Introduction Edge-preserving smoothing and super-resolution are classic and important

More information

(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17

(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17 Improving Generative Adversarial Networks with Denoising Feature Matching David Warde-Farley 1 Yoshua Bengio 1 1 University of Montreal, ICLR,2017 Presenter: Bargav Jayaraman Outline 1 Introduction 2 Background

More information

Kernel-Based Metric Adaptation with Pairwise Constraints

Kernel-Based Metric Adaptation with Pairwise Constraints Kernel-Based Metric Adaptation with Pairwise Constraints Hong Chang and Dit-Yan Yeung Department of Computer Science Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong {hongch,dyyeung}@cs.ust.hk

More information

Adaptive Sparse Kernel Principal Component Analysis for Computation and Store Space Constrained-based Feature Extraction

Adaptive Sparse Kernel Principal Component Analysis for Computation and Store Space Constrained-based Feature Extraction Journal of Information Hiding and Multimedia Signal Processing c 2015 ISSN 2073-4212 Ubiquitous International Volume 6, Number 4, July 2015 Adaptive Sparse Kernel Principal Component Analysis for Computation

More information

STATS306B STATS306B. Clustering. Jonathan Taylor Department of Statistics Stanford University. June 3, 2010

STATS306B STATS306B. Clustering. Jonathan Taylor Department of Statistics Stanford University. June 3, 2010 STATS306B Jonathan Taylor Department of Statistics Stanford University June 3, 2010 Spring 2010 Outline K-means, K-medoids, EM algorithm choosing number of clusters: Gap test hierarchical clustering spectral

More information

Robust Subspace Computation Using L1 Norm

Robust Subspace Computation Using L1 Norm Robust Subspace Computation Using L1 Norm Qifa Ke and Takeo Kanade August 2003 CMU-CS-03-172 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Abstract Linear subspace has many

More information

Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching

Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching Correlated Correspondences [ASP*04] A Complete Registration System [HAW*08] In this session Advanced Global Matching Some practical

More information

arxiv: v1 [cs.lg] 3 Jan 2018

arxiv: v1 [cs.lg] 3 Jan 2018 CLUSTERING OF DATA WITH MISSING ENTRIES Sunrita Poddar, Mathews Jacob Department of Electrical and Computer Engineering, University of Iowa, IA, USA arxiv:1801.01455v1 [cs.lg] 3 Jan 018 ABSTRACT The analysis

More information

Data Preprocessing. Javier Béjar AMLT /2017 CS - MAI. (CS - MAI) Data Preprocessing AMLT / / 71 BY: $\

Data Preprocessing. Javier Béjar AMLT /2017 CS - MAI. (CS - MAI) Data Preprocessing AMLT / / 71 BY: $\ Data Preprocessing S - MAI AMLT - 2016/2017 (S - MAI) Data Preprocessing AMLT - 2016/2017 1 / 71 Outline 1 Introduction Data Representation 2 Data Preprocessing Outliers Missing Values Normalization Discretization

More information

Model Criticism for Regression on the Grassmannian

Model Criticism for Regression on the Grassmannian Model Criticism for Regression on the Grassmannian Yi Hong, Roland Kwitt 3, and Marc Niethammer,2 University of North Carolina (UNC) at Chapel Hill, NC, US 2 Biomedical Research Imaging Center, UNC-Chapel

More information

Unsupervised Outlier Detection and Semi-Supervised Learning

Unsupervised Outlier Detection and Semi-Supervised Learning Unsupervised Outlier Detection and Semi-Supervised Learning Adam Vinueza Department of Computer Science University of Colorado Boulder, Colorado 832 vinueza@colorado.edu Gregory Z. Grudic Department of

More information

Introduction to Machine Learning

Introduction to Machine Learning Introduction to Machine Learning Isabelle Guyon Notes written by: Johann Leithon. Introduction The process of Machine Learning consist of having a big training data base, which is the input to some learning

More information

Subspace Clustering. Weiwei Feng. December 11, 2015

Subspace Clustering. Weiwei Feng. December 11, 2015 Subspace Clustering Weiwei Feng December 11, 2015 Abstract Data structure analysis is an important basis of machine learning and data science, which is now widely used in computational visualization problems,

More information

Interferogram Analysis using Active Instance-Based Learning

Interferogram Analysis using Active Instance-Based Learning Interferogram Analysis using Active Instance-Based Learning Olac Fuentes and Thamar Solorio Instituto Nacional de Astrofísica, Óptica y Electrónica Luis Enrique Erro 1 Santa María Tonantzintla, Puebla,

More information

CPSC 340: Machine Learning and Data Mining. Kernel Trick Fall 2017

CPSC 340: Machine Learning and Data Mining. Kernel Trick Fall 2017 CPSC 340: Machine Learning and Data Mining Kernel Trick Fall 2017 Admin Assignment 3: Due Friday. Midterm: Can view your exam during instructor office hours or after class this week. Digression: the other

More information