Kernel Principal Component Analysis: Applications and Implementation

Transcription

1 Kernel Principal Component Analysis: Applications and Daniel Olsson Royal Institute of Technology Stockholm, Sweden Examiner: Prof. Ulf Jönsson Supervisor: Prof. Pando Georgiev Master s Thesis Presentation February 4, / 24

2 / 24

3 Project goals Goals of the project: Implement kernel PCA in MATLAB. Apply the implementation to databases (Sugar and Iris data). Improve the method for these datasets. 3 / 24

4 Datamining Common problem Huge amounts of high-dimensional data (e.g. from research) Examples: Genetics, EEG-data, the Internet etc. How do we find meaningful patterns in vast amounts of data? 4 / 24

5 Dimension reduction methods Reduce the dimensionality of the data. Preserve most of the information. Examples: Principal Component Analysis (PCA) Random Projections 5 / 24

6 Kernel Principal Component Analysis Identifies patterns (features) in the data. Preserves the subspace that contains these patterns and discards the remaining space. 6 / 24

7 A basic example Figure: The data points (left) are embedded into the feature space (middle) and then projected onto a low-dimensional subspace that captures most of the information in the data (right). 7 / 24

8 The feature space The data is mapped from its original space (R d ) into the feature space (F) by the mapping function Φ: Φ : R d F. (1) A data point x i is represented as Φ(x i ) in the feature space. F and Φ are not known explicitly (!), but thanks to the kernel trick we can use them nevertheless. 8 / 24

9 The kernel trick The inner products of the data points in feature space, represented as a kernel matrix K ij = Φ(x i ) Φ(x j ). (2) The kernel matrix contains the relative distance between all the data points. For a data set in F The values of Φ(x 1 ),..., Φ(x n ) are unknown. The inner products Φ(x i ) Φ(x j ) are known (for all i, j). 9 / 24

10 The kernel trick The elements of K can be computed using a kernel function. The matrix K is symmetric and positive semidefinite. Mercer s Theorem: Any symmetric, positive semidefinite matrix can be regarded as an inner product matrix (kernel matrix) in some space. 10 / 24

11 The kernel function and kernel matrix Example of a kernel function K ij = Φ(x i ) Φ(x j ) = exp( σ x i x j 2 ), (3) where σ is a parameter. 11 / 24

12 Implementing kernel PCA in MATLAB s in MATLAB 1 : Nearest neighbor method Kernel regression Multiclass classification Random projections Optimizing label-weights using SDP (SeDuMi) 1 Available online: 12 / 24

13 Testing the implementation: The transduction problem Data points Each belongs to a class: Either +1 or -1. Training point: Known class. Test point: Unknown class. Transduction problem: Given the data and the class of each training points, determine which class each test point belongs to. 13 / 24

14 Nearest neighbor method Figure: Each test point has the same class as its nearest training point. (Figure by A. Ajanki.) 14 / 24

15 Kernel regression Determines the class of the points in the test set. Binary classifier Class of each test point is either +1 or -1. Find a vector c such that where γ is a parameter. (n tr γi ntr + K tr )c = y, (4) 15 / 24

16 Kernel regression The binary classification is performed by n tr f (x) := c i K tt,xi (x) (5) i=1 For a point x i If f (x i ) is positive class of x i is +1. If f (x i ) is negative class of x i is / 24

17 Kernel regression Figure: Kernel regression fits a function (green curve) to the set of training points (red points), and then uses the function to determine the class of each test points (located along the blue curve). (Poggio and Smale 2003) 17 / 24

18 The databases of kernel PCA tested on databases. Iris data 150 datapoints, 4-dimensional Three classes Sugar data 681 datapoints, 11-dimensional Four classes: Allose, Galactose, Glucose, Mannose 18 / 24

19 Result tools Prediction accuracy Percentage of correct predictions of the classes of the points in the test set. Colored plots The points are plotted in 3D. The data points are colored according to class. 19 / 24

20 for Iris data Figure: Four plots of the Iris data points for different values of the parameter σ. 20 / 24

21 for Sugar data Figure: Four plots of the sugar data points, using label-based weights (left) or no weights (right). 21 / 24

22 for Sugar data Figure: Allose (red), Galactose (green), Glucose (blue) and Mannose (black). 22 / 24

23 Finds nonlinear patterns in high dimensional data. Can be used for the labeling of a partially labeled set. Competitive with other dimension reduction methods. 23 / 24

24 Questions and comments 24 / 24