UNSUPERVISED LEARNING, CLUSTERING

Size: px

Start display at page:

Download "UNSUPERVISED LEARNING, CLUSTERING"

Theresa Daniels
5 years ago
Views:

1 UNSUPERVISED LEARNING, CLUSTERING

2 UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements, understanding data, identifying populations, problems, outliers (before modelling) Dimension reduction, important when working with high dimensional data Usually part of exploratory data analysis, which may lead to measuring the supervising signal when interesting structure is found in the X data Not a well defined problem

3 UNSUPERVISED LEARNING PCA - PRINCIPAL COMPONENT ANALYSIS PCA is a linear basis transformation from the original measured dimensions to new dimensions dictated by the variation in the data itself 1st component direction is along the largest variance in the data 2nd component is an orthogonal direction to the 1st with the largest variance 3rd component is an orthogonal direction to the 1st and the 2nd with the largest variance, etc.. The projections of the original data points give the principal components (N original dimensions: N components) y x σ 2 noise σ 2 signal Orthogonal directions lead to uncorrelated principal components Dimension reduction: the first some components capture the largest variation in the data, the interesting things! We can reveal some structure of the data using only few dimensions. 2-3 dimensions can be visualised and understood. Image from (Shlens)

4 UNSUPERVISED LEARNING PCA - PRINCIPAL COMPONENT ANALYSIS Standard use: 2D plots of loadings ( projections to components ) Original base directions may be useful to plot If clear clusters of data points are emerging, it should be investigated Second principal component First principal component Outliers: Sometimes components correspond to individual data points, outliers. These may be inspected and removed. And PCA can be repeated without the outliers. Second Principal Component UrbanPop Rape Assault Murder First Principal Component

5 UNSUPERVISED LEARNING PCA - PRINCIPAL COMPONENT ANALYSIS How many components? Proportion of variance explained. Details: Zero mean per dimension is assumed If different quantities are measured, units may not be comparable (Number of fingers or height in cm?) In this case, normalise original dimensions to have variance = 1 Only line of direction is defined: -1 flips might occur! Prop. Variance Explained Second Principal Component Principal Component Scaled UrbanPop Rape Assault Murder Cumulative Prop. Variance Explained Second Principal Component UrbanPop Rape Murder Assau First Principal Component First Principal Component Principal Component Unscaled

UNSUPERVISED LEARNING MORE DIMENSION REDUCTION, EMBEDDING

: In 3d space if distances can be measured, real positions

T-SNE, t-distributed stochastic neighbour embedding Usually

Getting very very common in genetic data analysis as a must

6 UNSUPERVISED LEARNING MORE DIMENSION REDUCTION, EMBEDDING MDS, Multi dimensional scaling Embed the points in low dimension, given their distances. E.g.: In 3d space if distances can be measured, real positions can be inferred. T-SNE, t-distributed stochastic neighbour embedding Usually gives the nicest separation of clear clusters. Getting very very common in genetic data analysis as a must have fancy plot. ICA, independent component analysis PCA: uncorrelated, ICA independent NMF Non-negative matrix factorisation (mutations) And more, left source stable/modules/manifold.html

7 CLUSTERING CLUSTERING Data points can be meaningfully categorised: clusters Classification: we have labels (y) for groups Clustering: labels are not measured, they are inferred from the (X) data Not a well defined problem Clusters inferred should be validated (with measurements, new data)

8 CLUSTERING K-MEANS CLUSTERING A priori fix the number of clusters Minimise the sum of intra-cluster distances Algorithm: 1. randomly data assign each data point to clusters 2. calculate cluster centroids, reassign each data point to the closest centroid, repeat until convergence Distance metric is generally Euclidean Local minimum is found, repeat multiple times to for best solution, and assessment of stability Left: possible failure modes. source ( cluster/plot_kmeans_assumptions.html)

9 CLUSTERING HIERARCHICAL CLUSTERING Number of clusters not fixed Iteratively agglomerate clusters from individual observations Important! It assumes nested clusters, this is often not true. e.g.: Hurricane lab data Algorithm: 1. assign each data point to a cluster 2. Join the two closest clusters Average Linkage Complete Linkage Single Linkage Cluster distance metric is super important Single (smallest pairwise distance) Average Complete (maximal distance) The result is not a clustering, it is a dendrogram. A horizontal cut defines a clustering. Where to cut? Well. Could be done divisively (Girwan-Newman graph clustering )

10 CLUSTERING MORE CLUSTERING DBSCAN, density thresholds define clusters Spectral clustering: using the eigenvectors of the pairwise distance matrix Gaussian mixture models And more, left source: clustering.html

11 SEMI-SUPERVISED LEARNING SEMI-SUPERVISED LEARNING Few data points have labels, most others not Exploit data structure of unlabelled examples for most effective supervised learning Use unsupervised learning to explore the data structure, clusters, and use few points to assign labels to cluster Hot topic, as data labelling is often much more expensive data unlabelled data collection: especially when professionals with high salaries are needed for labelling, e.g.: medical images Deep learning: Auto-encoders, Generative adversarial networks for one shot learning

5 Predicted SELF-SUPERVISED LEARNING Actual 6 SELF-SUPERVISED LEARNING Predicted Unsupervised learning, where a

Actual Supervised because the setup is the optimisation of an objective function 7 Unsupervised, because the

Grayscale image colorisation 1. Example input grayscale Denseand Optical Flow Prediction from a Static Image M.

Favaro Jacob Walker, Abhinav Gupta, and Martial Hebert Robotics Institute, Carnegie Mellon University Fig.

predicted These examples are cases where our model works especially well.

io/colorization/ to see the full range of results and to Jigsaw puzzle solving ry our model and code.

cv] 17 Dec 2015 Figure 4: PredNet Really hot topic, {jcwalker, abhinavg, hebert}@cs.cmu.

Such a question could be considered a non-semantic form of action prediction.

Given a static image, this (b) (c) CNN predicts the future (a) motion of each and every pixel in the image in

1: Learning image representations by solving Jigsaw puzzles. (a) The image our model.

(b) A puzzle obbeling and which is able to predict motion (marked based on the context the scene.

Some tiles might be directly identifiable as object about the underlying scene, it can predict future optical flow

are We outperform all previous approaches by large margins. predictions for car-cam videos.

The sequence below the red line was temporally scrambled.

sequences shown are from the CalTech Pedestrian dataset. we leverage data.

Humanlarge-scale is much more reliable when all tiles are jointly evaluated.

fication can be used as a training example, simply Introduction with reference to (c), determining the relative

12 5 Predicted SELF-SUPERVISED LEARNING Actual 6 SELF-SUPERVISED LEARNING Predicted Unsupervised learning, where a part of the data is predicted for another part of it. Actual Supervised because the setup is the optimisation of an objective function 7 Unsupervised, because the supervising signal is data itself Predicted Examples explain it 2 Future video frame prediction Scrambled Grayscale image colorisation 1. Example input grayscale Denseand Optical Flow Prediction from a Static Image M. Noroozi P. Favaro Jacob Walker, Abhinav Gupta, and Martial Hebert Robotics Institute, Carnegie Mellon University Fig. photos and output colorizations from our algo8 ithm.predicted These examples are cases where our model works especially well. Please visit Impainting Abstract ttp://richzhang.github.io/colorization/ to see the full range of results and to Jigsaw puzzle solving ry our model and code. Best viewed in color (obviously). time Motion direction predictions v2 [cs.cv] 17 Dec 2015 Figure 4: PredNet Really hot topic, {jcwalker, abhinavg, hebert}@cs.cmu.edu Given a scene, what is going to move, and in what direction will it move? Such a question could be considered a non-semantic form of action prediction. In this work, we present a convolutional neural network (CNN) based approach for motion prediction. Given a static image, this (b) (c) CNN predicts the future (a) motion of each and every pixel in the image in terms of optical flow. Our CNN model leverages the data in tens of thousands of realistic videos to train Fig. 1: Learning image representations by solving Jigsaw puzzles. (a) The image our model. Our method relies on absolutely no human lafrom the tiles withof green lines) are extracted. (b) A puzzle obbeling and which is able to predict motion (marked based on the context the scene. Because our CNN model makes no assumptions tained by shu ing the tiles. Some tiles might be directly identifiable as object about the underlying scene, it can predict future optical flow ambiguous (e.g., have similar patterns) and their identionparts, a diversebut set of others scenarios. are We outperform all previous approaches by large margins. predictions for car-cam videos. The first rows contain ground truth and the second rows contain predictions. The sequence below the red line was temporally scrambled. The model of magnitudes unsupervised data is collected (images videos) was orders trained on the KITTI dataset and sequences shown are from the CalTech Pedestrian dataset. we leverage data. Predicting color has the nice property that training visual learning is supposedly unsupervised (maybe it is self Humanlarge-scale is much more reliable when all tiles are jointly evaluated. In contrast, supervised) ata is practically free: any color photo1.fication can be used as a training example, simply Introduction with reference to (c), determining the relative position between the central tile Lotter et al, Zhang et al, Noroozi et Favaro, Walkerand et al the top two tiles from the left can be very challenging [10]. Images: the y taking image s L channel as input and its ab channels as the supervisory respectively, compared to the CNN-LSTM Encoder-Decoder. More details, as well as a thorough ignal. Others have noted the easy availability of training data, and previous investigation of systematically simplified models continuum between andandthe While it ison truethe that biological agents typically makethe use ofprednet multiple images Consider the images shown in Figure 1. Given the girl in front of the cake, we humans can easily predict that her head will move downward to extinguish the candle. The man with the discus is in a position to twist his body strongly to the right, and the squatting man on the bottom has nowhere to move but up. Humans have an amazing ability to not only recognize what is present in the image but also predict what

13 REFERENCES REFERENCES ISLR, chapter 10. ESL, chapter Shlens, J., A Tutorial on Principal Component Analysis. arxiv: [cs, stat]. Walker, J., Gupta, A., Hebert, M., Dense Optical Flow Prediction from a Static Image. arxiv: [cs]. Lotter, W., Kreiman, G., Cox, D., Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. arxiv: [cs, q-bio]. Zhang, R., Isola, P., Efros, A.A., Colorful Image Colorization. arxiv: [cs]. Noroozi, M., Favaro, P., Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. arxiv: [cs].

Network Traffic Measurements and Analysis

Network Traffic Measurements and Analysis DEIB - Politecnico di Milano Fall, 2017 Introduction Often, we have only a set of features x = x 1, x 2,, x n, but no associated response y. Therefore we are not interested in prediction nor classification,