Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University"

Transcription

1 Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University

2 Outline An overview on classification Basics of classification How to choose appropriate features (feature set) How to perform classification Classifiers

3 Where are we right now? Figure: Classification in image processing.

4 What is intelligence? Ability to separate relevant information Ability to learn from examples and to generalize knowledge so that it can be used in other situations Ability to draw conclusions from incomplete information

5 What is classification? Classification is a process in which individual items (objects/patterns/image regions/pixels) are grouped based on the similarity between the item and the description of the group Figure: Classification process.

6 What is classification? Distinguish between different types of objects Determine which parts of the image belong to the object of interest We would like to create an intelligent system that can draw conclusions from our image data Often used as the last stage in automated image analysis task

7 Classification example 1 Figure: Classification of leaves.

8 Classification example 2 Figure: Histological bone implant images.

9 Some important concepts Arrangements of descriptors are often called patterns Descriptors are often called features The most common pattern arrangement is a feature vector with n dimensions Patterns are placed in classes of objects which share common properties A collection of N classes are denoted ω 1, ω 2,..., ω N

10 Terminology Object = pattern = point = sample = vector Feature = attribute = measurement Classifier = decision function (boundary) Class Cluster

11 Dataset By measuring the features of many objects we construct dataset Object Perimeter Area Label Apple Apple Apple Pear Pear Pear

12 Data set and a representation in the feature space

13 Features Features are the individual measurable heuristic properties of the phenomena Discriminating (effective) features Independent features Features: area, perimeter texture color...

14 What are good features? Each pattern is represented in terms of n features x = (x 1,..., x n ) The goal is to choose those features that allow pattern vectors belonging to different classes to occupy compact and disjoint regions It is application dependent You might try many, many features, until you find the right ones Often, people compute 100s of features, and put them all in a classifier The classifier will figure out which ones are good This is wrong!!!

15 Peaking phenomenon (Curse of dimensionality) Additional features may actually degrade the performance of a classifier This paradox is called peaking phenomenon (curse of dimensionality) Figure: Curse of dimensionality.

16 Feature set Colour? Area? Perimeter?...

17 Dimensionality reduction Keep the number of features as small as possible Measurements cost Accuracy Simplify pattern representation The resulting classifier will be faster and will use less memory A reduction in the number of features may lead to a loss in the discriminatory power and thereby lower the accuracy

18 Feature extraction Feature extraction is the process of generating features to be used in the selection and classification tasks Reduce dimensionality by (linear or non-linear) projection of n dimensional vector onto m dimensional vector (m < n) May have not clear physical meaning Figure: Feature extraction.

19 Feature selection Feature selection methods choose features from the original set based on some criteria Reduce dimensionality by selecting subset of original features Have clear physical meaning Figure: Feature selection.

20 Choice of a criterion function The main issues in dimensionality reduction is the choice of a criterion function A most commonly used is the classification error of a feature subset Example: A measure of distance between two distributions f 1 and f 2 Mahalanobis distance: Main assumption is that Gaussian distributions have equal covariance matrix Σ D M (f 1, f 2 ) = (m 1 m 2 ) T Σ 1 (m 1 m 2 ) m 1 is a mean of objects in class 1 m 2 is a mean of objects in class 2

21 Feature selection methods Exhaustive search Best individual features Sequential forward selection Sequential backward selection

22 Exhaustive search Evaluate all ( n m) possible subsets Guaranteed to find the optimal subset Very expensive! Example: If n = 50, m = 5 then there are possible subset (classifications) to evaluate

23 Best individual features Evaluate all n features individually Select the best m n individual features Simple and not likely to lead to an optimal subset Example: Feature 1 is best Feature 2 is best Maybe features 3 and 4 outperform features 1 and 2!

24 Sequential forward selection Selects the best feature and then add one feature at a time Once a feature is retained, it cannot be discarded Computationally fast (for a subset of size 2 examine n 1 possible subsets)

25 Sequential backward selection Starts with all n features and successive delete one feature at a time Once a feature is deleted, it cannot be brought back into the optimal subset Require more computation time than sequential forward selection

26 Search algorithms The presented algorithms are suboptimal When should we use forward selection? When should we use backward selection? Is there any optimal algorithms? Branch and bound (divide and conquer method) Genetic algorithms Simulated annealing

27 Supervised vs. Unsupervised classification Supervised First apply knowledge, then classify Unsupervised First classify, then apply knowledge Figure: Supervised and unsupervised classification.

28 Object-wise and pixel-wise classification Object-wise classification Uses shape information to describe patterns Size, mean intensity, mean color, etc. Pixel-wise classification Uses information from individual pixels Intensity, color, texture, spectral information

29 Object-wise classification Lab 2

30 Object-wise classification Segment the image into regions and label them Extract features for each pattern Train classifier on examples with known class to find discriminant function in the feature space For new examples decide their class using the discriminant function

31 Pixel-wise classification 256*256 patterns (pixels) 3 features (red, green and blue color) Four classes (stamen, leaf, stalk and background)

32 Pixel-wise classification The pattern is a pixel in a non-segmented image Extract (calculate) features for each pattern (pixel) e.g., color, gray-level representation of texture, temporal changes. Train classifier New samples are classified by classifier

33 Train and classify Training Find rules and discriminant function that separate patterns in different classes using known examples Classification Take a new unknown example and put it into the correct class using discriminant function

34 Training data Training data can be obtained from available training samples Classify patterns which are not used during the training stage The performance of classifier depends on the number of available training samples as well as the specific values of the samples Number of training samples should not be too small!

35 Training data Conditional densities are unknown and they have to be learned from available training data Maybe density is known (for example multivariate Gaussian) A common strategy is to replace unknown parameters by the their estimated values Less available information - the difficulty of classification problem increases

36 How to choose appropriate training set?

37 How to choose appropriate training set? Figure: Original image and training image.

38 How to choose appropriate training set? Figure: Original image and training image.

39 Classifiers Once a feature selection finds a proper representation, a classifier can be designed using a number of possible approaches The performance of classifier depends on the interrelationship between sample size, number of features and classifier complexity The choice of a classifier is a difficult problem!!! It is often based on which classifier(s) happen to be available or best known to the user

40 Most commonly used classifiers The user can modify several associated parameters and criterion function There exist some classification problem for which they are the best choice Classifiers (covered in this lecture): Minimal distance classifier Maximal likelihood classifier Nearest neighbour classifier

41 Discriminant functions A discriminant function for a class is a function that will yield larger values than functions for other classes if the pattern belongs to the class d j (x) > d j (x) j = 1, 2,..., N; j i For N pattern classes, we have N discriminant functions The decision boundary between class i and class j is d j (x) d i (x) = 0

42 Decision boundary

43 Decision boundary Figure: Decision boundary.

44 Linear and quadratic classifier Figure: Linear (left) and quadratic (right) classifier.

45 Thresholding - simple classification Classify image into foreground and background Figure: Original image and tresholded image.

46 Box classification Generalized thresholding (multispectral thresholding) Interval for each class and feature All objects with feature vector within the same box belong to the same class

47 Bayesian classifiers Based on a priori knowledge of class probability Cost of errors Minimum distance classifier Maximum likelihood classifier

48 Minimum distance classifier Each class is represented by its mean vector Training is done using the objects (pixels) of known class Mean of the feature vectors for the object within the class is calculated New objects are classified by finding the closest mean vector

49 Minimum distance classifier

50 Limitations of Minimum distance classifier Useful when distance between means is large compared to randomness of each class with respect to its mean Optimal performance when distributions of the classes form spherical shape

51 Maximal likelihood classifier Classify according to the largest probability (taking variance and covariance into consideration) Assume that distribution within each class is Gaussian The distribution in each class can be described by a mean vector and covariance matrix

52 Maximal likelihood classifier For each class from training data compute: Mean vector Covariance matrix Form decision function for each class New objects are classified to class with highest probability

53 Variance and covariance Variance : spread a randomness for class Covariance : influence (dependency) between different features Described by covariance matrix Feature 1 Feature 2 Feature 3 Feature 1 1 1&2 1&3 Feature 2 1&2 2 2&3 Feature 3 1&3 2&3 3

54 Computation of covariance Features : x 1, x 2,..., x n Feature vector for object i: x 1,i, x 2,i,..., x n,i Mean for each class x mean1, x mean2,..., x meanm n Covariance: cov(x i, x j ) = 1 n 1 (x i,k x meani )(x j,k x meanj ) k=1 cov(x 1, x 1 ) cov(x 1, x 2 ) cov(x 1, x 3 ) Σ = cov(x 2, x 1 ) cov(x 2, x 2 ) cov(x 2, x 3 ) cov(x 3, x 1 ) cov(x 3, x 2 ) cov(x 3, x 3 )

55 Density functions Equal variance Different variance Covariance is different from zero

56 Assumptions on covariance matrix 1 Case 1 (Minimal distance) No covariance, equal variance 2 Case 2 (Equal covariance) Same covariance for all classes 3 Case 3 (Uncorrelated) No covariance, different variance 4 Case 4 (General) Different covariances

57 Case 1 Independent features Equal variance Minimal distance classifier

58 Case 2 Equal covariances for all classes

59 Case 3 Independent features - no covariance Different variance for different features

60 Case 4 General Complicated decision boundaries

61 Nearest neighbour Stores all training samples Assigns pattern to majority class among k nearest neighbour

62 k nearest neighbour Assigns pattern to majority class among k nearest neighbour

63 k nearest neighbour Metric dependent Lazy The function is approximated only locally The best choice of k depends on data Sensitive to outliers Larger values of k reduce effects of noise, but make boundaries between the classes less distinct

64 Combine classifiers! Each of the classifiers can be developed in a different context and for entirely different representation/description (identification of persons by their voice, face as well as hand writing) More than a single training set is available

65 Combine classifiers! Different classifiers trained on the same data may only differ in their global performances May show strong local differences Combination is very useful if the individual classifiers are independent

66 Combination Parallel All classifiers are evaluated independently Their results are combined (first weight then combine!!!) Serial - linear sequence The number of possible classes for a given pattern are gradually reduced as more classifiers in the sequence has be involved First consider simple classifiers (low computational and measurements cost) More accurate and expensive classifiers apply after Hierarchical (tree-like) Individual classifiers are combined into a structure (a tree-like structure) Advantage high efficiently and flexibility

67 Unsupervised classification Difficult, expensive or even impossible to rely label training sample with its true category (It is not possible to obtain ground truth) Patterns within a cluster are more similar to each other than are patterns belonging to different clusters

68 Unsupervised classification Difficult How to determine the number of clusters K? The number of clusters often depends on resolution (fine vs. coarse)

69 K means Step 1. Select an initial partition with K clusters. Repeat steps 2 through 4 until the cluster membership stabilizes Step 2. Generate a new partition by assigning each pattern to its closest cluster center Step 3. Compute new cluster centres as the centroids of the clusters Step 4. Repeat steps 2 and 3 until an optimum value of the criterion function is found (the cluster are stabilized)

70 K means

71 K means

72 K means - disadvantages Different initialization can result in different final clusters Fixed number of clusters can make it difficult to predict what K should be It is helpful to rerun the classification using the same as well as different K values, to compare the achieved results 10 different initializations for 2D data For N-dimensional data 10 different initializations is often not enough!

73 How to determine K? Davies-Bouldin index (DB index) It works for spherical clusters Intra-cluster and inter-cluster measure

74 Hierarchical clustering Hierarchical clustering - construct clustering tree (dendrogram) Start with each object/pixel as its own class Merge the classes that are closest according to some distance measure Continue until only one class is achieved Decide the number of classes based on the distances in the tree

75 Simple dendrogram

76 Distances measure and linking rules Distance measures: Euclidean City block Mahalonobis Linking rules - how distances are measured when cluster contains more than one object Single linkage (nearest neighbours) Complete linkage (furthest neighbours) Mean linkage (distance between cluster centres)

77 Linking rules Dependent on metric d complete - red d single - blue

78 Summary and conclusions Classification is needed in image processing It is highly application dependent Features and classifiers Using more features does not guarantee a better result!

Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University

Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate

More information

Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1

Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods

More information

Unsupervised Learning : Clustering

Unsupervised Learning : Clustering Unsupervised Learning : Clustering Things to be Addressed Traditional Learning Models. Cluster Analysis K-means Clustering Algorithm Drawbacks of traditional clustering algorithms. Clustering as a complex

More information

Unsupervised Learning

Unsupervised Learning Unsupervised Learning Unsupervised learning Until now, we have assumed our training samples are labeled by their category membership. Methods that use labeled samples are said to be supervised. However,

More information

10601 Machine Learning. Hierarchical clustering. Reading: Bishop: 9-9.2

10601 Machine Learning. Hierarchical clustering. Reading: Bishop: 9-9.2 161 Machine Learning Hierarchical clustering Reading: Bishop: 9-9.2 Second half: Overview Clustering - Hierarchical, semi-supervised learning Graphical models - Bayesian networks, HMMs, Reasoning under

More information

Unsupervised Learning. Presenter: Anil Sharma, PhD Scholar, IIIT-Delhi

Unsupervised Learning. Presenter: Anil Sharma, PhD Scholar, IIIT-Delhi Unsupervised Learning Presenter: Anil Sharma, PhD Scholar, IIIT-Delhi Content Motivation Introduction Applications Types of clustering Clustering criterion functions Distance functions Normalization Which

More information

Discriminate Analysis

Discriminate Analysis Discriminate Analysis Outline Introduction Linear Discriminant Analysis Examples 1 Introduction What is Discriminant Analysis? Statistical technique to classify objects into mutually exclusive and exhaustive

More information

Clustering. Mihaela van der Schaar. January 27, Department of Engineering Science University of Oxford

Clustering. Mihaela van der Schaar. January 27, Department of Engineering Science University of Oxford Department of Engineering Science University of Oxford January 27, 2017 Many datasets consist of multiple heterogeneous subsets. Cluster analysis: Given an unlabelled data, want algorithms that automatically

More information

Machine Learning. Unsupervised Learning. Manfred Huber

Machine Learning. Unsupervised Learning. Manfred Huber Machine Learning Unsupervised Learning Manfred Huber 2015 1 Unsupervised Learning In supervised learning the training data provides desired target output for learning In unsupervised learning the training

More information

Supervised vs unsupervised clustering

Supervised vs unsupervised clustering Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful

More information

Cluster Analysis. Ying Shen, SSE, Tongji University

Cluster Analysis. Ying Shen, SSE, Tongji University Cluster Analysis Ying Shen, SSE, Tongji University Cluster analysis Cluster analysis groups data objects based only on the attributes in the data. The main objective is that The objects within a group

More information

Introduction to Computer Science

Introduction to Computer Science DM534 Introduction to Computer Science Clustering and Feature Spaces Richard Roettger: About Me Computer Science (Technical University of Munich and thesis at the ICSI at the University of California at

More information

Segmentation of Images

Segmentation of Images Segmentation of Images SEGMENTATION If an image has been preprocessed appropriately to remove noise and artifacts, segmentation is often the key step in interpreting the image. Image segmentation is a

More information

Clustering and Visualisation of Data

Clustering and Visualisation of Data Clustering and Visualisation of Data Hiroshi Shimodaira January-March 28 Cluster analysis aims to partition a data set into meaningful or useful groups, based on distances between data points. In some

More information

Case-Based Reasoning. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. Parametric / Non-parametric.

Case-Based Reasoning. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. Parametric / Non-parametric. CS 188: Artificial Intelligence Fall 2008 Lecture 25: Kernels and Clustering 12/2/2008 Dan Klein UC Berkeley Case-Based Reasoning Similarity for classification Case-based reasoning Predict an instance

More information

Machine learning Pattern recognition. Classification/Clustering GW Chapter 12 (some concepts) Textures

Machine learning Pattern recognition. Classification/Clustering GW Chapter 12 (some concepts) Textures Machine learning Pattern recognition Classification/Clustering GW Chapter 12 (some concepts) Textures Patterns and pattern classes Pattern: arrangement of descriptors Descriptors: features Patten class:

More information

Figure 1: Workflow of object-based classification

Figure 1: Workflow of object-based classification Technical Specifications Object Analyst Object Analyst is an add-on package for Geomatica that provides tools for segmentation, classification, and feature extraction. Object Analyst includes an all-in-one

More information

Spectral Classification

Spectral Classification Spectral Classification Spectral Classification Supervised versus Unsupervised Classification n Unsupervised Classes are determined by the computer. Also referred to as clustering n Supervised Classes

More information

CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS

CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of

More information

Unsupervised Learning

Unsupervised Learning Networks for Pattern Recognition, 2014 Networks for Single Linkage K-Means Soft DBSCAN PCA Networks for Kohonen Maps Linear Vector Quantization Networks for Problems/Approaches in Machine Learning Supervised

More information

Clustering & Classification (chapter 15)

Clustering & Classification (chapter 15) Clustering & Classification (chapter 5) Kai Goebel Bill Cheetham RPI/GE Global Research goebel@cs.rpi.edu cheetham@cs.rpi.edu Outline k-means Fuzzy c-means Mountain Clustering knn Fuzzy knn Hierarchical

More information

Data: a collection of numbers or facts that require further processing before they are meaningful

Data: a collection of numbers or facts that require further processing before they are meaningful Digital Image Classification Data vs. Information Data: a collection of numbers or facts that require further processing before they are meaningful Information: Derived knowledge from raw data. Something

More information

PAM algorithm. Types of Data in Cluster Analysis. A Categorization of Major Clustering Methods. Partitioning i Methods. Hierarchical Methods

PAM algorithm. Types of Data in Cluster Analysis. A Categorization of Major Clustering Methods. Partitioning i Methods. Hierarchical Methods Whatis Cluster Analysis? Clustering Types of Data in Cluster Analysis Clustering part II A Categorization of Major Clustering Methods Partitioning i Methods Hierarchical Methods Partitioning i i Algorithms:

More information

6. Dicretization methods 6.1 The purpose of discretization

6. Dicretization methods 6.1 The purpose of discretization 6. Dicretization methods 6.1 The purpose of discretization Often data are given in the form of continuous values. If their number is huge, model building for such data can be difficult. Moreover, many

More information

Content-based Image and Video Retrieval. Image Segmentation

Content-based Image and Video Retrieval. Image Segmentation Content-based Image and Video Retrieval Vorlesung, SS 2011 Image Segmentation 2.5.2011 / 9.5.2011 Image Segmentation One of the key problem in computer vision Identification of homogenous region in the

More information

3 Feature Selection & Feature Extraction

3 Feature Selection & Feature Extraction 3 Feature Selection & Feature Extraction Overview: 3.1 Introduction 3.2 Feature Extraction 3.3 Feature Selection 3.3.1 Max-Dependency, Max-Relevance, Min-Redundancy 3.3.2 Relevance Filter 3.3.3 Redundancy

More information

Region-based Segmentation

Region-based Segmentation Region-based Segmentation Image Segmentation Group similar components (such as, pixels in an image, image frames in a video) to obtain a compact representation. Applications: Finding tumors, veins, etc.

More information

Image Segmentation. Srikumar Ramalingam School of Computing University of Utah. Slides borrowed from Ross Whitaker

Image Segmentation. Srikumar Ramalingam School of Computing University of Utah. Slides borrowed from Ross Whitaker Image Segmentation Srikumar Ramalingam School of Computing University of Utah Slides borrowed from Ross Whitaker Segmentation Semantic Segmentation Indoor layout estimation What is Segmentation? Partitioning

More information

Learning to Learn: additional notes

Learning to Learn: additional notes MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science 6.034 Artificial Intelligence, Fall 2008 Recitation October 23 Learning to Learn: additional notes Bob Berwick

More information

IBL and clustering. Relationship of IBL with CBR

IBL and clustering. Relationship of IBL with CBR IBL and clustering Distance based methods IBL and knn Clustering Distance based and hierarchical Probability-based Expectation Maximization (EM) Relationship of IBL with CBR + uses previously processed

More information

Finding Clusters 1 / 60

Finding Clusters 1 / 60 Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering Clustering by Partitioning, e.g. k-means Density Based Clustering, e.g. DBScan Grid Based Clustering 1 / 60

More information

DATA MINING LECTURE 7. Hierarchical Clustering, DBSCAN The EM Algorithm

DATA MINING LECTURE 7. Hierarchical Clustering, DBSCAN The EM Algorithm DATA MINING LECTURE 7 Hierarchical Clustering, DBSCAN The EM Algorithm CLUSTERING What is a Clustering? In general a grouping of objects such that the objects in a group (cluster) are similar (or related)

More information

Feature Extractors. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. The Perceptron Update Rule.

Feature Extractors. CS 188: Artificial Intelligence Fall Nearest-Neighbor Classification. The Perceptron Update Rule. CS 188: Artificial Intelligence Fall 2007 Lecture 26: Kernels 11/29/2007 Dan Klein UC Berkeley Feature Extractors A feature extractor maps inputs to feature vectors Dear Sir. First, I must solicit your

More information

[7.3, EA], [9.1, CMB]

[7.3, EA], [9.1, CMB] K-means Clustering Ke Chen Reading: [7.3, EA], [9.1, CMB] Outline Introduction K-means Algorithm Example How K-means partitions? K-means Demo Relevant Issues Application: Cell Neulei Detection Summary

More information

3. Cluster analysis Overview

3. Cluster analysis Overview Université Laval Analyse multivariable - mars-avril 2008 1 3.1. Overview 3. Cluster analysis Clustering requires the recognition of discontinuous subsets in an environment that is sometimes discrete (as

More information

Clustering (COSC 416) Nazli Goharian. Document Clustering.

Clustering (COSC 416) Nazli Goharian. Document Clustering. Clustering (COSC 416) Nazli Goharian nazli@cs.georgetown.edu 1 Document Clustering. Cluster Hypothesis : By clustering, documents relevant to the same topics tend to be grouped together. C. J. van Rijsbergen,

More information

Fuzzy Segmentation. Chapter Introduction. 4.2 Unsupervised Clustering.

Fuzzy Segmentation. Chapter Introduction. 4.2 Unsupervised Clustering. Chapter 4 Fuzzy Segmentation 4. Introduction. The segmentation of objects whose color-composition is not common represents a difficult task, due to the illumination and the appropriate threshold selection

More information

CS 521 Data Mining Techniques Instructor: Abdullah Mueen

CS 521 Data Mining Techniques Instructor: Abdullah Mueen CS 521 Data Mining Techniques Instructor: Abdullah Mueen LECTURE 2: DATA TRANSFORMATION AND DIMENSIONALITY REDUCTION Chapter 3: Data Preprocessing Data Preprocessing: An Overview Data Quality Major Tasks

More information

An Introduction to Cluster Analysis. Zhaoxia Yu Department of Statistics Vice Chair of Undergraduate Affairs

An Introduction to Cluster Analysis. Zhaoxia Yu Department of Statistics Vice Chair of Undergraduate Affairs An Introduction to Cluster Analysis Zhaoxia Yu Department of Statistics Vice Chair of Undergraduate Affairs zhaoxia@ics.uci.edu 1 What can you say about the figure? signal C 0.0 0.5 1.0 1500 subjects Two

More information

MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A

MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A. 205-206 Pietro Guccione, PhD DEI - DIPARTIMENTO DI INGEGNERIA ELETTRICA E DELL INFORMAZIONE POLITECNICO DI BARI

More information

ECG782: Multidimensional Digital Signal Processing

ECG782: Multidimensional Digital Signal Processing Professor Brendan Morris, SEB 3216, brendan.morris@unlv.edu ECG782: Multidimensional Digital Signal Processing Spring 2014 TTh 14:30-15:45 CBC C313 Lecture 10 Segmentation 14/02/27 http://www.ee.unlv.edu/~b1morris/ecg782/

More information

Introduction to Machine Learning CMU-10701

Introduction to Machine Learning CMU-10701 Introduction to Machine Learning CMU-10701 Clustering and EM Barnabás Póczos & Aarti Singh Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K-

More information

K-Means Clustering 3/3/17

K-Means Clustering 3/3/17 K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data points. We want to find underlying structure in the data. Examples: Identify groups of similar data points. Clustering

More information

CHAPTER 3 PRINCIPAL COMPONENT ANALYSIS AND FISHER LINEAR DISCRIMINANT ANALYSIS

CHAPTER 3 PRINCIPAL COMPONENT ANALYSIS AND FISHER LINEAR DISCRIMINANT ANALYSIS 38 CHAPTER 3 PRINCIPAL COMPONENT ANALYSIS AND FISHER LINEAR DISCRIMINANT ANALYSIS 3.1 PRINCIPAL COMPONENT ANALYSIS (PCA) 3.1.1 Introduction In the previous chapter, a brief literature review on conventional

More information

INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering

INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering Erik Velldal University of Oslo Sept. 18, 2012 Topics for today 2 Classification Recap Evaluating classifiers Accuracy, precision,

More information

University of Florida CISE department Gator Engineering. Clustering Part 2

University of Florida CISE department Gator Engineering. Clustering Part 2 Clustering Part 2 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville Partitional Clustering Original Points A Partitional Clustering Hierarchical

More information

How do microarrays work

How do microarrays work Lecture 3 (continued) Alvis Brazma European Bioinformatics Institute How do microarrays work condition mrna cdna hybridise to microarray condition Sample RNA extract labelled acid acid acid nucleic acid

More information

K-Nearest Neighbour Classifier. Izabela Moise, Evangelos Pournaras, Dirk Helbing

K-Nearest Neighbour Classifier. Izabela Moise, Evangelos Pournaras, Dirk Helbing K-Nearest Neighbour Classifier Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Reminder Supervised data mining Classification Decision Trees Izabela

More information

Idea. Found boundaries between regions (edges) Didn t return the actual region

Idea. Found boundaries between regions (edges) Didn t return the actual region Region Segmentation Idea Edge detection Found boundaries between regions (edges) Didn t return the actual region Segmentation Partition image into regions find regions based on similar pixel intensities,

More information

International Journal of Innovative Research in Computer and Communication Engineering

International Journal of Innovative Research in Computer and Communication Engineering Moving Object Detection By Background Subtraction V.AISWARYA LAKSHMI, E.ANITHA, S.SELVAKUMARI. Final year M.E, Department of Computer Science and Engineering Abstract : Intelligent video surveillance systems

More information

Clustering Part 4 DBSCAN

Clustering Part 4 DBSCAN Clustering Part 4 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville DBSCAN DBSCAN is a density based clustering algorithm Density = number of

More information

Workload Characterization Techniques

Workload Characterization Techniques Workload Characterization Techniques Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides are available on-line at: http://www.cse.wustl.edu/~jain/cse567-08/

More information

Unsupervised Learning: Clustering

Unsupervised Learning: Clustering Unsupervised Learning: Clustering Vibhav Gogate The University of Texas at Dallas Slides adapted from Carlos Guestrin, Dan Klein & Luke Zettlemoyer Machine Learning Supervised Learning Unsupervised Learning

More information

Lecture 7: Decision Trees

Lecture 7: Decision Trees Lecture 7: Decision Trees Instructor: Outline 1 Geometric Perspective of Classification 2 Decision Trees Geometric Perspective of Classification Perspective of Classification Algorithmic Geometric Probabilistic...

More information

Pit Pattern Classification of Zoom-Endoscopic Colon Images using D

Pit Pattern Classification of Zoom-Endoscopic Colon Images using D Pit Pattern Classification of Zoom-Endoscopic Colon Images using DCT and FFT Leonhard Brunauer Hannes Payer Robert Resch Department of Computer Science University of Salzburg February 1, 2007 Outline 1

More information

Articulated Pose Estimation with Flexible Mixtures-of-Parts

Articulated Pose Estimation with Flexible Mixtures-of-Parts Articulated Pose Estimation with Flexible Mixtures-of-Parts PRESENTATION: JESSE DAVIS CS 3710 VISUAL RECOGNITION Outline Modeling Special Cases Inferences Learning Experiments Problem and Relevance Problem:

More information

Quasi-thematic Features Detection & Tracking. Future Rover Long-Distance Autonomous Navigation

Quasi-thematic Features Detection & Tracking. Future Rover Long-Distance Autonomous Navigation Quasi-thematic Feature Detection And Tracking For Future Rover Long-Distance Autonomous Navigation Authors: Affan Shaukat, Conrad Spiteri, Yang Gao, Said Al-Milli, and Abhinav Bajpai Surrey Space Centre,

More information

CSE 5243 INTRO. TO DATA MINING

CSE 5243 INTRO. TO DATA MINING CSE 5243 INTRO. TO DATA MINING Cluster Analysis: Basic Concepts and Methods Huan Sun, CSE@The Ohio State University 09/25/2017 Slides adapted from UIUC CS412, Fall 2017, by Prof. Jiawei Han 2 Chapter 10.

More information

CS 664 Segmentation. Daniel Huttenlocher

CS 664 Segmentation. Daniel Huttenlocher CS 664 Segmentation Daniel Huttenlocher Grouping Perceptual Organization Structural relationships between tokens Parallelism, symmetry, alignment Similarity of token properties Often strong psychophysical

More information

Supervised and Unsupervised Learning (II)

Supervised and Unsupervised Learning (II) Supervised and Unsupervised Learning (II) Yong Zheng Center for Web Intelligence DePaul University, Chicago IPD 346 - Data Science for Business Program DePaul University, Chicago, USA Intro: Supervised

More information

Improved Performance of Unsupervised Method by Renovated K-Means

Improved Performance of Unsupervised Method by Renovated K-Means Improved Performance of Unsupervised Method by Renovated P.Ashok Research Scholar, Bharathiar University, Coimbatore Tamilnadu, India. ashokcutee@gmail.com Dr.G.M Kadhar Nawaz Department of Computer Application

More information

Introduction to Artificial Intelligence

Introduction to Artificial Intelligence Introduction to Artificial Intelligence COMP307 Machine Learning 2: 3-K Techniques Yi Mei yi.mei@ecs.vuw.ac.nz 1 Outline K-Nearest Neighbour method Classification (Supervised learning) Basic NN (1-NN)

More information

COMP33111: Tutorial and lab exercise 7

COMP33111: Tutorial and lab exercise 7 COMP33111: Tutorial and lab exercise 7 Guide answers for Part 1: Understanding clustering 1. Explain the main differences between classification and clustering. main differences should include being unsupervised

More information

Sequence clustering. Introduction. Clustering basics. Hierarchical clustering

Sequence clustering. Introduction. Clustering basics. Hierarchical clustering Sequence clustering Introduction Data clustering is one of the key tools used in various incarnations of data-mining - trying to make sense of large datasets. It is, thus, natural to ask whether clustering

More information

Exploratory data analysis for microarrays

Exploratory data analysis for microarrays Exploratory data analysis for microarrays Jörg Rahnenführer Computational Biology and Applied Algorithmics Max Planck Institute for Informatics D-66123 Saarbrücken Germany NGFN - Courses in Practical DNA

More information

Bagging for One-Class Learning

Bagging for One-Class Learning Bagging for One-Class Learning David Kamm December 13, 2008 1 Introduction Consider the following outlier detection problem: suppose you are given an unlabeled data set and make the assumptions that one

More information

Introduction to machine learning, pattern recognition and statistical data modelling Coryn Bailer-Jones

Introduction to machine learning, pattern recognition and statistical data modelling Coryn Bailer-Jones Introduction to machine learning, pattern recognition and statistical data modelling Coryn Bailer-Jones What is machine learning? Data interpretation describing relationship between predictors and responses

More information

Facial Expression Recognition Using Non-negative Matrix Factorization

Facial Expression Recognition Using Non-negative Matrix Factorization Facial Expression Recognition Using Non-negative Matrix Factorization Symeon Nikitidis, Anastasios Tefas and Ioannis Pitas Artificial Intelligence & Information Analysis Lab Department of Informatics Aristotle,

More information

Biometrics Technology: Image Processing & Pattern Recognition (by Dr. Dickson Tong)

Biometrics Technology: Image Processing & Pattern Recognition (by Dr. Dickson Tong) Biometrics Technology: Image Processing & Pattern Recognition (by Dr. Dickson Tong) References: [1] http://homepages.inf.ed.ac.uk/rbf/hipr2/index.htm [2] http://www.cs.wisc.edu/~dyer/cs540/notes/vision.html

More information

Kapitel 4: Clustering

Kapitel 4: Clustering Ludwig-Maximilians-Universität München Institut für Informatik Lehr- und Forschungseinheit für Datenbanksysteme Knowledge Discovery in Databases WiSe 2017/18 Kapitel 4: Clustering Vorlesung: Prof. Dr.

More information

Evaluation of Different Metrics for Shape Based Image Retrieval Using a New Contour Points Descriptor

Evaluation of Different Metrics for Shape Based Image Retrieval Using a New Contour Points Descriptor Evaluation of Different Metrics for Shape Based Image Retrieval Using a New Contour Points Descriptor María-Teresa García Ordás, Enrique Alegre, Oscar García-Olalla, Diego García-Ordás University of León.

More information

COMP 465: Data Mining Still More on Clustering

COMP 465: Data Mining Still More on Clustering 3/4/015 Exercise COMP 465: Data Mining Still More on Clustering Slides Adapted From : Jiawei Han, Micheline Kamber & Jian Pei Data Mining: Concepts and Techniques, 3 rd ed. Describe each of the following

More information

Machine Learning : Clustering, Self-Organizing Maps

Machine Learning : Clustering, Self-Organizing Maps Machine Learning Clustering, Self-Organizing Maps 12/12/2013 Machine Learning : Clustering, Self-Organizing Maps Clustering The task: partition a set of objects into meaningful subsets (clusters). The

More information

Clustering (COSC 488) Nazli Goharian. Document Clustering.

Clustering (COSC 488) Nazli Goharian. Document Clustering. Clustering (COSC 488) Nazli Goharian nazli@ir.cs.georgetown.edu 1 Document Clustering. Cluster Hypothesis : By clustering, documents relevant to the same topics tend to be grouped together. C. J. van Rijsbergen,

More information

Bioimage Informatics

Bioimage Informatics Bioimage Informatics Lecture 14, Spring 2012 Bioimage Data Analysis (IV) Image Segmentation (part 3) Lecture 14 March 07, 2012 1 Outline Review: intensity thresholding based image segmentation Morphological

More information

Lecture: k-means & mean-shift clustering

Lecture: k-means & mean-shift clustering Lecture: k-means & mean-shift clustering Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab Lecture 11-1 Recap: Image Segmentation Goal: identify groups of pixels that go together

More information

Unsupervised Learning

Unsupervised Learning Unsupervised Learning Unsupervised learning Until now, we have assumed our training samples are labeled by their category membership. Methods that use labeled samples are said to be supervised. However,

More information

A Framework for Efficient Fingerprint Identification using a Minutiae Tree

A Framework for Efficient Fingerprint Identification using a Minutiae Tree A Framework for Efficient Fingerprint Identification using a Minutiae Tree Praveer Mansukhani February 22, 2008 Problem Statement Developing a real-time scalable minutiae-based indexing system using a

More information

Data Fusion. Merging data from multiple sources to optimize data or create value added data

Data Fusion. Merging data from multiple sources to optimize data or create value added data Data Fusion Jeffrey S. Evans - Landscape Ecologist USDA Forest Service Rocky Mountain Research Station Forestry Sciences Lab - Moscow, Idaho Data Fusion Data Fusion is a formal framework in which are expressed

More information

On Sample Weighted Clustering Algorithm using Euclidean and Mahalanobis Distances

On Sample Weighted Clustering Algorithm using Euclidean and Mahalanobis Distances International Journal of Statistics and Systems ISSN 0973-2675 Volume 12, Number 3 (2017), pp. 421-430 Research India Publications http://www.ripublication.com On Sample Weighted Clustering Algorithm using

More information

PATTERN CLASSIFICATION AND SCENE ANALYSIS

PATTERN CLASSIFICATION AND SCENE ANALYSIS PATTERN CLASSIFICATION AND SCENE ANALYSIS RICHARD O. DUDA PETER E. HART Stanford Research Institute, Menlo Park, California A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS New York Chichester Brisbane

More information

What is machine learning?

What is machine learning? Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship

More information

Density estimation. In density estimation problems, we are given a random from an unknown density. Our objective is to estimate

Density estimation. In density estimation problems, we are given a random from an unknown density. Our objective is to estimate Density estimation In density estimation problems, we are given a random sample from an unknown density Our objective is to estimate? Applications Classification If we estimate the density for each class,

More information

Enhancing K-means Clustering Algorithm with Improved Initial Center

Enhancing K-means Clustering Algorithm with Improved Initial Center Enhancing K-means Clustering Algorithm with Improved Initial Center Madhu Yedla #1, Srinivasa Rao Pathakota #2, T M Srinivasa #3 # Department of Computer Science and Engineering, National Institute of

More information

Unsupervised Learning

Unsupervised Learning Unsupervised Learning A review of clustering and other exploratory data analysis methods HST.951J: Medical Decision Support Harvard-MIT Division of Health Sciences and Technology HST.951J: Medical Decision

More information

University of Florida CISE department Gator Engineering. Clustering Part 5

University of Florida CISE department Gator Engineering. Clustering Part 5 Clustering Part 5 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville SNN Approach to Clustering Ordinary distance measures have problems Euclidean

More information

Statistics 202: Data Mining. c Jonathan Taylor. Outliers Based in part on slides from textbook, slides of Susan Holmes.

Statistics 202: Data Mining. c Jonathan Taylor. Outliers Based in part on slides from textbook, slides of Susan Holmes. Outliers Based in part on slides from textbook, slides of Susan Holmes December 2, 2012 1 / 1 Concepts What is an outlier? The set of data points that are considerably different than the remainder of the

More information

Introduction to Medical Imaging (5XSA0) Module 5

Introduction to Medical Imaging (5XSA0) Module 5 Introduction to Medical Imaging (5XSA0) Module 5 Segmentation Jungong Han, Dirk Farin, Sveta Zinger ( s.zinger@tue.nl ) 1 Outline Introduction Color Segmentation region-growing region-merging watershed

More information

DS504/CS586: Big Data Analytics Big Data Clustering Prof. Yanhua Li

DS504/CS586: Big Data Analytics Big Data Clustering Prof. Yanhua Li Welcome to DS504/CS586: Big Data Analytics Big Data Clustering Prof. Yanhua Li Time: 6:00pm 8:50pm Thu Location: AK 232 Fall 2016 High Dimensional Data v Given a cloud of data points we want to understand

More information

Hierarchical and Ensemble Clustering

Hierarchical and Ensemble Clustering Hierarchical and Ensemble Clustering Ke Chen Reading: [7.8-7., EA], [25.5, KPM], [Fred & Jain, 25] COMP24 Machine Learning Outline Introduction Cluster Distance Measures Agglomerative Algorithm Example

More information

CS443: Digital Imaging and Multimedia Binary Image Analysis. Spring 2008 Ahmed Elgammal Dept. of Computer Science Rutgers University

CS443: Digital Imaging and Multimedia Binary Image Analysis. Spring 2008 Ahmed Elgammal Dept. of Computer Science Rutgers University CS443: Digital Imaging and Multimedia Binary Image Analysis Spring 2008 Ahmed Elgammal Dept. of Computer Science Rutgers University Outlines A Simple Machine Vision System Image segmentation by thresholding

More information

UNSUPERVISED STATIC DISCRETIZATION METHODS IN DATA MINING. Daniela Joiţa Titu Maiorescu University, Bucharest, Romania

UNSUPERVISED STATIC DISCRETIZATION METHODS IN DATA MINING. Daniela Joiţa Titu Maiorescu University, Bucharest, Romania UNSUPERVISED STATIC DISCRETIZATION METHODS IN DATA MINING Daniela Joiţa Titu Maiorescu University, Bucharest, Romania danielajoita@utmro Abstract Discretization of real-valued data is often used as a pre-processing

More information

Chapter 6: Cluster Analysis

Chapter 6: Cluster Analysis Chapter 6: Cluster Analysis The major goal of cluster analysis is to separate individual observations, or items, into groups, or clusters, on the basis of the values for the q variables measured on each

More information

6. Object Identification L AK S H M O U. E D U

6. Object Identification L AK S H M O U. E D U 6. Object Identification L AK S H M AN @ O U. E D U Objects Information extracted from spatial grids often need to be associated with objects not just an individual pixel Group of pixels that form a real-world

More information

Unsupervised Learning. Supervised learning vs. unsupervised learning. What is Cluster Analysis? Applications of Cluster Analysis

Unsupervised Learning. Supervised learning vs. unsupervised learning. What is Cluster Analysis? Applications of Cluster Analysis 7 Supervised learning vs unsupervised learning Unsupervised Learning Supervised learning: discover patterns in the data that relate data attributes with a target (class) attribute These patterns are then

More information

Introduction to Machine Learning. Xiaojin Zhu

Introduction to Machine Learning. Xiaojin Zhu Introduction to Machine Learning Xiaojin Zhu jerryzhu@cs.wisc.edu Read Chapter 1 of this book: Xiaojin Zhu and Andrew B. Goldberg. Introduction to Semi- Supervised Learning. http://www.morganclaypool.com/doi/abs/10.2200/s00196ed1v01y200906aim006

More information

Content Based Medical Image Retrieval Using Fuzzy C- Means Clustering With RF

Content Based Medical Image Retrieval Using Fuzzy C- Means Clustering With RF Content Based Medical Image Retrieval Using Fuzzy C- Means Clustering With RF Jasmine Samraj #1, NazreenBee. M *2 # Associate Professor, Department of Computer Science, Quaid-E-Millath Government college

More information

Latent Class Modeling as a Probabilistic Extension of K-Means Clustering

Latent Class Modeling as a Probabilistic Extension of K-Means Clustering Latent Class Modeling as a Probabilistic Extension of K-Means Clustering Latent Class Cluster Models According to Kaufman and Rousseeuw (1990), cluster analysis is "the classification of similar objects

More information

Information Retrieval and Web Search Engines

Information Retrieval and Web Search Engines Information Retrieval and Web Search Engines Lecture 7: Document Clustering May 25, 2011 Wolf-Tilo Balke and Joachim Selke Institut für Informationssysteme Technische Universität Braunschweig Homework

More information

Chapter 5: Outlier Detection

Chapter 5: Outlier Detection Ludwig-Maximilians-Universität München Institut für Informatik Lehr- und Forschungseinheit für Datenbanksysteme Knowledge Discovery in Databases SS 2016 Chapter 5: Outlier Detection Lecture: Prof. Dr.

More information