Machine Learning (CSMML16) (Autumn term, ) Xia Hong

Transcription

1 Machine Learning (CSMML16) (Autumn term, 28-29) Xia Hong 1

2 Useful books: 1. C. M. Bishop: Pattern Recognition and Machine Learning (2007) Springer. 2. S. Haykin: Neural Networks (1999) Prentice Hall. 3. R. O. Duda and P. E. Hart: Pattern Classification and Scene Analysis (1973) J. Wiley 2

3 Introduction Why study machine learning? The problem of searching for patterns in data is fundamental one and has a long and successful history. For example, extensive astronomical observations has led to Kepler to discover the empirical laws of planetary motion in the 16th century. The field of pattern recognition is concerned with the automatic discovery of regularities via the use of computer of these regularities to take actions such as classifying the data into different categories. We are entering the era of big data. This calls for automated methods of data analysis, which is what machine learning provides. 3

4 Over the last two decades there has been dramatic growth in practical applications for machine learning, e.g. in computer vision, signal processing, facial and speech recognition, and internet search engine, web security, biomedical data processing (DNA data), industrial condition monitoring and control systems, etc. Example: Handwritten digits recognition. If each digit corresponds to a image, so it can be represented a vector x comprising 784 real numbers. The goal is to build a machine that takes x and reproduce the identity of the digit 0,...,9 as the output. 4

5 A machine learning approach is to use a large number of N digits {x 1,..., x N }, called a training set to tune the parameters of a model. The categories of the training data set are known, by hand-labeling. The category of a digit is called target vector {t}. The result of machine learning algorithm can be expressed as a function {y(x)}, where y is the model output that aimed at {t}. The training or learning phase is to find the exact form of {y(x)}. 5

6 Once the model is trained it can then used to identify new digit images, which are said to comprise a test set. The ability to categorize correctly on fresh data set that has been used in the training phase is known as generalization, which is a central goal in pattern recognition. Applications in which training set comprises examples of the input vector and along with their target vectors are known as supervised problems. Cases such as digit recognition aims to assign each input vector to one of a finite number of discrete categories, are called classification problems. If the desired output consists of one or more continuous variables, then task is called regression. 6

7 An example of regression problem would be the prediction of the yield in a chemical manufactory process in which the inputs are concentration of reactants, the temperature and the pressure. In other pattern recognition problems, the training data set consists of only a set of input vector x without any corresponding target value. Then it is called unsupervised problems. Examples of unsupervised problems may be: to discover groups of similar patterns in the data, where it is called clustering, or to determine the distribution of data within the data space, known as density estimation, or to project data from a high-dimensional space to 2-D or 3-D for the purpose of visualization, known as dimensionality reduction. 7

8 In many practical applications, the original input variables are typically pre-processed, so that the pattern recognition problems is easier to solve. The pre-processing stage is sometimes also called feature selection. Example: A lumber mill producing assorted hardwoods wants to automate the process of sorting finished lumber according to the species of trees. Optical sensing is used to distinguish birch lumber from ash lumber. A camera takes a pictures of the lumber and passes to on to a feature extractor. lumber picture transducer feature extractor classifier decision Figure: A pattern classification system 8

9 The feature extractor reduces the data by measuring certain properties that distinguish pictures of birch lumber from pictures of ash lumber. These features are then passed to a classifier that evaluates the evidence presented and makes a final decision about the lumber type. Suppose that somebody at the lumber mill tells us that birth is often lighter colored than ash. Then brightness becomes an obvious feature. We might attempt to classify the lumber merely by seeing whether or not the average brightness x exceeds some critical value. Suppose that we obtain the following histogram based on these data samples. 9

10 number of samples Ash x brightness x Birch Figure: Histogram for the brightness feature. The histogram bears out the statement that birch is usually lighter than ash, but it is clear that this single criterion is not infallible. No matter how we choose x 0, we cannot reliably separate birch from ash by brightness alone. The second feature is based on the observation that ash typically has a more prominent grain pattern than birch. It is reasonable to assume that we can obtain a measure of this feature from the magnitude and frequency of occurrence of light-to-dark transitions in the picture. 10

11 The feature extractor has thus reduced each picture to a point or a feature vector x in a two dimensional space, where, x = [x 1, x 2 ] T where x 1 denotes the brightness, x 2 denotes the grain prominence. Our problem now is to partition the feature space into two regions for birth and ash. Suppose that we measure the feature vectors for our training data samples and obtain the following scatter diagram. This plot suggests the rule for classifying the data: Classify the lumber as ash if its feature vector falls above the line AB, and as birch otherwise. 11

12 grain prominence x 2 Ash Birch 00 11B A brightness x 1 Figure: Scatter diagram for the feature vector. In order to make sure that this rule performs well, we obtain more samples and adjust the position of line AB in order to minimize the probability of error. This also shows that pattern recognition problem has a statistical nature. 12

13 Often preprocessing is necessary in practical applications to speed up computations. For example, if the goal is real time face recognition in a high-resolution video stream, directly handing the raw data may be infeasible for machine learning algorithms. The feature selection that are fast to compute that are also preserves useful discriminatory information enabling faces to be distinguished from non-faces. Machine learning is closely related to and often overlaps with statistics, the study of the collection, analysis, interpretation, presentation, and organization of data. It has strong ties to mathematical optimization, which delivers methods, theory and application domains to the field. 13

14 Preliminaries: Calculus, linear algebra, and probability theory are required. Orgianization of the course: / sisxh/ teaching /CSMML16/Machine Learning Lecture Notes.htm Matlab (are used in numerical simulations) If you have no Matlab experience, please follow the link lab/ms /matlab/matlab.html to quick start Matlab and follow the codes provided. 14