CS 6140: Machine Learning Spring 2016
|
|
- Cory Osborne
- 5 years ago
- Views:
Transcription
1 CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa?on Science Northeastern University Webpage:
2 Logis?cs Assignment 2 will be released before next class (02/11). A couple more programming tasks For easier ones, you have to implement the classifiers and all preprocessing by yourselves For difficult ones, you can choose to use off-theshelf toolkits (need to acknowledge it) Also has analy?cal ques?ons. Due on 02/25
3 Logis?cs Feb 11 Topic: Deep Learning Feb 18 Topic: Deep Learning Feb 25 Topic: Structured Output Predic?on Assignment 2 due Mar 3 Topic: Mixture Models and Expecta?on Maximiza?on Mar 10 (No Class: Spring Break) Project progress report due Mar 17 Topic: Dimensionality Reduc?on Mar 24 Topic: Clustering Assignment 3 due Mar 31 Topic: Exam
4 Final Exams Short descrip?on ques?ons E.g what s the difference between supervised learning and unsupervised learning? Analy?cal ques?ons E.g. given a bunch of training samples (each has 3 features and 1 label), what is the predic?on of a naïve bayes classifier for a certain test sample? E.g. design a decision tree Open ques?ons E.g given a machine learning problem, how do you model it? What features you will use? What models you will use?
5 Final Exams 2 hours Open book Computer without Internet is okay Calculator is okay
6 What we learned last?me Bayesian Sta?s?cs Frequen?st Sta?s?cs Feature Selec?on
7 Bayesian vs. Frequen?st Frequen?st Bayesian
8 Feature Selec?on We oeen want to select a subset of the inputs that are most relevant for predic?ng the output, to get sparse models interpretability, speed, possibly beier predic?ve accuracy
9 Filter methods Compute relevance of each feature to the label marginally
10 Filter methods Compute relevance of each feature to the label marginally (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the 0.10
11 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng
12 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the 0.10
13 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set
14 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set - Then add class Correla,on with Label
15 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set - Then add class - Then add learning Correla,on with Label
16 Today s Outline Perceptron (and kernels) Support Vector Machines
17 Perceptron [Some of the slides are borrowed from Alex Smola s tutorial]
18 Biology and Learning Idea 1: Good behavior should be rewarded, bad behavior punished (or not awarded). Raising a dog. Idea 2: Correlated events should be combined. Babies learn language.
19 Biology and Learning Idea 1: Good behavior should be rewarded, bad behavior punished (or not awarded). Raising a dog. Idea 2: Correlated events should be combined. Babies learn language. Training Mechanisms Behavioral modifica?on of individuals (learning) Feeding the dog, then the dog learns to stand and sit. Hard-coded behavior in the genes (ins?nct) The wrongly coded animal dies.
20 Neurons
21 Perceptron
22 Perceptron Weighted combina?on The output of the neuron is a linear combina?on of the inputs Decision Func?on At the end the results are combined into
23 Perceptron An abstract model is to assume that Where w is the weight, x is the feature vectors b is the bias, Biological Interpreta?on The weights w i correspond to the synap?c weights, the mul?plica?on corresponds to the processing of inputs via the synapses, and the summa?on is the combina?on of signals in the cell body (soma).
24 Learning Goal: Linear Separa?on
25 Perceptron Algorithm
26 Perceptron Algorithm Nothing happens if we classify (x i, y i ) correctly If we see incorrectly classified observa?on we update w and b Posi?ve reinforcement of observa?ons
27 About the solu?on Perceptron Algorithm Weight vector is linear combina?on of observa?ons x i :
28 About the solu?on Perceptron Algorithm Classifica?on can be wriien in terms of dot products:
29
30 Pseudocode
31 The XOR Problem
32 The XOR Problem Perceptrons cannot learn such linearly inseparable func?ons!
33 Problem Linear func?ons are oeen too simple to provide good es?mators.
34 Problem Linear func?ons are oeen too simple to provide good es?mators. Idea: Map to a higher dimensional feature space via Replace every by in the perceptron algorithm.
35
36
37 Perceptron on Features
38 Problems with Construc?ng Features
39 Problems with Construc?ng Features Need to be an expert in the domain (e.g. Chinese characters). Can be expensive to compute.
40 Dimension = 1 Polynomial Features Dimension = 2 Dimension = d (skip proof)
41 Kernels Defini?on A kernel func?on is a symmetric func?on in its arguments for which the following property holds
42 Some choices of kernel func?ons
43 Linear Kernel
44 Laplacian Kernel
45 Gaussian Kernel
46 Kernel Perceptron
47 Linear Separators Which of these linear separators is op?mal? [Some of the slides are borrowed from David Sontag s lecture]
48
49 Support Vector Machine (SVM) SVMs (Vapnik, 1990 s) choose the linear separator with the largest margin. Hyperplane
50 Support Vector Machine (SVM) Reasons: Intui?on Theore?cal guarantee (skip here) In prac?cal tasks: SVM became famous when, using images as input, it gave accuracy comparable to neural-network with hand-designed features in a handwri?ng recogni?on task.
51 Support Vector Machine (SVM) How to find the hyperplane? Hyperplane
52 Planes A plane can be specified as the set of points given by
53 Planes A plane can be specified as the set of points given by Normal Vector: decide the direc?on of the plane
54 Normal to a plane Length of the vector
55 Scale invariance
56
57 What is the distance γ? Final result: can maximize margin by minimizing
58 Support vector machines
59 What if the data is not linearly separable?
60 What if the data is not linearly More features separable?
61 What if the data is not linearly separable? Old objec,ve New objec,ve
62 What if the data is not linearly separable? Old objec,ve New objec,ve Jointly minimize w.w and number of training mistakes!
63 What if the data is not linearly separable?
64 Allowing for slack: Soe margin SVM
65 Allowing for slack: Soe margin SVM
66 A Demonstra?on of SVMs hips://
67 Popular Tools for SVMs LIBSVM (c++) hips:// SVM light (c) hip://svmlight.joachims.org/ Scikit-learn (python) hip://scikit-learn.org/
68 Popular Tools for SVMs Torch (LuaJIT) hip://torch.ch/ Spider (Matlab) Weka (Java)
69 How do we op?mize the objec?ve? Quadra?c programming
70 Kernels Defini?on A kernel func?on is a symmetric func?on in its arguments for which the following property holds
71 How do we op?mize the objec?ve? Quadra?c programming No place to apply the kernel trick
72 Constrained op?miza?on
73 Constrained op?miza?on
74 Constrained op?miza?on
75 Lagrange mul?pliers Dual variables Lagrange Mul?pliers
76 How do we solve with constraints? Lagrange Mul?pliers
77 Lagrange mul?pliers Dual variables
78 Back to SVM (hard margin)
79 Dual SVM deriva?on
80 Dual SVM deriva?on Slater s condi?on from convex op?miza?on guarantees that these two op?miza?on problems are equivalent! (skip proof)
81 Dual SVM deriva?on
82 Dual SVM deriva?on
83 To get w and b
84 Classifica?on rule using dual solu?on Using dual solu?on dot product
85 Dual for the non-separable case
86 Dual for the non-separable case
87 How to interpret dual form
88 Back to the ques?on: What if the data is not linearly separable?
89 For example: Higher order polynomials
90 Dual formula?on only depends on dot-products of the features!
91 Dual formula?on only depends on dot-products of the features!
92 Kernels Defini?on A kernel func?on is a symmetric func?on in its arguments for which the following property holds
93 Kernel Trick
94 Soe margin SVM with kernel
95 Common kernels for SVM Aka Gaussian Radial basis func,on (RBF) kernel
96 Gaussian RBF kernel
97 SVM Classifica?on Python pseudocode
98 Overfilng Huge feature space with kernels: should we worry about overfilng? SVM objec?ve seeks a solu?on with large margin Good theore?cal guarantee But everything overfits some?mes
99 Overfilng Huge feature space with kernels: should we worry about overfilng? SVM objec?ve seeks a solu?on with large margin Good theore?cal guarantee But everything overfits some?mes Can control by: Selng C Choosing a beier kernel Varying parameters of the kernels
100 Dual for the non-separable case
101 Linear SVM C = 50
102 Linear SVM C = 50
103 Linear SVM C = 50
104 Linear SVM C = 50
105 Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to narrow margin (SVM tries to do a good job at separa?ng, even though it isn t possible)
106 Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to narrow margin (SVM tries to do a good job at separa?ng, even though it isn t possible) Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data
107 Dual for the non-separable case
108 Gaussian RBF Kernel with C = 0.1
109 Gaussian RBF Kernel with C = 0.2
110 Gaussian RBF Kernel with C = 0.4
111 Gaussian RBF Kernel with C = 0.8
112 Gaussian RBF Kernel with C = 1.6
113 Gaussian RBF Kernel with C = 3.2
114 Gaussian RBF Kernel with C = 6.4
115 Gaussian RBF Kernel with C = 12.8
116 Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to more complicated margin (SVM tries to do a good job at separa?ng, even though it isn t possible) OverfiNng for large C Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data
117 Common kernels for SVM Aka Gaussian Radial basis func,on (RBF) kernel
118 Gaussian RBF with different
119 Gaussian RBF with different
120 Gaussian RBF with different
121 Gaussian RBF with different
122 Insights Changing For clean data, doesn t maier much. For noisy data, small leads to more complicated margin (SVM tries to do a good job at separa?ng, even though it isn t possible) Lots of overfilng for small Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data
123 Dual for the non-separable case
124 Homework (part of assignment 2) Study the Sequen?al Minimal Op?miza?on algorithm and implement an SVM classifier by yourself References hip://cs229.stanford.edu/materials/smo.pdf Fast Training of Support Vector Machines using Sequen?al Minimal Op?miza?on hip://research.microsoe.com/pubs/68391/smobook.pdf
125 What we learned today Perceptron (and kernels) Support Vector Machines
126 Homework Read Murphy CH ,
CS 6140: Machine Learning Spring Final Exams. What we learned Final Exams 2/26/16
Logis@cs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Assignment
More informationCS 6140: Machine Learning Spring 2016
CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa?on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Logis?cs Exam
More informationML4Bio Lecture #1: Introduc3on. February 24 th, 2016 Quaid Morris
ML4Bio Lecture #1: Introduc3on February 24 th, 216 Quaid Morris Course goals Prac3cal introduc3on to ML Having a basic grounding in the terminology and important concepts in ML; to permit self- study,
More informationCSE 417T: Introduction to Machine Learning. Lecture 22: The Kernel Trick. Henry Chai 11/15/18
CSE 417T: Introduction to Machine Learning Lecture 22: The Kernel Trick Henry Chai 11/15/18 Linearly Inseparable Data What can we do if the data is not linearly separable? Accept some non-zero in-sample
More informationCAP5415-Computer Vision Lecture 13-Support Vector Machines for Computer Vision Applica=ons
CAP5415-Computer Vision Lecture 13-Support Vector Machines for Computer Vision Applica=ons Guest Lecturer: Dr. Boqing Gong Dr. Ulas Bagci bagci@ucf.edu 1 October 14 Reminders Choose your mini-projects
More informationMinimum Redundancy and Maximum Relevance Feature Selec4on. Hang Xiao
Minimum Redundancy and Maximum Relevance Feature Selec4on Hang Xiao Background Feature a feature is an individual measurable heuris4c property of a phenomenon being observed In character recogni4on: horizontal
More informationDeformable Part Models
Deformable Part Models References: Felzenszwalb, Girshick, McAllester and Ramanan, Object Detec@on with Discrimina@vely Trained Part Based Models, PAMI 2010 Code available at hkp://www.cs.berkeley.edu/~rbg/latent/
More informationIntroduction to Support Vector Machines
Introduction to Support Vector Machines CS 536: Machine Learning Littman (Wu, TA) Administration Slides borrowed from Martin Law (from the web). 1 Outline History of support vector machines (SVM) Two classes,
More informationSupport Vector Machines
Support Vector Machines RBF-networks Support Vector Machines Good Decision Boundary Optimization Problem Soft margin Hyperplane Non-linear Decision Boundary Kernel-Trick Approximation Accurancy Overtraining
More informationSupport Vector Machines
Support Vector Machines RBF-networks Support Vector Machines Good Decision Boundary Optimization Problem Soft margin Hyperplane Non-linear Decision Boundary Kernel-Trick Approximation Accurancy Overtraining
More informationFeature Extractors. CS 188: Artificial Intelligence Fall Some (Vague) Biology. The Binary Perceptron. Binary Decision Rule.
CS 188: Artificial Intelligence Fall 2008 Lecture 24: Perceptrons II 11/24/2008 Dan Klein UC Berkeley Feature Extractors A feature extractor maps inputs to feature vectors Dear Sir. First, I must solicit
More informationAll lecture slides will be available at CSC2515_Winter15.html
CSC2515 Fall 2015 Introduc3on to Machine Learning Lecture 9: Support Vector Machines All lecture slides will be available at http://www.cs.toronto.edu/~urtasun/courses/csc2515/ CSC2515_Winter15.html Many
More informationData Mining: Concepts and Techniques. Chapter 9 Classification: Support Vector Machines. Support Vector Machines (SVMs)
Data Mining: Concepts and Techniques Chapter 9 Classification: Support Vector Machines 1 Support Vector Machines (SVMs) SVMs are a set of related supervised learning methods used for classification Based
More informationSTA 4273H: Sta-s-cal Machine Learning
STA 4273H: Sta-s-cal Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! h0p://www.cs.toronto.edu/~rsalakhu/ Lecture 3 Parametric Distribu>ons We want model the probability
More informationPa#ern Recogni-on for Neuroimaging Toolbox
Pa#ern Recogni-on for Neuroimaging Toolbox Pa#ern Recogni-on Methods: Basics João M. Monteiro Based on slides from Jessica Schrouff and Janaina Mourão-Miranda PRoNTo course UCL, London, UK 2017 Outline
More informationAbout the Course. Reading List. Assignments and Examina5on
Uppsala University Department of Linguis5cs and Philology About the Course Introduc5on to machine learning Focus on methods used in NLP Decision trees and nearest neighbor methods Linear models for classifica5on
More informationClassification by Support Vector Machines
Classification by Support Vector Machines Florian Markowetz Max-Planck-Institute for Molecular Genetics Computational Molecular Biology Berlin Practical DNA Microarray Analysis 2003 1 Overview I II III
More informationSupport vector machines
Support vector machines When the data is linearly separable, which of the many possible solutions should we prefer? SVM criterion: maximize the margin, or distance between the hyperplane and the closest
More informationChakra Chennubhotla and David Koes
MSCBIO/CMPBIO 2065: Support Vector Machines Chakra Chennubhotla and David Koes Nov 15, 2017 Sources mmds.org chapter 12 Bishop s book Ch. 7 Notes from Toronto, Mark Schmidt (UBC) 2 SVM SVMs and Logistic
More informationSUPPORT VECTOR MACHINES
SUPPORT VECTOR MACHINES Today Reading AIMA 18.9 Goals (Naïve Bayes classifiers) Support vector machines 1 Support Vector Machines (SVMs) SVMs are probably the most popular off-the-shelf classifier! Software
More informationLinear methods for supervised learning
Linear methods for supervised learning LDA Logistic regression Naïve Bayes PLA Maximum margin hyperplanes Soft-margin hyperplanes Least squares resgression Ridge regression Nonlinear feature maps Sometimes
More informationTerraSwarm. A Machine Learning and Op0miza0on Toolkit for the Swarm. Ilge Akkaya, Shuhei Emoto, Edward A. Lee. University of California, Berkeley
TerraSwarm A Machine Learning and Op0miza0on Toolkit for the Swarm Ilge Akkaya, Shuhei Emoto, Edward A. Lee University of California, Berkeley TerraSwarm Tools Telecon 17 November 2014 Sponsored by the
More informationKernels + K-Means Introduction to Machine Learning. Matt Gormley Lecture 29 April 25, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Kernels + K-Means Matt Gormley Lecture 29 April 25, 2018 1 Reminders Homework 8:
More informationClassification by Support Vector Machines
Classification by Support Vector Machines Florian Markowetz Max-Planck-Institute for Molecular Genetics Computational Molecular Biology Berlin Practical DNA Microarray Analysis 2003 1 Overview I II III
More informationCS 6140: Machine Learning Spring 2017
CS 6140: Machine Learning Spring 2017 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Logis@cs Grades
More informationLecture 10: Support Vector Machines and their Applications
Lecture 10: Support Vector Machines and their Applications Cognitive Systems - Machine Learning Part II: Special Aspects of Concept Learning SVM, kernel trick, linear separability, text mining, active
More informationADVANCED CLASSIFICATION TECHNIQUES
Admin ML lab next Monday Project proposals: Sunday at 11:59pm ADVANCED CLASSIFICATION TECHNIQUES David Kauchak CS 159 Fall 2014 Project proposal presentations Machine Learning: A Geometric View 1 Apples
More informationAdvanced branch predic.on algorithms. Ryan Gabrys Ilya Kolykhmatov
Advanced branch predic.on algorithms Ryan Gabrys Ilya Kolykhmatov Context Branches are frequent: 15-25 % A branch predictor allows the processor to specula.vely fetch and execute instruc.ons down the predicted
More informationSUPPORT VECTOR MACHINES
SUPPORT VECTOR MACHINES Today Reading AIMA 8.9 (SVMs) Goals Finish Backpropagation Support vector machines Backpropagation. Begin with randomly initialized weights 2. Apply the neural network to each training
More informationSVM Classification in -Arrays
SVM Classification in -Arrays SVM classification and validation of cancer tissue samples using microarray expression data Furey et al, 2000 Special Topics in Bioinformatics, SS10 A. Regl, 7055213 What
More informationCSEP 573: Artificial Intelligence
CSEP 573: Artificial Intelligence Machine Learning: Perceptron Ali Farhadi Many slides over the course adapted from Luke Zettlemoyer and Dan Klein. 1 Generative vs. Discriminative Generative classifiers:
More informationLECTURE 5: DUAL PROBLEMS AND KERNELS. * Most of the slides in this lecture are from
LECTURE 5: DUAL PROBLEMS AND KERNELS * Most of the slides in this lecture are from http://www.robots.ox.ac.uk/~az/lectures/ml Optimization Loss function Loss functions SVM review PRIMAL-DUAL PROBLEM Max-min
More informationDM6 Support Vector Machines
DM6 Support Vector Machines Outline Large margin linear classifier Linear separable Nonlinear separable Creating nonlinear classifiers: kernel trick Discussion on SVM Conclusion SVM: LARGE MARGIN LINEAR
More informationSupport Vector Machines and their Applications
Purushottam Kar Department of Computer Science and Engineering, Indian Institute of Technology Kanpur. Summer School on Expert Systems And Their Applications, Indian Institute of Information Technology
More informationMore Learning. Ensembles Bayes Rule Neural Nets K-means Clustering EM Clustering WEKA
More Learning Ensembles Bayes Rule Neural Nets K-means Clustering EM Clustering WEKA 1 Ensembles An ensemble is a set of classifiers whose combined results give the final decision. test feature vector
More information10703 Deep Reinforcement Learning and Control
10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Policy Gradient II Used Materials Disclaimer: Much of the material and slides for this lecture
More informationMachine Learning Crash Course: Part I
Machine Learning Crash Course: Part I Ariel Kleiner August 21, 2012 Machine learning exists at the intersec
More informationECG782: Multidimensional Digital Signal Processing
ECG782: Multidimensional Digital Signal Processing Object Recognition http://www.ee.unlv.edu/~b1morris/ecg782/ 2 Outline Knowledge Representation Statistical Pattern Recognition Neural Networks Boosting
More informationApplication of Support Vector Machine In Bioinformatics
Application of Support Vector Machine In Bioinformatics V. K. Jayaraman Scientific and Engineering Computing Group CDAC, Pune jayaramanv@cdac.in Arun Gupta Computational Biology Group AbhyudayaTech, Indore
More informationRobot Learning. There are generally three types of robot learning: Learning from data. Learning by demonstration. Reinforcement learning
Robot Learning 1 General Pipeline 1. Data acquisition (e.g., from 3D sensors) 2. Feature extraction and representation construction 3. Robot learning: e.g., classification (recognition) or clustering (knowledge
More informationCSE 573: Artificial Intelligence Autumn 2010
CSE 573: Artificial Intelligence Autumn 2010 Lecture 16: Machine Learning Topics 12/7/2010 Luke Zettlemoyer Most slides over the course adapted from Dan Klein. 1 Announcements Syllabus revised Machine
More informationLecture #11: The Perceptron
Lecture #11: The Perceptron Mat Kallada STAT2450 - Introduction to Data Mining Outline for Today Welcome back! Assignment 3 The Perceptron Learning Method Perceptron Learning Rule Assignment 3 Will be
More informationKernel SVM. Course: Machine Learning MAHDI YAZDIAN-DEHKORDI FALL 2017
Kernel SVM Course: MAHDI YAZDIAN-DEHKORDI FALL 2017 1 Outlines SVM Lagrangian Primal & Dual Problem Non-linear SVM & Kernel SVM SVM Advantages Toolboxes 2 SVM Lagrangian Primal/DualProblem 3 SVM LagrangianPrimalProblem
More informationInstructor: Jessica Wu Harvey Mudd College
The Perceptron Instructor: Jessica Wu Harvey Mudd College The instructor gratefully acknowledges Andrew Ng (Stanford), Eric Eaton (UPenn), David Kauchak (Pomona), and the many others who made their course
More informationAnnouncements. CS 188: Artificial Intelligence Spring Classification: Feature Vectors. Classification: Weights. Learning: Binary Perceptron
CS 188: Artificial Intelligence Spring 2010 Lecture 24: Perceptrons and More! 4/20/2010 Announcements W7 due Thursday [that s your last written for the semester!] Project 5 out Thursday Contest running
More informationClassification: Feature Vectors
Classification: Feature Vectors Hello, Do you want free printr cartriges? Why pay more when you can get them ABSOLUTELY FREE! Just # free YOUR_NAME MISSPELLED FROM_FRIEND... : : : : 2 0 2 0 PIXEL 7,12
More informationCPSC 340: Machine Learning and Data Mining. Probabilistic Classification Fall 2017
CPSC 340: Machine Learning and Data Mining Probabilistic Classification Fall 2017 Admin Assignment 0 is due tonight: you should be almost done. 1 late day to hand it in Monday, 2 late days for Wednesday.
More informationCS 8520: Artificial Intelligence
CS 8520: Artificial Intelligence Machine Learning 2 Paula Matuszek Spring, 2013 1 Regression Classifiers We said earlier that the task of a supervised learning system can be viewed as learning a function
More informationAnnouncements. CS 188: Artificial Intelligence Spring Generative vs. Discriminative. Classification: Feature Vectors. Project 4: due Friday.
CS 188: Artificial Intelligence Spring 2011 Lecture 21: Perceptrons 4/13/2010 Announcements Project 4: due Friday. Final Contest: up and running! Project 5 out! Pieter Abbeel UC Berkeley Many slides adapted
More informationNon-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines
Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2007 c 2007,
More informationLogis&c Regression. Aar$ Singh & Barnabas Poczos. Machine Learning / Jan 28, 2014
Logis&c Regression Aar$ Singh & Barnabas Poczos Machine Learning 10-701/15-781 Jan 28, 2014 Linear Regression & Linear Classifica&on Weight Height Linear fit Linear decision boundary 2 Naïve Bayes Recap
More informationCS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #19: Machine Learning 1
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #19: Machine Learning 1 Supervised Learning Would like to do predicbon: esbmate a func3on f(x) so that y = f(x) Where y can be: Real number:
More informationParallel & Scalable Machine Learning Introduction to Machine Learning Algorithms
Parallel & Scalable Machine Learning Introduction to Machine Learning Algorithms Dr. Ing. Morris Riedel Adjunct Associated Professor School of Engineering and Natural Sciences, University of Iceland Research
More informationSupport vector machine (II): non-linear SVM. LING 572 Fei Xia
Support vector machine (II): non-linear SVM LING 572 Fei Xia 1 Linear SVM Maximizing the margin Soft margin Nonlinear SVM Kernel trick A case study Outline Handling multi-class problems 2 Non-linear SVM
More information1 Case study of SVM (Rob)
DRAFT a final version will be posted shortly COS 424: Interacting with Data Lecturer: Rob Schapire and David Blei Lecture # 8 Scribe: Indraneel Mukherjee March 1, 2007 In the previous lecture we saw how
More informationCS570: Introduction to Data Mining
CS570: Introduction to Data Mining Classification Advanced Reading: Chapter 8 & 9 Han, Chapters 4 & 5 Tan Anca Doloc-Mihu, Ph.D. Slides courtesy of Li Xiong, Ph.D., 2011 Han, Kamber & Pei. Data Mining.
More informationSupport Vector Machines (a brief introduction) Adrian Bevan.
Support Vector Machines (a brief introduction) Adrian Bevan email: a.j.bevan@qmul.ac.uk Outline! Overview:! Introduce the problem and review the various aspects that underpin the SVM concept.! Hard margin
More informationDecision Trees, Random Forests and Random Ferns. Peter Kovesi
Decision Trees, Random Forests and Random Ferns Peter Kovesi What do I want to do? Take an image. Iden9fy the dis9nct regions of stuff in the image. Mark the boundaries of these regions. Recognize and
More informationOptimal Separating Hyperplane and the Support Vector Machine. Volker Tresp Summer 2018
Optimal Separating Hyperplane and the Support Vector Machine Volker Tresp Summer 2018 1 (Vapnik s) Optimal Separating Hyperplane Let s consider a linear classifier with y i { 1, 1} If classes are linearly
More informationClassification by Support Vector Machines
Classification by Support Vector Machines Florian Markowetz Max-Planck-Institute for Molecular Genetics Computational Molecular Biology Berlin Practical DNA Microarray Analysis 2003 1 Overview I II III
More informationLecture 9: Support Vector Machines
Lecture 9: Support Vector Machines William Webber (william@williamwebber.com) COMP90042, 2014, Semester 1, Lecture 8 What we ll learn in this lecture Support Vector Machines (SVMs) a highly robust and
More informationSupport Vector Machines
Support Vector Machines . Importance of SVM SVM is a discriminative method that brings together:. computational learning theory. previously known methods in linear discriminant functions 3. optimization
More informationStructure and Support Vector Machines. SPFLODD October 31, 2013
Structure and Support Vector Machines SPFLODD October 31, 2013 Outline SVMs for structured outputs Declara?ve view Procedural view Warning: Math Ahead Nota?on for Linear Models Training data: {(x 1, y
More informationTerraSwarm. A Machine Learning and Op0miza0on Toolkit for the Swarm. Ilge Akkaya, Shuhei Emoto, Edward A. Lee. University of California, Berkeley
TerraSwarm A Machine Learning and Op0miza0on Toolkit for the Swarm Ilge Akkaya, Shuhei Emoto, Edward A. Lee University of California, Berkeley TerraSwarm Tools Telecon 17 November 2014 Sponsored by the
More informationSUPPORT VECTOR MACHINE ACTIVE LEARNING
SUPPORT VECTOR MACHINE ACTIVE LEARNING CS 101.2 Caltech, 03 Feb 2009 Paper by S. Tong, D. Koller Presented by Krzysztof Chalupka OUTLINE SVM intro Geometric interpretation Primal and dual form Convexity,
More informationMachine Learning Lecture 9
Course Outline Machine Learning Lecture 9 Fundamentals ( weeks) Bayes Decision Theory Probability Density Estimation Nonlinear SVMs 19.05.013 Discriminative Approaches (5 weeks) Linear Discriminant Functions
More informationCSCI 599 Class Presenta/on. Zach Levine. Markov Chain Monte Carlo (MCMC) HMM Parameter Es/mates
CSCI 599 Class Presenta/on Zach Levine Markov Chain Monte Carlo (MCMC) HMM Parameter Es/mates April 26 th, 2012 Topics Covered in this Presenta2on A (Brief) Review of HMMs HMM Parameter Learning Expecta2on-
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview 1. Overview of SVMs 2. Margin Geometry 3. SVM Optimization 4. Overlapping Distributions 5. Relationship to Logistic Regression 6. Dealing
More informationCPSC 340: Machine Learning and Data Mining. More Linear Classifiers Fall 2017
CPSC 340: Machine Learning and Data Mining More Linear Classifiers Fall 2017 Admin Assignment 3: Due Friday of next week. Midterm: Can view your exam during instructor office hours next week, or after
More informationContent-based image and video analysis. Machine learning
Content-based image and video analysis Machine learning for multimedia retrieval 04.05.2009 What is machine learning? Some problems are very hard to solve by writing a computer program by hand Almost all
More informationCS 8520: Artificial Intelligence. Machine Learning 2. Paula Matuszek Fall, CSC 8520 Fall Paula Matuszek
CS 8520: Artificial Intelligence Machine Learning 2 Paula Matuszek Fall, 2015!1 Regression Classifiers We said earlier that the task of a supervised learning system can be viewed as learning a function
More informationPerceptron Learning Algorithm (PLA)
Review: Lecture 4 Perceptron Learning Algorithm (PLA) Learning algorithm for linear threshold functions (LTF) (iterative) Energy function: PLA implements a stochastic gradient algorithm Novikoff s theorem
More informationLinear Models. Lecture Outline: Numeric Prediction: Linear Regression. Linear Classification. The Perceptron. Support Vector Machines
Linear Models Lecture Outline: Numeric Prediction: Linear Regression Linear Classification The Perceptron Support Vector Machines Reading: Chapter 4.6 Witten and Frank, 2nd ed. Chapter 4 of Mitchell Solving
More informationData Mining in Bioinformatics Day 1: Classification
Data Mining in Bioinformatics Day 1: Classification Karsten Borgwardt February 18 to March 1, 2013 Machine Learning & Computational Biology Research Group Max Planck Institute Tübingen and Eberhard Karls
More informationLecture 7: Support Vector Machine
Lecture 7: Support Vector Machine Hien Van Nguyen University of Houston 9/28/2017 Separating hyperplane Red and green dots can be separated by a separating hyperplane Two classes are separable, i.e., each
More informationSupport Vector Machines + Classification for IR
Support Vector Machines + Classification for IR Pierre Lison University of Oslo, Dep. of Informatics INF3800: Søketeknologi April 30, 2014 Outline of the lecture Recap of last week Support Vector Machines
More informationMachine Learning Lecture 9
Course Outline Machine Learning Lecture 9 Fundamentals ( weeks) Bayes Decision Theory Probability Density Estimation Nonlinear SVMs 30.05.016 Discriminative Approaches (5 weeks) Linear Discriminant Functions
More informationKernel Methods & Support Vector Machines
& Support Vector Machines & Support Vector Machines Arvind Visvanathan CSCE 970 Pattern Recognition 1 & Support Vector Machines Question? Draw a single line to separate two classes? 2 & Support Vector
More informationSVM in Analysis of Cross-Sectional Epidemiological Data Dmitriy Fradkin. April 4, 2005 Dmitriy Fradkin, Rutgers University Page 1
SVM in Analysis of Cross-Sectional Epidemiological Data Dmitriy Fradkin April 4, 2005 Dmitriy Fradkin, Rutgers University Page 1 Overview The goals of analyzing cross-sectional data Standard methods used
More informationCS229 Final Project: Predicting Expected Response Times
CS229 Final Project: Predicting Expected Email Response Times Laura Cruz-Albrecht (lcruzalb), Kevin Khieu (kkhieu) December 15, 2017 1 Introduction Each day, countless emails are sent out, yet the time
More informationNeural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer
More informationAn Introduction to Machine Learning
TRIPODS Summer Boorcamp: Topology and Machine Learning August 6, 2018 General Set-up Introduction Set-up and Goal Suppose we have X 1,X 2,...,X n data samples. Can we predict properites about any given
More informationPerceptron Introduction to Machine Learning. Matt Gormley Lecture 5 Jan. 31, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Perceptron Matt Gormley Lecture 5 Jan. 31, 2018 1 Q&A Q: We pick the best hyperparameters
More informationMachine Learning in Biology
Università degli studi di Padova Machine Learning in Biology Luca Silvestrin (Dottorando, XXIII ciclo) Supervised learning Contents Class-conditional probability density Linear and quadratic discriminant
More informationSearch Engines. Information Retrieval in Practice
Search Engines Information Retrieval in Practice All slides Addison Wesley, 2008 Classification and Clustering Classification and clustering are classical pattern recognition / machine learning problems
More informationMachine Learning: Think Big and Parallel
Day 1 Inderjit S. Dhillon Dept of Computer Science UT Austin CS395T: Topics in Multicore Programming Oct 1, 2013 Outline Scikit-learn: Machine Learning in Python Supervised Learning day1 Regression: Least
More informationApplying Supervised Learning
Applying Supervised Learning When to Consider Supervised Learning A supervised learning algorithm takes a known set of input data (the training set) and known responses to the data (output), and trains
More informationChap.12 Kernel methods [Book, Chap.7]
Chap.12 Kernel methods [Book, Chap.7] Neural network methods became popular in the mid to late 1980s, but by the mid to late 1990s, kernel methods have also become popular in machine learning. The first
More informationLogisBcs. CS 6140: Machine Learning Spring K-means Algorithm. Today s Outline 3/27/16
LogisBcs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and InformaBon Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Exam
More informationFeature Selec+on. Machine Learning Fall 2018 Kasthuri Kannan
Feature Selec+on Machine Learning Fall 2018 Kasthuri Kannan Interpretability vs. Predic+on Types of feature selec+on Subset selec+on/forward/backward Shrinkage (Lasso/Ridge) Best model (CV) Feature selec+on
More informationClassification Lecture Notes cse352. Neural Networks. Professor Anita Wasilewska
Classification Lecture Notes cse352 Neural Networks Professor Anita Wasilewska Neural Networks Classification Introduction INPUT: classification data, i.e. it contains an classification (class) attribute
More informationBayes Net Learning. EECS 474 Fall 2016
Bayes Net Learning EECS 474 Fall 2016 Homework Remaining Homework #3 assigned Homework #4 will be about semi-supervised learning and expectation-maximization Homeworks #3-#4: the how of Graphical Models
More informationCS 343H: Honors AI. Lecture 23: Kernels and clustering 4/15/2014. Kristen Grauman UT Austin
CS 343H: Honors AI Lecture 23: Kernels and clustering 4/15/2014 Kristen Grauman UT Austin Slides courtesy of Dan Klein, except where otherwise noted Announcements Office hours Kim s office hours this week:
More informationPreface to the Second Edition. Preface to the First Edition. 1 Introduction 1
Preface to the Second Edition Preface to the First Edition vii xi 1 Introduction 1 2 Overview of Supervised Learning 9 2.1 Introduction... 9 2.2 Variable Types and Terminology... 9 2.3 Two Simple Approaches
More informationInstance-based Learning
Instance-based Learning Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University October 15 th, 2007 2005-2007 Carlos Guestrin 1 1-Nearest Neighbor Four things make a memory based learner:
More information10601 Machine Learning. Model and feature selection
10601 Machine Learning Model and feature selection Model selection issues We have seen some of this before Selecting features (or basis functions) Logistic regression SVMs Selecting parameter value Prior
More informationSearch Engines. Informa1on Retrieval in Prac1ce. Annota1ons by Michael L. Nelson
Search Engines Informa1on Retrieval in Prac1ce Annota1ons by Michael L. Nelson All slides Addison Wesley, 2008 Evalua1on Evalua1on is key to building effec$ve and efficient search engines measurement usually
More informationLearning from Data. COMP61011 : Machine Learning and Data Mining. Dr Gavin Brown Machine Learning and Op<miza<on Research Group
Learning from Data COMP61011 : Machine Learning and Data Mining Dr Gavin Brown Machine Learning and Op
More informationCPSC 340: Machine Learning and Data Mining. Kernel Trick Fall 2017
CPSC 340: Machine Learning and Data Mining Kernel Trick Fall 2017 Admin Assignment 3: Due Friday. Midterm: Can view your exam during instructor office hours or after class this week. Digression: the other
More informationMul$variate classifica$on. Astro 585 Spring 2013
Mul$variate classifica$on Astro 585 Spring 2013 Concepts of classifica$on Here we consider situa9ons where the mul9variate dataset under study represents a new test set that is a mixture of classes that
More information