Introduction to Statistical Learning Theory
|
|
|
- Brittney Flynn
- 5 years ago
- Views:
Transcription
1 Introduction to Statistical Learning Theory Julia Kempe & David S. Rosenberg NYU CDS January 29, 2019 Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
2 Decision Theory: High Level View Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
3 What types of problems are we solving? In data science problems, we generally need to: Make a decision Take an action Produce some output Have some evaluation criterion Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
4 Actions Denition An action is the generic term for what is produced by our system. Examples of Actions Produce a 0/1 classication [classical ML] Reject hypothesis that θ = 0 [classical Statistics] Written English text [image captioning, speech recognition, machine translation ] What's an action for predicting where a storm will be in 3 hours? What's an action for a self-driving car? Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
5 Evaluation Criterion Decision theory is about nding optimal actions, under various denitions of optimality. Examples of Evaluation Criteria Is classication correct? Does text transcription exactly match the spoken words? Should we give partial credit? How? How far is the storm from the prediction location? [for point prediction] How likely is the storm's location under the prediction? [for density prediction] Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
6 Real Life: Formalizing a Business Problem First two steps to formalizing a problem: 1 Dene the action space (i.e. the set of possible actions) 2 Specify the evaluation criterion. Formalization may evolve gradually, as you understand the problem better Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
7 Inputs Most problems have an extra piece, going by various names: Inputs [ML] Covariates [Statistics] Examples of Inputs A picture A storm's historical location and other weather data A search query Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
8 Outcomes or Output or Label Inputs often paired with outputs or outcomes or labels Examples of outcomes/outputs/labels Whether or not the picture actually contains an animal The storm's location one hour after query Which, if any, of suggested the URLs were selected Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
9 Typical Sequence of Events Many problem domains can be formalized as follows: 1 Observe input x. 2 Take action a. 3 Observe outcome y. 4 Evaluate action in relation to the outcome (via a loss functionl(a, y)) Note Outcome y is often independent of action a But this is not always the case: search result ranking automated driving Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
10 Formalization: The Spaces The Three Spaces: Input space: X Action space: A Outcome space: Y Concept check: What are the spaces for linear regression? What are the spaces for logistic regression? What are the spaces for a support vector machine? Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
11 Some Formalization The Spaces X: input space Y: outcome space A: action space Prediction Function (or decision function) A prediction function (or decision function) gets input x X and produces an action a A : Loss Function f : X A x f (x) A loss function evaluates an action in the context of the outcome y. l : A Y R (a,y) l(a,y) Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
12 Evaluating a Prediction Function Loss function l evaluates a single action How to evaluate the prediction function as a whole? We will use the standard statistical learning theory framework. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
13 Statistical Learning Theory Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
14 A Simplifying Assumption Assume action has no eect on the output includes all traditional prediction problems what about stock market prediction? what about stock market investing? What about fancier problems where this does not hold? often can be reformulated or reduced to problems where it does hold see literature on reinforcement learning Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
15 Setup for Statistical Learning Theory Assume there is a data generating distribution P X Y. All input/output pairs (x,y) are generated i.i.d. from P X Y. Want prediction function f (x) that generally does well on average: l(f (x),y) is usually small, in some sense How can we formalize this? Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
16 The Risk Functional Denition The risk of a prediction function f : X A is R(f ) = El(f (x),y). In words, it's the expected loss of f on a new exampe (x,y) drawn randomly from P X Y. Risk function cannot be computed Since we don't know P X Y, we cannot compute the expectation. But we can estimate it... Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
17 The Bayes Prediction Function Denition A Bayes prediction function f : X A is a function that achieves the minimal risk among all possible functions: f arg minr(f ), f where the minimum is taken over all functions from X to A. The risk of a Bayes prediction function is called the Bayes risk. A Bayes prediction function is often called the target function, since it's the best prediction function we can possibly produce. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
18 Example: Least Squares Regression Spaces: A = Y = R Square loss: l(a,y) = (a y) 2 Risk: R(f ) = E [ (f (x) y) 2] (homework = ) = E [ (f (x) E[y x]) 2] + E [ (y E[y x]) 2] So Bayes prediction function is f (x) = E[y x] Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
19 Example 2: Multiclass Classication Spaces: A = Y = {1,...,k} 0-1 loss: Risk: l(a,y) = 1(a y) := { 1 if a y 0 otherwise. R(f ) = E[1(f (x) y)] = 0 P(f (x) = y) + 1 P(f (x) y) = P(f (x) y), which is just the misclassication error rate. Bayes prediction function is just the assignment to the most likely class: f (x) arg maxp(y = c x) 1 c k Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
20 But we can't compute the risk! Can't compute R(f ) = El(f (x),y) because we don't know P X Y. One thing we can do in ML/statistics/data science is assume we have sample data. Let D n = ((x 1,y 1 ),...,(x n,y n )) be drawn i.i.d. from P X Y. Let's draw some inspiration from the Strong Law of Large Numbers: If z,z 1,...,z n are i.i.d. with expected value Ez, then with probability 1. lim n 1 n n z i = Ez, i=1 Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
21 The Empirical Risk Let D n = ((x 1,y 1 ),...,(x n,y n )) be drawn i.i.d. from P X Y. Denition The empirical risk of f : X A with respect to D n is ˆR n (f ) = 1 n By the Strong Law of Large Numbers, almost surely. n l(f (x i ),y i ). i=1 lim ˆR n (f ) = R(f ), n But we want to nd the f that minimizes R(f ) - will minimizing ˆR n (f ) be good enough? Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
22 Empirical Risk Minimization We want risk minimizer, is empirical risk minimizer close enough? Denition A function ˆf is an empirical risk minimizer if ˆf arg min f where the minimum is taken over all functions. ˆR n (f ), Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
23 Empirical Risk Minimization P X = Uniform[0,1], Y 1 (i.e. Y is always 1) y x P X Y. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
24 Empirical Risk Minimization P X = Uniform[0,1], Y 1 (i.e. Y is always 1) y x A sample of size 3 from P X Y. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
25 Empirical Risk Minimization P X = Uniform[0,1], Y 1 (i.e. Y is always 1) y x A proposed prediction function: ˆf (x) = 1(x {0.25,0.5,0.75}) = { 1 if x {0.25,.5,.75} 0 otherwise Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
26 Empirical Risk Minimization P X = Uniform[0,1], Y 1 (i.e. Y is always 1) y x Under square loss or 0/1 loss: ˆf has Empirical Risk = 0 and Risk = 1. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
27 Empirical Risk Minimization ERM led to a function f that just memorized the data. How to spread information or generalize from training inputs to new inputs? Need to smooth things out somehow... A lot of modeling is about spreading and extrapolating information from one part of the input space X into unobserved parts of the space. One approach: Constrained ERM Instead of minimizing empirical risk over all prediction functions, constrain to a particular subset, called a hypothesis space. Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
28 Hypothesis Spaces Denition A hypothesis space F is a set of functions mapping X A. It is the collection of prediction functions we are choosing from. Want Hypothesis Space that... Includes only those functions that have desired regularity e.g. smoothness, simplicity Easy to work with Example hypothesis spaces? Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
29 Constrained Empirical Risk Minimization Hypothesis space F, a set of [prediction] functions mapping X A Empirical risk minimizer (ERM) in F is ˆf n arg min f F Risk minimizer in F is f F F, where f F 1 n n l(f (x i ),y i ). i=1 arg minel(f (x),y). f F Julia Kempe & David S. Rosenberg (NYU CDS) DS-GA 1003 / CSCI-GA 2567 January 29, / 29
Classification and Regression Trees
Classification and Regression Trees David S. Rosenberg New York University April 3, 2018 David S. Rosenberg (New York University) DS-GA 1003 / CSCI-GA 2567 April 3, 2018 1 / 51 Contents 1 Trees 2 Regression
k-means Clustering David S. Rosenberg April 24, 2018 New York University
k-means Clustering David S. Rosenberg New York University April 24, 2018 David S. Rosenberg (New York University) DS-GA 1003 / CSCI-GA 2567 April 24, 2018 1 / 19 Contents 1 k-means Clustering 2 k-means:
Machine Learning (CSE 446): Concepts & the i.i.d. Supervised Learning Paradigm
Machine Learning (CSE 446): Concepts & the i.i.d. Supervised Learning Paradigm Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 17 Review 1 / 17 Decision Tree: Making a
Large Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Machine Learning Basics - 1 U Kang Seoul National University U Kang 1 In This Lecture Overview of Machine Learning Capacity, overfitting, and underfitting
CSCI567 Machine Learning (Fall 2014)
CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 9, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 9, 2014 1 / 47
Applying Supervised Learning
Applying Supervised Learning When to Consider Supervised Learning A supervised learning algorithm takes a known set of input data (the training set) and known responses to the data (output), and trains
7. Boosting and Bagging Bagging
Group Prof. Daniel Cremers 7. Boosting and Bagging Bagging Bagging So far: Boosting as an ensemble learning method, i.e.: a combination of (weak) learners A different way to combine classifiers is known
Client Dependent GMM-SVM Models for Speaker Verification
Client Dependent GMM-SVM Models for Speaker Verification Quan Le, Samy Bengio IDIAP, P.O. Box 592, CH-1920 Martigny, Switzerland {quan,bengio}@idiap.ch Abstract. Generative Gaussian Mixture Models (GMMs)
Bayesian Methods. David Rosenberg. April 11, New York University. David Rosenberg (New York University) DS-GA 1003 April 11, / 19
Bayesian Methods David Rosenberg New York University April 11, 2017 David Rosenberg (New York University) DS-GA 1003 April 11, 2017 1 / 19 Classical Statistics Classical Statistics David Rosenberg (New
4.5 The smoothed bootstrap
4.5. THE SMOOTHED BOOTSTRAP 47 F X i X Figure 4.1: Smoothing the empirical distribution function. 4.5 The smoothed bootstrap In the simple nonparametric bootstrap we have assumed that the empirical distribution
Computer Experiments. Designs
Computer Experiments Designs Differences between physical and computer Recall experiments 1. The code is deterministic. There is no random error (measurement error). As a result, no replication is needed.
CSE 446 Bias-Variance & Naïve Bayes
CSE 446 Bias-Variance & Naïve Bayes Administrative Homework 1 due next week on Friday Good to finish early Homework 2 is out on Monday Check the course calendar Start early (midterm is right before Homework
Machine Learning Techniques for Data Mining
Machine Learning Techniques for Data Mining Eibe Frank University of Waikato New Zealand 10/25/2000 1 PART VII Moving on: Engineering the input and output 10/25/2000 2 Applying a learner is not all Already
Stat 602X Exam 2 Spring 2011
Stat 60X Exam Spring 0 I have neither given nor received unauthorized assistance on this exam. Name Signed Date Name Printed . Below is a small p classification training set (for classes) displayed in
Nearest Neighbor Classification
Nearest Neighbor Classification Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms January 11, 2017 1 / 48 Outline 1 Administration 2 First learning algorithm: Nearest
Data Analysis and Solver Plugins for KSpread USER S MANUAL. Tomasz Maliszewski
Data Analysis and Solver Plugins for KSpread USER S MANUAL Tomasz Maliszewski tmaliszewski@wp.pl Table of Content CHAPTER 1: INTRODUCTION... 3 1.1. ABOUT DATA ANALYSIS PLUGIN... 3 1.3. ABOUT SOLVER PLUGIN...
Introduction to Machine Learning
Introduction to Machine Learning Isabelle Guyon Notes written by: Johann Leithon. Introduction The process of Machine Learning consist of having a big training data base, which is the input to some learning
Computational Machine Learning, Fall 2015 Homework 4: stochastic gradient algorithms
Computational Machine Learning, Fall 2015 Homework 4: stochastic gradient algorithms Due: Tuesday, November 24th, 2015, before 11:59pm (submit via email) Preparation: install the software packages and
Divide and Conquer Kernel Ridge Regression
Divide and Conquer Kernel Ridge Regression Yuchen Zhang John Duchi Martin Wainwright University of California, Berkeley COLT 2013 Yuchen Zhang (UC Berkeley) Divide and Conquer KRR COLT 2013 1 / 15 Problem
Optimization Methods for Machine Learning (OMML)
Optimization Methods for Machine Learning (OMML) 2nd lecture Prof. L. Palagi References: 1. Bishop Pattern Recognition and Machine Learning, Springer, 2006 (Chap 1) 2. V. Cherlassky, F. Mulier - Learning
Transductive Learning: Motivation, Model, Algorithms
Transductive Learning: Motivation, Model, Algorithms Olivier Bousquet Centre de Mathématiques Appliquées Ecole Polytechnique, FRANCE olivier.bousquet@m4x.org University of New Mexico, January 2002 Goal
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University January 22 2019 Outline Practical issues in Linear Regression Outliers Categorical variables Lab
K-Means and Gaussian Mixture Models
K-Means and Gaussian Mixture Models David Rosenberg New York University June 15, 2015 David Rosenberg (New York University) DS-GA 1003 June 15, 2015 1 / 43 K-Means Clustering Example: Old Faithful Geyser
The Offset Tree for Learning with Partial Labels
The Offset Tree for Learning with Partial Labels Alina Beygelzimer IBM Research John Langford Yahoo! Research June 30, 2009 KDD 2009 1 A user with some hidden interests make a query on Yahoo. 2 Yahoo chooses
CS489/698 Lecture 2: January 8 th, 2018
CS489/698 Lecture 2: January 8 th, 2018 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D] Chapt. 3, [B] Sec. 2.5.2, [M] Sec. 1.4.2 CS489/698 (c) 2018 P. Poupart 1 Inductive Learning (recap) Induction
Locally Weighted Learning for Control. Alexander Skoglund Machine Learning Course AASS, June 2005
Locally Weighted Learning for Control Alexander Skoglund Machine Learning Course AASS, June 2005 Outline Locally Weighted Learning, Christopher G. Atkeson et. al. in Artificial Intelligence Review, 11:11-73,1997
What is Learning? CS 343: Artificial Intelligence Machine Learning. Raymond J. Mooney. Problem Solving / Planning / Control.
What is Learning? CS 343: Artificial Intelligence Machine Learning Herbert Simon: Learning is any process by which a system improves performance from experience. What is the task? Classification Problem
Tilings of the Euclidean plane
Tilings of the Euclidean plane Yan Der, Robin, Cécile January 9, 2017 Abstract This document gives a quick overview of a eld of mathematics which lies in the intersection of geometry and algebra : tilings.
Nonparametric regression using kernel and spline methods
Nonparametric regression using kernel and spline methods Jean D. Opsomer F. Jay Breidt March 3, 016 1 The statistical model When applying nonparametric regression methods, the researcher is interested
Machine Learning and Computational Statistics, Spring 2015 Homework 1: Ridge Regression and SGD
Machine Learning and Computational Statistics, Spring 2015 Homework 1: Ridge Regression and SGD Due: Friday, February 6, 2015, at 4pm (Submit via NYU Classes) Instructions: Your answers to the questions
Cellular Tree Classifiers. Gérard Biau & Luc Devroye
Cellular Tree Classifiers Gérard Biau & Luc Devroye Paris, December 2013 Outline 1 Context 2 Cellular tree classifiers 3 A mathematical model 4 Are there consistent cellular tree classifiers? 5 A non-randomized
Statistical topological data analysis using. Persistence landscapes
Motivation Persistence landscape Statistical topological data analysis using persistence landscapes Cleveland State University February 3, 2014 funded by AFOSR Motivation Persistence landscape Topological
Machine Learning and Computational Statistics, Spring 2016 Homework 1: Ridge Regression and SGD
Machine Learning and Computational Statistics, Spring 2016 Homework 1: Ridge Regression and SGD Due: Friday, February 5, 2015, at 6pm (Submit via NYU Classes) Instructions: Your answers to the questions
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University September 18 2018 Logistics HW 1 is on Piazza and Gradescope Deadline: Friday, Sept. 28, 2018 Office
Clustering. Mihaela van der Schaar. January 27, Department of Engineering Science University of Oxford
Department of Engineering Science University of Oxford January 27, 2017 Many datasets consist of multiple heterogeneous subsets. Cluster analysis: Given an unlabelled data, want algorithms that automatically
Inference and Representation
Inference and Representation Rachel Hodos New York University Lecture 5, October 6, 2015 Rachel Hodos Lecture 5: Inference and Representation Today: Learning with hidden variables Outline: Unsupervised
CSC 411 Lecture 4: Ensembles I
CSC 411 Lecture 4: Ensembles I Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 04-Ensembles I 1 / 22 Overview We ve seen two particular classification algorithms:
Nearest neighbors classifiers
Nearest neighbors classifiers James McInerney Adapted from slides by Daniel Hsu Sept 11, 2017 1 / 25 Housekeeping We received 167 HW0 submissions on Gradescope before midnight Sept 10th. From a random
Section 2.3: Simple Linear Regression: Predictions and Inference
Section 2.3: Simple Linear Regression: Predictions and Inference Jared S. Murray The University of Texas at Austin McCombs School of Business Suggested reading: OpenIntro Statistics, Chapter 7.4 1 Simple
Supervised Learning with Neural Networks. We now look at how an agent might learn to solve a general problem by seeing examples.
Supervised Learning with Neural Networks We now look at how an agent might learn to solve a general problem by seeing examples. Aims: to present an outline of supervised learning as part of AI; to introduce
Missing Data Analysis for the Employee Dataset
Missing Data Analysis for the Employee Dataset 67% of the observations have missing values! Modeling Setup Random Variables: Y i =(Y i1,...,y ip ) 0 =(Y i,obs, Y i,miss ) 0 R i =(R i1,...,r ip ) 0 ( 1
Generative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 2014-2015 Jakob Verbeek, November 28, 2014 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.14.15
Weka ( )
Weka ( http://www.cs.waikato.ac.nz/ml/weka/ ) The phases in which classifier s design can be divided are reflected in WEKA s Explorer structure: Data pre-processing (filtering) and representation Supervised
Nonparametric Methods Recap
Nonparametric Methods Recap Aarti Singh Machine Learning 10-701/15-781 Oct 4, 2010 Nonparametric Methods Kernel Density estimate (also Histogram) Weighted frequency Classification - K-NN Classifier Majority
ECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Sept 22, 2016 Course Information Website: http://www.stat.ucdavis.edu/~chohsieh/teaching/ ECS289G_Fall2016/main.html My office: Mathematical Sciences
Perceptron Introduction to Machine Learning. Matt Gormley Lecture 5 Jan. 31, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Perceptron Matt Gormley Lecture 5 Jan. 31, 2018 1 Q&A Q: We pick the best hyperparameters
Statistical Matching using Fractional Imputation
Statistical Matching using Fractional Imputation Jae-Kwang Kim 1 Iowa State University 1 Joint work with Emily Berg and Taesung Park 1 Introduction 2 Classical Approaches 3 Proposed method 4 Application:
Solution Sketches Midterm Exam COSC 6342 Machine Learning March 20, 2013
Your Name: Your student id: Solution Sketches Midterm Exam COSC 6342 Machine Learning March 20, 2013 Problem 1 [5+?]: Hypothesis Classes Problem 2 [8]: Losses and Risks Problem 3 [11]: Model Generation
Pattern Recognition. Kjell Elenius. Speech, Music and Hearing KTH. March 29, 2007 Speech recognition
Pattern Recognition Kjell Elenius Speech, Music and Hearing KTH March 29, 2007 Speech recognition 2007 1 Ch 4. Pattern Recognition 1(3) Bayes Decision Theory Minimum-Error-Rate Decision Rules Discriminant
Risk bounds for some classification and regression models that interpolate
Risk bounds for some classification and regression models that interpolate Daniel Hsu Columbia University Joint work with: Misha Belkin (The Ohio State University) Partha Mitra (Cold Spring Harbor Laboratory)
Machine Learning Lecture-1
Machine Learning Lecture-1 Programming Club Indian Institute of Technology Kanpur pclubiitk@gmail.com February 25, 2016 Programming Club (IIT Kanpur) ML-1 February 25, 2016 1 / 18 Acknowledgement This
Overview of various smoothers
Chapter 2 Overview of various smoothers A scatter plot smoother is a tool for finding structure in a scatter plot: Figure 2.1: CD4 cell count since seroconversion for HIV infected men. CD4 counts vs Time
Chapter 3. Bootstrap. 3.1 Introduction. 3.2 The general idea
Chapter 3 Bootstrap 3.1 Introduction The estimation of parameters in probability distributions is a basic problem in statistics that one tends to encounter already during the very first course on the subject.
Expectation Maximization (EM) and Gaussian Mixture Models
Expectation Maximization (EM) and Gaussian Mixture Models Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer 1 2 3 4 5 6 7 8 Unsupervised Learning Motivation
Model Selection and Assessment
Model Selection and Assessment CS4780/5780 Machine Learning Fall 2014 Thorsten Joachims Cornell University Reading: Mitchell Chapter 5 Dietterich, T. G., (1998). Approximate Statistical Tests for Comparing
Big Data Methods. Chapter 5: Machine learning. Big Data Methods, Chapter 5, Slide 1
Big Data Methods Chapter 5: Machine learning Big Data Methods, Chapter 5, Slide 1 5.1 Introduction to machine learning What is machine learning? Concerned with the study and development of algorithms that
CSCI567 Machine Learning (Fall 2018)
CSCI567 Machine Learning (Fall 2018) Prof. Haipeng Luo U of Southern California Aug 22, 2018 August 22, 2018 1 / 63 Outline 1 About this course 2 Overview of machine learning 3 Nearest Neighbor Classifier
CSCI 599: Applications of Natural Language Processing Information Retrieval Retrieval Models (Part 1)"
CSCI 599: Applications of Natural Language Processing Information Retrieval Retrieval Models (Part 1)" All slides Addison Wesley, Donald Metzler, and Anton Leuski, 2008, 2012! Retrieval Models" Provide
Slides for Data Mining by I. H. Witten and E. Frank
Slides for Data Mining by I. H. Witten and E. Frank 7 Engineering the input and output Attribute selection Scheme-independent, scheme-specific Attribute discretization Unsupervised, supervised, error-
Geometry I Can Statements I can describe the undefined terms: point, line, and distance along a line in a plane I can describe the undefined terms:
Geometry I Can Statements I can describe the undefined terms: point, line, and distance along a line in a plane I can describe the undefined terms: point, line, and distance along a line in a plane I can
Multi-label classification using rule-based classifier systems
Multi-label classification using rule-based classifier systems Shabnam Nazmi (PhD candidate) Department of electrical and computer engineering North Carolina A&T state university Advisor: Dr. A. Homaifar
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University January 24 2019 Logistics HW 1 is due on Friday 01/25 Project proposal: due Feb 21 1 page description
Homework. Gaussian, Bishop 2.3 Non-parametric, Bishop 2.5 Linear regression Pod-cast lecture on-line. Next lectures:
Homework Gaussian, Bishop 2.3 Non-parametric, Bishop 2.5 Linear regression 3.0-3.2 Pod-cast lecture on-line Next lectures: I posted a rough plan. It is flexible though so please come with suggestions Bayes
CSEP 573: Artificial Intelligence
CSEP 573: Artificial Intelligence Machine Learning: Perceptron Ali Farhadi Many slides over the course adapted from Luke Zettlemoyer and Dan Klein. 1 Generative vs. Discriminative Generative classifiers:
k-nearest Neighbor (knn) Sept Youn-Hee Han
k-nearest Neighbor (knn) Sept. 2015 Youn-Hee Han http://link.koreatech.ac.kr ²Eager Learners Eager vs. Lazy Learning when given a set of training data, it will construct a generalization model before receiving
Applying Machine Learning to Real Problems: Why is it Difficult? How Research Can Help?
Applying Machine Learning to Real Problems: Why is it Difficult? How Research Can Help? Olivier Bousquet, Google, Zürich, obousquet@google.com June 4th, 2007 Outline 1 Introduction 2 Features 3 Minimax
Machine Learning. Classification
10-701 Machine Learning Classification Inputs Inputs Inputs Where we are Density Estimator Probability Classifier Predict category Today Regressor Predict real no. Later Classification Assume we want to
Machine Learning / Jan 27, 2010
Revisiting Logistic Regression & Naïve Bayes Aarti Singh Machine Learning 10-701/15-781 Jan 27, 2010 Generative and Discriminative Classifiers Training classifiers involves learning a mapping f: X -> Y,
http://www.xkcd.com/233/ Text Clustering David Kauchak cs160 Fall 2009 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture17-clustering.ppt Administrative 2 nd status reports Paper review
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University September 20 2018 Review Solution for multiple linear regression can be computed in closed form
Regularization and model selection
CS229 Lecture notes Andrew Ng Part VI Regularization and model selection Suppose we are trying select among several different models for a learning problem. For instance, we might be using a polynomial
DATA ANALYSIS USING HIERARCHICAL GENERALIZED LINEAR MODELS WITH R
DATA ANALYSIS USING HIERARCHICAL GENERALIZED LINEAR MODELS WITH R Lee, Rönnegård & Noh LRN@du.se Lee, Rönnegård & Noh HGLM book 1 / 24 Overview 1 Background to the book 2 Crack growth example 3 Contents
Week 7 Picturing Network. Vahe and Bethany
Week 7 Picturing Network Vahe and Bethany Freeman (2005) - Graphic Techniques for Exploring Social Network Data The two main goals of analyzing social network data are identification of cohesive groups
Data Mining. ❷Chapter 2 Basic Statistics. Asso.Prof.Dr. Xiao-dong Zhu. Business School, University of Shanghai for Science & Technology
❷Chapter 2 Basic Statistics Business School, University of Shanghai for Science & Technology 2016-2017 2nd Semester, Spring2017 Contents of chapter 1 1 recording data using computers 2 3 4 5 6 some famous
How Learning Differs from Optimization. Sargur N. Srihari
How Learning Differs from Optimization Sargur N. srihari@cedar.buffalo.edu 1 Topics in Optimization Optimization for Training Deep Models: Overview How learning differs from optimization Risk, empirical
Data Analytics for. Transmission Expansion Planning. Andrés Ramos. January Estadística II. Transmission Expansion Planning GITI/GITT
Data Analytics for Andrés Ramos January 2018 1 1 Introduction 2 Definition Determine which lines and transformers and when to build optimizing total investment and operation costs 3 Challenges for TEP
Supervised Learning: The Setup. Spring 2018
Supervised Learning: The Setup Spring 2018 1 Homework 0 will be released today through Canvas Due: Jan. 19 (next Friday) midnight 2 Last lecture We saw What is learning? Learning as generalization The
GAMs semi-parametric GLMs. Simon Wood Mathematical Sciences, University of Bath, U.K.
GAMs semi-parametric GLMs Simon Wood Mathematical Sciences, University of Bath, U.K. Generalized linear models, GLM 1. A GLM models a univariate response, y i as g{e(y i )} = X i β where y i Exponential
Multi-label Classification. Jingzhou Liu Dec
Multi-label Classification Jingzhou Liu Dec. 6 2016 Introduction Multi-class problem, Training data (x $, y $ ) ( ), x $ X R., y $ Y = 1,2,, L Learn a mapping f: X Y Each instance x $ is associated with
Unsupervised Learning and Clustering
Unsupervised Learning and Clustering Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2009 CS 551, Spring 2009 c 2009, Selim Aksoy (Bilkent University)
Using Machine Learning to Optimize Storage Systems
Using Machine Learning to Optimize Storage Systems Dr. Kiran Gunnam 1 Outline 1. Overview 2. Building Flash Models using Logistic Regression. 3. Storage Object classification 4. Storage Allocation recommendation
Density estimation. In density estimation problems, we are given a random from an unknown density. Our objective is to estimate
Density estimation In density estimation problems, we are given a random sample from an unknown density Our objective is to estimate? Applications Classification If we estimate the density for each class,
Classification: Linear Discriminant Functions
Classification: Linear Discriminant Functions CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Discriminant functions Linear Discriminant functions
The exam is closed book, closed notes except your one-page cheat sheet.
CS 189 Fall 2015 Introduction to Machine Learning Final Please do not turn over the page before you are instructed to do so. You have 2 hours and 50 minutes. Please write your initials on the top-right
Introduction to Data Mining
Introduction to Data Mining Privacy preserving data mining Li Xiong Slides credits: Chris Clifton Agrawal and Srikant 4/3/2011 1 Privacy Preserving Data Mining Privacy concerns about personal data AOL
6.856 Randomized Algorithms
6.856 Randomized Algorithms David Karger Handout #4, September 21, 2002 Homework 1 Solutions Problem 1 MR 1.8. (a) The min-cut algorithm given in class works because at each step it is very unlikely (probability
Lecture 6-Decision Tree & MDL
6-Decision Tree & MDL-1 Machine Learning Lecture 6-Decision Tree & MDL Lecturer: Haim Permuter Scribes: Asaf Lavi and Ben Marinberg This lecture discusses decision trees and the minimum description length
MCMC Diagnostics. Yingbo Li MATH Clemson University. Yingbo Li (Clemson) MCMC Diagnostics MATH / 24
MCMC Diagnostics Yingbo Li Clemson University MATH 9810 Yingbo Li (Clemson) MCMC Diagnostics MATH 9810 1 / 24 Convergence to Posterior Distribution Theory proves that if a Gibbs sampler iterates enough,
The Law of Reflection *
OpenStax-CNX module: m42456 1 The Law of Reflection * OpenStax This work is produced by OpenStax-CNX and licensed under the Creative Commons Attribution License 3.0 Abstract Explain reection of light from
J. Weston, A. Gammerman, M. Stitson, V. Vapnik, V. Vovk, C. Watkins. Technical Report. February 5, 1998
Density Estimation using Support Vector Machines J. Weston, A. Gammerman, M. Stitson, V. Vapnik, V. Vovk, C. Watkins. Technical Report CSD-TR-97-3 February 5, 998!()+, -./ 3456 Department of Computer Science
Preprocessing, Multiphase Inference, and Massive Data in Theory and Practice
Preprocessing, Multiphase Inference, and Massive Data in Theory and Practice Alexander W Blocker Department of Statistics Harvard University MMDS 2012 July 13, 2012 Joint work with Xiao-Li Meng Alexander
Department of Electrical Engineering, Keio University Hiyoshi Kouhoku-ku Yokohama 223, Japan
Shape Modeling from Multiple View Images Using GAs Satoshi KIRIHARA and Hideo SAITO Department of Electrical Engineering, Keio University 3-14-1 Hiyoshi Kouhoku-ku Yokohama 223, Japan TEL +81-45-563-1141
Chapter 5. Understanding and Comparing Distributions. Copyright 2010, 2007, 2004 Pearson Education, Inc.
Chapter 5 Understanding and Comparing Distributions The Big Picture We can answer much more interesting questions about variables when we compare distributions for different groups. Below is a histogram
The Comparative Study of Machine Learning Algorithms in Text Data Classification*
The Comparative Study of Machine Learning Algorithms in Text Data Classification* Wang Xin School of Science, Beijing Information Science and Technology University Beijing, China Abstract Classification
Name Section Date. Experiment Reflection and Refraction
Name Section Date Introduction: Experiment Reflection and Refraction The travel of light is often represented in geometric optics by a light ray, a line that is drawn to represent the straight-line movement
NaysEddy ver 1.0. Example MANUAL. By: Mohamed Nabi, Ph.D. Copyright 2014 iric Project. All Rights Reserved.
NaysEddy ver 1.0 Example MANUAL By: Mohamed Nabi, Ph.D. Copyright 2014 iric Project. All Rights Reserved. Contents Introduction... 3 Getting started... 4 Simulation of flow over dunes... 6 1. Purpose of
The Optimized Physical Model for Real Rover Vehicle
모 The Optimized Physical Model for Real Rover Vehicle Jun-young Kwak The Robotics Institute Carnegie Mellon University Pittsburgh, PA 15213 junyoung.kwak@cs.cmu.edu Abstract This paper presents the way
Note Set 4: Finite Mixture Models and the EM Algorithm
Note Set 4: Finite Mixture Models and the EM Algorithm Padhraic Smyth, Department of Computer Science University of California, Irvine Finite Mixture Models A finite mixture model with K components, for
Learning in Medical Image Databases. Cristian Sminchisescu. Department of Computer Science. Rutgers University, NJ
Learning in Medical Image Databases Cristian Sminchisescu Department of Computer Science Rutgers University, NJ 08854 email: crismin@paul.rutgers.edu December, 998 Abstract In this paper we present several
Announcements. CS 188: Artificial Intelligence Spring Generative vs. Discriminative. Classification: Feature Vectors. Project 4: due Friday.
CS 188: Artificial Intelligence Spring 2011 Lecture 21: Perceptrons 4/13/2010 Announcements Project 4: due Friday. Final Contest: up and running! Project 5 out! Pieter Abbeel UC Berkeley Many slides adapted