Conditional Random Fields. Mike Brodie CS 778
|
|
|
- Amy Marcia Grant
- 6 years ago
- Views:
Transcription
1 Conditional Random Fields Mike Brodie CS 778
2 Motivation Part-Of-Speech Tagger 2
3 Motivation object 3
4 Motivation I object! 4
5 Motivation object Do you see that object? 5
6 Motivation Part-Of-Speech Tagger - Java CRFTagger INPUT FILE - Logan.txt Logan woke up this morning and ate a big bowl of Fruit Loops. On his way to school, a small dog chased after him. Fortunately, Logan's leg had healed and he outran the dog. 6
7 Motivation Part-Of-Speech Tagger - Java CRFTagger OUTPUT FILE - Logan.txt.pos Logan/NNP woke/vbd up/rp this/dt morning/nn and/cc ate/vb a/dt big/jj bowl/nn of/in Fruit/ NNP Loops/NNP./. On/IN his/prp$ way/nn to/ TO school/vb,/, a/dt small/jj dog/nn chased/ VBN after/in him/prp./. Fortunately/RB,/, Logan/ NNP 's/pos leg/nn had/vbd healed/vbn and/ CC he/prp outran/vb the/dt dog/nn./. 7
8 Motivation Stanford Named Entity Recognition Recognizes names, places, organizations 8
9 Motivation Image Applications Segmentation Multi-Label Classification Labels: Hat, Shirt Cup, Stanley 9
10 Motivation Bioinformatics Applications RNA Structural Alignment Protein Structure Prediction 10
11 Motivation Cupcakes with Rainbow Filling 11
12 Background Isn't it about time 12
13 Background Isn't it about time for Graphical Models? 13 Represent distribution as a product of local functions that depend on a smaller subset of variables
14 Background Generative vs Discriminative Generative models describe how a label vector y can probabilistically generate a feature vector x. Discriminative models describe how to take a feature vector x and assign it a label y. Either type of model can be converted to the other type using Bayes rule P (x y)p (y) =P (x, y) =P (y x)p (x) 14
15 Background Joint Distributions P (x, y) Problems:? 15
16 Background Joint Distributions P (x, y) Problems: - Modeling inputs => intractable models - Ignoring inputs => poor performance 16
17 Background Joint Distributions P (x, y) Naive Bayes Generative model Strong assumptions to simplify computation 17
18 Background Generative vs Discriminative Naive Bayes Logistic Regression Conditional 18
19 Background Generative vs Discriminative Naive Bayes Logistic Regression Conditional If trained to maximize conditional likelihood 19
20 Background Generative vs Discriminative Naive Bayes Logistic Regression Conditional If trained to maximize joint likelihood 20
21 Background Sequence Labeling Hidden Markov Model (HMM) 21
22 Background Sequence Labeling Hidden Markov Model (HMM) ASSUMPTIONS: y t? y 1,...,y t 2 y t 1 x t? Y \{y t },X \{x t } y t
23 Background Sequence Labeling Hidden Markov Model (HMM) PROBLEMS: - Later labels cannot influence previous labels - Cannot represent overlapping features 23
24 Background Improvements to HMMs Maximum Entropy Markov Model 24
25 Background Improvements to HMMs Maximum Entropy Markov Model St-1 St MEMM Ot P (S 1,...,S n O 1,...,O n )= 25 nx t=1 P (S t S t 1,O t )
26 Background Improvements to HMMs Conditional Markov Model PROS: More flexible form of context dependence Independence assumptions among labels, not observations CONS: Only models log-linear distribution over component transition distributions
27 Background Improvements to HMMs Label Bias Problem 27
28 Solution Conditional Random Fields - Drop Local Normalization Constant - Normalize GLOBALLY Over Full Graph Avoids Label Bias Problem 28
29 Solution Conditional Random Fields Calculate path probability by normalizing the path score by the sum of the scores of all possible paths Poorly matching path -> low score (probability) Well-matching path -> high score 29
30 Solution Conditional Random Fields Similar to CMM - Discriminative - Model Conditional Distribution p(y x) - Allow arbitrary, overlapping features 30
31 Solution Conditional Random Fields Similar to CMM - Discriminative - Model Conditional Distribution p(y x) - Allow arbitrary, overlapping features TAKEAWAY: Retains advantages of CMM over HMM Overcomes label bias problem of CMM and MEMM 31
32 Linear-Chain CRFs Conditional Random Fields Remember the HMM 32
33 Linear-Chain CRFs Conditional Random Fields Remember the HMM
34 Linear-Chain CRFs Conditional Random Fields By expanding Z, we have the conditional distribution p(y x) 34
35 Feature Functions students.cs.byu.edu/~mbrodie/778 35
36 36
37 Linear-Chain CRFs Conditional Random Fields 37
38 Feature Engineering Larger feature set benefits: Greater prediction accuracy More flexible decision boundary However, may lead to overfitting 38
39 Feature Engineering Label-Observation Features Where is an observation function rather than a specific word value For example: word word x t x t is capitalized ends in ing 39
40 Feature Engineering Unsupported Features For example: word x t is with and label is NOUN Greatly increases the number of parameters. y t 40
41 Feature Engineering Unsupported Features For example: word x t is with and label is NOUN Greatly increases the number of parameters. However, usually gives a slight boost in accuracy y t Can be useful -> Assign negative weights to prevent spurious labels from receiving high probability 41
42 Feature Engineering Boundary Labels Boundary labels (start/end of sentence, edge of image) can have different characteristics from other labels. For example: Capitalization in the middle of the sentence generally indicates a proper noun (but not necessarily at the beginning of a sentence). 42
43 Feature Engineering Other Methods Feature Induction Begin with a number of base features. Gradually add conjunctions to those features Normalize Features Convert categorical to binary features Features from Different Time Steps 43
44 Feature Engineering Other Methods Features as Model Combination Input Dependent Structure e.g. Skip-Chain CRFs Encourage identical words in a sentence to have the same label Do this by adding an edge in the graph 44
45 Linear-Chain CRFs Conditional Random Fields HMM-like CRF Previous observation affects transition 45
46 Feature Effects Pseudo Weighted Functions on the Brown Corpus: 38 Training, 12 Test Sentences Features Accuracy # Feature Functions yt-1 and yt xt and yt xt and yt AND xt and yt-1and yt xt and yt AND yt-1 and yt
47 Linear-Chain CRFs Parameter Estimation We want to find parameters for feature functions: 47
48 Linear-Chain CRFs Parameter Estimation We want to find parameters for feature functions: Maximize the conditional log likelihood: 48
49 Linear-Chain CRFs Parameter Estimation 49
50 Linear-Chain CRFs Parameter Estimation Which becomes 50
51 Linear-Chain CRFs Parameter Estimation Which becomes Penalize large weight vectors
52 Linear-Chain CRFs Parameter Estimation Cannot maximize `( ) in closed form Instead use numerical optimization => Find derivatives w.r.t. k 52
53 Linear-Chain CRFs Parameter Estimation Cannot maximize `( ) in closed form Instead use numerical optimization => Find derivatives w.r.t. k 53
54 Linear-Chain CRFs Parameter Estimation Cannot maximize `( ) in closed form Instead use numerical optimization => Find derivatives w.r.t. k 54
55 Linear-Chain CRFs Parameter Estimation Cannot maximize `( ) in closed form Instead use numerical optimization => Find derivatives w.r.t. k 55
56 Linear-Chain CRFs Parameter Estimation How to optimize `( )? Steepest ascent along Gradient Newton s Method (L-) BFGS 56
57 Linear-Chain CRFs Parameter Estimation How to optimize `( )? Steepest ascent along Gradient Newton s Method (L-) BFGS 57
58 Linear-Chain CRFs Parameter Estimation How to optimize `( )? Steepest ascent along Gradient Newton s Method (L-) BFGS 58
59 Linear-Chain CRFs Parameter Estimation How to optimize `( )? Steepest ascent along Gradient Newton s Method (L-) BFGS 59
60 Feature Effects Pseudo Weighted Functions on the Brown Corpus: 38 Training, 12 Test Sentences Features Accuracy # Feature Functions Trained Accuracy 10 Epochs yt-1 and yt xt and yt xt and yt AND xt and yt-1and yt xt and yt AND yt-1 and yt
61 Linear-Chain CRFs Inference During Training Marginal Distributions for each edge During Testing Viterbi Algorithm to compute the most likely labeling Compute Z(x) 61
62 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: 62
63 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: Factor Definition: 63
64 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: Factor Definition: Rewrite Using Distributive Law:
65 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: This leads to a set of forward variables 65
66 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: This leads to a set of forward variables Compute t by recursion
67 Linear-Chain CRFs Inference Use Forward-Backward Approach in HMM: Similarly, we computer a set of backward variables Compute t by recursion 67
68 Linear-Chain CRFs Inference Compute Marginals Needed for Gradient Using Forward and Backward Results 68
69 Linear-Chain CRFs Inference Compute Marginals Needed for Gradient Using Forward and Backward Results = 69
70 Linear-Chain CRFs Inference How does this compare to CRF training/inference? 70
71 Linear-Chain CRFs Inference How does this compare to CRF training/inference? 71
72 Linear-Chain CRFs Inference How does this compare to CRF training/inference? 72
73 Breakthrough 73
74 My Implementation 3 Different Implementations 1. Tensorflow 74
75 Tensorflow Tips Do NOT start with CRFs
76 Tensorflow Tips Be prepared for awkward, sometimes unintuitive formats 76
77 Tensorflow Gaaaaah!! len(x) => X.get_shape().dims[0].value Start out with Theano More widely used More intuitive 77
78 My Implementation 3 Different Implementations 1. Tensorflow 2. Scipy Optimization (fmin_l_bfgs_b) 78
79 My Implementation 3 Different Implementations 1. Tensorflow 2. Scipy Optimization (fmin_l_bfgs_b) 3. Gradient Descent A. Trainable Version B. Simplified, Web-based Version 79
80 Experiments Data Sets Brown Corpus News Articles Penn Treebank Corpus WSJ 5,000 POS Sentences 27,596 Tags 75/25 Train-Test Split 3,914 sentence SAMPLE $1,700 (or free BYU version) 75/25 Train-Test Split 80
81 Experiments Comparison Models Python Natural Language Toolkit (NLTK) Hidden Markov Model Conditional Random Field Averaged-Multilayer Perceptron Average the weight updates -> prevent radical changes due to different training batches 81
82 Experiments Brown Corpus Results Accuracy HMM Avg-MLP NLTK-CRF Brodie-CRF 82
83 Experiments Penn Corpus Results Accuracy ? 75 HMM Avg-MLP NLTK-CRF Brodie-CRF 83
84 Pros/Cons of Implementation Pros Can use forward/ backward and Viterbi Algorithm Learn weights for features Cons Slow Limited to gradient descent training 84
85 Extensions General CRFs Skip-chain CRFs Hidden Node CRFs 85
86 CRF-RNN + CNN Problem: Traditional CNNs produce coarse outputs for pixel-level labels. Solution: Apply CRF as a post-processing refinement
87 BI-LSTM-CRF
88 Future Directions New model combinations of CRFs Better training approximations (i.e. besides MCMC sampling methods) Additional hidden architectures 88
89 Bibliography 89
90 Bibliography 90
91 Bibliography Other Resources Tom Mitchell s Machine Learning. Online draft version of Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression. NBayesLogReg.pdf. Accessed January 25, Edwin Chen Blog. Introduction to Conditional Random Fields. Accessed February 2, LingPipe Tutorial: tutorial/crf/read-me.html. Accesed February 2,
92 Thank You Questions? 92
Motivation: Shortcomings of Hidden Markov Model. Ko, Youngjoong. Solution: Maximum Entropy Markov Model (MEMM)
Motivation: Shortcomings of Hidden Markov Model Maximum Entropy Markov Models and Conditional Random Fields Ko, Youngjoong Dept. of Computer Engineering, Dong-A University Intelligent System Laboratory,
Conditional Random Fields - A probabilistic graphical model. Yen-Chin Lee 指導老師 : 鮑興國
Conditional Random Fields - A probabilistic graphical model Yen-Chin Lee 指導老師 : 鮑興國 Outline Labeling sequence data problem Introduction conditional random field (CRF) Different views on building a conditional
Conditional Random Fields and beyond D A N I E L K H A S H A B I C S U I U C,
Conditional Random Fields and beyond D A N I E L K H A S H A B I C S 5 4 6 U I U C, 2 0 1 3 Outline Modeling Inference Training Applications Outline Modeling Problem definition Discriminative vs. Generative
Structured Learning. Jun Zhu
Structured Learning Jun Zhu Supervised learning Given a set of I.I.D. training samples Learn a prediction function b r a c e Supervised learning (cont d) Many different choices Logistic Regression Maximum
Structured Perceptron. Ye Qiu, Xinghui Lu, Yue Lu, Ruofei Shen
Structured Perceptron Ye Qiu, Xinghui Lu, Yue Lu, Ruofei Shen 1 Outline 1. 2. 3. 4. Brief review of perceptron Structured Perceptron Discriminative Training Methods for Hidden Markov Models: Theory and
Shallow Parsing Swapnil Chaudhari 11305R011 Ankur Aher Raj Dabre 11305R001
Shallow Parsing Swapnil Chaudhari 11305R011 Ankur Aher - 113059006 Raj Dabre 11305R001 Purpose of the Seminar To emphasize on the need for Shallow Parsing. To impart basic information about techniques
CS 6784 Paper Presentation
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data John La erty, Andrew McCallum, Fernando C. N. Pereira February 20, 2014 Main Contributions Main Contribution Summary
Modeling Sequence Data
Modeling Sequence Data CS4780/5780 Machine Learning Fall 2011 Thorsten Joachims Cornell University Reading: Manning/Schuetze, Sections 9.1-9.3 (except 9.3.1) Leeds Online HMM Tutorial (except Forward and
The Perceptron. Simon Šuster, University of Groningen. Course Learning from data November 18, 2013
The Perceptron Simon Šuster, University of Groningen Course Learning from data November 18, 2013 References Hal Daumé III: A Course in Machine Learning http://ciml.info Tom M. Mitchell: Machine Learning
Sequence Labeling: The Problem
Sequence Labeling: The Problem Given a sequence (in NLP, words), assign appropriate labels to each word. For example, POS tagging: DT NN VBD IN DT NN. The cat sat on the mat. 36 part-of-speech tags used
Recurrent Neural Networks. Nand Kishore, Audrey Huang, Rohan Batra
Recurrent Neural Networks Nand Kishore, Audrey Huang, Rohan Batra Roadmap Issues Motivation 1 Application 1: Sequence Level Training 2 Basic Structure 3 4 Variations 5 Application 3: Image Classification
Machine Learning. Chao Lan
Machine Learning Chao Lan Machine Learning Prediction Models Regression Model - linear regression (least square, ridge regression, Lasso) Classification Model - naive Bayes, logistic regression, Gaussian
Hidden Markov Models. Natural Language Processing: Jordan Boyd-Graber. University of Colorado Boulder LECTURE 20. Adapted from material by Ray Mooney
Hidden Markov Models Natural Language Processing: Jordan Boyd-Graber University of Colorado Boulder LECTURE 20 Adapted from material by Ray Mooney Natural Language Processing: Jordan Boyd-Graber Boulder
Hidden Markov Models. Gabriela Tavares and Juri Minxha Mentor: Taehwan Kim CS159 04/25/2017
Hidden Markov Models Gabriela Tavares and Juri Minxha Mentor: Taehwan Kim CS159 04/25/2017 1 Outline 1. 2. 3. 4. Brief review of HMMs Hidden Markov Support Vector Machines Large Margin Hidden Markov Models
Log- linear models. Natural Language Processing: Lecture Kairit Sirts
Log- linear models Natural Language Processing: Lecture 3 21.09.2017 Kairit Sirts The goal of today s lecture Introduce the log- linear/maximum entropy model Explain the model components: features, parameters,
D-Separation. b) the arrows meet head-to-head at the node, and neither the node, nor any of its descendants, are in the set C.
D-Separation Say: A, B, and C are non-intersecting subsets of nodes in a directed graph. A path from A to B is blocked by C if it contains a node such that either a) the arrows on the path meet either
School of Computing and Information Systems The University of Melbourne COMP90042 WEB SEARCH AND TEXT ANALYSIS (Semester 1, 2017)
Discussion School of Computing and Information Systems The University of Melbourne COMP9004 WEB SEARCH AND TEXT ANALYSIS (Semester, 07). What is a POS tag? Sample solutions for discussion exercises: Week
Conditional Random Fields : Theory and Application
Conditional Random Fields : Theory and Application Matt Seigel (mss46@cam.ac.uk) 3 June 2010 Cambridge University Engineering Department Outline The Sequence Classification Problem Linear Chain CRFs CRF
JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS. Puyang Xu, Ruhi Sarikaya. Microsoft Corporation
JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS Puyang Xu, Ruhi Sarikaya Microsoft Corporation ABSTRACT We describe a joint model for intent detection and slot filling based
MEMMs (Log-Linear Tagging Models)
Chapter 8 MEMMs (Log-Linear Tagging Models) 8.1 Introduction In this chapter we return to the problem of tagging. We previously described hidden Markov models (HMMs) for tagging problems. This chapter
Introduction to Hidden Markov models
1/38 Introduction to Hidden Markov models Mark Johnson Macquarie University September 17, 2014 2/38 Outline Sequence labelling Hidden Markov Models Finding the most probable label sequence Higher-order
COMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2016 Lecturer: Trevor Cohn 20. PGM Representation Next Lectures Representation of joint distributions Conditional/marginal independence * Directed vs
A Systematic Overview of Data Mining Algorithms
A Systematic Overview of Data Mining Algorithms 1 Data Mining Algorithm A well-defined procedure that takes data as input and produces output as models or patterns well-defined: precisely encoded as a
CSC 578 Neural Networks and Deep Learning
CSC 578 Neural Networks and Deep Learning Fall 2018/19 7. Recurrent Neural Networks (Some figures adapted from NNDL book) 1 Recurrent Neural Networks 1. Recurrent Neural Networks (RNNs) 2. RNN Training
Partitioning Data. IRDS: Evaluation, Debugging, and Diagnostics. Cross-Validation. Cross-Validation for parameter tuning
Partitioning Data IRDS: Evaluation, Debugging, and Diagnostics Charles Sutton University of Edinburgh Training Validation Test Training : Running learning algorithms Validation : Tuning parameters of learning
Structured Prediction Basics
CS11-747 Neural Networks for NLP Structured Prediction Basics Graham Neubig Site https://phontron.com/class/nn4nlp2017/ A Prediction Problem I hate this movie I love this movie very good good neutral bad
A Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
Discriminative Training with Perceptron Algorithm for POS Tagging Task
Discriminative Training with Perceptron Algorithm for POS Tagging Task Mahsa Yarmohammadi Center for Spoken Language Understanding Oregon Health & Science University Portland, Oregon yarmoham@ohsu.edu
Statistical Methods for NLP
Statistical Methods for NLP Information Extraction, Hidden Markov Models Sameer Maskey * Most of the slides provided by Bhuvana Ramabhadran, Stanley Chen, Michael Picheny Speech Recognition Lecture 4:
27: Hybrid Graphical Models and Neural Networks
10-708: Probabilistic Graphical Models 10-708 Spring 2016 27: Hybrid Graphical Models and Neural Networks Lecturer: Matt Gormley Scribes: Jakob Bauer Otilia Stretcu Rohan Varma 1 Motivation We first look
Machine Learning Classifiers and Boosting
Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Naïve
Computationally Efficient M-Estimation of Log-Linear Structure Models
Computationally Efficient M-Estimation of Log-Linear Structure Models Noah Smith, Doug Vail, and John Lafferty School of Computer Science Carnegie Mellon University {nasmith,dvail2,lafferty}@cs.cmu.edu
Part II. C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
Part II C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Converting Directed to Undirected Graphs (1) Converting Directed to Undirected Graphs (2) Add extra links between
Dynamic Programming. Ellen Feldman and Avishek Dutta. February 27, CS155 Machine Learning and Data Mining
CS155 Machine Learning and Data Mining February 27, 2018 Motivation Much of machine learning is heavily dependent on computational power Many libraries exist that aim to reduce computational time TensorFlow
CRF Feature Induction
CRF Feature Induction Andrew McCallum Efficiently Inducing Features of Conditional Random Fields Kuzman Ganchev 1 Introduction Basic Idea Aside: Transformation Based Learning Notation/CRF Review 2 Arbitrary
Detection and Extraction of Events from s
Detection and Extraction of Events from Emails Shashank Senapaty Department of Computer Science Stanford University, Stanford CA senapaty@cs.stanford.edu December 12, 2008 Abstract I build a system to
Gradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz
Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint
Conditional Random Fields for Word Hyphenation
Conditional Random Fields for Word Hyphenation Tsung-Yi Lin and Chen-Yu Lee Department of Electrical and Computer Engineering University of California, San Diego {tsl008, chl260}@ucsd.edu February 12,
Support Vector Machine Learning for Interdependent and Structured Output Spaces
Support Vector Machine Learning for Interdependent and Structured Output Spaces I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun, ICML, 2004. And also I. Tsochantaridis, T. Joachims, T. Hofmann,
Network Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Sources Hastie, Tibshirani, Friedman: The Elements of Statistical Learning James, Witten, Hastie, Tibshirani: An Introduction to Statistical Learning Andrew Ng:
A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition
A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition Théodore Bluche, Hermann Ney, Christopher Kermorvant SLSP 14, Grenoble October
Machine Learning. Decision Trees. Manfred Huber
Machine Learning Decision Trees Manfred Huber 2015 1 Decision Trees Classifiers covered so far have been Non-parametric (KNN) Probabilistic with independence (Naïve Bayes) Linear in features (Logistic
Discrete sequential models and CRFs. 1 Case Study: Supervised Part-of-Speech Tagging
0-708: Probabilisti Graphial Models 0-708, Spring 204 Disrete sequential models and CRFs Leturer: Eri P. Xing Sribes: Pankesh Bamotra, Xuanhong Li Case Study: Supervised Part-of-Speeh Tagging The supervised
Large-Scale Lasso and Elastic-Net Regularized Generalized Linear Models
Large-Scale Lasso and Elastic-Net Regularized Generalized Linear Models DB Tsai Steven Hillion Outline Introduction Linear / Nonlinear Classification Feature Engineering - Polynomial Expansion Big-data
I Know Your Name: Named Entity Recognition and Structural Parsing
I Know Your Name: Named Entity Recognition and Structural Parsing David Philipson and Nikil Viswanathan {pdavid2, nikil}@stanford.edu CS224N Fall 2011 Introduction In this project, we explore a Maximum
CS229 Final Project: Predicting Expected Response Times
CS229 Final Project: Predicting Expected Email Response Times Laura Cruz-Albrecht (lcruzalb), Kevin Khieu (kkhieu) December 15, 2017 1 Introduction Each day, countless emails are sent out, yet the time
CS6375: Machine Learning Gautam Kunapuli. Mid-Term Review
Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes
What is machine learning?
Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship
CS545 Project: Conditional Random Fields on an ecommerce Website
CS545 Project: Conditional Random Fields on an ecommerce Website Brock Wilcox December 18, 2013 Contents 1 Conditional Random Fields 1 1.1 Overview................................................. 1 1.2
A Systematic Overview of Data Mining Algorithms. Sargur Srihari University at Buffalo The State University of New York
A Systematic Overview of Data Mining Algorithms Sargur Srihari University at Buffalo The State University of New York 1 Topics Data Mining Algorithm Definition Example of CART Classification Iris, Wine
Simple Model Selection Cross Validation Regularization Neural Networks
Neural Nets: Many possible refs e.g., Mitchell Chapter 4 Simple Model Selection Cross Validation Regularization Neural Networks Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University February
Feature Extraction and Loss training using CRFs: A Project Report
Feature Extraction and Loss training using CRFs: A Project Report Ankan Saha Department of computer Science University of Chicago March 11, 2008 Abstract POS tagging has been a very important problem in
Probabilistic Graphical Models
10-708 Probabilistic Graphical Models Homework 4 Due Apr 27, 12:00 noon Submission: Homework is due on the due date at 12:00 noon. Please see course website for policy on late submission. You must submit
Machine Learning. Sourangshu Bhattacharya
Machine Learning Sourangshu Bhattacharya Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Curve Fitting Re-visited Maximum Likelihood Determine by minimizing sum-of-squares
Bayes Net Learning. EECS 474 Fall 2016
Bayes Net Learning EECS 474 Fall 2016 Homework Remaining Homework #3 assigned Homework #4 will be about semi-supervised learning and expectation-maximization Homeworks #3-#4: the how of Graphical Models
ADVANCED CLASSIFICATION TECHNIQUES
Admin ML lab next Monday Project proposals: Sunday at 11:59pm ADVANCED CLASSIFICATION TECHNIQUES David Kauchak CS 159 Fall 2014 Project proposal presentations Machine Learning: A Geometric View 1 Apples
CPSC 340: Machine Learning and Data Mining. Multi-Class Classification Fall 2017
CPSC 340: Machine Learning and Data Mining Multi-Class Classification Fall 2017 Assignment 3: Admin Check update thread on Piazza for correct definition of trainndx. This could make your cross-validation
(Multinomial) Logistic Regression + Feature Engineering
-6 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University (Multinomial) Logistic Regression + Feature Engineering Matt Gormley Lecture 9 Feb.
CSEP 573: Artificial Intelligence
CSEP 573: Artificial Intelligence Machine Learning: Perceptron Ali Farhadi Many slides over the course adapted from Luke Zettlemoyer and Dan Klein. 1 Generative vs. Discriminative Generative classifiers:
Classification: Linear Discriminant Functions
Classification: Linear Discriminant Functions CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Discriminant functions Linear Discriminant functions
Edinburgh Research Explorer
Edinburgh Research Explorer An Introduction to Conditional Random Fields Citation for published version: Sutton, C & McCallum, A 2012, 'An Introduction to Conditional Random Fields' Foundations and Trends
EECS 496 Statistical Language Models. Winter 2018
EECS 496 Statistical Language Models Winter 2018 Introductions Professor: Doug Downey Course web site: www.cs.northwestern.edu/~ddowney/courses/496_winter2018 (linked off prof. home page) Logistics Grading
Semi-Supervised Learning of Named Entity Substructure
Semi-Supervised Learning of Named Entity Substructure Alden Timme aotimme@stanford.edu CS229 Final Project Advisor: Richard Socher richard@socher.org Abstract The goal of this project was two-fold: (1)
Bayesian model ensembling using meta-trained recurrent neural networks
Bayesian model ensembling using meta-trained recurrent neural networks Luca Ambrogioni l.ambrogioni@donders.ru.nl Umut Güçlü u.guclu@donders.ru.nl Yağmur Güçlütürk y.gucluturk@donders.ru.nl Julia Berezutskaya
Computer Vision Group Prof. Daniel Cremers. 4a. Inference in Graphical Models
Group Prof. Daniel Cremers 4a. Inference in Graphical Models Inference on a Chain (Rep.) The first values of µ α and µ β are: The partition function can be computed at any node: Overall, we have O(NK 2
Pattern Recognition. Kjell Elenius. Speech, Music and Hearing KTH. March 29, 2007 Speech recognition
Pattern Recognition Kjell Elenius Speech, Music and Hearing KTH March 29, 2007 Speech recognition 2007 1 Ch 4. Pattern Recognition 1(3) Bayes Decision Theory Minimum-Error-Rate Decision Rules Discriminant
Neural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /10/2017
3/0/207 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/0/207 Perceptron as a neural
CS273 Midterm Exam Introduction to Machine Learning: Winter 2015 Tuesday February 10th, 2014
CS273 Midterm Eam Introduction to Machine Learning: Winter 2015 Tuesday February 10th, 2014 Your name: Your UCINetID (e.g., myname@uci.edu): Your seat (row and number): Total time is 80 minutes. READ THE
Generative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 013-014 Jakob Verbeek, December 13+0, 013 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.13.14
Cambridge Interview Technical Talk
Cambridge Interview Technical Talk February 2, 2010 Table of contents Causal Learning 1 Causal Learning Conclusion 2 3 Motivation Recursive Segmentation Learning Causal Learning Conclusion Causal learning
ECE521: Week 11, Lecture March 2017: HMM learning/inference. With thanks to Russ Salakhutdinov
ECE521: Week 11, Lecture 20 27 March 2017: HMM learning/inference With thanks to Russ Salakhutdinov Examples of other perspectives Murphy 17.4 End of Russell & Norvig 15.2 (Artificial Intelligence: A Modern
Semi-supervised Learning
Semi-supervised Learning Piyush Rai CS5350/6350: Machine Learning November 8, 2011 Semi-supervised Learning Supervised Learning models require labeled data Learning a reliable model usually requires plenty
Sum-Product Networks. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 15, 2015
Sum-Product Networks STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 15, 2015 Introduction Outline What is a Sum-Product Network? Inference Applications In more depth
Discriminative Training for Phrase-Based Machine Translation
Discriminative Training for Phrase-Based Machine Translation Abhishek Arun 19 April 2007 Overview 1 Evolution from generative to discriminative models Discriminative training Model Learning schemes Featured
Application of Support Vector Machine Algorithm in Spam Filtering
Application of Support Vector Machine Algorithm in E-Mail Spam Filtering Julia Bluszcz, Daria Fitisova, Alexander Hamann, Alexey Trifonov, Advisor: Patrick Jähnichen Abstract The problem of spam classification
Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center
Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction
Mini-project 2 CMPSCI 689 Spring 2015 Due: Tuesday, April 07, in class
Mini-project 2 CMPSCI 689 Spring 2015 Due: Tuesday, April 07, in class Guidelines Submission. Submit a hardcopy of the report containing all the figures and printouts of code in class. For readability
Semi-Markov Models for Named Entity Recognition
Semi-Markov Models for Named Entity Recognition Sunita Sarawagi Indian Institute of Technology Bombay, India sunita@iitb.ac.in William W. Cohen Center for Automated Learning & Discovery Carnegie Mellon
AT&T: The Tag&Parse Approach to Semantic Parsing of Robot Spatial Commands
AT&T: The Tag&Parse Approach to Semantic Parsing of Robot Spatial Commands Svetlana Stoyanchev, Hyuckchul Jung, John Chen, Srinivas Bangalore AT&T Labs Research 1 AT&T Way Bedminster NJ 07921 {sveta,hjung,jchen,srini}@research.att.com
Homework 2: HMM, Viterbi, CRF/Perceptron
Homework 2: HMM, Viterbi, CRF/Perceptron CS 585, UMass Amherst, Fall 2015 Version: Oct5 Overview Due Tuesday, Oct 13 at midnight. Get starter code from the course website s schedule page. You should submit
CS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Image Data: Classification via Neural Networks Instructor: Yizhou Sun yzsun@ccs.neu.edu November 19, 2015 Methods to Learn Classification Clustering Frequent Pattern Mining
Machine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
INTRODUCTION TO DEEP LEARNING
INTRODUCTION TO DEEP LEARNING CONTENTS Introduction to deep learning Contents 1. Examples 2. Machine learning 3. Neural networks 4. Deep learning 5. Convolutional neural networks 6. Conclusion 7. Additional
Neural Network Neurons
Neural Networks Neural Network Neurons 1 Receives n inputs (plus a bias term) Multiplies each input by its weight Applies activation function to the sum of results Outputs result Activation Functions Given
Lecture 20: Neural Networks for NLP. Zubin Pahuja
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Today s Lecture Feed-forward neural networks as classifiers simple
The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.
CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or
LSTM for Language Translation and Image Captioning. Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia
1 LSTM for Language Translation and Image Captioning Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia 2 Part I LSTM for Language Translation Motivation Background (RNNs, LSTMs) Model
Time series, HMMs, Kalman Filters
Classic HMM tutorial see class website: *L. R. Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition," Proc. of the IEEE, Vol.77, No.2, pp.257--286, 1989. Time series,
CSCE 478/878 Lecture 6: Bayesian Learning and Graphical Models. Stephen Scott. Introduction. Outline. Bayes Theorem. Formulas
ian ian ian Might have reasons (domain information) to favor some hypotheses/predictions over others a priori ian methods work with probabilities, and have two main roles: Optimal Naïve Nets (Adapted from
Encoding RNNs, 48 End of sentence (EOS) token, 207 Exploding gradient, 131 Exponential function, 42 Exponential Linear Unit (ELU), 44
A Activation potential, 40 Annotated corpus add padding, 162 check versions, 158 create checkpoints, 164, 166 create input, 160 create train and validation datasets, 163 dropout, 163 DRUG-AE.rel file,
Analysis: TextonBoost and Semantic Texton Forests. Daniel Munoz Februrary 9, 2009
Analysis: TextonBoost and Semantic Texton Forests Daniel Munoz 16-721 Februrary 9, 2009 Papers [shotton-eccv-06] J. Shotton, J. Winn, C. Rother, A. Criminisi, TextonBoost: Joint Appearance, Shape and Context
Problem Set #6 Due: 2:30pm on Wednesday, June 1 st
Chris Piech Handout #38 CS09 May 8, 206 Problem Set #6 Due: 2:30pm on Wednesday, June st Note: The last day this assignment will be accepted (late) is Friday, June 3rd As noted above, the last day this
Semi-Markov Conditional Random Fields for Information Extraction
Semi-Markov Conditional Random Fields for Information Extraction S U N I T A S A R A W A G I A N D W I L L I A M C O H E N N I P S 2 0 0 4 P R E S E N T E D B Y : D I N E S H K H A N D E L W A L S L I
Notes on Multilayer, Feedforward Neural Networks
Notes on Multilayer, Feedforward Neural Networks CS425/528: Machine Learning Fall 2012 Prepared by: Lynne E. Parker [Material in these notes was gleaned from various sources, including E. Alpaydin s book
Conditional Random Fields for Object Recognition
Conditional Random Fields for Object Recognition Ariadna Quattoni Michael Collins Trevor Darrell MIT Computer Science and Artificial Intelligence Laboratory Cambridge, MA 02139 {ariadna, mcollins, trevor}@csail.mit.edu
Predict the Likelihood of Responding to Direct Mail Campaign in Consumer Lending Industry
Predict the Likelihood of Responding to Direct Mail Campaign in Consumer Lending Industry Jincheng Cao, SCPD Jincheng@stanford.edu 1. INTRODUCTION When running a direct mail campaign, it s common practice
CS5670: Computer Vision
CS5670: Computer Vision Noah Snavely Lecture 33: Recognition Basics Slides from Andrej Karpathy and Fei-Fei Li http://vision.stanford.edu/teaching/cs231n/ Announcements Quiz moved to Tuesday Project 4
18 October, 2013 MVA ENS Cachan. Lecture 6: Introduction to graphical models Iasonas Kokkinos
Machine Learning for Computer Vision 1 18 October, 2013 MVA ENS Cachan Lecture 6: Introduction to graphical models Iasonas Kokkinos Iasonas.kokkinos@ecp.fr Center for Visual Computing Ecole Centrale Paris
CAP 6412 Advanced Computer Vision
CAP 6412 Advanced Computer Vision http://www.cs.ucf.edu/~bgong/cap6412.html Boqing Gong April 5th, 2016 Today Administrivia LSTM Attribute in computer vision, by Abdullah and Samer Project II posted, due
Classification Lecture Notes cse352. Neural Networks. Professor Anita Wasilewska
Classification Lecture Notes cse352 Neural Networks Professor Anita Wasilewska Neural Networks Classification Introduction INPUT: classification data, i.e. it contains an classification (class) attribute