Pointer Network. Oriol Vinyals. 박천음 강원대학교 Intelligent Software Lab.
|
|
- Brendan Hopkins
- 5 years ago
- Views:
Transcription
1 Pointer Network Oriol Vinyals 박천음 강원대학교 Intelligent Software Lab. Intelligent Software Lab.
2 Pointer Network 1 Pointer Network 2 Intelligent Software Lab. 2
3 Sequence-to-Sequence Model Train 학습학습학습학습학습 Test Intelligent Software Lab. 3
4 Encoder-Decoder Architecture Sentence representation: encoder is based on its property of sequence summarization. RNN h i = φ θ (h i 1, s i ) global RNN`s internal state z i based on the summary vector h T of the source sentence. Intelligent Software Lab. 4
5 Soft Attention Mechanism Bi-RNN: forward and backward RNN 으로전체입력문장을심볼 (symbol) 로요약 (summarization) 할수있음 h h Figure 2. Bidirectional recurrent neural networks for encoding a source sentence. - cho Intelligent Software Lab. 5
6 Soft Attention Mechanism But RNN은최근심볼에편향적인특성을보임 not good 따라서 Concat h and h Annotation vector h (1) 입력문장을가변길이표현으로저장하는문맥의존단어표현집합 (set of context-dependent word representation) 으로볼수있음 (as opposed to Fixed-length, dimensional summary) 번역된단어 (y 1, y 2,, y t 1 ) 와함께현재단어 x j (or h j ) 를번역 y t Figure 3. attention Mechanism takes into consideration... - cho Intelligent Software Lab. 6
7 Soft Attention Mechanism Attention mechanism: small neural network (+ single hidden layer) Attention vector: e j R (single scalar output) e ij = f z i 1, h j z i 1 : previous decoder`s hidden state h j : one of the source context-dependent word representations Single hidden layer e ij Intelligent Software Lab. 7
8 Soft Attention Mechanism 모든단어에대한 score 를구할때 softmax Attention weight: a ij = exp e ij j exp e ij Context vector (encoding): c ij = j=1 T a ij h j = Ε a ij [h j ] c ij Intelligent Software Lab. 8
9 Pointer Network Repurposing the attention mechanism to create pointers to input elements c: input(x 1, y 1 ) Input 과 output 의길이가서로다른경우 PtrNet 해결가능 Representing variable length dictionaries by using a softmax probability distribution as a pointer Intelligent Software Lab. 9
10 Pointer Network PtrNet u i j = v T tanh W 1 e j + W 2 d i, j 1,, n p C i C 1, C i 1, Ρ = softmax u i, (of length n) Blending attention a i j = softmax u i n d i = a i j e j j=1 PtrNet: Data structure Ρ: point sets Ρ = {Ρ 1,, Ρ n } Ρ j = (x j, y j ): cartesian coordinate v, W 1, W 2 : learnable parameters (input) u j i : use it as pointers to the input elements: C i 1 Ρ Ci 1 d i e j Intelligent Software Lab. 10
11 Pointer Network 1 Pointer Network 2 Intelligent Software Lab. 11
12 Pointer Network PtrNet: Architecture and Hyperparameters No extensive architecture or hyperparameter search Main message of the paper stronger Single layer LSTM Hidden units: 256 or 512 Stochastic gradient descent A learning rate of 1.0 Batch size of 128 Random uniform weight initialization: ~ 0.08 L2 gradient clipping of 2.0 1M training example pairs Intelligent Software Lab. 12
13 Pointer Network Convex Hull: Data structure Ρ: point sets Ρ = {Ρ 1,, Ρ n } (Input) Ρ j : uniform distribution in [0, 1] x [0, 1] C P = {C 1,, C m P } (Output) C i : between 1 and n corresponding to positions in the sequence Ρ Special token: beginning( ) or end( ) Example Input Ρ = Ρ 1,, Ρ 10 Output C Ρ = {, 2, 4, 3, 5, 6, 7, 2, } Intelligent Software Lab. 13
14 Pointer Network Convex Hull: Empirical Results Intelligent Software Lab. 14
15 Pointer Network Delaunay Triangulation: Data Structure Ρ: Delaunay triangulation for a set Ρ of points in a plane (every triangle is empty there is no point from Ρ in its interior) C Ρ = {C 1,, C m P }: the corresponding sequences representing the triangulation of the point set Ρ C i : three integers for each triangle representation Lexicographic order: we order the triangles C i by their incenter coordinates ( 정렬해야더좋은성능 ) Choose the increasing triangle representation Example C i = 1,2,4 (2,4,1) Input Ρ = Ρ 1,, Ρ 5 Output C Ρ = {, 1,2,4, 1,4,5, 1,3,5, (1,2,3), } Intelligent Software Lab. 15
16 Pointer Network Delaunay Triangulation: Empirical Results Given set of points triangulates the convex hull of these points Intelligent Software Lab. 16
17 Pointer Network Travelling Salesman Problem (TSP): Data structure Input/output pairs (Ρ, C Ρ ) as in the Convex Hull problem Ρ: the Cartesian coordinates representing the cities Chosen randomly in the [0, 1] x [0, 1] square C Ρ = {C 1,, C n }: permutation of integers from 1 to n representing the optimal path Intelligent Software Lab. 17
18 Pointer Network Travelling Salesman Problem (TSP): Data structure Valid tour 에대하여 beam search Invalid tour 제외..? 같은도시를반복 or 도착점무시 Intelligent Software Lab. 18
19 references Sequence to sequence Learning with Neural Networks, Stuskever at al. n-neural-machine-translation-gpus-part-2/ n-neural-machine-translation-gpus-part-3/ Pointer Networks, Vinyals at al. Intelligent Software Lab. 19
20 QA 감사합니다. 박천음, 이창기 Intelligent Software Lab., 강원대학교 Intelligent Software Lab.
16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning. Spring 2018 Lecture 14. Image to Text
16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning Spring 2018 Lecture 14. Image to Text Input Output Classification tasks 4/1/18 CMU 16-785: Integrated Intelligence in Robotics
More informationRecurrent Neural Nets II
Recurrent Neural Nets II Steven Spielberg Pon Kumar, Tingke (Kevin) Shen Machine Learning Reading Group, Fall 2016 9 November, 2016 Outline 1 Introduction 2 Problem Formulations with RNNs 3 LSTM for Optimization
More informationLSTM for Language Translation and Image Captioning. Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia
1 LSTM for Language Translation and Image Captioning Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia 2 Part I LSTM for Language Translation Motivation Background (RNNs, LSTMs) Model
More informationImage Captioning with Object Detection and Localization
Image Captioning with Object Detection and Localization Zhongliang Yang, Yu-Jin Zhang, Sadaqat ur Rehman, Yongfeng Huang, Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
More informationEncoding RNNs, 48 End of sentence (EOS) token, 207 Exploding gradient, 131 Exponential function, 42 Exponential Linear Unit (ELU), 44
A Activation potential, 40 Annotated corpus add padding, 162 check versions, 158 create checkpoints, 164, 166 create input, 160 create train and validation datasets, 163 dropout, 163 DRUG-AE.rel file,
More informationCS839: Probabilistic Graphical Models. Lecture 22: The Attention Mechanism. Theo Rekatsinas
CS839: Probabilistic Graphical Models Lecture 22: The Attention Mechanism Theo Rekatsinas 1 Why Attention? Consider machine translation: We need to pay attention to the word we are currently translating.
More informationTable of Contents. What Really is a Hidden Unit? Visualizing Feed-Forward NNs. Visualizing Convolutional NNs. Visualizing Recurrent NNs
Table of Contents What Really is a Hidden Unit? Visualizing Feed-Forward NNs Visualizing Convolutional NNs Visualizing Recurrent NNs Visualizing Attention Visualizing High Dimensional Data What do visualizations
More informationShow, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks
Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Zelun Luo Department of Computer Science Stanford University zelunluo@stanford.edu Te-Lin Wu Department of
More informationEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural
More informationCSC 578 Neural Networks and Deep Learning
CSC 578 Neural Networks and Deep Learning Fall 2018/19 7. Recurrent Neural Networks (Some figures adapted from NNDL book) 1 Recurrent Neural Networks 1. Recurrent Neural Networks (RNNs) 2. RNN Training
More informationSeq2SQL: Generating Structured Queries from Natural Language Using Reinforcement Learning
Seq2SQL: Generating Structured Queries from Natural Language Using Reinforcement Learning V. Zhong, C. Xiong, R. Socher Salesforce Research arxiv: 1709.00103 Reviewed by : Bill Zhang University of Virginia
More information27: Hybrid Graphical Models and Neural Networks
10-708: Probabilistic Graphical Models 10-708 Spring 2016 27: Hybrid Graphical Models and Neural Networks Lecturer: Matt Gormley Scribes: Jakob Bauer Otilia Stretcu Rohan Varma 1 Motivation We first look
More informationJOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS. Puyang Xu, Ruhi Sarikaya. Microsoft Corporation
JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS Puyang Xu, Ruhi Sarikaya Microsoft Corporation ABSTRACT We describe a joint model for intent detection and slot filling based
More informationModeling Sequences Conditioned on Context with RNNs
Modeling Sequences Conditioned on Context with RNNs Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 10. Topics in Sequence
More informationABC-CNN: Attention Based CNN for Visual Question Answering
ABC-CNN: Attention Based CNN for Visual Question Answering CIS 601 PRESENTED BY: MAYUR RUMALWALA GUIDED BY: DR. SUNNIE CHUNG AGENDA Ø Introduction Ø Understanding CNN Ø Framework of ABC-CNN Ø Datasets
More informationResidual Networks And Attention Models. cs273b Recitation 11/11/2016. Anna Shcherbina
Residual Networks And Attention Models cs273b Recitation 11/11/2016 Anna Shcherbina Introduction to ResNets Introduced in 2015 by Microsoft Research Deep Residual Learning for Image Recognition (He, Zhang,
More informationMachine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
More informationEmpirical Evaluation of RNN Architectures on Sentence Classification Task
Empirical Evaluation of RNN Architectures on Sentence Classification Task Lei Shen, Junlin Zhang Chanjet Information Technology lorashen@126.com, zhangjlh@chanjet.com Abstract. Recurrent Neural Networks
More informationTransition-Based Dependency Parsing with Stack Long Short-Term Memory
Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, Noah A. Smith Association for Computational Linguistics (ACL), 2015 Presented
More informationSEMANTIC COMPUTING. Lecture 9: Deep Learning: Recurrent Neural Networks (RNNs) TU Dresden, 21 December 2018
SEMANTIC COMPUTING Lecture 9: Deep Learning: Recurrent Neural Networks (RNNs) Dagmar Gromann International Center For Computational Logic TU Dresden, 21 December 2018 Overview Handling Overfitting Recurrent
More informationSequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015
Sequence Modeling: Recurrent and Recursive Nets By Pyry Takala 14 Oct 2015 Agenda Why Recurrent neural networks? Anatomy and basic training of an RNN (10.2, 10.2.1) Properties of RNNs (10.2.2, 8.2.6) Using
More informationVoronoi Diagrams. A Voronoi diagram records everything one would ever want to know about proximity to a set of points
Voronoi Diagrams Voronoi Diagrams A Voronoi diagram records everything one would ever want to know about proximity to a set of points Who is closest to whom? Who is furthest? We will start with a series
More informationDeep Learning. Architecture Design for. Sargur N. Srihari
Architecture Design for Deep Learning Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation
More informationCSCE 350: Chin-Tser Huang. University of South Carolina
CSCE 350: Data Structures and Algorithms Chin-Tser Huang huangct@cse.sc.edu University of South Carolina Announcement Homework 2 will be returned on Thursday; solution will be available on class website
More informationRecurrent Neural Network (RNN) Industrial AI Lab.
Recurrent Neural Network (RNN) Industrial AI Lab. For example (Deterministic) Time Series Data Closed- form Linear difference equation (LDE) and initial condition High order LDEs 2 (Stochastic) Time Series
More informationCS 6501: Deep Learning for Computer Graphics. Training Neural Networks II. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Training Neural Networks II Connelly Barnes Overview Preprocessing Initialization Vanishing/exploding gradients problem Batch normalization Dropout Additional
More informationCan Active Memory Replace Attention?
Google Brain NIPS 2016 Presenter: Chao Jiang NIPS 2016 Presenter: Chao Jiang 1 / Outline 1 Introduction 2 Active Memory 3 Step by Step to Neural GPU 4 Another two steps: 1. the Markovian Neural GPU 5 Another
More informationAsynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features
Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features Xu SUN ( 孙栩 ) Peking University xusun@pku.edu.cn Motivation Neural networks -> Good Performance CNN, RNN, LSTM
More informationConvolutional Sequence to Sequence Learning. Denis Yarats with Jonas Gehring, Michael Auli, David Grangier, Yann Dauphin Facebook AI Research
Convolutional Sequence to Sequence Learning Denis Yarats with Jonas Gehring, Michael Auli, David Grangier, Yann Dauphin Facebook AI Research Sequence generation Need to model a conditional distribution
More informationComputational Geometry
Computational Geometry 600.658 Convexity A set S is convex if for any two points p, q S the line segment pq S. S p S q Not convex Convex? Convexity A set S is convex if it is the intersection of (possibly
More informationS8822 OPTIMIZING NMT WITH TENSORRT Micah Villmow Senior TensorRT Software Engineer
S8822 OPTIMIZING NMT WITH TENSORRT Micah Villmow Senior TensorRT Software Engineer 2 100 倍以上速く 本当に可能ですか? 2 DOUGLAS ADAMS BABEL FISH Neural Machine Translation Unit 3 4 OVER 100X FASTER, IS IT REALLY POSSIBLE?
More informationGenerative Adversarial Text to Image Synthesis
Generative Adversarial Text to Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee Presented by: Jingyao Zhan Contents Introduction Related Work Method
More informationRNNs as Directed Graphical Models
RNNs as Directed Graphical Models Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 10. Topics in Sequence Modeling Overview
More informationGrounded Compositional Semantics for Finding and Describing Images with Sentences
Grounded Compositional Semantics for Finding and Describing Images with Sentences R. Socher, A. Karpathy, V. Le,D. Manning, A Y. Ng - 2013 Ali Gharaee 1 Alireza Keshavarzi 2 1 Department of Computational
More informationImageNet Classification with Deep Convolutional Neural Networks
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky Ilya Sutskever Geoffrey Hinton University of Toronto Canada Paper with same name to appear in NIPS 2012 Main idea Architecture
More informationA new approach for supervised power disaggregation by using a deep recurrent LSTM network
A new approach for supervised power disaggregation by using a deep recurrent LSTM network GlobalSIP 2015, 14th Dec. Lukas Mauch and Bin Yang Institute of Signal Processing and System Theory University
More informationIndex. Umberto Michelucci 2018 U. Michelucci, Applied Deep Learning,
A Acquisition function, 298, 301 Adam optimizer, 175 178 Anaconda navigator conda command, 3 Create button, 5 download and install, 1 installing packages, 8 Jupyter Notebook, 11 13 left navigation pane,
More informationDeep Learning Applications
October 20, 2017 Overview Supervised Learning Feedforward neural network Convolution neural network Recurrent neural network Recursive neural network (Recursive neural tensor network) Unsupervised Learning
More information4.12 Generalization. In back-propagation learning, as many training examples as possible are typically used.
1 4.12 Generalization In back-propagation learning, as many training examples as possible are typically used. It is hoped that the network so designed generalizes well. A network generalizes well when
More informationShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio Presented
More informationOn the Efficiency of Recurrent Neural Network Optimization Algorithms
On the Efficiency of Recurrent Neural Network Optimization Algorithms Ben Krause, Liang Lu, Iain Murray, Steve Renals University of Edinburgh Department of Informatics s17005@sms.ed.ac.uk, llu@staffmail.ed.ac.uk,
More informationDEEP LEARNING REVIEW. Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature Presented by Divya Chitimalla
DEEP LEARNING REVIEW Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature 2015 -Presented by Divya Chitimalla What is deep learning Deep learning allows computational models that are composed of multiple
More informationDCU-UvA Multimodal MT System Report
DCU-UvA Multimodal MT System Report Iacer Calixto ADAPT Centre School of Computing Dublin City University Dublin, Ireland iacer.calixto@adaptcentre.ie Desmond Elliott ILLC University of Amsterdam Science
More informationProject Final Report
Project Final Report Ye Tian Stanford University yetian@stanford.edu Tianlun Li Stanford University tianlunl@stanford.edu Abstract We plan to do image-to-sentence generation. This application bridges vision
More informationEncoding Techniques in Genetic Algorithms
Encoding Techniques in Genetic Algorithms Debasis Samanta Indian Institute of Technology Kharagpur dsamanta@iitkgp.ac.in 01.03.2016 Debasis Samanta (IIT Kharagpur) Soft Computing Applications 01.03.2016
More informationCUED-RNNLM An Open-Source Toolkit for Efficient Training and Evaluation of Recurrent Neural Network Language Models
CUED-RNNLM An Open-Source Toolkit for Efficient Training and Evaluation of Recurrent Neural Network Language Models Xie Chen, Xunying Liu, Yanmin Qian, Mark Gales and Phil Woodland April 1, 2016 Overview
More informationLifting Transform, Voronoi, Delaunay, Convex Hulls
Lifting Transform, Voronoi, Delaunay, Convex Hulls Subhash Suri Department of Computer Science University of California Santa Barbara, CA 93106 1 Lifting Transform (A combination of Pless notes and my
More informationNatural Language Processing CS 6320 Lecture 6 Neural Language Models. Instructor: Sanda Harabagiu
Natural Language Processing CS 6320 Lecture 6 Neural Language Models Instructor: Sanda Harabagiu In this lecture We shall cover: Deep Neural Models for Natural Language Processing Introduce Feed Forward
More informationRecurrent Convolutional Neural Networks for Scene Labeling
Recurrent Convolutional Neural Networks for Scene Labeling Pedro O. Pinheiro, Ronan Collobert Reviewed by Yizhe Zhang August 14, 2015 Scene labeling task Scene labeling: assign a class label to each pixel
More informationDeep Learning. Practical introduction with Keras JORDI TORRES 27/05/2018. Chapter 3 JORDI TORRES
Deep Learning Practical introduction with Keras Chapter 3 27/05/2018 Neuron A neural network is formed by neurons connected to each other; in turn, each connection of one neural network is associated
More informationCAP 6412 Advanced Computer Vision
CAP 6412 Advanced Computer Vision http://www.cs.ucf.edu/~bgong/cap6412.html Boqing Gong Feb 04, 2016 Today Administrivia Attention Modeling in Image Captioning, by Karan Neural networks & Backpropagation
More informationDeep Learning for Computer Vision II
IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L
More informationStructured Attention Networks
Structured Attention Networks Yoon Kim Carl Denton Luong Hoang Alexander M. Rush HarvardNLP ICLR, 2017 Presenter: Chao Jiang ICLR, 2017 Presenter: Chao Jiang 1 / Outline 1 Deep Neutral Networks for Text
More informationFastText. Jon Koss, Abhishek Jindal
FastText Jon Koss, Abhishek Jindal FastText FastText is on par with state-of-the-art deep learning classifiers in terms of accuracy But it is way faster: FastText can train on more than one billion words
More informationRecurrent Neural Networks and Transfer Learning for Action Recognition
Recurrent Neural Networks and Transfer Learning for Action Recognition Andrew Giel Stanford University agiel@stanford.edu Ryan Diaz Stanford University ryandiaz@stanford.edu Abstract We have taken on the
More informationMore NP-complete Problems. CS255 Chris Pollett May 3, 2006.
More NP-complete Problems CS255 Chris Pollett May 3, 2006. Outline More NP-Complete Problems Hamiltonian Cycle Recall a hamiltonian cycle is a permutation of the vertices v i_1,, v i_n of a graph G so
More informationLecture 2 Notes. Outline. Neural Networks. The Big Idea. Architecture. Instructors: Parth Shah, Riju Pahwa
Instructors: Parth Shah, Riju Pahwa Lecture 2 Notes Outline 1. Neural Networks The Big Idea Architecture SGD and Backpropagation 2. Convolutional Neural Networks Intuition Architecture 3. Recurrent Neural
More informationDeep Learning for Entity Matching: A Design Space Exploration. Sidharth Mudgal Sunil Kumar
Deep Learning for Entity Matching: A Design Space Exploration by Sidharth Mudgal Sunil Kumar A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science (Computer
More informationNatural Language Processing with Deep Learning CS224N/Ling284
Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 8: Recurrent Neural Networks Christopher Manning and Richard Socher Organization Extra project office hour today after lecture Overview
More informationBayesian model ensembling using meta-trained recurrent neural networks
Bayesian model ensembling using meta-trained recurrent neural networks Luca Ambrogioni l.ambrogioni@donders.ru.nl Umut Güçlü u.guclu@donders.ru.nl Yağmur Güçlütürk y.gucluturk@donders.ru.nl Julia Berezutskaya
More informationLecture 3. Brute Force
Lecture 3 Brute Force 1 Lecture Contents 1. Selection Sort and Bubble Sort 2. Sequential Search and Brute-Force String Matching 3. Closest-Pair and Convex-Hull Problems by Brute Force 4. Exhaustive Search
More informationVoronoi Diagrams and Delaunay Triangulations. O Rourke, Chapter 5
Voronoi Diagrams and Delaunay Triangulations O Rourke, Chapter 5 Outline Preliminaries Properties and Applications Computing the Delaunay Triangulation Preliminaries Given a function f: R 2 R, the tangent
More informationLecture 20: Neural Networks for NLP. Zubin Pahuja
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Today s Lecture Feed-forward neural networks as classifiers simple
More informationAdvanced Search Algorithms
CS11-747 Neural Networks for NLP Advanced Search Algorithms Daniel Clothiaux https://phontron.com/class/nn4nlp2017/ Why search? So far, decoding has mostly been greedy Chose the most likely output from
More informationKyoto-NMT: a Neural Machine Translation implementation in Chainer
Kyoto-NMT: a Neural Machine Translation implementation in Chainer Fabien Cromières Japan Science and Technology Agency Kawaguchi-shi, Saitama 332-0012 fabien@pa.jst.jp Abstract We present Kyoto-NMT, an
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationComputational Geometry
Lecture 12: Lecture 12: Motivation: Terrains by interpolation To build a model of the terrain surface, we can start with a number of sample points where we know the height. Lecture 12: Motivation: Terrains
More information18.S34 (FALL 2007) PROBLEMS ON HIDDEN INDEPENDENCE AND UNIFORMITY
18.S34 (FALL 2007) PROBLEMS ON HIDDEN INDEPENDENCE AND UNIFORMITY All the problems below (with the possible exception of the last one), when looked at the right way, can be solved by elegant arguments
More informationRecurrent Neural Networks
Recurrent Neural Networks Javier Béjar Deep Learning 2018/2019 Fall Master in Artificial Intelligence (FIB-UPC) Introduction Sequential data Many problems are described by sequences Time series Video/audio
More informationDeep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group
Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group 1D Input, 1D Output target input 2 2D Input, 1D Output: Data Distribution Complexity Imagine many dimensions (data occupies
More informationA Hybrid Neural Model for Type Classification of Entity Mentions
A Hybrid Neural Model for Type Classification of Entity Mentions Motivation Types group entities to categories Entity types are important for various NLP tasks Our task: predict an entity mention s type
More informationGraph Neural Network. learning algorithm and applications. Shujia Zhang
Graph Neural Network learning algorithm and applications Shujia Zhang What is Deep Learning? Enable computers to mimic human behaviour Artificial Intelligence Machine Learning Subset of ML algorithms using
More informationDeep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
More informationA Quick Guide on Training a neural network using Keras.
A Quick Guide on Training a neural network using Keras. TensorFlow and Keras Keras Open source High level, less flexible Easy to learn Perfect for quick implementations Starts by François Chollet from
More informationMachine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center
Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction
More informationChapter 8. Voronoi Diagrams. 8.1 Post Oce Problem
Chapter 8 Voronoi Diagrams 8.1 Post Oce Problem Suppose there are n post oces p 1,... p n in a city. Someone who is located at a position q within the city would like to know which post oce is closest
More informationNatural Language to Neural Programs
Natural Language to Neural Programs Daniel Simig Department of Engineering University of Cambridge This dissertation is submitted for the degree of Master of Philosophy in Machine Learning, Speech and
More informationNeural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer
More informationSimple and Efficient Learning with Automatic Operation Batching
Simple and Efficient Learning with Automatic Operation Batching Graham Neubig joint work w/ Yoav Goldberg and Chris Dyer in http://dynet.io/autobatch/ https://github.com/neubig/howtocode-2017 Neural Networks
More informationSentiment Classification of Food Reviews
Sentiment Classification of Food Reviews Hua Feng Department of Electrical Engineering Stanford University Stanford, CA 94305 fengh15@stanford.edu Ruixi Lin Department of Electrical Engineering Stanford
More informationPlan, Attend, Generate: Planning for Sequence-to-Sequence Models
Plan, Attend, Generate: Planning for Sequence-to-Sequence Models Francis Dutil, Caglar Gulcehre, Adam Trischler, Yoshua Bengio Presented by Xinyuan Zhang April 26, 2018 April 26, 2018 1 / 11 Introduction
More informationOptimal Compression of a Polyline with Segments and Arcs
Optimal Compression of a Polyline with Segments and Arcs Alexander Gribov Esri 380 New York Street Redlands, CA 92373 Email: agribov@esri.com arxiv:1604.07476v5 [cs.cg] 10 Apr 2017 Abstract This paper
More informationM. Sc. (Artificial Intelligence and Machine Learning)
Course Name: Advanced Python Course Code: MSCAI 122 This course will introduce students to advanced python implementations and the latest Machine Learning and Deep learning libraries, Scikit-Learn and
More informationNeural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision Anonymized for review Abstract Extending the success of deep neural networks to high level tasks like natural language
More informationImage Captioning and Generation From Text
Image Captioning and Generation From Text Presented by: Tony Zhang, Jonathan Kenny, and Jeremy Bernstein Mentor: Stephan Zheng CS159 Advanced Topics in Machine Learning: Structured Prediction California
More informationSpatial Data Structures
CSCI 420 Computer Graphics Lecture 17 Spatial Data Structures Jernej Barbic University of Southern California Hierarchical Bounding Volumes Regular Grids Octrees BSP Trees [Angel Ch. 8] 1 Ray Tracing Acceleration
More informationA Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition
A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition Théodore Bluche, Hermann Ney, Christopher Kermorvant SLSP 14, Grenoble October
More informationAn Experimental Evaluation of the Best-of-Many Christofides Algorithm for the Traveling Salesman Problem
An Experimental Evaluation of the Best-of-Many Christofides Algorithm for the Traveling Salesman Problem David P. Williamson Cornell University Joint work with Kyle Genova, Cornell University July 14,
More informationarxiv: v5 [stat.ml] 4 May 2017
Machine Learning on Sequential Data Using a Recurrent Weighted Average arxiv:1703.01253v5 [stat.ml] 4 May 2017 Jared Ostmeyer * Lindsay Cowell Department of Clinical Sciences UT Southwestern Medical Center
More informationA New Progressive Lossy-to-Lossless Coding Method for 2.5-D Triangle Mesh with Arbitrary Connectivity
A New Progressive Lossy-to-Lossless Coding Method for 2.5-D Triangle Mesh with Arbitrary Connectivity Dan Han University of Victoria October 31, 2016 New Mesh-Coding Method Dan Han 1 Outline 1 Introduction
More informationSpatial Data Structures
CSCI 480 Computer Graphics Lecture 7 Spatial Data Structures Hierarchical Bounding Volumes Regular Grids BSP Trees [Ch. 0.] March 8, 0 Jernej Barbic University of Southern California http://www-bcf.usc.edu/~jbarbic/cs480-s/
More informationStructured Attention Networks
Structured Attention Networks Yoon Kim Carl Denton Luong Hoang Alexander M. Rush HarvardNLP 1 Deep Neural Networks for Text Processing and Generation 2 Attention Networks 3 Structured Attention Networks
More informationFast Prediction of Solutions to an Integer Linear Program with Machine Learning
Fast Prediction of Solutions to an Integer Linear Program with Machine Learning CERMICS 2018 June 29, 2018, Fre jus ERIC LARSEN, CIRRELT and Universite de Montre al (UdeM) SE BASTIEN LACHAPELLE, CIRRELT
More informationOutline GF-RNN ReNet. Outline
Outline Gated Feedback Recurrent Neural Networks. arxiv1502. Introduction: RNN & Gated RNN Gated Feedback Recurrent Neural Networks (GF-RNN) Experiments: Character-level Language Modeling & Python Program
More informationHidden Units. Sargur N. Srihari
Hidden Units Sargur N. srihari@cedar.buffalo.edu 1 Topics in Deep Feedforward Networks Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation
More informationMIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius What is Mixed Precision Training? Reduced precision tensor math with FP32 accumulation, FP16 storage Successfully used to train a variety
More informationFundamentals of Integer Programming
Fundamentals of Integer Programming Di Yuan Department of Information Technology, Uppsala University January 2018 Outline Definition of integer programming Formulating some classical problems with integer
More informationMachine Learning for Natural Language Processing. Alice Oh January 17, 2018
Machine Learning for Natural Language Processing Alice Oh January 17, 2018 Overview Distributed representation Temporal neural networks RNN LSTM GRU Sequence-to-sequence models Machine translation Response
More informationGenerating composite SQL queries from natural language questions using recurrent neural networks. Matthias De Groote
Generating composite SQL queries from natural language questions using recurrent neural networks Matthias De Groote Supervisors: Prof. dr. ir. Joni Dambre, Prof. dr. Wesley De Neve Counsellors: Ir. Fréderic
More informationHow to Develop Encoder-Decoder LSTMs
Chapter 9 How to Develop Encoder-Decoder LSTMs 9.0.1 Lesson Goal The goal of this lesson is to learn how to develop encoder-decoder LSTM models. completing this lesson, you will know: After ˆ The Encoder-Decoder
More information