DL4J Components. Open Source DeepLearning Library CPU & GPU support Hadoop-Yarn & Spark integration Futre additions
|
|
|
- Violet Nash
- 5 years ago
- Views:
Transcription
1 DeepLearning4j
2 DL4J Components Open Source DeepLearning Library CPU & GPU support Hadoop-Yarn & Spark integration Futre additions
3 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn & Spark integration Futre additions
4 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn & Spark integration What components are interfaced? (Section 2) Futre additions
5 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions
6 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions
7 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions Section 4 / The End.
8 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions Section 4 / The End.
9 Lets first look at what actually Deep Learning is Generally: Deep Learning might be only an alternative term for greater sized artificial neural networks? Bigger Datasets are also required (scaling/performance issues?)
10 Deep Learning IN DeepLearning4J DL4J offers the following Deep Neural Networks: Restricted Boltzmann Machines Convolutional Nets (Images) Recurrent nets/lstms Recursive Autoencoders Deep Belief Networks Recursive Neural Tensor (~Vector/Array) Networks Stacked Denoising Autoencoders Data frame (/set) readers included for: MNIST, Labeled Faces in the Wild (LFW), IRIS And DL4J also supports pipelining / stacking of layers.
11 Deep Artificial Neural Networks (DNNs) More Info & Sources: ttp://stats.stackexchange.com/questions/114385/what-isthe-difference-between-convolutional-neural-networksrestricted-boltzma
12 Motivation: Recurrent Neural Networks Recurrent connections in Hippocampus enable sequential memory and different types of time depended learning
13 Uses For Recursive/Recurrent Neural Networks Left: Socher et al. (2011) - Parsing Natural Scenes and Natural Languagewith Recursive Neural Networks (Google) Top Middle: Hannun (2014) - Deep Speech: Scaling up end-to-endspeech recognition (Baidu) Rest: Karpathy (2015) - The Unreasonable Effectiveness of Recurrent Neural Networks [link] Further Readings: Gillick (2016) - Multilingual Language Processing From Bytes; Sak (2015) - Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition; Marek et al (2015) - Large-Scale Language Classification (200 Languages, Twitter)
14 Example: Recurrent Neural Networks In DL4J
15 Motivation: Sparsity & Extraction of Stimuly Statistics High information content (direction of high variance extracted by e.g. Denoising AE / ) Low overlap of feature maps, so more feat can be extracted
16 Example: UnSupervised Learning In DL4j
17 Motivation: Convlutional Neural Networks Top: Scannel (1993) - the connectional organization of the cat visual cortex Bottom Right: Fellemann & Van Essen (1991) - Distributed Hierarchical Processing in the Primate Cerebral Cortex The (human) visual sytem is hierarchically organized (each area thereof consists of multiple neuron layers [I-VI] with different types of connectivity.
18 Motivation: Convlutional Neural Networks Top: Scannel (1993) - the connectional organization of the cat visual cortex Bottom Right: Fellemann & Van Essen (1991) - Distributed Hierarchical Processing in the Primate Cerebral Cortex The (human) visual sytem is hierarchically organized (each area thereof consists of multiple neuron layers [I-VI] with different types of connectivity.
19 Motivation: Convlutional Neural Networks V1 V2 Hierarchical Processing for Extraction of different features (= Feature Maps) in the Visual System See next slide for how CNN do this... LIP V4 TL: Betsch & König et al. (2004) The world from a cat s perspective statistics of natural videos BL: Roe et al. (2009) - Visual System: Functional Architecture of Area V2 C: Roe et al. (2012) - Toward a Unified Theory of Visual Area V4 R: Patel (2014) - Topographic organization in the brain: searching for general principles
20
21 Real Uses for Convlutional Neural Networks Left: Socher et al. (2011) - Parsing Natural Scenes and Natural Languagewith Recursive Neural Networks (Google) Right: Szegedy (2014) - Going Deeper with Convolutions (GoogLeNet/Inception for Image Classification & Detection)
22 Example: Convlutional Neural Networks In DL4J
23 CoMbination of differen DNNs Can be done by stacking of different layer types Top: Vinyals (2015) - Show and Tell: A Neural Image Caption Generator Right: Sak (2015) Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks
24 Example: Chaining Neural Networks In DL4J Recurrent Network with Skip Connections Multiple Inputs and Merge Vertex Multi Task Learning See:
25 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions Section 4 / The End.
26 (Hadoop) and Spark Integration?
27 (Hadoop) and Spark Integration
28 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions Section 4 / The End.
29 But how is this code distributed? So far it is not only the datasets and interations are. Lets take a look at the source code to proof this.
30 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update Ansewers Request Worker 'waits' (asks) for new update Receives Update Ansewers Request Worker 1 receives update Startup Worker 2 Worker 2 finishes and sends Update Worker 'waits' (asks) for new update Worker 2 receives update
31 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
32 Source Code (MR/YARN) Worker main function after it has been first started: The worker initializeses and computes something during the first iteration with the given model on its data, the result is some kind of update.
33 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
34 Source Code (MR/YARN) Worker main function after it has been first started: We continue here two slides later The worker the worker then sends the update back to the master.
35 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
36 Source Code (MR/YARN) Just fullfilled if all workers Are done with the current iteration Once the master receives an update from the worker and checks if all of the workers have finished the iterated (that is sent updates). If so, it starts integrating the immidiate results and updates the internal update count.
37 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
38 Source Code (MR/YARN) Worker after having sent an update (within its main function) Loop with Timer inside Gets data for next iteration The worker goes over to wait for a next update...
39 Source Code (MR/YARN) Worker after having sent an update (waitonmasterupdate)... and it continutes to send periodical wait requests to the master service until it receivs a positive number as next update number.
40 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
41 Source Code (MR/YARN) MasterService.waiting function (on master, triggered remotely by worker) Only interesting if there Is a new update The master service registeres the worker as waiting and gives back the current update Id.
42 Distribution on MapReduce (now Decrapitated) Master receives application Master wairing for worker update Startup Worker 1 Receives Update Worker 1 finishes and sends Update 3 Ansewers Request 5a 1 Startup Worker 2 2 Worker 'waits' (asks) for new update Worker 2 finishes and sends Update Receives Update 4 Ansewers Request 5b Worker 1 receives update Worker 'waits' (asks) for new update 6 Worker 2 receives update
43 Source Code (MR/YARN) Goes back to worker main function If there is a new update, it fetches some kind of data (most probably related to the model parameters such as weights) from the master server for the current update.
44 Distribution on Spark Create a Job Run Iteration (on all workers) Finish Similiar to Mapreduce/Yarn, only that do not have to configure the worker separately.
45 Source Code (For Spark) Within a spark job for training a dataset on a multilayer (neural) network. - on master The master service registeres the worker as waiting and gives back the current update Id.
46 Source Code (Spark Integration) Run an iteration... (1) Seems like we are spreading some params and an 'updater' to the workers. RDD?
47 RDD's? We know them already! - Quick lookback:
48 Source Code (Spark Integration) Run an iteration... (2)... and do even more parallel stuff. In the end integrate the results (from all workers) that is, we also have to wait here until each worker has finished.
49 Iterative Reduce
50 Iterative Reduce (2) Does this even make sense if we need to wait always for the completion of all worker nodes?
51 DL4J Components Open Source Deep Learning Library What is DeepLearning and which part of it is covered by DL4j? (Section 1) CPU & GPU support Hadoop-Yarn (MR) & Spark integration What components are interfaced? (Section 2) How are the alogrithms distributed? (Section 3) Futre additions Section 4 / The End.
52 Outlook Google DistBelief + Apache Horn
53 GooGle Dist Belief Dean (2012) - Large Scale Distributed Deep Networks Figure 1: An example of model parallelism in DistBelief. A five layer deep neural network with local connectivity is shown here, partitioned across four machines (blue rectangles). Only those nodes with edges that cross partition boundaries (thick lines) will need to have their state transmitted between machines. Even in cases where a node has multiple edges crossing a partition boundary, its state is only sent to the machine on the other side of that boundary once. Within each partition, computation for individual nodes will the parallelized across all available CPU cores. Figure 2: Left: Downpour SGD. Model replicas asynchronously fetch parameters w and push gradients w to the parameter server. Right: Sandblaster L-BFGS. A single coordinator sends small messages to replicas and the parameter server to orchestrate batch optimization.
54 2012, DistBelief by Jeff Dean (Google) Taken from a Presentation by 윤진석
55 윤진석 Taken from a Presentation by 윤진석
56
Toward Scalable Deep Learning
한국정보과학회 인공지능소사이어티 머신러닝연구회 두번째딥러닝워크샵 2015.10.16 Toward Scalable Deep Learning 윤성로 Electrical and Computer Engineering Seoul National University http://data.snu.ac.kr Breakthrough: Big Data + Machine Learning
Large Scale Distributed Deep Networks
Large Scale Distributed Deep Networks Yifu Huang School of Computer Science, Fudan University huangyifu@fudan.edu.cn COMP630030 Data Intensive Computing Report, 2013 Yifu Huang (FDU CS) COMP630030 Report
CS 179 Lecture 16. Logistic Regression & Parallel SGD
CS 179 Lecture 16 Logistic Regression & Parallel SGD 1 Outline logistic regression (stochastic) gradient descent parallelizing SGD for neural nets (with emphasis on Google s distributed neural net implementation)
Parallel Implementation of Deep Learning Using MPI
Parallel Implementation of Deep Learning Using MPI CSE633 Parallel Algorithms (Spring 2014) Instructor: Prof. Russ Miller Team #13: Tianle Ma Email: tianlema@buffalo.edu May 7, 2014 Content Introduction
Deep Learning. Volker Tresp Summer 2014
Deep Learning Volker Tresp Summer 2014 1 Neural Network Winter and Revival While Machine Learning was flourishing, there was a Neural Network winter (late 1990 s until late 2000 s) Around 2010 there
Machine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
TensorFlow: A System for Learning-Scale Machine Learning. Google Brain
TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine learning is everywhere This is in large part due to: 1. Invention of more sophisticated machine learning models
Deep Learning. Volker Tresp Summer 2015
Deep Learning Volker Tresp Summer 2015 1 Neural Network Winter and Revival While Machine Learning was flourishing, there was a Neural Network winter (late 1990 s until late 2000 s) Around 2010 there
COMP 551 Applied Machine Learning Lecture 16: Deep Learning
COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all
Artificial Neural Networks. Introduction to Computational Neuroscience Ardi Tampuu
Artificial Neural Networks Introduction to Computational Neuroscience Ardi Tampuu 7.0.206 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition
Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda
Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda 1 Motivation And Intro Programming Model Spark Data Transformation Model Construction Model Training Model Inference Execution Model Data Parallel Training
Deep Learning Benchmarks Mumtaz Vauhkonen, Quaizar Vohra, Saurabh Madaan Collaboration with Adam Coates, Stanford Unviersity
Deep Learning Benchmarks Mumtaz Vauhkonen, Quaizar Vohra, Saurabh Madaan Collaboration with Adam Coates, Stanford Unviersity Abstract: This project aims at creating a benchmark for Deep Learning (DL) algorithms
Scaling Distributed Machine Learning
Scaling Distributed Machine Learning with System and Algorithm Co-design Mu Li Thesis Defense CSD, CMU Feb 2nd, 2017 nx min w f i (w) Distributed systems i=1 Large scale optimization methods Large-scale
Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech
Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:
CS-541 Wireless Sensor Networks
CS-541 Wireless Sensor Networks Lecture 14: Big Sensor Data Prof Panagiotis Tsakalides, Dr Athanasia Panousopoulou, Dr Gregory Tsagkatakis 1 Overview Big Data Big Sensor Data Material adapted from: Recent
Lecture 17: Neural Networks and Deep Learning. Instructor: Saravanan Thirumuruganathan
Lecture 17: Neural Networks and Deep Learning Instructor: Saravanan Thirumuruganathan Outline Perceptron Neural Networks Deep Learning Convolutional Neural Networks Recurrent Neural Networks Auto Encoders
Deep Learning Frameworks with Spark and GPUs
Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel,
Characterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
Asynchronous Parallel Stochastic Gradient Descent. A Numeric Core for Scalable Distributed Machine Learning Algorithms
Asynchronous Parallel Stochastic Gradient Descent A Numeric Core for Scalable Distributed Machine Learning Algorithms J. Keuper and F.-J. Pfreundt Competence Center High Performance Computing Fraunhofer
Characterizing and Benchmarking Deep Learning Systems on Modern Data Center Architectures
Characterizing and Benchmarking Deep Learning Systems on Modern Data Center Architectures Talk at Bench 2018 by Xiaoyi Lu The Ohio State University E-mail: luxi@cse.ohio-state.edu http://www.cse.ohio-state.edu/~luxi
COMP6237 Data Mining Data Mining & Machine Learning with Big Data. Jonathon Hare
COMP6237 Data Mining Data Mining & Machine Learning with Big Data Jonathon Hare jsh2@ecs.soton.ac.uk Contents Going to look at two case-studies looking at how we can make machine-learning algorithms work
End-to-End Training of Acoustic Models for Large Vocabulary Continuous Speech Recognition with TensorFlow
End-to-End Training of Acoustic Models for Large Vocabulary Continuous Speech Recognition with TensorFlow Ehsan Variani, Tom Bagby, Erik McDermott, Michiel Bacchiani Google Inc, Mountain View, CA, USA
LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University
LSTM and its variants for visual recognition Xiaodan Liang xdliang328@gmail.com Sun Yat-sen University Outline Context Modelling with CNN LSTM and its Variants LSTM Architecture Variants Application in
Harp-DAAL for High Performance Big Data Computing
Harp-DAAL for High Performance Big Data Computing Large-scale data analytics is revolutionizing many business and scientific domains. Easy-touse scalable parallel techniques are necessary to process big
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기 Introduction MapReduce is a programming model and an associated implementation for processing and generating large data set with parallel,
Deep Learning. Volker Tresp Summer 2017
Deep Learning Volker Tresp Summer 2017 1 Neural Network Winter and Revival While Machine Learning was flourishing, there was a Neural Network winter (late 1990 s until late 2000 s) Around 2010 there
Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies
http://blog.csdn.net/zouxy09/article/details/8775360 Automatic Colorization of Black and White Images Automatically Adding Sounds To Silent Movies Traditionally this was done by hand with human effort
Scaled Machine Learning at Matroid
Scaled Machine Learning at Matroid Reza Zadeh @Reza_Zadeh http://reza-zadeh.com Machine Learning Pipeline Learning Algorithm Replicate model Data Trained Model Serve Model Repeat entire pipeline Scaling
TENSORFLOW: LARGE-SCALE MACHINE LEARNING ON HETEROGENEOUS DISTRIBUTED SYSTEMS. by Google Research. presented by Weichen Wang
TENSORFLOW: LARGE-SCALE MACHINE LEARNING ON HETEROGENEOUS DISTRIBUTED SYSTEMS by Google Research presented by Weichen Wang 2016.11.28 OUTLINE Introduction The Programming Model The Implementation Single
Hadoop Map Reduce 10/17/2018 1
Hadoop Map Reduce 10/17/2018 1 MapReduce 2-in-1 A programming paradigm A query execution engine A kind of functional programming We focus on the MapReduce execution engine of Hadoop through YARN 10/17/2018
Deep Learning and Its Applications
Convolutional Neural Network and Its Application in Image Recognition Oct 28, 2016 Outline 1 A Motivating Example 2 The Convolutional Neural Network (CNN) Model 3 Training the CNN Model 4 Issues and Recent
Deep Learning with R. Francesca Lazzeri Data Scientist II - Microsoft, AI Research
with R Francesca Lazzeri - @frlazzeri Data Scientist II - Microsoft, AI Research Agenda with R What is Demo Better understanding of R DL tools Fundamental concepts in Forward Propagation Algorithm Activation
Review: The best frameworks for machine learning and deep learning
Review: The best frameworks for machine learning and deep learning infoworld.com/article/3163525/analytics/review-the-best-frameworks-for-machine-learning-and-deep-learning.html By Martin Heller Over the
Neural Network and Deep Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Neural Network and Deep Learning Early history of deep learning Deep learning dates back to 1940s: known as cybernetics in the 1940s-60s, connectionism in the 1980s-90s, and under the current name starting
Parallel Stochastic Gradient Descent: The case for native GPU-side GPI
Parallel Stochastic Gradient Descent: The case for native GPU-side GPI J. Keuper Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern, Germany Mark Silberstein Accelerated Computer
Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia
Deep learning for dense per-pixel prediction Chunhua Shen The University of Adelaide, Australia Image understanding Classification error Convolution Neural Networks 0.3 0.2 0.1 Image Classification [Krizhevsky
Residual Networks And Attention Models. cs273b Recitation 11/11/2016. Anna Shcherbina
Residual Networks And Attention Models cs273b Recitation 11/11/2016 Anna Shcherbina Introduction to ResNets Introduced in 2015 by Microsoft Research Deep Residual Learning for Image Recognition (He, Zhang,
Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,
Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image
SEMANTIC COMPUTING. Lecture 8: Introduction to Deep Learning. TU Dresden, 7 December Dagmar Gromann International Center For Computational Logic
SEMANTIC COMPUTING Lecture 8: Introduction to Deep Learning Dagmar Gromann International Center For Computational Logic TU Dresden, 7 December 2018 Overview Introduction Deep Learning General Neural Networks
Introduction to Deep Learning
ENEE698A : Machine Learning Seminar Introduction to Deep Learning Raviteja Vemulapalli Image credit: [LeCun 1998] Resources Unsupervised feature learning and deep learning (UFLDL) tutorial (http://ufldl.stanford.edu/wiki/index.php/ufldl_tutorial)
Face Recognition A Deep Learning Approach
Face Recognition A Deep Learning Approach Lihi Shiloh Tal Perl Deep Learning Seminar 2 Outline What about Cat recognition? Classical face recognition Modern face recognition DeepFace FaceNet Comparison
Machine Learning. MGS Lecture 3: Deep Learning
Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ Machine Learning MGS Lecture 3: Deep Learning Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ WHAT IS DEEP LEARNING? Shallow network: Only one hidden layer
Deep Learning for Vision: Tricks of the Trade
Deep Learning for Vision: Tricks of the Trade Marc'Aurelio Ranzato Facebook, AI Group www.cs.toronto.edu/~ranzato BAVM Friday, 4 October 2013 Ideal Features Ideal Feature Extractor - window, right - chair,
Deep Learning Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD.
Deep Learning 861.061 Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD asan.agibetov@meduniwien.ac.at Medical University of Vienna Center for Medical Statistics,
732A54/TDDE31 Big Data Analytics
732A54/TDDE31 Big Data Analytics Lecture 10: Machine Learning with MapReduce Jose M. Peña IDA, Linköping University, Sweden 1/27 Contents MapReduce Framework Machine Learning with MapReduce Neural Networks
Contents PART I: CLOUD, BIG DATA, AND COGNITIVE COMPUTING 1
Preface xiii PART I: CLOUD, BIG DATA, AND COGNITIVE COMPUTING 1 1 Princi ples of Cloud Computing Systems 3 1.1 Elastic Cloud Systems for Scalable Computing 3 1.1.1 Enabling Technologies for Cloud Computing
Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / ADD-IDAR
Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / 2017. 10. 31 syoh@add.re.kr Page 1/36 Overview 1. Introduction 2. Data Generation Synthesis 3. Distributed Deep Learning 4. Conclusions
Multi-Glance Attention Models For Image Classification
Multi-Glance Attention Models For Image Classification Chinmay Duvedi Stanford University Stanford, CA cduvedi@stanford.edu Pararth Shah Stanford University Stanford, CA pararth@stanford.edu Abstract We
Training Deep Neural Networks (in parallel)
Lecture 9: Training Deep Neural Networks (in parallel) Visual Computing Systems How would you describe this professor? Easy? Mean? Boring? Nerdy? Professor classification task Classifies professors as
Neural Networks for unsupervised learning From Principal Components Analysis to Autoencoders to semantic hashing
Neural Networks for unsupervised learning From Principal Components Analysis to Autoencoders to semantic hashing feature 3 PC 3 Beate Sick Many slides are taken form Hinton s great lecture on NN: https://www.coursera.org/course/neuralnets
Deep Learning for Vision
Deep Learning for Vision Presented by Kevin Matzen Quick Intro - DNN Feed-forward Sparse connectivity (layer to layer) Different layer types Recently popularized for vision [Krizhevsky, et. al. NIPS 2012]
Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved
Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop
Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY
Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company
SCALABLE DISTRIBUTED DEEP LEARNING
SEOUL Oct.7, 2016 SCALABLE DISTRIBUTED DEEP LEARNING Han Hee Song, PhD Soft On Net 10/7/2016 BATCH PROCESSING FRAMEWORKS FOR DL Data parallelism provides efficient big data processing: data collecting,
Programming Systems for Big Data
Programming Systems for Big Data CS315B Lecture 17 Including material from Kunle Olukotun Prof. Aiken CS 315B Lecture 17 1 Big Data We ve focused on parallel programming for computational science There
Decentralized and Distributed Machine Learning Model Training with Actors
Decentralized and Distributed Machine Learning Model Training with Actors Travis Addair Stanford University taddair@stanford.edu Abstract Training a machine learning model with terabytes to petabytes of
Spark Overview. Professor Sasu Tarkoma.
Spark Overview 2015 Professor Sasu Tarkoma www.cs.helsinki.fi Apache Spark Spark is a general-purpose computing framework for iterative tasks API is provided for Java, Scala and Python The model is based
Accelerating Convolutional Neural Nets. Yunming Zhang
Accelerating Convolutional Neural Nets Yunming Zhang Focus Convolutional Neural Nets is the state of the art in classifying the images The models take days to train Difficult for the programmers to tune
Neural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /10/2017
3/0/207 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/0/207 Perceptron as a neural
Lecture 19: Generative Adversarial Networks
Lecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling is a type of machine learning where the aim is to model the distribution that a given set of data (e.g. images,
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Grounded Compositional Semantics for Finding and Describing Images with Sentences R. Socher, A. Karpathy, V. Le,D. Manning, A Y. Ng - 2013 Ali Gharaee 1 Alireza Keshavarzi 2 1 Department of Computational
Putting it together. Data-Parallel Computation. Ex: Word count using partial aggregation. Big Data Processing. COS 418: Distributed Systems Lecture 21
Big Processing -Parallel Computation COS 418: Distributed Systems Lecture 21 Michael Freedman 2 Ex: Word count using partial aggregation Putting it together 1. Compute word counts from individual files
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Winter 215 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Due next Thursday evening Will send out reimbursement codes later
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
DeepFace: Closing the Gap to Human-Level Performance in Face Verification Report on the paper Artem Komarichev February 7, 2016 Outline New alignment technique New DNN architecture New large dataset with
Machine Learning in WAN Research
Machine Learning in WAN Research Mariam Kiran mkiran@es.net Energy Sciences Network (ESnet) Lawrence Berkeley National Lab Oct 2017 Presented at Internet2 TechEx 2017 Outline ML in general ML in network
Asynchronous Distributed Data Parallelism for Machine Learning
Asynchronous Distributed Data Parallelis for Machine Learning Zheng Yan, Yunfeng Shao Shannon Lab, Huawei Technologies Co., Ltd. Beijing, China, 100085 {yanzheng, shaoyunfeng}@huawei.co Abstract Distributed
Recurrent Neural Network (RNN) Industrial AI Lab.
Recurrent Neural Network (RNN) Industrial AI Lab. For example (Deterministic) Time Series Data Closed- form Linear difference equation (LDE) and initial condition High order LDEs 2 (Stochastic) Time Series
Machine Learning for Large-Scale Data Analysis and Decision Making A. Distributed Machine Learning Week #9
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Distributed Machine Learning Week #9 Today Distributed computing for machine learning Background MapReduce/Hadoop & Spark Theory
INTRODUCTION TO DEEP LEARNING
INTRODUCTION TO DEEP LEARNING CONTENTS Introduction to deep learning Contents 1. Examples 2. Machine learning 3. Neural networks 4. Deep learning 5. Convolutional neural networks 6. Conclusion 7. Additional
MapReduce Spark. Some slides are adapted from those of Jeff Dean and Matei Zaharia
MapReduce Spark Some slides are adapted from those of Jeff Dean and Matei Zaharia What have we learnt so far? Distributed storage systems consistency semantics protocols for fault tolerance Paxos, Raft,
MoonRiver: Deep Neural Network in C++
MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement
Deep Learning for Computer Vision with MATLAB By Jon Cherrie
Deep Learning for Computer Vision with MATLAB By Jon Cherrie 2015 The MathWorks, Inc. 1 Deep learning is getting a lot of attention "Dahl and his colleagues won $22,000 with a deeplearning system. 'We
Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context
1 Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes
Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python:
Code Mania 2019 Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: 1. Introduction to Artificial Intelligence 2. Introduction to python programming and Environment
Processing of big data with Apache Spark
Processing of big data with Apache Spark JavaSkop 18 Aleksandar Donevski AGENDA What is Apache Spark? Spark vs Hadoop MapReduce Application Requirements Example Architecture Application Challenges 2 WHAT
Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda
Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda 1 Observe novel applicability of DL techniques in Big Data Analytics. Applications of DL techniques for common Big Data Analytics problems. Semantic indexing
Machine Learning: Overview & Applications to Test
Machine Learning: Overview & Applications to Test 1st Lt Takayuki Iguchi 1st Lt Megan E. Lewis AFOTEC/Det 5/DTS Release Date: 6 MAR 17 Approved for Public Release: Distribution Unlimited. AFOTEC Public
Artificial Intelligence Introduction Handwriting Recognition Kadir Eren Unal ( ), Jakob Heyder ( )
Structure: 1. Introduction 2. Problem 3. Neural network approach a. Architecture b. Phases of CNN c. Results 4. HTM approach a. Architecture b. Setup c. Results 5. Conclusion 1.) Introduction Artificial
Autoencoder. Representation learning (related to dictionary learning) Both the input and the output are x
Deep Learning 4 Autoencoder, Attention (spatial transformer), Multi-modal learning, Neural Turing Machine, Memory Networks, Generative Adversarial Net Jian Li IIIS, Tsinghua Autoencoder Autoencoder Unsupervised
Convolutional Neural Networks
Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
Research challenges in data-intensive computing The Stratosphere Project Apache Flink
Research challenges in data-intensive computing The Stratosphere Project Apache Flink Seif Haridi KTH/SICS haridi@kth.se e2e-clouds.org Presented by: Seif Haridi May 2014 Research Areas Data-intensive
Parallel Deep Network Training
Lecture 26: Parallel Deep Network Training Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2016 Tunes Speech Debelle Finish This Album (Speech Therapy) Eat your veggies and study
Dynamic Routing Between Capsules
Report Explainable Machine Learning Dynamic Routing Between Capsules Author: Michael Dorkenwald Supervisor: Dr. Ullrich Köthe 28. Juni 2018 Inhaltsverzeichnis 1 Introduction 2 2 Motivation 2 3 CapusleNet
6.830 Lecture Spark 11/15/2017
6.830 Lecture 19 -- Spark 11/15/2017 Recap / finish dynamo Sloppy Quorum (healthy N) Dynamo authors don't think quorums are sufficient, for 2 reasons: - Decreased durability (want to write all data at
Neural Network Architectures for Large-scale 3d Reconstruction. Viren Jain Research & Machine Intelligence, Google Inc.
Neural Network Architectures for Large-scale 3d Reconstruction Viren Jain Research & Machine Intelligence, Google Inc. reconstructing neural wiring diagrams from 3d, nanoscale images of brain tissue connectomics
Spark, Shark and Spark Streaming Introduction
Spark, Shark and Spark Streaming Introduction Tushar Kale tusharkale@in.ibm.com June 2015 This Talk Introduction to Shark, Spark and Spark Streaming Architecture Deployment Methodology Performance References
COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 7. Image Processing COMP9444 17s2 Image Processing 1 Outline Image Datasets and Tasks Convolution in Detail AlexNet Weight Initialization Batch Normalization
CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 12 Map-Reduce and Spark Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Practice midterm posted on sakai First prepare and
Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version
Shallow vs. deep networks Restricted Boltzmann Machines Shallow: one hidden layer Features can be learned more-or-less independently Arbitrary function approximator (with enough hidden units) Deep: two
Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal
Object Detection Lecture 10.3 - Introduction to deep learning (CNN) Idar Dyrdal Deep Learning Labels Computational models composed of multiple processing layers (non-linear transformations) Used to learn
Know your data - many types of networks
Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for
Chapter 4: Apache Spark
Chapter 4: Apache Spark Lecture Notes Winter semester 2016 / 2017 Ludwig-Maximilians-University Munich PD Dr. Matthias Renz 2015, Based on lectures by Donald Kossmann (ETH Zürich), as well as Jure Leskovec,
Convolutional Layer Pooling Layer Fully Connected Layer Regularization
Semi-Parallel Deep Neural Networks (SPDNN), Convergence and Generalization Shabab Bazrafkan, Peter Corcoran Center for Cognitive, Connected & Computational Imaging, College of Engineering & Informatics,
~~ Read Deep Learning in Python: Master Data Science and Machine Learning with Modern Neural Networks written in Python... how to download books
~~ Read Deep Learning in Python: Master Data Science and Machine Learning with Modern Neural Networks written in Python... how to download books online for free ID:biahqe Click Here to Read Deep Learning
High-Performance Data Loading and Augmentation for Deep Neural Network Training
High-Performance Data Loading and Augmentation for Deep Neural Network Training Trevor Gale tgale@ece.neu.edu Steven Eliuk steven.eliuk@gmail.com Cameron Upright c.upright@samsung.com Roadmap 1. The General-Purpose
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space Sikai Zhong February 14, 2018 COMPUTER SCIENCE Table of contents 1. PointNet 2. PointNet++ 3. Experiments 1 PointNet Property
Machine Learning in WAN Research
Machine Learning in WAN Research Mariam Kiran mkiran@es.net Energy Sciences Network (ESnet) Lawrence Berkeley National Lab Oct 2017 Presented at Internet2 TechEx 2017 Outline ML in general ML in network
Distributed Computation Models
Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case
Matrix Computations and " Neural Networks in Spark
Matrix Computations and " Neural Networks in Spark Reza Zadeh Paper: http://arxiv.org/abs/1509.02256 Joint work with many folks on paper. @Reza_Zadeh http://reza-zadeh.com Training Neural Networks Datasets