Adam Paszke, Sam Gross, Soumith Chintala, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia

Transcription

1

2 Adam Paszke, Sam Gross, Soumith Chintala, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Alban Desmaison, Andreas Kopf, Edward Yang, Zach Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy & Team

3 What is PyTorch?

4 What is PyTorch efficient ndarray library with GPU support automatic differentiation engine machine learning primitives gradient based optimization package Machine Learning Ecosystem CUDA Probabilistic Modeling Data Loading NumPy like interface Deep Learning Visualization Reinforcement Learning Utility packages for image and text data

5 Tensor (ndarray) library

6 Tensor (ndarray) library NumPy-like API ( np.ndarray <-> torch.tensor ) Very fast acceleration on NVIDIA GPUs

7 Tensor (ndarray) library NumPy-like API ( np.ndarray <-> torch.tensor ) - Easy creation of Tensors with various dtypes and on different devices - Scientific computing methods (linear algebra, reduction, etc.) - Fast conversion from and to np.ndarray

8 Tensor (ndarray) library NumPy-like API ( np.ndarray <-> torch.tensor )

9 ndarray library Numpy PyTorch

10 Tensor (ndarray) library Very fast acceleration on NVIDIA GPUs - Efficient scientific computing on GPU - Clean device management - Seamless conversion to and from CPU

11 Tensor (ndarray) library Very fast acceleration on NVIDIA GPUs

12 Automatic Differentiation Engine

13 Automatic Differentiation Engine Computation as a graph built on-the-fly

14 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10)

15 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h i2h

16 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) h2h i2h

17 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() h2h Add Tanh i2h

18 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() h2h Add Tanh i2h loss = next_h.sum() loss

19 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() h2h Add Tanh i2h loss = next_h.sum() loss.backward() loss

20 Automatic Differentiation Engine Computation as a graph built on-the-fly x = torch.randn(1, 10) prev_h = torch.randn(1, 20) W_h = torch.randn(20, 20) W_x = torch.randn(20, 10) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() h2h Add Tanh i2h loss = next_h.sum() loss.backward() # compute gradients! Gradient w.r.t. the input Tensors is computed step-by-step from loss to top in reverse order loss

21 Automatic Differentiation Engine Computation as a graph built on-the-fly - Can use Python primitives to build the graph (e.g. for-loop) Functions with efficient backward implementations - torch.matmul, torch.fft, torch.svd, torch.trtrs, etc. Gradients by automatic backpropagation through the graph - Higher-order gradients (backward traversal is also a graph) - Multi-device graphs

22 Efficient Machine Learning Primitives

23 Machine Learning primitives Deep Learning - torch.nn Reinforcement Learning, Probabilistic Modeling - torch.distributions and more... - scientific computing methods + autograd

24 Neural Networks Figure by Md. Rezaul Karim

25 Neural Networks Figure by Md. Rezaul Karim

26 Convolutional Neural Networks (CNN) Figure by Yann LeCun et al.

27 Convolutional Neural Networks (CNN) Figure by Yann LeCun et al.

28 Recurrent Neural Networks (RNN) Figure by Pranoy Radhakrishnan

29 Recurrent Neural Networks (RNN) Figure by Pranoy Radhakrishnan

30 Gradient based optimization package

31 Gradient based optimization package State-of-the-art optimization algorithms - torch.optim.* - SGD, Adam, RMSProp, L-BFGS, etc. Learning Rate scheduler - torch.optim.lr_scheduler.* Extensible API

32 Machine Learning Ecosystem

33 Work items in practice Writing Building models Implementing Checkpointing Dataset loaders Training loop models Interfacing with Dealing with Building Building optimizers environments GPUs Baselines

34 Work items in practice Writing Building models Implementing Checkpointing Dataset loaders Training loop models Python + PyTorch - an environment to do all of this Interfacing with Dealing with Building Building optimizers environments GPUs Baselines

35 Writing Data Loaders every dataset is slightly differently formatted

36 Writing Data Loaders every dataset is slightly differently formatted have to be preprocessed and normalized differently

37 Writing Data Loaders every dataset is slightly differently formatted have to be preprocessed and normalized differently need a multithreaded Data loader to feed GPUs fast enough

38 Writing Data Loaders PyTorch solution: share data loaders across the community!

39 Writing Data Loaders PyTorch solution: share data loaders across the community!

40 Writing Data Loaders PyTorch solution: use regular Python to write Datasets: leverage existing Python code

41 Writing Data Loaders PyTorch solution: use regular Python to write Datasets: leverage existing Python code Example: ParlAI

42 Writing Data Loaders PyTorch solution: Code in practice

43 Writing Data Loaders PyTorch solution: Code in practice Research Workflows Pain Points Core Philisophy Upcoming Features

44 Writing Data Loaders PyTorch solution: Code in practice

45 Interfacing with environments Cars Simulations Video games Internet

46 Interfacing with environments Cars Simulations Video games Internet Pretty much every environment provides a Python API

47 Interfacing with environments Cars Simulations Video games Internet Natively interact with the environment directly

48 Visualization Tensorboard-PyTorch Visdom github.com/lanpa/tensorboard-pytorch github.com/facebookresearch/visdom

49 Debugging PyTorch is a Python extension

50 Debugging PyTorch is a Python extension Use your favorite Python debugger

51 Debugging PyTorch is a Python extension Use your favorite Python debugger

52 Debugging PyTorch is a Python extension Use your favorite Python debugger Use the most popular debugger:

53 Debugging PyTorch is a Python extension Use your favorite Python debugger Use the most popular debugger: print(foo)

54 Identifying bottlenecks PyTorch is a Python extension Use your favorite Python profiler

55 Identifying bottlenecks PyTorch is a Python extension Use your favorite Python profiler: Line_Profiler

56 Compilation Time PyTorch is written for the impatient

57 Compilation Time PyTorch is written for the impatient Absolutely no compilation time when writing your scripts whatsoever

58 Compilation Time PyTorch is written for the impatient Absolutely no compilation time when writing your scripts whatsoever All core kernels are pre-compiled

59 Ecosystem Use the entire Python ecosystem at your will

60 Ecosystem Use the entire Python ecosystem at your will Including SciPy, Scikit-Learn, etc.

61 Ecosystem Use the entire Python ecosystem at your will Including SciPy, Scikit-Learn, etc.

62 Machine Learning Ecosystem Shared model zoo (pretrained models) torchvision

63 Machine Learning Ecosystem Shared model zoo (pretrained models) github.com/aaron-xichen/pytorch-playground

64 Machine Learning Ecosystem Probabilistic programming github.com/probtorch/probtorch

65 Machine Learning Ecosystem Gaussian Processes

66 Machine Learning Ecosystem Image-to-Image Translation pix2pix & CycleGAN github.com/junyanz/pytorch-cyclegan-and-pix2pix pix2pixhd github.com/nvidia/pix2pixhd

67 Machine Learning Ecosystem Machine Translation OpenNMT-py: Open-Source Neural Machine Translation github.com/opennmt/opennmt-py FAIR Sequence-to-Sequence Toolkit github.com/facebookresearch/fairseq-py

68 Machine Learning Ecosystem General Linguistics Tasks

69 Machine Learning Ecosystem Sentiment Discovery

70 Machine Learning Ecosystem Optical Flow Estimation FlowNet 2.0 github.com/nvidia/flownet2-pytorch

71 and many more... Distributed training Profiling Extending autograd C++ interface (ATen) and C++ extensions ONNX JIT...

72 With from