Deep Learning for LSST

Transcription

1 6/20/18 Deep Learning for LSST Presented By: Aaron D. Saxton, PhD

2 Warm Up Getting Started Login to your account module load bwpy git clone Copy url s from /projects/training/baui/urls Quality of life ssh setting In file ~/.ssh/config Host * Controlmaster auto ControlPath ~/.ssh/master-%r@%h:%p ControlPersist yes ServerAliveInterval 50 2

3 ImageNet Large High Quality Dataset 14,197,122 Images synsets Runs the Large Scale Visual Recognition Challenge (ILSVRC) Annotated Bounding Boxes synset WordNet ( 3

4 ImageNet Blue Waters hosts copy of ImageNet Legal Term of Access Create account on Navigate to Term of Access Accept Term of Access Take screen shot or print to PDF Term of Access with your name on it. to After I receive your Term of Access I will give your Blue Waters user read permission to data 4

5 ImageNet Hands On: Explore Images (start train processing) 5

6 Blue Waters Overview Brief Summary AMD Interlagos NVIDIA Tesla 22,636 XE Compute Nodes 4,228 XK Compute Nodes Cray Gemini Interconnect 6

7 Blue Waters Overview 2 XK Nodes 4 XE Nodes 7

8 Blue Waters Overview PFLOPS 1.66 PB Scuba Subsystem: Storage Configuration for User Best Access 10/40/100 Gb Ethernet Switch 100 GB/sec External Servers IB Switch >1 TB/sec 400+ Gb/sec WAN Spectra Logic: 200 usable PB Sonexion: 26 usable PB

9 ImageNet Hands On: Explore Images (start validation processing) 9

10 Statistics Review Simple y = m % x + b regression Least Squares to find m,b With data set { x *, y * } *./,..,1 Very special, often hard to measure y * Let the error be R = 1 *./[(y * m % x * + b ] 8 Minimize R with respect to m and b. Simultaneously Solve R 9 m, b = 0 R ; (m, b) = 0 Linear System We will consider more general y = f(x) R 9 m, b = 0 and R ; m, b = 0 may not be linear 10

11 Statistics Review Regressions with parameterized sets of functions. e.g. y = ax 8 + bx + c (quadratic) y = a * x * (polynomial) y = Ne CD (exponential) y = /EF / G(HIJK) (logistic) After optimal parameters found, Use function for inference Have x, compute y 11

12 Statistics Review Polynomial model of degree n degrees of freedom - models capacity Deep Learning, Goodfellow et. al., MIT Press,

13 Neural Networks Activation functions Logistic ReLU (Rectified Linear Unit) 1.5 σ x = σ x = Arctan σ x = Softmax g M x /, x 8,, x O = FK P F K Q 13

14 Neural Networks Parameterized function Z T = σ α V9 + α 9 X T Y = β VM + β M Z f Y (X) = g M (T) β V*, β *, α V9, α 9 Weights to be optimized X Z T Y 14

15 Neural Networks Finding Weights β V*, β *, α V9,α 9 Back propagation Nothing more than chain rule Take partial derivative of error function R These text is a good reference for nitty gritty details The Elements of Statistical Learning, Second Eddition, by Trevor Hastie, Robert Tibshirani, Jerome Friedman Deep Learning, Goodfellow et. al., MIT Press, Back propagation give errors (or loss) Gradient Decent tells you how to update weights 15

16 Convolutions For two functions, f x, g x (f g)(x) = ` a` f y g x y dy g is the kernel to f Above is a rolling average 16

17 Convolutional Neural Network: Inception V3 About 6.8 Million Unknowns ( Can be half precision 2014, Achieved 6.67% Error in Top 5 Recall 17

18 Convolutional Neural Networks b.io/the-9-deep-learning-papers-you-need-to- Know-About.html Highlights AlexNet VGG Net GoogLeNet (Inception) Microsoft ResNet 18

19 Distributed Training Replicated Model Data Distributed Rank 1 Rank 2 19

20 Distributed Training MPI Collectives all_reduce(sum) Trainable parameters and gradients 20

21 Pytorch, virtual enviroments, TorchVision mkdir virenv/lsstsp2018 virtualenv --system-site-packages lsstsp2018 source virenv/lsstsp2018/activate pip install torchvision Demo 21

22 Extra Topics Generalization Gap 22

23 Extra Topics import torch.nn as nn import torch.nn.functional as F class Net(nn.Module): def init (self): super(net, self). init () self.conv1 = nn.conv2d(3, 6, 5) self.pool = nn.maxpool2d(2, 2) self.conv2 = nn.conv2d(6, 16, 5) self.fc1 = nn.linear(16 * 5 * 5, 120) self.fc2 = nn.linear(120, 84) self.fc3 = nn.linear(84, 10) def forward(self, x): x = self.pool(f.relu(self.conv1(x))) x = self.pool(f.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x Much simpler model net = Net() 23

24 Extra Topics Object Localization 24