Introduction to Deep Learning in Signal Processing & Communications with MATLAB

Transcription

1 Introduction to Deep Learning in Signal Processing & Communications with MATLAB Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India 2019 The MathWorks, Inc. 1

2 Different Types of Machine Learning Type of Machine Learning Categories of Algorithms Supervised Learning Classification Output is a choice between classes (True, False) (Red, Blue, Green) Machine Learning Develop predictive model based on both input and output data Regression Output is a real number (temperature, stock prices) Unsupervised Learning Clustering No output - find natural groups and patterns from input data only Discover an internal representation from input data only 2

3 What is Deep Learning? 3

4 Deep learning is a type of supervised machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is usually implemented using a neural network. The term deep refers to the number of layers in the network the more layers, the deeper the network. 4

5 Why is Deep Learning So Popular Now? Human Accuracy Source: ILSVRC Top-5 Error on ImageNet 5

6 Vision applications have been driving the progress in deep learning producing surprisingly accurate systems 6

7 Deep Learning success enabled by: Labeled public datasets Progress in GPU for acceleration World-class models and connected community AlexNet PRETRAINED MODEL Caffe I M P O R T E R VGG-16 PRETRAINED MODEL GoogLeNet PRETRAINED MODEL ResNet-50 PRETRAINED MODEL TensorFlow- Keras I M P O R T E R ONNX Converter MODEL CONVERTER Inception-v3 M O D E L S 7

8 Deep Learning is Versatile MATLAB Examples Available Here 8

9 Many Network Architectures for Deep Learning Series Network Directed Acyclic Graph Network Recurrent Network and more (GAN, DQN, ) AlexNet YOLO ResNet R-CNN LSTM 9

10 Convolutional Neural Networks Edges Shapes Objects A lot of data is required Convolution layers Fully-connected layers 10

11 Deep Learning Inference in 4 Lines of Code >> net = alexnet; >> I = imread('peacock.jpg') >> I1 = imresize(i,[ ]); >> classify(net,i1) ans = categorical peacock 11

12 Understanding network behavior using visualizations Filters Deep Dream Activations Custom visualizations Example: Class Activation Maps (See blog post) 12

13 Visualization Technique Deep Dream deepdreamimage(... net, fc8', channel, 'NumIterations', 50,... 'PyramidLevels', 4,... 'PyramidScale', 1.25); Synthesizes images that strongly activate a channel in a particular layer Example Available Here 13

14 What is Training? Images Labels Large Data Set Network Training Trained Deep Neural Network During training, neural network architectures learn features directly from the data without the need for manual feature extraction 14

15 What Happens During Training? AlexNet Example Training Data Labels Layer weights are learned during training 15

16 Visualize Network Weights During Training AlexNet Example Training Data First Convolution Layer Labels 16

17 Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 17

18 Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 18

19 Deep Learning Workflow ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH DATA DATA MODELS SYSTEMS Files Data Augmentation/ Transformation Hardware-Accelerated Training Desktop Apps Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems Sensors Import Reference Models Network Visualization Embedded Devices and Hardware 19

20 Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Accessing reference models from research Understanding network behaviour Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 20

21 Working with Signal Data Signals Labels Sequence Layers Fully Connected Layers Trained Model Signals Transform Train Convolutional Labels Network (CNN) Trained Model 21

22 Long Short Term Memory Networks from RNNs Recurrent Neural Network that carries a memory cell throughout the process Sequence Problems Long term dependency does not work well with RNNs c 0 C 1 C t 22

23 RNN to LSTM Cell state Forget gate Input gate Update Output gate 23

24 Test Data Training Data Example: Speaker Gender Recognition Using Deep Learning LSTM Network for Audio based Speaker Classification audiodatastore 1. Train Male Female Mozilla Common Voice Dataset Time-varying features: - MFCC - Pitch Predict Male Female 24

25 Some audio and speech applications following CNN workflows 27

26 28

27 Solution2: Speech Command Recognition with Deep Learning(MATLAB) Raw Data Preprocessed Data Models and Algorithms Integrated Analytic Systems... 29

28 43

29 44

30 45

31 46

32 Data augmentation allows training more advanced networks and generating more robust models Original Dataset Simulate: - lower voice - varying level of environmental interference - recording hardware effects - distortions due to use of different microphones - different speaker vs microphone positions or room sizes in enclosed environments

33 Parallel Serial % Cycle continuously and automatically through files in datastore mfccfile = zeros(numel(ads.files),nummfcc) while hasdata(ads) [data,info] = read(ads); % do something with data end % Partition and explicit parfor parallel processing pattern N = numpartitions(ads); parfor index=1:n subads = partition(ads,n,index); while hasdata(subads) data = read(subads); % do something with data end end % Tall array and "implicit" parallel processing pattern T = tall(ads); dcoffsets = cellfun(@(x) mean(x), T,'UniformOutput',false); gather(dcoffsets); 51

34 Package Speech Recognition App 52

35 53

36 BREAK 54

37 Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Not a deep learning expert Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Understanding network behavior Accessing reference models from research Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 55

38 BREAK 56

39 57

40 Speech 2 text intro 58

41 59

42 60

43 61

44 62

45 Audio Labeler Work on collections of recordings or record new audio directly within the app Navigate dataset and playback interactively Define and apply labels to Entire files Regions within files Import and export audio folders, label definitions and datastores 63

46 Apps for Data Labeling 64

47 Label Images Using Image Labeler App 65

48 Accelerate Labeling With Automation Algorithms Learn More 66

49 Perform Bootstrapping to Label Large Datasets Images Videos Labeling Apps Ground Truth Train Network Improve Network More Ground Truth Propose Labels Automation Algorithm 67

50 Example Semantic Segmentation Available Here Classify pixels into 11 classes Sky, Building, Pole, Road, Pavement, Tree, SignSymbol, Fence, Car, Pedestrian, Bicyclist CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in video: A high-definition ground truth database." Pattern Recognition Letters Vol 30, Issue 2, 2009, pp

51 Example: Singing Voice Separation Source separation Based on U-Net architecture 69

52 Synthetically generating labeled data 70

53 Modulation Classification with Deep Learning Generate synthetic modulated signals Apply channel impairments Train a CNN to classify modulation types 71

54 Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Understanding network behavior Accessing reference models from research Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 72

55 Transfer Learning 8 lines of MATLAB Code Load Training Data Load Pre-Trained Network Replace Final Layers Train Network on New Data %% Create a datastore imds = imagedatastore( Data,... 'IncludeSubfolders',true,'LabelSource','foldernames'); num_classes = numel(unique(imds.labels)); %% Load Referece Network net = alexnet; layers = net.layers %% Replace Final Layers layers(end-2) = fullyconnectedlayer(num_classes,'name',[ fc5']); layers(end) = classificationlayer('name','classout'); %% Set Training Options & Train Network trainopts = trainingoptions('sgdm,... 'MiniBatchSize', 64); net = trainnetwork(imds, layers, trainopts); 73

56 Tune Hyperparameters to Improve Training Many hyperparameters depth, layers, solver options, learning rates, regularization, Techniques Parameter sweep Bayesian optimization 74

57 Keras-Tensorflow Importer 75

58 Model Exchange and Co-execution MATLAB Python Model Exchange Co-execution 76

59 Model Exchange With Deep Learning Frameworks PyTorch Caffe2 TensorFlow TensorFlow- Keras MXNet ONNX MATLAB Core ML Chainer Cognitive Toolkit Caffe ONNX = Open Neural Network Exchange Format 77

60 Interoperate With Deep Learning Frameworks Use Cases PyTorch Caffe2 TensorFlow Deployment Export MXNet ONNX MATLAB Augmentation and Optimization Import Core ML Chainer Cognitive Toolkit Visualization and Debugging ONNX = Open Neural Network Exchange Format 78

61 Model Exchange With Deep Learning Frameworks Caffe Model Importer importcaffelayers importcaffenetwork TensorFlow-Keras Model Importer importkeraslayers importkerasnetwork ONNX Converter importonnxnetwork exportonnxnetwork ONNX Converter 79

62 Model Exchange and Co-execution MATLAB Python Model Exchange Co-execution 80

63 MATLAB-Python Co-Execution The How Call Python file from MATLAB Call TensorFlow commands from MATLAB TF_code.py 81

64 MATLAB-Python Co-Execution Method A 1. Copy the code into a.py file 2. Wrap entry point in a function import tensorflow as tf from tf import keras def mytfcode(): for x in y: a.foo() TF_code.py def foo(x): return x Add module to Python path and then call from MATLAB via: py.mymodule.mytfcode(); 82

65 Deep Learning on CPU, GPU, Multi-GPU and Clusters H OW TO TA R GET? Single CPU Single CPU Single GPU Single CPU, Multiple GPUs On-prem server with GPUs Cloud GPUs (AWS) 83

66 MATLAB Containers for NVIDIA GPU Cloud & DGX 84

67 Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Accessing reference models from research Understanding network behaviour Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 85

68 Deploying Deep Learning Application Embedded Hardware Application logic Desktop, Web, Cloud Code Generation Application Deployment 86

69 MATLAB Production Server is an application server that publishes MATLAB code as APIs that can be called by other applications Analytics Development MATLAB Compiler SDK MATLAB Deploy Package Code / test Access Integrate Data sources / applications MATLAB Production Server Worker processes Request Broker Enterprise Application < > Mobile / Web Application Scale and secure 3 rd party dashboard 87

70 MATLAB support for Cloud 88

71 89

72 IAAS to PAAS 90

73 Automatic Cluster Creation 91

74 Get Public IP for access 92

75 Server Machine VM access MPS console endpoint: 93

76 Deploying Deep Learning Application Embedded Hardware Application logic Desktop, Web, Cloud Code Generation Application Deployment 94

77 Solution- GPU/MATLAB Coder for Deep Learning Deployment Target Libraries Application logic GPU Coder TensorRT & cudnn Libraries MATLAB Coder Intel MKL-DNN Library ARM Compute Library 95

78 Deep Learning Deployment Workflows INFERENCE ENGINE DEPLOYMENT INTEGRATED APPLICATION DEPLOYMENT Trained DNN Preprocessing Postprocessing cnncodegen codegen Portable target code Portable target code 96

79 Single Image Inference on Titan Xp using cudnn TensorFlow (1.8.0) MXNet (1.2.1) GPU Coder (R2018b) PyTorch (0.3.1) Intel Xeon CPU 3.6 GHz - NVIDIA libraries: CUDA9 - cudnn 7 97

80 NVIDIA Hardware Support Package (HSP) Simple out-of-box targeting for NVIDIA boards: INFERENCE ENGINE DEPLOYMENT Jetson Drive platform Trained DNN cnncodegen Portable target code 98

81 Summary- MATLAB Deep Learning Framework DEPLOYMENT Intel MKL-DNN Library Access Data Design + Train NVIDIA TensorRT & cudnn Libraries Manage large image sets Automate image labeling Easy access to models Acceleration with GPU s Scale to clusters ARM Compute Library 99

82 Summary Create and validate Deep learning models Automate ground truth labeling Access large amount of data from cluster/cloud Interoperability with Deep learning frameworks Visualization and hyperparameter tuning Seamlessly scale training to GPUs, clusters and cloud Deployment on embedded targets and web services 100

83 Available Here 101

84 102

85 103