Introduction to Deep Learning in Signal Processing & Communications with MATLAB

Introduction to Deep Learning in Signal Processing & Communications with MATLAB Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India 2019 The MathWorks, Inc. 1

Different Types of Machine Learning Type of Machine Learning Categories of Algorithms Supervised Learning Classification Output is a choice between classes (True, False) (Red, Blue, Green) Machine Learning Develop predictive model based on both input and output data Regression Output is a real number (temperature, stock prices) Unsupervised Learning Clustering No output - find natural groups and patterns from input data only Discover an internal representation from input data only 2

What is Deep Learning? 3

Deep learning is a type of supervised machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is usually implemented using a neural network. The term deep refers to the number of layers in the network the more layers, the deeper the network. 4

Why is Deep Learning So Popular Now? Human Accuracy Source: ILSVRC Top-5 Error on ImageNet 5

Vision applications have been driving the progress in deep learning producing surprisingly accurate systems 6

Deep Learning success enabled by: Labeled public datasets Progress in GPU for acceleration World-class models and connected community AlexNet PRETRAINED MODEL Caffe I M P O R T E R VGG-16 PRETRAINED MODEL GoogLeNet PRETRAINED MODEL ResNet-50 PRETRAINED MODEL TensorFlow- Keras I M P O R T E R ONNX Converter MODEL CONVERTER Inception-v3 M O D E L S 7

Deep Learning is Versatile MATLAB Examples Available Here 8

Many Network Architectures for Deep Learning Series Network Directed Acyclic Graph Network Recurrent Network and more (GAN, DQN, ) AlexNet YOLO ResNet R-CNN LSTM 9

Convolutional Neural Networks Edges Shapes Objects A lot of data is required Convolution layers Fully-connected layers 10

Deep Learning Inference in 4 Lines of Code >> net = alexnet; >> I = imread('peacock.jpg') >> I1 = imresize(i,[227 227]); >> classify(net,i1) ans = categorical peacock 11

Understanding network behavior using visualizations Filters Deep Dream Activations Custom visualizations Example: Class Activation Maps (See blog post) 12

Visualization Technique Deep Dream deepdreamimage(... net, fc8', channel, 'NumIterations', 50,... 'PyramidLevels', 4,... 'PyramidScale', 1.25); Synthesizes images that strongly activate a channel in a particular layer Example Available Here 13

What is Training? Images Labels Large Data Set Network Training Trained Deep Neural Network During training, neural network architectures learn features directly from the data without the need for manual feature extraction 14

What Happens During Training? AlexNet Example Training Data Labels Layer weights are learned during training 15

Visualize Network Weights During Training AlexNet Example Training Data First Convolution Layer Labels 16

Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 17

Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 18

Deep Learning Workflow ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH DATA DATA MODELS SYSTEMS Files Data Augmentation/ Transformation Hardware-Accelerated Training Desktop Apps Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems Sensors Import Reference Models Network Visualization Embedded Devices and Hardware 19

Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Accessing reference models from research Understanding network behaviour Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 20

Working with Signal Data Signals Labels Sequence Layers Fully Connected Layers Trained Model Signals Transform Train Convolutional Labels Network (CNN) Trained Model 21

Long Short Term Memory Networks from RNNs Recurrent Neural Network that carries a memory cell throughout the process Sequence Problems Long term dependency does not work well with RNNs c 0 C 1 C t 22

RNN to LSTM Cell state Forget gate Input gate Update Output gate 23

Test Data Training Data Example: Speaker Gender Recognition Using Deep Learning LSTM Network for Audio based Speaker Classification audiodatastore 1. Train Male Female Mozilla Common Voice Dataset Time-varying features: - MFCC - Pitch -... 2. Predict Male Female 24

Some audio and speech applications following CNN workflows 27

https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf 28

Solution2: Speech Command Recognition with Deep Learning(MATLAB) Raw Data Preprocessed Data Models and Algorithms Integrated Analytic Systems... 29

43

44

45

46

Data augmentation allows training more advanced networks and generating more robust models Original Dataset Simulate: - lower voice - varying level of environmental interference - recording hardware effects - distortions due to use of different microphones - different speaker vs microphone positions or room sizes in enclosed environments -... 50

Parallel Serial % Cycle continuously and automatically through files in datastore mfccfile = zeros(numel(ads.files),nummfcc) while hasdata(ads) [data,info] = read(ads); % do something with data end % Partition and explicit parfor parallel processing pattern N = numpartitions(ads); parfor index=1:n subads = partition(ads,n,index); while hasdata(subads) data = read(subads); % do something with data end end % Tall array and "implicit" parallel processing pattern T = tall(ads); dcoffsets = cellfun(@(x) mean(x), T,'UniformOutput',false); gather(dcoffsets); 51

Package Speech Recognition App 52

53

BREAK 54

Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Not a deep learning expert Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Understanding network behavior Accessing reference models from research Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 55

BREAK 56

57

Speech 2 text intro 58

59

60

61

62

Audio Labeler Work on collections of recordings or record new audio directly within the app Navigate dataset and playback interactively Define and apply labels to Entire files Regions within files Import and export audio folders, label definitions and datastores 63

Apps for Data Labeling 64

Label Images Using Image Labeler App 65

Accelerate Labeling With Automation Algorithms Learn More 66

Perform Bootstrapping to Label Large Datasets Images Videos Labeling Apps Ground Truth Train Network Improve Network More Ground Truth Propose Labels Automation Algorithm 67

Example Semantic Segmentation Available Here Classify pixels into 11 classes Sky, Building, Pole, Road, Pavement, Tree, SignSymbol, Fence, Car, Pedestrian, Bicyclist CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in video: A high-definition ground truth database." Pattern Recognition Letters Vol 30, Issue 2, 2009, pp 88-97. 68

Example: Singing Voice Separation Source separation Based on U-Net architecture 69

Synthetically generating labeled data 70

Modulation Classification with Deep Learning Generate synthetic modulated signals Apply channel impairments Train a CNN to classify modulation types 71

Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Understanding network behavior Accessing reference models from research Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 72

Transfer Learning 8 lines of MATLAB Code Load Training Data Load Pre-Trained Network Replace Final Layers Train Network on New Data 1 2 3 4 5 6 7 8 %% Create a datastore imds = imagedatastore( Data,... 'IncludeSubfolders',true,'LabelSource','foldernames'); num_classes = numel(unique(imds.labels)); %% Load Referece Network net = alexnet; layers = net.layers %% Replace Final Layers layers(end-2) = fullyconnectedlayer(num_classes,'name',[ fc5']); layers(end) = classificationlayer('name','classout'); %% Set Training Options & Train Network trainopts = trainingoptions('sgdm,... 'MiniBatchSize', 64); net = trainnetwork(imds, layers, trainopts); 73

Tune Hyperparameters to Improve Training Many hyperparameters depth, layers, solver options, learning rates, regularization, Techniques Parameter sweep Bayesian optimization 74

Keras-Tensorflow Importer 75

Model Exchange and Co-execution MATLAB Python Model Exchange Co-execution 76

Model Exchange With Deep Learning Frameworks PyTorch Caffe2 TensorFlow TensorFlow- Keras MXNet ONNX MATLAB Core ML Chainer Cognitive Toolkit Caffe ONNX = Open Neural Network Exchange Format 77

Interoperate With Deep Learning Frameworks Use Cases PyTorch Caffe2 TensorFlow Deployment Export MXNet ONNX MATLAB Augmentation and Optimization Import Core ML Chainer Cognitive Toolkit Visualization and Debugging ONNX = Open Neural Network Exchange Format 78

Model Exchange With Deep Learning Frameworks Caffe Model Importer importcaffelayers importcaffenetwork TensorFlow-Keras Model Importer importkeraslayers importkerasnetwork ONNX Converter importonnxnetwork exportonnxnetwork ONNX Converter 79

Model Exchange and Co-execution MATLAB Python Model Exchange Co-execution 80

MATLAB-Python Co-Execution The How Call Python file from MATLAB Call TensorFlow commands from MATLAB TF_code.py 81

MATLAB-Python Co-Execution Method A 1. Copy the code into a.py file 2. Wrap entry point in a function import tensorflow as tf from tf import keras def mytfcode(): for x in y: a.foo() TF_code.py def foo(x): return x + 1 3. Add module to Python path and then call from MATLAB via: py.mymodule.mytfcode(); 82

Deep Learning on CPU, GPU, Multi-GPU and Clusters H OW TO TA R GET? Single CPU Single CPU Single GPU Single CPU, Multiple GPUs On-prem server with GPUs Cloud GPUs (AWS) 83

MATLAB Containers for NVIDIA GPU Cloud & DGX 84

Deep Learning Challenges Data Handling large amounts of data Labeling thousands of signals, images & videos Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks Accessing reference models from research Understanding network behaviour Optimizing hyperparameters Training takes hours-days Rapid and Optimized Deployment Desktop, web, cloud, and embedded hardware 85

Deploying Deep Learning Application Embedded Hardware Application logic Desktop, Web, Cloud Code Generation Application Deployment 86

MATLAB Production Server is an application server that publishes MATLAB code as APIs that can be called by other applications Analytics Development MATLAB Compiler SDK MATLAB Deploy Package Code / test Access Integrate Data sources / applications MATLAB Production Server Worker processes Request Broker Enterprise Application < > Mobile / Web Application Scale and secure 3 rd party dashboard 87

MATLAB support for Cloud 88

89

IAAS to PAAS 90

Automatic Cluster Creation 91

Get Public IP for access 92

Server Machine VM access MPS console endpoint: https://xxx.xx.xx.xx 93

Deploying Deep Learning Application Embedded Hardware Application logic Desktop, Web, Cloud Code Generation Application Deployment 94

Solution- GPU/MATLAB Coder for Deep Learning Deployment Target Libraries Application logic GPU Coder TensorRT & cudnn Libraries MATLAB Coder Intel MKL-DNN Library ARM Compute Library 95

Deep Learning Deployment Workflows INFERENCE ENGINE DEPLOYMENT INTEGRATED APPLICATION DEPLOYMENT Trained DNN Preprocessing Postprocessing cnncodegen codegen Portable target code Portable target code 96

Single Image Inference on Titan Xp using cudnn TensorFlow (1.8.0) MXNet (1.2.1) GPU Coder (R2018b) PyTorch (0.3.1) Intel Xeon CPU 3.6 GHz - NVIDIA libraries: CUDA9 - cudnn 7 97

NVIDIA Hardware Support Package (HSP) Simple out-of-box targeting for NVIDIA boards: INFERENCE ENGINE DEPLOYMENT Jetson Drive platform Trained DNN cnncodegen Portable target code 98

Summary- MATLAB Deep Learning Framework DEPLOYMENT Intel MKL-DNN Library Access Data Design + Train NVIDIA TensorRT & cudnn Libraries Manage large image sets Automate image labeling Easy access to models Acceleration with GPU s Scale to clusters ARM Compute Library 99

Summary Create and validate Deep learning models Automate ground truth labeling Access large amount of data from cluster/cloud Interoperability with Deep learning frameworks Visualization and hyperparameter tuning Seamlessly scale training to GPUs, clusters and cloud Deployment on embedded targets and web services 100

Available Here 101

102

103