Table of Contents. What Really is a Hidden Unit? Visualizing Feed-Forward NNs. Visualizing Convolutional NNs. Visualizing Recurrent NNs

Transcription

1

2 Table of Contents What Really is a Hidden Unit? Visualizing Feed-Forward NNs Visualizing Convolutional NNs Visualizing Recurrent NNs Visualizing Attention Visualizing High Dimensional Data What do visualizations get us now?

3 What Really is a Hidden Unit? Some operation that outputs a weighted value. Sometimes its between -1 and +1 or its just weighted by 1 resulting in a linear value Lets say between -1 and +1 for this example -1 This is really a projection into a weighted space +1

4 Let s Look at a Distribution We Want to Model Simple distribution of blue points totally surrounded by red points

5 Let s Put This Data Into Random Hidden Units Still not linearly separable Still not linearly separable Finally linearly separable We just visualized a simple hidden layer!

6 Let s Put This Data Into Random Hidden Units Still not linearly separable Still not linearly separable Finally linearly separable This distribution is obviously not linearly separable To separate this data with a NN we need multiple layers

7 Let s issee This how A you Network visualize of Layers a basic Like NN... These Learning... Let s imagine a network structured like this is it able to learn this distribution?

8 Visualize a basic NN... Let s look at some more networks

9 Paper Here: Visualize a Convolutional NN...

10 Paper Here: Visualize a Convolutional NN...

11 Paper Here: Visualize a Convolutional NN...

12 Paper Here: Visualize a Convolutional NN... Layer 4 Layer 5

13 Visualize a Convolutional NN... Let s look at a more interactive visualization

14 What about an RNN Turns out that its similar but different

15 Visualize a Recurrent NN... Activations that change dependent on the input Paper Here:

16 Let s See aarecurrent Visualize Network ofnn... Layers Like These Learning... Average LSTM Gate positions dependent on their inputs Paper Here:

17 Let s See aarecurrent Visualize Network ofnn... Layers Like These Learning... Average GRU Gate positions dependent on their inputs Paper Here:

18 Let s See AAttention... Visualizing Network of Layers Like These Learning... First let's talk about attention we have not covered it yet. This is going to be handwavy, but the professor will cover this in lecture shortly Note that there is only one value calculated per input Outputs Attention Values (Calculated by the attention model) Inputs Paper Here:

19 Let s See AAttention... Visualizing Network of Layers Like These Learning... a() is a learned neural network Paper Here:

20 Visualizing Attention... We can visualize these values!

21 Visualizing Attention... Soft Attention (Softmax Activation - Differentiable) Hard Attention (Step Function Activation - Not Differentiable) Paper Here:

22 Visualizing Attention... Paper Here:

23 Let s See AAttention... Visualizing Network of Layers Like These Learning... Sometimes this can show us where our errors are... Paper Here:

24 Let s See AAttention... Visualizing Network of Layers Like These Learning... You can also do this with Seq2Seq Models Don t these look a lot like word alignment charts Topic for another time Paper Here:

25 Let s See AHigh Visualizing Network Dimensional of LayersData Like These Learning... How do you view data that is in more than 3 dimensions? Projections! Paper Here:

26 Visualizing Let s See AHigh Network Dimensional of LayersData Like These Learning... (Dimensionality Reduction) Principal Component Analysis Mathematical Projection of Principal Components Common for visualizing embedding spaces (deterministic) Great PCA Reading here: T-Distributed Stochastic Neighbor Embedding (t-sne) Stochastic Projection Trained using Gradient Descent Using KL Divergence Common for visualizing training data (non-deterministic) Paper Here:

27 Visualizing Let s See AHigh Network Dimensional of LayersData Like These Learning... (Dimensionality Reduction) Let s look at some MNIST Examples Paper Here:

28 Visualizing Let s See AHigh Network Dimensional of LayersData Like These Learning... (Dimensionality Reduction) Let s look at some MNIST Examples Paper Here:

29 Visualizing Let s See AHigh Network Dimensional of LayersData Like These Learning... (Dimensionality Reduction) Let s look at some of our own examples ZwFPNxAhFlF6VMTy#scrollTo=ULqt5rdaQPoi Paper Here:

30 Let s See What do visualizations A Network of get Layers us now? Like These Learning... Right now nothing NNs are still black boxes No way to really use these visualizations to help improve our models They are currently just cool images to look at... maybe later it will be better

31 Let s See A Network of Layers Like These Learning... Conclusion Any Questions?