CIS 660. Image Searching System using CNN-LSTM. Presented by. Mayur Rumalwala Sagar Dahiwala

Transcription

1 CIS 660 using CNN-LSTM Presented by Mayur Rumalwala Sagar Dahiwala

2 AGENDA Problem in Image Searching? Proposed Solution Tools, Library and Dataset used Architecture of Proposed System Implementation of Algorithm CNN, LSTM

3 Problem in image searching? Current system search images, Based on title Based on description Based on META data Large Images set without description Instagram

4 Proposed Solution Two Different approaches we can think of, 1. Search image using similar image 2. Search image based on sentence (User query)

5 Tools, Library and Dataset used Image Dataset The Caltech Object Categories + Cluster At least 80 images per categories 30,608 images Text Dataset Cornell Movie Dialog corpus dataset 220,579 conversational exchanges between 10,292 pairs of movie characters involves 9,035 characters from 617 movies in total 304,713 utterances

6 Architecture of Proposed System

7 Image Processing - CNN Image representation based on RGB

8 Image Processing - CNN # Convolutional Layer 1. filter_size1 = 5 num_filters1 = 32 # Convolutional Layer 2. filter_size2 = 5 num_filters2 = 64 How filter is used

9 Image Processing - CNN

10 Image Processing - CNN # Rectified Linear Unit (ReLU). # It calculates max(x, 0) for each input pixel x. # This adds some nonlinearity to the formula layer = tf.nn.relu(layer) ReLu Replace negative values with zero

11 Image Processing - CNN # This is 2x2 max-pooling, which means that we # consider 2x2 windows and select the largest value # in each window. Then we move 2 pixels to the next window. layer = tf.nn.max_pool(value=layer, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='same') Max Pooling

12 Image Processing - CNN def new_conv_layer(input, # The previous layer. num_input_channels, # Num. channels in prev. layer. filter_size, # Width and height of each filter. num_filters, # Number of filters. use_pooling=true): # Use 2x2 max-pooling. # Shape of the filter-weights for the convolution. # This format is determined by the TensorFlow API. shape = [filter_size, filter_size, num_input_channels, num_filters] # Create new weights aka. filters with the given shape. weights = new_weights(shape=shape) # Create new biases, one for each filter. biases = new_biases(length=num_filters) # Create the TensorFlow operation for convolution. # Note the strides are set to 1 in all dimensions. # The first and last stride must always be 1, # because the first is for the image-number and # the last is for the input-channel. # But e.g. strides=[1, 2, 2, 1] would mean that the filter # is moved 2 pixels across the x- and y-axis of the image. # The padding is set to 'SAME' which means the input image # is padded with zeroes so the size of the output is the same. layer = tf.nn.conv2d(input=input, filter=weights, strides=[1, 1, 1, 1], padding='same')

13 Image Processing - CNN

14 User Query processing - RNN RNN Recurrent Neural Network RNN Recurrent Neural Network

15 User Query processing - RNN RNN Recurrent Neural Network

16 User Query processing - RNN RNN Recurrent Neural Network

17 User Query processing - RNN Vector Representation

18 User Query processing - RNN RNN Recurrent Neural Network

19 User Query processing - LSTM Element by Element addition (+) Element by Element Multiplication (X) Memory (M) Squashing function (f)

20 User Query processing - NLTK Sentence : cat with my car + = match 1 or more? = match 0 or 1 repetitions. * = match 0 or MORE repetitions. = Any character except a new line chunkgram = r"""chunk: {<RB.?>*<VB.?>*<NNP>+<NN>?}""" chunkparser = nltk.regexpparser(chunkgram) chunked = chunkparser.parse(tagged) <RB.?>* = "0 or more of any tense of adverb," followed by: <VB.?>* = "0 or more of any tense of verb," followed by: <NNP>+ = "One or more proper nouns," followed by <NN>? = 0 or one singular noun." ######### NLTK Chunking ######### (S cat/nn with/in my/prp$ car/nn) {'my': 0, 'with': 0, 'car': 1, 'cat': 1} /NN Singular Noun /IN Preposition /PRP Personal Pronoun NLTK Natural Language Tool-Kit

21 User Query processing NLTK

22 Predicting Images Cosine Similarity User Query cat with my car Term Value Cat 1 With 0 my 0 Car 1 Kitty 1 Dog 0 NLTK Output Vector Predicted Class Initial probability Add synonyms (Probability) Normalize Car /1.7=0.12 Cat /1.7=0.41 kitty /1.7=0.41 Dog /1.7=0.06 With my CNN Output Vector

23 Final Overview CNN Term Dictionary Cosine Similarity Cat with my car NLTK Term Dictionary

24 How its going to works?

25 Thank You Any Question?