Dynamic Convolutional Neural Network for Modelling Sentences NAl Kalchbrenner,Edward Grefenstette, Phil Blunsom

Transcription

1 Dynamic Convolutional Neural Network for Modelling Sentences NAl Kalchbrenner,Edward Grefenstette, Phil Blunsom Course-CS671 Presented by- Anurendra Kumar(12147)

2 AIM:-Accurately represent sentences capturing semantic content ABSTRACT/PROPOSAL:- A Convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) is adopted for the semantic modelling of sentences. It allows to handle sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long range relations. It does not rely on parse tree.

3 What is CNN Convolutional Neural Networks or CNN can be seen as a kind of neural network that uses many identical copies of the same neuron.it can express computationally large models with lesser number of parameters. The network has multiple interleaved convolutional and pooling layers. It captures local properties by using convolution to look only at particular segment of data to compute feature. Weights are shared in the particular layer,therefore lesser number of parameters.

4 CONVOLUTIONAL NEURAL NETWORK A feature map is obtained by repeated application of a function across subregions of the entire image, in other words, by convolution of the input image with a linear filter, adding a bias term and then applying a non-linear function.

5 CONVOLUTION LAYER It captures local properties by using convolution to look only at particular segment of data to compute feature.

6 DYNAMIC K-MAX POOLING A dynamic k-max pooling operation is a k-max pooling operation where we let k be a function of the length of the sentence and the depth of the network. Although many functions are possible, we simply model the pooling parameter as follows: l is number of current convolution layer,l is total number of convolution layer.s is length of sentence. ktop is the fixed pooling parameter for the topmost convolutional layer.

7 Architecture of dynamic CNN.

8 NON-LINEAR FEATURE FUNCTION After (dynamic) k-max pooling is applied to the result of a convolution, a bias b Rd and a nonlinear function g are applied component-wise to the pooled matrix. There is a single bias value for each row of the pooled matrix.the non-linearity of g allows to have non-linear feature extraction MULTIPLE FEATURE MAPS So far a wide convolution, a (dynamic) k-max pooling layer and a non-linear function is applied to the input sentence matrix to obtain a first order feature map. The three operations can be repeated to yield feature maps of increasing order and a network of increasing depth. We denote a feature map of the i-th order by Fi. As in convolutional networks for object recognition, to increase the number of learnt feature detectors of a certain order, multiple feature maps Fi1,..., Fin may be computed in parallel at the same layer.each feature map Fij is computed by convolving a distinct set of filters arranged in a matrix mi j,k with each feature map Fik 1 of the lower order i 1 and summing the results:

9 Properties of sentence model Word and n-gram model-following properties of model is desired:- Discriminative:-model should be able to discriminate whether a specific n- gram occurs in the input Relativity-Model should be able to tell the relative position of the most relevant n-grams. The network is designed to capture these two aspects. The filters m of the wide convolution in the first layer can learn to recognise specific n-grams that have size less or equal to the filter width m Generalised pooling operation induces invariance to absolute positions but maintains their order and relative position.

10 INDUCED FEATURE GRAPH In a DCNN, the convolution and pooling layers induce an internal feature graph over the input. The induced graph is a connected D.A.G. with weighted edges and root node.each subgraph may have a different shape that reflects the kind of relations that are detected in the subgraph. The peculiarity of feature graph of a DCNN is that dynamic k max can draw together features to the words that correspond to many positions apart in the sentence. Higher order features have highly variable ranges from short and focussed or global and long. The edges of subgraph reflect these varying ranges. In DCNN one learns a clear hierarchy of feature orders. Thus it is more general than parse tree in that it is not limited to syntactically dictated phrases;the graph structure can capture short or long range semantic relation between words that do not happen in parse tree.

11 Results