A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Transcription

1 A Neural Algorithm of Artistic Style Leon A. Gatys, Alexander S. Ecker, Matthias Bethge Presented by Shishir Mathur (1 Sept 2016)

2 What is this paper This is the research paper behind Prisma It creates artistic images of high perceptual quality. The system proposed uses neural representation to separate and then recombine content and style of arbitrary images, for creation of artistic images.

3 Introduction The key observation of the author was that the representation of content and style of an image in a Convolutional Neural Network are separable. When CNN are trained for object recognition tasks, then along the process hierarchy of the network the image is transformed into a representation that increasingly cares about the actual content of the image. To capture the style of the image correlations between different layers are mapped. This gives a multi-scale representation of the image which captures the texture information but not the global arrangement giving us the artistic style of the image.

4 Content of Image The feature responses of higher layers of the network capture the content representation. Higher layers in the network capture the high-level content in terms of objects and their arrangement in the input image but do not constrain the exact pixel values of the reconstruction.

5 Artistic Style of the Image To obtain style of the image the author used Gram matrices as Feature Space which were originally used to capture texture information. The Feature space is built on top of filter responses in each layer of the network. This in turn maps the correlations between different layers of the network. By including correlations between different layers, we obtain stationary, multi-scale representation of the image

6

7

8 Method The results were generated based on a pretrained VGG-Network for visual object recognition task. A layer with Nl distinct filters has Nl different feature maps each of size Ml (h x w of the feature map). So the response of in layer l can be stored in a matrix Fl RN x M.

9 Content Visualization To visualize the image information that is encoded at different layers we perform gradient descent on white noise image to find another image which matches the feature responses of the original image. Let p be a original image and x be the white noise image and Pl and Fl be their feature representation at layer l. We then define the squared-error loss between the two feature representation. The derivative of this loss with respect to the activations in layer l is

10 Using the derivative we can compute the the gradient with respect to the image x using standard error back propagation. In every iteration the the initial image x is changed till it generates the same response in certain layer of CNN as the original image p.

11 Style Representation and Visualization To map style Representation we map correlations between the different filter responses over the spatial extent of feature maps. This provide colours and local structures. These feature correlations are given by Gram Matrix Gl RN x N Where Gl is the inner product between vectorised feature map in layer l. To generate a texture that matches the style of the given image we use gradient descent from a white noise image to find another image that matches the style representation of the original image. This is done by minimising the error between entries of gram matrix.

12 Let a be the original image and x be the white noise image and Al and Gl be their respective style representation in layer l. The loss is modelled by And the total loss is Where wl are the weighting factor of contribution of each layer in the total loss. The derivative El W.R.T. activation layer l is This error can then be back propagated to the white noise image to get the style of the image

13 conv1_1 conv1_1 conv2_1 conv1_1 conv2_1 conv3_1 conv1_1 conv2_1 conv3_1 conv4_1 conv1_1 conv2_1 conv3_1 conv4_1 conv5_1

14 To generate image with content from one and artistic style of another we jointly minimise the distance of a white noise image with content representation of the image in one layer of the network and the style representation of the artwork in a number of layers of the CNN The ratio of α/β was either 1 x 10-3 or 1 x 10-4

15

16

17