Super-Resolution for Aliased Images

Transcription

1 Super-Resolution for Aliased Images

2 Who are we? Deep Learning Zurich, NVIDIA Switzerland Marco Foco Artem Rozantsev Dongho Kang 2

3 Task Given a lowresolution image Construct a highresolution image H upscaling n*h W n*w 3

4

5

6 Existing techniques Interpolation (bilinear, bicubic, lanczos, etc.) Interpolation + Sharpening (and other filtration) interpolation filter-based sharpening Such methods are data-independent Very rough estimation of the data behavior too general 6

7 Existing techniques (Deep) Perceptual Losses for Real-Time Style Transfer and Super-Res: Johnson et al Super-Res with Deep Adaptive Image Resampling: Jia et al A Fully Progressive Approach to Single-Image Super-Res: Wang et al EnhaceNet: Mehdi et al Image Super-Res via Deep Recursive ResNets: Tai et al and many others 7

8 Previously used method 4x upscaling model Conv2d [5, 5, 64] Conv2d 8 x Res Upconv x2 8 x Res Upconv x2 8 x Res [3, 3, 3] Up x4 + * Developed by Dmitry Korobchenko 8

9 Previously used method 4x upscaling model Conv2d [5, 5, 64] Conv2d 8 x Res Upconv x2 8 x Res Upconv x2 8 x Res [3, 3, 3] Up x4 + Low-pass: Bilinear up-scaling of the input image. * Developed by Dmitry Korobchenko 9

10 Previously used method 4x upscaling model Conv2d [5, 5, 64] Conv2d 8 x Res Upconv x2 8 x Res Upconv x2 8 x Res [3, 3, 3] Up x4 + Res Conv2d [3, 3, 64] Conv2d [3, 3, 64] + Upconv x2 NN x2 Conv2d [3, 3, 64] * Developed by Dmitry Korobchenko 10

11 How to train SuperRes Downscaling SR Downscaling model LR image HR image Reconstructed HR image 11

12 How to train SuperRes Downscaling SR Downscaling model LR image HR image Reconstructed HR image Solve the optimization problem:, - training set 12

13 Optimized for natural images Previously used method 4x 13

14 Previously used method Optimized for natural images introduces aliasing/ noise 4x 14

15 Optimized for natural images Previously used method 4x 15

16 Previously used method Optimized for natural images introduces aliasing/ noise 4x 16

17 Our approach 17

18 Problem Assumption Problem comes not from the architecture itself, but from the training data. One effective solution is to fine-tune the network on game images. Not very general, as the network will likely require retraining for different games Game footage is required, and needs to be balanced across the game content 18

19 Solution Assumption Problem comes not from the architecture itself, but from the training data. One effective solution is to fine-tune the network on game images. Not very general, as the network will likely require retraining for different games Game footage is required, and needs to be balanced across the game content We propose: Modify the downscaling procedure to include aliased images and retrain from scratch 19

20 Downscaling 20

21 Downscaling (one solution) Area downsampling -- average across the 4x4 image regions x 1/4 21

22 Downscaling (one solution) Area downsampling -- average across the 4x4 image regions x 1/4 Pros: Works well for natural images Cons: Is not able to correct for the aliasing issue (e.g. game images) 22

23 Uniform sampling of the 4x4 area Downscaling (our solution) x 1/4 23

24 Uniform sampling of the 4x4 area Downscaling (our solution) x 1/4 Benefits: Corrects for the aliasing artefacts that are introduced by the downsampling procedure Improves robustness of the network (data augmentation) 24

25 Downscaling (our solution) x 1/4 At every iteration the downscaled image is different! 25

26 Downscaling (comparison) Previous Ansel RTX (AI UP-RES) 4x Input LR image Area downscaling Random downscaling 26

27 Downscaling (comparison) Previous Ansel RTX (AI UP-RES) 4x Input LR image Area downscaling Random downscaling 27

28 Loss function MSE HFEN VGG TV GAN

29 Loss function MSE HFEN VGG TV GAN % down SR PSNR Peak Signal-to-Noise Ratio MSE loss: L ' ( " Related to 10! " #$ 29

30 Loss function MSE HFEN VGG TV GAN % HP: High-Pass filter down SR HFEN* loss: L) *+( " HFEN*: High Frequency Error Norm * 30

31 Loss function MSE HFEN VGG TV GAN % Perceptual features:, -,, down SR VGG* loss: L ) ", (, " VGG19 features taken after the 4 /0 convolutional layer (before 5 /0 max-pooling) * 31

32 Loss function MSE HFEN VGG TV GAN % down SR TV loss: 2) Serves as a regularizer and has little influence on the optimization 32

33 Loss function MSE HFEN VGG TV GAN down Generator SR % Discriminator real % fake GAN loss () 8 * 33