Introduction to patch-based approaches for image processing F. Tupin

Transcription

1 Introduction to patch-based approaches for image processing F. Tupin Athens week

2 Introduction Image processing Denoising and models Non-local / patch based approaches Principle Toy examples Limits and solutions Some applications in image processing Inpainting (image and video) HDR (High Dynamic Range) Texture synthesis page 1

3 Talk overview Introduction Image processing Denoising and models Non-local / patch based approaches Principle Toy examples Limits and solutions Some applications in image processing Inpainting (image and video) HDR (High Dynamic Range) Texture synthesis page 2

4 Image processing general scope Information extraction Interpretation Data enhancement and restoration (deblurring, denoising, irregular sampling inpainting) Data fusion Applications End-user interaction page 3

5 page 4 Visual system / Image processing to see is to think

6 Recent trends in image processing Applied mathematics: Increasingly sophisticated models Allowed by the increase of computational efficiency Real time processing Very simple and fast approaches Physics compensated by software page 5

7 page 6 Image denoising

8 Can we denoise? Temporal information page 7

9 Can we denoise? Spatial information page 8

10 Image models Hypothesis of signal / noise separation Hypothesis of signal smoothness Hypothesis of signal redundancy page 9

11 Image models Hypothesis of signal / noise separation Hypothesis of signal regularity Hypothesis of signal redundancy page 10

12 Denoising and «averaging» Average of many noisy values: estimation of the «true» reflectivity only if the selected values are coming from the same underlying noise-free value How can we select them on the image? page 11

13 Selection based filtering Where finding the «good» information? Locally (linear filtering) Locally (anisotropic diff.) Oracle page 12

14 Talk overview Introduction Image processing Denoising and models Non-local / patch based approaches Principle Toy examples Limits and solutions Some applications in image processing Inpainting (image and video) HDR (High Dynamic Range) Texture synthesis page 13

15 Selection-based filtering Non-local approaches: Relaxing locality and connexity constraints for pixel selection: selection based on similarity

16 Selection-based filtering Non-local approaches: Relaxing locality and connexity constraints for pixel selection: selection based on similarity [Yaroslavsky, 85] How computing d when having only noisy values? Use patches!

17 Non-local means [Buades 05] Algorithm : Similarity of pixels = similarity of patches

18 Selection-based filtering Non-local approaches: example of weight maps

19 Selection-based filtering Non-local approaches: example of weight maps

20 page 19 Non-local means

21 page 20 Selection based filtering H1 redundancy

22 Non-local approaches - patches H1 : Hypothesis of redundancy of patches in images page 21

23 page 22 Redundancy of patches

24 Non-local approaches H2 : similarity between patches similarity of central pixels page 23

25 page 24 Toy examples periodic texture

26 page 25 Toy examples periodic texture

27 page 26 Toy examples periodic texture

28 page 27 Toy examples periodic texture

29 page 28 Toy examples periodic texture

30 Isolated crenel s=7 page 29

31 Isolated crenel s=15 page 30

32 Limits and solutions Limits: Loss of weakly contrasted structures «rare patch effect»: noise halo Influence of NL-means parameters: Search window W Patch size s Kernel function (h parameter) Solution: Local adaptation of h Bias / variance trade-off page 31

33 Influence of W: loss of details W=11x11 W=61x61 page 32

34 page 33 Influence of patch size: «rare patch effect»

35 page 34 Influence of patch size

36 page 35 Influence of patch size

37 page 36 Results

38 page 37 Influence of h

39 page 38

40 page 39

41 page 40 h adaptation

42 Talk overview Introduction Image processing Denoising and models Non-local / patch based approaches Principle Toy examples Limits and solutions Some applications in image processing Inpainting (image and video) HDR (High Dynamic Range) Texture synthesis page 41

43 Patch-based inpainting Principle: Start by the boundary pixels of the region to fill Select a patch around the pixels Search for similar patch in the known image Fill the central pixel with the central value page 42

44 Patch-based et al. page 43

45 Patch-based et al. page 44

46 Patch-based et al. page 45

47 Video inpainting Principle Use space + time patches to fill gaps Multi-resolution framework Estimation of the dominant motion in the video page et al.

48 High Dynamic Range Imaging page et al.

49 HDR Loss of details in bright areas Loss of details in dark areas page et al.

50 Patch-based HDR (High Dynamic Range) page et al.

51 HDR principle static case page et al.

52 HDR principle static case page et al.

53 HDR dynamic case page et al.

54 Patch-based HDR page et al.

55 page et al.

56 page et al.

57 Aguerreberre et al. page et al.

58 Aguerreberre et al. page et al.

59 Texture synthesis page et al.

60 Texture synthesis page et al.

61 page et al.

62 Texture synthesis random phase page et al.

63 Textures synthesis random phase page et al.

64 Textures synthesis random phase page et al.

65 Patch-based synhesis page et al.

66 Synthesis with spectrum and patches page et al.

67 Examples page et al.

68 Examples page et al.

69 Conclusion Patch-based approach for image processing Very powerful and «weak» models General formulation Wide range of applications beyond denoising Spatial and temporal adaptation (video) Limits Additive gaussian noise Many parameters page 68

70 page 69 Acknowledgments / References PhD students : C. Deledalle, V. Duval, C. Aguerrebere, A. Newson, G. Tartavel Publications : Video inpainting of complex scenes, A. Newson et al., SIAM 2014 Simultaneous HDR reconstruction and denoising of dynamic scenes, C. Aguerrebere et al., ICCP 2013 A probabilistic patch based approach, C. Deledalle et al., IEEE IP 2009 Variational texture synthesis with sparcity and spectrum constraints, G. Tartavel et al., submitted HAL 2014