Inverse Problems and Machine Learning

Transcription

1 Inverse Problems and Machine Learning Julian Wörmann Research Group for Geometric Optimization and Machine Learning (GOL) 1

2 What are inverse problems? 2

3 Inverse Problems cause/ excitation 3

4 Inverse Problems cause/ excitation System/ Process 4

5 Inverse Problems cause/ excitation System/ Process effect/ measurement 5

6 Inverse Problems cause/ excitation System/ Process effect/ measurement 6

7 Inverse Problems cause/ excitation System/ Process effect/ measurement 7

8 Inverse Problems cause/ excitation System/ Process effect/ measurement 8

9 Inverse Problems cause/ excitation System/ Process effect/ measurement 9

10 cause/ excitation Goal System/ Process effect/ measurement 1

11 cause/ excitation Goal System/ Process System/ -1 Process effect/ measurement 11

12 cause/ excitation Model: Goal System/ Process System/ -1 Process effect/ measurement 12

13 cause/ excitation Model: Goal System/ Process System/ -1 Process effect/ measurement noise 13

14 Inverse Problems in Image Processing Denoising Inpainting Deblurring? 14

15 Tasks 1. Determine the model parameters 15

16 Tasks 1. Determine the model parameters 16

17 Tasks 1. Determine the model parameters 2. Reconstruct from 17

18 Tasks 1. Determine the model parameters 2. Reconstruct from? 18

19 Approaches to solve inverse problems 19

20 Least Squares approach 2

21 Least Squares approach Problems: ill-conditioned Example: Signal Deconvolution/Deblurring 21

22 Least Squares approach Problems: ill-conditioned Example: Signal Deconvolution/Deblurring n m System under-/overdetermined infinitely many/no solutions Example: Signal Inpainting 22

23 Least Squares approach Problems: ill-conditioned Example: Signal Deconvolution/Deblurring n m System under-/overdetermined infinitely many/no solutions Example: Signal Inpainting No AWGN 23

24 Least Squares approach Problems: ill-conditioned Example: Signal Deconvolution/Deblurring n m System under-/overdetermined infinitely many/no solutions Example: Signal Inpainting No AWGN Solutions: Exploiting structures and properties of the data 24

25 Least Squares approach Problems: ill-conditioned Example: Signal Deconvolution/Deblurring n m System under-/overdetermined infinitely many/no solutions Example: Signal Inpainting No AWGN Solutions: Exploiting structures and properties of the data Optimization under constraints 25

26 Optimization under constraints 26

27 Optimization under constraints Constraint set encoded in function 27

28 Optimization under constraints Constraint set encoded in function assumed noise energy 28

29 What are suitable constraints? - Pixelvalues are always positive - Images contain homogeneous regions, i.e. neighbouring pixels often have the same value - Signals can be composed of Basissignals (e.g. sinusoids) 29

30 Synthesis Operator (Dictionary) idealised 3

31 Synthesis Operator (Dictionary) idealised atoms 31

32 Synthesis Operator (Dictionary) idealised 32

33 Synthesis Operator (Dictionary) idealised atoms 33

34 Synthesis Operator (Dictionary) idealised atoms 34

35 Synthesis Operator (Dictionary) idealised signal atoms = 1 35

36 Synthesis Operator (Dictionary) idealised signal atoms =

37 Synthesis Operator (Dictionary) idealised signal atoms =

38 Synthesis Model 38

39 Synthesis Model = Dictionary 39

40 Synthesis Model Assumption: Signal has a sparse representation = Dictionary 4

41 Synthesis Model Assumption: Signal has a sparse representation = Dictionary 41

42 Synthesis Model Assumption: Signal has a sparse representation = Dictionary redundant Columns are called atoms 42

43 JPEG Compression Natural images are compressible signals with a compressible representation in a DCT (JPEG) or Wavelet Basis (JPEG-2) Compressible Signals can be well approximated through sparse signals 43

44 JPEG Compression Image from: Gregory K. Wallace, The JPEG Still Picture Compression Standard, IEEE Transactions on Consumer Electronics, vol. 38 no.1, Feb

45 Input patch Forward DCT coefficients Quantization table Normalized quantized coefficients Denormalized quantized coefficients Reconstructed patch 45

46 Input patch Forward DCT coefficients Quantization table Normalized quantized coefficients Denormalized quantized coefficients Reconstructed patch 46

47 Synthesis Model Goal: Find sparsest, that explains the measurements 47

48 Synthesis Model Goal: Find sparsest, that explains the measurements Signal is synthesized from sparse vector Synthesis Model 48

49 Synthesis Model Goal: Find sparsest, that explains the measurements Signal is synthesized from sparse vector Synthesis Model 49

50 p k x = x p p j p 2 f ( x) = x α 1 j=1 As p we get a count of the non-zeros in the vector α p α p p 2 α α 1 p p p < 1 x 5

51 From Synthesis Model to the Analysis Model Synthesis Model = Signal is synthesized from a few atoms ( = sparse vektor) 51

52 From Synthesis Model to the Analysis Model Synthesis Model = Signal is synthesized from a few atoms ( = sparse vektor) Analysis Model Assumption: Signal is mapped to sparse vector via an Analysis Operator = 52

53 From Synthesis Model to the Analysis Model Synthesis Model = Signal is synthesized from a few atoms ( = sparse vektor) Analysis Model Assumption: Signal is mapped to sparse vector via an Analysis Operator = Rows are called atoms 53

54 Analysis Operator exemplarily signal 54

55 Analysis Operator exemplarily finite differences of adjacent pixels signal 55

56 Analysis Operator exemplarily = sparse analysed vektor finite differences of adjacent pixels signal 56

57 Analysis Model Goal: Find signal, such that the analysed vector is sparse and such that explains the measurements 57

58 Find a solution via BMP FOCUSS QN SPARSA CG FISTA NESTA L1-magic TWIST ISTA CVX C-SALSA OMP YALL1 SALSA SAMP 58

59 What are appropriate Synthesis/Analysis Operators? 1. Analytically given Advantages: Fast implementation + generalisation Drawback: Sparse representation is not optimal Examples: Wavelets, Bandlets, Curvelets, Dicrete Cosine Transform, Fourier Transform, Finite Difference Operator (Total Variation) 59

60 What are appropriate Synthesis/Analysis Operators? 1. Analytically given Advantages: Fast implementation + generalisation Drawback: Sparse representation is not optimal Examples: Wavelets, Bandlets, Curvelets, Dicrete Cosine Transform, Fourier Transform, Finite Difference Operator (Total Variation) 2. Learned from trainingdata Advantages: Optimal sparse representation, performance Drawback: Slow implementation Examples : CURRENT RESEARCH! 6

61 What are appropriate Synthesis/Analysis Operators? 1. Analytically given Advantages: Fast implementation + generalisation Drawback: Sparse representation is not optimal Examples: Wavelets, Bandlets, Curvelets, Dicrete Cosine Transform, Fourier Transform, Finite Difference Operator (Total Variation) 2. Learned from trainingdata Advantages: Optimal sparse representation, performance Drawback: Slow implementation Examples : CURRENT RESEARCH! Example: Dictionary Learning 61

62 Analysis Operator Learning 62

63 Operator Learning example for Image Processing vectorised patches Operator Learning 63

64 Analysis Operator Learning Basics Required: N representative training signals 64

65 Analysis Operator Learning Basics Required: N representative training signals Sought: Analysis Operator, such that N analysed vectors are sparse 65

66 Analysis Operator Learning Basics Required: N representative training signals Sought: Analysis Operator, such that N analysed vectors are sparse Analysis Operator Atoms 66

67 Analysis Operator Learning Basics Required: N representative training signals Sought: Analysis Operator, such that N analysed vectors are sparse Analysis Operator Atoms 67

68 Analysis Operator Learning Basics Required: N representative training signals Sought: Analysis Operator, such that N analysed vectors are sparse Analysis Operator Atoms Constraint set to avoid trivial solution 68

69 Geometric Analysis Operator Learning (GOAL) 69

70 Geometric Analysis Operator Learning (GOAL) Constraints 7

71 Geometric Analysis Operator Learning (GOAL) Constraints 1. Atoms/rows of are normalised, i.e. 71

72 Geometric Analysis Operator Learning (GOAL) Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e. 72

73 Geometric Analysis Operator Learning (GOAL) Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e. 3. Rows are not trivially linear dependent, 73

74 Geometric Analysis Operator Learning (GOAL) Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e. 3. Rows are not trivially linear dependent, From constraints 1+2 element of a special manifold efficient method to find a solution (e.g. Conjugate Gradient, Quasi-Newton) 74

75 Example: Manifold Learning Normalised rows lie on the surface of a sphere (with radius = 1) Step along geodesics 75

76 Denoising Applied to solve inverse problems Inpainting Deblurring? 76

77 Denoising Applied to solve inverse problems Inpainting Deblurring! 77

78 Demo: GOAL + Lena 78

79 Bimodal signal reconstruction 79

80 Application: 3D Reconstruction in HD 3D scene analysis with high-resolution camera and depth sensor 8

81 Application: 3D Reconstruction in HD 3D scene analysis with high-resolution camera and depth sensor 81

82 Application: 3D Reconstruction in HD 3D scene analysis with high-resolution camera and depth sensor Bimodal Analysis Operator 82

83 Learning from bimodal signals Intensity Depth 83

84 Learning from bimodal signals signal pair Intensity bright dark Depth 84

85 Learning from bimodal signals signal pair Intensity bright dark Depth Intensity operator Ω I minimize Ω I,Ω D G Ω I S I, Ω D S D Depth operator Ω D learn Ω I and Ω D such that both analyzed vectors Ω I s i and Ω D s d are maximally sparse 85

86 Bimodal reconstruction Unimodal s arg min g Ωs subject to As y 2 2 ε s R N 86

87 Bimodal reconstruction Unimodal s arg min g Ωs subject to As y 2 2 ε s R N Bimodal (s I, s D ) arg min s I,s D R N G Ω I s I, Ω D s D subj. to A I s I y I A D s D y D 2 2 ε 87

88 Bimodal reconstruction Unimodal s arg min g Ωs subject to As y 2 2 ε s R N Bimodal (s I, s D ) arg min s I,s D R N G Ω I s I, Ω D s D subj. to A I s I y I A D s D y D 2 2 ε Intensity image fixed c s D arg min λg c, Ω D s D + A D s D y D 2 2 s D R N 88

89 Results of the bimodal reconstruction (JID) Depth Map Super-Resolution 8x NN interpolation bicubic interpolation JID 3D Scene Reconstruction original bicubic interpolation JID 89

90 Analysis Based Blind Compressive Sensing 9

91 Concept of Compressive Sensing Only a few linear and non-adaptive measurements are sufficient to reconstruct the signal with high accuracy. 91

92 Concept of Compressive Sensing Only a few linear and non-adaptive measurements are sufficient to reconstruct the signal with high accuracy. Exploitation of the sparse representation with a (analytically) given Dictionary or Operator. 92

93 Analysis Based Compressive Sensing Reconstruction of the signals under the assumption that there exists a sparse representation 93

94 Analysis Based Blind Compressive Sensing Exploiting the property that learned operators admit a sparser representation Adaptive, signal dependent regularisation of the inverse problem under consideration of the error model 94

95 Analysis Based Blind Compressive Sensing Exploiting the property that learned operators admit a sparser representation Adaptive, signal dependent regularisation of the inverse problem under consideration of the error model 95

96 Analysis Based Blind Compressive Sensing Image reconstruction (1) ABCS (2) TV Operator 96

97 Analysis Based Blind Compressive Sensing Learned Analysis Operators (1) Random Input (2) Barbara (3) Piecewise constant 97

98 Analysis Based Blind Compressive Sensing Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure 98

99 Analysis Based Blind Compressive Sensing Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure The Analysis Operator does not need to be learned before the reconstruction 99

100 Analysis Based Blind Compressive Sensing Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure The Analysis Operator does not need to be learned before the reconstruction Ability to reconstruct different signal/image classes by simply exchanging the error model 1

101 Take Home Messages 11

102 Take Home Messages Structure in data is extremely important and can be utilized to regularize inverse problems 12

103 Take Home Messages Structure in data is extremely important and can be utilized to regularize inverse problems Sparsity is a valuable property of many signals 13

104 Take Home Messages Structure in data is extremely important and can be utilized to regularize inverse problems Sparsity is a valuable property of many signals Machine Learning can help to find such structures 14

105 Take Home Messages Structure in data is extremely important and can be utilized to regularize inverse problems Sparsity is a valuable property of many signals Machine Learning can help to find such structures Geometric aspects of a problem can be exploited in the optimization 15

106 Weiterführende Literatur S. Hawe, M. Kleinsteuber, and K. Diepold. Analysis Operator Learning and its Application to Image Reconstruction. IEEE Transactions on Image Processing, vol.22, no.6, pp , June 213. J. Wörmann, S. Hawe, and M. Kleinsteuber. Analysis Based Blind Compressive Sensing. IEEE Signal Processing Letters, 2(5) , 213. M. Kiechle, S. Hawe, and M. Kleinsteuber. A Joint Intensity and Depth Co-Sparse Analysis Model for Depth Map Super-Resolution. IEEE International Conference on Computer Vision 213. M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing, vol. 54, no.11,