Super-Resolution for Hyperspectral Remote Sensing Images with Parallel Level Sets

Transcription

1 Super-Resolution for Hyperspectral Remote Sensing Images with Parallel Level Sets Leon Bungert 1, Matthias J. Ehrhardt 2, Marc Aurele Gilles 3, Jennifer Rasch 4, Rafael Reisenhofer 5 (1) Applied Mathematics 1, Friedrich-Alexander Universität Erlangen-Nürnberg (2) Department for Applied Mathematics and Theoretical Physics, University of Cambridge (3) Center for Applied Mathematics, Cornell University (4) Fraunhofer Heinrich Hertz Institute, Berlin (5) AG Computational Data Analysis, Universität Bremen AIP 2017, Zhejiang University 31 May 2017

2 Outline Problem Model Algorithmic Realization Numerical Results

3 Two Imaging Modalities low-res hyperspectral image (126 channels, px) channel 20 channel 47 channel 63 channel 90 high-res RGB image ( px) Eventually, biologists aim to correctly segment different kinds of trees. 3/31

4 Basic Approach Goal: Increase the resolution of the hyperspectral image to the resolution of the RGB image. Assumption: Structures that are visible in the hyperspectral image are also visible in the RGB image. We express the structural information of the RGB image in terms of level sets or gradients and use this as additional a-priori knowledge in the super-resolution process. 4/31

5 Model We consider solutions (u, k ) of the minimization problem where 1 argmin (u,k) U K 2 S C k u f 2 + α dtv v (u) + β TV(k), N l, N h, N k N denote numbers of pixels, U = [0, 1] N h, K is the N k -dimensional unit simplex, C k is the convolution operator with respect to a kernel k, S is a subsampling operator, f R N l is a single channel of the hyperspectral image, dtv v denotes directional total variation penalty with respect to the vector-field v, TV denotes the total variation penalty and α, β > 0 are regularization parameters. 5/31

6 Model - Forward Operator argmin (u,k ) U K 1 S Ck u f 2 + α dtvv (u ) + β TV(k ) 2 Cyclic convolution operator with a kernel k K Ck : RNh RNh, F 2D (Ck u )n = F 2D (u )n F 2D (k )n Subsampling operator S : RNh RNl, (S u )n = uπ [n ] with a mapping π {1,..., Nh }Nl. high-res channel u Ck (u ) S Ck (u ) 6/31

7 Model - Directional Total Variation 1 argmin (u,k) U K 2 S C k u f 2 + αdtv v (u) + β TV(k) Let : R N h R N h 2 denote the discrete gradient operator based on forward differences. For a paremter η > 0 and a high-res image u R N h, we consider u the vectorfield v = u. 2 +η 2 image u vectorfield v Ehrhardt & Betcke, /31

8 Model - Directional Total Variation 1 argmin (u,k) U K 2 S C k u f 2 + αdtv v (u) + β TV(k) Let a vectorfield v be given as in the previous slide and denote P vn = I v n v t n where I is the 2 2 identity matrix, then the directional TV penalty is given by dtv v (u) = N h N h P vn u n = u n v n, u n v n. n=1 n=1 Ehrhardt & Betcke, /31

9 Model - Regularization of the Kernel 1 argmin (u,k) U K 2 S C k u f 2 + α dtv v (u) + βtv(k) We assume that the kernel k has a sparse gradient and include the total variation penalty TV(k) = N h k n. n=1 Additionally, we assume that the kernel satisfies N k k n = 1 and k n 0, n N k. n=1 Rudin, Osher & Fatemi, /31

10 Model 1 argmin (u,k) U K 2 S C k u f 2 + α dtv v (u) + β TV(k) The kernel k is mainly associated to the PSF of the imaging device but can also be used to model motion blur stemming from the movement of the aircraft and to resolve translational differences between the hyperspectral image and the RGB image. The data fidelty term, dtv v as well as TV are convex and the energy functional is convex in u and convex in k. However, the energy functional is not convex in (u, k). 10/31

11 Algorithmic Realization We apply inertial alternating linearized minimization (ipalm; Pock & Sabach, 2016) which was recently proposed for a general class of nonconvex and nonsmooth problems. Guarantees convergence to a critical point for problems of the form argmin H(x, y) + f 1 (x) + f 2 (y), (x,y) R N 1 R N 2 where H(x, y) is smooth and its gradient Lipschitz continuous in both x and y and f 1 and f 2 can be general nonsmooth and nonconvex functions albeit with efficiently computable proximal mappings. 1 argmin (u,k) U K 2 S C k u f 2 }{{} H(u,k) + α dtv v (u) }{{} f 1 (u) + β TV(k) }{{} f 2 (k) 11/31

12 Algorithmic Realization ipalm includes inertial term in proximal alternating linearized minimization (PALM; Bolte, Sabach & Teboulle, 2014) The update step j j + 1 for the variable u is given by ũ = u (j ) + α (j ) (u (j ) u (j 1) ) ŭ = u (j ) + β (j ) (u (j ) u (j 1) ) ( ) ũ 1 u S C k (j ) ŭ f 2 u (j +1) prox dtv v τ (j ) 2τ (j ) 1 where α (j ) and β (j ) are inertial parameters and τ (j ) controls the step size and should be chosen proportional to the Lipschitz constant of u S C k (j ) ŭ f 2. The update step for the variable k is given analogously. Pock & Sabach, /31

13 Algorithmic Realization In order to apply the update step ũ = u (j ) + α (j ) (u (j ) u (j 1) ) ŭ = u (j ) + β (j ) (u (j ) u (j 1) ) ( ) ũ 1 u S C k (j ) ŭ f 2, u (j +1) prox dtv v τ (j ) We need to compute 2τ (j ) 1 prox dtv v τ (j ) (and prox TV τ (j ) for k) u S C k (j ) ŭ f 2 (and k S C k (j ) ŭ f 2 ) and the Lipschitz constant of u S C k (j ) ŭ f 2. 13/31

14 Algorithmic Realization u (j +1) prox dtv v τ (j ) ( ũ 1 2τ (j ) u S Ck (j ) ŭ f ) 2 1 prox dtv v and prox TV can by computed via fast gradient τ (j ) τ (j ) projection (Beck & Teboulle, 2009) u S C k (j ) ŭ f 2 = 2 C k S (S C k u f )) Let L be the Lipschitz constant of u S C k (j ) ŭ f 2 then it holds that L max F 2D 2 (k) n n Alternatively, a backtracking scheme can be applied to obtain smaller values for τ (j ) and thereby larger step sizes. 14/31

15 Numerical Results - Test Case 1 low-res hyperspectral channel ( px) high-res RGB image ( px) 15/31

16 Numerical Results - Test Case 1 input f argmin (u,k ) U K vectorfield v (η = 0.001) 1 S Ck u f 2 + α dtvv (u ) + β TV(k ), 2 with α = 0.5, β = 10 16/31

17 Numerical Results - Test Case 1 (u, k ) after 5000 iterations of ipalm (with backtracking) 17/31

18 Numerical Results - Test Case 1 Comparison of the RGB image, f and the recovered image u : RGB image f u 18/31

19 Numerical Results - Test Case 1 Comparison of the RGB image, f and u (zoom): RGB image f u 19/31

20 Numerical Results - Test Case 1 How well does the solution u fit the given data f? f S C k u f S C k u 20/31

21 Numerical Results - Test Case 2 low-res hyperspectral channel ( px) high-res RGB image ( px) 21/31

22 Numerical Results - Test Case 2 (u, k ) after 5000 iterations of ipalm (with backtracking) 22/31

23 Numerical Results - Test Case 2 Comparison of the RGB image, f and the recovered image u : RGB image f u 23/31

24 Numerical Results - Test Case 2 Comparison of the RGB image, f and u (zoom): RGB image f u 24/31

25 Numerical Results - Test Case 2 How well does the solution u fit the given data f? f S C k u f S C k u 25/31

26 Numerical Results - Blind Deconvolution vs. Fixed Kernel with blind deconvolution with fixed Gaussian kernel 26/31

27 Numerical Results - TV vs. dtv dtv TV 27/31

28 Conclusion and Outlook dtv regularization can be used to combine two imaging modalities in a super resolution task. Including blind deconvolution in the optimization process can be crucial in correctly linking the two modalities. Reconstructing one pixel channel in Matlab currently takes about 2 hours. This could be improved by a C++ or GPU-based implementation. Our solutions still suffer from high-frequency point artifacts. 28/31

29 Thank you! 29/31

30 Shameless Advertising 30/31