PIPA: A New Proximal Interior Point Algorithm for Large-Scale Convex Optimization

Transcription

1 PIPA: A New Proximal Interior Point Algorithm for Large-Scale Convex Optimization Marie-Caroline Corbineau 1, Emilie Chouzenoux 1,2, Jean-Christophe Pesquet 1 1 CVN, CentraleSupélec, Université Paris-Saclay, France 2 LIGM, UMR CNRS 8049, Université Paris-Est Marne la Vallée, France 19 April 2018 Calgary, ICASSP 2018 Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

2 In collaboration with E. Chouzenoux J.-C. Pesquet Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

3 About IPMs Interior Point Methods Many problems in signal/image processing (image restoration, enhancement, denoising/deblurring, spectral unmixing) can be formulated as constrained minimization problems need efficient methods for solving those. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

4 About IPMs Interior Point Methods Many problems in signal/image processing (image restoration, enhancement, denoising/deblurring, spectral unmixing) can be formulated as constrained minimization problems need efficient methods for solving those. Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex ( i {1,..., p}) c i : R n ], + ] convex, smooth Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

5 About IPMs Interior Point Methods Many problems in signal/image processing (image restoration, enhancement, denoising/deblurring, spectral unmixing) can be formulated as constrained minimization problems need efficient methods for solving those. Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex ( i {1,..., p}) c i : R n ], + ] convex, smooth How to minimize f while ensuring that every iterate is feasible? Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

6 About IPMs Interior Point Methods Many problems in signal/image processing (image restoration, enhancement, denoising/deblurring, spectral unmixing) can be formulated as constrained minimization problems need efficient methods for solving those. Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex ( i {1,..., p}) c i : R n ], + ] convex, smooth How to minimize f while ensuring that every iterate is feasible? Add a barrier function Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

7 About IPMs Logarithmic Barrier Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 Unconstrained Subproblem P µ : Where µ > 0 is the barrier parameter. minimize f (x) µ x R n p ln ( c i (x)) i=1 }{{} + as c i (x) 0 P 0 is replaced by a sequence of subproblems (P µj ) j N. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

8 About IPMs Logarithmic Barrier Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 Unconstrained Subproblem P µ : Where µ > 0 is the barrier parameter. minimize f (x) µ x R n p ln ( c i (x)) i=1 }{{} + as c i (x) 0 P 0 is replaced by a sequence of subproblems (P µj ) j N. Subproblems are solved approximately for a sequence µ j 0. Main advantage : every iterate is feasible. Primal-dual algorithm : superlinear convergence for NLP. [Gould et al., 2001] Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

9 About IPMs Logarithmic Barrier Constrained Problem P 0 : minimize f (x) x R n s.t. ( i {1,..., p}) c i (x) 0 Unconstrained Subproblem P µ : Where µ > 0 is the barrier parameter. minimize f (x) µ x R n p ln ( c i (x)) i=1 }{{} + as c i (x) 0 P 0 is replaced by a sequence of subproblems (P µj ) j N. Subproblems are solved approximately for a sequence µ j 0. Main advantage : every iterate is feasible. Primal-dual algorithm : superlinear convergence for NLP. [Gould et al., 2001] Require the inversion of an n n matrix at each step : medium size applications. First or second order methods : limited to smooth functions. [Armand et al., 2000] Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

10 About IPMs Problem of Interest Quality of the solution and robustness against noise can be improved by adding a non-differentiable term (l 1, TV,... ). Composite Constrained Problem P 0 : minimize f (x) + g(x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex, non-differentiable g : R n ], + ] convex, smooth ( i {1,..., p}) c i : R n ], + ] convex, smooth Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

11 About IPMs Problem of Interest Quality of the solution and robustness against noise can be improved by adding a non-differentiable term (l 1, TV,... ). Composite Constrained Problem P 0 : minimize f (x) + g(x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex, non-differentiable g : R n ], + ] convex, smooth ( i {1,..., p}) c i : R n ], + ] convex, smooth How to address the non-smooth term while ensuring that every iterate is feasible? Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

12 About IPMs Problem of Interest Quality of the solution and robustness against noise can be improved by adding a non-differentiable term (l 1, TV,... ). Composite Constrained Problem P 0 : minimize f (x) + g(x) x R n s.t. ( i {1,..., p}) c i (x) 0 where f : R n ], + ] convex, non-differentiable g : R n ], + ] convex, smooth ( i {1,..., p}) c i : R n ], + ] convex, smooth How to address the non-smooth term while ensuring that every iterate is feasible? Combine the logarithmic barrier method with proximal tools. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

13 About IPMs Notation and Definitions Let S + (R N ) be the set of symmetric positive definite matrices of R N N. The weighted norm induced by U S + (R N ) is. U =. U.. Let Γ 0 (R N ) denote the set of proper lower semicontinuous convex functions from R N to ], + ]. Proximity Operator The proximal operator a prox U f (x) of f Γ 0(R N ) at x R N relative to the metric induced by U S + (R N ) is the unique vector ŷ R N such that f (ŷ) ŷ x 2 U = inf f (y) + 1 y x U(y x). y R N 2 a. http ://proximity-operator.net/ Example : Indicator function : projection. l 1 norm : soft-thresholding. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

14 Approach Proposed Approach P 0 is replaced by a sequence of subproblems (P µj ) j N. Unconstrained Subproblems P µj : minimize x R n f (x) + g(x) µ j Our algorithm comprises two interlocked loops. p ln ( c i (x)) i=1 } {{ } smooth Given µ j > 0, (x j,k ) k is produced via several forward-backward (proximal gradient) steps. Once x j,k is close enough to the solution of P µj, the barrier parameter µ j is updated. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

15 Algorithm Iteration Scheme Forward-Backward Step For j fixed, p where ϕ µj (x) = g(x) µ j ln( c i (x)). i=1 x j,k+1 = prox A j,k γ j,k f (x j,k γ j,k A 1 j,k ϕµ j (x j,k)) Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

16 Algorithm Iteration Scheme Forward-Backward Step For j fixed, x j,k+1 = prox A j,k γ j,k f (x j,k γ j,k A 1 j,k ϕµ j (x j,k)) p where ϕ µj (x) = g(x) µ j ln( c i (x)). i=1 Gradient step on the smooth term ; Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

17 Algorithm Iteration Scheme Forward-Backward Step For j fixed, p where ϕ µj (x) = g(x) µ j ln( c i (x)). i=1 x j,k+1 = prox A j,k γ j,k f (x j,k γ j,k A 1 j,k ϕµ j (x j,k)) Gradient step on the smooth term ; Proximal step on the non-differentiable function f ; Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

18 Algorithm Iteration Scheme Forward-Backward Step For j fixed, x j,k+1 = prox A j,k γ j,k f (x j,k γ j,k A 1 j,k ϕµ j (x j,k)) p where ϕ µj (x) = g(x) µ j ln( c i (x)). i=1 Gradient step on the smooth term ; Proximal step on the non-differentiable function f ; Preconditioner A j,k for acceleration [Chouzenoux et al., 2016] ; Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

19 Algorithm Iteration Scheme Forward-Backward Step For j fixed, x j,k+1 = prox A j,k γ j,k f (x j,k γ j,k A 1 j,k ϕµ (x j j,k)) p where ϕ µj (x) = g(x) µ j ln( c i (x)). i=1 Gradient step on the smooth term ; Proximal step on the non-differentiable function f ; Preconditioner A j,k for acceleration [Chouzenoux et al., 2016] ; Stepsize γ j,k > 0 found using a backtracking strategy [Salzo, 2017] since ϕ µj not Lipschitz-differentiable. is Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

20 Algorithm Algorithm Proximal Interior point Algorithm (PIPA) Initialization Let γ > 0, (δ, θ) ]0, 1[ 2, µ 0 > 0; Initialize x 0,0 such that ( i {1,..., p}) c i (x 0,0 ) < 0; For j = 0, 1,... For k = 0, 1,... Choose A j,k satisfying a boundedness condition ; For l = 0, 1,... x j,k l = proxa j,k (x γθ l f j,k γθ l A 1 j,k ϕµ (x j j,k)); Stop if the backtracking condition is met ; x j,k+1 = x j,k l ; γ j,k = γθ l ; Stop if precision conditions are met ; x j+1,0 = x j,k+1 ; Update µ j ; Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

21 Algorithm Theoretical Results Assumptions The set of solutions to P 0 is nonemtpy and bounded ; f, g and the constraints are convex, g is Lipschitz-differentiable and the constraints are continuously twice-differentiable ; The strict interior of the feasible set is nonempty ; ( j N) f + ϕ µj is a Kurdyka-Lojasiewicz (KL) function ; ( j N) (A j,k ) k are bounded from above and from below ; lim j µ j = 0 and ( i {1,..., 4}) lim j ɛ i,j /µ j = 0. Convergence Under some mild technical assumptions : for all j N, (x j,k ) k N converges to a solution to P µj ; (x j,0 ) j N is bounded and every cluster point of it is a solution to P 0 ; if in addition strict complementarity holds, and if there exists i {1,..., p} such that c i is strictly convex (or alternatively, for linear constraints, if some full rank property is satisfied) then (x j,0 ) j N converges to a solution to P 0. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

22 Hyperspectral Unmixing Hyperspectral Unmixing Problem Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

23 Hyperspectral Unmixing Hyperspectral Unmixing Model Optimization Problem p 1 minimize X R p n 2 Y SX κ (WX i ) d 1 i=1 p s.t. ( j {1,..., n}) X i,j 1 i=1 ( i {1,..., p})( j {1,..., n}) X i,j 0 p, n, s : number of endmembers, pixels, spectral bands Y R s n : observation S R s p : library X R p n : abundance matrix W R n n : orthogonal wavelet basis (.) d 1 : l 1 norm of the detail wavelet coefficients Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

24 Hyperspectral Unmixing Experimental Setting Realistic Data Simulation Urban 1 data set : p = 6 endmembers (known spectral signatures), s = 162 spectral bands, n = pixels Gaussian noise : σ 2 = Asphalt Road Grass Tree Roof Metal Dirt 1. http :// Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

25 Hyperspectral Unmixing Experimental Setting Realistic Data Simulation Urban 1 data set : p = 6 endmembers (known spectral signatures), s = 162 spectral bands, n = pixels large-scale pb : > variables. Gaussian noise : σ 2 = Asphalt Road Grass Tree Roof Metal Dirt 1. http :// Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

26 Hyperspectral Unmixing Experimental Setting Realistic Data Simulation Urban 1 data set : p = 6 endmembers (known spectral signatures), s = 162 spectral bands, n = pixels large-scale pb : > variables. Gaussian noise : σ 2 = Reconstruction Model Regularization weight : κ = W : orthogonal Daubechies 4 wavelet decomposition over 2 resolution levels. 1. http :// Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

27 Hyperspectral Unmixing Experimental Setting Realistic Data Simulation Urban 1 data set : p = 6 endmembers (known spectral signatures), s = 162 spectral bands, n = pixels large-scale pb : > variables. Gaussian noise : σ 2 = Reconstruction Model Regularization weight : κ = W : orthogonal Daubechies 4 wavelet decomposition over 2 resolution levels. Algorithm Parameters Variable metric : A j,k := 2 ϕ µj (x j,k ) [Becker et al., 2012]. The barrier parameter (µ j ) j N and the stopping criteria {ɛ i,j /µ j } i {1,...,4} follow a geometric decrease. 1. http :// Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

28 Hyperspectral Unmixing Experimental Setting Realistic Data Simulation Urban 1 data set : p = 6 endmembers (known spectral signatures), s = 162 spectral bands, n = pixels large-scale pb : > variables. Gaussian noise : σ 2 = Reconstruction Model Regularization weight : κ = W : orthogonal Daubechies 4 wavelet decomposition over 2 resolution levels. Algorithm Parameters Variable metric : A j,k := 2 ϕ µj (x j,k ) [Becker et al., 2012]. The barrier parameter (µ j ) j N and the stopping criteria {ɛ i,j /µ j } i {1,...,4} follow a geometric decrease. Implementation Matlab R2016b, Intel Xeon 3.2 GHz processor and 16 GB of RAM. Code will be available soon on 1. http :// Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

29 Hyperspectral Unmixing Comparison State-of-the-Art Algorithms No reg : interior point least squares algorithm without regularization [Chouzenoux et al., 2014] ADMM : alternating direction of multipliers method [Setzer et al., 2010] PDS : primal-dual splitting algorithm [Combettes et al., 2014] GFBS : generalized forward-backward splitting algorithm [Raguet et al., 2013] Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

30 Hyperspectral Unmixing Evaluation Metric Signal-to-Noise Ratio SNR = 10 log 10 ( p i=1 X i X i 2 2 X i 2 2 ) ; SNR i = 10 log 10 ( Xi X i 2 2 X i 2 2 ) where X i is the true abundance map of the i th endmember. Distance from the Iterates to the Solution x j,k x 2 x 2 where x is the vectorization of X and x is obtained after a very large number of iterations. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

31 Results Quantitative Results No reg : SNR = 1.96 db / With regularization : SNR = 3.65 db sec No reg Figure Left : global SNR versus time. Right : distance from the iterates to the solution versus time. Asphalt Road Grass Tree Roof Metal Dirt No reg ADMM PDS GFBS PIPA Table SNR (db) for all endmembers after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

32 Visual Results Asphalt Road Groundtruth No reg ADMM No reg ADMM 6.75 PDS 2.06 GFBS 2.17 PIPA Table SNR (db) PDS GFBS PIPA Figure Abundance map of asphalt road after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

33 Visual Results Grass Groundtruth No reg ADMM No reg ADMM PDS 3.33 GFBS 3.57 PIPA Table SNR (db) PDS GFBS PIPA Figure Abundance map of grass after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

34 Visual Results Dirt Groundtruth No reg ADMM No reg ADMM PDS GFBS PIPA Table SNR (db) PDS GFBS PIPA Figure Abundance map of dirt after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

35 Conclusion Application of a new proximal interior point algorithm to hyperspectral unmixing with a non-differentiable regularization. Convergence guaranteed under mild assumptions. Possibility to include an arbitrary preconditioner. Good performance in the context of a large-scale image recovery application. Extension of the convergence proof to inexact proximity operator. Other applications. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

36 References N. I. M. Gould, D. Orban, A. Sartenaer and P. L. Toint. Superlinear convergence of primal-dual interior point algorithms for nonlinear programming SIAM Journal on Optimization, Vol. 11, No. 4, pp , P. Armand, J. C. Gilbert and S. Jan-Jégou. A feasible BFGS interior point algorithm for solving convex minimization problems SIAM Journal on Optimization, Vol. 11, No. 1, pp , E. Chouzenoux, J.-C. Pesquet and A. Repetti. A block coordinate variable metric forward-backward algorithm Journal of Global Optimization, Vol. 66, No. 3, pp , S. Salzo. The variable metric forward-backward splitting algorithm under mild differentiability assumptions SIAM Journal on Optimization, Vol. 27, No. 4, pp , S. Setzer, G. Steidl and T. Teuber. Deblurring Poissonian images by split Bregman techniques Journal of Visual Communication and Image Representation, Vol. 21, No. 3, pp , E. Chouzenoux, M. Legendre, S. Moussaoui and J. Idier. Fast constrained least squares spectral unmixing using primal-dual interior-point optimization IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 7, No. 1, pp 59 69, P. L. Combettes, L. Condat, J.-C. Pesquet and B. C. Vũ. A forward-backward view of some primal-dual optimization methods in image recovery Proceedings of the IEEE International Conference on Image Processing (ICIP 2014), pp , H. Raguet, J. Fadili, G. and Peyré. A generalized forward-backward splitting SIAM Journal on Imaging Sciences, Vol. 6, No. 3, pp , S. Becker and J. Fadili. A forward-backward view of some primal-dual optimization methods in image recovery Proceedings of the 25th Advances in Neural Information Processing Systems Conference (NIPS 2012), pp , Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

37 Thank you! Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 22

38 Stopping Criteria Backtracking [Salzo, 2017] For j and k fixed, the backtracking procedure stops if : ϕ µj ( x l j,k ) ϕµ j (x j,k) x l j,k x j,k ϕ µj (x j,k ) If f := 0, Armijo linesearch along the steepest direction. δ γθ l x l j,k x j,k 2 A j,k Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 4

39 Stopping Criteria Backtracking [Salzo, 2017] For j and k fixed, the backtracking procedure stops if : ϕ µj ( x l j,k ) ϕµ j (x j,k) x l j,k x j,k ϕ µj (x j,k ) If f := 0, Armijo linesearch along the steepest direction. δ γθ l x l j,k x j,k 2 A j,k Accuracy for Solving P µj The barrier parameter is decreased as soon as the following criteria are met : 1 x j,k x j,k+1 ɛ 1,j A γ j,k (x j,k x j,k+1 ) ɛ 2,j j,k p c i (x j,k+1 ) p c 1 c i (x j,k ) ɛ 3,j µ i (x j,k ) c i (x j,k+1 ) j c i (x j,k ) ɛ 4,j i=1 i=1 where {(ɛ i,j ) j N } i {1,...,4} and (µ j ) j N are strictly positive sequences converging to 0 such that ( i {1,..., 4}) lim ɛ i,j /µ j = 0. j Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 4

40 Tree Groundtruth No reg ADMM No reg ADMM PDS 4.73 GFBS 4.76 PIPA Table SNR (db) PDS GFBS PIPA Figure Abundance map of tree after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 4

41 Roof Groundtruth No reg ADMM No reg ADMM PDS 6.63 GFBS 7.66 PIPA Table SNR (db) PDS GFBS PIPA Figure Abundance map of roof after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 4

42 Metal Groundtruth No reg ADMM No reg 4.90 ADMM 7.57 PDS GFBS 0.05 PIPA 7.06 Table SNR (db) PDS GFBS PIPA Figure Abundance map of metal after 13 sec. Corbineau, Chouzenoux, Pesquet PIPA: Proximal Interior Point Algorithm ICASSP / 4