p-norm MINIMIZATION OVER INTERSECTIONS OF CONVEX SETS İlker Bayram

Transcription

1 p-norm MINIMIZATION OVER INTERSECTIONS OF CONVEX SETS İlker Bayram Istanbl Technical University, Department of Electronics and Telecommnications Engineering, Istanbl, Trkey ABSTRACT We consider the imization of the l p norm sbject to conve constraints. The problem considered in this paper may be regarded as a relaation of a similar problem that employs the l norm. We derive the dal problem, which is nconstrained and devise an algorithm for the dal problem by adapting the Doglas-Rachford algorithm. We demonstrate the tility of the algorithm on an eperiment and discss its differences with an eisting algorithm. Inde Terms Basis prsit, Doglas-Rachford algorithm, imm norm soltion, Dykstra s algorithm, bridge estimate.. INTRODUCTION Let K, K 2,..., K k be closed, conve sets (with nonempty intersection) in R n. In this paper, we consider the imization problem, p () K K 2... K k for < p <. We derive a dal problem and making se of the dal problem, propose an algorithm to obtain the imizer. Althogh solving () with p cold be of interest per se (see e.g. [8], Sec.6), we consider it as a variation of the basis prsit (BP) problem [3], s.t. A = d. (2) BP is eqivalent to a linear program and can be solved as sch. However, if the observations d are noisy, instead of BP, it is desirable to consider the problem s.t. A d 2 ɛ, (3) where ɛ is related to the noise level. One approach to address this problem is to consider the nconstrained problem ˆ λ = arg A d λ. (4) The athor thanks TÜBİTAK for partially fnding this research. For each λ, if we set ɛ λ = A λ d 2, then λ is the imizer of (3) for ɛ = ɛ λ. Moreover, for fied ɛ, we can find λ ɛ s.t. A λɛ d = ɛ that is, λɛ solves (3). There are methods (see, e.g. [7]) to trace the soltions λ as λ is varied. As a variant of this, [] traces the soltions of the LASSO problem A d 2 s.t. τ (5) for different τ. We also refer to [8] for a relevant interesting approach, which obtains an algorithm for LASSO throgh a stdy of the so-called bridge estimators. In this paper, we replace (i) the l norm with the l p norm, (ii) consider a more general constraint of the form K... K k where K i s are closed conve sets. These changes lead to the following advantages. (i) The soltion of the dal-problem for the l norm does not directly lead to the soltion of the primal problem. Relaing the l norm allows s to link the primal and the dal soltion directly. (ii) Decomposing the constraint set K as K = K... K k, allows s to tilize the projections onto K i s which can be easier to realize than projections onto K. As a final remark, we note that another approach to solving () might be to employ more general parallel proimal algorithms as described in [4, 3]. These algorithms employ projections or proimal mappings that are as easy to realize as the ones in the proposed algorithm. Parallel proimal algorithms are also attractive as they can work with p =. However, they are different from the proposed algorithm in that they do not differentiate between the l p term and the constraint projections in (). We provide a frther discssion on the possible conseqences in Section 4. Otline In Section 2 we derive the dal problem and Proposition. The Doglas-Rachford algorithm is briefly reviewed and adapted to the dal-problem in Section 3. In Section 4 we provide two different applications of the algorithm. Section 5 is the conclsion.

2 Notation Throghot the paper, bold variables, like z, denote vectors in R n. We denote the i th component of a vector z as either z i or z(i). We cation that, terms like z i denote vectors in R n. For a set K, the spport fnction of K, denoted by σ K (), is defined as σ K () = sp z K, z. We refer to [2] for a detailed investigation. 2. THE DUAL PROBLEM In this section, we derive the dal problem and discss how the soltion of the dal problem is related to the soltion of the primal problem for p. Specifically, we show that for /p + /q =, and K K 2... K k p p p (6) z,z 2,...,z k q z + z z k q q + sp, z K + sp 2, z sp k, z k. (7) 2 K 2 k K k are dal to each other. The imizers of the two problems are related as, Proposition. Sppose that < p <. If {z z 2..., z k } imize (7), then for z = z + z z k, sign(z) (z) q/p = sign(z ) z q sign(z2) (z2) q/p :=. (8) sign(zn) (zn) q/p imizes (6). Remark. For p = 2 (and hence q = 2), the primal-dal pair is well known. Indeed, Dykstra s algorithm [5, 0] can be interpreted as a coordinate-descent type algorithm working on the dal problem [9]. Remark 2. For p =, the soltion of the dal gives the signs of the soltion for the primal problem. The dal-problem consists of terms for which the proimal operators are feasible to realize. This in trn makes the Doglas-Rachford algorithm feasible. Or plan is to adapt the Doglas-Rachford algorithm to the dal problem and obtain the imizer of the primal problem by sing Prop.. Derivation of the Dal Problem We start by noting that P = inf K p p p = inf sp z p p p +, z σ K (z) }{{} h(,z) In words, the imization problem () is eqivalent to finding the saddle point of h(, z) (where we take K = i K i ). Now if we define the dal fnction g(z) as, g(z) = inf p p p +, z σ K (z) (9) we also have P = sp z g(z) [2]. If the imm of the lhs of (9) is achieved at, then, it can be shown that, 0 sign( i ) i p + z i for i =, 2,..., n. (0) Here, sign(t) is a set valed mapping defined as if t < 0, sign(t) = [, ] if t = 0, if t > 0. From this, we obtain, () z i sign( i ) i p for i =, 2,..., n. (2) Noting that, p = p/q, this is tre if i = sign(z i ) z i q/p for i =, 2,..., n. (3) Inserting this into the lhs of (9), and noting that q/p = q, we get g(z) = p z q q + sign(z) z q/p, z σ K (z) = p z q q z q q σ K (z) ( ) = q z q q + σ K (z) The dal problem ma z g(z), is therefore eqivalent to, (4) z q z q q + σ K (z). (5) Now if K = K K 2... K k then we have [2], σ K (z) = σ K (z ) σ Kk (z k ) z,...,z k sbject to z + z z k = z. (6) Inserting (6) into (5), the dal problem becomes z q z q q + σ K (z ) σ Kk (z k ) sbject to z + z z k = z, (7) which is eqivalent to (7). We see that if z is the imizer of (7), the relation stated in Prop. now follows from (2). Notice that, nlike the primal problem, the dal problem is nconstrained. This allows s to adapt known schemes to come p with an algorithm that converges to the imizer. In the following, we adapt the Doglas-Rachford algorithm [, 6] to this problem. Actally, sign( i ) i p is the i th entry of the sbgradient [2] of p p.

3 3. ADAPTING THE DOUGLAS-RACHFORD ALGORITHM Given a imization problem of the form, z f(z) + g(z) (8) the Doglas-Rachford algorithm finds the imizer throgh sccessive application of some combination of proimal operators of f and g. For a fnction h, its proimal operator with parameter λ is a mapping that maps a point z to the imizer of a fnctional defined in terms of h. More precisely, it is the operator Jh λ ( ) defined as, J λ h (z) = arg 2λ z h(). (9) In terms of J λ f, J λ g, the proimal operators for f and g, the Doglas-Rachford algorithm for solving (8) is [, 6], Algorithm The Doglas-Rachford Algorithm repeat ( z Jf λ 2 J λ g (z) z ) + ( z Jg λ (z) ) ntil convergence z Jg λ (z) Application of the Doglas-Rachford Algorithm to the Dal Problem For or problem, we define f and g as, f ( ) z, z 2,..., z k = z + z z k q q, q g ( z, z 2,..., z k ) = (20a) sp, z + sp 2, z K 2 K 2 + sp k K k k, z k (20b) Algorithm reqires s to sccessively apply J λ f and J λ g. Let s now derive what these operators correspond to, in or case. Comptation of J λ f To compte J λ f (z, z 2,..., z k ), we need to solve, 2,..., k 2 z z k k λ/q k q q. (2) The point J λ f (z, z 2,..., z k ) is the niqe point (, 2,..., k ) that satisfies, 0 i z i +λ sign ( ) k k q (22) for i =,..., k. 2 Smg these over i, we obtain, 0 z +λ k sign( ) q, (23) where = k and z = z z k. Notice that if we can solve for, then, i = z i λ sign( ) q. (24) Let s therefore look at (23). For the j th entry of, namely j, we need to solve j + (λ k) sign( j ) j q = z j. (25) To simplify notation, consider the problem of finding R sch that + d sign() q = z. (26) for d > 0. We note that sign() = sign(z). Now taking z > 0, again to frther simplify the eqality, we need to solve, for > 0, + d }{{ q = z. (27) } s() s() is a strictly increasing fnction with s(0) = 0 < z, and s(z) > z. Therefore, there is a niqe that satisfies s() = z with (0, z). This can be fond by an iterative algorithm. An algorithm that solves (2) is ths, Algorithm 2 Comptation of Jf λ Inpt : z, z 2,..., z k R n Otpt :, 2,..., k R n that imizes (2) z k i= z i d λ k for j = to n do a 0 b z(j) repeat c (a + b)/2 if c + d c q > z(j) then b c else a c end if ntil a b (j) sign ( z(j) ) (a + b)/2 for i = to k do i z i λ sign() q 2 Notice that, here the argment of the sign fnction is a vector the fnction is applied componentwise.

4 Comptation of J λ g To compte J λ g (z, z 2,..., z k ), we need to solve,..., k 2 z z k k λ σ K ( ) λ σ Kk ( k ). (28) This fnctional is separable with respect to i s. Therefore, we will only stdy the following imization problem. = arg 2 z λ σ K () (29) The following proposition (see [2] for a derivation of a similar reslt) will be sed to obtain a description of the imizer. Proposition 2. Let K be a closed conve set. Then, arg 2 z σ K () = z P K (z) (30) (a) (b) where P K (z) is the projection of z to K (i.e. the closest point in K to z). Using this proposition, and noting that λ σ K () = σ λ K (), we can write = z P λ K (z) (3) Here it is important that the projection operator be easily realizable. Algorithm 3 Comptation of J λ g Inpt : z, z 2,..., z k R n Otpt :, 2,..., k R n that imizes (28) for i = to k do i z i P λ Ki (z i ) Remark 3. The projections in Algorithm 3 can be performed in parallel. Taking this into accont leads to significant redction in rnning times, especially when the nmber of constraint sets is high, as in the Eperiment discssed below. 4. EXPERIMENTS We consider a denoising application where side information is also available. We have at hand the noisy image, namely y shown in Fig. a. We also have -D (Radon) projections of the image along the angles θ = k π/8 for k = 0,..., 7. We denote the linear operator which comptes the -D projection along the angle k π/8 as P k and the given projection data at k π/8 as d k. Or primary variables are the wavelet coefficients of the image (we sed an orthonormal wavelet transform with Dabechies filters that have 3 vanishing moments). (c) Fig.. Denoising with side information. The side information consists of Radon projections along the angles k π/8 for k = 0,..., 7. (a) Noisy image, RMSE = 0.2. Soltion for (b) p = 2, RMSE = , (c) p = 4/3, RMSE = , (d) p = 8/7, RMSE = We denote the synthesis operator corresponding to the wavelet basis as W. We also define the constraint sets as, (d) K i = { : P i W = d i } for i = 0,..., 7 (32) K 8 = { : W y σ} (33) where σ is taken as the standard deviation of the noise. Based on these definitions, we consider the imization problem, K p, where K = K 0... K 8. The reslts are shown in Fig. (b,c) for p = 2, p = 4/3 and p = 8/7. We observe that p leads to an improved reconstrction. Note that the formlation allows s to easily take into accont data Log-Dist. to Primal Soln. 5 0 p = 8/7 p = 4/3 Spingarn s Alg Iterations Fig. 2. Log-distance to the primal limit for the proposed algorithm and Spingarn s algorithm, derived from [3].

5 which cannot be epressed as an affine space. Comparison with Parallel Proimal Algorithms As we noted in the Introdction, there are other algorithms available for the primal problem (). Specifically, let s consider Spingarn s method of partial inverses [3] applied to the problem at hand (see Algorithm 4 below). For another similar algorithm, see [4]. Algorithm 4 Spingarn s Algorithm for () Initialize 0, y i 0 for i = 0,,..., m. repeat t 0 arg z 2 + y0 z z p 0 + y 0 t 0 for i = to m do t i P Ki ( + y i) i + y i t i m+ m i=0 ti for i = to m do y i i m+ m i=0 i ntil convergence Algorithm 4 consists of simple sbsteps like the proposed algorithm. However, this algorithm treats the proimal mapping of the l p norm and the projections onto K i s in the same manner. The pdated information from the sbsteps are merged only at the end of each iteration. On the other hand, the proposed algorithm treats the two parts of the dal problem differently. Once the projections onto K i s are performed, this information is fed into the proimal mapping of q q. In order to compare the performances of the two algorithms we performed the following eperiment on the phantom denoising problem described above. We ran both algorithms for 5000 iterations to obtain the limit in each case. We note that there are two limit points of interest one for the primal, one for the dal problem, related to each other throgh (2). Althogh the proposed algorithm works on the dal, we are interested in the ability of the algorithm to approach the imizer of the primal problem in as few iterations as possible. In order to provide a fair comparison, we normalized distances by setting the norm of the corresponding limit to nity. Fig.2 depicts the log-distance to the primal-limit for the proposed algorithm and Spingarn s algorithm. Since the proposed algorithm works on the dal-problem, we cannot claim that the distance to the primal-limit will be monotone decreasing. Indeed, the algorithm starts very far from the primal limit and the distance to the primal limit does increase in the first few iterations. However, interestingly, in 00 iterations, the distance to the primal-limit for the proposed algorithm is mch lower, compared to Spingarn s method. Considering the parallel natre of Spingarn s method, the scheme presented in this paper allows, in a sense, to trade the parallel strctre for faster convergence (given limited resorces). 5. CONCLUSION We considered a variation of the sparse reconstrction problem. Specifically, we replaced the the l norm with the l p norm for p >, and stdied a fleible formlation that allows mltiple constraints to be placed on the reconstrction. Althogh imm l p norm reconstrctions, for p >, are no longer sparse (in the strict sense of the word), this formlation can also be feasible. For p, the soltion of the primal problem provides an approimation to the imm l reconstrction. We also demonstrated that the proposed algorithm performs comparably with eisting alternatives. 6. REFERENCES [] E. Van Den Berg and M. P. Friedlander. Probing the pareto frontier for basis prsit soltions. SIAM J. Sci. Compt., 3(2):890 92, [2] A. Chambolle. An algorithm for total variation imization and applications. Jornal of Mathematical Imaging and Vision, 20(-2):89 97, Janary-March [3] S. S. Chen, D. L. Donoho, and M. A. Sanders. Atomic decomposition by basis prsit. SIAM J. on Scientific Compting, 20():33 6, 998. [4] P. L. Combettes and J.-C. Pesqet. A proimal decomposition method for solving conve variational inverse problems. Inverse Problems, 24(6), December [5] R. L. Dykstra. An algorithm for restricted least sqares regression. Jornal of the American Statistical Association, 78: , 983. [6] J. Eckstein and D. P. Bertsekas. On the Doglas-Rachford splitting method and the proimal point algorithm for maimal monotone operators. Mathematical Programg, 55(3):293 38, 992. [7] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32(2): , [8] W. J. F. Penalized regressions : The bridge verss the lasso. J. Compt. Graph. Statist., 7(3):397 46, 998. [9] N. Gaffke and R. Mathar. A cyclic projection algorithm via dality. Metrika, 36:29 54, 989. [0] S.-P. Han. A sccessive projection method. Mathematical Programg, 40: 4, 988. [] P. L. Lions and B. Mercier. Splitting algorithms for the sm of two nonlinear operators. SIAM J. on Nmer. Anal., 6(6): , 979. [2] R. T. Rockafellar. Conve Analysis. Princeton University Press, 996. [3] J. E. Spingarn. Partial inverse of a monotone operator. Applied Math. and Optimization, 0(): , 983.