Y. C. PATI Information Systems Laboratory Dept. of Electrical Engineering Stanford University, Stanford, CA 94305

Transcription

1 i 4ô- / To appear in Proc. of the Annual Asilomar Conference on Signals Systems and Computers, Nov. 1-3, 1993/ Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to Wavelet Decomposition Y. C. PATI Information Systems Laboratory Dept. of Electrical Engineering Stanford University, Stanford, CA R. REZAIIFAR AND P. S. KRISHNAPRASAD Institute for Systems Research Dept. of Electrical Engineering University of Maryland, College Park, MD 074 Abstract In this paper we describe a recursive algorithm to compute representations of functions with respect to nonorthogonal and possibly overcomplete dictionaries of elementary building blocks e.g. aiene (wa.velet) frames. We propoeea modification to the Matching Pursuit algorithm of Mallat and Zhang (199) that maintains full backward orthogonality of the residual (error) at every step and thereby leads to improved convergence. We refer to this modified algorithm as Orthogonal Matching Pursuit (OMP). It is shown that all additional computation required for the OMP al gorithm may be performed recursively. where fk is the current approximation, and Rkf the current residual (error). Using initial values of R0f = 1, fo = 0, and k = 1, the MP algorithm is comprised of the following steps,.,.41) Compute the inner-products {(Rkf,z)}. (H) Find flki such that (III) Set, I(R*f,1:n 1+,)l asupl(rkf,z,)i, where 0 < a < 1. 1 Introduction and Background Given a collection of vectors D = {z} in a Hubert space 1i, let us define V = Span{z}, and W = V1 (in E). We shall refer to D as a dictionary, and will aasume the vectors z,,, are normalized (jlxii = 1). In [3] Mal lat and Zhang proposed an iterative algorithm that they termed Matching Pursuit (MP) to construct rep resentations of the form Pyf=Eanzn, (1) where P1, is the orthogonal projection operator onto V. Each iteration of the MP algorithm results in an intermediate representation of the form f=ea:n+r*ffk+pf, ft fk+1 = 1k + (Rkf, z+1) Zflk+j = Rkf (Rkf,zflk+l)zflk+, (IV) Increment k, (k + k + 1), and repeat steps (I) (IV) until some convergence criterion has been satisfied. The proof of convergence [3] of MP relies essentially on the fact that (Rk+lf,zflk+l) = 0. This orthogonality of the residual to the last vector selected leads..±o the following energy conservation equation. lifii jj, + 1fjI + l(kf,zn4+1)i. () It has been noted that the MP algorithm may be d& rived as a special case of a technique known as Pro jection Pursuit (c.f. []) in the statisti literature. A shortcoming of the Matching Pursuit algorithm in its originally proposed form is that although asymp t.otic convergence is guaranteed, the resulting approxi mation after any finite number of iterations will in gen eral be suboptimal in the following sense. Let N < co 1

2 have 1,.. f Ii, f=azn+rkf, with (Rkf,s) =0, n 1,...k r./ (3) k+1,i=1 n=1,...k+1. (4) (b) Ill with <7k,Zn)O, n=1,...k. (5)...,z}. {z 11 b 1 k (5 5+1 k_ an a 3,..., and at = (R5f, Zk+i) (Rkf, Zk+j) = where a = (7s,zsi) gives the optimal approximation with respect to the selected subset of the dictionary. This is achieved by ensuring full backward orthogonality of the error i.e. at each iteration Rkf E V. For the example in Fig ure 1, OMP ensures convergence in exactly two itera tions. It is also shown that the additional computation required for OMP, takes a simple recursive form. We demonstrate the utility of OMP by example of applications to representing functions with respect to time-frequency localized affine wavelet dictionaries. We also compare the performance of OMP with that of MP on two numerical examples. Orthogonal Matching Pursuit Assume we have the following kthorder model for The superscript k, in the coefficients a, show the de pendence of these coefficients on the model-order. We would like to update this model to a model of order k+1, f = + Rk+lf, with (Rk+lf, z,) = 0, Since elements of the dictionary D are not required to be orthogonal, to perform such an update, we also require an auxiliary model for the dependence of zk+1 on the previous x s (n = 1,...k). Let, Thus = Pvbzk+a, and = Pvk+1, is the component of xk1 which is unexplained by Using the auxiliary model (5), it may be shown that the correct update from the ktorder model to the model of order k + 1, is given by be the number of MP iterations performed. Thus we fn = (f,zflk+i) z,. Define VN = Span{z,,. We shall refer to fn as an optima! N-term apprnximation if fri VNf i.e. fri is the best approximation we can construct using the selected subset {z.,zfln} of the dictionary D. (Note that this notion of optimal ity does not involve the problem of selecting an op timal N-element subset of the dictionary.) In this sense, fri is an optimal N-term approximation, if and only if RN! E V. As MP only guarantees that RNf L x,,, fri as generated by MP will in general be suboptimal. The difficulty with such suboptimality is easily illustrated by a simple example in JR. Let z1, and x be two vectors in JR and take f i as shown in Figure 1. Figure 1(b) is a plot of IIRsfll (7 thermore after any finite number of iterations, OMP A aimlai difficulty with the Projection Pursuit algorithm noted by Donoho elal [11 who suggested that bec.kfitting may be used to improve the convergence o( PPR Although the R5f, Xt+i) tedinique is not fully de.ibed in (ij it appears that it is in the b (z,,, Z54.j) Ilzt+ i11 E1 same spirit a. the tednique we present here. a a Figure 1: Matching pursuit example in 1R: Dic 1,z} and a vector f 1R tionary D = {z versus k. Hence although asymptotic convergence is guaranteed, after any finite number of steps, the error In this paper we propose a refinement of the Match may still be quite large. ing Pursuit (MP) algorithm that we refer to as Or thogonal Matching Pursuit (OMP). For nonorthogo nal dictionaries, OMP will in general converge faster than MP. For any finite size dictionary of N elements, OMP converges to the projection onto the span of the dictionary elements in no more than N steps. Fur-

3 3 convergence properties of OMP. The results of the previous section may be used to construct the following algorithm that we will refer to the form (7). The following theorem summarizes the on an ener conservation equation that now takes As in the case of MP, convergence of OMP relies. Some Properties of OMP The key difference between MP and OMP lies in Prop erty (ii/) of Theorem.1. Property (iif) implies that tions, and = T can get using the N vectors we have selected from arises in determining the b a of the auxiliary model where (5). To compute the b s we rewrite the normal equa (8) (f,z) = (f-fk,zf) = (f,x) a (x,z). we may express these recursions implicitly in the for at the N step we have the best approximation we {(Rkf, x,)) may be computed recursively. For OMP Remarks:.3 Some Computational Details iterations to the projection of f onto the span of the Pursuit does not possess this property. naries of size M, OMP converges in no more than M as Orthogonal Matching Pursuit (OMP). of Theorem 1 in [3]. The proof of the the second prop erty follows immediately from the orthogonality con ditions of Equation (3). the dictionary. Therefore in the case of finite dictio Proof: The proof of convergence parallels the proof.1 The OMP Algorithm (ii)fn=pv,,,f, N=O,1, = IIR+ifil dictionary elements. As mentioned earlier, Matching mula The only additional computation required for OMP, IIRkfH + 117k II tions associated with (5) as a system of k linear equa (7) k-.oo (1) urn IIRkf.PvfIIO. gerzeruted by OMP. Then As in the case of MP, the inner products (II) Find Xllb+ E D \ Dk such that VII) Set k k + 1, and repeat (I) (V1I). fk+1 k+1 (III) If (Rkf,xnh+,)I <, ( > 0) then stop. J (I) Compute {(Rkf, x,) ;z D \ Dk). 1itiali.ation Theorem.1 For f?t, let Rkf be the residuals R*+af = = DkU{zk+1). = a1z,* and update the model, a1=a akb, n=1,...,k, (VI) Set, cz =ak = 117k11 (Rkf,Zk+l), and (7k,z)=0, n=1,...,k. =E1 bz +7k (V) Compute {b} 1, such that, mutation k k k+1 (IV) Reorder the dictionary D, by applying the per 1(R*f,znh+jIasupl(Rkf,xi)I, O< a 1. Zo=O, a=o, k=o foo, R0ff, D0={) Rkf = Rk+lf + akj k, and It also follows that the residual Rk+jf satisfies, (X1,Xk) (z,zk)..: (:i,x (z1,x1) (z,z1)... (x,z) ) (x 1 x)... (z,z) = [(Zk+1,Z1),(Xk+j,Z).. Vk = Akbk (9)

4 4 derivate of a Gaussian, i.e. D = {m,,,, m, n E Z} and the analyzing wavetet 0 is given by, convolution followed by sampling at each dilation level a total of 351 vectors. Figure (b) that OMP clearly converges in far fewer Figure 3. We could also compare the two algorithms \1/ tations with repect to an a.ffine wavelet frame con structed from dilates and translates of the second In the following examples we consider represen where, Note that for wavelet dictionaries, the initial set of in error. However such a comparison can only be made ner products {(f, Om,n)}, are readily computed by one m. The dictionary used in these examples consists of In our first example, both OMP and MP were ap signal of Example I, we see from Figure 4 that it is 3 iterations than MP. The squared magnitude of the co pute representations of signals to within a prespecified erations required for each algorithm depends on both the signal and the dictionary. For example, for the efficients 0k, of the resulting representation is shown in on the basis of required computational effort to com plied to the signal shown in Figure. We see from for a given signal and dictionary, as the number of it plying OMP in Example 1. Shading is proportional to to apply MP. In this case OMP converges in exactly with respect to arbitrary dictionaries of elementary Figure 3: Distribution of coefficients obtained by ap squared magnitude of the coefficients 0k, with dark three iterations. functions. The algorithm we have described is a mod ification of the Matching Pursuit (MP) algorithm of iterations. On the other hand for the signal shown Pursuit (OMP), to compute representations of signals colors indicating large magnitudes. to 8 times more expensive to achieve a prespecified er In this paper we have described a recursive al 4 Summary and Conclusions in Figure 5, which lies in the span of three dictionary ror using OMP even though OMP converges in fewer vectors, it is approximately 0 times more expensive Tr.i W4 14 gorithm, which we refer to as Orthogonal Matching Mallat and Zhang [3] that improves convergence us- I 3 Examples (solid lini) and MP (dashed line)..bk from the previous step. bk j, may be computed recursively using A1, and bk = A vk. (10) hence we may write Note that the positive constant used in Step (111) of OMP guarantees nonsingulazity of the matrix Ak, (where * denotes conjugate transpose) it may be where fi = 1/(1 vzbk). Hence A1, and therefore Figure : Example I: Original signal f, with OMP residual Rkf versus iteration number k, for both OMP approximation superimposed, (b) Squared L norm of Oir.4 Sp w.d 0.W Amcn = (i.;;:;) (: 1) e_ / 4 = m/,(m1 k+11 flb shown using the block matrix inversion formula that I. V r A1 + bkb,3bk 1 (1) i I, (11) r Ak k 1 However, since Ak+1 may be written as o too a 40 to to

5 line). References Figure 4: Computational cost (FLOPS) versus ap ta z.d 1.,w This research of Y.C.P. was supported in part by NASA Headquarters, Center for Aeronautics and Space Informa tion Sciences (CASIS) under Grant NAGW419,S6, and in Acknowledgements to7 at each step. dictionaries, as knowledge of the redundancy is ex ploited in OMP to reduce the error as much as poible computationally cheaper than MP for very redundant Although we do not provide a rigorous argument here, it seems reasonable to conjecture that OMP will be proximation error for both OMP (solid line) and MP (dashed line) applied to the signal in Example 1. 5_ t0 0 0 (b) ing an additional orthogonalization step. The main finite dictionary. We also demonstrated that all addi on two simple examples of decomposition with respect on both the class of signals and choice of dictionary. putational effort required for each algorithm depends performed recursively. to a wavelet dictionary. It was noted that although OMP converges in fewer iterations than MP, the com The two algorithms, MP and OMP, were compared benefit of OMP over MP is the fact that it is guar anteed to converge in a finite number of steps for a tional computation that is required for OMP may be number k, for both OMP (solid line) and MP (dashed Figure 5: Example II: Original signal f, (b) Squared L norm of residual R.f versus iteration ZO 5 time-frequency dictionaries. Preprint. Submitted Statistics, 13() : , partment of Defense monitored by the Air Force Office of Scientific Research under Contract F I-Oog5, This research of R.R. and P.S.K was supported in part by the Air Force Office of Scientific Research under con Initiative Program under Grant AFOSR , by the Army Research Office under Smart Structures URI Con ence Foundation s Engineering Research Centers Program, NSFD CDR [1) D. Donoho, I. Johnstone, P. Rousseeuw, and W. Stahel. The Annals of Statistics, 13(): , Discussion following article by P. Hu ber. [) P..1. Huber. Projection pursuit. The Annals of [3] S. Mallat and Z. Zhang. Matching pursuits with to IEEE Trzrnsactions on Signal Processing, 199. part by the Advanced Research Projects Agency of the De tract F J-0500, the AFOSR University Research tract no. DAALO3-9-G-011, and by the National Sci