EE 364B Convex Optimization An ADMM Solution to the Sparse Coding Problem. Sonia Bhaskar, Will Zou Final Project Spring 2011

Transcription

1 EE 364B Convex Optiization An ADMM Solution to the Sparse Coding Proble Sonia Bhaskar, Will Zou Final Project Spring 20 I. INTRODUCTION For our project, we apply the ethod of the alternating direction of ultipliers and sequential convex optiization to sparse coding of iages. The otivation behind sparse coding of iages is to odel how the brain is able to efficiently utilize the huan visual syste for a variety of tasks, such as separating a car fro a background, as well as general classification tasks. Sparse coding ais to deterine a generalized set of overcoplete bases Φ to represent any natural iage, where we desire that each iage can be represented by a linear cobination of a few of these bases, i.e. represented by a sparse vector of coefficients α. When learned on natural iages, the sparse coding bases are localized edge detectors in space, orientation and frequency. Feature learning on natural iages with sparse coding can be challenging because there can arbitrarily large input diensions, dependent on how big the visual field for the algorith is. Further, in any situations, an over-coplete set of bases is desired [], which eans that the nuber of features to learn exceeds the nuber of input diensions. Both aspects deands faster algoriths to ake the sparse coding proble feasible, either on a single coputer or on ultiple in a parallel coputing setting. Aong odels for natural iages, sparse coding is not a convex objective but is convex when fixing either Φ or α. This forulation akes it slow to train especially when the input diensions are high. We wish to consider a distributed solution using ADMM, which we believe will speed up the algorith. Additionally we wish to try out a heuristic in order to be able to use the desired optiization ethods, which we will soon discuss. II. PROBLEM DESCRIPTION First, we present the proble forulation for sparse coding below. The optiization proble, for x (i R n, Φ R n k, and α (i R k, where i refers to the i th training exaple, is ( iniize Φ, α (i i 2 x(i Φα (i λ α (i subject to Φ j 2 2, j =,..., k. ( where Φ j refers to the coluns of Φ, α (i j refers to the j th entry of α (i, Φ j 2 2 j is called the non-degeneracy constraint, and where we define the objective function f as given in Eqn. (2: f(x (i, α (i, Φ = ( 2 x(i Φα (i λ α (i (2 = n 2 x(i α (i j Φ j λ α (i (3 Thus this proble answers the question Given an input x (i, what is the optial vector of coefficients α (i that will give us the best linear reconstruction of the input using the coluns of a atrix of the optial bases, Φ? Looking at both of these cases, we see that the objective function is not convex if we are solving for both Φ and α (i i. However, if we only solve for either the bases Φ or the coefficients α (i while holding the other constant, we have a convex proble with convex constraints. Sequential convex prograing can be used here, but it is often slow. So we now explore soe solutions to this issue. j=

2 2 III. SEQUENTIAL CONVEX PROGRAMMING We see that for Eqn. (2, this is not a convex objective, although the constraint on the bases is convex. Often sequential convex prograing is used to solve this proble, that is, the proble is separated into first solving for the α (i s, which is then a least-squares proble regularized by an L-nor (sparsity, coonly known as the lasso proble, and then solving for Φ, which is a constrained least-squares proble. Thus the convex proble would need to be solved sequentially, since when we fix Φ and solve for the α (i s it is convex, and when we fix the α (i s and then solve for Φ it is convex. The sequential progra would go as follows. First, we would optiize for the α (i s: iniize α (i i Then we would solve for the bases: iniize Φ ( 2 x(i Φα (i λ α (i 2 x(i Φα (i 2 2 subject to Φ j 2 2, j =,..., n. As entioned in Section I, this iterative optiization is slow we now turn to the alternating direction ethod of ultipliers (ADMM in order to solve it. Previous ethods have used dual decoposition, but ADMM is a ore robust ethod with faster convergence which we hope to exploit for speed. IV. ALTERNATING DIRECTION METHOD OF MULTIPLIERS We can forulate the convex optiizaton probles into their ADMM for, resulting in what we believe will be a uch faster solution. A. Solving for the coefficients Solving for the α (i s, we have iniize α (i ( 2 x(i Φα (i λ ˆα (i subject to α (i = ˆα (i, i =,...,. The ADMM algorith updates would run as follows - note that we have a set of updates for each of the training exaples. We can do the in parallel we can do the siultaneously in atrix forat (for our ipleentation we chose the latter: α (i, k+ := (Φ T Φ + ρi (Φ T x (i + ρˆα k y k ˆα (i, k+ := ax (α (i, k+ + ρ yk λρ, 0 ax ( α (i, k+ ρ yk λρ, 0 y (i, k+ := y (i, k + ρ(α (i, k+ ˆα (i, k+ B. Solving for the bases Optiizing for the bases Φ, where each basis is a colun Φ j, the proble is cast as in Section III. Since the proble is convex, we can solve it directly using CVX, however, the found epirically that solving the proble directly is tie-consuing. This is because when we solve for the bases Φ R n k, the nuber of variables is nk. An alternative solution is given in [2]. Instead for solving nk variables, we can for the dual proble and solve with only n variables. We ipleented this ethod and report results in the results section.

3 Further, we can use the ADMM for for the constrained convex proble. The updates are, where U k R n k : Φ k+ := arg in Φ 2 x(i Φα (i ρ 2 Φ ˆΦ k + U k 2 2 ˆΦ k+ := Π C (Φ k+ + U k U k+ := U k + Φ k+ ˆΦ k+ where the projection Π C is applied to each colun of the atrix individually as { Xj X Π C (X = Π C (X j j, Π C (X j = j for X 2 j 2 > X j for X j 2 and X j is the jth colun of X. Unfortunately this is very slow to solve for the first step in the iteration, and CVX cannot handle probles where we have chosen the nuber of bases to be very large and thus we did not experience uch success because of the aount of tie we had to wait for each iteration. So next we present a heuristic solution which is uch faster. 3 (4 V. ALTERNATIVE SOLUTION TO NON-DEGENERACY CONSTRAINT In order to eploy an ADMM solution to solving for the bases where we are considering the coluns as bases, we recast the iniization proble using a heuristic. We recast the proble given in Eqn. (. We replace the non-degeneracy constraint and regularize the least-squares proble with a Frobenius nor: iniize Φ 2 x(i Φα (i β Φ 2 F This objective function is a convex function with a nice solution. First taking the gradient with respect to Φ we find that ( Φ 2 x(i Φα (i β Φ 2 F ( = Φ = ( x (i α (it + Φ(α (i α (it + 2βΦ ( = Φ α (i α (it + 2βI x (i α (it 2 (x(it x (i 2x (it Φα (i + α (it Φ T Φα (i + β Φ 2 F Solving for the global iniu, we find ( ( Φ = x (i α (it α (i α (it + 2βI (5 This heuristic is approxiately equivalent to the original proble and gives relevant solutions. Using this heuristic is advantageous because this convex optiization proble is a convex quadratic that has an analytical solution and the inverse is over a k k atrix, which is not terribly large.

4 4 VI. RESULTS To illustrate the practicality of our ethods, experients are perfored by learning a sparse coding dictionary fro natural iage patches. Copared to other feature learning algoriths for iages, such as Independent Coponent Analysis and its variants [3], sparse coding on natural iages is uch ore coputationally expensive. At the sae tie, ultiple papers have shown that sparse coding yields better results in tasks such as object recognition and detection. We show that our propsed ethods are able to significantly speed up the sparse coding algorith and our ipleentation is a useful tool for coputer vision researchers. A. CPU ipleentation In our experients, 4x4 iage patches were extracted fro the sae iage dataset as provided in the original work on sparse coding []. We run sequential convex optiization for 00 iterations to learn the coefficients and 96 bases. To iprove the speed of convergence, the data was split into inibatches of 000 and the algorith iteratively optiizes with respect to each inibatch. For ADMM we select ρ = 0 and run for 5-0 iterations in each lasso solve; we found ρ = 0 to give us the fastest convergence. Figure shows the bases learned by ADMM lasso solve with (left Frobenius nor regularization on bases and (right dual solve for bases. The two ethods yield very nice features (we desire edge-like features note that there is not a : correspondence between each block in Figure. Fig.. Bases learned fro (left Frobenius nor regularization on the bases and (right dual solve for bases cobined with ADMM lasso solve in sequential convex optiization. The regularization paraeter is selected as β = 5e 3 For the proposed ethod of using Frobenius nor regularization to avoid degeneracy of the dictionary, we plot convergence, sparsity, Frobenius nor, as well as histogra of coefficients in Figure 2. The objective of the optiization converges soothly with sequential convex optiization, a significant extent of sparsity is aintained in the coefficients, and the Frobenius nor is well controlled. We note that even though the plot shows sparsity at 65%, we look at the histogra and see that we have the desired distribution with the vast ajority of coefficients being zero, and very few of the being greater than zero. The resulting coefficients for a sparse epirical distribution. In Table I, tiing results are presented for learning the dictionary (Φ with 5 bases. Table II shows tiing results for learning the dictionary with the sae nuber of bases as the nuber of input diensions (96. In both tables, evaluation is perfored against popular sparse coding algoriths: the feature-sign algorith [4] and SPAMS toolbox [5]. In Table II, NF indicates not feasible in coputation tie or in eory. TABLE I LEARNING 5 BASES: TIME COMPARISON WITH SPAMS AND FEATURE-SIGN # Algorith tie (sec speedup Feature-sign authors code [4] e5 SPAMS authors code [5] (ex e5 CVX lasso + CVX Prial solve for bases base ADMM lasso + CVX Prial solve for bases ADMM lasso + Dual solve for bases e3 ADMM lasso + Frobenius nor regularization 2.4.3e4

5 5 Fig. 2. Plots for the Frobenius nor regularization ethod cobined with ADMM lasso solve. On the left we show the reconstruction cost convergence, sparsity (percentage of zeros in coefficients, and the Frobenius nor of the dictionary atrix. On the right, we show the histogra of resulting coefficients. TABLE II LEARNING 96 BASES: TIME COMPARISON WITH SPAMS AND FEATURE-SIGN # Algorith tie (sec speedup Feature-sign authors code [4] 95. base SPAMS authors code [5] (ex CVX lasso + CVX prial solve for bases NF ADMM lasso + CVX prial solve for bases NF ADMM lasso + Dual solve for bases ADMM lasso + Frobenius nor regularization In the lasso solve, using ADMM to quickly solve L/L2 sub-probles significantly iproves the speed of inferring coefficients. The ADMM lasso solver only needs to run a sall nuber of iterations (5-0 for efficient learning of the dictionary. As can be seen in rows 4 and 5 in Table I and Table II, the ethod of solving for bases using the dual proble with any fewer nuber of variables is faster than solving for the prial proble by a large argin. Coparing rows 5 and 6 in both tables, our ethod of using Frobenius nor regularization for solving for the bases further further iproves speed by a considerable factor. Fro the above results, the speed-up factor of our ethod copared to the naive ipleentation is.3e4 ties. The Feature-sign algorith perfors well when the nuber of bases is sall. However, it becoes a lot slower as the nuber of bases increase (Table II, and in fact we use it as the baseline and our ethod perfor 0 faster. Of all ethods, the SPAMS sparse coding tool box is still fastest. We note that copared to our siple ipleentation, SPAMS uses a highly optiized block-coordinate descent algorith and with ex ipleentation. Further, since the algorith heavily exploits sparsity with block-coordinate descent, its perforance drops when we reduce the value of λ. We copare to our algorith with different λ settings in Figure 3. To suarize, our ethod perfors reasonably well with a siple ipleentation. Iportantly, it is not affected by the extent of sparsity or nuber of feature diensions. This is advantageous when we hope to learn an over-coplete set of bases to represent natural iages. Further, we can think of two technical ways to speed up our algorith: GPU and parallel coputing using ADMM. We eployed a GPU ipleentation and obtained a significant speedup. B. GPU ipleentation We ipleented our algorith with ADMM lasso solve and Frobenius nor regularied solver for the bases with Jacket ( a GPU ipleentation for MATLAB. The result is shown in Table III. Since our proposed ethod iterates between two fast steps with ultiple atrixvector ultiplications, the GPU is able to significantly speed up our algorith. More concretely, it speeds

6 SPAMS Our ethod elapsed tie labda Fig. 3. Effect of sparsity setting on speed of SPAMS: since SPAMS heavily exploits sparsity, we experient with a large range of λ values. Our algorith is unaffected by the sparsity setting up our algorith by 8 ties. With GPU, our ethod is 8 faster than the Feature-sign algorith and even 2 faster than the state-of-the-art SPAMS toolbox. TABLE III LEARNING 96 BASES: TIME COMPARISON WITH SPAMS AND FEATURE-SIGN # Algorith tie (sec speedup Feature-sign authors code [4] 95. base SPAMS authors code [5] (ex GPU ADMM lasso + Frobenius nor regularization VII. CONCLUSION In the report, we analyzed the ethod of sequential convex optiization to solve the sparse coding proble applied on natural iages. The ADMM lasso solver significantly speeds up the practical probles of learning the sparse coding dictionary and inferring the sparse representation of iages, when the dictionary is known. Further, we experiented with fast ways of solving the second step, solving for the bases using the dual as proposed in [4], and developed a ethod of using the Frobenius nor regularization on the dictionary. With the regularization ethod cobined with ADMM lasso solve, we presented a fast iplentation of sparse coding on natural iages. When coputation is ported on GPUs, the proposed ethod achieves state-of-the-art speed-up for sparse coding on natural iages. VIII. FUTURE DIRECTIONS The ethods we experiented with in this report is run on a single coputer. The proposed advantage of the ADMM ethod extends to the capability of running ultiple sub-probles on ultiple coputers. This is yet to be explored and we would expect a slightly less than linear-speed up with the nuber of coputers used, when we use ADMM to parallelize data and solve for the lasso step. Another interesting future direction is local receptive fields with sparse coding on large iages. There has been recent work in artificial intelligence literature showing encouraging results using this ethod [6]. The idea of applying ADMM to speed-up and parallelize local sub-probles is rather siple. However, fro our experients, ipleentation of this idea is technically challenging. REFERENCES [] B. Olshausen and D. Field. Eergence of siple-cell receptive field properties by learning a sparse code for natural iages. Nature, 996. [2] Rajat Raina, Alexis Battle, Honglak Lee, Benjain Packer, and Andrew Y. Ng. Self-taught learning: Transfer learning fro unlabeled data. In ICML, 2007.

7 [3] A. Hyvarinen, J. Hurri, and P. Hoyer. Natural Iage Statistics. Springer, [4] H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algoriths. In NIPS, [5] J. Mairal, F. Bachand J. Ponce, and G. Sapiro. Online learning for atrix factorization and sparse coding. Journal of Machine Learning Research, 200. [6] K. Gregor and Y. LeCun. Eergence of coplex-like cells in a teporal product network with local receptive fields. arxiv: ,