Iterative Improvement of the Multiplicative Update NMF Algorithm using Nature-inspired Optimization

Transcription

1 2011 Seventh International Conference on Natural Computation Iterative Improvement of the Multiplicative Update NMF Algorithm using Nature-inspired Optimization Andreas Janecek Key Laboratory of Machine Perception (MOE), and Department of Machine Intelligence; School of Electronics Engineering and Computer Science, Peking University, Beijing, , China Ying Tan Key Laboratory of Machine Perception (MOE), and Department of Machine Intelligence; School of Electronics Engineering and Computer Science, Peking University, Beijing, , China Abstract Low-rank approximations of data (e. g. based on the Singular Value Decomposition) have proven very useful in various data mining applications. The Non-negative Matrix Factorization (NMF) leads to special low-rank approximations which satisfy non-negativity constraints. The Multiplicative Update (MU) algorithm is one of the two original NMF algorithms and is still one of the fastest NMF algorithms per iteration. Nevertheless, MU demands a quite large number of iterations in order to provide an accurate approximation of the original data. In this paper we present a new iterative update strategy for the MU algorithm based on nature-inspired optimization algorithms. The goal is to achieve a better accuracy per runtime compared to the standard version of MU. Several properties of the NMF objective function underlying the MU algorithm motivate the utilization of heuristic search algorithms. Indeed, this function is usually non-differentiable, discontinuous, and may possess many local minima. Experimental results show that our new iterative update strategy for the MU algorithm achieves the same approximation error than the standard version in significantly fewer iterations and in faster overall runtime. I. INTRODUCTION The Non-negative Matrix Factorization (NMF, [13]) leads to a low-rank approximation which satisfies non-negativity constraints. Contrary to other low-rank approximations these constraints may improve the sparseness of the factors and due to the additive parts-based representation also improve interpretability [2], [8], [13]. NMF consists of reduced rank nonnegative factors W R m k and H R k n with k min{m, n} that approximate a matrix A R m n by A W H. NMF requires all entries in A, W and H to be zero or positive. Due to its nonnegativity constraints, NMF produces so-called additive parts-based representations of the data (in contrast to other linear representations such as SVD). This is an impressive benefit of NMF, since it makes the interpretation of the NMF factors much easier than for factors containing positive and negative entries. The Multiplicative Update (MU) algorithm is one of the two original NMF algorithms presented in [13] and is still one of the fastest NMF algorithms per iteration. Hence, it can be used to achieve a rough but very fast approximation [7]. Over the last decades, nature-inspired metaheuristic algorithms have gained much popularity due to their applicability for various optimization problems. The main advantage of these algorithms is the fact that they are able to find acceptable results within a reasonable amount of time for many complex, large and dynamic problems. Although they lack the ability to guarantee the optimal solution for a given problem, it has been shown that they are able to tackle various kinds of real-world optimization problems [5]. The goal of this paper is to iteratively improve the approximation quality of the MU algorithm by utilizing natureinspired metaheuristcs. This improvement is achieved by optimizing selected rows of W and/or selected columns of H, respectively, during the first iterations of the factorization. The idea of using optimization heuristics is strengthened by several properties of the NMF objective function underlying the MU algorithm. This function is usually non-differentiable, discontinuous, and may possess many local minima. Since metaheuristic algorithms are know to be able to deal well with such difficulties they seem to be a promising choice for the optimization process. The presented iterative update strategy is successful if it is able to achieve a better approximation error than the standard version of the MU algorithm in faster overall runtime. The iterative update process of the MU algorithm and the sequential optimization of selected rows of W and/or selected columns of H in each iteration allow for parallel and/or distributed computation. The tasks of optimizing different rows of W and different columns of H can be split up into several partly indepent sub-tasks and can thus be executed concurrently. Related Work. In 1994 Paatero and Tapper [16] published an early article on positive matrix factorization, but the work by Lee and Seung [13] five years later achieved much more popularity and is known as a standard reference for NMF. The original Multiplicative Update algorithm introduced in [13] provides a good baselines against which other algorithms (e. g. the Alternating Least Squares algorithm [16], the Gradient Descent algorithm [15], ALSPGRAD [15], quasi Newton-type NMF [11], fastnmf and bayesnmf [19], etc.) have to be judged. Although other algorithms are often able to achieve a better approximation at convergence, the MU algorithm is still the fastest NMF algorithm per iteration and a good choice if a very fast and rough approximation is needed [7] /11/$ IEEE 1668

2 Only few studies can be found in the literature that aim at combining NMF and metaheuristics, most of them are based on genetic Algorithms (GAs). In [21], the authors have investigated the application of GAs on sparse NMF for microarray analysis, while [20] have applied GAs for boolean matrix factorization, a variant of NMF for binary data based on Boolean algebra. However, their work differs significantly from the techniques introduced in this paper. In a preceding study we successfully applied nature-inspired algorithms as initialization enhancers for NMF [9]. We are not aware of any studies that investigate iterative update strategies for NMF algorithms using nature-inspired metaheuristics. Notation: A matrix is represented by an uppercase italic letter (A, B, Σ,... ), a vector by a lowercase bold letter (u, x, q 1,... ), and a scalar by a lowercase Greek letter (λ, µ,... ). The i th row vector of a matrix D is represented as d r i, and the j th column vector of D as d c j. II. REVIEW OF METHODS Non-negative Matrix Factorization (NMF). Low-rank approximations replace a data matrix with a related matrix of much lower rank in order to reduce the required storage space and/or to achieve a more efficient representation of the relationship between data elements. Although there are several well known techniques (e.g., the SVD), a main problem of most techniques refers to the interpretability of the transformed features after the approximation. The resulting orthogonal matrix factors generated by the approximation are often difficult to interpret because they contain positive and negative coefficients. A negative quantification is often meaningless in the application domain and the information about how much an original attribute contributes is lost. The NMF leads to special low-rank approximations which satisfy non-negativity constraints. These constraints require that all entries in A, W and H are zero or positive. This makes the interpretation of the NMF factors much easier than for factors containing positive and negative entries, and enables NMF a non-subtractive combination of parts to form a whole [13]. Optimization Problem. The nonlinear optimization problem underlying NMF can generally be stated as 1 min f(w, H) = min W,H W,H 2 A W H 2 F. (1) The Frobenius norm. F is commonly used to measure the error between the original data A and the approximation W H (other measures, e.g. based on the Kullback-Leibler divergence, are also possible [14]). Unlike the SVD, the NMF is not unique, and convergence is not guaranteed for all NMF algorithms. If they converge, then usually to local minima only (potentially different ones for different algorithms). Nevertheless, the data compression achieved with only local minima has been shown to be of desirable quality for many data mining applications [12]. Algorithms for computing NMF are iterative and require initialization of W and H. A proper initialization can lead to faster error reduction and better overall error at convergence. Although the benefits of good initialization techniques are well known in the literature, most studies use random initialization [3]. A possible problem of most initialization techniques is the fact that the initialization procedure can be rather costly in terms of runtime [8]. The Multiplicative Update Algorithm. The general structure of the MU algorithm is given in Alg. 1. Usually, W and H are initialized randomly and the whole algorithm is repeated several times (maxrepetition). In each repetition, the update steps of the MU algorithm are processed until a maximum number of iterations is reached (maxiter). If the approximation error drops below a pre-defined threshold, or if the shift between two iterations is very small, the algorithm might stop before all iterations are processed. The update steps taken from [14] are based on the mean squared error objective function. The parameter ε in each iteration is a small value (usually ε = 10 9 ) used to avoid division by zero. The divisions in Algorithm 1 are to be performed element-wise. Comments about the convergence of the MU algorithm can be found, for example, in [2], [15]. Given matrix A R m n and k min{m, n}; for rep = 1 to maxrepetition do W = rand(m, k); H = rand(k, n); for iter = 1 to maxiter do % perform MU specific update steps W = W (AH )./(W HH + ε); H = H (W A)./(W W H + ε); check termination criterion; Algorithm 1: General Structure of the MU Algorithm Nature-inspired Metaheuristics. We use five nature-inspired optimization heuristics for iteratively improving the approximation quality of the MU algorithm. Details about the heuristics can be found in the references provided below. Genetic Algorithms (GA, [6]) are adaptive heuristic search algorithms based on evolutionary processes such as natural selection, mutation, and crossover. Particle Swarm Optimization (PSO, [10]) is inspired by the social behavior of swarms. Particles adjust their position in the search-space based on their own best history position as well as on the global best position found so far. Differential Evolution (DE, [18]), is a simple yet effective population based function minimizer, which moves particles around in the search-space using simple mathematical formula. Fish School Search (FSS, [1]) mimics the movement of fish schools. The main operators are feeding (fish can gain/loose weight, deping on the region they swim in) and swimming. Fireworks Algorithm (FWA, [22]) is a recently developed metaheuristic that simulates the explosion process of fireworks. 1669

3 III. METAHEURISTICS FOR OPTIMIZING NMF FACTORS We use the Frobenius norm (1) as NMF objective function for the MU algorithm to measure the error between A and W H, because it offers some properties which can be utilized to apply metaheuristics for optimizing NMF. The Frobenius norm of a matrix D is defined as D F = ( d ij 2 ) 1 /2, where d ij is the element in the i th row and j th column of D. Since a reduction of the Frobenius norm of any row or any column of D leads to a reduction of the total Frobenius norm D F, the Frobenius norm of D can also be computed row wise, by summing up the norms of all row vectors d r i of D, or column wise, by summing up the norms of all column vectors d c j of D. Hence, we are interested in reducing the norm of selected rows and/or columns of D, which is the distance matrix between the original data matrix A and the approximation W H, such that D = A W H. In order to improve the approximation as fast as possible we identify rows of D with highest norm (i.e. the approximation of this row is worse than for other rows of D) and optimize the corresponding rows of W. The same procedure is used to identify the columns of H that should be optimized. for iter = 1 to maxiter do W = W (AH )./(W HH + ε); H = H (W A)./(W W H + ε); if (iter < m) then % Update rows of W d r i is the i th row vector of D = A W H; [V al, IX W ]= sort(norm(d r i ), desc ); IX W = IX W (1 : c); i IX W : minimize a r i wr i H F ; W (i, :) = w r i ; % Update columns of H d c j is the j th column vector of D = A W H; [V al, IX H] = sort(norm(d c j), desc ); IX H = IX H(1 : c); j IX H: minimize a c j W hc j F ; H(:, j) = h c j; c = c c check termination criterion; Algorithm 3: Iterative Optimization of the MU Algorithm The iterative optimization procedure for the MU algorithm using metaheuristics is summarized in Alg. 3 (which exts the inner loop of Alg. 1). Variables used in Alg. 3: m: number of iterations in which the optimization using metaheuristics is applied. For example, if m = 5, the optimization is applied in the first 5 iterations. If iter >= m, only the standard MU update steps are executed. c: the number of rows and/or columns that are optimized in the current iteration. For example, if c = 10, the 10 row vectors of D having the highest norm are identified and the 10 corresponding rows of W are optimized. c: the value of c is decreased by c in each iteration. c = round(c initial /m) Functions used in Algorithm 3: [V al, IX W ] = sort(norm(d r i ), desc ): returns the values (V al) and the corresponding indizes (IX W ) of the norm of all row vectors d r i of D in descing order. IX W = IX W (1 : c): returns only the first c elements of the vector IX W. minimize a r i wr i H F : in this line a r i (the ith row of A) and H are used as input parameters for the optimization algorithms, the output is the optimized row vector w r i. W (i, :) = w r i : the ith row of W is replaced with w r i. Updating columns of H is similar to updating the rows of W. The dimension of the optimization problem is identical to the rank k of the NMF and is given by the first dimension of H and the second dimension of W, respectively. Parallelism. As the metaheuristics have relatively high computational cost compared to the simple MU update steps it is necessary to parallelize the optimization tasks. Since all rows of W are indepent from each other the optimization of any row of W does not influence the optimization of any other row of W (identical for columns of H). This allows for a parallel implementation of the proposed method. IV. EXPERIMENTAL EVALUATION The software used for the experiments in this paper was written in Matlab. The MU implementation is based on the nnmf() function included in Matlab s Statistic Toolbox since version v6.2. For the optimization algorithms, we adapted or implemented the following Matlab codes: For PSO and DE we adapted the codes from [17] for our needs. For GA we adapted the continuous genetic algorithm available in the appix of [6]. Due to space limitations and in order to provide unbiased evaluation results we present results achieved with a randomly created, square matrix. Parameter Setup. All heuristics use a population size of ten and run for ten iterations (100 fitness evaluations). These parameters were chosen to be significantly smaller than for most other optimization problems due to runtime limitations a too expensive optimization may outweigh the performance gain of the proposed method. The dimension of the optimization problem is identical to the rank k of the NMF. The upper/lower bound of the search space was set to the interval [0, 4] and upper/lower bound of the initialization to [0, max(a)]. Algorithm specific parameters: GA: mutation rate of 0.5; selection rate of 0.65 PSO: following [4] ϕ = 4.1, and c 1 = c 2 = 2.05 DE: crossover probability (pc) set to upper limit 1 FSS: step ind initial =1, step ind final =0.001, W scale =10 FWA: number of sons (sonnum) set to

4 Hardware. Runtimes were measured on a SUN Fire X4600 M2 with eight 3.2GHz quad-core processors and 32GB RAM. We implemented all algorithms in Matlab exploiting Matlab s parallel computing potential. This allows for parallelizing the optimization process over up to 32 workers (each Matlab worker runs on one core). The runtimes presented in this section are based on this parallel implementation. the metaheuristics we applied our optimization procedure here only on the rows of W, while the columns in H remained unchanged. Experiments showed that with this setting the loss in accuracy compared to optimizing both, W and H, is relatively small while the runtime can be increased significantly. m was set to 2 which indicates that the optimization is only applied in the first two iterations (compared to m = 10 in Fig. 1). All optimization algorithms are still able to achieve a much faster reduction of the approximation error in terms of accuracy per iteration, although the values of the parameters m and c were chosen to be very small. Fig. 1. Accuracy per iteration when updating W and H; m=10, c=20, k=5. Accuracy per Iteration. Fig. 1 shows the accuracy per iteration for the basic MU algorithm and the proposed optimization method based on the different metaheuristics when optimizing both factors W and H. Results in this figure were achieved with m set to 10 and c set to 20 (Section III). The results show that the approximation error per iteration can be reduced significantly if metaheuristics are used. However, Fig. 1 does not take into account the overall runtime needed to achieve a given accuracy. Actually, the computational expenses for optimizing rows of W and columns of H using the settings mentioned above are rather large and overwhelm the benefits of the faster error reduction per iteration. Fig. 2. Accuracy per iteration when updating only W ; m=2, c=20, k=5. Fig. 2 shows similar information as Fig. 1 for different parameter setting. Due to the relatively high computational cost of Fig. 3. Proportional runtimes for achieving the same accuracy as basic MU after 15 (30) iterations. Only rows of W are updated; m=2, c=20, k=2. Accuracy per Runtime. Fig. 3 shows the reduction in runtime when the same accuracy as for basic MU after 15 (30) iterations should be achieved (for k = 2). The proposed iterative optimization strategy only needs about 60% of the runtime to achieve the same accuracy as basic MU after 15 iterations, and between 40% and 50% of the runtime to achieve the same accuracy as basic MU after 30 iterations (deping on the applied meta-heuristic). In Fig. 3, the runtime of basic MU sets the baseline (1 = 100%), the runtimes of the meta-heuristics are given as speedup over basic MU: t opt XX /t Basic MU Table I compares the number of iterations needed to achieve a given accuracy as well as the necessary runtime to achieve a given accuracy for different values of rank k and the same parameters as in Fig. 2 and Fig. 3. It can be interpreted as follows: The most-left column is the abbreviation of the optimization algorithm. The first row of any optimization algorithm shows the number of iterations needed to achieve the same accuracy as the basic version of MU for a given rank k when using the proposed iterative optimization update strategy. E.g. for rank k = 2, the iterative optimization of MU using DE as optimization algorithm needs only 4 iterations to achieve the same accuracy as the basic MU algorithm after 15 iterations (7 iterations for k = 3, 8 iterations for k = 5, and so on...). The second row of any optimization algorithm shows the overall runtime needed to achieve the same accuracy as the basic version of MU for a given rank k. For example, for rank k = 2, the iterative optimization of MU using DE as optimization algorithm needs only 0.58 (cf. Fig. 3!) of the amount of time as the basic version of MU (0.69 of the runtime of basic MU for k = 3, 0.80 for k = 5, and so on...). The best result for a given rank k is highlighted in bold letters. The third and fourth row of any optimization algorithm show 1671

5 TABLE I COMPARISON OF ITERATION PER ACCURACY AND RUNTIME PER ACCURACY FOR DIFFERENT VALUES OF RANK k. (m=2, c=20, k=5) DE k = 2 k = 3 k = 5 k = 7 k = 10 acc(15 iters) t(15) = acc(30 iters) t(30) = FSS k = 2 k = 3 k = 5 k = 7 k = 10 acc(15 iters) t(15) = acc(30 iters) t(30) = FWA k = 2 k = 3 k = 5 k = 7 k = 10 acc(15 iters) t(15) = acc(30 iters) t(30) = GA k = 2 k = 3 k = 5 k = 7 k = 10 acc(15 iters) t(15) = acc(30 iters) t(30) = PSO k = 2 k = 3 k = 5 k = 7 k = 10 acc(15 iters) t(15) = acc(30 iters) t(30) = the similar information for 30 iterations. Summarizing, the proposed optimization strategy is able to clearly decrease the overall runtime per accuracy compared to the basic version of the MU algorithm. With increasing k the reduction of runtime ts to decrease. For rank k > 15 our method is slower than the basic MU algorithm. This is caused by the higher computational cost of the optimization algorithms for larger problem dimension. When comparing the optimization algorithm, no big gap between them can be found. Generally, DE achieved slightly better results as the other metaheuristics, for small rank (k = 2 or 3), FSS and PSO achieve the best resutls. V. CONCLUSION In this paper we presented a new iterative update strategy for the MU algorithm based on nature-inspired optimization algorithms. The proposed update strategy allows for efficiently computing the optimization of single rows of W and/or single columns of H in parallel. Our results show that especially for small rank k our method achieves the same approximation error than the standard version of MU in significantly fewer iterations and in faster overall runtime. This indicates the applicability of our method as a rough and very fast approximation method. We are currently exting our study on other NMF algorithms in order to further utilize the potential of the proposed update strategy. Acknowledgments. This work was supported by National Natural Science Foundation of China (NSFC), Grant No Andreas wants to thank the Erasmus Mundus External Coop. Window, Lot 14 ( / ECW). REFERENCES [1] C. BASTOS FILHO, F. LIMA NETO, M. SOUSA, M. PONTES, AND S. MADEIRO, On the influence of the swimming operators in the fish school search algorithm, in SMC 2009: Proceedings of Systems, Man and Cybernetics., 2009, pp [2] M. W. BERRY, M. BROWNE, A. N. LANGVILLE, P. V. PAUCA, AND R. J. PLEMMONS, Algorithms and applications for approximate nonnegative matrix factorization, Computational Statistics & Data Analysis, 52 (2007), pp [3] C. BOUTSIDIS AND E. GALLOPOULOS, SVD based initialization: A head start for nonnegative matrix factorization, Pattern Recogn., 41 (2008), pp [4] D. BRATTON AND J. KENNEDY, Defining a standard for particle swarm optimization, in Swarm Intelligence Symposium, SIS IEEE, 2007, pp [5] R. CHIONG, ed., Nature-Inspired Algorithms for Optimisation, vol. 193 of Studies in Computational Intelligence, Springer, [6] R. L. HAUPT AND S. E. HAUPT, Practical Genetic Algorithms (2nd ed.):, John Wiley & Sons, Inc., [7] A. JANECEK, S. SCHULZE-GROTTHOFF, AND W. N. GANSTERER, libnmf - a library for nonnegative matrix factorizatrion, Computing and Informatics, 22 (2011). [8] A. G. JANECEK AND W. N. GANSTERER, Utilizing nonnegative matrix factorization for classification problems, in Survey of Text Mining III: Application and Theory, M. W. Berry and J. Kogan, eds., John Wiley & Sons, Inc., [9] A. G. JANECEK AND Y. TAN, Using population based algorithms for initializing nonnegative matrix factorization, in to appear in ICSI 2011: Second International Conference on Swarm Intelligence, [10] J. KENNEDY AND R. EBERHART, Particle swarm optimization, in Proceedings of IEEE International Conference on Neural Networks, vol. 4, 1995, pp [11] H. KIM AND H. PARK, Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method, SIAM J. Matrix Anal. Appl., 30 (2008), pp [12] A. N. LANGVILLE, C. D. MEYER, AND R. ALBRIGHT, Initializations for the nonnegative matrix factorization, in SIGKDD 06: 12th ACM Int. Conf. on Knowledge Discovery and Data Mining, [13] D. D. LEE AND H. S. SEUNG, Learning parts of objects by non-negative matrix factorization., Nature, 401 (1999), pp [14], Algorithms for non-negative matrix factorization, Advances in Neural Information Processing Systems, 13 (2001), pp [15] C.-J. LIN, Projected gradient methods for nonnegative matrix factorization, Neural Comput., 19 (2007), pp [16] P. PAATERO AND U. TAPPER, Positive matrix factorization: A nonnegative factor model with optimal utilization of error estimates of data values, Environmetrics, 5 (1994), pp [17] M. E. H. PEDERSEN, SwarmOps. Online: projects/swarmops/matlab, 04/2010, [18] K. V. PRICE, R. M. STORN, AND J. A. LAMPINEN, Differential Evolution A Practical Approach to Global Optimization, Springer, [19] M. N. SCHMIDT AND H. LAURBERG, Non-negative matrix factorization with Gaussian process priors, Comp. Intelligence and Neuroscience, 2008 (2008), pp [20] V. SNÁSEL, J. PLATOS, AND P. KRÖMER, Developing genetic algorithms for boolean matrix factorization, in DATESO 2008, [21] K. STADLTHANNER, D. LUTTER, F. THEIS, AND ET AL., Sparse nonnegative matrix factorization with genetic algorithms for microarray analysis, in IJCNN 2007: Proceedings of the International Joint Conference on Neural Networks, 2007, pp [22] Y. TAN AND Y. ZHU, Fireworks algorithm for optimization, in Advances in Swarm Intelligence, Y. Tan, Y. Shi, and K. Tan, eds., vol of Lecture Notes in Computer Science, Springer, 2010, pp