A label distance maximum-based classifier for multi-label learning

Transcription

1 Bio-Medical Materials and Engineering 26 (2015) S1969 S1976 DOI /BME IOS ress S1969 A label distance maximum-based classifier for multi-label learning Xiaoli Liu a,b, Hang Bao a, Dazhe Zhao a,b and eng Cao a,b,* a Medical Image Comuting Laboratory of Ministry of Education, Northeastern University, , Shenyang, China b College of Information Science and Engineering, Northeastern University, , Shenyang, China Abstract. Multi-label classification is useful in many bioinformatics tasks such as gene function rediction and rotein site localization. This aer resents an imroved neural network algorithm, Max Label Distance Back roagation Algorithm for Multi-Label Classification. The method was formulated by modifying the total error function of the standard B by adding a enalty term, which was realized by maximizing the distance between the ositive and negative labels. Extensive exeriments were conducted to comare this method against state-of-the-art multi-label methods on three oular bioinformatic benchmark datasets. The results illustrated that this roosed method is more effective for bioinformatic multilabel classification comared to commonly used techniques. Keywords: Multi-label classification, neural networks, max label distance, self-adative learning rate 1. Introduction Bioinformatics analysis oses classification challenges because one gene can be associated with several functional classes [1]. This is a tyical multi-label classification roblem. Multi-label classification is concerned with learning from a set of instances that is associated with a set of labels [2]. Traditional single-label classification generally assigns instances to a single category, where each instance is associated with a single label. Multi-label classification is a comlex and challenging task in machine learning [3]. Many researchers have roosed various methods for multi-label learning [2, 4-10], but their effectiveness has not been satisfactory for bioinformatics classification. This aer resents an imroved neural network algorithm, Max Label Distance Back roagation Algorithm (named MaxLDB) for Multi-Label Classification. The method was formulated by modifying the total objective function of the standard Back roagation (B) by adding a enalty term, which was realized by maximizing the distance between the ositive and negative labels. Hence, the total loss of the roosed method was comrised by standard error and the maximum label distance. Also, an adative learning rate was used to imrove efficiency. The rest of the aer is organized as follows: Section 2 reviews a standard back roagation neural network. Section 3 details the method. Section 4 resents exerimental results and comarisons. * Address for Corresonding: eng Cao, College of Information Science and Engineering, Northeastern University, , Shenyang, China. Tel.: (86 24) ; Fax: (86 24) ; caoeng@ise.neu.edu.cn /15/$ IOS ress and the authors. This article is ublished with Oen Access and distributed under the terms of the Creative Commons Attribution and Non-Commercial License.

2 S1970 X. Liu et al. / A label distance maximum-based classifier for multi-label learning Finally, conclusions are summarized in Section Standard back roagation neural network Consider a three-layered B of which the l-th layer contains N l units, l=1,,m. The outut of the j- th unit at the l-th layer is: y j l = f net l l ( j +θ j ) l whereθ jl is the bias for the j-th unit, and ( ) activation function for the j-th unit: f net is the activation function. net j is the inut of the j (1) net j l N l 1 = w ij i=1 l 1,l y i l 1 (2) where w ij l-1,l is the weight from the i-th unit at the (l-1)-th layer to the j-th unit at the the l-th layer. E s denotes the training error of the Standard B, as follows: N M ( ) 2 E S = 1 y 2 j, t j, (3) =1 j=1 where is a attern over inut-outut airs and is the amount of the training data. The y j, and t j, are the network and the target outut vectors at the j-th outut layer unit for the attern. 3. roosed method 3.1. Error function Many activation functions can be used in B. The tangent activation function f ( x)= ( e x e x ) ( e x + e x ) was used due to its fast convergence. So, the label distance maximum error L function E is defined as follows: E L N = ex 1 2 min ( y M ( j,k ) max( y j,l )) (4) =1,k Y,k Y j=1 where Y denotes the labels for x. Hence, the total error of the roosed method is comrised by the standard error, which is introduced in Section 2 and the label distance maximum error:

3 X. Liu et al. / A label distance maximum-based classifier for multi-label learning S1971 ( ) E = E S + λe L N = E S L M + λe = =1 j=1 =1 N M j=1 1 2 y t j j ( ) 2 + λ ex 1 ( 2 min ( y j,k ) max ( y j,l )) (5) where λ is the regularization arameter that controls the tradeoff between the training error and label distance maximum error. This can be written as: E = E S + λ E L = E S + λ E L net w ij w ij w j ij net j net (6) j w ij and by Eq. (2), net j Eq. (7) is drawn: S w ij = y i is obtained; when δ j = ( E net j + E L net j ) is defined by Eq. (1), δ j = E S + λ E L y j y j y = E S + λ E L j net j y j y f net j +θ j j ( ) (7) considering E S, we get E S y j = y j t j. Considering E L, we get: 1 1 ( ( y, ) ( y, )) ex ( min ( ) max ( )) 1 L ex min jk max y y j Y jl E = = yj yj 1 1 ex ( min ( y ) max ( y )) j Y 2 2 j l k j (8) since f ( net j +θ j )= ( 1+ c j )( 1 c j ), we get: δ j = ( t j y j )+ 1 2 λ ex 1 2 min ( y j ) max y l ( ( )) ( t j y j ) 1 2 λ ex 1 2 min ( y k ) max y j ( ( )) 1+ c j ( )( 1 c j ) j Y 1+ c j ( )( 1 c j ) j Y (9) The delta rule used in the standard B was used to udate the weights. According to the gradient descent strategy, the weight and bias are changed as follows: E Δ wij = η = ηδ j yi, Δ θ j = ηδ j w ij (10) where is the learning rate, introduced in the next section Learning rate

4 S1972 X. Liu et al. / A label distance maximum-based classifier for multi-label learning Vogl, et al. roosed an adative learning-rate back-roagation with adative momentum (AB) [11]. He modified the weight after each eoch over all the atterns as follows: Δw ij ( K +1)= η δ j y i + mc Δw ij ( K) (11) where K reresents the iteration number rather than the resentation number and mc is the momentum factor. The learning rate is determined by whether the total error of all atterns decreases the erformance after one iteration or not. If an udate successively reduces the total error, is increased by multilying a factor a>1 for the next iteration. If the total error exceeds the revious value by a certain ercentage, is decreased by multilying a factor b<1. The learning rate is constrained between 0.05 and 0.3. A modified adative learning rate method was used based on AB, considering the learning rate but not the momentum. The net was udated when the total error reduced successively. The method can be denoted by the following equation: Algorithm 1 Max label distance back roagation algorithm (MaxLDB) for multi-label classification Algorithm MaxLDB Initialization weight matrix W,, total error E,, eoch,a, b,, while E > & K < eochs do do K = K+1; E = 0; for = 1 to do for the j-th unit at the l-th layer do Δw ij = η E = ηδ j y ; i w ij w ij = w ij + Δw ij ; E = E S + λe L ; if E(K +1) < E(K),η(K +1) = aη(k),net = net(k +1) ; else if E(K +1) ϕe(k),η(k +1) = bη(k) ; else if E(K +1) = E(K),η(K +1) = bη(k) ; η( K +1)= aη ( K), net = net K +1 ( ) E( K+1)< E( K) ( ) E( K+1) ϕe( K) ( ) E( K+1)= E( K) ( ) otherwise bη K bη K η K (12) where is the ratio between the udate total error and the revious value. The arameters used in AB and the roosed method are a=1.05, b=0.7, and =1.04. starts with Formally, Algorithm 1 describes the roosed method. The overall comutational cost of the roosed algorithm is O(M E), where M is the total amount of the architectural arameters (weights and biases) of the network, is the amount of training instances, and E is the total amount of training eochs.

5 X. Liu et al. / A label distance maximum-based classifier for multi-label learning S1973 Table 1 The roerty of the training and test datasets (Cardinality is the average number of labels er instance) Datasets Attributes Classes Training Test Cardinality Yeast Human lant Exeriments This section rovides an emirical evaluation of the roosed method derived from exerimental analysis on three oular bioinformatics benchmark datasets. Table 1 resents the main roerty of the training and test datasets emloyed in the exeriments. Dataset Yeast1 was samled from biological data; it redicted Yeast Saccharomyces cerevisiae on gene function [12]. Each gene was comrised with multi-label functions, so it could be used as a multi-label dataset. Datasets Human and lant2 redicted the subcellular locations of roteins according to their sequence. It has been observed that some multilex roteins can be assigned to multile locations sites simultaneously. MaxLDB was emirically assessed against the state-of-the-art methods for multi-label classification, such as SB (Standard B) [13], AB (Adative Learning-rate B with Adative Momentum) [11], the algorithm level for multi-label learning: BMLL (Backroagation for Multi- Label Learning) [2], and the strategy level for multi-label learning: BR [4], RakEL [7], CC (Classifier Chain) [14], and ECC (Ensemble of Classifier Chain) [14]. For SB, AB, and MaxLDB, the number of units in the hidden layer was fixed at 10; the number of training eochs was set at For BMLL, the arameters defined in [2] were used, which were set to be 20% of the number of inut units and 200, resectively. For the strategy level algorithms, a neural network with the same configuration was chosen as the base classifier, and was imlemented by a MULAN software ackage [15]. In addition, a L2 regularization term of all network weights was added to the global error function for each comarable method to avoid overfitting. Tables 2-4 shows the results of the roosed method and other multi-label classification algorithms on the Yeast, Human, and lant datasets, where the values in bold indicate the best result obtained by the corresonding method. Table 5 shows the number of wins, losses, and ties for MaxLDB comared to the other methods across 3 datasets and six evaluation metrics, thereby comosing 18 cometitions. MaxLDB achieved the best overall erformance for 3 datasets in six evaluation metrics. The roosed method beat the other methods on all datasets in One-Error, Hamming Loss, Table 2 Exerimental results of each multi-label classification algorithms on the Yeast dataset (the evaluation metrics can be referenced in [3]) Methods One- Ranking Average Hamming F1 AUC Error Loss recision Loss SB AB BMLL BR CC ECC RakEL MaxLDB

6 S1974 X. Liu et al. / A label distance maximum-based classifier for multi-label learning Table 3 Exerimental results of each multi-label classification algorithms on the Human dataset Methods One-Error Ranking Average Hamming F1 AUC Loss recision Loss SB AB BMLL BR CC ECC RakEL MaxLDB Table 4 Exerimental results of each multi-label classification algorithms on the lant dataset Methods One- Ranking Average recision Hamming Loss F1 AUC Error Loss SB AB BMLL BR CC ECC RakEL MaxLDB Table 5 MaxLDB comared to the rest of the algorithms on three bioinformatics datasets in six different metrics Algorithm wins losses Algorithm win losses s MaxLDB vs. SB 17 1 MaxLDB vs. CC 16 2 MaxLDB vs. AB 15 3 MaxLDB vs. ECC 13 5 MaxLDB vs MaxLDB vs BMLL RakEL MaxLDB vs RakEL and AUC. MaxLDB articularly erformed better than other algorithms on the lant dataset in all metrics excet for F1. The roosed method outerformed the other algorithms in One-Error, Average recision, Hamming Loss, and AUC, but B_MLL erformed better in Ranking Loss and F1. Moreover, the AB obtained comarable results, although it did not account for the mechanization of multi-label learning. By contrast, BMLL only erformed well on the Yeast dataset with caturing the characteristics of multi-label learning. The true and false ositive rates of SB, AB, BMLL, and MaxLDB were calculated to illustrate the erformance of the various training algorithms based on B methods. This was achieved by building a label confusion matrix for each label of each examle. Figure 1 shows the ROC dataset curves (Yeast, Human and lant) to visualize the erformance over all instances. For the Yeast dataset, MaxLDB exhibited similar behavior to AB and BMLL, but erformed better than SB in the AUC measurement. On the Human dataset, the roosed method erformed

7 X. Liu et al. / A label distance maximum-based classifier for multi-label learning S1975 (a) (b) (c) Fig. 1. The true and false ositive rate (ROC) curves on the datasets. (a) Yeast; (b) Human; (c) lant. (a) (b) (c) Fig. 2. Convergence behavior on the datasets. (a) Yeast; (b) Human; (c) lant. similarly with SB and AB. The BMLL algorithm was the worst method. For the lant dataset, the roosed method outerformed the other methods; the comlex correlation among labels was difficult to cature due to the limited training set size. Moreover, the issue of high dimensionality comlicated otimization. The roosed method beat the other methods on the lant dataset because it exlored label correlations by maximizing the distance between the ositive and negative labels. Figure 2 illustrates how the global training error changed as the number of training eochs increased. MaxLDB achieved the otimal solution with fewer iteration numbers comared with SB and AB. The convergence behavior demonstrates that the roosed method can imrove multi-label data learning network efficiency. 5. Conclusions This aer rooses an imroved neural network algorithm, Max Label Distance Back roagation Algorithm for Multi-Label Classification. This method was formulated by modifying the total error function of the standard B by adding a enalty term, which was realized by maximizing the distance between the ositive and negative labels. It controlled the magnitude of the weights and imroved the network s generalization erformance. The method was comared against state-of-the-art multi-label classification methods through extensive exeriments; the results illustrated that the roosed method was effective for multi-label classification in bioinformatics, and obtained cometitive or better erformance comared with five tyical multi-label learning algorithms.

8 S1976 X. Liu et al. / A label distance maximum-based classifier for multi-label learning Acknowledgment This research was suorted by the National Key Technology Research and Develoment rogram of the Ministry of Science and Technology of China under grant 2014BAI17B01, the Fundamental Research Funds for the Central Universities under Grant N as well as N , and the ostdoctoral Science Foundation of China 2015M References [1] A. Clare, Machine learning and data mining for yeast functional genomics, h.d. Dissertation, University of Wales Aberystwyth, [2] M.L. Zhang and Z.H. Zhou, Multi-label neural networks with alications to functional genomics and text categorization, IEEE Transactions on Knowledge and Data Engineering 18 (2006), [3] M.L. Zhang and Z.H. Zhou, A review on multi-label learning algorithms, IEEE Transactions on Knowledge and Data Engineering 26 (2014), [4] M.R. Boutell, J. Luo, X. Shen and C.M. Brown, Learning multi-label scene classification, attern Recognition 37 (2004), [5] E. Hüllermeier, J. Fürnkranz, W. Cheng and K. Brinker, Label ranking by learning airwise references, Artificial Intelligence 172 (2008), [6] J. Fürnkranz, E. Hüllermeier, E.L. Mencía and K. Brinker, Multilabel classification via calibrated label ranking, Machine Learning 73 (2008), [7] G. Tsoumakas, I. Katakis and I. Vlahavas, Random k-label sets for multi-label classification, IEEE Transactions on Knowledge and Data Engineering 23 (2011), [8] A. Clare and R.D. King, Knowledge discovery in multi-label henotye data, rocessing of the 5 th Euroean Conference on rinciles of Data Mining and Knowledge Discovery 2168 (2001), [9] B.E. Schaire and Y. Singer, Boostexter: A boosting-based system for text categorization, Machine Learning 39 (2000), [10] F.D. Comité, R. Gilleron and M. Tommasi, Learning multi-label alternating decision trees from texts and data, roceedings of the 3 rd International Conference on Machine Learning and Data Mining in attern Recognition 2734 (2003), [11] T.. Vogl, J.K. Mangis, J.K. Rigler, W.T. Zink and D.L. Alkon, Accelerating the convergence of the backroagation method, Biological Cybernetics 59 (1988), [12] A. Elisseeff and J. Weston, A kernel method for multi-labelled classification, in: Advances in Neural Information rocessing Systems, MIT ress Cambridge, MA, USA, 2011, [13] D.E. Rumelhart, G.E. Hinton and R.J. Williams, Learning internal reresentations by error roagation, in: arallel Distributed rocessing: Exlorations in the Microstructure of Cognition, MIT ress Cambridge, MA, USA, 1985, [14] J. Reed, B. fahringer and G. Holmes, Classifier chain for multi-label classification, Machine Learning 85 (2011), [15] G. Tosumakas, E. Syromitros-Xioufis, J. Vilcek and I. Vlahavas, MULAN: A java library for multi-label learning, The Journal of Machine Learning Research 12 (2011),