MACHINE learning techniques can, automatically,

Transcription

1 Proceedings of Internationa Joint Conference on Neura Networks, Daas, Texas, USA, August 4-9, 203 High Leve Data Cassification Based on Network Entropy Fiipe Aves Neto and Liang Zhao Abstract Traditiona data cassification is based ony on physica features of input data. They are caed ow eve cassification. Data cassification by considering not ony physica attributes but aso pattern formation is denominated high eve cassification. In this paper, we propose a new technique that performs high eve cassification by extracting information of networks constructed from the input data. Specificay, we cacuate the network entropies before and after the insertion of a data item to be cassified. Then, we cassify it as beonging to the cass which resuts in the argest increase of the entropy measures. We show that the proposed method can execute cassification tasks according to both simiarity and pattern formation of input data to reach good resuts in the experiments with artificia and rea data sets. In summary, our technique can cacuate how significant a data item is for each cass performing a new way to cassify data. I. INTRODUCTION MACHINE earning techniques can, automaticay, buid modes that are abe to organize the existing knowedge or to mimic specific human behavior by using computing representation of the data retrieved from severa domains []. Machine earning techniques are traditionay divided in two main casses: supervised earning and unsupervised earning [2]. Supervised earning aims to induce concepts from abeed sampes from known casses, that is, the earning process buids an input-output map based on the observation of training data. When the abes consist in discrete vaues, the probem is denominated cassification and, when the vaues are continuous, regression. The constructed map used to predict new data abes is caed cassifier. In unsupervised earning, aso entited custering, the main task is to group the data by some simiarity criteria. Learning process of the custering is ony based on data features, because there is no foreknowedge about the data casses [2], [3]. Artificia Neura Networks, K-Nearest Neighbors (KNN) and Support Vector Machines (SVM) are data cassification techniques broady studied nowadays. However these techniques cassify the data based on the physica features (e.g. simiarity or distance) of the input. This earning process is caed ow eve cassification [4]. Rea data items tend to have an impicit tendency to form patterns in the feature space. We denominate this property as pattern formation, which has a semantic meaning. Because of that, many others researches focus on training and cassifying data considering not ony the physica features but aso the Fiipe Aves Neto and Liang Zhao are with the Department of Computer Sciences, Institute of Mathematics and Computer Science (ICMC), University of São Pauo (USP), Av. Trabahador São-carense, 400, , São Caros, SP, Brazi (emai: fiipeneto@usp.br, zhao@icmc.usp.br). This work is supported by São Pauo State Research Foundation (FAPESP) and the Braziian Nationa Research Counci (CNPq). semantic meaning of the input data, such as co-training technique and Semantic Web, but there is not an expicit and genera scheme to dea with this approach in iterature yet. Data cassification by considering the physica features and the pattern formation is known as high eve cassification [4]. A fundamenta research area in this work is compex network. Formay, a compex network can be represented as a arge scae graph with non trivia connection pattern [5]. Here, the terms network and graph are equivaent. Many natura phenomena can be characterized by a network, ike brain structures, socia interactions and the internet. These systems can be represented by using vertices and edges indicating connections between them. In internet, for exampe, the vertices coud represent routers and the edges its physica connection [6]. Siva and Zhao [4] propose a data cassification technique where a network characterizes the pattern formation of the input data by exporing its topoogica properties. The cassification process, initiay, buids a network for each cass in order to represent each pattern formation, after that, the process verifies the accordance between the pattern formation of the network and the input data instance for cassification. The accordance is verified by the comparison of the network measurements before and after the insertion of the input in the network. Another important topic to this work is random wak which is a fundamenta dynamic process [7]. Given a network and a start point, we seect a neighbor of it at random, and move to this neighbor; then we seect a neighbor of this ast chosen node at random, and move to it, and so on. The sequence of seected points is a random wak [8]. A random wak is a finite Markov chain that is time-reversibe, in other words, every Markov chain can be viewed as random wak on a directed network with weighted edges [8]. Markov chain is the stochastic process defined by a countabe or finite number of states where the conditiona probabiity that the process is in a given state in the current time depends ony on the previous state and not on the sequence of states that preceded it [9]. The matrix that contains the vaues of these probabiities is caed stochastic matrix of the process. In this paper, we propose a high eve data cassification technique based on the extraction of dynamica and topoogica features of the networks constructed from the input data by their entropy. Network entropy is the entropy of a stochastic matrix associated with the adjacency matrix of the network [0] and is reated to the robustness of networks, i.e. their resiience to random changes [6]. Furthermore, network entropy aso characterizes the mutipicity of interna pathways and is negativey correated with the shortest av /3/$ IEEE 2405

2 erage path ength [0]. Consequenty, it captures from oca to goba features of the data, making it ideay suited for high eve cassification. The achieved resuts show that the proposed technique can not ony cassify according to the simiarity, but aso considering the pattern formation. The remainder of the paper is organized as foows. The proposed mode definition is described in section II. In section III, we provide computer simuation resuts to show the particuar characteristics of the proposed technique. Finay, section IV concudes the paper. II. MODEL DESCRIPTION In this section, we present a new cassification technique that anayzes the semantic meaning of the data by using their patterns reationships in networks. Consider a given training set denoted here as X training = {(x,y ),...,(x n,y n )}, where the first component of the ith tupe denotes the attributes of the ith data item. If there are d attributes in each data item, it is said to be a d-dimensiona data set. The second component of the ith tupe characterizes the cass abe associated to the ith data item. The goa in machine earning is to buid a mapping from x to y, this mapping is caed cassifier. The constructed cassifier can be checked by using a test set X test = {x,...,x m }, in which abes are not provided and X training X test =. Here we wi dea ony with discrete abes - y i {,...,L}, i {,...,n} -, so the task is denominated as cassification. From these definitions, the foowing subsections wi describe the cassification technique proposed in this paper. The cassification process is divided in three phases: Network Formation Phase, Entropy Cacuation Phase and Decision Phase. A. Network Formation Phase Initiay, an undirected and weighted network with a singe connected component is buid for each cass. We wi ca G the network that represents the cass {,...,L}. To construct the network G, every data item of the training set, X training, beonging to cass is represented as a vertex. There wi be a connection between two vertices if the Eucidean distance between their feature vectors is ess than a predefined threshod vaue. Moreover, the weight of the connection shoud be inversey proportiona to the distance of the vertices, i.e., the coser the vertices, the stronger must be the connection. So, the weight of the connection between two vertices, i and j, is cacuated by a ij = a ji = ( x i x j )α if if x i x j = 0 0 < x i x j ǫ 0 if x i x j > ǫ Thereby, the constructed network, represented by the adjacency matrix A = (a ij ), preserves the simiarity and the topoogy of the data. The parameter ǫ > 0 determines which vertices shoud be connected and the parameter α 0 indicates how much the distance of the vertices wi be considered, that is, if α 0, the weight of the connections () tends to be binary. It is important to note that the network of each cass needs to have a singe connected component, so the choice of the parameter ǫ must refect this condition. After that, we buid the networks G (i), {,...,L} and i {,...,u} that are based on the data items, beonging to cass, of the training set and the ith data item of the test set aso foowing Equation (), but in the cases when the extra vertex do not have any connection with other vertices, we consider G (i) = G. A these networks wi be used in the next phase of the cassification process. B. Entropy Cacuation Phase In this phase, the entropy of each constructed network is cacuated. Network entropy is the entropy of a stochastic matrix associated with the adjacency matrix A [0]. Then, firsty, we must obtain the stochastic matrix P = (p ij ) associated with these networks. Let λ denote the dominant eigenvaue of A and et (v i ) be the corresponding eading eigenvector. The stochastic matrix is defined by p ij = a ijv j λv i (2) With it at hands, we are abe to cacuate the network entropy by using the dynamica entropy of this stochastic process where p ij describes the transition rates from state i j and its stationary distribution, π = πp. The dynamica entropy, H(P), is defined as H(P) = i π i H i, where H i = j p ij ogp ij (3) Here H i is the Shannon entropy of the distribution [p i,...,p in ] and H is the weighted average over a stationary states [0]. For the next phase, we denominate H(G ) the entropy of the network that represents the cass and H(G (i) ) the entropy of the network G appended the ith item of the test set X test. C. Decision Phase Finay, we wi define the fuzzy cassifier C that decides what cass the data item beong. The hypothesis of the technique is: if the ith data item of the test set X test beongs to cass {,...,L}, the proportiona increase of entropy, (i), is maximum. This idea is expained by the property of networks to dea with random changes, so a data item which does not beong to certain cass wi not affect the respective network. Thus, the item is cassified by its importance in the cass. Mathematicay, the probabiity C(i, ) that the ith data item of the test set X test is cassified as beonging to the cass {,...,L} is given by C(i,) = (i) k L (i) k, where (i) = H(G(i) ) H(G ) (4) 2406

3 The vaues of C(i, ) ies between 0 and and C(i,k) =. k L III. SIMULATION RESULTS In this section, we present three simuations in order to iustrate the efficiency of the proposed high eve cassification method when appied in rea and artificia data sets and compare its resuts with traditiona cassification techniques. Firsty, we use a we known and broady studied database in pattern recognition iterature, the Iris Data Set [], secondariy, other database from UCI, the Seeds Data Set, and, asty, an artificia data set that represents a difficut situation for ow eve cassification Iris Setosa cass Iris Versicoour cass Iris Virginica cass Iris Setosa items from test set (a) A. Simuation by using Iris Data Set Iris is a database which contains 3 casses (Iris Setosa, Iris Versicoour, Iris Virginica) and 50 instances, where each cass refers to a kind of pant. One cass is ineary separabe from the other two, but these are not ineary separabe from each other []. In this simuation, we tested our technique in this database. We chose the parameters ǫ ying between and 2, and α = 0,,2 for Equation () in network formation phase. Figure shows our resuts of this simuation. In this case, we reaize that the smaer the vaues of ǫ, the higher are the accuracy of the cassifier. Moreover, the best resuts are achieved by using ow vaues of α reaching unti 96% of accuracy in 0-fod cross-vaidation. In this way, the proposed technique gets good resuts in this simuation. cassifier accuracy ε α = 0 α = α = 2 Fig.. An anaysis of the cassifier accuracy on different vaues of the parameters ǫ and α appied to the Iris database in 0-fod cross-vaidation. Three different vaues for α were chosen in order to test the cassifier in binary (bue ine with circes) and weighted (green ine with diamonds and red ine with crosses) networks. It is important to notice that vaues ǫ < does not create networks with a singe connected component, thus we do not use these vaues. In order to expicit the cassification detais, Figure 2 shows the proportiona increase of network entropy, (i), due to insertion of data items from test set in the networks of each cass. The parameters are set as: ǫ = and α = 0, and we used 66% of data in training set. Figure 2a shows in detais the ratio between the vaues of network entropy before and after the insertion of the Iris Setosa data item from test set in networks of each cass. In.05 Iris Setosa cass Iris Versicoour cass Iris Virginica cass Iris Versicoour items from test set Iris Setosa cass Iris Versicoour cass Iris Virginica cass (b) Iris Virginica items from test set Fig. 2. Comparison of the proportiona increase of network entropy for each cass by inserting the data items from test set. Networks that represent the casses are buit by using 66% of data and parameters ǫ = and α = 0. Each figure shows the resuts obtained by inserting data items from test set, where Figure 2a expose the Iris Setosa items, Figure 2b the Iris Versicoour items and Figure 2c the Iris Virginica items. Bue ines indicate the vaues (i) IrisSetosa, i. Green and red ines indicate, respectivey, (i) IrisV ersicoour and (i) IrisV irginica, i. this figure, we observe that these data items increased the entropy ony in the network of the same cass. It happens because the Iris Setosa cass is ineary separabe of the other two casses. Simiary, the increase of the network entropy due to Iris Versicoour data items from test set is showed in Figure 2b. Finay, Figure 2c present the same resuts by inserting the Iris Virginica data items from test set. Emphasized in this figure, we can see the data items that are miscassified as beonging to Iris Versicoour cass. (c) B. Simuation by using Seeds Data Set Seeds database contains 3 casses (Kama, Rosa, Canadian), 20 instances and 7 attributes for each data item. The casses refer to three different varieties of wheat []. In this simuation, we provide an anaysis simiar to previous 2407

4 subsection for this database. Here, we chose the parameters ǫ ying between 2.5 and 3.5, and α = 0,,2 for Equation () in network formation phase. Figure 3 shows our resuts of this simuation. In this case, the best resut was achieved by using α = 0 and ǫ = 3. reaching 9% of accuracy in 0-fod cross-vaidation. Like in the ast simuation, some traditiona cassification techniques reach simiar resuts to ours Kama cass Rosa cass Canadian cass (a) cassifier accuracy α = 0 α = α = ε Kama cass Rosa cass Canadian cass Fig. 3. An anaysis of the cassifier accuracy on different vaues of the parameters ǫ and α appied to the Seeds database in 0-fod cross-vaidation. Three different vaues for α were chosen in order to test the cassifier in binary (bue ine with circes) and weighted (green ine with diamonds and red ine with crosses) networks (b) It is important to notice that vaues ǫ < 2.5 can not create networks with a singe connected component. In order to expicit the cassification detais, Figure 4 shows the proportiona increase of network entropy, (i), due to insertion of data items from test set in the networks of each cass. The parameters are set as: ǫ = 3. and α = 0, and we used 66% of data in training set. Figure 4a shows in detais the ratio between the vaues of network entropy before and after the insertion of the Kama data item from test set in networks of each cass. Simiary, the increase of the network entropy due to Rosa data items from test set is showed in Figure 4b. The data items that were incorrecty cassified are highighted in these both figure. Finay, Figure 4c present the same resuts by inserting the Canadian data items from test set. In this figure, we observe that a Canadian data items were correcty cassified. C. Simuation by using an Artificia Data Set In this simuation, we tested our cassification method in a situation where traditiona techniques can not extract enough information to correcty cassify the data. The artificia data set is described by Figure 5. The database consists in two casses forming two grids where approximatey 4% of them are intersected. Despite the difficut of this probem, our cassification method obtained better resuts than other techniques. Figure 6 demonstrate how we each one of the techniques cassify the data with different sizes of the training set. The traditiona cassification techniques used here are Mutiayer Perceptron (MLP) and Support Vector Machine (SVM) Kama cass Rosa cass Canadian cass Fig. 4. Comparison of the proportiona increase of network entropy for each cass by inserting the data items from test set. Networks that represent the casses are buit by using 66% of data and parameters ǫ = 3. and α = 0. Each figure shows the resuts obtained by inserting data items from test set, where Figure 4a expose the Kama items, Figure 4b the Rosa items and Figure 4c the Canadian items. Bue ines indicate the vaues (i) Kama, i. Green and red ines indicate, respectivey, (i) Rosa and (i) Canadian, i. The MLP cassifier obtained accuracy between 96.3% and 97.5% whie SVM obtained cassification rate between 90.4% and 96.3%. It is worth mentioning that these vaues of accuracy are not good because more than 92% of data in this database coud be considered easy to be cassified. Our technique, in contrast, coud cassify correcty more than 98% of data in test set by using ony 40% of a data in training set and 00% of accuracy with 90% of data in training set. Besides, the bigger are the training set size, the better are the resuts of the appication of our technique whist it does not happen in the other techniques appications. (c) IV. CONCLUSIONS Here, we propose a high eve data cassification technique based on network entropy. This method cassifies the data 2408

5 Fig. 5. Representation of the artificia database constructed to evauate our technique. The database has two bi-dimensiona casses that intersect themseves. The casses has a simiar pattern formation, in fact, both casses are grids with distance between two adjacent data item, but the red cass data items are shifted in reation to bue cass data items. items by their significance in the network that represent the casses. As a resut, the technique can cassify according the physica and semantic features of the input data. The choice of the parameters, ǫ and α, in the network formation phase is crucia to the cassifier achieves good accuracy. An interesting fact in this work is that the bigger the vaues of the parameters, the bigger is the amount of information you obtain from data, but not aways a huge amount of information obtain better resuts. For exampe, the seection of the parameters in the simuation by using Iris database was α = 0 and ǫ = to focus at distance of the data, i.e., the resuting network is binary with few connections, and this approach achieved better resuts than other. Furthermore, we expose a situation (subsection III-C) which provides evidences that increasing the size of training set in traditiona data cassification techniques can not improve accuracy or induce errors when the simiarity is not enough to cassify the data since these techniques consider ony oca aspects. This probem can be overcome when the proposed technique is used. Nevertheess, there are some imitations in this technique. Firsty, the agorithm is not scaabe to arge data sets because the computationa costs of constructing networks and cacuating entropies are very expensive. Secondy, imbaanced data set may have unexpected effects on this method. Lasty, the performance of the technique is very sensitive to the parameters. Concuding, this work is on track to improve current resuts in rea data cassification appications presenting innovation for machine earning theory. cassifier accuracy 4 2 Network Entropy SVM MLP training set size Fig. 6. A detaied study of our technique performance compared to other traditiona techniques appied in the artificia database described by Figure 5 varying the size of the training set used. The bue ine with circes indicates the resuts of our technique appication with parameters ǫ = and α =. The red ine with crosses shows the performance reached by a Mutiayer Perceptron cassifier with 4 neuron in output ayer and earning rate 0.3. The green ine with diamonds depicts the resuts achieved by a Support Vector Machine cassifier with a radia basis function as kerne. REFERENCES [] C.M. Bishop, Pattern recognition and machine earning. Springer, [2] T.M. Mitche, Machine earning. McGraw-Hi Series in Computer Science. McGraw-Hi, 997. [3] S.J. Russe and P. Norvig, Artificia Inteigence: A Modern Approach. pp , Prentice Ha, 3 edition, 200. [4] T.C. Siva and L. Zhao, Network-Based High Leve Data Cassification. IEEE Transactions on Neura Networks and Learning Systems, v. 23, pp , 202. [5] S. Boccaetti, V. Latora, Y. Moreno, M. Chavez and D.U. Hwang, Compex networks: Structure and dynamics. Physics Reports, vo. 424, pp , [6] L.F. Costa, F.A. Rodrigues, G. Travieso and P.R. Vias, Characterization of compex networks: A survey of measurements. Advances in Physics, vo. 56, pp , [7] J.D. Noh and H. Rieger, Random Waks on Compex Networks. Physica Review Letters, vo. 92, pp. 870, [8] L. Lovász, Random Waks on Graphs: a Survey. Yae University, 994. [9] S.M. Ross, Introduction to probabiity modes. Academic Press, 0 edition, [0] L. Demetrius and T. Manke, Robustness and network evoution - an entropic principe. Physica A, vo. 346, pp , [] A. Frank and A. Asuncion. UCI Machine Learning Repository [ Irvine, CA: University of Caifornia, Schoo of Information and Computer Science