Combining Neural Networks Based on Dempster-Shafer Theory for Classifying Data with Imperfect Labels

Transcription

1 Combining Neural Networks Based on Dempster-Shafer Theory for Classifying Data with Imperfect Labels Mahdi Tabassian 1,2, Reza Ghaderi 1, and Reza Ebrahimpour 2,3 1 Faculty of Electrical and Computer Engineering, Babol University of Technology, Babol, Iran r_ghaderi@nit.ac.ir 2 School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM), Tehran, Iran, P.O. Box {tabasian,ebrahimpour}@ipm.ir 3 Department of Electrical and Computer Egineering, Shahid Raaee Teacher Training University, Tehran, Iran Abstract. This paper addresses the supervised learning in which the class membership of training data are subect to uncertainty. This problem is tackled in the framework of the Dempster-Shafer theory. In order to properly estimate the class labels, different types of features are extracted from the data. The initial labels of the training data are ignored and by utilizing the main classes' prototypes, each training pattern, in each of the feature spaces, is reassigned to one class or a subset of the main classes based on the level of ambiguity concerning its class label. Multilayer perceptrons neural network is used as base classifier and for a given test sample, its outputs are considered as basic belief assignment. Finally, the decisions of the base classifiers are combined using Dempster's rule of combination. Experiments with artificial and real data demonstrate that considering ambiguity in class labels can provide better results than classifiers trained with imperfect labels. Keywords: Data with imperfect labels, Dempster-Shafer theory, Classifier combination, Neural network. 1 Introduction In the classical supervised classification framework, a classifier is trained based on a learning set in the form of labeled patterns. However, in some applications, unambiguous label assignment may be difficult, imprecise and expensive. Such situations can occur when differentiating between two or more classes is not easy due to lack of information required for specifying certain labels to data or the difficulty of labeling complicated data of the problem at hand. Combination of multiple classifiers' decisions induced from different information sources has proved to be a promising approach for improving the performance of a classification system that deals with imprecise and uncertain information. The Dempster-Shafer (D-S) theory of evidence [1] is a well suited framework for representation of partial knowledge. Compared to the Bayesian approach, it provides a more flexible G. Sidorov et al. (Eds.): MICAI 2010, Part II, LNAI 6438, pp , Springer-Verlag Berlin Heidelberg 2010

2 234 M. Tabassian, R. Ghaderi, and R. Ebrahimpour mathematical tool for dealing with imperfect information. It offers various tools for combining several items of evidence and, as understood in the transferable belief model (TBM) [2], allows to make decision about the class of a given pattern through transforming a belief function into a probability function. Thanks to its flexible characteristics to represent different kind of knowledge, the D-S theory provides a suitable theoretical framework for combining classifiers especially those learned using imprecise and/or uncertain data [3-7]. In this paper we propose a new approach within the belief functions and combining classifiers frameworks for dealing with supervised classification problems in which the labels of the learning data are imperfect. Several representations of the data are used and an approach is suggested, based on the supervised information, for detecting inconsistencies in the labels of the learning data and assigning crisp and soft labels to them. Multilayer perceptrons (MLP) neural network is used as base classifier and its outputs are interpreted as belief function. Final decision about the class of a test pattern is made by combining the beliefs produced by the base classifiers using Dempster's rule of combination. The paper is organized as follows. In Section 2 basic concepts of the D-S theory are reviewed. Details of our proposed method are described in Section 3. It is followed by the experimental results on artificial and real data in Section 4 and finally, Section 5 draws conclusion and summarizes the paper. 2 Dempster-Shafer Theory The Dempster-Shafer (D-S) theory of evidence [1] is a theoretical framework for reasoning with uncertain and partial information. Several models for uncertain reasoning has been proposed based on the D-S theory. An example is the transferable belief model (TBM) proposed by Smets [2]. In this section, the main notions of the D-S theory are briefly reviewed. Let Ω = {ω 1,...,ω M} be a finite set of mutually exclusive and exhaustive hypotheses called the frame of discernment. A basic belief assignment (BBA) is a function m: 2 Ω [0,1] verifying m (A) = 1. A BBA m such that m( φ ) = 0 is called normal. The subsets A of Ω with nonzero masses are called the focal elements of m and A Ω m(a) indicates the degree of belief that is assigned to the exact set of A and not to any of its subsets. There are belief and plausibility functions associated with a BBA and are defined respectively, as follow: Bel( A) = m ( B ) B A Pl ( A ) = m ( B ) A B φ Bel(A) represents the total amount of probability that must be allocated to A, while Pl(A) can be interpreted as the maximum amount of support that could be given to A. Let m 1 and m 2 be two BBAs induced by two independent items of evidence. These pieces of evidence can be combined using Dempster s rule of combination which is defined as: (1) (2)

3 Combining Neural Networks Based on Dempster-Shafer Theory 235 m 1( A ) m 2( B ) A B = C m ( C ) = 1 m 1( A ) m 2( B ) A B = φ (3) Combining BBAs using Dempster's rule of combination is possible only if the sources of belief are not totally contradictory which means that there exist two subsets A Ω and B Ω with A B φ such that m ( A ) > 0 and m ( B ) > Discounting When an information source S is used in the belief function framework, its reliability can be taking into account by the discounting operation in which the original BBA m α is weakened by a discounting rate α [0,1]. The resulting BBA m is defined by Shafer [1] and Smets [8]: α ma ( ) = (1 α) ma ( ) A Ω, A Ω α m( Ω= ) α+ (1 α) m( Ω) (4) The coefficient (1- α) cab be regarded as a confidence degree one has in the source of information. If α = 1, it means that S is not reliable and the information provided by this source is discarded. On the other hand, α = 0 indicates that the S is fully reliable and the belief function remains unchanged. 2.2 Decision Making The approach adopted in this paper for decision making is based on the pignistic transformation which is defined in the TBM. By uniformly distributing the mass of belief m(a) among its elements for all A Ω, a pignistic probability distribution is defined as: 1 m(a) BetP(ω) =, ω Ω (5) A 1 m( φ) { A Ω,ω A } where A is the cardinality of subset A and for normal BBAs, would be ma ( ). Finally a test sample is assigned to class with the largest pignistic probability. 3 The Proposed Method Fig. 1 shows the architecture of our proposed method. There are two main phases involved in the implementation of the method including relabeling the learning data and classifying an input test sample by combining decisions of neural networks trained on the learning data with new labels.

4 236 M. Tabassian, R. Ghaderi, and R. Ebrahimpour Fig. 1. Architecture of the proposed classification scheme. In the training phase, crisp and soft labels are assigned to the learning data and MLPs are trained on the data with new labels. In the test phase, the outputs of the MLPs are converted in the form of BBAs by making use of softmax operator. Final decision on a given test sample is made by combining the experts' beliefs with Dempster's rule of combination and using pignistic probability. 3.1 Relabeling Let Ω = {ω 1,...,ω M} be a set of M classes and X = [x 1,...,x n ] be a data point described by n features in the training set which is associated to one class in Ω with certainty. The goals of this stage are (i) to detect inconsistencies in the labels of the training data using the supervised information carried by the data and, (ii) to reassign each train sample to ust one main class or any subset of the main classes based on the level of ambiguity concerning the class membership of that sample. Let P be a M n matrix containing prototype vectors of the main classes and let Di = [ di1,..., dim] be the set of distances between train sample X i and the M prototypes according to some distance measure (e.g. the Euclidian one). Initial label of X i is ignored and by utilizing the information provided by the vector D i, uncertainty detection and class reassignment for this sample are performed in a three-step procedure: Step 1: The minimum distance between X i and the class prototypes is taken from the vector D i, d = min( d ), k = 1,..., M (6) i ik and d i is called d min. Step 2: A value 0< μ k 1 is calculated for each of M classes using the following function: d min + β μk ( X i) =, k = 1,..., M (7) d ik + β in which 0 < β < 1 is a small constant value and ensures that the utilized function allocates a value greater than zero to each of M classes even if d min = 0. μ is a k

5 Combining Neural Networks Based on Dempster-Shafer Theory 237 decreasing function of the difference between d min and d k and has values close to 1 for small differences. Step 3: A threshold value 0< τ <1 is defined and based on the level of ambiguity regarding the class membership of the train sample X i, this sample may be assigned to (i) a set of classes if the corresponding values of μ for these classes are greater than k or equal to τ or (ii) ust one main class, which has the closest prototype to X i and μ = 1, if X i be far away from other main classes prototypes. k Close distances between a train pattern and some of the class prototypes can be interpreted as an indication of ambiguity in the pattern s label and in such cases a soft label is assigned to that train pattern. The above procedure is repeated for all training data and for all feature spaces. Since several representations of the data are employed to provide complementary information, it can be expected that if a soft label has been assigned to a train sample in one of the feature spaces, this sample could belong to one main class or a subset of the main classes with less uncertainty in the other feature spaces. In this way, the negative effects of crisp but imperfect labels of some learning samples on the classification performance can be reduced. 3.2 Training and Classification MLP neural network is used as base classifier. The learning set of each feature space, which consists data with new crisp or soft labels, is employed to train a base classifier. Since different types of features are extracted from the data, each feature space has its own level of uncertainty in class labels. As a result, after the relabeling procedure, the number and type of classes in each feature space could be different from the other one and base classifiers with different models (different number of output nodes) are trained on these feature spaces. In this way, diversity among the base classifiers can be achieved. In the test phase, same types of features as the training stage are extracted from the test data and they are applied to their corresponding trained base classifiers. The output values of each base classifier can be interpreted as measures of confidence associated with different decisions. For combining the evidences induced by the feature spaces using Dempster s rule of combination, the decision of each base classifier should be converted in the form of BBA. This can be done through normalizing the outputs of the base classifiers by a softmax operator as: exp( O i ) mi({ ω }) =, = 1,..., C (8) C exp( O ) = 1 i where O i is the th output value of the ith base classifier, C is the number of classes in the ith feature space after the relabeling stage and m i({ ω }) is the mass of belief given to class ω.

6 238 M. Tabassian, R. Ghaderi, and R. Ebrahimpour Note that, although our method allows to compute a BBA m i that could has any focal set over the set of the main classes, the BBA contains only a set of states which have corresponding classes in the ith feature space after the relabeling stage. Before combining the opinions of different evidences, the belief function of each classifier is weakened by its discounting rate (the method of evaluation of the discounting factor will be explained in Section 3.2.1) and the resulting BBAs are merged using Dempster s rule of combination. The decision about the class of a given test sample is then made using the pignistic probability derived from the final BBA by the pignistic transformation. Note that, although the main contribution of our method is to classify data with imperfect labels, its application can be extended to classification problems which involve heavily overlapping class distributions and nonlinear class boundaries Evaluation of the Discounting Factor To assess the reliability of information sources provided by different feature spaces, the method proposed by Eloudi et.al. [9] is used. This method is based on the TBM and for finding the discounting factor, minimizes the distance between the pignistic probabilities computed from the discounted beliefs and the actual values of data. The outline of this approach is summarized below. Let Ω= {ω 1,...,ω M} be a set of classes and χ = {X 1,...,X n} be a set of n samples. Let BBA m (X ) be the normalized belief function for sample X which is defined on the set of classes. The class of each sample is known and c Ω denotes the class of sample X. Let BetP α (X ) denotes the pignistic probability obtained from the dis- counted BBA α m ( X ). For each sample X, an indicator function δ, i 1 if c = ωi, = 0 otherwise, δ,i is defined as: and sum of the Euclidian distances between the pignistic probabilities computed from the discounted BBAs and the indicator functions, is calculated as follows: n M 2 TotalDist = ( BetP α (X )( ωi) δ, i) (10) =1 i=1 The discounting factor α that minimizes the above distance is given by: α n M ( δ - BetP(X )( ω )) BetP(X )( ω ), i i i =1 i=1 = min 1,max 0, (11) n M 2 n/m - ( BetP(X )( ωi )) =1 i=1 (9) 4 Experimental Results In this section, we report experimental results on artificial and real datasets to highlight the main aspects of our proposed method. In our experiments, the centers of the

7 Combining Neural Networks Based on Dempster-Shafer Theory 239 main classes are taken by averaging the learning data of each class and the resulting vectors are considered as the main classes prototypes. A value of β = 0.01 was adopted in the relabeling stage and validation sets were used to calculate the discounting factor. 4.1 Artificial Data We used a two-dimensional data so that the results could be easily represented and interpreted. The dataset was made of three classes with equal sample size, Gaussian distribution and common identity covariance matrix and 150, 300 and 1500 samples were generated independently for training, validation and testing sets, respectively. The center of each class was located at one of the vertices of an equilateral triangle. In order to use different representations of the data, the classes were transferred to their near vertices in clockwise direction and in this fashion two other feature spaces were generated. In each feature space, a unique subset of each class overlapped with data of other classes. It means that the high uncertainty pertaining to class membership of a pattern in one of the feature spaces can be reduced because this pattern may be located in non-overlapping or less ambiguous areas in the other feature spaces. To evaluate the performance of the proposed method in classification tasks with different levels of uncertainty in class labels, the length of the equilateral triangle (the distance between the neighborhood classes) was varied in {1,2,3} and in this way, three cases from strongly overlapping to almost separated classes were studied. Fig. 2 represents graphically the explained above procedure for generating the artificial data. Fig. 2. Generic representation of the artificial data. For generating new feature spaces, the classes are transferred to their near corners in the clockwise direction. To demonstrate how different parts of a class overlap with other classes in different feature spaces, class 1 is divided to four parts and the positions of these parts in the three feature spaces are shown Soft and Crisp Label Generation There are two main issues in the relabeling stage including choice of the main classes prototypes and selecting the threshold value (τ ). Although an appropriate selection of prototype can play an important role in accurate assignments of crisp or soft labels to data, we focus our attention on the influence of considering uncertainty in the labels of data on the performance of a classification method that confronts data with imperfect class labels or highly overlapping class distributions.

8 240 M. Tabassian, R. Ghaderi, and R. Ebrahimpour In order to examine how training samples with different levels of uncertainty in their labels are treated in the relabeling procedure, the results of partitioning the first training set with τ = 0.8 is demonstrated in Fig. 3. Each partition is represented by a convex hull and its class label indicates that the samples of that partition were assigned to which subset of the main classes. It can be seen that soft labels were assigned to samples situated in the boundaries of the main classes or those located at ambiguous regions. Fig. 3. Representation of partitioning the first learning set into crisp and soft subsets by the proposed relabeling approach with τ = 0.8. a) first feature space, b) second feature space, c) third feature space Performance Comparison The performance of our proposed method was compared to three MLP neural networks trained separately on one of the feature spaces and ensemble networks constructed by merging the decisions of the single MLPs using three fixed combining methods (Averaging, Product and Max rules). The single MLPs and the ensemble networks discarded the possible uncertainties in class labels and employed data with initial certain labels. The MLPs were trained by Levenberg-Marquardt algorithm with default parameters and 80 epochs of training and had one hidden layer. Each neural network was trained 50 times with random initializations. To evaluate the performance of our proposed method based on different threshold values, we generated 19 training sets from each original training data by varying τ from 0.05 to 0.95 with a step size of The average test error rates of the employed classifiers on the three datasets and for different number of hidden nodes are represented in Fig. 4. Note that, the classification results of our proposed method with the best τ for each dataset are shown. As can be seen, our method yields considerably better classification results than the other classifiers when the first dataset, as the most difficult set with highest level of ambiguity in class labels, was used (Fig. 4 (a)). As shown in Fig. 4(b) and Fig. 4(c), the differences between the test performances of our proposed scheme and the three ensemble networks on the second and third datasets are small. So, it can be concluded that there is not much benefit to be gained from considering uncertainty in perfectly labeled data or in a classification problem with separated classes.

9 Combining Neural Networks Based on Dempster-Shafer Theory 241 Fig. 4. Classification error rates as a function of number of hidden nodes for single MLPs, ensemble networks constructed by combining the MLPs using fixed combining method and our proposed method. a) first training set ( τ = 0.8 for our method), b) second training set ( τ = 0.75 for our method), c) third feature space ( τ = 0.7 for our method). 4.2 Real Data We applied our proposed method to the problem of classifying circular knitted fabric defects. The data consisted of five classes of knitted defect samples and defect-free fabric images (Fig. 5). Fig. 5. Samples of fabric images in our six-class problem. a) Vertical Strip, b) Horizontal Strip, c) Soil freckle, d) Crack, e) Hole, f) Defect-free. As can be seen from Fig. 5, four classes belong to either horizontal or vertical defects which means that the extracted features from samples of these classes can make overlapping areas in feature space. Moreover, little differences between samples of some defect classes and defect-free images may also cause vagueness in information provided by the learning data.

10 242 M. Tabassian, R. Ghaderi, and R. Ebrahimpour The feature vectors were computed from pixels gray level images in three stages including wavelet decomposition, binary thresholding and morphological processing. Three wavelet filters of Daubechies family (db2, db5, db10) and three levels of decomposition were used based on extensive experiments. Horizontal and vertical detailed subimages at level 3 were used for further analysis since most of the defective fabrics had either horizontal or vertical defects. Detailed subimages were then converted to binary, binary level can vary from 0 to 1 and it was chosen to be 0.2 in our work, and a morphological filter (Opening) was applied to them to eliminate irrelevant parts which could be considered as defective areas. White areas in the final subimages were considered as defects and total number of white pixels in the final horizontal and vertical images made the first and second dimensions of the feature vector, respectively. Fig.6 illustrates three two-dimensional feature spaces which were achieved using the above feature extraction approach. Fig. 6. Representations of the fabric samples in the two-dimensional feature spaces obtained by making use of three wavelet filters. a) db2, b) db5, c) db10. The dataset involved 90 samples and all classes had equal number of patterns. Overall classification performance was evaluated using a K-fold cross validation method with K = 5. The data was divided into five subsets and in each round, three of the five subsets were employed for training while the fourth and fifth were used for validation and testing, respectively. The above procedure was repeated for all five subsets and the average classification rate on the test patterns was considered as a figure of merit. Similar to experiments carried out using the artificial dataset, we compared the performance of our proposed method on the real data with those of single and ensemble neural networks that relied on the initial imperfect labels. All MLPs were trained 100 times with random initial weights and 50 epochs of training and their architectures were similar to the former experiment. Table 1 gives the average test error rates of the proposed method (with the best threshold value ( τ = 0.75)), the ensemble networks and the single MLPs as a function of number of hidden nodes.

11 Combining Neural Networks Based on Dempster-Shafer Theory 243 Table 1. Error rates as a function of number of hidden nodes for single MLPs, ensemble networks and our method ( τ = 0.75 ). The minimum error rate of each network is typed in bold. No. of hidden neurons Average error rate (%) First MLP Second MLP Third MLP Averaging Product Max Proposed method Small differences between error rates of the proposed method and the ensemble networks can be interpreted by considering the poor choice of the main classes' prototypes (center of classes) in the relabeling stage of our method. As mentioned earlier, our maor interest is to investigate the effect of considering uncertainty in class labels in improving the classification results and utilizing complementariness between information sources in the framework of the D-S theory. So, the expectation is that by employing more complex algorithm for generating prototypes of the main classes, which is an open area of research, better classification results would be achieved using the proposed method We used the real data to examine the effect of utilizing discounting strategy on the performance of the proposed method and to explore how diverse classifiers were generated from the feature spaces using our proposed relabeling approach. Error reduction rates obtained by employing the discounting factor in our method for different threshold values and the best number of hidden neurons are presented in Table 2. Overall, discounting brought better performances compared with the situation that all information sources assumed to be fully reliable. Table 2. Error reduction rates (%) achieved by employing the discounting factor in the proposed method as a function of the threshold value for the best number of hidden nodes (9 hidden nodes) Threshold value (τ) Error reduction rate (%) Table 3 lists the generated classes from each feature space after the relabeling stage for the best threshold value ( τ = 0.75 ). Here, we randomly selected one of the training sets produced by the five-fold cross validation. It can be seen that the number and type of generated soft class labels for each training set are almost different from another one which indicates that the ensemble network was made by a set of diverse classifiers. Apart from the influence of τ on the label generation procedure, form of the produced class labels is related to the number of overlapping areas and amount of uncertainty pertaining to samples of each area.

12 244 M. Tabassian, R. Ghaderi, and R. Ebrahimpour Table 3. Generated classes from the three feature space after the relabeling stage for the best threshold value ( 0.75 τ = ) Produced classes Feature Set 1 ω 1 ω 2 ω 3 ω 4 ω 5 ω 6 ω 2,4 ω 3,6 ω 1,3,6 ω 3,5,6 - Feature Set 2 ω 1 ω 2 ω 3 ω 4 ω 5 ω 6 ω 2,6 ω 3,5 ω 3,6 - - Feature Set 3 ω 1 ω 2 ω 3 ω 4 ω 5 ω 6 ω 2,4 ω 2,6 ω 3,6 ω 2,3,6 ω 1,2,3,6 5 Conclusion In this paper, a method for handling imperfect labels using belief functions has been presented. By extracting different types of features from data, the proposed method takes advantage of information redundancy and complementariness between sources. In each feature space and by making use of the proposed relabeling technique, the initial labels of the learning data are ignored and each train pattern is then reassigned to a class with crisp or soft label based on its closeness to prototypes of the main classes. MLP neural network is used as base classifier and its outputs are interpreted as BBA and in this way, partial knowledge about the class of a test pattern is encoded. The BBAs are then discounted based on the reliability of the base classifiers in identifying validation patterns and are pooled using Dempster's rule of combination. Experiments carried out on controlled simulated data and a dataset of knitted fabric defects. It was shown that by considering the ambiguity in labels of the data, our method can outperform classifiers that rely on the initial imperfect labels. References 1. Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976) 2. Smets, P., Kennes, R.: The Transferable Belief Model. Artif. Intell. 66, (1994) 3. Rogova, G.: Combining the result of several neural network classifiers. Neural Networks 7(5), (1994) 4. Denoeux, T.: A k-nearest Neighbor Classification Rule Based on Dempster-Shafer Theory. IEEE Trans. Syst., Man, Cybern. 25(3), (1995) 5. Denoeux, T.: A Neural Network Classifier Based on Dempster-Shafer Theory. IEEE Trans. Syst., Man, Cybern. A, Syst., Humans 30, (2000) 6. Basir, O., Karray, F., Zhu, H.: Connectionist-Based Dempster-Shafer Evidential Reasoning for Data Fusion. IEEE Trans. on Neural Net. 16, (2005) 7. Quost, B., Denoeux, T., Masson, M.-H.: Pairwise classifier combination using belief functions. Pattern Recognition Letters 28, (2007) 8. Smets, P.: Belif Functions: The Disunctive Rule of Combination and the Generalized Bayesian Theorem. International Journal of Approximate Reasoning 9, 1 35 (1993) 9. Elouedi, Z., Mellouli, K., Smets, P.: Assessing Sensor Reliability for Multisensor Data Fusion Within the Transferable Belief Model. IEEE Trans. Syst., Man, Cybern. B 34, (2004)