Bayesian Network Structure Learning from Attribute Uncertain Data

Transcription

1 Bayesia Network Structure Learig from Attribute Ucertai Data Wetig Sog 1,2, Jeffrey Xu Yu 3, Hog Cheg 3, Hogya Liu 4, Ju He 1,2,*, ad Xiaoyog Du 1,2 1 Key Labs of Data Egieerig ad Kowledge Egieerig, Miistry of Educatio, Chia 2 School of Iformatio, Remi Uiversity of Chia {sogwt,heju,duyog}@ruc.edu.c 3 The Chiese Uiversity of Hog Kog {yu,hcheg}@se.cuhk.edu.hk 4 Departmet of Maage. Sci. & Eg., Tsighua Uiversity hyliu@tsighua.edu.c Abstract. I recet years there has bee a growig iterest i Bayesia Network learig from ucertai data. While may researchers focus o Bayesia Network learig from data with tuple ucertaity, Bayesia Network structure learig from data with attribute ucertaity gets little attetio. I this paper we make a clear defiitio of attribute ucertai data ad Bayesia Network Learig problem from such data. We propose a structure learig method amed DTAU based o iformatio theory. The algorithm assumes that the structure of a Bayesia etwork is a tree. It avoids eumeratig all possible worlds. The depedecy tree is computed with polyomial time complexity. We coduct experimets to demostrate the effectiveess ad efficiecy of our method. The experimets show the clusterig results o ucertai dataset by our depedecy tree are acceptable. Keywords: ucertaity, Bayesia Network structure, depedecy. 1 Itroductio I the past, researchers i data miig ad machie learig area usually assume data is certai or precise, however that is ot always the case. Data ucertaity arises i may applicatios such as sesor etwork ad user privacy protectio. Ucertai data ca be divided ito two categories by ucertaity source: artificial ucertai data ad iheret ucertai data. People sometimes add oise to data for some purpose such as user privacy protectio. As a result, the data become artificially ucertai. There are also iheretly ucertai data. For example the scietific measuremet techiques ad tools are iheretly imprecise ad they are resposible for the geeratio of iheret ucertai data. May researchers focus o ucertai data maagemet ad miig i database area ad data miig area i recet years. I database area, there are three * Correspodig author. H. Gao et al. (Eds.): WAIM 2012, LNCS 7418, pp , Spriger-Verlag Berli Heidelberg 2012

2 Bayesia Network Structure Learig from Attribute Ucertai Data 315 models of ucertai data. The first oe is tuple ucertaity model [1] [2]. Each tuple i a probabilistic database is associated with a probability which represets the likelihood the tuple exists i the relatio. The secod oe is attribute ucertaity model [3]. I attribute ucertaity model, each attribute i a tuple is subject to a idepedet probability distributio. Correlated ucertaity model [5] is the third oe. Attributes are described by a joit probability distributio. May data miig algorithms have bee proposed to aalyze ucertai data, for example, miig frequet patters from ucertai trasactio database [2], aïve Bayesia classifiers for correlated ucertai data [7] [8], ad clusterig ucertai objects [6]. However there are few works o data miig from attribute ucertai data, ad to the best of our kowledge o work has focused o how to lear Bayesia Network (BN) structure from such data. Attribute idepedecy is a commo assumptio i database ad data miig area, but it is ot always reasoable, because there are depedecy relatioships amog attributes. Attribute ucertaity due to measuremet error or iheret ucertaity should t be the reaso for idepedece assumptio. The structure learig from attribute ucertai data ca reveal the essetial relatioship betwee attributes. I this paper, we propose the problem of BN structure learig from attribute ucertai data ad a algorithm amed DTAU to solve the problem. Experimets demostrate the effectiveess of our proposed algorithm. The rest of the paper is orgaized as follows. Sectio 2 discusses related work o Bayesia Network structure learig for ucertai data. Sectio 3 gives relevat defiitios. Sectio 4 itroduces our structure learig algorithm DTAU. Sectio 5 is experimetal study ad Sectio 6 cocludes the paper. 2 Related Work Bayesia Network (BN) is a powerful tool to represet joit probability distributio over a set of variables or attributes. A BN is made up of two compoets: a directed acyclic graph (DAG), whose odes represet variables ad a set of coditioal probability tables (CPTs) which specifies the coditioal distributio of each variable give its paret i the DAG. Give a BN structure (DAG), there have bee may algorithms for parameter learig. However, sometimes the BN structures are ukow for lack of domai kowledge. Thus the BN learig problem is of great importace. It has bee proved that BN structure learig problem is NP completed [9]. May researchers have proposed approximatio algorithms to solve the structure learig from certai data problem. These methods are divided ito two categories. The methods i the first category are based o iformatio theory which is used to measure depedecy relatioships betwee odes [10]. The methods [11] [12] i the secod category aim to maximize score fuctio of the possible structure cosiderig that each ode has o more tha K parets. As this problem is NP-hard whe K is bigger tha 1, heuristic rules based methods are usually used. For the attribute ucertai data, we make use of the iformatio theory to solve the problem ad assume the structure is a tree.

3 316 W. Sog et al. 3 Problem Defiitio I this sectio we describe some cocepts about the problem of learig BN structure from attribute ucertai data. The term observatio is a cocept i BN learig from certai data problem. Defiitio 1 (Attribute). A attribute X i is a compoet or aspect of a object O. X i ca take ay value i D(X i ) which is the possible value domai of X i. D i represets the size of D(X i ). The attribute X i is represeted by a ode (a radom variable) i the BN structure. Defiitio 2 (ucertai example). A vector ue = {P 1 (X 1 ), P 2 (X 2 ),, P m (X m )} is a ucertai example if each P(X i ) is a probability distributio or probability desity fuctio over D(X i ). Defiitio 3 (Ucertai observatio). Give a attribute X i, a observatio P i (X i ) of X i (1 i m) i a ucertai example ue is a ucertai observatio. Defiitio 4 (Attribute Ucertai traiig dataset). A ucertai traiig dataset D is composed of ucertai examples, D = {ue 1, ue 2,, ue }. I this paper we focus o the problem of learig a BN structure from attribute ucertai traiig dataset. We assume the structure of Bayesia Network is a tree. I the followig parts of the paper we study the problem of learig structure from discrete attribute ucertai data. If they is cotiuous, we ca discrete them. 4 Bayesia Network Structure Learig Algorithm DTAU I this sectio, we start with a brief itroductio of a aïve BN structure learig algorithm based o the expoetial possible worlds. The we will explai why the aïve method is uacceptable. At last we will show our approximatio method which takes polyomial time. Defiitio 5 (Possible world). Give a ucertai dataset about m attributes, it geerates possible worlds, where each world is a certai dataset about the m attributes which has the same size of examples with the ucertai oe. Each possible world W i is associated with a probability Pr(W i ) that the world exists. The aïve method is based o possible world over all attributes (PWAA). The idea is covertig the attribute ucertai dataset to some certai datasets. Give a ucertai discrete dataset AUD with N examples ad m attributes, first we compute every possible world W i of AUD. Each observatios of example e ij i possible world W i is a possible value i the domai of the correspodig attribute. Those possible worlds form a set W ad its size is П m i=1(d i ) N. Secod, we treat each possible world W i as a certai traiig dataset, ad the we lear a depedecy tree uder the correspodig traiig data set. The tree with the highest score ca be recogized as the right oe. The total umber of trees (obviously cotaiig the duplicates) is the umber of the possible worlds. The score of a tree T i is Σ Wj.tree=Ti Pr(W j ).

4 Bayesia Network Structure Learig from Attribute Ucertai Data 317 The umber of possible worlds is expoetial, so the solutio preseted above costs expoetial time complexity to costruct the depedecy tree. We propose a algorithm amed DTAU (Depedecy Tree learig from Attribute Ucertai data) to costruct a depedecy tree without eumeratio of possible worlds ad reduce the eormous computatio. Our idea is to make use of the attribute ucertai dataset directly. For a traditioal certai traiig dataset, a popular way to costruct the depedecy tree with the closest probability approximatio is called Chow-Liu tree [10]. The kerel idea i [10] is how to compute the depedecy betwee each two attributes. The depedecy betwee odes X i ad X j is measured by mutual etropy I(X i, X j ). The computatio is defied by Equatio 1. The value of mutual etropy I(X i, X j ) is always positive or zero. The value is more close to zero, the the depedecy betwee the two attribute is weaker. The zero value meas they are idepedet. We propose the DTAU algorithm to lear a depedecy tree from attribute ucertai data. The DTAU algorithm is cosistet with Chow-Liu tree uder certai traiig data. The key poit i the DTAU algorithm is how to compute the depedecy betwee each two attributes uder attribute ucertai traiig data. Equatio 2 shows a iitial approximatio of I(X i, X j ). The two equatios are from [10]. PX ( = x, X = x) I( X, X ) = P( X = x, X = x )log ( ) ( ) i i j j i j i i j j xi, x PX j i = xi PX j = xj (1) I ( X, X ) N ( X = s, X = t ) lo g i j i j s D ( X i ), t D ( X j ) i N ( X i = s, X j = t ) N ( X = s ) N ( X j = t ) Equatio 3, 4 ad 5 shows how to approximate the frequecy i equatio 2 ad we get the fial approximatio of I(X i, X j ) by equatio 3, 4 ad 5. N ( X = s, X = t ) = P ( X = s ) P ( X = t ) i j k i i k j j k = 1 = = = i k i i k = 1 = = = j k j j k = 1 N ( X s ) P ( X s ) N ( X t ) P ( X t ) From the equatios above we ca lear that if the probability P ki (X i =s) is the highest for attribute X i ad the probability P kj (X j = t) is the highest for attribute X j, the the occurrece probability for pair (s, t) may be the highest. The idea behid the equatio is that the idepedece assumptio does t have effect o the overall depedecy computatio. I other words, if the two attributes are idepedet, the computatio result is zero. If they are ot, the computatio result is positive. The equatio shows the cosistece with certai data. We prove the result ad we do t describe the details of the proof for the limitatio of space i this paper. The DTAU algorithm is divided ito three steps. The first step is to compute the depedecy betwee each two ucertai attributes ad costruct a weighted udirected graph. The we follow the tree costructio method i the Chow-Liu tree algorithm. The secod step is to get a maximum spaig tree by a greedy algorithm (2) (3) (4) (5)

5 318 W. Sog et al. which is 2-ratio approximatio of the optimal tree. The last step is to add the directio by width first traverse. The pseudo-code is give below: Algorithm DTAU Iput: a attribute ucertai dataset AUD with examples ad m variables Output: a depedecy tree T Major steps: Costruct a weighted udirected completed graph G = (V,E ),V ={X1,X2,...,Xm}; for i =1 to m for j =1 to i Compute depedecy I(X i,x j ) by equatio 1,2,3,4 ad 5; e(x i,x j ) = I(X i,x j ) ed for ed for T = greedy_max_spaig_tree(g ); Select a radom ode X k i T as begiig ode; Add directio for each edge by breadth-first traverse begiig with X k ; The time complexity of DTAU algorithm is O(m 2 ), which is smaller tha the aïve solutio. We ca cofirm that if the differece betwee I(X i, X j ) ad I(X i, X k ) satisfies a t-coditio that the absolute value of the differece betwee I(X i, X k ) ad I(X i, X k ) is bigger tha t, the depedecy tree created by the DTAU method is the same with the oe i the possible world with the highest probability. Because if the t-coditio is satisfied, the partial orders for all mutual etropy are the same. The partial orders ca determie the structure. We prove this result ad we do t describe the details of the proof or the computatio of t for the limitatio of space. Experimets show that our method performs well eve whe the t-coditio ca t be satisfied. 5 Experimets As there has t bee ay public attribute ucertai dataset, the attribute ucertai datasets we use are geerated from certai datasets artificially. We geerate the attribute ucertai datasets by addig oise to the UCI machie learig datasets which are stadard for traditioal BN learig problems. We covert the Letter recogitio ad Balace datasets to attribute ucertai datasets. The oise additio strategy is described as follows. For each traiig example i the origial certai dataset, we assig a probability p which is ot smaller tha a boud α to the correspodig attribute s observatio i the origial certai traiig example, ad assig a low probability to other possible values for this attribute. Table 1 shows a example of the origial certai dataset, where D(Attribute 1) = {a, b, c} ad D(Attribute 2) = {d, e, f}. Table 2 shows a ucertai dataset obtaied after oise additio with α beig 0.5. By this way we get attribute ucertai traiig datasets deoted by AU-Letter-α, ad AU-Balace-α respectively.

6 Bayesia Network Structure Learig from Attribute Ucertai Data 319 Table 1. A example of certai data Table 2. A example of oise additio Attribute 1 Attribute 2 Attribute 1 Attribute 2 a d a: 0.8 b: 0.1 c: 0.1 d: 0.5 e: 0.25 f: 0.25 b e a: 0.2 b: 0.6 c: 0.2 d: 0.15 e: 0.7 f: 0.15 For each ucertai dataset we geerate, the ucertai observatios i a attribute ucertai dataset are closer to the oes of the origial certai dataset whe α is closer to 1. The experimet o AU-Letter-0.5 shows that the partial orders of the depedecy measure (iformatio etropy) i attribute ucertai data is almost the same with the oe i the certai dataset. Figure 1 shows the tree we lear from AU- Letter-0.5 ad the Chow-Liu tree leared from the certai dataset. Fig. 1. Depedecy tree from the dataset Letter ad the correspodigly certai dataset The two trees share the black solid edges. Edge<X 13, X 12 > belogs to the tree from ucertai dataset ad edge <X 4, X 14 > belogs to the tree from certai dataset. We fid that oly edge <X 4, X 14 > ad edge <X 13, X 12 > are differet. The differece betwee I(X 4, X 14 ) ad I(X 12, X 13 ) accouts for the bigger oe of the two iformatio etropy less tha 1.3%. We desig experimets o AU-Balace to demostrate the effectiveess of the depedecy tree by clusterig results. We use DTAU algorithm to lear a depedecy tree from AU-balace data. The we geerate a certai sample dataset i for the ucertai traiig example ue i i the dataset. The we treat the sample dataset i as the traiig dataset ad the depedecy tree as BN structure to lear the joit probability distributio o all attributes i the ucertai dataset ad the the ucertai example ue i turs to be a ucertai object. We cluster those ucertai objects by algorithm UK-meas [4] ad the origial certai objects by K-meas ad compare the two clusterig results. We do experimets o the AU-Balace dataset with differet values of α. Table 3. Clusterig precisio uder differet parameters Dataset Cluster 0 Cluster 1 Cluster 2 precisio AU-Balace % AU-Balace % AU-Balace % Balace certai %

7 320 W. Sog et al. We test three differet values of parameter α, 0.6, 0.8 ad 0.9. For each dataset, the cluster result is compared with the true class labels. Table 3 shows the results uder differet ucertai dataset. The umbers i colum 2, colum 3 ad colum 4 represet the size of correct examples i the correspodig cluster. The measure precisio shows the percetage of correct clustered examples. From this table we ca see that the precisio for each of the three ucertai dataset is quite close to the certai oe, for the certai oe is always the possible world with the highest probability. The experimets show that the depedecy tree geerated by our method is acceptable ad α is a importat factor to the cluster results. 6 Coclusio I this paper we propose the Bayesia Network structure learig problem o attribute ucertai dataset, ad we propose algorithm DTAU by which we ca lear a depedecy tree i polyomial time. We coducted experimets to demostrate the effectiveess of our proposed algorithm. The experimet results show the depedecy trees are acceptable ad the proposed algorithm is effective. I the future, we will further aalyze the effect of parameters α. Ackowledgemet. This work was supported by the NSFC uder Grat No ad ad the Major Natioal Sci. ad Tech. Project of Chia uder Grat No. 2010ZX Refereces 1. Dalvi, N., Suciu, D.: Effciet query evaluatio o probabilistic databases. The VLDB Joural 16(4), (2007) 2. Berecker, T., Kriegel, H.P., Rez, M., Verhei, F., Zuefle, A.: Probabilistic frequet item set miig i ucertai databases. I: 15th ACM SIGKDD Iteratioal Coferece o Kowledge Discovery ad Data Miig, pp ACM Press, Paris (2009) 3. Sigh, S., Mayfield, C., Shah, R., Prabhakar, S., Hambrusch, S., Neville, J., Cheg, R.: Database support for probabilistic attributes ad tuples. I: 24th IEEE ICDE Iteratioal Coferece o Data Egieerig, pp IEEE Press, Cacú (2008) 4. Lee, S.D., Kao, B., Cheg, R.: Reducig UK-meas to K-meas. I: 7th IEEE ICDM Workshops Iteratioal Coferece o Data Miig Workshops, pp IEEE Press, Omaha (2008) 5. Gullo, F., Poti, G., Tagarelli, A., Greco, S.: A Hierarchical Algorithm for Clusterig Ucertai Data via a Iformatio-Theoretic Approach. I: 8th IEEE ICDM Iteratioal Coferece o Data Miig, pp IEEE Press, Pisa (2008) 6. Güema, S., Kremer, H., Seidl, T.: Subspace Clusterig for Ucertai Data. I: SIAM SDM SIAM Coferece o Data Miig, pp SIAM Press, Ohio (2010) 7. He, J., Zhag, Y., Li, X., Wag, Y.: Naive Bayes classifier for positive ulabeled learig with ucertaity. I: SIAM SDM SIAM Coferece o Data Miig, pp SIAM Press, Ohio (2010)

8 Bayesia Network Structure Learig from Attribute Ucertai Data Re, J., Lee, S.D., Che, X., Kao, B., Cheg, R., Cheug, D.: Naive bayes classificatio of ucertai data. I: 9th IEEE ICDM Iteratioal Coferece o Data Miig, pp IEEE Press, Miami (2009) 9. Dalvi, N., Suciu, D.: Learig Bayesia etworks is NP-complete. Lecture Notes I Statistics, pp Spriger, New York (1996) 10. Chow, C., Liu, C.: Approximatig discrete probability distributios with depedece trees. IEEE Trasactios o Iformatio Theory Joural 14(3), (1968) 11. Heckerma, D., et al.: A tutorial o learig with Bayesia etworks. I: Learig i Graphical Models, Michael I. Jorda, Massachusetts (1999) 12. Friedma, N., Nachma, I., Peér, D.: Learig Bayesia Network Structure from Massive Datasets: The Sparse Cadidate Algorithm. I: UAI 14th Coferece o Ucertaity i Artificial Itelligece, Madiso, pp (1999)