Learning Bayesian Networks from Inaccurate Data

Transcription

1 Learnin Bayesian Networks from Inaccurate Data A Study on the Effect of Inaccurate Data on Parameter Estimation and Structure Learnin of Bayesian Networks Valerie Sessions Marco Valtorta Department of Computer Science and Enineerin, University of South Carolina Astract The wealth of data collected y automated systems, or pen and paper processes, and availale on the World Wide We, is staerin. With these vast amounts of data come amazin possiilities for data minin and learnin of data patterns throuh automated and semi-automated tools. However, these tools are inherently reliant on the quality of the data with which they are supplied. Without a thorouh understandin of our data s quality and the effects of poor quality on our learnin alorithms, we are consequently unale to accurately jude the quality of our models. We have distilled the ideas of prominent data quality researchers into four features of data sources: data accuracy, completeness, timeliness, and consistency. After understandin various dimensions of data quality we were ale to test the effects of inaccurate data on learnin of Bayesian Networks, specifically in Parameter Estimation and Structure Learnin. By testin the learnin alorithms with differin percentaes and amounts of inaccurate data, we have eun to develop uidelines for adjustin the learnin techniques and judin the quality of the learned models. 1

2 1. Data Quality Metrics Assessin the quality of our data is a difficult process and is ripe with sujectivity. Researchers have een attemptin for some time to create quantitative metrics to accurately jude the quality of our data. There are many different measurements that these researchers use to assess data quality. These rane from the sujective value-added and understandaility, to those that are more easily quantifiale, such as completeness and timeliness. While some researchers have up to sixteen different metrics [Pipino et al 00], we will concentrate on a core set of four for this research accuracy/precision, completeness, consistency/elievaility, and timeliness, distilled from [Wand and Wan 1996], [Stron et al 1997], and [Tayi and Ballou 1998]. We will explain these terms in the context of our research. Accuracy and precision are taken from their common scientific meanins. Accuracy represents how close a measurement, or data record, is to the real-world situation that it measures. Precision will refer to two similar ideas in this research. Firstly, it will refer to the standard deviation or variation in a numerical data record with multiple readins. As an example, if a weather sensor is calirated efore its use to have a variation of ± 0.01 C, we would use this calirated rane as the precision of the instrument. The second use for precision is to quantify the deree to which a sensor, or other data input, ives a consistent data output for a iven input, reardless of the accuracy of these results. Of course, we would wish our data to e oth accurate and precise. While accuracy is a metric that must e calculated over time and with much dilience to discover discrepancies etween what is contained in the real world and what

3 is represented in the data record, precision can more easily e calculated from the data at hand. Completeness in this research will refer to the amount of missin data records. This can e computed as the total numer of missin data, or the total numer of rows containin missin data. Each can e useful in our research. This metric is easily calculated from the data at hand, and requires no sujective insertions from the user or data manaer. Completeness is especially important in our research, as we can aue the effects of usin approximation alorithms, such as Gis Samplin or the Expectation- Maximization (EM) alorithm, in our learnin techniques. Lare amounts of missin data can also point to prolems in the data collection method and the data collection tools. Timeliness of the data is also important in the context of our data set. Data that is outdated is often useless in many fields of study, such as weather data records, or stock market data that must e used to make uyin and sellin decisions in seconds. However, in other fields the timeliness of data is not as crucial. If historical data is ein used to track consumer spendin over the last decade, havin data within minutes of the realworld event is not as important. This metric is therefore at once oth sujective and ojective. If the data is time-stamped, it is not difficult to determine the time difference etween the entry and the present time. However, we must have allowed a Suject Matter Expert to then uide the proram to determine how current the data should e for that particular purpose. Lastly, our research will employ the idea of consistency or elievaility. These metrics are similar to precision, except will e used here to compare data from different data sets. Where precision applies to one particular data source a particular sensor or 3

4 manually inputted data source, consistency/elievaility will apply to multiple data sets reportin on the same real-world situation. If we have three sensors reportin weather information from one location (or small radius) we can measure the consistency of the data for that reion as the deviation etween the data records from each sensor. Similarly, if we have eyewitness accounts at the scene of a crime, we can jude the consistency of the data set y the similarity of the accounts. This metric is computed separately from the accuracy and precision calculations and ives no weiht to one data source over the other. If there are three data sources, we will calculate one consistency metric for that data record which takes into account differences from all the data sets, without declarin which data source is most accurate. This allows us to account for differences in data without knowin the accuracy of the data set. This may seem faulty, and indeed if we have the resources to jude the accuracy of the data sets then we can assin data quality ased on those findins. However, consistency allows us a metric to determine that there are indeed differences in data sources, without tracin the data sets ack to the sources and havin to exhaustively determine which data set is most accurate. Consistency is an easier metric to calculate and more likely to e availale for use in our alorithms.. Testin the Effects of Inaccuracy While all four data quality dimensions will eventually e thorouhly studied in our research, the focus of this paper will remain on inaccuracy and its effect on learned Bayesian Networks (BNs). In order to understand these effects, a set of tests was constructed. Two test BNs were modeled one that will e considered the true state of the BN and another, ad data set in which all of the potentials were set at 0.5 (equal 4

5 likelihood of either result). Data sets were then enerated with 0%, 10%, 5%, 50%, 75%, and 90% dirty data y creatin one data set with the true data records and a second with the ad data and meldin them toether. For example, to enerate a 10% dirty set for 100 records, one would enerate 90 cases from the true data set and 10 cases from the ad data set and comine them for a 100 record set. This was done for 100, 500, 1000, and 10,000 record sets. For purposes of this research, two traditional BNs were studied Trip to Asia and the Stud Farm network. The structures of these networks are shown in Fiures 1 and respectively. Fiure 1 Trip to Asia 5

6 Fiure Stud Farm Network After creatin the data sets, the enerated data were iven to three BN learnin prorams a Parameter Estimation alorithm, the Huin Researcher Structure Learnin tool that implements the NPC and PC alorithms, and Visual CB, an implementation of the CB alorithm developed y [Sinh and Valtorta 1995]. See also [Neapolitan 004]. The pseudocode for the Parameter Learnin alorithm is displayed elow in Fiure 3. The input to the code is a Huin.net file as well as the data sources either in a flat file or XML schema. The alorithm uses either the linear calculation, or the mean of the eta distriution for the potential calculation. This code can e easily manipulated for testin our new alorithms y chanin the potential calculations of each data set. 6

7 Parameter Estimation Pseudo code Input: Bayesian Network structure in form of Huin.net file, flat file or XML schema of data sets Parse structure et nodes and states of nodes from Huin structure For each Node If Node with no parents i = count total numer of entries for that node For each state of node (yes, no, other, etc) j = Count entries of that state Calculate eta distriution for that state Potential for the state = effective value, E(F) = j/i, or mean of eta distriution If Node with parents For each state of parent i = count total numer of entries where parent = that state For each state of node j = count numer of entries where parent = current state and node = current state Calculate eta distriution for that state Potential for the state = effective value, E(F) = j/i, or mean of eta distriution Output: Bayesian Network with learned potentials in form of Huin.net file Fiure 3 Parameter Estimation Pseudocode The potentials enerated y the alorithm were compared to the aseline of learnin from 0% dirty data. The variance etween the aseline and the dirty data sets was then calculated (standard variance calculation - σ αβ =, where α ( α + β ) ( α + β + 1) and β are standard parameters of the eta distriution). Then the averae variance was calculated for each size of the data set - 100, 500, 1000, and 10,000 data records. Also, the data sets potential calculations were raphed to show differences in oriinal and dirty data sets. In order to test the effects of inaccurate data on the Huin Structure Learnin alorithms, the dirty data sets were inputted to this proram and compared to a structure 7

8 learned from a aseline of 0% dirty data. These structures were compared for missin links, reversed links, and extra links to determine the effects of this inaccurate data on the learned structures. Further, the data sets were run throuh a second Structure Learnin alorithm developed y [Sinh and Valtorta 1995] the CB alorithm. This alorithm is a comination of a conditional independence testin alorithm with the traditional K alorithm for determinin BN structure. These structures were also compared for missin links, reversed links, and extra links aainst the aseline data. 3. Parameter Estimation Results The resultant potential calculations for the Trip to Asia experiments are shown in taular form in Fiure 4 and raphical form in Fiure 5. Averae Variance Baseline 10 percent 5 percent 50 percent 75 percent 90 percent Fiure 4 Averae Variance From Baseline 8

9 Comparison Of Learned Potentials 1. 1 Learned Potential Series1 Series Series3 Series4 Series5 Series Data Point Fiure 5 Comparison of learned potentials from dirty data sets. Series 1 Baseline Data Potentials learned from 0% dirty data on 100 records Series Baseline Data Potentials learned from 0% dirty data on 10,000 records Series 3 Potentials learned from 10% dirty data on 100 records Series 4 Potentials learned from 10% dirty data on 10,000 records Series 5 Potentials learned from 90% dirty data on 100 records Series 6 Potentials learned from 90% dirty data on 10,000 records The results from the raph are surprisin at first, ecause it seems as thouh the smaller data sets yield etter results than the larer ones. This is not the case however, and the apparent rise in variance as the data set ets larer is due to normal variances of the eta distriution. While the distriution of ood and ad data sets is important, the results do not chane on averae with a rise in the total numer of data records used. We will prove this in two parts. Firstly, we will prove that the potential of a state is due to the distriutions of the data, and is not affected y the amount of data used to learn the data sets. Secondly, we will show that the variance from the oriinal learned data sets is directly related to the inherent properties of the eta distriution. 9

10 Theorem 1 The learned potential of a state is invariant with respect to the numer of n ood and ad data sets, n and n, respectively, when the ratio is fixed. n Proof: Consider a inary variale, say X, in state yes. The potential of X = yes is φ ood. If a sample of size n is otained y simulation, the numer of cases in which X = yes has mean n * φ ood. Consider a second distriution in which the proaility of X = yes is φ ad. If a sample of n size is otained from this distriution, the numer of cases for which X = yes has mean n * φ ad. Considerin a sample that is an areate of the aove two, we otain the fraction of the cases for which X = yes is ( n ood ) + ( n ad ) φ overall =. (1) ( n + n ) To prove that this ratio is invariant with respect to n + n when replace n with k * n and n with k * n. (( k n ) ood ) + (( k n ) ad ) φ overall =. (( k n ) + ( k n )) This reduces to k *(( n ood ) + ( n ad )) φ overall =. k *( n + n ) n n is fixed, we will The factor k cancels out, provin that the φ overall is invariant with respect to the size of the n overall data set, when is fixed. n Lemma 1 - The variances in the data sets that are apparent in the experiment are due to the inherent properties of the eta distriution. 10

11 Explanation The formula for the eta distriution is ( α 1)!( β 1)! B ( α, β ) =. ( α + β 1)! α The mean is iven y µ = and the variance y α + β αβ σ =. ( α + β ) ( α + β + 1) We proved in Theorem 1 that the mean is invariant with respect to additional overall data when the ratio of ood data to ad data is constant. Now we will see if the variance is also invariant with respect to n + n when the ratio n n is fixed. We have the same n, n, φ ood, and φ ad from the previous proof. Sustitutin for α and β we have σ [( n = ood ) + ( n ad ( n )] [( n + n ( n ) ( n + n ood + 1) )) + ( n ( n ad ))]. Sustitutin n with k*n and n with k * n we have σ [( kn = ood ) + ( kn ad ( kn )] [( kn + kn ( kn ) ( kn + kn ood + 1) )) + ( kn ( kn ad ))]. Factorin k we have σ = k [( n ood ) + ( n k ad ( n )] [( n + n ( n ) ( kn + kn ood + 1) )) + ( n ( n ad ))] σ [( n = ood ) + ( n ad ( n )] [( n + n ( n ) ( kn + kn ood + 1) )) + ( n ( n ad ))]. We cannot factor out the k from the denominator. The lim σ = 0. k Therefore, as k ets larer the variance ecomes smaller. 11

12 Results from the Stud Farm Network are similar, ut were run on only 100 and 10,000 data records, for the aseline, 10%, 50%, and 90% dirty data. The Parameter Estimation results are shown in taular form in Fiure 6, and raphically in Fiure 7. Averae Variance Baseline 10 Percent 50 Percent 90 Percent Fiure 6 Stud Farm Averae Variance from Baseline Comparison of Learnt Potentials - 10,000 Data Records 1. Learnt Potential Baseline Percent Percent Percent Parameter Fiure 7 Stud Farm Learnt Potentials Graph 4. Structure Learnin Results The results of the structure learnin tests for the NPC/PC alorithm usin the Huin Researcher tool, and the CB alorithm, are shown in taular form in Fiure 8. Because the results from the NPC and PC tests were very similar, we will display only the NPC results for ease. 1

13 NPC-100 NPC-500 NPC-1000 NPC CB-100 CB-500 CB-1000 CB Total Wron Baseline Percent Percent Percent Percent Percent Fiure 8 NPC and CB Structure Learnin Results The total incorrect links are calculated y addin the extra links, reversed links, and missin links. This is displayed in raphical form for 100 and 10,000 data records in Fiure 9. Comparison of Incorrect Links Numer of Incorrect Links NPC-100 NPC CB-100 CB Percentae of Dirty Data Fiure 9 Comparison of Incorrect Links NPC and CB Alorithms These results are also deceivin at first, as it appears that the total amount of wron data increases until approximately 75% of dirty data, and then declines at 75-90% dirty data. This is not the case, however, as revealed y a careful examination of the total correct, missin, reversed, and extra links. At around 5-75% missin data, the learnin 13

14 alorithms ein to discover aout twice as many total links as in the 0-10% and 75-90% dirty data sets. With the larer amounts of conflictin data, the alorithm finds links in oth the ad data set and the true data set ivin us twice as many links oth correct and incorrect. As the percentae of 50-50, or ad data increases to 75-90%, or as the ad data percentae is low, 0-10%, the alorithms finds only one set of links. This can e seen as we raph the total numer of incorrect links divided y the total numer of links found. This is shown in taular form in Fiure 10 and raphically for 100 and 10,000 data records in Fiure 11. A similar result is noted in [Spirtes et al 000]. The authors discovered that if the PC alorithm fails to find an ede from the true raph in the second stae of the alorithm, then the alorithm fails in susequent steps to remove nonexistent edes. They also discovered that the failure to find an ede in the true raph could lead to orientation mistakes in susequent staes. Total Wron/Total Links NPC-100 NPC-500 NPC-1000 NPC CB-100 CB-500 CB-1000 CB Baseline Percent Percent Percent Percent Percent Fiure 10 NPC and CB Structure Learnin Results Total Incorrect/Total Links 14

15 Comparison of Learnt Structures 3.5 Total Incorrect Links/Total Links NPC-100 NPC CB-100 CB Percentae of Dirty Data Fiure 11 Comparison of Incorrect Links Total Incorrect/Total Links Results from the Stud Farm network were similar and are shown in taular form in Fiure 1 and raphically in Fiure 13. Total Incorrect Links/Total Links NPC-100 NPC CB-100 CB Baseline Percent Percent Percent Fiure 1 Stud Farm NPC and CB Structure Learnin Results 15

16 Comparison of Learnt Structures 7 6 Total Wron/Total Links NPC-100 NPC CB-100 CB Percentae Dirty Data Fiure 13 Stud Farm Comparison of Incorrect Links Total Incorrect/Total Links 5. Conclusions Even from a cursory overview of the results of these experiments, it is evident that Parameter Estimation alorithms are more tolerant of inaccuracies in data than the Structure Learnin alorithms we have studied NPC and CB. At only 10% dirty data, the CB alorithm with 10,000 data records increased the numer of incorrect links from (aseline) to 16 - an 800% increase. NPC also had an increase in incorrect links of 35% for 10,000 records with only 10% dirty data. The results after 10% dirty data, while not increasin at a sinificant rate, also do not ecome closer to the oriinal structure, as would e expected. There does not seem to e a sinificant advantae to either of these Structure Learnin alorithms in terms of handlin inaccurate data, althouh the CB alorithm appears, in these two examples, to create etter structures for smaller data records, and NPC seems to create etter structures with larer amounts of data. Therefore when dealin with small data sets of inaccurate data, the CB alorithm may e a etter 16

17 choice for learnin structures. The most sinificant findin of this research, thouh, is that even at small amounts of inaccurate data, these Structure Learnin alorithms are unale to enerate correct structures. Therefore a thorouh understandin of the quality of the data is very important when usin Structure Learnin alorithms, as even small inaccuracies can create completely incorrect models. Our parameter estimation alorithm, however, does follow a radual increase in inaccurate modelin when learned from inaccurate data sets. As the inaccuracy of the data increases, the learned potentials also deviate from the oriinal and increase radually accordin to percentae of missin data. Therefore, unlike the Structure Learnin alorithms we have studied, BNs learned from inaccurate data can e useful when the potentials are iven a rane of accuracy ased on the elieved accuracy of the data. Therefore if a model is elieved to e ased on data that is 90% accurate with 100 data records, the potentials can e iven with a standard deviation of approximately 0.18 ( variance = ). This variance was taken from the Trip to Asia data results, however we realize that the Stud Farm standard deviation for 10% missin data would e hiher 0.8. Our future research will strive to verify an averae potential deviation for the various percentaes of inaccurate data that can e applied to a wide rane of new models. 6. Future Research Future research shall include methods for incorporatin the results of these experiments into learnin alorithms for BNs. For Parameter Estimation this opens two areas of future research quantifyin averae standard deviation fiures for expressin the rane of accuracy of a model learned from partially inaccurate data, and investiation into 17

18 methods for discountin those sets of data that are deemed to e partially inaccurate. We elieve this will add to the adaptation work of [Olesen et al 199] which was incorporated into the Huin enine. The results of these experiments are discourain for improvement of Structure Learnin alorithms for learnin from inaccurate data sets, as even small amounts of inaccurate data appear to reatly impact the usefulness and accuracy of the learned models. Our research will continue to test further Structure Learnin alorithms with the intent of findin an alorithm that handles inaccuracies more effectively. Alon with this research, it is also our elief that the inaccuracies of the data itself can e modeled effectively to show the dependence of the model upon data quality. For example, each node or link in the learned Bayesian Network can also have a quality node that quantifies the overall accuracy, completeness, timeliness, and consistency of the data set that was used for learnin. There exists a causal link from the accuracy of the data to the learned potentials and structure. An example of this idea is shown in Fiure

19 Fiure 14 Modelin Inaccuracies as a Causal Entity The potential tale for Tuerculosis chanes to add the parameter of Link Dependence on Data Accuracy. As the accuracy of the data decreases, the causal relationship etween Trip to Asia and Has Tuerculosis eins to fade and the potentials ecome evenly split no dependence on one parameter over another. This allows us to chane our model dynamically as our eliefs aout the quality of the data chanes. This is a challenin prolem, as more experiments will e needed to confirm the relationship etween accuracy of the data and existence of a learned causal link. Usin this initial research as a uide, however, it is our elief that we can ein to understand and account for data quality in the automated learnin of BNs. 19

20 7. Bilioraphy Neapolitan, R.E. (004). Learnin Bayesian Networks. Upper Saddle River, NJ: Pearson Education, Inc. Olesen, K., Lauritsen, S., Jensen, F. (199) ahugin: A System Creatin Adaptive Causal Proailistic Networks. Proceedins of the Eihth Conference on Uncertainty in Artificial Intellience, 3-9. Pipino, L., Lee, Y., and Wan, R. (00). Data Quality Assessment. Communications of the ACM, Vol Sinh, M., and Valtorta, M. (1995). Construction of Bayesian Network Structures from Data: a Brief Survey and an Efficient Alorithm. International Journal of Approximate Reasonin. Spirtes, P., Glymour, C., and Scheines, R. Causation, Prediction, and Search. MIT Press, Camride, Massachusetts, 000. Stron, D., Lee, Y., and Wan, R. (1997). Data Quality in Context. Communications of the ACM. Vol Tayi, G. and Ballou, D. (1998). Examinin Data Quality. Communications of the ACM. Vol Wand, Y. and Wan, R. (1996). Anchorin Data Quality Dimensions in Ontoloical Foundations. Communications of the ACM Vol