Statistically Analyzing the Impact of Automated ETL Testing on Data Quality

Transcription

1 Chapter 5 Statisticall Analzing the Impact of Automated ETL Testing on Data Qualit 5.0 INTRODUCTION In the previous chapter some prime components of hand coded ETL prototpe were reinforced with automated testing procedures to enhance data qualit at the target database. In simplest words usefulness of data defines its qualit but from research viewpoint, data qualit has been addressed differentl in different research areas, some of such prominent research areas include statistics, management, and computer science. Statisticians were the first to investigate some of the problems related to data qualit, the have proposed some mathematical theories for considering duplicates in statistical data sets, in the late 1960 s. At the beginning of the 1980 s, management gurus focused on how to control data manufacturing sstems in order to detect and eliminate data qualit problems. It was onl at the beginning of the 1990 s computer scientists instigate considering the problem of defining, measuring, and improving the qualit of electronic data stored in databases, data warehouses, and legac sstems. It is a common practice among researchers to reduce data qualit definition just to accurac. Undeniabl, data are normall considered to be of poor qualit if tpos are present or wrong values are associated with a concept instance, for eample an erroneous birth date associated with a person. However, data qualit is more than simpl data accurac. Other significant dimensions such as completeness, consistenc, and concurrenc are necessar in order to full characterize the qualit

2 of data. Data qualit can be stated as a multifaceted concept, as in whose definition different dimensions concur. These data qualit dimensions, e.g., accurac, can sometimes be easil detected for eample misspellings but are more difficult to detect in cases where admissible but not correct values are provided. Completeness, the second data qualit dimension highlights the missing values within a tuple whereas consistenc deals with enforcing integrit constraints. Data preserves real world objects, in a format that can be stored, retrieved, and elaborated b a software procedure, and communicated through a network. The process of preserving the real world b means of data is applicable to a large number of phenomena, such as measurements, events, characteristics of people, the environment, sounds, and smells. Data are etremel versatile in such preservation. Besides primar data of objects, other supporting information also needs to be preserved in real-life and business processes, such as dimensions related to a fact. Since researchers in the area of data qualit must deal with a wide spectrum of possible data representations, hence the have proposed several classifications for data. The data qualit literature distinguishes, implicitl or eplicitl, three tpes of data: a) Structured: when each data element has an associated fied structure like tables in a relational database. b) Semi structured: when data has a structure with some degree of fleibilit. Semi structured data are of self-describing nature. XML is the language widel used to represent semi structured data. Some common characteristics of semi structured data are:

3 (i) Managing data fields not known at design time; for instance, a flat file does not have an associated schema file. (ii) Similar data can be represented in multiple was; for eample, a date might be represented b one field or b multiple fields, even within a single set of data (iii) Among fields declared at design time, man fields ma not have values. c) Unstructured, when data are epressed in natural language with no specific structure or domain tpes, It is intuitive that dimensions and techniques for data qualit have to be adapted for the three tpes of data described above because it is progressivel more comple to conceive and use structured to unstructured data simultaneousl. This chapter substantiates the need of automated ETL testing through statistical analsis of recorded ETL performance before and after the introduction of automated testing. Following the null hpothesis a paired t test was chosen to adjudicate the performance of customized ETL routine. According to paired t test for difference of means, the calculated value of t should be higher than the tabulated values of t. The null hpothesis presumes that there is no significant difference in data qualit before and after the introduction of automated testing. The simplest eperimental design for a t test is to have two conditions: an eperimental condition in which subjects receives some kind of treatment, and a "control" condition in which the do not. If one wants to compare performance in the two conditions then t test is the optimal choice. Sometimes, the difference between the two conditions is ver clear-cut: i.e. the eperimental treatment has made a clear difference to subject s behaviour. In other conditions as in case of pschological analsis the difference between the conditions is not so obvious; in these

4 circumstances, use of a t-test can help to decide whether the difference between the conditions is "real" or whether it is due merel to chance fluctuations from one time of testing to another. There are two versions of the t-test: a) Dependent-means t-test (also known as the "matched pairs", paired or "repeated measures" t-test): this variant is used when the same subjects participate in both conditions of the eperiment. b) Independent-means t-test (also known as an "independent measures" t- test): this variant is used when there are two different groups of subjects, one group performing one condition in the eperiment, and the other group performing the other condition. If there is a case in which sample sizes are equal i.e. n1 = n2 = n and there is a need to eamine if there is an shift of location parameter i.e. mean, in both samples then paired t-test is the optimal choice. 5.2 STATISTICAL ANALYSIS Electronic data pla a crucial role in the success of an data warehouse application. Data warehouse applications are business driven entities and because of the eplicit data requirements of an business domain it is hard to devise a generalized data qualit assurance model for data warehouse applications. Electronic data is so widel diffused that the qualit of such data and its related effects on ever kind of decision making activit of the data warehouse are becoming more and more critical. To understand the gravit of this situation it was decided to simulate such environment with the help of ETL prototpe developed earlier during the course of this research. Near to real snthetic test data was generated and placed at different logical locations. The test data was later snthesized with the help of ETL prototpe

5 to identif the probable data qualit issues. Following are the data qualit concerns/ dimensions identified from the user s perspective after analsing the snthesized data. Sno. Data Qualit Concern Concern Tpe Desired Feature 1 Correctness Data Value Data are correct and reliable 2 Completeness Data Value Degree to which values are present 3 Consistenc Data Value Follow integrit constraints and rules 4 Timeliness Data Value The age of the data is appropriate for the task at hand 5 Interpretabilit Data Format Correct interpretation of data values based on clear and unique data definitions 6 Handling Values Data Format Abilit to unambiguous characterization of null and default values from applicable values of the domain 7 Robust Representation Data Format Toning representation of similar tpe of data. 8 Ease of understanding Data Format Data are clear without ambiguit and can be easil comprehended 9 Relevanc Data Format Data are applicable and useful for the task at hand Table 5.1 Data Qualit Concerns from User s Perspective Keeping in view the user s perspective of data qualit and the tpes of anomalies discussed in the previous chapter, here the data qualit problems have been alienated in seven categories. The snthetic database of thirt five thousand records was snthesized using the customized ETL routine. These snthesized records were divided into seven samples of five thousand records each. Among these five thousand records, five hundred tuples were selected randoml. The manuall observed value of each data qualit issue in the seven samples has been recorded before and after the introduction of automated testing with the intension to prove the enhancement in data qualit Table 5.2, represents the observed counts of data anomalies before and after the introduction of automated testing. To verif the effectiveness of automated testing

6 one has to check whether with the introduction of automated technique the count of errors have decreased considerabl or not. Thus, to test the shift of location parameter paired t-test was applied following a generalized statistical hpothesis described below: a) H 0 : there is no shift of location parameter, i.e. mean value of samples taken before and after the introduction of automated testing is same, i.e. μ1 = μ2. b) H A : location parameter is shifted on the lower side i.e. mean value of samples taken after the introduction of automated testing is less than that of the samples taken before the introduction of automated testing. i.e. μ1 > μ2. Sp No Leical Anomalies Format Errors Irregularities Integrit Constraints Duplicates Semantic Errors Contradictions (=sample values before automated testing, = sample values after the introduction of automated testing) Sp NO:= Sample Number Table 5.2 Anomalies Observed and the Introduction of Automated Testing in Corresponding Samples. Here both μ1 and μ2 are the mean values of each data set of values before and after the introduction of automated testing. Initiall H 0 (null hpothesis) was followed with a perception that mean values of samples taken before and after the induction of automated testing are same. But a decrease in the mean value of samples taken after

7 the induction of automated testing clearl represented a substantial decrease in errors. Table 5.3 below provides the details of applied paired t-test and its tabulated values at 5% and 1% level of significance along with the p-values to confirm the results. According to p-value concept lesser the value of p for each test there are more chances of selecting the alternative hpothesis, i.e. H 1. Moreover, if the calculated value of t-statistic is more then the tabulated t-value at 5% and 1% level of significance then one has to reject the null hpothesis. Anomalies Actual calculated value of t test Tabulated value of t using 5% level of significance Tabulated value of t using 1% level of significance p-values Decision Leical Format Irregularities Integrit Constraints Duplicates Semantic Errors Contradictions Hpothesis Hpothesis Hpothesis Hpothesis Hpothesis Hpothesis Hpothesis Table 5.3 Comparisons of Calculated Values of t and p with Tabulated Values at 5% and 1% Level of Significance. Thus the analsis of above table showed that in each categor of error component we had rejected the null hpothesis and its alternative was accepted i.e. there is shift on the lower side of mean. This indicates a considerable decrease in data anomalies after the introduction of automated testing. The results of the statistical analsis performed earlier have been summarized with the help of Figure

8 Data Anomalies and The Automated Testing Leical Anomalies Format Errors Irregularities Integrit Constraints Duplicates Semantic Errors Contradictions Sample Wise Categories of Errors Sample1 Sample2 Sample3 Sample4 Sample5 Sample6 Sample7 Count of Errors Figure 5.1 Bar Chart Representation of Statistical Analsis

9 The aforesaid analsis has revealed the fact that there is a significant gap between the perception and realit of prevailing data qualit standards. The Y2K problem which led to modif software applications and databases using a two digit field to represent ears is an archetpe data qualit issue turned up because of poor data qualit standards. The concept of data is rapidl evolving, from structured data of relational databases, to semi structured data, unstructured data, documents, images, sounds, and maps resulting in a continuous change of the concept of data qualit. Due to relative immaturit of data qualit research area and absence of enforced de facto standards enacted b international organizations, it is etremel hard to formulate an standard data qualit model for data warehouses. Hence there is an utmost need to classif and define the data qualit dimensions and metrics from data warehouse point of view