Building a Data Warehouse: Data Quality is key for BI. Werner Daehn

Transcription

1 [ Building a Data Warehouse: Data Quality is key for BI Werner Daehn

2 [ Learning Points A DWH project is about discovering new information Not having a good quality counterfeits that purpose Actually it is dangerous as you might draw wrong conclusions With little effort amazing things can be achieved using today s ETL tools 2

3 [ Agenda Develop sensibility for the problem Things you can do about it The holistic view 3

4 [ THE PROBLEM 4

5 [ How does a typical report look like? It shows the data is as Duplicates with different spelling Different formatting Missing attributes The report does not look nice 5

6 [ How does a typical report look like? But the looks is not everything, inner beauty is important too How much revenue did I make in Denver? Zero, it is DENVER Except for one major customer, there it is Denver CO Of course we can ask to correct that in the source system but It will never be done It is no long term solution as it will happen at any time again What can we do about it? Change the Universe formula? 6

7 [ How does a typical report look like? Same problem everywhere Missing data Comma or not Duplicates Upper case/lower case word 7

8 [ DO SOMETHING ABOUT IT 8

9 [ Categories of Data Problems Technical inconsistencies like Foreign Keys, Primary Keys, overlaps in from-to-date ranges, NULL values in fields used as LoV, blanks or empty strings, Missing information in fields Logical inconsistencies like product weight greater 1000 tons Duplicates two keys for the same natural object Incorrect or different spelling Too little relevance in field values are used in queries A few years back you would not have had the option to optimize above 9

10 [ Technical Inconsistencies Fixing these is a standard task ETL tools are built for that We talked about that in the ETL session already 10

11 [ Missing Information in fields Options Set the value to a constant, e.g.? Set the value to the key then, e.g. name is IAD-1000 In the Universe the object Material Name is a combination of name and key, e.g. unknown (IAD-1000) The text column contains this combination already Notify the source system, e.g. automatically send and No solution is perfect, pick any 11

12 [ Logical Inconsistencies Can be broken down into different categories again Multiple values mean the same thing Values being out of range Two fields contain mutual exclusive value combinations 12

13 [ Multiple values mean the same thing Do I have more male, female or companies as customers? Comp., Company, Firma, Empresa, Using this information as-is does not make sense Lookup table converting the input value to a standardized output Maintain a rule like ifthenelse( ) Unknown values map to a default like? 13

14 [ Values are out of range On row/field level like the example of weight Too expensive to validate everything Too expensive to figure out ourselves On aggregated level e.g. sum(revenue) was just 100USD yesterday? Very Unlikely Called Dataflow Auditing in DataServices Define a set of conditions tested at the dataflow end Execute script actions if violated e.g. if violated consider the dataflow failed 14

15 [ Mutual exclusive value combinations Require extensive lookup data being maintained Complex conditions with patterns and expressions But rules not coded in the ETL logic for maintenance reasons Lookup function is commonly used for that Not only equal operations supported but patterns too So the pattern formula(!) is in the lookup table And the column to be returned can be a formula(!) Control over what row to be returned via return-policy 15

16 [ Mutual exclusive value combinations For some areas DataServices provides Data Quality Transforms plus all the required reference data Address Cleanse looks at all address related fields and fixes missing data, inconsistency, spellings, standardization problems Name parsing for customer name, company name, , and other related fields 16

17 [ Mutual exclusive value combinations - Addresses Assign input fields to their type of information e.g. STREET input field is an ADDRESS_LINE Worst case use MULTILINE and the transform has to figure out Pick additional output fields from a list of more than 100 generated values Packed by up 30 GB of official postal office data for all major countries worldwide Plus a Worldwide directory at postal code/city level 17

18 [ Mutual exclusive value combinations - Addresses 18

19 [ Duplicates In our Material table there is definitely something wrong Either multiple material numbers for the same product Or the text is not precise enough DataServices has a Match Transform Identify potential duplicates Assign the same Key Number to all duplicates This key can then be the parent of an hierarchy Create a golden record Makes 1 million comparisons per second/cpu Use break keys, e.g. MATERIAL_TYPE to limit what has to be compared with what Some fuzzy logic options for the comparisons 19

20 [ Duplicates Configured Material_Type to be a Break-Key, only records within one type can even be potential duplicates STD_MATERIAL output column is the material s parent 20

21 [ Incorrect or different spelling DataService Data Cleanse transform is for text cleansing Supports common text requirements, e.g. convert the word five to 5, is the order of the words important, etc The reference data has to be created yourself for non-standard cases With the Information Steward product there is an interactive Cleansing Package Builder based on data profiling results 21

22 [ Incorrect or different spelling 22

23 [ Relevance criteria Where do we buy our Supply goods (COST_ELEMENT=403000) from? Hm, ½ the costs are with US based vendors, just ¼ with DE based ones But another ¼ is assigned to vendors with no country So is it save to say the majority is US? Yes, all cost bookings unrelated to vendors are for vendor? used for internal cost assignments 23

24 [ Relevance criteria ETL dashboard 24

25 [ THE HOLISTIC VIEW 25

26 [ Responsibilities Who gets the blame if a report contains duplicates? The DWH team Who can ultimately fix the problem The person responsible for the record in the source system Who wants to see correct data but does not? The management Who has the work fixing all records? The person responsible for the record in the source system There is are disconnects! 26

27 [ Fixing the disconnects Using shown ETL techniques we can limit the impact of DQ problems We can correct those tiny issues while loading the DWH, like standardized names We can even correct things in the source if we are confident enough using the ETL tool For the rest we have to enable team work Make the users of the source system aware that their data is used Enable the DWH users to correct the data themselves?! 27

28 [ Business User Collaboration Enable them to look at the data And to figure out what the problem is They will ask the source system people directly 28

29 [ Business User Collaboration Enable them to monitor Data Quality trends themselves Is important data getting more precise? What state is it? Product-Master is in state green, but customer is yellow! 29

30 [ Business User Collaboration Business User does define allowed values, what to do with problems, all the rules DataServices is just using those rules 30

31 [ Best Practices Do not change the data, add it in an extra column Hide columns in the Universe if they are too sparsely set Standardize the data automatically Correct data as much as possible Validate the data for feasibility only the most obvious things Enable collaboration between the Business Users to put the DWH out of the line 31

32 [ Key Learnings Garbage In Garbage out principle does not work for us Doing something about wrong data is possible Use that advantage whenever appropriate Measure the quality of the loaded data In case of problems ask for changes in the Universe or the source system Ideas for Business User collaboration 32

33 [ ] Thank you for participating. Please remember to complete and return your evaluation form following this session. For ongoing education on this area of focus, visit the Year-Round Community page at [ SESSION CODE: