Data Quality / Data Cleansing in BW

Transcription

1 Data Quality / Data Cleansing in BW Lothar Schubert, BW RIG 8/

2 Agenda About Data Quality Data Cleansing Data Validation Data Repair 02 SAP AG 2001, Title of Presentation, Speaker Name 2

3 Why Data Cleansing / Validation? BW data are highly integrated. BW data are queried frequently. BW data are expected to be of high quality. BW requires high data accuracy for effective decision support. BW data often serve as foundation for further processing. Data Quality / Information Quality what it means: Data / Information is relevant. Data / Information is timely. Data / Information is correct. 03 SAP AG 2001, Title of Presentation, Speaker Name 3

4 Sources for Dirty Data Data are incorrect in source system Data consolidation causes issues Technical platforms are different (code pages, etc.) Administration issues (double loadings, ) Custom logic Technology issues (SW, DB, O/S, HW, ) 04 SAP AG 2001, Title of Presentation, Speaker Name 4

5 Data Contaminants Cup Holder, green US Cup Holder, red US Lighter, black US white cover US green Cup Holder US red wheel, type " " CA blue wheel, type " " CA white wheel CA inconsistent keys free form fields multiple keys Cup Holder, green JP phone plug JP channel JP plastic cover, red JP (pink version of above) JP surprises invalid characters 05 SAP AG 2001, Title of Presentation, Speaker Name 5

6 Data Contaminants - 2 data format data redundancy XYZ.com Ltd. 10/10/2000 $ XYZ.com Ltd. 10/10/2000 $ XYZ.com Ltd. 10/10/2000 $ XYZ.com Ltd. 10/12/2000 $ XYZ.com Ltd. 10/14/2000 $ XYZ.com Ltd. 10/17/2000 $ XYZ.com Ltd. 10/19/2000 $ data anomalies XYZ.com Ltd. 10/10/99 $ XYZ.com Ltd. 10/12/99 $ ABC Co. 10/14/2000 $ 4333 LMN Ltd. 10/14/2000 $ 9000 XYZ.com Ltd. 10/14/2000 $ ZZZ Sl. 10/14/2000 $ data redundancy 06 SAP AG 2001, Title of Presentation, Speaker Name 6

7 Data Contaminants - 3 Data Contamination during upload via Exits Application Exits Generic BW Exit RSAP0001 Transfer- / Update-Routines Virtual Exits Consider the following: Timeliness of Data Check for Versions Check for Return Codes Delta Trigger Capabilities Performance and General Architecture 07 SAP AG 2001, Title of Presentation, Speaker Name 7

8 Where, When and by Whom to implement Data Cleansing? Where: In the Source System? During Data Extraction? In the BW System? When: In the productive phase? In the test phase? In the blueprint phase? Who: Is it a technical issue? Is it a project issue? Is it an organizational issue? Data cleansing occurs at all levels. Avoid tendency to attempt cleanse only within the BW extraction process. Often data cleansing is best performed at the legacy / source system level. Data cleansing is one of the greatest risks in data movement efforts. Design belongs into blueprint phase. Test data are often cleaner than real data. Often data quality and inconsistency issues are systemic in the organization and must be addressed at higher level in the organization to get resolved. 08 SAP AG 2001, Title of Presentation, Speaker Name 8

9 ROI of Data Quality Data Quality as an Investment You should ask What is the risk of incomplete / incorrect data sets? What is the cost to fix data, once contaminated? What are corporate quality standard? However, also you should ask What is the reliability of source data? Where is the point of diminishing returns? 09 SAP AG 2001, Title of Presentation, Speaker Name 9

10 BW Architecture Business Information Warehouse any source Extraction / Open Staging Persistent Staging Area Transformation dwh ods object ods object ods object ods object ods Business Rules Info Cube Info Cube ods object data marts Business Rules master data Info Cube Info Cube Synchronous Access Asynchronous Distribution - Open HUB Services portal/ application any target Integration Cleansing Granularity 10 SAP AG 2001, Title of Presentation, Speaker Name 10

11 Agenda About Data Quality Data Cleansing Data Validation Data Repair 11 SAP AG 2001, Title of Presentation, Speaker Name 11

12 Referential Integrity Assuring Referential Integrity can be a major challenge in DWH design Relax! BW does it for you. Automated checks. Central Metadata Dictionary. Integrated Architecture. 12 SAP AG 2001, Title of Presentation, Speaker Name 12

13 Master Data Validation 13 SAP AG 2001, Title of Presentation, Speaker Name 13

14 Check for Permitted Characters Case A: characters not permitted Case B: characters permitted C Permitted by standard:!"%&'()*+,-/:;<=>?_ ABCDEFGHIJKLMNOPQRSTUVWXYZ 14 SAP AG 2001, Title of Presentation, Speaker Name 14

15 Consistency Check for Characteristic Values Checking for use of character values in the Data type NUMC fields correct consideration of the conversion routine ALPHA use of lower case letters use of special characters plausibility of date / time fields Consider Performance Impacts! 15 SAP AG 2001, Title of Presentation, Speaker Name 15

16 Data Integrity Checks on Packages APIs are available to read PSA contents Function RSAR_ODS_MAINTAIN,. Check for reference between records Summary checks,. 16 SAP AG 2001, Title of Presentation, Speaker Name 16

17 Handling of Invalid Data Records Tip: Consider Automation Via Event Chains. Business Information Warehouse Error Handling: 1- No Update, No Reporting 2- Valid Records Update, No Reporting 3- Valid Records Update, Reporting Possible Scheduler Extract OK PSA Staging Engine Error PSA Correction of invalid data: within source System manually in PSA by Rule (see RS_ERRORLOG_EXAMPLE) 17 SAP AG 2001, Title of Presentation, Speaker Name 17

18 Local Master Data 18 SAP AG 2001, Title of Presentation, Speaker Name 18

19 Deletion Features during Update 19 SAP AG 2001, Title of Presentation, Speaker Name 19

20 Agenda About Data Quality Data Cleansing Data Validation Data Repair 20 SAP AG 2001, Title of Presentation, Speaker Name 20

21 Aggregate Check Tool Report RRX_TRACE_CHECK_AGGREGATE Check OSS Note for details. 21 SAP AG 2001, Title of Presentation, Speaker Name 21

22 Custom Check Points Identify check points in source system Write check point data to custom table Use generic extractor for load Populate check cube Perform Compress with 0 suppression Execute exception report Key figures Article Cashier Number Article Cashier Number Sales (POS Receipts) Sales (Receipt) Overall result $ 0.00 $ 0.00 $ $ 0.00 $ 0.00 $ $ 0.00 $ 0.00 $ 0.00 Overall result $ 0.00 $ 0.00 $ SAP AG 2001, Title of Presentation, Speaker Name 22

23 Audit Dimensions / Data Modelling Audit Dimensions can identify: When were the data created? Which source did the data come from? Which tools where used for extraction? Which rules had touched the data? 23 SAP AG 2001, Title of Presentation, Speaker Name 23

24 Display of individual Requests Key figures Amount Request ID Request ID Amount $ 80, # $ 28,078, Overall result $ 28,158, You can use the REQUEST ID to display and analyze individual requests. 24 SAP AG 2001, Title of Presentation, Speaker Name 24

25 Check Programs (RSRV) Infocubes: Fact, SID, MID, Hierarchies Infoobjects DDIC Definitions Characteristic Values 25 SAP AG 2001, Title of Presentation, Speaker Name 25

26 Data Quality Check Flags 26 SAP AG 2001, Title of Presentation, Speaker Name 26

27 Agenda About Data Quality Data Cleansing Data Validation Data Repair 27 SAP AG 2001, Title of Presentation, Speaker Name 27

28 Request Deletion Infocube / ODS Consider Limitations! 28 SAP AG 2001, Title of Presentation, Speaker Name 28

29 Selective Deletion 29 SAP AG 2001, Title of Presentation, Speaker Name 29

30 InfoCube Reconstruction 30 SAP AG 2001, Title of Presentation, Speaker Name 30

31 InfoCube Request Reversal Posting Works still fine after compression / roll-up! 31 SAP AG 2001, Title of Presentation, Speaker Name 31

32 Data Quality / Data Cleansing in BW Lothar Schubert, BW RIG 8/