Fun and Useful Features in SAP Information Steward 4.2

Transcription

1 Fun and Useful Features in SAP Information Steward 4.2 The latest release of SAP Information Steward includes the Data Cleansing Advisor, which has some great new features that help organizations get a jump start on their Information Steward implementation. Today I ll show an example process for using the Data Cleansing Advisor to suggest Information Steward validation rules and create a cleansing solution. Profiling the Data The first step in generating cleansing rules is to have Information Steward profile the data. First, create a column profiling task (select table Profile Columns) and make sure to select the advanced profiling options Median & Distribution and Word distribution. Next, create a Content Type profiling task (select table Profile Content Type). Data Validation Advisor Suggestions Once these tasks have been run, returning to the Workspace Home Profile Results Advanced tab will indicate with icons in the Advisor column if any validation or data cleansing rules have been

2 suggested. As shown in the figure below, the golden gear icon indicates suggested validation rules, and the broom icon indicates cleansing advisor rules. Clicking on the Data Validation Advisor (golden gear icon) opens up the list of suggested validation rules, as shown in the image below. These rules typically suggest non-null nor non-zero columns, and sometimes pick up on patterns (for example that a column should have only Y or N). Since this is a small set of fake data, the rules are relatively simplistic, but potentially still useful. The user can see the exact code the rule produces in the bottom pane, along with a summary of the profile data pass rate on the bottom right. Clicking on the blue link for the <Null> values will display example records that would have failed this rule, so the user can review. Clicking on Accept creates a new validation rule and binding, which can be viewed on the Rules tab within the project. This rule can then be updated with any additional business logic that needs to be included.

3 Data Cleansing Advisor The Content Type profiling task generated suggested data cleanse rules. Clicking on the broom icon in the Advisor column on the Profile Results tab opens the Data Cleansing Advisor Wizard, pictured below. After naming the cleansing solution, the wizard displays all of the auto-detected content columns and allows the user to select which should be considered in the cleansing solution. The next portion allows the user to select the matching theme (options include Individual, Family, ID Only, or No Match, if the user does not want to assess matching in the cleansing solution). I was also able to designate that the ID column be used for the Person + ID matching.

4 In the last step of the wizard, I can create additional simple transformations, such as stripping nonprintable characters, leading or trailing spaces, replacing blanks, or removing quotes.

5 Upon clicking Finish, the cleansing solution is created on the Data Cleansing Solutions tab and the cleansing package begins running, as shown in the picture below. Once the cleanse process finishes, I can review a summary of the cleanse and match results:

6 This system does not have address directories installed, so no postal addresses can be validated. Additionally, this data has many duplicates, so each person in the dataset appears more than once (all with different IDs). I can drill into the cleanse results by clicking on View Cleans Results and then picking the column for cleansing. For example, reviewing the results for Phone Number below shows that there are 5 records with Invalid phones. We can drill further and review those records by clicking on the View Data option. This shows us that the phone numbers that were declared invalid will be blanked out by the cleansing package. If we had address cleansing in place, we would be able to review the before and after addresses side-by-side. Back on the cleansing overview, clicking View Match Results shows that 100% of records in the dataset were matched to at least one other record, most of those with a high level of confidence.

7 We can further review the matching results by clicking on View Data and seeing the records that were grouped together, and optionally filter for High and Medium matches.

8 Selecting multiple records and selecting Compare Records opens a side-by-side dialog box that highlights differences between the records in red. Once I have reviewed the cleansing results and am satisfied, I can click Publish Solution, which generates a Data Services Workbench solution on the same CMC instance that Information Steward is using. Authorized Data Services Workbench users can access this cleansing solution. Hillary Bliss, Analytics Practice Lead Decision First Technologies Hillary.bliss@decisionfirst.com Hillary Bliss is the Analytics Practice Lead at Decision First Technologies, and specializes in data warehouse design, ETL development, statistical analysis, and predictive modeling. She works with clients and vendors to integrate business analysis and predictive modeling solutions into the organizational data warehouse and business intelligence environments based on their specific operational and strategic business needs. She has a master s degree in statistics and an MBA from Georgia Tech.