EDQ Product Data Extensions Essentials

Size: px

Start display at page:

Download "EDQ Product Data Extensions Essentials"

Sara Parks
6 years ago
Views:

2 EDQ Product Data Extensions Essentials January, 2015

3 Contents Part 1 Product Data and EDQ s Product Data Extensions: a Conceptual Overview Part 2 Create a Data Lens Using the Knowledge Studio Part 3 Develop a Data Services Application and Integrate it with Enterprise Data Quality Part 4 How to Quickly Create a Starter Data Lens Using AutoBuild

4 Part 1 Product Data and EDQ s Product Data Extensions: a Conceptual Overview

5 Why Use EDQ s Product Data Extensions? Product Data is often stored in an Item Master system, such as the Oracle Product Information Master (PIM). Item Master data needs to be high quality: In the right structure. Consistent in terms of format and content. Complete. Free of duplicates. Search and discovery tools, such as Oracle Endeca, have similar requirements.

6 Item Master Requirements Item Masters require products organized into a logical hierarchy. Let s look at an example: the Oracle Product Information Master (PIM). Products are held in an Item Catalog. The Item Catalog consists of many Item Catalog Categories (ICCs). You can think of each ICC as a product family. Each ICC holds products that have similar characteristics. Each ICC can have several hierarchical levels. E.g.: Computer Parts and Components (The ICC Level 1). Memory (Level 2) SRAM (Level 3) DRAM (Level 3)

7 Item Catalog Category (ICC) - 4 Level Example Office Supplies Writing Instruments Category Family Category Group Pencils Pens Mechanical Pencils Wooden Pencils Ballpoint Pens Fountain Pens Roller Pens Category Line Category

8 About Categories Each Category (Mechanical_Pencils): Could belong to a Category Line (Pencils), which Might belong to a Category Group (Writing Instruments), which Might belong to a Category Family (Office Supplies). Can be classified against a schema: UNSPSC_11_ Mechanical pencils* Can be given: An Item Definition Name (Mechanical_Pencils). An Item Definition Description (Pencil Mechanical Blue Barrel.7MM). A Standardized Description (Mechanical Pencil 0.7 MM Clear Blue Barrel). A set of Attributes (Item Name, Item Type, Barrel Color, Size etc.). *UNSPSC = United Nations Standard Products and Services Code.

9 Items, Categories and Attributes Category and Item are synonymous. Each category has a set of attributes. Each category can have a different set of attributes. For example: Category Mechanical Pencils Wooden Pencils Ball Point Pens Attributes Item Name, Item Type, Barrel Color, Size, Units Per Package Item Name, Item Type, Lead Hardness, Units Per Package Item Name, Item Type, Point Type, Barrel Color, Ink Color, Units Per Package

10 The Data Quality Challenge To go from this: To this:

11 Schemas You may want to classify your items against a pre-existing schema, such as the United Nations Standard Products and Services Code (UNSPSC):

12 Oracle s Data Quality Offering Oracle Enterprise Data Quality (EDQ): Business user friendly. Enables you to work with data from any domain: Understand, check, clean, standardize, parse and match. Ability to build sophisticated data quality processes. EDQ includes Product Data Extensions Specialized set of extensions for classifying, structuring (parsing) and standardizing unstructured fields in multi-category product data.

13 Make up of Product Data Product data typically consists of both structured and unstructured fields.

14 When to Use EDQ Alone, and When to Use the Product Data Extensions Use Oracle Enterprise Data Quality (EDQ) if: You want to understand and improve data from any domain, including singlecategory product data. You want to understand and check multi-category product data (in particular its structured fields). Use the Product Data Extensions if you want to: Classify and structure (parse) unstructured product data fields that contain numerous categories. Prepare product data for a Product Master Data Management Implementation. E.g. Oracle Product Item Master (PIM). Prepare product data for a structured data discovery tool. E.g. Oracle Endeca.

15 Multi-Category Product Data Workflow: Best Practice EDQ Import the Data Profile the Data Standardize Structured Data Sanity Check Data (Audit) Parse Unstructured Multi-Category Data Product Data Extensions Use AutoBuild to create Data Lens. Refine Lens in Knowledge Studio. Create DSA in Application Studio. EDQ Profile the standardized Data Deduplicate the data. Export the Data

16 What do the Product Data Extensions Do? Specialized functionality to deal with multi-category product data: Item identification. Attribute identification and extraction. Standardization. Classification against a schema.

17 Product Data Extensions Development Process Flow

18 Part 2 Create a Data Lens Using the Knowledge Studio

19 Aims of a Data Lens Recognize terms. Recognize phrases. Identify Items. Identify, extract and order attributes. Standardize terms, phrases and attributes. Classify items against a schema.

20 Data Lens Roadmap Recognize Terms and Phrases Define Items and their Attributes Standardize Terms, Phrases and Items Classify Items Against a Schema Configure Your Data Lens for Matching Check-In Your Data Lens Where should I output my results?

21 Recognize Terms and Phrases (1) You configure a data lens by working with sample data to build semantic intelligence. You start with green text nodes: You want to determine which item each row of data relates to. E.g. This row is a Smooth Highlighter, that one is a Mechanical Pencil, the next is a Rollerball Pen and so on. The first step is to recognize terms.

22 Recognize Terms and Phrases (2) A term consists of one or more text nodes that mean the same thing. You can associate a single term with several different text nodes (if they mean the same thing). Known as creating term variants or productions.

23 Recognize Terms and Phrases (3) The next step is to recognize phrases. A phrase consists of one or more terms. Phrases will equip your data lens to recognize item attributes (and so determine items).

24 Recognize Terms and Phrases (4) It is good practice to prefix your phrase names with letters that indicate their eventual use:

25 Define Items and their Attributes (1) Items should be placed in a logical hierarchy. (Remember the Item Master System s requirements.) First, create your hierarchical tree :

26 Define Items and their Attributes (2) Next, tell your data lens how to recognize which rows belong to which item definition. You do this by creating attributes. Attributes can be: Required: Scoring. Optional.

27 Define Items and their Attributes (3) Finally, you associate each attribute with a phrase. Now your data lens knows this row is a smooth highlighter: it has defined the item.

28 Define Items and their Attributes (4) How do required, scoring and optional attributes work? Required attributes: In order for a row of data to be defined as a particular item, it must have that item s required attributes. Scoring attributes: The more scoring attributes a row has, the higher its Quality Index (QI). Items with a low QI may require remediation. Optional attributes: These are possible attributes that do not affect a row s QI. All attributes can be output. Later, you will define the order in which attributes are output.

29 Smart Glossaries The Product Data Extensions include pre-configured smart glossaries. These are libraries of terms* and phrases* that are commonly found in product data. Examples of Product Data Extension smart glossaries: Colors. Units of Measure. Packaging. Using the Smart Glossaries will save you time. *We are using the Product Data Extensions definition of terms and phrases here.

30 Data Lens Roadmap Recognize Terms and Phrases Define Items and their Attributes Standardize Terms, Phrases and Items Classify Items Against a Schema Configure Your Data Lens for Matching Check-In Your Data Lens Where should I output my results?

31 Standardize Terms, Phrases and Items The item master requires data that is standardized. We must not output any of Highlighters or HI-LITERS or hilighter, but always one standard form: Highlighter. You should: Standardize all variants. You can use your existing lists of productions. Standardize case. Set the output order for attributes. This applies to: Terms and Phrases. Item Attributes, Item Definition Name and Description. The Description.

32 Data Lens Roadmap Recognize Terms and Phrases Define Items and their Attributes Standardize Terms, Phrases and Items Classify Items Against a Schema Configure Your Data Lens for Matching Check-In Your Data Lens Where should I output my results?

33 Classify Items Against a Schema (1) You may want to classify your items against a pre-existing schema, such as the United Nations Standard Products and Services Code (UNSPSC):

34 Classify Items Against a Schema (Continued) You can classify each item against a schema. This is done via drag and drop.

35 Data Lens Roadmap Recognize Terms and Phrases Define Items and their Attributes Standardize Terms, Phrases and Items Classify Items Against a Schema Configure Your Data Lens for Matching Check-In Your Data Lens Where should I output my results?

36 Configure Your Data Lens to Enable Deduplication Configure a match standardization type. Set handling for null and multiple instances of attribute values: Set Null Handling to either Allow Null or Replace Null (with suitable replacement text, such as none or N/A ). Set Multiple Instance Handling to Concatenate Unordered, and enter a comma into the adjacent field. Do not select a separator character that will be used in your data. To apply the Null and Multiple Instance Handling rules globally, click Reset All Rules. Create a Match Type.

About Semantic Keys The Product Data Extensions can pass EDQ two fields that can be especially useful for deduplication: Semantic Key Hashed value based on the required attributes for matching for

37 About Semantic Keys The Product Data Extensions can pass EDQ two fields that can be especially useful for deduplication: Semantic Key Hashed value based on the required attributes for matching for each item only. Unpopulated if any required attribute is NULL. Configure required attributes on the Match Weights sub-tab of the Standardize Items tab. Semantic Key 2 Hashed value based on all attributes. Populated even if some attributes are NULL. 37

38 Data Lens Roadmap Recognize Terms and Phrases Define Items and their Attributes Standardize Terms, Phrases and Items Classify Items Against a Schema Configure Your Data Lens for Matching Check-In Your Data Lens Where should I output my results?

39 Check-In Your Data Lens Your Data-Lens will only be available to Data Service Applications (DSAs) once it has been checked in.

40 Part 3 Develop a Data Services Application and Integrate it with Enterprise Data Quality

41 Product Data Extensions Use data lens: Process Orchestration Data Service Application orchestrates the DQ process as a callable service Overall Process Orchestration Sub-processes Pre-built functions Category Knowledge & Standards 41

42 Creating a Data Services Application (DSA) 42

43 Creating a Data Services Application (DSA) 43

Creating a Data Services Application (DSA) In the Item Definition

attributes to pass back to EDQ, and specify how they should be

At a minimum, ensure you pass back the following information to EDQ:

44 Creating a Data Services Application (DSA) In the Item Definition Transformation, you embed a Data Lens in the DSA, select which attributes to pass back to EDQ, and specify how they should be standardized. At a minimum, ensure you pass back the following information to EDQ: Item Definition Name Attribute Name Attribute Value Semantic Key Semantic Key 2 Standardized Description Classification information 44

45 Creating a Data Services Application (DSA) 45

46 Creating a Data Services Application (DSA) Your DSA will only be available to an EDQ process once it has been checked in. 46

47 Embed a Product Data Application Within an EDQ Process Product Data Services Application (DSA) called from an EDQ Process via the Process Product Data processor. The EDQ process can include deduplication and profiling.

48 Deduplicating Parsed Product Data in EDQ 48

49 Deduplicating Parsed Product Data in EDQ 49

50 Deduplicating Parsed Product Data in EDQ 50

51 Deduplicating Parsed Product Data in EDQ 51

52 Deduplicating Parsed Product Data in EDQ See EDQ Matching Essentials for more information about how to configure deduplication. 52

53 Use Standard EDQ Functionality to Export the Parsed and Deduplicated Product Data 53

54 Part 4 How to Quickly Create a Starter Data Lens Using AutoBuild

55 Product Data Extensions Development Process Flow With AutoBuild

56 Autobuild: Reverse-Engineer a Data Lens from a Sample of Structured Product Data You may have a sample of product data that is already parsed and well-structured, ready for a product hub.

Autobuild Methodology Data Gold data (from MDM, item master etc.

Harvest data samples, metadata & standards from across the business 2.

documentation Governance standards Supplier SLAs Etc.

Load into Data Lens System as Item Definition 5.

Put into production and continue to learn Legacy Systems ERP, PLM, Item

57 Autobuild Methodology Data Gold data (from MDM, item master etc.) Raw data (datafeeds etc.) 1. Harvest data samples, metadata & standards from across the business 2. Add knowledge from Oracle and other sources Company Standards System documentation Governance standards Supplier SLAs Etc. Category Hierarchy Refine data lens 3. Load into Excel for review 4. Load into Data Lens System as Item Definition 5. Use raw data to recognize & learn variations 6. Put into production and continue to learn Legacy Systems ERP, PLM, Item master etc. Lookup tables, X-ref tables Search system Website SMEs Tribal knowledge New requirements/changes Attributes Values Semantic Model (Item Definition) Knowledge Industry Standards Industry best practice examples DataLens Smart Glossaries 57

58 AutoBuild Structured Data Input Category Columns Attributes as column headers Categories associated with attributes AutoBuild requires already-structured product data as its input: Multi-column Category names Attribute names in the header row Vertical list of attributes Valid Values expressed in full form (unabbreviated)

59 Use AutoBuild to Create a New Data Lens Create a new data lens Extract and capture the taxonomy* (hierarchy or the structure of the information) as you fill in the Category Information in the AutoBuild template Extract and capture the attribute name and values to associate with the hierarchy in the AutoBuild template to complete the Semantic Model Leverage a Smart Glossary to provide horizontal knowledge to jump start the recognition. Smart Glossaries, for example for units of measure, colors, and so son, are part of the Product Data Extensions. * Taxonomy an item classification system that includes the definition of category specific attributes

60 AutoBuild Generate data lens Generate Item definitions Attributes Phrases and terms Attribute to phrase associations Leverage Smart Glossary

61 Refine Your Data Lens in the Knowledge Studio Remove Any Remaining Ambiguities Scan sample data files for ambiguities Resolve ambiguities using rule merge and refactoring Adjust Standardization Use full form to standardize terms Include variants of full forms to improve recognition Confirm, order and refine extracted attributes Include additional Unit conversions if necessary Setup match rules if necessary Quality Assurance Create and validate lens level regression test sets for each production standardization Deploy to development and regression test against defined test cases

Oracle Product Data Quality

Oracle Product Data Quality PIM Connector User's Guide Version 5.6.1 E23405-01 April 2011 Oracle Product Data Quality PIM Connector User's Guide, Version 5.6.1 E23405-01 Copyright 2001, 2011 Oracle and/or