Configuration for Registry*, Repository or Hybrid** Master Data Management. *aka Federated or Virtual **aka Coexistence

Size: px

Start display at page:

Download "Configuration for Registry*, Repository or Hybrid** Master Data Management. *aka Federated or Virtual **aka Coexistence"

Maud Perkins
6 years ago
Views:

1 Configuration for Registry*, Repository or Hybrid** Master Data Management *aka Federated or Virtual **aka Coexistence October 2017 (Best viewed in slideshow mode) Revision 3.6 Copyright 2017 WhamTech, Inc. 1

2 Seven keys to successful MDM Data quality Data/entity relationships Optimal merge bin size and process Data access Real-time updates Data governance Wildly successful MDM Seamless integration with applications Ultimate Goal Revision 3.6 Copyright 2017 WhamTech, Inc. 2

3 Conflicting goals with bins Bins large enough to evaluate all possible contributing data Bins small enough to limit n x (n-1) editdistance calculation overhead Revision 3.6 Copyright 2017 WhamTech, Inc. 3

4 Binning Methods Currently, expand fuzzy LAST NAME + DOB with high cardinality entity matches from Link Tend towards composite-weighted multi-attribute probabilistic match Revision 3.6 Copyright 2017 WhamTech, Inc. 4

5 How to retain a master data record (MDR)? Master Data Record Template CUSTOMER RECNO LAST_NAME FIRST_NAME KNOWN_AS ACCOUNT_NO PHONE_BEST ADDRESS CITY ZIP STATE EFF_DATETIME MDR.CUSTOMER RECNO bsmith@gmail.com 300 Any St. New Town AL 1/1/ :32 Master Data Record - based on latest and missing data Centralized or distributed? Repository or registry? DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 Revision 3.6 Copyright 2017 WhamTech, Inc. 5

7 WhamTech DDVP can accommodate any combination 100% Distributed 100% Centralized WhamTech MDM solution is 100% distributed, 100% registry and limited repository no additional data to accommodate = INTEGRATED + HIGH PERFORMANCE + LEAN A practical MDM solution would be 100% centralized and 100% repository, but difficult to connect to source data = SEPARATE An ideal MDM solution would be 100% distributed and 100% registry, but difficult to implement and performance issues = DIFFICULT + LOW PERFOMANCE Typical MDM solution is 100% centralized, 100% repository, limited registry and additional data to accommodate = SEPARATE + BLOAT 100% Repository 100% Registry Revision 3.6 Copyright 2017 WhamTech, Inc. 7

8 Benefits of a Distributed Hybrid* MDM Approach Distributed Data security and privacy, e.g., PII, master data stays in the same environment as sources Adapter-level rules can integrate master data automatically no application rewrites Local processing - fast and secure Use master data at multiple levels Add/remove data source adapters with little or no impact No central bottleneck or dependencies Can still be centrally managed and consolidated Limited Repository Best master data may not exist as such in data sources Master data immediately available avoids (1) access to multiple sources and (2) source data transformation Allows retention of historic master data Full Registry Updateable Traceable Avoids repository for some master data *Hybrid = Limited Repository and Full Registry Revision 3.6 Copyright 2017 WhamTech, Inc. 8

10 Basic WhamTech DDVP Configuration Applications* *Applications include normal operational, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model EIQ Federation Server EIM EIQ Federation Server EIM EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter EIQ Conventional Adapter EIM EIM EIM EIM Data Source Data Source Data Source Data Source Revision 3.6 Copyright 2017 WhamTech, Inc. 10

11 Master data available at all levels update and propagate Applications* *Applications include normal operational, MDM, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model EIQ Federation Server Seamless integration of master, operational/transactional and other data EIQ Federation Server EIQ HybridAdapter EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter Conventional Queries EIQ Queries MDR MDR MDR MDR MDRs MDRs MDRs MDRs Data Source Data Source Data Source Data Source Revision 3.6 Copyright 2017 WhamTech, Inc. 11

12 Master data available for all access levels EIQ SuperAdapter EIQ HybridAdapter EIQ HybridAdapter EIQ SuperAdapter MDR Conventional Queries EIQ Queries Conventional Queries EIQ Queries MDRs MDR Data Source Changed Data Capture or Data Source Polling Partial MDR MDR MDRs MDRs MDRs Data Source Data Source Data Source Data Source Data Schema-level Access < Results-level Access > Revision 3.6 Copyright 2017 WhamTech, Inc. 12

13 The End Appendices What is distributed hybrid master data? How to manage distributed hybrid master data? How to use distributed hybrid master data? How do WhamTech DDVP and MDM processes combine? Two main forms of MDR repository and registry Hybrid master data record creation Revision 3.6 Copyright 2017 WhamTech, Inc. 13

15 Distributed hybrid master data description (1 of 2) Master data is either an, or part, of the Enterprise Information Model (EIM) Master data defined separately from normal content data in EIM, or Normal content data in EIM also tagged as master data, which may also be tagged as link entity data One or more master data entities are defined and used to find any and all associated source and external data Example master data entities are PERSON (e.g., patient, doctor, nurse and administrator), ORGANIZATION (e.g., hospital, lab, insurance co., insured co. and healthcare provider), ADDRESS, PHONE, , SYMPTOMS, AILMENT, TREATMENT, OUTCOME, etc. Master data, e.g., PERSON, can comprise multiple attributes that may also be entities, e.g., LNAME, FNAME, MNAME, DOB, SSN, PHONE, and ADDRESS, aka a complex entity Any source data or master data can be masked, tokenized or encrypted, either dynamically or in indexes, including format-preserved encryption (FPE) combination of tokenization and encryption EOS Revision 3.6 Copyright 2017 WhamTech, Inc. 15

16 Distributed hybrid master data description (2 of 2) Access to data in general, or specific data, may depend on role-based access control (RBAC) Dynamic masking, tokenization or encryption may also depend on RBAC Encryption keys can be specific to an individual entity or entities, e.g., PERSON or others, and may be separately passed from result-sets to the recipient(s) for decryption Hybrid master data table contains values, any phonetic tokens (for fuzzy match), links to records containing values and date-time of when values were either added to data sources, indexes (assuming near real-time indexing) or master data table Allows for rapid real-time updates as master data tables are all that is needed no need to query data sources All entries in master data tables are indexed, queryable and can be joined to normal content indexes and Link for a seamless integration of master data and source data EOS Revision 3.6 Copyright 2017 WhamTech, Inc. 16

18 Managing distributed hybrid master data (1 of 8) Applications* *Applications include normal operational, BI and analytics DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter EIQ Federation Server 3. Link mapping confirms that there are no other probableweighted multi-attribute composite matches on other data sources, i.e., this appears to be a new customer to the organization 2. indexes are updated 1. An apparently new customer interacts with one of the organization s contact points and a record is created in data source 1 Revision 3.6 Copyright 2017 WhamTech, Inc. 18

19 Managing distributed hybrid master data (2 of 8) Applications* *Applications include normal operational, BI and analytics 5. The unique master customer ID is indexed and associated with the content index for the customer record EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter EIQ Federation Server MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 MDR.CUSTOMER RECNO Main St. Any Town TX 1/1/ :26 4. A MDR with an unique master customer ID is created, stored and indexed in the EIQ Index layer for data source 1 no other data sources Revision 3.6 Copyright 2017 WhamTech, Inc. 19

20 Managing distributed hybrid master data (3 of 8) 9. The unique master customer ID index for the existing customer is associated with the customer index for the new record MAST_CUST_ID = 323 EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter MDR Applications* EIQ Federation Server *Applications include normal operational, BI and analytics 8. Link mapping confirms a match with the existing record, that there are no other matches in other data sources and Link are updated DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO Main St. Any Town TX 1/1/ :26 7. indexes are updated 6. One year later, the same customer interacts with data source 1 again, but is not governed as an existing customer Revision 3.6 Copyright 2017 WhamTech, Inc. 20

21 Managing distributed hybrid master data (4 of 8) Applications* *Applications include normal operational, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter EIQ Federation Server 10. The MDM merge process updates the MDR MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO Any St. New Town AL 1/1/ :15 Revision 3.6 Copyright 2017 WhamTech, Inc. 21

22 Managing distributed hybrid master data (5 of 8) Applications* *Applications include normal operational, BI and analytics 13. Link mapping confirms that there are two record matches in data source 1, and Link for data sources 1 and 2 are updated EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter EIQ Federation Server 12. indexes for data source 2 are updated 11. One year later, the same customer interacts with a different customer touch-point EIQ SuperAdapter and EIM with a MDM new record is created in data source 2 MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO Any St. New Town AL 1/1/ :15 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 Revision 3.6 Copyright 2017 WhamTech, Inc. 22

23 Managing distributed hybrid master data (6 of 8) Applications* *Applications include normal operational, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model 14. The MDR associated with the two linked records in data source 1 is read and merged with the new record in data source 2 EIQ SuperAdapter EIQ Federation Server EIQ SuperAdapter MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO Any St. New Town AL 1/1/ :15 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 Revision 3.6 Copyright 2017 WhamTech, Inc. 23

24 Managing distributed hybrid master data (7 of 8) Applications* *Applications include normal operational, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model EIQ SuperAdapter EIQ Federation Server 16. The unique master customer ID is indexed and associated with the customer index for the new record 15. The MDM merge process creates a MDR in data source 2 EIQ SuperAdapter MAST_CUST_ID = 323 MDR MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO Any St. New Town AL 1/1/ :15 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 MDR.CUSTOMER RECNO bsmith@gmail.com 300 Any St. New Town AL 1/1/ :32 Revision 3.6 Copyright 2017 WhamTech, Inc. 24

25 Managing distributed hybrid master data (8 of 8) Applications* *Applications include normal operational, BI and analytics EIM = Enterprise Information Model aka Standard/Common Data Model 17. The MDR in data source 1 (and any other data sources) is updated EIQ SuperAdapter EIQ Federation Server EIQ SuperAdapter MAST_CUST_ID = 323 MDR MAST_CUST_ID = 323 MDR DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 MDR.CUSTOMER RECNO bsmith@gmail.com 300 Any St. New Town AL 1/1/ :32 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 MDR.CUSTOMER RECNO bsmith@gmail.com 300 Any St. New Town AL 1/1/ :32 Revision 3.6 Copyright 2017 WhamTech, Inc. 25

27 Using distributed hybrid master data (1 of 2) Query master data indexes and tables to access and read source data Typically, results are a combination of source and a small amount of master data, where source data is replaced with its respective master data Optionally, use source data only - do not replace any of it with its respective master data Optionally, use both source and its respective master data Regardless of source/master data combination, consolidate multiple data source results at higher federation levels Master data essential for almost any meaningful BI and analytics for virtual single patient index, other entity-centric views, marketing, CRM, etc. Query master data indexes to manage master data, run BI/analytics, and/or populate a higher-level master data repository Automatically work with existing applications using query transforms no application rewrites Revision 3.6 Copyright 2017 WhamTech, Inc. 27

28 Using distributed hybrid master data (2 of 2) Seamlessly and transparently integrate any and all data, including master, mainframe, operational, warehouse, cloud, Big, Fast, public, government, etc. Include data sources without WhamTech indexed adapters through WhamTech and other vendor conventional federated adapters Have to manage separate, but associated, master data tables and indexes Use results directly as is with Link for degrees of separation and link queries, link analysis/graph database representation and highly interactive visualization, for all types of analytics Master data is required to represent multiple records of same entities as single nodes, e.g., PERSON, VEHICLE, ADDRESS, ORGANIZATION, etc. Alleviates manual involvement tends towards automation Revision 3.6 Copyright 2017 WhamTech, Inc. 28

30 WhamTech DDVP and MDM Processes (1 of 3) Automate as a much as possible: Network asset/device discovery Data source discovery Data discovery Optionally, with raw Link to automatically connect same data in multiple data sources Data profiling to develop data transforms for typos, transpositions and nonstandard data, e.g., name, address, phone and correction Lookup dictionaries and thesauri USPS and/or other address correction Regular expressions Revision 3.6 Copyright 2017 WhamTech, Inc. 30

31 WhamTech DDVP and MDM Processes (2 of 3) Multiple indexes and types, e.g., basic content, aggregations, calculations, fuzzy, text, extracted entities, indexed views and Link MDM: Data source-specific tables containing unique indexed primary entity IDs, master data (including phonetic representations), links to source master data and date-time Create with Link, multi-attribute fuzzy match and master data rules Option of pure registry-based master data table, but limits options and more complex Hierarchies honored through joins and/or Link Inferred ontologies Reasons for hierarchies change depending on application, e.g., one vendor has multiple products and one product from multiple vendors MDM: Master data versioning with access to historic master data Revision 3.6 Copyright 2017 WhamTech, Inc. 31

32 WhamTech DDVP and MDM Processes (3 of 3) MDM: Option to replace either data source indexes or source data itself (automatically updates indexes) with master data Execute analytics and/or visually represent master data combined with other data and search/query filters, e.g., BI and link analysis/graph database Include aggregations, calculations and other data, e.g., external MDM: Write back selective master data updates/corrections to data sources (see above) Continuous access to metadata, including latest data profiles Helps identify anomalies/outliers for event processing, alerts and/or data transform modifications Revision 3.6 Copyright 2017 WhamTech, Inc. 32

34 How to retain a master data record (MDR)? Master Data Record Template CUSTOMER RECNO LAST_NAME FIRST_NAME KNOWN_AS ACCOUNT_NO PHONE_BEST ADDRESS CITY ZIP STATE EFF_DATETIME MDR.CUSTOMER RECNO bsmith@gmail.com 300 Any St. New Town AL 1/1/ :32 Master Data Record - based on latest and missing data DS1.CUSTOMER RECNO Main St. Any Town TX 1/1/ :25 DS1.CUSTOMER RECNO Any St. New Town AL 1/1/ :14 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 Revision 3.6 Copyright 2017 WhamTech, Inc. 34

35 Two main forms of MDR repository and registry Master Data Model (Repository) Master Data Model (Registry) Data Source Record(s) Master Data Record (Repository) Master Data Record (Registry) MDR.RECNO MDR.RECNO_EFF_DT CUSTOMER MAST_CUST_ID CUST_TYPE LAST_NAME LST_NAM_MAST_URN LST_NAM_EFF_DT FIRST_NAME FST_NAM_MAST_URN FST_NAM_EFF_DT KNOWN_AS KNOWN_AS_MAST_URN KNOWN_AS_EFF_DT PHONE_PREFERRED PHONE_TYPE PHONE_NO PHONE_MAST_URN PHONE_EFF_DT _PREFERRED _TYPE _MAST_URN _EFF_DT ADDRESS_MAILING ADD_TYPE ADDRESS1 ADDRESS2 CITY ZIP STATE ADD_MAST_URN ADD_EFF_DT CUSTOMER MAST_CUST_ID CUST_TYPE LST_NAM_MAST_URN LST_NAM_EFF_DT FST_NAM_MAST_URN FST_NAM_EFF_DT KNOWN_AS_MAST_URN KNOWN_AS_EFF_DT OPTIONAL MDR.RECNO MDR.RECNO_EFF_DT PHONE_PREFERRED PHONE_TYPE PHONE_MAST_URN PHONE_EFF_DT _PREFERRED _TYPE _MAST_URN _EFF_DT ADDRESS_MAILING OADD_TYPE ADD_MAST_URN OADD_EFF_DT Note: A minimal MDR could be an integer list or set of bitmaps consisting of ones and zeros. DS1.CUSTOMER RECNO Home Main St. Any Town TX 1/1/ :25 DS2.CUSTOMER RECNO bsmith@gmail.com 1/1/ :31 DS1.CUSTOMER RECNO Mobile 300 Any St. New Town AL 1/1/ :14 MDR.RECNO01 1/1/ :32 CUSTOMER 323 Web only DS1.RECNO04 1/1/ :25 DS1.RECNO04 1/1/ :25 DS2.RECNO12 1/1/ :31 PHONE_PREFERRED Mobile DS1.RECNO12 1/1/ :31 _PREFERRED bsmith@gmail.com DS2.RECNO09 1/1/ :31 ADDRESS_MAILING 300 Any St. New Town AL DS1.RECNO12 1/1/ :14 MDR.RECNO01 CUSTOMER DS1.RECNO04 DS1.RECNO04 DS1.RECNO12 PHONE_PREFERRED DS1.RECNO12 _PREFERRED DS2.RECNO09 ADDRESS_MAILING DS1.RECNO12 Revision 3.6 Copyright 2017 WhamTech, Inc. 35

36 Advantages and disadvantages of conventional REPOSITORY MDM ADVANTAGES Single best version of master data Queryable DISADANTAGES Applications need to be modified to use master data difficult, cost and time questionable success Complex data model maintenance and modification Single (typically) relational data models do not serve many applications multiple different materialized views required Difficult to combine master data with nonmaster data, in particular, operational/transactional data Revision 3.6 Copyright 2017 WhamTech, Inc. 36

37 Advantages and disadvantages of conventional REGISTRY MDM ADVANTAGES Use of keys and link tables to point to master data in data sources avoids ETL and synchronization Normal applications indirectly maintain master data not a separate external application Difficult, but possible that existing applications automatically use and substitute master data New applications can be written to take advantage of registry MDM easier than with repository MDM DISADANTAGES Requires access to multiple data sources to retrieve master data Query loads on adapters and data sources Bandwidth and query performance Dirty and potentially unusable non-standard master data requires additional processing Pointers tend to be limited to single records instead of multiple attributes in multiple records Difficult to maintain historic master data Revision 3.6 Copyright 2017 WhamTech, Inc. 37

39 Hybrid master data record creation (1 of 2) Normal content indexes are generated using data quality transforms, including address correction In the future, may include initial binning key index, e.g., for PERSON, LAST_NAME_FUZZY + DOB Fuzzy match indexes are generated using various algorithms, including latest Metaphone 3 for international names Link are generated using designated entities in the EIM to capture relationships among similar content data, i.e., same-table self-joins, internal PK-FKs and external joins, on simple entities Exact matches on very high cardinality entities, e.g., ADDRESS, SSN, PHONE and future test for transpositions In the future, exact matches on binning key, e.g., for PERSON, LAST_NAME_FUZZY + DOB For complex entities, use binning and edit-distance algorithms In the future, may avoid some or all edit-distance algorithms Use Link to see if one or more very high cardinality entities match records across data sources, and if so, pull records into bin and re-bin based on high cardinality entity matches If no very high cardinality records, run edit-distance algorithms without just more resource intensive Revision 3.6 Copyright 2017 WhamTech, Inc. 39

40 Hybrid master data record creation (2 of 2) Run edit-distance algorithms within each bin and composite score for matching probabilities, e.g., PERSON based on names and other entity attributes DOB, SSN, PHONE, and ADDRESS If the composite score is above a certain threshold: Unique ID is assigned to the primary master entity, e.g., PERSON Unique ID is virtually indexed for each occurrence of the primary master entity in all indexes for all data sources A master data record is created in a separate master table in each adapter containing the PERSON and the unique ID is stored and indexed in the master data table Run a process, aka MDM merge process, to determine the best master data values for each PERSON ID Write best master data values back to every master data table, along with any associated data, e.g., any phonetic token, links to records containing values and date-time Automatically index all master data Revision 3.6 Copyright 2017 WhamTech, Inc. 40

Distributed Hybrid MDM, aka Virtual MDM Optional Add-on, for WhamTech SmartData Fabric

Distributed Hybrid MDM, aka Virtual MDM Optional Add-on, for WhamTech SmartData Fabric Revision 2.1 Page 1 of 17 www.whamtech.com (972) 991-5700 info@whamtech.com August 2018 Contents Introduction... 3