SmartData Fabric (SDF) aka SmartData Lake (SDL) aka Distributed Data Virtualization (DDV) Basic Overview

Transcription

1 Smart Fabric (SDF) aka Smart Lake (SDL) aka Distributed Virtualization (DDV) Basic Overview October 2018 Revision 8.6 Copyright 2018 WhamTech, Inc. 1

2 Smart Fabric Unique indexed adapter-based data virtualization, federation and integration for: Advanced data access and data security seen as a security solution discovery, quality, standardization, governance and relationships mapping Virtual data warehouse and/or virtual data mart lake + data management + master data management = data reservoir provisioning for highly curated, self-serve reporting, BI and analytics Interoperability with write-back to data sources not just app to app Seamless, automatic and near real-time updateable distributed master data management Virtual graph database and link analysis, and interactive graph/link visualization Standards such as ODBC, JDBC, REST APIs and SQL, and standard applications Revision 8.6 Copyright 2018 WhamTech, Inc. 2

3 Smart Fabric (SDF) Transparent virtual distributed data management layer that plugs-and-plays in existing IT infrastructures Complements and leverages existing IT systems, tools and applications Key differentiator: Federated adapters that Read, Transform and Index (RTI) data, wherever it resides, process queries against these indexes, and read and transform results data from sources Addresses upfront data discovery, security, quality, standards, MDM and other data-related processes Leaves and guards data in sources, copies (e.g., data lake) or stored in indexes a hybrid approach ORGANIZATION B EXTERNAL COMMERCIAL/PUBLIC DATA SOURCE ORGANIZATION A = SDF Federation Server = SDF Adapter = SDF CLOUD DATA SOURCE ORGANIZATION C Governance ORGANIZATION S OWN SYSTEMS OF REFERENCE DS1 DS2 DS3 DS4 DS5 DS6 DS7 DS8 DS9 DS10 DS11 DS12 Revision 8.6 Copyright 2018 WhamTech, Inc. 3 ETL MDM

4 Smartphone CRM app invoking new BPM-based workflows with write-back to legacy systems through standard APIs/data services Patient-centric smartphone app interacts with legacy data sources through new workflows developed and orchestrated by BPM software BPM workflows interact with data source through standard FHIR REST APIs provided as data services BPM workflows both read and write back to legacy data sources Public Cloud Applications Applications EIQ Adapter EIQ Adapter EIQ Federation Server EIQ Adapter Applications Applications Web Server BPM Workflow Web Server FHIR REST APIs Web Server EIQ Federation Server Applications Applications Hybrid Cloud 2.0 Applications EIQ Federation Server Applications EIQ Adapter EIQ Adapter Applications Patient Management EHR Type 1 Labs Hybrid Cloud 1.0 sources remain on premise Local Smart Fabric deployed on premise sources remain on premise Local Smart Fabric deployed on Cloud Applications EIQ Federation Server Applications EIQ Adapter EIQ Adapter Applications Patient Management EHR Type 2 Applications Patient Management EHR Type 3 ORGANIZATION A ORGANIZATION B ORGANIZATION C Revision 8.6 Copyright 2018 WhamTech, Inc. 4

5 New paradigm read, transform and index (RTI) ETL is No Longer King, Long Live SDD (aka RTI) - How to Close the Loop from Discovery (data) to Information (value-added data) to Insights (analytics) to Outcomes (business improvements) Paper presented at DATAVERSITY Architecture Summit in Chicago, IL, in November 2017 DATA analytics DATA integration Master DATA management DATA relationships DATA management The increasing value of DATA Raw DATA Revision 8.6 Copyright 2018 WhamTech, Inc. 5

6 are powerful as they enable processes and drive value Build and maintain DATA PROFILES using raw indexes for data discovery (metadata), data matching within and across data sources, and developing and testing data transforms Support FORRESTER ZERO TRUST DATA SECURITY FRAMEWORK discover, INDEX, classify and secure PCI, PHI, PII, GDPR, etc. PRE-PROCESS DATA to build and maintain production indexes to address data management fundamentals data usually discarded Index POINTERS REFER TO DATA in sources, copies, e.g., Lake, or stored in indexes pointers retained in results data = full traceability Enable HIGH PERFORMANCE, DISTRIBUTED PARALLEL QUERY PROCESSSING through standard drivers, APIs, Web/data services, SQL and other query languages MONITOR DATA SOURCES for content and relationships in near real-time, and process events Enable VIRTUAL GRAPH DATABASE, link analysis and graph/link visualization EOS Revision 8.6 Copyright 2018 WhamTech, Inc. 6

7 SDF EIQ Adapter index and query process DISCOVERY INITIAL INDEX BUILD CONTINUOUS INDEX UPDATE QUERY PROCESSING RESULTS RETRIEVAL BI / Analytics / Application(s) STANDARD DRIVER SQL Query Transform EIQ Product front-end Application to Standard View Mapping EIQ Federation Server EIQ Adapter Standard View Mapping to EIQ DEVELOP and TEST Profiler Read Transform Index (RTI) Tool USED BY BUILD Discovery Automatic Query Processing EIQ Transforms/clean-ups Update Server Result-set data source pointers Retrieval source-specific Other data source EIQ Adapters and EIQ Federation Servers EIQ Adapter EIQ Federation Server CONVENTIONAL DRIVER OR BULK LOAD USER API / DRIVER MESSAGE QUEUE Source Transaction Log Source Revision 8.6 Copyright 2018 WhamTech, Inc. 7

8 Basic Smart Fabric architecture Applications WhamTech Smart Fabric (SDF) ADMINISTRATION/ CONFIGURATION Virtual network, data source and advanced data discovery Standard drivers, APIs, Web/data services, SQL and other query languages Virtual data security Virtual event processing Virtual cybersecurity Virtual MDM WhamTech EIQ Federation Servers Virtual reporting, BI and analytics WhamTech EIQ Adapters (indexed and conventional federated) Independent structured and unstructured, and indexed views preprocessing (read, but not usually stored) Virtual interactive link analysis and visualization ACCESS SECURITY Optional Lake Centralized or Distributed Sources Mainframes bases Files Logs Office docs Applications Web docs Social media Big Streaming Cloud IoT Revision 8.6 Copyright 2018 WhamTech, Inc. 8

9 Smart Fabric Keyword Descriptions (1 of 2) DISTRIBUTED aka FEDERATED (not centralized) VIRTUAL (leaves data where it resides uses federated adapters) SECURITY (data and access to it) DATA MANAGEMENT (data discovery, classification, security, processing/analytics, cleansing, transformation, standardization, mapping to a standard data view, linking/matching within and across data sources, indexing and query processing) MASTER DATA MANAGEMENT (hybrid [limited repository + full registry] near realtime distributed, seamless and automatic integration with data access) ANALYTICS (built-in, externally run against and highly curated data provisioning for) INDEPENDENT (of where data resides and associated systems, and configurations) INTEROPERABLE (can write back to data sources) Revision 8.6 Copyright 2018 WhamTech, Inc. 9

10 Smart Fabric Keyword Descriptions (2 of 2) INDEXES SYNCED TO DATA SOURCES IN REAL-TIME TO BATCH (twelve change data capture options) VIRTUAL GRAPH DATABASE (semantic model) VIRTUAL LINK ANALYSIS (find connections between entities, n degrees of separation) GRAPH/LINK VISUALIZATION (highly interactive thin client, OEM tool) RUNS IN CLOUD, ON PREMISE, IN DATA CENTERS OR AS HYBRID (including Hybrid Cloud 2.0) EVENT PROCESSING (as indexes and index views are being updated) ULTIMATE METADATA MANAGEMENT (complete on all data) BUILDS AND SUPPORTS DATA GOVERNANCE (bottom-up/edge-in) EOS Revision 8.6 Copyright 2018 WhamTech, Inc. 10

11 SDF better than or as good as alternatives No. Feature Smart Fabric Warehouse Conventional Federated Search 1 Minimal time to implement and add new sources 2 Relatively low cost and high ROI 3 Flexibility of use 4 Actively monitor data sources ( ) or 5 Unstructured data 6 Unlimited query options and performance ( )* 7 Denormalized views ( )* ( ) 8 Relationship/link mapping or or 9 Write back to data sources or 10 No major schema transforms or 11 source changes readily accommodated 12 Full text search *with data marts Revision 8.6 Copyright 2018 WhamTech, Inc. 11 Lake

12 SDF combines best of and overcomes worst of alternatives No. Feature Smart Fabric Warehouse Conventional Federated Search 13 Clean and usable data 14 Consistent and multiple indexes and types ( ) 15 Install nothing on data source systems 16 Pre-aggregated, calculated and join views 17 Results when data sources unavailable or 18 Row, column and data element security ( ) 19 Structured data or 20 stays in original format 21 remains in source or 22 User-level access to source data 23 Latest data available 24 Drill-down capability ( ) Lake Revision 8.6 Copyright 2018 WhamTech, Inc. 12

13 SDF slightly disadvantaged compared to alternatives No. Feature Smart Fabric Warehouse Conventional Federated Search 25 No index or query load on data sources ( ) 26 source owners not aware of queries ( ) 27 Archive options ( )* ( ) 28 Good for application data sources ( ) 29 Minimal additional system cost ( ) 30 No need for data or index update process *Can store and index either in own or third-party database: 1. Changed data for archive 2. Derived data (aggregations, calculations) 3. Any designated data Can index original format archived data, e.g., mainframe files stored for SOX compliance Lake Revision 8.6 Copyright 2018 WhamTech, Inc. 13

14 F I R E W A L L F I R E W A L L Smart Fabric security-centric distributed virtual data, master data and graph data management, and analytics Example multiple data source SDF configuration RDBMS Social Media Feed EIQ SuperAdapter EIQ SuperAdapter EIQ Federation Server Adapters and federation servers independently configurable and accessible at multiple levels Potential LIFO/FIFO query processing Hadoop EIQ SuperAdapter Mainframe ERP System EIQ Conventional Adapter EIQ SuperAdapter EIQ Federation Server EIQ Federation Server TCP / IP WhamTech ODBC/JDBC Driver, APIs, Web/data services Application(s) Salesforce 3 rd Party Adapter Revision 8.6 Copyright 2018 WhamTech, Inc. 14

15 Example shared-nothing architecture Out-of-the-box configurable backup, failover and load balancing = high availability Application(s) EIQ Federation Server can be multiple sharded segments or replicated copies EIQ Federation Server EIQ Federation Server EIQ Federation Server EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter EIQ SuperAdapter Source Revision 8.6 Copyright 2018 WhamTech, Inc. 15

16 Initial EIQ Adapter configuration, index build and data view mapping Discovery and raw indexbased Profiling Metadata Discovery and Semantic Mapping Classification and Security Develop and test Transforms using profiles Distributed Metadata Repository, incl. Governance Source Discovery Network Asset and Device Discovery Source Read, Transform/ clean-up (and Index) EIQ EIQ Adapter* w/sdv** Automated Discovery and Classification (ADDC) thus far Twelve ways to build and maintain indexes Index schema and names usually same as data source usually do not store data only queryable representations Alternate use of raw indexes to initially build EIQ mapped to SDV *EIQ SuperAdapter and EIQ TurboAdapter **Standard View Revision 8.6 Copyright 2018 WhamTech, Inc. 16

17 EIQ Adapter index update, query and results retrieval Distributed Metadata Repository, incl. Governance Queries resolved in the EIQ Adapter and EIQ Result-set pointers to data in source Source Read, Transform/ clean-up (and Index) User-level access Continual EIQ updates EIQ Federation Server EIQ Multiple other data sources EIQ EIQ Federation EIQ Server Server Adapter* (sub- (submiddleware) w/sdv** Middleware) w/sdv Raw results data usually transformed/cleaned-up from source Applications / middleware connect with standard drivers or Web Services and SQL*** Middleware Application(s) Results provided in almost any format *EIQ SuperAdapter and EIQ TurboAdapter **Standard View ***Future OQL, SPARQL and NoSQL options Revision 8.6 Copyright 2018 WhamTech, Inc. 17

18 Smart Fabric for data federation/abstraction Discovery Profiling Classification Categorization Access and Security Cleansing Transformation Standardization Masking Tokenization Encryption Indexing (data discarded) Standard data view mapping Indexed views BI, analytics, CRM and BPM support Link mapping/ indexing Master Management (MDM) Event processing Event correlation [Anomaly detection] High performance parallel query processing Integration (results read only) Interoperability (results read and write) Link Analysis/Graph base WhamTech key differentiators addressed upfront Revision 8.6 Copyright 2018 WhamTech, Inc. 18

19 Enterprise conventional data life-cycle Mart (DM)/ Analytics base/ Link Analysis- Graph base Acted on Analyzed Reported Related Indexed Stored Quality/improved Discarded/retained Copied Indexed Stored Created Stored Copied Quality/improved Stored Indexed Copied Quality/improved copied multiple times Log/Transaction System Operational Store (ODS) Warehouse (ETL and DW) Revision 8.6 Copyright 2018 WhamTech, Inc. 19

20 Big life-cycle similar to enterprise data life-cycle Big / Analytics base/ Link Analysis/ Graph base Acted on Analyzed Reported Related Indexed Stored Quality/improved Discarded/retained Copied Indexed Stored Created Stored Copied x 3 Quality/improved Stored Indexed Copied Quality/improved copied multiple times Log/Transaction System Big Lake/Reservoir (similar to ODS) Big Refinery (similar to ETL) Revision 8.6 Copyright 2018 WhamTech, Inc. 20

21 SDF eliminates most conventional data life-cycle stages Reported Related Indexed Stored Quality/improved Discarded/retained Acted on Analyzed Copied Indexed Stored Created Stored Copied Quality/improved Stored Indexed Copied Quality/improved Reported Discarded/retained Acted on Analyzed WhamTech Smart Fabric Created Stored Quality/improved Indexed Related Master Capabilities in the Smart Fabric support applications Revision 8.6 Copyright 2018 WhamTech, Inc. 21

22 provisioning for Big and other analytics Big / Analytics base/ Link Analysis/ Graph base Reported Indexed Stored Quality/improved Discarded/retained Acted on Analyzed Copied Created Stored Quality/improved Indexed Related Master data mapping, quality, security, masking tokenization, encryption and link mapping, and master data, addressed Eliminate up to 80% of time spent by expensive data scientists/analysts preparing data Tend towards real-time analytics and feedback to operational/transactional systems Log/Transaction System WhamTech Smart Fabric Revision 8.6 Copyright 2018 WhamTech, Inc. 22

23 SDF bridges the gap between the enterprise, and reporting, BI, analytics and other apps Integration (read only) Mapping Quality Masking, Tokenization and Security Encryption Individual Group Single customer view (read only) Interoperability (read and write) Enterprise (operational, transactional, other and external data sources) Aggregation Links/ Relationships Smart Fabric Provisioning Governance Master Reporting, BI, analytics and other apps Population Multiple entity centricities Local Queries/Results Regional Other Predictive rules/interactive CRM and BPM Global Revision 8.6 Copyright 2018 WhamTech, Inc. 23

24 Smart Fabric basic solutions Revision 8.6 Copyright 2018 WhamTech, Inc. 24

25 Smart Fabric incremental solutions Revision 8.6 Copyright 2018 WhamTech, Inc. 25

26 Sounds too good to be true where is this applied? (1 of 2) Major project(s) with a very large healthcare provider, starting with selective patient indexing/segmentation for contributing network providers 3 completed -> 100 -> 600 -> 20,000 data sources = incremental delivery Multiple other incremental use cases, e.g., matching prescriptions to diagnoses, and potential for becoming the basis for a Smart Lake Major project(s) with a very large system integrator, starting with enabling a highly security-centric single person view across multiple disparate HR systems using standard semantic view data access, MDM, event processing, BPM workflows and operational dashboards Past work with major DoD and intel government contractors high performance and complex query processing, including up to 60 billion records/day on HBase Includes federal healthcare projects with CMS, VA and other agencies Federal Fraud Fusion Center Unemployment insurance claims, Medicare, Medicaid, SSA, SARS, NIBRs, etc., MDM, data stays in sources (fed, state and local), graph and link visualization, and support SPARK and ML analytics Revision 8.6 Copyright 2018 WhamTech, Inc. 26

27 Sounds too good to be true where is this applied? (2 of 2) Work with Northrop Grumman on a number of projects POC for single patient view for NHS Trust in the UK 3 organizations, 7 data sources = 4 on premise and 2 in Cloud - Hybrid Cloud, use NHS MPI and own MPI, FHIR APIs, services and AWS POC for patient smartphone app support for interactive CRM, new BPM workflows on REST API data services, with read and write back to legacy data sources Message Bank for very large medical academic delivery system for HL7 and other messages Cassandra target data source, real-time, parse and index, VMPI, FHIR APIs, and future support for reporting, BI and analytics, including SPARK and ML Bitcoin/Blockchain transaction reporting, BI and analytics particularly, graph visualization, link analysis, SNA and MDM Potential integration and matching of unstructured healthcare insurance benefits documents with structured claims data Virtual graph database, link analysis and graph visualization using simple SQL OEM KeyLines visualization Revision 8.6 Copyright 2018 WhamTech, Inc. 27

28 Conventional vs. SDF adapters costs and ROI comparison Attribute Conventional Federated WhamTech SDF Adapters Access Adapters Costs - TCO Up to 1000 % of WhamTech 100% ROI assuming TCO as basis, and revenue gains and cost savings Capabilities Basic Advanced more capabilities for less cost Perpetual License IBM and others > 200% of 100%, starting at $10K per data source Costs CAPEX WhamTech; some freeware and Red Lease/SaaS Costs Hat < WhamTech Assume 40% of perpetual license costs per year, including maintenance and support Implementation Costs Up to 500% of WhamTech, long duration to implement Maintenance and 18% of perpetual license costs Support Costs included in lease/saas costs 40% of perpetual license costs per year, including maintenance and support 100%, relatively simple to implement = low costs and short duration 18% of perpetual license costs included in lease/saas costs Revision 8.6 Copyright 2018 WhamTech, Inc. 28

29 The End Appendix: Backup material Revision 8.6 Copyright 2018 WhamTech, Inc. 29

30 Simplified* layer capabilities *Detailed layer diagram in Appendix, slide 39 Applications Standard drivers, APIs, Web/data services, SQL and other query languages Built-in advanced capabilities RBAC and data loss prevention Link analysis/graph database Visualization Big /analytics data provisioning Built-in support for common applications Post-index, standard data view, multi-record processing Pre-index, single record processing Support for BI/analytics Master data management Indexing content, security, extracted entities, indexed views, unstructured and Link Security classification, data masking, tokenization and encryption Support for CRM Support for BPM/ decision support Event processing Event correlation [Anomaly detection] Standard data view mapping more than one possible Quality cleansing, transformation and standardization Discovery, profiling and correlation device, source and data Support for interoperability Analytics parsing, categorization, entity extraction and other analytics AUTOMATION Sources Mainframes bases Files Logs Optional Lake Centralized or Distributed Office docs Applications Web docs Social media Big Streaming Cloud IoT Revision 8.6 Copyright 2018 WhamTech, Inc. 30

31 Advanced data management Combine the best and overcome the worst of conventional approaches of data warehousing, federated data access and enterprise search through index-based federated adapters to create a Hybrid Smart Fabric/Lake without needing to copy or move data from source systems, although that is an option Access adapters and federation servers at any level and from any location with access control Include all data sources from mainframes to IoT devices, on premise, Cloud, Hybrid Cloud, external, etc. Use multiple types of indexes and indexed views distributed, 100% contiguous across data sources, columnar, file-based and contain pointers to source data Federate/distribute data governance built and maintained from the bottom up as systems are discovered, read, indexed and metadata captured combine with IAM, RBAC, etc. - can obtain (and store) a complete centralized data governance view at any time, and intervene and impose as needed Federate/distribute metadata repository, data discovery, security, quality, transforms, relationships and mapping to standard data views Seamless, automatic and near real-time updateable integration of master data management to enable single customer/patient and other entity views across the extended organization can also federate/distribute master data Revision 8.6 Copyright 2018 WhamTech, Inc. 31

32 Advanced application support Combine queries on structured, semi-structured and unstructured data Accelerate query processing on existing systems, but almost no load on data sources High performance parallel/edge query processing Enable direct access to data sources through indexes and/or use indexes to represent and use data logically as (1) objects, (2) relational, (3) hierarchical and (4) NoSQL/Big Table Built-in virtual graph database, link analysis and graph visualization - use simple SQL Event processing - monitor changes to data sources through indexes and indexed views, trigger workflows, and update applications and visualizations, e.g., operational dashboards and graphs True interoperability based on single customer/patient views with both read and write-back to data sources goal to have almost any application working with almost any data source(s) Provision highly curated data to Big /analytics in near real-time Bridge the gap between enterprise operational/transactional systems and reporting, BI, analytics and other applications tend towards closing the loop in near real-time Partnering with unique analytics management vendor, Aginity (aginity.com) Revision 8.6 Copyright 2018 WhamTech, Inc. 32

33 Advanced data security Leverage centrally managed IAM and attribute-based RBAC, e.g., Active Directory (AD) with Kerberos source stewards can have ultimate veto Supports Single Sign-On (SSO) for all data sources regardless of data source system support Be a data security gatekeeper for data sources Follow Forrester Zero Trust Security Model = Discover, index, classify and secure All results data traceable to source records Row, column and data element security Dynamic data masking, tokenization and encryption (third-party Format-Preserving Encryption (FPE)) governance from the bottom-up and/or can support a top-down tool Full auditability Support for third-party User Behavior Analytics (UBA) Alleviates/prevents insider data thefts (25%) and external origin (hacks) data thefts (75%) Leverage user logs, including queries made Revision 8.6 Copyright 2018 WhamTech, Inc. 33

34 Other advanced capabilities NEAR REAL-TIME ARCHITECTURE Edge process data Enable a near real-time data/event-driven architecture Build new workflows using BPM software on top of legacy systems to support operations, CRM, smartphone apps, IOT devices, reporting, BI and analytics STANDARDS Standard drivers, APIs, Web/data services, REST APIs, ANSI SQL and other query languages with conversion, Cloud, VMs, PMs, Windows, Linux and soon-to-be IBM Power Systems Standard data models/views, e.g., HL7 and FHIR for healthcare, NIEM for government and other areas, XBR and others for financial services, ACORD for insurance or organization s own Revision 8.6 Copyright 2018 WhamTech, Inc. 34

35 Smart Fabric example Bluemix deployment Multiple access methods Multiple query language options Multiple ways to represent data Standard data view, e.g., FHIR APIs and NIEM Cloud platform-based data services New BPM workflows running on legacy data sources Write-back to data sources VMPI-governed data access Multiple legacy data sources sources could be in multiple organizations sources could be on premise and in the Cloud Hybrid Cloud access Revision 8.6 Copyright 2018 WhamTech, Inc. 35

36 Multi-level value contribution Reduce costs Increase revenue Increase profit Improve customer experience Upsell and cross-sell customers Gain customers Improve compliance Improve reporting Tend to realtime Reduce waste Reduce liability Convert knowledge to SUCCESS OUTCOMES Visualization Link analysis Social analytics Ontology representation BI/ analytics BPM Decision support CRM/ MDM EHR/ HIE Convert information to KNOWLEDGE Indexed views Advanced text search Entity extraction Entity resolution Link / mapping Realtime alerts CEP Categorization discovery profiling Access almost any data source Integrate multiple data sources Convert data to value-added INFORMATION Leave data in sources Work with structured and unstructured data Improve data quality Build structured and unstructured indexes Update in near realtime Map to a virtual standard data view Provide basic indexed virtual DATA access, integration, sharing and interoperability Scale with distributed parallel processing Almost no load on data sources Revision 8.6 Copyright 2018 WhamTech, Inc. 36

37 Smart Fabric is location agnostic Revision 8.6 Copyright 2018 WhamTech, Inc. 37

38 Smart Fabric is configuration agnostic Revision 8.6 Copyright 2018 WhamTech, Inc. 38

39 -driven bottom-up vs top-down approach For each data source: Application/Middleware External Query in SQL WhamTech s Standard Drivers/APIs/Web Services WhamTech s Mapping Layer WhamTech s Automatic Query Processor EIQ SuperAdapter WhamTech Content WhamTech Link Quality/Parser/Entity Extraction/Other discovery and profiling Transaction Log Reader, MQ or similar Source Driver/API/Web Service Source Revision 8.6 Copyright 2018 WhamTech, Inc. 39

40 Detailed Smart Fabric layer capabilities Applications Real-time Business Intelligence Distributed Analytics Complex Event Processing Enterprise and Web Search Virtual Graph base Link Analysis Social Network Analysis Living Networks CDI-MDM/ Single Entity View Other Applications Standard drivers, APIs, Web/data services, SQL and other query languages EIQ SuperAdapter Security Layer Query Side Automatic Query Processing Real-time Monitoring and Event Processing Master Management Smart Fabric Administration and Configuration Tools Structured Master Text Extracted Entity Semantic Mapping to Standard View(s) Fuzzy Match Preaggregated Precalculated Embedded Value Join Link Denormalized Security and Privacy Access Controls Security Layer Side Cleansing, Transformation, Standardization, Masking, Tokenization and Encryption Entity Extraction/NLP/Categorization/Other Text Processing Metadata Discovery/ Profiling Device, Source and Discovery Metadata Management and Repository, incl. Governance Changed Capture Intelligent Spider Sources Network Assets and Devices Relational bases Standard, Proprietary and Web Service Drivers Mainframe data Files Changed Capture Web Services Application Drivers Applications Network Assets and Devices Enterprise documents and Spidered files from Web and other sources Revision 8.6 Copyright 2018 WhamTech, Inc. 40

41 Smart Fabric impact diagram Intra and Inter- Source Correlation Profiling Source Discovery Semantic Identification Structured Unstructured Index Preparation and Results Transformation Transform Development and Testing Masking, Tokenization and Encryption Parser Transform Entity Extraction Security Classification Categorization Security Classification Content Structured (most data discarded) Distributed Metadata Repository Distributed Metadata Standard View Standard View (indexes semantically mapped) Link Results Pointers Link Master and Results Master Master Results Link Analytics and Visualization Device/Host Unstructured (most data discarded) Results Revision 8.6 Copyright 2018 WhamTech, Inc. 41

42 Smart Fabric impact diagram for query submission APPLICATION Discovery Index Preparation and Results Transformation Content Standard View Link Master and Results 3 Structured (most data discarded) 1 1 Master Structured Unstructured Distributed Metadata Repository 3 Distributed Metadata 2 Standard View (indexes semantically mapped) Link Master 3 Link Analytics and Visualization 3 Unstructured (most data discarded) Revision 8.6 Copyright 2018 WhamTech, Inc. 42

43 Smart Fabric impact diagram for results retrieval APPLICATION Discovery Index Preparation and Results Transformation Content Standard View Link Master and Results 4 Structured Unstructured Masking, Tokenization and Encryption 5 5 Transform Structured (most data discarded) Distributed Metadata Repository 4 Distributed Metadata 6 Standard View (indexes semantically mapped) 7 Results Pointers Link Master Master 4 9 Link Analytics and Visualization Unstructured (most data discarded) 8 Results Revision 8.6 Copyright 2018 WhamTech, Inc. 43

44 Smart Fabric processes (1 of 5) Automate (using BPM) as much as possible: Deploy on AWS, Azure, IBM Bluemix, OpenStack, VMs, physical servers other cloud options available Instantiate an EIQ System Administration and Configuration Tool Instantiate a distributed network asset/device, data source and metadata repository Network asset/device discovery source discovery Using network asset/device discovery tool Using spiders for ediscovery-type documents, files, , etc. Instantiate EIQ Adapters on demand Revision 8.6 Copyright 2018 WhamTech, Inc. 44

45 Smart Fabric processes (2 of 5) discovery Optionally, with raw Link (internal and external pre-joins) identification DSL: risk classification CS: Event correlation DSL = Security Layer CS = Cybersecurity profiling for data transforms for typos, transpositions and non-standard data, e.g., name, address, phone and correction Lookup dictionaries and thesauri, USPS or other address correction, regular expressions, APIs, DLLs, transformation server, etc. DSL: Masking, tokenization or encryption for indexed data or dynamically depending on access controls Revision 8.6 Copyright 2018 WhamTech, Inc. 45

46 Smart Fabric processes (3 of 5) Multiple indexes and types, e.g., basic content, DSL: security (classification), aggregations, calculations, fuzzy, text, extracted entities and Link DSL: Can encrypt entire disc volumes, individual indexes or entire sets of indexes MDM: source-specific tables containing unique indexed primary entity IDs, and master data, links and date-time MDM = Master Management Create using Link Index process, with multi-attribute fuzzy match for composite scoring and master data rules DSL: WhamTech Security and Privacy Access Profiles (SPAPs) or other Role- Based Access Control Current: Source organization, user, role, application, target organization and data source profiles available Future: Extend for application processes Revision 8.6 Copyright 2018 WhamTech, Inc. 46

47 Smart Fabric processes (4 of 5) Hierarchies honored through joins and/or Link Inferred ontologies Reasons for hierarchies change depending on application, e.g., one vendor has multiple products and one product from multiple vendors MDM: Versioning with access to historic master data Combine with other data sources, tending towards EDW/enterprise solutions MDM: Pure registry option to replace either data source indexes or source data itself (automatically updates indexes) with master data Pure registry-based master data table, but limits options, lower performance and more complex Execute analytics, combined with other data and search/query filters, e.g., reporting, BI and link analysis/graph database Include aggregations, calculations, master data (if available) and other data, e.g., external Revision 8.6 Copyright 2018 WhamTech, Inc. 47

48 Smart Fabric processes (5 of 5) Write back selective updates/corrections to data sources with possible inverse data transforms (MDM: See previous slide) Continuously monitor metadata (index tree profiles) using stored procedures with triggers Helps identify anomalies/outliers Event processing enabled (federated solution for Oracle Event Processing) Open source and commercial BPM software for non-oracle solutions Interoperability query transformation to avoid rewriting applications Goal to enable almost any application(s) to work with almost any data source(s) Mainframe data source option files and live systems Hadoop (HBase/Hive and HDFS levels) and Cassandra options Revision 8.6 Copyright 2018 WhamTech, Inc. 48

49 Smart Fabric key differentiators expanded (1 of 2) 1. Index data regardless of where it resides in sources not copied or moved, work with copies or store in indexes a hybrid approach 2. Any data source mainframes, legacy, databases, office documents, Web, social media, streaming, Hadoop, Cassandra, structured, unstructured, semi-structured, etc. 3. Automation as much as possible detect new devices, and discover data sources and data, e.g., for IoT and networks 4. quality and other data management fundamentals addressed before any data access cleansed, transformed and standardized indexes, and results 5. Own indexes, indexed views and query processing, as part of adapters to multiple data sources parallel distributed processing linearly scalable maintained in real-time using changed data capture, incrementally and batch 6. Event processing (past federated data access partner with Oracle ) Revision 8.6 Copyright 2018 WhamTech, Inc. 49

50 Smart Fabric key differentiators expanded (2 of 2) 7. Link mapping finds relationships between entities within and across data sources and captures them in Link using (i) exact, (ii) fuzzy and (iii) composite weighted probability matches 8. Link used to (i) locate and read entities for master data, (ii) execute degrees of separation queries, (iii) accelerate joins, (iv) represent ontological/semantic models and (v) perform link analytics 9. Seamless combination of master data with operational, other enterprise and external data systems, and of use by any application 10. Link analytics for single customer/patient views, network visualization, fraud detection, debottlenecking, social media analysis, etc. 11. Highly interactive browser-based query, results and link visualization tool 12. Support for active CRM, BPM, individual analytics and machine learning Revision 8.6 Copyright 2018 WhamTech, Inc. 50

51 Smart Fabric data processes Analyzed Discarded/retained Acted on Note: not copied or moved - only results retrieved Reported (Results data quality/ transformation) Results retrieval Query processing Master data management Device/host discovery source discovery discovery and profiling security classification and link indexing quality/transformation/ masking/tokenization/encryption Revision 8.6 Copyright 2018 WhamTech, Inc. 51

52 Solutions to multiple challenging problems in one platform Applications Virtual network, data source and advanced data discovery Virtual data security Virtual event processing Virtual cybersecurity Virtual MDM Virtual reporting, BI and analytics Virtual interactive link analysis and visualization Optional Modules Distributed Virtualization (and Federation, Integration and Interoperability) Platform, aka Smart Fabric Basic Product Revision 8.6 Copyright 2018 WhamTech, Inc. 52

53 Solutions to multiple challenging problems in one platform 1. SMARTDATA FABRIC (SDF) for basic data discovery, profiling, quality, mapping, indexing, virtualization, federation, integration and interoperability as basis for capability support modules and applications 2. EXTEND SDF WITH AUTOMATED NETWORK, DATA SOURCE AND ADVANCED DATA DISCOVERY including relationships and eventual automated mapping to a standard data view 3. EXTEND SDF WITH EVENT PROCESSING to keep track of significant changes occurring in data 4. EXTEND SDF WITH DISTRIBUTED (preferably HYBRID) MDM to seamlessly combine with operational/transactional data and maintain in near real-time 5. EXTEND SDF WITH IMPROVED CYBERSECURITY through indexed federated log and other data source access, including automated anomaly detection and automated event correlation 6. EXTEND SDF WITH VIRTUAL DATA SECURITY LAYER to defend and protect data of value (i) as it is created, (ii) at rest in the source, (iii) in transit, (iv) at the recipient and (v) after no longer needed 7. EXTEND SDF WITH BI/ANALYTICS oriented virtual and materialized real-time updateable hierarchical indexed views, text analytics including entity extraction, and locally executed algorithms 8. EXTEND SDF WITH LINK ANALYSIS (and OEM LINK VISUALIZATION) for link analysis/graph database for almost any type of analytics, including virtual MDM (master patient index), cybersecurity and data security Revision 8.6 Copyright 2018 WhamTech, Inc. 53