MarkLogic as a Real-Time Data Hub

Transcription

1 MarkLogic as a Real-Time Data Hub Prepared by Mike Bowers 5/9/2018 version 1.3

2 About the Author Michael Bowers Principal Engineer Data Architect Using NoSQL professionally for 10 years Author Pro CSS and HTML Design Patterns Pro HTML5 and CSS3 Design Patterns 2

3 Church of Jesus Christ of Latter-day Saints Hundreds of NoSQL servers running 190+ applications and websites with billions of page views annually 15.9 million members (30,304 congregations worldwide) 10,238 Humanitarian Service Missionaries in 189 countries Thousands of documents in 188 published languages

4 What is a Data Hub? A data hub is populated with data from one or more sources and from which data is taken to one or more destinations.

5 Why use a Data Hub? Malcolm Chisholm Ph.D. The more that data is an enterprise resource that needs to be shared the more likely data hubs will appear Data can indeed be shared via point-to-point interfaces between pairs of applications, and this is a valid architectural alternative [and] is much simpler to implement than any hub, and it seems that all enterprises now have gigantic spider webs of such interfaces that have grown up over the years.

6 Why use a Real-time Data Hub? To replace the gigantic spider webs of real-time APIs SOAP REST Database Links

7 Why is a Data Hub better? A data hub untangles the spider web that comes with point-to-point solutions In my experience (Malcolm Chisholm Ph.D.), point-to-points have the following issues (amongst others): They often needlessly replicate movement of the same data. They typically have poor controls around them, and minimal governance. They are typically difficult to modify. They are typically poorly documented and understood. They tend to promote coupling and fusion of applications into a giant monolithic enterprise silo that is very difficult to evolve in line with business changes. They rely on the application pair involved to do things like data integration and transaction integration.

8 Dr. Chrisholm's Six Flavors of Data Hub 1. Publish Subscribe Hub (Real-time or Batch) 3. Operational Data Hub for Data Warehouse (Batch) 5. Message Hub (Real-time) Cache ODS DW 2 3 ESB Operational Data Hub for Integrated Reporting 4. Master Data Management Hub (Real-time or Batch) 6. Integration Hub (Real-time or Batch) ODS reports 2 MDM 5 2 DB DW

9 Where is MarkLogic often used as a Data Hub? 1. Publish Subscribe Hub (Real-time or Batch) 3. Operational Data Hub for Data Warehouse (Batch) 5. Message Hub (Real-time) Cache ODS DW 2 3 ESB Operational Data Hub for Integrated Reporting 4. Master Data Management Hub (Real-time or Batch) 6. Integration Hub (Real-time or Batch) ODS reports 2 MDM 5 2 DB DW

10 A Real-time Data Hub is an Advanced Cache Engineers who love REST, Messaging, and Service Busses Can be strongly opposed to the idea of a "Data Hub" They believe it is an anti-pattern to replicate data to use database technologies for integration because databases are below the API layer that contains business rules Like the idea of intelligently caching data delivered by REST endpoints It is a best practice to cache data for performance, scale, and availability to use REST APIs to share data after applying business rules

11 How can we use MarkLogic as a Real-time Data Hub? Sources Consumers Source Cache to Scale/Filter REST API's Reads Central Cache to Share Enterprise Data Consuming Cache to Combine REST Data

12 Use Case: Source caches its shared data to offload delivery Source app uses Pub-Sub Data Hub Pattern to publish data to many consuming apps sources across many channels One Source Publish-Subscribe Hub Many Consuming s MarkLogic Advantages 1. Sources write REST data as is 2. Consumers read REST data as desired 3. Built-in, standard, powerful REST API 4. Scale out performance & availability 5. Fully decouple consuming applications from non-standard REST from sources from data centers 6. Shard data for data sovereignty 7. Low cost and quick to deploy, maintain 8. Queries 9. Joins 10. Security Filters MarkLogic Considerations 1. Developer centric 2. Needs more integrations with message queues and relational databases

13 Use Case: System integrates data from many sources Consuming app uses Integration Data Hub Pattern to collect and harmonize data from many sources across many channels Many Source s Integration Hub One Consuming MarkLogic Advantages 1. Load data as is 2. Data transforms 3. Data quality 4. Data validation 5. Data harmonization 6. Data lineage and provenance 7. Data discovery: queries, joins, search 10. Temporal tracking 11. Semantic views 12. Ontologies 13. Real-time and batch 14. Integrated with Java and Node.JS 15. Scale out 8. Get data from remote data centers as if it were local 9. Get data from common integration channels: REST, Queue, Database, File MarkLogic Considerations 1. Developer centric 2. Needs more integrations with message queues and relational databases

14 Use Case: Central location to share enterprise data Enterprise creates Integration Data Hub to collect and harmonize data from many sources for many consuming systems Many Source s Central Integration Hub Many Consuming s Combines the strengths of the previous two use cases for publishing and integrating data Allows data from many sources to be integrated and queried together Centralizes the work of integrating data

15 Global Real-time Data Hub Data Hubs replicate some or all of their data around the world for local, highly available, near real-time access US Europe Synced Data Africa

16 Data Hub Conceptual Architecture Sources Publishing Shared Data 3 rd Party Tools Consumers Getting Read-only Filtered Data 3 rd Party Tools REST Message Q Relational DB Queryable REST Cache REST / s Message Q Relational DB Files Files 1. Real-time and Batch 2. REST first 3. Doc/Graph Queries 4. Doc/Graph Joins 5. Security Filters 6. Data quality 7. Data transforms 8. Low cost to maintain 9. Quick to provision 10. Build once and reuse 11. Multiple data centers 12. Data Sovereignty 13. Integrate IDs 14. Be a data source 15. Backwards Compatibility 16. Support REST with JSON/XML 17. Support Message Queues 18. Support Relational DBs 19. Support File transfers 20. Real-time & scheduled data

17 SOURCES MarkLogic Data Hub Physical Architecture MuleSoft MuleSoft CONSUMERS REST push or pull REST pull or push REST pull or push REST APIs & s JMS Q push EMX: API Gateway REST push REST push or pull EMX: API Gateway JMS Q mostly push REST APIs & s JMS Q mostly push Currently Enhancing Databases ODBC mostly pull EMX ESB MarkLogic Data Hub DSA Filter EMX ESB Currently Building ODBC mostly push Databases Data Sync Work Area Published Area Data Sync File Import pull File Export push Bulk Files File Transfer push File Transfer push Bulk Files EMX: Sterling File Gateway EMX: Sterling File Gateway