Andreas Weininger,

Transcription

1 Db2 Event Store and Queryplex How do they fit into the Db2 Family? Andreas Weininger,

2 Agenda New requirements Db2 Event Store and the Fast Data Platform IBM Project Queryplex 2

3 New Requirements

4 Perishable Insights: Insights That Are Actionable Source: Forrester Perishable Insights-Stop Wasting Money on Unactionable Insights Report 2016 Being able to act on insights in real-time can provide maximum differentiation and value to businesses. To act on real time events, businesses need an Application Architecture that can ingest, analyze and act upon events as they happen in sub-second to seconds response time. Machine Learning combined with Reactive Programing frame works enable insights to be actioned automatically and stay current so business can react to events in sub-second timer frames 4 4

5 Big Data and Microservices are Converging Microservices Reactive Systems Fast Fast Data Data Applications Big Data Increasing requirements for scalability & resilience Streaming data pipelines are being served by Microservices Increasing requirements for streaming vs. batch 5 5

6 All businesses have become Data Driven Competitive Advantage Key Decisions 6

7 Data is Everywhere and increasingly heterogeneous 7

8 Db2 Event Store and the Fast Data Platform

9 Success with Db2 Event Store Fastest Growing segment in the Market Event Driven Apps are needed now Db2 Event Store is the database for fast data management New digital business and IoT creates new opportunities for Event Store Gartner: By 2020, participation in 80% of new business ecosystems will require support for event processing. By 2020, for the majority of global enterprise CIOs, achieving broad competence in event-driven IT will be a top-three priority. Data originates in real time. Why shouldn t you analyze and act on that data in real time too? 9

10 Use Cases Retailer Loyalty Program Integrate streamed payment, couponing events, climate, calendar, mobile data. measure refine, deliver better couponing and loyalty system Smart Metering/Smart Grid Deliver a Integrated platform for optimizing energy usage, capacity and billing across a smart grid system Banking Risk Exposure Combine account transactions from across the bank to provide a master ledger for real-time risk exposure and fraud identification Satellite Tracking System Track satellites in real time and produce analytics on operations and performance Intelligent Manufacturing Deliver real-time monitoring framework for automated production lines, providing productivity, preventive maintenance, and reporting Transactional to Analytics Consolidation Capture your transactions and augment with external data into an analytics platform for deeper analytics 10

11 Smarter Energy Energy Companies: Millions of meter to monitor Traditional: one reading per month offline Trend: one reading every minutes eventually online Volume: 2,920x to 4,380x larger: Collect in 2-3 hours what used to be collected in one year! Benefits Real-time response to energy demand Better management of energy sources (ex: home solar panels) Better peak predictions Better response to black out/brown out needs Dynamic billing Needs Fast ingest, quick access to current state, historical analysis 11

12 Connected Cars Telematics Information: Speed, gearbox, ABS, airbag, tire pressure, wiper seed, location Over 1500 readings, some at millisecond intervals Derive weather, local events Location: traffic patterns Demographic, driving patterns Benefits: Customer sentiment, driving experience Understand how customer use their products Maintenance planning Geo-fenced event notification 12

13 Consumer Products Brand Management Buzz, sentiment analysis Instrumented Assets Immediate access to issues Preventive maintenance Just-in-time inventory replenishment Detect out of stock or non-compliant stores Alerts to store managers, sales team Instant availability to sales trends (reduce out-of-stock) Marketing Real-time campaign tweaking Target marketing location, time 13

14 Retail Six Billion Mobile Phone Subscribers Social data analytics Buzz, sentiment analytics Brand management ibeacon-type technologies Brick-and-mortar click stream analysis Security Fraud detection Micro segmentation marketing Customer location and time opportunities 14

15 Use Cases: Summary Cyber Security Retail and Logistics Banking and Finance Internet of Things Telecommunications Industrial & Manufacturing Operational Efficiency Risk Management Problem Detection Fraud Customer Service Energy Savings Smarter Planet 15

16 IBM Db2 Event Store Characteristics 1 Lightning Fast Ingest 1 1 Million inserts per second per node Ingest rate scales linearly with added nodes Memory-optimized indexing for fast lookups 2 Real-time Analytics Real-time analytics over ALL ingested data OLAP queries faster than SparkSQL Integrated machine learning capabilities 3 Highly Available Atomic Transactions Replication for HA Configurable consistency Full insert Capabilities 4 Built for Open Hybrid Cloud Query elasticity through Spark Writes to shared storage in Parquet format No vendor lock-in 1 Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. 16

17 IBM Cloud Db2 Event Store Operating in Concert on the Edge 17

18 Why Not Hadoop? Hadoop is not designed for transactional systems Hadoop is designed for large batch ingests Db2 Event Store is a high-performance hybrid relational/analytics system Db2 Event Store continuously optimizes storage for high performance access 18

19 Real-Time Analytics: Db2 Event Store and IBM Streams Operational Analytics Data at rest Real-Time Analytics Data in motion Store the data Continuous No storage required Ask a question Query the data Analytics on Streaming Data in context Analyze In-memory analytics Dynamic applications 19

20 IBM Cloud IBM Db2 Event Store Management System 20

21 IBM Db2 Event Store System Data Scientist Using Spark Analytical Tooling Historical and Operational Reporting With traditional Tools 21

22 IBM EventStore Ingest 1. Ingest occurs using Scala API or streaming sink Scala API accessible through Spark-friendly Context 2. Batches of rows are formed and sent asynchronously from client to the appropriate EventStore nodes 3. Rows are placed in the queryable log, replicated to replica nodes, and reply is sent to client 4. After some time, data in logs is formed into Parquet blocks and written to the storage layer 1 Log (on SSD) IBM EventStore OLTP Context Transactional Application Node A Node B Node C IBM EventStore Engine IBM EventStore Engine Log (on SSD) Compressed Parquet data Shared Storage 2 IBM EventStore Engine) Log (on SSD) Configurable Replication 3 Share 4 22

23 Ingest: Groomer/Roller Process Background process Wakes up at configurable interval (default: 1 second) Reformat log data into Parquet format and write to permanent storage Logs are expected to be stored on SSD drives Re-grooming Re-process the data due to: small file size, poor compression, and more Optimizing ingestion and storage 23

24 IBM EventStore Analytics 1. Analytics queries leverage the EventStore SQLContext which extents the Spark SQLContext 2. Queries are sent to either EventStore nodes, or vanilla Spark nodes depending on performance requirements and whether most recent (just inserted) data is required a) Query is sent to EventStore nodes to retrieve most recent data and combine with shared data in cache or in storage layer b) Query is sent to Spark node(s) to read data all but most recent data from storage Cache 2a IBM EventStore SQLContext Analytical Application Node A Node B Node C Spark Executor 1 IBM BLUSpark EventStore Engine Engine Log (on SSD) Spark Executor IBM BLUSpark EventStore Engine Engine Cache 2a Log (on SSD) Compressed Parquet data 2b Spark Executor Storage 24

25 Event Store Indexing All data indexed by primary key for fast lookups Indexes formed asynchronously to avoid insert latency Stored as a Log Structured Merge (LSM) Tree Sequences of sorted keys (called runs) written to disk during Share processing Runs are merged together to improve scan and I/O efficiency Scans search runs in reverse sequential order until desired key is found (or ruled out) Synopsis maintained for each run to allow for run elimination Scan Order Recent Runs Older Runs Oldest Run 25

26 IBM Db2 Event Store Components Watson Studio Local (DSX Local) Provides Spark, Machine Learning, etc. Extends the SparkSQL interface IBM Data Platform Manager Comes with DSX. Includes a monitoring dashboard and management for nodes, services, pods, and users ELK Elasticsearch, Logstash, Kibana Used for logging Alluxio 1 Transparent support of distributed file systems. Zookeeper Manages the cluster metadata Prometheus Monitoring and alerting 1 Alluxio is currently part of the product but disabled 26

27 IBM Db2 Event Store Storage GlusterFS Distributed file system, posix compliant Used for data storage Swift ObjectStore For user artifacts (DSX) IBM Cloudant Services meta database (DSX) Elasticsearch Used for logging (DSX) 27

28 Client API Data loading and analytics available through DSX More info in later slides Scala API Available as one jar file Other available APIs Python Java (requires a Scala jar) REST Only available for Enterprise and Standard editions IBM Streams operator 28

29 IBM Fast Data Platform Enables the developer and data scientists to efficiently bring to market fast data solutions Incorporates microservices, streaming data analytics, and machine learning Lightbend Fast Data Platform Build streaming analytic solutions and microservices IBM Db2 Event Store For fast, scalable persistence IBM DSX Local For Data Scientists to collaborate, build and export machine learning models 29

30 IBM Fast Data Platform - Architecture 30

31 IBM Project Queryplex

32 All businesses have become Data Driven Competitive Advantage Key Decisions 32

33 Data is Everywhere and increasingly heterogeneous 33

34 Performing Analytics Today Costly and Complex High Latency Does not meet Business need Analytics Application Compute resources at source not utilized Data Lakes Data Warehouses Data Marts Error prone, data integrity challenges Applications expect homogeneity Not all data needs to be moved or copied 34

35 Resulting Data Architectures Numerous ETLs Unnecessary duplication, replication Data governance issues accelerating 35

36 Project Queryplex A new approach to Data Virtualization Now in beta trial. Coming to ICP for Data in November 1 Query anything, anywhere. Query many heterogenous data sources as one across cloud, on-premise and mobile with advanced analytics using the most popular languages and tools Queryplex Constellation Data Source Node 2 Extreme simplicity. Automatically discover, and connect few to many devices and data stores into a single self balancing constellation. Avoid the complexity of centralized copies. Data only persists at the source. 4 Security. Fully secure and encrypted communication and preservation of data access rights at source. Service Node Cluster 3 Execution speedup. Many times acceleration using the power of every device to compute and aggregate results. Analytics Application Queryplex service node Caching Policy 36

37 Federation vs. Queryplex Classic Federation & Edge Computing Query Query Result coordinator Query issued against the system A coordinator receives the request and fans the work out to edge nodes Edge nodes individually perform as much work as they can based on their own data. Individual results are sent back to the coordinator for final merging and remaining analytics. Coordinator receives intermediary results from all edge nodes, merges results, and performs remaining analytics Queryplex s Computational Mesh Query Query Result coordinator Query issued against the system A coordinator receives the request and fans the work out to edge nodes Edge nodes self organize into a constellation where they can communicate with a small number of peers. Nodes collaborate to perform almost all analytics, not only analytics on their own data. Coordinator receives mostly finalized results from just a fraction of nodes. Completes the final work for the query result. 37

38 Early Performance Benchmark Query with Copy & Load Query includes six aggregates and grouping on 2.5 years of data, with 2,500 data points per day. San Jose: Head node Toronto: Edge devices RDBMS and Hive tests run on Intel CPU, roughly 4x faster than ARM. Edge/Federation and Queryplex data resides on MySQL databases inside of 36 Raspberry Pi devices, with ARM processors. San Jose: Head node Toronto: Edge devices 38

39 Early Performance Benchmark Query only Query includes six aggregates and grouping on 2.5 years of data, with 2,500 data points per day. San Jose: Head node Toronto: Edge devices RDBMS and Hive tests run on Intel CPU, roughly 4x faster than ARM. Edge/Federation and Queryplex data resides on MySQL databases inside of 36 Raspberry Pi devices, with ARM processors. San Jose: Head node Toronto: Edge devices 39

40 Early Performance Benchmark additional geographic latency Adding devices that reside in Hursley UK did not impact query time San Jose: Queryplex service node (application connection point) Toronto, Canada: 36 Raspberry Pis Hursley, UK: 2 Raspbery Pis 40

41 Supported Languages & Data Sources Query Languages SQL (ANSI) SQL (Oracle) SQL (DB2) SQL (PostgreSQL, Netezza) Spark SQL SQL Python SQL in R & SparkR PL/SQL (stored procedures) SQL PL (stored procedures) Future Future Mix Any Combination of Data Sources Oracle Excel Db2 (software, appliance, cloud) CSV (delimited text) Netezza Cloudera PostgreSQL Teradata Informix Db2/Z MySQL MongoDB Future SQLServer Accumulo Future DerbyDB Redis Future Big SQL Cloudant Future Db2 Event Store Greenplum Future 41

42 Release & Platform Timeline Supported Customer beta trials now - more at queryplex.com or via a request to info@queryplex.com Individual Developer unsupported digital download and go trial available: Service Node is on-premise (Linux x86, PPCLE). Mac and PC support for developers Queryplex End Node agent can run anywhere with connectivity to Service Node (Java or Docker for x86/arm/ppcle) GA release in Q4 (November) initially packaged as Data Virtualization within IBM Cloud Private for Data (ICP for Data) Subsequent platforms and packaging of Data Virtualization with Project Queryplex under consideration for 2019 release 42

43 Questions? 43

44 Thank you ITALIAN HINDI FRENCH JAPANESE BRAZILIAN PORTUGUESE SIMPLIFIED CHINESE TRADITIONAL CHINESE SPANISH RUSSIAN TAMIL THAI GERMAN ARABIC 44

45 Notices and disclaimers Copyright 2018 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed as is without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. 45

46 Notices and disclaimers continued Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such thirdparty products to interoperate with IBM s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera, Bluemix, Blueworks Live, CICS, Clearcase, Cognos, DOORS, Emptoris, Enterprise Document Management System, FASP, FileNet, Global Business Services, Global Technology Services, IBM ExperienceOne, IBM SmartCloud, IBM Social Business, Information on Demand, ILOG, Maximo, MQIntegrator, MQSeries, Netcool, OMEGAMON, OpenPower, PureAnalytics, PureApplication, purecluster, PureCoverage, PureData, PureExperience, PureFlex, purequery, purescale, PureSystems, QRadar, Rational, Rhapsody, Smarter Commerce, SoDA, SPSS, Sterling Commerce, StoredIQ, Tealeaf, Tivoli Trusteer, Unica, urban{code}, Watson, WebSphere, Worklight, X-Force and System z Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: 46