Rickard Linck Client Technical Professional Core Database and Lifecycle Management Common Analytic Engine Cloud Data Servers On-Premise Data Servers

Size: px

Start display at page:

Download "Rickard Linck Client Technical Professional Core Database and Lifecycle Management Common Analytic Engine Cloud Data Servers On-Premise Data Servers"

Donald Bryant
6 years ago
Views:

1 Rickard Linck Client Technical Professional Core Database and Lifecycle Management Common Analytic Engine Cloud Data Servers On-Premise Data Servers

2 Watson Data Platform Reference Architecture Business Analyst Data Scientist Data Engineer App Developer Watson Analytics Cognos Analytics Data Science Experience Embedded Throughout DataWorks Forge Find Share Collaborate Bluemix Ecosystem as underlying engine Data Sources On-premises / cloud Unstructured / relational Streaming / batch Feeds (e.g., ) Cloud Information Governance Catalog Ingest DataWorks Forge Message Hub (Kafka) Streams High Speed Ingest (TWC) Store BigInsights Cloudant Compose dashdb DB2 Graph S3 / SWIFT Govern Analyze Open Metadata Services Cognos Analytics Data Science Experience Graph Processing Spark (Python, R, SQL) Text Analysis (AQL) Watson Analytics Deploy App deploy Model deploy Report deploy Pipeline deploy 2 On premises Information Server Quarks Streams BigInsights Cloudant dashdb DB2 Cognos Analytics Data Science Experience Spark (Python, R, SQL) SPSS Text Analysis (AQL) SPSS

ANALYTICAL TRANSACTIONAL BigInsights Enterprise class Hadoop and real-time BigSQL for easier analytics IBM differentiators like GPFS Cloudant Local Massively scalable Eventual consistency model Built

3 Data Repository Portfolio On-Premise Offerings for structured and semi-structured data, transactional and analytics workloads. Knit together with the InfoSphere IIG technologies for data movement and integration. ANALYTICAL TRANSACTIONAL BigInsights Enterprise class Hadoop and real-time BigSQL for easier analytics IBM differentiators like GPFS Cloudant Local Massively scalable Eventual consistency model Built for Systems of Engagement UNSTRUCTURED DataStage On-Prem to Cloud ETL Connector 4 STRUCTURED DB2 with BLU SQL interface ACID compliance Columnar, in-memory performance DB2 Built for Systems of Insight PDA (Netezza) SQL interface ACID compliance Appliance form factor DB2 Built for Systems of Insight DB2 with purescale SQL interface ACID compliance Flexible HA and DR options Built for Systems of Record

4 Data Repository Portfolio - Cloud IBM CDS solutions combine workloads and data types for true hybrid services ANALYTICAL TRANSACTIONAL 5 BigInsights on Cloud Spark for in-memory Hadoop Built on IBM Open Platform Bare metal performance BigInsights enterprise features UNSTRUCTURED STRUCTURED Watson Analytics Analytics & Visualization Services dashdb for Analytics SQL interface, ACID compliant Columnar, in-memory performance BLU augmented with Netezza in-db analytics Built for Systems of Insight MPP Private Cloud option DB2 on Cloud Hosted DB2 on SoftLayer Flexible deployments for OLTP -or- analytics Secure Cloudant Database as a Service (DBaaS) Massively scalable for global data distribution Eventually consistent data model Built for mobile, Systems of Engagement DataWorks Data Refinery Services dashdb for Transaction SQL interface, ACID compliant Row organized tables Built for Systems of Record

DB2 - Highlights and Strategic Investment Areas Petabyte Scale

operational warehouse with petabyte scale BLU Technology Best

most cost effective solution for SAP Alternative to Oracle Minimized

Oracle SQL and NOSQL Integration Expand applications to new data

structured data 6 Never Down Systems of Record and Systems of

5 DB2 - Highlights and Strategic Investment Areas Petabyte Scale In-Memory Warehousing - Systems of Insight Build or expand your operational warehouse with petabyte scale BLU Technology Best Database for SAP Leverage the highest performing, most optimized and most cost effective solution for SAP Alternative to Oracle Minimized cost and risk of migration when breaking free from high costs of Oracle SQL and NOSQL Integration Expand applications to new data types and application paradigms while integrated with existing structured data 6 Never Down Systems of Record and Systems of Engagement purescale ensures your business critical systems are always available, performant and scalable

dashdb for Analytics In-database analytics capabilities for best performance atop a fully-managed warehouse 7 BLU Acceleration Netezza In-Database Analytics Cloudant NoSQL Integration dashdb MPP For

6 dashdb for Analytics In-database analytics capabilities for best performance atop a fully-managed warehouse 7 BLU Acceleration Netezza In-Database Analytics Cloudant NoSQL Integration dashdb MPP For apps that need: Elastic scalability Fully-managed High availability data warehouse on cloud Data model flexibility BLU Acceleration Data mobility columnar technology Text search + Netezza in-database analytics Geospatial Choice of SoftLayer or Amazon Web Services Available as: BLU in-memory processing, data skipping, actionable compression, parallel vector processing, Load & Go administration Netezza predictive analytic algorithms Fully managed DBaaS Fully integrated RStudio & R language On-premises private cloud Oracle compatibility Hybrid architecture Massively Parallel Processing (MPP) On disk data encryption and secure connectivity for Analytics

dashdb for Transactions Transactional database capabilities for best performance atop a fully-managed instance Excellent Transactional Performance Oracle Compatibility Robust Security For apps that

7 dashdb for Transactions Transactional database capabilities for best performance atop a fully-managed instance Excellent Transactional Performance Oracle Compatibility Robust Security For apps that need: Elastic scalability Fully-managed High transactional availability database Data model flexibility as a service Data mobility Row-organized tables for high Text search transactional performance Geospatial Oracle compatibility Available as: On disk data encryption and for Transactions secure connectivity Fully managed DBaaS On-premises private cloud Two Enterprise plans plus HA versions Hybrid architecture 2 cores, 8 GB memory, 500 GB SAN 12 cores, 128 GB memory, 1.4 TB SSD 9

dashdb for Transactions High Availability Configuration Reduced outage times for unplanned outages and maintenance activities Configuration includes both a primary server plus a warm standby server

8 dashdb for Transactions High Availability Configuration Reduced outage times for unplanned outages and maintenance activities Configuration includes both a primary server plus a warm standby server Log shipping keeps standby up-to-date for fast failovers Simplified client configuration floating IP address always points to current primary Not logged operations that compromise data recoverability are blocked Integrated Cluster Management Health of servers monitored Automated takeover Automatic Client Reroute Clients transparently connect to standby when primary fails for Transactions High Availability Configuration Primary Server Transaction Log Record Shipping Keeps standby current Standby Server 10

dashdb Local for Analytics Benefits of dashdb Technology with Fast Deployment into Private Cloud Environment Private or Virtualized Private Cloud Docker Container Technology dashdb Technology MPP

9 dashdb Local for Analytics Benefits of dashdb Technology with Fast Deployment into Private Cloud Environment Private or Virtualized Private Cloud Docker Container Technology dashdb Technology MPP with Automatic Scaling For apps that need: Highly flexible Elastic data scalability warehouse Optimized for High fast availability and flexible deployment into private or Data virtual model private flexibilityclouds Data mobility Uses Docker container technology Text search Built on top of dashdb technology, it Geospatial shares the benefits of Available as: BLU Acceleration in-memory columnar technology Netezza In-database Analytics Fully managed DBaaS Oracle Compatibility On-premises private cloud Massively Parallel Processing (MPP) with Hybrid architecture automated scaling capabilities to increase infrastructure efficiency 11

10 WHY dashdb Local? Ease of consumption Docker for super-fast provisioning and in-place update (<30min for MPP cluster setup) From a small laptop deployment for development purposes To large production clusters up to 100TB initially Management simplicity requires less skills and reduces TCO Automatic orchestrated configuration and adaption to infrastructure resources Embedded high availability and DR Market leading performance and agility Available in single node and MPP configurations of IBM s in-memory columnar database technology with automated scale up and out Enables economies of scale Increase utilization of private cloud infrastructures (and commodity hardware, including VMware) Host OS agnostic - For x86 processor architectures (others to be considered) Enjoy the privacy and control of on-premises and hosted private clouds Integrated with unified DW family including Hadoop providing a consistent user / application experience Built-in Fluid Query for seamless integration x-data stores Same DW engine supporting a write-once, run anywhere paradigm 12 Compatibility with Oracle and Pure Data for Analytics/ Netezza offering

11 Docker vs VMware (Virtual Machine) Docker is containerization technology It complements VMware, although NOT the same as VMware dashdb Local uses Docker for mainly packaging and shipping Portable and modular software packaging which is what Docker is great at Docker provides OS-Level process isolation While VMware offers Hardware virtualization (abstraction layer) Both technologies can be used in conjunction For example, Docker containers can be created inside VMs to make a solution ultra-portable Unlike VMware, Docker does not create an entire virtual operating system Host kernel is shared amongst Docker containers Applications only ship with what they need to run no more, no less Docker applications are therefore easier and more lightweight to deploy and faster to start up than virtual machines 13

12 Architecture: Host System on Customer s Cloud or On-Premises Data Center Docker Container Customer s Infrastructure Management Software or Agents Docker Supported OS dashdb Engine (columnar, in-memory) DSM Workload Monitoring dashdb Console CPU MIN: 2-N Cores(laptop) REC: 2 x 12 core Xeon 2690 v3 Memory Min: 8-N GB REC: 256GB Clustered Filesystem ie GPFS, GFS, VxFS, CXFS REC: 3000 MB/sec IO rate Scale Up CPU, Memory, IO according to your infrastructure availability 14

13 Elasticity Scale Out For expansion, a new node will be added to an HA group and the partitions will be re-balanced across all the containers Additional storage capacity can be added to the clustered file system by customer in the host OS HA Group #1 New Node Container A Container B Container C Container D Partition 1 Partition 7 Partition 13 Partition 19 Partition 2 Partition 8 Partition 14 Partition 20 Partition 3 Partition 9 Partition 15 Partition 21 Partition 4 Partition 10 Partition 16 Partition 22 Partition 5 Partition 11 Partition 17 Partition 23 Partition 6 Partition 12 Partition 18 Partition 24 Added Capacity Clustered File System 15

14 DB2 and dashdb Who Does What? Activity Provision hardware infrastructure Manage hardware infrastructure Database manger software installation Database manager instance creation DB2 Software (on-premise) DB2 Software (BYOL on IaaS) DB2 on Cloud Customer IaaS Vendor IBM Customer IaaS Vendor IBM dashdb Local (on-premise or private cloud) Customer or IaaS Vendor Customer or IaaS Vendor dashdb for Analytics Customer IBM IBM * IBM Customer IBM IBM * IBM Database creation Customer Customer IBM * IBM Database configuration Manage database environment Database manager software maintenance Customer Customer IBM * IBM Customer Customer Customer IBM Customer Customer IBM * IBM OS maintenance Customer Customer Customer IBM Setup encryption Customer Customer IBM IBM Database backup and restore Customer Customer Customer IBM IBM IBM dashdb for Transactions 18 Control and Flexibility Simplicity * For initial setup or subsequent updates of dashdb Local, customer simply deploys an IBMprovided Docker image with all software (including the database environment) pre-configured

15 Common Analytics Engine What is the Common Analytics Engine? + + SQL Engine MPP Scalability BLU Acceleration High Concurrency Built-in Deep Analytics INZA Functionality PostgreSQL compatibility Simplicity LOAD and GO Integrated Spark Engine Cloudant Integration R Integration SQL and NOSQL Support Netezza Application Compatibility All Analytic Workloads Advanced Security & Encryption DB2 Application Compatibility Oracle Application Compatibility Data Virtualization 19

16 Common Analytics Engine Where can a customer get the common analytics engine? Managed Public Cloud Service Software-defined Appliance Custom Deployable Software Hadoop / Spark Environment dashdb dashdb Local dashdb appliance DB2 BigSQL A Common Analytics SQL engine enabling true hybrid data warehousing solutions with portable analytics Data Virtualization Load and Go deployment Built-in analytics SQL and NOSQL Capabilities Cloudant Integration Consistent Management 21 Full MPP scalability DB2 Applications Application portability Oracle Applications Hybrid by design Netezza Applications In-memory performance Advanced Security Advanced Workload Management

17 Common Analytics Engine What applications work with the common analytics engine? APPLICATIONS Managed Public Cloud Service Software-defined Appliance Custom Deployable Software Hadoop / Spark Environment dashdb dashdb Local dashdb appliance DB2 BigSQL 22

18 Common Analytics Engine How do I choose which offering to use? Managed Public Cloud Service Software-defined Appliance Custom Deployable Software Hadoop / Spark Environment dashdb dashdb Local dashdb appliance DB2 BigSQL WORKLOAD DEPLOYMENT Fully managed Agile, Simple Integration with other cloud data services Analytics and reporting Structured and semistructured data GB -> TB On-premise or hosted Customer manages IaaS Customize compute / storage ratios Software only Docker-based provisioning Analytics and reporting Structured and semistructured data GB -> TB On-premise Appliance Software and hardware Analytics and reporting FPGA-powered performance Structured data GB -> TB -> PB On-premise Customized Software only Analytics and reporting Operational analytics Operational data store Structured and semistructured GB -> TB -> PB On-premise or hosted or fully managed Software only Leverages open source environment Analytics and reporting Minimal update/delete Structured, semistructured and unstructured data GB -> TB -> PB -> ZB WORKLOAD DEPLOYMENT IBM Corporation

19 Common Analytics Engine Data Virtualization Fluid Query Federation Layer High performance Enterprise readiness Heterogeneity Oracle compatibility High functionality Applications 24 You can leverage any of the above offerings as a federation server This image shows only a subset of all supported data sources

20 Common SQL Engine Covering a broader set of workloads than just analytics Managed Public Cloud Service Software-defined Appliance Custom Deployable Software Hadoop / Spark Environment ANALYTIC WORKLOADS dashdb dashdb Local dashdb appliance DB2 BigSQL A Common SQL Engine enabling true application and ecosystem portability for all workloads TRANSACTION WORKLOADS Managed Public Cloud Service Software-defined Custom Deployable Software dashdb for TX dashdb Local for TX DB2 25

dashdb Local for Analytics convergence with PDA Cloud first, inmemory, SIMD acceleration (BLU) for analytics Enhanced elasticity, UDX integration, and managed service

generation Netezza Appliance N3001 Integration with Hadoop for high speed query routing and bulk data movement Continued enhancements to caching algorithms, optimizer,

21 dashdb Local for Analytics convergence with PDA Cloud first, inmemory, SIMD acceleration (BLU) for analytics Enhanced elasticity, UDX integration, and managed service Further UDX integration and optimization SQL, Analytic, Application, ETL & Client Convergence Managed Service S/W Private Cloud H/W Accelerated Appliance 26 Latest generation Netezza Appliance N3001 Integration with Hadoop for high speed query routing and bulk data movement Continued enhancements to caching algorithms, optimizer, workload optimizations, etc. Next Generation PDA/Netezza/dashDB Appliance: Sailfish Big Data optimized Elastic Mixed workloads Tiered storage Extreme concurrency Real time data ingest

PDA client options towards hybrid data warehouse Democratize analytics with new cloud applications or augment your PDA system for DR or backup use cases dashdb Managed PDA Purchase or continue to

22 PDA client options towards hybrid data warehouse Democratize analytics with new cloud applications or augment your PDA system for DR or backup use cases dashdb Managed PDA Purchase or continue to realize value with PDA if an appliance is required 27 Integrate structured data with semistructured and unstructured data on Hadoop. Leverage Hadoop as an exploratory landing zone. Big Insights BigSQL Create flexible DWH environments on std. infrastructure, on-premise or hosted to replace or augment your PDA system Upgrade from PDA to the next-generation system (GA outlook Q317) dashdb Local Consume your dashdb Local entitlement on the Sailfish infrastructure dashdb Appliance

allowed All sync modes supported Integrated Cluster Management TSA monitors primary and performs automated takeover Automatic

23 DB2 on Cloud HADR Support HADR provides redundant copy(s) of database(s) to protect against server or storage failure Regular features and capabilities of HADR all apply to DB2 on Cloud Multiple standbys supported (up to three) Reads on Standby allowed All sync modes supported Integrated Cluster Management TSA monitors primary and performs automated takeover Automatic Client Reroute Client application transparently connects to standby 28 Primary Server HADR Keeps standby(s) in sync Standby Server

24 DB2 BYOL to Cloud - DB2 purescale in the Cloud purescale s scale-out architecture is a natural fit to exploit typical cloud infrastructure Powerful, commodity bare-metal servers e.g. 12 cores, 128 GB RAM 10 Gb TCP sockets member interconnect High-performance shared storage Three systems provide tie-breaking for HA For cases where tie-breaker disks aren t available Bare metal servers for performance and HA 29 With VMs, two members / CFs etc. might end up in the same physical machine SoftLayer Performance network-attached storage Capacity and IOPS for data and transaction logs shared across the systems M0 CFp M1 CFs App App App App M2 SoftLayer Performance Storage

25 IDAA on Cloud V1.1 The Foundation for Open and Easy Access to DB2 Data with dashdb under the covers! Rapid acceleration of existing business-critical queries Improve performance and enable new insights while retaining z Systems security and reliability Derive business insight from z/os transaction systems One integrated, hybrid platform, optimized to run mixed workload Simplicity and time to value With secure data movement and access Analytics Workload SQL IBM DB2 Analytics Accelerator on Cloud Transaction Processing SQL 30

IBM dashdb Local. Using a software-defined environment in a private cloud to enable hybrid data warehousing. Evolving the data warehouse

IBM dashdb Local. Using a software-defined environment in a private cloud to enable hybrid data warehousing. Evolving the data warehouse IBM dashdb Local Using a software-defined environment in a private cloud to enable hybrid data warehousing Evolving the data warehouse Managing a large-scale, on-premises data warehouse environments to