Data Warehouse appliances: IBM Pure Data for Analytics

Transcription

1 May 2017 Data Warehouse appliances: IBM Pure Data for Analytics Fabio Bresciani, Cloud & Cognitive, IBM Italia

2 1997 IBM ebusiness 1927 Italy 1956 Data storage industry creation 1969 IBM technology guided Apollo mission to the moon 1981 The IBM PC 2011 IBM Watson 1911 Computing-Tabulating- Recording (CTR) 1924 International Business Machines 1935 Training courses for Women 1944 First machine to handle long calculations automatically 1961 The Selectric Typewriter 1962 First computerdriven airline reservation system 1971 Floppy disk 1969 Magnetic strips on credit cards 1973 UPC bar codes 1986 IBM scientists won the Nobel Prize 1997 Supercomputer defeated the best chess player 2

3 Cosa fa IBM? Analytics Consulting Services Security IBM Technical and Infrastructure Services Research Cloud Commerce Healthcare Systems Internet of Things

4 IBM Research Centri di Ricerca IBM 5.7 B$ in R&D (6% del fatturato) Almaden Austin New York São Paulo/ Rio de Janeiro Dublin Zurich Haifa Delhi/Bengaluru Nairobi Johannesburg Beijing/Shanghai Tokyo Melbourne 13 centri di Ricerca in 6 continenti, fra cui quello di Zurigo guidato dall italiano Alessandro Curioni Per 24 anni consecutivi l impresa leader nei brevetti brevetti U.S. nel premi Nobel master inventor in 43 paesi Concentrati in aree strategiche: Cloud Computing, Analytics, Security Cognitive Computing, Healthcare 4

5 Traditional Data Warehouses are just too complex They do NOT meet the demands of advanced analytics on big data. Too complex an infrastructure Too complicated to deploy Too much tuning required Too long to get answers Too inefficient at analytics Too many people needed to maintain Too costly to operate 5

6 Big Data Floods Traditional Database Systems 6

7 Let s Simplify This Mess 7

8 And Bring Analytics In To The Warehouse 8

9 Netezza Simplicity Legacy RDBMS Create Table - Logical Model Netezza DDL Create Table - Logical Model CREATE TABLE CRRADMIN.OT_ORDER_EVENTS CREATE TABLE CRRADMIN.OT_ORDER_EVENTS ( ( TRADE_DATE DATE NOT NULL, ORIGIN_SYS_CD VARCHAR2(32 BYTE) NOT NULL, TRADE_DATE DATE NOT NULL, ORIGIN_SYS_EVENT_SEQ VARCHAR2(32 BYTE) NOT NULL, ORIGIN_SYS_CD VARCHAR (32) NOT NULL, EVENT_ID Allocate Space NUMBER(9) NOT NULL, ORIGIN_SYS_EVENT_SEQ VARCHAR (32) NOT NULL, EVENT_CLASS_CD VARCHAR2(32 BYTE) NOT NULL, EVENT_ID INTEGER NOT NULL, EVENT_DATETIME TABLESPACE OTR_DATA" DATE LOCAL NOT NULL, EVENT_CLASS_CD VARCHAR (32) NOT NULL, ORIGIN_SYS_REF (PARTITION BY RANGE VARCHAR2(32 (TRADE_DATE) BYTE) NOT ( NULL, EVENT_DATETIME TIMESTAMP NOT NULL, ORIGIN_SYS_PARENT_REF VARCHAR2(32 BYTE), PARTITION P VALUES LESS THAN ( ) ORIGIN_SYS_REF VARCHAR (32) NOT NULL, ORIGIN_SYS_ORDER_REF VARCHAR2(32 BYTE), ORIGIN_SYS_RELATED_REF PCTFREE 10 INITRANS VARCHAR2(32 2 MAXTRANS BYTE), 255 ORIGIN_SYS_PARENT_REF VARCHAR (32), Create Indexes ORIGIN_SYS_GROUP_REF STORAGE(INITIAL VARCHAR2( NEXT BYTE), MINEXTENTS ORIGIN_SYS_ORDER_REF 1 VARCHAR (32), ORIGIN_SYS_DATETIME MAXEXTENTS DATE NOT NULL, ORIGIN_SYS_RELATED_REF VARCHAR (32), CREATE INDEX OTOE_EVENT_ID TRADE_ID PCTINCREASE 0 FREELISTS NUMBER(9), 1 FREELIST GROUPS 1 BUFFER_POOL ORIGIN_SYS_GROUP_REF VARCHAR (32), BASKET_ID DEFAULT, PCTFREE NUMBER(9), ON 10 INITRANS 2 MAXTRANS 255 ORIGIN_SYS_DATETIME TIMESTAMP NOT NULL, ORDER_ID CRRADMIN.OT_ORDER_EVENTS(EVENT_ID) STORAGE(INITIAL NUMBER(9), NEXT MINEXTENTS TRADE_ID 1 INTEGER, BASKET_NAME TABLESPACE VARCHAR2(32 OTR_IDX BYTE), MAXEXTENTS BASKET_ID INTEGER, SQC_SQN NOLOGGING VARCHAR2(20 BYTE), EXECFAC_ID PCTINCREASE 0 FREELISTS NUMBER(9), 1 FREELIST GROUPS 1 BUFFER_POOL ORDER_ID INTEGER, PCTFREE 10 CUSTOMER_REF DEFAULT, INITRANS VARCHAR2(255 2 BYTE), BASKET_NAME VARCHAR (32), Logical INSTRUMENT_ID PARTITION Model Only P NUMBER(9), VALUES LESS THAN ( ) SQC_SQN VARCHAR (20), MAXTRANS 255 No SYMBOL indexes PCTFREE 10 INITRANS VARCHAR2(64 2 MAXTRANS BYTE), 255 EXECFAC_ID INTEGER, STORAGE(BUFFER_POOL DEFAULT) STORAGE(INITIAL NEXT MINEXTENTS CUSTOMER_REF 1 VARCHAR (255), ); No Physical Tuning/Admin NOPARALLEL MAXEXTENTS INSTRUMENT_ID INTEGER, NOCOMPRESS Distribute PCTINCREASE Data by Columns 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL / or Round Robin SYMBOL VARCHAR (64), DEFAULT, CREATE INDEX OTOE_TRADE_ID PARTITION P VALUES LESS THAN ( ) ) 9 ON PCTFREE 10 INITRANS 2 MAXTRANS 255 DISTRIBUTE ON (ORIGIN_SYS_REF); CRRADMIN.OT_ORDER_EVENTS(TRADE_ID)

10 IBM PureData System for Analytics The Simple Data Warehouse Appliance for Serious Analytics Purpose-built analytics appliance Integrated database, server and storage Standard interfaces Low total cost of ownership What makes it different? Speed x faster than traditional custom systems 1 Simplicity - minimal administration and tuning Scalability - petabyte+ scale user data capacity Smart - high performance, advanced analytics 10

11 Massively Parallel Processing Architecture Divide and conquer MPP Shared Nothing concept Divides the work in smaller tasks A big task is sliced vertically into a series of smaller tasks Benefits The smaller tasks run independently The work is automatically balanced among the tasks to minimize the time to complete Each task is assigner the same amount of physical resources Communication between is made only at the beginning and end of the task A large task completes in a short elapsed time Maximizes use of resources Points of Attention Complexity on administration and management Communication bottlenecks 11

12 Data Warehouse Workload Fewer requests, lots of data manipulation Transactional System used for BI Request Request CPU General Purpose Storage 12

13 Data Warehouse Workload Transaction systems are inefficient for data shuffling Transactional System used for BI Results Request CPU General Purpose Storage 13

14 Data Warehouse Blades Designed for Tera-scale Business Intelligence IBM Pure Data System Results Request CPU Intelligent Storage Asymmetric Massively Parallel Processing 14

15 Data Warehouse Blades Highly efficient data movement IBM Pure Data System Results 2% of CPU requirements 1% of network traffic Request CPU Intelligent Storage Asymmetric Massively Parallel Processing 15

16 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza Appliance Client TRU64 HP-UX WINDOWS LINUX ODBC 3.X JDBC Type 4 OLE-DB SQL/92 SQL Compiler 1 2 S-Blade Processor & streaming DB logic S-Blade Processor & streaming DB logic Query Plan Execution Engine 3 S-Blade Processor & streaming DB logic Source Systems ETL Server DBA CLI 3rd Party Apps High-Speed Loader/Unloader Optimize Admin Front End DBOS SMP Host Network Fabric Ÿ Ÿ Ÿ 920 High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader 16

17 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza TwinFin Appliance Client TRU64 HP-UX Source Systems WINDOWS ETL Server DBA CLI 3rd Party Apps LINUX SQL High-Speed Loader/Unloader SQL Compiler Query Plan Optimize Admin SQL Front End SMP Host Snippets Execution Engine DBOS Network Fabric Ÿ Ÿ Ÿ 920 S-Blade S-Blade S-Blade Processor & streaming DB logic High-Performance Database Engine Streaming joins, aggregations, sorts S-Blade Processor & streaming DB logic Processor & streaming DB logic Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader 17

18 S-Blade Data Stream Processing select DISTRICT, PRODUCTGRP, sum(nrx) from MTHLY_RX_TERR_DATA where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' FPGA Core CPU Core Slice of table MTHLY_RX_TERR_DATA (compressed) 18 Uncompress Project select DISTRICT, PRODUCTGRP, sum(nrx) Restrict, Visibility Complex Joins, Aggs, etc. where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' sum(nrx)

19 Asymmetric Massively Parallel Processing SOLARIS AIX Netezza TwinFin Appliance Client TRU64 HP-UX Source Systems WINDOWS ETL Server DBA CLI 3rd Party Apps LINUX ODBC 3.X JDBC Type 4 OLE-DB SQL/92 High-Speed Loader/Unloader SQL Compiler Query Plan Optimize Admin Front End SMP Host Consolidate Execution Engine DBOS Network Fabric Ÿ Ÿ Ÿ 920 S-Blade S-Blade S-Blade Processor & streaming DB logic High-Performance Database Engine Streaming joins, aggregations, sorts, etc. S-Blade Processor & streaming DB logic Processor & streaming DB logic Processor & streaming DB logic Massively Parallel Intelligent Storage High Performance Loader 19

20 Inside the IBM PureData System for Analytics N3001 Optimized Hardware + Software Hardware accelerated AMPP Purpose-built for high performance analytics Requires no tuning SMP Hosts SQL Compiler Query Plan Optimize Admin Disk Enclosures User data, mirror, swap partitions High speed data streaming Snippet Blades Hardware-based query acceleration with FPGAs Blistering fast results Complex analytics executed as the data streams from disk 20

21 Disk Mirroring and Failover Primary Mirror Temp All user data and temp space mirrored Disk failures transparent to queries and transactions Failed drives automatically regenerated Bad sectors automatically rewritten or relocated 21

22 S-Blade Failover and Query Continuity S-Blades Drives automatically reassigned to remaining S-Blades within a chassis Read-only queries (that have not returned data yet) automatically restarted Transactions and loads interrupted Loads automatically restarted from last successful checkpoint 22

23 23

24 ZoneMap Pure Data's Anti-Index: Automatic Query Acceleration Col 1: Date Col 2: Zip Zone Maps Base Table Data Blocks Indexes are additional structures on disk, derived from the base table to accelerate locating information ZoneMaps are a method within the storage system without the need for add l structures on the disk to indicate where data DOES NOT reside The NPS system > Automatically stores min. & max. values of all integer columns in each file extent > Uses the ZoneMap information to determine if a given extent should be read Zone Maps: 18 out of 48 Extents Read 24 Both indices and ZoneMaps are techniques to avoid full table scans, but Netezza s ZoneMap approach is: Automatic; and Does not require a separate structure to create, tune & maintain

25 25

26 26

27 Response Time Distributions and Performance SPU Node 1 CPU Disk I/O Network Response time is affected by the completion time for all of the SPUs in the AMPP array. A distribution method that distributes data evenly across all SPUs is the single most important factor that can influence overall performance! 27

28 Response Time Hash Distributions and Data Skew SPU Node 1 CPU Disk I/O Network Gender = M or F will distribute all table records on 2 SPUs 6 7 Select a distribution key with unique values and high cardinality 28

29 Response Time Hash Distributions and Processing Skew SPU Node CPU Disk I/O Network Jan Feb Mar Apr May Jun Jul Using a DATE column as the distribution key may distribute rows evenly across all S- Blades. However, most analysis (queries) is performed on a date range. Massive parallel processing won t be achieved when all of the records to be processed for a given date range are located on a single or a few S-Blades) 29

30 Commonly JOINed Tables: Use the Same Distribution Key For tables commonly joined (WHERE clause) use the same column/distribution key used in the JOIN! CREATE TABLE customer ( c_custkey integer, c_name character varying(25), c_address character varying(40), c_nationkey integer, c_phone character(15), c_acctbal numeric(15,2), c_mktsegment character(10), c_comment character varying(117) ) DISTRIBUTE ON ( c_custkey ); CREATE TABLE orders ( o_orderkey integer, o_custkey integer, o_orderstatus character(1), o_totalprice numeric(15,2), o_orderdate date, o_orderpriority character(15), o_clerk character(15), o_shippriority integer, o_comment character varying(79) ) DISTRIBUTE ON ( o_custkey ); 30

31 Impact of Distribution Key on Table Join Performance Identical Distribution Keys CREATE TABLE ORDERS (ORDER_NO, CUST_NO, ) DISTRIBUTE BY HASH (CUST_NO) CREATE TABLE CUSTOMERS (CUST_NO, ) DISTRIBUTE BY HASH (CUST_NO) SELECT FROM ORDERS O, CUSTOMERS C WHERE O.CUST_NO = C.CUST_NO ORDERS Table 100, 1, 135, 4, 190, 4, 222, 8, 118, 6, 149, 7, 206, 3, 282, 11, 112, 2, 168, 5, 174, 12, 211, 2, No data movement is required Join Processing Join Processing Join Processing CUSTOMERS Table 1, 4, 8, 10, 3, 6, 7, 11, 2, 5, 9, 12, 31

32 Impact of Distribution Key on Table Join (cont.) Different Distribution Keys CREATE TABLE ORDERS (ORDER_NO, CUST_NO, ) DISTRIBUTE BY HASH (ORDER_NO) CREATE TABLE CUSTOMERS (CUST_NO, ) DISTRIBUTE BY HASH (CUST_NO) SELECT FROM ORDERS O, CUSTOMERS C WHERE O.CUST_NO = C.CUST_NO ORDERS Table 118, 6, 135, 4, 174, 12, 282, 11, 112, 2, 168, 5, 206, 3, 222, 8, 100, 1, 149, 7, 190, 4, 211, 2, Data shipping Data movement is required Shipped rows Shipped rows Shipped rows Join Processing Join Processing Join Processing CUSTOMERS Table 1, 4, 8, 10, 3, 6, 7, 11, 2, 5, 9, 12, 32

33 Workload Management Workload Management (WLM) provided optional functionality to manage resources and prioritize usage across a diverse multi-user environment to meet the need of mixed user workloads Guaranteed Resource Allocation (GRA) Mechanism to allocate NPS resources among groups of users in a multi-user environment Prioritized Query Execution (PQE) Finer control over resource allocation by extending the notion of query priorities from scheduling to execution Short Query Bias (SQB) Ensures users with short queries receive faster, higher, biased query response time under heavy system workloads Workload Limits (GRA) You can use the JOB MAXIMUM attribute of the group definition to control the number of actively running jobs submitted by that group User Requests Request Queues Power User Minimum Resource Guarantees Departmental User Admin Tasks 33

34 Appliances are easy to monitor 34

35 Traditional storage is not ready for the digital transformation Object storage solves the problems of scale, management and costs BLOCK & FILE Traditional Storage Block storage = fixed size blocks in rigid arrangement, ideal for enterprise databases. File storage - sharing files in hierarchically nested folders, ideal for active documents. OBJECT Storage for unstructured data (photos, videos, audios, ) and big data. Object is data with metadata. Basis for cloud storage, spans geographies. High scalability (seamless, multi-dimensional scaling). Ease of use. Lower cost of operations. 35 Page 35

36 36 Page 36

37 37 Page 37

38 Writing Data to IBM Cloud Object Storage text Original Data Let s store a Video! $ Accesser 1 Objects are sent to the Accesser via the S3 Compatible API or Openstack Swift Compatible API $ 2 Each object is segmented into 4MB segments e.g a 1GB object will be segmented into 250 segments. 4MB 4MB 4MB 4MB 4MB 38 38

39 Writing Data to IBM Cloud Object Storage text $ 4MB 4MB 4MB 4MB Each segment is encrypted and then sliced. $ 4MB 4MB 4MB 4MB Erasure Coding Expansion 4 Erasure coding is used to transform the data into a customizable number of slices 39 39

40 Writing Data to IBM Cloud Object Storage text $ 4MB 4MB 4MB 4MB Each slice is written to a separate storage node. In this example, the storage nodes are geographically dispersed across 3 sites. SITE 1 SITE 2 SITE 3 Storage Nodes SITE 1 SITE 2 SITE 3 SITE 2 SITE

41 Reading Data from IBM Cloud Object Storage text 4MB 4MB 4MB 4MB 4MB Storage Nodes SITE 1 SITE 2 SITE 3 SITE 1 SITE 2 SITE 3 SITE 2 SITE 3 With this 12/7 Information Dispersal Algorithm, a read can still be executed with any five storage nodes being unavailable

42 Reading Data from IBM Cloud Object Storage text $ Storage Nodes SITE 1 SITE 2 SITE 3 SITE 1 SITE 2 SITE 3 SITE 2 SITE 3 Even an entire site outage (plus one additional storage node outage) can be tolerated

43 IBM Cloud Object Storage EFFICIENCY How to build a highly reliable storage system for 1 Petabyte of usable data? RAID 6 + Replication IBM Cloud Object Storage Original 1.20 PB Raw Onsite mirror 1.20 PB Raw Remote copy 1.20 PB Raw 1 PB 3.6 PB x 3.6x 3 FTE Replication/backup Usable Storage Raw Storage 6TB Disks Racks Required Floor Space Ops Staffing Extra Software 1 PB 1.7 PB x 1.7x.5 FTE None $ 70% + TCO Savings Page 43