OLTP on Hadoop: Reviewing the first Hadoop- based TPC- C benchmarks

Transcription

1 OLTP on Hadoop: Reviewing the first Hadoop- based TPC- C benchmarks Monte Zweben Co- Founder and Chief Execu6ve Officer John Leach Co- Founder and Chief Technology Officer September 30, 2015

2 The Tradi6onal Database Market Opera2onal - $24B Analy2cal - $11B 2

3 Origins of Hadoop Big Batch Processing Jobs to Batch AnalyGcs Batch Processing for Web Search Index Batch AnalyGcs Google File System (GFS) paper published Google Map Reduce paper published Hadoop created based on GFS and Map Reduce Hive created by Facebook for Analy6cs 3

4 Hadoop Not Just for Data Scien6sts Anymore Moving Hadoop Beyond Batch AnalyGcs to Power Real- Time Apps Distributed File System Java MapReduce Programs Read- Only Batch Analy6cs Distributed RDBMS SQL- 99 Queries Real- Time Updates with ACID Transac6ons Real- Time Concurrent Apps and Analy6cs 4

5 SQL- on- Hadoop Market OLTP OLAP 5

6 OLTP Requirements ACID Transac6ons High concurrency Secondary indexes Joins Stored procedures Triggers Constraints Foreign keys Sub- queries Views 6

7 ACID What is it? Why is it important? Reliable updates across mulgple rows and tables Atomicity Transac6ons are all or nothing Consistency Isola2on Durability Only valid data is saved Transac6ons do not affect each other Wrigen data will not be lost Use Cases Update a value and a secondary index Transfer $ between bank accounts $ $ Recover from batch update error without reload 7

8 Doesn t Hive Have ACID Transac6ons? Hive Can t Power an App: High Concurrency vs Batch TransacGons ~10 concurrent users Batch updates Table locking Up to 100,000 concurrent users Mul6- Version Concurrency Control Power applica6ons Opera2onal RDBMSs designed for an analy2c workload low concurrency users upda6ng Mul6ple writers wait behind each other Hive is not meant for low latency updates and deletes Hortonworks blog, Apache Hive ACID TransacGons in HDP 2.2 8

9 Snapshot Isola6on: High- Concurrency MVCC Transac6ons Leverages Mul6- Version Concurrency Control (MVCC) Each update creates a new version with a new 6mestamp Each transac6on can see its own virtual snapshot of database Writers don t block readers 9

10 Splice Machine The RDBMS on Hadoop Replace Oracle to scale out your applica2ons " Affordable, Scale- Out Commodity hardware " Elas2c Easy to expand or scale back " Transac2onal Real- 6me updates & ACID Transac6ons " ANSI SQL Leverage exis6ng SQL code, tools, & skills " Flexible Support opera6onal and analy6cal workloads 10

11 Compelling TCO: Sample Oracle Replacement Oracle RAC Costs List Price Unit 3 Year Cost Splice Machine Costs List Price Unit 3 Year Cost Oracle Database Enterprise Edi6on with RAC $37, $2,416,000 Splice Machine Annual Subscrip6on $10,000 7 $210,000 3 years DB Maintenance (22% list price/yr) 3 years Opera6ng System Support (Oracle Linux) $24, $1,594,560 $6,897 4 $27,588 Cloudera Enterprise Edi6on Annual Subscrip6on 3 years Opera6ng System Support (Oracle Linux) $7,500 8 $180,000 $6,897 4 $27,588 Server Costs (mid- range, Intel Xeon- based) $16,000 4 $64,000 Server Costs with Storage $5,000 8 $40,000 Primary Storage $143,360 1 $143,360 TOTAL $22,500 $457,588 TOTAL $228,922 $4,245,508 Assumes Oracle Enterprise Edi6on ($47.5K/CPU) and RAC ($23K/CPU) 90% TCO Reduc6on ($3.8M) 3-7x faster 11

12 TPC- C Benchmark Gold standard for OLTP Requires high concurrency transac6ons 5 very complex queries Models ERP order- entry: 12

13 Ini6al TPC- C Results* on Commodity Hardware Linear scalability for transacgonal workload on Hadoop * Unaudited Transac2ons per Minute (tmpc) 70,000 60,000 50,000 40,000 30,000 20,000 10, Nodes 13

14 Experimental TPC- C Results: Splice Machine HBase Fork 160,000 Linear scalability for transacgonal workload on Hadoop * Unaudited Transac2ons per Minute (tmpc) 140, , ,000 80,000 60,000 40,000 20,000 - tpmc tpmc (Hbase Patch) Nodes 14

15 3 rd - Party Applica6ons vs Ad- Hoc Queries Far more difficult to accommodate than analygc queries Dynamically Generated SQL No workarounds since generated code Not human- friendly Machine- generated symbols not meant to be interpreted Code not indented or styled No comments Complex Sub- Queries Object Rela6onal Mappings create many levels of sub- queries Applica6ons: object oriented to achieve code efficiency, reuse, and understandability Databases: rela6onal to achieve performance, ACID proper6es, and minimal storage High Concurrency Must support 1,000s to 10,000s of concurrent users

16 OLTP Applica6on: Unica Splice Machine powers the Unica ApplicaGon Campaign Defini2on Customer Segmenta6on by Household Output of 8 Segments By Country By Previous 12 Months Use of Select, Merge and Audience Processes Data Flow Process Select, Merge & Audience Processes Selec6ons Rules Household that Have Opted in Grouped by Loyalty and Non Loyalty SQL for Complex Selec2on Rule INSERT INTO UAC_639_1c SELECT A.CUSTOMER_MASTER_ID FROM UAC_639_v A WHERE A.CUSTOMER_MASTER_ID NOT IN ( SELECT UAC_639_14.CUSTOMER_MASTER_ID FROM UAC_639_14 UNION SELECT UAC_639_11.CUSTOMER_MASTER_ID FROM UAC_639_11 UNION SELECT UAC_639_12.CUSTOMER_MASTER_ID FROM UAC_639_12 U NION SELECT UAC_639_13.CUSTOMER_MASTER_ID FROM UAC_639_13); 16

17 OLTP Applica6on: Unica Splice Machine powers the Unica ApplicaGon Cross Channel Campaigns Architecture Real- Time Personaliza6on Ini2al Results vs. Oracle RAC 3-7x faster through parallelized queries Consumers Real- Time Ac6ons ¼ cost with commodity scale out 17

18 OLTP Applica6on: Redpoint Splice Machine powers the RedPoint Convergent MarkeGng ApplicaGon Campaign Defini2on Audience selec6on Dataflow Offers Data Flow Process Suppression Split rules 18

19 OLTP Applica6on: Redpoint Splice Machine powers the RedPoint Convergent MarkeGng ApplicaGon SQL for Complex Selec2on Rule INSERT INTO "AMEX"."RP_BC_69_2" SELECT a4."pid" AS "PID", a4."hhid" AS "HHID", (CASE WHEN a5."pid" IS NULL THEN 'N' ELSE 'Y' END) AS "Standard_Suppression_2044", (CASE WHEN a10."pid" IS NULL THEN 'N' ELSE 'Y' END) AS "Low_Value_Customer_1729", (CASE WHEN a13."pid" IS NULL THEN 'N' ELSE 'Y' END) AS "Not_Mailable_1128" FROM "AMEX"."RP_BC_69_1" a4 LEFT OUTER JOIN (SELECT a6."pid" FROM "AMEX"."RP_BC_69_1" a6 INNER JOIN "AMEX"."PERSON" a7 ON a6."pid" = a7."pid" WHERE EXISTS (SELECT a8."pid" FROM "AMEX"."PERSON_ACCOUNT_MASTER_FINANCE" a8 INNER JOIN "AMEX"."UD_ACCOUNT_DETAILS_FINANCE" a9 ON a8."rp_account_id" = a9."rp_account_id" WHERE a7."pid" = a8."pid" AND a9."account_status" = 'O')) AS a5 ON a4."pid" = a5."pid" LEFT OUTER JOIN (SELECT a11."pid" FROM "AMEX"."RP_BC_69_1" a11 INNER JOIN "AMEX"."PERSON" a12 ON a11."pid" = a12."pid" WHERE a12."customer_segment" = 1) AS a10 ON a4."pid" = a10."pid" LEFT OUTER JOIN (SELECT a14."pid" FROM "AMEX"."RP_BC_69_1" a14 INNER JOIN "AMEX"."PERSON" a15 ON a14."pid" = a15."pid" WHERE EXISTS (SELECT a16."pid" FROM "AMEX"."PERSON_ADDRESS" a16 INNER JOIN "AMEX"."ADDRESS" a17 ON a16."addr_id" = a17."addr_id" WHERE a15."pid" = a16."pid" AND a17."std_status_code" IN ('M', 'X', '7'))) AS a13 ON a4."pid" = a13."pid" ; 19

20 OLTP Applica6on: Redpoint Splice Machine powers the RedPoint Convergent MarkeGng ApplicaGon Architecture Ini2al Results vs. Oracle RAC Stream or Batch Updates Real- Time Offers 5-10x faster through parallelized queries Consumers Real- Time Data ¼ cost with commodity scale out 20

21 Digital Marke6ng ETL/Opera6onal Data Lake Use Cases Precision Medicine Fraud Detec6on Internet of Things 21

22 Sneak Peek: Splice Machine 2.0 First Hybrid, In- Memory RDBMS Powered by Hadoop and Spark Advantages OLAP + OLTP Massive scalability Spark in- memory compu6ng engine High- concurrency ACID transac6ons ANSI SQL Seamless integra6on Isolated resource management Benchmarks Simultaneous TPC- C and TPC- DS Never done before 22

23 Summary Power OLTP Apps on Hadoop First TPC- C benchmark run on Hadoop Leverage Hadoop for both OLTP and OLAP Stay Tuned! Hybrid, in- memory RDBMS powered by Hadoop and Spark Support mixed OLTP & OLAP workloads Look for simultaneous TPC- C & TPC- DS benchmark results

24 Ques6ons? Monte Zweben CEO Splice Machine John Leach CTO Splice Machine 24

25 OLTP on Hadoop: Reviewing the first Hadoop- based TPC- C benchmarks Monte Zweben Co- Founder and Chief Execu6ve Officer John Leach Co- Founder and Chief Technology Officer September 30, 2015

26 OLTP on Hadoop: Reviewing the first Hadoop- based TPC- C benchmarks Appendix

27 Unica Customer Marke6ng Service Provider Pilot Original number of records in each table: Household Master million Customer Preference million Household Computed Value million Customer Address Quality - 71 million Unica 6mings Strategic Segments EM (36 processes) (oracle: 5 hours - splice: 2 hours, 10 minutes) Strategic Segments DM (23 processes) (oracle: 3 hours - splice: 2 hours, 15 minutes) 27

28 Unica Demo Campaign Flowchart 1 " 8 segments of household IDs created " Preference indicated to receive direct mail " Iden6fied as valid (vs. ghost) household in advance " Segments used as inputs for other Flowcharts " Valid name and address " Loyalty customers converted to unique list " Household ID iden6fier by audience " Addi6onal Extract of Canadian household details " Real 6me updates to Splice DB table 28

29 Unica Demo Campaign (cont.) Household Segments: " a list of US based households (US DM HH) " the list of US based household just men6oned but where there has been at least one transac6on in the last 12 months. (US DM HH 12M) " a list of US based loyalty households (US PP DM HH) " a list of US based loyalty households with at least one transac6on in the last 12 months (US PP DM HH 12M) " a list of Canadian house holds (Canada DM) " a list of Canadian house holds with at least one transac6on in the last 12 months (Canada DM PP) " a list of Puerto Rico based households (Puerto Rico DM) " a list of Puerto Rico based loyalty households with at least one transac6on in the last 12 months (PR PP DM) 29