Trafodion Enterprise-Class Transactional SQL-on-HBase

Size: px

Start display at page:

Download "Trafodion Enterprise-Class Transactional SQL-on-HBase"

Abel Leon Cobb
5 years ago
Views:

1 Trafodion Enterprise-Class Transactional SQL-on-HBase

2 Trafodion Introduction (Welsh for transactions) Joint HP Labs & HP-IT project for transactional SQL database capabilities on Hadoop Leveraging 20+ years of database investment! Transactional SQL Complete: Full-function SQL Reuse existing SQL skills and improve developer productivity Protected: Distributed ACID transactions Data consistency across multiple rows, tables, SQL statements Efficient: Low-latency R/W transactions Optimized for real-time transaction processing applications Interoperable: Standard ODBC/JDBC access Works with existing tools and applications Data federation: Trafodion/HBase/Hive tables Enables multiple data model deployment + Hadoop Scalable: Elastic scale for high concurrency Provides elastic scalability as number of users / data grows Highly Available: For enterprise applications Leverages HBase / Hadoop replication Open: Hadoop and Linux distribution neutral Easy to add to existing infrastructure with no vendor lock-in Eco-system: Leverages large Hadoop eco-system Can use any tool or database accessing Hadoop 2

Trafodion Software architecture (3 layers) Client User and ISV Operational Applications Driver JDBC ODBC SQL CMP Database Connectivity Master Future WMS Compiler and Optimizer Workload Management

3 Trafodion Software architecture (3 layers) Client User and ISV Operational Applications Driver JDBC ODBC SQL CMP Database Connectivity Master Future WMS Compiler and Optimizer Workload Management Distributed Transaction Management DTM * ESP.... ESP SQL Parallelism Storage Engine HBase HBase Hive Data Store Integration Native HBase Tables KVS, Columnar via HBase API + coprocessors Trafodion Tables HDFS Relational Schema Direct HDFS access to Hive tables using HCatalog 3 * Executor Server Process

4 Optimized for varying operational workloads Optimized performance and efficiency OLT optimization for directed keyed access ESPs for parallel SQL operations Multi-layered ESP for complex plans Data Flow pipeline parallel architecture Reusable Masters and ESPs for efficiency Cached SQL plans eliminates recompilation Sophisticated Optimizer Leverages Equal Height Histograms Pushes down Filters e.g. row selection (start-stop key) Coprocessors e.g. aggregates Multi-Dimensional Access (MDAM) Secondary index access Service persistence (via Zookeeper) and automatic query resubmission Ethernet Node 1 Node 2 Node n Master Client Application FILTERS HBase HBase HBase COPROCESSORS Master HDFS HDFS HDFS HDFS Type 2 and Type 4 ODBC/JDBC driver ESP ESP ESP ESP ESP ESP ESP ESP ESP ESP HDFS 4

5 Trafodion Differences with HBase Data abstraction Physical Layout Column values ACID Guarantee Language API Row Key Index Secondary Indexes Salting Trafodion + HBase Relational schema Single column family with space-saving column encoding Explicitly defined and enforced data types Transactions across multiple SQL statements, tables, and rows SQL (Trafodian invokes native HBase API) Composite (multi-column) row key Arbitrary secondary key columns Allows specification of salt key-prefix for load balancing 5

6 Comprehensive DDL, DML, TCL, and utility support ANSI Core SQL 99 complaint + other SQL 99 and SQL 2003 support with Trafodion extensions Full featured DDL - CREATE/DROP/ALTER statements for tables, views, indexes, constraints Comprehensive data type support - numeric, character, varchar, date, time, interval Unicode encoding including UTF8, UCS2, and ISO8859-1for user data; UTF8 for metadata Full featured DML SELECT, INSERT, UPDATE, DELETE, UPSERT and MERGE statements JOIN (INNER, LEFT/RIGHT/FULL OUTER), UNION, WHERE, GROUP BY, HAVING, ORDER BY, SAMPLING, etc. Correlated and nested sub-queries Cursor support (non-holdable) Extensive SQL function support - aggregate, date/time, character, mathematical, OLAP, sequence, etc. Utilities Update Statistics, Explain, Control Query Shape, Command Line Interface Transaction Control BEGIN WORK, COMMIT WORK, ROLLBACK WORK, SET TRANSACTION Work-in-progress: Stability, Performance, Triggers, Referential Integrity, C++ UDFs, Java Stored Procedures, Bulk Loader, node/system failure Transaction Recovery, Grant/Revoke, 6

7 Trafodion Distributed Transaction Management Scalable Architecture Node 1 Node 2 Node n Transaction Resource Manager Library Manager Library Transaction Manager Transaction Resource Manager Library Manager Library Transaction Manager... Transaction Resource Manager Library Manager Library Transaction Manager HBase trx extends HBase classes 7

8 See for yourself Come discover and develop on Trafodion 8

Hadoop Beyond Batch: Real-time Workloads, SQL-on- Hadoop, and thevirtual EDW Headline Goes Here

Hadoop Beyond Batch: Real-time Workloads, SQL-on- Hadoop, and thevirtual EDW Headline Goes Here Marcel Kornacker marcel@cloudera.com Speaker Name or Subhead Goes Here 2013-11-12 Copyright 2013 Cloudera