TRANSACTIONS AND ABSTRACTIONS

Size: px

Start display at page:

Download "TRANSACTIONS AND ABSTRACTIONS"

Dale Bell
5 years ago
Views:

1 TRANSACTIONS AND ABSTRACTIONS OVER HBASE Andreas Continuuity

2 AGENDA Transactions over HBase: Why? What? Implementation: How? The approach Transaction Manager Abstractions Future

3 WHO WE ARE Simple Access to Powerful Technology Continuuity Reactor: the world s first scale-out application server for Hadoop Fast, easy development, deployment and management of Hadoop and HBase apps Collect, Process, Store, and Query data Real-time Stream Processing

4 REAL-TIME STREAM PROCESSING Queue... Flowlet... HBase Table

5 REAL-TIME STREAM PROCESSING Queue... Flowlet... HBase Table

6 WRAP IN TRANSACTION! Queue... Flowlet... HBase Table...

7 TRANSACTIONS: WHAT? Atomic - Entire transaction is committed as one Consistent - No partial state change due to failure Isolated - No dirty reads, transaction is only visible after commit Durable - Once committed, data is persisted reliably

8 HBASE: QUICK WHAT? Open-source non-relational distributed columnoriented database modeled after Google s BigTable Named Tables Row key x Column key x timestamp -> Value Massive Scale Key space partitioned into Regions.

9 WHAT ABOUT HBASE? Atomic operations on cell value: checkandput, checkanddelete, increment, append Atomic batch of operations on rows within region No cross region atomic operations support No cross table atomic operations support No multi-rpc atomic operations support

10 IMPLEMENTATION OMID style Snapshot Isolation Multi-Version Concurrency Control Cell version (timestamp) = transaction ID Reads exclude uncommitted transactions (for isolation) Optimistic Concurrency Control Conflict detection at commit of transaction Rollback in case of conflict (whichever commits later)

11 OPTIMISTIC CONCURRENCY CONTROL Avoids cost of locking rows and tables No deadlocks or lock escalations Cost of conflict detection and possible rollback is higher Good if conflicts are rare: short transaction, disjoint partitioning of work

12 TRANSACTIONS IN CONTEXT ZooKeeper Client 1 Tx Manager (active) Tx Manager (standby) Client 2 HBase Master 1 RS 1 RS 3 Client N Master 2 RS 2 RS 4

13 TRANSACTION LIFE CYCLE Client RPC API Tx Manager start tx start start tx write to HBase do work in progress try commit commit check conflicts roll back in HBase try abort failed none time out failed succeeded abort complete V invalidate invalid X

14 TRANSACTION MANAGER Create new transactions Provides monotonically increasing write pointers Maintains all in-progress, committed, and invalid transactions Detect conflicts Transaction = Write Pointer: Timestamp for HBase writes Read pointer: Upper bound timestamp for reads Excludes: List of timestamps to exclude from reads

15 TRANSACTION MANAGER Simple & Fast All required state is in-memory Single point of failure? Periodically persist snapshot of all state Write-ahead log for all changes since last snapshot Secondary Tx Manager watches for failure of Primary Failover can happen quickly

16 TRANSACTION CLEANUP Some transactions time out or fail to roll back Invalid transactions must be excluded from reads Exclude list can get large over time Old versions may not be visible to any transaction TTL (time-to-live) expires old versions

updates from transaction snapshot in HDFS Purges data from

17 DATA JANITOR RegionObserver coprocessor Maintains in-memory snapshot of recent invalid & in-progress sets Periodically updates from transaction snapshot in HDFS Purges data from invalid transactions and older versions on flush & compaction

18 ABSTRACTION Dataset implements TransactionAware interface void starttx(transaction tx); Collection<byte[]> gettxchanges(); boolean committx() throws Exception; boolean rollbacktx() throws Exception; Other Data Stores than HBase HBase, LevelDB, HyperSQL, In-Memory,

19 TRANSACTION AWARE Modulation of ACID by dataset implementation Granularity of keys - row, column, In-memory caching of writes Skip rollback

20 WHAT S NEXT? Continue Scaling Tx Manager Transaction Groups? Integration across other transactional stores Open Source

21 ` Looking for the chance to work with a team that is defining a new category within Big Data? We are hiring!! careers[at]continuuity.com!! Andreas andreas[at]continuuity.com

TRANSACTIONS OVER HBASE

TRANSACTIONS OVER HBASE Alex Baranau @abaranau Gary Helmling @gario Continuuity WHO WE ARE We ve built Continuuity Reactor: the world s first scale-out application server for Hadoop Fast, easy development,