Rails on HBase. Zachary Pinter and Tony Hillerson RailsConf 2011

Transcription

1 Rails on HBase Zachary Pinter and Tony Hillerson RailsConf 2011

2 What we will cover

3 What is it?

4 What are the tradeoffs that HBase makes?

5 Why HBase is probably the wrong choice for your app

6 Why HBase might be the right choice for your app

7 + How to use HBase with Rails

8 What we won t cover

9 This is not a deep dive

10 This is not a guide to setting up a production HBase cluster *See

11 So... What is HBase?

12 HBase is based on Bigtable

13 HBase Grew out of Hadoop

14 Core Concepts

15 Distributed Column Oriented Database

16 Tables column key core:link row llb0ga llb260 schedule/detail/17886

17 Column Families core column family stats column family key core:link stats:hits llb0ga llb260 public/schedule/detail/

18 Column Families Must be defined at creation or added to schema later Should group data accessed and sized similarly

19 Columns Can be dynamically added given that the family exists Stored as a byte array - do what you need with it

20 Column Families: Data Locality

21 Regions master llb0ga... llbee9 lld92q... lq1rro Regionserver Regionserver

22 Main Operations Get Put Incr Delete Scan

23 JRuby based shell

24 hbase shell ruby p180 :002 > status 1 servers, 0 dead, average load ruby p180 :003 > (1..10).each{ status } 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load 1 servers, 0 dead, average load => ruby p180 :004 >

25 API REST Thrift Apache Avro Java API

26 What are the tradeoffs that HBase makes? NoSQL = Tradeoffs

27 Highly Distributed Pros Built-in sharding No single server holds all the data Different servers operate on different slices of the data Cons Makes joins difficult/impossible Limited real-time queries/sorting

28 Sorted Row Keys Bytearray keys Any kind of data you want Sorted in byte order All row access through the key

29 Sorted Row Keys Pros Real-time random lookups by key Real-time range queries Cons Conflates unique identifier with default sort Keys are an integral part of schema design, requiring extra consideration

30 Map / Reduce Pros Scales linearly: more servers = faster response times Operates on huge datasets (billions of rows) Works well with dynamic/non-uniform schemas Cons Scans every row per query * Not real-time (batch operation)

31 HBase vs. X Seven Databases in Seven Weeks Eric Redmond Don t Miss His Talk!!

32 Why HBase is probably the wrong choice for your app

33 HBase Limitations

34 HBase Limitations Limited real-time queries (key lookup, range lookup) Limited sorting (one default sort per table) Limited transactions Manual indexing Joins/Normalization Storage of large binary data *

35 HBase Requirements At least 5 Servers (Zookeeper, Region Server, Name Nodes, Data Nodes, etc) Memory Intensive (2-4gb bar minimum per server, depending on server role) CPU Intensive Rough EC2 Cost: 5 c1.xlarge reserved for 1 year = over $850/month (not counting bandwidth!)

36 Why HBase might be the right choice for your app

37 Large data sets Ability to aggregate and analyze billions of rows Starts to shine when relational databases break down Is your db already sharded? Are you using SQL queries that take hours to run?

38 Hadoop Integration HDFS for storing files Native Map/Reduce Supports data locality Doesn t require data to be transferred

39 Flexible schema Supports non-heterogeneous data through column families with mixed key/value pairs Ability to refactor the data store via map/reduce (beats a multi-day ALTER TABLE command)

40 Rails on HBase

41 hbase-stargate Uses Stargate, HBase REST server

42 Rhino Thrift API Seems to be abandoned?

43 Massive Record It works! Uses Thrift API

44 Kwisatz: An URL Shortener

45 Demo: JRuby Scripts

46 Demo: Rails Front-End

47 Source Demo source code github.com/effectiveui/jruby-hbase-demo github.com/effectiveui/rails-hbase-massive-record

48 Getting Set Up (Mac) brew install hbase Cloudera VM

49 Recap

50 Where Would I Use HBase?

51 Where is HBase s Sweet Spot

52 Where Would I Use Rails With HBase?

53 Questions?

54 Thank You! EffectiveUI.com