HBase... And Lewis Carroll! Twi:er,

Transcription

1 HBase... And Lewis Carroll! Twi:er, 1

2 2010: Cloudera Architect 2011: Cloudera TAM/DSE : Cloudera Training focusing on Partners and Newbies 2H/2013: Partner Engineering focusing on ISV Prior experience as an SE in Business Intelligence, Middleware and Data 2

3 Our Discussion Crash Coure in HBase HBase Best (from Lewis Carroll) Goal: A broad understanding of HBase A feeling for the tradeoffs and considera@ons 3

4 The Three V s More Volume: Data than ever. (Bo:lenecks and costly storage) Variety: Types of data than ever. (Expensive/ ineffec@ve to model/schema) Velocity: Faster than ever. (Hard to capture, move, analyze in way) 4

5 Crash course in HDFS 1 NameNode METADATA: /user/diana/bar -> 3, 5 block 3 -> N3, N4, N6 block 5 -> N2, N3, N4 2 3 bar? 3, DataNode DataNode DataNode 5 5 Client Big blocks (64MB) Write once Batch Parallel processing via MapReduce Schema on Read DataNode DataNode DataNode

6 Cloudera s Distribu@on including Apache Hadoop (CDH) File System Mount UI Framework SDK FUSE-DFS HUE HUE SDK Workflow Scheduling Metadata APACHE OOZIE APACHE OOZIE APACHE HIVE Languages / Compilers APACHE PIG, APACHE HIVE, APACHE MAHOUT Data Integration Fast Read/Write Access APACHE FLUME, APACHE SQOOP APACHE HBASE HDFS, MAPREDUCE Coordination APACHE ZOOKEEPER

7 What is HBase Apache Managed Open Source Project Sparse, Sorted Map Based on Google s Big Table paper Implemented on top of HDFS Linearly Scalable Fault Tolerant Strongly Consistent Uses a Log Structured Merge Tree

8 When to use HBase When you need random access To huge data sets with huge concurrency With a well- defined access pa:ern (or a scalable cache)

9 An HBase Table Row Key Column Family One Column Family Two Row data Key Column contents=foo Family One Column fname=jeffrey Family Two col2=someval lname=bean Row data Key Column contents=foo Family One Column fname=jeffrey mname1=william Family Two col2=someval lname=bean data contents=foo fname=jeffrey mname2=francis mname1=william even more col2=someval col921=random lname=bean mname2=francis mname1=william even more more data col921=random mname2=francis fname=data even more more data col921=random fname=data more data Many rows, split into regions fname=data

10 Visualized by Hue

11 Tables distributed across regions (AKA shards ) From kiji.org

12 RegionServers and HDFS

13 Flushes and

14 Gemng at Data Very constrained query (get/put/scan) Data stored as un- typed byte arrays Lexicographically sorted by row key Co- accessed data co- located by column family Java API, HBase shell, REST, Thril, HUE Hive or Impala support for ad hoc query

15 A REAL HBase table h:ps:// engineering/inside- facebook- messages- server/

16 We get like... "Can HBase handle 2 million queries per second?" "Can I have some HBase performance numbers?" "Can I have sub- second query response?" "What's the maximum write throughput supported by HBase?" "Can HBase serve 500,000 concurrent queries over 4 petabytes of data?" 16

17 Or like... What's the right value for: hbase.hregion.max.filesize? hbase.hregion.memstore.flush.size? Java heap size for the regionserver process? 17

18 We get like... How big should my cluster be? What kind of nodes should I use? What column famillies do I need? What should my row key be? 18

19 And the answer...? It depends! 19

20 `Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where- - ' said Alice. `Then it doesn't ma:er which way you go,' said the Cat. 20

21 21 Tune for the workload!

22 Write heavy workload: bigger memstore Block Updates Global Limit Flush size MemStore put put put put flush HFile HFile HFile 22

23 Read Heavy Workload: bigger block cache get get get Block Cache HFile Bulk Load 23

24 Mixed Workload: Tuned get get get Block Cache Block Global Flush MemStore put put put put flush HFile HFile HFile 24

25 25 Design Schema for Access Pa:ern

26 26 Web Clicks Schema: An RDBMS

27 27 Choosing a Row Key and Column Families

28 28 Avoiding Hot Spots with Promoted Field Key

29 29 Controlling Display order by Reverse Timestamp

30 30 Design Schema for Performance

31 31 Design Schema for Performance

32 32 Bloom Filters Help Reads...

33 33...Unless you frequently update most rows

34 34 Tradeoffs

35 Tall/Narrow tables Split efficiently Logical rows span physical rows Good for scans Generally recommended 35

36 But... Flat- Wide Tables... Are good for random gets 36

37 Cell size Big Cells: Grow HBase block size But that breaks performance on small cells 37

38 38 Other Concerns

39 Don't Colocate MapReduce and HBase...Unless it's to read or write from HBase 39

40 Do major not as specified in the default 40

41 Split regions......but not as is default 41

42 WAL on puts trades performance for durability Random gets v sequen@al scans affect cache considera@ons Region size and cluster size affect query throughput 42

43 Improper choices... "In that lives instability: and in that lives unavailability. Visit either you like. They're both mad." 43

44 So the Test environment matches Test suite matches Expect to iterate Expect to redesign Use Cloudera Manager to detect HBase is dependent upon the 44

45 Lastly A good use case Properly configured With a well understood applica@on Will scale! Ad hoc query won't (use Impala). 45

46 More HUE: h:p://gethue.com Oreilly s HBase, the Defini<ve Guide by Clouderan Lars George Cloudera University: h:p://university.cloudera.com/ h:p://hbase.apache.org and mailing lists Cloudera Forums HBase at ebay, Facebook, StumbleUpon 46

47 47