HBase 在 hulu 的使用和实践. hulu

Transcription

1 HBase 在 hulu 的使用和实践 hulu qianxi.zhang@hulu.com

2 About hulu

3 About me 张虔熙 ü 大数据平台组 ü 专注于分布式计算和存储技术 ü 热衷于参与开源社区贡献代码

4 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

5 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

6 Overview HBase version : Hadoop nodes :1000+ HBase nodes:200+ HBase table: 200+ HBase data size:700tb Cluster:4

7 Scenario Audience Platform( 用户画像系统 ) Log Storage( 日志存储系统 ) Online Bill Storage( 订单信息存储系统 ) OpenTSDB

8 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

9 Audience Platform( 用户画像系统 ) 用户画像 : 根据用户行为抽象出的一个标签化的用户模型 Data üprofile( 基本属性 ) üuser behavior( 用户行为 ) üthird party data( 第三方数据 ) ülabel( 标签 )

10 Audience Platform( 用户画像系统 ) Data characteristic üsparse(10^6 qualifier) ümulti-version(user behavior) Purpose ümarketing decision üpersonalized recommendation üadvertisement

11 Audience Platform( 用户画像系统 ) Kafka Spark Streaming Spark Service HDFS HBase Cache DB Bulk Load MapReduce HDFS

12 Audience Platform( 用户画像系统 ) Key technology üauto balance InputFormat üsnapshot

13 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

14 Region Size Distribution

15 Application Performance Problem ü Task execution time in MapReduce and Spark is positive correlation with Region Size ü Task execution timevaries wildly Resolve Bug ü Enable TableInputFormat auto balance(hbase.mapreduce.input.autobalance) ü Split large Region and merge small Region for InputFormat ü HBASE-15357(Wrong split/middle key)

16 Snapshot Snapshot ütable Meta ühfile Link Why Snapshot? üperformance üthe view of data at specific time

17 Snapshot Problem ücreate one snapshot per application? ühow toshare snapshot between application? Snapshot Service ümanage snapshot lifecycle üassign the reasonable snapshot tothe application

18 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

19 Online Bill Storage( 订单信息存储系统 ) Characteristic übill information üonline service üwrite more, readless üread delay < 1s

20 Online Bill Storage( 订单信息存储系统 ) Key technology üreplication ürpc Queue üreplica

21 Agenda Overview Audience Platform( 用户画像系统 ) Auto Balance InputFormat and Snapshot Online Bill Storage( 订单信息存储系统 ) Replication, RPC Queue and Replica

22 Replication Two datacenter,master-master Replication Write Read Write Read Replication Cluster A Cluster B Replication

23 Replication Problem ü Replication Table and CF configuration will be wrong if the table name includes namespace ü Previous design did not consider namespace üuse : when parsing tables and family, such as usertable:family1 übut Namespace and table segmentation is also :, such as namespace1:usertable:faimly1 Resolve ü HBASE-11386, HBASE-11393(Use Protobuf instead of string)

24 Replication Problem üsome data couldn t be replicated üpeerclusterznode under is of removed peer may never be deleted üif some regionserver crash, other regionserver couldn t take over the rest replication work since the method copyqueuesfromrsusingmulti fails Resolve ühbase-16135, HBASE-14476

25 RPC Queue Improve Performance ümulti RPC Queue ühbase Write Queue Get Queue Scan Queue More ücontrolling Queue Delay(CoDel) ühbase-15136

26 Replica Problem üwhen a RegionServer crash, the region on it is unavailable for a period Resolve üregion replicas ü There could be more than one replica for one region ü One primary replica could accept write and read operation ü Multi secondary replica only accepts read operation ü HBASE-10070

27 Replica Client Read and Write RegionServer RegionServer Read Only HBase Region (Primary) Region (Secondary) HDFS WAL HFile-1 HFile-2

28 Replica Client strategy ü Query primaryregion first ü If don t get the result in 10ms, add a query to the secondary replicas ü Take the firstanswer and cancel others Problem More ü The data in secondary replica may be stale. ü HBASE-11568(Async WAL to secondary replica)

29 Future Multi-Tenancy(HBASE-10994) Strong schema High availability

30 Reference

31 Thank you