HBase在Hulu的使用和实践

1 .HBase在hulu的使用和实践 张虔熙 @ hulu qianxi.zhang@hulu.com 

2 .About hulu 

3 .About me • 张虔熙 ü软件工程师@Hulu大数据平台组 ü专注于分布式计算和存储技术 ü热衷于参与开源社区贡献代码 üqianxi.zhang@hulu.com 

4 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

5 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

6 .Overview • HBase version ： 1.2.0 • Hadoop nodes ：1000+ • HBase nodes：200+ • HBase table： 200+ • HBase data size：700TB • Cluster：4 

7 .Scenario • Audience Platform(用户画像系统) • Log Storage(日志存储系统) • Online Bill Storage(订单信息存储系统) • OpenTSDB 

8 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

9 .Audience Platform(用户画像系统) • 用户画像：根据用户行为抽象出的一个标签化的用户模型 • Data üProfile(基本属性) üUser behavior(用户行为) üThird party data(第三方数据) üLabel(标签) 

10 .Audience Platform(用户画像系统) • Data characteristic üSparse(10^6 qualifier) üMulti-version(User behavior) • Purpose üMarketing decision üPersonalized recommendation üAdvertisement 

11 .Audience Platform(用户画像系统) Spark Service Spark Kafka Streaming HDFS HBase Cache DB Bulk Load HDFS MapReduce 

12 .Audience Platform(用户画像系统) • Key technology üAuto balance InputFormat üSnapshot 

13 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

14 .Region Size Distribution 

15 .Application Performance • Problem ü Task execution time in MapReduce and Spark is positive correlation with Region Size ü Task execution time varies wildly • Resolve ü Enable TableInputFormat auto balance(hbase.mapreduce.input.autobalance) ü Split large Region and merge small Region for InputFormat • Bug ü HBASE-15357(Wrong split/middle key) 

16 .Snapshot • Snapshot üTable Meta üHFile Link • Why Snapshot? üPerformance üThe view of data at specific time 

17 .Snapshot • Problem üCreate one snapshot per application？ üHow to share snapshot between application？ • Snapshot Service üManage snapshot lifecycle üAssign the reasonable snapshot to the application 

18 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

19 .Online Bill Storage(订单信息存储系统) • Characteristic üBill information üOnline service üWrite more, read less üRead delay < 1s 

20 .Online Bill Storage(订单信息存储系统) • Key technology üReplication üRPC Queue üReplica 

21 .Agenda • Overview • Audience Platform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica 

22 .Replication • Two datacenter，Master-Master Replication Write Read Write Read Replication Cluster A Cluster B Replication 

23 .Replication • Problem ü Replication Table and CF configuration will be wrong if the table name includes namespace ü Previous design did not consider namespace ü Use “:” when parsing tables and family, such as “userTable:family1” ü But Namespace and table segmentation is also “:”, such as “namespace1:userTable:faimly1” • Resolve ü HBASE-11386, HBASE-11393(Use Protobuf instead of string) 

24 .Replication • Problem üSome data couldn’t be replicated üPeerClusterZnode under is of removed peer may never be deleted üIf some regionserver crash, other regionserver couldn’t take over the rest replication work since the method “copyQueuesFromRSUsingMulti” fails • Resolve üHBASE-16135, HBASE-14476 

25 .RPC Queue • Improve Performance Write Queue üMulti RPC Queue Get Queue üHBASE-11355 Scan Queue • More üControlling Queue Delay（CoDel） üHBASE-15136 

26 .Replica • Problem ü When a RegionServer crash, the region on it is unavailable for a period • Resolve ü Region replicas ü There could be more than one replica for one region ü One primary replica could accept write and read operation ü Multi secondary replica only accepts read operation ü HBASE-10070 

27 .Replica Client Read and Write Read Only RegionServer RegionServer HBase Region Region (Primary) (Secondary) HDFS WAL HFile-1 HFile-2 

28 .Replica • Client strategy ü Query primary region first ü If don’t get the result in 10ms, add a query to the secondary replicas ü Take the first answer and cancel others • Problem ü The data in secondary replica may be stale. • More ü HBASE-11568(Async WAL to secondary replica) 

29 .Future • Multi-Tenancy(HBASE-10994) • Strong schema • High availability