展开查看详情
1 .HBase Disaster Recovery Solution
at Huawei
Ashish Singhi
1
2 .About.html
• Senior Technical Leader at Huawei
• Around 6 years of experience in Big Data related projects
• Apache HBase Committer
2
3 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
3
4 .Why Disaster Recovery ?
Cost of
Downtime
4
5 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
5
6 .Backup Vs Disaster Recovery
Two different problems and solutions
Backup Disaster Recovery
Process Archive items to Replicate to secondary
cold media site
Infrastructure Medium level Duplicate of active
cluster (high level)
Cost Affordable Expensive
Restore process One to few at a One to everything
time
Restore time Slow Fast
Production usage Common Rare
6
7 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
7
8 .HBase Disaster Recovery
• HBase Disaster recovery is based on replication, which
mirrors data across a network in real time.
• The technology is used to move data from a local source
location to one or more target locations.
• Replication over WAN has become an ideal technology
for disaster recovery to prevent data loss in the event of
failure.
8
9 .Deployment Strategies
9
10 .Active – Standby Cluster
Active Cluster Standby Cluster
HBase HBase
Read
Replication
Write
/hbase/clusterStat Serves Read Serves only /hbase/clusterStat
e: active and Write Read Client e: standby
Client Requests
Requests
ZooKeeper ZooKeeper
10
11 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
11
12 .Replication
Source Cluster Peer Cluster 1 [tableCfs - 1]
Region Server Region Server Bulk load
1
Replication 1 1 Table
Replication Sink
Source/End Point 1
WAL
Batch
Batch 1
Replication Source
Manager
2
Bulk load
1 Bulk load
1
Replication 31 21 1 Replication Sink Table
Source/End Point 2 1
…/peers/
Batch
…/rs/ Region Server
…/hfile-refs/
Peer Cluster 2 [tableCfs - ]
ZooKeeper
12
13 .Sync DDL Operations
• Synchronize the table properties across clusters
• Any change in the source cluster, reflects immediately in
the peer clusters.
• Does not break the replication.
• An additional option with DDL command to sync
• Internally sync those changes to peer clusters.
13
14 .Sync Security related Data
• Synchronize security related HBase data across the
clusters
• Any update in the source cluster ACL, Quota or Visibility
Labels table, reflects immediately in peer clusters.
• A custom WAL entry filter is added in replication for this.
• Does not break the security for HBase data access.
14
15 .Read Only Cluster
• Enable a cluster to serve only read requests
• A coprocessor based solution
• Standby cluster will serve all the read requests
• Standby cluster will serve write requests only if the requests
is coming from a,
• Super user
• From a list of accepted IPs
15
16 .Cluster Recovery
Active Standby Cluster Standby Active Cluster
HBase HBase
Read
Replication
Write
/hbase/clusterStat Serves only Serves Read /hbase/clusterStat
e: active standby Read Client and Write e: standby active
Requests Client
Requests
ZooKeeper ZooKeeper
16
17 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
17
18 .Miscellaneous
• Increased the default replication.source.ratio to 0.5
• Adaptive hbase.replication.rpc.timeout
• Active cluster HDFS server configurations are maintained
in Standby cluster ZooKeeper for bulk loaded data
replication.
18
19 .Agenda
• Why Disaster Recovery ?
• Backup Vs Disaster Recovery
• HBase Disaster Recovery
• Solution
• Miscellaneous
• Future Work
19
20 .Future work
• Move HBase Replication tracking from ZooKeeper to
HBase table (HBASE-15867)
• Copy bulk loaded data to peer with data locality
• Replication data network bandwidth throttling.
20
21 .Thank You！
mailto: ashishsinghi@apache.org
Twitter: ashishsinghi89
21
热门城市
