07_kafka_cloner

1 .Kafka Kloner Michael Natkovich 

2 .Kafka Kloner A Dynamic High-Speed Inter-Cluster Kafka Replicator Propane replicates around 150 billion events per day cross colo across its clusters with an average latency of less than 2 seconds. 

3 .Motivation ■ Demand for Yahoo Streaming in multiple data centers ■ Kafka MirrorMaker operability issues Finally, we borrowed the MirrorMaker approach and implemented replication using Storm. Yahoo Confidential & Proprietary 

4 .MirrorMaker vs Kloner MirrorMaker Kloner Performance High High Operability Low High Fault Tolerance Low High Topic Management Low High Monitoring Low High Yahoo Confidential & Proprietary 

5 .Overview Yahoo Confidential & Proprietary 

6 .Features Design ■ Storm based approach ■ One topology per topic being cloned ■ One worker per topic partition ■ One kafka consumer + producer per worker Core Logic ■ Same as MirrorMaker ● Read; Commit Offsets; Send Message; Yahoo Confidential & Proprietary 

7 .Features Manager ■ Runs based on a DSL input with list of topics and their destinations ■ Polls Storm for list of topics already being replicated and number of workers ■ Polls Kafka clusters for topic metadata such as partitions per topic Then, 1. Starts replication by starting new topologies 2. Scales replication by rebalancing topologies (Restarts with updated worker count) 3. Stops replication by stopping topologies ■ The Manager can be a cron or simply invoked after DSL update Yahoo Confidential & Proprietary 

8 .Features Security ■ Cross colo data transfers are encrypted ■ Kafka SSL + Yahoo Certificate Authority. Monitoring & Alerting ■ Yahoo Monitoring Framework ■ Kloner msg rates ■ Kafka msg rates ■ Kloner consumer lags Yahoo Confidential & Proprietary 

9 .Numbers Clusters 2 Topics 15 Msgs/sec 1 million Bytes/sec 1.5 GB Replication Lag < 2 seconds Yahoo Confidential & Proprietary 

10 .Limitations ■ No exact replication ● Offsets differ between clusters ■ No replay or spooling ● No message level failure handling ● But we are tracking number of failed events ● There is rate limiting Yahoo Confidential & Proprietary 

11 .QUESTIONS