Apache Pulsar:打造新一代云原生流数据平台

1 .

2 .Apache Pulsar 打造新一代云原生流数据平台 李鹏辉 2021/10/16 streamnative.io 

3 . StreamNative，Apache Pulsar 新特性设 计及开发。 智联招聘，负责消息系统设计及开发工作。 2018年推推动 Apache Pulsar 在智联招聘 Penghui Li 落地，成为国内首个 Pulsar 落地案例。支 持智联招聘日均数百亿级消息投递。 StreamNative 开源爱好者，Apache Pulsar， Apache Apache Pulsar Committer Bookkeeper，SkyWalking。 PMC member streamnative.io 

4 .About Pulsar 2012 2016 2018 2019 - TODAY Yahoo! Open Source Apache TLP High Growth Period Developed inside Pulsar is Pulsar becomes StreamNative is founded in 2019. Yahoo! as “Cloud committed to Apache top level Broad rapid Pulsar adoption and Messaging Open Source. project. growth in contributors. Service” Launch cloud, KoP and AoP. EcoSystem expands. streamnative.io 

5 .Pulsar Growth 10X Growth 5X Growth September 2018: Apache TLP 10,000,000+ Docker Images Contributors Github Stars Downloads streamnative.io 

6 .Pulsar Adoption streamnative.io 

7 .About StreamNative Apache Pulsar Pulsar Core Open core, community support StreamNative Platform StreamNative Platform Enterprise features, self-managed StreamNative Cloud StreamNative Cloud Fully managed service, SaaS streamnative.io 

8 .为云而生的流数据平台架构 streamnative.io 

9 .存储计算分离 streamnative.io 

10 .数据分片 分片架构 存储和计算分离架构为 Apache Pulsar 带 来了灵活的横向扩展能力 ● 计算扩容 ● 存储扩容 Topic 不和物理设备绑定 ● 更快的故障恢复 ● 更高的磁盘延迟容忍能力 streamnative.io 

11 .Parallel Write streamnative.io 

12 .Tailing read streamnative.io 

13 .Catch-up read streamnative.io 

14 .Parallel batch read (bypass broker) S5 S0 S1 S2 S3 S4 active S0 S2 S1 S2 S0 S1 S0 S2 S1 S3 S4 S5 S3 S5 S3 S4 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 streamnative.io 

15 .IO Isolation Write to the Journal Read from write cache Write Cache Write to write cache Entry Log Entry Log Entry Log Journal Read from Ledger disk streamnative.io 

16 .Scale Broker Topic - 0 Topic - 2 Topic - 3 Topic - 4 Topic - 6 Topic - 1 Topic - 5 Broker 0 Broker 1 streamnative.io 

17 .Scale Up - Broker Topic - 2 Topic - 0 Topic - 4 Topic - 3 Topic - 6 Topic - 1 Topic - 5 Broker 0 Broker 1 Broker 2 streamnative.io 

18 .Scale Down - Broker Topic - 2 Topic - 0 Topic - 4 Topic - 3 Topic - 6 Topic - 1 Topic - 5 Broker 2 Broker 1 streamnative.io 

19 .Scale Bookie Segments of a Topic S0 S1 S2 S3 S4 S5 S0 S2 S1 S2 S0 S1 S0 S2 S1 S3 S4 S5 S3 S5 S3 S4 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 streamnative.io 

20 .Scale Up Bookie Segments of a Topic S0 S1 S2 S3 S4 S5 S6 S0 S2 S1 S2 S0 S1 S0 S2 S1 S3 S6 S4 S5 S3 S5 S3 S6 S4 S6 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 New Bookie streamnative.io 

21 .Scale Down Bookie Segments of a Topic S0 S1 S2 S3 S4 S5 S6 S0 S2 S0 S1 S0 S2 S1 S3 S6 S4 S5 S3 S6 S4 S6 S4 S5 Bookie 0 Bookie 2 Bookie 3 Bookie 4 New Bookie streamnative.io 

22 .Scale Down Bookie Segments of a Topic S0 S1 S2 S3 S4 S5 S6 S0 S2 S0 S1 S0 S2 S1 S3 S6 S1 S4 S5 S3 S6 S4 S6 S4 S5 S3 S5 S2 Bookie 0 Bookie 2 Bookie 3 Bookie 4 New Bookie streamnative.io 

23 .Geo-Replication 跨云数据多备 streamnative.io 

24 .云平台之上的分层存储 streamnative.io 

25 .Segments Segments of a Topic S5 S0 S1 S2 S3 S4 active streamnative.io 

26 .Segments Distribution Segments of a Topic S5 S0 S1 S2 S3 S4 active S0 S2 S1 S2 S0 S1 S0 S2 S1 S3 S4 S5 S3 S5 S3 S4 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 streamnative.io 

27 .Segments Offload Offloaded Segments of a Topic S5 S0 S1 S2 S3 S4 active S2 S2 S2 S3 S4 S5 S3 S5 S3 S4 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 S0 S1 S2 streamnative.io 

28 .Segments Deletion Cursor 0 Segments of a Topic S5 S2 S3 S4 active Cursor 1 Only segment 0 can be deleted safely streamnative.io 

29 .Segments Deletion Offloaded Segments of a Topic S5 S1 S2 S3 S4 active S2 S2 S2 S3 S4 S5 S3 S5 S3 S4 S4 S5 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 S1 S2 streamnative.io