8.魏彬-Apache Pulsar 在日志场景的最佳实践

1 .Apache Pulsar 在⽇志场景的最佳实践 魏彬@StreamNative streamnative.i o 

2 . StreamNative is the company behind the Pulsar community. streamnative.i o 

3 .Executive Team ✓ Data veterans with extensive industry experienc Sijie Guo Matteo Merli Jia Zhai ✓ Original creators of Apache Pulsar Founder and CEO CTO Co-Founder & BookKeepe Apache Software ✓ Apache Software Pulsar/BookKeeper PMC Foundation Membe Pulsar/BookKeeper PMC Foundation Membe Operated the largest Pulsar/ Pulsar PMC Chair BookKeeper cluster streamnative.i r r r o e 

4 . StreamNative: Host of the Pulsar Summits 2020 202 ● 2 Global Pulsar Summit ● 3 Global Pulsar Summit ● 1,600 attendee ○ North America – June 16-1 ● 80+ speaker ○ Europe – Sept. 2 ● Attendees included: ○ Asia – Nov 28-2 ● Speakers from: streamnative.i 1 s o s 9 1 s s 7 

5 .Apache Pulsar: Github Stars 9000+ 6X Growth Sep 2018: Apache Top-level project streamnative.i o 

6 .Apache Pulsar: Contributors 400+ 11X Growth Sep 2018: Apache Top-level project streamnative.i o 

7 .Apache Pulsar: Monthly Active Contributors streamnative.i o 

8 . 魏彬 (@rockybean) Solution Engineer@StreamNative Elastic Certified Engineer & Analyst 阿⾥云 MVP streamnative.i o 

9 .⽬录 I. 常⻅⽇志架构(ELK II. ⽇志场景的常⻅挑战 III. 解决⽅案(Apache Pulsar IV. Q&A streamnative.i o ) ) 

10 .I. 常⻅⽇志架构 streamnative.i o 

11 .常⻅⽇志架构 ELK Elasticsearch Beats Logstash Kafka FILEBEAT Kibana WINGLOGBEAT AUDITBEAT Messaging Queue PACKETBEAT streamnative.i o 

12 .消息队列在⽇志场景中的作⽤ 削峰解耦 数据分发 streamnative.i o 

13 .II. ⽇志场景的常⻅挑战 streamnative.i o 

14 .⽇志场景的常⻅挑战 ⽇志分级 流量不确定性 种类繁多 如何保证 如何实现 如何灵活地适配 核⼼业务⽇志的可⽤性？ ⽣产和消费处理能⼒的即时扩容？ 复杂多变的⽇志种类？ streamnative.i o 

15 .消息队列在⽇志场景中的更多功能性要求 ⽇志分级 流量不确定性 种类繁多 限速 即时扩缩容能⼒ ⽆限 Topic/Partition Auto Scaling streamnative.i o 

16 .III. 解决⽅案(Apache Pulsar) streamnative.i o 

17 .Apache Pulsar • 计算和存储分离的分层架构 • Broker - 计算节点 • Bookie - 存储节点 • 计算层⽆状态，即时扩容 • 存储层具有低延时、⾼吞吐、持久化、强⼀ 致的特性 • 即时故障转移和容错机制 • 天⽣适合 K8S 等容器编排管理平台 streamnative.i o 

18 .限速 •仅⽀持 user/client-id 级别设定 •namespace/topic 级别 •Broker 级别 streamnative.i o 

19 .即时扩缩容 •增加新节点后，需要 reassign partition 才能使⽤ •存储与计算分离，可以按需增加计算或存储节点，增加即 •消费能⼒受 partition 数⽬限制，⽽且有 consumer ⽣效，不需要 reassign rebalance 的问题 •消费能⼒不受 partition 数⽬限制，可以不断增加 consumer 来提升消费能⼒ streamnative.i o 

20 .⽆限 Topic/Partition •随 Partition 数增多，请求延迟下降严重 •⽀持百万级别 topic，请求延迟稳定 •追加写模式退化为随机写 •topic/partition 仅是逻辑概念，保证追加写模式 streamnative.i o 

21 .⽆限 Topic/Partition https://www.infoq.cn/article/xeyeEeNNY5CG0PGyxeVD streamnative.i o 

22 .ELK 架构 Filebeat Filebeat Filebeat Filebeat Kafka Logstash Elasticsearch Kibana Grafana Monitoring API streamnative.i o 

23 .Kafka on Pulsar 兼容 kafka 协议 > 1.0 https://github.com/streamnative/kop streamnative.i o 

24 .Pulsar 引⼊后的架构 V1 Filebeat Filebeat Filebeat Filebeat 架构不需要变动！ kop Apache Pulsar kop * ⽆法使⽤部分 Pulsar 的特性 Logstash * 限速限流 Elasticsearch Kibana Grafana Monitoring API streamnative.i o 

25 .Pulsar 引⼊后的架构 V2 Filebeat * Filebeat * Filebeat * Filebeat * Filebeat Output Pulsar Plugin * Apache Pulsar Logstash * Logstash/Pulsar Input Plugin Input Pulsar Plugin Elasticsearch Kibana Grafana Monitoring API streamnative.i o 

26 .Pulsar 引⼊后的架构 V2 ● https://github.com/streamnative/pulsar-beat-outpu ● https://github.com/streamnative/logstash-input-pulsa ● https://github.com/streamnative/logstash-output-pulsar t r 

27 .Pulsar Function 轻量级计算引擎：ETL、数据丰富、数据过滤、数据脱敏、动态路由、统计分析等 streamnative.i o 

28 .Pulsar Function 轻量级计算引擎：ETL、数据丰富、数据过滤、数据脱敏、动态路由、统计分析等 streamnative.i o 

29 .Pulsar Function Function Pipeline -> Function Mesh streamnative.i o