TGIP-CN 028: Apache Pulsar 延时队列的设计与实现

播放视频

视频文档

TGIP-CN 028: Apache Pulsar 延时队列的设计与实现

下载 8

快召唤伙伴们来围观吧
微博 QQ QQ空间 贴吧
视频嵌入链接文档嵌入链接
<iframe src="https://www.slidestalk.com/ApachePulsar/TGIPCN28DelayedMessageDelivery69518?embed&video" frame border="0" width="640" height="360" scrolling="no" allowfullscreen="true">复制
微信扫一扫分享
已成功复制到剪贴板

StreamNative

发布于

4年前

1320

人观看

#信息技术

本周分享简介

本周将由 StreamNative 李鹏辉为大家带来「Apache Pulsar 延时队列的设计与实现」主题分享，大家可以在直播时留言和互动。

关于 TGIP

TGIP 就是 Thank God It’s Pulsar 的缩写，灵感来自于谷歌团队的 Thank God It’s Friday，表示工作周快结束了，很快又会有一个欢快的周末。TGIP 主要是为了向大家科普关于 Pulsar 的一些基础概念，不管是它的架构模式，还是消息之间的传递动态等，你都可以从 TGIP 的直播中获取到。TGIP 的直播通常会安排在周日，不会占用大家太多的时间。利用周末来了解一下 Apache Pulsar，岂不快哉？

关于 Apache Pulsar

Apache Pulsar 是 Apache 软件基金会顶级项目，是下一代云原生分布式消息流平台，集消息、存储、轻量化函数式计算为一体，采用计算与存储分离架构设计，支持多租户、持久化存储、多机房跨区域数据复制，具有强一致性、高吞吐、低延时及高可扩展性等流数据存储特性。
GitHub 地址：http://github.com/apache/pulsar/

关于组织方 StreamNative

StreamNative 是一家开源基础软件公司，由 Apache 顶级项目 Apache Pulsar 、Apache BookKeeper 原生核心开发者组建而成，围绕 Apache Pulsar 及 Apache BookKeeper 打造下一代流数据平台。SteamNative 专注于开源生态和社区构建，致力于前沿技术领域的创新，创始团队成员曾就职于 Yahoo、Twitter、EMC 等知名大公司。
公司创建于 2019 年，创立初期即获得红杉中国种子基金天使轮投资，在近期已完成数百万美元的 Pre-A 轮融资，本轮融资由源码资本领投、红杉中国种子基金跟投。

展开查看详情

1 .TGIP-CN - 28 Delayed Message Delivery Penghui Li 2020/12/13 streamnative.io

2 .Community 1. 2.6.3 Coming soon 2. PIP 70 Lightweight broker entry metadata 3. PIP 73 Conﬁgurable data source priority for message reading streamnative.io

3 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

4 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

5 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

6 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

7 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

8 .Delayed Message Delivery 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 1 4 2 3 0 streamnative.io

9 .Delayed Message Delivery Different delay granularity on a partition Delayed messages and Non-delayed messages in a partition streamnative.io

10 .Usage producer.newMessage() .deliverAt(long timestamp) .sendAsync(); producer.newMessage() .deliverAfter(long delay, TimeUnit unit) .sendAsync(); streamnative.io

11 . M0 streamnative.io M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

12 . streamnative.io M0 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

16 .Disadvantages Delayed index memory limitation Rebuilding delayed index The index only available for a subscription streamnative.io

17 .PIP 26 Hierarchical Timing Wheels Time Partition Time Partition Time Partition Time Partition Time Partition Tailer ... 2 1 0 http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf https://blog.acolyer.org/2015/11/23/hashed-and-hierarchical-timing-wheels/ streamnative.io

18 . streamnative.io M0 Build delayed index M0 M1 | Delay 5 min M2 Subscription How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

19 . streamnative.io M0 M2 Time Partition 0 M1 M3 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

20 . streamnative.io M0 M2 Time Partition 0 M1 M3 Time Partition 1 M4 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

21 . streamnative.io M0 M2 M5 M6 Time Partition 0 M7 M1 M3 M8 Time Partition 1 M4 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

22 . streamnative.io M0 M2 M5 M6 Time Partition 0 M7 M1 M3 M8 M8 M9 Time Partition 1 M4 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

23 . streamnative.io M0 M2 M5 M6 M7 M8 M9 M3 Time Partition 1 M4 M1 M0 M1 | Delay 5 min M2 How it works? M3 | Delay 2 min M4 | Delay 10 min M5 M6 M7 M8 | Delay 1 min M9

24 .Advantages Load delayed indexes more efﬁciently Reduce the memory usage for delayed indexes Avoid rebuilding indexed by replaying too many entries streamnative.io

25 .Challenges Data cleaning Safe position for subscription to start read Too much individual acks introduced much memory overhead streamnative.io

26 .Wechat Apache Pulsar / StreamNative Bilibili https://space.bilibili.com/391380821 Email dev@pulsar.apache.org users@pulsar.apache.org Slack https://apache-pulsar.slack.com (#general / #china) register: https://apache-pulsar.herokuapp.com/ Github https://github.com/apache/pulsar https://github.com/apache/bookkeeper https://github.com/streamnative streamnative.io

27 . Thank You streamnative.io

2点赞

0收藏

8下载