3.使用 Ray 构建高可用和可扩展的在线应用-蔡腾纬

1 .⽤Ray构建⾼可⽤和可扩展的 在线应⽤ 蔡腾纬 蚂蚁集团 1 

2 . 初露头⻆ - ⽤户场景 // pseudocode public class Strategy { • ⼤量⽀付机构（渠道）的策略和指标计算 public void calc(long time, Map banksAndIndicator){ for(Entry e : banksAndIndicator.entrySet()){ String bank = e.getKey(); for(String indicator : e.getValues()){ • 单机瓶颈 } calcIndicator(time, bank, indicator); } } • 机构数量上涨 public void calcIndicator(long time, String bank, List indicators){ • 数据量、计算量上涨 for(String indicator : indicators){ calcBankIndicators(time, bank, indicator); } // do indicators' data calculation • 复杂的计算逻辑，重构代价 } public void calcBankIndicators(long time, String bank, • 有没有⾼性价⽐的分布式计算⽅案？ // do bank data calculation String indicator){ } } 2 

3 . 初露头⻆ - 期望⽅案 What’s What Ray running can do • 我们期望的新⽅案 Service Ray Worker Trigger Trigger • RPC 或者消息触发, 低时延完 Process Results gather results 成计算 Task #0 Result #0 Ray Worker • parallel exec 容易在原有代码上改写 Task #1 Result #1 Task #0 Result #0 Ray Worker • 动态的任务执⾏ Task #N Result #N Task #1 Result #1 Ray Worker Process Results Task #N Result #N 3 

4 . 初露头⻆ - 基于 Ray 的系统设计 AppMaster Actor User App create & control • AppMaster - 控制所有计算 User Job #0 Actor ⽣命周期与状态 Worker Actor L1 User Code Dispatcher • 创建 Trigger Actor 来接收外 trigger task Trigger Actor RPC Server Dispatcher ExecutionRuntime Worker Actor L2 User Code Dispatcher 部计算任务 & wait for results (within 2xx ms) Worker Actor L1 ExecutionRuntime ExecutionRuntime User Code Dispatcher • 预拉起计算 Actor，减少初始 Trigger Actor RPC Server Dispatcher ExecutionRuntime Worker Actor L2 化时间 ExecutionRuntime Worker Actor L1 User Code Dispatcher ExecutionRuntime User Code Dispatcher • Dispatcher 负责将上层 Actor L1 worker ExecutionRuntime L2 worker 路由到下层计算 Actor execute execute `calcIndicator` `calcBankIndicators` 4 

5 . 模型服务 User App RPC Call • 在线机器学习 Streaming Serving Ray Actor • 分钟级的持续模型导出 Ensamble Prediction LR Engine Engine Model • 模型尽快更新到线上服务 Serving Runtime Ray Actor Ray Actor • NLP/CV Training Prediction NN Prediction NN Engine Model Engine Model • 异构 model serving Serving Runtime Serving Runtime Update Model • Ensemble model serving ~ 10-30 minutes per Iterate 5 

6 . FaaS Eventing (Knative) Serving Ray Actor • 事件驱动平台（如 Knative Event FaaS Runtime User Func A Source Eventing）将消息转为 RPC RPC Send Serving Runtime Trigger Events • 基于 Ray Actor 的 FaaS 实例 Event Source Ray Actor 根据 workload ⾃动扩缩容 FaaS Runtime User Func A （建设中） Serving Runtime Send/Pull Send Events Events Ray Actor Event User Platform App FaaS Runtime User Func B RPC Direct Call Serving Runtime 6 

7 . 在线资源分配（运筹优化） • 解决⼤规模线性规划问题 • 详情可以观看蚂蚁集团在 Ray Summit 上的 Talk "Application of online resource allocation based on Ray" 7 

8 . Ant Ray Serving: 设计⽬标 • Ant Ray Serving：⼀个基于Ray的通⽤在线服务框架 • 挑战: • ⾼可⽤, SLA > 99.9% • 快速更新（10分钟内更新 1k+ 个实例） • ⾜够的灵活度和扩展性 8 

9 . 跨集群架构 - 集群编排 • ⾼可⽤：多集群容灾 Start/Stop Service Serving ServingKeeper Client • 引⼊跨集群 Ray 作业编排：Serving Keeper Scale/Upgrade • 多集群的服务管理 Serving Cluster I Serving Cluster II • ⾃动编排（单元化部署架构） AppMaster AppMaster Actor I Actor II • 代理多集群的服务管理（扩缩容、升级） Proxy Proxy Proxy Proxy • 跨集群的协调 Actor Actor Actor Actor • 根据集群、容量事件协调集群间流量⽐例 Backend Backend Backend Backend Actor Actor Actor Actor • ⽀持蓝绿发布、⾦丝雀发布 9 

10 . 跨集群架构 - 服务发现 User Application RPC Trigger select living instance • 跨集群的服务发现 Serving Client (local- rst load balance) pull instances • 两层服务发现 RPC call selected instance service.global.xxx.com • 本地、时延优先的负载均衡策略 weight: 2 weight: 1 • 弹性调整流量 service.i.xxx.com service.ii.xxx.com • ⽀持 Ray 集群的容灾和滚动更新 Serving Cluster I Serving Cluster II Proxy Proxy Proxy Actor Actor Actor 10 fi 

11 . AppMaster 状态管理 Scene #1: AppMaster Actor 容灾 • AppMaster 状态 1. Ray 重建 AppMaster actor • 服务元数据 2. AppMaster 发现只是⾃身重建, 恢复所有状态到内存并继续服 务 • Proxy & backend actor handler Scene #2: 整个服务的容灾 • 服务配置如 replica 数量以及 backend con g 1. 整个服务作业被⽤户或者 Serving Keeper 重启 等 2. AppMaster 发现是作业重启, ⾃动恢复服务元数据和配置等， 重置 runtime 状态 • Proxy, backend 及服务本身的 status 等 3. AppMaster 根据配置重建 proxy/backend actor，并更新状态 • AppMaster 状态⾼可⽤ change AppMaster persistent State • 可扩展的 AppMaster state backend: MySQL, State ⽂件, Redis 等等 AppMaster Backend restore 11 fi 

12 . 服务更新 • 根据更新步⻓、策略等配置确定更 Remove itself Pre- Prepare resource & from service con guration Updating discovery 新步骤 Detache Serving • d 更新前先摘除流量 Rollback to Upgrade State Machine of Reload/Upgrade • previous 在 serving backend runtime ⽀持 version Serving Backend after no tra c 并尽量采⽤原地更新 Rollbacki Upgradin Validate ng Success g • 在线学习场景中更新后的 Validatin Self validate Validating ⾮常重要 Validate Fail with recorded requests g 12 fi ffi 

13 . 可扩展性 • 基于 K8S/Yarn 的弹性调度 • Autoscale (探索中) • 难点：准确的策略和⾜够快扩容速度 • 基于时序的策略或模型预测 • SLA - Serving or Scheduling level agreement 13 

14 . 整体架构 User Application User Application (Data Plane) (Controll Plane) Serving Client ServingKeeper Serving Client pull instances Service Discovery register Serving Cluster I Serving Cluster II Serving Cluster III Control Instruction Ray Node Ray Node AppMaster AppMaster Data Requests Proxy Proxy … Java Backend Python Backend … Java Backend Python Backend … Serving Cluster X Ray Node Ray Node Ray Node Ray Node Proxy Proxy Ray Node Ray Node Backend Backend … Backend Backend … 14 

15 . 未来 • ⽀持在线学习模型服务秒级更新 • 在⽣产场景⽀持⾃动扩缩容 • ⽀持在线复杂、有状态计算 • 与 Ray Serve 合作 • 在 Ray Serve 中增加 Java ⽀持 • 构建可插拔的 controller, 服务发现, rpc 实现等 15 

16 . 谢谢! Email: tengwei.ctw@antgroup.com Slack: tengweicai@gmail.com We’re hiring :-) 16