徐骁_Apache Flink 和 Elasticsearch 助⼒实时 OLAP 平台

1 .Flink Forward China 2018 公司：去哪⼉儿⽹网 职位：实时计算负责⼈人 演讲者：徐骁 

2 .Apache Flink 和 Elasticsearch 助⼒力力实时 OLAP 平台 Apache Flink and Elasticsearch power realtime OLAP platform & 

3 . ⾃自我介绍 About me 2016 加⼊入 Qunar Joined Qunar in 2016 主要负责: Focus on: ELK ELK Elasticsearch Elasticsearch Kafka Kafka Flink 实时计算平台 Flink platform 

4 . ⽬目录 Outline 1. 起因 1. Motivation 2. 如何写⼊入 ES 2. How to sink to ES 3. Flink 和 ES 构建实时 OLAP 平台 3. Flink & ES OLAP platform 4. 实时⽹网络拓拓扑 4. Realtime network topology 

5 .1 Motivation 

6 . Motivation Log Kafka Logstash Elasticsearch Kibana 

7 . Motivation 基于 Lucene 上⼿手容易易, 易易编程 Based on Lucene Easy to start 分⽚片机制 灵活 Sharding Flexible ⾼高可⽤用 伸缩性 Logstash Elasticsearch High available Scalable Kibana Kibana 

8 . Motivation 可靠性 High available 性能 Performance Logstash < 5.0 复杂需求 Complex needs 

9 . Motivation 可靠性 Exactly Once High available 性能 JVM Performance 复杂需求 Streaming Model Flink Complex needs Stateful Window Failover 

10 . Motivation Flink 

11 . Motivation Logstash Log Kafka Elasticsearch Kibana Flink 

12 .2 How to sink to ES? 

13 . How to sink to ES Config ES address ElasticsearchSinkFunction https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/elasticsearch.html 

14 . How to sink to ES bulk.flush.max.actions Config bulk.flush.max.size.mb bulk.flush.interval.ms bulk processor 

15 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

16 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

17 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend OffsetCommitMode: • DISABLED • ON_CHECNKPOINTS Offset:0 Offset:0 • KAFKA_PERIODIC 

18 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

19 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

20 . How to sink to ES Enable checkpoint: At least once bulk.flush.max.actions=2 Kafka Source bulk processor ES flush real offset Zookeeper Statebackend Offset:0 Offset:0 

21 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

22 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0 

23 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Trigger checkpoint Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:0 

24 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending) 

25 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:5(Pending) 

26 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending) 

27 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier flush() snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending) 

28 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending) 

29 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)