Flink 技术介绍与未来展望

1 . 1 8 ( 2 ) 9.. 0 

2 . L B . . F J 2015 2016 2017 2018 C B F B 

3 .

4 .Flink 

5 .Stateful Computations Over Streams 

6 .Fast Scalable Event-time Fault-tolerant Large-state In-memory Exaclty-once Stateful Computations Over Streams 

7 .Checkpoint State Time Window Chandy-Lamport State API Watermark ValueState, ListState, MapState BroadcastState 

8 .

9 .• M • > > • e e • F • C - • J S • EP S 

10 .Blink is an unified engine for large-scale data processing. 

11 .

12 . - Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 Table API v1.0 tumbling, sliding, session Join in Table API/SQL CEP library added group-windows for Table API/SQL SQL CLI Flink 1.1.0 Flink 1.3.0 Refactor Table API/SQL Retractions in Table API/SQL to base on Apache Calcite Rescalable CEP operators 

13 . Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 State API (ValueState, ProcessFunction Broadcast State ReducingState, ListState) (low-level API) Flink 1.1.0 Flink 1.3.0 Flink 1.6.0 Session windows Side outputs State TTL Late-arriving events DataStream Interval Join 

14 . - - - ) ) - ) AB - )) - - )) )) - - ()- ) ) ()- 

15 . & Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 RocksDB for out-of-core Rescalable keyed and Task local state recovery state support operator state Flink 1.1.0 Flink 1.3.0 Fully async RocksDB Incremental checkpoint snapshots Fine-grained recovery 

16 .

17 . Flink 1.4.0 Flink 1.2.0 End-to-End exactly once Async I/O processing (with kafka, too) Flink 1.3.0 Flink 1.5.0 HistoryServer: FLIP-6: New Deployment and Process Model archive status and statistics Rewrite Network Stack of completed jobs 

18 .• • 2 )(1 -/ / 2 /- • 2 

19 .

20 .0 0 

21 .

22 .1 /-1 ) ( • • B • • • 

23 .

24 .1 Spark 2 Stream SQL - Join TopN CEP on SQL 3 Stream SQL MicroBatch, Local-Global, CountDistinct 4 SQL Python UDF 

25 .