Apache Flink® 1.7 and Beyond Part I

1 .Apache Flink ® 1.7 and Beyond ‫ݪل‬ғ data Artisans ᘳ֖ғ Solutions Architect ᄍᦖᘏғ Konstantin Knauf (Slides by Till Rohrmann) @snntrable 

2 . What is Apache Flink? Data Stream Event-driven Batch Processing Processing process static and realtime results Applications historic data from data streams data-driven actions and services Stateful Computations Over Data Streams 

3 .Flink 1.7: What happened so far? 

4 . Flink 1.7.0 in Numbers • Contributors: 112 • Resolved issues: 430 • Commits: 970 • Changes LOC: +103824/-63124 

5 . Flink Applications Need to Evolve • E.g. changing requirements, new algorithms, better serializers, bug fixes, etc. • Expensive to restart application from scratch (maintain state) 

6 . State Schema Evolution • Support for changing state schema “Upgrading Stateful Flink Streaming Applications: State of • Adding/Removing fields the Union” by Tzu-Li Tai Today @ 5:20 pm Room 1 • Changing type of fields • Currently fully supported when using Avro types 

7 . Converting Currencies 7:12pm 9:37am 8:45am € 1 $ 1.13 CN¥ 7.8 

8 . Temporal Tables and Joins 13 11 7 7 4 Currency Rate Time CN¥ 7.8 3 CN¥ 7.89 5 15 14 12 CN¥ 7.75 9 

9 .SQL for Pattern Analysis SELECT * from ? 

10 .SELECT * MATCH_RECOGNIZE FROM TaxiRides MATCH_RECOGNIZE ( PARTITION BY driverId ORDER BY rideTime MEASURES S.rideId as sRideId AFTER MATCH SKIP PAST LAST ROW PATTERN (S M{2,} E) DEFINE S AS S.isStart = true, M AS M.rideId <> S.rideId, E AS E.isStart = false AND E.rideId = S.rideId ) 

11 . More SQL Improvements • ElasticSearch 6 Table Sink “Flink Streaming SQL 2018” by Piotr Nowojski Today @ • Support for views in SQL Client 4:00 pm Room 1 • More built-in functions: TO_BASE64, LOG2, REPLACE, COSH,… 

12 . Other Notable Features • Scala 2.12 Support • Exactly-once S3 StreamingFileSink • Kafka 2.0 connector • Versioned REST API • Removal of legacy mode 

13 .Flink 1.8+: What is happening next? 

14 . End-to-end SQL Only Pipelines Hive Meta Store • Support for external catalogs (Confluent Schema Registry, Hive Meta Store) Input schema information Output schema information • Data definition language (DDL) SQL Table Source Table Sink Query