Challenges and Opportunities of Apache Flink® Ecosystem Part II

1 .,QWHUDFWLYH3URJUDPPLQJ)OLQN Flink Env Cached/Materialized TableAPI Flink Cluster Table Submit job1 points job uses generat es lines Result Submit ses job2 u job generates surfaces S ub Result mi t es job job3 us … Result 

2 .,QWURGXFH5RZEDVHG7DEOH$3, )/,3)OLQN 

3 .,PSURYH)OLQN 0/3LSHOLQH)OLQN HPEUDFLQJ7DEOH$3,ғ ๅ਻ฃቘᥴ޾ӤಋҔಢၞᕹӞҔ(7/॒ቘ޾$,ᦇᓒ‫ݢ‬զ෫ᖲᵞ౮ VXSSRUWVWUHDPLQJ0/ ඪ೮ਫ෸ᦒᕞ҅ਫ෸ᶼၥ SURYLGHDEHWWHU$,H[SHULHQFH 

4 .7HQVRUIORZ RQ)OLQN Data Process and Model Testing and Data Ingestion Deployment Transformation Training Validation Change model or parameters 

5 .$ERXW0HŃ ᒍ‫ڻ‬᲋ -HII=KDQJ Ø Joined Alibaba in 2018/10 Ø Worked in Hortonworks Pivotal & eBay Ø Apache member PMC of Apache Tez, Livy & Zeppelin 

6 .$SDFKH=HSSHOLQ चԭZHEጱԻ԰ୗ‫ړ‬ຉ1RWHERRNඪ೮ 6TO 6FDODᒵᒵ :HEEDVHGQRWHERRNWKDWHQDEOHV GDWDGULYHQ LQWHUDFWLYHGDWDDQDO\WLFV DQGFROODERUDWLYHGRFXPHQWVZLWK64/ 6FDODDQGPRUH 

7 .(QKDQFH)OLQN ([SHULHQFHLQ$SDFKH=HSSHOLQ =HSSHOLQ පሲ ᵞ౮ մӱᕆᇙ௔ ҁ(IILFLHQF\҂ ҁ,QWHJUDWLRQ҂ ҁ,QWHJUDWLRQ҂ 

8 . පሲ (IILFLHQF\ Ø ள᭛୏‫ݎ‬ӱ‫ۓ‬᭦ᬋ҅෫ᵱ಑۱҅᮱ᗟ҅ᯈᗝᒵᒵ 4XLFNGHYHORSPHQWQREXLOGQRSDFNDJLQJDQGHDV\FRQILJXUDWLRQ Ø ள᭛ਧ֖Კ᧏҅෫ᵱ᪡᫨‫)ک‬OLQNᵞᗭ๐‫࢏ۓ‬ ,GHQWLI\HUURUIDVWQRQHHGWRVZLWFKWR-0GDVKERDUGIRUH[FHSWLRQ Ø ṛපᓕቘ඙֢)OLQN -REҁ&DQFHO -RE҅ 6DYH3RLQW ᒵᒵ҂ 0DQLSXODWH)OLQN MREHDVLO\LQFOXGLQJVXEPLVVLRQFDQFHODQGUHVXPHIURPVDYHSRLQW 

9 .ᵞ౮ ,QWHJUDWLRQ Ø හഝ‫ݢ‬ᥤ۸ ҁᶉாහഝ  ۖாහഝ҂ 'DWD 9LVXDOL]DWLRQ ҁ6WDWLF  '\QDPLF҂ Ø ٌ՜ૡٍጱᵞ౮ҁ+LYH҅3LJ҅ 3\WKRQ҅0DUNGRZQ ᒵ҂ ,QWHJUDWLRQ ZLWK RWKHU GDWD WRROV ҁ+LYH3LJ3\WKRQ0DUNGRZQDQGHWF Ø ๢࢏਍ԟ໛ຝጱᵞ౮ҁ7HQVRUIORZ҅ .HUDV ᒵ҂ ,QWHJUDWLRQZLWK0DFKLQHOHDUQLQJIUDPHZRUNV7HQVRUIORZ.HUDV DQGHWF 

10 .մӱᕆᇙ௔ (QWHUSULVH)HDWXUH Ø ग़ᑄಁҁ0XOWLWHQDQF\҂ Ø ਞ‫ق‬ḵᦤ ҁ$XWKHQWLFDWLRQ $XWKRUL]DWLRQ Ø ᶱፓդᎱᓕቘ 3URMHFW 0DQDJHPHQW Ø Өԯ๐‫ۓ‬ጱᵞ౮ҁ,QWHJUDWLRQ ZLWK &ORXG҂ 

11 .7DNH$ZD\ Batch Processing • Unified Batch and Stream Processing • Integrate Flink with Hive Ecosystem (FLINK-10556) AI Processing • Interactive Programming (FLINK-11199) • Iterative Processing • Row-based TableAPI (FLIP29, FLINK-10972) • Tensorflow on Flink • Flink ML pipeline (FLINK-11095) Zeppelin • Enhance Flink Experience in Apache Zeppelin (ZEPPELIN-3913) 

12 . Demo • Query Hive Meta and Data via Flink SQL_CLI • Flink Batch and Stream Processing on Zeppelin 

13 .