Driver Location Intelligence at Scale using Apache Spark, Delta Lake, and MLflow on Databricks

1 .WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics 

2 .Driver Location Intelligence at Scale using Apache Spark, Delta Lake and MLflow on Databricks Sergio Ballesteros, TomTom Kia Eisinga, TomTom #UnifiedDataAnalytics #SparkAISummit 

3 .Our vision A safe, connected, autonomous world that is free of congestion and emissions. 

4 .Big data drives our business, but data privacy always comes first 4 

5 .Data • Anonymous location (GPS) traces 5 

6 .742.000.000 km every day 18.000 x 6 

7 .7 

8 .Data • Anonymous location (GPS) Traces • Community inputs • User events • Journalistic data • Car sensor data 8 

9 .Data flow ~80 billion data points per day ~150 trillion data points 9 

10 .Data flow 10 

11 .Data flow 11 

12 .Use case 1: IQMaps analytics In dash systems are outperformed by smartphones The embedded system is expected to be up-to-date, with no user interaction. And the most visible component of it is a map. 12 

13 .Drivers do not update their maps Today’s solutions provide manual updates, often with a necessity to drive to the dealer. This is way too complex and inefficient. 13 

14 .14 

15 .OEMs require data efficient solutions While drivers expect up-to-date system, the carmakers are usually concerned about the data cost required for the map management. 15 

16 .99.8% OF TRIPS ARE DRIVEN WITHIN 1000KM RADIUS 98% OF TRIPS ARE DRIVEN WITHIN 150KM RADIUS 16 

17 .When radius is 0 km • User drives within 2 regions every week day • Radius of 0 km. • Download and install just home regions • Cellular data usage kept to a minimum 17 

18 .When radius is 150 km • User drives within 2 update regions every week day • Radius of 150 km. • Home region: 6 update regions. • Cellular data usage increased 18 

19 .IQMapsdemo with MLflow 19 

20 .20 

21 .Real results using 0.5M trips “This insight has led me to the conclusion that a default radius of 150km is unnecessary, and a small radius of ~10km would already satisfy most drivers while keeping cellular data usage low for OEMs.” - Rolf Dorland, PM at TomTom 21 

22 .Going on holidays • User goes for his holiday (less frequent updated region) • Once user starts driving, updates for all update regions the route goes through are downloaded and installed. 22 

23 .Destination prediction 23 

24 .24 

25 .Opportunity Past: Rule-based solution Delta Lake pipelines Present: Machine Learning 25 

26 .Data Original trace data from 1 source 227K device serials Filtering out invalid trips 143K device serials Users with at least 50 trips 3.6K device serials Devices feasible for modelling 2.5K device serials 26 

27 .Features What do we use in the end? For each trip, we have the following information: • Where did the trip start? • At what speed were you driving when the trip started? • What was the time of day (morning/afternoon/evening) when the trip started? • Was it rush hour when the trip started? • What day of the week was it? • Was it a weekend day? • What was the season? • Which driver profile do you belong to? Historical information: • Which destination did you go to your last trip? And the one before that? And the one before that? • If it is a, let's say Monday, where did you go to the last Monday you made a trip? (do this for every weekday) To predict: To which destination are you going? 27 

28 .Labels How do we define where you are going? • We are given the latitude and longitude of a destination of a trip. • In order to find out which latitude and longitudes belong to the same destination, we apply a clustering algorithm called DBSCAN. • DBSCAN clusters together destinations that are within 500 meters from each other. We should have at least 5 trips to a destination in order to call it a cluster. 28 

29 .29