Fast Cars, Big Data - How Streaming Can Help Formula 1 - Tugdual Grall - Codemotion Amsterdam 2017
1 like•802 views
The document discusses how streaming data can enhance Formula 1 racing by utilizing real-time analytics from various sensors in cars, which can generate billions of data points during a race weekend. It outlines the architecture and tools necessary for effective data processing and storage, emphasizing the need for a scalable and distributed approach. Additionally, it highlights the potential applications of this data in other industries beyond motorsports.
![© 2017 MapR Technologies@tgrall 34
Produce Messages
ProducerRecord<String, byte[]> rec = new ProducerRecord<>(
“/apps/racing/stream:sensors_data“,
eventName,
value.toString().getBytes());
producer.send(rec, (recordMetadata, e) -> {
if (e != null) { … });
producer.flush();](https://image.slidesharecdn.com/ejxgn6oqz6idegicf3qg-signature-753cb6f16dd9e78187c27f5c16d36b5d35566cf18d25ee83278c932237b78616-poli-170522152114/85/Fast-Cars-Big-Data-How-Streaming-Can-Help-Formula-1-Tugdual-Grall-Codemotion-Amsterdam-2017-34-320.jpg)
Fast Cars, Big Data - How Streaming Can Help Formula 1 - Tugdual Grall - Codemotion Amsterdam 2017
- 1. © 2017 MapR TechnologiesMapR Confidential 1 Fast Cars, Big Data How Streaming Can Help Formula1 ? Tugdual Grall @tgrall
- 2. © 2017 MapR Technologies@tgrall {“about” : “me”} Tugdual “Tug” Grall • MapR : Technical Evangelist • MongoDB, Couchbase, eXo, Oracle • NantesJUG co-founder • @tgrall • http://tgrall.github.io • [email protected] / [email protected]
- 3. © 2017 MapR Technologies@tgrall 3 Open Source Engines & Tools Commercial Engines & Applications Utility-Grade Platform Services DataProcessing Web-Scale Storage MapR-FS MapR-DB Search and Others Real Time Unified Security Multi-tenancy Disaster Recovery Global NamespaceHigh Availability MapR Streams Cloud and Managed Services Search and Others UnifiedManagementandMonitoring Search and Others Event StreamingDatabase Custom Apps MapR Converged Data Platform
- 4. © 2017 MapR Technologies@tgrall 4 Agenda • What’s the point of data in motorsports? • Demo • Architecture • What’s next? • Q&A
- 5. © 2017 MapR Technologies@tgrall 5 What’s the point of data in motorsports?
- 6. © 2017 MapR Technologies@tgrall 6
- 7. © 2017 MapR Technologies@tgrall 7 Data in Motorsports F1 Framework - http://f1framework.blogspot.de/2013/08/short-guide-to-f1-telemetry-spa-circuit.html
- 8. © 2017 MapR Technologies@tgrall 8 Race Strategy F1 Framework - http://f1framework.blogspot.be/2013/08/race-strategy-explained.html
- 9. © 2017 MapR Technologies@tgrall 9 Got examples?
- 10. © 2017 MapR Technologies@tgrall 10 Got examples? • Up to 300 sensors per car
- 11. © 2017 MapR Technologies@tgrall 11 Got examples? • Up to 300 sensors per car • Up to 2000 channels
- 12. © 2017 MapR Technologies@tgrall 12 Got examples? • Up to 300 sensors per car • Up to 2000 channels • Sensor data are sent to the paddock in 2ms
- 13. © 2017 MapR Technologies@tgrall 13 Got examples? • Up to 300 sensors per car • Up to 2000 channels • Sensor data are sent to the paddock in 2ms • 1.5 billions of data points for a race
- 14. © 2017 MapR Technologies@tgrall 14 Got examples? • Up to 300 sensors per car • Up to 2000 channels • Sensor data are sent to the paddock in 2ms • 1.5 billions of data points for a race • 5 billions for a full race weekend
- 15. © 2017 MapR Technologies@tgrall 15 Got examples? • Up to 300 sensors per car • Up to 2000 channels • Sensor data are sent to the paddock in 2ms • 1.5 billions of data points for a race • 5 billions for a full race weekend • 5/6Gb of compressed data per car for 90mn
- 16. © 2017 MapR Technologies@tgrall 16 Got examples? • Up to 300 sensors per car • Up to 2000 channels • Sensor data are sent to the paddock in 2ms • 1.5 billions of data points for a race • 5 billions for a full race weekend • 5/6Gb of compressed data per car for 90mn US Grand Prix 2014 : 243 Tb (race teams combined)
- 17. © 2017 MapR Technologies@tgrall 17 How does that work?
- 18. © 2017 MapR Technologies@tgrall 18 Production System Outline
- 19. © 2017 MapR Technologies@tgrall 19 Simplified Demo System Outline
- 20. © 2017 MapR Technologies@tgrall 20 TORCS for Cars, Physics & Drivers • The Open Racing Car Simulator – http://torcs.sourceforge.net/ • Car racing game • AI & Research Platform
- 21. © 2017 MapR Technologies@tgrall 21 Demo
- 22. © 2017 MapR Technologies@tgrall 22 Architecture Events Producers Events Consumers sensors data Real Time Analytics Analytics with SQL https://github.com/mapr-demos/racing-time-series
- 23. © 2017 MapR Technologies@tgrall 23 Architecture Events Producers Events Consumers sensors data Real Time Analytics https://github.com/mapr-demos/racing-time-series JSONProducer Consumer
- 24. © 2017 MapR Technologies@tgrall 24 Where/How to store data ?
- 25. © 2017 MapR Technologies@tgrall 25 Big Datastore Distributed File System HDFS/MapR-FS NoSQL Database HBase/MapR-DB ….
- 26. © 2017 MapR Technologies@tgrall 26 Data Streaming
- 27. © 2017 MapR Technologies@tgrall 27 Data Streaming Moving millions of events per h|mn|s
- 28. © 2017 MapR Technologies@tgrall 28 What is Kafka? • http://kafka.apache.org/ • Created at LinkedIn, open sourced in 2011 • Implemented in Scala / Java • Distributed messaging system built to scale
- 29. © 2017 MapR Technologies@tgrall 29 Big Picture Producer Producer Producer Consumer Consumer Consumer
- 30. © 2017 MapR Technologies@tgrall 30 Topic & Partitions 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 0 1 2 3 4 5 6 7 Partition 0 Partition 1 Partition 2 Writes
- 31. © 2017 MapR Technologies@tgrall 31 Consumer Groups
- 32. © 2017 MapR Technologies@tgrall 32 More real life Kafka … Zookeeper Broker 1 Topic A Topic B Broker 2 Topic A Topic B Broker 3 Topic A Topic B Producer Producer Producer Consumer Consumer Consumer
- 33. © 2017 MapR Technologies@tgrall 33 MapR Streams • Distributed messaging system built to scale • Use Apache Kafka API 0.9.0 • No code change • Does not use the same “broker” architecture – Log stored in MapR Storage (Scalable, Secured, Fast, Multi DC) – No Zookeeper
- 34. © 2017 MapR Technologies@tgrall 34 Produce Messages ProducerRecord<String, byte[]> rec = new ProducerRecord<>( “/apps/racing/stream:sensors_data“, eventName, value.toString().getBytes()); producer.send(rec, (recordMetadata, e) -> { if (e != null) { … }); producer.flush();
- 35. © 2017 MapR Technologies@tgrall 35 Consume Messages long pollTimeOut = 800; while(true) { ConsumerRecords<String, String> records = consumer.poll(pollTimeOut); if (!records.isEmpty()) { Iterable<ConsumerRecord<String, String>> iterable = records::iterator; StreamSupport .stream(iterable.spliterator(), false) .forEach((record) -> { // work with record object record.value(); … }); consumer.commitAsync(); } }
- 36. © 2017 MapR Technologies@tgrall 36 What’s next?
- 37. © 2017 MapR Technologies@tgrall 37 Capture more data
- 38. © 2017 MapR Technologies@tgrall 38 Sensor Data V1 • 3 main data points: – Speed (m/s) – RPM – Distance (m) • Buffered { "_id":"1.458141858E9/0.324", "car" = "car1", "timestamp":1458141858, "racetime”:0.324, "records": [ { "sensors":{ "Speed":3.588583, "Distance":2003.023071, "RPM":1896.575806 }, "racetime":0.324, "timestamp":1458141858 }, { "sensors":{ "Speed":6.755624, "Distance":2004.084717, "RPM":1673.264526 }, "racetime":0.556, "timestamp":1458141858 },
- 39. © 2017 MapR Technologies@tgrall 39 Sensor Data V2 • 3 main data points: – Speed (m/s) – RPM – Distance (m) – Throttle – Gears – … • Buffered { "_id":"1.458141858E9/0.324", "car" = "car1", "timestamp":1458141858, "racetime”:0.324, "records": [ { "sensors":{ "Speed":3.588583, "Distance":2003.023071, "RPM":1896.575806, "Throttle" : 33, "Gear" : 2 }, "racetime":0.324, "timestamp":1458141858 }, { "sensors":{ "Speed":6.755624, "Distance":2004.084717, “RPM”:1673.264526, "Throttle" : 37, "Gear" : 2 }, "racetime":0.556, "timestamp":1458141858 },
- 40. © 2017 MapR Technologies@tgrall 40 Add new services
- 41. © 2017 MapR Technologies@tgrall 41 New Data Service sensors data Alerts Stream Processing
- 42. © 2017 MapR Technologies@tgrall 42 • Cluster Computing Platform • Extends “MapReduce” with extensions – Streaming – Interactive Analytics • Run in Memory • http://spark.apache.org/
- 43. © 2017 MapR Technologies@tgrall 43 • Streaming Dataflow Engine – Datastream/Dataset APIs – CEP, Graph, ML • Run in Memory • https://flink.apache.org/
- 44. © 2017 MapR Technologies@tgrall 44 Demo Stream Processing with Flink
- 45. © 2017 MapR Technologies@tgrall 45 Streaming Architecture & Formula 1 • Stream & Process data in real time – Use distributed & scalable processing and storage – NoSQL Database, Distributed File System • Decouple the source from the consumer(s) – Dashboard, Analytics, Machine Learning – Add new use case….
- 46. © 2017 MapR Technologies@tgrall 46 Streaming Architecture & Formula 1 • Stream & Process data in real time – Use distributed & scalable processing and storage – NoSQL Database, Distributed File System • Decouple the source from the consumer(s) – Dashboard, Analytics, Machine Learning – Add new use case…. This is not only about Formula 1! (Telco, Finance, Retail, Content, IT)
- 47. © 2017 MapR Technologies@tgrall 47 One Last Thing… Events Producers Events Consumers Web Socket https://github.com/tgrall/anki-mapr-demo BLE MapR-Streams Apache Kafka
- 48. © 2017 MapR Technologies@tgrall Streaming Architecture http://mapr.com/ebooks/ Free ebooks & Online training http://mapr.com/training/
- 49. © 2017 MapR TechnologiesMapR Confidential 49 Fast Cars, Big Data How Streaming Can Help Formula1 ? Tugdual Grall @tgrall