Luigi Selmi - The Big Data Integrator Platform
0 likes•2,074 views
The document discusses the SC4 pilot project, which aims to build a scalable and fault-tolerant platform for processing large datasets using open source frameworks. The platform utilizes a microservices architecture and processes real-time floating car data for tasks like map matching and short-term traffic forecasting using algorithms like feedforward artificial neural networks. It also discusses how semantic technologies from projects like SANSA-Stack and LinkedGeoData could enable additional use cases for the SC4 platform.
Luigi Selmi - The Big Data Integrator Platform
- 1. Pilot SC4 SC4 Final Hangout 2018-02-02 Big Data Europe Empowering Communities with Data Technologies
- 2. Objective of the Pilot SC4 SC4 Final Hangout 2018-02-02 A scalable, fault-tolerant and flexible platform based on open source frameworks that can process unbounded data sets.
- 3. Microservice Architecture SC4 Final Hangout 2018-02-02
- 4. The FCD Pipeline SC4 Final Hangout 2018-02-02
- 5. Pilot Architecture SC4 Final Hangout 2018-02-02
- 6. BDE Components SC4 Final Hangout 2018-02-02
- 7. Parallelization: input and output data SC4 Final Hangout 2018-02-02 device_id timestamp lat lon speed orientation transit The mapMatch subtask keeps the time order so that the next task keyBy(road_seg)/window(15’)/apply(avera ge_speed) will return the correct result within the time window for each road segment. road_seg_id start_date num_vehicles avg_speed
- 8. Parallelization: map-match subtasks SC4 Final Hangout 2018-02-02 1. source() 2. mapMatch() 3. keyBy()/window()/apply() 4. sink() The subtasks can be distributed in slots with different parallelism (e.g. from 1 to 12)
- 9. Parallelization: map-match subtasks SC4 Final Hangout 2018-02-02 A slot can process all the subtasks in a pipeline
- 10. SC4 Pilot Pipeline SC4 Final Hangout 2018-02-02
- 11. Data Upload SC4 Final Hangout 2018-02-02
- 12. Producer and Consumer SC4 Final Hangout 2018-02-02
- 13. Visualization SC4 Final Hangout 2018-02-02 The pilot SC4 can process real-time FCD data for map-matching and classify a road segment according to the traffic level.
- 14. Short-Term Traffic Forecast SC4 Final Hangout 2018-02-02 Algorithm: Feedforward ANN
- 15. Short-Term Traffic Forecast SC4 Final Hangout 2018-02-02 Algorithm: Feedforward ANN Hyperparameters (spatial and temporal correlation): Input layer units: (Dd*24*60*Cr)/Tw Dd = number of days (e.g. working days, 5 days) Tw = time window (e.g. 30’) Cr = connected road segments (e.g. 3) -> 720 input units
- 16. SANSA-Stack: Big Data + Machine Learning + Semantic Technologies SC4 Final Hangout 2018-02-02 SANSA-Stack, part of the BDE project, and RDF data sets based on semantic technologies such as LinkedGeoData, will enable more use cases related to SC4
- 17. Thanks SC4 Final Hangout 2018-02-02 BDE project website: https://www.big-data-europe.eu/ Code repository: https://github.com/big-data-europe?q=sc4 Contact: [email protected]