"How to Test and Validate an Automated Driving System," a Presentation from MathWorks
5 likes1,231 views
For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/mathworks/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit-nehemiah For more information about embedded vision, please visit: http://www.embedded-vision.com Avinash Nehemiah, Product Marketing Manage for, Computer Vision at MathWorks, presents the "How to Test and Validate an Automated Driving System" tutorial at the May 2017 Embedded Vision Summit. Have you ever wondered how ADAS and autonomous driving systems are tested? Automated driving systems combine a diverse set of technologies and engineering skill sets from embedded vision to control systems. This technological diversity and complexity makes it especially challenging to test these systems. This session describes the main challenges engineers face in testing and validating autonomous cars and driver assistance systems, and uses case studies to share best practices used in the automotive industry.
Ad
More Related Content
What's hot (20)
Similar to "How to Test and Validate an Automated Driving System," a Presentation from MathWorks (20)
Ad
More from Edge AI and Vision Alliance (20)
Ad
Recently uploaded (20)
"How to Test and Validate an Automated Driving System," a Presentation from MathWorks
- 1. Copyright © 2017 MathWorks, Inc 1 How to Test and Validate an Automated Driving System Avinash Nehemiah May 2017
- 2. Copyright © 2017 MathWorks, Inc 2 1. Obvious Reasons • Safety critical software driving cars on public roads • Human lives depend on this • Integration of new sensors and software into existing vehicles 2. Less Obvious Reasons • Test the interaction of automated driving with human drivers • Understand the driving passenger experience Why Test Automated Driving Systems
- 3. Copyright © 2017 MathWorks, Inc 3 1. In-vehicle testing on a road or test track 2. Open loop testing of defined scenarios 3. Virtual driving Audience Poll: Current Testing Methodologies Answer: All are necessary for different phases of development.
- 4. Copyright © 2017 MathWorks, Inc 4 • Automated driving sub-systems • Most common challenges faced testing automated driving • Perception system testing • Controls and system-level testing • Summary Talk Outline How to Test and Validate an Automated Driving System
- 5. Copyright © 2017 MathWorks, Inc 5 Simplified Automated Driving Sub-Systems Embedded Perception Software SensorFusion PlanningandControl (Steering,Brakingetc.) LiDAR Camera Radar Actuation/ Driving
- 6. Copyright © 2017 MathWorks, Inc 6 • Automated driving sub-systems • Challenges faced testing automated driving • Perception system testing • Controls and system-level testing • Summary Talk Outline How to Test and Validate an Automated Driving System
- 7. Copyright © 2017 MathWorks, Inc 7 1. In-vehicle testing is expensive and time consuming 2. Too dangerous for testing corner-cases 3. Difficult to validate “black box” perception algorithms Challenges Faced Test and Validate an Automated Driving System
- 8. Copyright © 2017 MathWorks, Inc 8 • Automated driving sub-systems • Challenges faced testing automated driving • Perception system testing • Controls and system-level testing • Summary Talk Outline How to Test and Validate an Automated Driving System
- 9. Copyright © 2017 MathWorks, Inc 9 Perception systems include vision, deep learning, sensor fusion, etc. Testing Perception Systems Different flavors of testing: 1. Ground truth labeling • Used for: computer vision , radar and deep learning 2. Scenario testing with synthetic data • Used for: sensor fusion, control algorithms
- 10. Copyright © 2017 MathWorks, Inc 10 How do auto companies verify changes to perception algorithms ?
- 11. Copyright © 2017 MathWorks, Inc 11 Test Fleet Video Data RADAR Data CAN Logs t=1454335205, {‘Pedestrian’,10m,(12 3,450)},{‘Car’,100m,(1 12,235)} … … … t= 1454335215, {‘Road’,’wet’}, light=‘Day’ What is Ground Truth Labeling ? Query specific driving scenarios ( E.g., Car approaching stop sign with speed 30 mph ) Characterize system performance (E.g., pedestrian detection in day vs. night ) Comprehensive regression testing new ADAS features Recorded Sensor Data Human-verified ground “truth” Manual labeling (1000s of hours of tedious work)
- 12. Copyright © 2017 MathWorks, Inc 12 Re-simulation and Comparison vs. Ground Truth Recorded Sensor Data Ground Truth Perception System Embedded Hardware, C Code, MATLAB model COMPARE Most common workflow used to test “black box” perception systems Uses millions of miles of test data
- 13. Copyright © 2017 MathWorks, Inc 13 Testing vs. Ground Truth How can I verify this detection is correct?
- 14. Copyright © 2017 MathWorks, Inc 14 Example: Ground Truth Labeling & Re-Simulation
- 15. Copyright © 2017 MathWorks, Inc 15 Case Study: Traffic Sign Recognition (Continental) System Includes: • Computer vision • Machine learning Traffic Sign Recognition for Driver Assistance Systems Dr. Alexander Behrens, Continental AG MAC Stuttgart , 2015
- 16. Copyright © 2017 MathWorks, Inc 16 Case Study: Traffic Sign Recognition (Continental) Traffic Sign Recognition for Driver Assistance Systems Dr. Alexander Behrens, Continental AG MAC Stuttgart, 2015 Testing vs. ground truth Perception Systems
- 17. Copyright © 2017 MathWorks, Inc 17 Case Study: Advanced Emergency Braking (Scania) Developing Advanced Emergency Braking Systems at Scania – Jonny Andersson Control systemPerception system
- 18. Copyright © 2017 MathWorks, Inc 18 Case Study: Advanced Emergency Braking (Scania) Developing Advanced Emergency Braking Systems at Scania – Jonny Andersson In-vehicle testing
- 19. Copyright © 2017 MathWorks, Inc 19 Case Study: Advanced Emergency Braking (Scania) Developing Advanced Emergency Braking Systems at Scania – Jonny Andersson Review Recorded Data Testing Workflow Test case: • Interesting scenarios • Expected output (ground truth)
- 20. Copyright © 2017 MathWorks, Inc 20 How would you test a dangerous driving situation ?
- 21. Copyright © 2017 MathWorks, Inc 21 When to use synthetic data? Recorded data does not exist for • Dangerous scenarios • Different sensor configurations Enables testing of dangerous scenarios and corner cases Scenario Testing with Synthetic Data
- 22. Copyright © 2017 MathWorks, Inc 22 How does my vision algorithm respond to changes to acceleration and steering ?
- 23. Copyright © 2017 MathWorks, Inc 23 • Automated driving sub-systems • Challenges faced testing automated driving • Perception system testing • Controls and system-level testing • Summary Talk Outline How to Test and Validate an Automated Driving System
- 24. Copyright © 2017 MathWorks, Inc 24 Algorithm Models Vehicle and Environment Models • All testing methodologies shown previously are open loop • Open loop testing: Input does not change with algorithm changes Open Loop Testing Forward Collision Warning Autonomous Emergency Braking Dynamic Static
- 25. Copyright © 2017 MathWorks, Inc 25 • All testing methodologies shown previously are open loop • Open loop testing: Input does not change with algorithm changes Testing Control Systems – What Else? • Closed loop testing is required to test control systems • Closed loop testing: Environment changes with algorithm.
- 26. Copyright © 2017 MathWorks, Inc 26 Algorithm Models Vehicle and Environment Models Closed Loop Testing Forward Collision Warning Autonomous Emergency Braking Dynamic Dynamic
- 27. Copyright © 2017 MathWorks, Inc 27 Ego Vehicle Dynamics Target Vehicle Dynamics Coordinate Transforms Sensor Models Closed loop simulation – reduces need for in-vehicle testing
- 28. Copyright © 2017 MathWorks, Inc 28 Closed Loop Simulation with Hardware in Loop Algorithm Models Vehicle and Environment Models Forward Collision Warning Autonomous Emergency Braking CAN Cable Why Use HIL ? • Run algorithm in lab first • Move hardware in-vehicle for test-drive
- 29. Copyright © 2017 MathWorks, Inc 29 Closed Loop Simulation with Hardware in Loop
- 30. Copyright © 2017 MathWorks, Inc 30 Challenge #1: In-vehicle testing is expensive and time consuming • Closed-loop simulation for virtual test-drives Challenge #2: Too dangerous for testing corner- cases • Simulate dangerous driving scenarios Challenge #3: Difficult to validate “black box” perception algorithms • Ground truth labeling and re-simulation How Engineers Solve Common Challenges Test and Validate an Automated Driving System
- 31. Copyright © 2017 MathWorks, Inc 31 • Simulation and offline testing augment (nor replace) in-vehicle tests • Two key methodologies used for testing: • Ground truth labeling and re-simulation – testing perception • Closed loop simulation with a gaming engine – system level testing Key Takeaways
- 32. Copyright © 2017 MathWorks, Inc 32 Automated Driving System Toolbox Design, simulate, and test ADAS and autonomous driving systems Resources More Information (Links) • Traffic Sign Recognition for Driver Assistance Systems, Dr. Alexander Behrens, Continental AG, MAC Stuttgart , 2015 • Developing Advanced Emergency Braking Systems at Scania - Jonny Andersson • Jayaraman, A., Micks, A., and Gross, E., "Creating 3D Virtual Driving Environments for Simulation-Aided Development of Autonomous Driving and Active Safety," SAE Technical Paper 2017-01-0107, 2017, doi:10.4271/2017-01-0107.