Scene Complexity Scores for Identifying Challenging Scenarios
Author: Mr. Sumanth Nagaraj, Product Manager of NavInfo Europe
Advances in automotive technology are guiding the transitioning from driving assistance functions to enable partial autonomy features. To ensure that fully automated vehicles are ready for the road, testing and validating autonomous driving systems (ADS) is essential. However, it can be very challenging and highly complex to test vehicles on an infinite number of scenarios. Creating relevant minimum and extended edge case scenarios that target specific ADS system features and functions thus becomes a need.
Simulation-based testing offers a highly efficient, cost-effective alternative which allows infinite scenarios to expand the verification and validation scope in a simulated virtual environment. Moreover, simulation scenarios can complement behaviors observed from real drive-based situations, or from knowledge-based insights. Identifying critical scenarios occurs when the system fails to behave as intended given the specific set of specifications for a test scenario. These events can be used to improve the system and make it error-free before deployment to the real-world environment.
Challenging Scenarios to Identify Critical Scenarios
To arrive at the scenarios which can lead towards critical behavior of AV system and software features, it is important to focus on the scene of Vehicle Under Test (VUT) operation. The complex scene environment in which the VUT operates based on its perception, challenges and interpretation is used to identify challenging scenarios. Offered by the significant difficulty/complexity levels of the scene, the challenging scenarios can then be tested for validating AV feature behavior, thus resulting to identifying the critical scenarios where the AV feature behavior tends to fail.
“Identification of Challenging Highway-Scenarios for the Safety Validation of Automated Vehicles Based on Real Driving Data”, Thomas Ponn, Matthias Breitfuß, Xiao Yu and Frank Diermeyer, Aug 2020
Identifying Challenging Scenarios based on the Complexity of the Scene
To quantify the complexity of a scene in which the VUT operates, different aspects of the static and dynamic components of scene and the VUT interactions are considered. Regulatory test attributes as featured in the L3 system of ALKS (Automated Lane Keeping System) - TTC (Time to Collision), THW (Time Head Way) provide a formulation of threat perception levels for the imminent danger for systems to test relative to the VUT and actors in the scene.
However, that alone may not suffice for L4 and L5 systems where the complexity of the ODD in operation is multifold with the challenges from the actor dynamics in the scene, complex infrastructure conditions, and path planning decision making of the VUT. Validating AV system behavior for optimal performance and robustness is vital under such challenging conditions while factoring various complexity metrics from the scene of operation
The following are complexity metrics that are used to quantify different complexities in the scene:
These metrics can be extended or limited for different ADS system and feature test use cases which can closely focus on the intended scenario needs to validate AV system or software. They can also provide highly complex scenarios to test the criticality of system and software behavior.
Complexity Scores to Identify Challenging Scenarios
The following images demonstrate a highly complex scene compared to a non-complex scene for AV system and software testing.
Complex and Non-Complex Scene in a Drive Scenario
In conclusion, assigning complexity scores from scene dynamics for identifying challenging scenarios provides a systematic approach to segment huge volumes of real drive data to focus on challenging scenarios which can test the AV feature behaviors to the limit. These extracted scenarios allow for the identification of challenging scenarios which are used to test critical behaviors of AV system and software features with rigorous tests and validation through simulation. This approach overlaps with the real-world scene dynamics under which the VUT operates.
In our last blog post, we discussed how real drive data can be extracted to create relevant, appropriate scenarios in NavInfo Europe’s simulation solutions. Our scenario extractor can assign complexity scores in a fully automated way to the collected real drive data. This can help setup a database of useful scenarios for intended testing for AV system and software.