Author: Mr. Sumanth Nagaraj, Product Manager of NavInfo Europe

Advances in automotive technology are guiding the transition from vehicles having driving assistance functions to enabling partial autonomy features. To ensure that fully automated vehicles are ready for the road, testing and validating autonomous driving systems (ADS) is essential. However, it can be very challenging and highly complex to test vehicles on an infinite amount of scenarios. Therefore, it is important to create relevant minimum and extended edge case scenarios that target specific ADS system features and functions thus becoming a need.

Simulation-based testing offers a highly efficient, cost-effective solution that allows infinite scenarios to expand the verification and validation scope in a simulated virtual environment. Moreover, simulation scenarios can complement behaviors observed from real drive-based situations, or from knowledge-based insights. Identifying critical scenarios occurs when the system fails to behave as intended given the specific set of specifications for a test scenario. These events can be used to improve the system and rid the system of major errors before deployment to the real-world environment.

Challenging Scenarios to Identify Critical Scenarios

To arrive at the scenarios which can lead towards critical behavior of AV system and software features, it is important to focus on the scene of Vehicle Under Test (VUT) operation. The complex scene environment in which the VUT operates based on its perception, challenges, and interpretation is used to identify challenging scenarios. Offered by the significant complexity levels of the scene, the challenging scenarios can then be tested to validate AV feature behavior. Thus, resulting in the identification of the critical scenarios where the AV feature behavior tends to fail.

Identifying Challenging Scenarios Based On The Complexity Of The Scene

To quantify the complexity of a scene in which the VUT operates, different aspects of the static and dynamic components of the scene and the VUT interactions are considered. Regulatory test attributes as featured in the L3 system of ALKS (Automated Lane Keeping System) – TTC (Time to Collision) and THW (Time Head Way) provide a formulation of threat perception levels for the imminent danger. This allows systems to test relative to the VUT as well as other actors in the scene.

However, that alone may not suffice for L4 and L5 systems where the complexity of the ODD in operation is multifold. This includes the challenges from the actor dynamics in the scene, complex infrastructure conditions, and path planning decision-making of the VUT. Validating AV system behavior for optimal performance and robustness is vital under such challenging conditions while factoring various complexity metrics from the scene of operation.

It is possible to quantify various complexities in the scene using the following complexity metrics: 

1. The number of actors in the scene of the region of interest.
2. Types of actors in the scene of the region of interest.
3. Individual actors in the scene – velocity, acceleration, deceleration relative to the ego-vehicle influence
4. Maximum & minimum influence of the longitudinal and latitudinal velocity, acceleration, deceleration of actors in the scene.
5. Mutual space influence the scene between the traffic actor’s space for understanding the traffic participants’ possible interactions.
6. Time headway between ego and actor in a longitudinal direction
7. The time between a collision between the ego and surrounding vehicles.
8. Possible actions of an ego vehicle.

These metrics can be extended or limited for different ADS systems and feature test use cases that can focus on the intended scenario needed to validate AV systems or software. They can also provide highly complex scenarios to test the criticality of system and software behavior.

Complexity Scores to Identify Challenging Scenarios

The following images demonstrate a highly complex scene compared to a non-complex scene for AV system and software testing.

Conclusion

In conclusion, assigning complexity scores from scene dynamics for identifying challenging scenarios provides a systematic approach to segment huge volumes of real drive data to focus on challenging scenarios. These can test the AV feature behaviors to the limit, and allow for the identification of challenging scenarios. Afterward, these scenarios are used to test critical behaviors of AV systems and software features with rigorous testing and validation through simulation. This approach overlaps with the real-world scene dynamics under which the VUT operates.

In our last blog post, we discuss how real drive data can be extracted to create relevant, appropriate scenarios in NavInfo Europe’s simulation solutions. Our Scenario Extractor can assign complexity scores to real drive data in a fully automated manner. This can help set up a database of useful scenarios for intended testing for AV systems and software.