ADAS Testing for Robust Verification & Validation Process

Advanced Driver Assistance Systems (ADAS) have become integral features for automobiles on road. Their inclusion was not just to reduce driver stress but also to promote safety. Existing ADAS features such as adaptive cruise control, lane-keep assist, collision mitigation braking system, and many more have proven results that suggest their inclusion has significantly reduced head on, rear end, and cross traffic collisions.

Table-1

Depending on the level of autonomy, ADAS features vary in their functionality. As per SAE, Level 0 is where a driver has full control of the vehicle, and Level 5 is where the vehicle can perform driving functions under all driving conditions. In the current scenario most of the ADAS features fall within L0-L2 autonomy which still requires the driver to take control, even in the presence of combined automated controls. The development of ADAS features as described by NHTSA is shown in Figure 3. These features are heavily present in new production vehicles as they have been shown to aid in fostering a safe driving environment. Original Equipment Manufacturers (OEMs) must be even more careful due to the user’s reliance on these technologies to perform and aid some driving-related functions. As a result, OEMs should instill a safety-by-design approach through which any shortcomings associated with sensor technology that could lead to hazards can be evaluated during the early design phase.


5 pahses of Automation

Some OEMs follow best practices through standards such as ISO 26262 which covers Electronic/Electric (E/E) safety-critical malfunctions associated with a system. Understanding system behavior and evaluating hazards are two crucial aspects that contribute towards safe system design. To ensure these factors are evaluated for any E/E system, for L0-L2, ISO 26262 suggests a strategic framework starting from the design phase through testing and verification & validation (V&V). This process should be deeply rooted within an organization during the early design phase to help understand any potential risks beforehand, as it can impact system confidence. Due to dynamic behavior associated with L3-L5, it is difficult to address these aspects during the early design. The rigor provided by ISO 26262, along with a strategic framework, was proven to be effective if implemented from the design phase to testing to early production phase for any Automotive Safety Integrity Level (ASIL) yielded item.

For both current and legacy systems under development, engineers rely on test strategy and V&V criteria. The testing schema proposed by ISO 26262 satisfies the testing requirements to verify and validate L0-L2 and partially L3 systems since its faults were already known. However, the ISO 26262 V&V process falls shorts when testing L3-L5 systems. The vehicle operates in varying environmental conditions. Such change in landscape can introduce new risks during operation where not all testing scenarios can evaluate the unknown risks associated. It is challenging for OEMs to develop a robust V&V process to make their system fail safe and comply with safety-by-design criteria.

Due to the residual behavior of risks involved in unknown hazards, it is challenging for engineers to test for any of these using the current simulation techniques. To help understand system behavior for corner case scenarios – scenarios that do not show up during testing – ISO PAS 21448: Safety of the Intended Functionality (SOTIF) has a theoretical approach and set of guidelines.

 For L0-L2 autonomy, ISO 26262 is robust enough to identify E/E failures. What is the case for autonomy L3-L5 where unknown system hazards are more prevalent? ISO 26262 along with ISO PAS 21448 helps to identify and promote risk-free approaches for all levels of autonomy.

Figure-2

Figure 2 System design with known/unknown hazard scenarios and corresponding safety standard

 

Timeline

Figure 3: Safety features by NHTSA

V&V Strategy ISO 26262/ISO PAS 21448

ISO 26262 Process Overview:

The ISO 26262 process starts by providing guidelines for promoting a safety culture by defining the management of functional safety. In the next steps, the item is defined through the concept phase in which all the possible hazards associated to the system are evaluated. This helps in calculating the ASIL rank. Further, a safety goal is derived based on the ASIL ranked hazards which are essential for providing and generating requirements. In ISO terminology, all the requirements will be derived under the Functional Safety Concept (FSC), and they form the basis for lower-level system requirements. FSC is critical for developing a safe system design and helps to develop requirements defined in Technical Safety Concept (TSC). This forms the basis for hardware and software safety requirements. Through the safety analysis approach, the residual risks associated can be further evaluated. In the following sections a detailed testing and V&V strategy will be discussed.

Table-2

 

Validation and Verification:                                  

Validation is used to ensure that a system is free from any unreasonable risks due to unknown hazards. Generally, in the V&V process, the hazards and risks that were not evaluated during the initial hazard and risk analysis can be done using a safety-analysis approach – Failure Mode and Effect Analysis (FMEA) or Fault Tree Analysis (FTA). To satisfy the validation coverage criteria, simulations help to explore a wide range of possible relevant scenarios. Through this process, the ASIL rank allocated to the system can be even more robust, which helps in implementing an appropriate testing strategy applicable for L0-L2 systems.

On the contrary, when the vehicle takes control, for L3-L5, there is a high possibility for mode confusion and system responsibilities. These risks cannot be tested during V&V. To address the shortcomings of dynamic risks associated to the system, ISO PAS 21448, draft, is setting V&V guidelines. However, ISO PAS 21448 must make sure to address the follow questions in order to establish a robust V&V process.

  • Is the safety analysis enough to evaluate hazards associated with the system?
  • Does the test specification cover the required use cases to evaluate system performance?
  • Is the testing strategy enough to complete validation and verification of the system?

The following sections introduce the V&V scope which includes verifying system requirements, leveraging the V&V process to increase system confidence, and system observation. These sections will also further address challenges associated with L3-L5 due to corner scenarios. The testing approach for defining testing goals, testing use cases, test targets, etc., which are clearly defined by 5 Who’s and 2 How’s (5W-2H) approach that can be taken to overcome challenges posed by V&V will be discussed as well.

Section 1: Understanding V&V Scope

The scope of V&V demonstrated in this section addresses advanced autonomy levels – L3-L5. The validation process is critical for safe feature deployment especially in the cases of L3-L5 autonomy. For L0-L2 autonomous features, testing and validation strategies provided by ISO 26262 are enough to validate the system design. But in cases where system misuse is deliberate, SAE J31408/ISO 21434 (under development) standards suggest guidelines for security related issues.

Entire system validation is not achieved by testing alone, but it can be achieved by robust implementation of quality audits, expert reviews, or through safety analysis techniques. The validation plan schematic shown above is an example illustration of how it can be implemented using ISO 26262 standard. With increased complexity, testing helps to determine any hidden faults and increases systems confidence for use in real-time driving conditions. The following steps map out the required criteria that the V&V process must satisfy.

STEP 1: Requirements Verification to Satisfy System Design

With the help of readily-available and known scenarios, engineers can further ensure system safety through testing. The verification process mainly focuses on testable requirements and can rely on safety by design processes that were long used in production vehicles. The concept of safety by design is fundamental, but for autonomous vehicles it can be difficult due to unknown scenarios which cannot be directly verifiable. For example, scenarios such as brightness, construction zones, and tunnels can occlude the sensors and create an unsafe driving environment. To address these issues validation of both known and unknown scenarios is necessary.

STEP 2: Leverage Validation Process to Maximize System Confidence

In real-time conditions, a system with 100-percent reliability and 0-percent free of fault is difficult to achieve due to uncertainties which cannot be avoided. Factors such as the Operational Design Domain (ODD) and the Object and Event Detection Response (OEDR) are crucial to ensure system functions are within the design domain and the user’s required response. Typically, validation is accompanied by testing the verified system either by controlled on-road testing (e.g. proving grounds) or through simulations such as Hardware in the Loop (HIL), Software in the Loop (SIL), and Driver in the Loop (DIL). During validation, the system’s function is to be stable to avoid any unknown scenarios.

STEP 3: Post Deployment Observation.
Post deployment observation is an important step which enables safety engineers to introduce and update any changes required to the system functions. The changes or updates can be the result of long-terms tests, field monitoring, or security tests. Safety engineers need to make sure that introducing changes and updates should not introduce any news risks or alter the system functions, which can impact safety.

Section 2: V&V Challenges Associated with Highly Autonomous Features

It is highly challenging to test all the scenarios for any system, as some sensors can present challenges due to the dynamic nature and uncertainty associated with most of them. A positive balance between detecting all the known and unknown scenarios and system performance is necessary. These are the five challenges that were presented in this paper (as adapted from Intel Safety First for Automated Driving).

Challenge 1: System Safety without Driver Interaction

For L0-L2 automated vehicles, since the vehicle is constantly under driver control, all the worst-case scenarios can be accounted for to ensure system safety. But in L3-L5, the vehicle is in control and the driver does not need to be fully alert. Safe system operation requires a unique approach that has the highest rigor which allows the vehicle to drive by itself. Statistical validation methods such as regression analysis and standard deviation can be applied to the obtained data sets through testing.

Challenge 2: System Safety with Driver Interaction

Complex maneuvers such as overtaking, lane changing, and tunnel driving can be challenging tasks for autonomous vehicles. As part of the V&V process for L3-L5, all these complex maneuvers tasks need to be assessed. This list can increase once autonomous vehicles mature. Mode transitions, autonomous to human, for complex scenarios requires the user to be available to constantly monitor system performance and take control if any system anomalies are found. Also, long-term autonomous driving makes the user complacent and it is safe practice to include these scenarios in the V&V process and system impact analysis.

Challenge 3: Scenarios Not Known in Traffic

Sensor interaction is a normative procedure for L3-L5 to perform critical driving tasks. It is important that sensors capture all the essential information and respond instantaneously to generate necessary vehicle control functions actuation. Driving patterns are always shifting and autonomous vehicles should be able to adapt to these unknowns. Security-related risks in traffic also arise when the Vehicle-to-Vehicle, Vehicle-to-Pedestrian, etc. (V2X) interactions are in full implementation. All these factors are inputs into validation.

Figure 4 is an example of a simplified illustration of normal/unsafe operation condition for collision avoidance system equipped with a camera sensor. In this scenario the camera sensor relays information in the form of images at current time (t) to perform a driving brake operation. The decision algorithm helps to classify images based on a familiarity index which includes preloaded route geography, environmental conditions, speed range, and other factors. In a normal operation case, where the image is identified as familiar, the ECU can send signals for brake actuation. For images that cannot be identified as known then the ECU might generate unwanted driving control action which can lead to a hazard.

Figure-4

 

Function

Collison Avoidance Using Camera Sensor

System Composition

Input: Images (t) – Images at current time, t

 

Processing: Decision Algorithm – Provides decision on image classification based on familiarity index

 

Processing/Actuation Initiation: ECU – Processes the information and arms appropriate driving control function

 

Scenario

An L4 system trying to maneuver from point A to point B is performing a driving control action based on the received images from camera sensor

Possible result

Normal Operation – System safely brakes the vehicle based on the identified images

 

Unsafe Operation – Dynamic environmental conditions resulting in decision confusion and unwanted control generation leading to a hazard


Challenge 4: Validation of various system configurations and variants
.
During the lifetime of autonomous vehicles, ADAS features require software updates or hardware changes. Due to the complex nature of ADAS systems configurations for L3-L5, it is critical to address how the updates and modifications to the system are handled during their service time in the V&V process.

Challenge 5: Validation of systems/subsystems based on machine learning.
Algorithm complexity in L2 features are less in comparison to L3-L5. Since L3-L5 autonomous vehicles rely heavily on complex algorithms that include extensive machine learning or artificial intelligence, V&V efforts requires a different approach. Implementing validation for subsystems within a system configuration can be challenging which increases testing effort.

Section 3: Approach for V&V Automated Driving Systems

Testing is a key requirement for V&V to ensure proper design implementation. Testing within validation also helps to verify whether the specified functional requirements are met. For L3-L5, validating autonomous driving features cannot be completely achieved by the current test criteria proposed by the functional safety standard and safety of the intended functionality.

Establishing a robust test strategy is critical for the testing process. To understand crucial elements that contribute towards testing strategy, 5W-2H (Ohno, 1998; Tague 2005), elaborated as 5W: Who?, What?, Where?, When?, Why?, and 2H: How?, How well?, needed to be understood. For L0-L2, the when and who questions can be answered by the current process development standards. Validating a system design alongside answering the remaining questions, excluding those for L0-L2, by the 5W-2H concepts can improve the overall V&V process and is defined in Table 1.

 

WHY? and HOW WELL?

Test goals, completion criteria, metrics

HOW?

Test techniques

WHERE?

Test platforms

WHAT?

Test elements or objects under test


Table 1: 5W-2H Concept explaining Testing strategy

These factors help in establishing a refined testing process for L3-L5 autonomous driving to support validation.

In Conclusion:

Risk identification and hazard evaluation as early as the design phase is critical as it can impact systems integrity. Overall, safer systems help promote safer roads. A systematic safety approach can be possible through an integrated ISO 26262 process implementation. Increased user demand for ADAS features in automobiles requires OEMs to tackle new challenges and comply to regulatory requirements. Due to system complexity associated with ADAS sensors, a unique safety approach rather than a traditional process is required.

The operation lifecycle for ADAS sensors is mostly in a dynamic environment; testing is required to ensure system safety. Testing and V&V processes plays an important role as they provide visibility on the system intended functionality, system failure modes behavior, and safety mechanisms robustness. SOTIF suggests the same procedure for V&V of unknown system behavior for corner case scenarios, which is common with higher autonomy levels.

LHP has helped customers with ISO 26262 process, work products development, and functional safety assessments for the past 8 years. With our expertise in ADAS and process development standards such as ISO 26262, ISO PAS 21448, Cybersecurity, AUTOSAR, and ALM tools, LHP can play a crucial role for an organization’s technical and process implementation needs.

 

Appendix 

 

Term

Definition

Corner Case

Scenarios that are dynamic and there is no visibility to them during the standard process implementation

Scenario

Description of a temporal development between several scenes in a sequence of scenes influenced by actions and events

Safety by Design

Detecting and minimizing risks as early as the design phase of any system/item/component

ODD

Specific condition under which a given driving automation system is designed to function

OEDR

Dynamic driving task that includes monitoring the driving environment and executing an appropriate response to objects and events

Levels of Autonomy

Level 0 (L0) – Driver is in full control, no automation

Level 1 (L1) – Driver is in full control, some driver assistance features included in vehicle design

Level 2 (L2) – Partial automation with combined automated functions (e.g., steering, braking, etc.), driver must be engaged at all times and monitor environment

Level 3 (L3) – Conditional automation; driver is necessary but not required to monitor environment

Level 4 (L4) – High automation; vehicle has the capability to perform certain functions

Level 5 (L5) – Full automation; vehicle can perform all driving functions

 

 

References

APTIV, A. (2019). Safety First for Automated Driving. APTIV,AUDI,CONTINENTAL,DAIMLER,INTEL,INFINEON,FCA.

NHTSA. (2016). Assesment of safety standards for Automotive Electronic Control systems. NHTSA.

NHTSA. (2016). Fedral Automated Vehicle Policy. NHTSA.

NHTSA. (n.d.). Automated Vehicles for Safety. Retrieved from NHTSA: https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety

Organisation, I. S. (2011). Road-Vehicles Functional safety: ISO 26262.

Saenz, A. (n.d.). Developing Test Solutions to Safely Operate an ADAS In Varying Real-World Conditions . LHP Engineering Solutions.

T.Ohno. (1988). Toyota Production System, Beyond Largescale production. Oregon: Productivity press.

TAGUE, N. (2005). The Quality Tool Box. Wisconsin : Quality press.

Steve Neemeh

Written by Steve Neemeh

Steve joined LHP in 2015 to lead the expansion of the west coast operations. He is the leader of the strategy and solutions architects as well as president of the delivery consulting organization. Steve has over 25 years of Functional Safety experience prior to joining LHP. Steve has launched multiple start-up operations and has taken them to full production. Notably, a complete ground up electronics and software development group to service commercial aerospace electronics and military vehicle power electronics. For LHP, Steve pioneered the implementation of safety critical applications in California, launching functional safety for autonomous driving applications as well as air mobility.