ISO 34502:2022

Road vehicles - Test scenarios for automated driving systems - Scenario based safety evaluation framework

This document provides guidance for a scenario-based safety evaluation framework for automated driving systems (ADSs). The framework elaborates a scenario-based safety evaluation process that is applied during product development. The guidance for the framework is intended to be applied to ADS defined in ISO/SAE PAS 22736 and to vehicle categories 1 and 2 according to Reference [10]. This scenario-based safety evaluation framework for ADS is applicable for limited access highways. This document does not address safety-related issues involving misuse, human machine interface and cybersecurity. This document does not address non-safety related issues involving comfort, energy efficiency or traffic flow efficiency.

Véhicules routiers — Scénarios d'essai pour les systèmes de conduite automatisée — Cadre d'évaluation de la sécurité basé sur des scénarios

General Information

• Status: Published
• Publication Date: 01-Nov-2022
• ICS: 43.020 - Road vehicles in general
• Technical Committee: ISO/TC 22/SC 33 - Vehicle dynamics and chassis components
• Drafting Committee: ISO/TC 22/SC 33/WG 9 - Test scenarios of automated driving systems
• Current Stage: 6060 - International Standard published
• Start Date: 02-Nov-2022
• Due Date: 27-Oct-2022
• Completion Date: 02-Nov-2022

Overview

ISO 34502:2022 - "Road vehicles - Test scenarios for automated driving systems - Scenario based safety evaluation framework" - provides guidance for a scenario-based safety evaluation process for automated driving systems (ADS). Intended for ADS as defined in ISO/SAE PAS 22736 and for vehicle categories 1 and 2, the standard is specifically applicable to limited access highways. It complements SOTIF (ISO 21448) and ISO 26262 by focusing on scenario-driven evaluation of the intended functionality rather than human–machine interface, cybersecurity, misuse, or non-safety aspects (comfort, energy, traffic flow).

Key topics and technical scope

• A structured, product-development integrated process for scenario-based safety evaluation, including:
  • Integration into development and safety test objectives
  • Specification of the relevant scenario space
  • Derivation of critical scenarios based on risk factors
  • Derivation and allocation of test scenarios and concrete test cases
  • Test execution and safety evaluation
• The standard contains clauses with requirements and recommendations, plus specified work products and conformity requirements.
• Informative annexes cover practical methods and supporting topics:
  • Physics principles for scenarios
  • Traffic-, perception-, and vehicle-control related critical scenarios
  • Structuring scenarios with criticality analysis
  • Qualification of virtual test platforms
  • Scenario databases, parameter variation methods, segmentation, behavioural safety assessment, risk evaluation, constrained-random testing, and traffic-data sufficiency

Practical applications

• Use ISO 34502:2022 to build a repeatable, evidence-based safety evaluation program for ADS on highways:
  • Define scenario spaces and parameter ranges for virtual and physical testing
  • Identify and prioritize critical scenarios that challenge perception, decision-making and control
  • Allocate test cases across simulation, closed-course and on-road verification
  • Evaluate whether an ADS is free from unreasonable risks within its intended operational design domain (limited access highways)
• Helps demonstrate compliance with SOTIF objectives and provides inputs for functional safety planning (ISO 26262 scope separation)

Who should use this standard

• Automotive OEMs and tier-1 system suppliers developing automated driving systems
• ADS safety engineers, test engineers and verification teams
• Simulation and test-platform providers (virtual test platform qualification)
• Regulators, certification bodies and independent test labs that assess ADS safety claims

Related standards and keywords

• Related: ISO 21448 (SOTIF), ISO 26262, ISO/SAE PAS 22736
• SEO keywords: ISO 34502:2022, automated driving systems, scenario-based safety evaluation, ADS test scenarios, critical scenarios, limited access highways, virtual test platforms, behavioural safety, scenario database.

Standards Content (Sample)

INTERNATIONAL ISO
STANDARD 34502
First edition
2022-11
Road vehicles — Test scenarios
for automated driving systems —
Scenario based safety evaluation
framework
Véhicules routiers — Scénarios d'essai pour les systèmes de conduite
automatisée — Cadre d'évaluation de la sécurité basé sur des
scénarios
Reference number
© ISO 2022
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii
Contents Page
Foreword .v
Introduction . vi
1 S c op e . 1
2 Nor m at i ve r ef er enc e s . 1
3 Terms and definitions . 1
4 T est scenario-based safety evaluation process . 2
4.1 I ntegration into the overall development process . 2
4 .1.1 Obje c t i ve s . 2
4.1.2 G eneral . 2
4.1.3 Requirements and recommendations . 8
4.1.4 Requirements for conformity . 9
4.2 S afety test objectives . 9
4 . 2 .1 Obje c t i ve s . 9
4.2.2 G eneral . 9
4.2.3 I nput to this clause . 9
4.2.4 R equirements and recommendations . 10
4 . 2 . 5 Work pr o duc t s . 10
4.3 S pecification of the relevant scenario space . 10
4 . 3 .1 Obje c t i ve s . 10
4.3.2 G eneral . 10
4.3.3 Input to this clause . 10
4.3.4 R equirements and recommendations . 11
4 . 3 . 5 Work pr o duc t s . 11
4.4 D erivation of critical scenarios based on risk factors . 11
4 .4 .1 Obje c t i ve s . 11
4.4.2 G eneral . 11
4.4.3 Input to this clause . 11
4.4.4 R equirements and recommendations .12
4 .4 . 5 Work pr o duc t s .12
4.5 D erivation of test scenarios based on covering the relevant scenario space .12
4 . 5 .1 Obje c t i ve s .12
4.5.2 General .12
4.5.3 Input to this clause . 13
4.5.4 R equirements and recommendations .13
4 . 5 . 5 Work pr o duc t s .13
4.6 D erivation of concrete test scenarios and test scenario allocation .13
4 . 6 .1 Obje c t i ve s .13
4.6.2 G eneral .13
4.6.3 Input to this clause . 13
4.6.4 R equirements and recommendations . 14
4 . 6 . 5 Work pr o duc t s .15
4.7 T est execution . .15
4.7.1 Ob jectives .15
4.7.2 I nput to this clause . 16
4.7.3 Requirements and recommendations . 16
4.7.4 W ork products . 17
4.8 S afety evaluation . 17
4.8.1 Objectives . 17
4.8.2 General . 17
4.8.3 Input to this clause . 17
4.8.4 Requirements and recommendations . 18
4.8.5 Work products . 18
iii
Annex A (informative) Physics principles scenario-based approach .19
Annex B (informative) Traffic-related critical scenarios .22
Annex C (informative) Perception-related critical scenarios .28
Annex D (informative) Vehicle control related critical scenarios .49
Annex E (informative) Derivation and structuring of scenarios using criticality analysis.53
Annex F (informative) Qualification of virtual test platforms .62
Annex G (informative) Scenario database and parameter variation methods .66
Annex H (informative) Segmentation of test space .69
Annex I (informative) Evaluation of test scenarios based on behavioural safety assessment .71
Annex J (informative) Risk evaluation based on positive risk balance .75
Annex K (informative) Constrained random testing to identify unknown critical scenarios .77
Annex L (informative) Sufficiency of traffic data to develop parameter ranges .79
Bibliography .80
iv
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html.
This document was prepared by Technical Committee ISO/TC 22, Road Vehicles, Subcommittee SC 33,
Vehicle dynamics and chassis components.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
v
Introduction
In order to safely introduce automated driving systems (ADS) into the market, socially acceptable and
technically sound scenario-based safety evaluation methodologies need to be developed. A number
of national and international governmental institutions are gradually releasing technical safety
[7][8][9]
guidelines to support the development of these methodologies, as well as associated regulations
and standards.
In order to evaluate whether ADSs are free from unreasonable risks, it is beneficial to develop safety
evaluation methodologies. Considering emphasis on limited access highways, scenario-based safety
evaluation methodologies are suitable for assessing safety in a repeatable, objective and evidence-
based manner and that is compatible with existing standards.
Functional safety is defined as the absence of unreasonable risks that arise from malfunctions of an
electric/electronic (E/E) system. The ISO 26262 series specifies a hazard analysis and risk assessment
to determine vehicle level hazards. This evaluates the potential risks due to malfunctioning behaviour
of the system and enables the definition of top-level safety requirements, i.e. the safety goals, necessary
to mitigate the risks.
For some E/E systems, which rely on sensing the external or internal environment to build situational
awareness, there can be potentially hazardous behaviour caused by or within the intended functionality.
Examples of the causes of such potentially hazardous behaviour include the inability of the function
to correctly comprehend the situation and operate safely or insufficient robustness of the function,
system, or algorithm. The absence of unreasonable risk resulting from hazardous behaviours related to
functional insufficiencies is defined as the safety of the intended functionality (SOTIF).
Functional safety (the ISO 26262 series) and SOTIF (ISO 21448) are distinct, necessary, and
complementary aspects of safety. This document is conformant with SOTIF and adds specificity to its
content, by incorporating a scenario-based safety evaluation process that identifies risk factors and
related critical scenarios that affect the intended functionality, and apply them to evaluate whether the
ADS is free from unreasonable risks.
The International Organization for Standardization (ISO) draws attention to the fact that it is claimed
that compliance with this document may involve the use of a patent.
ISO takes no position concerning the evidence, validity and scope of this patent right.
The holder of this patent right has assured ISO that he/she is willing to negotiate licences under
reasonable and non-discriminatory terms and conditions with applicants throughout the world. In
this respect, the statement of the holder of this patent right is registered with ISO. Information may be
obtained from the patent database available at www.iso.org/patents.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights other than those in the patent database. ISO shall not be held responsible for identifying
any or all such patent rights.
vi
INTERNATIONAL STANDARD ISO 34502:2022(E)
Road vehicles — Test scenarios for automated driving
systems — Scenario based safety evaluation framework
1 S cope
This document provides guidance for a scenario-based safety evaluation framework for automated
driving systems (ADSs). The framework elaborates a scenario-based safety evaluation process that
is applied during product development. The guidance for the framework is intended to be applied to
ADS defined in ISO/SAE PAS 22736 and to vehicle categories 1 and 2 according to Reference [10]. This
scenario-based safety evaluation framework for ADS is applicable for limited access highways.
This document does not address safety-related issues involving misuse, human machine interface and
cybersecurity.
This document does not address non-safety related issues involving comfort, energy efficiency or traffic
flow efficiency.
2 Normat ive references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO 34501, Road vehicles — Test scenarios for automated driving systems — Vocabulary
ISO 21448, Road vehicles — Safety of the intended functionality
ISO 26262-3, Road vehicles — Functional safety — Part 3: Concept phase
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 34501 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
critical scenario
scenario including one or more risk factors (3.3)
3.2
hazardous scenario
scenario in which harm occurs unless prevented by an entity other than the ADS
3.3
risk factor
factor or condition of a scenario that, if present, increases either the probability of the occurrence of
harm, or the severity of harm, or both
3.4
safety test objective
safety property of the ADS to be shown via a set of tests
Note 1 to entry: The safety test objectives can be derived from the validation targets or the acceptance criteria of
ISO 21448.
Note 2 to entry: The safety test objectives also include the aspect of the test end criteria.
Note 3 to entry: Depending on the kind of the safety test objectives the pass/fail-criteria of a concrete test
scenario can be included within the safety test objectives.
4 T est scenario-based safety evaluation process
4.1 Int egration into the overall development process
4.1.1 Objectives
The objectives of this clause are:
a) to provide an overview of the overall safety tasks and content of this document;
b) to provide an overview of the scenario-based safety evaluation process;
c) to explain the relationship between this framework and other standards and legislation.
4.1.2 General
4.1.2.1 Overall safety tasks and content of this document
Figure 1 presents the overall safety task “Identification and risk evaluation of the hazardous scenarios
of the ADS” and its derived subtasks.
Figure 1 — Overview of the different safety tasks to identify hazardous scenarios for the ADS
This document proposes to address the identification of potential hazardous scenarios via analysis
from two different starting points:
1. the relevant scenario space (task 1.1);
2. the system (task 1.2).
This approach is similar to the approach found in functional safety where the safety analysis is executed
from two different and complementary perspectives: The deductive approach (e.g. Fault Tree Analysis,
FTA) and the inductive approach (e.g. Failure Modes and Effects Analysis, FMEA).
In system-based approaches (task 1.2), the starting point of the analysis is the system itself. In scenario-
based approaches (task 1.1), which are the focus of this document, the starting point is the analysis of
the scenarios belonging to the relevant scenario space. For this approach the relevant scenario space
is analysed to identify risk factors. Only general physical limitations of the systems are considered,
for example, a sensor has a field of view based on the physics of its detection system, but other
implementation specific issues, e.g. the limitations of a machine learning algorithm to classify a detected
object correctly or sensor failures due to random hardware faults, are neglected. These system specific
aspects can be better analysed with system-based approaches. One advantage of the scenario-based
approach is that it can be applied with minimal dependency on the implementation of the system itself
(e.g. for regulatory use). As such, the results of a given analysis can be reused for different systems as
long as the relevant scenario space is the same, considering that the concrete parameters maximizing
the risk factor for a given scenario still have system dependencies (e.g. exact number and positions of
sensors).
NOTE 1 Knowledge gained during the execution of one approach (e.g. the system-based approach) can be used
to support the analysis by another approach (e.g. the scenario-based approach).
NOTE 2 The results of the system-based safety analysis can also be test scenarios to be executed.
Not all the relevant tasks for ADS safety evaluation are addressed by this document. This document
predominantly focuses on:
— task 1.1.1: identification and risk evaluation of potential hazardous scenarios via analysis of the
relevant scenario space (see 4.3); and
— task 1.1.2: derivation of a representative set of test scenarios to argue a sufficient coverage of the
relevant scenario space in search for unknown hazardous scenarios (see Annex K).
Guidelines for the execution of the remaining safety tasks can be found in other standards, e.g.
— task 2: ISO 21448;
— task 3: ISO/SAE 21434;
— task 1.2 and task 1.3: ISO 21448, the ISO 26262 series.
NOTE 3 Some safety issues can be assigned to multiple tasks.
EXAMPLE An adversarial attack, also known as “physical hack”, for example, in which sensors are spoofed
with the help of stickers on traffic signs, can be assigned to task 3 and task 1. Within task 3, the relevant attack
scenarios are identified. Within task 1.1 and task 1.2, it is evaluated whether the system is sufficiently robust
against the identified relevant attack scenarios.
NOTE 4 The result of task 1.2, the system based analysis, can also be scenarios that need to be tested in order
to evaluate the safety of the system.
NOTE 5 Overall guidance concerning safety for ADS considering SOTIF, functional safety and security can be
found in, e.g. ISO/TR 4804.
4.1.2.2 Overall flow of this document
Figure 2 shows the overall flow of this document within the scope of product development processes.
Within the figure:
— the first column from the left represents the inputs to the scenario-based safety evaluation process
elaborated within this document;
— the second column represents the preparation phase preceding the identification of critical scenarios
phase in which safety test objectives are specified;
— the third column provides an overview of the specification of the relevant scenario space, and
identification of risk factors and critical scenarios for safety evaluation according to the scenario-
based safety evaluation framework;
— the fourth column shows the interconnections among the scenario-based safety testing and
evaluation process (safety analysis phase) and the remaining product development phases;
— the fifth column represents how the output of the scenario-based safety evaluation framework fits
into the overall vehicle safety approval process that includes other safety validation steps;
— lines indicate iteration loops and influence conditions; they can contain new findings and trigger
necessary adaptations, when, for example, functional modifications are necessary due to safety
reasons.
The subclauses in Clause 4 aim at addressing the following points to contribute to an overall scenario-
based safety evaluation process.
— 4.1 Integration into the overall development process: How the framework integrates into
existing product development processes.
— 4.2 Safety test objectives: Specification of safety test objectives that the system needs to fulfil.
— 4.3 Specification of the relevant scenario space: How the relevant scenario space is defined.
— 4.4 Derivation of critical scenarios based on risk factors: How to define a set of critical scenarios
from which a set of test scenarios are derived.
— 4.5 Derivation of test scenarios based on covering the relevant scenario space: The
identification of critical scenarios to potentially be tested.
— 4.6 Derivation of concrete test scenarios and test scenario allocation: How test scenarios are
generated and allocated to different testing platforms.
— 4.7 Test execution: Requirements that need to be fulfilled while running test scenarios.
— 4.8 Safety evaluation: How the test results are evaluated to achieve an overall result.
Key
input
step in this document (clause number)
decision in this document
decision in this document
external decision
Figure 2 — ISO 34502 flow
Figure 3 illustrates the relationship between ISO 21448 and this document.
4.3 adds specificity to ISO 21448:2022, Clause 7, by identifying reasonably foreseeable risk factors that
may lead to hazardous scenarios. By structuring these risk factors, critical scenarios are generated and
compiled into a scenario catalogue for testing purposes. Therefore, the approach to identifying and
structuring risk factors in this document contributes to maximize the coverage of known hazardous
scenarios in SOTIF.
4.5 contributes to address ISO 21448:2022, Clause 9, by defining the concrete scenarios that need to
be tested and their corresponding platforms, which is an essential step to define the verification and
validation strategy.
Finally, 4.3 to 4.8 contribute to address ISO 21448:2022, Clauses 10 and 11. By using the known
hazardous scenario as additional input to the safety evaluation process, and varying some of the
properties/attributes of these scenarios, unknown hazardous scenarios can also be explored, and the
space and amount of unknown scenarios can be reduced.
NOTE The scenario-based safety evaluation process or parts of it can be applied to the system, subsystem
or component level, in addition to the vehicle level. Accordingly, the process is adapted to the corresponding ADS
under test.
Figure 3 — Relationship between ISO 21448 (left) and ISO 34502 (right) flow charts
4.1.3 Requirements and recommendations
This document shall be applied in combination with:
— ISO 21448.
4.1.4 Requirements for conformity
When claiming conformance with this document, each requirement shall be met unless a rationale
is provided, demonstrating that the non-conformity is deemed acceptable, i.e. the corresponding
objectives are still achieved.
4.2 S afety test objectives
4.2.1 Objectives
The objective of 4.2 is to specify the relevant safety test objectives for the ADS safety evaluation.
4.2.2 General
The safety test objectives represent the safety properties of the ADS to be shown via a set of tests. The
objectives are derived from general risk acceptance criteria like the principles of ‘as low as reasonably
practicable’ (ALARP), of ‘minimal endogenous mortality’ (MEM), of ‘positive risk balance’ (PRB), and
of applicable regulations. The safety test objectives are either derived from or provided by an external
[11]
source like ISO 21448 or by a related regulation . The safety test objectives are typically expressed
by using, for example, one of the two following procedures.
a) Safety test objectives specified as a boundary value (upper, or depending on the formulation, lower
boundary value) of the acceptable and demonstratable occurrence rate of a measurable safety-
related behaviour of the ADS (or its elements) during operation within the operational domain.
EXAMPLE 1 A hazardous behaviour of the system that is evaluated as critical, does not occur during x
hours of test operation within the operational domain.
EXAMPLE 2 The perception element forwards incorrectly perceived objects to the control element less
than once per y hours during operation within the operational domain.
EXAMPLE 3 The relative frequency of undesired behaviour in a given scenario is lower than x.
b) Safety test objectives specified as a performance reference model regarding the capability of the
ADS to handle certain scenarios safely, based on minimum performance levels required for these
scenarios.
EXAMPLE 4 The ADS is capable of preventing any accident that would be preventable according to a
reference performance model of a competent and careful human driver.
The safety test objectives are chosen in such a way that their fulfilment supports the overall safety
argument of the ADS. They represent a measurable or observable property of the ADS.
NOTE Additional safety arguments (e.g. safety analysis) can be a necessary part the fulfilment of the safety
test objectives to demonstrate that the overall safety argument is valid.
4.2.3 Input to this clause
4.2.3.1 Prerequisites
The following information shall be considered if available:
— industry standards (e.g. ISO 21448, the ISO 26262 series);
— operational design domain (ODD);
— design and functionality of the ADS, including the intended behaviour;
— other safety-relevant scenario catalogues (e.g. NCAP).
4.2.3.2 Further supporting information
The following information can be considered:
— traffic rules and regulations (e.g. Reference [11]);
— government guidelines (e.g. References [7][8][9]);
— regional specific social norms (e.g. Reference [12]).
4.2.4 Requirements and recommendations
The safety test objectives derived from external sources shall be specified.
NOTE Multiple safety test objectives can be specified to reflect the requirements from different external
sources or for different levels of abstraction.
4.2.5 Work products
Safety test objective(s) resulting from requirement 4.2.4.
4.3 Specification of the relevant scenario space
4.3.1 Objectives
The objective of 4.3 is to define and specify the relevant scenario space.
4.3.2 General
The relevant scenario space describes the possible scenarios that the ADS can encounter, in
consideration of the ODD and the possible manoeuvres of the ADS.
4.3.3 Input to this clause
4.3.3.1 Prerequisites
The following information shall be available:
— safety test objectives in accordance with 4.2;
— item definition in accordance with ISO 26262-3;
— specification of the functionality in accordance with ISO 21448;
— capabilities of the ADS (e.g. according to ISO/TR 4804);
— ODD;
— description of the design and the functionality of the ADS, including the intended behaviour;
— other safety-relevant scenario catalogues (e.g. NCAP);
— sources of information based on which parameter ranges can be defined (e.g. traffic monitoring
data, accident data, field operational test, naturalistic driving data, insurance data, map and road
data, expert knowledge, coverage requirements).
4.3.3.2 F urther supporting information
The following information can be considered:
— regulations (e.g. Reference [11]);
— government guidelines (e.g. References [7][8][9]);
— regional specific social norms (e.g. Reference [12]);
— scenario attributes.
4.3.4 Requirements and recommendations
4.3.4.1 The relevant scenario space shall be specified.
NOTE 1 Functional, abstract, logical and concrete scenario definitions can be used to support the specification
of the relevant scenario space.
[13]
NOTE 2 The technical representation can be the ASAM OpenSCENARIO format .
NOTE 3 The specification of the relevant scenario space can include parameter ranges and statistical
distributions.
4.3.5 Work products
Specification of the relevant scenario space resulting from 4.3.4.
4.4 Deri vation of critical scenarios based on risk factors
4.4.1 Objectives
The objectives of 4.4 are:
a) to analyse the relevant scenario space to identify risk factors;
b) to determine critical scenarios with the help of risk factors.
4.4.2 General
There are different possible approaches to identify safety critical scenarios via analysis (see Figure 1,
ISO 21448 and the ISO 26262 series). This document focuses on the scenario-based approach to identify
critical scenarios with the help of risk factors.
4.4.3 Input to this clause
4.4.3.1 Prerequisites
The following information shall be available:
— the information listed in 4.3.3.1;
— relevant scenario space in accordance with 4.3.5.
4.4.3.2 F urther supporting information
The following information can be considered:
— test results from previously tested scenarios.
4.4.4 Requirements and recommendations
4.4.4.1 Identification of risk factors
4.4.4.1.1 The relevant scenario space shall be analysed to identify risk factors.
NOTE 1 The physics principles approach can be used to identify the risk factors relevant to the ADS. See
Annexes A, B, C and D for detailed examples.
NOTE 2 The criticality analysis approach can be used to identify risk factors relevant to the ADS. See Annex E
for a detailed example.
4.4.4.2 Derivation of critical scenarios based on the analysis of the risk factors
4.4.4.2.1 The critical scenarios shall be identified under consideration of the identified risk factors.
NOTE 1 A structured approach can be used to fulfil this requirement. See Annexes A, B, C and D for detailed
examples.
NOTE 2 For this analysis, specific system issues resulting from, for example, machine learning algorithms, are
not considered. System restrictions are considered as far as physics principle aspects are concerned, e.g. the field
of view of a sensor or the limited ability to decelerate in case of low friction coefficient between the tyres and the
road surface. As such they reflect the general technical and physical limitations of the system.
NOTE 3 The critical scenarios can be described as functional, abstract, logical or concrete scenarios.
4.4.4.2.2 For the critical scenarios identified in 4.4.4.2.1 a representative set of scenarios shall be
specified.
NOTE 1 A methodology for the determination of parameter ranges from real traffic data can be found in
Reference [14].
NOTE 2 A methodology for the determination of traffic data sufficiency, by establishing a relationship between
the amount of data collected and the accuracy of the parameter ranges defined from the data can be found in
Annex L.
NOTE 3 Regional, national and international ordinance, guidelines and regulations can be used to determine
parameter ranges and statistical distributions.
NOTE 4 A set of logical scenarios can be used to derive a set of concrete test scenarios.
4.4.5 Work products
Set of critical scenarios resulting from requirements in 4.4.4.
4.5 Deri vation of test scenarios based on covering the relevant scenario space
4.5.1 Objectives
The objective of this clause is to derive a set of test scenarios. The set of test scenarios is chosen in such
a way that the relevant scenario space is sufficiently covered.
4.5.2 General
This approach addresses the task 1.1.2 mentioned in Figure 1.
4.5.3 Input to this clause
4.5.3.1 Prerequisites
The following information shall be available:
— the information listed in 4.3.3.1;
— relevant scenario space in accordance with 4.3.5.
4.5.3.2 F urther supporting information
The following information can be considered:
— set of critical scenarios based on risk factors in accordance with 4.4.5;
— test results from previously tested scenarios.
4.5.4 Requirements and recommendations
4.5.4.1 In case of safety test objectives based on a performance reference model, a set of test scenarios
shall be specified under the consideration of the performance reference model (see 4.2.2).
NOTE The definition of test scenarios can be supported by a scenario database (see Annex G).
4.5.4.2 A set of test scenarios shall be specified to ensure a sufficient coverage of the relevant scenario
space (see 4.2.2).
4.5.5 Work products
Set of test scenarios resulting from requirements in 4.5.4.
4.6 Deri vation of concrete test scenarios and test scenario allocation
4.6.1 Objectives
The objectives of 4.6 are:
a) to define general requirements for testing concrete scenarios;
b) to provide guidance for the allocation of test scenarios to different testing platforms;
c) to define general capability requirements for tools used for verification and validation.
4.6.2 General
In general, different platforms, including simulation/virtual test platforms (VTP), track-test platforms,
and real-world test platforms, can be used individually or in combination for scenario-based safety
evaluation. While using them, each platform fulfils different requirements relating to accuracy,
repeatability and traceability (see 4.6.4.3).
4.6.3 Input to this clause
4.6.3.1 Prerequisites
The following information shall be available:
— the information listed in 4.3.3.1;
— relevant scenario space in accordance with 4.3.5;
— information about the capability of the testing platforms to be used (see Annex F).
4.6.3.2 Further supporting information
The following information can be considered:
— set of test scenarios in accordance with 4.5.5;
— results from previously tested scenarios;
— exposure data (e.g. based on recorded data, statistics, etc.).
4.6.4 Requirements and recommendations
4.6.4.1 Derivation of a set of concrete scenarios to be tested
4.6.4.1.1 To achieve a test coverage of the scenario space sufficient for the required safety argument,
parameter ranges and their combinations shall be defined for testing.
NOTE 1 An approach to segment the scenario space and to define relevant representative scenarios can be
found in Annex E.
NOTE 2 If the databases of the parameter distribution are insufficient, parameter variation methods can be
[15]
used (see Annex G), e.g. Monte Carlo methods, methods considering parameter dependencies or risk-based
[16]
model methods .
NOTE 3 When selecting representative test scenarios, scenarios involving multiple risk factors can be
given special consideration. The combination of multiple mild factors can potentially lead to a severe case. For
example, for snowy weather, non-favourable light conditions, and an aggressive cut-in by a surrounding vehicle,
the combined scenario considering all these factors can potentially be very dangerous.
4.6.4.1.2 The definition of concrete parameter values and their combinations shall be based on the
relevant safety test objectives provided in 4.2.
NOTE 1 Using randomly selected concrete test scenarios can support the safety argument against unknown
hazardous scenarios. See Annex K for more details.
NOTE 2 Parameter variation methods can also help to identify new risk factors or the worst case conditions
for these risk factors. See Annex G for more details.
NOTE 3 In the case of logical scenario, adaptive generation of concrete scenarios can be done for each logical
scenario, dynamically, based on results from previous tests.
4.6.4.1.3 All test scenarios that have been identified from 4.3 to 4.5 as relevant with respect to the
safety test objectives defined in 4.2, shall be tested.
4.6.4.2 Allocation of tests to different test platforms
4.6.4.2.1 Relevant test scenarios shall be allocated to at least one test platform.
4.6.4.2.2 The selected test platform shall be suitable for the assigned test scenarios.
NOTE 1 The allocation of tests to VTP can consider large number of scenarios. These platforms are particularly
suitable to execute tests that would be too dangerous or too complicated to execute in real life.
NOTE 2 The allocation of tests to track-test platforms can be based on pre-selected tests, e.g. certification
tests, test with a high relevance regarding drive dynamics and real sensor performance, rare events which can
hardly be seen in real-world tests or events on public road with less repeatability.
NOTE 3 The allocation of tests to real-world platforms can consider public road safety and can be based on
high relevance regarding real system performance. Depending on the possibility to control each single parameter
the surrounding conditions can vary more or less randomly.
4.6.4.3 Capability requirements for qualification of the used platforms (including models)
4.6.4.3.1 VTP and track-test platforms shall deliver the same repeatable and reproducible results,
within reasonable tolerances.
4.6.4.3.2 VTP track-test platforms and real-world platforms shall deliver traceable results.
NOTE Traceable refers to the relation between safety test objectives, test scenarios and test results.
4.6.4.3.3 Platforms shall be suitable t
...