ISO/FDIS 39003

Road traffic safety (RTS) — Guidance on ethical considerations relating to safety for autonomous vehicles

Sécurité routière — Recommandations relatives aux considérations éthiques en matière de sécurité pour les véhicules autonomes

General Information

• Status: Not Published
• ICS: 03.220.20 - Road transport
• Technical Committee: ISO/TC 241 - Road traffic safety management systems
• Drafting Committee: ISO/TC 241 - Road traffic safety management systems
• Current Stage: 5020 - FDIS ballot initiated: 2 months. Proof sent to secretariat
• Start Date: 07-Apr-2023
• Completion Date: 07-Apr-2023

Standards Content (Sample)

DRAFT INTERNATIONAL STANDARD
ISO/DIS 39003
ISO/TC 241 Secretariat: SIS
Voting begins on: Voting terminates on:
2022-07-11 2022-10-03
Road Traffic Safety (RTS) — Guidance on ethical
considerations relating to safety for autonomous vehicles
ICS: 03.220.20
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/DIS 39003:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. © ISO 2022
---------------------- Page: 1 ----------------------
ISO/DIS 39003:2022(E)
DRAFT INTERNATIONAL STANDARD
ISO/DIS 39003
ISO/TC 241 Secretariat: SIS
Voting begins on: Voting terminates on:
Road Traffic Safety (RTS) — Guidance on ethical
considerations relating to safety for autonomous vehicles
ICS: 03.220.20
COPYRIGHT PROTECTED DOCUMENT
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
© ISO 2022
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

NOT BE REFERRED TO AS AN INTERNATIONAL

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on STANDARD UNTIL PUBLISHED AS SUCH.

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

IN ADDITION TO THEIR EVALUATION AS

or ISO’s member body in the country of the requester. BEING ACCEPTABLE FOR INDUSTRIAL,

TECHNOLOGICAL, COMMERCIAL AND
ISO copyright office
USER PURPOSES, DRAFT INTERNATIONAL
CP 401 • Ch. de Blandonnet 8
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
CH-1214 Vernier, Geneva
POTENTIAL TO BECOME STANDARDS TO
Phone: [phone]
WHICH REFERENCE MAY BE MADE IN
Reference number
Email: [e-mail]
NATIONAL REGULATIONS.
Website: www.iso.org ISO/DIS 39003:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
Published in Switzerland
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
© ISO 2022 – All rights reserved
PROVIDE SUPPORTING DOCUMENTATION. © ISO 2022
---------------------- Page: 2 ----------------------
ISO/DIS 39003:2022(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction .............................................................................................................................................................................................................................. vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ..................................................................................................................................................................................... 1

3 Terms and definitions .................................................................................................................................................................................... 1

4 External factors affecting autonomous vehicle safety ................................................................................................ 2

4.1 General ........................................................................................................................................................................................................... 2

4.2 The road environment ..................................................................................................................................................................... 2

5 Stakeholders in AV design and operations ............................................................................................................................... 3

5.1 Producers – Manufacturers; designers and their suppliers ........................................................................... 3

5.2 Distribution chain – Distributors; Sellers ...................................................................................................................... 3

5.3 Purchasers, owners, and operators ..................................................................................................................................... 4

5.4 Government agencies and interested parties ............................................................................................................. 4

6 Governance, assessment, and evaluation .................................................................................................................................. 4

6.1 Ethical reference for assessment ........................................................................................................................................... 4

6.2 Additional Standards ........................................................................................................................................................................ 4

6.3 General ........................................................................................................................................................................................................... 5

6.3.1 Higher organizational level ....................................................................................................................................... 5

6.3.2 Development Organizational level...................................................................................................................... 5

6.3.3 Specific Development and implementation processes...................................................................... 6

6.3.4 Post Implementation Checking Against Ethical Criteria ................................................................ 7

6.4 Conducting the assessment ......................................................................................................................................................... 7

6.5 Expression of results and conclusion ................................................................................................................................. 8

7 Operationalization of Ethics - Discussion on values and ethics to consider ........................................8

7.1 Ethical Framework for the design of AVs (driving rules and ethical design) ................................. 8

7.1.1 Purpose ....................................................................................................................................................................................... 9

7.1.2 Values ......................................................................................................................................................................................... 10

7.1.3 Principles ............................................................................................................................................................................... 10

7.1.4 Methods for Construction and Evaluation of Maxims ...................................................................13

7.2 The Background of Maxims .....................................................................................................................................................13

7.2.1 Maxim Design and Construction ....................................................................................................................... 13

7.2.2 Evaluations of Maxims ............................................................................................................................................... 14

7.3 Driving Rules ......................................................................................................................................................................................... 15

7.3.1 Need vs. desire Driving Rule ................................................................................................................................. 16

7.3.2 Once on the road space .............................................................................................................................................. 16

7.3.3 In the lane behaviour (includes braking)....................................................................................................... 17

7.3.4 Lane switching .................................................................................................................................................................... 17

7.3.5 In the presence of the other ...................................................................................................................................... 18

7.3.6 Road/Infrastructure Use Cases ............................................................................................................................. 19

7.3.7 Resolving Conflict .............................................................................................................................................................20

7.3.8 Negotiations ......................................................................................................................................................................... 20

7.3.9 AV unable to function as intended ...................................................................................................................... 21

7.3.10 Yielding to first responders and emergency response vehicles .................................................... 21

7.3.11 Protecting other road users ..................................................................................................................................... 22

7.3.12 Unavoidable collision with other road users .............................................................................................. 23

7.3.13 Other Issues ........................................................................................................................................................................... 24

8 A Framework for rule construction and dealing with violations and deviations .......................24

8.1 Framework .............................................................................................................................................................................................. 24

8.2 Goals ............................................................................................................................................................................................................. 25

8.3 Primary rules ........................................................................................................................................................................................ 25

8.4 Supporting Rules ............................................................................................................................................................................... 26

iii
© ISO 2022 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/DIS 39003:2022(E)

8.5 Precautionary and Disabling Rules (prevention) ................................................................................................. 26

8.6 Reinforcing and Enabling Rules (performance) .................................................................................................... 26

8.7 Counter rules ......................................................................................................................................................................................... 26

8.8 Exceptions (prevention and performance) ................................................................................................................ 26

8.9 Discretionary and Compensatory rules ........................................................................................................................ 27

8.10 Misconduct .............................................................................................................................................................................................. 27

8.11 Violation .................................................................................................................................................................................................... 27

8.12 Deviations ................................................................................................................................................................................................ 27

8.13 Breakdowns ...........................................................................................................................................................................................28

8.14 RULE STRATEGY ................................................................................................................................................................................28

8.15 Boundaries (for prevention of unwanted behavior) ..........................................................................................28

8.15.1 Margins ....................................................................................................................................................................................28

8.15.2 Barriers ....................................................................................................................................................................................28

8.15.3 Buffers ......................................................................................................................................................................................28

8.16 Promoters (for performance) .................................................................................................................................................29

8.17 Further considerations ................................................................................................................................................................ 29

9 External/Internal Design ........................................................................................................................................................................29

10 Sustainability .......................................................................................................................................................................................................30

11 Review and Re-evaluation following controls system updates ......................................................................31

Annex A ........................................................................................................................................................................................................................................32

Annex B Sustainability Issues ...............................................................................................................................................................................34

Annex C .........................................................................................................................................................................................................................................35

Annex D Action plan – an example ...................................................................................................................................................................41

Bibliography .............................................................................................................................................................................................................................43

© ISO 2022 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/DIS 39003:2022(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www.iso.org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following

URL: www.iso.org/iso/foreword.html.

This document was prepared by Technical Committee ISO/TC 241, Road traffic safety management

systems.
A list of all parts in the ISO 39000 series can be found on the ISO website.
© ISO 2022 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/DIS 39003:2022(E)
Introduction
0.1 General

A long established and commonly held view is that the single most significant factor in road traffic

safety are the actions of the driver. It must however be emphasized that the road transport system is

a complex socio-technical system which places high demands on humans to negotiate. Crashes occur

since human beings due to finite cognitive capacity and physiological limitations cannot always cope

with these demands. To increase road safety the road transport system therefore has to be designed

to support the road user to cope with this complexity and to mitigate the effects of crashes when this

is not possible. When technology exceeds human capability autonomous vehicles have the potential to

replace the human driver and to further increase road safety

Autonomous vehicles have the potential to replace the human driver and to increase road safety by

reducing the opportunity of poor decision making and improper conduct in daily operations. This will

probably take a long time and in the meantime the capabilities of technology and humans should be

combined and integrated in such a way that the strengths of both are utilized efficiently. Important

safety improvements were made over the last century, but AD technology provides new opportunities

to improve safety even further.

Autonomous vehicles (AV) are unlikely to gain widespread acceptance until the travelling public feels

assured of their safety and security, not only of passengers but also other vehicles and vulnerable

road users. This includes a behaviour of the AV according to the desires and requirements of society.

However, despite contemporary expectations and a very optimistic view on technology, humans have

an extremely valuable faculty that machines will probably never possess -- ethical decision making and

judgement. For real traffic situations the human does not always have the prerequisites to take a rational

decision based on ethics since the time frame is often too narrow. For that reason, many “decisions”

are made instinctively without the possibility of making a well-reasoned and balanced ethical decision.

Defining this element and imparting it on machines is critical for the success of autonomous vehicles.

This can only be achieved by ensuring that autonomous vehicles are equipped with driving rules that

align with the general ethical beliefs, needs, and desires of humanity on a global level, subject to local

specific nuances.

To achieve the imprint of global and local ethical considerations in AV design, there is a need for a

framework of ethics involving the necessary stakeholders of different areas. To that objective’s end,

it is important to develop standards for AV behaviour. The objective is to ensure that these vehicles

are designed to minimize risk and behave in all circumstances in a way that is aligned with universal

expectations of a fair society. While there are few standards available, or under development, that

address the engineering and technological aspects of autonomous vehicles, there are no international

standards that address aspects concerning the general topics of driving policy and ethical behaviour,

which are also important. By driving policy we mean a general approach of how an AV makes a decision

and performs manoeuvres. Ethical relevant behaviour represents positive or potentially negative

impact on road users and especially the vulnerable ones as well as the public space at large.

The objective of this work is to lay out a framework for the development of a standard for ethical and

societally accepted driving policy (e.g., those relating to fairness and equity with respect to other road

users, the public space, and the environment).
0.2 The Concept of Autonomous Vehicles

Autonomous vehicles have the objective to substitute driver, including tasks, decisions, and

responsibilities. Hence the driver behaviour model proposed by Michon (1985) and applied to

autonomous vehicles by Ulbrich and colleagues (2013) (Figure 1) can be a start to design and operate

autonomous vehicles. The driving task consists of three levels: the strategic level concerned with the

higher-level trip goals (e.g., route choice), the tactical level concerned with the maneuvering decisions,

including negotiations and interactions with other vehicles, and operational level concerned with the

execution of these tactical and operational behaviours at the level of vehicle control. There is no strictly

hierarchical relationship among these levels.
© ISO 2022 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/DIS 39003:2022(E)
Figure 1 — Hierarchical model of driving task (source Ulbrich et al., 2013)

Beside this consideration which has a timely sequence, the autonomous vehicle must be designed

thoroughly to ensure safe and secure operation, the decision base for the desired behavior need to be

established. Therefore, all necessary information needs to be available, situation needs to be captured

and understood while different perspectives will support the final decision.

Hence a functional decomposition into six layers (Amersbach, Winner, 2017) (Figure 2) can provide the

first step into solution space.
vii
© ISO 2022 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/DIS 39003:2022(E)
Figure 2 — Decomposition Layers (source Amersbach, Winner, 2017)
— Information access
Information needs to be available and accessible
— Information reception
All necessary information can be captured
— Information processing

Captured information need to contain all required classification and identification for further

process
— Situational understanding
Based on information, situation need to be captured and understood
— Behavioral decision

Based on designed or trained situational awareness, the desired behavior needs to be chosen

— Action
Vehicle transforms whished behavior into action

Within this framework, technical realization can differ while the focus of the situation decision will

be comprehensible, accountable, and comparable among different designs. A thoroughly development

will benefit transparency for action and taken decision. The work described in this standard focuses

primarily on Layer 4 – behavioral decision
0.3 The Concept of Ethical considerations for autonomous vehicles
viii
© ISO 2022 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/DIS 39003:2022(E)

Philosophy helps people question, understand and make sense of the world so they can act properly in

it. This means that decisions and actions are not only intrinsically fair, but that they are also performed

in a way that is balanced with respect with other’s needs, the societal needs and humanistic values,

as well as with respect to the physical world around us. It is widely accepted that philosophy can be

divided into theoretical philosophy and practical philosophy. Theoretical philosophy is concerned with

understanding and making sense of the world and includes topics such as ontology, epistemology, and

meta-ethics amongst others. Practical philosophy focuses on topics such as human agency, practices,

values, as well as ethical dimensions of (human) behaviour and conduct.

Within practical philosophy there are two main branches: Normative and applied ethics. Normative

ethics is the study of ethical action and determining standards for decision making and conduct (e.g.

deontology, virtue ethics, etc. - see Annex B). Applied ethics is the application of standards of ethics to

real life situations (e.g. biomedical ethics, AI ethics, political ethics, etc. - see Annex B). In this document,

ethics refers to both normative and applied ethics but not to meta-ethics (which is more concerned

with the nature and basis of ethical concepts and the underlying reasoning and assumptions behind

moral theories).

Applied ethics is a flexible and practical way to address ethical considerations in the development of

new technologies because an applied ethics field, such as AV ethics, can borrow from more than one

normative school of ethics. Therefore, the framework offered in this document, although based on “a

principle-based approach, can use ideas from deontology and virtue ethics, for example, to help solve

problems. This is a balanced approach; it does not condone one type of normative ethics over another,

it offers a range of perspectives that will help the designer/developer in choosing the best (or better)

decision possible for specific situations.

In summary, ethics is the study of how to choose to act in situations. To make decisions, we need clarity

on what matters and to understand what is considered good, bad, right, and wrong. Therefore, ethics

may be viewed as a tool that helps us create the difference between a “good” decision and a “bad” one.

This is of great importance in the development of autonomous vehicles because choices made during

the design and development of AV systems determine its “driving behaviour” and how it caters for its

passengers and interacts with other road users (e.g., vulnerable road users). “Driving behaviour” is

what was designed for and programmed into the machine, “conduct” is what actually transpires as a

consequence of applying the driving behaviour to the real world.

This document offers a particular framework for AV ethics which is intended to support the practical

integration of ethics into the AV development process. The framework we suggest builds upon

Socrates conception of the hierarchical nature of philosophy for practical use which has three levels

(See Figure 3). The base level is the conduct that transpires as a consequence of decision and actions

(behaviour), the second level is how these behaviours are governed. Namely what kind of policy and

arrangements are in place to make decisions. The top level is Purpose, which Behaviour and Conduct

must align with, in order to achieve the goals and objectives of the entire endeavour. Finally, knowledge

is the “tool” for making sure that the purpose is reasonable and balanced and that the governance and

behaviour/conduct are indeed feasible and appropriate.
Figure 3 — Socrates’ hierarchical nature of philosophy
© ISO 2022 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/DIS 39003:2022(E)

Figure 4 is a graphical depiction of the framework advocated here for AV ethics. It shares similarities

with other ethical approaches for emerging technologies, it is also unique as it has been created

specifically for the AV context. Essentially, the AV ethics framework provides guidance for reflective

and critical decision making and is composed of four main elements: purpose, values, principles,

and process. The framework approach provides structure and guidance, yet it is flexible enough to

...