ISO/IEC PRF TR 29119-11

TECHNICAL ISO/IEC TR
REPORT 29119-11
First edition
Software and systems engineering —
Software testing —
Part 11:
Testing of AI-based systems
PROOF/ÉPREUVE
Reference number
ISO/IEC TR 29119-11:2020(E)
ISO/IEC 2020
---------------------- Page: 1 ----------------------
ISO/IEC TR 29119-11:2020(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2020

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms, definitions and abbreviated terms ................................................................................................................................ 1

3.1 Terms and definitions ....................................................................................................................................................................... 1

3.2 Abbreviated terms ............................................................................................................................................................................10

4 Introduction to AI and testing .............................................................................................................................................................11

4.1 Overview of AI and testing .........................................................................................................................................................11

4.2 Artificial intelligence (AI) ...........................................................................................................................................................11

4.2.1 Definition of ‘artificial intelligence’ ..............................................................................................................11

4.2.2 AI use cases ........................................................................................................................................................................11

4.2.3 AI usage and market ..................................................................................................................................................12

4.2.4 AI technologies ...............................................................................................................................................................12

4.2.5 AI hardware .......................................................................................................................................................................15

4.2.6 AI development frameworks ..............................................................................................................................15

4.2.7 Narrow vs general AI ........................................................................................................................................... ......15

4.3 Testing of AI-based systems .....................................................................................................................................................16

4.3.1 The importance of testing for AI-based systems .................. .............................................................16

4.3.2 Safety-related AI-based systems .....................................................................................................................16

4.3.3 Standardization and AI ............................................................................................................................................16

5 AI system characteristics ..........................................................................................................................................................................18

5.1 AI-specific characteristics ..........................................................................................................................................................18

5.1.1 General...................................................................................................................................................................................18

5.1.2 Flexibility and adaptability ..................................................................................................................................19

5.1.3 Autonomy ............................................................................................................................................................................20

5.1.4 Evolution ..............................................................................................................................................................................20

5.1.5 Bias ...........................................................................................................................................................................................20

5.1.6 Complexity .........................................................................................................................................................................21

5.1.7 Transparency, interpretability and explainability ...........................................................................21

5.1.8 Non-determinism .........................................................................................................................................................22

5.2 Aligning AI-based systems with human values .......................................................................................................22

5.3 Side-effects ..............................................................................................................................................................................................22

5.4 Reward hacking ...................................................................................................................................................................................23

5.5 Specifying ethical requirements for AI-based systems .....................................................................................23

6 Introduction to the testing of AI-based systems ...............................................................................................................24

6.1 Challenges in testing AI-based systems ..........................................................................................................................24

6.1.1 Introduction to challenges testing AI-based systems ...................................................................24

6.1.2 System specifications ................................................................................................................................................24

6.1.3 Test input data ................................................................................................................................................................25

6.1.4 Self-learning systems ................................................................................................................................................25

6.1.5 Flexibility and adaptability ..................................................................................................................................25

6.1.6 Autonomy ............................................................................................................................................................................25

6.1.7 Evolution ..............................................................................................................................................................................25

6.1.8 Bias ...........................................................................................................................................................................................26

6.1.9 Transparency, interpretability and explainability ...........................................................................26

6.1.10 Complexity .........................................................................................................................................................................26

6.1.11 Probabilistic and non-deterministic systems ......................................................................................26

6.1.12 The test oracle problem for AI-based systems ....................................................................................26

6.2 Testing AI-based systems across the life cycle .........................................................................................................27

6.2.1 General...................................................................................................................................................................................27

6.2.2 Unit/component testing .........................................................................................................................................27

6.2.3 Integration testing .......................................................................................................................................................27

6.2.4 System testing .................................................................................................................................................................28

6.2.5 System integration testing ....................................................................................................................................28

6.2.6 Acceptance testing ...................................................................... .................................................................................28

6.2.7 Maintenance testing ...................................................................................................................................................28

7 Testing and QA of ML systems ..............................................................................................................................................................28

7.1 Introduction to the testing and QA of ML systems ................................................................................................28

7.2 Review of ML workflow ................................................................................................................................................................29

7.3 Acceptance criteria ...........................................................................................................................................................................29

7.4 Framework, algorithm/model and hyperparameter selection ..................................................................29

7.5 Training data quality .......................................................................................................................................................................29

7.6 Test data quality ..................................................................................................................................................................................29

7.7 Model updates ......................................................................................................................................................................................29

7.8 Adversarial examples and testing ........................................................................................................................................29

7.9 Benchmarks for machine learning ......................................................................................................................................30

8 Black-box testing of AI-based systems ........................................................................................................................................30

8.1 Combinatorial testing .....................................................................................................................................................................30

8.2 Back-to-back testing ........................................................................................................................................................................31

8.3 A/B testing ...............................................................................................................................................................................................32

8.4 Metamorphic testing .......................................................................................................................................................................32

8.5 Exploratory testing ...........................................................................................................................................................................33

9 White-box testing of neural networks.........................................................................................................................................33

9.1 Structure of a neural network .................................................................................................................................................33

9.2 Test coverage measures for neural networks ............................................................................................................35

9.2.1 Introduction to test coverage levels .............................................................................................................35

9.2.2 Neuron coverage ...........................................................................................................................................................35

9.2.3 Threshold coverage ....................................................................................................................................................35

9.2.4 Sign change coverage ................................................................................................................................................36

9.2.5 Value change coverage .............................................................................................................................................36

9.2.6 Sign-sign coverage .......................................................................................................................................................36

9.2.7 Layer coverage ................................................................................................................................................................36

9.3 Test effectiveness of the white-box measures ...........................................................................................................36

9.4 White-box testing tools for neural networks .............................................................................................................37

10 Test environments for AI-based systems .................................................................................................................................37

10.1 Test environments for AI-based systems ......................................................................................................................37

10.2 Test scenario derivation ...............................................................................................................................................................38

10.3 Regulatory test scenarios and test environments .................................................................................................38

Annex A Machine learning .........................................................................................................................................................................................40

Bibliography .............................................................................................................................................................................................................................49

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

Any trade name used in this document is information given for the convenience of users and does not

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

A list of all parts in the ISO/IEC/IEEE 29119 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

The testing of traditional systems is well-understood, but AI-based systems, which are becoming more

prevalent and critical to our daily lives, introduce new challenges. This document has been created to

This document is primarily provided for those testers who are new to AI-based systems, but it can also

be useful for more experienced testers and other stakeholders working on the development and testing

As a Technical Report, this document contains data of a different kind from that normally published as

an International Standard or Technical Specification, such as data on the “state of the art”.

This document provides an introduction to AI-based systems. These systems are typically complex

(e.g. deep neural nets), are sometimes based on big data, can be poorly specified and can be non-

This document explains those characteristics which are specific to AI-based systems and explains the

corresponding difficulties of specifying the acceptance criteria for such systems.

This document presents the challenges of testing AI-based systems, the main challenge being the test

oracle problem, whereby testers find it difficult to determine expected results for testing and therefore

whether tests have passed or failed. It covers testing of these systems across the life cycle and gives

guidelines on how AI-based systems in general can be tested using black-box approaches and introduces

white-box testing specifically for neural networks. It describes options for the test environments and

In this document an AI-based system is a system that includes at least one AI component.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

statistical testing approach that allows testers to determine which of two systems or components

output of an activation function (3.1.3) of a node in a neural network

formula associated with a node in a neural network that determines the

ability of a system to react to changes in its environment in order to continue meeting both functional

deliberate use of adversarial examples (3.1.6) to cause a ML model (3.1.46) to fail

Note 1 to entry: Typically targets ML models in the form of a neural network (3.1.49).

input to an ML model (3.1.46) created by applying small perturbations to a working example that results

Note 1 to entry: Typically applies to ML models in the form of a neural network (3.1.49).

testing approach based on the attempted creation and execution of adversarial examples (3.1.6) to

Note 1 to entry: Typically applied to ML models in the form of a neural network (3.1.49).

situation when a previously labelled AI (3.1.12) system is no longer considered to be AI as technology

algorithm used to create an ML model (3.1.46) from the training data

EXAMPLE ML algorithms include linear regression (3.1.62), logistic regression, decision tree (3.1.25), SVM,

capability of an engineered system to acquire, process and apply knowledge and skills

approach to testing whereby an alternative version of the system is used as a pseudo-oracle (3.1.59) to

EXAMPLE The pseudo-oracle may be a system that already exists, a system developed by an independent

method used in artificial neural networks to determine the weights to be

used on the network connections based on the computed error at the output of the network

collection of benchmarks, where a benchmark is a set of tests used to compare the performance of

measure of the distance between the predicted value provided by the ML

machine learning function that predicts the output class for a given input

grouping of a set of objects such that objects in the same group (i.e. a cluster) are more similar to each

black-box test design technique in which test cases are designed to execute specific combinations of

EXAMPLE Pairwise testing (3.1.53), all combinations testing, each choice testing, base choice testing.

table used to describe the performance of a classifier (3.1.20) on a set of test data (3.1.75) for which the

part of the ML workflow that transforms raw data into a state ready for use

Note 1 to entry: Pre-processing can include analysis, normalization, filtering, reformatting, imputation, removal

supervised-learning model (3.1.46) for which inference can be represented

approach to creating rich hierarchical representations through the training of neural networks (3.1.49)

Note 1 to entry: Deep learning uses multi-layered networks of simple computing units (or “neurons”). In these

neural networks each unit combines a set of input values to produce an output value, which in turn is passed on

system which, given a particular set of inputs and starting state, will always produce the same set of

distance between the training data (3.1.80) distribution and the desired

Note 1 to entry: The effect of distributional shift often increases as the users’ interaction with the system (and so

changes to ML model (3.1.46) behaviour that occur over time

Note 1 to entry: These changes typically make predictions (3.1.57) less accurate and may require the model to be

level of understanding how the AI-based system (3.1.8) came up with a given result

experience-based testing in which the tester spontaneously designs and executes tests based on the

tester's existing relevant knowledge, prior exploration of the test item (including the results of previous

tests), and heuristic "rules of thumb" regarding common software behaviours and types of failure

Note 1 to entry: Exploratory testing hunts for hidden properties (including hidden behaviours) that, while quite

possibly benign by themselves, could interfere with other properties of the software under test, and so constitute