Software and systems engineering — Software testing — Part 1: General concepts

This document specifies general concepts in software testing and presents key concepts for the ISO/IEC/IEEE 29119 series.

Ingénierie du logiciel et des systèmes — Essais du logiciel — Partie 1: Concepts généraux

Le présent document définit les concepts généraux relatifs aux tests logiciels et présente les principaux concepts employés dans la série ISO/IEC/IEEE 29119.

General Information

Status
Published
Publication Date
03-Feb-2022
Current Stage
6060 - International Standard published
Start Date
10-Dec-2021
Completion Date
04-Feb-2022
Ref Project

RELATIONS

Buy Standard

Standard
ISO/IEC/IEEE 29119-1:2022 - Software and systems engineering — Software testing — Part 1: General concepts Released:2/4/2022
English language
47 pages
sale 15% off
Preview
sale 15% off
Preview
Standard
ISO/IEC/IEEE 29119-1 - Software and systems engineering -- Software testing
English language
55 pages
sale 15% off
Preview
sale 15% off
Preview
Standard
ISO/IEC/IEEE 29119-1:2022 - Software and systems engineering — Software testing — Part 1: General concepts Released:2/4/2022
French language
51 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/
STANDARD IEC/IEEE
29119-1
Second edition
2022-01
Software and systems engineering —
Software testing —
Part 1:
General concepts
Ingénierie du logiciel et des systèmes — Essais du logiciel —
Partie 1: Concepts généraux
Reference number
ISO/IEC/IEEE 29119-1:2022(E)
© ISO/IEC 2022
© IEEE 2022
---------------------- Page: 1 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2022
© IEEE 2022

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO or IEEE at the

respective address below or ISO’s member body in the country of the requester.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
CP 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Geneva NY 10016-5997, USA
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org Email: stds.ipr@ieee.org
Website: www.iso.org Website: www.ieee.org
Published in Switzerland
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction .............................................................................................................................................................................................................................vii

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ..................................................................................................................................................................................... 1

3 Terms and definitions .................................................................................................................................................................................... 1

4 Software testing concepts .......................................................................................................................................................................16

4.1 Introduction to software testing......................................................................................................................................... 16

4.1.1 Overview ................................................................................................................................................................................ 16

4.1.2 Relationship to quality management ............................................................................................................ 16

4.1.3 Verification and validation ..................................................................................................................................... 16

4.1.4 Test item .................................................................................................................................................................................. 17

4.1.5 Static and dynamic testing ..................................................................................................................................... 17

4.1.6 Exhaustive testing and sampling ...................................................................................................................... 18

4.1.7 Testing as a heuristic ................................................................................................................................................... 18

4.1.8 Purpose of testing .......................................................................................................................................................... 18

4.1.9 Test basis ................................................................................................................................................................................ 19

4.1.10 Test oracle .................. ......................................................................................................................... ................................... 19

4.1.11 Test independence .......................................................................................................................................................... 19

4.2 Test plans and test strategies ................................................................................................................................................. 19

4.2.1 General ..................................................................................................................................................................................... 19

4.2.2 Risks and risk management .................................................................................................................................. 20

4.2.3 Risks and requirements as the basis of a test strategy ................................................................20

4.2.4 Test approaches ................................................................................................................................................................ 21

4.2.5 Testing in development and maintenance life cycles ......................................................................22

4.2.6 Domains and system characteristics ............................................................................................................ 23

4.2.7 Test strategy contents ................................................................................................................................................ 23

4.3 Test frameworks ................................................................................................................................................................................ 24

4.3.1 Test processes .................................................................................................................................................................... 24

4.3.2 Test documentation ......................................................................................................................................................28

4.3.3 Documentation requirements .............................................................................................................................30

4.3.4 Configuration management and testing .....................................................................................................30

4.3.5 Tool support .........................................................................................................................................................................30

4.3.6 Process improvement and testing ...................................................................................................................30

4.3.7 Test metrics ........................................................................................................................................... ............................... 31

4.4 Test design and execution ......................................................................................................................................................... 31

4.4.1 Test model ............................................................................................................................................................................. 31

4.4.2 Model-based testing ..................................................................................................................................................... 32

4.4.3 Scripted and exploratory testing ...................................................................................................................... 33

4.4.4 Test design techniques ............................................................................................................................................... 33

4.4.5 Experience-based testing ........................................................................................................................................34

4.4.6 Retesting and regression testing ...................................................................................................................... 35

4.4.7 Manual and automated testing ........................................................................................................................... 35

4.4.8 Continuous testing ......................................................................................................................................................... 35

4.4.9 Back-to-back testing ..................................................................................................................................................... 35

4.4.10 A/B testing ............................................................................................................................................................................ 36

4.4.11 Mathematical-based and fuzz testing...........................................................................................................36

4.4.12 Test environments .........................................................................................................................................................36

4.4.13 Test data management ............................................................................................................................................... 37

4.5 Project management and testing ........................................................................................................................................ 37

4.6 Communication and reporting .............................................................................................................................................. 37

4.7 Defects and incident management .....................................................................................................................................38

Annex A (informative) System characteristics and testing – examples ......................................................................39

iii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)

Annex B (informative) Testing roles .................................................................................................................................................................46

Bibliography .............................................................................................................................................................................................................................47

IEEE Notices and Abstract..........................................................................................................................................................................................48

© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed

for the different types of ISO/IEC documents should be noted. This document was drafted in

accordance with the rules given in the ISO/IEC Directives, Part 2 (see www.iso.org/directives or

www.iec.ch/members_experts/refdocs).

IEEE Standards documents are developed within the IEEE Societies and the Standards Coordinating

Committees of the IEEE Standards Association (IEEE-SA) Standards Board. The IEEE develops its

standards through a consensus development process, approved by the American National Standards

Institute, which brings together volunteers representing varied viewpoints and interests to achieve the

final product. Volunteers are not necessarily members of the Institute and serve without compensation.

While the IEEE administers the process and establishes rules to promote fairness in the consensus

development process, the IEEE does not independently evaluate, test, or verify the accuracy of any of

the information contained in its standards.

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC

list of patent declarations received (see https://patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to

the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see

www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

ISO/IEC/IEEE 29119-1 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information

technology, Subcommittee SC 7, Software and systems engineering, in cooperation with the Systems and

Software Engineering Standards Committee of the IEEE Computer Society, under the Partner Standards

Development Organization cooperation agreement between ISO and IEEE.

This second edition cancels and replaces the first edition (ISO/IEC/IEEE 29119-1:2013), which has been

technically revised.
The main changes are as follows:

— Testing terms and their definitions that are not covered within this document have been removed.

This has led to this document being renamed from ‘Concepts and definitions’ to ‘General concepts’.

— The coverage of test concepts has been made more concise and re-ordered.

— The concept of test sub-processes has been removed due to its complexity and replaced with

additional coverage of the instantiation of test processes.
— The expected content of a test strategy has been clarified.
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)

— A simplified test design process is described, with the derivation of test cases now based on test

models rather than on test conditions.

— The coverage of metrics and measures has been moved from an annex into the body of the document.

— The annex explaining how testing fits into different life cycle models has been removed.

— A new annex providing examples of how systems from different domains are associated with certain

characteristics and test approaches has been added.

A list of all parts in the ISO/IEC/IEEE 29119 series can be found on the ISO and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards

body. A complete listing of these bodies can be found at www.iso.org/members.html and

www.iec.ch/national-committees.
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
Introduction

The purpose of the ISO/IEC/IEEE 29119 series is to define an internationally agreed set of standards for

software testing that can be used by any organization when performing any form of software testing

and using any life cycle.

It is recognized that there are many different types of software, software organizations, and

methodologies. Software domains include information technology (IT), personal computers (PC),

embedded, mobile, scientific and many other classifications. Software organizations range from

small to large, co-located to world-wide, and commercial to those providing a public service. Software

development methodologies include object-oriented, traditional, agile and DevOps. These and other

factors influence software testing. The ISO/IEC/IEEE 29119 series can support testing in many different

contexts.

This document facilitates the use of other parts in the ISO/IEC/IEEE 29119 series by introducing the

general concepts on which the ISO/IEC/IEEE 29119 series is built.

A general introduction to software testing is provided. The role of software testing in quality

management and as part of verification and validation is described; and its implementation in the form

of both static and dynamic testing is defined. The impracticality of exhaustive testing and the need for

sampling are explained; and the importance of the test basis and test oracle are described. The benefits

of test independence are introduced.

Test plans and test strategies are described in the context of risk-based testing, which is the

recommended approach to strategizing and managing testing that underlies the ISO/IEC/IEEE 29119

series and provides the basis for test prioritization and focus. Test levels, test types and test design

techniques (and corresponding measures) are described in the context of their inclusion as part of the

test strategy.

Various test frameworks are presented, including test processes (and test process improvement), test

metrics, test documentation, configuration management and tool support.

The performance of test design and execution based on the use of a test model is described. Several of

the most important test design and execution choices are considered, including scripted and exploratory

testing approaches, the importance of test design techniques for the creation of test cases, test patterns,

retesting and regression testing, manual and automated testing, back-to-back and A/B testing.

Several activities that directly support test design and executions are introduced, including test

environments, test data management, communications and reporting and defect and incident

management.

Annex A briefly describes a number of system characteristics and suggested associated test approaches.

If a tester can identify which of the system characteristics apply to the system they are testing, then

they should consider whether the specialized testing listed for the characteristic is appropriate for

inclusion in their test strategy.

Annex B introduces several generic testing roles and briefly describes their responsibilities.

The test process model that the ISO/IEC/IEEE 29119 series is based on is defined in detail in

ISO/IEC/IEEE 29119-2. ISO/IEC/IEEE 29119-2 covers the software testing processes at the

organizational level, test management level and for dynamic test levels. Testing is the primary approach

to risk treatment in software development. This document defines a risk-based approach to testing.

Templates and examples of test documentation that are produced during the testing process are defined

in ISO/IEC/IEEE 29119-3. Software testing techniques that can be used during testing are defined in

ISO/IEC/IEEE 29119-4.

While this document is informative, ISO/IEC/IEEE 29119-2, ISO/IEC/IEEE 29119-3 and

ISO/IEC/IEEE 29119-4 are normative, meaning that they include requirements for anyone wanting to

claim conformance to these standards. Users who want to use the standards but have good reasons

vii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)

for not following every requirement (e.g. for someone following an agile approach to development and

testing) can claim tailored conformance as long as the level of tailoring and its rationale are described

and agreed. Specific details of conformance are provided in the relevant conformance clause in each of

the standards.

The ISO/IEC/IEEE 29119 series can be used in isolation or can be used as part of a larger set of standards

that cover other aspects of the software life cycle. For instance, some users use ISO/IEC/IEEE 12207

to define software system life cycle models appropriate to their products and services (and some

may use the corresponding systems engineering standard, ISO/IEC/IEEE 15288), and reference the

ISO/IEC/IEEE 29119 series for their software testing needs.

Together, the ISO/IEC/IEEE 29119 series aims to provide stakeholders with the ability to manage and

perform software testing in any organization.
viii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 8 ----------------------
INTERNATIONAL STANDARD ISO/IEC/IEEE 29119-1:2022(E)
Software and systems engineering — Software testing —
Part 1:
General concepts
1 Scope

This document specifies general concepts in software testing and presents key concepts for the

ISO/IEC/IEEE 29119 series.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO, IEC and IEEE maintain terminology databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
— IEEE Standards Dictionary Online: available at https:// dictionary .ieee .org

NOTE For additional terms and definitions in the field of systems and software engineering, see

ISO/IEC/IEEE 24765, which is published periodically as a “snapshot” of the SEVOCAB (Systems and software

Engineering Vocabulary) and is publicly accessible at https:// www .computer .org/ sevocab.

3.1
A/B testing
split-run testing

statistical testing (3.131) approach that allows testers to determine which of two systems or components

performs better
3.2
accessibility testing

type of usability testing (3.131) used to measure the degree to which a test item (3.107) can be operated

by users with the widest possible range of characteristics and abilities
3.3
activation function
transfer function

formula associated with a node in a neural network (3.50) that determines

the output of the node (activation value (3.4)) from the inputs to the neuron
3.4
activation value

output of an activation function (3.3) of a node in a neural network (3.50)

© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
3.5
actual result

set of behaviours or conditions of a test item (3.107), or set of conditions of associated data or the test

environment (3.95), observed as a result of test execution (3.99)

EXAMPLE Outputs to screen, outputs to hardware, changes to data, reports, and communication messages

sent.
3.6
AI-based system
system including one or more components implementing AI (3.7)
3.7
artificial intelligence

capability of an engineered system to acquire, process and apply knowledge and skills

3.8
autonomous system

system capable of working without human intervention for sustained

periods
3.9
autonomy

ability of a system to work for sustained periods without human

intervention
3.10
back-to-back testing
differential testing

approach to testing (3.131) whereby an alternative version of the system is used to generate expected

results (3.35) for comparison from the same test inputs

EXAMPLE The alternative version can be a system that already exists, a system developed by an independent

team or a system implemented using a different programming language.
3.11
base choice
base value

input parameter value that is normally selected based on being a representative or typical value for the

parameter
3.12
boundary value analysis

specification-based test design technique (3.94) based on exercising the boundaries of equivalence

partitions (3.30)
3.13
branch testing

structure-based test case (3.85) design technique based on exercising branches in the control flow

(3.22) of the test item (3.107)
3.14
cause-effect graph

graphical representation of decision rules (3.25) between causes (inputs described as Boolean conditions

(3.21)) and effects (outputs described as Boolean expressions)
3.15
cause-effect graphing

specification-based test design technique (3.94) based on exercising decision rules (3.25) in a cause-effect

graph (3.14)
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
3.16
classification

machine learning (3.44) function that predicts the output class for a given

input
3.17
classification tree

hierarchical tree model of the input data to a program in which the inputs are represented by distinct

classifications (relevant test aspects) and classes (input values)
3.18
combinatorial testing
combinatorial test design techniques

class of specification-based test design techniques (3.94) based on exercising combinations of P-V pairs

(3.56)
EXAMPLE Pairwise testing (3.57), base choice (3.11) testing.
3.19
compatibility testing

type of testing (3.131) that measures the degree to which a test item (3.107) can function satisfactorily

alongside other independent products in a shared environment (co-existence), and where necessary,

exchanges information with other systems or components (interoperability)
3.20
completion criteria
conditions under which the testing (3.131) activities are considered complete
3.21
condition
Boolean expression containing no Boolean operators
EXAMPLE “A < B” is a condition but “A and B” is not.
3.22
control flow

sequence in which operations are performed during the execution of a test item (3.107)

3.23
decision

type of statement in which a choice between two or more possible outcomes controls which set of

actions will result

Note 1 to entry: Typical decisions are simple selections (e.g. if-then-else), to decide when to exit loops (e.g. while-

loop), and in case (switch) statements (e.g. case-1–2-3-...-N).
3.24
decision outcome
result of a decision (3.23) that determines the branch to be executed
3.25
decision rule

combination of conditions (3.21) (also known as causes) and actions (also known as effects) that produce

a specific outcome in decision table testing (3.27) and cause-effect graphing (3.15)

3.26
decision table

tabular representation of decision rules (3.25) between causes (inputs described as Boolean conditions

(3.21)) and effects (outputs described as Boolean expressions)
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC/IEEE 29119-1:2022(E)
3.27
decision table testing

specification-based test design technique (3.94) based on exercising decision rules (3.25) in a decision

table (3.26)
3.28
decision testing

structure-based test case (3.85) design technique based on exercising decision outcomes (3.24) in the

control flow (3.22) of the test item (3.107)
3.29
dynamic testing
testing (3.131) in which a test item (3.107) is evaluated by executing it
3.30
equivalence partition
equivalence class

class of inputs or outputs that are expected to be treated similarly by the test item (3.107)

3.31
equivalence partitioning

test design technique (3.94) in which test cases (3.85) are designed to exercise equivalence partitions

(3.30) by using one or more representative members of each partition
3.32
error guessing

test design technique (3.94) in which test cases (3.85) are derived on the basis of the tester’s knowledge

of past failures, or general knowledge of failure modes

Note 1 to entry: The relevant knowledge can be gained from personal experience, or can be encapsulated in, for

example, a defects database or a “bug taxonomy”.
3.33
executable statement

statement which, when compiled, is translated into object code, which will be executed procedurally

when the test item (3.107) is running and may perform an action on program data
3.34
exhaustive testing

test approach (3.83) in which all combinations of input values and preconditions are tested

Note 1 to entry: In nearly all non-trivial situations, exhaustive testing is impossible, due to the large number of

possible tests.
3.35
expected result

observable predicted behaviour of the test item (3.107) under specified conditions based on its

specification or another source
3.36
experience-based testing

class of test case (3.85) design techniques based on using the experience of testers to generate test

cases
EXAMPLE Error guessing (3.32).
Note 1
...

INTERNATIONAL ISO/IEC/
STANDARD IEEE
29119-1
Second edition
Software and systems engineering —
Software testing —
Part 1:
General concepts
Ingénierie du logiciel et des systèmes — Tests logiciels —
Partie 1: Concepts généraux
PROOF/ÉPREUVE
Reference number
ISO/IEC/IEEE 29119-1:2021(E)
ISO/IEC 2021
IEEE 2021
---------------------- Page: 1 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
© IEEE 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO or IEEE at the

respective address below or ISO’s member body in the country of the requester.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
CP 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Geneva NY 10016-5997, USA
Phone: +41 22 749 01 11
Email: copyright@iso.org Email: stds.ipr@ieee.org
Website: www.iso.org Website: www.ieee.org
Published in Switzerland
© ISO/IEC 2021 – All rights reserved
ii PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ..............................................................................................................................................................................................................................vii

1 Scope ................................................................................................................................................................................................................................. 9

2 Normative references ...................................................................................................................................................................................... 9

3 Terms and definitions ..................................................................................................................................................................................... 9

4 Software testing concepts ........................................................................................................................................................................24

4.1 Introduction to software testing ...........................................................................................................................................24

4.1.1 Overview ..............................................................................................................................................................................24

4.1.2 Relationship to quality management ..........................................................................................................24

4.1.3 Verification and validation ...................................................................................................................................24

4.1.4 Test item ...............................................................................................................................................................................25

4.1.5 Static and dynamic testing ...................................................................................................................................25

4.1.6 Exhaustive testing and sampling ....................................................................................................................26

4.1.7 Testing as a heuristic .................................................................................................................................................26

4.1.8 Purpose of testing ........................................................................................................................................................26

4.1.9 Test basis .............................................................................................................................................................................27

4.1.10 Test oracle ...........................................................................................................................................................................27

4.1.11 Test independence.......................................................................................................................................................27

4.2 Test plans and test strategies ..................................................................................................................................................27

4.2.1 General...................................................................................................................................................................................27

4.2.2 Risks and risk management ................................................................................................................................28

4.2.3 Risks and requirements as the basis of a test strategy ................................................................28

4.2.4 Test approaches .............................................................................................................................................................29

4.2.5 Testing in development and maintenance life cycles ....................................................................30

4.2.6 Domains and system characteristics ...........................................................................................................31

4.2.7 Test strategy contents ..............................................................................................................................................31

4.3 Testing frameworks .........................................................................................................................................................................32

4.3.1 Test processes .................................................................................................................................................................32

4.3.2 Test documentation ....................................................................................................................................................36

4.3.3 Documentation requirements ...........................................................................................................................38

4.3.4 Configuration management and testing ...................................................................................................38

4.3.5 Tool support ......................................................................................................................................................................38

4.3.6 Process improvement and testing .................................................................................................................38

4.3.7 Test metrics .......................................................................................................................................................................39

4.4 Test design and execution ..........................................................................................................................................................39

4.4.1 Test model ..........................................................................................................................................................................39

4.4.2 Model-based testing ...................................................................................................................................................40

4.4.3 Scripted and exploratory testing ....................................................................................................................41

4.4.4 Test design techniques.............................................................................................................................................41

4.4.5 Experience-based testing ......................................................................................................................................42

4.4.6 Retesting and regression testing .....................................................................................................................43

4.4.7 Manual and automated testing .........................................................................................................................43

4.4.8 Continuous testing ......................................................................................................................................................43

4.4.9 Back-to-back testing ..................................................................................................................................................43

4.4.10 A/B testing .........................................................................................................................................................................44

4.4.11 Mathematical-based and fuzz testing .........................................................................................................44

4.4.12 Test environments .......................................................................................................................................................44

4.4.13 Test data management .............................................................................................................................................45

4.5 Project management and testing ..........................................................................................................................................45

4.6 Communication and reporting ...............................................................................................................................................45

4.7 Defects and incident management......................................................................................................................................46

Annex A (informative) System characteristics and testing – examples .........................................................................47

© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE iii
---------------------- Page: 3 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)

Annex B (informative) Testing roles ..................................................................................................................................................................54

Bibliography .............................................................................................................................................................................................................................55

IEEE Notices and Abstract ...........................................................................................................................................................................................56

© ISO/IEC 2021 – All rights reserved
iv PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO/IEC documents should be noted. This document was drafted in accordance with

the rules given in the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members

_experts/ refdocs).

IEEE Standards documents are developed within the IEEE Societies and the Standards Coordinating

Committees of the IEEE Standards Association (IEEE-SA) Standards Board. The IEEE develops its

standards through a consensus development process, approved by the American National Standards

Institute, which brings together volunteers representing varied viewpoints and interests to achieve the

final product. Volunteers are not necessarily members of the Institute and serve without compensation.

While the IEEE administers the process and establishes rules to promote fairness in the consensus

development process, the IEEE does not independently evaluate, test, or verify the accuracy of any of

the information contained in its standards.

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see https:// patents .iec .ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/

iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.

ISO/IEC/IEEE 29119-1 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information

technology, Subcommittee SC 7, Software and systems engineering, in cooperation with the Systems and

Software Engineering Standards Committee of the IEEE Computer Society, under the Partner Standards

Development Organization cooperation agreement between ISO and IEEE.

This second edition cancels and replaces the first edition (ISO/IEC/IEEE 29119-1:2013), which has been

technically revised.
The main changes compared to the previous edition are as follows:

— Testing terms and their definitions that are not covered within this document have been removed.

This has led to this document being renamed from ‘Concepts and definitions’ to ‘General concepts’.

— The coverage of test concepts has been made more concise and re-ordered.

— The concept of test sub-processes has been removed due to its complexity and replaced with

additional coverage of the instantiation of test processes.
— The expected content of a test strategy has been clarified.
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE v
---------------------- Page: 5 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)

— A simplified test design process is described, with the derivation of test cases now based on test

models rather than on test conditions.

— The coverage of metrics and measures has been moved from an annex into the body of the document.

— The annex explaining how testing fits into different life cycle models has been removed.

— A new annex providing examples of how systems from different domains are associated with certain

characteristics and test approaches has been added.

A list of all parts in the ISO/IEC/IEEE 29119 series can be found on the ISO and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national

-committees.
© ISO/IEC 2021 – All rights reserved
vi PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
Introduction

The purpose of the ISO/IEC/IEEE 29119 series is to define an internationally agreed set of standards for

software testing that can be used by any organization when performing any form of software testing

and using any life cycle.

It is recognized that there are many different types of software, software organizations, and

methodologies. Software domains include information technology (IT), personal computers (PC),

embedded, mobile, scientific and many other classifications. Software organizations range from

small to large, co-located to world-wide, and commercial to those providing a public service. Software

development methodologies include object-oriented, traditional, agile and DevOps. These and other

factors influence software testing. The ISO/IEC/IEEE 29119 series can support testing in many different

contexts.

This document facilitates the use of other parts in the ISO/IEC/IEEE 29119 series by introducing the

general concepts on which the ISO/IEC/IEEE 29119 series is built.

A general introduction to software testing is provided. The role of software testing in quality

management and as part of verification and validation is described; and its implementation in the form

of both static and dynamic testing is defined. The impracticality of exhaustive testing and the need for

sampling are explained; and the importance of the test basis and test oracle are described. The benefits

of test independence are introduced.

Test plans and test strategies are described in the context of risk-based testing, which is the

recommended approach to strategizing and managing testing that underlies the ISO/IEC/IEEE 29119

series and provides the basis for test prioritization and focus. Test levels, test types and test design

techniques (and corresponding measures) are described in the context of their inclusion as part of the

test strategy.

Various test frameworks are presented, including test processes (and test process improvement), test

metrics, test documentation, configuration management and tool support.

The performance of test design and execution based on the use of a test model is described. Several of

the most important test design and execution choices are considered, including scripted and exploratory

testing approaches, the importance of test design techniques for the creation of test cases, test patterns,

retesting and regression testing, manual and automated testing, back-to-back and A/B testing.

Several activities that directly support test design and executions are introduced, including test

environments, test data management, communications and reporting and defect and incident

management.

Annex A briefly describes a number of system characteristics and suggested associated test approaches.

If a tester can identify which of the system characteristics apply to the system they are testing, then

they should consider whether the specialized testing listed for the characteristic is appropriate for

inclusion in their test strategy.

Annex B introduces several generic testing roles and briefly describes their responsibilities.

The test process model that the ISO/IEC/IEEE 29119 series is based on is defined in detail in

ISO/IEC/IEEE 29119-2. ISO/IEC/IEEE 29119-2 covers the software testing processes at the

organizational level, test management level and for dynamic test levels. Testing is the primary approach

to risk treatment in software development. This document defines a risk-based approach to testing.

Templates and examples of test documentation that are produced during the testing process are defined

in ISO/IEC/IEEE 29119-3. Software testing techniques that can be used during testing are defined in

ISO/IEC/IEEE 29119-4.

While this document is informative, ISO/IEC/IEEE 29119-2, ISO/IEC/IEEE 29119-3 and

ISO/IEC/IEEE 29119-4 are normative, meaning that they include requirements for anyone wanting to

claim conformance to these standards. Users who want to use the standards but have good reasons

© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE vii
---------------------- Page: 7 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)

for not following every requirement (e.g. for someone following an agile approach to development and

testing) can claim tailored conformance as long as the level of tailoring and its rationale are described

and agreed. Specific details of conformance are provided in the relevant conformance clause in each of

the standards.

The ISO/IEC/IEEE 29119 series can be used in isolation or can be used as part of a larger set of standards

that cover other aspects of the software life cycle. For instance, some users use ISO/IEC/IEEE 12207

to define software system life cycle models appropriate to their products and services (and some

may use the corresponding systems engineering standard, ISO/IEC/IEEE 15288), and reference the

ISO/IEC/IEEE 29119 series for their software testing needs.

Together, the ISO/IEC/IEEE 29119 series aims to provide stakeholders with the ability to manage and

perform software testing in any organization.
© ISO/IEC 2021 – All rights reserved
viii PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
---------------------- Page: 8 ----------------------
INTERNATIONAL STANDARD ISO/IEC/IEEE 29119-1:2021(E)
1 Scope

This document specifies general concepts in software testing and presents key concepts for the

ISO/IEC/IEEE 29119 series.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO, IEC and IEEE maintain terminology databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org

— IEEE Standards Dictionary Online: available at https:// ieeexplore .ieee .org/ xpls/ dictionary .jsp

NOTE For additional terms and definitions in the field of systems and software engineering, see

ISO/IEC/IEEE 24765, which is published periodically as a “snapshot” of the SEVOCAB (Systems and software

Engineering Vocabulary) and is publicly accessible at www .computer .org/ sevocab. ISO/IEC/IEEE 29119-1

includes all terms that are used in ISO/IEC/IEEE 29119 (all parts).
3.1
A/B testing
split-run testing

statistical testing (3.131) approach that allows testers to determine which of two systems or components

performs better
3.2
accessibility testing

type of usability testing (3.131) used to measure the degree to which a test item (3.107) can be operated

by users with the widest possible range of characteristics and abilities
3.3
activation function
transfer function

formula associated with a node in a neural network (3.50) that determines

the output of the node (activation value (3.4)) from the inputs to the neuron
3.4
activation value

output of an activation function (3.3) of a node in a neural network (3.50)

3.5
actual result

set of behaviours or conditions of a test item (3.107), or set of conditions of associated data or the test

environment (3.95), observed as a result of test execution (3.99)

EXAMPLE Outputs to screen, outputs to hardware, changes to data, reports, and communication messages

sent.
3.6
AI-based system
system including one or more components implementing AI (3.7)
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 9
---------------------- Page: 9 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
3.7
artificial intelligence

capability of an engineered system to acquire, process and apply knowledge and skills

3.8
autonomous system

system capable of working without human intervention for sustained

periods
3.9
autonomy

ability of a system to work for sustained periods without human

intervention
3.10
back-to-back testing
differential testing

approach to testing (3.131) whereby an alternative version of the system is used to generate expected

results (3.35) for comparison from the same test inputs

EXAMPLE The alternative version can be a system that already exists, a system developed by an independent

team or a system implemented using a different programming language.
3.11
base choice
base value

input parameter value that is normally selected based on being a representative or typical value for the

parameter
3.12
boundary value analysis

specification-based test design technique (3.94) based on exercising the boundaries of equivalence

partitions (3.30)
3.13
branch testing

structure-based test case (3.85) design technique based on exercising branches in the control flow

(3.22) of the test item (3.107)
3.14
cause-effect graph

graphical representation of decision rules (3.25) between causes (inputs described as Boolean conditions

(3.21)) and effects (outputs described as Boolean expressions)
3.15
cause-effect graphing

specification-based test design technique (3.94) based on exercising decision rules (3.25) in a cause-effect

graph (3.14)
3.16
classification

machine learning (3.44) function that predicts the output class for a given

input
3.17
classification tree

hierarchical tree model of the input data to a program in which the inputs are represented by distinct

classifications (relevant test aspects) and classes (input values)
© ISO/IEC 2021 – All rights reserved
10 PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
3.18
combinatorial testing
combinatorial test design techniques

class of specification-based test design techniques (3.94) based on exercising combinations of P-V pairs

(3.56)
EXAMPLE Pairwise testing (3.57), base choice (3.11) testing.
3.19
compatibility testing

type of testing (3.131) that measures the degree to which a test item (3.107) can function satisfactorily

alongside other independent products in a shared environment (co-existence), and where necessary,

exchanges information with other systems or components (interoperability)
3.20
completion criteria
conditions under which the testing (3.131) activities are considered complete
3.21
condition
Boolean expression containing no Boolean operators
EXAMPLE “A < B” is a condition but “A and B” is not.
3.22
control flow

sequence in which operations are performed during the execution of a test item (3.107)

3.23
decision

type of statement in which a choice between two or more possible outcomes controls which set of

actions will result

Note 1 to entry: Typical decisions are simple selections (e.g. if-then-else), to decide when to exit loops (e.g. while-

loop), and in case (switch) statements (e.g. case-1–2-3-...-N).
3.24
decision outcome
result of a decision (3.23) that determines the branch to be executed
3.25
decision rule

combination of conditions (3.21) (also known as causes) and actions (also known as effects) that produce

a specific outcome in decision table testing (3.27) and cause-effect graphing (3.15)

3.26
decision table

tabular representation of decision rules (3.25) between causes (inputs described as Boolean conditions

(3.21)) and effects (outputs described as Boolean expressions)
3.27
decision table testing

specification-based test design technique (3.94) based on exercising decision rules (3.25) in a decision

table (3.26)
3.28
decision testing

structure-based test case (3.85) design technique based on exercising decision outcomes (3.24) in the

control flow (3.22) of the test item (3.107)
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 11
---------------------- Page: 11 ----------------------
ISO/IEC/IEEE 29119-1:2021(E)
3.29
dynamic testing
testing (3.131) in which a test item (3.107) is evaluated by executing it
3.30
equivalence partition
equivalence class

class of inputs or outputs that are expected to be treated similarly by the test item (3.107)

3.31
equivalence partitioning

test design technique (3.94) in which test cases (3.85) are designed to exercise equivalence partitions

(3.30) by using one or more representative members of each partition
3.32
error guessing

test design technique (3.94) in which test cases (3.85) are derived on the basis of the tester’s knowledge

of past failures, or general knowledge of failure modes

Note 1 to entry: The relevant knowledge can be gained from personal experience, or can be encapsulated in, for

example, a defects database or a “bug taxonomy”.
3.33
executable statement

statement which, when compiled, is translated into object code, which will be executed procedurally

when the test item (3.107) is running and may perform an action on program data
3.34
exhaustive testing

test approach (3.83) in which all combinations of input values and preconditions are tested

Note 1 to entry: In nearly all non-trivial situations, exhaustive testing is impossible, due to the large number of

possible tests.
3.35
expected result

observable predicted behaviour of the test item (3.107) under specified conditions based on its

specification or another source
3.36
experience-based testing
class of test case (3.85) design techniques based on using the
...

NORME ISO/
INTERNATIONALE IEC/IEEE
29119-1
Deuxième édition
2022-01
Ingénierie du logiciel et des
systèmes — Essais du logiciel —
Partie 1:
Concepts généraux
Software and systems engineering — Software testing —
Part 1: General concepts
Numéro de référence
ISO/IEC/IEEE 29119-1:2022(F)
© ISO/IEC 2022
© IEEE 2022
---------------------- Page: 1 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
DOCUMENT PROTÉGÉ PAR COPYRIGHT
© ISO/IEC 2022
© IEEE 2022

Tous droits réservés. Sauf prescription différente ou nécessité dans le contexte de la mise en œuvre, aucune partie de cette

publication ne peut être reproduite ni utilisée sous quelque forme que ce soit et par aucun procédé, électronique ou mécanique, y

compris la photocopie, ou la diffusion sur l’internet ou un intranet, sans autorisation écrite soit de l’ISO soit de l’IEEE, à l’une ou

l’autre des adresses ci-après.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
Case postale 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Genève NY 10016-5997, USA
Tél.: +41 22 749 01 11
Fax: +41 22 749 09 47
E-mail: copyright@iso.org E-mail: stds.ipr@ieee.org
Web: www.iso.org Web: www.ieee.org
Publié en Suisse
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 2 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
Sommaire Page

Avant-propos ...............................................................................................................................................................................................................................v

Introduction .............................................................................................................................................................................................................................vii

1 Domaine d'application ...................................................................................................................................................................................1

2 Références normatives ..................................................................................................................................................................................1

3 Termes et définitions ...................................................................................................................................................................................... 1

4 Concepts des tests logiciels ...................................................................................................................................................................17

4.1 Introduction aux tests logiciels ............................................................................................................................................ 17

4.1.1 Vue d'ensemble.................................................................................................................................................................. 17

4.1.2 Relation avec le management de la qualité .............................................................................................. 17

4.1.3 Vérification et validation .......................................................................................................................................... 17

4.1.4 Élément de test .................................................................................................................................................................. 18

4.1.5 Tests statiques et dynamiques ............................................................................................................................ 18

4.1.6 Tests exhaustifs et échantillonnage ............................................................................................................... 19

4.1.7 Tests heuristiques .......................................................................................................................................................... 19

4.1.8 Finalité du test ................................................................................................................................................................... 19

4.1.9 Base de test........................................................................................................................................................................... 20

4.1.10 Oracle de test ...................................................................................................................................................................... 20

4.1.11 Indépendance des tests ............................................................................................................................................. 20

4.2 Plans de test et stratégies de test ....................................................................................................................................... 21

4.2.1 Généralités ............................................................................................................................................................................ 21

4.2.2 Risques et gestion du risque ................................................................................................................................. 21

4.2.3 Risques et exigences comme base d'une stratégie de test .........................................................22

4.2.4 Approches de test ........................................................................................................................................................... 22

4.2.5 Tests dans les cycles de vie du développement et de la maintenance ..............................23

4.2.6 Domaines et caractéristiques du système ................................................................................................ 24

4.2.7 Contenu de la stratégie de test ............................................................................................................................ 25

4.3 Frameworks de test.........................................................................................................................................................................26

4.3.1 Processus de test ............................................................................................................................................................. 26

4.3.2 Documentation des tests ..........................................................................................................................................30

4.3.3 Exigences relatives à la documentation ...................................................................................................... 32

4.3.4 Gestion de la configuration et tests ................................................................................................................ 32

4.3.5 Support outillé ...................................................................................................................................................................33

4.3.6 Amélioration des processus et tests .............................................................................................................. 33

4.3.7 Métriques de test.............................................................................................................................................................34

4.4 Conception et exécution des tests ......................................................................................................................................34

4.4.1 Modèle de test ....................................................................................................................................................................34

4.4.2 Tests basés sur un modèle ...................................................................................................................................... 35

4.4.3 Tests avec script et tests exploratoires .......................................................................................................36

4.4.4 Techniques de conception de tests ..................................................................................................................36

4.4.5 Tests basés sur l'expérience .................................................................................................................................. 37

4.4.6 Re-tests et tests de régression ............................................................................................................................38

4.4.7 Tests manuels et automatisés ..............................................................................................................................38

4.4.8 Tests en continu ...............................................................................................................................................................39

4.4.9 Tests dos-à-dos ........................................................................................................................................... .......................39

4.4.10 Tests A/B ................................................................................................................................................................................ 39

4.4.11 Tests mathématiques et tests à données aléatoires.........................................................................39

4.4.12 Environnements de test ............................................................................................................................................40

4.4.13 Gestion des données de test ..................................................................................................................................40

4.5 Gestion de projet et tests ............................................................................................................................................................40

4.6 Communication et génération de rapports ................................................................................................................ 41

4.7 Gestion des défauts et des incidents ................................................................................................................................ 41

Annexe A (informative) Caractéristiques du système et tests — Exemples ...........................................................42

iii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 3 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)

Annexe B (informative) Rôles de test ...............................................................................................................................................................50

Bibliographie ...........................................................................................................................................................................................................................51

Notes et résumés IEEE ...................................................................................................................................................................................................52

© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 4 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
Avant-propos

L'ISO (Organisation internationale de normalisation) et l’IEC (Commission électrotechnique

internationale) forment le système spécialisé de la normalisation mondiale. Les organismes

nationaux membres de l'ISO ou de l’IEC participent au développement de Normes internationales

par l'intermédiaire des comités techniques créés par l'organisation concernée afin de s'occuper des

domaines particuliers de l'activité technique. Les comités techniques de l'ISO et de l’IEC collaborent

dans des domaines d'intérêt commun. D'autres organisations internationales, gouvernementales et non

gouvernementales, en liaison avec l'ISO et l’IEC, participent également aux travaux.

Les procédures utilisées pour élaborer le présent document et celles destinées à sa mise à jour sont

décrites dans les Directives ISO/IEC, Partie 1. Il convient, en particulier de prendre note des différents

critères d'approbation requis pour les différents types de documents ISO/IEC. Le présent document a

été rédigé conformément aux règles de rédaction données dans les Directives ISO/IEC, Partie 2 (voir

www.iso.org/directives ou www.iec.ch/members_experts/refdocs).

Les documents normatifs de l’IEEE sont développés au sein des sociétés de l’IEEE et des Comités de

Coordination des Normes du Conseil des Normes de l’Association des normes IEEE (IEEE-SA). L’IEEE

élabore ses normes par le biais d’un processus d'élaboration du consensus approuvé par l’American

National Standards Institute, qui rassemble des volontaires représentant divers points de vue et divers

intérêts pour parvenir au produit final. Les volontaires ne sont pas nécessairement des membres

de l’Institut et aucune compensation ne leur est attribuée. Bien que l’IEEE administre le processus

et établisse des règles pour favoriser l’équité au cours du processus d'élaboration du consensus,

l’IEEE n’évalue pas, ne soumet pas à essai ou ne vérifie pas de manière indépendante l’exactitude des

informations contenues dans ses normes.

L'attention est attirée sur le fait que certains des éléments du présent document peuvent faire l'objet

de droits de propriété intellectuelle ou de droits analogues. L'ISO et l’IEC ne sauraient être tenues pour

responsables de ne pas avoir identifié de tels droits de propriété et averti de leur existence. Les détails

concernant les références aux droits de propriété intellectuelle ou autres droits analogues identifiés

lors de l'élaboration du document sont indiqués dans l'Introduction et/ou dans la liste des déclarations

de brevets reçues par l'ISO (voir www.iso.org/brevets) ou dans la liste des déclarations de brevets

reçues par l'IEC (voir https://patents.iec.ch).

Les appellations commerciales éventuellement mentionnées dans le présent document sont données

pour information, par souci de commodité, à l’intention des utilisateurs et ne sauraient constituer un

engagement.

Pour une explication de la nature volontaire des normes, la signification des termes et expressions

spécifiques de l'ISO liés à l'évaluation de la conformité, ou pour toute information au sujet de

l'adhésion de l'ISO aux principes de l’Organisation mondiale du commerce (OMC) concernant les

obstacles techniques au commerce (OTC), voir www.iso.org/iso/avant-propos. Pour l'IEC, voir

www.iec.ch/understanding-standards.

ISO/IEC/IEEE 29119-1 a été élaborée par le comité technique mixte ISO/IEC JTC 1, Technologies de

l'information, sous-comité SC 7, Ingénierie du logiciel et des systèmes, en coopération avec le Systems and

Software Engineering Standards Committee de l'IEEE Computer Society, dans le cadre de l'accord de

coopération entre les Organisations Partenaires de Développement de Normes que sont l'ISO et l'IEEE.

Cette deuxième édition annule et remplace la première édition (ISO/IEC/IEEE 29119-1:2013), qui a fait

l'objet d'une révision technique.
Les principales modifications sont les suivantes:

— les termes relatifs aux tests et leurs définitions non couverts dans le présent document ont été

supprimés. Le présent document intitulé «Concepts et définitions» a donc été renommé «Concepts

généraux»;
— la couverture des concepts de test a été réduite et réorganisée;
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 5 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)

— le concept de sous-processus de test a été supprimé en raison de sa complexité et remplacé par la

couverture supplémentaire d'instanciation des processus de test;
— le contenu attendu de la stratégie de test a été clarifié;

— un processus de conception simplifié des tests est décrit, l'origine des cas de test étant désormais

basée sur les modèles de test plutôt que sur les conditions de test;

— la couverture des indicateurs et des mesures, qui se trouvait dans une annexe, a été déplacée dans

le corps du document;

— l'annexe expliquant de quelle manière les tests s'adaptent aux différents modèles de cycle de vie a

été supprimée;

— une nouvelle annexe fournissant des exemples de la manière dont les systèmes issus de différents

domaines sont associés à certaines caractéristiques et approches de test a été ajoutée.

Une liste de toutes les parties de la série ISO/IEC/IEEE 29119 se trouve sur les sites Web de l'ISO et de

l'IEC.

Il convient que l’utilisateur adresse tout retour d’information ou toute question concernant le présent

document à l’organisme national de normalisation de son pays. Une liste exhaustive desdits organismes

se trouve à l’adresse www.iso.org/fr/members.html et www.iec.ch/national-committees.

© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 6 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
Introduction

L'objet de la série ISO/IEC/IEEE 29119 est de définir un ensemble de normes de tests logiciels

approuvées au plan national pouvant être utilisées par toute organisation qui entreprend toute forme

de test logiciel, quel que soit le cycle de vie.

Il est indéniable qu'il existe un grand nombre de logiciels, d'organisations logicielles et de types de

méthodologies différents. Les logiciels couvrent divers domaines, notamment les technologies de

l'information (TI), les ordinateurs personnels (PC), les technologies intégrées, mobiles et scientifiques,

ainsi que de nombreuses autres classifications. Les organisations logicielles vont des petites aux

grandes entreprises, des entreprises colocalisées aux multinationales, et des entreprises commerciales

aux services publics. Les méthodologies de développement logiciel sont tantôt orientées objets, tantôt

traditionnelles, tantôt en Agile, tantôt axées sur le DevOps. Tous ces facteurs, combinés à d'autres,

influencent les tests logiciels. La série ISO/IEC/IEEE 29119 peut faciliter la conduite de tests dans de

nombreux contextes différents.

Le présent document aide à l'utilisation des autres parties de la série ISO/IEC/IEEE 29119 en

introduisant les concepts généraux sur lesquels la série ISO/IEC/IEEE 29119 est bâtie.

Il fournit une introduction générale à la notion de tests logiciels. Il décrit le rôle des tests logiciels dans

le management de la qualité et dans le cadre des processus de vérification et de validation, et définit

leur mise en œuvre sous forme de tests statiques et dynamiques. Il explique, par ailleurs, la difficulté de

réaliser des tests exhaustifs et la nécessité de procéder par échantillonnage, et souligne l'importance de

la base de test et de l'oracle de test. Il décrit en outre les avantages de l'indépendance des tests.

Les plans de test et les stratégies de test sont décrits dans le contexte de tests basés sur les risques,

ce qui constitue l'approche recommandée pour l'élaboration d'une stratégie et la gestion des tests,

approche sous-jacente à la série ISO/IEC/IEEE 29119 et fournissant une base pour définir les priorités

et les objectifs de test. Les niveaux de test, les types de test et les techniques de conception de tests

(ainsi que les mesures correspondantes) sont décrits en tant que partie intégrante de la stratégie de

test.

Différents frameworks de test sont présentés, notamment les processus de test (et l'amélioration de ces

processus), les métriques de test, la documentation de test, la gestion de la configuration ainsi que le

support outillé.

Cette norme décrit également la conception et l'exécution des tests à partir d'un modèle de test. Elle

évalue plusieurs choix de conception et d'exécution de tests parmi les plus importants, notamment

les approches de tests basés sur des scripts et les approches de tests exploratoires, l'importance des

techniques de conception de tests pour la création de cas de test, les patterns de test, les re-tests et les

tests de régression, les tests manuels et automatisés, les tests dos-à-dos (back-to-back) et les tests A/B.

Elle présente plusieurs activités qui soutiennent directement la conception et l'exécution des tests,

notamment les environnements de test, la gestion des données de test, la communication et la

production de rapports, et la gestion des défauts et incidents.

L'Annexe A décrit brièvement certaines caractéristiques du système et suggère les approches de test

associées. Si un testeur peut identifier les caractéristiques du système qui s'appliquent au système qu'il

est en train de tester, il convient alors qu'il détermine s'il est utile d'inclure dans sa stratégie de test les

tests spécialisés répertoriés pour les caractéristiques en question.

L'Annexe B introduit plusieurs rôles de test génériques et en décrit brièvement les responsabilités.

Le modèle de processus de test sur lequel est fondée la série de normes ISO/IEC/IEEE 29119 est défini

en détail dans l'ISO/IEC/IEEE 29119-2. L'ISO/IEC/IEEE 29119-2 traite des processus de test logiciel au

niveau organisationnel, au niveau de la gestion des tests et au niveau des tests dynamiques. Le test

forme l'approche essentielle du traitement des risques dans le développement de logiciels. Le présent

document définit une approche des tests basée sur le risque.
vii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 7 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)

L'ISO/IEC/IEEE 29119-3 définit des modèles et des exemples de documentation de test produite au

cours du processus de test. Les techniques de test pouvant être utilisées au cours des tests sont définies

dans l'ISO/IEC/IEEE 29119-4.

Bien que le présent document soit informatif, l'ISO/IEC/IEEE 29119-2, l'ISO/IEC/IEEE 29119-3 et

l'ISO/IEC/IEEE 29119-4 sont normatives, ce qui signifie qu'elles stipulent des exigences obligatoires

pour quiconque souhaiterait revendiquer la conformité à ces normes. Les utilisateurs qui souhaitent

utiliser les normes, mais qui ont de bonnes raisons de ne pas en suivre chaque exigence (par exemple,

quelqu'un qui suit une approche de développement et de test en Agile) peuvent revendiquer une

conformité adaptée, à condition que le niveau d'adaptation et sa justification soient décrits et acceptés.

Les détails de conformité spécifiques sont donnés dans l'article de chaque norme applicable à la

conformité.

La série ISO/IEC/IEEE 29119 peut être utilisée de manière isolée ou dans le cadre d'un ensemble

plus vaste de normes qui couvrent d'autres aspects du cycle de vie du logiciel. Par exemple,

certains utilisateurs utiliseront l'ISO/IEC/IEEE 12207 pour définir des modèles de cycle de vie du

système logiciel qui s'appliquent à leurs produits et services (et d'autres peuvent utiliser la norme

correspondante ISO/IEC/IEEE 15288 relative à l'ingénierie des systèmes), et se référer à la série de

normes ISO/IEC/IEEE 29119 pour leurs besoins en matière de tests logiciels.

Collectivement, la série ISO/IEC/IEEE 29119 vise à donner aux parties prenantes la possibilité de gérer

et d'exécuter des tests logiciels dans n'importe quelle organisation.
viii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 8 ----------------------
NORME INTERNATIONALE ISO/IEC/IEEE 29119-1:2022(F)
Ingénierie du logiciel et des systèmes — Essais du
logiciel —
Partie 1:
Concepts généraux
1 Domaine d'application

Le présent document définit les concepts généraux relatifs aux tests logiciels et présente les principaux

concepts employés dans la série ISO/IEC/IEEE 29119.
2 Références normatives
Le présent document ne contient aucune référence normative.
3 Termes et définitions

Pour les besoins du présent document, les termes et définitions suivants s'appliquent.

L'ISO, l'IEC et l'IEEE tiennent à jour des bases de données terminologiques destinées à être utilisées en

normalisation, consultables aux adresses suivantes:

— ISO Online browsing platform: disponible à l’adresse https:// www .iso .org/ obp

— IEC Electropedia: disponible à l’adresse https:// www .electropedia .org/

— IEEE Standards Dictionary Online: disponible à l'adresse https:// dictionary .ieee .org

NOTE Pour d'autres termes et définitions relevant du domaine de l'ingénierie des systèmes et des logiciels,

voir l'ISO/IEC/IEEE 24765, qui est régulièrement publiée sous la forme d'un «instantané» du SEVOCAB (Systems

and software Engineering Vocabulary) et est accessible publiquement à l'adresse https:// www .computer .org/

sevocab.
3.1
test A/B
test dédoublé

approche de tests (3.131) statistique qui permet aux testeurs d'identifier parmi deux systèmes ou

composants celui qui présente les meilleures performances
3.2
test d'accessibilité

type de tests (3.131) d'utilisabilité servant à mesurer le degré selon lequel des utilisateurs présentant

la plus grande diversité possible de caractéristiques et d'aptitudes peuvent faire fonctionner un

élément de test (3.107)
3.3
fonction d'activation
fonction de transfert

formule associée à un nœud dans un réseau neuronal (3.50) qui détermine

la sortie du nœud (valeur d'activation [3.4]) à partir des entrées du neurone
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 9 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
3.4
valeur d'activation

sortie d'une fonction d'activation (3.3) d'un nœud dans un réseau neuronal

(3.50)
3.5
résultat réel

ensemble de comportements ou de conditions d'un élément de test (3.107), ou ensemble de conditions

des données associées ou de l'environnement de test (3.95), observés à la suite de l'exécution du test (3.99)

EXEMPLE Sorties sur un écran, sorties matérielles, modifications de données, rapports et messages de

communication envoyés.
3.6
système basé sur l'IA
système comprenant un ou plusieurs composants qui met en œuvre l'IA (3.7)
3.7
intelligence artificielle

capacité d'un système d'ingénierie à acquérir, traiter et appliquer des connaissances et des aptitudes

3.8
système autonome

système capable de fonctionner pendant des périodes prolongées sans

intervention humaine
3.9
autonomie

aptitude d'un système à fonctionner pendant des périodes prolongées

sans intervention humaine
3.10
test dos-à-dos
test différentiel

approche de tests (3.131) dans laquelle une version alternative du système est utilisée pour générer des

résultats attendus (3.35) en vue de les comparer avec les mêmes entrées utilisées pour le test

EXEMPLE La version alternative peut être un système qui existe déjà, un système développé par une équipe

indépendante ou un système mis en œuvre à l'aide d'un langage de programmation différent.

3.11
choix de base
valeur de base

valeur de paramètre d'entrée qui est normalement sélectionnée pour sa nature représentative ou

typique du paramètre en question
3.12
analyse des valeurs limites

technique de conception de tests (3.94) basée sur une spécification, qui consiste à exercer les valeurs aux

limites de partitions d'équivalence (3.30)
3.13
test de branches

technique de conception d'un cas de test (3.85) basée sur une structure, qui consiste à exercer des

branches du flux de contrôle (3.22) de l'élément de test (3.107)
3.14
graphe de causes à effets

représentation graphique des règles de décision (3.25) entre des causes (entrées décrites sous forme de

conditions [3.21] booléennes) et des effets (sorties décrites sous forme d'expressions booléennes)

© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
---------------------- Page: 10 ----------------------
ISO/IEC/IEEE 29119-1:2022(F)
3.15
mise en graphe de causes à effets

technique de conception de tests (3.94) basée sur une spécification, qui consiste à exercer des règles de

décision (3.25) dans un graphe de causes à effets (3.14)
3.16
classification

fonction d'apprentissage automatique (3.44) qui prédit la classe de sortie

d'une entrée donnée
3.17
classification arborescente

modèle d'arborescence hiérarchique des données d'entrée d'un programme dans lequel les sorties sont

représentées par des classifications (aspects pertinents pour les tests) et des classes (valeurs d'entrée)

distinctes
3.18
tests combinatoires
techniques de conception de tests combinatoires

catégorie de technique de conception de tests (3.94) basées sur une spécification, qui consistent à exercer

des combinaisons de paires P-V (3.56)
EXEMPLE Test par paires (3.57), tests par choix de base (3.11).
3.19
test de compatibilité

type de tests (3.131) qui évalue dans quelle mesure un élément de test (3.107) peut fonctionner de

manière satisfaisante parallèlement à d'autres produits indépendants dans un environnement partagé

(coexistence) et, si nécessaire, échanger des informations avec d'autres systèmes ou composants

(interopérabilité)
3.20
critères de clôture

conditions dans lesquelles les activités de tests (3.131) sont considérées comme clôturées

3.21
condition
expression booléenne ne contenant aucun opérateur booléen
EXEMPLE «A < B» est une condition, contrairement à «A and B».
3.22
flux de contrôle

ordre de déroulement des opérations au cours de l'exécution d'un élément de test (3.107)

3.23
décision

type d'instruction dans lequel un choix entre deux résultats possibles ou plus contrôle l'ensemble des

actions qui en résultera

Note 1 à l'article: Les décisions typiques prennent la forme de sélections simples (par exemple, if-then-else)

permettant de décider à quel moment sortir de
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.