ISO/IEC/IEEE 29119-1:2022
(Main)Software and systems engineering — Software testing — Part 1: General concepts
Software and systems engineering — Software testing — Part 1: General concepts
This document specifies general concepts in software testing and presents key concepts for the ISO/IEC/IEEE 29119 series.
Ingénierie du logiciel et des systèmes — Essais du logiciel — Partie 1: Concepts généraux
Le présent document définit les concepts généraux relatifs aux tests logiciels et présente les principaux concepts employés dans la série ISO/IEC/IEEE 29119.
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/
STANDARD IEC/IEEE
29119-1
Second edition
2022-01
Software and systems engineering —
Software testing —
Part 1:
General concepts
Ingénierie du logiciel et des systèmes — Essais du logiciel —
Partie 1: Concepts généraux
Reference number
© ISO/IEC 2022
© IEEE 2022
© ISO/IEC 2022
© IEEE 2022
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO or IEEE at the
respective address below or ISO’s member body in the country of the requester.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
CP 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Geneva NY 10016-5997, USA
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org Email: stds.ipr@ieee.org
Website: www.iso.org Website: www.ieee.org
Published in Switzerland
ii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Contents Page
Foreword .v
Introduction .vii
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Software testing concepts .16
4.1 Introduction to software testing. 16
4.1.1 Overview . 16
4.1.2 Relationship to quality management . 16
4.1.3 Verification and validation . 16
4.1.4 Test item . 17
4.1.5 Static and dynamic testing . 17
4.1.6 Exhaustive testing and sampling . 18
4.1.7 Testing as a heuristic . 18
4.1.8 Purpose of testing . 18
4.1.9 Test basis . 19
4.1.10 Test oracle . . . 19
4.1.11 Test independence . 19
4.2 Test plans and test strategies . 19
4.2.1 General . 19
4.2.2 Risks and risk management . 20
4.2.3 Risks and requirements as the basis of a test strategy .20
4.2.4 Test approaches . 21
4.2.5 Testing in development and maintenance life cycles .22
4.2.6 Domains and system characteristics . 23
4.2.7 Test strategy contents . 23
4.3 Test frameworks . 24
4.3.1 Test processes . 24
4.3.2 Test documentation .28
4.3.3 Documentation requirements .30
4.3.4 Configuration management and testing .30
4.3.5 Tool support .30
4.3.6 Process improvement and testing .30
4.3.7 Test metrics . . 31
4.4 Test design and execution . 31
4.4.1 Test model . 31
4.4.2 Model-based testing . 32
4.4.3 Scripted and exploratory testing . 33
4.4.4 Test design techniques . 33
4.4.5 Experience-based testing .34
4.4.6 Retesting and regression testing . 35
4.4.7 Manual and automated testing . 35
4.4.8 Continuous testing . 35
4.4.9 Back-to-back testing . 35
4.4.10 A/B testing . 36
4.4.11 Mathematical-based and fuzz testing.36
4.4.12 Test environments .36
4.4.13 Test data management . 37
4.5 Project management and testing . 37
4.6 Communication and reporting . 37
4.7 Defects and incident management .38
Annex A (informative) System characteristics and testing – examples .39
iii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Annex B (informative) Testing roles .46
Bibliography .47
IEEE Notices and Abstract.48
iv
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed
for the different types of ISO/IEC documents should be noted. This document was drafted in
accordance with the rules given in the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
IEEE Standards documents are developed within the IEEE Societies and the Standards Coordinating
Committees of the IEEE Standards Association (IEEE-SA) Standards Board. The IEEE develops its
standards through a consensus development process, approved by the American National Standards
Institute, which brings together volunteers representing varied viewpoints and interests to achieve the
final product. Volunteers are not necessarily members of the Institute and serve without compensation.
While the IEEE administers the process and establishes rules to promote fairness in the consensus
development process, the IEEE does not independently evaluate, test, or verify the accuracy of any of
the information contained in its standards.
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
ISO/IEC/IEEE 29119-1 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software and systems engineering, in cooperation with the Systems and
Software Engineering Standards Committee of the IEEE Computer Society, under the Partner Standards
Development Organization cooperation agreement between ISO and IEEE.
This second edition cancels and replaces the first edition (ISO/IEC/IEEE 29119-1:2013), which has been
technically revised.
The main changes are as follows:
— Testing terms and their definitions that are not covered within this document have been removed.
This has led to this document being renamed from ‘Concepts and definitions’ to ‘General concepts’.
— The coverage of test concepts has been made more concise and re-ordered.
— The concept of test sub-processes has been removed due to its complexity and replaced with
additional coverage of the instantiation of test processes.
— The expected content of a test strategy has been clarified.
v
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
— A simplified test design process is described, with the derivation of test cases now based on test
models rather than on test conditions.
— The coverage of metrics and measures has been moved from an annex into the body of the document.
— The annex explaining how testing fits into different life cycle models has been removed.
— A new annex providing examples of how systems from different domains are associated with certain
characteristics and test approaches has been added.
A list of all parts in the ISO/IEC/IEEE 29119 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
vi
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Introduction
The purpose of the ISO/IEC/IEEE 29119 series is to define an internationally agreed set of standards for
software testing that can be used by any organization when performing any form of software testing
and using any life cycle.
It is recognized that there are many different types of software, software organizations, and
methodologies. Software domains include information technology (IT), personal computers (PC),
embedded, mobile, scientific and many other classifications. Software organizations range from
small to large, co-located to world-wide, and commercial to those providing a public service. Software
development methodologies include object-oriented, traditional, agile and DevOps. These and other
factors influence software testing. The ISO/IEC/IEEE 29119 series can support testing in many different
contexts.
This document facilitates the use of other parts in the ISO/IEC/IEEE 29119 series by introducing the
general concepts on which the ISO/IEC/IEEE 29119 series is built.
A general introduction to software testing is provided. The role of software testing in quality
management and as part of verification and validation is described; and its implementation in the form
of both static and dynamic testing is defined. The impracticality of exhaustive testing and the need for
sampling are explained; and the importance of the test basis and test oracle are described. The benefits
of test independence are introduced.
Test plans and test strategies are described in the context of risk-based testing, which is the
recommended approach to strategizing and managing testing that underlies the ISO/IEC/IEEE 29119
series and provides the basis for test prioritization and focus. Test levels, test types and test design
techniques (and corresponding measures) are described in the context of their inclusion as part of the
test strategy.
Various test frameworks are presented, including test processes (and test process improvement), test
metrics, test documentation, configuration management and tool support.
The performance of test design and execution based on the use of a test model is described. Several of
the most important test design and execution choices are considered, including scripted and exploratory
testing approaches, the importance of test design techniques for the creation of test cases, test patterns,
retesting and regression testing, manual and automated testing, back-to-back and A/B testing.
Several activities that directly support test design and executions are introduced, including test
environments, test data management, communications and reporting and defect and incident
management.
Annex A briefly describes a number of system characteristics and suggested associated test approaches.
If a tester can identify which of the system characteristics apply to the system they are testing, then
they should consider whether the specialized testing listed for the characteristic is appropriate for
inclusion in their test strategy.
Annex B introduces several generic testing roles and briefly describes their responsibilities.
The test process model that the ISO/IEC/IEEE 29119 series is based on is defined in detail in
ISO/IEC/IEEE 29119-2. ISO/IEC/IEEE 29119-2 covers the software testing processes at the
organizational level, test management level and for dynamic test levels. Testing is the primary approach
to risk treatment in software development. This document defines a risk-based approach to testing.
Templates and examples of test documentation that are produced during the testing process are defined
in ISO/IEC/IEEE 29119-3. Software testing techniques that can be used during testing are defined in
ISO/IEC/IEEE 29119-4.
While this document is informative, ISO/IEC/IEEE 29119-2, ISO/IEC/IEEE 29119-3 and
ISO/IEC/IEEE 29119-4 are normative, meaning that they include requirements for anyone wanting to
claim conformance to these standards. Users who want to use the standards but have good reasons
vii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
for not following every requirement (e.g. for someone following an agile approach to development and
testing) can claim tailored conformance as long as the level of tailoring and its rationale are described
and agreed. Specific details of conformance are provided in the relevant conformance clause in each of
the standards.
The ISO/IEC/IEEE 29119 series can be used in isolation or can be used as part of a larger set of standards
that cover other aspects of the software life cycle. For instance, some users use ISO/IEC/IEEE 12207
to define software system life cycle models appropriate to their products and services (and some
may use the corresponding systems engineering standard, ISO/IEC/IEEE 15288), and reference the
ISO/IEC/IEEE 29119 series for their software testing needs.
Together, the ISO/IEC/IEEE 29119 series aims to provide stakeholders with the ability to manage and
perform software testing in any organization.
viii
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
INTERNATIONAL STANDARD ISO/IEC/IEEE 29119-1:2022(E)
Software and systems engineering — Software testing —
Part 1:
General concepts
1 Scope
This document specifies general concepts in software testing and presents key concepts for the
ISO/IEC/IEEE 29119 series.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO, IEC and IEEE maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
— IEEE Standards Dictionary Online: available at https:// dictionary .ieee .org
NOTE For additional terms and definitions in the field of systems and software engineering, see
ISO/IEC/IEEE 24765, which is published periodically as a “snapshot” of the SEVOCAB (Systems and software
Engineering Vocabulary) and is publicly accessible at https:// www .computer .org/ sevocab.
3.1
A/B testing
split-run testing
statistical testing (3.131) approach that allows testers to determine which of two systems or components
performs better
3.2
accessibility testing
type of usability testing (3.131) used to measure the degree to which a test item (3.107) can be operated
by users with the widest possible range of characteristics and abilities
3.3
activation function
transfer function
formula associated with a node in a neural network (3.50) that determines
the output of the node (activation value (3.4)) from the inputs to the neuron
3.4
activation value
output of an activation function (3.3) of a node in a neural network (3.50)
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.5
actual result
set of behaviours or conditions of a test item (3.107), or set of conditions of associated data or the test
environment (3.95), observed as a result of test execution (3.99)
EXAMPLE Outputs to screen, outputs to hardware, changes to data, reports, and communication messages
sent.
3.6
AI-based system
system including one or more components implementing AI (3.7)
3.7
artificial intelligence
AI
capability of an engineered system to acquire, process and apply knowledge and skills
3.8
autonomous system
system capable of working without human intervention for sustained
periods
3.9
autonomy
ability of a system to work for sustained periods without human
intervention
3.10
back-to-back testing
differential testing
approach to testing (3.131) whereby an alternative version of the system is used to generate expected
results (3.35) for comparison from the same test inputs
EXAMPLE The alternative version can be a system that already exists, a system developed by an independent
team or a system implemented using a different programming language.
3.11
base choice
base value
input parameter value that is normally selected based on being a representative or typical value for the
parameter
3.12
boundary value analysis
specification-based test design technique (3.94) based on exercising the boundaries of equivalence
partitions (3.30)
3.13
branch testing
structure-based test case (3.85) design technique based on exercising branches in the control flow
(3.22) of the test item (3.107)
3.14
cause-effect graph
graphical representation of decision rules (3.25) between causes (inputs described as Boolean conditions
(3.21)) and effects (outputs described as Boolean expressions)
3.15
cause-effect graphing
specification-based test design technique (3.94) based on exercising decision rules (3.25) in a cause-effect
graph (3.14)
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.16
classification
machine learning (3.44) function that predicts the output class for a given
input
3.17
classification tree
hierarchical tree model of the input data to a program in which the inputs are represented by distinct
classifications (relevant test aspects) and classes (input values)
3.18
combinatorial testing
combinatorial test design techniques
class of specification-based test design techniques (3.94) based on exercising combinations of P-V pairs
(3.56)
EXAMPLE Pairwise testing (3.57), base choice (3.11) testing.
3.19
compatibility testing
type of testing (3.131) that measures the degree to which a test item (3.107) can function satisfactorily
alongside other independent products in a shared environment (co-existence), and where necessary,
exchanges information with other systems or components (interoperability)
3.20
completion criteria
conditions under which the testing (3.131) activities are considered complete
3.21
condition
Boolean expression containing no Boolean operators
EXAMPLE “A < B” is a condition but “A and B” is not.
3.22
control flow
sequence in which operations are performed during the execution of a test item (3.107)
3.23
decision
type of statement in which a choice between two or more possible outcomes controls which set of
actions will result
Note 1 to entry: Typical decisions are simple selections (e.g. if-then-else), to decide when to exit loops (e.g. while-
loop), and in case (switch) statements (e.g. case-1–2-3-.-N).
3.24
decision outcome
result of a decision (3.23) that determines the branch to be executed
3.25
decision rule
combination of conditions (3.21) (also known as causes) and actions (also known as effects) that produce
a specific outcome in decision table testing (3.27) and cause-effect graphing (3.15)
3.26
decision table
tabular representation of decision rules (3.25) between causes (inputs described as Boolean conditions
(3.21)) and effects (outputs described as Boolean expressions)
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.27
decision table testing
specification-based test design technique (3.94) based on exercising decision rules (3.25) in a decision
table (3.26)
3.28
decision testing
structure-based test case (3.85) design technique based on exercising decision outcomes (3.24) in the
control flow (3.22) of the test item (3.107)
3.29
dynamic testing
testing (3.131) in which a test item (3.107) is evaluated by executing it
3.30
equivalence partition
equivalence class
class of inputs or outputs that are expected to be treated similarly by the test item (3.107)
3.31
equivalence partitioning
test design technique (3.94) in which test cases (3.85) are designed to exercise equivalence partitions
(3.30) by using one or more representative members of each partition
3.32
error guessing
test design technique (3.94) in which test cases (3.85) are derived on the basis of the tester’s knowledge
of past failures, or general knowledge of failure modes
Note 1 to entry: The relevant knowledge can be gained from personal experience, or can be encapsulated in, for
example, a defects database or a “bug taxonomy”.
3.33
executable statement
statement which, when compiled, is translated into object code, which will be executed procedurally
when the test item (3.107) is running and may perform an action on program data
3.34
exhaustive testing
test approach (3.83) in which all combinations of input values and preconditions are tested
Note 1 to entry: In nearly all non-trivial situations, exhaustive testing is impossible, due to the large number of
possible tests.
3.35
expected result
observable predicted behaviour of the test item (3.107) under specified conditions based on its
specification or another source
3.36
experience-based testing
class of test case (3.85) design techniques based on using the experience of testers to generate test
cases
EXAMPLE Error guessing (3.32).
Note 1 to entry: Experience-based testing can include concepts such as test attacks, tours, and error taxonomies
which target potential problems such as security, performance, and other quality areas.
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.37
exploratory testing
experience-based testing (3.36) in which the tester spontaneously designs and executes tests based on
the tester's existing relevant knowledge, prior exploration of the test item (3.107) (including the results
of previous tests), and heuristic “rules of thumb” regarding common software behaviours and types of
failure
Note 1 to entry: Exploratory testing hunts for hidden properties (including hidden behaviours) that, while quite
possibly benign by themselves, can interfere with other properties of the software under test, and so constitute
a risk that the software will fail.
3.38
fuzz testing
software testing (3.131) approach in which high volumes of random (or
near random) data, called fuzz, are used to generate inputs to the test item (3.107)
3.39
incident
anomalous or unexpected event, set of events, condition, or situation at any time during the life cycle of
a project, product, service, or system
3.40
incident report
documentation of the occurrence, nature, and status of an incident (3.39)
Note 1 to entry: Incident reports are also known as anomaly reports, bug reports, defect reports, error reports,
issues, problem reports and trouble reports, amongst other terms.
3.41
keyword
one or more words used as a reference to a specific set of actions
intended to be performed during the execution of one or more test cases (3.85)
Note 1 to entry: The actions include interactions with the User Interface during the test, verification, and specific
actions to set up a test scenario (3.123).
Note 2 to entry: Keywords are named using at least one verb.
Note 3 to entry: Composite keywords can be constructed based on other keywords.
3.42
keyword-driven testing
testing (3.131) using test cases (3.85) composed from keywords (3.41)
3.43
load testing
type of performance testing (3.58) conducted to evaluate the behaviour of a test item (3.107) under
anticipated conditions of varying load, usually between anticipated conditions of low, typical, and peak
usage
3.44
machine learning
ML
process using computational techniques to enable systems to learn from data or experience
3.45
maintainability testing
test type (3.130) conducted to evaluate the degree of effectiveness and efficiency with which a test item
(3.107) may be modified
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.46
manual testing
humans performing tests by entering information into a test item (3.107) and verifying the results
Note 1 to entry: Automated testing (3.131) uses tools, robots (3.70), and other test execution engines (3.100) to
perform tests. Manual testing does not use these items.
3.47
MC/DC testing
modified condition decision testing
structure-based test case (3.85) design technique based on demonstrating that a single Boolean
condition (3.21) within a decision (3.23) can independently affect the outcome of the decision
3.48
metamorphic relation
description of how changes to the test inputs for a test case (3.85) affect the expected outputs based on
the required behaviour of a test item (3.107)
3.49
metamorphic testing
specification-based test case (3.85) design technique based on generating test cases based on existing
test cases and metamorphic relations (3.48)
3.50
neural network
artificial neural network
network of primitive processing elements connected by weighted links
with adjustable weights, in which each element produces a value by applying a nonlinear function to its
input values, and transmits it to other elements or presents it as an output value
Note 1 to entry: Whereas some neural networks are intended to simulate the functioning of neurons in the
nervous system, most neural networks are used in artificial intelligence as realizations of the connectionist
model.
Note 2 to entry: Examples of nonlinear functions are a threshold function, a sigmoid function, and a polynomial
function.
[SOURCE: ISO/IEC 2382:2015, 2120625, modified — The admitted term “neural net” has been removed;
notes 3 to 5 to entry have been removed.]
3.51
neuron coverage
proportion of activated neurons divided by the total number of neurons in
the neural network (3.50) (normally expressed as a percentage) for a set of tests
Note 1 to entry: A neuron is considered to be activated if its activation value (3.4) exceeds zero.
3.52
non-deterministic system
system which, given a particular set of inputs and starting state, will not always produce the same set
of outputs and final state
3.53
organizational test practices
documentation that expresses the recommended approaches or methods for the testing (3.131) to be
performed within an organization, providing detail on how the testing is to be performed
Note 1 to entry: The organizational test practices are aligned with the organizational test policy (3.118).
Note 2 to entry: An organization can have more than one organizational test practices document to cover
markedly different contexts, such one for mobile apps and one for safety (3.71) critical systems.
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Note 3 to entry: The organizational test practices can incorporate the context of the test policy where no separate
test policy is available
3.54
organizational test process
test process (3.121) for developing and managing organizational test specifications (3.55)
3.55
organizational test specification
document that provides information about testing (3.131) for an organization, i.e. information that is
not project specific
EXAMPLE The most common examples of organizational test specifications are the organizational test policy
(3.118) and the organizational test practices (3.53).
3.56
P-V pair
parameter-value pair
combination of a test item (3.107) parameter with a value assigned to that parameter, used as a test
coverage item (3.90)
3.57
pairwise testing
black-box test design technique (3.94) in which test cases (3.85) are designed to execute all possible
discrete combinations of each pair of input parameters
Note 1 to entry: Pairwise testing is the most popular form of combinatorial testing (3.18).
3.58
performance testing
type of testing (3.131) conducted to evaluate the degree to which a test item (3.107) accomplishes its
designated functions within given constraints of time and other resources
3.59
portability testing
type of testing (3.131) conducted to evaluate the ease with which a test item (3.107) can be transferred
from one hardware or software environment to another, including the level of modification needed for
it to be executed in various types of environment
3.60
procedure testing
type of functional suitability testing (3.131) conducted to evaluate whether procedural instructions
for interacting with a test item (3.107) or using its outputs meet user requirements and support the
purpose of their use
3.61
product risk
risk that a product can be defective in some specific aspect of its function, quality, or structure
3.62
project risk
risk related to the management of a project
EXAMPLE Lack of staffing, strict deadlines, changing requirements.
3.63
random testing
specification-based test design technique (3.94) based on generating test cases (3.85) to exercise
randomly selected test item (3.107) inputs
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.64
regression testing
testing (3.131) performed following modifications to a test item (3.107) or to its operational
environment, to identify whether failures in unmodified parts of the test item occur
Note 1 to entry: Regression testing differs from retesting (3.68) in that it does not test that the modification
works correctly, but that other parts of the system have not been accidentally affected by the change.
Note 2 to entry: The adequacy of a set of regression test cases (3.85) depends on the item under test and on the
modifications to that item or its operational environment.
3.65
regulatory standard
standard promulgated by a regulatory agency
3.66
reliability testing
type of testing (3.131) conducted to evaluate the ability of a test item (3.107) to perform its required
functions, including evaluating the frequency with which failures occur, when it is used under stated
conditions for a specified period of time
3.67
requirements-based testing
specification-based test case (3.85) design technique based on exercising atomic requirements
EXAMPLE An atomic requirement can be ‘The system shall collect and store the date of birth of all registered
users.’
3.68
retesting
confirmation testing
testing (3.131) performed to check that modifications made to correct a fault have successfully removed
the fault
Note 1 to entry: When retesting is performed often it is also complemented by regression testing (3.64), to help
ensure that other unmodified parts of the test item (3.107) have not been accidentally adversely affected by the
modifications.
3.69
risk-based testing
testing (3.131) in which the management, selection, prioritization, and use of testing activities and
resources are consciously based on corresponding types and levels of analysed risk
3.70
robot
programmed actuated mechanism with a degree of autonomy (3.9), moving
within its environment, to perform intended tasks
Note 1 to entry: A robot includes the control system and interface of the control system.
Note 2 to entry: The classification of robot into industrial robot or service robot is done according to its intended
application.
3.71
safety
expectation that a system does not, under defined conditions, lead to a state in which human life, health,
property, or the environment is endangered
[SOURCE: ISO/IEC/IEEE 12207:2017, 3.1.48]
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
3.72
scenario testing
specification-based test case (3.85) design technique based on exercising sequences of interactions
between the test item (3.107) and other systems
Note 1 to entry: Users are considered to be other systems in this context.
3.73
scripted testing
testing (3.131) performed based on a documented test script (3.124)
Note 1 to entry: This term normally applies to manually executed testing, rather than the execution of an
automated script.
3.74
security testing
test type (3.130) conducted to evaluate the degree to which a test item (3.107), and associated data and
information, are protected so that unauthorized persons or systems cannot use, read, or modify them,
and authorized persons or systems are not denied access to them
3.75
specification-based testing
black-box testing
closed box testing
testing (3.131) in which the principal test basis (3.84) is the external inputs and outputs of the test item
(3.107), commonly based on a specification, rather than its implementation in source code or executable
software
3.76
state transition testing
specification-based test case (3.85) design technique based on exercising transitions in a state model
EXAMPLE Example state models are state transition diagram and state table.
3.77
statement testing
structure-based test case (3.85) design technique based on exercising executable statements (3.33) in
the source code of the test item (3.107)
3.78
static testing
testing (3.131) in which a test item (3.107) is examined against a set of quality or other criteria without
the test item being executed
EXAMPLE Reviews, static analysis.
3.79
stress testing
type of performance efficiency testing (3.131) conducted to evaluate a test item's (3.107) behaviour
under conditions of loading above anticipated or specified capacity requirements, or of resource
availability below minimum specified requirements
3.80
structure-based testing
white box testing
glass-box testing
structural testing
dynamic testing (3.29) in which the tests are derived from an examination of the structure of the test
item (3.107)
Note 1 to entry: Structure-based testing is not restricted to use at component level and can be used at all levels,
e.g. menu item coverage as part of a system test.
© ISO/IEC 2022 – All rights reserved
© IEEE 2022 – All rights reserved
Note 2 to entry: Techniques include branch testing (3.13), decision testing (3.28), and statement testing (3.77).
3.81
suspension criteria
criteria used to (temporarily) stop all or a portion of the testing (3.131) activities
3.82
test
activity in which a system or component is executed under specified conditions, the results are observed
or recorded, and an evaluation is made of some aspect of the system or component
3.83
test approach
high-level test implementation choice, typically made as part of the test strategy (3.127) design activity
EXAMPLE 1 The use of model-based testing (3.131) for the functional system testing.
EXAMPLE 2 Typical choices made as test approaches are test level (3.108), test type (3.130), test technique
(3.94), test practice (3.119) and the form of static testing (3.78) to be used.
3.84
test basis
information used as the basis for designing and implementing test cases (3.85)
Note 1 to entry: The test basis may take the form of documentation, such as a requirements specification,
design specification, or module specification, but may also be an undocumented understanding of the required
behaviour.
3.85
test case
set of preconditions, inputs and expected results (3.35), developed to drive the execution of a test item
(3.107) to meet test objectives (3.114)
Note 1 to entry: A test case is the lowest level of test implementation documentation (i.e. test cases are not made
up of test cases) for the test level (3.108) or test type (3.130) for which it is intended.
Note 2 to entry: Test case preconditions include the required state of the test environment (3.95), data (e.g.
databases) used by the test item, and the test item itself.
Note 3 to entry: Inputs are the data information and actions, where applicable, used to drive test execution (3.99).
3.86
test completion process
test management process (3.110) that aims to ensure that useful test assets are made available for later
use, test envi
...
INTERNATIONAL ISO/IEC/
STANDARD IEEE
29119-1
Second edition
Software and systems engineering —
Software testing —
Part 1:
General concepts
Ingénierie du logiciel et des systèmes — Tests logiciels —
Partie 1: Concepts généraux
PROOF/ÉPREUVE
Reference number
ISO/IEC/IEEE 29119-1:2021(E)
©
ISO/IEC 2021
©
IEEE 2021
ISO/IEC/IEEE 29119-1:2021(E)
© ISO/IEC 2021
© IEEE 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO or IEEE at the
respective address below or ISO’s member body in the country of the requester.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
CP 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Geneva NY 10016-5997, USA
Phone: +41 22 749 01 11
Email: copyright@iso.org Email: stds.ipr@ieee.org
Website: www.iso.org Website: www.ieee.org
Published in Switzerland
© ISO/IEC 2021 – All rights reserved
ii PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
Contents Page
Foreword .v
Introduction .vii
1 Scope . 9
2 Normative references . 9
3 Terms and definitions . 9
4 Software testing concepts .24
4.1 Introduction to software testing .24
4.1.1 Overview .24
4.1.2 Relationship to quality management .24
4.1.3 Verification and validation .24
4.1.4 Test item .25
4.1.5 Static and dynamic testing .25
4.1.6 Exhaustive testing and sampling .26
4.1.7 Testing as a heuristic .26
4.1.8 Purpose of testing .26
4.1.9 Test basis .27
4.1.10 Test oracle .27
4.1.11 Test independence.27
4.2 Test plans and test strategies .27
4.2.1 General.27
4.2.2 Risks and risk management .28
4.2.3 Risks and requirements as the basis of a test strategy .28
4.2.4 Test approaches .29
4.2.5 Testing in development and maintenance life cycles .30
4.2.6 Domains and system characteristics .31
4.2.7 Test strategy contents .31
4.3 Testing frameworks .32
4.3.1 Test processes .32
4.3.2 Test documentation .36
4.3.3 Documentation requirements .38
4.3.4 Configuration management and testing .38
4.3.5 Tool support .38
4.3.6 Process improvement and testing .38
4.3.7 Test metrics .39
4.4 Test design and execution .39
4.4.1 Test model .39
4.4.2 Model-based testing .40
4.4.3 Scripted and exploratory testing .41
4.4.4 Test design techniques.41
4.4.5 Experience-based testing .42
4.4.6 Retesting and regression testing .43
4.4.7 Manual and automated testing .43
4.4.8 Continuous testing .43
4.4.9 Back-to-back testing .43
4.4.10 A/B testing .44
4.4.11 Mathematical-based and fuzz testing .44
4.4.12 Test environments .44
4.4.13 Test data management .45
4.5 Project management and testing .45
4.6 Communication and reporting .45
4.7 Defects and incident management.46
Annex A (informative) System characteristics and testing – examples .47
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE iii
ISO/IEC/IEEE 29119-1:2021(E)
Annex B (informative) Testing roles .54
Bibliography .55
IEEE Notices and Abstract .56
© ISO/IEC 2021 – All rights reserved
iv PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO/IEC documents should be noted. This document was drafted in accordance with
the rules given in the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
IEEE Standards documents are developed within the IEEE Societies and the Standards Coordinating
Committees of the IEEE Standards Association (IEEE-SA) Standards Board. The IEEE develops its
standards through a consensus development process, approved by the American National Standards
Institute, which brings together volunteers representing varied viewpoints and interests to achieve the
final product. Volunteers are not necessarily members of the Institute and serve without compensation.
While the IEEE administers the process and establishes rules to promote fairness in the consensus
development process, the IEEE does not independently evaluate, test, or verify the accuracy of any of
the information contained in its standards.
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see https:// patents .iec .ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
ISO/IEC/IEEE 29119-1 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software and systems engineering, in cooperation with the Systems and
Software Engineering Standards Committee of the IEEE Computer Society, under the Partner Standards
Development Organization cooperation agreement between ISO and IEEE.
This second edition cancels and replaces the first edition (ISO/IEC/IEEE 29119-1:2013), which has been
technically revised.
The main changes compared to the previous edition are as follows:
— Testing terms and their definitions that are not covered within this document have been removed.
This has led to this document being renamed from ‘Concepts and definitions’ to ‘General concepts’.
— The coverage of test concepts has been made more concise and re-ordered.
— The concept of test sub-processes has been removed due to its complexity and replaced with
additional coverage of the instantiation of test processes.
— The expected content of a test strategy has been clarified.
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE v
ISO/IEC/IEEE 29119-1:2021(E)
— A simplified test design process is described, with the derivation of test cases now based on test
models rather than on test conditions.
— The coverage of metrics and measures has been moved from an annex into the body of the document.
— The annex explaining how testing fits into different life cycle models has been removed.
— A new annex providing examples of how systems from different domains are associated with certain
characteristics and test approaches has been added.
A list of all parts in the ISO/IEC/IEEE 29119 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
© ISO/IEC 2021 – All rights reserved
vi PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
Introduction
The purpose of the ISO/IEC/IEEE 29119 series is to define an internationally agreed set of standards for
software testing that can be used by any organization when performing any form of software testing
and using any life cycle.
It is recognized that there are many different types of software, software organizations, and
methodologies. Software domains include information technology (IT), personal computers (PC),
embedded, mobile, scientific and many other classifications. Software organizations range from
small to large, co-located to world-wide, and commercial to those providing a public service. Software
development methodologies include object-oriented, traditional, agile and DevOps. These and other
factors influence software testing. The ISO/IEC/IEEE 29119 series can support testing in many different
contexts.
This document facilitates the use of other parts in the ISO/IEC/IEEE 29119 series by introducing the
general concepts on which the ISO/IEC/IEEE 29119 series is built.
A general introduction to software testing is provided. The role of software testing in quality
management and as part of verification and validation is described; and its implementation in the form
of both static and dynamic testing is defined. The impracticality of exhaustive testing and the need for
sampling are explained; and the importance of the test basis and test oracle are described. The benefits
of test independence are introduced.
Test plans and test strategies are described in the context of risk-based testing, which is the
recommended approach to strategizing and managing testing that underlies the ISO/IEC/IEEE 29119
series and provides the basis for test prioritization and focus. Test levels, test types and test design
techniques (and corresponding measures) are described in the context of their inclusion as part of the
test strategy.
Various test frameworks are presented, including test processes (and test process improvement), test
metrics, test documentation, configuration management and tool support.
The performance of test design and execution based on the use of a test model is described. Several of
the most important test design and execution choices are considered, including scripted and exploratory
testing approaches, the importance of test design techniques for the creation of test cases, test patterns,
retesting and regression testing, manual and automated testing, back-to-back and A/B testing.
Several activities that directly support test design and executions are introduced, including test
environments, test data management, communications and reporting and defect and incident
management.
Annex A briefly describes a number of system characteristics and suggested associated test approaches.
If a tester can identify which of the system characteristics apply to the system they are testing, then
they should consider whether the specialized testing listed for the characteristic is appropriate for
inclusion in their test strategy.
Annex B introduces several generic testing roles and briefly describes their responsibilities.
The test process model that the ISO/IEC/IEEE 29119 series is based on is defined in detail in
ISO/IEC/IEEE 29119-2. ISO/IEC/IEEE 29119-2 covers the software testing processes at the
organizational level, test management level and for dynamic test levels. Testing is the primary approach
to risk treatment in software development. This document defines a risk-based approach to testing.
Templates and examples of test documentation that are produced during the testing process are defined
in ISO/IEC/IEEE 29119-3. Software testing techniques that can be used during testing are defined in
ISO/IEC/IEEE 29119-4.
While this document is informative, ISO/IEC/IEEE 29119-2, ISO/IEC/IEEE 29119-3 and
ISO/IEC/IEEE 29119-4 are normative, meaning that they include requirements for anyone wanting to
claim conformance to these standards. Users who want to use the standards but have good reasons
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE vii
ISO/IEC/IEEE 29119-1:2021(E)
for not following every requirement (e.g. for someone following an agile approach to development and
testing) can claim tailored conformance as long as the level of tailoring and its rationale are described
and agreed. Specific details of conformance are provided in the relevant conformance clause in each of
the standards.
The ISO/IEC/IEEE 29119 series can be used in isolation or can be used as part of a larger set of standards
that cover other aspects of the software life cycle. For instance, some users use ISO/IEC/IEEE 12207
to define software system life cycle models appropriate to their products and services (and some
may use the corresponding systems engineering standard, ISO/IEC/IEEE 15288), and reference the
ISO/IEC/IEEE 29119 series for their software testing needs.
Together, the ISO/IEC/IEEE 29119 series aims to provide stakeholders with the ability to manage and
perform software testing in any organization.
© ISO/IEC 2021 – All rights reserved
viii PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
INTERNATIONAL STANDARD ISO/IEC/IEEE 29119-1:2021(E)
1 Scope
This document specifies general concepts in software testing and presents key concepts for the
ISO/IEC/IEEE 29119 series.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO, IEC and IEEE maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org
— IEEE Standards Dictionary Online: available at https:// ieeexplore .ieee .org/ xpls/ dictionary .jsp
NOTE For additional terms and definitions in the field of systems and software engineering, see
ISO/IEC/IEEE 24765, which is published periodically as a “snapshot” of the SEVOCAB (Systems and software
Engineering Vocabulary) and is publicly accessible at www .computer .org/ sevocab. ISO/IEC/IEEE 29119-1
includes all terms that are used in ISO/IEC/IEEE 29119 (all parts).
3.1
A/B testing
split-run testing
statistical testing (3.131) approach that allows testers to determine which of two systems or components
performs better
3.2
accessibility testing
type of usability testing (3.131) used to measure the degree to which a test item (3.107) can be operated
by users with the widest possible range of characteristics and abilities
3.3
activation function
transfer function
formula associated with a node in a neural network (3.50) that determines
the output of the node (activation value (3.4)) from the inputs to the neuron
3.4
activation value
output of an activation function (3.3) of a node in a neural network (3.50)
3.5
actual result
set of behaviours or conditions of a test item (3.107), or set of conditions of associated data or the test
environment (3.95), observed as a result of test execution (3.99)
EXAMPLE Outputs to screen, outputs to hardware, changes to data, reports, and communication messages
sent.
3.6
AI-based system
system including one or more components implementing AI (3.7)
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 9
ISO/IEC/IEEE 29119-1:2021(E)
3.7
artificial intelligence
AI
capability of an engineered system to acquire, process and apply knowledge and skills
3.8
autonomous system
system capable of working without human intervention for sustained
periods
3.9
autonomy
ability of a system to work for sustained periods without human
intervention
3.10
back-to-back testing
differential testing
approach to testing (3.131) whereby an alternative version of the system is used to generate expected
results (3.35) for comparison from the same test inputs
EXAMPLE The alternative version can be a system that already exists, a system developed by an independent
team or a system implemented using a different programming language.
3.11
base choice
base value
input parameter value that is normally selected based on being a representative or typical value for the
parameter
3.12
boundary value analysis
specification-based test design technique (3.94) based on exercising the boundaries of equivalence
partitions (3.30)
3.13
branch testing
structure-based test case (3.85) design technique based on exercising branches in the control flow
(3.22) of the test item (3.107)
3.14
cause-effect graph
graphical representation of decision rules (3.25) between causes (inputs described as Boolean conditions
(3.21)) and effects (outputs described as Boolean expressions)
3.15
cause-effect graphing
specification-based test design technique (3.94) based on exercising decision rules (3.25) in a cause-effect
graph (3.14)
3.16
classification
machine learning (3.44) function that predicts the output class for a given
input
3.17
classification tree
hierarchical tree model of the input data to a program in which the inputs are represented by distinct
classifications (relevant test aspects) and classes (input values)
© ISO/IEC 2021 – All rights reserved
10 PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
3.18
combinatorial testing
combinatorial test design techniques
class of specification-based test design techniques (3.94) based on exercising combinations of P-V pairs
(3.56)
EXAMPLE Pairwise testing (3.57), base choice (3.11) testing.
3.19
compatibility testing
type of testing (3.131) that measures the degree to which a test item (3.107) can function satisfactorily
alongside other independent products in a shared environment (co-existence), and where necessary,
exchanges information with other systems or components (interoperability)
3.20
completion criteria
conditions under which the testing (3.131) activities are considered complete
3.21
condition
Boolean expression containing no Boolean operators
EXAMPLE “A < B” is a condition but “A and B” is not.
3.22
control flow
sequence in which operations are performed during the execution of a test item (3.107)
3.23
decision
type of statement in which a choice between two or more possible outcomes controls which set of
actions will result
Note 1 to entry: Typical decisions are simple selections (e.g. if-then-else), to decide when to exit loops (e.g. while-
loop), and in case (switch) statements (e.g. case-1–2-3-.-N).
3.24
decision outcome
result of a decision (3.23) that determines the branch to be executed
3.25
decision rule
combination of conditions (3.21) (also known as causes) and actions (also known as effects) that produce
a specific outcome in decision table testing (3.27) and cause-effect graphing (3.15)
3.26
decision table
tabular representation of decision rules (3.25) between causes (inputs described as Boolean conditions
(3.21)) and effects (outputs described as Boolean expressions)
3.27
decision table testing
specification-based test design technique (3.94) based on exercising decision rules (3.25) in a decision
table (3.26)
3.28
decision testing
structure-based test case (3.85) design technique based on exercising decision outcomes (3.24) in the
control flow (3.22) of the test item (3.107)
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 11
ISO/IEC/IEEE 29119-1:2021(E)
3.29
dynamic testing
testing (3.131) in which a test item (3.107) is evaluated by executing it
3.30
equivalence partition
equivalence class
class of inputs or outputs that are expected to be treated similarly by the test item (3.107)
3.31
equivalence partitioning
test design technique (3.94) in which test cases (3.85) are designed to exercise equivalence partitions
(3.30) by using one or more representative members of each partition
3.32
error guessing
test design technique (3.94) in which test cases (3.85) are derived on the basis of the tester’s knowledge
of past failures, or general knowledge of failure modes
Note 1 to entry: The relevant knowledge can be gained from personal experience, or can be encapsulated in, for
example, a defects database or a “bug taxonomy”.
3.33
executable statement
statement which, when compiled, is translated into object code, which will be executed procedurally
when the test item (3.107) is running and may perform an action on program data
3.34
exhaustive testing
test approach (3.83) in which all combinations of input values and preconditions are tested
Note 1 to entry: In nearly all non-trivial situations, exhaustive testing is impossible, due to the large number of
possible tests.
3.35
expected result
observable predicted behaviour of the test item (3.107) under specified conditions based on its
specification or another source
3.36
experience-based testing
class of test case (3.85) design techniques based on using the experience of testers to generate test
cases
EXAMPLE Error guessing (3.32).
Note 1 to entry: Experience-based testing can include concepts such as test attacks, tours, and error taxonomies
which target potential problems such as security, performance, and other quality areas.
3.37
exploratory testing
experience-based testing (3.36) in which the tester spontaneously designs and executes tests based on
the tester's existing relevant knowledge, prior exploration of the test item (3.107) (including the results
of previous tests), and heuristic “rules of thumb” regarding common software behaviours and types of
failure
Note 1 to entry: Exploratory testing hunts for hidden properties (including hidden behaviours) that, while quite
possibly benign by themselves, can interfere with other properties of the software under test, and so constitute
a risk that the software will fail.
© ISO/IEC 2021 – All rights reserved
12 PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
3.38
fuzz testing
software testing (3.131) approach in which high volumes of random (or
near random) data, called fuzz, are used to generate inputs to the test item (3.107)
3.39
incident
anomalous or unexpected event, set of events, condition, or situation at any time during the life cycle of
a project, product, service, or system
3.40
incident report
documentation of the occurrence, nature, and status of an incident (3.39)
Note 1 to entry: Incident reports are also known as anomaly reports, bug reports, defect reports, error reports,
issues, problem reports and trouble reports, amongst other terms.
3.41
keyword
one or more words used as a reference to a specific set of actions
intended to be performed during the execution of one or more test cases (3.85)
Note 1 to entry: The actions include interactions with the User Interface during the test, verification, and specific
actions to set up a test scenario (3.123).
Note 2 to entry: Keywords are named using at least one verb.
Note 3 to entry: Composite keywords can be constructed based on other keywords.
3.42
keyword-driven testing
testing (3.131) using test cases (3.85) composed from keywords (3.41)
3.43
load testing
type of performance testing (3.58) conducted to evaluate the behaviour of a test item (3.107) under
anticipated conditions of varying load, usually between anticipated conditions of low, typical, and peak
usage
3.44
machine learning
ML
process using computational techniques to enable systems to learn from data or experience
3.45
maintainability testing
test type (3.130) conducted to evaluate the degree of effectiveness and efficiency with which a test item
(3.107) may be modified
3.46
manual testing
humans performing tests by entering information into a test item (3.107) and verifying the results
Note 1 to entry: Automated testing (3.131) uses tools, robots (3.70), and other test execution engines (3.100) to
perform tests. Manual testing does not use these items.
3.47
MC/DC testing
modified condition decision testing
structure-based test case (3.85) design technique based on demonstrating that a single Boolean
condition (3.21) within a decision (3.23) can independently affect the outcome of the decision
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 13
ISO/IEC/IEEE 29119-1:2021(E)
3.48
metamorphic relation
description of how changes to the test inputs for a test case (3.85) affect the expected outputs based on
the required behaviour of a test item (3.107)
3.49
metamorphic testing
specification-based test case (3.85) design technique based on generating test cases based on existing
test cases and metamorphic relations (3.48)
3.50
neural network
artificial neural network
network of primitive processing elements connected by weighted links
with adjustable weights, in which each element produces a value by applying a nonlinear function to its
input values, and transmits it to other elements or presents it as an output value
Note 1 to entry: Whereas some neural networks are intended to simulate the functioning of neurons in the
nervous system, most neural networks are used in artificial intelligence as realizations of the connectionist
model.
Note 2 to entry: Examples of nonlinear functions are a threshold function, a sigmoid function, and a polynomial
function.
[SOURCE: ISO/IEC 2382:2015, 2120625, modified — The admitted term “neural net” has been removed;
notes 3 to 5 to entry have been removed.]
3.51
neuron coverage
proportion of activated neurons divided by the total number of neurons in
the neural network (3.50) (normally expressed as a percentage) for a set of tests
Note 1 to entry: A neuron is considered to be activated if its activation value (3.4) exceeds zero.
3.52
non-deterministic system
system which, given a particular set of inputs and starting state, will not always produce the same set
of outputs and final state
3.53
organizational test practices
documentation that expresses the recommended approaches or methods for the testing (3.131) to be
performed within an organization, providing detail on how the testing is to be performed
Note 1 to entry: The organizational test practices are aligned with the organizational test policy (3.118).
Note 2 to entry: An organization can have more than one organizational test practices document to cover
markedly different contexts, such one for mobile apps and one for safety (3.71) critical systems.
Note 3 to entry: The organizational test practices can incorporate the context of the test policy where no separate
test policy is available
3.54
organizational test process
test process (3.121) for developing and managing organizational test specifications (3.55)
© ISO/IEC 2021 – All rights reserved
14 PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
3.55
organizational test specification
document that provides information about testing (3.131) for an organization, i.e. information that is
not project specific
EXAMPLE The most common examples of organizational test specifications are the organizational test policy
(3.118) and the organizational test practices (3.53).
3.56
P-V pair
parameter-value pair
combination of a test item (3.107) parameter with a value assigned to that parameter, used as a test
coverage item (3.90)
3.57
pairwise testing
black-box test design technique (3.94) in which test cases (3.85) are designed to execute all possible
discrete combinations of each pair of input parameters
Note 1 to entry: Pairwise testing is the most popular form of combinatorial testing (3.18).
3.58
performance testing
type of testing (3.131) conducted to evaluate the degree to which a test item (3.107) accomplishes its
designated functions within given constraints of time and other resources
3.59
portability testing
type of testing (3.131) conducted to evaluate the ease with which a test item (3.107) can be transferred
from one hardware or software environment to another, including the level of modification needed for
it to be executed in various types of environment
3.60
procedure testing
type of functional suitability testing (3.131) conducted to evaluate whether procedural instructions
for interacting with a test item (3.107) or using its outputs meet user requirements and support the
purpose of their use
3.61
product risk
risk that a product can be defective in some specific aspect of its function, quality, or structure
3.62
project risk
risk related to the management of a project
EXAMPLE Lack of staffing, strict deadlines, changing requirements.
3.63
random testing
specification-based test design technique (3.94) based on generating test cases (3.85) to exercise
randomly selected test item (3.107) inputs
3.64
regression testing
testing (3.131) performed following modifications to a test item (3.107) or to its operational
environment, to identify whether failures in unmodified parts of the test item occur
Note 1 to entry: Regression testing differs from retesting (3.68) in that it does not test that the modification
works correctly, but that other parts of the system have not been accidentally affected by the change.
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 15
ISO/IEC/IEEE 29119-1:2021(E)
Note 2 to entry: The adequacy of a set of regression test cases (3.85) depends on the item under test and on the
modifications to that item or its operational environment.
3.65
regulatory standard
standard promulgated by a regulatory agency
3.66
reliability testing
type of testing (3.131) conducted to evaluate the ability of a test item (3.107) to perform its required
functions, including evaluating the frequency with which failures occur, when it is used under stated
conditions for a specified period of time
3.67
requirements-based testing
specification-based test case (3.85) design technique based on exercising atomic requirements
EXAMPLE An atomic requirement can be ‘The system shall collect and store the date of birth of all registered
users.’
3.68
retesting
confirmation testing
testing (3.131) performed to check that modifications made to correct a fault have successfully removed
the fault
Note 1 to entry: When retesting is performed often it is also complemented by regression testing (3.64), to help
ensure that other unmodified parts of the test item (3.107) have not been accidentally adversely affected by the
modifications.
3.69
risk-based testing
testing (3.131) in which the management, selection, prioritization, and use of testing activities and
resources are consciously based on corresponding types and levels of analysed risk
3.70
robot
programmed actuated mechanism with a degree of autonomy (3.9), moving
within its environment, to perform intended tasks
Note 1 to entry: A robot includes the control system and interface of the control system.
Note 2 to entry: The classification of robot into industrial robot or service robot is done according to its intended
application.
3.71
safety
expectation that a system does not, under defined conditions, lead to a state in which human life, health,
property, or the environment is endangered
[SOURCE: ISO/IEC/IEEE 12207:2017, 3.1.48]
3.72
scenario testing
specification-based test case (3.85) design technique based on exercising sequences of interactions
between the test item (3.107) and other systems
Note 1 to entry: Users are considered to be other systems in this context.
© ISO/IEC 2021 – All rights reserved
16 PROOF/ÉPREUVE © IEEE 2021 – All rights reserved
ISO/IEC/IEEE 29119-1:2021(E)
3.73
scripted testing
testing (3.131) performed based on a documented test script (3.124)
Note 1 to entry: This term normally applies to manually executed testing, rather than the execution of an
automated script.
3.74
security testing
test type (3.130) conducted to evaluate the degree to which a test item (3.107), and associated data and
information, are protected so that unauthorized persons or systems cannot use, read, or modify them,
and authorized persons or systems are not denied access to them
3.75
specification-based testing
black-box testing
closed box testing
testing (3.131) in which the principal test basis (3.84) is the external inputs and outputs of the test item
(3.107), commonly based on a specification, rather than its implementation in source code or executable
software
3.76
state transition testing
specification-based test case (3.85) design technique based on exercising transitions in a state model
EXAMPLE Example state models are state transition diagram and state table.
3.77
statement testing
structure-based test case (3.85) design technique based on exercising executable statements (3.33) in
the source code of the test item (3.107)
3.78
static testing
testing (3.131) in which a test item (3.107) is examined against a set of quality or other criteria without
the test item being executed
EXAMPLE Reviews, static analysis.
3.79
stress testing
type of performance efficiency testing (3.131) conducted to evaluate a test item's (3.107) behaviour
under conditions of loading above anticipated or specified capacity requirements, or of resource
availability below minimum specified requirements
3.80
structure-based testing
white box testing
glass-box testing
structural testing
dynamic testing (3.29) in which the tests are derived from an examination of the structure of the test
item (3.107)
Note 1 to entry: Structure-based testing is not restricted to use at component level and can be used at all levels,
e.g. menu item coverage as part of a system test.
Note 2 to entry: Techniques include branch testing (3.13), decision testing (3.28), and statement testing (3.77).
3.81
suspension criteria
criteria used to (temporarily) stop all or a portion of the testing (3.131) activities
© ISO/IEC 2021 – All rights reserved
© IEEE 2021 – All rights reserved PROOF/ÉPREUVE 17
ISO/IEC/IEEE 29119-1:2021(E)
3.82
test
activity in which a system or component is executed under specified conditions, the results are observed
or recorded, and an evaluation is made of some aspect of the system or component
3.83
test approach
high-level test implementation choice, typically made as part of the test strategy (3.127) design activity
EXAMPLE 1 The use of model-based testing (3.131) for the functional system testing.
EXAMPLE 2 Typical choices made as test approaches are test level (3.108), test type (3.130), test technique
(3.94), test practice (3.119) and the form of static testing (3.78) to be used.
3.84
test basis
information used as the basis for designing and implementing test cases (3.85)
Note 1 to entry: The test basis may take the form of documentation, such as a requirements specification,
design specification, or module specification, but may also be an undocumented understanding of the required
behaviour.
3.85
test case
set of preconditions, inputs and expected results (3.35), developed to drive the execution of a test item
(3.107) to meet test objectives (3.114)
Note 1 to entry: A test case is the lowest level of test implementation documentation (i.e. test cases are not made
up of test cases) for the test level (3.108) or test type (3.130) for which it is intended.
Note 2 to entry: Test case preconditions include the required state of the test environment (3.95), data (e.g.
databases) used by the test item, and the test item itself.
Note 3 to entry: Inputs are the data information and actions, where applicable, used to drive tes
...
NORME ISO/
INTERNATIONALE IEC/IEEE
29119-1
Deuxième édition
2022-01
Ingénierie du logiciel et des
systèmes — Essais du logiciel —
Partie 1:
Concepts généraux
Software and systems engineering — Software testing —
Part 1: General concepts
Numéro de référence
© ISO/IEC 2022
© IEEE 2022
DOCUMENT PROTÉGÉ PAR COPYRIGHT
© ISO/IEC 2022
© IEEE 2022
Tous droits réservés. Sauf prescription différente ou nécessité dans le contexte de la mise en œuvre, aucune partie de cette
publication ne peut être reproduite ni utilisée sous quelque forme que ce soit et par aucun procédé, électronique ou mécanique, y
compris la photocopie, ou la diffusion sur l’internet ou un intranet, sans autorisation écrite soit de l’ISO soit de l’IEEE, à l’une ou
l’autre des adresses ci-après.
ISO copyright office Institute of Electrical and Electronics Engineers, Inc
Case postale 401 • Ch. de Blandonnet 8 3 Park Avenue, New York
CH-1214 Vernier, Genève NY 10016-5997, USA
Tél.: +41 22 749 01 11
Fax: +41 22 749 09 47
E-mail: copyright@iso.org E-mail: stds.ipr@ieee.org
Web: www.iso.org Web: www.ieee.org
Publié en Suisse
ii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
Sommaire Page
Avant-propos .v
Introduction .vii
1 Domaine d'application .1
2 Références normatives .1
3 Termes et définitions . 1
4 Concepts des tests logiciels .17
4.1 Introduction aux tests logiciels . 17
4.1.1 Vue d'ensemble. 17
4.1.2 Relation avec le management de la qualité . 17
4.1.3 Vérification et validation . 17
4.1.4 Élément de test . 18
4.1.5 Tests statiques et dynamiques . 18
4.1.6 Tests exhaustifs et échantillonnage . 19
4.1.7 Tests heuristiques . 19
4.1.8 Finalité du test . 19
4.1.9 Base de test. 20
4.1.10 Oracle de test . 20
4.1.11 Indépendance des tests . 20
4.2 Plans de test et stratégies de test . 21
4.2.1 Généralités . 21
4.2.2 Risques et gestion du risque . 21
4.2.3 Risques et exigences comme base d'une stratégie de test .22
4.2.4 Approches de test . 22
4.2.5 Tests dans les cycles de vie du développement et de la maintenance .23
4.2.6 Domaines et caractéristiques du système . 24
4.2.7 Contenu de la stratégie de test . 25
4.3 Frameworks de test.26
4.3.1 Processus de test . 26
4.3.2 Documentation des tests .30
4.3.3 Exigences relatives à la documentation . 32
4.3.4 Gestion de la configuration et tests . 32
4.3.5 Support outillé .33
4.3.6 Amélioration des processus et tests . 33
4.3.7 Métriques de test.34
4.4 Conception et exécution des tests .34
4.4.1 Modèle de test .34
4.4.2 Tests basés sur un modèle . 35
4.4.3 Tests avec script et tests exploratoires .36
4.4.4 Techniques de conception de tests .36
4.4.5 Tests basés sur l'expérience . 37
4.4.6 Re-tests et tests de régression .38
4.4.7 Tests manuels et automatisés .38
4.4.8 Tests en continu .39
4.4.9 Tests dos-à-dos . .39
4.4.10 Tests A/B . 39
4.4.11 Tests mathématiques et tests à données aléatoires.39
4.4.12 Environnements de test .40
4.4.13 Gestion des données de test .40
4.5 Gestion de projet et tests .40
4.6 Communication et génération de rapports . 41
4.7 Gestion des défauts et des incidents . 41
Annexe A (informative) Caractéristiques du système et tests — Exemples .42
iii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
Annexe B (informative) Rôles de test .50
Bibliographie .51
Notes et résumés IEEE .52
iv
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
Avant-propos
L'ISO (Organisation internationale de normalisation) et l’IEC (Commission électrotechnique
internationale) forment le système spécialisé de la normalisation mondiale. Les organismes
nationaux membres de l'ISO ou de l’IEC participent au développement de Normes internationales
par l'intermédiaire des comités techniques créés par l'organisation concernée afin de s'occuper des
domaines particuliers de l'activité technique. Les comités techniques de l'ISO et de l’IEC collaborent
dans des domaines d'intérêt commun. D'autres organisations internationales, gouvernementales et non
gouvernementales, en liaison avec l'ISO et l’IEC, participent également aux travaux.
Les procédures utilisées pour élaborer le présent document et celles destinées à sa mise à jour sont
décrites dans les Directives ISO/IEC, Partie 1. Il convient, en particulier de prendre note des différents
critères d'approbation requis pour les différents types de documents ISO/IEC. Le présent document a
été rédigé conformément aux règles de rédaction données dans les Directives ISO/IEC, Partie 2 (voir
www.iso.org/directives ou www.iec.ch/members_experts/refdocs).
Les documents normatifs de l’IEEE sont développés au sein des sociétés de l’IEEE et des Comités de
Coordination des Normes du Conseil des Normes de l’Association des normes IEEE (IEEE-SA). L’IEEE
élabore ses normes par le biais d’un processus d'élaboration du consensus approuvé par l’American
National Standards Institute, qui rassemble des volontaires représentant divers points de vue et divers
intérêts pour parvenir au produit final. Les volontaires ne sont pas nécessairement des membres
de l’Institut et aucune compensation ne leur est attribuée. Bien que l’IEEE administre le processus
et établisse des règles pour favoriser l’équité au cours du processus d'élaboration du consensus,
l’IEEE n’évalue pas, ne soumet pas à essai ou ne vérifie pas de manière indépendante l’exactitude des
informations contenues dans ses normes.
L'attention est attirée sur le fait que certains des éléments du présent document peuvent faire l'objet
de droits de propriété intellectuelle ou de droits analogues. L'ISO et l’IEC ne sauraient être tenues pour
responsables de ne pas avoir identifié de tels droits de propriété et averti de leur existence. Les détails
concernant les références aux droits de propriété intellectuelle ou autres droits analogues identifiés
lors de l'élaboration du document sont indiqués dans l'Introduction et/ou dans la liste des déclarations
de brevets reçues par l'ISO (voir www.iso.org/brevets) ou dans la liste des déclarations de brevets
reçues par l'IEC (voir https://patents.iec.ch).
Les appellations commerciales éventuellement mentionnées dans le présent document sont données
pour information, par souci de commodité, à l’intention des utilisateurs et ne sauraient constituer un
engagement.
Pour une explication de la nature volontaire des normes, la signification des termes et expressions
spécifiques de l'ISO liés à l'évaluation de la conformité, ou pour toute information au sujet de
l'adhésion de l'ISO aux principes de l’Organisation mondiale du commerce (OMC) concernant les
obstacles techniques au commerce (OTC), voir www.iso.org/iso/avant-propos. Pour l'IEC, voir
www.iec.ch/understanding-standards.
ISO/IEC/IEEE 29119-1 a été élaborée par le comité technique mixte ISO/IEC JTC 1, Technologies de
l'information, sous-comité SC 7, Ingénierie du logiciel et des systèmes, en coopération avec le Systems and
Software Engineering Standards Committee de l'IEEE Computer Society, dans le cadre de l'accord de
coopération entre les Organisations Partenaires de Développement de Normes que sont l'ISO et l'IEEE.
Cette deuxième édition annule et remplace la première édition (ISO/IEC/IEEE 29119-1:2013), qui a fait
l'objet d'une révision technique.
Les principales modifications sont les suivantes:
— les termes relatifs aux tests et leurs définitions non couverts dans le présent document ont été
supprimés. Le présent document intitulé «Concepts et définitions» a donc été renommé «Concepts
généraux»;
— la couverture des concepts de test a été réduite et réorganisée;
v
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
— le concept de sous-processus de test a été supprimé en raison de sa complexité et remplacé par la
couverture supplémentaire d'instanciation des processus de test;
— le contenu attendu de la stratégie de test a été clarifié;
— un processus de conception simplifié des tests est décrit, l'origine des cas de test étant désormais
basée sur les modèles de test plutôt que sur les conditions de test;
— la couverture des indicateurs et des mesures, qui se trouvait dans une annexe, a été déplacée dans
le corps du document;
— l'annexe expliquant de quelle manière les tests s'adaptent aux différents modèles de cycle de vie a
été supprimée;
— une nouvelle annexe fournissant des exemples de la manière dont les systèmes issus de différents
domaines sont associés à certaines caractéristiques et approches de test a été ajoutée.
Une liste de toutes les parties de la série ISO/IEC/IEEE 29119 se trouve sur les sites Web de l'ISO et de
l'IEC.
Il convient que l’utilisateur adresse tout retour d’information ou toute question concernant le présent
document à l’organisme national de normalisation de son pays. Une liste exhaustive desdits organismes
se trouve à l’adresse www.iso.org/fr/members.html et www.iec.ch/national-committees.
vi
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
Introduction
L'objet de la série ISO/IEC/IEEE 29119 est de définir un ensemble de normes de tests logiciels
approuvées au plan national pouvant être utilisées par toute organisation qui entreprend toute forme
de test logiciel, quel que soit le cycle de vie.
Il est indéniable qu'il existe un grand nombre de logiciels, d'organisations logicielles et de types de
méthodologies différents. Les logiciels couvrent divers domaines, notamment les technologies de
l'information (TI), les ordinateurs personnels (PC), les technologies intégrées, mobiles et scientifiques,
ainsi que de nombreuses autres classifications. Les organisations logicielles vont des petites aux
grandes entreprises, des entreprises colocalisées aux multinationales, et des entreprises commerciales
aux services publics. Les méthodologies de développement logiciel sont tantôt orientées objets, tantôt
traditionnelles, tantôt en Agile, tantôt axées sur le DevOps. Tous ces facteurs, combinés à d'autres,
influencent les tests logiciels. La série ISO/IEC/IEEE 29119 peut faciliter la conduite de tests dans de
nombreux contextes différents.
Le présent document aide à l'utilisation des autres parties de la série ISO/IEC/IEEE 29119 en
introduisant les concepts généraux sur lesquels la série ISO/IEC/IEEE 29119 est bâtie.
Il fournit une introduction générale à la notion de tests logiciels. Il décrit le rôle des tests logiciels dans
le management de la qualité et dans le cadre des processus de vérification et de validation, et définit
leur mise en œuvre sous forme de tests statiques et dynamiques. Il explique, par ailleurs, la difficulté de
réaliser des tests exhaustifs et la nécessité de procéder par échantillonnage, et souligne l'importance de
la base de test et de l'oracle de test. Il décrit en outre les avantages de l'indépendance des tests.
Les plans de test et les stratégies de test sont décrits dans le contexte de tests basés sur les risques,
ce qui constitue l'approche recommandée pour l'élaboration d'une stratégie et la gestion des tests,
approche sous-jacente à la série ISO/IEC/IEEE 29119 et fournissant une base pour définir les priorités
et les objectifs de test. Les niveaux de test, les types de test et les techniques de conception de tests
(ainsi que les mesures correspondantes) sont décrits en tant que partie intégrante de la stratégie de
test.
Différents frameworks de test sont présentés, notamment les processus de test (et l'amélioration de ces
processus), les métriques de test, la documentation de test, la gestion de la configuration ainsi que le
support outillé.
Cette norme décrit également la conception et l'exécution des tests à partir d'un modèle de test. Elle
évalue plusieurs choix de conception et d'exécution de tests parmi les plus importants, notamment
les approches de tests basés sur des scripts et les approches de tests exploratoires, l'importance des
techniques de conception de tests pour la création de cas de test, les patterns de test, les re-tests et les
tests de régression, les tests manuels et automatisés, les tests dos-à-dos (back-to-back) et les tests A/B.
Elle présente plusieurs activités qui soutiennent directement la conception et l'exécution des tests,
notamment les environnements de test, la gestion des données de test, la communication et la
production de rapports, et la gestion des défauts et incidents.
L'Annexe A décrit brièvement certaines caractéristiques du système et suggère les approches de test
associées. Si un testeur peut identifier les caractéristiques du système qui s'appliquent au système qu'il
est en train de tester, il convient alors qu'il détermine s'il est utile d'inclure dans sa stratégie de test les
tests spécialisés répertoriés pour les caractéristiques en question.
L'Annexe B introduit plusieurs rôles de test génériques et en décrit brièvement les responsabilités.
Le modèle de processus de test sur lequel est fondée la série de normes ISO/IEC/IEEE 29119 est défini
en détail dans l'ISO/IEC/IEEE 29119-2. L'ISO/IEC/IEEE 29119-2 traite des processus de test logiciel au
niveau organisationnel, au niveau de la gestion des tests et au niveau des tests dynamiques. Le test
forme l'approche essentielle du traitement des risques dans le développement de logiciels. Le présent
document définit une approche des tests basée sur le risque.
vii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
L'ISO/IEC/IEEE 29119-3 définit des modèles et des exemples de documentation de test produite au
cours du processus de test. Les techniques de test pouvant être utilisées au cours des tests sont définies
dans l'ISO/IEC/IEEE 29119-4.
Bien que le présent document soit informatif, l'ISO/IEC/IEEE 29119-2, l'ISO/IEC/IEEE 29119-3 et
l'ISO/IEC/IEEE 29119-4 sont normatives, ce qui signifie qu'elles stipulent des exigences obligatoires
pour quiconque souhaiterait revendiquer la conformité à ces normes. Les utilisateurs qui souhaitent
utiliser les normes, mais qui ont de bonnes raisons de ne pas en suivre chaque exigence (par exemple,
quelqu'un qui suit une approche de développement et de test en Agile) peuvent revendiquer une
conformité adaptée, à condition que le niveau d'adaptation et sa justification soient décrits et acceptés.
Les détails de conformité spécifiques sont donnés dans l'article de chaque norme applicable à la
conformité.
La série ISO/IEC/IEEE 29119 peut être utilisée de manière isolée ou dans le cadre d'un ensemble
plus vaste de normes qui couvrent d'autres aspects du cycle de vie du logiciel. Par exemple,
certains utilisateurs utiliseront l'ISO/IEC/IEEE 12207 pour définir des modèles de cycle de vie du
système logiciel qui s'appliquent à leurs produits et services (et d'autres peuvent utiliser la norme
correspondante ISO/IEC/IEEE 15288 relative à l'ingénierie des systèmes), et se référer à la série de
normes ISO/IEC/IEEE 29119 pour leurs besoins en matière de tests logiciels.
Collectivement, la série ISO/IEC/IEEE 29119 vise à donner aux parties prenantes la possibilité de gérer
et d'exécuter des tests logiciels dans n'importe quelle organisation.
viii
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
NORME INTERNATIONALE ISO/IEC/IEEE 29119-1:2022(F)
Ingénierie du logiciel et des systèmes — Essais du
logiciel —
Partie 1:
Concepts généraux
1 Domaine d'application
Le présent document définit les concepts généraux relatifs aux tests logiciels et présente les principaux
concepts employés dans la série ISO/IEC/IEEE 29119.
2 Références normatives
Le présent document ne contient aucune référence normative.
3 Termes et définitions
Pour les besoins du présent document, les termes et définitions suivants s'appliquent.
L'ISO, l'IEC et l'IEEE tiennent à jour des bases de données terminologiques destinées à être utilisées en
normalisation, consultables aux adresses suivantes:
— ISO Online browsing platform: disponible à l’adresse https:// www .iso .org/ obp
— IEC Electropedia: disponible à l’adresse https:// www .electropedia .org/
— IEEE Standards Dictionary Online: disponible à l'adresse https:// dictionary .ieee .org
NOTE Pour d'autres termes et définitions relevant du domaine de l'ingénierie des systèmes et des logiciels,
voir l'ISO/IEC/IEEE 24765, qui est régulièrement publiée sous la forme d'un «instantané» du SEVOCAB (Systems
and software Engineering Vocabulary) et est accessible publiquement à l'adresse https:// www .computer .org/
sevocab.
3.1
test A/B
test dédoublé
approche de tests (3.131) statistique qui permet aux testeurs d'identifier parmi deux systèmes ou
composants celui qui présente les meilleures performances
3.2
test d'accessibilité
type de tests (3.131) d'utilisabilité servant à mesurer le degré selon lequel des utilisateurs présentant
la plus grande diversité possible de caractéristiques et d'aptitudes peuvent faire fonctionner un
élément de test (3.107)
3.3
fonction d'activation
fonction de transfert
formule associée à un nœud dans un réseau neuronal (3.50) qui détermine
la sortie du nœud (valeur d'activation [3.4]) à partir des entrées du neurone
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.4
valeur d'activation
sortie d'une fonction d'activation (3.3) d'un nœud dans un réseau neuronal
(3.50)
3.5
résultat réel
ensemble de comportements ou de conditions d'un élément de test (3.107), ou ensemble de conditions
des données associées ou de l'environnement de test (3.95), observés à la suite de l'exécution du test (3.99)
EXEMPLE Sorties sur un écran, sorties matérielles, modifications de données, rapports et messages de
communication envoyés.
3.6
système basé sur l'IA
système comprenant un ou plusieurs composants qui met en œuvre l'IA (3.7)
3.7
intelligence artificielle
IA
capacité d'un système d'ingénierie à acquérir, traiter et appliquer des connaissances et des aptitudes
3.8
système autonome
système capable de fonctionner pendant des périodes prolongées sans
intervention humaine
3.9
autonomie
aptitude d'un système à fonctionner pendant des périodes prolongées
sans intervention humaine
3.10
test dos-à-dos
test différentiel
approche de tests (3.131) dans laquelle une version alternative du système est utilisée pour générer des
résultats attendus (3.35) en vue de les comparer avec les mêmes entrées utilisées pour le test
EXEMPLE La version alternative peut être un système qui existe déjà, un système développé par une équipe
indépendante ou un système mis en œuvre à l'aide d'un langage de programmation différent.
3.11
choix de base
valeur de base
valeur de paramètre d'entrée qui est normalement sélectionnée pour sa nature représentative ou
typique du paramètre en question
3.12
analyse des valeurs limites
technique de conception de tests (3.94) basée sur une spécification, qui consiste à exercer les valeurs aux
limites de partitions d'équivalence (3.30)
3.13
test de branches
technique de conception d'un cas de test (3.85) basée sur une structure, qui consiste à exercer des
branches du flux de contrôle (3.22) de l'élément de test (3.107)
3.14
graphe de causes à effets
représentation graphique des règles de décision (3.25) entre des causes (entrées décrites sous forme de
conditions [3.21] booléennes) et des effets (sorties décrites sous forme d'expressions booléennes)
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.15
mise en graphe de causes à effets
technique de conception de tests (3.94) basée sur une spécification, qui consiste à exercer des règles de
décision (3.25) dans un graphe de causes à effets (3.14)
3.16
classification
fonction d'apprentissage automatique (3.44) qui prédit la classe de sortie
d'une entrée donnée
3.17
classification arborescente
modèle d'arborescence hiérarchique des données d'entrée d'un programme dans lequel les sorties sont
représentées par des classifications (aspects pertinents pour les tests) et des classes (valeurs d'entrée)
distinctes
3.18
tests combinatoires
techniques de conception de tests combinatoires
catégorie de technique de conception de tests (3.94) basées sur une spécification, qui consistent à exercer
des combinaisons de paires P-V (3.56)
EXEMPLE Test par paires (3.57), tests par choix de base (3.11).
3.19
test de compatibilité
type de tests (3.131) qui évalue dans quelle mesure un élément de test (3.107) peut fonctionner de
manière satisfaisante parallèlement à d'autres produits indépendants dans un environnement partagé
(coexistence) et, si nécessaire, échanger des informations avec d'autres systèmes ou composants
(interopérabilité)
3.20
critères de clôture
conditions dans lesquelles les activités de tests (3.131) sont considérées comme clôturées
3.21
condition
expression booléenne ne contenant aucun opérateur booléen
EXEMPLE «A < B» est une condition, contrairement à «A and B».
3.22
flux de contrôle
ordre de déroulement des opérations au cours de l'exécution d'un élément de test (3.107)
3.23
décision
type d'instruction dans lequel un choix entre deux résultats possibles ou plus contrôle l'ensemble des
actions qui en résultera
Note 1 à l'article: Les décisions typiques prennent la forme de sélections simples (par exemple, if-then-else)
permettant de décider à quel moment sortir des boucles (par exemple, boucle while), et d'instructions in case
(switch) (par exemple, case-1–2-3-.-N).
3.24
résultat de décision
résultat d'une décision (3.23) qui détermine quelle branche exécuter
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.25
règle de décision
combinaison de conditions (3.21) (également appelées causes) et d'actions (également appelées effets)
qui produisent un résultat spécifique dans un test par table de décision (3.27) et dans une mise en graphe
de causes à effets (3.15)
3.26
table de décision
représentation tabulaire des règles de décision (3.25) entre des causes (entrées décrites sous forme de
conditions [3.21] booléennes) et des effets (sorties décrites sous forme d'expressions booléennes)
3.27
test par table de décision
technique de conception de tests (3.94) basée sur une spécification, qui consiste à exercer des règles de
décision (3.25) dans une table de décision (3.26)
3.28
test des décisions
technique de conception de cas de test (3.85) basée sur une structure, qui consiste à exercer des résultats
de décisions (3.24) dans le flux de contrôle (3.22) de l'élément de test (3.107)
3.29
test dynamique
tests (3.131) au cours desquels un élément de test (3.107) est évalué par son exécution
3.30
partition d'équivalence
classe d'équivalence
classe d'entrées ou de sorties qui sont supposées être traitées de la même manière par l'élément de test
(3.107)
3.31
partitionnement en classes d'équivalence
technique de conception de tests (3.94) dans laquelle les cas de test (3.85) sont conçus pour exercer des
partitions d'équivalence (3.30) en utilisant un ou plusieurs membres représentatifs de chaque partition
3.32
estimation d'erreur
technique de conception de tests (3.94) dans laquelle les cas de test (3.85) sont dérivés d'après la
connaissance du testeur de défaillances passées ou d'après la connaissance générale de modes de
défaillance
Note 1 à l'article: Les connaissances pertinentes peuvent être acquises par expérience personnelle ou peuvent
être encapsulées, par exemple, dans une base de données de défauts ou dans une «taxonomie de défauts».
3.33
instruction exécutable
instruction qui, une fois compilée, est traduite en code objet qui sera exécuté de façon procédurale lors
de l'exécution de l'élément de test (3.107) et qui peut effectuer une action sur les données du programme
3.34
test exhaustif
approche de test (3.83) dans laquelle sont testées toutes les combinaisons de valeurs d'entrée et de
préconditions
Note 1 à l'article: Dans pratiquement toutes les situations majeures, les tests exhaustifs sont impossibles en
raison du nombre important de tests possibles.
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.35
résultat attendu
comportement attendu observable de l'élément de test (3.107) dans des conditions spécifiées, compte
tenu de sa spécification ou d'une autre source
3.36
test basé sur l'expérience
catégorie de techniques de conception de cas de test (3.85) qui s'appuient sur l'expérience des testeurs
pour générer des cas de test
EXEMPLE Estimation d'erreur (3.32).
Note 1 à l'article: Les tests basés sur l'expérience peuvent couvrir des concepts tels que les attaques-tests, les
tours (ou types d'usage) et les taxonomies de défauts qui ciblent des problèmes potentiels comme la sécurité, les
performances et d'autres domaines de la qualité.
3.37
test exploratoire
test basé sur l'expérience (3.36) dans lequel le testeur conçoit et exécute des tests spontanément à partir
de ses connaissances existantes, d'une exploration préalable de l'élément de test (3.107) (y compris des
résultats des tests précédents) et de « règles empiriques» heuristiques concernant les comportements
logiciels et les types de défaillance courants
Note 1 à l'article: Les tests exploratoires recherchent les propriétés cachées (y compris les comportements
cachés) qui, bien que sans doute inoffensives en soi, peuvent interférer avec d'autres propriétés du logiciel testé,
et donc représenter un risque de défaillance du logiciel.
3.38
test à données aléatoires
approche des test (3.131) logiciels dans laquelle d'importants volumes
de données aléatoires (ou pseudo-aléatoires), appelées fuzz en anglais, sont utilisés pour générer des
entrées pour l'élément de test (3.107)
3.39
incident
événement, ensemble d'événements, condition ou situation anormal(e) ou inattendu(e) survenant à
n'importe quel moment pendant le cycle de vie d'un projet, d'un produit, d'un service ou d'un système
3.40
rapport d'incident
documentation de la survenue, de la nature et de l'état d'un incident (3.39)
Note 1 à l'article: Les Rapports d'Incident sont également appelés rapports d'anomalie, rapports de défaillance,
rapports de défaut, rapports d'erreur, problèmes, rapports de problème et rapports de dysfonctionnement, entre
autres désignations.
3.41
mot-clé
un ou plusieurs mot(s) utilisé(s) comme référence à un ensemble
spécifique d'actions appelées à être effectuées au cours de l'exécution d'un ou de plusieurs cas de test
(3.85)
Note 1 à l'article: Les actions comprennent les interactions avec l'interface utilisateur au cours du test, de la
vérification et des actions spécifiquement exécutées pour configurer un scénario de test (3.123).
Note 2 à l'article: Les mots-clés sont nommés avec au moins un verbe.
Note 3 à l'article: Des mots-clés composites peuvent être construits sur la base d'autres mots-clés.
3.42
tests axés sur des mots-clés
tests (3.131) qui utilisent des cas de test (3.85) composés de mots-clés (3.41)
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.43
test de charge
type de test de la performance (3.58) effectué pour évaluer le comportement d'un élément de test (3.107)
dans des conditions anticipées de charges variables, généralement dans des conditions anticipées de
faible utilisation, d'utilisation typique et de pic d'utilisation
3.44
apprentissage automatique
ML
procédé qui utilise des techniques de calcul pour permettre à des systèmes d'apprendre à partir de
données ou de l'expérience
3.45
test de maintenabilité
type de test (3.130) réalisé afin d'évaluer le degré d'efficacité et d'efficience avec lequel un élément de
test (3.107) peut être modifié
3.46
test manuel
test effectué par l'homme, par saisie d'informations dans un élément de test (3.107) et vérification des
résultats
Note 1 à l'article: Les tests (3.131) automatisés utilisent des outils, des robots (3.70) et d'autres moteurs d'exécution
de tests (3.100) pour réaliser des tests. Les tests manuels n'emploient pas ces moyens.
3.47
test MC/DC
test des conditions/décisions modifiées
technique de conception de cas de test (3.85) basée sur une structure, qui consiste à démontrer qu'une
seule condition (3.21) booléenne dans une décision (3.23) peut affecter indépendamment le résultat de
la décision
3.48
relation métamorphique
description de la manière dont des modifications apportées aux entrées d'un cas de test (3.85) affectent
les sorties attendues en fonction du comportement exigé d'un élément de test (3.107)
3.49
test métamorphique
technique de conception de cas de test (3.85) basée sur une spécification, qui consiste à générer des cas
de test à partir de cas de test et de relations métamorphiques (3.48) existants
3.50
réseau neuronal
réseau de neurones artificiels
réseau d'éléments de traitement primitifs connectés par des liaisons
pondérées avec des poids ajustables, dans lequel chaque élément produit une valeur en appliquant à
ses valeurs d'entrée une fonction non linéaire, et la transmet à d'autres éléments ou la présente sous la
forme d'une valeur de sortie
Note 1 à l'article: Si certains réseaux neuronaux sont conçus pour simuler le fonctionnement des neurones dans
le système nerveux, la plupart sont utilisés dans l'intelligence artificielle comme moyen de matérialiser le modèle
connexionniste.
Note 2 à l'article: Une fonction seuil, une fonction sigmoïde et une fonction polynomiale sont des exemples de
fonctions non linéaires.
[SOURCE: ISO/IEC 2382:2015, 2120625, modifiée — Le terme toléré «neural net» a été supprimé de
l'article anglais; les Notes 3 à 5 à l'article ont été supprimées.]
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.51
couverture de neurones
proportion de neurones activés divisés par le nombre total de neurones
dans le réseau neuronal (3.50) (normalement exprimé en pourcentage) pour un ensemble de tests
Note 1 à l'article: Un neurone est considéré comme activé si sa valeur d'activation (3.4) est supérieure à zéro.
3.52
système non déterministe
système qui, à partir d'un ensemble d'entrées et d'un état initial donnés, ne produit pas toujours le
même ensemble de sorties et le même état final
3.53
pratiques de tests organisationnelles
documentation qui exprime les approches ou méthodes recommandées pour les tests (3.131) à effectuer
au sein d'une organisation, et qui fournit des détails sur la manière dont le test doit être effectué
Note 1 à l'article: Les pratiques de tests organisationnelles sont alignées sur la politique de tests organisationnelle
(3.118).
Note 2 à l'article: Une organisation peut avoir plusieurs documents relatifs aux pratiques de tests
organisationnelles pour couvrir des contextes sensiblement différents, par exemple un document pour les
applications mobiles et un document pour les systèmes critiques pour la sûreté (3.71).
Note 3 à l'article: Les pratiques de tests organisationnelles peuvent incorporer le contexte de la politique de tests
lorsqu'aucune politique de tests distincte n'est disponible.
3.54
processus de test au niveau organisationnel
processus de test (3.121) servant à développer et gérer des spécifications de tests organisationnelles
(3.55)
3.55
spécification de tests organisationnelle
document qui fournit des informations sur les tests (3.131) pour une organisation, c'est-à-dire des
informations qui ne sont pas spécifiques à un projet
EXEMPLE La politique de tests organisationnelle (3.118) et les pratiques de tests organisationnelles (3.53) sont
les exemples les plus courants de spécifications de tests organisationnelles.
3.56
paire P-V
paire paramètre-valeur
combinaison d'un paramètre d'élément de test (3.107) et d'une valeur affectée à ce paramètre, qui est
utilisée comme élément de couverture de test (3.90)
3.57
test par paires
technique de conception de tests (3.94) boîte noire dans laquelle les cas de test (3.85) sont conçus pour
exécuter toutes les combinaisons possibles de chaque paire de paramètres d'entrée
Note 1 à l'article: Le test par paires est la forme de test combinatoire (3.18) la plus populaire.
3.58
test de performance
type de tests (3.131) effectué pour évaluer dans quelle mesure un élément de test (3.107) accomplit des
fonctions désignées dans des contraintes de temps et d'autres ressources données
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.59
test de portabilité
type de tests (3.131) effectué dans le but d'évaluer le degré de facilité avec lequel un élément de test
(3.107) peut être transféré d'un environnement matériel ou logiciel à un autre, ainsi que le niveau de
modification nécessaire pour l'exécuter dans divers types d'environnement
3.60
test procédural
type de tests (3.131) d'aptitude fonctionnelle effectué afin d'évaluer si des instructions de procédure
concernant les interactions avec un élément de test (3.107) ou l'utilisation de ses sorties satisfont aux
exigences de l'utilisateur et répondent aux finalités de son utilisation
3.61
risque produit
risque qu'un produit soit défectueux du point de vue de certains aspects spécifiques de sa fonction, de
sa qualité ou de sa structure
3.62
risque projet
risque associé à la gestion d'un projet
EXEMPLE Manque de personnel, délais stricts, exigences changeantes.
3.63
test aléatoire
technique de conception de tests (3.94) basée sur une spécification, qui consiste à générer des cas de test
(3.85) pour exercer des entrées d'un élément de test (3.107) sélectionnées de manière aléatoire
3.64
test de régression
tests (3.131) effectués à la suite de modifications introduites dans un élément de test (3.107) ou dans son
environnement d'exploitation, afin d'identifier si des défaillances se produisent au niveau des parties
non modifiées de l'élément de test
Note 1 à l'article: Le test de régression est différent du re-test (3.68) en ce sens qu'il ne vérifie pas que la
modification fonctionne correctement, mais que d'autres parties du système n'ont pas été accidentellement
affectées par le changement.
Note 2 à l'article: L'adéquation d'un ensemble de cas de test (3.85) de régression dépend de l'élément testé et des
modifications apportées à cet élément ou à son environnement d'exploitation.
3.65
norme réglementaire
norme promulguée par un organisme réglementaire
3.66
test de fiabilité
type de tests (3.131) effectué pour évaluer la capacité d'un élément de test (3.107) à exécuter ses
fonctions requises, notamment en évaluant la fréquence d'occurrence des défaillances, lorsqu'il est
utilisé dans les conditions indiquées pendant une durée spécifiée
3.67
test basé sur les exigences
technique de conception de cas de test (3.85) basée sur une spécification, qui consiste à exercer des
exigences atomiques
EXEMPLE Une exigence atomique peut être «Le système doit collecter et stocker la date de naissance de tous
les utilisateurs inscrits.»
© ISO/IEC 2022 – Tous droits réservés
© IEEE 2022 – Tous droits réservés
3.68
re-test
test de confirmation
tests (3.131) effectués pour vérifier que des modifications introduites dans le but de corriger un défaut
ont effectivement permis de supprimer le défaut
Note 1 à l'article: Un re-test, lorsqu'il est effectué, est souvent complété également par un test de régression (3.64)
pour aider à s'assurer que d'autres parties non modifiées de l'élément de test (3.107) n'ont pas été accidentellement
affectées par les modifications.
3.69
test basé sur les risques
tests (3.131) au cours desquels la gestion, la sélection, la priorisation et l'utilisation des activités
et ressources de test sont consciemment fondées sur les types et niveaux correspondants du risque
analysé
3.70
robot
mécanisme act
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...