ETSI TR 103 910 V1.1.1 (2025-02)

Methods for Testing and Specification (MTS); AI Testing; Test Methodology and Test Specification for ML-based Systems

DTR/MTS-103910

General Information

• Status: Not Published
• Technical Committee: MTS - Methods for Testing & Specification
• Current Stage: 12 - Completion
• Due Date: 25-Feb-2025
• Completion Date: 14-Feb-2025

Standards Content (Sample)

TECHNICAL REPORT
Methods for Testing and Specification (MTS);
AI Testing;
Test Methodology and Test Specification for
ML-based Systems
2 ETSI TR 103 910 V1.1.1 (2025-02)

Reference
DTR/MTS-103910
Keywords
AI, ML, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from the
ETSI Search & Browse Standards application.
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format on ETSI deliver repository.
Users should be aware that the present document may be revised or have its status changed,
this information is available in the Milestones listing.
If you find errors in the present document, please send your comments to
the relevant service listed under Committee Support Staff.
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure (CVD) program.
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2025.
All rights reserved.
ETSI
3 ETSI TR 103 910 V1.1.1 (2025-02)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
Executive summary . 6
Introduction . 6
1 Scope . 8
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 8
3 Definition of terms, symbols and abbreviations . 16
3.1 Terms . 16
3.2 Symbols . 18
3.3 Abbreviations . 18
4 General conditions of testing ML-based systems . 19
4.1 Machine Learning. 19
4.2 Classification of ML methods . 20
4.3 ML-based systems and its integration . 21
4.4 Testing ML-based systems . 22
5 Challenges and specifics of testing ML-based systems . 23
5.1 Open context and technology . 23
5.2 Stochastic solution approach and deep learning . 23
5.3 Robustness issue and missing transparency of neural networks . 24
5.4 Need for fair decision making . 24
5.5 Fault and failure model for testing ML-based systems. 25
5.6 Verification vs. validation of ML-based systems . 26
6 Quality criteria addressed by testing ML-based systems . 27
6.1 General . 27
6.2 Model relevance . 28
6.2.1 General . 28
6.2.2 Criteria for model relevance . 29
6.2.3 Assessing model relevance . 30
6.2.3.1 General . 30
6.2.3.2 Assessing ML model methods . 30
6.2.3.3 Assessing ML model capabilities . 30
6.2.3.4 Assessing suitability for tasks . 31
6.2.3.5 Assessing application context adaptability . 32
6.2.3.6 Assessing accountability . 32
6.3 Correctness . 32
6.3.1 Criteria for correctness . 32
6.3.2 Assessing correctness . 33
6.4 Robustness . 34
6.4.1 Criteria for robustness . 34
6.4.2 Assessing robustness. 35
6.5 Avoidance of unwanted bias . 36
6.5.1 Criteria for avoidance of unwanted bias . 36
6.5.2 Assessing avoidance of unwanted bias . 36
6.6 Information security . 37
6.6.1 Criteria for information security . 37
6.6.2 Assessing information security . 38
6.7 Safeguards against exploitation of ML models . 39
6.7.1 General . 39
6.7.2 Criteria for safeguards against exploitation . 39
ETSI
4 ETSI TR 103 910 V1.1.1 (2025-02)
6.7.3 Assessing safeguards against exploitation . 40
6.8 Security from vulnerabilities . 40
6.8.1 General . 40
6.8.2 Criteria for security from vulnerabilities . 41
6.8.3 Assessing security from vulnerabilities . 42
6.9 Explainability . 42
6.9.1 Criteria for explainability . 42
6.9.1.1 General . 42
6.9.1.2 Consistency of information . 43
6.9.1.3 Clarity about ML model methods . 43
6.9.1.4 Human understandability . 43
6.9.1.5 Temporal continuity of explanations . 44
6.9.2 Assessing explainability . 44
7 Workflow integration, test methods and definition of test items . 44
7.1 General . 44
7.2 A workflow perspective for developing and operating ML-based systems. 45
7.3 Overview on test methods for testing ML-based systems . 47
7.4 Considerations in defining adequate test items for testing ML-based systems . 47
8 Detailed test item identification and definition of test activities within the workflow perspective . 48
8.1 General . 48
8.2 Test items of the bus
...