CEN/CLC ISO/IEC/TR 24029-1:2023
(Main)Artificial Intelligence (AI) - Assessment of the robustness of neural networks - Part 1: Overview (ISO/IEC TR 24029-1:2021)
Artificial Intelligence (AI) - Assessment of the robustness of neural networks - Part 1: Overview (ISO/IEC TR 24029-1:2021)
This document provides background about existing methods to assess the robustness of neural networks.
Informationstechnik - Künstliche Intelligenz - Bewertung der Robustheit neuronaler Netze (ISO/IEC TR 24029-1:2021)
Intelligence artificielle (IA) - Évaluation de la robustesse des réseaux de neurones - Partie 1: Vue d'ensemble (ISO/IEC TR 24029-1:2021)
Umetna inteligenca (UI) - Ocenjevanje robustnosti nevronskih omrežij - 1. del: Pregled (ISO/IEC TR 24029-1:2021)
Ta dokument podaja dodatne informacije o obstoječih metodah za ocenjevanje robustnosti nevronskih omrežij.
General Information
Standards Content (Sample)
SLOVENSKI STANDARD
01-maj-2024
Umetna inteligenca (UI) - Ocenjevanje robustnosti nevronskih omrežij - 1. del:
Pregled (ISO/IEC TR 24029-1:2021)
Artificial Intelligence (AI) - Assessment of the robustness of neural networks - Part 1:
Overview (ISO/IEC TR 24029-1:2021)
Informationstechnik - Künstliche Intelligenz - Bewertung der Robustheit neuronaler Netze
(ISO/IEC TR 24029-1:2021)
Intelligence artificielle (IA) - Évaluation de la robustesse des réseaux de neurones -
Partie 1: Vue d'ensemble (ISO/IEC TR 24029-1:2021)
Ta slovenski standard je istoveten z: CEN/CLC ISO/IEC/TR 24029-1:2023
ICS:
35.020 Informacijska tehnika in Information technology (IT) in
tehnologija na splošno general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
TECHNICAL REPORT CEN/CLC ISO/IEC/TR
24029-1
RAPPORT TECHNIQUE
TECHNISCHER REPORT
December 2023
ICS 35.020
English version
Artificial Intelligence (AI) - Assessment of the robustness
of neural networks - Part 1: Overview (ISO/IEC TR 24029-
1:2021)
Intelligence artificielle (IA) - Évaluation de la Informationstechnik - Künstliche Intelligenz -
robustesse des réseaux de neurones - Partie 1: Vue Bewertung der Robustheit neuronaler Netze (ISO/IEC
d'ensemble (ISO/IEC TR 24029-1:2021) TR 24029-1:2021)
This Technical Report was approved by CEN on 17 December 2023. It has been drawn up by the Technical Committee
CEN/CLC/JTC 21.
CEN and CENELEC members are the national standards bodies and national electrotechnical committees of Austria, Belgium,
Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy,
Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of North Macedonia, Romania, Serbia,
Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and United Kingdom.
CEN-CENELEC Management Centre:
Rue de la Science 23, B-1040 Brussels
© 2023 CEN/CENELEC All rights of exploitation in any form and by any means
Ref. No. CEN/CLC ISO/IEC/TR 24029-1:2023 E
reserved worldwide for CEN national Members and for
CENELEC Members.
Contents Page
European foreword . 3
European foreword
The text of ISO/IEC TR 24029-1:2021 has been prepared by Technical Committee ISO/IEC JTC 1
"Information technology” of the International Organization for Standardization (ISO) and has been
taken over as CEN/CLC ISO/IEC/TR 24029-1:2023 by Technical Committee CEN-CENELEC/ JTC 21
“Artificial Intelligence” the secretariat of which is held by DS.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN-CENELEC shall not be held responsible for identifying any or all such patent rights.
Any feedback and questions on this document should be directed to the users’ national standards body.
A complete listing of these bodies can be found on the CEN and CENELEC websites.
Endorsement notice
The text of ISO/IEC TR 24029-1:2021 has been approved by CEN-CENELEC as
TECHNICAL ISO/IEC TR
REPORT 24029-1
First edition
2021-03
Artificial Intelligence (AI) —
Assessment of the robustness of
neural networks —
Part 1:
Overview
Reference number
ISO/IEC TR 24029-1:2021(E)
©
ISO/IEC 2021
ISO/IEC TR 24029-1:2021(E)
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
ISO/IEC TR 24029-1:2021(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope .1
2 Normative references .1
3 Terms and definitions .1
4 Overview of the existing methods to assess the robustness of neural networks .3
4.1 General . 3
4.1.1 Robustness concept . 3
4.1.2 Typical workflow to assess robustness . 3
4.2 Classification of methods . 6
5 Statistical methods .7
5.1 General . 7
5.2 Robustness metrics available using statistical methods . 8
5.2.1 General. 8
5.2.2 Examples of performance measures for interpolation . 8
5.2.3 Examples of performance measures for classification . 9
5.2.4 Other measures .13
5.3 Statistical methods to measure robustness of a neural network .14
5.3.1 General.14
5.3.2 Contrastive measures .14
6 Formal methods .14
6.1 General .14
6.2 Robustness goal achievable using formal methods.15
6.2.1 General.15
6.2.2 Interpolation stability .15
6.2.3 Maximum stable space for perturbation resistance.15
6.3 Conduct the testing using formal methods .16
6.3.1 Using uncertainty analysis to prove interpolation stability .16
6.3.2 Using solver to prove a maximum stable space property .16
6.3.3 Using optimization techniques to prove a maximum stable space property .16
6.3.4 Using abstract interpretation to prove a maximum stable space property .17
7 Empirical methods .17
7.1 General .17
7.2 Field trials .17
7.3 A posteriori testing .18
7.4 Benchmarking of neural networks .19
Annex A (informative) Data perturbation .20
Annex B (informative) Principle of abstract interpretation .25
Bibliography .26
© ISO/IEC 2021 – All rights reserved iii
ISO/IEC TR 24029-1:2021(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence.
A list of all parts in the ISO/IEC 24029 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv © ISO/IEC 2021 – All rights reserved
ISO/IEC TR 24029-1:2021(E)
Int
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.