prEN ISO/IEC 23282
(Main)Artificial Intelligence - Evaluation methods for accurate natural language processing systems (ISO/IEC DIS 23282:2026)
- Abstract
This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
- Status
- Not Published
- Publication Date
- 21-Feb-2028
- Technical Committee
- CEN/CLC/JTC 21 - Artificial Intelligence
- Drafting Committee
- CEN/CLC/JTC 21/WG 3 - Engineering aspects
- Current Stage
- 4020 - Submission to enquiry - Enquiry
- Start Date
- 03-Sep-2026
- Completion Date
- 03-Sep-2026
prEN ISO/IEC 23282 is the CEN reference for evaluating artificial intelligence (AI) natural language processing (NLP) systems. It specifies how to measure the quality of system outputs so that users can judge functional suitability, and it gives guidance on selecting, implementing and interpreting those evaluation methods. It is aimed at people who need to compare, validate or report on NLP system results in a reproducible way.
What does prEN ISO/IEC 23282 specify?
prEN ISO/IEC 23282 specifies evaluation methods for NLP systems, including quantitative metrics and human evaluation protocols, and it sets requirements for how those methods are implemented and supported by technical resources. The document is organized around generic methods first, then task prototypes, then task-by-task application, and finally requirements on resources.
| Clause | What it covers |
|---|---|
| 1-3 | Scope, normative reference, and terms and definitions |
| 4 | Abbreviations used throughout the document |
| 5 | Overall approach to NLP evaluation and how the document is intended to be used |
| 6 | Evaluation methods: automated metrics, human protocols, and multi-dimensional analysis criteria |
| 7 | Shared evaluation methods per task prototype |
| 8 | Evaluation methods per task |
| 9 | Evaluation of multi-component NLP systems |
| 10 | Requirements on test data and computing resources |
The document is task-based, not metric-based alone. That matters in practice because the same metric, such as F1-score, is used differently depending on whether the task is categorisation, segmentation, identification, or sequence generation.
What are the key requirements of prEN ISO/IEC 23282?
prEN ISO/IEC 23282 requires evaluators to choose methods that fit the task, define how those methods are applied, and report enough detail to make results comparable and reproducible. It also requires the evaluation setup to be technically sound, especially for test data and computational resources.
Clause 6 - choose and define the evaluation method
Clause 6 describes both automated metrics and human evaluation protocols. Automated methods include accuracy, precision, recall, F1-score, edit distance metrics, BLEU, ROUGE, BERTscore, COMET, EER, DCF, minDCF and actDCF; human methods include MOS, DRT, MRT and MUSHRA. In practice, this means the evaluator must pick a method that matches the output type, such as translation, recognition, clustering or speech quality, rather than assuming one generic score fits every NLP task.
Clause 6 also requires the metric computation details that affect reproducibility to be specified. That matters because two implementations can produce different numbers if they handle matching, averaging, thresholds or reference handling differently.
Clause 6.4 - use multi-dimensional analysis when a single score is not enough
Clause 6.4 defines quality criteria and error criteria for multi-dimensional analysis, with both generic and task-specific criteria. In practice, this lets teams look beyond a single headline score and examine specific strengths and weaknesses, for example by checking which error types are responsible for a result.
Clause 7 - apply metrics according to the task prototype
Clause 7 groups tasks into prototypes such as categorisation, identification, segmentation, sequence labelling, per-element identification, set creation, ranking, sequence generation and structure generation. This matters because the same metric is not used in the same way for every prototype. For example, F1-score may be applied differently to segmentation than to identification, and the document gives the extra rules needed for consistent use.
Clause 7 also shows which generic methods from ISO/IEC DIS 4213 are reused, especially for categorisation and identification. For an evaluator, this means the standard is not only about new NLP metrics; it also tells you when the generic AI metrics can be used and what extra NLP-specific rules must be followed.
Clause 8 - match methods to real NLP tasks
Clause 8 maps evaluation methods to concrete NLP tasks such as language identification, tokenization, lemmatization, part-of-speech tagging, named entity recognition, coreference resolution, machine translation, automatic speech recognition, optical character recognition and question answering. It also covers document recommendation, clustering, semantic tasks, text generation, and conversions among modalities.
This matters because the clause tells you which methods are relevant for a task and how to interpret them there. For example, the document uses BLEU, ROUGE, BERTscore and COMET for machine translation and other generation tasks, while CER, WER and PER are used for speech and transcription-related tasks.
Clause 9 - evaluate combined systems at component and system level
Clause 9 gives three combination schemes for multi-component NLP systems: multi-task, pipeline and helper task. In practice, this means an end-to-end NLP system can be assessed at component level, at combined-system level, or both, depending on how the system is built and whether intermediate outputs are available.
This is important for real deployments because many NLP products are chains of tasks, such as recognition followed by translation or post-processing. Clause 9 helps evaluators decide whether to score each step separately, score the whole chain, or do both to expose interaction effects.
Clause 10 - control the evaluation resources
Clause 10 requires test data to be clearly identified and independent from data used in design and development. It also says multilingual test data shall include at least 3 languages, or 2 if the system only supports 2, and each included language shall be evaluated separately.
That matters because data leakage or poor language coverage can make results look better than they really are. Clause 10 also addresses computing resources, so the evaluation setup can be reproduced rather than depending on hidden implementation conditions.
What terms does prEN ISO/IEC 23282 define?
prEN ISO/IEC 23282 defines the core vocabulary used to describe NLP evaluation. These terms matter because the document ties its methods to functional suitability, task structure and data quality.
- Artificial intelligence (AI) - research and development of mechanisms and applications of AI systems.
- Natural language processing (NLP) - the discipline concerned with how systems acquire, process and interpret natural language, and also information processing based on natural language understanding or natural language generation.
- Natural language understanding (NLU) - extracting information from text or speech and producing a description of what it represents.
- Natural language generation (NLG) - converting data carrying semantics into natural language.
- Functional suitability - the capability of a product to provide functions that meet intended users’ needs under specified conditions.
- Functional correctness - the degree to which a product or system provides the correct results with the needed degree of precision.
- Test data - data used to assess the performance of a final model.
- Independent - a dataset property meaning no characteristic is shared with another dataset other than the same statistical distribution; lack of independence can cause data leakage.
Who uses prEN ISO/IEC 23282?
prEN ISO/IEC 23282 is used by people who need to evaluate NLP systems in a structured and comparable way. That includes system developers, test and validation engineers, quality managers, technical buyers and researchers working on AI evaluation.
It is especially relevant in information technology, speech technology, machine translation, search and retrieval, document analysis, dialogue systems and multilingual processing. It also helps teams that need to report evaluation results for procurement, benchmarking, audits or internal acceptance testing.
Which standards are used with prEN ISO/IEC 23282?
prEN ISO/IEC 23282 is designed to be used with ISO/IEC DIS 4213, which provides the generic performance-measurement methods for AI classification, regression, clustering and recommendation tasks. This document builds on those methods and adds NLP-specific rules where generic AI metrics need extra specification.
The overview also says prEN ISO/IEC 23282 should be read with ISO/IEC TR 23281, which supplies the task hierarchy, task definitions and combination schemes used throughout Clauses 7 to 9. The definitions and introduction also draw on ISO/IEC 22989:2022, ISO/IEC 25059:2023 and ISO/IEC 25010:2023 for AI terminology and for functional correctness and functional suitability.
What does the prEN ISO/IEC 23282 document contain?
prEN ISO/IEC 23282 contains mathematical definitions and implementation guidance for automated metrics, plus structured human evaluation protocols. It also contains task-by-task applicability rules, calibration guidance and interpretation notes so that results can be compared across systems and implementations.
The document includes metric families for matching, distance, clustering, ranking, generation, speech and translation, together with the conditions under which they are applied. It also gives prototype-specific rules for tasks such as segmentation, identification and sequence generation, and it addresses special cases like combined NLP systems and multilingual test data.
In practice, the document is used for evaluation planning, benchmarking, reporting and acceptance of NLP systems. It is written to reduce ambiguity in what a score actually means, rather than to prescribe one universal evaluation method for every NLP problem.
Relations
- Effective Date
- 28-Jan-2026
- Effective Date
- 28-Jan-2026
- Effective Date
- 28-Jan-2026
Get Certified
Connect with accredited certification bodies for this standard
BSI Group
BSI (British Standards Institution) is the business standards company that helps organizations make excellence a habit.
NYCE
Mexican standards and certification body.
Sponsored listings
Frequently Asked Questions
prEN ISO/IEC 23282 is a draft published by the European Committee for Standardization (CEN). Its full title is "Artificial Intelligence - Evaluation methods for accurate natural language processing systems (ISO/IEC DIS 23282:2026)". This standard covers: This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
prEN ISO/IEC 23282 is classified under the following ICS (International Classification for Standards) categories: 35.240.01 - Application of information technology in general. The ICS classification helps identify the subject area and facilitates finding related standards.
prEN ISO/IEC 23282 has the following relationships with other standards: It is inter standard links to EN ISO/IEC 24760-3:2022, EN ISO/IEC 24760-2:2022, EN ISO/IEC 29146:2023. Understanding these relationships helps ensure you are using the most current and applicable version of the standard.
prEN ISO/IEC 23282 is associated with the following European legislation: EU Directives/Regulations: 2024/1689; Standardization Mandates: M/593, M/593, M/613, M/XXX AI, M/XXX AI. When a standard is cited in the Official Journal of the European Union, products manufactured in conformity with it benefit from a presumption of conformity with the essential requirements of the corresponding EU directive or regulation.
prEN ISO/IEC 23282 is available in PDF format for immediate download after purchase. The document can be added to your cart and obtained through the secure checkout process. Digital delivery ensures instant access to the complete standard document.
Standards Content (Sample)
SLOVENSKI STANDARD
01-november-2026
Umetna inteligenca (UI) - Metode vrednotenja za natančne sisteme za obdelavo
naravnega jezika (ISO/IEC DIS 23282:2026)
Artificial Intelligence - Evaluation methods for accurate natural language processing
systems (ISO/IEC DIS 23282:2026)
Evaluationsmethoden für die Genauigkeit von Systemen zur Verarbeitung natürlicher
Sprache (ISO/IEC DIS 23282:2026)
Intelligence artificielle - Méthodes d’évaluation pour des systèmes de traitement
automatique du langage naturel performants (ISO/IEC DIS 23282:2026)
Ta slovenski standard je istoveten z: prEN ISO/IEC 23282
ICS:
35.240.01 Uporabniške rešitve Application of information
informacijske tehnike in technology in general
tehnologije na splošno
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
DRAFT
International
Standard
ISO/IEC DIS 23282
ISO/IEC JTC 1/SC 42
Artificial Intelligence — Evaluation
Secretariat: ANSI
methods for accurate natural
Voting begins on:
language processing systems
2026-09-02
Intelligence artificielle — Méthodes d’évaluation pour des
Voting terminates on:
systèmes de traitement automatique du langage naturel
2026-11-25
performants
ICS: 35.240.01
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENTS AND APPROVAL. IT
IS THEREFORE SUBJECT TO CHANGE
Member bodies are requested to consult relevant national interests in ISO/TC AND MAY NOT BE REFERRED TO AS AN
INTERNATIONAL STANDARD UNTIL
37 before casting their ballot to the e-Balloting application.
PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
This document has not been edited by the ISO Central Secretariat.
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
ISO/CEN PARALLEL PROCESSING
WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS.
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION.
Reference number
© ISO/IEC 2026
ISO/IEC DIS 23282:2026(en)
DRAFT
ISO/IEC DIS 23282:2026(en)
International
Standard
ISO/IEC DIS 23282
ISO/IEC JTC 1/SC 42
Artificial Intelligence — Evaluation
Secretariat: ANSI
methods for accurate natural
Voting begins on:
language processing systems
Intelligence artificielle — Méthodes d’évaluation pour des
Voting terminates on:
systèmes de traitement automatique du langage naturel
performants
ICS: 35.240.01
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENTS AND APPROVAL. IT
IS THEREFORE SUBJECT TO CHANGE
Member bodies are requested to consult relevant national interests in ISO/TC AND MAY NOT BE REFERRED TO AS AN
INTERNATIONAL STANDARD UNTIL
37 before casting their ballot to the e-Balloting application.
PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
This document has not been edited by the ISO Central Secretariat.
BEING ACCEPTABLE FOR INDUSTRIAL,
© ISO/IEC 2026
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
STANDARDS MAY ON OCCASION HAVE TO
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
BE CONSIDERED IN THE LIGHT OF THEIR
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
POTENTIAL TO BECOME STANDARDS TO
ISO/CEN PARALLEL PROCESSING
WHICH REFERENCE MAY BE MADE IN
or ISO’s member body in the country of the requester.
NATIONAL REGULATIONS.
ISO copyright office
RECIPIENTS OF THIS DRAFT ARE INVITED
CP 401 • Ch. de Blandonnet 8
TO SUBMIT, WITH THEIR COMMENTS,
CH-1214 Vernier, Geneva
NOTIFICATION OF ANY RELEVANT PATENT
Phone: +41 22 749 01 11
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION.
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland Reference number
© ISO/IEC 2026
ISO/IEC DIS 23282:2026(en)
© ISO/IEC 2026 – All rights reserved
ii
ISO/IEC DIS 23282:2026(en)
Contents Page
Foreword .vii
Introduction .viii
1 Scope .1
2 Normative references .1
3 Terms and definitions .1
4 Abbreviations .4
5 Overview .5
5.1 General .5
5.2 Types of evaluation methods .5
5.3 Contents of this document .6
6 Description of evaluation methods .7
6.1 General .7
6.2 Automated metrics .7
6.2.1 General .7
6.2.2 Accuracy, precision and recall, F1 score .8
6.2.3 Precision @ K, recall @ K, F1 @ K .8
6.2.4 Purity, inverse purity .8
6.2.5 Edit distance metrics .8
6.2.6 Entropy, mutual information and normalized mutual information .9
6.2.7 N-gram and subsequence matching metrics .10
6.2.8 Embeddings-based metrics . 13
6.2.9 Learned metrics .14
6.2.10 EER (equal error rate) .16
6.2.11 Jaccard coefficient, Rand score, adjusted Rand score, Fowlkes-Mallows index .16
6.2.12 Silhouette coefficient, silhouette index, Davies-Bouldin index, Calinski-
Harabasz index .16
6.2.13 B .17
6.2.14 AER (alignment error rate) .18
6.2.15 METEOR .18
6.2.16 Perplexity .18
6.2.17 DCF, minDCF and actDCF .18
6.3 Human evaluation protocols .19
6.3.1 General .19
6.3.2 Types of human evaluation protocols. 20
6.3.3 MOS (mean opinion score) .21
6.3.4 DRT (diagnostic rhyme test) .21
6.3.5 MRT (modified rhyme test) . . .21
6.3.6 MUSHRA . 22
6.4 Multi-dimensional analysis criteria . 22
6.4.1 General . 22
6.4.2 Quality criteria . 22
6.4.3 Error criteria. 23
7 Shared evaluation methods per task prototype .24
7.1 General .24
7.2 Categorisation .24
7.2.1 General .24
7.2.2 Automated metric: accuracy .24
7.2.3 Automated metrics: precision, recall, F1-score .24
7.3 Identification . 25
7.3.1 General . 25
7.3.2 Automated metric: accuracy . 25
7.3.3 Automated metrics: precision, recall, F1-score . 26
© ISO/IEC 2026 – All rights reserved
iii
ISO/IEC DIS 23282:2026(en)
7.3.4 Automated metric: EER . 26
7.3.5 Automated metrics: DCF, minDCF, actDCF . 26
7.4 Segmentation . 26
7.4.1 General . 26
7.4.2 Automated metrics: precision, recall, F1-score .27
7.5 Sequence labelling . 28
7.5.1 General . 28
7.5.2 Automated metric: accuracy . 28
7.5.3 Automated metrics: precision, recall, F1-score . 28
7.6 Per-element identification . 29
7.6.1 General . 29
7.6.2 Automated metric: accuracy . 29
7.6.3 Automated metrics: precision, recall, F1-score . 29
7.7 Set creation . 30
7.7.1 General . 30
7.7.2 Automated metrics: silhouette coefficient, silhouette index, Davies-Bouldin
index, Calinski-Harabasz index . 30
7.7.3 Automated metrics: purity, inverse purity . 30
7.7.4 Automated metrics: Jaccard coefficient, Rand score, adjusted Rand score,
Fowlkes-Mallows index . 30
7.7.5 Automated metrics: mutual information, normalized mutual information .31
7.7.6 Automated metric: MUC.31
7.7.7 Automated metric: B .32
7.8 Ranking .32
7.8.1 General .32
7.8.2 Automated metrics: precision @ K, recall @ K, F1 @ K.32
7.9 Sequence generation.32
7.9.1 General .32
7.9.2 Automated metric: BLEU . 33
7.9.3 Automated metrics: ROUGE-1, ROUGE-2, ROUGE-n . 33
7.9.4 Automated metric: ROUGE-L . 33
7.9.5 Automated metric: perplexity . 33
7.9.6 Automated metrics: CER, WER, PER . 34
7.9.7 Automated metrics: BERTscore . 34
7.9.8 Automated metrics: COMET regression metric, COMET ranking metric . 34
7.10 Structure generation . 34
8 Evaluation methods per task .35
8.1 General . 35
8.2 Tasks related to shallow analysis of natural language contents . 35
8.2.1 General . 35
8.2.2 Language identification . 35
8.2.3 Sentence boundary detection .37
8.2.4 Tokenization .37
8.2.5 Lemmatization . 38
8.2.6 Part-of-speech tagging . 39
8.2.7 Syntactic parsing . . 40
8.2.8 Morphological analysis . 40
8.2.9 Word sense disambiguation . 40
8.2.10 Coreference resolution .43
8.2.11 Anaphora resolution . 46
8.2.12 Discourse parsing .47
8.2.13 Document segmentation .47
8.2.14 Parallel corpora alignment . 48
8.2.15 Speech activity detection . 48
8.2.16 Utterance boundary detection . . 49
8.2.17 Phrase boundary detection . 49
8.2.18 Speaker change detection . 50
8.2.19 Speaker diarization . 50
© ISO/IEC 2026 – All rights reserved
iv
ISO/IEC DIS 23282:2026(en)
8.3 Tasks related to author identification and profiling . 50
8.3.1 General . 50
8.3.2 Authorship attribution . 50
8.3.3 Speaker recognition .52
8.3.4 Handwriting identification . 55
8.3.5 Style analysis .57
8.3.6 Speaker role identification .57
8.3.7 Speaker gender identification . 58
8.3.8 Impersonation detection .59
8.4 Tasks related to document analysis and mining .59
8.4.1 General .59
8.4.2 Topic analysis .59
8.4.3 Named entity recognition . 60
8.4.4 Entity linking .61
8.4.5 Relation extraction .62
8.4.6 Slot filling .62
8.4.7 Claim extraction and fact extraction .62
8.4.8 Sentiment analysis . 63
8.4.9 Emotion recognition . 64
8.4.10 User intent detection . 65
8.4.11 Author intention recognition . 66
8.4.12 Offensive speech detection .67
8.5 Tasks related to corpus-level analysis and mining . 68
8.5.1 General . 68
8.5.2 Document recommendation . 68
8.5.3 Content-based document search . 68
8.5.4 Semantic clustering . 69
8.5.5 Thematic clustering . 69
8.6 Tasks related to semantics and meaning .70
8.6.1 General .70
8.6.2 Automatic term extraction .70
8.6.3 Word sense induction.70
8.6.4 Term set expansion .71
8.6.5 Semantic relation extraction .71
8.6.6 Embeddings .71
8.6.7 Representation learning .71
8.6.8 Language modelling . 72
8.7 Tasks related with questions and dialogue . 72
8.7.1 General . 72
8.7.2 Question answering . 72
8.7.3 Generation of multimedia contents from prompts .74
8.8 Tasks involving the generation of linguistic contents .74
8.8.1 General .74
8.8.2 Automatic summarization .74
8.8.3 Machine translation . 75
8.8.4 Document denoising . 77
8.8.5 Language error correction . 78
8.8.6 Text normalization . 79
8.8.7 Sensitive data treatment . 79
8.8.8 Voice anonymization . 79
8.8.9 Description generation . 79
8.8.10 Vanilla text generation . 80
8.8.11 Constrained text generation . 82
8.8.12 Captioning . 84
8.8.13 Source code generation . 84
8.8.14 Source code documentation . 84
8.8.15 Paraphrase generation . 85
8.8.16 Question generation . 85
8.8.17 Distractor generation . 85
© ISO/IEC 2026 – All rights reserved
v
ISO/IEC DIS 23282:2026(en)
8.9 Tasks involving conversions among modalities . 85
8.9.1 General . 85
8.9.2 Automatic speech recognition . 86
8.9.3 Text-to-speech synthesis . 86
8.9.4 Optical character recognition . 88
8.9.5 Lip reading . 89
8.10 Transversal functionalities . 89
8.10.1 General . 89
8.10.2 Domain adaptation . 89
9 Evaluation methodology for multi-component NLP systems .90
10 Requirements on resources .91
10.1 General .91
10.2 Requirements on test data .91
10.3 Requirements on computing resources .91
Bibliography .92
© ISO/IEC 2026 – All rights reserved
vi
ISO/IEC DIS 23282:2026(en)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out through
ISO technical committees. Each member body interested in a subject for which a technical committee
has been established has the right to be represented on that committee. International organizations,
governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely
with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the different types
of ISO documents should be noted. This document was drafted in accordance with the editorial rules of the
ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of any patent
rights identified during the development of the document will be in the Introduction and/or on the ISO list of
patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment,
as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the
Technical Barriers to Trade (TBT) see the following URL: www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
© ISO/IEC 2026 – All rights reserved
vii
ISO/IEC DIS 23282:2026(en)
Introduction
This document focuses on the evaluation of natural language processing (NLP) systems, as regards their
[1]
functional suitability as defined in ISO/IEC 25059:2023 . This includes functional correctness (often
[1]
informally referred to as ‘performance’ by NLP practitioners, which ISO/IEC 25059:2023 , Annex C
discourages due to ambiguity, or 'accuracy' in some regulation, which can be inconsistent with technical
usage), as well as other quality characteristics such as functional adaptability.
While other standards exist on the evaluation of AI systems, such as ISO/IEC DIS 4213, they only contain
generic metrics, targeting the evaluation of some base AI tasks like classification, and those metrics are
either not sufficiently specified to be applied in turn on NLP systems or not applicable to part of these NLP
systems.
This document therefore builds on ISO/IEC DIS 4213 to provide the NLP-specific complements that are
necessary to enable the evaluation of NLP systems in particular. This includes additional specifications on
generic metrics used beyond NLP, as well as NLP-specific evaluation methods, encompassing both automated
and human evaluation.
© ISO/IEC 2026 – All rights reserved
viii
DRAFT International Standard ISO/IEC DIS 23282:2026(en)
Artificial Intelligence — Evaluation methods for accurate
natural language processing systems
1 Scope
This document specifies the evaluation of natural language processing systems, in the sense of measuring
the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation
methods for those systems, together with guidance on how to select, implement and interpret those
evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It
includes requirements on the implementation of the described metrics, and further requirements on the
technical resources involved in the evaluation process.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC DIS 4213, Information technology — Artificial Intelligence — Performance measurement for AI
classification, regression, clustering and recommendation tasks
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC
...