oSIST prEN ISO/IEC 23282:2026
(Main)Artificial Intelligence - Evaluation methods for accurate natural language processing systems (ISO/IEC DIS 23282:2026)
- Abstract
This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
- Status
- Not Published
- Public Enquiry End Date
- 17-Nov-2026
- Technical Committee
- UMI - Artificial intelligence
- Current Stage
- 4020 - Public enquire (PE) (Adopted Project)
- Start Date
- 17-Sep-2026
- Due Date
- 04-Feb-2027
oSIST prEN ISO/IEC 23282:2026 is the SIST publication of ISO/IEC DIS 23282:2026, a draft International Standard on evaluating natural language processing (NLP) systems. It specifies how to measure the quality of NLP outputs to judge functional suitability, and it covers both automated metrics and human evaluation methods. The document is for people who need to select, apply, report or compare NLP evaluation results, such as engineers, quality managers and buyers.
What does oSIST prEN ISO/IEC 23282:2026 specify?
oSIST prEN ISO/IEC 23282:2026 specifies evaluation methods for NLP systems and the way to use them so results can be compared and interpreted consistently. The focus is on measuring output quality in a way that supports functional suitability, not on prescribing one single metric for every use case.
The document is organized as follows:
- Clauses 1 to 3 set the scope, normative reference and terms and definitions.
- Clause 4 lists abbreviations used in the document.
- Clause 5 gives the overview and explains how the document relates to other AI evaluation work.
- Clause 6 describes evaluation methods, including automated metrics, human evaluation protocols and multi-dimensional analysis criteria.
- Clause 7 adapts shared evaluation methods to task prototypes such as categorisation, identification, segmentation, sequence labelling, ranking and sequence generation.
- Clause 8 maps evaluation methods to specific NLP tasks and explains when they apply.
- Clause 9 covers evaluation of multi-component NLP systems.
- Clause 10 sets requirements for resources such as test data and computing resources.
What are the key requirements of oSIST prEN ISO/IEC 23282:2026?
oSIST prEN ISO/IEC 23282:2026 requires evaluators to treat NLP evaluation as task-based and method-specific. The document does not ask users to apply every method everywhere. Instead, it asks them to choose a method that fits the task, then apply and report it in a way that avoids ambiguity.
Metric choice and reporting
Clause 6 defines a broad set of automated metrics and human evaluation protocols, and it states that the implementation details needed for reproducibility must be reported. That matters in practice because the same named metric can produce different results if the matching rules, averaging rules or handling of edge cases differ.
Clause 6 covers, among others, accuracy, precision, recall and F1-score, precision @ K, recall @ K and F1 @ K, edit distance metrics, BLEU, ROUGE, BERTscore, COMET, EER, DCF, minDCF, actDCF, MOS, DRT, MRT and MUSHRA. In practice, this means the evaluator must know whether the task is better measured by exact matching, sequence similarity, ranking quality, clustering quality, detection cost, or human judgement.
Clause 6 also introduces multi-dimensional analysis criteria for quality and error. These criteria are meant to complement a single score with finer-grained analysis, so teams can compare subsets of data or inspect particular error types rather than relying on one overall number.
Shared rules by task prototype
Clause 7 groups tasks into prototypes and gives shared evaluation rules for each prototype. The prototypes include categorisation, identification, segmentation, sequence labelling, per-element identification, set creation, ranking, sequence generation and structure generation.
This matters because a metric such as F1-score is not applied the same way in every setting. For example, clause 7 distinguishes between application rules and implementation rules, including how variables in a formula map to task elements, how to average over a test set, and how to handle ill-formatted outputs and edge cases. That is the part a team needs when it wants two implementations to produce the same score.
Task-specific evaluation
Clause 8 maps methods to concrete NLP tasks and adds task-specific interpretation guidance. It covers shallow analysis tasks such as language identification, sentence boundary detection, tokenization, lemmatization, part-of-speech tagging, syntactic parsing, morphological analysis, word sense disambiguation, coreference resolution, anaphora resolution, discourse parsing, document segmentation, parallel corpora alignment, speech activity detection, utterance boundary detection, phrase boundary detection, speaker change detection and speaker diarization.
It also covers author identification and profiling, document analysis and mining, corpus-level analysis and mining, semantics and meaning, questions and dialogue, generation of linguistic contents, conversions among modalities, and transversal functionalities such as domain adaptation. In practice, this is the part that tells a user which metrics are relevant for a specific NLP task, and what aspect of the task each metric is measuring.
A few representative examples show the kind of guidance it gives:
- Language identification uses accuracy, precision, recall, F1-score, EER and DCF-based measures.
- Named entity recognition and segmentation tasks use precision, recall and F1-score with task-specific matching rules.
- Coreference resolution uses metrics such as MUC and B, and the document also names CEAF, MELA and BLANC.
- Machine translation uses BLEU, BERTscore, COMET and human direct assessment protocols.
- ASR, text-to-speech synthesis and OCR use CER, WER and PER, with human protocols also listed for text-to-speech synthesis.
Combined systems and resources
Clause 9 addresses systems built from multiple components. It distinguishes multi-task systems, pipeline systems and helper-task systems, and it recommends evaluating components separately and, where useful, also evaluating the combined system.
This matters because end-to-end score alone can hide where errors come from. A pipeline may look weak because one component feeds bad output to the next, or it may look stronger because later processing corrects earlier errors. Clause 9 explains when component-level evaluation is enough and when the combination itself should also be scored.
Clause 10 then sets requirements on test data and computing resources. Test data used for evaluation must be clearly identified and independent of data used for design and development. For multilingual NLP systems, the test data must contain at least 3 languages, or 2 if the system only supports 2, and each included language must be evaluated separately. Random seeds and other resource conditions that can affect reproducibility also need to be controlled and reported.
What terms does oSIST prEN ISO/IEC 23282:2026 define?
oSIST prEN ISO/IEC 23282:2026 uses a small set of core terms that frame the evaluation problem.
- Artificial intelligence (AI) - the discipline of researching and developing mechanisms and applications of AI systems.
- Natural language processing (NLP) - a discipline concerned with how systems acquire, process and interpret natural language.
- Natural language understanding (NLU) - extraction of information from text or speech and production of a description of what it represents.
- Natural language generation (NLG) - the task of converting data carrying semantics into natural language.
- Functional suitability - the capability of a product to provide functions that meet stated and implied user needs under specified conditions.
- Functional correctness - the degree to which a product or system provides the correct results with the needed degree of precision.
- Test data - data used to assess the performance of a final model.
- Independent - for a dataset, a condition where no characteristic is shared with another dataset other than the same statistical distribution.
Who uses oSIST prEN ISO/IEC 23282:2026?
oSIST prEN ISO/IEC 23282:2026 is used by teams that build, test, compare or procure NLP systems. That includes AI engineers, NLP researchers, evaluation specialists, quality managers, procurement teams and organizations that need a defensible way to assess system output quality.
It is relevant in areas such as machine translation, speech recognition, text generation, information extraction, clustering, recommendation, dialogue systems, speaker and author identification, and multimodal systems that combine NLP with speech or computer vision. In practice, these users apply it to design test plans, choose metrics, document results and compare alternative systems on a common basis.
Which standards are used with oSIST prEN ISO/IEC 23282:2026?
| Standard | What it contributes |
|---|---|
| ISO/IEC DIS 4213 | The base set of performance measurement methods for AI classification, regression, clustering and recommendation tasks. The document builds on it where NLP tasks reuse or adapt those methods. |
| ISO/IEC TR 23281 | The task hierarchy and the combination schemes used throughout the document. Clause 5 and later clauses refer to it for task definitions and prototypes. |
| ISO/IEC 22989:2022 | Source for core AI, NLP, NLU, NLG and related definitions used in Clause 3. |
| ISO/IEC 25059:2023 | Source for the definition of functional correctness used in the explanation of what is being measured. |
| ISO/IEC 25010:2023 | Source for the definition of functional suitability, which is the quality aspect the evaluation aims to assess. |
What does the oSIST prEN ISO/IEC 23282:2026 document contain?
oSIST prEN ISO/IEC 23282:2026 contains formal descriptions of evaluation methods and guidance on how to apply them to NLP tasks. Clause 6 gives the metrics and human protocols, including the conditions needed for reproducible computation and interpretation.
Clause 7 and Clause 8 provide the practical mapping from method to task, including which metrics fit which task families and how those metrics should be applied in the NLP context. The document also includes task-by-task interpretation guidance, so a result can be read in terms of the aspect of language processing it measures.
Clause 9 explains how to evaluate systems made of several NLP components, including pipelines and multi-task models. Clause 10 covers the evaluation resources, especially test data independence, multilingual coverage and computing conditions that affect reproducibility.
Relations
- Effective Date
- 28-Jan-2026
- Effective Date
- 28-Jan-2026
- Effective Date
- 28-Jan-2026
Get Certified
Connect with accredited certification bodies for this standard
BSI Group
BSI (British Standards Institution) is the business standards company that helps organizations make excellence a habit.
NYCE
Mexican standards and certification body.
Sponsored listings
Frequently Asked Questions
oSIST prEN ISO/IEC 23282:2026 is a draft published by the Slovenian Institute for Standardization (SIST). Its full title is "Artificial Intelligence - Evaluation methods for accurate natural language processing systems (ISO/IEC DIS 23282:2026)". This standard covers: This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
This document specifies the evaluation of natural language processing systems, in the sense of measuring the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation methods for those systems, together with guidance on how to select, implement and interpret those evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It includes requirements on the implementation of the described metrics, and further requirements on the technical resources involved in the evaluation process.
oSIST prEN ISO/IEC 23282:2026 is classified under the following ICS (International Classification for Standards) categories: 35.240.01 - Application of information technology in general. The ICS classification helps identify the subject area and facilitates finding related standards.
oSIST prEN ISO/IEC 23282:2026 has the following relationships with other standards: It is inter standard links to SIST EN ISO/IEC 29146:2023, SIST EN ISO/IEC 24760-2:2023, SIST EN ISO/IEC 24760-3:2023. Understanding these relationships helps ensure you are using the most current and applicable version of the standard.
oSIST prEN ISO/IEC 23282:2026 is associated with the following European legislation: EU Directives/Regulations: 2024/1689; Standardization Mandates: M/593, M/593, M/613, M/XXX AI, M/XXX AI. When a standard is cited in the Official Journal of the European Union, products manufactured in conformity with it benefit from a presumption of conformity with the essential requirements of the corresponding EU directive or regulation.
oSIST prEN ISO/IEC 23282:2026 is available in PDF format for immediate download after purchase. The document can be added to your cart and obtained through the secure checkout process. Digital delivery ensures instant access to the complete standard document.
Standards Content (Sample)
SLOVENSKI STANDARD
01-november-2026
Umetna inteligenca (UI) - Metode vrednotenja za natančne sisteme za obdelavo
naravnega jezika (ISO/IEC DIS 23282:2026)
Artificial Intelligence - Evaluation methods for accurate natural language processing
systems (ISO/IEC DIS 23282:2026)
Evaluationsmethoden für die Genauigkeit von Systemen zur Verarbeitung natürlicher
Sprache (ISO/IEC DIS 23282:2026)
Intelligence artificielle - Méthodes d’évaluation pour des systèmes de traitement
automatique du langage naturel performants (ISO/IEC DIS 23282:2026)
Ta slovenski standard je istoveten z: prEN ISO/IEC 23282
ICS:
35.240.01 Uporabniške rešitve Application of information
informacijske tehnike in technology in general
tehnologije na splošno
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
DRAFT
International
Standard
ISO/IEC DIS 23282
ISO/IEC JTC 1/SC 42
Artificial Intelligence — Evaluation
Secretariat: ANSI
methods for accurate natural
Voting begins on:
language processing systems
2026-09-02
Intelligence artificielle — Méthodes d’évaluation pour des
Voting terminates on:
systèmes de traitement automatique du langage naturel
2026-11-25
performants
ICS: 35.240.01
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENTS AND APPROVAL. IT
IS THEREFORE SUBJECT TO CHANGE
Member bodies are requested to consult relevant national interests in ISO/TC AND MAY NOT BE REFERRED TO AS AN
INTERNATIONAL STANDARD UNTIL
37 before casting their ballot to the e-Balloting application.
PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
This document has not been edited by the ISO Central Secretariat.
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
ISO/CEN PARALLEL PROCESSING
WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS.
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION.
Reference number
© ISO/IEC 2026
ISO/IEC DIS 23282:2026(en)
DRAFT
ISO/IEC DIS 23282:2026(en)
International
Standard
ISO/IEC DIS 23282
ISO/IEC JTC 1/SC 42
Artificial Intelligence — Evaluation
Secretariat: ANSI
methods for accurate natural
Voting begins on:
language processing systems
Intelligence artificielle — Méthodes d’évaluation pour des
Voting terminates on:
systèmes de traitement automatique du langage naturel
performants
ICS: 35.240.01
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENTS AND APPROVAL. IT
IS THEREFORE SUBJECT TO CHANGE
Member bodies are requested to consult relevant national interests in ISO/TC AND MAY NOT BE REFERRED TO AS AN
INTERNATIONAL STANDARD UNTIL
37 before casting their ballot to the e-Balloting application.
PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
This document has not been edited by the ISO Central Secretariat.
BEING ACCEPTABLE FOR INDUSTRIAL,
© ISO/IEC 2026
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
STANDARDS MAY ON OCCASION HAVE TO
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
BE CONSIDERED IN THE LIGHT OF THEIR
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
POTENTIAL TO BECOME STANDARDS TO
ISO/CEN PARALLEL PROCESSING
WHICH REFERENCE MAY BE MADE IN
or ISO’s member body in the country of the requester.
NATIONAL REGULATIONS.
ISO copyright office
RECIPIENTS OF THIS DRAFT ARE INVITED
CP 401 • Ch. de Blandonnet 8
TO SUBMIT, WITH THEIR COMMENTS,
CH-1214 Vernier, Geneva
NOTIFICATION OF ANY RELEVANT PATENT
Phone: +41 22 749 01 11
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION.
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland Reference number
© ISO/IEC 2026
ISO/IEC DIS 23282:2026(en)
© ISO/IEC 2026 – All rights reserved
ii
ISO/IEC DIS 23282:2026(en)
Contents Page
Foreword .vii
Introduction .viii
1 Scope .1
2 Normative references .1
3 Terms and definitions .1
4 Abbreviations .4
5 Overview .5
5.1 General .5
5.2 Types of evaluation methods .5
5.3 Contents of this document .6
6 Description of evaluation methods .7
6.1 General .7
6.2 Automated metrics .7
6.2.1 General .7
6.2.2 Accuracy, precision and recall, F1 score .8
6.2.3 Precision @ K, recall @ K, F1 @ K .8
6.2.4 Purity, inverse purity .8
6.2.5 Edit distance metrics .8
6.2.6 Entropy, mutual information and normalized mutual information .9
6.2.7 N-gram and subsequence matching metrics .10
6.2.8 Embeddings-based metrics . 13
6.2.9 Learned metrics .14
6.2.10 EER (equal error rate) .16
6.2.11 Jaccard coefficient, Rand score, adjusted Rand score, Fowlkes-Mallows index .16
6.2.12 Silhouette coefficient, silhouette index, Davies-Bouldin index, Calinski-
Harabasz index .16
6.2.13 B .17
6.2.14 AER (alignment error rate) .18
6.2.15 METEOR .18
6.2.16 Perplexity .18
6.2.17 DCF, minDCF and actDCF .18
6.3 Human evaluation protocols .19
6.3.1 General .19
6.3.2 Types of human evaluation protocols. 20
6.3.3 MOS (mean opinion score) .21
6.3.4 DRT (diagnostic rhyme test) .21
6.3.5 MRT (modified rhyme test) . . .21
6.3.6 MUSHRA . 22
6.4 Multi-dimensional analysis criteria . 22
6.4.1 General . 22
6.4.2 Quality criteria . 22
6.4.3 Error criteria. 23
7 Shared evaluation methods per task prototype .24
7.1 General .24
7.2 Categorisation .24
7.2.1 General .24
7.2.2 Automated metric: accuracy .24
7.2.3 Automated metrics: precision, recall, F1-score .24
7.3 Identification . 25
7.3.1 General . 25
7.3.2 Automated metric: accuracy . 25
7.3.3 Automated metrics: precision, recall, F1-score . 26
© ISO/IEC 2026 – All rights reserved
iii
ISO/IEC DIS 23282:2026(en)
7.3.4 Automated metric: EER . 26
7.3.5 Automated metrics: DCF, minDCF, actDCF . 26
7.4 Segmentation . 26
7.4.1 General . 26
7.4.2 Automated metrics: precision, recall, F1-score .27
7.5 Sequence labelling . 28
7.5.1 General . 28
7.5.2 Automated metric: accuracy . 28
7.5.3 Automated metrics: precision, recall, F1-score . 28
7.6 Per-element identification . 29
7.6.1 General . 29
7.6.2 Automated metric: accuracy . 29
7.6.3 Automated metrics: precision, recall, F1-score . 29
7.7 Set creation . 30
7.7.1 General . 30
7.7.2 Automated metrics: silhouette coefficient, silhouette index, Davies-Bouldin
index, Calinski-Harabasz index . 30
7.7.3 Automated metrics: purity, inverse purity . 30
7.7.4 Automated metrics: Jaccard coefficient, Rand score, adjusted Rand score,
Fowlkes-Mallows index . 30
7.7.5 Automated metrics: mutual information, normalized mutual information .31
7.7.6 Automated metric: MUC.31
7.7.7 Automated metric: B .32
7.8 Ranking .32
7.8.1 General .32
7.8.2 Automated metrics: precision @ K, recall @ K, F1 @ K.32
7.9 Sequence generation.32
7.9.1 General .32
7.9.2 Automated metric: BLEU . 33
7.9.3 Automated metrics: ROUGE-1, ROUGE-2, ROUGE-n . 33
7.9.4 Automated metric: ROUGE-L . 33
7.9.5 Automated metric: perplexity . 33
7.9.6 Automated metrics: CER, WER, PER . 34
7.9.7 Automated metrics: BERTscore . 34
7.9.8 Automated metrics: COMET regression metric, COMET ranking metric . 34
7.10 Structure generation . 34
8 Evaluation methods per task .35
8.1 General . 35
8.2 Tasks related to shallow analysis of natural language contents . 35
8.2.1 General . 35
8.2.2 Language identification . 35
8.2.3 Sentence boundary detection .37
8.2.4 Tokenization .37
8.2.5 Lemmatization . 38
8.2.6 Part-of-speech tagging . 39
8.2.7 Syntactic parsing . . 40
8.2.8 Morphological analysis . 40
8.2.9 Word sense disambiguation . 40
8.2.10 Coreference resolution .43
8.2.11 Anaphora resolution . 46
8.2.12 Discourse parsing .47
8.2.13 Document segmentation .47
8.2.14 Parallel corpora alignment . 48
8.2.15 Speech activity detection . 48
8.2.16 Utterance boundary detection . . 49
8.2.17 Phrase boundary detection . 49
8.2.18 Speaker change detection . 50
8.2.19 Speaker diarization . 50
© ISO/IEC 2026 – All rights reserved
iv
ISO/IEC DIS 23282:2026(en)
8.3 Tasks related to author identification and profiling . 50
8.3.1 General . 50
8.3.2 Authorship attribution . 50
8.3.3 Speaker recognition .52
8.3.4 Handwriting identification . 55
8.3.5 Style analysis .57
8.3.6 Speaker role identification .57
8.3.7 Speaker gender identification . 58
8.3.8 Impersonation detection .59
8.4 Tasks related to document analysis and mining .59
8.4.1 General .59
8.4.2 Topic analysis .59
8.4.3 Named entity recognition . 60
8.4.4 Entity linking .61
8.4.5 Relation extraction .62
8.4.6 Slot filling .62
8.4.7 Claim extraction and fact extraction .62
8.4.8 Sentiment analysis . 63
8.4.9 Emotion recognition . 64
8.4.10 User intent detection . 65
8.4.11 Author intention recognition . 66
8.4.12 Offensive speech detection .67
8.5 Tasks related to corpus-level analysis and mining . 68
8.5.1 General . 68
8.5.2 Document recommendation . 68
8.5.3 Content-based document search . 68
8.5.4 Semantic clustering . 69
8.5.5 Thematic clustering . 69
8.6 Tasks related to semantics and meaning .70
8.6.1 General .70
8.6.2 Automatic term extraction .70
8.6.3 Word sense induction.70
8.6.4 Term set expansion .71
8.6.5 Semantic relation extraction .71
8.6.6 Embeddings .71
8.6.7 Representation learning .71
8.6.8 Language modelling . 72
8.7 Tasks related with questions and dialogue . 72
8.7.1 General . 72
8.7.2 Question answering . 72
8.7.3 Generation of multimedia contents from prompts .74
8.8 Tasks involving the generation of linguistic contents .74
8.8.1 General .74
8.8.2 Automatic summarization .74
8.8.3 Machine translation . 75
8.8.4 Document denoising . 77
8.8.5 Language error correction . 78
8.8.6 Text normalization . 79
8.8.7 Sensitive data treatment . 79
8.8.8 Voice anonymization . 79
8.8.9 Description generation . 79
8.8.10 Vanilla text generation . 80
8.8.11 Constrained text generation . 82
8.8.12 Captioning . 84
8.8.13 Source code generation . 84
8.8.14 Source code documentation . 84
8.8.15 Paraphrase generation . 85
8.8.16 Question generation . 85
8.8.17 Distractor generation . 85
© ISO/IEC 2026 – All rights reserved
v
ISO/IEC DIS 23282:2026(en)
8.9 Tasks involving conversions among modalities . 85
8.9.1 General . 85
8.9.2 Automatic speech recognition . 86
8.9.3 Text-to-speech synthesis . 86
8.9.4 Optical character recognition . 88
8.9.5 Lip reading . 89
8.10 Transversal functionalities . 89
8.10.1 General . 89
8.10.2 Domain adaptation . 89
9 Evaluation methodology for multi-component NLP systems .90
10 Requirements on resources .91
10.1 General .91
10.2 Requirements on test data .91
10.3 Requirements on computing resources .91
Bibliography .92
© ISO/IEC 2026 – All rights reserved
vi
ISO/IEC DIS 23282:2026(en)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out through
ISO technical committees. Each member body interested in a subject for which a technical committee
has been established has the right to be represented on that committee. International organizations,
governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely
with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the different types
of ISO documents should be noted. This document was drafted in accordance with the editorial rules of the
ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of any patent
rights identified during the development of the document will be in the Introduction and/or on the ISO list of
patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment,
as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the
Technical Barriers to Trade (TBT) see the following URL: www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
© ISO/IEC 2026 – All rights reserved
vii
ISO/IEC DIS 23282:2026(en)
Introduction
This document focuses on the evaluation of natural language processing (NLP) systems, as regards their
[1]
functional suitability as defined in ISO/IEC 25059:2023 . This includes functional correctness (often
[1]
informally referred to as ‘performance’ by NLP practitioners, which ISO/IEC 25059:2023 , Annex C
discourages due to ambiguity, or 'accuracy' in some regulation, which can be inconsistent with technical
usage), as well as other quality characteristics such as functional adaptability.
While other standards exist on the evaluation of AI systems, such as ISO/IEC DIS 4213, they only contain
generic metrics, targeting the evaluation of some base AI tasks like classification, and those metrics are
either not sufficiently specified to be applied in turn on NLP systems or not applicable to part of these NLP
systems.
This document therefore builds on ISO/IEC DIS 4213 to provide the NLP-specific complements that are
necessary to enable the evaluation of NLP systems in particular. This includes additional specifications on
generic metrics used beyond NLP, as well as NLP-specific evaluation methods, encompassing both automated
and human evaluation.
© ISO/IEC 2026 – All rights reserved
viii
DRAFT International Standard ISO/IEC DIS 23282:2026(en)
Artificial Intelligence — Evaluation methods for accurate
natural language processing systems
1 Scope
This document specifies the evaluation of natural language processing systems, in the sense of measuring
the quality of a system’s results to assess its functional suitability. It provides a definition of evaluation
methods for those systems, together with guidance on how to select, implement and interpret those
evaluation methods. This document covers quantitative metrics as well as other evaluation methods. It
includes requirements on the implementation of the described metrics, and further requirements on the
technical resources involved in the evaluation process.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC DIS 4213, Information technology — Artificial Intelligence — Performance measurement for AI
classification, regression, clustering and recommendation tasks
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC
...