ISO/IEC TR 42106:2026

• Abstract:

  This document provides an overview of conceptual frameworks for graded benchmarking of artificial intelligence (AI) system quality characteristics. The aim is to examine the feasibility of using differentiated benchmarking of quality characteristics based on the complexity and context of use of an AI system.

• Status: Published
• Publication Date: 19-Jul-2026
• ICS: 35.240.01 - Application of information technology in general
• Technical Committee: ISO/IEC JTC 1/SC 42 - Artificial intelligence
• Drafting Committee: ISO/IEC JTC 1/SC 42 - Artificial intelligence
• Current Stage: 6060 - International Standard published
• Start Date: 20-Jul-2026
• Due Date: 05-May-2026
• Completion Date: 20-Jul-2026

Overview

ISO/IEC TR 42106:2026 provides an essential overview of conceptual frameworks for graded or differentiated benchmarking of artificial intelligence (AI) system quality characteristics. Developed by ISO/IEC JTC 1, SC 42, this technical report addresses the diversity, complexity, and varied contexts where AI systems are applied, emphasizing the importance of tailored benchmarking criteria to ensure system quality and trustworthiness. By examining various approaches and existing frameworks, the standard assesses the feasibility of applying differentiated benchmarking based on the complexity and context of each AI system.

This document is highly relevant for organizations developing, deploying, or assessing AI systems, as it helps rationalize efforts in standardization, maintain system trustworthiness, and ensure that conformance requirements scale appropriately with the potential risk AI systems present.

Key Topics

• Benchmarking Fundamentals

  • Definitions and terminology related to "benchmark" and "benchmarking"
  • Types of benchmarking: activity-/component-based and process-based
  • The importance of reference points and use of metrics, measures, and criteria

• AI System Quality Characteristics

  • Methods for benchmarking different AI quality attributes (accuracy, reliability, robustness, etc.)
  • Challenges in benchmarking due to AI system diversity and application contexts
  • Example metrics for tasks like classification, regression, object detection, and more

• Context and Complexity

  • The role of context of use in determining relevant quality characteristics and benchmarks
  • Unique risks and sociotechnical considerations in AI systems versus traditional software
  • Need for quality assurance aligned with the expected impact and potential harm

• Frameworks for Differentiated Benchmarking

  • AI management frameworks and risk-based approaches
  • Classification-based frameworks and examples from international practice (e.g., risk classification in German Data Ethics Commission, EU AI Act)
  • Use of tiered and flexible controls based on potential impact and application sector

• Limitations and Challenges

  • Overfitting to benchmarks, dataset bias, and the challenge of generalizability
  • Difficulties in comparing diverse quality metrics across applications
  • Resource constraints, human factors, and transparency issues in large-scale AI system evaluation

Applications

ISO/IEC TR 42106:2026 is designed to be practically useful for:

• AI Developers and Providers: Guiding the selection of appropriate benchmarks for different types and complexities of AI systems, ensuring reliable and trustworthy deployments.
• Regulators and Standards Bodies: Informing regulatory approaches and helping develop policies for risk assessment, transparency, and oversight of AI technologies.
• AI Customers and Partners: Supporting procurement and evaluation by providing frameworks for assessing system quality characteristics relevant to their use cases.
• Researchers and Academics: Serving as a reference for developing new benchmarking methods and studying the impact of benchmarking on AI innovation and safety.
• Public Sector Organizations: Assisting with impact assessments and governance of automated decision-making systems, ensuring public trust and accountability.

By enabling differentiated benchmarking, organizations can efficiently allocate resources, focus conformance efforts where risks are highest, and foster both innovation and safety in AI deployment.

Related Standards

For a comprehensive approach to AI quality and benchmarking, consider these complementary standards:

• ISO/IEC 22989 - Artificial intelligence concepts and terminology
• ISO/IEC 23053 - Framework for AI systems using machine learning
• ISO/IEC 25059 - Quality model for AI systems
• ISO/IEC 42001 - AI management systems
• ISO/IEC 25040 - Systems and software Quality Requirements and Evaluation (SQuaRE)
• NIST AI RMF - AI Risk Management Framework
• IEEE 7010 - Standard for ethically driven system design
• EU AI Act - European AI regulation framework

These documents, together with ISO/IEC TR 42106:2026, form a robust foundation for AI benchmarking, quality evaluation, and risk management, supporting conformance and fostering trust in AI across diverse sectors.

Technical
Report
ISO/IEC TR 42106
First edition
Information technology — Artificial
2026-07
intelligence (AI) — Overview of
differentiated benchmarking of AI
system quality characteristics
Technologies de l'information — Intelligence artificielle (IA) —
Aperçu de l'analyse comparative différenciée des caractéristiques
de qualité des systèmes d'IA
Reference number
© ISO/IEC 2026
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2026 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Overview of relevant benchmarking methods . 2
4.1 Review of benchmarking definitions .2
4.2 Types of benchmarking . .3
4.3 Metrics, measures and criteria .3
5 Benchmarking AI systems . 4
5.1 Benchmarking AI system quality . .4
5.2 Context of use .5
5.3 Complex adaptive systems .5
5.4 Limitations in benchmarking AI systems .6
6 Frameworks for differentiated benchmarking . 7
6.1 AI management frameworks . . .7
6.2 Classification-based frameworks .7
6.3 Levels of specification .9
7 Feasibility analysis .11
7.1 Case Example 1: On-job training recommendation system .11
7.2 Case Example 2: User intent recognition . 12
7.3 Case Example 3: Generation of clinical pathways . 13
Annex A (informative) Sample levels of specification .15
Annex B (informative) Descriptions of measures .16
Bibliography .18

© ISO/IEC 2026 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2026 – All rights reserved
iv
Introduction
Artificial intelligence (AI) systems are diverse in nature and heterogeneous in terms of their potential
effect on consumers and third-parties. For example, a company’s use of disposable income estimates using
a linear regression model to serve ads targeted at different socio-economic profiles reflects a system with
low complexity and potential impact. Conversely, a bank’s use of disposable income estimates using a large
neural network model to make housing loan decisions reflects a system with both high complexity and high
impact.
Benchmarking is often used to compare quality characteristics of software systems against reference or
target values. Given the diverse nature and heterogeneous impact of AI systems, such reference or target
values can differ widely across systems and across deployments of similar systems across contexts of use.
To accommodate this diversity and heterogeneity, differentiated benchmarking can target different quality
characteristics at different reference values. This document reviews AI management frameworks for their
ability to offer guidance for differentiated benchmarking of AI system quality characteristics.
By evaluating frameworks for specifying differing levels of benchmarks and benchmarking, commensurate
with expected social impact of AI systems, this document identifies gaps in current frameworks that, when
filled, can yield guidance for differentiated benchmarking of AI systems, which can help rationalize the
standardization implementation effort of AI providers, while maintaining system trustworthiness for AI
customers and AI partners.
© ISO/IEC 2026 – All rights reserved
v
Technical Report ISO/IEC TR 42106:2026(en)
Information technology — Artificial intelligence (AI) —
Overview of differentiated benchmarking of AI system quality
characteristics
1 Scope
This document provides an overview of conceptual frameworks for graded benchmarking of artificial
intelligence (AI) system quality characteristics. The aim is to examine the feasibility of using differentiated
benchmarking of quality characteristics based on the complexity and context of use of an AI system.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 22989, Information technology — Artificial intelligence — Artificial intelligence concepts and
terminology
ISO/IEC 23053, Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)
ISO/IEC 25059, Software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE)
— Quality model for AI systems
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 22989, ISO/IEC 23053,
ISO/IEC 25059 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
benchmark
reference point against which comparisons can be made
[1]
Note 1 to entry: For AI system benchmarking, an AI system quality characteristic (ISO/IEC 25059 ) is the object of
comparison.
[2]
[SOURCE: ISO/IEC 29155-1:2017 , 3.2, modified —Note 1 to entry has been replaced.]
3.2
benchmarking
activity of comparing objects of interest to each other or against a benchmark (3.1) to evaluate (a)
characteristic(s)
Note 1 to entry: For AI system benchmarking, the object of interest is an AI system quality characteristic
[1]
(ISO/IEC 25059 ).
[2]
[SOURCE: ISO/IEC 29155-1:2017 , 3.3, modified — Note 1 to entry has been replaced.]

© ISO/IEC 2026 – All rights reserved
4 Overview of relevant benchmarking methods
4.1 Review of benchmarking definitions
1) 2) 3)
When searching for "benchmarking" in the ISO , IEC and ITU terminology databases, there were 76
results from the ISO OBP and none from the IEC and ITU-T databases. By deleting not relevant terms and
definitions and merging the same definitions together, 14 definitions were collected.
These 14 definitions include several instances that define "benchmark" and "benchmarking" as a pair, with
the definition of benchmarking relying on the paired definition of benchmark. A clustered view of the objects
of interest for each of these definitions of benchmark/benchmarking is given in Table 1.
Table 1 — Clustered objects and characteristics relevant for benchmark and benchmarking
Cluster Object Description of characteristics Sources
[3]
1 reference point/tool/method comparisons can be made, pro- ISO 41011:2024 , 3.8.5;
[4]
cess, performance or quality can
(metric against; ISO 14031:2021 , 3.4.8;
be measured; other characteristics
[2]
any standard or reference; ISO/IEC 29155-1:2017 , 2.1 and 3.2;
can be measured
[5]
point of fixed location; ISO 17258:2015 , 3.1;
[6]
permanent mark) ISO 14050:2020 , 3.2.15;
[7]
(benchmark) ISO 21678:2020 , 3.2;
[8]
ISO 21931-1:2022 , 3.2.16;
[9]
ISO/IEC/IEEE 24765:2017 , 3.362;
[10]
ISO/TS 18667:2018 , 3.1.1;
[11]
ISO 20468-1:2018 , 3.1.2;
[12]
ISO 13053-2:2011 , 2.1;
[13]
ISO/IEC 25040
[2]
2 activity of comparing, evaluat- objects of interest to each other or ISO/IEC 29155-1:2017 , 3.2 and 3.3;
[14]
ing and analysis against a benchmark,
ISO/IEC 18520:2019 , 3.1.1;
[15]
(activity of comparing or characteristic; similar operational
ISO/TR 24514:2018 , 3.1;
evaluating; comparative eval- practices; an organization can use
[16]
ISO 14644-16:2019 , 3.3.1;
uation or analysis; activity of to search for and compare practic-
[17]
ISO 10010:2022 , 3.4;
measurement and analysis) es inside and outside the organiza-
[18]
ISO 10014:2021 , 3.8;
tion, with the aim of improving its
[19]
ISO 30400:2022 , 3.1.18 and 3.17;
performance; similar operational
[20]
practices
ISO 32210:2022 , 3.34;
[9]
ISO/IEC/IEEE 24765:2017 , 3.362
and 3.363;
[21]
ISO/IEC TS 25058
[3]
3 process of comparing process- the same nature, under the same ISO 41011:2024 , 3.8.5.1
es, performances or quali- circumstances and with similar
ty against practices measures
[5]
4 single value (benchmark) used for orientation ISO 17258:2015 , 3.2;
[22]
ISO 24523:2017 , 3.2;
[23]
ISO 24513:2019 , 3.7.1.1.2;
[15]
ISO/TR 24514:2018 , 3.1
Core concepts about "benchmarking" are reflected in the repetition of words and phrases across these 14
definitions. Among these, "comparisons" (10 times), “performances” (7 times), "can be measured" (6 times),
and "practices" (5 times) are the most frequently used core concepts.
1) ISO Online browsing platform (OBP): available at https:// www .iso .org/ obp
2) IEC Electropedia (IEV): available at https:// www .electropedia .org/
3) ITU-T Terms and Definitions available at: https:// www .itu .int/ br _tsb _terms/ #/

© ISO/IEC 2026 – All rights reserved
The terms “process” (4 times), “organization” (4 times), "a reference point" (3 times), “metric against"
(3 times), “evaluate (3 times)”, and “standard” (3 times) also contribute to an understanding of the basic
concept of "benchmarking”.
4.2 Types of benchmarking
From the review of uses of benchmarking in standardization literature, it is evident that there are primarily
two types of benchmarking in existing definitions of ISO deliverables: benchmarking can be intended in
[24] [3]
terms of an activity and components (ISO/IEC 29155-1 ) or for processes (ISO 41011:2024 ) as objects
of benchmarking.
With regards to benchmarking activity and components, the focus lies on comparing objects of interest
[24]
with each other or with a benchmark to evaluate characteristics (ISO/IEC 29155-1 ). Such activities
have characteristics of similar operation practice, similar attributes, processes or performance that are
comparable. The benchmark refers to a reference point against which comparisons can be made. For the
relevant stakeholder, a reference point can be any of the following:
— a tool for performance improvement through systematic search and adaptation of leading practice;
— a standard against which results can be measured or evaluated;
— a method for comparing the performance of organizations in a market segment;
— a test procedure that can be used to compare systems or components with each other or with a standard.
[24] [25]
ISO/IEC 29155-1 relies for product quality evaluation on ISO/IEC 25000 and in this perspective
[13]
ISO/IEC 25040 defines the context for using benchmarks as quality criteria.
With regards to benchmarking processes, the focus lies on comparing any combination of processes,
performances and quality with practices of the same nature, under the same circumstances and with
similar measures. Its special considerations are the systematic process for the identification of, becoming
acquainted with and for adoption of successful practices of benchmarking partners. This concept is used in
[3]
the domain of facility management (ISO 41011:2024 ).
Within this document, the concept of benchmarking is used to focus on the activity of comparing objects of
interest against a benchmark to evaluate characteristics, and the concept of benchmark to focus on a reference
point to which comparisons can be made. Such concepts are used widely in the domain of information
technology project performance benchmarking framework of systems and software engineering, as defined
[13] [21]
in ISO/IEC 25040 and ISO/IEC TS 25058 .
Therefore, the definitions of benchmark and benchmarking given in this document are adapted from
[2] [2]
ISO/IEC 29155-1:2017 , 3.2 and ISO/IEC 29155-1:2017 , 3.3 respectively, for reflecting the emphasis on
product benchmarking most clearly.
4.3 Metrics, measures and criteria
AI system functional correctness is measured using a vast array of quantitative metrics. Additionally, a
number of measures, such as loss functions, are relevant for measuring functional reliability during model
training, but not during actual system usage. In addition, some criteria are used for model size determination,
model selection, model training time and so forth, but are not directly reported as functional correctness
indices.
This document describes metrics, measures and criteria used for common tasks performed by AI systems.
The given list is not comprehensive, but is intended to provide a useful overview of tasks and their
corresponding measures.
© ISO/IEC 2026 – All rights reserved
Table 2 — Metrics and measures for common AI system tasks
Task Measures
[26]
Classification ISO/IEC TS 4213 Accuracy
Cross validation
Precision
Recall
Confusion matrix
ROC (receiver operating characteristic) curve
[27]
Regression Root mean squared error (RMSE)
[27]
Prediction ranking Spearman’s rank correlation
[28]
Localisation (bounding box around an object) Intersection over Union (IoU)
[29][28]
Object detection Mean average precision (mAP)
[29]
Image - semantic segmentation Mean intersection over union (mIoU)
[30]
Time-series forecasting MSE, MAE, MAPE (mean absolute percentage error), Mean abso-
lute scaled error (MASE)
[31]
POS (part-of-speech) tagging Accuracy
[31]
Named entity recognition Precision, Recall, F1 Score
[31]
Dependency parsing Labelled attachment score (LAS), unlabelled attachment score
(UAS), label accuracy score (LS)
[31]
Information retrieval PR curve, interpolated precision,
Mean average precision
[32]
Summarisation ROGUE (F1 score from the n-gram precision and recall)
Mathematical descriptions of measures used in Table 2 are given in Annex B.
Additionally, several criteria are used for determining the efficacy of model training procedures, without a
direct correlation with the functional correctness of the model. Some such criteria include:
— model size,
— training time,
— convergence rate.
It is notable that, whereas AI system quality encompasses multiple characteristics and sub-characteristics,
[1]
as described in ISO/IEC 25059 , existing metrics are mainly relevant to measurement of functional
suitability, and measures of several other important characteristics, such as reliability, maintainability,
usability and security appear not to be addressed. The consequences of this imbalance are reviewed further
in 5.4.
5 Benchmarking AI systems
5.1 Benchmarking AI system quality
Benchmarking the quality characteristics of AI systems is crucial for several reasons. Firstly, it allows
measurement and comparison of the quality characteristics of different AI models objectively, providing
valuable insights into their strengths and weaknesses. By benchmarking quality characteristics such
as accuracy, efficiency, reliability, and robustness, stakeholders can identify areas for improvement and
innovation, driving advancements in AI technology. Additionally, benchmarking facilitates standardization
and transparency within the AI ecosystem, enabling stakeholders to make informed decisions about
which models are most suitable for their specific needs. Furthermore, benchmarking helps to establish
benchmarks against which future AI systems can be evaluated, fostering a continuous cycle of improvement
and innovation. In addition to this, benchmarks can be used in AI management system controls.

© ISO/IEC 2026 – All rights reserved
Several methods exist for benchmarking AI systems, each tailored to measure specific quality characteristics.
[33]
IEEE 2937 provides formalized methods for benchmarking of hardware-related metrics of AI server
systems, emphasizing the measurement of training time, power consumption and inference latency. To
measure the functional correctness and suitability of AI systems, the general approach is the use of reference
datasets and evaluation metrics, where AI models are tested on established reference datasets, such as
[34] [35]
ImageNet for image classification or MNIST for handwritten digit recognition . These datasets come
with predefined training and testing datasets, enabling consistent evaluation across different methods.
Another method involves organizing competitions and challenges, such as the ImageNet Large Scale
[36] [37]
Visual Recognition Challenge (ILSVRC) or the Common Objects in Context (COCO) challenge , where
researchers and developers submit their AI models to compete against each other on specific tasks. These
competitions provide a platform for rigorous evaluation and comparison of AI systems in diverse scenarios.
Most relevant for this document, organizations like the National Institute of Standards and Technology
[38] [39]
(NIST) FRVT framework and the AI Benchmarking Initiative (AIBench) have developed reference
methodologies and benchmarks for evaluating AI systems in specific domains, promoting transparency and
reproducibility in AI research.
While current approaches for benchmarking AI systems are valuable, they also have several limitations. An
important limitation is dataset bias, where the quality of AI models can be skewed due to biases present
in the training data. This can lead to overfitting to specific datasets and poor generalization to real-world
[40]
scenarios . Leader-board competitions, the most common form of accuracy benchmarking for AI systems,
are particularly sensitive to dataset decay, and require careful handling to prevent overfitting to held out
[41]
data . Another challenge is the proliferation of evaluation metrics across domains, making it difficult
to compare the quality of AI models across different tasks. Reference datasets and competitions typically
focus on narrow tasks or domains, limiting the scope of evaluation and potentially overlooking important
aspects of AI systems, such as ethical considerations and societal impact. Moreover, the reproducibility
of benchmarking results can be challenging, particularly when details about model architectures,
hyperparameters, and training procedures are not adequately documented.
5.2 Context of use
Some software quality standards have historically co-evolved with reliability engineering, which in turn
historically focused on the maintenance and upkeep of mechanical systems. For such mechanical systems,
component reliability tends to correlate well with nearly all desirable quality metrics, such as functional
correctness, safety and resilience. For the most part, since software systems also have conceptually
enumerable input-output characteristics, approaches rooted in reliability engineering have translated well
to them.
However, this historical provenance of software quality standards systematically under-emphasises the role
of context of use on the quality characteristics of software-based systems. This is a significant limitation,
as the context of use offers considerable information about the possible hazards of a system’s use, which
is necessary to design appropriate requirements for the system. As Nancy Leveson observes, “system
and software requirements development are necessarily a system engineering problem, not a software
[42]
engineering problem.”
It is therefore helpful to consider AI systems from a sociotechnical perspective, ensuring that the degree of
quality assurance is aligned with the degree of quality expected of the system based on the context of use.
5.3 Complex adaptive systems
[43]
NIST AI 100-1:2023 , Annex B summarizes key aspects in which risks from AI systems are different from
risks from traditional software systems. These differences include the following:
a) Data used in model training is not always representative of the context of use of the system.
b) It is possible that real ground truth data does not exist, or is not available.
c) Data distributions can drift over time, and become detached from the original context in which the
system was trained.
d) Use of pre-trained models limits controllability of data quality and bias mitigation strategies.

© ISO/IEC 2026 – All rights reserved
In addition to these risks to system correctness, multiple additional sociotechnical considerations apply for
other quality characteristics of AI systems, such as:
e) Humans interacting with AI systems can change their behaviours to work around the narrow intelligence
of such systems replacing human operators.
f) AI systems can be subjected to data poisoning and spoofing attacks, reducing their effectiveness, when
deployed.
g) Human operators working alongside AI decision support systems can become overconfident and accept
system suggestions by default.
h) Human operators working alongside AI decision support systems can mistrust and ignore AI system
suggestions.
i) AI system integration into legacy IT systems can expand the cybersecurity threat envelope of the
existing system in ways that are difficult to detect with an audit of the two systems in isolation.
This list of considerations is not comprehensive, and is presented primarily to emphasize the thematic point
that AI systems are best validated with a sociotechnical systems approach, accounting for the fact that they
interact with users and third parties in complex ways, and that other entities adapt to being interacted with
by AI systems in ways that are not always foreseeable. Thus, new methods and approaches for benchmarking
complex and adaptive systems can be proposed.
5.4 Limitations in benchmarking AI systems
While benchmarking is already a challenging activity for simpler systems, involving multiple facets of data,
processes and measurements, benchmarking AI systems poses novel challenges that are to be addressed
with care. In particular, it is challenging to benchmark AI systems due to the following factors:
— AI systems are applied in a variety of sectors and contexts of use, each with different sources of risk and
uncertainty. Benchmarking such systems can either be adaptive to these differences, or be sufficiently
comprehensive to address all of them. For example, object detection models are frequently benchmarked
[33]
using mean average precision (mAP) across object classes, described for instance in IEEE 2937 .
However, there are several contexts of use, (e.g. object recognition for driverless vehicles) wherein
misidentification of some classes of objects, (e.g. pedestrians at risk), is of greater importance than other
objects, (e.g. street signs). In such contexts, mAP can exaggerate the functional suitability of the system,
since low importance classes are more likely to natively be encountered in the data environment than
high importance classes.
— Designers report AI system quality characteristics using a variety of metrics which makes comparisons
across metrics infeasible. For example, classification models for healthcare frequently report functional
correctness using an F1 score or the area under an ROC curve. However, such measures assume the
availability of plentiful clinical resources to act upon model predictions. In reality, clinicians are only
able to act upon a limited number of inputs from such classification models, thus favouring evaluation
metrics drawing upon the recommender systems literature, such as mean reciprocal rank, top-k
precision, etc. These metrics are mutually incommensurable, making it difficult to assess the true value
[44]
of such systems in use .
— Reference datasets used for benchmarking can contain noise, imbalance, and bias in unknown quantities.
Quality characteristic measurements inherit these problems in the form of fragility, inaccuracy and
algorithmic bias respectively. Examples of AI algorithms perpetuating societal and demographic biases
abound in the academic literature, and a vast literature on fairness in machine learning has emerged in
[45]
response to this problem .
— Evaluating very large models requires specialized techniques and infrastructure, which are not equally
accessible under resource constraints. Particularly for large language models, the computational
and energy requirements necessary to train models are very large, and inaccessible to most public
[46] [47] [48]
institutions , with the evaluations of such models also not homogeneous , .

© ISO/IEC 2026 – All rights reserved
— Many AI applications involve interaction with humans, and the nature of this interaction changes
reflexively as humans adapt to the use of the system. For example, automation of actions in cockpits has
[49]
been shown to be associated with atrophy of flying skills in human pilots , and similar deficits are
[50]
anticipated in the use case of driverless cars . Benchmarking AI systems in such contexts is predicated
on careful consideration of human factors and user experience, which adds considerable complexity to
any possible evaluation.
These inter-related problems are caused by the fact that modern AI systems are developed using very large
datasets and very large models, with downstream sociotechnical considerations not clearly known at the
time of system benchmarking. While it is possible to develop comprehensive benchmarking references
that accommodate the large scale and complexity of AI systems in use, the application of such standards is
predicated on high levels of expertise and resource allocations.
Alternatively, it is possible to conceive of approaches for differentiated benchmarking of AI systems, such
that quality characteristics of systems are benchmarked at different levels, and with different degrees of
standardization, adaptive to sociotechnical consideration of where such a system lies on a spectrum of
potential for harm. In this way, the conformity burden can rationally scale with the harm potential of AI
systems, thus simultaneously enabling innovation while maintaining safety.
6 Frameworks for differentiated benchmarking
6.1 AI management frameworks
AI management frameworks can potentially be used for differentiated benchmarking. For example,
[51]
ISO/IEC 42001 specifies requirements for establishing, maintaining and improving AI management
systems within organizations.
[52]
The AI Risk Management Framework (AI RMF) is another representative example . This framework
utilizes a descriptive methodology, offering flexibility in implementation. It focuses on assessing hazards,
exposures, and vulnerabilities associated with AI systems, allowing organizations to manage risks
effectively across various use cases and sectors.
[53]
ALTAI is a procedural framework that can be used as a tool for management frameworks. It attempts
to cover all principles and stages of AI implementation, and attempts to offer region-agnostic and sector-
agnostic perspectives. ALTAI follows a procedural approach, emphasizing trustworthiness in AI systems.
It assesses hazards, exposures, and vulnerabilities to ensure the ethical and trustworthy deployment of AI
technologies.
[54]
The Algorithm Impact Assessment Tool (AIA) addresses planning, requirements analysis, design, and
testing stages. The framework takes a procedural approach, ensuring that AI implementations are region-
agnostic and sector-agnostic. AIA assesses hazards, exposures, and vulnerabilities without specifying
particular domains.
[55]
IEEE 7010 follows a descriptive approach. While specific focus areas are not explicitly mentioned, the
practices are designed to be region-agnostic and sector-agnostic. The framework provides guidance on
assessing hazards, exposures, and vulnerabilities associated with autonomous and intelligent systems,
emphasizing their impact on human well-being.
6.2 Classification-based frameworks
[56]
Classification is a very common form of standardization activity . The standardization of IT systems,
in particular, seems to lend itself well to classification-based frameworks, as is evidenced by NIST's
cybersecurity framework subcategories, which enables an organization to standardize processes
[57]
relevant for its specific needs . The judgement of relevance provides the source of differentiation in the
standardization process in such frameworks, with the most common frame of relevance judgment being risk
or impact assessment.
© ISO/IEC 2026 – All rights reserved
A number of frameworks for risk or impact assessment of AI systems pre-exist. Some of these frameworks
use risk-based classification to differentiate benchmarking treatment for various AI products. Some such
frameworks are reviewed below.
[58]
The German Data Ethics Commission has created a guidance document describing five criticality classes
for AI systems depicting harm for, i.e. the physical as well as psychical well-being, finance, date, manipulation
of information as well as a negative form of nudging. Based on this guidance, regulation classes for AI systems
can vary depending on the jurisdiction and specific regulations in place. The document describes these five
regulation classes with corresponding duties for responsible parties, such as providers and manufacturers
as well as concerns which justify the placement of an AI system into a specific class.
a) Class 1: No or minimal potential for harm:
Duties: correctness checks, transparency, system analyses in cases of suspicion.
Concerns: potential for unexpected or unintended consequences.
b) Class 2: Low risk:
Duties: risk assessment, transparency obligations, and basic safety standards.
Concerns: undue risks to individuals or society.
c) Class 3: Moderate risk:
Duties: oversight, risk assessments, third-party audits, and adherence to specific industry standards.
Concerns: harm to individuals, privacy violations, diffusion in accountability, fairness in AI decision-
making.
d) Class 4: High risk:
Duties: thorough risk assessments, continuous monitoring, and robust fail-safes, independent audits,
conformity with strict safety and security standards, and regular reporting to regulatory authorities.
Concerns: significant harm to individuals, society, or critical infrastructure as well as negative ethical,
legal, and social implications, including, i.e. discrimination, bias, and transparency.
e) Class 5: Forbidden:
Duties: extraction of product from market by supervision authorities.
Concerns: extreme potential for harm, including threats to human life, national security, or global
stability. Immediate detection of product of such classes, as well clarity in accountability are
indispensable.
The German Data Ethics Commission framework can be adapted in several ways, for instance by making
higher risk criticality classes subject to the duties of lower risk or otherwise tweaking specific duties and
tying the terms "transparency" and "system analyses" to specific definitions.
The EU AI Act also adopts a risk-based classification approach, categorizing AI systems into different risk
[59]
levels based on their potential impact on rights, safety and societal values . High risk systems are expected
to be subject to stricter requirements and oversight, and providers of high risk systems are expected to meet
additional requirements related to data quality, documentation, transparency and traceability throughout
the AI system's life-cycle. The Act also allows for conformity assessment to verify conformity with the
requirements set forth in the regulation.
The Automated Decision-Making Systems in the Public Sector: An Impact Assessment Tool for Public
[60]
Authorities is designed for public authorities in Germany . It provides guidelines for assessing hazards,
exposures, and vulnerabilities associated with AI implementations in the public sector. This framework
follows a tiered procedural approach, wherein systems that exceed a set threshold score in a first-level
checklist during evaluation are taken through a more extensive set of controls than systems that do not.

© ISO/IEC 2026 – All rights reserved
The Artificial Intelligence Impact Assessment framework provides guidelines for assessing the impact of
[61]
AI technologies . The framework is designed to be region-agnostic and sector-agnostic. This framework
also follows a tiered procedural approach, offering organizations guidance on assessing hazards, exposures,
vulnerabilities, and mitigation risks associated with AI implementations based on the perceived criticality
of the deployment.
[62]
The Model Rules on Impact Assessment of Algorithmic Decision-Making Systems are designed to be
region-agnostic and applicable to public sectors. This framework also follows a tiered procedural approach,
guiding organizations in conducting impact assessments related to hazard, exposure, and vulnerability
associated with algorithmic decision-making systems based on the perceived criticality of the deployment.
Classification-based approaches to risk and impact assessment have the advantage of being commonly
known, easily reproducible, easily documentable, and intuitive to work with for regulatory and governance
bodies. However, there are also several limitations to such approaches.
For example, risk matrix based approaches to risk assessment presuppose a utilitarian view of risk, such that
[63]
there exists an implicit acceptance of severe risks provided the likelihood of such risks is acceptably low .
It is, however, well known that people systematically underestimate the probability of unlikely events in
[64]
decisions they make from experience . Therefore, analysts' likelihood estimates inevitably understate
[65]
expected risk when risk matrix approaches are applied in the determination of risk .
Additionally, classification is inherently a unidimensional approach to standardization, which is reasonable
in cases where the dimension along which risk or impact is expected to vary is clearly understood, but not
in cases where the dimensionality of risk variability itself is complex and multi-dimensional. Risk-based
classification of AI systems fundamentally inherits this defect.
6.3 Levels of specification
Risk-based classification schemes can guide entities towards specific actions appropriate for a relevant
context-of-use, but such schemes are based on a relatively unidimensional evaluation of context and risk.
The complex sociotechnical nature of AI system deployments can be more effectively addressed by adapting
generalized classification-based approaches, such as the use of levels of specification.
Software specifications offer guidance for ensuring that programs actually do implement the logic they
are expected to implement. These specifications can vary in their level of detail, with standard modules
specified mostly as flowcharts, and critical subunits specified with additional information about data ranges,
exception possibilities, etc. Repurposing this software idiom for the task of benchmarking AI systems, levels
of specification can be created to specify the set of benchmarking actions appropriate for AI systems with
different potentials for harm.

© ISO/IEC 2026 – All rights reserved
Quality characteristics
Standard ISO XXX Clause 5.17 is applicable
Standard ISO XXY Clause 8.9 is applicable
Figure 1 — Schematic illustration of levels of specification
A levels-based approach to benchmarking, as illustrated in Figure 1, can specify the set of benchmarks or
benchmarking procedures helpful to establish quality characteristics for AI systems with a particular level
of harm potential. The levels approach, therefore, is a generalization of the classification-based approach,
seeking to map AI systems with different harm potentials to benchmarks targeted at an enumerated set of
quality characteristics.
The construction of AI specification levels is based on:
a) designing property–action matrices, mapping software quality characteristics for AI systems to
standards,
NOTE 1 Property is intended as a quality characteristic.
b) modelling the ecosystem of AI applications at a general level to identify common patterns of
standardization needs for different stakeholders, and
c) defining property-action matrices particularized for systems with different levels of complexity.
Specification levels themselves emerge as nominal categories describing the specific property-outcome
matrix to be applied while specifying or designing any given system. Indicative sample property action
matrices are presented in Annex A.
By associating commonly occurring patterns of quality characteristic requirements within AI systems
with a rubric of specification levels, stakeholders can communicate the trustworthiness of AI systems to
observers succinctly, yet accurately. At the same time, the development of such specification levels can ease
the development and execution of regulatory frameworks within specific application domains. This approach
[13]
can be considered as a part of the quality evaluation planning and rating as defined in ISO/IEC 25040 .
However, defining specification levels involves discretization of an intrinsically continuous spectrum of
potential fo
...