ISO/IEC TR 17903:2024

Information technology - Artificial intelligence - Overview of machine learning computing devices

This document surveys machine learning (ML) computing devices, including the following: - ML computing device terminology and characteristics; - existing approaches to the setting and use of characteristics for optimizing ML computing device performance. The information provided in this document is relevant for organizations of all types and sizes.

Technologies de l'information — Intelligence artificielle — Aperçu des dispositifs informatiques pour l'apprentissage automatique

General Information

• Status: Published
• Publication Date: 05-May-2024
• ICS: 35.020 - Information technology (IT) in general
• Technical Committee: ISO/IEC JTC 1/SC 42 - Artificial intelligence
• Drafting Committee: ISO/IEC JTC 1/SC 42 - Artificial intelligence
• Current Stage: 6060 - International Standard published
• Start Date: 06-May-2024
• Due Date: 16-Nov-2023
• Completion Date: 06-May-2024

Overview

ISO/IEC TR 17903:2024 - "Information technology - Artificial intelligence - Overview of machine learning computing devices" is a Technical Report that surveys terminology, representative characteristics and existing approaches for optimizing performance of machine learning (ML) computing devices. Published as a first edition in 2024 by ISO/IEC JTC 1/SC 42, the report is an informational resource intended to help AI stakeholders understand the building blocks and performance factors of ML computing devices. It is relevant to organizations of all sizes and does not prescribe mandatory requirements.

Key topics

The report organizes content across conceptual, technical and optimization topics:

• Concepts and terminology: ML computing, AI computing, computing device, infrastructure, functional units and services (Clause 3–5).
• ML computing device characteristics (Clause 6), including:
  • Datatypes and their impact on effectiveness and efficiency
  • ML operators (primitives used by models)
  • Memory access and addressing mechanisms
  • Scheduling and job orchestration mechanisms
  • Topologies (interconnects and device layouts)
  • Streams, buffering and cache mechanisms
  • Data exchange and memory interoperability mechanisms
• Performance optimization approaches (Clause 7):
  • Computing resource–level tactics (hardware configuration, interconnects, memory usage)
  • Enabling software–level tactics (runtime, compiler and scheduling strategies)
  • Measures and metrics for evaluating device performance

Practical applications

This Technical Report helps stakeholders make practical decisions around ML computing devices:

• Hardware architects and vendors - understand device characteristics important for ML workloads and how design choices affect efficiency.
• System integrators and IT managers - select and configure accelerators, memory, and interconnects to meet performance and cost targets.
• ML engineers and platform teams - tune datatypes, operators and scheduling to improve model training and inference throughput and latency.
• Procurement and operations - evaluate compatibility, scalability and interoperability of ML computing devices across infrastructures.
• Researchers and standards developers - build on a common vocabulary and survey of current optimization approaches.

By clarifying ML computing device terminology and surveying characteristic-setting approaches, ISO/IEC TR 17903:2024 supports better alignment between ML workloads, device capabilities and performance optimization strategies.

Related standards

• ISO/IEC 22989 - AI concepts and terminology
• ISO/IEC 23053 - Framework for AI systems using ML

These documents are cited as normative references and provide complementary guidance on AI/ML terminology and system-level frameworks.

Standards Content (Sample)

Technical
Report
ISO/IEC TR 17903
First edition
Information technology — Artificial
2024-05
intelligence — Overview of machine
learning computing devices
Technologies de l'information — Intelligence artificielle —
Aperçu des dispositifs informatiques pour l'apprentissage
automatique
Reference number
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2024 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 5
5 ML computing device concepts . 5
5.1 Processing .5
5.2 Computing.5
5.3 Device .6
5.4 Infrastructure .6
5.5 Service .6
5.6 Performance .7
5.7 Computing device . .7
6 ML computing device characteristics . 7
6.1 Datatypes .7
6.1.1 General .7
6.1.2 Effectiveness and efficiency .8
6.2 ML operators .8
6.2.1 General .8
6.2.2 Effectiveness and efficiency .9
6.3 Memory access and addressing mechanisms .10
6.3.1 General .10
6.3.2 Effectiveness and efficiency .10
6.4 Scheduling .10
6.4.1 General .10
6.4.2 Effectiveness and efficiency .11
6.5 Topologies . 12
6.5.1 General . 12
6.5.2 Effectiveness and efficiency . 12
6.6 Streams . 13
6.6.1 General . 13
6.6.2 Effectiveness and efficiency . 13
6.7 Buffering mechanisms . 13
6.7.1 General . 13
6.7.2 Effectiveness and efficiency .14
6.8 Cache mechanisms .14
6.8.1 General .14
6.8.2 Effectiveness and efficiency .14
6.9 Data exchange mechanisms . 15
6.9.1 General . 15
6.9.2 Effectiveness and efficiency . 15
7 Approaches and measures for performance optimization .16
7.1 Approaches .16
7.1.1 Overview .16
7.1.2 Computing resource-level .16
7.1.3 Enabling software-level .16
7.2 Measures .17
Annex A (informative) Relationships between ML computing device-related definitions . 19
Bibliography .21

© ISO/IEC 2024 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2024 – All rights reserved
iv
Introduction
As an important approach for realizing artificial intelligence (AI), machine learning (ML) has been applied
to and has improved productivity in multiple domains (e.g. education, finance, environment protection). ML
computing devices can be essential to the development and deployment of many types of AI systems.
An ML computing device can have a set of characteristics, including supported datatypes, ML operators,
buffer settings, access and share mechanisms, memory access and addressing mechanisms, virtualization
and sharing mechanisms, job scheduling mechanisms, topologies, data exchange mechanisms and memory
interoperability mechanisms. Use and setting of these characteristics can affect the overall performance
of an ML computing device. The performance of an ML computing device used to develop and deploy an AI
system can be crucial to business effectiveness and efficiency.
This document surveys and provides information to AI stakeholders to assist them in understanding the
representative characteristics of ML computing devices. In Clause 5, ML computing device-related concepts
are discussed. Clause 6 summarizes ML computing device characteristics. In Clause 7, existing approaches
for optimizing ML computing devices’ performance are discussed.

© ISO/IEC 2024 – All rights reserved
v
Technical Report ISO/IEC TR 17903:2024(en)
Information technology — Artificial intelligence — Overview
of machine learning computing devices
1 Scope
This document surveys machine learning (ML) computing devices, including the following:
— ML computing device terminology and characteristics;
— existing approaches to the setting and use of characteristics for optimizing ML computing device
performance.
The information provided in this document is relevant for organizations of all types and sizes.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 22989, Information technology — Artificial intelligence — Artificial intelligence concepts and
terminology
ISO/IEC 23053, Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 22989 and ISO/IEC 23053 and
the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
artificial intelligence computing
AI computing
processing that leverages infrastructures to enable the set of methods or automated entities that together
build, optimize and apply an AI model
3.2
machine learning computing
ML computing
processing that leverages infrastructures to train and execute ML models
3.3
infrastructure
hardware and software environment to support computer system and software design, development, and
modification
Note 1 to entry: Network facilities can be also infrastructure working with hardware and software.

© ISO/IEC 2024 – All rights reserved
[1]
[SOURCE: ISO/IEC/IEEE 12207:2017, 3.1.25, modified: note 1 to entry is added.]
3.4
computing device
functional unit (3.6) that can perform substantial computations, including numerous arithmetic operations
and logic operations with or without human intervention
Note 1 to entry: A computing device can consist of a stand-alone unit (3.7), or several interconnected units. It can also
be a device that provides a specific set of functions, such as a phone or a personal organizer, or more general functions
such as a laptop or desktop computer.
Note 2 to entry: A computing device contains at least one unit (3.7).
[2]
[SOURCE: ISO/IEC 19770-5:2015, 3.6, modified: note 2 to entry is replaced.]
3.5
hardware
all or part of the physical components of an information processing system
EXAMPLE Computers, peripheral devices.
[3]
[SOURCE: ISO/IEC 2382:2015, 2121277, modified: removed note to entry 2 and 3]
3.6
functional unit
entity of hardware or software, or both, capable of accomplishing a specified purpose
Note 1 to entry: In IEC 50 (191), the more general term item is used in place of functional unit. An item may sometimes
include people.
[SOURCE: ISO/IEC 2382:2015, 2123022, modified: removed notes 2 and 3 to entry.]
3.7
unit
lowest level of hardware assembly for which acceptance and qualification tests are required
[4]
[SOURCE: ISO 24917:2020, 3.4]
3.8
service
kind of application which encapsulates one or more computing modules and can be accessed through a
specified interface
[5]
[SOURCE: ISO/IEC 19763-5:2015, 3.1.18]
3.9
artificial intelligence computing device
AI computing device
computing device (3.4) that can be specifically used for accelerating some or all of artificial intelligence
computing
Note 1 to entry: An artificial intelligence computing device often contains and works with specific enabling software.
3.10
machine learning computing device
ML computing device
computing device (3.4) that can be specifically used for accelerating machine learning computing
Note 1 to entry: A machine learning computing device often contains and works with specific enabling software.
Note 2 to entry: Machine learning computing device is a subset of AI computing device (3.9).
EXAMPLE A neural network process unit and its associated enabling software.

© ISO/IEC 2024 – All rights reserved
3.11
performance
measurable result
Note 1 to entry: Performance can relate either to quantitative or qualitative findings.
Note 2 to entry: Performance can relate to the management of activities, processes, products (including services),
systems or organizations.
[6]
[SOURCE: ISO/IEC 27000:2018, 3.52]
3.12
effectiveness
extent to which planned activities are realized and planned results are achieved
[7]
[SOURCE: ISO/IEC 33001:2015, 3.1.3]
3.13
efficiency
resources expended in relation to the accuracy and completeness with which users achieve goals
[8]
[SOURCE: ISO/IEC 25063:2014, 3.4]
3.14
datatype
set of distinct values, characterized by properties of those values and by operations on those values
[9]
[SOURCE: ISO/IEC 11404:2007, 3.12]
3.15
operator
mathematical or logical symbol that represents an action to be performed in an operation, or a symbol
representing the name of a function
[12]
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.2716, modified — 3.2716.3 and 3.2716.4 are combined.]
3.16
schedule
methods for controlling the timing of the execution of an activity within or represented by a managed object
[33]
[SOURCE: ISO/IEC 10164-15:2002, 3.1.27]
3.17
topology
structure of the communication paths between the medium attachment points
[48]
[SOURCE: ISO/IEC 14543-2-1:2006, 3.2.30]
3.18
stream
sequence of representations of jobs or parts of jobs to be performed, as submitted to an operating system
[3]
[SOURCE: ISO/IEC 2382:2015, 2122886]
3.19
computing
actions performed or approaches implemented by a system, aiming at processing specific kinds of data or
providing solutions to certain tasks
Note 1 to entry: Computing can also be a system providing functionalities or characteristics.

© ISO/IEC 2024 – All rights reserved
3.20
heterogeneous computing
system that uses more than one kind of processor or cores
[SOURCE: ISO/IEC 30145-3:2020, 3.1.5]
3.21
distributed computing
spreading of computation and data across a number of computers connected by a network
[12]
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.1240]
3.22
quantum computing
use of quantum phenomena for computational purposes
[SOURCE: ISO/TS 80004-12:2016, 6.5]
3.23
cloud computing
paradigm for enabling network access to a scalable and elastic pool of shareable physical or virtual resources
with self-service provisioning and administration on-demand
Note 1 to entry: Examples of resources include servers, operating systems, networks, software, applications and
storage equipment.
[SOURCE: ISO/IEC 22123-1:2021, 3.2.1]
3.24
inter-cloud computing
paradigm for enabling the interworking between two or more cloud service providers
[SOURCE: ISO/IEC 22123-1:2021, 3.12.1]
3.25
edge computing
distributed computing in which processing and storage takes place at or near the edge, where the nearness
is defined by the system's requirements
[SOURCE: ISO/IEC TR 23188:2020, 3.1.3]
3.26
processing
run of an algorithm, treatment on data or a sequence of them performed by AI systems
3.27
buffer
device or storage area used to store data temporarily to compensate for differences in rates of data flow,
time of occurrence of events, or amounts of data that can be handled by the devices or processes involved in
the transfer or use of the data
[12]
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.430.1]
3.28
cache
temporary storage in computer memory, to improve operations by having frequently used data readily
available for retrieval
[12]
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.452.1]

© ISO/IEC 2024 – All rights reserved
4 Abbreviated terms
AI artificial intelligence
FIFO first-in-first-out
GPU graphics processing unit
IT information technology
ML machine learning
NPU neural-network processing unit
PCIe peripheral component interconnect express
SoC system on chip
UML unified modelling language
5 ML computing device concepts
5.1 Processing
Processing can refer to the following:
— an activity, or treatment, or a sequence of them to be conducted for a specific purpose;
— a run of an algorithm, which can provide certain treatment or trigger some activity for a specific purpose.
In comparison with computing, processing emphasizes specific actions, while computing can be a system
that contains actions as well as approaches to be realized via actions. In the context of an ML computing
device, process can refer to the action that stakeholders take for an ML task by using ML computing devices.
5.2 Computing
A prerequisite for deriving the meaning of AI computing device and ML computing device is to define AI
computing and ML computing. In International Standards, a definition of computing does not exist. Based on
the definitions in sub-areas of IT, the meaning of computing can be at minimum the following:
— a system having specific functionalities or characteristics aiming at processing a specific kind of data or
providing solutions to certain tasks. This meaning of computing is used in definitions of heterogeneous
computing and cognitive computing, where the term system explicitly appears;
— a set of approaches or actions that can be performed by a system aiming at processing a specific kind
of data or providing solutions to certain tasks. This meaning of computing is used in definitions of
semantic computing, quantum computing, cloud computing, inter-cloud computing, soft computing,
edge computing and distributed computing.
AI contains a wide range of branches (e.g. ML, semantic computing and planning). Computing approaches
and solutions are driven by the characteristics of data (e.g. tensor, vector and scalar). For ML tasks, when
large number of vectors or tensors are involved, hardware modules and software components can be applied
to compute elements in a vector or a tensor in parallel rather than sequentially.

© ISO/IEC 2024 – All rights reserved
5.3 Device
An ML computing device is a kind of device. Analysis of the meaning of device helps to define and understand
the concept of ML computing device. Many definitions of device occur in ISO/IEC and IEEE standards. The
meanings of a device can be the following:
— a physical appliance or apparatus or a combination of multiple of them that can provide functionalities for
[64]-[65]
a specific purpose. This meaning of device is used in multiple areas (e.g. home electronic system,
[68] [69] [70] [72]
health informatics, nanotechnologies, cloud computing and industrial automation systems );
— a combination of hardware and software components or an instance of software solely that can provide
functionalities for a specific purpose. This meaning of device is used in multiple areas (e.g. operating
[66] [67] [12]
system, rich media user interface and software engineering ).
A device provides functionalities for a specific purpose. Not all devices are for the purpose of computing. In
the context of this document, a device can be a composition of collaborative devices that perform different
computing tasks or different parts of one computing task. Each of these devices can contain hardware and
associated enabling software.
EXAMPLE A server is a device that contains collaborative devices such as memory, disk and bus for generic
computing, as well as those specifically designed for tensor multiplication for neural network processing. Memory,
disk, bus and tensor computation components can work with associated enabling software. Some software is
provided as drivers within an operating system, while other software (e.g. the enabling software library for tensor
multiplication) needs to be additionally provided by a specific vendor.
From the perspective of an application, a device can be logical, which is a form of representation mapping
to one, a part of one device, or a set of devices. The use of such a logical device triggers the functioning of
mapped devices.
5.4 Infrastructure
In the information technology domain, an infrastructure can be software and hardware (including network
facilities), even data and files, regardless of scale (see ISO/IEC 16350). In non-IT domain, the meaning of
infrastructure can be wider. Infrastructures can be facilities, services, or supporting resources for the
[75] [74]
operation of an organization or a system (e.g. robotics domain, space system domain and management
[76]
domain ).
Based on the meaning of infrastructure in IT domain, an infrastructure in ML domain can be anything that
can be used by an ML application or its formation, including ML computing devices, datasets, ML software
toolkits, or even ML models.
The main difference between device and infrastructure is that the latter can contain data, file, even services
provided by a human or a group of humans. In the context of this document, a device can be a specific kind
of infrastructure whose main purpose is not to provide data, files, or services and whose purpose does not
include management or operation of an organization.
5.5 Service
By investigating the definitions of service specified in ISO/IEC standards, a service can be:
— a set of actions or values delivered from a provider (e.g. an organization) to a user or a customer. This
[78] [80]
meaning is used in multiple areas, including health informatics, social management, etc.;
— an application or software that encapsulates one or more computing modules and provides an interface to
apply its functions to the input from users. This meaning is used in multiple areas, including information
[77],[84],[85] [79] [81]
technology, health informatics and geomatics ;
— a set of operations provided by an entity (e.g. web application server). This meaning is used in multiple
[82] [83] [86]
areas, including software engineering, web service and ergonomics .

© ISO/IEC 2024 – All rights reserved
A service can be provided by utilizing IT or non-IT infrastructures, depending on the content of the service:
— When it takes use of certain IT infrastructures, it does not necessarily leverage ML techniques. When a
service is an application that provides some function via an interface, its scope is larger than a computing
device bound to specific business requirements. A service can also be a software component or an
assembly of components at higher layer (e.g. operating system layer or application layer). When a service
is provided that depends on an ML process, the service can take advantage of ML computing devices.
— A service can also takes use of non-IT infrastructures only, meaning some actions (e.g. transportation of
goods) provided by a vendor using certain machines (e.g. a truck) to users or customers.
5.6 Performance
Characteristics of ML computing devices can affect the efficiency and use of ML. From this perspective,
performance is an overall term that includes efficiency and effectiveness metrics. In some ISO/IEC standards
(e.g. [6] and [87]), the term performance is defined as measurable results.
The meaning of ML computing devices’ performance is different from the meaning of the performance of
[100]
classification systems. Specified in ISO/IEC/TS 4213 the term performance emphasizes classification
accuracy-related measurable results. The former puts more attention on the efficiency (e.g. running speed,
throughput, power consumption and reliability) and effectiveness (e.g. compatibility to datatypes or
protocols). An algorithm’s performance in terms of running speed is largely determined by its execution
environment. Speed of execution is not a characteristic of an algorithm, but subject to complexity (e.g. time
complexity and space complexity) and execution environment.
An important target of study and manufacturing is to improve the efficiency of user applications. The
performance of ML computing devices can be represented and measured by multiple metrics, defined in
[101]
IEEE 2937 which covers training as well as inference processes. They provide options for users to choose
in accordance with specific business requirements and restrictions. The absolute value of running speed
(e.g. number of floating point operations per second) of an ML computing device does not systematically
represent the actual outcome of computing on a specific task (e.g. natural language processing with an ML
[11] [41]
model BERT or image classification with an ML model RESNET ). The actual throughput (e.g. number
of images or sentences processed per second) can provide an overall evidence of the performance of ML
computing device, meaning the magnitude of outcomes a user can obtain per a time unit.
5.7 Computing device
A computing device is a functional unit that can perform substantial computations. A functional unit can be
hardware-only or a combination of hardware and software. A functional unit is a unit which is the lowest
level of hardware assembly for which acceptance and qualification tests are required. Both hardware and
software are derived from entity, as shown in Annex A.
An ML computing device is a kind of AI computing device since ML is a sub-area of AI and ML computing
devices specifically accelerate ML computing. The concept AI computing device is a descendant of the
concept computing device. This also holds for the concept ML computing device.
6 ML computing device characteristics
6.1 Datatypes
6.1.1 General
Datatype is an attribute of a piece of data that tells a computing system how to interpret its value. A
computing device, based on the widths or the structures of its buffers, registers or calculators, consumes a
piece of data in the form of sequences of bits.
The question of whether a datatype is acceptable or process-able by an ML system is focused on application
and program aspects. An ML computing device only provides instructions (e.g. vector addition, matrix

© ISO/IEC 2024 – All rights reserved
multiplication) for calculation over bit sequences. Advanced encoding or decoding and datatypes (e.g. string,
integer, list, set) can be implemented by associated software libraries. These are useable by upper layer
applications and programs but not directly applicable to devices.
In contrast, on the device side, it is important that an input bit sequence with certain width can be calculated
in fewest clock signals. Specific hardware design for acceleration can be performed, driven by requirements.
From this perspective, different parts of a computing device can have different designs to be competent to
accelerate computations with intended widths of bit sequences.
[10]
Representative and frequently used widths of bit sequences are standardized by IEEE 754, which
specifies interchange and arithmetic formats and methods for binary and decimal floating-point arithmetic
in computer programming environments. Depending on the requirements of the scenarios where ML
computing devices are used, datatypes can be designed, implemented and applied for specific purposes.
Typical precisions of data include 8-bit signed integer, 16-bit half-precision signed float point and 32-bit
signed floating point.
In some cases, a specific ML computing device cannot directly process a piece of data in certain datatype
defined in upper layer. For example, some ML computing devices whose maximum width of register is 16-bit
cannot directly process data in 64-bit floating point. In such a case, if supported by the device’s associated
software libraries, additional operations are performed including but not limited to:
— casting data from higher precision to lower precision;
— filling unused bits with zeros;
— splitting data in higher precision into parts, processing one part at a time and composing the stepwise
outcomes into the final result.
These operations can lead to performance loss, in terms of accuracy or overall time latency.
6.1.2 Effectiveness and efficiency
From the perspective of definitions, there are no advantages and disadvantages across datatypes, since
the definitions of datatypes are driven by business requirements. However, from the perspective of ML
computing device, micro architecture design, instruction sets and APIs can be prepared to support a certain
datatype. If a type of ML computing device supports a relatively wide range of datatypes, the design of
its registers, calculators, buffers and other components can be more flexible but can also be harder and
less efficient. The risk of insufficient use of computing resources (e.g. unused bits in a 32-bit register when
dealing with an 8-bit number) can also exist. In contrast, if a kind of ML computing device processes just one
kind of datatype, the utilization can be higher and its design from this perspective can be simpler.
When datatypes are selected and applied for a specific ML task. Stakeholders can consider the trade-
offs between efficiency and effectiveness in terms of the selection of datatypes. The target is to apply
datatypes which allows to complete the task with as minimum resources as possible. For training, mixed-
precision approaches (e.g. [89] and [88]) can be applied to decrease the resources (e.g. time) consumption
and in the meanwhile guarantee the accuracy of the trained model with qualified precision. For inference,
quantification techniques (e.g. [73]) can be conducted to achieve efficiency if the quantified model can still
provide adequate accuracy that makes the AI system effective.
6.2 ML operators
6.2.1 General
An operator is a symbol representing a certain functionality. Operators can be implemented by software
and wrapped as static and dynamic libraries. The software implementation can be regarded as a kind of
abstraction that wraps those specific details about interactions with a certain hardware. In this way,
manipulations on hardware (e.g. physically transmitting a signal to a port of hardware) can be done by
invoking software functions.
© ISO/IEC 2024 – All rights reserved
[13]
For ML computing, specific operators are designed to implement mathematical or logical calculations
used in ML training (during forward propagation and backward propagation sub-processes) and inference.
This kind of operator for ML computing is referred to as an ML operator. ML operators can be implemented
by software and wrapped as libraries for programmable invocations for training and inference. ML operators
[14]
can use fundamental mathematical operators, which are not specifically for ML processes and can be
used for another technical domain. ML operators include the ones for traditional ML process (e.g. regression
[15]-[16]
and clustering) as well as deep learning .
In an ML system, ML operators are defined by AI computing devices and implemented by the associated
enabling software libraries. Such implementation of operators exposes a set of supported functions that
[17]
the ML computing device can execute. A normative International Standard does not exist specifying a
common set of operators, except the ones provided by ML computing device providers, such as the sets of
[18] [19]
operators defined in compute unified device architecture, compute architecture for neural networks
and other libraries with similar functionalities. In Reference [20] an open, cross-industry standards-based
initiative unified programming model is specified. It provides specifications and open-source projects. The
sets of operators from ML computing device manufacturers can differ in terms of contents, semantics of
similar functions, function signatures and approaches of use.
In an ML system, ML operators can be further wrapped and extended by the following upper layers:
— ML software toolkits can provide encapsulations over ML operators that are used by software internal
computation and user procedures. ML software toolkits vendors also implement their own extensions
on ML operators, which are the sets of functions executable by a specific toolkit, using the operators
supported by the underlying hardware. In this layer, there is no formal standard specifying a complete
and common set of operators that a considerable number of ML software toolkits (e.g. [21]-[23]) follow.
Standards for operators in this layer are emerging (e.g. [24]), followed by a group of collaborative
enterprises. Such standards can be associated with specific model compilers that can compile sequences
of computation onto specific backend hardware.
— ML model can be in a specific representation formed by a set of specified ML operators that work as an
intermediate form for ML model format exchange and transition to assist the portability of models. In
this layer, a representative standard for model exchange is Reference [25]. It works as a bridge between
models in different formats generated by different ML software toolkits. There does not exist a full
consistency between the operators in an ML model intermediate representation with the encapsulated
ones used by an ML software toolkit.
In addition, open standards (e.g. [26]) exist for general purpose parallel programming agnostic to the type
of processors, giving software developers portable and efficient access to the power of these heterogeneous
processing platforms. Reference [26] is aimed at supporting parallel computing, which works as a base for
[26]
ML. Complex operators (e.g. K-mean and Conv), particularly in the scope of ML, have not been specified .
Reference [26] can benefit the following:
— To implement necessary sets of operators, such as linear algebra, to enable hardware-agnostic mathematic
[27]
computations (e.g. MAGMA ).
— By ML computing device manufacturers and implemented with modifications in their accelerating
libraries for enabling computation of mathematics via specific computing device.
A set of functionalities (e.g. communication, command and data exchange) exist which benefit calculations
but do not directly carryout the calculation. They are the base of computing and can be used for multiple
technical domains, not only for ML. In this area, operators can refer to the abstractions supported by static
or dynamic libraries that execute on specific hardware.
6.2.2 Effectiveness and efficiency
The definitions of the operators of a specific platform (e.g. the ML library of an ML computing device, ML
software framework and intermediate representation) provides advantages for improving applications’
compliance to that platform. However, AI computing device and ML software toolkit manufacturers

© ISO/IEC 2024 – All rights reserved
have already designed, implemented or extended their own operators. The following difficulties exist in
establishing a unified and public set of ML operators for all kinds of AI computing devices and software:
— The soundness of the operator set is hard to guarantee if these operators come out of a merge of those
existing operator sets. It is highly probable that a subset of operators exist that cannot be successfully
executed by a subset of hardware and software. Conflicts and inconsistencies can exist among the
operators, since the settings, requirements and use of parameters and contexts of similar operators
across manufacturers can be different.
— The execution efficiency of a unified and public operator set cannot be guaranteed. The more abstract
or general, the more adaptation or specification can be needed. This principle is true in ML computing
devices, where multiple kinds of hardware and software can be involved by various users and application
systems. To achieve compliance between operator sets to be merged, a loss of efficiency can happen in
cases such as unused and redundant parameters as well as cascaded data.
— The workload for maintaining a set of unified and public ML operators can be massive, given that the
underlying ML operator sets by different manufacturers can continuously and independently evolve.
6.3 Memory access and addressing mechanisms
6.3.1 General
ML computing needs memory for temporarily storing intermediate results of computations for further use.
In modern settings, there exist at least two representative kinds of memory settings:
[28]-[31]
— ML computing devices can be equipped with on-chip memory encapsulated with processor
together.
— ML computing devices can b
...