ISO/IEC 25059

Software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Quality model for AI systems

This document outlines a quality model for AI systems and is an application-specific extension to the standards on SQuaRE. The characteristics and sub-characteristics detailed in the model provide consistent terminology for specifying, measuring and evaluating AI system quality. The characteristics and sub-characteristics detailed in the model also provide a set of quality characteristics against which stated quality requirements can be compared for completeness.

Titre manque

General Information

• Status: Not Published
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 42 - Artificial intelligence
• Drafting Committee: ISO/IEC JTC 1/SC 42/WG 3 - Trustworthiness
• Current Stage: 6000 - International Standard under publication
• Completion Date: 04-May-2023

Standards Content (Sample)

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 25059
ISO/IEC JTC 1/SC 42 Secretariat: ANSI
Voting begins on: Voting terminates on:
2022-07-12 2022-10-04
Software engineering — Systems and software Quality
Requirements and Evaluation (SQuaRE) — Quality model
for AI systems
ICS: 35.080
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 25059:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022
---------------------- Page: 1 ----------------------
ISO/IEC DIS 25059:2022(E)
DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 25059
ISO/IEC JTC 1/SC 42 Secretariat: ANSI
Voting begins on: Voting terminates on:
Software engineering — Systems and software Quality
Requirements and Evaluation (SQuaRE) — Quality model
for AI systems
ICS: 35.080
COPYRIGHT PROTECTED DOCUMENT
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
© ISO/IEC 2022
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

NOT BE REFERRED TO AS AN INTERNATIONAL

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on STANDARD UNTIL PUBLISHED AS SUCH.

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

IN ADDITION TO THEIR EVALUATION AS

or ISO’s member body in the country of the requester. BEING ACCEPTABLE FOR INDUSTRIAL,

TECHNOLOGICAL, COMMERCIAL AND
ISO copyright office
USER PURPOSES, DRAFT INTERNATIONAL
CP 401 • Ch. de Blandonnet 8
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
CH-1214 Vernier, Geneva
POTENTIAL TO BECOME STANDARDS TO
Phone: [phone]
WHICH REFERENCE MAY BE MADE IN
Reference number
Email: [e-mail]
NATIONAL REGULATIONS.
Website: www.iso.org ISO/IEC DIS 25059:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
Published in Switzerland
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
© ISO/IEC 2022 – All rights reserved
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022
---------------------- Page: 2 ----------------------
ISO/IEC DIS 25059:2022(E)
1 © ISO 2023

2 All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this

3 publication may be reproduced or utilized otherwise in any form or by any means, electronic or mechanical,

4 including photocopying, or posting on the internet or an intranet, without prior written permission. Permission

5 can be requested from either ISO at the address below or ISO’s member body in the country of the requester.

6 ISO copyright office
7 CP 401 • Ch. de Blandonnet 8
8 CH-1214 Vernier, Geneva
9 Phone: [phone]
10 Fax: [phone]
11 Email: [e-mail]
12 Website: www.iso.org
13 Published in Switzerland
© ISO 2022 – All rights reserved i
---------------------- Page: 3 ----------------------
ISO/IEC DIS 25059:2022(E)
14 Contents

15 Foreword ........................................................................................................................................................................ iii

16 Introduction.................................................................................................................................................................... iv

17 1 Scope .......................................................................................................................................................................... 1

18 2 Normative references .......................................................................................................................................... 1

19 3 Terms and definitions .......................................................................................................................................... 1

20 3.1 General ...................................................................................................................................................................... 1

21 3.2 Product quality ....................................................................................................................................................... 2

22 3.3 Quality in use ........................................................................................................................................................... 2

23 4 Abbreviations .......................................................................................................................................................... 3

24 5 Product quality model ......................................................................................................................................... 3

25 5.1 General ...................................................................................................................................................................... 3

26 5.2 Controllability ......................................................................................................................................................... 3

27 5.3 Functional adaptability ....................................................................................................................................... 3

28 5.4 Functional correctness ........................................................................................................................................ 4

29 5.5 Robustness ............................................................................................................................................................... 4

30 5.6 Transparency .......................................................................................................................................................... 4

31 5.7 Intervenability ........................................................................................................................................................ 5

32 6 Quality in use model ............................................................................................................................................. 5

33 6.1 General ...................................................................................................................................................................... 5

34 6.2 Societal and ethical risk mitigation ................................................................................................................ 6

35 6.3 Transparency .......................................................................................................................................................... 7

36 Annex A (informative) SQuaRE series .................................................................................................................... 8

37 A.1. SQuaRE series divisions ...................................................................................................................................... 8

38 Annex B (informative) How a risk-based approach relates to a quality-based approach and

39 quality models ............................................................................................................................................... 10

40 B.1. General ................................................................................................................................................................... 10

41 B.2. Relationship with standards .......................................................................................................................... 10

42 B.3. Comparison of approaches ............................................................................................................................. 11

43 Annex C (informative) Performance .................................................................................................................... 13

44 Bibliography ................................................................................................................................................................. 14

ii © ISO 2022 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC DIS 25059:2022(E)
46 Foreword

47 ISO (the International Organization for Standardization) is a worldwide federation of national standards

48 bodies (ISO member bodies). The work of preparing International Standards is normally carried out

49 through ISO technical committees. Each member body interested in a subject for which a technical

50 committee has been established has the right to be represented on that committee. International

51 organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO

52 collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

53 electrotechnical standardization.

54 The procedures used to develop this document and those intended for its further maintenance are

55 described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

56 different types of ISO documents should be noted. This document was drafted in accordance with the

57 editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

58 Attention is drawn to the possibility that some of the elements of this document may be the subject of

59 patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of any

60 patent rights identified during the development of the document will be in the Introduction and/or on

61 the ISO list of patent declarations received (see www.iso.org/patents).

62 Any trade name used in this document is information given for the convenience of users and does not

63 constitute an endorsement.

64 For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

65 expressions related to conformity assessment, as well as information about ISO's adherence to the World

66 Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see

67 www.iso.org/iso/foreword.html.

68 This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,

69 Subcommittee SC 42, Artificial intelligence.

70 Any feedback or questions on this document should be directed to the user’s national standards body. A

71 complete listing of these bodies can be found at www.iso.org/members.html.
© ISO 2022 – All rights reserved iii
---------------------- Page: 5 ----------------------
ISO/IEC DIS 25059:2022(E)
72 Introduction

73 High-quality software products and computer systems are crucial to stakeholders. Quality models, quality

74 requirements, quality measurement, and quality evaluation are standardized within the SQuaRE series

75 of international standards (ISO/IEC 25000 [1] to ISO/IEC 25099).

76 AI systems require additional properties and characteristics of systems to be considered, and

77 stakeholders have varied needs. This is because the AI system can be used for decision-making tasks, can

78 be based on noisy and incomplete data, can give probabilistic predictions in some cases, can learn from

79 data and can adapt during operation. Also relevant is the increased automation that occurs in such

80 systems.

81 According to ISO/IEC TR 24028:2020 [2], trustworthiness has been understood and treated as both an

ongoing organizational process as well as a non-functional requirement specifying emergent properties

83 of a system — that is, a set of inherent characteristics with their attributes — within the context of quality

84 of use as indicated in ISO/IEC 25010:2011 [3].

85 ISO/IEC TR 24028:2020 discusses the applicability to AI systems of the ISO/IEC 250xx – SQuaRE series

86 that have been developed for conventional software. According to ISO/IEC TR 24028:2020, the SQuaRE

87 series does not sufficiently address the data-driven unpredictable nature of AI systems. While

88 considering the existing body of work, ISO/IEC TR 24028:2020 identifies the need for developing new

89 standards for AI systems that can go beyond the characteristics and requirements of conventional

90 software development.

91 ISO/IEC TR 24028:2020 contains a related discussion on different approaches to testing and evaluation

92 of AI systems. It states that for testing of an AI system, modified versions of existing software and

93 hardware verification and validation techniques are needed. It identifies several conceptual differences

94 between many AI systems and conventional systems and concludes that “the ability of the [AI] system to

95 achieve the planned and desired result … may not always be measurable by conventional approaches to

96 software testing”. Testing of AI systems is addressed in ISO/IEC TR 29119-11:2020 [4].

97 This document outlines an application-specific AI system extension to the SQuaRE series quality model

98 specified in ISO/IEC 25010:2011 [3]. This document is meant to be used in conjunction with the SQuaRE

99 series.

100 AI systems perform tasks. One or more tasks can be defined for an AI system. For the evaluation of task

101 fulfillment it is necessary to specify quality requirements with evaluation measures.

102 In Clause 3 this document defines terminology that can be used to define quality requirements for AI

103 systems. In Clause 5 and 6 the relevance of these terms is explained, and links to other standardization

104 deliverables (e.g. the ISO/IEC 24029 series [5][6]) are highlighted.

105 ISO/IEC 25012:2008 [7] contains a model for data quality that is complementary to the model defined in

106 this document. ISO/IEC 25012:2008 is being extended for AI systems by the ISO/IEC 5259 series of

107 standards [8].
iv © ISO 2022 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC DIS 25059:2022(E)
108 1 Scope

109 This document outlines a quality model for AI systems and is an application-specific extension to the

110 SQuaRE series. The characteristics and sub-characteristics detailed in the model provide consistent

111 terminology for specifying, measuring and evaluating AI system quality. The characteristics and sub-

112 characteristics detailed in the model also provide a set of quality characteristics against which stated

113 quality requirements can be compared for completeness.
114 2 Normative references

115 The following documents are referred to in the text in such a way that some or all of their content

116 constitutes requirements of this document. For dated references, only the edition cited applies. For

117 undated references, the latest edition of the referenced document (including any amendments) applies.

118 ISO/IEC 25010:2011, Systems and software engineering — Systems and software Quality Requirements and

119 Evaluation (SQuaRE) — System and software quality models

120 ISO/IEC 22989:— , Information technology — Artificial intelligence — Artificial intelligence concepts and

121 terminology

122 ISO/IEC 23053:— , Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)

123 ISO/IEC 24029-2:— , Artificial intelligence (AI) — Assessment of the robustness of neural networks — Part

124 2: Methodology for the use of formal methods
125 3 Terms and definitions

126 For the purposes of this document, the terms and definitions given in ISO/IEC 22989:—, ISO/IEC

127 23053:—, and the following apply.

128 ISO and IEC maintain terminological databases for use in standardization at the following addresses:

129 — ISO Online browsing platform: available at https://www.iso.org/obp
130 — IEC Electropedia: available at http://www.electropedia.org/
131 3.1 General
132 3.1.1
133 measure, noun
134 variable to which a value is assigned as the result of measurement

135 Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and

136 indicators.
137 [SOURCE: ISO/IEC 15939:2007, 3.15]
138 3.1.2
139 measure, verb
140 make a measurement
141 [SOURCE: ISO/IEC 25010:2011, 4.4.6]
142 3.1.3
143 software quality measure

144 measure of internal software quality, external software quality or software quality in use

145 Note 1 to entry: Internal software quality, external software quality and software quality in use are described in the

146 quality model in ISO/IEC 9126-1
Under preparation. Stage at the time of publication ISO/IEC FDIS 22989:2022.
Under preparation. Stage at the time of publication ISO/IEC FDIS 23053:2022.
Under preparation. Stage at the time of publication ISO/IEC CD 24029-2:2021.
© ISO 2022 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC DIS 25059:2022(E)
147 [ISO/IEC 25030:2007, A.55]
148 3.1.4
149 risk treatment measure
150 protective measure
151 action or means to eliminate hazards or reduce risks

152 [SOURCE: ISO/IEC Guide 51:2014, 3.13], modified to change reduction to treatment in order to align with

153 ISO/IEC 23894:— ]
154 3.2 Product quality
155 3.2.1
156 user controllability

157 degree to which a user can appropriately intervene in an AI system’s functioning in a

158 timely manner
159 3.2.2
160 functional adaptability

161 degree to which an AI system can accurately acquire information from data, or the result

162 of previous actions, and use that information in future predictions
163 3.2.3
164 functional correctness

165 degree to which a product or system provides the correct results with the needed degree of precision

166 Note 1 to entry: AI systems, and particularly those machine learning methods, do not usually provide functional

167 correctness in all observed circumstances. Therefore, it is necessary to measure the correctness and incorrectness

168 carefully.
169 [SOURCE ISO/IEC 25010:2011, 4.2.1.2, added note]
170 3.2.4
171 robustness

172 degree to which an AI system can maintain its level of performance under any

173 circumstances

174 [SOURCE ISO/IEC 22989:—, modified to be the degree to which the system has the property, 3.5.12]

175 3.3 Quality in use
176 3.3.1
177 societal and ethical risk mitigation

178 degree to which an AI system mitigates potential risk to society

179 Note 1 to entry: Societal and ethical risk mitigation includes accountability, fairness, transparency and

180 explainability, professional responsibility, promotion of human value, privacy, human control of technology,

181 community involvement and development, respect for the rule of law, respect for international norms of behaviour

182 and labour practices.
183 3.3.2
184 transparency

185 degree to which appropriate information about the AI system is communicated to

186 relevant stakeholders

187 Note 1 to entry: Appropriate information for AI system transparency can include aspects such as features,

188 components, procedures, measures, design goals, design choices and assumptions.

189 [SOURCE ISO/IEC 22989:—, modified to be the degree to which the system has the property, 3.5.14]

Under preparation. Stage at the time of publication ISO/IEC DIS 23894:2022.
2 © ISO 2022 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC DIS 25059:2022(E)
190 3.3.3
191 intervenability

192 degree to which an operator can intervene in an AI system’s functioning in a timely

193 manner to prevent harm or hazard
194 4 Abbreviations
195 AI artificial intelligence
196 ML machine learning
197 5 Product quality model
198 5.1 General

199 An AI system product quality model is detailed in Figure 1 below. The model is based on a modified

200 version of a general system model provided in ISO/IEC 25010:2011. New and modified sub-

201 characteristics are identified using an asterisk. Some of the sub-characteristics have different meanings

202 or contexts as compared to the ISO/IEC 25010:2011 model. The modifications, additions and differences

203 are described in this Clause. The original characteristics shall be interpreted as defined in ISO/IEC

204 25010:2011.
205
206 Figure 1 — AI system product quality model

207 Each of these modified or added sub-characteristics is listed in the remainder of this Clause.

208 5.2 Controllability

209 User controllability is a new sub-characteristic of usability. User controllability is a property of an AI

210 system such that a human or another external agent can intervene in its functioning. Enhanced

211 controllability is helpful if unexpected behaviour cannot be completely avoided and that would lead to

212 negative consequences.

213 User controllability is related to controllability, which is described in ISO/IEC 22989:—, 5.12.

214 5.3 Functional adaptability

215 Functional adaptability is a new sub-characteristic of functional suitability. Functional adaptability of an

216 AI system is the ability of the system to adapt itself to a changing dynamic environment it is deployed in.

217 AI systems can learn from new training data, operational input data and the results of previous actions

© ISO 2022 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO/IEC DIS 25059:2022(E)

218 taken by the system. The concept of functional adaptability is broader than that of continuous learning

219 systems, as defined in ISO/IEC 22989:—, 5.11.9.2.

220 Continuous learning is not a mandatory requirement for functional adaptability. For example, a system

221 that switches classification models based on events in its environment can also be considered functionally

222 adaptive.

223 Functional adaptability in AI systems is unlike other quality characteristics as there are system specific

224 consequences that cannot be interpreted using a straight-line linear scale (e.g. bad to good). Generally,

225 higher functional adaptability can result in improvements for the outcomes enacted by AI systems.

226 For some systems, high functional adaptability can modify the AI system based on added information that

227 can improve the overall output of the system. In other systems, high functional adaptability can cause

228 additional unhelpful outcomes to become more likely based on the system’s previous choices (i.e.

229 weightings of a decision path with relatively high uncertainty that is reinforced based on decisions

230 previously selected by the AI system) providing a higher likelihoods of unintended negative outcomes

231 (e.g. reinforcing a negative human cognitive bias).

232 While conventional algorithms usually produce the same result for the same set of inputs, AI systems,

233 due to continuous learning, can exhibit different behaviour and therefore can produce different results.

234 5.4 Functional correctness

235 Functional correctness exists in ISO/IEC 25010:2011. This model amends the description since AI

236 systems, and particularly those probabilistic ML methods, do not usually provide functional correctness

237 because a certain error rate is expected in their outputs. Therefore, it is necessary to measure correctness

238 and incorrectness carefully. Numerous measurements exist for these purposes in the context of ML

239 methods and examples of these as applicable to a classification model with accompanying relevant

240 examples can be found in ISO/IEC TS 4213:— [12].

241 Additionally, there can be a trade-off between characteristics such as performance efficiency [13],

242 robustness [14] and functional correctness.
243 5.5 Robustness

244 Robustness is a new sub-characteristic of reliability. It is used to describe the ability of a system to

245 maintain its level of functional correctness under any circumstances including:

246 • the presence of unseen, biased, adversarial or invalid data inputs;
247 • external interference;

248 • harsh environmental conditions encompassing generalization, resilience, reliability;

249 • attributes related to the proper operation of the system as intended by its developers.

250 The proper operation of a system is important for the security of the system and safety of its stakeholders

251 in a given environment or context. Information about functional safety in the context of AI systems can

252 be found in ISO/IEC 5469:— [15].

253 Robustness is discussed in ISO/IEC TR 24028:2020, 10.7 [2], and methods for assessment are described

254 in ISO/IEC TR 24029-1:2021 [5] and defined in ISO/IEC 24029-2:—.
255 5.6 Transparency

256 Transparency is sub-characteristic of usability. It relates to the availability of data regarding AI system

257 processes and how amenable that data is to external inspection. Transparency of AI systems can help

258 potential users of AI systems to choose a system fitting their requirements, improving relevant

259 stakeholders’ knowledge about the applicability and the limitations of an AI system as well as assisting

260 with the explainability of AI systems.

261 Transparency relates to the availability of information about an AI system and the way this information

262 is communicated to relevant stakeholders in accordance with their objectives and knowhow. The

Under preparation. Stage at the time of publication ISO/IEC DTS 4213.2:2022.
Under preparation. Stage at the time of publication ISO/IEC AWI 5469:2020.
4 © ISO 2022 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC DIS 25059:2022(E)

263 transparency information can include a description of an AI system functionality, the system’s

264 decomposition, interfaces, ML model(s) used, training data, verification and validation data, performance

265 benchmarks, logs and the management practices of an organization responsible for the system.

266 Transparent systems document, log or display their internal processes using introspection tools and data

267 files. The flow of data can be trackable at each step, with applied decisions, exceptions and rules

268 documented. Log output can track processes in the pipeline as they permute data, as well as system level

269 calls. Errors are logged explicitly, particularly in transform steps. Highly transparent and modular

270 systems can be built of well-documented subcomponents whose interfaces are explicitly described.

271 A system with low transparency has internal workings which are difficult to inspect externally.

272 Transparency of AI systems eases investigations of system malfunctions. Unavailability of detailed

273 processing records can impair testability and societal and ethical impact assessment and risk treatment.

274 Ultimately, transparency of AI systems contributes to establishing of trust, accountability and

275 communication among stakeholders. Some aspects of transparency are discussed in ISO/IEC TR

276 24028:2020, 10.2 [2].

277 Transparency is also a sub-characteristic of satisfaction in the Quality in Use model. See 6.3 for further

278 information.
279 5.7 Intervenability

280 The extent of intervenability can be determined depending on the scenarios where the AI system can be

281 used. The key to intervenability is to enable state observation and transition from an unsafe state to a

282 safe state. Operability is the degree to which an AI system has attributes that make it easy to operate and

283 control, which emphasizes the importance of an AI system’s interface. Compared to operability,

284 intervenability is more fundamental from a quality perspective and intended to prevent an AI system

285 from doing harm or hazard.

286 Intervenability is related to controllability, which is described in ISO/IEC 22989:—, 5.15.5

287 6 Quality in use model
288 6.1 General

289 An AI system quality in use model is detailed in Figure 2. The model is based on a modified version of a

290 general quality in use model provided in ISO/IEC 25010:2011. New and modified sub-characteristics are

291 identified using an asterisk. Some of the sub-characteristics have different meanings or contexts as

292 compared to the ISO/IEC 25010:2011 model. The modifications, additions and differences are described

293 in this Clause. The original characteristics shall be interpreted as defined in ISO/IEC 25010:2011.

© ISO 2022 – All rights reserved 5
---------------------- Page: 11 ----------------------
ISO/IEC DIS 25059:2022(E)
294
295 Figure 2 — AI system quality in use model
296 6.2 Societal and ethical risk mitigation

297 Societal and ethical risk mitigation is a new sub-characteristic of freedom from risk.

298 ISO/IEC TR 24368:— [16], explores this topic and outlines the following themes:

299 — accountability;
300 — fairness and non-discrimination;
301 — transparency and explainability;
302 — professional responsibility;
303 — promotion of human values;
304 — privacy;
305 — human control of technology;
306 — community involvement and development;
307 — human centred design;
308 — respect for the rule of law;
309 — respect for internation
...