ISO/IEC PRF 4944
(Main)Information technology — User interfaces — Evaluating usability of natural user interfaces
Information technology — User interfaces — Evaluating usability of natural user interfaces
This document provides a framework of evaluating usability of natural user interface (NUI) for systems, products or services. The usability evaluation focuses on efficiency, effectiveness, and satisfaction of NUI. Requirements and recommendations on evaluating usability of NUI and other UI for emerging technologies are also presented. Measuring and reporting the usability of NUI are described in the document.
Technologie de l'information — Interfaces utilisateur — Cadre pour l'évaluation de l'utilisabilité d'une interface utilisateur naturelle
General Information
Buy Standard
Standards Content (Sample)
International
Standard
ISO/IEC 4944
First edition
Information technology — User
interfaces — Evaluating usability of
natural user interfaces
Technologie de l'information — Interfaces utilisateur — Cadre
pour l'évaluation de l'utilisabilité d'une interface utilisateur
naturelle
PROOF/ÉPREUVE
Reference number
ISO/IEC 4944:2024(en) © ISO/IEC 2024
ISO/IEC 4944:2024(en)
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
PROOF/ÉPREUVE
© ISO/IEC 2024 – All rights reserved
ii
ISO/IEC 4944:2024(en)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative reference . 1
3 Terms and definitions . 1
3.1 Terms related to natural user .1
3.2 Terms related to usability .2
4 Natural user interfaces (NUI). 2
5 Evaluating usability of a natural user interface (NUI) . 3
5.1 Framework .3
5.2 NUI as a system, product or service being evaluated .4
5.3 Evaluating usability for diverse NUI users .4
5.4 Evaluating usability for goals, tasks and interactions of NUI.5
5.5 Evaluating usability for diverse contexts of use of NUI .5
5.6 Components of usability.6
5.7 Evaluation based on use .6
6 Measuring usability of a NUI . 7
6.1 Purposes of and measures for evaluation .7
6.2 Measures for evaluating the user interface of NUI .8
6.2.1 Evaluating user interface of NUI for accomplishing tasks .8
6.2.2 Effectiveness measures .8
6.2.3 Efficiency measures.9
6.2.4 Satisfaction measures . .10
6.3 Measures for evaluating naturalness of interactions with NUI.10
6.3.1 Evaluating naturalness of interactions with NUI for accomplishing tasks .10
6.3.2 Effectiveness measures .10
6.3.3 Efficiency measures.11
6.3.4 Satisfaction measures . . 12
7 Reporting usability evaluation results .12
7.1 Content elements for reporting usability evaluations . 12
7.2 Format for reporting usability evaluations . 12
Bibliography .13
PROOF/ÉPREUVE
© ISO/IEC 2024 – All rights reserved
iii
ISO/IEC 4944:2024(en)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 35, User interfaces.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
PROOF/ÉPREUVE
© ISO/IEC 2024 – All rights reserved
iv
ISO/IEC 4944:2024(en)
Introduction
As smart devices such as smart phones and smart speakers have become popular, new forms of user
interfaces have been introduced. One aspect of the new interfaces is mimicking natural ways of interacting
with the devices and utilizing speech, gestures, body language, eye gaze and other natural features of
human communication through natural user interfaces (NUIs), which allow for faster, smoother, intuitive,
and smarter interaction. However, while the concept of NUI is well defined, practical applications of NUI still
need to be developed. This is strongly related to the lack of usability of NUI in smart devices.
When specified users of systems, products or services achieve their specified goals with effectiveness,
efficiency and satisfaction in a specified context of use (as defined in ISO/IEC 9241-11), usability can be
evaluated. Effectiveness is considered as the ability of users to complete tasks with the system (e.g. the NUI).
Efficiency measures resources taken by users to complete specified tasks. Satisfaction measures comfort
levels of user experience along with acceptance.
It is important that the usability evaluation of NUI concentrates on recognizing the user's intention and
focuses on multiple modalities of information flows between the users and the devices.
This document describes a framework for evaluating the usability of NUI. Although the framework can be
applied in most general contexts, it is specifically focused on optimizing user experiences in terms of NUI.
PROOF/ÉPREUVE
© ISO/IEC 2024 – All rights reserved
v
International Standard ISO/IEC 4944:2024(en)
Information technology — User interfaces — Evaluating
usability of natural user interfaces
1 Scope
This document provides a framework, requirements and recommendations for evaluating the usability of
a natural user interface (NUI) for systems, products or services. Measuring and reporting the usability of
the NUI are also described. The usability evaluation focuses on the efficiency, effectiveness and satisfaction
of the NUI. This document can also be applied to the usability evaluation of other user interfaces (UIs) for
emerging technologies.
2 Normative reference
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO 9241-11, Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 9241-11 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1 Terms related to natural user
3.1.1
user interface
UI
set of all the components of an interactive system that provide information and controls for the user to
accomplish specific tasks with the interactive system
[SOURCE: ISO 9241-110:2020, 3.10]
3.1.2
natural user interface
NUI
user interface where the user operates the interface through a manner related to everyday human
behaviours in a real world
Note 1 to entry: Natural refers to the user's expectations, behaviour and feelings that are involved in interacting with a NUI.
Note 2 to entry: Everyday human behaviours refer to behaviours that the user can be expected to have already learned.
Note 3 to entry: Interactions with NUIs typically model interactions performed in the natural world, including (but not
limited to) touch, gestures or vocal interactions, or a combination thereof.
PROOF/ÉPREUVE
© ISO/IEC 2024 – All rights reserved
ISO/IEC 4944:2024(en)
3.1.3
natural user interface interaction
NUI interaction
user-system interaction making use of a natural user interface
3.1.4
naturalness
correspondence with everyday interactions between humans and the real-world objects with which they
interact
Note 1 to entry: Naturalness relates to everyday behaviours that the user can be expected to have already learned.
EXAMPLE To perform a “hello” gesture, the user refers to an object by pointing; to order something the user
issues a vocal command; or to agree the user nods in the affirmative or voices a vocal onomatopoeia for agreement.
3.2 Terms related to usability
3.2.1
interaction
user-system interaction
exchange of information between a user and an interactive system via the user interface to complete the
intended task
Note 1 to entry
...
Date: 2023-04
ISO/IEC DIS PRF 4944:2023(E)
ISO/IEC TC JTC1 JTC 1/SC 35/WG 9
Secretariat: AFNOR
Date: 2024-07-22
Information technology — User interfaces — Framework for
evaluating the — Evaluating usability of a natural user interfaces
Technologie de l'information — Interfaces utilisateur — Cadre pour l'évaluation de l'utilisabilité d'une
interface utilisateur naturelle
Copyright notice
ThisFDIS stage
ISO/IEC DIS 4944:2022(E)
© ISO document is a working draft/IEC 2024
All rights reserved. Unless otherwise specified, or committee draft and is copyright-protected by ISO. While the
reproductionrequired in the context of working drafts or committee drafts in any form for use by participants in the ISO
standards development process is permitted without prior permission from ISO, neither its implementation, no part of
this document nor any extract from itpublication may be reproduced, stored or utilized otherwise in any form or
transmitted in any form for any other purposeby any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission from ISO. Permission can be requested from either ISO
at the address below or ISO’s member body in the country of the requester.
Requests for permission to reproduce this document for the purpose of selling it should be addressed as shown below
or to ISO's member body in the country of the requester:
[Indicate the full address, telephone number, fax number, telex number, and electronic mail address, as appropriate,
of the Copyright Manager of the ISO member body responsible for the secretariat of the TC or SC within the framework
of which the working document has been prepared.]
Reproduction for sales purposes may be subject to royalty payments or a licensing agreement.
Violators may be prosecuted.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: + 41 22 749 01 11
E-mail: copyright@iso.org
Website: www.iso.org
Published in Switzerland
iii © ISO/IEC 2022 – All rights reserved
ISO/IEC DISPRF 4944:2022(E2024(en)
Contents Page
Foreword . vi
Introduction . vii
1 Scope . 1
2 Normative reference . 1
3 Terms and definitions . 1
3.1 Terms related to natural user . 1
3.2 Terms related to usability . 2
4 Natural user interfaces (NUI) . 4
5 Evaluating usability of a natural user interface (NUI) . 5
5.1 Framework . 5
5.2 NUI as a system, product or service being evaluated . 7
5.3 Evaluating usability for diverse NUI users . 8
5.4 Evaluating usability for goals, tasks and interactions of NUI . 8
5.5 Evaluating usability for diverse contexts of use of NUI . 9
5.6 Components of usability . 9
5.7 Evaluation based on use . 10
6 Measuring usability of a NUI . 11
6.1 Purposes of and measures for evaluation . 11
6.2 Measures for evaluating the user interface of NUI . 12
6.2.1 Evaluating user interface of NUI for accomplishing tasks. 12
6.2.2 Effectiveness measures . 12
6.2.3 Efficiency measures . 13
6.2.4 Satisfaction measures . 14
6.3 Measures for evaluating naturalness of interactions with NUI . 14
6.3.1 Evaluating naturalness of interactions with NUI for accomplishing tasks . 14
6.3.2 Effectiveness measures . 15
6.3.3 Efficiency measures . 16
6.3.4 Satisfaction measures . 16
7 Reporting usability evaluation results . 17
7.1 Content elements for reporting usability evaluations . 17
7.2 Format for reporting usability evaluations . 17
Bibliography . 18
Foreword . 3
Introduction . 4
1 Scope . 5
2 Normative reference . 5
3 Terms and definitions . 5
3.1 Natural user interface terms and definitions . 5
3.2 Usability terms and definitions . 6
4 Natural user interfaces . 8
iv © ISO/IEC 2022 – All rights reserved
© ISO/IEC 2024 – All rights reserved
iv
ISO/IEC DISPRF 4944:2022(E2024(en)
5 Evaluating usability of a Natural User Interface (NUI) . 9
5.1 Framework . 9
5.2 NUI as a system, product, or service being evaluated . 10
5.3 Evaluating usability for diverse NUI users . 10
5.4 Evaluating usability for goals, tasks, and interactions of NUI . 11
5.5 Evaluating usability for diverse contexts of use of NUI . 11
5.6 Components of usability . 12
5.7 Evaluation based on use . 13
6 Measuring usability of a NUI . 14
6.1 Purposes of and measures for evaluation . 14
6.2 Measures for evaluating the user interface of NUI . 14
6.2.1 Evaluating user interface of NUI for accomplishing tasks. 14
6.2.2 Effectiveness measures . 14
6.2.3 Efficiency measures . 15
6.2.4 Satisfaction measures . 16
6.3 Measures for evaluating naturalness of interactions with NUI . 17
6.3.1 Evaluating naturalness of interactions with NUI for accomplishing tasks . 17
6.3.2 Effectiveness measures . 17
6.3.3 Efficiency measures . 18
6.3.4 Satisfaction measures . 19
7 Reporting usability evaluation results . 19
7.1 Content elements for reporting usability evaluations . 19
7.2 Format for reporting usability evaluations . 19
© ISO/IEC 2022 – All rights reserved v
© ISO/IEC 2024 – All rights reserved
v
ISO/IEC DISPRF 4944:2022(E2024(en)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members
of ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types of
document should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC
Directives, Part 2 (see www.iso.org/directiveswww.iso.org/directives or
www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the use of
(a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any claimed
patent rights in respect thereof. As of the date of publication of this document, ISO and IEC [had/had not]
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch.www.iso.org/patents and
https://patents.iec.ch. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html.www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-
standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 35, User interfaces.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html and www.iec.ch/national-
committeeswww.iso.org/members.html and www.iec.ch/national-committees.
vi © ISO/IEC 2022 – All rights reserved
© ISO/IEC 2024 – All rights reserved
vi
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.