Human Factors (HF) - User Experience Guidelines for real-time communication services expressed in Quality of Service terms

The present document provides guidelines for the user experience of real-time communication services. The services include person-to-person (two-way) communication and person-to-machine (one-way) communication. The present document is revised from EG 202 534 [i.11] that was restricted to person-to-person communication services. The revision adds new guidelines that have been requested by stakeholders and omits some of the previous guidelines that are no longer considered relevant. The guidelines are based on empirical data about user experience. Most of the data is obtained from scientific papers. A minority of guidelines are based on existing standardisation documents to ensure that relevant normative and informative standards material is clear amongst results available from journal articles, conference papers, research reports, etc. The present document does not replace any existing standards. Most of the guidelines derived from scientific papers are specific to a particular context, in that the original user tests were for specific tasks, users and technical parameters and therefore the results may not be generalisable. Although the guidelines provide information about the main user experience measure(s) and technical parameter(s) of a particular test result, it is beyond the scope of the present document to provide all of the variables concerned with each user test. However, the origin of each empirical source of a guideline is shown and listed in the References.

Človeški dejavniki (HF) - Smernice iz uporabniških izkušenj za realnočasovne komunikacijske storitve, izražene z izrazi za kakovost storitve (QoS)

Pričujoči dokument zagotavlja smernice iz uporabniških izkušenj za realnočasovne komunikacijske storitve. Storitve vključujejo komunikacijo med dvema osebama (dvosmerno) in komunikacijo med osebo in napravo (enosmerno). Pričujoči dokument je revidiran iz EG 202 534 [i.11], ki je bil omejen na komunikacijske storitve med dvema osebama. Revizija dodaja nove smernice, ki so jih zahtevali deležniki, in opušča nekatere predhodne smernice, ki ne veljajo več za pomembne. Smernice temeljijo na empiričnih podatkih iz uporabniških izkušenj. Večina podatkov izhaja iz znanstvenih člankov. Manjši delež smernic temelji na obstoječih standardizacijskih dokumentih, ki zagotavljajo, da je material o pomembnih normativnih in informativnih standardih jasno ločen od rezultatov, ki so na voljo v člankih iz revij, prispevkih na konferencah, raziskovalnih poročilih itd. Pričujoči dokument ne nadomešča nobenega od obstoječih standardov. Večina smernic, ki izhajajo iz znanstvenih člankov, je specifična za določen kontekst, v smislu, da so bili izvirni uporabniški preskusi namenjeni specifičnim nalogam, uporabnikom in tehničnim parametrom, zato rezultatov morda ni mogoče posplošiti. Čeprav smernice zagotavljajo informacije o glavnih merilih uporabniških izkušenj in tehničnih parametrih rezultata določenega preskusa, ni v obsegu uporabe pričujočega dokumenta, da bi podal vse spremenljivke, povezane z vsakim uporabniškim preskusom. Vir vsakega empiričnega vira smernice pa je prikazan in naveden v referencah.

General Information

Status
Published
Publication Date
08-Nov-2010
Current Stage
6060 - National Implementation/Publication (Adopted Project)
Start Date
07-Sep-2010
Due Date
12-Nov-2010
Completion Date
09-Nov-2010

Buy Standard

Standard
ETSI EG 202 670 V1.1.1 (2010-03) - Human Factors (HF); User Experience Guidelines for real-time communication services expressed in Quality of Service terms
English language
98 pages
sale 15% off
Preview
sale 15% off
Preview
Standard
ETSI EG 202 670 V1.1.1 (2009-12) - Human Factors (HF); User Experience Guidelines for real-time communication services expressed in Quality of Service terms
English language
98 pages
sale 15% off
Preview
sale 15% off
Preview
Guide
SIST EG 202 670 V1.1.1:2010
English language
98 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day

Standards Content (sample)

ETSI EG 202 670 V1.1.1 (2010-03)
ETSI Guide
Human Factors (HF);
User Experience Guidelines
for real-time communication services
expressed in Quality of Service terms
---------------------- Page: 1 ----------------------
2 ETSI EG 202 670 V1.1.1 (2010-03)
Reference
DEG/HF-00112
Keywords
interaction, quality, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org

The present document may be made available in more than one electronic version or in print. In any case of existing or

perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).

In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive

within ETSI Secretariat.

Users of the present document should be aware that the document may be subject to revision or change of status.

Information on the current status of this and other ETSI documents is available at

http://portal.etsi.org/tb/status/status.asp

If you find errors in the present document, please send your comment to one of the following services:

http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2010.
All rights reserved.
TM TM TM TM

DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered

for the benefit of its Members.

3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.

LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.

GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.

ETSI
---------------------- Page: 2 ----------------------
3 ETSI EG 202 670 V1.1.1 (2010-03)
Contents

Intellectual Property Rights ................................................................................................................................ 5

Foreword ............................................................................................................................................................. 5

Introduction ........................................................................................................................................................ 5

1 Scope ........................................................................................................................................................ 7

2 References ................................................................................................................................................ 7

2.1 Normative references ......................................................................................................................................... 7

2.2 Informative references ........................................................................................................................................ 8

3 Definitions and abbreviations ................................................................................................................. 11

3.1 Definitions ........................................................................................................................................................ 11

3.2 Abbreviations ................................................................................................................................................... 15

4 Overview of the real-time communication services addressed in the present document ....................... 16

4.1 Text communication ......................................................................................................................................... 20

4.2 Speech communication ..................................................................................................................................... 20

4.3 Video communication (including speech) ........................................................................................................ 20

4.3.1 Face-to-face video communication ............................................................................................................. 20

4.3.2 Remote inspection or 'see what I see' (SWIS) video communication ......................................................... 20

4.4 Multimedia communication.............................................................................................................................. 21

4.5 Real-time games ............................................................................................................................................... 21

4.6 TV including PC TV and Mobile TV ............................................................................................................... 21

5 Overview of user experience guidelines expressed in Quality of Service terms .................................... 21

6 Advice on using the Guide ..................................................................................................................... 24

6.1 Requirements for guidelines ............................................................................................................................. 25

6.2 The structure of the guidelines ......................................................................................................................... 25

6.3 Maturity and restrictions of the guidelines ....................................................................................................... 26

7 The guidelines ........................................................................................................................................ 26

7.1 Text communication ......................................................................................................................................... 27

7.2 Speech communication ..................................................................................................................................... 30

7.3 Video communication (including speech) ........................................................................................................ 41

7.3.1 Video communication: Face-to-face ........................................................................................................... 41

7.3.2 Video communication: Remote inspection ................................................................................................. 57

7.3.3 Video communication: Multi-point and heterogeneous networks .............................................................. 63

7.4 Multimedia communication.............................................................................................................................. 64

7.5 Real-time games ............................................................................................................................................... 72

7.6 TV .................................................................................................................................................................... 74

7.6.1 PC TV ......................................................................................................................................................... 74

7.6.2 Mobile TV .................................................................................................................................................. 76

7.7 Service selection ............................................................................................................................................... 85

Annex A: Requirements for guidelines and a web-based guideline and tutorial system ................. 92

A.1 Requirement derivation process ............................................................................................ ................. 92

A.2 Requirements for guidelines ................................................................................................................... 92

A.2.1 Provide information on key topics of concern that will aid development choices ........................................... 92

A.2.2 Provide information on related concepts .......................................................................................................... 93

A.2.3 Provide user experience data that can be used from different perspectives ...................................................... 93

A.2.4 Link QoS and user experience variables .......................................................................................................... 93

A.2.5 Provide information about user behaviour that is feasible to apply .................................................................. 94

A.3 Requirements for a web-based system ................................................................................................... 94

Annex B: Sources for the guidelines ..................................................................................................... 95

ETSI
---------------------- Page: 3 ----------------------
4 ETSI EG 202 670 V1.1.1 (2010-03)

B.1 Bespoke user studies .............................................................................................................................. 95

B.1.1 Laboratory experiments .................................................................................................................................... 95

B.1.2 Field studies...................................................................................................................................................... 96

B.1.3 Survey .............................................................................................................................................................. 96

B.1.4 Expert review and expert panels ....................................................................................................................... 96

B.2 Existing literature ................................................................................................................................... 97

History .............................................................................................................................................................. 98

ETSI
---------------------- Page: 4 ----------------------
5 ETSI EG 202 670 V1.1.1 (2010-03)
Intellectual Property Rights

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information

pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found

in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in

respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web

server (http://webapp.etsi.org/IPR/home.asp).

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee

can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web

server) which are, or may be, or may become, essential to the present document.
Foreword

This ETSI Guide (EG) has been produced by ETSI Technical Committee Human Factors (HF).

The present document is primarily for the HF community.

The present document updates and replaces EG 202 534 [i.11]. New guidelines have been added and some guidelines

from EG 202 534 [i.11] were not considered relevant for the present document (e.g. because of being out of date with

technology developments). In addition, whereas EG 202 534 [i.11] focused on person-to-person (two-way)

communication services, the present document extends to real-time person-to-machine (one-way) communication

services.
Introduction

The real-time communication services that are currently available and in development place different demands on the

communication channel and terminal equipment. Also they continue to evolve for both mobile and fixed usage. The

services offer great potential but also complex choices regarding the most appropriate technologies and media that are

suitable for different communication situations. Taking video communication as an example, there are advances at the

two extremes of usage: for video calls with mobile telephones and conference-room 'Telepresence' systems and between

these extremes video communication is increasingly available through personal computer-based 'Web-conferencing'

services.

TR 102 353 [i.64] identified the need to develop guidelines on real-time communication services. The main aim is to

provide guidelines for network operators, equipment manufacturers and service providers that address the user

experience of different communication services in terms of either:

• the configuration and quality of service (QoS) of a particular communication service;

• the best choice between different communication services, where user experience data exists to enable

recommendations on the basis of of set-up time, reliability, type of user tasks and user situations.

The general requirements for the guidelines contained in the present document are / main inclusion criteria for the

guidelines are:

• Empirical basis - the guidelines are based on user test results where data has been collected on user

performance and/or user opinion. The guidelines are therefore justified on the basis of either a laboratory

experiment, field study, survey or an expert panel of people with empirical knowledge gained through

experience with users or user tests.

• Known QoS - the test systems were of specified QoS parameter values, thereby enabling a statement of

expected user experience for a given service with particular QoS.

• Technology independent - because the user experience results were related to test systems with specified QoS

rather than a particular technology.
ETSI
---------------------- Page: 5 ----------------------
6 ETSI EG 202 670 V1.1.1 (2010-03)

• Vendor independent - because the user experience results were related to test systems with specified QoS

rather than a particular vendor.

• User communication phase - the guidelines concern user experience once communication is established,

therefore the guidelines do not address interface design and call control;

• Stakeholder interest - priority has been given to developing guidelines in areas where stakeholder interest has

been shown or is anticipated. Therefore Guidelines are provided for topics that have been identified as

important for intended guideline users and for which user-based data existed or could be collected. For this

reason, the development of guidelines from published literature is not exhaustive.

The guidelines are grouped into topics that include different user tasks (e.g. decision making, negotiation, persuasion),

technical parameters (e.g. delay, packet loss, frame-rate) and special user groups (e.g. people with speech impairments,

deaf and hearing impaired people, people with cognitive impairments). The user test results are derived primarily from

industry and European Commission Framework Programme research reports, journal articles, conference proceedings

and standards documents.
ETSI
---------------------- Page: 6 ----------------------
7 ETSI EG 202 670 V1.1.1 (2010-03)
1 Scope

The present document provides guidelines for the user experience of real-time communication services. The services

include person-to-person (two-way) communication and person-to-machine (one-way) communication.

The present document is revised from EG 202 534 [i.11] that was restricted to person-to-person communication

services. The revision adds new guidelines that have been requested by stakeholders and omits some of the previous

guidelines that are no longer considered relevant.

The guidelines are based on empirical data about user experience. Most of the data is obtained from scientific papers. A

minority of guidelines are based on existing standardisation documents to ensure that relevant normative and

informative standards material is clear amongst results available from journal articles, conference papers, research

reports, etc.
The present document does not replace any existing standards.

Most of the guidelines derived from scientific papers are specific to a particular context, in that the original user tests

were for specific tasks, users and technical parameters and therefore the results may not be generalisable. Although the

guidelines provide information about the main user experience measure(s) and technical parameter(s) of a particular test

result, it is beyond the scope of the present document to provide all of the variables concerned with each user test.

However, the origin of each empirical source of a guideline is shown and listed in the References.

Those readers interested in the details of a particular guideline empirical source are also directed to a web-based system

(http://portal.etsi.org/STFs/STF_HomePages/STF354/STF354.asp) that provides this information.

2 References

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific.
• For a specific reference, subsequent revisions do not apply.

• Non-specific reference may be made only to a complete document or a part thereof and only in the following

cases:

- if it is accepted that it will be possible to use all future changes of the referenced document for the

purposes of the referring document;
- for informative references.

Referenced documents which are not found to be publicly available in the expected location might be found at

http://docbox.etsi.org/Reference.

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee

their long term validity.
2.1 Normative references

The following referenced documents are indispensable for the application of the present document. For dated

references, only the edition cited applies. For non-specific references, the latest edition of the referenced document

(including any amendments) applies.
Not applicable.
ETSI
---------------------- Page: 7 ----------------------
8 ETSI EG 202 670 V1.1.1 (2010-03)
2.2 Informative references

The following referenced documents are not essential to the use of the present document but they assist the user with

regard to a particular subject area. For non-specific references, the latest version of the referenced document (including

any amendments) applies.

[i.1] Anderson, A., Smallwood, S., MacDonald, R., Mullin, J., Fleming, A. & O'Malley, C. (2000):

"Video data and video links in mediated communication: What do users value? International

Journal of Human-Computer Studies", 52(1), 165-187.

[i.2] ANSI T1.522-2000: "Quality of Service for Business Multimedia Conferencing".

[i.3] Bekkering, E. and Shim, J. (2006). "i2i Trust in Videoconferencing", July 2006/Vol. 49, No. 7

Communications of the ACM, pp. 103-107.

[i.4] Blythe, M.; Overbeeke, K., Monk, A., Wright, P. (Eds) (2004). "Funology: From Usability to

Enjoyment" Kluwer Academic.

[i.5] Brennan, D., Georgeadis, A., Baron, C., Barker, L. (2004): "The Effect of Videoconference-Based

Telerehabilitation on Story Retelling Performance by Brain-Injured Subjects and Its Implications

for Remote Speech-Language Therapy", Telemedicine Journal and e-Health. June 1, 2004,

10(2): 147-154.

[i.6] Brooks, P., Schliemann, T., Hestnes, B., Frowein, H., Aaby, C., O'Malley, C. (2003): ""Final

Report Project IST-1999-11577 Eye-2-Eye: Fitness-for-Purpose of Person-Person Communication

Technologies". EC Deliverable IST11577/SEF/DIS/DS/Pub/008/b1, June 2003.

[i.7] Brooks, P. & Hestnes, B. (2003): "User-centred technical guidelines for real-time human

communication services: Requirements and derivation". Proceedings of the 19th International

Symposium on Human Factors in Telecommunication, Berlin, Germany, December 1-4 2003,

pp. 11-18.

[i.8] Cullum, C., Weiner, M. Gehrmann, H. Hynan, L. (2006): "Feasibility of Telecognitive Assessment

in Dementia. Assessment", Volume 13, No. 4, December 2006 385-390.

[i.9] Cuny, R., Li, M., Kristensson, M. (2006): "Mobile service applications and performance in

UMTS", In Soldani, D., Li, M., Cuny, R. (Eds) "QoS and QoE Management in UMTS Cellular

Systems", John Wiley & Sons, Ltd.

[i.10] Doherty-Sneddon, G., Anderson, A., O' Malley, C., Langton, S., Garrod, S. & Bruce, V. (1997):

"Face-to-face and video mediated communication: A comparison of dialogue structure and task

performance", Journal of Experimental Psychology: Applied , 3(2), 105-125.

[i.11] ETSI EG 202 534: "Human Factors (HF); Guidelines for real-time person-to-person

communication services".

[i.12] ETSI ES 202 667 : "Speech and multimedia Transmission Quality (STQ); Audiovisual QoS for

communication over IP networks".

[i.13] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.14] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective

as perceived by the user".

[i.15] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission

requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.16] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as

perceived by the user".
ETSI
---------------------- Page: 8 ----------------------
9 ETSI EG 202 670 V1.1.1 (2010-03)

[i.17] ETSI ETR 292: "Terrestrial Trunked Radio (TETRA); Voice plus Data (V+D); Technical

requirements specification; Network management".

[i.18] ETSI TR 102 535: "Human Factors (HF); Guidelines for real-time person-to-person

communication services; Future requirements".

[i.19] ETSI ETR 334: "Human Factors (HF); The implications of human ageing for the design of

telephone terminals".
[i.20] ETSI ETR 297: "Human Factors (HF); Human Factors in Videotelephony".

[i.21] ETSI TS 122 105: "Universal Mobile Telecommunications System (UMTS); Services and service

capabilities (3GPP TS 22.105)".

[i.22] ETSI TS 123 107: "Digital cellular telecommunications system (Phase 2+); Universal Mobile

Telecommunications System (UMTS); LTE; Quality of Service (QoS) concept and architecture

(3GPP TS 23.107 Release 8)".

[i.23] Følstad, A., Brooks, P., Heim, J., Schliemann, T., Wiig., S., Hestnes, B., Heiestad, S., Ulseth, T.,

Frowein, H., Aaby, C., O'Malley, C., Brundell, P., Lonsdale, P. (2002): "Results of Field

Experiments of Communication Media. IST Project 1999-11577. Eye-2-Eye: Fitness-for-purpose

of Person-Person Communication Technologies", CEC Deliverable
IST11577/SEF/DIS/DS/Pub/004/b1, October 2002.

[i.24] France, E. F., Anderson, A. H. & Gardner, M. (2001): "The impact of status and audio

conferencing technology on business meetings", International Journal of Human Computer

Studies, 54: 857-876.

[i.25] Frowein, H., Donker, W., Devoldere, P., Schliemann, T., Brooks, P., Følstad, A., Hestnes, B. et al.

(2003): "Specification of Cost-Benefit Analysis Tool. Project IST-1999-11577 Eye-2-Eye:

Fitness-for-Purpose of Person-Person Communication Technologies", EC Deliverable
IST11577/IvD/RAD/DS/FP5/052/b1, May 2003.

[i.26] Ghinea, G., Thomas, J. (2005). "Quality of Perception: User Quality of Service in Multimedia

Presentations", IEEE Transactions on Multimedia, Vol. 7, No. 4, August 2005, pp. 786-789.

[i.27] Grayson, D. & Coventry, L. (1998) "The Effects of Visual Proxemic Information in Video

Mediated Communication", SIGCHI Bulletin Vol.30 No.3, July 1998.

[i.28] Hellström, G., Aström, L. (2008): "The value of total conversation call with emergency services,

Proceedings of International Symposuim on Human Factors in Telecommunications", Kuala

Lumpa, Malaysia, 17-20 March 2008.

[i.29] Hestnes, B., Brooks, P., Heiestad, S. (2004): "Mobile Eye-phone - a study of relevance,

effectiveness and user-perceived suitability", Fornebu, Telenor R&D Report (Scientific Report

R2/2004).

[i.30] Hestnes, B., Brooks, P., Heiestad, S., Ulseth, T., Aaby., C. (2003): "Quality of Experience in

real-time person-person communication services - User based QoS expressed in technical network

QoS terms", proceedings of the 19th International Symposium on Human Factors in
Telecommunication, Berlin, Germany, December 1-4 2003, pp. 3-10.

[i.31] ISO 9241 (all parts): "Ergonomic requirements for office work with visual display terminals

(VDTs)".

[i.32] ITU-R Recommendation BT.1359-1 (2003): "Method for the Subjective Assessment of

Intermediate Quality Level of Coding Systems".

[i.33] ITU-T Series H: "Audiovisual and multimedia systems, Supplement 1 (05/99) (1999) Application

profile - Sign language and lip-reading real-time conversation using low bit-rate video

communication".

[i.34] ITU-T Recommendation E.800 (2008): "Definitions of terms related to quality of service".

[i.35] ITU-T Recommendation F.700 (2000): Framework Recommendation for multimedia services.

ETSI
---------------------- Page: 9 ----------------------
10 ETSI EG 202 670 V1.1.1 (2010-03)

[i.36] ITU-T Recommendation G.108 (1999): "Application of the E-model: A planning guide".

[i.37] ITU-T Recommendation G.114 (2003): "One-way transmission time", International.

[i.38] ITU-T Recommendation G.1010 (2001): "End-user multimedia QoS categories".

[i.39] ITU-T Recommendation G.1080: "Quality of experience requirements for IPTV services".

[i.40] ITU-T Recommendation P.10/G.100: "Amendment 2: New definitions for inclusion in

Recommendation ITU-T P.10/G.100", International Telecommunication Union, Geneva,
Switzerland, 2008.

[i.41] ITU-T Recommendation Y.1541 (2006): "Network performance objectives for IP-based services",

International Telecommunication Union, Geneva, Switzerland.

[i.42] Jumisko-Pyykkö, S., Vinod Kumar, M. V., Liinasuo, M., & Hannuksela, M. (2006): Acceptance of

audiovisual quality in erroneous television sequences over a DVB-H channel. In Proceedings of

the Second International Workshop in Video Processing and Quality Metrics for Consumer

Electronics".

[i.43] Kilgore, R.M., and Chignell, M., (2006): "Listening to unfamiliar voices in spatial audio: Does

visualization of spatial position enhance voice identification? Proceedings of the 20th International

Symposium on Human Factors in Telecommunication", Sophia Antipolis, France, March 21-23

2006.
NOTE: Available from http://www.hft.org/HFT06/HFT_06_programme.htm).

[i.44] Knoche, H., McCarthy, J. D. and Sasse, M.A. (2005): "Can small be beautiful?: assessing image

resolution requirements for mobile TV", Proc. ACM Multimedia 2005, ACM Press, 829-838.

[i.45] Knoche, H., McCarthy J., Sasse, M. A. (2006): "Reading the Fine Print: The Effect of Text

Legibility on Perceived Video Quality in Mobile TV", ACM Multimedia, ACM, 727-730.

[i.46] Knoche, H.O., Sasse, M.A. (2008): "The sweet spot: how people trade off size and definition on

mobile devices. In Proceeding of the 16th ACM international conference on Multimedia",

Vancouver, British Columbia, Canada: ACM, pp. 21-30. .

[i.47] Mashima, P., Birkmire-Peters, D., Syms, M., Holtel, M., Burgess, L., Peters, L. (2003):

"Telehealth: Voice Therapy Using Telecommunications Technology", American Journal of

Speech-Language Pathology Vol.12 432-439.

[i.48] Mason, S. (2006): "Mobile TV - results from the DVB-H trial in Oxford", EBU TECHNICAL

REVIEW - April 2006.
NOTE: Available at http://tech.ebu.ch/docs/techreview/trev_306-mason.pdf.

[i.49] McCarthy, J., Sasse, M. A. & Miras, D. (2004): "Sharp or Smooth? Comparing the effects of

quantization vs. frame rate for streamed video. Proceedings of CHI 2004", Vienna, Austria,

April 20-24, pp. 535-542.

[i.50] Mulbach, L., Bocker, M. & Prussog, A. (1995): "Telepresence in Videocommunications: A Study

on Stereoscopy and Individual Eye Contact", Human Factors, 37(2), 290-305.

[i.51] Nokia (2004): "Quality of Experience (QoE) of mobile services: Can it be measured and

improved?", White Paper, Nokia Corporation, Finland, 2004.

[i.52] O' Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G & Bruce, V. (1996): "Comparison

of Face-to-Face and Video-Mediated Interaction. Interacting with Computers", 8(2), 177-192.

[i.53] O'Malley, C., Brundell, P., McFadzean, J., Lonsdale, P., Schliemann, T., Brooks, P. et al. (2002):

"Results of Laboratory Experiments of Communication Media. IST Project 1999-11577.

Eye-2-Eye: Fitness-for-purpose of Person-Person Communication Technologies", CEC
Deliverable IST11577/UON/SOP/DS/Pub/003/b1, December 2002.
ETSI
---------------------- Page: 10 ----------------------
11 ETSI EG 202 670 V1.1.1 (2010-03)

[i.54] Post & Telestyrelsen (2005): "Mobile video communications for people who are deaf: Report on

trial operations with broadband for people with disability", Swedish National Post and Telecom

Agency Report No. PTS-ER-2005:14.
[i.55] Raake, A. (2006). Short- and Long-Term Packet Loss Beha
...

Final draft ETSI EG 202 670 V1.1.1 (2009-12)
ETSI Guide
Human Factors (HF);
User Experience Guidelines
for real-time communication services
expressed in Quality of Service terms
---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 202 670 V1.1.1 (2009-12)
Reference
DEG/HF-00112
Keywords
interaction, quality, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org

The present document may be made available in more than one electronic version or in print. In any case of existing or

perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).

In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive

within ETSI Secretariat.

Users of the present document should be aware that the document may be subject to revision or change of status.

Information on the current status of this and other ETSI documents is available at

http://portal.etsi.org/tb/status/status.asp

If you find errors in the present document, please send your comment to one of the following services:

http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM

DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered

for the benefit of its Members.

3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.

LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.

GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.

ETSI
---------------------- Page: 2 ----------------------
3 Final draft ETSI EG 202 670 V1.1.1 (2009-12)
Contents

Intellectual Property Rights ................................................................................................................................ 5

Foreword ............................................................................................................................................................. 5

Introduction ........................................................................................................................................................ 5

1 Scope ........................................................................................................................................................ 7

2 References ................................................................................................................................................ 7

2.1 Normative references ......................................................................................................................................... 7

2.2 Informative references ........................................................................................................................................ 8

3 Definitions and abbreviations ................................................................................................................. 11

3.1 Definitions ........................................................................................................................................................ 11

3.2 Abbreviations ................................................................................................................................................... 15

4 Overview of the real-time communication services addressed in the present document ....................... 16

4.1 Text communication ......................................................................................................................................... 20

4.2 Speech communication ..................................................................................................................................... 20

4.3 Video communication (including speech) ........................................................................................................ 20

4.3.1 Face-to-face video communication ............................................................................................................. 20

4.3.2 Remote inspection or 'see what I see' (SWIS) video communication ......................................................... 20

4.4 Multimedia communication.............................................................................................................................. 21

4.5 Real-time games ............................................................................................................................................... 21

4.6 TV including PC TV and Mobile TV ............................................................................................................... 21

5 Overview of user experience guidelines expressed in Quality of Service terms .................................... 21

6 Advice on using the Guide ..................................................................................................................... 24

6.1 Requirements for guidelines ............................................................................................................................. 25

6.2 The structure of the guidelines ......................................................................................................................... 25

6.3 Maturity and restrictions of the guidelines ....................................................................................................... 26

7 The guidelines ........................................................................................................................................ 26

7.1 Text communication ......................................................................................................................................... 27

7.2 Speech communication ..................................................................................................................................... 30

7.3 Video communication (including speech) ........................................................................................................ 41

7.3.1 Video communication: Face-to-face ........................................................................................................... 41

7.3.2 Video communication: Remote inspection ................................................................................................. 57

7.3.3 Video communication: Multi-point and heterogeneous networks .............................................................. 63

7.4 Multimedia communication.............................................................................................................................. 64

7.5 Real-time games ............................................................................................................................................... 72

7.6 TV .................................................................................................................................................................... 74

7.6.1 PC TV ......................................................................................................................................................... 74

7.6.2 Mobile TV .................................................................................................................................................. 76

7.7 Service selection ............................................................................................................................................... 85

Annex A: Requirements for guidelines and a web-based guideline and tutorial system ................. 92

A.1 Requirement derivation process ............................................................................................ ................. 92

A.2 Requirements for guidelines ................................................................................................................... 92

A.2.1 Provide information on key topics of concern that will aid development choices ........................................... 92

A.2.2 Provide information on related concepts .......................................................................................................... 93

A.2.3 Provide user experience data that can be used from different perspectives ...................................................... 93

A.2.4 Link QoS and user experience variables .......................................................................................................... 93

A.2.5 Provide information about user behaviour that is feasible to apply .................................................................. 94

A.3 Requirements for a web-based system ................................................................................................... 94

Annex B: Sources for the guidelines ..................................................................................................... 95

ETSI
---------------------- Page: 3 ----------------------
4 Final draft ETSI EG 202 670 V1.1.1 (2009-12)

B.1 Bespoke user studies .............................................................................................................................. 95

B.1.1 Laboratory experiments .................................................................................................................................... 95

B.1.2 Field studies...................................................................................................................................................... 96

B.1.3 Survey .............................................................................................................................................................. 96

B.1.4 Expert review and expert panels ....................................................................................................................... 96

B.2 Existing literature ................................................................................................................................... 97

History .............................................................................................................................................................. 98

ETSI
---------------------- Page: 4 ----------------------
5 Final draft ETSI EG 202 670 V1.1.1 (2009-12)
Intellectual Property Rights

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information

pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found

in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in

respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web

server (http://webapp.etsi.org/IPR/home.asp).

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee

can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web

server) which are, or may be, or may become, essential to the present document.
Foreword

This ETSI Guide (EG) has been produced by ETSI Technical Committee Human Factors (HF), and is now submitted

for the ETSI standards Membership Approval Procedure.
The present document is primarily for the HF community.

The present document updates and replaces EG 202 534 [i.11]. New guidelines have been added and some guidelines

from EG 202 534 [i.11] were not considered relevant for the present document (e.g. because of being out of date with

technology developments). In addition, whereas EG 202 534 [i.11] focused on person-to-person (two-way)

communication services, the present document extends to real-time person-to-machine (one-way) communication

services.
Introduction

The real-time communication services that are currently available and in development place different demands on the

communication channel and terminal equipment. Also they continue to evolve for both mobile and fixed usage. The

services offer great potential but also complex choices regarding the most appropriate technologies and media that are

suitable for different communication situations. Taking video communication as an example, there are advances at the

two extremes of usage: for video calls with mobile telephones and conference-room 'Telepresence' systems and between

these extremes video communication is increasingly available through personal computer-based 'Web-conferencing'

services.

TR 102 353 [i.64] identified the need to develop guidelines on real-time communication services. The main aim is to

provide guidelines for network operators, equipment manufacturers and service providers that address the user

experience of different communication services in terms of either:

• the configuration and quality of service (QoS) of a particular communication service;

• the best choice between different communication services, where user experience data exists to enable

recommendations on the basis of of set-up time, reliability, type of user tasks and user situations.

The general requirements for the guidelines contained in the present document are / main inclusion criteria for the

guidelines are:

• Empirical basis - the guidelines are based on user test results where data has been collected on user

performance and/or user opinion. The guidelines are therefore justified on the basis of either a laboratory

experiment, field study, survey or an expert panel of people with empirical knowledge gained through

experience with users or user tests.

• Known QoS - the test systems were of specified QoS parameter values, thereby enabling a statement of

expected user experience for a given service with particular QoS.

• Technology independent - because the user experience results were related to test systems with specified QoS

rather than a particular technology.
ETSI
---------------------- Page: 5 ----------------------
6 Final draft ETSI EG 202 670 V1.1.1 (2009-12)

• Vendor independent - because the user experience results were related to test systems with specified QoS

rather than a particular vendor.

• User communication phase - the guidelines concern user experience once communication is established,

therefore the guidelines do not address interface design and call control;

• Stakeholder interest - priority has been given to developing guidelines in areas where stakeholder interest has

been shown or is anticipated. Therefore Guidelines are provided for topics that have been identified as

important for intended guideline users and for which user-based data existed or could be collected. For this

reason, the development of guidelines from published literature is not exhaustive.

The guidelines are grouped into topics that include different user tasks (e.g. decision making, negotiation, persuasion),

technical parameters (e.g. delay, packet loss, frame-rate) and special user groups (e.g. people with speech impairments,

deaf and hearing impaired people, people with cognitive impairments). The user test results are derived primarily from

industry and European Commission Framework Programme research reports, journal articles, conference proceedings

and standards documents.
ETSI
---------------------- Page: 6 ----------------------
7 Final draft ETSI EG 202 670 V1.1.1 (2009-12)
1 Scope

The present document provides guidelines for the user experience of real-time communication services. The services

include person-to-person (two-way) communication and person-to-machine (one-way) communication.

The present document is revised from EG 202 534 [i.11] that was restricted to person-to-person communication

services. The revision adds new guidelines that have been requested by stakeholders and omits some of the previous

guidelines that are no longer considered relevant.

The guidelines are based on empirical data about user experience. Most of the data is obtained from scientific papers. A

minority of guidelines are based on existing standardisation documents to ensure that relevant normative and

informative standards material is clear amongst results available from journal articles, conference papers, research

reports, etc.
The present document does not replace any existing standards.

Most of the guidelines derived from scientific papers are specific to a particular context, in that the original user tests

were for specific tasks, users and technical parameters and therefore the results may not be generalisable. Although the

guidelines provide information about the main user experience measure(s) and technical parameter(s) of a particular test

result, it is beyond the scope of the present document to provide all of the variables concerned with each user test.

However, the origin of each empirical source of a guideline is shown and listed in the References.

Those readers interested in the details of a particular guideline empirical source are also directed to a web-based system

(http://portal.etsi.org/STFs/STF_HomePages/STF354/STF354.asp) that provides this information.

2 References

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific.
• For a specific reference, subsequent revisions do not apply.

• Non-specific reference may be made only to a complete document or a part thereof and only in the following

cases:

- if it is accepted that it will be possible to use all future changes of the referenced document for the

purposes of the referring document;
- for informative references.

Referenced documents which are not found to be publicly available in the expected location might be found at

http://docbox.etsi.org/Reference.

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee

their long term validity.
2.1 Normative references

The following referenced documents are indispensable for the application of the present document. For dated

references, only the edition cited applies. For non-specific references, the latest edition of the referenced document

(including any amendments) applies.
Not applicable.
ETSI
---------------------- Page: 7 ----------------------
8 Final draft ETSI EG 202 670 V1.1.1 (2009-12)
2.2 Informative references

The following referenced documents are not essential to the use of the present document but they assist the user with

regard to a particular subject area. For non-specific references, the latest version of the referenced document (including

any amendments) applies.

[i.1] Anderson, A., Smallwood, S., MacDonald, R., Mullin, J., Fleming, A. & O'Malley, C. (2000):

"Video data and video links in mediated communication: What do users value? International

Journal of Human-Computer Studies", 52(1), 165-187.

[i.2] ANSI T1.522-2000: "Quality of Service for Business Multimedia Conferencing".

[i.3] Bekkering, E. and Shim, J. (2006). "i2i Trust in Videoconferencing", July 2006/Vol. 49, No. 7

Communications of the ACM, pp. 103-107.

[i.4] Blythe, M.; Overbeeke, K., Monk, A., Wright, P. (Eds) (2004). "Funology: From Usability to

Enjoyment" Kluwer Academic.

[i.5] Brennan, D., Georgeadis, A., Baron, C., Barker, L. (2004): "The Effect of Videoconference-Based

Telerehabilitation on Story Retelling Performance by Brain-Injured Subjects and Its Implications

for Remote Speech-Language Therapy", Telemedicine Journal and e-Health. June 1, 2004,

10(2): 147-154.

[i.6] Brooks, P., Schliemann, T., Hestnes, B., Frowein, H., Aaby, C., O'Malley, C. (2003): ""Final

Report Project IST-1999-11577 Eye-2-Eye: Fitness-for-Purpose of Person-Person Communication

Technologies". EC Deliverable IST11577/SEF/DIS/DS/Pub/008/b1, June 2003.

[i.7] Brooks, P. & Hestnes, B. (2003): "User-centred technical guidelines for real-time human

communication services: Requirements and derivation". Proceedings of the 19th International

Symposium on Human Factors in Telecommunication, Berlin, Germany, December 1-4 2003, pp.

11-18.

[i.8] Cullum, C., Weiner, M. Gehrmann, H. Hynan, L. (2006): "Feasibility of Telecognitive Assessment

in Dementia. Assessment", Volume 13, No. 4, December 2006 385-390.

[i.9] Cuny, R., Li, M., Kristensson, M. (2006): "Mobile service applications and performance in

UMTS", In Soldani, D., Li, M., Cuny, R. (Eds) "QoS and QoE Management in UMTS Cellular

Systems", John Wiley & Sons, Ltd.

[i.10] Doherty-Sneddon, G., Anderson, A., O' Malley, C., Langton, S., Garrod, S. & Bruce, V. (1997):

"Face-to-face and video mediated communication: A comparison of dialogue structure and task

performance", Journal of Experimental Psychology: Applied , 3(2), 105-125.

[i.11] ETSI EG 202 534: "Human Factors (HF); Guidelines for real-time person-to-person

communication services".

[i.12] ETSI ES 202 667 : "Speech and multimedia Transmission Quality (STQ); Audiovisual QoS for

communication over IP networks".

[i.13] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.14] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective

as perceived by the user".

[i.15] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission

requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.16] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as

perceived by the user".
ETSI
---------------------- Page: 8 ----------------------
9 Final draft ETSI EG 202 670 V1.1.1 (2009-12)

[i.17] ETSI ETR 292: "Terrestrial Trunked Radio (TETRA); Voice plus Data (V+D); Technical

requirements specification; Network management".

[i.18] ETSI TR 102 535: "Human Factors (HF); Guidelines for real-time person-to-person

communication services; Future requirements".

[i.19] ETSI ETR 334: "Human Factors (HF); The implications of human ageing for the design of

telephone terminals".
[i.20] ETSI ETR 297: "Human Factors (HF); Human Factors in Videotelephony".

[i.21] ETSI TS 122 105: "Universal Mobile Telecommunications System (UMTS); Services and service

capabilities (3GPP TS 22.105)".

[i.22] ETSI TS 123 107: "Digital cellular telecommunications system (Phase 2+); Universal Mobile

Telecommunications System (UMTS); LTE; Quality of Service (QoS) concept and architecture

(3GPP TS 23.107 Release 8)".

[i.23] Følstad, A., Brooks, P., Heim, J., Schliemann, T., Wiig., S., Hestnes, B., Heiestad, S., Ulseth, T.,

Frowein, H., Aaby, C., O'Malley, C., Brundell, P., Lonsdale, P. (2002): "Results of Field

Experiments of Communication Media. IST Project 1999-11577. Eye-2-Eye: Fitness-for-purpose

of Person-Person Communication Technologies", CEC Deliverable
IST11577/SEF/DIS/DS/Pub/004/b1, October 2002.

[i.24] France, E. F., Anderson, A. H. & Gardner, M. (2001): "The impact of status and audio

conferencing technology on business meetings", International Journal of Human Computer

Studies, 54: 857-876.

[i.25] Frowein, H., Donker, W., Devoldere, P., Schliemann, T., Brooks, P., Følstad, A., Hestnes, B. et al.

(2003): "Specification of Cost-Benefit Analysis Tool. Project IST-1999-11577 Eye-2-Eye:

Fitness-for-Purpose of Person-Person Communication Technologies", EC Deliverable
IST11577/IvD/RAD/DS/FP5/052/b1, May 2003.

[i.26] Ghinea, G., Thomas, J. (2005). "Quality of Perception: User Quality of Service in Multimedia

Presentations", IEEE Transactions on Multimedia, Vol. 7, No. 4, August 2005, pp. 786-789.

[i.27] Grayson, D. & Coventry, L. (1998) "The Effects of Visual Proxemic Information in Video

Mediated Communication", SIGCHI Bulletin Vol.30 No.3, July 1998.

[i.28] Hellström, G., Aström, L. (2008): "The value of total conversation call with emergency services,

Proceedings of International Symposuim on Human Factors in Telecommunications", Kuala

Lumpa, Malaysia, 17-20 March 2008.

[i.29] Hestnes, B., Brooks, P., Heiestad, S. (2004): "Mobile Eye-phone - a study of relevance,

effectiveness and user-perceived suitability", Fornebu, Telenor R&D Report (Scientific Report

R2/2004).

[i.30] Hestnes, B., Brooks, P., Heiestad, S., Ulseth, T., Aaby., C. (2003): "Quality of Experience in

real-time person-person communication services - User based QoS expressed in technical network

QoS terms", proceedings of the 19th International Symposium on Human Factors in
Telecommunication, Berlin, Germany, December 1-4 2003, pp. 3-10.

[i.31] ISO 9241 (all parts): "Ergonomic requirements for office work with visual display terminals

(VDTs)".

[i.32] ITU-R Recommendation BT.1359-1 (2003): "Method for the Subjective Assessment of

Intermediate Quality Level of Coding Systems".

[i.33] ITU-T Series H: "Audiovisual and multimedia systems, Supplement 1 (05/99) (1999) Application

profile - Sign language and lip-reading real-time conversation using low bit-rate video

communication".

[i.34] ITU-T Recommendation E.800 (2008): "Definitions of terms related to quality of service".

[i.35] ITU-T Recommendation F.700 (2000): Framework Recommendation for multimedia services.

ETSI
---------------------- Page: 9 ----------------------
10 Final draft ETSI EG 202 670 V1.1.1 (2009-12)

[i.36] ITU-T Recommendation G.108 (1999): "Application of the E-model: A planning guide".

[i.37] ITU-T Recommendation G.114 (2003): "One-way transmission time", International.

[i.38] ITU-T Recommendation G.1010 (2001): "End-user multimedia QoS categories".

[i.39] ITU-T Recommendation G.1080: "Quality of experience requirements for IPTV services".

[i.40] ITU-T Recommendation P.10/G.100: "Amendment 2: New definitions for inclusion in

Recommendation ITU-T P.10/G.100", International Telecommunication Union, Geneva,
Switzerland, 2008.

[i.41] ITU-T Recommendation Y.1541 (2006): "Network performance objectives for IP-based services",

International Telecommunication Union, Geneva, Switzerland.

[i.42] Jumisko-Pyykkö, S., Vinod Kumar, M. V., Liinasuo, M., & Hannuksela, M. (2006): Acceptance of

audiovisual quality in erroneous television sequences over a DVB-H channel. In Proceedings of

the Second International Workshop in Video Processing and Quality Metrics for Consumer

Electronics".

[i.43] Kilgore, R.M., and Chignell, M., (2006): "Listening to unfamiliar voices in spatial audio: Does

visualization of spatial position enhance voice identification? Proceedings of the 20th International

Symposium on Human Factors in Telecommunication", Sophia Antipolis, France, March 21-23

2006.
NOTE: Available from http://www.hft.org/HFT06/HFT_06_programme.htm).

[i.44] Knoche, H., McCarthy, J. D. and Sasse, M.A. (2005): "Can small be beautiful?: assessing image

resolution requirements for mobile TV", Proc. ACM Multimedia 2005, ACM Press, 829-838.

[i.45] Knoche, H., McCarthy J., Sasse, M. A. (2006): "Reading the Fine Print: The Effect of Text

Legibility on Perceived Video Quality in Mobile TV", ACM Multimedia, ACM, 727-730.

[i.46] Knoche, H.O., Sasse, M.A. (2008): "The sweet spot: how people trade off size and definition on

mobile devices. In Proceeding of the 16th ACM international conference on Multimedia",

Vancouver, British Columbia, Canada: ACM, pp. 21-30. .

[i.47] Mashima, P., Birkmire-Peters, D., Syms, M., Holtel, M., Burgess, L., Peters, L. (2003):

"Telehealth: Voice Therapy Using Telecommunications Technology", American Journal of

Speech-Language Pathology Vol.12 432-439.

[i.48] Mason, S. (2006): "Mobile TV - results from the DVB-H trial in Oxford", EBU TECHNICAL

REVIEW - April 2006.
NOTE: http://www.ebu.ch/en/technical/trev/trev_306-mason.pdf

[i.49] McCarthy, J., Sasse, M. A. & Miras, D. (2004): "Sharp or Smooth? Comparing the effects of

quantization vs. frame rate for streamed video. Proceedings of CHI 2004", Vienna, Austria, April

20-24, pp. 535-542.

[i.50] Mulbach, L., Bocker, M. & Prussog, A. (1995): "Telepresence in Videocommunications: A Study

on Stereoscopy and Individual Eye Contact", Human Factors, 37(2), 290-305.

[i.51] Nokia (2004): "Quality of Experience (QoE) of mobile services: Can it be measured and

improved?", White Paper, Nokia Corporation, Finland, 2004.

[i.52] O' Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G & Bruce, V. (1996): "Comparison

of Face-to-Face and Video-Mediated Interaction. Interacting with Computers", 8(2), 177-192.

[i.53] O'Malley, C., Brundell, P., McFadzean, J., Lonsdale, P., Schliemann, T., Brooks, P. et al. (2002):

"Results of Laboratory Experiments of Communication Media. IST Project 1999-11577.

Eye-2-Eye: Fitness-for-purpose of Person-Person Communication Technologies", CEC
Deliverable IST11577/UON/SOP/DS/Pub/003/b1, December 2002.
ETSI
---------------------- Page: 10 ----------------------
11 Final draft ETSI EG 202 670 V1.1.1 (2009-12)

[i.54] Post & Telestyrelsen (2005): "Mobile video communications for people who are deaf: Repor

...

SLOVENSKI STANDARD
SIST EG 202 670 V1.1.1:2010
01-december-2010
ýORYHãNLGHMDYQLNL +) 6PHUQLFHL]XSRUDEQLãNLKL]NXãHQM]DUHDOQRþDVRYQH
NRPXQLNDFLMVNHVWRULWYHL]UDåHQH]L]UD]L]DNDNRYRVWVWRULWYH 4R6

Human Factors (HF) - User Experience Guidelines for real-time communication services

expressed in Quality of Service terms
Ta slovenski standard je istoveten z: EG 202 670 Version 1.1.1
ICS:
03.120.99 Drugi standardi v zvezi s Other standards related to
kakovostjo quality
35.020 Informacijska tehnika in Information technology (IT) in
tehnologija na splošno general
SIST EG 202 670 V1.1.1:2010 en

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST EG 202 670 V1.1.1:2010
---------------------- Page: 2 ----------------------
SIST EG 202 670 V1.1.1:2010
ETSI EG 202 670 V1.1.1 (2010-03)
ETSI Guide
Human Factors (HF);
User Experience Guidelines
for real-time communication services
expressed in Quality of Service terms
---------------------- Page: 3 ----------------------
SIST EG 202 670 V1.1.1:2010
2 ETSI EG 202 670 V1.1.1 (2010-03)
Reference
DEG/HF-00112
Keywords
interaction, quality, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org

The present document may be made available in more than one electronic version or in print. In any case of existing or

perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).

In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive

within ETSI Secretariat.

Users of the present document should be aware that the document may be subject to revision or change of status.

Information on the current status of this and other ETSI documents is available at

http://portal.etsi.org/tb/status/status.asp

If you find errors in the present document, please send your comment to one of the following services:

http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2010.
All rights reserved.
TM TM TM TM

DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered

for the benefit of its Members.

3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.

LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.

GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.

ETSI
---------------------- Page: 4 ----------------------
SIST EG 202 670 V1.1.1:2010
3 ETSI EG 202 670 V1.1.1 (2010-03)
Contents

Intellectual Property Rights ................................................................................................................................ 5

Foreword ............................................................................................................................................................. 5

Introduction ........................................................................................................................................................ 5

1 Scope ........................................................................................................................................................ 7

2 References ................................................................................................................................................ 7

2.1 Normative references ......................................................................................................................................... 7

2.2 Informative references ........................................................................................................................................ 8

3 Definitions and abbreviations ................................................................................................................. 11

3.1 Definitions ........................................................................................................................................................ 11

3.2 Abbreviations ................................................................................................................................................... 15

4 Overview of the real-time communication services addressed in the present document ....................... 16

4.1 Text communication ......................................................................................................................................... 20

4.2 Speech communication ..................................................................................................................................... 20

4.3 Video communication (including speech) ........................................................................................................ 20

4.3.1 Face-to-face video communication ............................................................................................................. 20

4.3.2 Remote inspection or 'see what I see' (SWIS) video communication ......................................................... 20

4.4 Multimedia communication.............................................................................................................................. 21

4.5 Real-time games ............................................................................................................................................... 21

4.6 TV including PC TV and Mobile TV ............................................................................................................... 21

5 Overview of user experience guidelines expressed in Quality of Service terms .................................... 21

6 Advice on using the Guide ..................................................................................................................... 24

6.1 Requirements for guidelines ............................................................................................................................. 25

6.2 The structure of the guidelines ......................................................................................................................... 25

6.3 Maturity and restrictions of the guidelines ....................................................................................................... 26

7 The guidelines ........................................................................................................................................ 26

7.1 Text communication ......................................................................................................................................... 27

7.2 Speech communication ..................................................................................................................................... 30

7.3 Video communication (including speech) ........................................................................................................ 41

7.3.1 Video communication: Face-to-face ........................................................................................................... 41

7.3.2 Video communication: Remote inspection ................................................................................................. 57

7.3.3 Video communication: Multi-point and heterogeneous networks .............................................................. 63

7.4 Multimedia communication.............................................................................................................................. 64

7.5 Real-time games ............................................................................................................................................... 72

7.6 TV .................................................................................................................................................................... 74

7.6.1 PC TV ......................................................................................................................................................... 74

7.6.2 Mobile TV .................................................................................................................................................. 76

7.7 Service selection ............................................................................................................................................... 85

Annex A: Requirements for guidelines and a web-based guideline and tutorial system ................. 92

A.1 Requirement derivation process ............................................................................................ ................. 92

A.2 Requirements for guidelines ................................................................................................................... 92

A.2.1 Provide information on key topics of concern that will aid development choices ........................................... 92

A.2.2 Provide information on related concepts .......................................................................................................... 93

A.2.3 Provide user experience data that can be used from different perspectives ...................................................... 93

A.2.4 Link QoS and user experience variables .......................................................................................................... 93

A.2.5 Provide information about user behaviour that is feasible to apply .................................................................. 94

A.3 Requirements for a web-based system ................................................................................................... 94

Annex B: Sources for the guidelines ..................................................................................................... 95

ETSI
---------------------- Page: 5 ----------------------
SIST EG 202 670 V1.1.1:2010
4 ETSI EG 202 670 V1.1.1 (2010-03)

B.1 Bespoke user studies .............................................................................................................................. 95

B.1.1 Laboratory experiments .................................................................................................................................... 95

B.1.2 Field studies...................................................................................................................................................... 96

B.1.3 Survey .............................................................................................................................................................. 96

B.1.4 Expert review and expert panels ....................................................................................................................... 96

B.2 Existing literature ................................................................................................................................... 97

History .............................................................................................................................................................. 98

ETSI
---------------------- Page: 6 ----------------------
SIST EG 202 670 V1.1.1:2010
5 ETSI EG 202 670 V1.1.1 (2010-03)
Intellectual Property Rights

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information

pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found

in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in

respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web

server (http://webapp.etsi.org/IPR/home.asp).

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee

can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web

server) which are, or may be, or may become, essential to the present document.
Foreword

This ETSI Guide (EG) has been produced by ETSI Technical Committee Human Factors (HF).

The present document is primarily for the HF community.

The present document updates and replaces EG 202 534 [i.11]. New guidelines have been added and some guidelines

from EG 202 534 [i.11] were not considered relevant for the present document (e.g. because of being out of date with

technology developments). In addition, whereas EG 202 534 [i.11] focused on person-to-person (two-way)

communication services, the present document extends to real-time person-to-machine (one-way) communication

services.
Introduction

The real-time communication services that are currently available and in development place different demands on the

communication channel and terminal equipment. Also they continue to evolve for both mobile and fixed usage. The

services offer great potential but also complex choices regarding the most appropriate technologies and media that are

suitable for different communication situations. Taking video communication as an example, there are advances at the

two extremes of usage: for video calls with mobile telephones and conference-room 'Telepresence' systems and between

these extremes video communication is increasingly available through personal computer-based 'Web-conferencing'

services.

TR 102 353 [i.64] identified the need to develop guidelines on real-time communication services. The main aim is to

provide guidelines for network operators, equipment manufacturers and service providers that address the user

experience of different communication services in terms of either:

• the configuration and quality of service (QoS) of a particular communication service;

• the best choice between different communication services, where user experience data exists to enable

recommendations on the basis of of set-up time, reliability, type of user tasks and user situations.

The general requirements for the guidelines contained in the present document are / main inclusion criteria for the

guidelines are:

• Empirical basis - the guidelines are based on user test results where data has been collected on user

performance and/or user opinion. The guidelines are therefore justified on the basis of either a laboratory

experiment, field study, survey or an expert panel of people with empirical knowledge gained through

experience with users or user tests.

• Known QoS - the test systems were of specified QoS parameter values, thereby enabling a statement of

expected user experience for a given service with particular QoS.

• Technology independent - because the user experience results were related to test systems with specified QoS

rather than a particular technology.
ETSI
---------------------- Page: 7 ----------------------
SIST EG 202 670 V1.1.1:2010
6 ETSI EG 202 670 V1.1.1 (2010-03)

• Vendor independent - because the user experience results were related to test systems with specified QoS

rather than a particular vendor.

• User communication phase - the guidelines concern user experience once communication is established,

therefore the guidelines do not address interface design and call control;

• Stakeholder interest - priority has been given to developing guidelines in areas where stakeholder interest has

been shown or is anticipated. Therefore Guidelines are provided for topics that have been identified as

important for intended guideline users and for which user-based data existed or could be collected. For this

reason, the development of guidelines from published literature is not exhaustive.

The guidelines are grouped into topics that include different user tasks (e.g. decision making, negotiation, persuasion),

technical parameters (e.g. delay, packet loss, frame-rate) and special user groups (e.g. people with speech impairments,

deaf and hearing impaired people, people with cognitive impairments). The user test results are derived primarily from

industry and European Commission Framework Programme research reports, journal articles, conference proceedings

and standards documents.
ETSI
---------------------- Page: 8 ----------------------
SIST EG 202 670 V1.1.1:2010
7 ETSI EG 202 670 V1.1.1 (2010-03)
1 Scope

The present document provides guidelines for the user experience of real-time communication services. The services

include person-to-person (two-way) communication and person-to-machine (one-way) communication.

The present document is revised from EG 202 534 [i.11] that was restricted to person-to-person communication

services. The revision adds new guidelines that have been requested by stakeholders and omits some of the previous

guidelines that are no longer considered relevant.

The guidelines are based on empirical data about user experience. Most of the data is obtained from scientific papers. A

minority of guidelines are based on existing standardisation documents to ensure that relevant normative and

informative standards material is clear amongst results available from journal articles, conference papers, research

reports, etc.
The present document does not replace any existing standards.

Most of the guidelines derived from scientific papers are specific to a particular context, in that the original user tests

were for specific tasks, users and technical parameters and therefore the results may not be generalisable. Although the

guidelines provide information about the main user experience measure(s) and technical parameter(s) of a particular test

result, it is beyond the scope of the present document to provide all of the variables concerned with each user test.

However, the origin of each empirical source of a guideline is shown and listed in the References.

Those readers interested in the details of a particular guideline empirical source are also directed to a web-based system

(http://portal.etsi.org/STFs/STF_HomePages/STF354/STF354.asp) that provides this information.

2 References

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific.
• For a specific reference, subsequent revisions do not apply.

• Non-specific reference may be made only to a complete document or a part thereof and only in the following

cases:

- if it is accepted that it will be possible to use all future changes of the referenced document for the

purposes of the referring document;
- for informative references.

Referenced documents which are not found to be publicly available in the expected location might be found at

http://docbox.etsi.org/Reference.

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee

their long term validity.
2.1 Normative references

The following referenced documents are indispensable for the application of the present document. For dated

references, only the edition cited applies. For non-specific references, the latest edition of the referenced document

(including any amendments) applies.
Not applicable.
ETSI
---------------------- Page: 9 ----------------------
SIST EG 202 670 V1.1.1:2010
8 ETSI EG 202 670 V1.1.1 (2010-03)
2.2 Informative references

The following referenced documents are not essential to the use of the present document but they assist the user with

regard to a particular subject area. For non-specific references, the latest version of the referenced document (including

any amendments) applies.

[i.1] Anderson, A., Smallwood, S., MacDonald, R., Mullin, J., Fleming, A. & O'Malley, C. (2000):

"Video data and video links in mediated communication: What do users value? International

Journal of Human-Computer Studies", 52(1), 165-187.

[i.2] ANSI T1.522-2000: "Quality of Service for Business Multimedia Conferencing".

[i.3] Bekkering, E. and Shim, J. (2006). "i2i Trust in Videoconferencing", July 2006/Vol. 49, No. 7

Communications of the ACM, pp. 103-107.

[i.4] Blythe, M.; Overbeeke, K., Monk, A., Wright, P. (Eds) (2004). "Funology: From Usability to

Enjoyment" Kluwer Academic.

[i.5] Brennan, D., Georgeadis, A., Baron, C., Barker, L. (2004): "The Effect of Videoconference-Based

Telerehabilitation on Story Retelling Performance by Brain-Injured Subjects and Its Implications

for Remote Speech-Language Therapy", Telemedicine Journal and e-Health. June 1, 2004,

10(2): 147-154.

[i.6] Brooks, P., Schliemann, T., Hestnes, B., Frowein, H., Aaby, C., O'Malley, C. (2003): ""Final

Report Project IST-1999-11577 Eye-2-Eye: Fitness-for-Purpose of Person-Person Communication

Technologies". EC Deliverable IST11577/SEF/DIS/DS/Pub/008/b1, June 2003.

[i.7] Brooks, P. & Hestnes, B. (2003): "User-centred technical guidelines for real-time human

communication services: Requirements and derivation". Proceedings of the 19th International

Symposium on Human Factors in Telecommunication, Berlin, Germany, December 1-4 2003,

pp. 11-18.

[i.8] Cullum, C., Weiner, M. Gehrmann, H. Hynan, L. (2006): "Feasibility of Telecognitive Assessment

in Dementia. Assessment", Volume 13, No. 4, December 2006 385-390.

[i.9] Cuny, R., Li, M., Kristensson, M. (2006): "Mobile service applications and performance in

UMTS", In Soldani, D., Li, M., Cuny, R. (Eds) "QoS and QoE Management in UMTS Cellular

Systems", John Wiley & Sons, Ltd.

[i.10] Doherty-Sneddon, G., Anderson, A., O' Malley, C., Langton, S., Garrod, S. & Bruce, V. (1997):

"Face-to-face and video mediated communication: A comparison of dialogue structure and task

performance", Journal of Experimental Psychology: Applied , 3(2), 105-125.

[i.11] ETSI EG 202 534: "Human Factors (HF); Guidelines for real-time person-to-person

communication services".

[i.12] ETSI ES 202 667 : "Speech and multimedia Transmission Quality (STQ); Audiovisual QoS for

communication over IP networks".

[i.13] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.14] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective

as perceived by the user".

[i.15] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission

requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as

perceived by the user".

[i.16] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission

requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as

perceived by the user".
ETSI
---------------------- Page: 10 ----------------------
SIST EG 202 670 V1.1.1:2010
9 ETSI EG 202 670 V1.1.1 (2010-03)

[i.17] ETSI ETR 292: "Terrestrial Trunked Radio (TETRA); Voice plus Data (V+D); Technical

requirements specification; Network management".

[i.18] ETSI TR 102 535: "Human Factors (HF); Guidelines for real-time person-to-person

communication services; Future requirements".

[i.19] ETSI ETR 334: "Human Factors (HF); The implications of human ageing for the design of

telephone terminals".
[i.20] ETSI ETR 297: "Human Factors (HF); Human Factors in Videotelephony".

[i.21] ETSI TS 122 105: "Universal Mobile Telecommunications System (UMTS); Services and service

capabilities (3GPP TS 22.105)".

[i.22] ETSI TS 123 107: "Digital cellular telecommunications system (Phase 2+); Universal Mobile

Telecommunications System (UMTS); LTE; Quality of Service (QoS) concept and architecture

(3GPP TS 23.107 Release 8)".

[i.23] Følstad, A., Brooks, P., Heim, J., Schliemann, T., Wiig., S., Hestnes, B., Heiestad, S., Ulseth, T.,

Frowein, H., Aaby, C., O'Malley, C., Brundell, P., Lonsdale, P. (2002): "Results of Field

Experiments of Communication Media. IST Project 1999-11577. Eye-2-Eye: Fitness-for-purpose

of Person-Person Communication Technologies", CEC Deliverable
IST11577/SEF/DIS/DS/Pub/004/b1, October 2002.

[i.24] France, E. F., Anderson, A. H. & Gardner, M. (2001): "The impact of status and audio

conferencing technology on business meetings", International Journal of Human Computer

Studies, 54: 857-876.

[i.25] Frowein, H., Donker, W., Devoldere, P., Schliemann, T., Brooks, P., Følstad, A., Hestnes, B. et al.

(2003): "Specification of Cost-Benefit Analysis Tool. Project IST-1999-11577 Eye-2-Eye:

Fitness-for-Purpose of Person-Person Communication Technologies", EC Deliverable
IST11577/IvD/RAD/DS/FP5/052/b1, May 2003.

[i.26] Ghinea, G., Thomas, J. (2005). "Quality of Perception: User Quality of Service in Multimedia

Presentations", IEEE Transactions on Multimedia, Vol. 7, No. 4, August 2005, pp. 786-789.

[i.27] Grayson, D. & Coventry, L. (1998) "The Effects of Visual Proxemic Information in Video

Mediated Communication", SIGCHI Bulletin Vol.30 No.3, July 1998.

[i.28] Hellström, G., Aström, L. (2008): "The value of total conversation call with emergency services,

Proceedings of International Symposuim on Human Factors in Telecommunications", Kuala

Lumpa, Malaysia, 17-20 March 2008.

[i.29] Hestnes, B., Brooks, P., Heiestad, S. (2004): "Mobile Eye-phone - a study of relevance,

effectiveness and user-perceived suitability", Fornebu, Telenor R&D Report (Scientific Report

R2/2004).

[i.30] Hestnes, B., Brooks, P., Heiestad, S., Ulseth, T., Aaby., C. (2003): "Quality of Experience in

real-time person-person communication services - User based QoS expressed in technical network

QoS terms", proceedings of the 19th International Symposium on Human Factors in
Telecommunication, Berlin, Germany, December 1-4 2003, pp. 3-10.

[i.31] ISO 9241 (all parts): "Ergonomic requirements for office work with visual display terminals

(VDTs)".

[i.32] ITU-R Recommendation BT.1359-1 (2003): "Method for the Subjective Assessment of

Intermediate Quality Level of Coding Systems".

[i.33] ITU-T Series H: "Audiovisual and multimedia systems, Supplement 1 (05/99) (1999) Application

profile - Sign language and lip-reading real-time conversation using low bit-rate video

communication".

[i.34] ITU-T Recommendation E.800 (2008): "Definitions of terms related to quality of service".

[i.35] ITU-T Recommendation F.700 (2000): Framework Recommendation for multimedia services.

ETSI
---------------------- Page: 11 ----------------------
SIST EG 202 670 V1.1.1:2010
10 ETSI EG 202 670 V1.1.1 (2010-03)

[i.36] ITU-T Recommendation G.108 (1999): "Application of the E-model: A planning guide".

[i.37] ITU-T Recommendation G.114 (2003): "One-way transmission time", International.

[i.38] ITU-T Recommendation G.1010 (2001): "End-user multimedia QoS categories".

[i.39] ITU-T Recommendation G.1080: "Quality of experience requirements for IPTV services".

[i.40] ITU-T Recommendation P.10/G.100: "Amendment 2: New definitions for inclusion in

Recommendation ITU-T P.10/G.100", International Telecommunication Union, Geneva,
Switzerland, 2008.

[i.41] ITU-T Recommendation Y.1541 (2006): "Network performance objectives for IP-based services",

International Telecommunication Union, Geneva, Switzerland.

[i.42] Jumisko-Pyykkö, S., Vinod Kumar, M. V., Liinasuo, M., & Hannuksela, M. (2006): Acceptance of

audiovisual quality in erroneous television sequences over a DVB-H channel. In Proceedings of

the Second International Workshop in Video Processing and Quality Metrics for Consumer

Electronics".

[i.43] Kilgore, R.M., and Chignell, M., (2006): "Listening to unfamiliar voices in spatial audio: Does

visualization of spatial position enhance voice identification? Proceedings of the 20th International

Symposium on Human Factors in Telecommunication", Sophia Antipolis, France, March 21-23

2006.
NOTE: Available from http://www.hft.org/HFT06/HFT_06_programme.htm).

[i.44] Knoche, H., McCarthy, J. D. and Sasse, M.A. (2005): "Can small be beautiful?: assessing image

resolution requirements for mobile TV", Proc. ACM Multimedia 2005, ACM Press, 829-838.

[i.45] Knoche, H., McCarthy J., Sasse, M. A. (2006): "Reading the Fine Print: The Effect of Text

Legibility on Perceived Video Quality in Mobile TV", ACM Multimedia, ACM, 727-730.

[i.46] Knoche, H.O., Sasse, M.A. (2008): "The sweet spot: how people trade off size and definition on

mobile devices. In Proceeding of the 16th ACM international conference on Multimedia",

Vancouver, British Columbia, Canada: ACM, pp. 21-30. .

[i.47] Mashima, P., Birkmire-Peters, D., Syms, M., Holtel, M., Burgess, L., Peters, L. (2003):

"Telehealth: Voice Therapy Using Telecommunications Technology", American Journal of

Speech-Language Pathology Vol.12 432-439.

[i.48] Mason, S. (2006): "Mobile TV - results from the DVB-H trial in Oxford", EBU TECHNICAL

REVIEW - April 2006.
NOTE: Available at http://tech.ebu.ch/docs/techreview/trev_306-mason.pdf.

[i.49] McCarthy, J., Sasse, M. A. & Miras, D. (2004): "Sharp or Smooth? Comparing the effects of

quantization vs. frame rate for streamed video. Proceedings of CHI 2004", Vienna, Austria,

April 20-24, pp. 535-542.
[i.50] Mulbach, L., Bocker, M. & Prussog, A. (1995): "Teleprese
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.