ISO/DIS 5060

Translation services — Evaluation of translation output — General guidance

Services de traduction — Évaluation des résultats de traduction — Recommandations générales

Prevajalske storitve - Vrednotenje rezultatov prevajanja - Splošna navodila

General Information

• Status: Not Published
• ICS: 01.020 - Terminology (principles and coordination)
• Technical Committee: ISO/TC 37/SC 5 - Translation, interpreting and related technology
• Drafting Committee: ISO/TC 37/SC 5/WG 1 - Translation
• Current Stage: 4099 - Full report circulated: DIS approved for registration as FDIS
• Completion Date: 14-Oct-2022
• Ref Project: oSIST ISO/DIS 5060:2022 - Translation services — Evaluation of translation output — General guidance

Standards Content (Sample)

SLOVENSKI STANDARD
oSIST ISO/DIS 5060:2022
01-december-2022
Prevajalske storitve - Vrednotenje rezultatov prevajanja - Splošna navodila
Translation services — Evaluation of translation output — General guidance

Services de traduction — Évaluation des résultats de traduction — Recommandations

générales
Ta slovenski standard je istoveten z: ISO/DIS 5060:2022
ICS:
01.020 Terminologija (načela in Terminology (principles and
koordinacija) coordination)
03.080.99 Druge storitve Other services
oSIST ISO/DIS 5060:2022 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
oSIST ISO/DIS 5060:2022
---------------------- Page: 2 ----------------------
oSIST ISO/DIS 5060:2022
DRAFT INTERNATIONAL STANDARD
ISO/DIS 5060
ISO/TC 37/SC 5 Secretariat: DIN
Voting begins on: Voting terminates on:
2022-07-20 2022-10-12
Translation services — Evaluation of translation output —
General guidance

Services de traduction — Évaluation d'un texte résultant d'une traduction — Recommandations générales

ICS: 01.020
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/DIS 5060:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. © ISO 2022
---------------------- Page: 3 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
DRAFT INTERNATIONAL STANDARD
ISO/DIS 5060
ISO/TC 37/SC 5 Secretariat: DIN
Voting begins on: Voting terminates on:
Translation services — Evaluation of translation output —
General guidance

Services de traduction — Évaluation d'un texte résultant d'une traduction — Recommandations générales

ICS: 01.020
COPYRIGHT PROTECTED DOCUMENT
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
© ISO 2022
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

NOT BE REFERRED TO AS AN INTERNATIONAL

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on STANDARD UNTIL PUBLISHED AS SUCH.

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

IN ADDITION TO THEIR EVALUATION AS

or ISO’s member body in the country of the requester. BEING ACCEPTABLE FOR INDUSTRIAL,

TECHNOLOGICAL, COMMERCIAL AND
ISO copyright office
USER PURPOSES, DRAFT INTERNATIONAL
CP 401 • Ch. de Blandonnet 8
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
CH-1214 Vernier, Geneva
POTENTIAL TO BECOME STANDARDS TO
Phone: [phone]
WHICH REFERENCE MAY BE MADE IN
Reference number
Email: [e-mail]
NATIONAL REGULATIONS.
Website: www.iso.org ISO/DIS 5060:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
Published in Switzerland
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
© ISO 2022 – All rights reserved
PROVIDE SUPPORTING DOCUMENTATION. © ISO 2022
---------------------- Page: 4 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction .................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ..................................................................................................................................................................................... 1

3 Terms and definitions .................................................................................................................................................................................... 1

3.1 General concepts ................................................................................................................................................................................... 1

3.2 Concepts related to the people involved in the evaluation process ........................................................ 3

3.3 Concepts related to the evaluation process .................................................................................................................. 4

3.4 Concepts related to errors ........................................................................................................................................................... 5

3.5 Concepts related to penalty points....................................................................................................................................... 6

4 Human resources ................................................................................................................................................................................................ 6

4.1 General ........................................................................................................................................................................................................... 6

4.2 Professional competences of evaluators ......................................................................................................................... 6

4.3 Qualifications of evaluators ........................................................................................................................................................ 7

5 Pre-evaluation phase ...................................................................................................................................................................................... 7

5.1 Evaluation methods ........................................................................................................................................................................... 7

5.2 Implementing the evaluation of translation output .............................................................................................. 8

5.3 Specifications ........................................................................................................................................................................................... 8

5.4 Error typology ........................................................................................................................................................................................ 8

5.5 Error type weights ........................................................................................................................................................................... 10

5.6 Severity levels of error types .................................................................................................................................................. 10

5.7 Critical errors ....................................................................................................................................................................................... 10

5.8 Counting repeated errors .......................................................................................................................................................... 10

6 Evaluation phase ...............................................................................................................................................................................................10

6.1 General ........................................................................................................................................................................................................ 10

6.2 Basic evaluation principles ....................................................................................................................................................... 10

6.3 Sampling the translation output to be evaluated ................................................................................................. 10

6.4 Translation evaluation scorecard ....................................................................................................................................... 11

6.5 Quality rating ........................................................................................................................................................................................12

7 Post-evaluation phase .................................................................................................................................................................................12

7.1 Feedback ...................................................................................................................................................................................................12

7.2 Dispute resolution ............................................................................................................................................................................12

Annex A (informative) Translation use case, evaluation purpose and constraints.......................................13

Annex B (informative) Developing a translation evaluation strategy ..........................................................................14

Annex C (informative) Developing an evaluation system ..........................................................................................................15

Annex D (informative) Sampling ..........................................................................................................................................................................16

Annex E (informative) Cohesion and coherence ...................................................................................................................................17

Annex F (informative) Examples of scorecards for evaluating translation output ........................................18

Bibliography .............................................................................................................................................................................................................................22

iii
© ISO 2022 – All rights reserved
---------------------- Page: 5 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www.iso.org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to

the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see

www.iso.org/iso/foreword.html.

This document was prepared by Technical Committee ISO/TC 37, Language and terminology,

Subcommittee SC 5, Translation, interpreting and related technology.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www.iso.org/members.html.
© ISO 2022 – All rights reserved
---------------------- Page: 6 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
Introduction

Globalization, the accompanying growth of the translation market, the professionalization of translation

services and the growing availability of online machine translation tools have driven the demand for

standardized and harmonized principles and procedures for the evaluation of translation output.

In an industrial world in which products have to comply with clients’ and legal specifications,

translations have to be regarded as a product that has to fulfil certain requirements. The systematic

comparison of a target language content with the source language content giving regard to the

translation project specifications provides the means to arrive at a valuable and objective judgment of

achievement.

This document provides guidance for the evaluation of human translation output, post-edited machine

translation output, and unedited machine translation output. It also provides guidance on the

qualifications of the evaluators. The role of sampling is also discussed in this document.

The guidance provided in this document can also support the evaluation of source texts intended for

translation.
© ISO 2022 – All rights reserved
---------------------- Page: 7 ----------------------
oSIST ISO/DIS 5060:2022
---------------------- Page: 8 ----------------------
oSIST ISO/DIS 5060:2022
DRAFT INTERNATIONAL STANDARD ISO/DIS 5060:2022(E)
Translation services — Evaluation of translation output —
General guidance
1 Scope

This document provides guidance for the evaluation of human translation output, post-edited machine

translation output, and unedited machine translation output. Its focus lies on an analytic translation

evaluation method using error types and penalty points configured to produce an error score and a

quality rating.

This document focusses on the human evaluation of the translation output only and follows an approach

designed to reflect minimum complexity. The reason behind this approach is to keep this document

applicable for as many users as possible in the translation industry.

This document does not deal with related elements such as the processes of assuring the quality of the

translation output and corrective actions.

This document is intended to be used by translation service providers (TSPs, be they individual

translators, translation companies or inhouse translation services), their clients and other interested

parties in the translation industry.
This document does not apply to interpreting.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 20539, Translation, interpreting and related technology — Vocabulary
3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO 20539 and the following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1 General concepts
3.1.1
translate

render source language content (3.1.7) into target language content (3.1.8) in written form or signed

language
[SOURCE: ISO 20539:2019, 3.1.7]
© ISO 2022 – All rights reserved
---------------------- Page: 9 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
3.1.2
translation

set of processes to render source language content (3.1.7) into target language content (3.1.8) in written

form

Note 1 to entry: A translation can refer to formats other than text-based formats (e.g. an audio file, image, etc.).

[SOURCE: ISO 17100:2015, 2.1.2, modified – “may” changed to “can” in Note 1 to entry]

3.1.3
revision
bilingual editing

bilingual examination of target language content (3.1.8) against source language content (3.1.7) for its

suitability for the agreed purpose

[SOURCE: ISO 17100:2015, 2.2.6, modified – content of note changed to admitted term]

3.1.4
evaluation of translation output
evaluation

bilingual examination of target language content (3.1.8) against source language content (3.1.7) while

classifying any errors (3.4.1) against translation project specifications (3.3.2) and for the purpose of

reaching a quality rating (3.3.10)

Note 1 to entry: The term “assessment” is frequently used as a synonym for evaluation. While both involve

inspection, “evaluation” is the preferred term, since formally there are some differences. Evaluation is product-

oriented and judgmental (to accept or reject a product), whereas assessment is process-oriented and diagnostic

(to provide feedback on how to improve the process used to produce the product). The preference for “evaluation”

is also to avoid the confusion that the abbreviation “QA” gives rise to, mostly referring to quality assurance.

3.1.5
corrective action

action taken to eliminate the cause of nonconformities or errors (3.4.1) during translation (3.1.2) or

target language content (3.1.8)

Note 1 to entry: Corrective action involves an investigation to identify what went wrong and what action can be

taken to ensure that it does not happen in the same way again.

[SOURCE: ISO 17100:2015, 2.5.5, modified –plural deleted in “nonconformity” and “in the translation

process” changed to “translation”]
3.1.6
content
information in any form
EXAMPLE Text, audio, video, etc.
[SOURCE: ISO 20539:2019, 3.1.2]
3.1.7
source language content
language content (3.1.6) to be translated (3.1.1)
[SOURCE: ISO 17100:2015, 2.3.2]
3.1.8
target language content
language content (3.1.6) translated (3.1.1) from source language content (3.1.7)
[SOURCE: ISO 17100:2015, 2.3.3]
© ISO 2022 – All rights reserved
---------------------- Page: 10 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
3.1.9
translation output
result of translation (3.1.2)
[SOURCE: ISO 20539:2019, 3.3.3]
3.1.10
client-TSP agreement

mutual acknowledgement that sets forth the provisions and conditions of a translation service

Note 1 to entry: This can take the form of a contract, memorandum of understanding (MOU), verbal agreements,

e-mail, etc. A verbal agreement can be documented in writing.
3.1.11
translation service

production and delivery of target language content (3.1.8) according to a client-TSP agreement (3.1.10)

[SOURCE: ISO 20539:2019, 3.3.4, modified – “client specification” changed to “client-TSP agreement”]

3.1.12
locked segment

bitext segment in a translation memory for which access has been restricted in order to prevent it from

being changed or edited
3.2 Concepts related to the people involved in the evaluation process
3.2.1
evaluator

person who performs evaluation of translation output (3.1.4) in accordance with the rules of the

evaluating entity (3.2.2)
3.2.2
evaluating entity

individual or group of people within an organization which determines the translation evaluation

strategy (3.3.3)
3.2.3
translator
person who translates (3.1.1)
[SOURCE: ISO 20539:2019, 3.1.11]
3.2.4
reviser
person who performs revision (3.1.3)
[SOURCE: ISO 20539:2019, 3.3.10]
3.2.5
translation service provider
TSP
person or organization that performs translation service
[SOURCE: ISO 20539:2019, 3.3.5]
© ISO 2022 – All rights reserved
---------------------- Page: 11 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
3.3 Concepts related to the evaluation process
3.3.1
requirement
need or expectation that is stated, generally implied or obligatory
[SOURCE: ISO 9000:2015, 3.6.4, modified – notes deleted]
3.3.2
translation project specifications

set of agreed upon and defined requirements (3.3.1) for producing translation output (3.1.9)

Note 1 to entry: ISO 17100, Annex B, lists a set of potential project specifications.

3.3.3
translation evaluation strategy

organization-wide policy implemented on the basis of the specific purpose for the evaluation of

translation output (3.1.4)

Note 1 to entry: The purpose for the evaluation of translation output can vary from project to project. Therefore,

it might be necessary to apply various translation evaluation strategies.
3.3.4
translation evaluation system

set of steps and actions performed during the evaluation of translation output (3.1.4) implemented on

the basis of the translation evaluation strategy (3.3.3)
3.3.5
translation evaluation specifications

principles for the evaluation of translation output (3.1.4), based on the translation project specifications

(3.3.2)
3.3.6
translation evaluation sample
sample

amount of content (3.1.6) taken from target language content (3.1.8) to be submitted to an evaluation of

translation output (3.1.4)
3.3.7
sampling

defined procedure whereby a part of the translation output (3.1.9) is selected to provide a translation

evaluation sample (3.3.6)
3.3.8
translation evaluation scorecard
translation evaluation scoresheet

table containing all relevant error types (3.4.3) with their assigned penalty points (3.5.1), for the purpose

of producing an error score (3.3.9) for the evaluated translation output (3.1.9)
3.3.9
error score

result of the evaluation penalty point total (3.5.4), in relation to the number of characters, number of

words, or number of lines evaluated in the translation output (3.1.9)
3.3.10
quality rating

classification of the evaluation of translation output (3.1.4), based on comparing the error score (3.3.10)

with an applicable error score threshold (3.5.9) and rating system
© ISO 2022 – All rights reserved
---------------------- Page: 12 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
3.3.11
risk assessment

process of identifying and assessing the potential risk level based on the likelihood that an error (3.4.1)

will occur according to the perceived severity of its impact
3.4 Concepts related to errors
3.4.1
error
failure to adhere to translation project specifications (3.3.2)
3.4.2
error typology
classification of errors (3.4.1)
3.4.3
error type

class of translation error (3.4.1) identified by name, definition and position in an error typology (3.4.2)

3.4.4
main error type
superordinate error type (3.4.3)
3.4.5
error sub-type
subordinate error type (3.4.3)
3.4.6
severity level

rating of the impact of an error (3.4.1) on the target language content (3.1.8) and the suitability for the

intended purpose
Note 1 to entry: Example of severity levels: neutral, minor, major, critical.
3.4.7
minor error

error (3.4.1) assignable to a severity level (3.4.6) that does not prevent the translation output (3.1.9) from

fulfilling its intended purpose
3.4.8
major error

error (3.4.1) assignable to a severity level (3.4.6) that might prevent the translation output (3.1.9) from

fulfilling its intended purpose
3.4.9
critical error

error (3.4.1) assignable to a severity level (3.4.6) that prevents the translation output (3.1.9) from

fulfilling its intended purpose
3.4.10
repeated error

error (3.4.1), recurring within a text, that is annotated, but may or may not be attributed with penalty

points (3.5.1)
3.4.11
severity multiplier value

numerical value reflecting severity levels (3.4.6) of a given error (3.4.1) to be multiplied with error counts

(3.4.12)
© ISO 2022 – All rights reserved
---------------------- Page: 13 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)
3.4.12
error count
number of errors (3.4.1) in an evaluation of translation output (3.1.4)
3.5 Concepts related to penalty points
3.5.1
penalty point

numerical value calculated for each error type (3.4.3) based on the severity multiplier value (3.4.11)

3.5.2
error type penalty point
penalty point (3.5.1) assigned to a given error type (3.4.3)
3.5.3
error type penalty point total
sum of penalty points (3.5.1) for a given error type (3.4.3)
3.5.4
evaluation penalty point total

sum of error type penalty point totals (3.5.3) of an evaluation of translation output (3.1.4)

3.5.5
error type weight

numerical value used to modify the unweighted error type penalty point total (3.5.3) per error type

(3.4.3)
3.5.6
weight
applying error type weights (3.5.5)
3.5.7
weighted error type penalty point total
weight of an error type (3.4.3) after applying error type weights (3.5.5)
3.5.8
unweighted error type penalty point total
weight of an error type (3.4.3) before applying error type weights (3.5.5)
3.5.9
error score threshold

target value above or below which an evaluation of translation output (3.1.4) is passed

4 Human resources
4.1 General

The TSP should have a process in place to ensure that people performing evaluation of translation

output have the competences and qualifications outlined below.
4.2 Professional competences of evaluators
Evaluators should have the following competences:

a) Translation competence: The ability to translate content, including the ability to address the

problems of language content comprehension and language content production and the ability

to render the target language content in accordance with the client-TSP agreement and other

translation project specifications.
© ISO 2022 – All rights reserved
---------------------- Page: 14 ----------------------
oSIST ISO/DIS 5060:2022
ISO/DIS 5060:2022(E)

b) Revision competence: The ability to objectively compare target language content against source

language content and the ability to identify potential errors and suggest or make corrections in

order to assure that the revised target language content meets the client’s specifications and the

specifications agreed upon by the client-TSP agreement.

c) Evaluation competence: This competence requires attention to detail, the ability to follow

instructions precisely, and the ability to communicate feedback positively. This competence

includes the ability to objectively classify and annotate any errors found during the evaluation

process, assign them to their relevant error type and severity level in order to establish an error

score by filling out the translation evaluation scorecard.

d) Linguistic and textual competence in the source language and the target language: The

ability to understand the source language, fluency in the target language, and general or specialized

knowledge of text-type conventions. This linguistic and textual competence includes the ability to

apply this knowledge when producing translation output or other target language content.

e) Competence in information acquisition and processing: The ability to understand the processes

required for the evaluation and to apply specifications efficiently. In some cases, research

competence could also be required to verify decisions taken and to develop suitable strategies for

efficiently accomplishing the tasks required.

f) Cultural competence: The ability to make use of information on the behavioural standards, up-

to-date terminology, value systems, and locale that characterize both source and target language

cultures.

g) Technical competence: The knowledge, abilities, and skills required to perform the technical

tasks in the evaluation process by employing technical resources including the tools and IT systems

that support the entire evaluation process.

h) Domain competence: The ability to understand source language content and to evaluate the

correspondence in the target language verifying the appropriate style and terminology.

4.3 Qualifications of evaluators

The TSP should determine the evaluator’s qualifications by obtaining documented evidence that the

evaluator fulfils at least one of the following criteria:

a) The evaluator has obtained a degree in translation, linguistics, language studies or an equivalent

degree from a recognized institution of higher education; the degree should include significant

training in translation and revision.

b) The evaluator has obtained a degree in any other field from a recognized institution of higher

education and has the equivalent of two years documented experience in translation and revision.

...