Information and documentation -- Quality assessment for national libraries

This document defines terms for the quality assessment of national libraries and specifies the following methods for the assessment: — performance measurement, and — impact assessment. The results of both methods are of special interest for comparison over time within the same library. Comparisons between libraries are possible if differences in the mandate, tasks and constituencies of the libraries are taken into account. Not all methods described in this document apply to all national libraries. Limitations of the applicability of individual methods are specified in the descriptions. This document is not intended to exclude the use of performance indicators or of methods for impact assessment not specified in it. This document does not cover web archiving, but refers to ISO/TR 14873 for statistics and quality issues for this new task of national libraries.

Information et documentation -- Evaluation de qualité pour les bibliothèques nationales

Informatika in dokumentacija - Ocena kakovosti za nacionalne knjižnice

Ta dokument opredeljuje izraze za oceno kakovosti nacionalnih knjižnic in določa naslednje
metode ocenjevanja:
– merjenje učinkovitosti in
– oceno vpliva.
Rezultati obeh metod so še posebej zanimivi za primerjavo, kako so se rezultati v isti knjižnici spreminjali skozi čas.
Primerjave med knjižnicami so možne, če se upošteva razlike v pooblastilih, nalogah in bralcih
oz. obiskovalcih knjižnic.
Vendar se vse metode, ki so opisane v tem dokumentu, ne uporabljajo za vse nacionalne knjižnice. Omejitve uporabe
posameznih metod so določene v opisih.
Ta dokument naj ne bi izključeval uporabe kazalnikov uspešnosti ali metod za
ocenjevanje vpliva, ki niso navedene v njem.
Ta dokument ne zajema spletnega arhiviranja, ampak se glede statistike in kakovosti navezuje na ISO/TR 14873
za to novo nalogo nacionalnih knjižnic.

General Information

Status
Published
Publication Date
03-Mar-2019
Current Stage
6060 - International Standard published
Start Date
30-Jan-2019
Completion Date
04-Mar-2019

Buy Standard

Standard
ISO 21248:2019 - Information and documentation -- Quality assessment for national libraries
English language
139 pages
sale 15% off
Preview
sale 15% off
Preview
Standard
ISO 21248:2019
English language
145 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day

Standards Content (sample)

INTERNATIONAL ISO
STANDARD 21248
First edition
2019-03
Information and documentation —
Quality assessment for national
libraries
Information et documentation — Evaluation de qualité pour les
bibliothèques nationales
Reference number
ISO 21248:2019(E)
ISO 2019
---------------------- Page: 1 ----------------------
ISO 21248:2019(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2019

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2019 – All rights reserved
---------------------- Page: 2 ----------------------
ISO 21248:2019(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Quality assessment in national libraries .................................................................................................................................17

4.1 Mission and functions of national libraries .................................................................................................................17

4.1.1 General...................................................................................................................................................................................17

4.1.2 Mission statement for national libraries ..................................................................................................18

4.1.3 Core functions of national libraries ..............................................................................................................18

4.1.4 Additional functions of national libraries ...............................................................................................19

4.2 Target groups of national libraries .....................................................................................................................................20

4.3 Quality criteria in national libraries ..................................................................................................................................21

4.3.1 Quality criteria that apply to all types of libraries ...........................................................................21

4.3.2 Specific quality criteria for national libraries ......................................................................................21

4.4 Methods for quality assessment in national libraries ........................................................................................21

5 Performance indicators for national libraries ...................................................................................................................22

5.1 General ........................................................................................................................................................................................................22

5.2 Use of performance indicators ...............................................................................................................................................22

5.2.1 General...................................................................................................................................................................................22

5.2.2 Selection of performance indicators ............................................................................................................22

5.2.3 Limitations of performance measurement ............................................................................................23

5.3 Criteria for performance indicators ...................................................................................................................................23

5.4 Descriptive framework .................................................................................................................................................................24

5.4.1 General...................................................................................................................................................................................24

5.4.2 Name .......................................................................................................................................................................................24

5.4.3 Background........................................................................................................................................................................24

5.4.4 Objective of the performance indicator ....................................................................................................24

5.4.5 Definition of the performance indicator ..................................................................................................24

5.4.6 Method(s)............................................................................................................................................................................24

5.4.7 Interpretation and use of results ....................................................................................................................25

5.4.8 Source(s) ..............................................................................................................................................................................25

5.4.9 Examples and further reading (optional)................................................................................................25

5.5 List of performance indicators for national libraries .........................................................................................25

6 Impact assessment in national libraries ..................................................................................................................................27

6.1 Overview ...................................................................................................................................................................................................27

6.2 Definition and description of library impact .............................................................................................................27

6.2.1 General...................................................................................................................................................................................27

6.2.2 Definition of library impact .................................................................................................................................27

6.2.3 Effects of library impact .........................................................................................................................................27

6.2.4 Challenges in assessing impact ........................................................................................................................30

6.2.5 Use of impact assessment results .................. .................................................................................................30

6.3 Methods for assessing impact of national libraries ..............................................................................................31

6.3.1 General...................................................................................................................................................................................31

6.3.2 Inferred evidence .........................................................................................................................................................31

6.3.3 Observed evidence ......................................................................................................................................................33

6.3.4 Solicited evidence ........................................................................................................................................................34

6.3.5 Combined methods .....................................................................................................................................................36

6.4 Assessing the economic value of national libraries .............................................................................................37

6.4.1 General...................................................................................................................................................................................37

6.4.2 Calculating the value of library benefits to users .............................................................................37

6.4.3 Economic impact analysis .....................................................................................................................................40

© ISO 2019 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO 21248:2019(E)

6.5 Assessing impact on a specified target population ..............................................................................................40

6.5.1 General...................................................................................................................................................................................40

6.5.2 Assessing impact on researchers ....................................................................................................................40

6.5.3 Assessing impact on educators and learners .......................................................................................41

6.5.4 Assessing impact on the general public ....................................................................................................43

6.5.5 Assessing impact on the library and information network .....................................................44

6.5.6 Assessing impact on publishers and authors .......................................................................................45

6.5.7 Assessing impact on public administration ...........................................................................................46

6.5.8 Assessing impact on business ...........................................................................................................................47

Annex A (normative) Description of performance indicators ................................................................................................50

Annex B (informative) Examples of impact surveys ......................................................................................................................123

Annex C (informative) Performance indicators in the structure of the balanced scorecard ..............133

Bibliography .........................................................................................................................................................................................................................135

iv © ISO 2019 – All rights reserved
---------------------- Page: 4 ----------------------
ISO 21248:2019(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso

.org/iso/foreword .html.

This document was prepared by Technical Committee ISO/TC 46, Information and documentation,

Subcommittee SC 8, Quality — Statistics and performance evaluation.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/members .html.
© ISO 2019 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO 21248:2019(E)
Introduction
This document is concerned with the evaluation of national libraries.

It provides guidance on the use of performance indicators regarding the quality of services in national

libraries and on methods for assessing the impact and value of national libraries.

Clause 5 resumes the work done in ISO/TR 28118:2009 where, for the first time, a specific evaluation

method was described for national libraries. This document establishes a set of performance indicators

that are adapted to the mission and functions of national libraries. Annex A specifies how the indicators

are intended to be calculated and used.

In Clause 6, this document describes methods for identifying and proving the impact of national

libraries on individuals, institutions and on society. The methods described in Clause 6 do not reflect all

possible methods or evaluation techniques, but are those seen to be most effective for assessing impact

of national libraries. Annex B gives examples of impact surveys, considering different user groups of

national libraries.
The texts in this document are partly based on ISO 11620 and ISO 16439.
vi © ISO 2019 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO 21248:2019(E)
Information and documentation — Quality assessment for
national libraries
1 Scope

This document defines terms for the quality assessment of national libraries and specifies the following

methods for the assessment:
— performance measurement, and
— impact assessment.

The results of both methods are of special interest for comparison over time within the same library.

Comparisons between libraries are possible if differences in the mandate, tasks and constituencies of

the libraries are taken into account.

Not all methods described in this document apply to all national libraries. Limitations of the applicability

of individual methods are specified in the descriptions.

This document is not intended to exclude the use of performance indicators or of methods for impact

assessment not specified in it.

This document does not cover web archiving, but refers to ISO/TR 14873 for statistics and quality

issues for this new task of national libraries.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https: //www .iso .org/obp
— IEC Electropedia: available at http: //www .electropedia .org/
3.1
access
successful request of a library-provided online service

Note 1 to entry: An access is one cycle of user activities that typically starts when a user connects to a library-

provided online service and ends by a terminating activity that is either explicit (by leaving the database through

log-out or exit) or implicit (timeout due to user inactivity).

Note 2 to entry: Accesses to the library website (3.46) are counted as virtual visits (3.97).

Note 3 to entry: Requests of a general entrance or gateway page should be excluded.

Note 4 to entry: If possible, requests by search engines should be excluded.
[SOURCE: ISO 2789:2013, 2.2.1]
© ISO 2019 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO 21248:2019(E)
3.2
accessibility
ease of reaching and using a service or facility
[SOURCE: ISO 11620:2014, 2.2]
3.3
anecdotal evidence

evidence based on anecdotes or stories, obtained informally from personal observations and

experiences, not collected systematically nor empirically tested
[SOURCE: ISO 16439:2014, 3.5]
3.4
availability

degree to which content, documents, facilities or services are actually provided by the library at the

time required by users
[SOURCE: ISO 11620:2014, 2.6]
3.5
benefit
helpful or good effect, or something intended to help
[SOURCE: ISO 16439:2014, 3.7]
3.6
book
non-serial printed document in codex form
[SOURCE: ISO 2789:2013, 2.3.4]
3.7
capital expenditure
expenditure which results in the acquisition of, or addition to, fixed assets

Note 1 to entry: This includes expenditure on building sites, new buildings and extensions, furnishings and

equipment for new and expanded buildings, computer systems (hardware and software), etc. When applicable,

local and national sales/purchase taxes [e.g. value added tax (VAT)] are included.

[SOURCE: ISO 2789:2013, 2.6.1]
3.8
closed stacks
stacks that are not accessible to users
Note 1 to entry: Antonym: open stacks (3.63).
[SOURCE: ISO/TR 11219:2012, 2.19]
3.9
commercial publication

document in print or electronic format that is generally produced, distributed and sold for the purpose

of profit
Note 1 to entry: Excludes publications by non-profit organizations.
[SOURCE: ISO 2789:2013, 2.3.7, modified — Note 1 to entry has been added.]
2 © ISO 2019 – All rights reserved
---------------------- Page: 8 ----------------------
ISO 21248:2019(E)
3.10
conservation

preservation measures and actions applied to prevent, arrest or delay deterioration of a document or

other material object
[SOURCE: ISO 5127:2017, 3.12.1.01]
3.11
content unit

computer-processed uniquely identifiable textual or audio-visual piece of published work that may be

original or a digest of other published work

Note 1 to entry: This includes documents or parts of documents (e.g. articles, abstracts, content tables, images)

and descriptive records.
Note 2 to entry: Adapted from COUNTER code of practice, Release 3:2008.

Note 3 to entry: PDF, Postscript, HTML and other formats of the same content unit will be counted as separate items.

[SOURCE: ISO 2789:2013, 2.3.9]
3.12
contingent valuation

method for assessing the economic value of non-profit institutions and projects of public utility

depending on potential users’ responses to survey questions such as what they are willing to pay for a

benefit (3.5) or feature (willingness-to-pay), or what they would accept as financial compensation if a

certain benefit or feature was missing (willingness-to-accept)

Note 1 to entry: The method is used in environmental protection and health care, for example.

[SOURCE: ISO 16439:2014, 3.9]
3.13
cultural economics

branch of economics that studies the relation of culture to economic outcomes and explores cultural

phenomena as economic factors
[SOURCE: ISO 16439:2014, 3.12]
3.14
data mining

computational process that identifies patterns by analysing quantitative data (3.73) from different

perspectives and dimensions, categorizing it, and summarizing potential relationships and impacts

[SOURCE: ISO 16439:2014, 3.13]
3.15
digital document

information unit with a defined content, born digital or digitized, that has been created or digitized by

the library or acquired in digital form as part of the library collection (3.43)

Note 1 to entry: This includes eBooks, electronic patents, networked audio-visual documents and other digital

documents, e.g. reports, cartographic and music documents, preprints, etc. Databases and electronic serials are

excluded.
Note 2 to entry: A digital document can be structured into one or more files.

Note 3 to entry: A digital document consists of one or more content units (3.11). Before digitization (3.16), the

library must decide which content units should be searchable afterwards, e.g. articles in serials, or songs on

records.
[SOURCE: ISO 2789:2013, 2.3.13]
© ISO 2019 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO 21248:2019(E)
3.16
digitization
process of converting analogue materials into digital form

Note 1 to entry: Digitization for document supply from the library collection (3.43)to a user or institution is

excluded.
Note 2 to entry: Digitization for preservation purposes is included.
Note 3 to entry: Mass digitization is included.

Note 4 to entry: Purchase of electronic copies for replacing print copies is excluded.

[SOURCE: ISO 2789:2013, 2.3.15]
3.17
document

recorded information or material object which can be treated as a unit in a documentation process

Note 1 to entry: Documents can differ in form and characteristics.

[SOURCE: ISO 5127:2017, 3.1.1.38, modified — Note 1 to entry of ISO 2789:2013, 2.3.17 has been

included.]
3.18
document processing

everything done to a document after it is acquired by a library, before it is placed on the shelves, including

acquisition, descriptive cataloguing, subject indexing, if applicable binding, physical processing, and

shelving

Note 1 to entry: For electronic documents, the end of processing will either coincide with cataloguing when

the URL for a document on an external server has been added, or with the installation of the document on the

library’s server and the URL being added in the catalogue.
[SOURCE: ISO/TR 28118:2009, 2.13, modified — Note 1 to entry has been added.]
3.19
document retrieval
process of recovering specific documents from a storage (3.87) place

[SOURCE: ISO 5127:2017, 3.10.1.03, modified — The term “placement” has been excluded in the

definition.]
3.20
economic impact

effect of a policy, decision, institution, or event on the economy of a given area

Note 1 to entry: Economic impact is usually measured in terms of changes in economic growth (output or value

added) and associated changes in jobs (employment) and income (wages).

[SOURCE: ISO 16439:2014, 3.16, modified — The term “decision” has been added while “programme”

has been excluded in the definition.]
3.21
economic impact of libraries
positive influence of libraries on the local, regional or national economic life
4 © ISO 2019 – All rights reserved
---------------------- Page: 10 ----------------------
ISO 21248:2019(E)
3.22
educational services

learning sessions and learning materials and programmes in all formats for children and adults for the

purpose of enhancing skills in library and information use

Note 1 to entry: This includes the provision of services for schools and the cooperation with other libraries in

preparing and offering educational services.
Note 2 to entry: Education of librarians is excluded.
3.23
effectiveness
measure of the degree to which given objectives are achieved

Note 1 to entry: An activity is effective if it maximizes the results it was established to produce.

[SOURCE: ISO 11620:2014, 2.15]
3.24
efficiency
measure of the utilisation of resources to realize a given objective

Note 1 to entry: An activity is efficient if it minimizes the use of resources, or produces better performance with

the same resources.
[SOURCE: ISO 11620:2014, 2.16]
3.25
electronic collection
digital collection

all resources in electronic form in the library collection (3.43), whether born digital or digitized

Note 1 to entry: The electronic collection includes databases, electronic serials, and digital documents (3.15). Free

Internet resources (3.31) which have been catalogued by the library in its online catalogue (3.61) or a database are

excluded.

Note 2 to entry: The resources can be networked, installed on stand-alone workstations or stored on physical

carriers.

[SOURCE: ISO 2789:2013, 2.3.21, modified — The term “digital collection” has been added as an

accepted term.]
3.26
electronic document delivery

electronic transmission of a document or part of a document from the library collection

(3.43) to a user, mediated by library staff not necessarily via another library

Note 1 to entry: Electronic transmission of documents to members of the population to be served (3.67) is

included. Fax transmission is excluded.

Note 2 to entry: May be split up as to transmission with or without charge to the user.

Note 3 to entry: Unmediated downloading by users from the electronic collection (3.25) of the library is excluded.

[SOURCE: ISO 2789:2013, 2.2.7]
3.27
electronic service

library service delivered via electronic means whether from local servers or provided via networks

Note 1 to entry: Electronic library services include the online catalogue (3.61), the library website (3.46), the

electronic collection (3.25), electronic lending, electronic document delivery (mediated), electronic reference

service (3.77), user training (3.94) by electronic means, services for mobile devices (3.51), services for interactive

use (including services on social networks), and Internet access offered via the library.

© ISO 2019 – All rights reserved 5
---------------------- Page: 11 ----------------------
ISO 21248:2019(E)

Note 2 to entry: This does not include booking physical services (e.g. rooms or library tours) by electronic means.

[SOURCE: ISO 2789:2013, 2.2.8]
3.28
evaluation

process of estimating the effectiveness, efficiency, utility and relevance of a service or facility

[SOURCE: ISO 11620:2014, 3.19]
3.29
event

pre-arranged activity with cultural, educational, social, political, scholarly, or other intent

EXAMPLE Exhibitions, author visits, literary discussions, workshops, etc.

Note 1 to entry: Only events arranged by the library on its own or in partnership with other institutions are

included, whether inside or outside the library premises. Events inside the library premises organized by

institutions outside the library without the library’s cooperation are excluded.
Note 2 to entry: User training (3.94) lessons and library tours are excluded.

Note 3 to entry: Ongoing programmes are included. Each session of a programme is counted as one event.

Note 4 to entry: Virtual events are included.
[SOURCE: ISO 2789:2013, 2.2.9]
3.30
focus group interview
focus group discussion

group interview in the form of a moderated discussion among a small number of selected individuals on

topics introduced by the moderator
[SOURCE: ISO 16439:2014, 3.23]
3.31
free Internet resource

Internet resource with unrestricted (open) access for which no payment is required

[SOURCE: ISO 2789:2013, 2.3.23]
3.32
full-time equivalent
FTE

measurement equal to one staff person working a full-time work schedule for one year

Note 1 to entry: For example, if out of three persons employed as librarians, one works quarter-time, one works half-

time, and one works full-time, then the FTE of these three persons would be 0,25 + 0,5 + 1,0 = 1,75 librarians (FTE).

Note 2 to entry: Not all libraries may use the same number of hours per year to determine an FTE. Thus, any

comparative measures between libraries may need to consider any differences in hours.

[SOURCE: ISO 11620:2014, 2.23]
3.33
goal
desired state of affairs to be achieved by the implementation of agreed policies
[SOURCE: ISO 11620:2014, 2.24]
6 © ISO 2019 – All rights reserved
---------------------- Page: 12 ----------------------
ISO 21248:2019(E)
3.34
homepage

main or opening screen of a website to which all other pages for users are linked

[SOURCE: ISO 5127:2017, 3.3.3.26]
3.35
impact

difference or change in an individual or group resulting from the contact with library services

Note 1 to entry: The change can be tangible or intangible.
[SOURCE: ISO 16439:2014, 3.25]
3.36
income generated
income generated by library operations and fro
...

SLOVENSKI STANDARD
SIST ISO 21248:2019
01-oktober-2019
Informatika in dokumentacija - Ocena kakovosti za nacionalne knjižnice
Information and documentation -- Quality assessment for national libraries

Information et documentation -- Evaluation de qualité pour les bibliothèques nationales

Ta slovenski standard je istoveten z: ISO 21248:2019
ICS:
01.140.20 Informacijske vede Information sciences
03.120.10 Vodenje in zagotavljanje Quality management and
kakovosti quality assurance
SIST ISO 21248:2019 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST ISO 21248:2019
---------------------- Page: 2 ----------------------
SIST ISO 21248:2019
INTERNATIONAL ISO
STANDARD 21248
First edition
2019-03
Information and documentation —
Quality assessment for national
libraries
Information et documentation — Evaluation de qualité pour les
bibliothèques nationales
Reference number
ISO 21248:2019(E)
ISO 2019
---------------------- Page: 3 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2019

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2019 – All rights reserved
---------------------- Page: 4 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Quality assessment in national libraries .................................................................................................................................17

4.1 Mission and functions of national libraries .................................................................................................................17

4.1.1 General...................................................................................................................................................................................17

4.1.2 Mission statement for national libraries ..................................................................................................18

4.1.3 Core functions of national libraries ..............................................................................................................18

4.1.4 Additional functions of national libraries ...............................................................................................19

4.2 Target groups of national libraries .....................................................................................................................................20

4.3 Quality criteria in national libraries ..................................................................................................................................21

4.3.1 Quality criteria that apply to all types of libraries ...........................................................................21

4.3.2 Specific quality criteria for national libraries ......................................................................................21

4.4 Methods for quality assessment in national libraries ........................................................................................21

5 Performance indicators for national libraries ...................................................................................................................22

5.1 General ........................................................................................................................................................................................................22

5.2 Use of performance indicators ...............................................................................................................................................22

5.2.1 General...................................................................................................................................................................................22

5.2.2 Selection of performance indicators ............................................................................................................22

5.2.3 Limitations of performance measurement ............................................................................................23

5.3 Criteria for performance indicators ...................................................................................................................................23

5.4 Descriptive framework .................................................................................................................................................................24

5.4.1 General...................................................................................................................................................................................24

5.4.2 Name .......................................................................................................................................................................................24

5.4.3 Background........................................................................................................................................................................24

5.4.4 Objective of the performance indicator ....................................................................................................24

5.4.5 Definition of the performance indicator ..................................................................................................24

5.4.6 Method(s)............................................................................................................................................................................24

5.4.7 Interpretation and use of results ....................................................................................................................25

5.4.8 Source(s) ..............................................................................................................................................................................25

5.4.9 Examples and further reading (optional)................................................................................................25

5.5 List of performance indicators for national libraries .........................................................................................25

6 Impact assessment in national libraries ..................................................................................................................................27

6.1 Overview ...................................................................................................................................................................................................27

6.2 Definition and description of library impact .............................................................................................................27

6.2.1 General...................................................................................................................................................................................27

6.2.2 Definition of library impact .................................................................................................................................27

6.2.3 Effects of library impact .........................................................................................................................................27

6.2.4 Challenges in assessing impact ........................................................................................................................30

6.2.5 Use of impact assessment results .................. .................................................................................................30

6.3 Methods for assessing impact of national libraries ..............................................................................................31

6.3.1 General...................................................................................................................................................................................31

6.3.2 Inferred evidence .........................................................................................................................................................31

6.3.3 Observed evidence ......................................................................................................................................................33

6.3.4 Solicited evidence ........................................................................................................................................................34

6.3.5 Combined methods .....................................................................................................................................................36

6.4 Assessing the economic value of national libraries .............................................................................................37

6.4.1 General...................................................................................................................................................................................37

6.4.2 Calculating the value of library benefits to users .............................................................................37

6.4.3 Economic impact analysis .....................................................................................................................................40

© ISO 2019 – All rights reserved iii
---------------------- Page: 5 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)

6.5 Assessing impact on a specified target population ..............................................................................................40

6.5.1 General...................................................................................................................................................................................40

6.5.2 Assessing impact on researchers ....................................................................................................................40

6.5.3 Assessing impact on educators and learners .......................................................................................41

6.5.4 Assessing impact on the general public ....................................................................................................43

6.5.5 Assessing impact on the library and information network .....................................................44

6.5.6 Assessing impact on publishers and authors .......................................................................................45

6.5.7 Assessing impact on public administration ...........................................................................................46

6.5.8 Assessing impact on business ...........................................................................................................................47

Annex A (normative) Description of performance indicators ................................................................................................50

Annex B (informative) Examples of impact surveys ......................................................................................................................123

Annex C (informative) Performance indicators in the structure of the balanced scorecard ..............133

Bibliography .........................................................................................................................................................................................................................135

iv © ISO 2019 – All rights reserved
---------------------- Page: 6 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso

.org/iso/foreword .html.

This document was prepared by Technical Committee ISO/TC 46, Information and documentation,

Subcommittee SC 8, Quality — Statistics and performance evaluation.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/members .html.
© ISO 2019 – All rights reserved v
---------------------- Page: 7 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
Introduction
This document is concerned with the evaluation of national libraries.

It provides guidance on the use of performance indicators regarding the quality of services in national

libraries and on methods for assessing the impact and value of national libraries.

Clause 5 resumes the work done in ISO/TR 28118:2009 where, for the first time, a specific evaluation

method was described for national libraries. This document establishes a set of performance indicators

that are adapted to the mission and functions of national libraries. Annex A specifies how the indicators

are intended to be calculated and used.

In Clause 6, this document describes methods for identifying and proving the impact of national

libraries on individuals, institutions and on society. The methods described in Clause 6 do not reflect all

possible methods or evaluation techniques, but are those seen to be most effective for assessing impact

of national libraries. Annex B gives examples of impact surveys, considering different user groups of

national libraries.
The texts in this document are partly based on ISO 11620 and ISO 16439.
vi © ISO 2019 – All rights reserved
---------------------- Page: 8 ----------------------
SIST ISO 21248:2019
INTERNATIONAL STANDARD ISO 21248:2019(E)
Information and documentation — Quality assessment for
national libraries
1 Scope

This document defines terms for the quality assessment of national libraries and specifies the following

methods for the assessment:
— performance measurement, and
— impact assessment.

The results of both methods are of special interest for comparison over time within the same library.

Comparisons between libraries are possible if differences in the mandate, tasks and constituencies of

the libraries are taken into account.

Not all methods described in this document apply to all national libraries. Limitations of the applicability

of individual methods are specified in the descriptions.

This document is not intended to exclude the use of performance indicators or of methods for impact

assessment not specified in it.

This document does not cover web archiving, but refers to ISO/TR 14873 for statistics and quality

issues for this new task of national libraries.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https: //www .iso .org/obp
— IEC Electropedia: available at http: //www .electropedia .org/
3.1
access
successful request of a library-provided online service

Note 1 to entry: An access is one cycle of user activities that typically starts when a user connects to a library-

provided online service and ends by a terminating activity that is either explicit (by leaving the database through

log-out or exit) or implicit (timeout due to user inactivity).

Note 2 to entry: Accesses to the library website (3.46) are counted as virtual visits (3.97).

Note 3 to entry: Requests of a general entrance or gateway page should be excluded.

Note 4 to entry: If possible, requests by search engines should be excluded.
[SOURCE: ISO 2789:2013, 2.2.1]
© ISO 2019 – All rights reserved 1
---------------------- Page: 9 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
3.2
accessibility
ease of reaching and using a service or facility
[SOURCE: ISO 11620:2014, 2.2]
3.3
anecdotal evidence

evidence based on anecdotes or stories, obtained informally from personal observations and

experiences, not collected systematically nor empirically tested
[SOURCE: ISO 16439:2014, 3.5]
3.4
availability

degree to which content, documents, facilities or services are actually provided by the library at the

time required by users
[SOURCE: ISO 11620:2014, 2.6]
3.5
benefit
helpful or good effect, or something intended to help
[SOURCE: ISO 16439:2014, 3.7]
3.6
book
non-serial printed document in codex form
[SOURCE: ISO 2789:2013, 2.3.4]
3.7
capital expenditure
expenditure which results in the acquisition of, or addition to, fixed assets

Note 1 to entry: This includes expenditure on building sites, new buildings and extensions, furnishings and

equipment for new and expanded buildings, computer systems (hardware and software), etc. When applicable,

local and national sales/purchase taxes [e.g. value added tax (VAT)] are included.

[SOURCE: ISO 2789:2013, 2.6.1]
3.8
closed stacks
stacks that are not accessible to users
Note 1 to entry: Antonym: open stacks (3.63).
[SOURCE: ISO/TR 11219:2012, 2.19]
3.9
commercial publication

document in print or electronic format that is generally produced, distributed and sold for the purpose

of profit
Note 1 to entry: Excludes publications by non-profit organizations.
[SOURCE: ISO 2789:2013, 2.3.7, modified — Note 1 to entry has been added.]
2 © ISO 2019 – All rights reserved
---------------------- Page: 10 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
3.10
conservation

preservation measures and actions applied to prevent, arrest or delay deterioration of a document or

other material object
[SOURCE: ISO 5127:2017, 3.12.1.01]
3.11
content unit

computer-processed uniquely identifiable textual or audio-visual piece of published work that may be

original or a digest of other published work

Note 1 to entry: This includes documents or parts of documents (e.g. articles, abstracts, content tables, images)

and descriptive records.
Note 2 to entry: Adapted from COUNTER code of practice, Release 3:2008.

Note 3 to entry: PDF, Postscript, HTML and other formats of the same content unit will be counted as separate items.

[SOURCE: ISO 2789:2013, 2.3.9]
3.12
contingent valuation

method for assessing the economic value of non-profit institutions and projects of public utility

depending on potential users’ responses to survey questions such as what they are willing to pay for a

benefit (3.5) or feature (willingness-to-pay), or what they would accept as financial compensation if a

certain benefit or feature was missing (willingness-to-accept)

Note 1 to entry: The method is used in environmental protection and health care, for example.

[SOURCE: ISO 16439:2014, 3.9]
3.13
cultural economics

branch of economics that studies the relation of culture to economic outcomes and explores cultural

phenomena as economic factors
[SOURCE: ISO 16439:2014, 3.12]
3.14
data mining

computational process that identifies patterns by analysing quantitative data (3.73) from different

perspectives and dimensions, categorizing it, and summarizing potential relationships and impacts

[SOURCE: ISO 16439:2014, 3.13]
3.15
digital document

information unit with a defined content, born digital or digitized, that has been created or digitized by

the library or acquired in digital form as part of the library collection (3.43)

Note 1 to entry: This includes eBooks, electronic patents, networked audio-visual documents and other digital

documents, e.g. reports, cartographic and music documents, preprints, etc. Databases and electronic serials are

excluded.
Note 2 to entry: A digital document can be structured into one or more files.

Note 3 to entry: A digital document consists of one or more content units (3.11). Before digitization (3.16), the

library must decide which content units should be searchable afterwards, e.g. articles in serials, or songs on

records.
[SOURCE: ISO 2789:2013, 2.3.13]
© ISO 2019 – All rights reserved 3
---------------------- Page: 11 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
3.16
digitization
process of converting analogue materials into digital form

Note 1 to entry: Digitization for document supply from the library collection (3.43)to a user or institution is

excluded.
Note 2 to entry: Digitization for preservation purposes is included.
Note 3 to entry: Mass digitization is included.

Note 4 to entry: Purchase of electronic copies for replacing print copies is excluded.

[SOURCE: ISO 2789:2013, 2.3.15]
3.17
document

recorded information or material object which can be treated as a unit in a documentation process

Note 1 to entry: Documents can differ in form and characteristics.

[SOURCE: ISO 5127:2017, 3.1.1.38, modified — Note 1 to entry of ISO 2789:2013, 2.3.17 has been

included.]
3.18
document processing

everything done to a document after it is acquired by a library, before it is placed on the shelves, including

acquisition, descriptive cataloguing, subject indexing, if applicable binding, physical processing, and

shelving

Note 1 to entry: For electronic documents, the end of processing will either coincide with cataloguing when

the URL for a document on an external server has been added, or with the installation of the document on the

library’s server and the URL being added in the catalogue.
[SOURCE: ISO/TR 28118:2009, 2.13, modified — Note 1 to entry has been added.]
3.19
document retrieval
process of recovering specific documents from a storage (3.87) place

[SOURCE: ISO 5127:2017, 3.10.1.03, modified — The term “placement” has been excluded in the

definition.]
3.20
economic impact

effect of a policy, decision, institution, or event on the economy of a given area

Note 1 to entry: Economic impact is usually measured in terms of changes in economic growth (output or value

added) and associated changes in jobs (employment) and income (wages).

[SOURCE: ISO 16439:2014, 3.16, modified — The term “decision” has been added while “programme”

has been excluded in the definition.]
3.21
economic impact of libraries
positive influence of libraries on the local, regional or national economic life
4 © ISO 2019 – All rights reserved
---------------------- Page: 12 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)
3.22
educational services

learning sessions and learning materials and programmes in all formats for children and adults for the

purpose of enhancing skills in library and information use

Note 1 to entry: This includes the provision of services for schools and the cooperation with other libraries in

preparing and offering educational services.
Note 2 to entry: Education of librarians is excluded.
3.23
effectiveness
measure of the degree to which given objectives are achieved

Note 1 to entry: An activity is effective if it maximizes the results it was established to produce.

[SOURCE: ISO 11620:2014, 2.15]
3.24
efficiency
measure of the utilisation of resources to realize a given objective

Note 1 to entry: An activity is efficient if it minimizes the use of resources, or produces better performance with

the same resources.
[SOURCE: ISO 11620:2014, 2.16]
3.25
electronic collection
digital collection

all resources in electronic form in the library collection (3.43), whether born digital or digitized

Note 1 to entry: The electronic collection includes databases, electronic serials, and digital documents (3.15). Free

Internet resources (3.31) which have been catalogued by the library in its online catalogue (3.61) or a database are

excluded.

Note 2 to entry: The resources can be networked, installed on stand-alone workstations or stored on physical

carriers.

[SOURCE: ISO 2789:2013, 2.3.21, modified — The term “digital collection” has been added as an

accepted term.]
3.26
electronic document delivery

electronic transmission of a document or part of a document from the library collection

(3.43) to a user, mediated by library staff not necessarily via another library

Note 1 to entry: Electronic transmission of documents to members of the population to be served (3.67) is

included. Fax transmission is excluded.

Note 2 to entry: May be split up as to transmission with or without charge to the user.

Note 3 to entry: Unmediated downloading by users from the electronic collection (3.25) of the library is excluded.

[SOURCE: ISO 2789:2013, 2.2.7]
3.27
electronic service

library service delivered via electronic means whether from local servers or provided via networks

Note 1 to entry: Electronic library services include the online catalogue (3.61), the library website (3.46), the

electronic collection (3.25), electronic lending, electronic document delivery (mediated), electronic reference

service (3.77), user training (3.94) by electronic means, services for mobile devices (3.51), services for interactive

use (including services on social networks), and Internet access offered via the library.

© ISO 2019 – All rights reserved 5
---------------------- Page: 13 ----------------------
SIST ISO 21248:2019
ISO 21248:2019(E)

Note 2 to entry: This does not include booking physical services (e.g. rooms or library tours) by electronic means.

[SOURCE: ISO 2789:2013, 2.2.8]
3.28
evaluation

process of estimating the effectiveness, efficiency, utility and relevance of a service or facility

[SOURCE: ISO 11620:2014, 3.19]
3.29
event

pre-arranged activity with cultural, educational, social, political, scholarly, or other intent

EXAMPLE Exhibitions, author visits, literary discussions, workshops, etc.

Note 1 to entry: Only events arranged by the library on its own or in partnership with other institutions are

included, whether inside or outside the library premises. Events inside the library premises organized by

institutions outside the library without the library’s cooperation are excluded.
Note 2 to entry: User training (3.94) lessons and library tours are excluded.

Note 3 to entry: Ongoing programmes are included. Each session of a programme is counted as one event.

Note 4 to entry: Virtual events are included.
[SOURCE: ISO 2789:2013, 2.2.9]
3.30
focus group interview
focus group discussion

group interview in the form of a moderated discussion among a small number of selected individuals on

topics introduced by the moderator
[SOURCE: ISO 16439:2014, 3.23]
3.31
free Internet resource

Internet resource with unrestricted (open) access for which no payment is required

[SOURCE: ISO 2789:2013, 2.3.23]
3.32
full-time equivalent
FTE

measurement equal to one staff person working a full-time work schedule for one year

Note 1 to entry: For example, if out of three persons employed as librarians, one works quarter

...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.