ISO/IEC 29155-1:2017
(Main)Systems and software engineering — Information technology project performance benchmarking framework — Part 1: Concepts and definitions
Systems and software engineering — Information technology project performance benchmarking framework — Part 1: Concepts and definitions
ISO/IEC 29155-1:2017 identifies a framework for information technology (IT) project performance benchmarking (e.g. development or maintenance productivity) and related aspects (e.g. data collection and software classification). The framework consists of activities and components that are necessary to successfully identify, define, select, apply, and improve benchmarking for IT project performance. It also provides definitions for IT project performance benchmarking terms, which are also applicable to other parts of the ISO/IEC 29155 series. The target audience of this document are stakeholders of IT project performance benchmarking. NOTE The following are examples of how this document can be used: - by a benchmarking service provider who wants to align their benchmarking process to be consistent with this document; - by a benchmarking user (or third-party agents) for evaluating the performance of an IT project; - by an organization internally to answer specific information needs. ISO/IEC 29155-1:2017 does not cover how to organize benchmarking. It is out of the scope of this document to prescribe the name, format, or explicit content of the documentation to be produced from the benchmarking process.
Ingénierie des systèmes et du logiciel — Cadre de conduite de tests de performance de projet de technologies de l'information — Partie 1: Concepts et définitions
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 29155-1
Second edition
2017-12
Systems and software engineering —
Information technology project
performance benchmarking
framework —
Part 1:
Concepts and definitions
Ingénierie des systèmes et du logiciel — Cadre de conduite de tests de
performance de projet de technologies de l'information —
Partie 1: Concepts et définitions
Reference number
ISO/IEC 29155-1:2017(E)
©
ISO/IEC 2017
---------------------- Page: 1 ----------------------
ISO/IEC 29155-1:2017(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2017 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 29155-1:2017(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 4
5 Overview of the framework . 4
5.1 General . 4
5.2 Concepts of IT project performance benchmarking . 4
5.2.1 Overall framework . 4
5.2.2 Core benchmarking activities . 6
5.2.3 Supporting activities. 6
5.2.4 Benchmarking information base . 7
5.2.5 Benchmarking instruments . 8
5.2.6 External reference base . 8
5.2.7 Benchmarking experience base . 8
5.3 Stakeholders . 8
5.4 Types of benchmarking methods .10
5.5 IT project categories.11
6 Requirements for benchmarking .11
7 Benchmarking reporting .12
8 Data collection and maintenance .13
9 Additional standardization areas of benchmarking framework .14
9.1 General .14
9.2 Benchmarking domains .14
9.3 Data characteristics .15
9.4 Data analysis .15
9.5 Data exchange .15
Bibliography .16
© ISO/IEC 2017 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 29155-1:2017(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following
URL: www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
This second edition cancels and replaces the first edition (ISO/IEC 29155-1:2011), of which it constitutes
a minor revision. The changes compared to the previous edition are as follows:
— the introduction was restructured and updated to add descriptions for ISO/IEC 29155-2, -3, and -4,
which were published after the publication of the first edition;
— Clause 2 (Normative references) was added following the new requirements of ISO/IEC Directives,
Part 2 (Seventh edition);
— definitions were added to include major terms and definitions of other parts of ISO/IEC 29155;
— abbreviations of names of benchmarking activities were added;
— minor editorial improvements were made to increase understandability and readability throughout
the document; and
— Clause 5 of the first edition was restructured to Clauses 5 to 9.
iv © ISO/IEC 2017 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 29155-1:2017(E)
Introduction
Benchmarking consists of comparing “objects of interest” to each other, or against a benchmark, to
evaluate characteristic(s). In the context of the ISO/IEC 29155 series, the “object of interest” is the
performance of information technology (IT) projects, and the characteristic is a particular aspect of an
IT project such as productivity.
Benchmarking is one of the fastest-growing techniques in the area of IT project management. Instances
of IT project performance benchmarking are initiated and conducted for various reasons. Among the
most common reasons are:
— the need to compare project productivity between similar industries;
— the need to compare productivity between different project types and technologies;
— the need to find the most effective targets for IT development process improvement;
— the need to compare productivity between different suppliers;
— the need to improve project management maturity;
— the need to improve project estimation capability.
Much has been written regarding the trials of establishing IT project performance benchmarking,
and statistics bear witness to the high failure rate of measurement and benchmarking programs. The
most likely causes for failure have been disappointment in the benchmarking outcomes due to a lack
of alignment between the selected measurements and business goals, and the misunderstanding of
project level measurements in relation to program and portfolio management levels. When there is no
alignment between executed measurements and provided outcomes, unnecessary effort is required
from the IT project teams collecting the project data. This results in decreased motivation to continue
and institutionalize benchmarking.
As is shown in Figure 1, the ISO/IEC 29155 series contains multiple parts:
— Part 1 provides the overall framework model for IT project performance benchmarking. It consists
of activities and components that are necessary to successfully identify, define, select, apply, and
improve benchmarking. It also provides definitions for IT project performance benchmarking terms;
— Part 2 describes the required tasks in individual benchmarking activities that are necessary to
execute various activities to conduct and/or support successful benchmarking in an organization;
— Part 3 provides general requirements and guidance for reporting processes and contents of typical
reports;
— Part 4 provides general requirements and guidance for the activities to collect data of IT project to
be entered into and maintained in a benchmarking repository.
It is possible that further parts follow in the future.
This document is intended to provide a framework about issues and considerations for data selection
and comparison in IT project performance benchmarking.
The starting point for this document and the ISO/IEC 29155 series was the concept outlined by the draft
ISBSG (International Software Benchmarking Standards Group) benchmarking standard. IT project
performance benchmarking is a combination of several different advanced technologies and practices
in the area of quantitative analysis and management. Thus, the framework introduced in this document
can be built on the basis of various standardized key technologies such as:
— project management (e.g. PMBOK Guide and ISO 10006);
— systems and software measurements (e.g. ISO/IEC/IEEE 15939);
© ISO/IEC 2017 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 29155-1:2017(E)
— software life cycle process (e.g. ISO/IEC 12207);
— system life cycle process (e.g. ISO/IEC/IEEE 15288);
— functional size measurement (e.g. ISO/IEC 14143 series and related methods);
— systems and software quality evaluations (e.g. the ISO/IEC 25000 family and ISO/IEC 9126 series).
This document is designed to conform to the concepts of ISO/IEC 12207 (Software life cycle processes),
ISO/IEC/IEEE 15288 (Systems life cycle processes), the ISO/IEC 14143 series (Functional size
measurement), the ISO/IEC 15504 series and the ISO/IEC 33000 family (Process assessment), ISO/IEC/
TR 12182 (Categorization of systems and software products), or ISO/IEC 14764 (Software life cycle
processes — Maintenance).
Figure 1 — IT project performance benchmarking framework overview
vi © ISO/IEC 2017 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 29155-1:2017(E)
Systems and software engineering — Information
technology project performance benchmarking
framework —
Part 1:
Concepts and definitions
1 Scope
This document identifies a framework for information technology (IT) project performance
benchmarking (e.g. development or maintenance productivity) and related aspects (e.g. data collection
and software classification).
The framework consists of activities and components that are necessary to successfully identify,
define, select, apply, and improve benchmarking for IT project performance. It also provides definitions
for IT project performance benchmarking terms, which are also applicable to other parts of the
ISO/IEC 29155 series.
The target audience of this document are stakeholders of IT project performance benchmarking.
NOTE The following are examples of how this document can be used:
— by a benchmarking service provider who wants to align their benchmarking process to be consistent with
this document;
— by a benchmarking user (or third-party agents) for evaluating the performance of an IT project;
— by an organization internally to answer specific information needs.
This document does not cover how to organize benchmarking. It is out of the scope of this document
to prescribe the name, format, or explicit content of the documentation to be produced from the
benchmarking process.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https://www.iso.org/obp
— IEC Electropedia: available at http://www.electropedia.org/
3.1
attribute
property or characteristic of an entity that can be distinguished quantitatively or qualitatively by
human or automated means
[SOURCE: ISO/IEC/IEEE 15939:2017, 3.2]
© ISO/IEC 2017 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC 29155-1:2017(E)
3.2
benchmark
reference point against which comparisons can be made
Note 1 to entry: In the context of the ISO/IEC 29155 (all parts), IT project performance(s) is the object of
comparison.
3.3
benchmarking
activity of comparing objects of interest to each other or against a benchmark (3.2) to evaluate
characteristic(s)
Note 1 to entry: In the context of the ISO/IEC 29155 (all parts), the object of interest is IT project performance,
and the characteristic is a particular aspect of an IT project such as productivity.
3.4
benchmarking analyst
person or organization that conducts benchmarking (3.3) activities
3.5
benchmarking experience base
information store that contains the evaluation of the information products and the benchmarking (3.3)
activity as well as any lessons learned during benchmarking and analysis
Note 1 to entry: ISO/IEC/IEEE 15939:2007 defines “information product” as “one or more indicators and their
associated interpretations that address an information need”. For example, information product can include
templates, graphs, statistical algorithms, and interpretation guidelines.
3.6
benchmarking method
logical sequence of general steps to describe the process of comparing one or more attributes (3.1)
against a reference attribute with respect to a specified scale
3.7
benchmarking report
document showing the results of an instance of benchmarking (3.15)
Note 1 to entry: Document usually consists of various formats (e.g. textual descriptions, numeric values,
statistical charts and tables), and is exchanged via various media (e.g. electronic documents, electronic dataset,
printed documents, and embedded data within specific computer software).
3.8
benchmarking repository
organized and persistent data storage which is designated for benchmarking (3.3)
3.9
benchmarking user
person or organization that utilizes the outcome of benchmarking (3.3)
3.10
core report
document for providing descriptions of the process and outcomes of the benchmarking (3.3) activity
Note 1 to entry: Two kinds of core reports (executive summary and detailed report) are often produced for
reporting results of an instance of benchmarking activity.
3.11
data element
smallest unit of data of an IT project
Note 1 to entry: Data element is usually implemented to be a data cell in a benchmarking repository and/or an “IT
project data”,
2 © ISO/IEC 2017 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 29155-1:2017(E)
Note 2 to entry: Data element is defined by a responsible person who plans data collection or benchmarking.
3.12
data record
defined group of related data elements, in which all the necessary data elements are included to
represent attributes (3.1) of interest
Note 1 to entry: One data record usually corresponds to a specific IT project with in IT project performance
benchmarking framework.
3.13
data submitter
person or organization that provides IT project (3.16) data to be included into a benchmarking
repository (3.8)
3.14
explanatory report
document attached to a product and which provides complementary information in order to assist
understanding and avoid inappropriate usage of the product
Note 1 to entry: Examples of an explanatory report are data element definitions, data demographics, data source
information which are attached to benchmarking repositories or benchmarks.
Note 2 to entry: Examples of the product are benchmarking repository, benchmark(s), or software tools to
support benchmarking activities.
3.15
instance of benchmarking
set of operations, described specifically, used in the execution of a particular benchmarking (3.3)
according to a given method
3.16
IT project
information technology project
temporary endeavor undertaken to create or change a unique information technology product, system,
or service
Note 1 to entry: The PMBOK Guide defines “project” as “a temporary endeavor undertaken to create a unique
product, service or result”. “IT project” is a specified subset of projects.
3.17
IT project dataset
classified group of data records, into which collected data records are selected by pre-defined criteria
Note 1 to entry: Classification criteria for an IT project dataset can be based on the information needs of the
owner and/or users of data.
3.18
project performance
derived measure that gives an indication of some attribute (3.1) associated with how well, how quickly,
how effectively or how efficiently a project is carried out
3.19
repository owner
person or organization that owns and maintains a benchmarking repository
Note 1 to entry: Also called repository administrator.
© ISO/IEC 2017 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO/IEC 29155-1:2017(E)
3.20
task
smallest unit of work subject to management accountability; a well-defined work assignment for one or
more project members
Note 1 to entry: Related tasks are usually grouped to form activities.
[SOURCE: IEEE 829-2008 IEEE Standard for Software and System Test Documentation, 3.1.38]
4 Abbreviated terms
CBa “conduct benchmarking” activity
IBa “issue benchmarks” activity
IT information technology
MBLa “manage benchmarking business level” activity
MPa “measure IT project” activity
MPLa “manage benchmarking program level” activity
MRa “maintain repository” activity
PIa “provide instruments” activity
SDa “submit data” activity
URa “utilize benchmarking results” activity
5 Overview of the framework
5.1 General
This clause presents an overview of the IT project performance benchmarking framework. The
objective is to orient the users of this document so that they can apply benchmarking properly within
context.
5.2 Concepts of IT project performance benchmarking
5.2.1 Overall framework
This clause outlines the overall framework of IT project performance benchmarking.
As shown in Figure 2, the framework consists of the following categories of components:
— [C1] “Core benchmarking activities”, which collectively illustrate an instance of benchmarking;
— [C2] “Supporting activities”, which provides the base and instruments for benchmarking information
for use in an instance of benchmarking;
— [C3] “Benchmarking instruments”, used by stakeholders to conduct an instance of benchmarking;
— [C4] “Benchmarking information base”, which contains data for use during an instance of
benchmarking;
— [C5] “External reference base”, which provides alternative or additional external data (e.g. repository
and/or benchmarks) for an instance of benchmarking;
4 © ISO/IEC 2017 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 29155-1:2017(E)
— [C6] “Benchmarking experience base”, which contains knowledge and lessons learned from present
and/or past benchmarking experiences.
Key
activity
benchmark
organized data store
generic data store
© ISO/IEC 2017 – All rights reserved 5
---------------------- Page: 11 ----------------------
ISO/IEC 29155-1:2017(E)
tool
guide
method
component category
data flow
utility flow
Figure 2 — IT project performance benchmarking framework
5.2.2 Core benchmarking activities
The core benchmarking activities mainly address information needs of the benchmarking users. Each
activity has one or more processes to initiate, plan, execute, review and improve the activity. The
activities can be grouped as follows:
— The “conduct benchmarking” activity, which initiate and execute an instance of benchmarking [CBa];
— The “utilize benchmarking results” activity, which utilizes benchmarking results for various
business purposes [URa].
There are two approaches for conducting benchmarking:
— extracting benchmark from benchmarking repository and comparing it to data of specific IT
project(s);
— referring a benchmark (internal benchmarks or external benchmarks) and comparing it to data of
specific IT project(s).
NOTE Examples of the usage are estimating, comparing, evaluating and managing performance of a specific
IT project.
5.2.3 Supporting activities
The supporting activities provide the framework to assist benchmarking users. Each activity has one or
more processes to initiate, plan, execute, review and improve the activity. The activities can be grouped
as follows:
— the “measure IT project” activity, which measures IT project artifacts and maintains data [MPa];
— the “submit data” activity, which selects and provides data of IT projects to be included into the
benchmarking repository [SDa];
— the “maintain repository” activity, which accepts, verifies, and stores data of IT projects into the
benchmarking repository, and later manages the benchmarking repository [MRa];
— the “issue benchmarks” activity, which analyzes data of IT projects within the benchmarking
repository and provides internal benchmarks [IBa];
6 © ISO/IEC 2017 – All rights reserved
---------------------- Page: 12 ----------------------
ISO/IEC 29155-1:2017(E)
— the “provide instruments” activity, which develops and provides tools, methods, and guides to
support every activity in the benchmarking framework [PIa].
In this framework, the activity of measuring IT project can be planned and conducted as an existing
routine task of a project, or be done as an ad hoc task in support of an instance of benchmarking.
The activity of submitting data usually includes tasks to select and review data before submission.
The management of a benchmarking repository usually includes different tasks for different purposes,
e.g. to assure data quality, control data security, ensure data anonymity (i.e. protect privacy of data
submitter), back up data, etc.
The activity of issuing internal benchmarks provides pre-determined benchmarks which are authorized
and shared by benchmarking users.
5.2.4 Benchmarking information base
Three types of components exist in this category:
— the benchmarking repository, which is a data store to maintain reliable data of IT projects used to
produce a benchmark;
— the IT project data, which is a data store to maintain data related to various IT projects;
— the internal benchmarks, which provide a group of commonly usable and authorized pre-determined
benchmarks.
The benchmarking repository is the organized and persistent data store to maintain verified data
related to IT projects. Data in the benchmarking repository are extracted in order to derive a benchmark
in an instance of benchmarking.
The IT project data, in Figure 2, is the data store in which data related to various IT project are stored.
This data store in this framework is a conceptual collection of all available information, and it is possible
that it is neither organized nor persistent. Data are not always stored in electronic storage in systematic
forms. Data in IT project data within this framework can be:
— collected in a variety of formats for different business purposes such as project tracking, time
recording for payment, its existence;
— stored in a variety of media including electronic or paper copy;
— independently maintained by individual IT project teams.
Some measured data can be temporarily stored for a specific purpose e.g. project management, quality
management, etc. Such data may be deleted when the purpose has been achieved or data have become
obsolete.
The third component of benchmarking information base is the internal benchmark. An internal
benchmark can be:
— a pre-extracted dataset;
— a set of data extraction conditions; or
— documented results of an instance of benchmarking.
Such a pre-determined benchmark is created by analyzing a specific benchmarking repository and
usually issued as white papers, data books, technical reports, study papers, research papers, academic
articles, etc.
A pre-determined benchmark can only be an internal benchmark if it is authorized by the
benchmarking user.
© ISO/IEC 2017 – All rights reserved 7
---------------------- Page: 13 ----------------------
ISO/IEC 29155-1:2017(E)
Created internal benchmarks can be:
— published in public (across organizations);
— distributed within an organization; or
— built-in into specific tools.
Internal benchmarks enable benchmarking users to prepare reliable benchmarks without needing to
extract benchmarks from the benchmarking repository.
5.2.5 Benchmarking instruments
Three types of components exist in benchmarking instruments category:
— the tools, which provide means to support stakeholders;
— the methods, which provide procedures;
— the guides, which provide instructive information.
The tools can support various activities, e.g. to measure, collect, maintain, extract and analyze data, as
well as to report results. These tools are not necessarily automated or computer-supported.
A method in this framework is a pre-defined procedure to conduct activities. Examples of methods are
statistical analyses which are applied and utilized to conduct an instance of benchmarking.
Guides give instructive information for stakeholders to establish and operate within benchmarking
framework. Guides can be directives, implemented as textbooks, handbooks, comment boxes, or
tooltips, for example.
5.2.6 External reference base
Two types of components exist in the external reference base category:
— the external repository, which is a repository maintained by an external organization;
— the external benchmarks, which are issued in some external organizations by analyzing external
repositories.
An external repository is the dataset which is maintained as a repository in ex
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.