ISO/IEC 33003:2015
(Main)Information technology — Process assessment — Requirements for process measurement frameworks
Information technology — Process assessment — Requirements for process measurement frameworks
ISO/IEC 33003:2015 sets out the requirements for process measurement frameworks for use in process assessment. The requirements defined in ISO/IEC 33003:2015 form a structure which a) establish the requirements for process measurement frameworks in the context of process assessment, b) establish the requirements for the validation of process measurement frameworks for use in process assessment, and c) establish requirements that are applicable to any process measurement frameworks to develop composite measures across domains. ISO/IEC 33003:2015 is applicable to the development of process measurement frameworks for any process quality characteristic across all application domains.
Technologies de l'information — Évaluation du processus — Exigences relatives au cadres de mesure du processus
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 33003
Second edition
2015-03-01
Information technology — Process
assessment — Requirements for
process measurement frameworks
Technologies de l’information — Évaluation du processus —
Exigences relatives au cadres de mesure du processus
Reference number
ISO/IEC 33003:2015(E)
©
ISO/IEC 2015
---------------------- Page: 1 ----------------------
ISO/IEC 33003:2015(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2015 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 33003:2015(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Requirements for process measurement frameworks . 4
4.1 Conceptualization . 4
4.1.1 Requirements . 4
4.1.2 Guidance. 4
4.2 Construct definition . 4
4.2.1 Requirements . 4
4.2.2 Guidance. 5
4.3 Operationalization . 5
4.3.1 Requirements . 5
4.3.2 Guidance. 5
4.4 Construct specification examination . 5
4.4.1 Requirements . 5
4.4.2 Guidance. 5
4.5 Rating process attributes . 6
4.5.1 Requirements . 6
4.5.2 Guidance. 6
4.6 Aggregation . 7
4.6.1 Requirements . 7
4.6.2 Guidance. 7
4.7 Sensitivity analysis . 8
4.7.1 Requirements . 8
4.7.2 Guidance. 8
5 Requirements for the validation of process measurement frameworks .8
5.1 Requirements . 8
5.2 Guidance . 8
5.2.1 Reliability . 9
5.2.2 Construct validity . 9
5.2.3 Construct specification . 9
6 Verifying conformity of process measurement frameworks . 9
Annex A (informative) A terminology map.11
Annex B (informative) Construct specification: Reflective or formative .13
Annex C (informative) Some statistical validation methods .15
Annex D (informative) Methods for implementing the requirements for process
measurement frameworks .18
Bibliography .20
© ISO/IEC 2015 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 33003:2015(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Details of any patent rights identified during the development of the document will be in the Introduction
and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 7, Software
and systems engineering.
iv © ISO/IEC 2015 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 33003:2015(E)
Introduction
This International Standard provides requirements for process measurement frameworks that support
and enable the assessment of process quality characteristics, from conceptualization to empirical
validation. In process measurement frameworks, measurement of a process quality characteristic
produces a composite measure (e.g. process capability levels of ordinal scale in ISO/IEC 33020). Examples
of process quality characteristics that are constructs (theoretical concepts) include process capability,
process security, process agility, and process safety. The main users of this International Standard are
developers of process measurement frameworks and process assessment models. Conformity to this
International Standard ensures that any process measurement framework is developed with reliable
structures or elements which will generate quality composite measures.
This International Standard is part of a set of International Standards designed to provide a consistent and
coherent framework for the assessment of process quality characteristics, based on objective evidence
resulting from implementation of the processes. The framework for assessment covers processes
employed in the development, maintenance, and use of systems across the information technology
domain and those employed in the design, transition, delivery, and improvement of services. The set of
International Standards, as a whole, addresses process quality characteristics of any type. Results of
assessment can be applied for improving process performance, or for identifying and addressing risks
associated with application of processes.
This International Standard provides requirements for the development of process measurement
frameworks, such as ISO/IEC 33020. These can then be used to define process assessment models,
conformant to ISO/IEC 33004, that can be employed for process assessments conformant with
ISO/IEC 33002. The overall architecture and content of the series is described in ISO/IEC 33001.
Several International Standards in the ISO/IEC 330xx family of standards for process assessment are intended
to replace and extend parts of the ISO/IEC 15504 series of Standards. ISO/IEC 33001, Annex A provides a
detailed record of the relationship between the ISO/IEC 330xx family and the ISO/IEC 15504 series.
© ISO/IEC 2015 – All rights reserved v
---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 33003:2015(E)
Information technology — Process assessment —
Requirements for process measurement frameworks
1 Scope
This International Standard sets out the requirements for process measurement frameworks for use in
process assessment. The requirements defined in this International Standard form a structure which
a) establish the requirements for process measurement frameworks in the context of process assessment,
b) establish the requirements for the validation of process measurement frameworks for use in process
assessment, and
c) establish requirements that are applicable to any process measurement frameworks to develop
composite measures across domains.
This International Standard is applicable to the development of process measurement frameworks for
any process quality characteristic across all application domains.
Annex A presents a map of terminologies used in this International Standard. Annex B provides an
explanation of construct specifications. Annex C reviews statistical validation methods. Annex D
provides some methods including references that can be utilized in implementing the requirements for
process measurement frameworks. These Annexes will be moved to a guide for constructing process
measurement frameworks to be developed as part of the set of International Standards.
NOTE ISO/IEC 33020 is a process measurement framework for assessment of process capability based on
this International Standard.
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies
ISO/IEC 15939:2007, Systems and software engineering — Measurement process
ISO/IEC 33001:2015, Information technology — Process assessment — Concepts and terminology
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 33001, ISO/IEC 15939,
and the following apply:
3.1
aggregation method
method that combines a set of measurement values to create a composite value
Note 1 to entry: Aggregation methods are based on compensatory or non-compensatory models.
© ISO/IEC 2015 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO/IEC 33003:2015(E)
3.2
compensatory model
MCDM model in which a composite measure is composed of individually weighted terms and where
criteria (also refer to attribute terms) with a high value can compensate for those of a low value in
proportion to each weight
Note 1 to entry: A compensatory model suggests that improving the more important measures (those with a
higher weighting) is more likely to increase or improve the overall composite value than improving the less
important ones. This model assumes that the weight (influence level) of criteria remains the same regardless of
the measured level of the criteria.
3.3
composite measure
variable derived from a set of operations of a construct’s multi-item measures defined according to
construct specification (either reflective or formative) that is the way in which the latent variable
representing the construct of interest is linked to its measures
3.4
composite value
value from a composite measure
Note 1 to entry: A composite value can be from an ordinal, interval, or ratio scale.
3.5
construct
concept such as the abstract idea, image, underlying theme, or subject matter that one wishes to measure
using process assessments
Note 1 to entry: In process measurement frameworks, constructs (also refers to latent constructs) are theoretical
concepts such as the process quality characteristics and process attributes.
Note 2 to entry: The meaning that one assigns to a construct is called theoretical definition, which should explain
its meaning, as well as discuss its distinct dimensions (facets).
3.6
dimension
distinct components that a multidimensional construct encompasses
3.7
formative construct
construct that is formed from its observed measures in the relationship between a construct and its measures
Note 1 to entry: The construct is a consequence of its measures and each measure is a determinant of the construct.
3.8
latent variable
variable representing a unidimensional construct
Note 1 to entry: There should be a separate latent variable for each dimension of a construct and a minimum of
one measure per latent variable.
3.9
MCDM
Multiple-Criteria Decision Making or Multi-Attribute Decision Making
making preference decisions (e.g., evaluation, prioritization, and selection) of available alternatives
characterized by multiple criteria
Note 1 to entry: A criterion in MCDM corresponds to measure.
Note 2 to entry: An MCDM with one alternative is the same as the development of a composite measure.
2 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 33003:2015(E)
3.10
measurement model
the implicit or explicit relationship between a latent variable and its (multi-item) measures
Note 1 to entry: The relationship between a reflective (formative) construct and its measure(s) is called a reflective
(formative) measurement model.
3.11
multidimensional construct
construct that consists of a number of unidimensional constructs.
Note 1 to entry: Each dimension of a multidimensional construct is called unidimensional and is represented by
one latent variable. Each dimension can have multiple measures. In a multidimensional construct, for example,
the meaning of capability when it is defined as the common factor underlying its process attributes is different
from the case when capability is defined as a simple sum of its process attributes. The former is called a reflective
multidimensional construct and the latter is formative. A multidimensional construct can span an indeterminate
number of levels.
3.12
non-compensatory model
MCDM model that does not allow criteria to compensate for each other in proportion to their weights
Note 1 to entry: Strongly positive or negative terms influence the overall composite value disproportionately,
although the weight stays the same. There are various non-compensatory models depending on the evaluation
policy, the purpose of the composite measure, and/or the measurement scale.
3.13
reflective construct
construct that is viewed as the cause of measures in the relationship between a construct and its measures
Note 1 to entry: Reflective construct is an underlying factor of the variation of its measures.
3.14
scale
ordered set of values, continuous, or discrete, or a set of categories to which the attribute is mapped
Note 1 to entry: The type of scale depends on the nature of the relationship between values on the scale. Four
types of scales are commonly defined:
Nominal ─ the measurement values are categorical. For example, the classification of defects by their type does
not imply order among the categories.
Ordinal ─ the measurement values are rankings. For example, the assignment of defects to a severity level is a ranking.
Interval ─ the measurement values have equal distances corresponding to equal quantities of the attribute. For
example, cyclomatic complexity has the minimum value of one, but each increment represents an additional path.
The value of zero is not possible.
Ratio ─ the measurement values have equal distances corresponding to equal quantities of the attribute where
the value of zero corresponds to none of the attribute. For example, the size of a software component in terms
of LOC is a ratio scale because the value of zero corresponds to no lines of code and each additional increments
represents equal amounts of code.
[SOURCE: ISO/IEC 15939:2007]
3.15
unidimensionality
existence of a single trait or construct underlying a set of measures
© ISO/IEC 2015 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO/IEC 33003:2015(E)
4 Requirements for process measurement frameworks
This clause defines the requirements for developing process measurement frameworks. Guidance in this
International Standard is limited to providing a better understanding of these requirements. Figure A.1
provides a mapping of the relationships between some terms used in this Clause.
NOTE Guidance on achieving conformance to these requirements, including examples and methods, will be
provided in a guide for constructing process measurement frameworks to be developed as part of the set of Standards.
4.1 Conceptualization
4.1.1 Requirements
a) A measurement framework shall identify and address a single process quality characteristic;
b) A process quality characteristic in a process measurement framework shall be defined on the basis
of a multidimensional construct;
c) A process quality characteristic in a process measurement framework shall be defined as a set of
process attributes;
d) Each process attribute shall define a property of the process quality characteristic;
e) Each process attribute that is not directly measurable shall be considered as a construct;
f) Process attributes in a process measurement framework shall be defined as either reflective or
formative.
g) The measurement framework shall document the policies and assumptions underlying its use
and application;
4.1.2 Guidance
The process of identifying and clarifying concepts is called conceptualization. A concept is an idea
or image that unites phenomena of interest (e.g., traits, behaviour traits) under a single term. It is a
summarizing device to replace a list of specific traits. Most process quality characteristics (e.g., process
capability) are not observable but are theoretical concepts called constructs.
The composite measures (e.g., process capability level) used in process measurement frameworks are
defined on the basis of a construct composed of process attributes. A measurement framework may be
structured into a series of levels of achievement.
When a process attribute is not directly measurable, it may also be defined as a construct. The set of
process attributes for any construct may be either reflective or formative.
Participation of experts and stakeholders can increase the validity of the process quality characteristic
and its process attributes; aspects of validity are discussed in C.3.
A multidimensional construct can be depicted with a path diagram including a set of dimensions and
their relationships. Use of a path diagram improves the understandability of model scope and structures.
4.2 Construct definition
4.2.1 Requirements
a) The construct definition shall define the meaning of the process quality characteristic and its
process attributes in a process measurement framework;
b) The construct definition shall clarify the specification of the process quality characteristic and its
process attributes as dimensions;
4 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 33003:2015(E)
c) The construct definition shall provide a guide for the operationalization of the process quality
characteristic and its process attributes;
d) The construct definition shall state the scales of composite measures such as categorical (e.g., a
series of ordinal values such as capability level) or numeric;
e) At least one of the process attributes shall comprise the achievement of the defined process purpose
and process outcomes for the process; this is termed the process performance attribute;
4.2.2 Guidance
Although a process quality characteristic or process attribute should convey an intuitive understanding
of what it represents, interpretation may vary according to the observer. Thus, a definition is required
to explain and provide the meaning of a construct. This is called the construct definition.
Clarification of a construct implies that for example the definition of the process quality characteristic
as specified super-ordinate fully covers all of process attributes on the basis of construct specification,
where process attributes as sub-ordinates are its distinct dimensions. A latent variable can be assigned to a
unidimensional construct in the model. Statistical methods related to dimensionality are introduced in C.1.
4.3 Operationalization
4.3.1 Requirements
a) All process attributes shall be defined according to their construct specification;
b) Achievement of process attributes shall be verifiable through objective evidence.
4.3.2 Guidance
When a process attribute is directly observable through formal assessments, self-reports, surveys
(including questionnaires and interviews), observations, or other empirical means, it is a base measure
that is functionally independent of other measures. If a process attribute is measured with its several
sub-constructs or measures, it can be considered as a construct. Four or more base measures are
recommended to measure a construct and perform a set of statistical tests (including model validation
and construct specifications) in reflective specification.
NOTE Refer to Clause 6.3.4 of ISO/IEC 33004 for assessment indicators that are utilized for process
attribute rating.
4.4 Construct specification examination
4.4.1 Requirements
Construct specifications of the process quality characteristic and its associated process attributes shall
be examined through operationalization and with rationale.
4.4.2 Guidance
There are two kinds of construct specifications that refer to the way in which the latent variable
representing the construct is linked to its measures (i.e., the relationship between a unidimensional
construct and its measures): reflective and formative measurement models. A process quality
characteristic or process attribute can be viewed either as underlying factors or indices produced by
observed measures. The former is referred to as reflective (effect) constructs or reflective measurement
models, and the latter formative (causal) constructs or formative measurement models.
The objective of a reflective measurement model is to measure a single property by using multiple
measures, whereas a formative model attempts to summarize multiple properties with a single
© ISO/IEC 2015 – All rights reserved 5
---------------------- Page: 10 ----------------------
ISO/IEC 33003:2015(E)
composite value. In Annex B, these two specifications can be represented as Figure B.1 (a) and Figure B.1
(b), respectively.
Decision rules to examine reflective or formative, construct specification, are summarized in Table 1.
These decision rules can be applied to the process quality characteristic and its associated process
attributes. They can be assessed a priori statistical validation of construct specification. Annex B
provides the construct specification in detail.
Table 1 — Decision rules to examine reflective or formative measurement model
Decision rule Reflective measurement model Formative measurement model
• Measures are manifestations of the con- • Measures are defining characteristics (aspects)
struct. of the construct.
• Measures share a common theme. • Measures need not share a common theme.
• Measures should be interchangeable. • Measures need not be interchangeable.
Characteristics of
• Measures should have the same or similar • Measures need not have the same or similar
measures of the
content. content.
construct
• Excluding a measure should not alter the • Excluding a measure may alter the conceptual
conceptual domain of the construct. domain of the construct.
• Measures are expected to co-vary with one • Measures need not co-vary with one another.
other.
• The direction of causality is from the con- • The direction of causality is from measures to
Direction of causality
struct to its multi-item measures. the construct.
between construct
• Changes in a measure should not cause to • Changes in the construct should not cause
and measures
changes in the construct. changes in the measures.
In some instances, the relationships depicted in Figure B.1 (Annex B) can have a higher-order level,
i.e., conceptual definitions of constructs are often specified at a more abstract level, which sometimes
include multiple reflective and/or formative first-order dimensions. The definition of a higher-order
model should be theory-driven in a reflective measure model. Statistical analyses should be used to
support or validate the definition.
4.5 Rating process attributes
4.5
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.