ISO/IEC 33003:2015

Information technology - Process assessment - Requirements for process measurement frameworks

ISO/IEC 33003:2015 sets out the requirements for process measurement frameworks for use in process assessment. The requirements defined in ISO/IEC 33003:2015 form a structure which a) establish the requirements for process measurement frameworks in the context of process assessment, b) establish the requirements for the validation of process measurement frameworks for use in process assessment, and c) establish requirements that are applicable to any process measurement frameworks to develop composite measures across domains. ISO/IEC 33003:2015 is applicable to the development of process measurement frameworks for any process quality characteristic across all application domains.

Technologies de l'information — Évaluation du processus — Exigences relatives au cadres de mesure du processus

General Information

• Status: Published
• Publication Date: 26-Feb-2015
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 10 - Process assessment
• Current Stage: 9060 - Close of review
• Completion Date: 02-Dec-2030

Relations

• Revises: ISO/IEC TR 15504-7:2008 - Information technology - Process assessment - Part 7: Assessment of organizational maturity
• Effective Date: 05-Nov-2015
• Revises: ISO/IEC 15504-2:2003 - Information technology - Process assessment - Part 2: Performing an assessment
• Effective Date: 04-Nov-2015

Overview

ISO/IEC 33003:2015 - Information technology - Process assessment - Requirements for process measurement frameworks - defines the requirements for developing and validating process measurement frameworks used in process assessment. The standard provides a structured approach from conceptualization to empirical validation so frameworks produce reliable composite measures (for example process capability levels). ISO/IEC 33003:2015 is applicable to any process quality characteristic across all application domains and supports consistent process assessment results.

Key topics and technical requirements

ISO/IEC 33003:2015 specifies requirements and guidance in these core areas:

• Conceptualization: define the construct (process quality characteristic) and its theoretical meaning and dimensions.
• Construct definition and specification: identify whether constructs are reflective or formative, and document latent variables and measures.
• Operationalization: translate constructs into measurable items and criteria for assessment.
• Construct specification examination: assess the internal consistency and appropriateness of the construct-to-measure relationships.
• Rating process attributes: define how individual process attributes are evaluated (scales, evidence, and rating rules).
• Aggregation methods: requirements for combining measures into composite values using compensatory or non‑compensatory models.
• Sensitivity analysis: evaluate how changes in measures or weights affect the composite measure.
• Validation of measurement frameworks: requirements for empirical validation including reliability, construct validity, and verification procedures.
• Verification of conformity: procedures for confirming a measurement framework meets the standard’s requirements.

Annexes provide supporting material: a terminology map, construct specification guidance (reflective vs formative), statistical validation methods, and implementation methods and references.

Practical applications and users

ISO/IEC 33003:2015 is intended for practical use by:

• Developers of process measurement frameworks (e.g., ISO/IEC 33020-style frameworks).
• Process assessment model designers and conformity assessors who need validated composite measures.
• Quality managers, auditors, and process improvement teams seeking objective process capability, security, safety, or agility measures.
• Tool vendors implementing assessment workflows and scoring algorithms that require defensible aggregation and sensitivity methods.

Adoption helps organizations produce consistent, validated assessments for process improvement, benchmarking, risk identification, and decision‑making.

Related standards

• ISO/IEC 33001:2015 (Concepts and terminology for process assessment)
• ISO/IEC 15939:2007 (Measurement process)
• ISO/IEC 33020 (example framework for process capability built on ISO/IEC 33003)

Keywords: ISO/IEC 33003:2015, process measurement framework, process assessment, composite measures, construct specification, aggregation, validation, process capability.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 33003
Second edition
2015-03-01
Information technology — Process
assessment — Requirements for
process measurement frameworks
Technologies de l’information — Évaluation du processus —
Exigences relatives au cadres de mesure du processus
Reference number
©
ISO/IEC 2015
© ISO/IEC 2015
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail [e-mail]
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2015 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Requirements for process measurement frameworks . 4
4.1 Conceptualization . 4
4.1.1 Requirements . 4
4.1.2 Guidance. 4
4.2 Construct definition . 4
4.2.1 Requirements . 4
4.2.2 Guidance. 5
4.3 Operationalization . 5
4.3.1 Requirements . 5
4.3.2 Guidance. 5
4.4 Construct specification examination . 5
4.4.1 Requirements . 5
4.4.2 Guidance. 5
4.5 Rating process attributes . 6
4.5.1 Requirements . 6
4.5.2 Guidance. 6
4.6 Aggregation . 7
4.6.1 Requirements . 7
4.6.2 Guidance. 7
4.7 Sensitivity analysis . 8
4.7.1 Requirements . 8
4.7.2 Guidance. 8
5 Requirements for the validation of process measurement frameworks .8
5.1 Requirements . 8
5.2 Guidance . 8
5.2.1 Reliability . 9
5.2.2 Construct validity . 9
5.2.3 Construct specification . 9
6 Verifying conformity of process measurement frameworks . 9
Annex A (informative)  A terminology map.11
Annex B (informative) Construct specification: Reflective or formative .13
Annex C (informative) Some statistical validation methods .15
Annex D (informative) Methods for implementing the requirements for process
measurement frameworks .18
Bibliography .20
© ISO/IEC 2015 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Details of any patent rights identified during the development of the document will be in the Introduction
and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 7, Software
and systems engineering.
iv © ISO/IEC 2015 – All rights reserved

Introduction
This International Standard provides requirements for process measurement frameworks that support
and enable the assessment of process quality characteristics, from conceptualization to empirical
validation. In process measurement frameworks, measurement of a process quality characteristic
produces a composite measure (e.g. process capability levels of ordinal scale in ISO/IEC 33020). Examples
of process quality characteristics that are constructs (theoretical concepts) include process capability,
process security, process agility, and process safety. The main users of this International Standard are
developers of process measurement frameworks and process assessment models. Conformity to this
International Standard ensures that any process measurement framework is developed with reliable
structures or elements which will generate quality composite measures.
This International Standard is part of a set of International Standards designed to provide a consistent and
coherent framework for the assessment of process quality characteristics, based on objective evidence
resulting from implementation of the processes. The framework for assessment covers processes
employed in the development, maintenance, and use of systems across the information technology
domain and those employed in the design, transition, delivery, and improvement of services. The set of
International Standards, as a whole, addresses process quality characteristics of any type. Results of
assessment can be applied for improving process performance, or for identifying and addressing risks
associated with application of processes.
This International Standard provides requirements for the development of process measurement
frameworks, such as ISO/IEC 33020. These can then be used to define process assessment models,
conformant to ISO/IEC 33004, that can be employed for process assessments conformant with
ISO/IEC 33002. The overall architecture and content of the series is described in ISO/IEC 33001.
Several International Standards in the ISO/IEC 330xx family of standards for process assessment are intended
to replace and extend parts of the ISO/IEC 15504 series of Standards. ISO/IEC 33001, Annex A provides a
detailed record of the relationship between the ISO/IEC 330xx family and the ISO/IEC 15504 series.
© ISO/IEC 2015 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 33003:2015(E)
Information technology — Process assessment —
Requirements for process measurement frameworks
1 Scope
This International Standard sets out the requirements for process measurement frameworks for use in
process assessment. The requirements defined in this International Standard form a structure which
a) establish the requirements for process measurement frameworks in the context of process assessment,
b) establish the requirements for the validation of process measurement frameworks for use in process
assessment, and
c) establish requirements that are applicable to any process measurement frameworks to develop
composite measures across domains.
This International Standard is applicable to the development of process measurement frameworks for
any process quality characteristic across all application domains.
Annex A presents a map of terminologies used in this International Standard. Annex B provides an
explanation of construct specifications. Annex C reviews statistical validation methods. Annex D
provides some methods including references that can be utilized in implementing the requirements for
process measurement frameworks. These Annexes will be moved to a guide for constructing process
measurement frameworks to be developed as part of the set of International Standards.
NOTE ISO/IEC 33020 is a process measurement framework for assessment of process capability based on
this International Standard.
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies
ISO/IEC 15939:2007, Systems and software engineering — Measurement process
ISO/IEC 33001:2015, Information technology — Process assessment — Concepts and terminology
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 33001, ISO/IEC 15939,
and the following apply:
3.1
aggregation method
method that combines a set of measurement values to create a composite value
Note 1 to entry: Aggregation methods are based on compensatory or non-compensatory models.
© ISO/IEC 2015 – All rights reserved 1

3.2
compensatory model
MCDM model in which a composite measure is composed of individually weighted terms and where
criteria (also refer to attribute terms) with a high value can compensate for those of a low value in
proportion to each weight
Note 1 to entry: A compensatory model suggests that improving the more important measures (those with a
higher weighting) is more likely to increase or improve the overall composite value than improving the less
important ones. This model assumes that the weight (influence level) of criteria remains the same regardless of
the measured level of the criteria.
3.3
composite measure
variable derived from a set of operations of a construct’s multi-item measures defined according to
construct specification (either reflective or formative) that is the way in which the latent variable
representing the construct of interest is linked to its measures
3.4
composite value
value from a composite measure
Note 1 to entry: A composite value can be from an ordinal, interval, or ratio scale.
3.5
construct
concept such as the abstract idea, image, underlying theme, or subject matter that one wishes to measure
using process assessments
Note 1 to entry: In process measurement frameworks, constructs (also refers to latent constructs) are theoretical
concepts such as the process quality characteristics and process attributes.
Note 2 to entry: The meaning that one assigns to a construct is called theoretical definition, which should explain
its meaning, as well as discuss its distinct dimensions (facets).
3.6
dimension
distinct components that a multidimensional construct encompasses
3.7
formative construct
construct that is formed from its observed measures in the relationship between a construct and its measures
Note 1 to entry: The construct is a consequence of its measures and each measure is a determinant of the construct.
3.8
latent variable
variable representing a unidimensional construct
Note 1 to entry: There should be a separate latent variable for each dimension of a construct and a minimum of
one measure per latent variable.
3.9
MCDM
Multiple-Criteria Decision Making or Multi-Attribute Decision Making
making preference decisions (e.g., evaluation, prioritization, and selection) of available alternatives
characterized by multiple criteria
Note 1 to entry: A criterion in MCDM corresponds to measure.
Note 2 to entry: An MCDM with one alternative is the same as the development of a composite measure.
2 © ISO/IEC 2015 – All rights reserved

3.10
measurement model
the implicit or explicit relationship between a latent variable and its (multi-item) measures
Note 1 to entry: The relationship between a reflective (formative) construct and its measure(s) is called a reflective
(formative) measurement model.
3.11
multidimensional construct
construct that consists of a number of unidimensional constructs.
Note 1 to entry: Each dimension of a multidimensional construct is called unidimensional and is represented by
one latent variable. Each dimension can have multiple measures. In a multidimensional construct, for example,
the meaning of capability when it is defined as the common factor underlying its process attributes is different
from the case when capability is defined as a simple sum of its process attributes. The former is called a reflective
multidimensional construct and the latter is formative. A multidimensional construct can span an indeterminate
number of levels.
3.12
non-compensatory model
MCDM model that does not allow criteria to compensate for each other in proportion to their weights
Note 1 to entry: Strongly positive or negative terms influence the overall composite value disproportionately,
although the weight stays the same. There are various non-compensatory models depending on the evaluation
policy, the purpose of the composite measure, and/or the measurement scale.
3.13
reflective construct
construct that is viewed as the cause of measures in the relationship between a construct and its measures
Note 1 to entry: Reflective construct is an underlying factor of the variation of its measures.
3.14
scale
ordered set of values, continuous, or discrete, or a set of categories to which the attribute is mapped
Note 1 to entry: The type of scale depends on the nature of the relationship between values on the scale. Four
types of scales are commonly defined:
Nominal ─ the measurement values are categorical. For example, the classification of defects by their type does
not imply order among the categories.
Ordinal ─ the measurement values are rankings. For example, the assignment of defects to a severity level is a ranking.
Interval ─ the measurement values have equal distances corresponding to equal quantities of the attribute. For
example, cyclomatic complexity has the minimum value of one, but each increment represents an additional path.
The value of zero is not possible.
Ratio ─ the measurement values have equal distances corresponding to equal quantities of the attribute where
the value of zero corresponds to none of the attribute. For example, the size of a software component in terms
of LOC is a ratio scale because the value of zero corresponds to no lines of code and each additional increments
represents equal amounts of code.
[SOURCE: ISO/IEC 15939:2007]
3.15
unidimensionality
existence of a single trait or construct underlying a set of measures
© ISO/IEC 2015 – All rights reserved 3

4 Requirements for process measurement frameworks
This clause defines the requirements for developing process measurement frameworks. Guidance in this
International Standard is limited to providing a better understanding of these requirements. Figure A.1
provides a mapping of the relationships between some terms used in this Clause.
NOTE Guidance on achieving conformance to these requirements, including examples and methods, will be
provided in a guide for constructing process measurement frameworks to be developed as part of the set of Standards.
4.1 Conceptualization
4.1.1 Requirements
a) A measurement framework shall identify and address a single process quality characteristic;
b) A process quality characteristic in a process measurement framework shall be defined on the basis
of a multidimensional construct;
c) A process quality characteristic in a process measurement framework shall be defined as a set of
process attributes;
d) Each process attribute shall define a property of the process quality characteristic;
e) Each process attribute that is not directly measurable shall be considered as a construct;
f) Process attributes in a process measurement framework shall be defined as either reflective or
formative.
g) The measurement framework shall document the policies and assumptions underlying its use
and application;
4.1.2 Guidance
The process of identifying and clarifying concepts is called conceptualization. A concept is an idea
or image that unites phenomena of interest (e.g., traits, behaviour traits) under a single term. It is a
summarizing device to replace a list of specific traits. Most process quality characteristics (e.g., process
capability) are not observable but are theoretical concepts called constructs.
The composite measures (e.g., process capability level) used in process measurement frameworks are
defined on the basis of a construct composed of process attributes. A measurement framework may be
structured into a series of levels of achievement.
When a process attribute is not directly measurable, it may also be defined as a construct. The set of
process attributes for any construct may be either reflective or formative.
Participation of experts and stakeholders can increase the validity of the process quality characteristic
and its process attributes; aspects of validity are discussed in C.3.
A multidimensional construct can be depicted with a path diagram including a set of dimensions and
their relationships. Use of a path diagram improves the understandability of model scope and structures.
4.2 Construct definition
4.2.1 Requirements
a) The construct definition shall define the meaning of the process quality characteristic and its
process attributes in a process measurement framework;
b) The construct definition shall clarify the specification of the process quality characteristic and its
process attributes as dimensions;
4 © ISO/IEC 2015 – All rights reserved

c) The construct definition shall provide a guide for the operationalization of the process quality
characteristic and its process attributes;
d) The construct definition shall state the scales of composite measures such as categorical (e.g., a
series of ordinal values such as capability level) or numeric;
e) At least one of the process attributes shall comprise the achievement of the defined process purpose
and process outcomes for the process; this is termed the process performance attribute;
4.2.2 Guidance
Although a process quality characteristic or process attribute should convey an intuitive understanding
of what it represents, interpretation may vary according to the observer. Thus, a definition is required
to explain and provide the meaning of a construct. This is called the construct definition.
Clarification of a construct implies that for example the definition of the process quality characteristic
as specified super-ordinate fully covers all of process attributes on the basis of construct specification,
where process attributes as sub-ordinates are its distinct dimensions. A latent variable can be assigned to a
unidimensional construct in the model. Statistical methods related to dimensionality are introduced in C.1.
4.3 Operationalization
4.3.1 Requirements
a) All process attributes shall be defined according to their construct specification;
b) Achievement of process attributes shall be verifiable through objective evidence.
4.3.2 Guidance
When a process attribute is directly observable through formal assessments, self-reports, surveys
(including questionnaires and interviews), observations, or other empirical means, it is a base measure
that is functionally independent of other measures. If a process attribute is measured with its several
sub-constructs or measures, it can be considered as a construct. Four or more base measures are
recommended to measure a construct and perform a set of statistical tests (including model validation
and construct specifications) in reflective specification.
NOTE Refer to Clause 6.3.4 of ISO/IEC 33004 for assessment indicators that are utilized for process
attribute rating.
4.4 Construct specification examination
4.4.1 Requirements
Construct specifications of the process quality characteristic and its associated process attributes shall
be examined through operationalization and with rationale.
4.4.2 Guidance
There are two kinds of construct specifications that refer to the way in which the latent variable
representing the construct is linked to its measures (i.e., the relationship between a unidimensional
construct and its measures): reflective and formative measurement models. A process quality
characteristic or process attribute can be viewed either as underlying factors or indices produced by
observed measures. The former is referred to as reflective (effect) constructs or reflective measurement
models, and the latter formative (causal) constructs or formative measurement models.
The objective of a reflective measurement model is to measure a single property by using multiple
measures, whereas a formative model attempts to summarize multiple properties with a single
© ISO/IEC 2015 – All rights reserved 5

composite value. In Annex B, these two specifications can be represented as Figure B.1 (a) and Figure B.1
(b), respectively.
Decision rules to examine reflective or formative, construct specification, are summarized in Table 1.
These decision rules can be applied to the process quality characteristic and its associated process
attributes. They can be assessed a priori statistical validation of construct specification. Annex B
provides the construct specification in detail.
Table 1 — Decision rules to examine reflective or formative measurement model
Decision rule Reflective measurement model Formative measurement model
• Measures are manifestations of the con- • Measures are defining characteristics (aspects)
struct. of the construct.
• Measures share a common theme. • Measures need not share a common theme.
• Measures should be interchangeable. • Measures need not be interchangeable.
Characteristics of
• Measures should have the same or similar • Measures need not have the same or similar
measures of the
content. content.
construct
• Excluding a measure should not alter the • Excluding a measure may alter the conceptual
conceptual domain of the construct. domain of the construct.
• Measures are expected to co-vary with one • Measures need not co-vary with one another.
other.
• The direction of causality is from the con- • The direction of causality is from measures to
Direction of causality
struct to its multi-item measures. the construct.
between construct
• Changes in a measure should not cause to • Changes in the construct should not cause
and measures
changes in the construct. changes in the measures.
In some instances, the relationships depicted in Figure B.1 (Annex B) can have a higher-order level,
i.e., conceptual definitions of constructs are often specified at a more abstract level, which sometimes
include multiple reflective and/or formative first-order dimensions. The definition of a higher-order
model should be theory-driven in a reflective measure model. Statistical analyses should be used to
support or validate the definition.
4.5 Rating process attributes
4.5.1 Requirements
a) The process attributes shall be rated;
b) A measurement scale, i.e., nominal, ordinal, interval, or ratio, shall be defined for the process attributes;
c) A measurement method shall be identified that objectively assigns a value to each process attribute.
4.5.2 Guidance
Some assessments can generate the ratings of for example a process quality characteristic or process
attributes for individual process instances assessed. On the other hand, others providing an overall
picture without ratings can simultaneously assess a set of process instances under the same context as a
process. Rating of process attributes can be based on formal assessments, self-reports, surveys (including
questionnaires and interviews), observations, or other empirical means. Thus, a measurement scale for
rating base measures should be consistent with the granularity of assessment. Occasionally, rating in
self-reports or surveys is on the base of perception rather than objective evidence. Rating scale for the
process quality characteristic and its process attributes should be addressed with rationale, consistent
with the construct specification (refer to 4.3).
A well-established documented assessment process for rating process attributes provides credible
measurement results. The approach to rating the process attributes shall be defined in the documented
assessment process, and may depend on the class of the assessment, based on the assessment objectives.
Thus, for this purpose, a documented assessment process will guide the process for establishing,
planning, performing and evaluating assessment under an integrated assessment scheme. If there
6 © ISO/IEC 2015 – All rights reserved

is consensus in the community, a validated documented assessment process can be adopted after
examining its conformity with measurement purposes.
4.6 Aggregation
4.6.1 Requirements
Aggregation derives a composite value or rating by combining a set of measurement values.
a) All aggregations required within the measurement framework shall be identified;
b) Aggregation methods shall be specified;
c) Aggregation methods shall be statistically valid.
d) Aggregation methods shall utilize consistent measurement scales;
e) Aggregation methods shall be consistent with the measurement framework policies and assumptions;
f) Aggregation methods shall be consistent with construct specifications.
4.6.2 Guidance
The scale of composite measure for the process quality characteristic or process attribute should be
stated in accordance with its construct specification. The number of aggregation required depends on
the structure of a multidimensional construct of process quality characteristic in 4.1. Principally, the
point of aggregation can be the hierarchical order of constructs such as process quality characteristics
and its process attributes. Each process quality characteristic level on the scale is defined in terms of the
achievements of a set of process attributes.
A MCDM with one alternative can also be regarded as the aggregation method to derive a value of
composite measure. An aggregation method may be based on compensatory or non-compensatory models
depending on the construct specification, evaluation policy, the purpose of the composite measure,
and/or the measurement scale. A formative model with no measurement error can be considered as a
compensatory type MCDM which aggregates different aspects or dimensions into a composite value.
An aggregation example related would be the combination of a set of process attribute ratings to a
level of process capability, assuming a formative specification. In a mutidimensional construct such
as the process quality characteristic and its process attributes, aggregation is used to determine a
process capability level from a set of process attribute ratings. In addition, if process attribute rating is
performed for each of multiple process instances, aggregation methods should be provided.
A rating scale of a process quality characteristic or process attribute represents the extent of its
achievement. The scale expressed as an ordinal scale can be transformed from an interval or ratio scale
to provide anchor points for the rating. For example, the rating scale may be applied to express the
extent of achievement of a process attribute for a process instance in a specific organizational context,
or to express the extent of achievement of a process attribute across multiple process instances within
the defined organizational unit scope.
Consistency in measurement scale implies that lower level transformation, from a higher measurement
level to lower level, is possible such that, (i) a ratio scale can be transformed to an interval, ordinal, or
nominal scale, (ii) an interval to an ordinal or nominal, and (iii) an ordinal to a nominal. However, the
inverse direction is not allowed.
A composite value of the reflective construct can be computed by averaging or summing the values of
measures if associated assumptions are satisfied. Those methods can also be applied to the aggregation
of sub-constructs to obtain a composite value of higher-level construct in the multidimensional construct.
The presence of outliers should be examined, and highly skewed measurement values should be
transformed, if necessary. True outliers may be removed from the aggregation. If measures have a
© ISO/IEC 2015 – All rights reserved 7

different range of values, normalization is required before any data manipulation because of differences
in measurement units. An appropriate normalization method should be used with respect to both the
theoretical basis and data properties.
4.7 Sensitivity analysis
4.7.1 Requirements
Sensitivity analysis aims to examine the robustness of the composite value. The kinds and methods of
sensitivity analysis depend on rating and aggregation method in process measurement frameworks.
a) Sensitivity analysis shall be performed for measurement scales of process attributes;
b) Sensitivity analysis shall be performed for aggregation methods;
c) Sensitivity analysis shall be performed for weights, if applicable.
4.7.2 Guidance
The robustness of the composite value can be evaluated by uncertainty analysis or by sensitivity
analysis. Uncertainty analysis examines how uncertainty in input factors such as measurement values
propagates through the structure of the composite measure and affects the composite value. Sensitivity
analysis examines the extent to which each individual source of uncertainty contributes to the output
variance. Sensitivity analysis can be performed on the basis of process attributes.
Including weights in a composite measure, for example most compensatory models, requires sensitivity
analysis for weights, where a weight assignment method should be specified. Non-compensatory MCDM
models do not require weights.
5 Requirements for the validation of process measurement frameworks
5.1 Requirements
a) Plans for reliability and validity of process measurement frameworks shall be established at the
beginning of standardization. These plans shall include post-standardization activities;
b) Claims on reliability and validity of process measurement frameworks shall be consistent with
construct specification;
c) Consistency (also refers to equivalence) as a reliability measure shall be examined for process
attributes, if reflective;
d) Validities shall be examined for the process quality characteristic and its process attributes in a
process measurement framework;
e) Construct specification shall be empirically examined for the process quality characteristic and its
measures in a process measurement framework;
f) External measures (e.g., goals, criteria, and/or achievements) of a process measurement framework
under development shall be documented for validity investigation.
5.2 Guidance
The quality of the process quality characteristic and its process attributes can be examined by using
empirical methods such as reliability estimation (especially if reflective) and validity tests. Process
measurement frameworks state their reliability and validity claims and how those claims shall be
corroborated. Statistical validation of requirements specified in this Clause can be provided by a
separate document or an Annex of a process measurement framework.
8 © ISO/IEC 2015 – All rights reserved

5.2.1 Reliability
A general definition of the reliability of a measure is the variance of the true (latent variable) variance
divided by the total measure variance. Reliability concerns the degree of repeatability (stability) and
consistency (equivalence) of a measure in terms of its ability to capture latent variables. Repeatability
implies that “repeated assessments [at two different points in time, of the same process to the same
or alternative instrument by the same assessor] should produce results that can be accepted as being
identical. Consistency (equivalence) focuses on multiple measures of a construct measured at a single
point in time, where each measure is considered a separate but equivalent measure of the underlying
concept. C.2 briefly introduces statistical methods for estimating reliability.
A satisfactory level of reliability depends on how assessment results from a process measurement
framework are used. For instance, in applied settings where important decisions are made with respect
to the composite value, a high value of consistency (e.g., 0.9) is usually recommended as the minimally
acceptable value.
In reflective constructs, unidimensionality is a required condition for the reliability analysis and
construct validity.
5.2.2 Construct validity
The quality of the process quality characteristic and process attributes can be examined by using
empirical methods such as reliability estimation (if reflective) and validity tests. The validation of a
process measurement framework is a procedure for determining whether there is objective evidence
that the process quality characteristic and process attributes what they are intended to measure, and
that they are useful for their intended purposes. Some validation methods can be performed during
standard development as indicated by Clause 4. However, this clause addresses post hoc statistical
analyses to validate process measurement frameworks during trials and/or after publication.
If the process quality characteristic and its process attributes in a process measurement framework
are not correctly operationalized, measured, or statistically validated, any composite measure may be
weak or inappropriate. Thus, the process quality characteristic and its process attributes in a process
measurement framework should be linked to its statistical validation, although statistical tests are not
the purpose of International Standards. Threats to the validity should be addressed when evaluating the
validity of process measurement frameworks. Validity tests depend on construct specifications.
Data of external measures can be objectively or subjectively collected. They are used for examining the
predictive validity. Construct validities are briefly explained in C.3.
5.2.3 Construct specification
Construct specifications (also referred to as specification models) can be statistically tested to
determine whether the relationship between process quality characteristic and its process attributes is
formative or reflective. A simulation study can be performed if necessary. All aggregation should meet
the rationale of construct specification.
NOTE Confirmatory tetrad analysis, (addressed in B.3), can be used to statistically test construct
specifications.
6 Verifying conformity of process measurement frameworks
This clause is concerned with the mechanisms that may be used to verify that the requirements of this
International Standard have been fulfilled.
Conformity to the requirements of this International Standard may be verified by:
— self-declaration (first party);
— a second party;
© ISO/IEC 2015 – All rights reserved 9

— a third party.
The party performing verification shall obtain objective evidence that the process measurement
framework fulfils the requirements set forth in Clause 4. Objective evidence shall be provided to
demonstrate the integrity and consistency of the process measurement framework.
10 © ISO/IEC 2015 – All rights reserved

Annex A
(informative)
A terminology map
Figure A.1 presents a map of terms defined in Clause 3 and includes requirements and guidance defined in
Clause 4 (the arrows present reading direction). A process quality characteristic is explained, described,
and organized by a multidimensional construct that consists of a set of unidimensional constructs. Each
of the unidimensional constructs assigns meaning to phenomenon of interest. A construct is identified
and clarified via conceptualization and embodied by construct definition.
Conceptualization
[Clause 4.1]
Phenomenon
is a process of identifying
is term to describe,
and clarifying
Multidimensional construct
organize, and assign
[Clause 4.1]
meaning to
Process quality is a
consist of
defines the meaning of
characteristic
and clarifies
(Construct)
Construct
the construct
A set of unidimensional constructs
[Clause 4.1]
definition
specification of
[Clause 4.2]
are evaluated
represents
are factored into
for
or are an unidimensional
construct
Validities & Construct
are operationalization of
Reliability specification test
Latent variable
[Clause 5.1] [Clause 5.1]
Sensitivity
[Clause 4.2]
Analysis
are evaluated
empirically evaluate
[Clause 4.7]
for
process attributes whether
they are reflective or
is performed for
Assessment
formative.
Process Compensatory and
the rating scale
attributes non-compensatory models
Composite measure
is obtained by
is assigned to [Clause 4.3]
[Clause 4.2]
Rating
includes
is examined by
[Clause 4.5]
determines a
is performed for
composite value of
MCDM
Decision rules of
ratings of
construct
is a kind of
specification
Aggregation
[Clause 4.4]
[Clause 4.6]
Figure A.1 — Map of terminologies
The process quality characteristic is operationalized as a set of process attributes, which are specified
as either reflective or formative. Decision rules provide guides on the definition of process attributes. If
process attributes are defined as reflective measures, factor analysis can support to determine a set of
unidimensional constructs. On the other hand, process attributes defined as formative may treat each
process attribute as a dimension. E
...