ISO/IEC 25040:2011
(Main)Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Evaluation process
Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Evaluation process
ISO/IEC 25040:2011 contains requirements and recommendations for the evaluation of software product quality and clarifies the general concepts. It provides a process description for evaluating software product quality and states the requirements for the application of this process. The evaluation process can be used for different purposes and approaches. The process can be used for the evaluation of the quality of pre-developed software, commercial-off-the-shelf software or custom software and can be used during or after the development process. ISO/IEC 25040:2011 establishes the relationship of the evaluation reference model to the SQuaRE documents as well as shows how each SQuaRE document should be used during the activities of the evaluation process. It is intended for those responsible for software product evaluation and is appropriate for developers, acquirers and independent evaluators of software products. These three different approaches are detailed in ISO/IEC 14598-3, ISO/IEC 14598-4, and ISO/IEC 14598-5. It is not intended for evaluation of other aspects of software products (such as functional requirements, process requirements, business requirements, etc.).
Ingénierie des systèmes et du logiciel — Exigences de qualité et évaluation des systèmes et du logiciel (SQuaRE) — Modèle de référence d'évaluation et guide
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 25040
First edition
2011-03-01
Systems and software engineering —
Systems and software Quality
Requirements and Evaluation
(SQuaRE) — Evaluation process
Ingénierie des systèmes et du logiciel — Exigences de qualité et
évaluation des systèmes et du logiciel (SQuaRE) — Modèle de
référence d'évaluation et guide
Reference number
ISO/IEC 25040:2011(E)
©
ISO/IEC 2011
---------------------- Page: 1 ----------------------
ISO/IEC 25040:2011(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2011
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2011 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 25040:2011(E)
Contents Page
Foreword .iv
Introduction.v
1 Scope.1
2 Conformance .1
3 Normative references.1
4 Terms and definitions .1
5 Software product quality evaluation reference model .10
5.1 Reference model - general .10
5.2 Reference model - evaluation processes.11
5.3 Roles.13
5.4 Quality in the life cycle.13
5.5 Support for the evaluation.13
6 Software product quality evaluation process .14
6.1 General requirements .14
6.2 Documentation .14
6.3 Establish the evaluation requirements .15
6.4 Specify the evaluation.17
6.5 Design the evaluation .19
6.6 Execute the evaluation.20
6.7 Conclude the evaluation.21
Annex A (informative) Evaluation levels.25
Annex B (informative) Evaluation methods.29
Annex C (informative) Example of Cost-Effectiveness Ranking of Evaluation Methods .34
Annex D (informative) Relationships between software product quality evaluation process
reference model and software and system life cycle processes .35
Annex E (informative) Evaluation report template .37
Annex F (informative) Diagrams of inputs, outcomes, constraints and resources for activities.39
Bibliography.44
© ISO/IEC 2011 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 25040:2011(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electro technical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 25040 is part of the SQuaRE series of standards and was prepared by Joint Technical Committee
ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software and systems engineering.
iv © ISO/IEC 2011 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 25040:2011(E)
Introduction
As the use of information technology grows, the number of critical computer systems also grows. Such
systems include, for example, security critical, life critical, economically critical and safety critical systems. The
quality of software in these systems is particularly important because software faults can lead to serious
consequences.
Evaluation is the systematic determination of the extent to which an entity meets its specified criteria. The
evaluation of software product quality is vital to both the acquisition and development of software. The relative
importance of the various characteristics of software quality depends on the intended usage or objectives of
the system of which the software is a part; software products need to be evaluated to decide whether relevant
quality characteristics meet the requirements of the system.
This document is part of the SQuaRE series of standards and contains general requirements for software
product quality evaluation as well as clarifies the associated general concepts.
The general goal of creating the SQuaRE set of standards is to move to a logically organized, enriched and
unified series covering two main processes: software quality requirements specification and software quality
evaluation, supported by a software quality measurement process. The purpose of the SQuaRE set of
standards is to assist those developing and acquiring software products with the specification and evaluation
of quality requirements. It establishes criteria for the specification of software product quality requirements,
their measurement, and evaluation. It includes a quality model for aligning customer definitions of quality with
attributes of the development process. In addition, the series provides recommended measures of software
product quality attributes that can be used by developers, acquirers, and evaluators.
SQuaRE provides
• terms and definitions,
• reference models,
• general guide,
• individual division guides, and
• standards for requirements specification, planning and management, measurement and evaluation
purposes.
SQuaRE includes International Standards on quality model and measures, as well as on quality requirements
and evaluation.
SQuaRE replaces the current ISO/IEC 9126 series and the ISO/IEC 14598 series.
This International Standard is intended to be used in conjunction with the other parts of the SQuaRE series of
standards, and with the ISO/IEC 14598 series and the ISO/IEC 9126 series until superseded by the
ISO/IEC 25000 series of standards.
The SQuaRE series of standards consists of the following divisions under the general title Systems and
software product Quality Requirements and Evaluation:
• ISO/IEC 2500n - Quality Management Division,
• ISO/IEC 2501n - Quality Model Division,
© ISO/IEC 2011 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 25040:2011(E)
• ISO/IEC 2502n - Quality Measurement Division,
• ISO/IEC 2503n - Quality Requirements Division, and
• ISO/IEC 2504n - Quality Evaluation Division.
Annex A provides an explanation on levels of evaluation, aspects to be considered when defining evaluation
levels and suggestions on evaluation techniques to be applied according to the rank of evaluation level.
Annex B provides examples of evaluation methods.
Annex C provides a table showing relationships between some evaluation methods, possible cost rank and
effectiveness per software quality characteristics.
Annex D provides relationships between the software product quality evaluation process reference model and
the software and system life cycle processes.
Annex E provides an example template of an evaluation report.
Annex F provides the diagrams of inputs, outcomes, constraints and resources for each evaluation activity.
Figure 1 illustrates the organization of the SQuaRE series representing families of standards, further called
Divisions.
Quality Model
Division
2501n
Quality
Quality
Quality Management
Requirements Evaluation
Division
Division Division
2500n
2503n 2504n
Quality
Measurement Division
2502n
Extension Division
25050 - 25099
2503n 2504n
Figure 1 — Organization of the SQuaRE series of International Standards
The Divisions within the SQuaRE model are as follows.
• ISO/IEC 2500n - Quality Management Division. The International Standards that form this division
define all common models, terms and definitions referred to by all other standards from the SQuaRE
series. Referring paths (guidance through SQuaRE documents) and high-level practical suggestions in
applying proper standards to specific application cases offer help to all types of users. The division also
provides requirements and guidance for a supporting function which is responsible for the management
of software product requirements, specification and evaluation.
vi © ISO/IEC 2011 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 25040:2011(E)
• ISO/IEC 2501n - Quality Model Division. The International Standard that forms this division presents
detailed quality models for software, quality in use and data. Practical guidance on the use of the quality
model is also provided.
• ISO/IEC 2502n - Quality Measurement Division. The International Standards that form this division
include a software product quality measurement reference model, mathematical definitions of quality
measures, and practical guidance for their application. This division presents internal measures of
software quality, external measures of software quality and quality in use measures. Quality measure
elements (QME) forming foundations for the latter measures are defined and presented.
• ISO/IEC 2503n - Quality Requirements Division. The International Standard that forms this division
helps specifying quality requirements. These quality requirements can be used in the process of quality
requirements, elicitation for a software product to be developed or as inputs for an evaluation process.
The requirements definition process is mapped to technical processes defined in ISO/IEC 15288.
• ISO/IEC 2504n - Quality Evaluation Division. The International Standards that form this division provide
requirements, recommendations and guidelines for software product evaluation, whether performed by
independent evaluators, acquirers or developers. The support for documenting a measure as an
evaluation module is also presented.
ISO/IEC 25050 to ISO/IEC 25099 are reserved to be used for SQuaRE extension International Standards
and/or Technical Reports.
This International Standard is part of the 2504n series on quality evaluation division that currently consists of
the following International Standards:
• ISO/IEC 25040 - Evaluation process: contains general requirements for specification and evaluation of
software quality and clarifies the general concepts. Provides a process description for evaluating quality of
software product and states the requirements for the application of this process. The evaluation process is
the basis for software product quality evaluation for different purposes and approaches. Therefore, the
process can be used for the evaluation of quality in use, external measure of software quality and internal
measure of software quality and can be applied to evaluate the quality of pre-developed software or
custom software during its development process. The software product quality evaluation can be
conducted, for instance, by an acquirer, a developer organization, or an independent evaluator.
• ISO/IEC 25041 - Evaluation guides for developers, acquirers and evaluators: contains specific
requirements and recommendations for developers, acquirers and evaluators.
• ISO/IEC 25042 - Evaluation modules: defines the structure and content of the documentation to be used
to describe an evaluation module. These evaluation modules contain the specification of the quality model
(i.e. characteristics, subcharacteristics and corresponding internal, external or quality in use measures),
the associated data and information about the planned application of the model and the information about
its actual application. Appropriate evaluation modules are selected for each evaluation. In some cases it
may be necessary to develop new evaluation modules. Guidance for developing new evaluation modules
is found in ISO/IEC 25042. This International Standard can also be used by organizations producing new
evaluation modules.
• ISO/IEC 25045 - Evaluation module for recoverability: provides the specification to evaluate the
subcharacteristic of recoverability defined under the characteristic of reliability of the quality model. It
determines the external measures of software quality of resiliency and autonomic recovery index when the
information system composed of one or more software products' execution transactions is subjected to a
series of disturbances. A disturbance could be an operational fault (e.g. an abrupt shutdown of an OS
process that brings down a system) or an event (e.g. a significant increase of users to the system).
ISO/IEC 25040 is a revised version and replaces the current ISO/IEC 14598-1.
© ISO/IEC 2011 – All rights reserved vii
---------------------- Page: 7 ----------------------
INTERNATIONAL STANDARD ISO/IEC 25040:2011(E)
Systems and software engineering — Systems and software
Quality Requirements and Evaluation (SQuaRE) — Evaluation
process
1 Scope
This International Standard contains requirements and recommendations for the evaluation of software
product quality and clarifies the general concepts. It provides a process description for evaluating software
product quality and states the requirements for the application of this process. The evaluation process can be
used for different purposes and approaches. The process can be used for the evaluation of the quality of pre-
developed software, commercial-off-the-shelf software or custom software and can be used during or after the
development process.
This International Standard establishes the relationship of the evaluation reference model to the SQuaRE
documents as well as shows how each SQuaRE document should be used during the activities of the
evaluation process.
It is intended for those responsible for software product evaluation and is appropriate for developers, acquirers
and independent evaluators of software products. These three different approaches are detailed in
ISO/IEC 14598-3, ISO/IEC 14598-4, and ISO/IEC 14598-5.
It is not intended for evaluation of other aspects of software products (such as functional requirements,
process requirements, business requirements, etc.).
2 Conformance
Evaluation of software product quality conforms to this International Standard if it complies with the
requirements of Clause 6.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
There are no normative references in this document.
4 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
4.1
acquirer
individual or organization that acquires or procures a system, software product or software service from a
supplier
NOTE Adapted from ISO/IEC 12207:2008.
© ISO/IEC 2011 – All rights reserved 1
---------------------- Page: 8 ----------------------
ISO/IEC 25040:2011(E)
4.2
analysis model
algorithm or calculation combining one or more base and/or derived measures with associated decision
criteria
4.3
attribute
inherent property or characteristic of an entity that can be distinguished quantitatively or qualitatively by
human or automated means
NOTE 1 Adapted from ISO/IEC 15939:2007.
NOTE 2 ISO 9000 distinguishes two types of attributes: a permanent characteristic existing inherently in something;
and an assigned characteristic of a product, process or system (e.g. the price of a product, the owner of a product). The
assigned characteristic is not an inherent quality characteristic of that product, process or system.
4.4
attribute for quality measure
attribute that relates to software product itself, to the use of the software product or to its development process
NOTE Attributes for quality measure are used in order to obtain quality measure elements.
4.5
base measure
measure defined in terms of an attribute and the method for quantifying it
NOTE 1 A base measure is functionally independent of other measures.
NOTE 2 Adapted from the International Vocabulary of Basic and General Terms in Metrology, 1993.
[ISO/IEC 15939:2007]
4.6
commercial-off-the-shelf software product
software product defined by a market-driven need, commercially available, and whose fitness for use has
been demonstrated by a broad spectrum of commercial users
4.7
context of use
users, tasks, equipment (hardware, software and materials), and the physical and social environments in
which a product is used
[ISO 9241-11:1998]
4.8
custom software
software product developed for a specific application from a user requirements specification
4.9
data
collection of values assigned to base measures, derived measures and/or indicators
[ISO/IEC 15939:2007]
4.10
decision criteria
thresholds, targets, or patterns used to determine the need for action or further investigation, or to describe
the level of confidence in a given result.
[ISO/IEC 15939:2007]
2 © ISO/IEC 2011 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 25040:2011(E)
4.11
derived measure
measure that is defined as a function of two or more values of base measures
[ISO/IEC 15939:2007]
NOTE 1 Adapted from the International Vocabulary of Basic and General Terms in Metrology, 1993.
NOTE 2 A transformation of a base measure using a mathematical function can also be considered as a derived
measure.
4.12
developer
individual or organization that performs development activities (including requirements analysis, design,
testing through acceptance) during the software life-cycle process
NOTE Adapted from the definition in ISO/IEC 12207:2008.
4.13
division of standards
division forms a family of standards serving complementary purposes
4.14
end user
individual person who ultimately benefits from the outcomes of the system
NOTE The end user can be a regular operator of the software product or a casual user such as a member of the
public.
4.15
entity
object that is to be characterized by measuring its attributes
EXAMPLE An object can be a process, product, project, or resource.
[ISO/IEC 15939:2007]
4.16
evaluation
systematic determination of the extent to which an entity meets its specified criteria
[ISO/IEC 12207:2008]
4.17
evaluation coverage
degree to which the evaluation covers the specified software product quality requirements
4.18
evaluation level
rigour to be applied during the evaluation that defines the depth or thoroughness of the evaluation in terms of
evaluation techniques to be applied and evaluation results to be achieved
4.19
evaluation method
procedure describing actions to be performed by the evaluator in order to obtain results for the specified
measurement applied to the specified product components or on the product as a whole
© ISO/IEC 2011 – All rights reserved 3
---------------------- Page: 10 ----------------------
ISO/IEC 25040:2011(E)
4.20
evaluation module
package of evaluation technology for measuring software quality characteristics, subcharacteristics or
attributes
NOTE The package includes evaluation methods and techniques, input to be evaluated, data to be measured and
collected and supporting procedures and tools.
4.21
evaluation records
documented objective evidence of all activities performed and of all results achieved within the evaluation
process
4.22
evaluation requester
person or organization that requests an evaluation
4.23
evaluation tool
instrument that can be used during evaluation to collect data, to perform interpretation of data or to automate
part of the evaluation
NOTE Examples of such tools are source code analysers to compute code metrics, CASE tools to produce
formalized models, test environments to run the executable programs, checklists to collect inspection data or
spreadsheets to produce syntheses of measures.
4.24
evaluation stringency
degree required for the software product quality characteristics and subcharacteristics to fulfil the expected
use criticality of the software product
4.25
evaluator
individual or organization that performs an evaluation
4.26
failure
termination of the ability of a product to perform a required function or its inability to perform within previously
specified limits
NOTE Adapted from IEEE 610.12-1990.
4.27
fault
incorrect step, process or data definition in a computer program
[IEEE 610.12-1990]
4.28
functional requirement
requirement that specifies a function that a system or system component must be able to perform
[IEEE 610.12-1990]
NOTE The software quality characteristic “functionality” can be used to specify or evaluate the suitability, accuracy,
interoperability, security and compliance of a function.
4 © ISO/IEC 2011 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 25040:2011(E)
4.29
implied needs
needs that may not have been stated but are actual needs
NOTE Some implied needs only become evident when the software product is used in particular conditions.
EXAMPLE Implied needs include: needs not stated but implied by other stated needs and needs not stated because
they are considered to be evident or obvious.
4.30
independent evaluator
individual or organization that performs an evaluation independently from developers and acquirers
NOTE The individual or organization acting as developer or acquirer for the target system to be evaluated cannot
become the independent evaluator for the system. The independent evaluator can be an organization. Independent
evaluators can belong to the same organization as the developers as long as they are independent from developers and
acquirers.
4.31
indicator
measure that provides an estimate or evaluation of specified attributes derived from a model with respect to
defined information needs
[ISO/IEC 15939:2007]
NOTE In ISO/IEC 14598-1 this definition was “a measure that can be used to estimate or predict another measure”.
4.32
information need
insight necessary to manage objectives, goals, risks, and problems
[ISO/IEC 15939:2007]
4.33
information product
one or more indicators and their associated interpretations that address an information need
EXAMPLE A comparison of a measured defect rate to planned defect rate along with an assessment of whether or
not the difference indicates a problem.
[ISO/IEC 15939:2007]
4.34
information system needs
needs that can be specified as quality requirements by external measures and sometimes by internal
measures
4.35
intermediate software product
product of the software development process that is used as input to another stage of the software
development process
EXAMPLE Intermediate software products can include static and dynamic models, other documents and source
code.
4.36
intermediate software product needs
needs that can be specified as quality requirements by internal measures
© ISO/IEC 2011 – All rights reserved 5
---------------------- Page: 12 ----------------------
ISO/IEC 25040:2011(E)
4.37
maintainer
individual or organization that performs maintenance activities
NOTE Adapted from ISO/IEC 12207:2008.
4.38
measure, noun
variable to which a value is assigned as the result of measurement
NOTE 1 The term “measures” is used to refer collectively to base measures, derived measures, and indicators.
NOTE 2 Adapted from ISO/IEC 14598-1:1999.
4.39
measure, verb
make a measurement
[ISO/IEC 14598-1:1999]
4.40
measurement
set of operations having the object of determining a value of a measure
[ISO/IEC 15939:2007]
NOTE 1 Adapted from the International Vocabulary of Basic and General Terms in Metrology, 1993.
NOTE 2 Measurement can include assigning a qualitative category such as the language of a source program (ADA,
C, COBOL, etc.).
4.41
measurement function
algorithm or calculation performed to combine two or more base measures
[ISO/IEC 15939:2007]
4.42
measurement method
logical sequence of operations, described generically, used in quantifying an attribute with respect to a
specified scale
[ISO/IEC 15939:2007]
NOTE Adapted from the International Vocabulary of Basic and General Terms in Metrology, 1993.
4.43
measurement procedure
set of operations, described specifically, used in the performance of a particular measurement according to a
given method
[ISO/IEC 15939:2007]
NOTE Adapted from the International Vocabulary of Basic and General Terms in Metrology, 1993.
4.44
measurement process
process for establishing, planning, performing and evaluating software measurement within an overall project
or organizational measurement structure
NOTE Adapted from ISO/IEC 15939:2007.
6 © ISO/IEC 2011 – All rights reserved
---------------------- Page: 13 ----------------------
ISO/IEC 25040:2011(E)
4.45
observation
instance of applying a measurement procedure to produce a value for a base measure
[ISO/IEC 15939:2007]
4.46
operator
individual or organization that operates the system
NOTE Adapted from ISO/IEC 12207:2008.
4.47
process
system of activities, which uses resources to transform inputs into outputs
NOTE Adapted from ISO 9000:2005.
4.48
quality in use (measure)
the extent to which a product used by specific users meets the users' needs to achieve specific goals with
effectiveness, productivity, safety and satisfaction in specific cont
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.