ISO/IEC 14598-3:2000

Software engineering - Product evaluation - Part 3: Process for developers

Ingénierie du logiciel — Évaluation du produit — Partie 3: Procédés pour les développeurs

General Information

• Status: Withdrawn
• Publication Date: 09-Feb-2000
• Withdrawal Date: 09-Feb-2000
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 6 - Software Product and System Quality
• Current Stage: 9599 - Withdrawal of International Standard
• Start Date: 09-Oct-2012
• Completion Date: 30-Oct-2025

Relations

• Revised: ISO/IEC 25041:2012 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Evaluation guide for developers, acquirers and independent evaluators
• Effective Date: 03-Dec-2011

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 14598-3
First edition
2000-02-01
Software engineering — Product
evaluation —
Part 3:
Process for developers
Ingénierie du logiciel — Évaluation du produit —
Partie 3: Procédés pour développeurs
Reference number
©
ISO/IEC 2000
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading
this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in
this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the
unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
©  ISO/IEC 2000
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 734 10 79
E-mail [e-mail]
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 2000 – All rights reserved

© ISO/IEC
Contents
1 SCOPE 1
2 CONFORMANCE 1
3 NORMATIVE REFERENCES 2
4 TERMS AND DEFINITIONS 2
5 EVALUATION CONCEPTS 2
5.1 General aspects 2
5.2 User needs 3
5.3 External attributes 3
5.4 Internal attributes 3
5.5 Quality indicators 4
5.6 Evaluation process 4
5.7 Relation between evaluation and life cycle processes 4
6 EVALUATION PROCESS REQUIREMENTS 5
6.1 General requirements 5
6.1.1 Organizational requirements 5
6.1.2 Project requirements 5
6.2 Establish evaluation requirements 5
6.2.1 Quality requirements identification 5
6.3 Specification of the evaluation 6
6.3.1 External quality requirements 6
6.3.2 Internal quality requirements 7
6.4 Design of the evaluation 8
6.4.1 Planning the external evaluation 8
6.4.2 Planning the internal evaluation 8
iii
© ISO/IEC
6.5 Execution of the evaluation 9
6.5.1 Internal evaluation 9
6.5.2 Evaluation of the end product 9
6.6 Quality evaluation review and feedback to the organization 10
ANNEX A DEFINITIONS FROM OTHER STANDARDS 11
BIBLIOGRAPHY
iv
© ISO/IEC
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
International Standard ISO/IEC 14598-3 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts, under the general title Software engineering — Product evaluation:
— Part 1: General overview
— Part 2: Planning and management
— Part 3: Process for developers
— Part 4: Process for acquirers
— Part 5: Process for evaluators
— Part 6: Documentation of evaluation modules
Annex A of this part of ISO/IEC 14598 is for information only.
v
© ISO/IEC
Introduction
This part of ISO/IEC 14598 is intended for use during software development. It is applicable to all software
development activities requiring a disciplined process. This part of ISO/IEC 14598 is particularly aimed at those
measuring and evaluating the quality of software.
This part of ISO/IEC 14598 provides guidelines for clarifying quality requirements and for implementing and
analysing software quality measures. This part of ISO/IEC 14598 applies to all software at all phases of the
development life cycle. It focuses on the selection and reporting of those indicators that are useful to predict end
product quality by measuring the quality of intermediate products. It also focuses on measuring end product quality.
vi
INTERNATIONAL STANDARD  © ISO/IEC ISO/IEC 14598-3:2000(E)
Software engineering - Product evaluation -
Part 3: Process for developers
1 Scope
This part of ISO/IEC 14598 provides requirements and recommendations for the practical
implementation of software product evaluation when the evaluation is conducted in parallel with the
development and carried out by the developer. In particular, it may be used to apply the concepts
described in ISO/IEC 9126-1, 2, 3 and ISO/IEC 14598-1, 2, 6.
The process described in this part of ISO/IEC 14598 defines the activities needed to analyse
evaluation requirements, to specify, design, and perform evaluation actions and to conclude the
evaluation of any kind of software product.
The evaluation process is designed to be used concurrently with the development. The evaluation
process needs to be synchronised with the software development process and the entities be
evaluated as they are delivered.
This part of ISO/IEC 14598 may be used by
• a project manager to clarify quality requirements, to monitor and control the quality of the software
during development and to make decisions to assure that the required quality is built in,
• a software designer to identify specific features that should be built into the software or changed in
order to meet the quality requirements,
• a quality assurance / control / audit responsible to evaluate whether the quality requirements are
met,
• a maintainer to make decisions for the implementation of changes and redesign/reengineering,
• a software acquirer as part of an agreement with a developer when acquiring software (e.g. in the
case of outsourcing software development) when an independent evaluation is not required.
Acquirers may be personnel in a purchasing role, developers outsourcing a part of the software
product, or end-users. The role of the acquirer depends on the agreement between the acquirer
and the developer. ISO/IEC 14598-4 describes evaluation from the acquirers point of view.
This part of ISO/IEC 14598 is intended for application at the project level. In order to obtain full benefit
from this standard the organization should be involved. This aspect is covered in ISO/IEC 14598-2.
This part of ISO/IEC 14598 does not prescribe specific indicators or metrics nor does it prescribe any
particular development method.
2 Conformance
In order to conform to this part of ISO/IEC 14598, an organization shall review all requirements and
recommendations in clause 6, to identify which are applicable, and state which requirements have not
been implemented.
© ISO/IEC
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute
provisions of this part of ISO/IEC 14598. For dated references, subsequent amendments to, or
revisions of, any of these publications do not apply. However, parties to agreements based on this part
of ISO/IEC 14598 are encouraged to investigate the possibility of applying the most recent editions of
the normative documents indicated below. For undated references, the latest edition of the normative
document referred to applies. Members of ISO and IEC maintain registers of currently valid
International Standards.
ISO/IEC 9126-1, Information technology - Software product quality - Part 1: Quality model.
ISO/IEC 12207, Information technology - Software life cycle processes.
ISO/IEC 14598-1:1999, Information technology - Software product evaluation - Part 1: General
overview.
ISO/IEC 14598-2:2000, Information technology - Software product evaluation - Part 2: Planning and
management.
ISO/IEC 14598-6, Software engineering - Product evaluation - Part 6: Documentation of evaluation
modules.
4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the definitions given in ISO/IEC 14598-1 and the
following definitions apply.
4.1
counting rule
conditions and procedures under which the measurement value is obtained
4.2
external attribute
a measurable property of an entity which can only be derived with respect to how it relates to its
environment
Note: External attributes are those that relate to requirements (external properties of the software). External
attributes can only be derived from the operational behaviour of the system of which it is a part.
4.3
internal attribute
a measurable property of an entity which can be derived purely in terms of the entity itself
Note: Internal attributes are those that relate to the internal organization of the software and its development.
4.4
unit
a quantity adopted as a standard of measurement
Note: Each unit has an associated scale.
5 Evaluation concepts
5.1 General aspects
The quality of software products can be described in terms of quality characteristics.
© ISO/IEC
Note: A set of quality characteristics is defined in ISO/IEC 9126-1.
However, in general it is not practical to assign measurement values directly to these characteristics.
Instead, a set of software quality attributes of the software product is selected that represents the main
aspects of the characteristics. Measurement values of these attributes give a quantitative
representation of the quality of the software product.
The emphasis of this part of ISO/IEC 14598 is to support the developer when applying software
measurement and evaluation during the development life cycle. This is done by identifying attributes of
the intermediate products and development activities and by measuring these attributes. This provides
a means for quantitatively monitor and control the quality of the software product under development
during the development process. The goal is to identify problems in achieving the desired quality as
early as possible in the development process.
Today’s knowledge of software measurement and evaluation does not justify the recommendation of a
single set of attributes that apply to every software product and every software developing
organization. Therefore, the selection of attributes of the software product, the intermediate products
and development activities is based on the experience of the organization developing the software.
5.2 User needs
Identification of the user needs is an important aspect of establishing general quality requirements.
This is done by identifying the user needs for quality in use in particular contexts of use. These general
requirements are informal by nature and needs to be formalised. They can be quantified and evaluated
using quality in use metrics.
Note : A set of quality in use metrics is described in ISO/IEC 9126-4.
The approach taken in this part of ISO/IEC 14598 is to formulate the general requirements in terms of
external attributes.
5.3 External attributes
External quality attributes represent the quality characteristics of the software product. They are used
to express external quality requirements quantitatively. This is done by assigning a target
measurement value to each attribute.
When the software product is developed actual measurement values of the attributes are collected,
hereby providing a quantitative expression of the quality characteristics of the software. Quality
evaluation is done by comparing the actual measured values with the target values of all attributes.
Note: A set of external software quality metrics is provided in ISO/IEC 9126-2.
5.4 Internal attributes
In order to monitor and control the software quality during development the external quality
requirements are translated into requirements of intermediate products and development activities.
This is done by translating the target measurement values of the external attributes of the software
product into target measurement values of internal attributes of intermediate products and
development activities.
The selection of internal attributes and translation of external target values into internal target values is
a non-trivial activity. It depends primarily on personal experience unless the developer provides an
infrastructure for collecting and analysing experience from previously completed projects. In that case,
the experience of the developer can support the activity.
Note 1:The organizational aspect is described in ISO/IEC 14598-2.
During development actual values of internal attributes are measured. The values are compared to the
target values. This provides a control of the software quality during development.
© ISO/IEC
Internal attributes can be used to identify anomalies or outliers (i.e. attribute values that deviate from
what would normally be expected). General experience tells that such entities are worth examining
more closely.
Some internal attributes can be used to monitor trends in the development when they are measured
periodically (e.g. every week). Trends measures are used for identifying problems early, both related to
the product and to the development process.
Note 2: A set of internal metrics is provided in ISO/IEC 9126-3.
5.5 Quality indicators
Internal quality attributes can be used as quality indicators. In particular, internal attributes are often
used as indicators of external attributes; but no general, direct relationship between quality indicators
and external quality attributes has been validated yet. However, it is commonly accepted that quality
indicators provide useful guidance when used with care.
Use of quality indicators allows the software developer to identify possible quality problems early in the
development and to take corrective actions immediately.
There is no known universal set of quality indicators that is suitable for every software development
effort. There are differences in applications, development methods and tools, project organizations and
cultural differences to mention some examples. Therefore, some indicators may be useful in one
organization, but not work in another organization.
5.6 Evaluation process
The evaluation process described in this part of 14598 consists of a set of activities which are
conducted by the developer. These activities are performed on the basis of measurement values
obtained during the development process.
Note 1: The generic evaluation process is described in ISO/IEC 14598-1.
Note 2 : The organizational aspects of the evaluation is described in ISO/IEC 14598-2.
The evaluation process comprises the five activities listed below:
- Establishment of evaluation requirements which consists of identifying the general quality
requirements according to an agreed quality model. This activity is described in 6.2.
- Specification of the evaluation which consists of determining the external metrics and target
measurement values (criteria for evaluation). This activity is described in 6.3.1. Specification also
consists of determining the internal metrics and target measurement values (criteria for evaluation).
This activity is described in 6.3.2.
- Design of the evaluation which consists of planning data collection actions. This activity is described
in 6.4.1 and 6.4.2.
- Execution of the evaluation which consists of collecting internal measurement values during
development and comparing them with target values (evaluation during development). Internal attribute
values (quality indicators) are used to estimate end product quality. This is described in 6.5.1. It also
consists of collecting external measurement values when they become available and comparing with
target values (evaluation of product quality). This activity is described in 6.5.2.
- Feedback to the organization which is based on a of review of evaluation results. This activity is
described in 6.6.
5.7 Relation between evaluation and life cycle processes
Evaluation of a software product can be performed within the context of any life cycle process.
© ISO/IEC
Note 1: Software life cycle processes are defined in ISO/IEC 12207: 1995.
This part of ISO/IEC 14598 relates primarily to the development processes.
Note 2 : Development processes are described in ISO/IEC 12207 subclause 5.3. As stated in ISO/IEC 12207,
this implies that it may also be necessary to consider the maintenance process (subclause 5.5) and supporting
life cycle processes (clause 6) and organizational life cycle processes (clause 7). When this standard is used in
the case of outsourcing software development it also relates to the acquisition process and the supply process
as described in ISO/IEC 12207 subclause 5.1 and 5.2
6 Evaluation process requirements
6.1 General requirements
This clause relates to the organizational and project-specific requirement.
6.1.1 Organizational requirements
The developer shall build up an infrastructure that allows for data collection and process modifications
based on data analysis.
Note : Organizatorial aspects of evaluation are described in ISO/IEC 14598-2
6.1.2 Project requirements
The developer shall develop the software following a disciplined development process that allows for
planning and conducting software measurement and evaluation.
Note 1: Life cycle processes are described in ISO/IEC 12207. Development are described in subclause 5.3.
Note 2: An overview of software product evaluation can be found in ISO/IEC 14598-1.
The developer shall coordinate evaluation activities with supporting processes and activities.
Note 3 : Supporting processes are described in ISO/IEC 12207, including in particular the quality assurance
process (subclause 6.3), the verification process (subclause 6.4), the validation process (subclause 6.5) and the
audit process (subclause 6.7).
Many data analysis methods require data from previous projects developed under similar conditions
and with comparable quality requirements. The developer should, therefore, apply a development
model similar to one that has been used in previous projects in the developers organization. Also the
same set of attributes should be applied in the projects to allow for data analysis.
6.2 Establish evaluation requirements
This clause relates to the establishment of the general quality requirements and the analysis of their
feasibility.
6.2.1 Quality requirements identification
The developer shall ensure that general quality requirements applicable to the software system are
identified. User needs, organizational experience, application area experience, software integrity
requirements, required standards, regulations, laws etc. should be considered when identifying general
requirements.
Note 1: Software integrity levels are described in ISO/IEC 15026.
The developer shall ensure that an agreed quality model is used for structuring the quality
requirements.
Note 2 : A quality model is described in ISO/IEC 9126-1.
A list of other system requirements that may affect the feasibility of the quality requirements shall be
produced. Acquisition concerns, such as cost and schedule constraints, warranties, and organizational
concerns should be considered. Mutually exclusive requirements should be resolved.
© ISO/IEC
Note 3: Focus should be on external product attributes.
All parties involved in the creation and use of the software system should participate or be represented
in the quality requirements identification process.
The relative priorities of the requirements should be discussed with all involved parties. Each group
should weigh the quality requirements against the other system requirements and constraints. All
viewpoints should be considered.
The identified quality requirements may be conflicting or cooperative. Conflicts between the
requirements shall be resolved. In addition, if the choice of quality requirements is in conflict with cost,
schedule or system functionality, one or the other shall be altered.
The developer shall execute a feasibility analysis of the quality requirements. Experience from previous
projects with similar quality requirements carried out in the developers organization should be
considered.
The developer shall ensure that the requirements are technically feasible, reasonable, complementary,
achievable and verifiable.
The quality requirements shall be resolved into a single set of quality requirements formulated
according to the agreed quality model. Agreement to the final list of general requirements should be
sought from all involved parties.
6.3 Specification of the evaluation
This subclause relates to quantification of quality requirements. For each requirement, one or more
external attributes are selected to represent the requirement. Assigned target values serve as
quantitative representations of the requirements (evaluation criteria).
For each external require
...