ISO/IEC 14598-6:2001

Software engineering - Product evaluation - Part 6: Documentation of evaluation modules

This part of ISO/IEC 14598 defines the structure and content of the documentation to be used to describe an Evaluation Module. Evaluation modules are intended to be used within the context of the ISO/IEC 9126 and the ISO/IEC 14598 multipart standards. This part of ISO/IEC 14598 is intended to be used by experts in evaluation technology such as testing laboratories, research institutes and others when producing new evaluation modules.

Ingénierie du logiciel — Évaluation du produit — Partie 6: Documentation des modules d'évaluation

General Information

• Status: Published
• Publication Date: 20-Jun-2001
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 6 - Software Product and System Quality
• Current Stage: 9093 - International Standard confirmed
• Start Date: 29-Jul-2008
• Completion Date: 30-Oct-2025

Relations

• Revised: ISO/IEC 25041:2012 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Evaluation guide for developers, acquirers and independent evaluators
• Effective Date: 07-May-2011

Overview

ISO/IEC 14598-6:2001 - "Software engineering - Product evaluation - Part 6: Documentation of evaluation modules" defines a standardized format and content for documenting an evaluation module. An evaluation module packages the evaluation technology (methods, techniques, inputs, data, tools and procedures) required to measure software quality characteristics within the context of the ISO/IEC 9126 and ISO/IEC 14598 multipart standards. The standard is intended for experts in evaluation technology - e.g., testing laboratories, research institutes, evaluators and QA groups - creating reusable, repeatable modules for software product evaluation.

Key Topics

• Evaluation module concept: a self-contained package that links evaluation techniques, metrics and measures to a specific software quality aspect.
• Documentation format (EM0–EM5, optional EMA):
  • EM0: Foreword and introduction (rationale, relationship to other documents)
  • EM1: Scope - characteristics, sub-characteristics, level of evaluation, techniques, applicability
  • EM2: References
  • EM3: Terms and definitions
  • EM4: Inputs and metrics - required inputs, data elements, metrics and measures
  • EM5: Interpretation of results - mapping measures to quality, reporting
  • EMA (optional): Detailed application procedure, resources, evaluation instructions, documentation
• Requirements for conformance: documentation must follow the clause 6 format in the standard to be conformant.
• Support for repeatability and objectivity: standardized definitions and procedures help ensure evaluations are reproducible and comparable.
• Annex guidance and examples: informative annexes provide module development guidance and concrete examples (e.g., fault density, functionality, usability/quality in use).

Applications and Who Uses It

• Testing laboratories and research institutes developing new evaluation modules and metrics.
• Software evaluators and consultants applying consistent, documented techniques during product assessments.
• Developers, acquirers and QA teams who need standardized metrics and reporting for procurement, certification, or quality assurance.
• Organisations building libraries of evaluation modules to ensure consistent internal reuse and to catalog evaluation technology. Practical uses include standardizing measurement procedures, supporting metric development, framing evaluation level and confidence, and creating repeatable test/reporting workflows.

Related Standards

• ISO/IEC 9126-1 (Product quality model) - referenced as the default quality model
• ISO/IEC 14598 parts 1–5 (Product evaluation processes: overview, planning, developer/acquirer/evaluator processes)
• ISO/IEC 12207 (Software life cycle processes) and related lifecycle standards (See ISO/IEC 14598-6 for authoritative normative references and annex examples.)

Keywords: ISO/IEC 14598-6:2001, evaluation module, software product evaluation, ISO/IEC 9126, documentation of evaluation modules, metrics, testing laboratories, evaluation technology.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 14598-6
First edition
2001-06-01
Software engineering — Product
evaluation —
Part 6:
Documentation of evaluation modules
Ingénierie du logiciel — Évaluation du produit —
Partie 6: Documentation des modules d'évaluation
Reference number
©
ISO/IEC 2001
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2001
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 � CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail [e-mail]
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 2001 – All rights reserved

Contents Page
1 Scope .1
2 Conformance.1
3 Normative references .1
4 Terms and definitions .2
5 The evaluation module concept.2
6 Format for documentation of an evaluation module.3
6.1 EM0 : Foreword and introduction .3
6.1.1 Foreword.3
6.1.2 Introduction.3
6.2 EM1 : Scope.3
6.2.1 Characteristics.3
6.2.2 Level of evaluation .3
6.2.3 Techniques.4
6.2.4 Applicability.4
6.3 EM2 : References.4
6.4 EM3 : Terms and definitions.4
6.5 EM4 : Inputs and metrics .4
6.5.1 Input for the evaluation .4
6.5.2 Data elements.5
6.5.3 Metrics and measures .5
6.6 EM5 : Interpretation of results.5
6.6.1 Mapping of measures.5
6.6.2 Reporting.5
6.7 EMA : Application procedure .5
6.7.1 Definition of Technical Terms Used.5
6.7.2 Resources Required.5
6.7.3 Evaluation instructions .6
6.7.4 Documentation.6
Annex A (informative) Development of evaluation modules .7
Annex B (informative) Example of an evaluation module – Fault density.8
Annex C (informative) Example of an evaluation module – Functionality .12
Annex D (informative) Example of an evaluation module – Usability and Quality in Use .25
Bibliography.31
© ISO/IEC 2001 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO/IEC 14598 may be the subject of
patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
International Standard ISO/IEC 14598-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598-6 is intended for use in conjunction with ISO/IEC 9126-1 (in preparation) which will replace
ISO/IEC 9126:1991.
ISO/IEC 14598 consists of the following parts, under the general title Software engineering — Product evaluation:
� Part 1: General overview
� Part 2: Planning and management
� Part 3: Process for developers
� Part 4: Process for acquirers
� Part 5: Process for evaluators
� Part 6: Documentation of evaluation modules
Annexes A, B, C and D of this part of ISO/IEC 14598 are for information only.
iv © ISO/IEC 2001 – All rights reserved

Introduction
Software product evaluation depends on a set of evaluation techniques and metrics that provide information about
the quality characteristics of the software. Many metrics and associated methods for using the measurement
results can be used for specific software product evaluation. ISO/IEC 9126-2 and ISO/IEC 9126-3 provide example
metrics that correspond to one sub-characteristic. It is difficult to use these metrics consistently in an organisation.
It may be necessary to develop new metrics for specific use. Therefore, it is necessary that a supporting function
(see 14598-2) in the organisation specifies each metric for correct and consistent use within the organisation. The
format for documenting a metric and associated methods, as well as guides for their use, should be standardised.
The concept of an evaluation module provides a solution to this need.
An evaluation module specifies the evaluation methods applicable for evaluating a quality characteristic and
identifies the evidence it needs. It also defines the elementary evaluation procedure and the format for reporting the
measurements resulting from the application of the techniques.
A consistent way of documenting evaluation modules has a number of advantages:
� It provides a common reference in the description of the theoretical basis of evaluation modules.
� It identifies a minimal set of requirements for documenting and developing evaluation modules.
� It provides a necessary tool in collecting and cataloguing the large number of evaluation modules anticipated.
Evaluation modules provide a flexible and structured approach to making metrics applicable for evaluating
intermediate and completed products. The use of evaluation modules produced according to this part of ISO/IEC
14598 helps to ensure that software product evaluations can be repeatable, reproducible and objective.
The format for documenting an evaluation module takes into account the following:
� It will be applied within the context of the evaluation of software products.
� The format supports the need for developing new metrics with respect to state of the art.
� The format provides a precise definition of metrics and their application.
� It provides the information needed for those who will use them for an evaluation.
Annex A provides guidance for the development process for new evaluation modules.
Annexes B, C and D are examples of evaluation modules.
© ISO/IEC 2001 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 14598-6:2001(E)
Software engineering — Product evaluation —
Part 6:
Documentation of evaluation modules
1 Scope
This part of ISO/IEC 14598 defines the structure and content of the documentation to be used to describe an
Evaluation Module. Evaluation modules are intended to be used within the context of the ISO/IEC 9126 and the
ISO/IEC 14598 multipart standards.
This part of ISO/IEC 14598 is intended to be used by experts in evaluation technology such as testing laboratories,
research institutes and others when producing new evaluation modules.
2 Conformance
The documentation of an evaluation module conforms to this part of ISO/IEC 14598 if it meets the requirements of
clause 6 (format for documentation of an evaluation module).
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this part of ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this part of ISO/IEC 14598 are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO/IEC 9126-1, Software engineering — Product quality — Part 1: Quality model.
ISO/IEC 12207, Information technology — Software life cycle processes.
ISO/IEC 14598-1, Information technology — Software product evaluation — Part 1: General overview.
ISO/IEC 14598-2, Product evaluation — Part 2: Planning and management.
ISO/IEC 14598-3, Product evaluation — Part 3: Process for developers.
ISO/IEC 14598-4, Software engineering — Product evaluation — Part 4: Process for acquirers.
ISO/IEC 14598-5, Information technology — Software product evaluation — Part 5: Process for evaluators.
© ISO/IEC 2001 – All rights reserved 1

4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the following terms and definitions apply.
4.1
evaluation module
a package of evaluation technology for measuring software quality characteristics, sub-characteristics or attributes
NOTE The package includes.
� evaluation methods and techniques
� inputs to the evaluation
� data to be measured and collected
� supporting procedures and tools
4.2
evaluation technology (technology used for evaluation)
the techniques, tools, metrics, measures and other technical information used for evaluation [ISO/IEC 14598-2]
5 The evaluation module concept
Evaluation of software products can be a comprehensive task. Different aspects of quality characteristics and sub-
characteristics may require different evaluation techniques to be applied and different data to be collected. In order
to manage this complexity, an evaluation should be structured into manageable units. Each of these units can
cover one or more quality aspects. However, each unit should be focused on the evaluation of one specific quality
aspect applying a specific evaluation technique. The information necessary for conducting one of these evaluations
should be collected and packaged for future use. Such a package is called an evaluation module.
The benefit of using a standardized format of an evaluation module is that
� it supports the development of evaluation modules since it provides a table of contents which makes visible
what information is necessary for an evaluation and how this information is handled (principles, metrics, tools,
...)
� it supports the use of evaluation modules since information will be available in a homogenous way
� it supports the reuse of evaluation modules since it facilitates the establishment and maintenance of libraries of
evaluation modules
� it supports standardization of evaluation modules since the format is compliant with standards requirements
An evaluation module collects all information necessary to perform an evaluation of a specific aspect of a quality
characteristic applying a specific evaluation technique. It clarifies which specific aspect of a software quality
characteristic is being measured. The procedure for making the measurement is defined as well as the
preconditions and accuracy of the measurement.
Evaluation modules provide the link between evaluation techniques, metrics and measures. When parts 3, 4 or 5 of
ISO/IEC 14598 recommend the application of an evaluation technique, an appropriate evaluation module can be
selected from an evaluation module library (see ISO/IEC 14598-2).
The documentation of an evaluation module has six parts EM0 - EM5 serving different purposes, and an optional
annex EMA.
2 © ISO/IEC 2001 – All rights reserved

EM0 provides formal information about the evaluation module and gives an introduction to the evaluation technique
described in the evaluation module.
EM1 defines the scope of applicability of the evaluation module.
EM2 provides relevant references.
EM3 includes the definitions needed for the evaluation module.
EM4 specifies the input products required for the evaluation and defines the data to be collected and measures to
be calculated.
EM5 contains information about how to interpret measurement results.
The optional annex EMA includes the detailed procedure for applying the evaluation module. Although EMA is
optional it is suggested that it be included.
NOTE The format for the documentation of an evaluation module complies with the formal requirements of a standard as
described in ISO Directives Part 3. This facilitates standardization of evaluation modules.
6 Format for documentation of an evaluation module
The documentation of an evaluation module shall be formatted according to 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 and 6.7.
6.1 EM0 : Foreword and introduction
6.1.1 Foreword
This clause shall provide information about
� preparation, approval, contributions and changes,
� relationship to other standards or other documents.
6.1.2 Introduction
This clause should introduce the principles, background and technical rationale underlying the evaluation module.
NOTE A formal description of evaluation method is provided in 6.2.3.
6.2 EM1 : Scope
6.2.1 Characteristics
This clause shall identify the characteristics, sub-characteristics or attributes that the evaluation module can
evaluate.
NOTE The evaluation module may contribute to one or more characteristics or sub-characteristics.
The quality model describing the characteristics, sub-characteristics or attributes shall be identified. The model in
ISO/IEC 9126-1 should be used, unless there is a particular reason to use another model.
6.2.2 Level of evaluation
This clause shall describe the level of the evaluation defined by the evaluation module. Evaluation levels are
related to the importance of the characteristic, sub-characteristics or attribute evaluated. The level should be
© ISO/IEC 2001 – All rights reserved 3

described taking into account the assumed usage of the software and the environment of the software product (for
example, safety conditions, security constraints, economic risks, and application constraints).
The level defines the depth or thoroughness of the evaluation in terms of the evaluation techniques to be applied
and evaluation results to be achieved. Different evaluation levels give different levels of confidence in the quality of
the software product.
NOTE Levels can be formulated as A, B, C, or D as described in ISO/IEC 14598-5. Software integrity levels are described
in ISO/IEC 15026.
6.2.3 Techniques
This clause shall describe the evaluation techniques applied by the evaluation module. The relevant theory, models
or heuristics underlying the evaluation shall be included or adequately referenced.
NOTE Examples of evaluation techniques are reliability growth models, benchmark testing, static analysis of code.
6.2.4 Applicability
This clause shall identify the scope of application of the evaluation module.
NOTE 1 For example, the evaluation module may be applicable to a particular programming language, or to the class of all
imperative languages.
It shall be described where in the software life cycle the evaluation module can be used. If it is intended to be used
in a specific software life cycle process, this should be identified.
NOTE 2 Software life cycle processes are described in ISO/IEC 12207.
6.3 EM2 : References
This clause shall provide references to normative and technical documents. If the evaluation module depends on
the result of other evaluation modules, then these shall be mentioned here.
6.4 EM3 : Terms and definitions
This clause shall define technical terms used in the evaluation module. Alternatively, references to sources in which
definitions can be found shall be made.
6.5 EM4 : Inputs and metrics
6.5.1 Input for the evaluation
This clause shall identify the inputs required for the evaluation. These shall be classified as product component,
product information, supporting information and product in use information.
NOTE 1 Information classified as product component includes software requirements specification, software design
description, program description, source code, executable code, and user documentation.
NOTE 2 Information classified as product information includes software requirements review report, software design review
report, program review report, unit test report, and user documentation review report.
NOTE 3 Information classified as supporting information includes quality assurance plan, configuration management plan,
program test plan, and description of programming language and compiler. Supporting information is not evaluated but only
used as background information necessary for conducting the evaluation.
NOTE 4 Information classified as product in use information includes testing report and operation report describing the
behavior of the system. System includes any associated hardware, software and users.
4 © ISO/IEC 2001 – All rights reserved

6.5.2 Data elements
This clause shall specify which data elements are to be extracted from the inputs.
NOTE 1 Examples of data elements are: Number of lines of code containing comments; frequency distribution of sentence
length in the user-manual; number of words in each help message; number of failures observed per hour of operation; number
of tokens of specified types found in each module by a specified lexical scan.
NOTE 2 In general, the data elements are the material from which measures are computed. But in some cases, the raw data
may itself constitute metrics.
6.5.3 Metrics and measures
This clause shall describe how the measures are calculated from the data elements using metrics.
If the metrics are to be combined to obtain ‘higher-order’ metrics, then the dependency shall be made explicit.
All assumptions and preconditions to be satisfied before measuring shall be stated.
6.6 EM5 : Interpretation of results
6.6.1 Mapping of measures
This clause shall specify the meaning of the measures, that is, the interpretation of the results of the measurement.
This includes the evaluation scale onto which the values obtained by the defined metrics are to be mapped. If the
mapping is non-trivial then the details of the algorithms (functions) needed to make the mappings shall be defined
or references shall be made to their sources. If several measures are obtained for a single characteristic, sub-
characteristics or attribute, then the clause shall describe how they can be combined into a rating for the
characteristic, sub-characteristics or attribute.
The accuracy of the measure shall be specified.
6.6.2 Reporting
This clause shall describe the contents of the report providing the result of applying the evaluation module. In some
cases, visualization of obtained values is important and should be encouraged.
6.7 EMA : Application procedure
NOTE The inclusion of EMA is optional but, if included, it shall have the following contents.
6.7.1 Definition of Technical Terms Used
This clause shall define technical terms that are not defined in 6.4, but used in part EMA of the evaluation module,
or referenced sources.
6.7.2 Resources Required
This clause shall specify the resources required when applying the evaluation module. This should include:
Software tools required (any software tools required should be identified and both generic tool types and
proprietary tools should be referenced); hardware/software needed; test-harness or other equipment; skills and
qualifications (any special skills and qualifications (for example, certification) required by the evaluator or the
evaluating organization should be identified); effort of application (the effort required for typical application of the
evaluation module should be estimated and, if this effort depends on attributes of the product (for example, number
of lines of code), an estimating algorithm should be given); and any other resources required.
© ISO/IEC 2001 – All rights reserved 5

6.7.3 Evaluation instructions
This clause shall describe full details of the procedure to be followed. This should include the selection of evidence
(for example, sampling of code), the generation and recording of raw data, counting rules, algorithms for computing
metrics from raw data, the recording of results, and requirements for the retention of working and final
documentation. In particular, the steps to be taken to ensure traceability and repeatability of results should be
highlighted. The described procedure should be compliant with ISO Guide 25.
6.7.4 Documentation
This clause shall outline the internal documentation resulting from the usage of the evaluation module. The outlined
report should be compliant with ISO Guide 25.
6 © ISO/IEC 2001 – All rights reserved

Annex A
(informative)
Development of evaluation modules
This informative annex provides guidance for the process of developing new evaluation modules. The new ISO/IEC
9126 part 2, part 3 and part 4 may serve as input for this process. The evaluation module development process
should include five steps:
A.1 Identification of the evaluation module requirements
When the need for a new evaluation module is identified and a decision is made to develop the evaluation module,
the first step should be to identify the requirements for the new evaluation module. This includes identification of a
quality model and quality characteristic or sub-characteristic. Also, the stringency of the evaluation should be
decided.
A.2 Specification of the evaluation module
Based on the identified requirements for the evaluation module the next step is to specify the evaluation techniques
and inputs to the evaluation (for example, source code) as well as the set of metrics and the fundamental set of
data elements. Here ISO/IEC 9126 part 2, part 3 and part 4 may be useful.
A.3 Development of the evaluation procedure
This step takes the formal specification from the previous step and adds some procedural aspects. The
interpretation of metrics and data elements should be explained in the context of the evaluation. The resources
required should be estimated and detailed evaluation procedures should be developed. Trial application of the
evaluation procedure may be necessary.
A.4 Description of the evaluation procedure
In this step the evaluation procedure developed in the previous step should be described according to the
evaluation module format, that is, the description should conform to this part of ISO/IEC 14598.
A.5 Verification and validation of the evaluation module
The evaluation module should be reviewed (verified) against its specification. Experts in the area covered by the
evaluation technique described should do this. The validation should ensure that the evaluation module represents
state of the art technology and existing technology that is unfamiliar to the organization.
The evaluation module should be tested (validated) by different groups of people in different environments. The
experience obtained should be fed back to the evaluation module development team as input for an update.
© ISO/IEC 2001 – All rights reserved 7

Annex B
(informative)
Example of an evaluation module – Fault density
Information technology - Software product evaluation - Evaluation Module: Fault density
B.0 Introduction
This evaluation module is used to determine the fault density of a program. Detecting a high number of faults
during design and testing phase reduces potential faults in the software, which will cause failures in the operation
phase. A high number of faults at the beginning of the operation phases cause frequent failures and decreases
product reliability. Therefore it should be ensured that the fault density is below a specified threshold value before
the program is put into operation.
In general, it is impossible to count the number of faults remaining in a software product. But by using the model
and the historical data of fault detection, the number can be estimated. The fault density is calculated using this
estimate. The general estimation procedure is as below:
1) Choose an appropriate Reliability Growth Model RGM to adopt, for example, an exponential RGM or S-shaped
RGM.
2) Record the cumulative number of detected faults with time at a particular point in the test period.
3) Decide on the number of parameters of the RGM equation that are required to fit the curve to the set of data
recorded.
4) As time (t) of the RGM equation tends to infinity, the estimated number of potential faults can be calculated.
B.1 Scope
B.1.1 Characteristics
Reliability – Maturity – Fault density
B.1.2 Level of evaluation
Level B as defined in ISO/IEC 14598-5.
B.1.3 Technique
A reliability growth modelling technique is used. The number of faults in the final software product is predicted using
the reliability growth model.
B.1.4 Applicability
1) This is generally used during system testing and is applicable for all types of programming languages.
2) When it is necessary to compare the fault density value with another value for a program written in a different
programming language, the size values should be normalized.
8 © ISO/IEC 2001 – All rights reserved

B.2 References
�
B.3 Terms and definitions
The following definitions apply to the evaluation module:
Fault
A defect in the software.
Failure
Any occurrence of an event (or non-occurrence of some events) usually from a set of defined events.
LOC (Lines of Code)
The number of lines of code.
ELOC (Erroneous Lines Of Code)
The number of lines of code, in which a failure is detected and modified.
EELOC (Estimated Erroneous Lines Of Code)
The estimated number of erroneous lines of code (ELOC).
NCLOC (Non-Commented Lines Of Code)
The number of lines of code without comments.
FDV (Fault Density Value)
The value that indicates the number of faults per unit of product volume.
B.4 Inputs and metrics
B.4.1 Input for the evaluation
The following sources are used as input to the evaluation:
1) Product Component : Source code
2) Product Information : Program test report, Program review report, Program verification report
B.4.2 Data elements
The following data elements must be collected in order to apply the evaluation technique:
1) The number of detected faults (ELOC).
2) Detected time of each failure. Time must be measured in a consistent way, for example, CPU time or calendar
time.
3) The number of lines of code (NCLOC).
B.4.3 Metrics and measures
The fault density value is calculated from
FDV = ( EELOC - ELOC ) / NCLOC
© ISO/IEC 2001 – All rights reserved 9

B.5 Interpretation of results
B.5.1 Mapping of measures
FDV (No mapping of metrics in this case).
Experimental source of scale inside the company:
FDV<=10E-4 Excellent
10E-4 10E-3 FDV>10E-2 Poor
NOTICE : The threshold values must be tailored in accordance with applied phase or software domain.
B.5.2 Reporting
The following information is reported as the result of applying the evaluation module:
1) Identification of source code
2) Fault density value: FDV
3) The corresponding rating {Excellent, Good, Fair, Poor}
B.6 Application procedure
B.6.1 Definition of technical terms used
RGM
Reliability Growth Model. The Reliability Growth Model used to estimate the number of erroneous lines of code.
B.6.2 Resources required
The following resources must be available when the evaluation module is applied:
Software Tools Required :
1) LOC counting tool
2) Modified line counting tool
3) Reliability data collection tool and analysis tool (optional)
Hardware/software Platform
No special requirements.
Test-harness or other Equipment
No special requirements, but use of a reliability data collection and analysis tool is recommended.
10 © ISO/IEC 2001 – All rights reserved

Skills and Qualification
Knowledge of applying the RGM is required.
Man-power Effort of Application
Most of the effort is related to testing and recording of failures. If a reliability and data collection tool is applied, the
application of the RGM and calculating the FDV only requires limited effort.
Other Special Resources
No special requirements.
B.6.3 Evaluation instructions
1) Selection of Sample
Select sample source code. Sampling ratio should be more than half.
2) Generation of Raw Data
Failure data is extracted from test reports. If test reports are not available, testing is performed. To apply the
reliability growth model RGM a minimum of XXX testing time is required.
For each failure, the location and time of occurrence is recorded.
Count modified ELOC with modification time and total NCLOC of each sample.
3) Algorithm
Estimate the number of potential erroneous LOC using the reliability growth model RGM. The estimated number of
erroneous lines of code is found by applying the RGM model:
EELOC = RGM-({(Failure,Time)})
Calculate the fault density value:
FDV = ( EELOC - ELOC ) / NCLOC
Notice: How to use the RGM model or tool must be explained here in all details or alternatively references to a
software tool and manual for making the estimation must be provided.
4) Requirements for retention of working and final document
Measure once a week, and analyse the trend during the testing phase.
Action: If the calculated value of fault density is larger than some specified value (rating=poor), the source code is
reviewed, re-tested and debugged, and fault density is calculated again.
B.6.4 Documentation (Internal)
The following information is recorded for internal documentation:
1) Identification and version of source code sample
2) Identification and version of test documentation
3) The failure set used for estimating of EELOC
4) Date of application and responsible person
© ISO/IEC 2001 – All rights reserved 11

Annex C
(informative)
Example of an evaluation module – Functionality
Information technology - Software product evaluation - Evaluation module: Functionality
C.0 Introduction
This evaluation module is used to determine the “functionality” of a software system/component, where functionality
is intended as the extent to which software component provides functions which meet stated and implied needs
under specified conditions.
A correct evaluation is possible if there is a well-documented definition of the software/system requirements and if
stated needs are well described. In every case, the evaluation will consider how it is documented.
ISO/IEC 9126 defines the “functionality” in terms of sub-characteristics, as described in following paragraphs. Each
sub-characteristic is measurable through the measure of more sub-items.
The elementary items hereby described for each sub-characteristic will be an approximate level for the definition of
the sub-characteristic value. A formula for this evaluation does not currently exist and the adoption of an emerging
standard will give confidence that the work will find a common agreement in the context of his application.
The quality model relevant to this evaluation module considers more than one metric for each elementary item
pertinent to each sub-characteristic. Metrics for each elementary item to be measured are described in “checklists”,
with (y/n/d) possible answer for each question: y means yes, n means no and d means discard because not
applicable.
The answer y/n to a question is related to the comparison of the metric value with an expected value: if the
measured value is equal or better then the expected value then the answer is “yes”, otherwise the answer is “no”.
The “functionality” characteristic is to be evaluated with following rules:
Vc =� Vsc /nsc
i
|Vsc|=� m /(n-nd)
j i
Vc is the measured value of the characteristic
Vsc is the measured value of the i-sub-characteristic
i
nsc is the number of sub-characteristics.
m is 1 if the i-answer is positive, otherwise is 0
i
n is the total number of the measures
nd is the number of discarded questions
12 © ISO/IEC 2001 – All rights reserved

C.1 Scope
C.1.1 Characteristics
Functionality metrics indicate if the software component satisfies the defined requirements. Also the software
component shall satisfy implied user needs; in other words, implicit requirements for the typology of the software
component under evaluation.
Metrics for functionality include indicators for 5 sub-characteristics:
� suitability
� accuracy
� interoperability
� compliance
� security.
Suitability metrics measure the ratio of the satisfied functions during testing and user operations to the required
functions; in other words, functions performing tasks which do not conform to documented requirements.
Accuracy metrics measure the precision:
� the range of erroneous of calculation
� the difference between the actual and expected results of task performed
� inconsistency between actual operation procedure and documented procedure (for example, in manuals)
Interoperability metrics measure the communicability level of the software component with other systems, other
software products other equipment:
� data transfer capability
� command exchange capability
Compliance metrics measure the standardisation level of the software component against regulation or rules of the
environment.
Security metrics measure the defence performance level against illegal access and/or illegal operations.
C.1.2 Level of evaluation
ISO/IEC 14598-5 in annex B (informative) describes the criteria for the selection of the evaluation level.
The criticality of the characteristic measured in this module in respect to different aspects (security, economy,.)
influences the definition of the accuracy of the evaluation, the definition of the quantity of measures to be executed,
and the techniques to be used. The question that determines the evaluation level is: if the “functionality” does not
meet the requirements, what kind of problem exists?
The following table indicates levels and conditions for each level, considering:
� safety aspects;
� economy aspects;
© ISO/IEC 2001 – All rights reserved 13

� security aspects;
� environmental aspects.
The choice of the level shall be made adopting at least the higher level resulting from the analysis of each aspect.
Levels safety aspects economy aspects Security aspects Environmental
aspects
many people financial disaster Protection of stra- unrecoverable
A
killed (company will not tegic data and environmental
survive) services damage
B threat to human large economic Protection of recoverable environ-
lives loss (company critical data and mental damage
engaged services
C damage to Significant Protection against local pollution
property, few economic loss error risk
people injured (company
affected)
D small damage Negligible no specific risk no environmental risk
to property, no economic loss identified
risk to people
C.1.3 Technique
The following table indicates the evaluation techniques adopted to evaluate the FUNCTIONALITY, starting from the
row relative to the chosen level and going down; that is, if the evaluation level to be adopted is B, the techniques
from level B to level D are applied for the evaluation.
Evaluation techniques for FUNCTIONALITY
Level A formal proofs (adequate techniques does not currently exist for the evaluation
the FUNCTIONALITY at level A)
Level B component testing (white-box testing)
Level C review, code inspection
Level D functional testing (black-box testing)
The following is an introductory description of the possible techniques indicated in the table.
Formal proofs
The most common method of program proving is the method of “inductive assertions”. The goal is the development
of a set of theorems about the software component under evaluation. The method begins by writing assertions
about the software component’s input conditions and correct results. A separate proof is necessary to show that the
program will always eventually terminate. Other proof techniques are the methods of “predicate transformers, “sub-
goal induction”, “computation induction”, “structural induction”, and “intermittent assertions”.
In the future, this evaluation module would be enlarged to cover this evaluation level, but presently this level for
FUNCTIONALITY is not measurable.
Component testing
Each software component is tested against the requirements. Also, it is tested to the completely integrated software
components.
14 © ISO/IEC 2001 – All rights reserved

White-box testing
This technique permits one to examine the internal structure of the software component: the tester derives test data
from an examination of the program’s logic.
White-box testing is concerned with the degree to which test cases exercise or cover the logic (source code) of the
program. A useful coverage criterion is the “statement coverage”; that is to require every statement in the program
to be executed at least once. Stronger logic-coverage criteria are “path coverage”, “decision coverage”, “condition
coverage”, “decision/condition coverage” and “multiple condition coverage”.
Review
The process that involves the visual inspection/analysis of software components and all the pertinent
documentation.
Code inspections
Code inspections involves the reading or visual inspection of a software component. The objective is to find errors,
but not to find solutions to the errors. However, this technique is not able to find “high level” errors, such as in the
design. Code inspection has to be considered as complementary to the computer-based inspection, such as static
analysis of code.
Checklists
The review activity is based on a defined checklist that permits the activity to be repeated with few subjective
criteria. Questions in the checklist must be as simple as possible; the goal of the question has to be elementary
information. Checklists are subjected to revision, integration, and deletion based on the application’s experience.
Static analysis of code
It is possible to deduce the existence (possible) of errors in software by analysing its source code. One type of
static analysis is the “control flow”, in which the source code is subdivided into segments and the relationship
between the segments are examined to verify, for example, that there are no segments that cannot be executed or
that there are no paths which do not reach a “STOP” statement.
Another type of analysis concerns the “call graph”,or “structure of the software system”, that describes the nesting
of all the software units.
Functional testing
This is a process of attempting to find discrepancies between the software product and its external specification.
The specification is analysed to derive a set of test cases. It is important to well define the test cases and apply
specific techniques and methods (equivalence partitioning, boundary value, analysis, cause-effect graphing, error-
guessing methods, .).
Black-box testing
The software is viewed as a black box; that is, there is no concern about the internal behaviour and structure of the
program. The tester is only interested in finding circumstances in which the program does not behave according to
its specifications. If there is not adequate exhaustive testing
...