ISO/IEC 14598-5:1998

Information technology - Software product evaluation - Part 5: Process for evaluators

This part of ISO/IEC 14598 provides requirements and recommendations for the practical implementation of software product evaluation when several parties need to understand, accept and trust evaluation results. In particular, it may be used to apply the concepts described in ISO/IEC 9126. The process described in this part of ISO/IEC 14598 defines the activities needed to analyse evaluation requirements, to specify, design and perform evaluation actions and to conclude the evaluation of any kind of software product. The evaluation process may be used to evaluate already existing products, provided the needed product components are available, or to evaluate products in development. NOTE For the evaluation of a product in development, the evaluation process needs to be synchronized with the software development process and product components are evaluated as they are delivered. This part of ISO/IEC 14598 may be used by - testing laboratory evaluators, when providing software product evaluation services, - software suppliers, when planning evaluation of their products, including evaluation to be carried out by independent testing services, - software acquirers, when requesting evaluation information from a supplier or testing service, - software users when evaluating products or when using evaluation reports provided by testing laboratories, - certification bodies in defining new certification schemes for software products.

Technologies de l'information — Évaluation du produit logiciel — Partie 5: Procédés pour les évaluateurs

General Information

• Status: Withdrawn
• Publication Date: 01-Jul-1998
• Withdrawal Date: 01-Jul-1998
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 6 - Software Product and System Quality
• Current Stage: 9599 - Withdrawal of International Standard
• Start Date: 17-Nov-2020
• Completion Date: 30-Oct-2025

Relations

• Consolidated By: ISO 8835-3:2007 - Inhalational anaesthesia systems - Part 3: Transfer and receiving systems of active anaesthetic gas scavenging systems
• Effective Date: 06-Jun-2022
• Revised: ISO/IEC 25041:2012 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Evaluation guide for developers, acquirers and independent evaluators
• Effective Date: 03-Dec-2011

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 14598-5
First edition
1998-07-01
Information technology — Software product
evaluation —
Part 5:
Process for evaluators
Technologies de l’information — Évaluation du produit logiciel —
Partie 5: Procédés pour les évaluateurs
Reference number
B C
Contents
Foreword.iv
Introduction.v
1 Scope.1
2 Conformance .1
3 Normative references.2
4 Definitions.2
5 Evaluation Concepts.3
5.1 General aspects . 3
5.2 Evaluation starting point. 3
5.2.1 Initial agreement.3
5.2.2 Parties involved in the evaluation .4
5.3 Characteristics of the evaluation process . 4
5.4 Evaluation process. 4
5.4.1 Evaluation activities.5
5.4.2 Input to the evaluation process.5
5.4.3 Output of the evaluation process.6
5.5 Relations between evaluation and life-cycle. 7
6 Evaluation process requirements.8
6.1 General requirements. 8
6.1.1 Organisation and quality system .8
6.1.2 Requester’s responsibilities.8
6.1.3 Evaluator’s responsibilities .9
6.2 Establishment of evaluation requirements . 9
6.2.1 Purpose of the establishment of evaluation requirements.9
6.2.2 Elaboration of the evaluation requirements .9
6.2.3 Contents of the evaluation requirements.10
6.2.4 Approbation and reporting .10
©  ISO/IEC 1998
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or
utilized in any form or by any means, electronic or mechanical, including photocopying and micro-
film, without permission in writing from the publisher.
ISO/IEC Copyright Office • Case postale 56 • CH-1211 Genève 20 • Switzerland
Printed in Switzerland
ii
©
ISO/IEC ISO/IEC 14598-5:1998(E)
6.3 Specification of the evaluation.11
6.3.1 Purpose of the specification of the evaluation.11
6.3.2 Elaboration of the evaluation specification .11
6.3.3 Contents of the evaluation specification .13
6.3.4 Approbation and reporting .14
6.4 Design of the evaluation.14
6.4.1 Purpose of designing the evaluation.14
6.4.2 Elaboration of the evaluation plan.14
6.4.3 Contents of the evaluation plan.16
6.4.4 Approbation and reporting .16
6.5 Execution of the evaluation.16
6.5.1 Purpose of the evaluation execution.16
6.5.2 Performing the evaluator actions.16
6.5.3 Reviewing and reporting .18
6.6 Conclusion of the evaluation.19
6.6.1 Purpose of the evaluation conclusion.19
6.6.2 Joint review of the evaluation report.19
6.6.3 Disposition of evaluation data and documents.19
Annex A (normative) Template evaluation report.20
Annex B (informative) Levels of evaluation .22
Annex C (informative) Software product components.26
Annex D (informative) Interactions between requester and evaluator .30
Annex E (informative) Evaluation contract .32
Annex F (informative) Bibliography .35
iii
©
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialised system for worldwide standardisation. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organisation to deal with particular fields of mutual
interest. Other international organisations, governmental and non-governmental, in liaison with
ISO and IEC, also take part in the work.
In the field of information technology, ISO and IEC have established a joint technical committee
ISO/IEC JTC 1. Draft International Standards adopted by the joint technical committee are
circulated to national bodies for voting. Publication as an International Standard requires approval
by at least 75% of the national bodies casting a vote.
International Standard ISO/IEC 14598-5 was prepared by Joint Technical Committee ISO/IEC
JTC1, Information technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts, under the general title Information Technology -
Software product evaluation :
- Part 1: General overview
- Part 2: Planning and management
- Part 3: Process for developers
- Part 4: Process for acquirers
- Part 5: Process for evaluators
- Part 6: Evaluation modules
Annex A forms an integral part of this part of ISO/IEC 14598. Annexes B, C, D, E and F are for
information only.
iv
©
ISO/IEC ISO/IEC 14598-5:1998(E)
Introduction
Software products are becoming more and more important in all domains of industry and services.
It is therefore necessary to be able to evaluate the quality of these software products.
Software products are extremely varied. They are produced to fulfil very diverse requirements in
terms, for example, of functionality. Their context for use can also be very varied such as in the case
of application software in a management information system, of software embedded in other
products or of game software, to cite a few examples.
Potential benefits from evaluation are:
- the developer can use the results of the evaluation of its product to identify corrective actions, in
order to improve the product or to make decisions about the evolution strategy for the product;
- for the supplier of a product the benefit from an evaluation can be to get confidence in the value
of the product; in addition the evaluation report can be used for commercial purposes;
- for software product acquirers, evaluation results may be used as objective data on which to base
acquiring decisions;
- for the industry at large, the spread of software product evaluation will help the use of quality as
a marketing argument.
The primary purpose of software product evaluation is to provide quantitative results concerning
software product quality that are comprehensible, acceptable to and can be depended on by any
interested party.
The evaluation process is described as a step-wise procedure that allows expression of evaluation
requirements in terms of quality characteristics as defined in ISO/IEC 9126. The evaluation takes
into account various documents that can be considered as part of the software product, e.g. design
documentation, test or validation reports, source code or user documentation. It is recommended
that the evaluator uses a library of evaluation modules that define evaluation methods. These
evaluation modules could be standardised, although no provision for that is proposed in this
standard. The evaluation leads to the production by the evaluator of an evaluation report.
This evaluation process is a generic abstract process that follows the model defined in ISO/IEC
9126. Therefore, this process is applicable within all primary life-cycle processes defined in
ISO/IEC 12207. Specific supporting life-cycle processes defined in ISO/IEC 12207 are directly
related to the evaluation process. They are quality assurance, verification, validation, joint review
and audit.
The tailoring process defined in ISO/IEC 12207 is built in the evaluation process defined in this
standard by allowing the user to specify and design the evaluation activities.
The evaluation process described here may be used to test the conformity to standards such as
ISO/IEC 12119.
v
©
INTERNATIONAL STANDARD  ISO/IEC ISO/IEC 14598-5:1998(E)
Information technology —
Software product evaluation —
Part 5: Process for evaluators
1 Scope
This part of ISO/IEC 14598 provides requirements and recommendations for the practical
implementation of software product evaluation when several parties need to understand, accept and
trust evaluation results. In particular, it may be used to apply the concepts described in
ISO/IEC 9126.
The process described in this part of ISO/IEC 14598 defines the activities needed to analyse
evaluation requirements, to specify, design and perform evaluation actions and to conclude the
evaluation of any kind of software product.
The evaluation process may be used to evaluate already existing products, provided the needed
product components are available, or to evaluate products in development.
NOTE  For the evaluation of a product in development, the evaluation process needs to be
synchronized with the software development process and product components are evaluated as they
are delivered.
This part of ISO/IEC 14598 may be used by
- testing laboratory evaluators, when providing software product evaluation services,
- software suppliers, when planning evaluation of their products, including evaluation to be
carried out by independent testing services,
- software acquirers, when requesting evaluation information from a supplier or testing service,
- software users when evaluating products or when using evaluation reports provided by testing
laboratories,
- certification bodies in defining new certification schemes for software products.
2 Conformance
Because of the freedom of choice afforded to the user by the general nature of its recommendations,
a simple claim of compliance with this part of ISO/IEC 14598 is not valid. Any organization
imposing this part of ISO/IEC 14598 as a condition of trade is responsible for specifying and
making public a set of requirements which constitute the terms for compliance for a given
application of this part of ISO/IEC 14598. All requirements of clause 6 should be considered for
applicability.
©
3 Normative references
The following standards contain provisions, which through reference in this text, constitute
provisions of this part of ISO/IEC 14598. At the time of publication, the editions indicated were
valid. All standards are subject to revision, and parties to agreements based on this part of
ISO/IEC 14598 are encouraged to investigate the possibility of applying the most recent editions of
the standards indicated below. Members of IEC and ISO maintain registers of currently valid
International Standards.
ISO/IEC 9126:1991, Information technology — Software product evaluation — Quality
characteristics and guidelines for their use.
1)
ISO/IEC 14598-1:— ,
Information technology — Software product evaluation — Part 1: General
overview.
1)
ISO/IEC 14598-6:— , Information technology — Software product evaluation — Part 6:
Evaluation modules.
4 Definitions
For the purposes of this part of ISO/IEC 14598, the following definitions apply.
4.1 evaluation method: a procedure describing the action to be performed by the evaluator in
order to obtain the result for the specified measurement or verification applied on the specified
product components or on the product as a whole.
4.2 evaluation report: the document that presents evaluation results and other information
relevant to an evaluation.
4.3 evaluation records: documented objective evidence of all activities performed and of all
results achieved within the evaluation process.
: the person or organisation that requests an evaluation.
4.4 evaluation requester
4.5 evaluation tool: an instrument that can be used during evaluation to collect data, to perform
interpretation of data or to automate part of the evaluation.
NOTE  Examples of such tools are source code analysers to compute code metrics, CASE tools to
produce formalised models, test environments to run the executable programs, checklists to collect
inspection data or spreadsheets to produce syntheses of measures.
4.6 evaluator: the organisation that performs an evaluation.
NOTE  An evaluator may, for example, be a testing laboratory, the quality department of a software
development organisation, a government organisation or a user.
4.7 software product developer: the person or organisation that manufactures a software
product.
___________
1)  To be published.
©
ISO/IEC ISO/IEC 14598-5:1998(E)
4.8 software product evaluation: technical operation that consists of producing an assessment of
one or more characteristics of a software product according to a specified procedure.
NOTE 1  This definition can be compared to that of testing in ISO/IEC Guide 2. However, in this part
of ISO/IEC 14598, the term evaluation is preferred in order to avoid confusion with the notion of
testing widely accepted in the field of software engineering.
NOTE 2  Software product evaluation is not necessarily conformity testing (as defined in ISO/IEC
Guide 2, 13.3.2) in the context of a certification scheme. However, conformity testing can be part of
an evaluation.
5 Evaluation Concepts
5.1 General aspects
The quality of software products can be described in terms of quality characteristics as defined in
ISO/IEC 9126. However, the state of the art in software measurement is such that, in general, the
direct measurement of these characteristics is not practical. What is possible is to assess these
characteristics based on the measurement of lower abstraction attributes of the product.
In this context, the evaluator can use his or her experience in software engineering to make the
assessment. This might reduce the objectivity of the evaluation. Another aspect to be considered is
the possibility of using non-deterministic evaluation methods; although precisely defined, such a
method can require the evaluator to make choices which cannot be pre-defined.
NOTE  An example of a non-deterministic evaluation method is the one that consists of translating a
specification component of the product into a formal model and of performing performance or
reliability evaluation of this model; the translation phase involves many choices to be made by the
evaluator.
Therefore, provisions in this part of ISO/IEC 14598 are provided to maintain the level of objectivity
of evaluation as high as possible in all circumstances. These provisions bear on the organisation of
reviews of intermediate and final evaluation results and the keeping of records of the evaluation
process.
5.2 Evaluation starting point
5.2.1 Initial agreement
The evaluation of a software product occurs when the requester of the evaluation requests the
evaluator to perform an evaluation of this software product.
NOTE  When requesting the evaluation, the requester expresses evaluation requirements which are
analysed by the evaluator. The requester and the evaluator subsequently agree on the evaluation
specification.
©
5.2.2 Parties involved in the evaluation
Potential requesters of evaluations are, for example,
- software developers,
- software suppliers,
- software acquirers,
- software users,
- system integrators in their role of software acquirers.
Potential evaluators are, for example,
- third party testing laboratories,
- testing entities within software producing or distributing organisations,
- testing entities within software buying or using organisations,
- testing entities within system integration organisations,
- organisations making comparisons between products.
In some cases, the developer of the software product is involved in the evaluation even if the
developer is not the requester of the evaluation.
5.3 Characteristics of the evaluation process
A principal objective of the evaluation process described in this part of ISO/IEC 14598 is to
promote the following desirable evaluation process characteristics:
- repeatability: repeated evaluation of the same product to the same evaluation specification by the
same evaluator should produce results that can be accepted as being identical,
- reproducibility: evaluation of the same product to the same evaluation specification by a
different evaluator should produce results that can be accepted as being identical,
- impartiality: the evaluation should not be biased towards any particular result,
- objectivity: the evaluation results should be factual, i.e. not coloured by the feelings or the
opinions of the evaluator.
NOTE  Evaluations of the same product can be conducted with different evaluation specifications.
They are therefore not comparable and may lead to different results.
5.4 Evaluation process
The evaluation process (see clause 6) consists of a set of activities which are conducted in co-
operation with the requester and the evaluator. These activities are performed on the basis of data
provided by the requester and the evaluator or produced by other activities. They produce data
which is used by other activities or which is the result of the evaluation process.
©
ISO/IEC ISO/IEC 14598-5:1998(E)
The activities are designed to take into account the following issues:
- objectives vary from one evaluation case to another since software products are developed to
fulfil varied requirements and an evaluation requester may agree particular evaluation
requirements (see 6.2.1),
- software products are composed of components, the form and nature of which depend on
development methods which can be very different,
- possible evaluation techniques are numerous and need to be selected taking into account the
objectives of the evaluation and the composition of the product.
All these considerations impose a high flexibility for the process.
5.4.1 Evaluation activities
The evaluation process (see clause 6) comprises the five activities listed below:
- establishment of evaluation requirements (see 6.2.1);
- specification of the evaluation based on the evaluation requirements and on the description of the
product provided by the requester (see 6.3.1);
- design of the evaluation which produces an evaluation plan on the basis of the evaluation
specification; this activity takes into account the components of the software product to be
evaluated and the evaluation methods proposed by the evaluator;
- execution of the evaluation plan which consists of inspecting, modelling, measuring and testing
the products and its components according to the evaluation plan; these actions can be performed
using software tools (which are usually provided by the evaluator); the actions performed by the
evaluator are recorded and the results obtained are put in a draft evaluation report;
- conclusion of the evaluation, which consists of the delivery of the evaluation report and the
disposal by the evaluator of the product evaluated as well as its components when they have
been transmitted independently.
5.4.2 Input to the evaluation process
The requester provides the requester’s requirements which are an initial version of the evaluation
requirements.
The requester provides, during the evaluation, the following input to the evaluation process:
- the product description,
- the product components.
The product description identifies the software product as well as its components submitted for
evaluation.
©
NOTE 1  The product may include documents related to planning, process or development methods
used for its production. A planning document may include schedule, organisation structure or
estimated costs.
NOTE 2  If the requester is a user, he or she should agree with the developer to support the evaluator
and may require the developer to deliver to the evaluator the description of the software component
and software product to be evaluated.
The evaluator provides the following input to the evaluation process:
- pre-defined evaluation specifications,
- evaluation methods and
- evaluation tools.
5.4.3 Output of the evaluation process
During the evaluation process, the evaluator provides the following output products:
- evaluation records, including evaluation plan and records of evaluation actions,
- the draft evaluation report, including evaluation requirements, evaluation specification and
synthetised evaluation results,
- the reviewed evaluation report.
The evaluation requirements, specification and plan are the intermediate products of the evaluation
process. The evaluation records and evaluation report are the final products of the evaluation
process.
The evaluation requirements describe the objectives of the evaluation; in particular, quality
requirements for the product are described.
The evaluation specification defines all analyses and measurements to be performed on the product
and on its components. The components of the product that will be analysed and measured are
identified.
The evaluation plan describes operational procedures needed to implement the evaluation
specification; in particular all the methods and tools to be used in the evaluation are described.
The evaluation records consist of the evaluation plan and a detailed account of actions performed by
the evaluator while executing the evaluation plan; these records are kept by the evaluator.
NOTE 1  The evaluation records are kept in order to allow re-processing of the evaluation results.
The evaluation report contains evaluation requirements, the evaluation specification, results from
the measurements and analyses performed and any other information necessary to be able to repeat
or reproduce the evaluation. The evaluation report is first issued as a draft for review. When in final
form, it is delivered to the requester.
©
ISO/IEC ISO/IEC 14598-5:1998(E)
NOTE 2  The figure below gives an overview of the process described above. The information flow
between activities is identified.
Requester's
Evaluation
Requirements
Establishment Requirements
of Evaluation
Requirements
Evaluation
Specification
Specification
of the
Evaluation
Pre-defined
Product
Evaluation
Evaluation
Evaluation
Description
Design of
Specifications
Records
Plan
the
Record of
Evaluation
Evaluation
Evaluation
Actions
Evaluator's
Methods
Evaluation Tools
Input
Execution
of the
Evaluation
Draft Evaluation
Report
Reviewed
Synthetised
Product
Evaluation
Evaluation
Conclusion
Components
Report
Results
of the
Evaluation
Requester's
Input
Figure 1 — The evaluation process
5.5 Relations between evaluation and life-cycle
Evaluation of a software product can be performed within the context of any life-cycle process as
defined in ISO/IEC 12207. In particular, evaluation can occur within one of the acquisition, supply,
development, operating or maintenance processes.
The decision as to whether a software product evaluation is to be performed may be taken as early
as possible in the product development process. If this is done right at the beginning of the
development process, it is possible to build into the software development process the
measurements and tests to be performed for the evaluation. This would ensure the maximum
likelihood for the product to satisfy all requirements concerning the evaluation results, as well as
minimising the risk of extra, unexpected costs being incurred.
When the requester is the product developer, early contact with the evaluator to discuss the
intention of submitting a product for evaluation would also help the developer to anticipate any
special needs (such as particular documents or evidence which might be required) which the
evaluators could have.
It is possible that some (or even all) of the evaluation actions will have to be done on site rather than
at the evaluator’s. In this case, the actions will still be controlled by the evaluator to ensure that the
results are impartial.
For very large, complex software projects it would be beneficial for the developer to have
continuous, detailed co-operation with the evaluator during the whole development of the product to
©
minimise the duration and cost of the evaluation process. This co-operation should be such that it
does not reduce the impartiality of the evaluator.
6 Evaluation process requirements
6.1 General requirements
6.1.1 Organisation and quality system
In order to satisfy the characteristics expressed in 5.3, i.e. repeatability, reproducibility, impartiality
and objectivity of the evaluation results, the evaluator shall act in an organisational context that
provides all necessary assurance to obtain sufficient quality for its activities. In order to satisfy this
requirement, the evaluator’s organisation may comply with the requirements prescribed in ISO/IEC
Guide 25.
6.1.2 Requester’s responsibilities
The responsibilities of the requester of the evaluation shall be
- to establish necessary legal rights in the software product for the purpose of the evaluation,
- to provide information necessary for identification and description of the product,
- to state initial evaluation requirements and to negotiate with the evaluator to determine the actual
evaluation requirements; these requirements for the evaluation should comply with relevant
regulations and standards,
- to state confidentiality requirements concerning the information submitted to the evaluation,
- to act, whenever necessary, as an intermediary between the developer and the evaluator,
- to provide the evaluator, whenever necessary, with suitable access to computers and other
equipment used for development and for operational use of the software product,
- to provide, whenever necessary, support to the evaluator, including training and access to
suitable staff,
- to ensure the timely supply, whenever necessary, of the software product, its description and
components, including documentation and other material,
- to inform, whenever necessary, the evaluator of any factor that might invalidate the evaluation
results.
©
ISO/IEC ISO/IEC 14598-5:1998(E)
6.1.3 Evaluator’s responsibilities
The responsibilities of the evaluator shall be
- to check that the requester has the sufficient legal rights in the software product for the
evaluation to be performed; to do so, the evaluator may require an attestation from the requester,
- to keep the confidentiality as required, of all the information provided by the requester,
including, for example, the product under evaluation, the evaluation records and the evaluation
report,
- to provide qualified and trained staff to conduct the evaluation,
- to provide the evaluation tools and technology,
- to conduct the evaluation in accordance with the evaluation requirements,
- to maintain records of any work performed during the evaluation which has an impact on the
evaluation results,
- to ensure timely delivery of the evaluation report to the requester,
- to provide the visibility into the conduct of the evaluation to the extent requested by the
requester.
6.2 Establishment of evaluation requirements
6.2.1 Purpose of the establishment of evaluation requirements
The purpose of the establishment of evaluation requirements is to describe the objectives of the
evaluation. Such objectives relate to the software product’s intended use and its associated risks (for
example, see annex B). Several viewpoints may be considered: those of different product users such
as the product acquirer, its supplier, its developer, its operator or its maintainer.
6.2.2 Elaboration of the evaluation requirements
The activity of analysis of the evaluation requirements is composed of the following sub-activities:
- proposing requester’s requirements by the requester;
- expressing the extent of the coverage of the evaluation by the requester;
- supporting the requester in analyzing the reason for evaluation and in describing the
evaluation requirements by the evaluator;
- explaining the extent of confidence and stringency of evaluation by the evaluator;
- agreeing on the evaluation requirements.
The requester of the evaluation shall provide the requester’s requirements which are an initial
version of the evaluation requirements. The evaluator should support the requester in analysing the
reasons for evaluating the product and in describing the evaluation requirements.
©
The application domain for the product submitted to the evaluation should be considered, as well as
the general description of its purpose. Critical issues such as safety, security, economic or
environment aspects may be taken into account. Applicable regulations and laws should be
considered.
In the requester’s requirements, the requester shall express requirements on how extensive the
coverage of the evaluation should be. At the same time the evaluator should ensure that the
evaluation be stringent enough to provide real confidence in the software product quality.
Therefore, the evaluator and the requester shall agree on the evaluation requirements as a pre-
requisite for continuing with the evaluation process.
NOTE  For certification of a software product or of its components the requester of the evaluation
specifies the normative document containing the requirements for the product.
6.2.3 Contents of the evaluation requirements
The evaluation requirements shall contain a general description of the application domain for the
product submitted to the evaluation. A general description of the product purpose shall be provided.
The evaluation requirements shall also consist of a list of quality requirements referring, for
example, to quality characteristics as defined in ISO/IEC 9126. In this context, sub-characteristics
may also be used. When a requirement refers to a characteristic not defined by ISO/IEC 9126, a
reference to authoritative literature defining it shall be made and the requester and the evaluator
should explicitly state their mutual understanding of this characteristic.
The relative importance of each quality characteristic in the evaluation requirements should be
given. This applies when some part of the product needs to be evaluated with different evaluation
requirements. To express this importance, the notion of an evaluation level as suggested in annex B
may be used.
For each requirement in the evaluation requirements, the specification of information to be
contained in the software product and in its components to be evaluated shall be provided. This
specification should, as much as possible, refer to a software engineering standard. In addition, the
type of formalism used in the components or the type of software development methods used to
produce them may be specified.
NOTE  The extent and the form of information required for the evaluation can be related to the cost
of the evaluation, on the one hand, and to the importance of a specific quality requirement on the
product, on the other hand.
6.2.4 Approbation and reporting
The evaluation requirements shall be approved as a result of a joint review between the requester
and the evaluator.
The evaluation requirements shall be included in the evaluation report and in the evaluation records.
©
ISO/IEC ISO/IEC 14598-5:1998(E)
6.3 Specification of the evaluation
6.3.1 Purpose of the specification of the evaluation
The purpose of specifying the evaluation shall be to define the scope of the evaluation and the
measurements to be performed on the product submitted for evaluation and on its various com-
ponents. The level of details in the evaluation specification should be such that, on its basis, the
repeatability and the reproducibility of the evaluation be ensured.
NOTE 1  The evaluation specified may be non-deterministic. In that case, it should be such that the
results obtained from repeated or reproduced evaluations be consistent.
However, the evaluation specification should not contain proprietary information of the evaluator.
NOTE 2  The evaluation report, which contains the evaluation specification, is delivered to the
evaluation requester who may disclose it to other parties. Therefore it would not be advisable for the
evaluator to try to protect some proprietary information.
6.3.2 Elaboration of the evaluation specification
The activity of specifying the evaluation is composed of three sub-activities:
- analysing the product description,
- specifying the measurements to be performed on the product and its components,
- verifying the specification produced with regards to the evaluation requirements.
NOTE  The verification sub-activity may be conducted in parallel with the others in order to identify
potential problems as early as possible.
6.3.2.1 Analysing the product description
The requester shall provide a description of the product submitted for evaluation. The goal of this
description is
- to allow to define the scope of the evaluation, i.e. the identification of those software product
components that are to be considered as part of the product and the identification of those
software product components that are not to be considered as part of the product and which are
only referred to for the ease of understanding the product.
NOTE 1  Such an identification may be based on specifying which parts of the documents belong to
the product, which function is implemented or not by the product.
NOTE 2  Defining the scope of the evaluation is important when the software product submitted for
evaluation is embedded in a system consisting of hardware, other software products, networks or
organisations because the separation between such products is not always obvious.
- to give to the evaluator the identification of product components submitted for evaluation, to
understand their structure and to identify the information provided as well as how to access it.
©
This description shall contain the list of product components actually submitted for evaluation, a
rationale about the structure of the product and a list of product related documents. The components
listed may contain other smaller components which need not be listed. For each component and
product related document in the lists, the following information shall be provided:
- description of the nature of the component,
- information about formalisms used within the component,
- information about the size of the component,
- relationship with other components,
- information about availability of the product component to the evaluator.
In any case, reference to appropriate software engineering standards should be made.
The evaluator shall check that the product description conforms to the above mentioned
requirements.
The evaluator shall analyse the rationale provided as well as the description of the components in
order to identify their relationship with the components identified in the evaluation requirements.
NOTE 3  In the evaluation requirements, components may be specified from a theoretical point of
view, with regard to quality characteristics to be assessed. In the product description, actual
components are listed. It may happen that some actual components of the product contain information
that the evaluation requirements specifies as being in several components.
NOTE 4  This information is needed in order to identify which evaluation can be performed. This will
be used, together with the evaluation requirements, to build the evaluation specification.
NOTE 5  The analysis of the product description may be improved by consultation with the developer
of the product. This would provide an opportunity for the evaluator to establish whether an evaluation
to the depth required will be possible, by performing a brief audit.
6.3.2.2 Specifying measurements
The evaluator shall allocate the evaluation requirements on the product itself and the various
components identified in the product description. This should lead to a decomposition of the
evaluation requirements into e.g. sub-characteristics. The result of this decomposition may be
different for different components submitted for evaluation. At this stage, some components listed
in the product description may not be considered further.
The evaluator shall then specify the measurements intended to be used to assess the characteristics,
sub-characteristics and attributes of the product and the selected components. These specifications
should be formulated as a combination of the following types of statements:
- a formalised specification of a metric to be applied on the product or on an identified set of
components, together with instruction to present the resulting measures in the evaluation report,
- a reference to statements in a product component specifying software requirements that will be
verified and the specification of the procedure to be used to verify these requirements,
©
ISO/IEC ISO/IEC 14598-5:1998(E)
- the specification of a requirement for the software product which is either missing in the
software requirement documents or needs to be explained in more detail for the evaluation and
the specification of the procedure to be used to verify this requirement,
- a reference to statements in identified standards or regulations where additional software require-
ments are provided and the specification of the procedure to be used to verify these
requirements.
For each of these statements, reference should be made to the nature of and to the formalisms used
in the components to be measured or verified.
For this task, the evaluator may use pre-defined evaluation specifications. These elementary speci-
fications should be in the form of evaluation module specifications as recommended in ISO/IEC
14598-6.
6.3.2.3 Verifying the evaluation specification
The evaluator shall perform a verification of the evaluation specification with regard to the
evaluation requirements.
The evaluator shall check that the components listed in the product description provide all the
necessary information to perform the evaluation according to the evaluation requirements. The
evaluator shall also verify that the measurements and verifications specified are sufficient to meet
the objectives of the evaluation as expressed in the evaluation requirements.
The first check can lead to the identification of missing information in the components listed in the
product description. This may be solved in one of the following ways:
- a reference to a product component containing the missing information shall be added in the
product description; this means that the requester will provide this component together with the
others for the performing of the evaluation;
- the objectives of the evaluation shall be reduced, which means that the evaluation requirements
are revised.
The second check aims at confirming that the measurements and verifications proposed in the
evaluation specification are consistent with the technical state of the art. This may be done in one of
the following ways:
- by identifying relevant measurement standards,
NOTE  Such standards may be evaluation modules as prescribed in ISO/IEC 14598-6.
- by providing a detailed justification, referencing appropriate authoritative literature in the field;
this justification shall be included in the evaluation specification.
6.3.3 Contents of the evaluation specification
The evaluation specification shall comprise:
- the scope of the evaluation referring to the product components as identified in the product
description,
- a cross-reference between the information needed to perform the evaluation and the product
components and other related documents listed in the product description,
©
- a specification of measurements and verifications to be performed and the references to product
components on which they are to be performed,
- a mapping between the specification of measurements and verifications and the evaluation
requirements together with the reference to standards or the justification for each measurement
or verification listed.
6.3.4 Approbation and reporting
The evaluation specification shall be approved as a result of a joint review between the requester
and the evaluator.
The evaluation specification shall be included in the evaluation report and in the evaluation records.
In addition, any modification of the evaluation requirements as specified in sub-clause 6.3.2.3 shall
be reported in the evaluation records.
6.4 Design of the evaluation
6.4.1 Purpose of designing the evaluation
The design of the evaluation shall document the procedures to be used by the evaluator to perform
the measurements specified in the evaluation specification. The evaluator shall produce an
evaluation plan that describes the resourc
...