ISO/IEC 14598-4:1999

Software engineering - Product evaluation - Part 4: Process for acquirers

Ingéniérie du logiciel — Évaluation du produit — Partie 4: Procédé pour les acquéreurs

General Information

• Status: Withdrawn
• Publication Date: 13-Oct-1999
• Withdrawal Date: 13-Oct-1999
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 6 - Software Product and System Quality
• Current Stage: 9599 - Withdrawal of International Standard
• Start Date: 09-Oct-2012
• Completion Date: 30-Oct-2025

Relations

• Consolidated By: ISO 8835-2:2007 - Inhalational anaesthesia systems - Part 2: Anaesthetic breathing systems
• Effective Date: 06-Jun-2022
• Revised: ISO/IEC 25041:2012 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Evaluation guide for developers, acquirers and independent evaluators
• Effective Date: 03-Dec-2011

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 14598-4
First edition
1999-10-01
Software engineering — Product
evaluation —
Part 4:
Process for acquirers
Ingéniérie du logiciel — Évaluation du produit —
Partie 4: Procédé pour les acquéreurs
Reference number
©
ISO/IEC 1999
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 1999
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 � CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 734 10 79
E-mail [e-mail]
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 1999 – All rights reserved

Contents
1 Scope .1
2 Conformance.1
3 Normative references .2
4 Terms and definitions .2
5 Software product evaluation - General considerations.2
5.1 Correlation between evaluation and acquisition processes.2
5.2 Inputs to the evaluation process.3
5.2.1 System requirements .3
5.2.2 Integrity level requirements.4
5.2.3 Software requirements specification.4
5.2.4 Evaluations performed by others.5
5.3 Tailoring.5
6 Evaluation during acquisition of “off-the-shelf” software products.6
6.1 Step 1 - Establish evaluation requirements .7
6.1.1 Establish the purpose and scope of the evaluation.7
6.1.2 Specify evaluation requirements .8
6.2 Step 2 - Specify the evaluation.9
6.2.1 Select metrics.9
6.2.2 Select the evaluation methods.10
6.2.3 Evaluations performed by others.11
6.3 Step 3 - Design the evaluation.11
6.4 Step 4 - Execute the evaluation.13
6.4.1 Execute the evaluation methods.13
6.4.2 Analyze the evaluation results .13
6.4.3 Draw conclusions .14
7 Evaluation during acquisition of custom software and modifications to existing software .14
7.1 Step 1 - Establish evaluation requirements .15
© ISO/IEC 1999 – All rights reserved iii

7.2 Step 2 - Specify the evaluation.15
7.3 Step 3 - Design the evaluation.15
7.4 Step 4 - Execute the evaluation.15
Annex A (informative) Definitions from other standards .16
Annex B (informative) Tables.21
Annex C (informative) Figures .25
Annex D (informative) Evaluation methods.27
Annex E (informative) Example of staged evaluation process.32
Bibliography .34
iv © ISO/IEC 1999 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO/IEC 14598 may be the subject of
patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
International Standard ISO/IEC 14598-4 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts under the general title Software engineering — Product evaluation:
� Part 1: General overview
� Part 2: Planning and management
� Part 3: Process for developers
� Part 4: Process for acquirers
� Part 5: Process for evaluators
� Part 6: Documentation of evaluation modules
Annexes A to E of this part of ISO/IEC 14598 are for information only.
© ISO/IEC 1999 – All rights reserved v

Introduction
Software has become increasingly pervasive. The demand for added software functionality and faultless software
products has grown as more processes are automated to take advantage of the power of the computer. Today's
modern systems are so complex, that they are unable to perform their functions without software. The use of
commercially available “off-the-shelf” software products is accelerating as the variety of available products grows
and the rapid evolution of software engineering technology reduces reliance on custom-coded software. The
object-oriented development approach, which is based on the development of an application system through the
extension of existing libraries of self-contained units, has also reduced requirements for custom-coded software.
This has led to intense focus on concomitant software product quality or self-contained software unit quality.
Development of custom software is prone to rework as a result of failure to meet user requirements. The use of
custom software may also require a larger than anticipated effort with respect to deployment, implementation,
training, and maintenance support activities. Acquisition of commercial “off-the-shelf” software products, or, reuse
of in-house existing software products, is also not without risk. Problems can be encountered because the “off-the-
shelf” software products may require customizing; testing and analysis requirements may be large; product
maintenance and support is doubtful when the product becomes obsolete or revised; it may be difficult to integrate
software products into larger systems; and the quality of the product may not be consistent with the required quality
of the target system.
Commercial “off-the-shelf” software products are extremely varied. They can be:
� a) used as stand-alone products (i.e., payroll, accounting software, consumer software or 'shrink-wrapped
software' [i.e., word-processing software, spreadsheets]);
� b) integrated as components into a larger system which consists of other software and hardware
components (i.e., operating system, relational data base management system, graphical users interface
[GUI]);
� c) embedded in hardware (i.e., communication data link, programmable array logic [PAL]);
� d) embedded as part of a configurable software/hardware system that can be used for the development of a
specific application (i.e., distributed control system);
� e) CASE tools used to support the software development and maintenance process (i.e., compilers,
configuration management tools).
Errors in stand-alone software products can impact productivity, cause financial loss, or cause unnecessary rework.
Software components can be difficult to integrate, affect the reliability of the overall system, or be incompatible with
system objectives. CASE tools may introduce an error into a product under development or be difficult to use.
It is therefore essential to be able to evaluate the quality of software products during acquisition, or when making a
decision on reusing an existing software product or component. Evaluation may be used to accept or reject a single
product, or to select one product, from among alternative products, that meets the quality requirements established
for the target application. The level of rigor of the evaluation process is necessarily commensurate with the integrity
requirements for the product. The highest level of rigor is required when performing evaluation of software products
that are mission critical.
vi © ISO/IEC 1999 – All rights reserved

INTERNATIONAL STANDARD © ISO/IEC ISO/IEC 14598-4 :1999(E)
Software engineering — Product evaluation — Part 4: Process for
acquirers
1 Scope
This part of ISO/IEC 14598 contains requirements, recommendations and guidelines for the systematic
measurement, assessment and evaluation of software product quality during acquisition of “off-the-shelf” software
products, custom software products, or modifications to existing software products. It uses the software quality
model described in ISO/IEC 9126-1; expands on the general process for evaluating software quality that is defined
in ISO/IEC 14598-1; and uses the process for acquisition that is defined in ISO/IEC 12207. It can be used in
conjunction with ISO/IEC 12119, ISO/IEC 14598-2 (new), ISO/IEC 14598-3 (new) and ISO/IEC 14598-6. The steps
of the evaluation process are similar between this part of ISO/IEC 14598 and ISO/IEC 14598-5, but the context of
use is quite different. In the case that acquirers entrust second or third parties with evaluations, ISO/IEC 14598-5 is
required to be applied. In the case that acquirers require third party testing of software packages against the quality
requirements for the package, ISO/IEC 12119 may be applied.
The evaluation process described in this part of ISO/IEC 14598 also helps to meet the objectives of deciding on the
acceptance of a single product, or for selecting a product from among alternate products. The evaluation process
may be tailored to the nature and integrity level of the application. It is also sufficiently flexible to accommodate the
wide range of forms and uses of software products in a cost-effective manner.
This part of ISO/IEC 14598 is intended for, but not limited to, project managers, system engineers, development
and maintenance software engineering staff and end users who plan to acquire software products, and also
suppliers who provide such products.
The target software products of the evaluation process in this part of ISO/IEC 14598 can be integrated into a larger
system as components or can be used stand-alone. They are classified as:
a) Commercial off-the-shelf software products;
b) Existing software products developed or acquired for other applications, or for a wide range of common
applications;
c) Custom software products or modifications to existing software products.
The evaluation process that is defined in this part is also applicable to CASE tools. Because evaluation of CASE
tools is specifically addressed in ISO/IEC 14102, CASE tools are considered out of scope of this part of ISO/IEC
14598.
ISO/IEC 14598-4 is designed to work in partnership with other standards. For systems with high integrity
requirements, additional requirements may be included in the evaluation process described in ISO/IEC 14598-4,
that are derived from sector-specific standards, e.g., IEC 880, DOA-167A , MOD-55, etc.
2 Conformance
Because of the freedom of choice afforded to the user by the general nature of its recommendations, a simple
claim of compliance with this part of ISO/IEC 14598 is not valid. Any organization imposing this part of ISO/IEC
14598 as a condition of trade is responsible for specifying and making public the evaluation process that meets the
mandatory objectives specified in 6.1.1. The specified evaluation process constitutes the terms for compliance for a
given application of this part of ISO/IEC 14598. All activities of clauses 6 and 7 shall be considered for applicability.
© ISO/IEC
Requirements on the evaluation process can also be established contractually during execution of the acquisition
process. Compliance with the evaluation process described in this part of ISO/IEC 14598 is then easily established.
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this part of ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this part of ISO/IEC 14598 are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO/IEC 9126-1, Information technology - Software quality characteristics and metrics – Part 1: Quality
1)
characteristics and subcharacteristics.
ISO/IEC 12207:1995, Information technology - Software life cycle processes.
ISO/IEC 14598-1:1999, Information technology – Software product evaluation - Part 1: General overview.
ISO/IEC 14598-5:1998, Information technology – Software product evaluation - Part 5: Process for evaluators.
ISO/IEC 15026:1998, Information technology – System and software integrity levels.
4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the following definitions apply. Key definitions from other standards
used in this part of ISO/IEC 14598 are reproduced in Annex A for convenient reference.
4.1
commercial-off-the-shelf software (COTS)
software defined by a market-driven need, commercially available, and whose fitness for use has been
demonstrated by a broad spectrum of commercial users
NOTE See also definition in IEEE Std 1062-1993.
4.2
custom software
software developed for a specific application from a user requirements specification
4.3
existing software
software that is already developed and available; is usable either “as is” or with modifications; and which is
provided by the supplier, acquirer, or a third party
NOTE See also definition of “off-the-shelf product” [ISO/IEC 12207: 1995].
5 Software product evaluation - General considerations
5.1 Correlation between evaluation and acquisition processes
The acquisition process activities (defined in ISO/IEC 12207) summarized below, are combined with the general
evaluation process activities (defined in ISO/IEC 14598-1) in Clauses 6 and 7. Clause 6 focuses on the application
of the evaluation of end product quality during acquisition of COTS products. Clause 7 focuses on the application of
the evaluation process during acquisition of custom software or modifications to existing software.
1)
To be published. Until this part is published ISO/IEC 9126 :1991 should be used.
© ISO/IEC
a) Initiation - identification of software requirements for the product to be acquired, the acquisition plan, and the
acceptance strategy and criteria;
b) Request-for-proposal (-tender) preparation - specification and documentation of acquisition requirements;
c) Contract preparation and update - supplier selection, contract preparation and negotiation, and contract
change control;
d) Supplier monitoring - evaluation activities performed during contract execution leading to acceptance and
delivery of the software product;
e) Acceptance and completion - activities performed during product acceptance and delivery of the final software
product.
Note, that, the general evaluation process defined in ISO/IEC 14598-1, is not defined as a process in ISO/IEC
12207, but as an elementary function equivalent to the “check” portion of the plan-do-check-act (PDCA) cycle that
is implemented by each life-cycle process. However, the general evaluation process may be implemented in any
ISO/IEC 12207 process (i.e., development, maintenance, acquisition, validation); it is therefore at a different level of
abstraction from the sense of “process” used in ISO/IEC 12207.
This distinction is important when implementing the general evaluation process. The acquirer needs to define both
the evaluation process and the acquisition process that he/she will follow to achieve the evaluation requirements
during acquisition. In the context of larger system development, the acquisition and evaluation activities to be
followed need to be integrated with other development and integration activities, and identified in a project
measurement plan as specified in ISO/IEC 14598-2 (in preparation); i.e., specific acquisition implementation
considerations for evaluation include the following considerations:
� a software requirements specification required for evaluation can form the basis for acquisition requirements
required for the request-for-proposal (-tender);
� separate preliminary evaluation activities may be needed to preselect software products and suppliers;
� supplier and product information requirements for evaluation need to be specified in the acquisition
requirements or during contract preparation;
� evaluation activities can be executed as part of proposal evaluation, during monitoring of contract execution,
as part of product development, as part of formal product acceptance, or after product delivery.
5.2 Inputs to the evaluation process
5.2.1 System requirements
The starting point for determining evaluation requirements for target software begins with the overall system
requirements. The system requirements identify the user, user goals, tasks and characteristics, including the
environment in which the product is to be used, in addition to functional and other requirements for the product or
system. They form the basis for subsequent system architecture design, specification of software requirements,
and software architecture design. Relevant legal and regulatory requirements need to be identified at this stage as
they impact on the rigor and formality of the acquisition and evaluation processes.
During system requirements decomposition and design, system requirements are allocated to hardware and
software configuration items, and to user operations including system procedures. Design activities during a system
development life-cycle result in subsequent decisions to acquire or reuse “off-the-shelf” software products. Some of
the evaluation work is actually a part of these design activities, since it plays a role in the decision making process.
Evaluation of software products to be acquired is performed separately. During system integration and testing of
the end product, software configuration items are integrated with other software, and with the hardware
configuration items (Refer to ISO/IEC 12207). Figure 1 shows the larger systems engineering context for evaluation
and acquisition.
Candidates for use and acquisition in this part of ISO/IEC 14598 are software products which can be integrated into
a larger system as components or which can be used stand-alone. They are classified as:
© ISO/IEC
a) Commercial-off-the-shelf software products;
b) Existing software products developed or acquired for other applications, or for a wide range of common
applications;
c) Custom software products or modifications to existing software products.
In the case of software configuration items that are to be integrated into a larger system, software requirements
need to be defined for each item. In other cases, the system and the software configuration items coincide, and
may be considered equivalent.
Hardware configuration items to be acquired may contain software such as an operating system resident in
firmware (i.e. ROM, PROM). When the existing software forms an integral part of the hardware in this fashion it
usually needs to be evaluated with the hardware configuration item.
Initial
System Requirements
Mandate
System
Capture & System Design
Procedures
•user objectives & environment
•functional and other
requirements
•integrity level Non-Computer
Subsystems
Engineering
Computer Subsystems
Engineering
Subsystem
Integration &
Software Hardware
System
Engineering Engineering
Validation
System
Ergonomics & Product
Operation
Human Factors Evaluation &
Engineering Acquisition
Figure 1 — Systems engineering context for software product evaluation and acquisition
5.2.2 Integrity level requirements
If the software is critical in containing non-trivial safety, security, financial, environmental and societal risk of a
system within acceptable limits, its requred integrity level must have been established and properly documented
prior to procurement and evaluation. Guidance to the integrity level determination process is given in ISO/IEC
15026. The resulting integrity level determines how the software is dealt with in the evaluation process.
5.2.3 Software requirements specification
Software requirements shall be defined using an appropriate well defined quality model. For this purpose the
quality model and definitions in ISO/IEC 9126-1 should be used, unless there is a particular reason to use another
© ISO/IEC
model. This model defines six broad categories of characteristics of software in use: functionality, reliability,
usability, efficiency, maintainability and portability. These can be further broken down into subcharacteristics which
have measurable or assessable attributes.
Requirements should be defined in terms of external metrics (external metrics are defined in ISO/IEC 9126-2)
which directly relate to user needs and should be documented in a requirements specification. Documenting the
user needs can vary from producing an informal list of required functional and performance requirements to
preparing a complete requirements specification for the product (or system if the product is embedded). The
requirements specification may then form the basis for acquisition requirements used during the tendering step in
the acquisition process and the basis against which subsequent product evaluation is performed.
5.2.4 Evaluations performed by others
The scope of the present evaluation process can be reduced through access to the results of evaluation activities
performed by second or third parties as long as the results are trustworthy. Such evaluation activities can comprise
preexisting certifications, product evaluations and/or process assessments. For example:
� software engineering processes for product development may be standardized to meet the requirements of
ISO/IEC 12207, ISO 9000-3, or other sector-specific standards;
� the supplier’s quality system under which the software is developed may be certified to ISO 9001 requirements
by a third party;
� the software product may be evaluated by second or third parties to ISO/IEC 14598-5 or ISO/IEC 12119
requirements;
� the supplier’s software process capability for acceptable product development may be assessed by third
parties to ISO/IEC 15504-8 (in preparation);
� the software may be functionally evaluated as part of a larger system development phase;
� the software product may have been previously evaluated for another application with different integrity
requirements;
� evaluations on the product may have been performed by other parties within the organization through informal
or formal evaluation activities.
The additional costs and time required to obtain and interpret external evaluation results for the target application
may affect the feasibility of this method. It may still be necessary to consult the with the evaluator or supplier in
order to attain adequate confidence in the results of others.
NOTE Evaluation results of the supplier software engineering process, supplier quality system, or supplier capability alone are
not sufficient criteria to demonstrate that a software product contains the required quality characteristics. Other product
evaluation methods [e.g., such as those that specifically measure factors and attributes of quality that are appropriate to the
requirements of the end-user] need to be executed.
5.3 Tailoring
The evaluation process can be applied to a wide range of acquisition requirements, integrity requirements, and
evaluator objectives. For example:
� acquirers of software packages may wish to evaluate a software package using only ISO/IEC 12119;
� acquirers of software products may use ISO/IEC 14598-5 for independent evaluation;
� a small or sole acquirer may require a less formal evaluation process with minimum documentation of the
evaluation;
� for consumer software the evaluation process objective may simply be to select, test and acquire one product
from among a number of similar products. The formal acquisition process is then reduced to outright purchase,
and does not include contract preparation.
© ISO/IEC
The evaluation process should have the flexibility to accommodate the uniqueness of each application, to avoid
unnecessary work or work that adds no value, and to provide a practical means of establishing the requisite
confidence in the software. The required integrity level of the software largely determines the rigor and formality of
theevaluationprocess.
The acquisition process can also be tailored as for the evaluation process using the tailoring guidance given in
ISO/IEC 12207 and the required integrity level for the specific software product to be acquired. Acquisition of
complete software systems with high integrity requirements will usually invoke the full set of acquisition activities
and tasks, along with corresponding supply process activities and tasks, that are specified in ISO/IEC 12207.
Generally, as the integrity level increases, so should the amount of rigor, and the number of activities and tasks
associated with the acquisition process.
Table B.1 in Annex B shows an example of tailoring of integrated acquisition and evaluation process activities according to the
target software integrity level requirements.
6 Evaluation during acquisition of “off-the-shelf” software products
The general software product evaluation process defined in ISO/IEC 14598-1 consists of four major steps, which
are specifically implemented and refined to focus on the evaluation of end product quality during acquisition of “off-
the-shelf” products in this part of ISO/IEC 14598. However, this does not preclude evaluation of intermediate
products for specific quality characteristics. Hence, the details in the implementation of the steps differ, but are not
inconsistent, with the details described in ISO/IEC 14598-1. The evaluation process is summarized in Table 1
below in terms of its steps and key tasks, as well as inputs and outputs.
© ISO/IEC
Table 1 — Evaluation Process During Acquisition of “Off-the-Shelf” Products
Inputs Evaluation Key Tasks Outputs
Step
System/ Software Establish Specify objectives, purpose, and scope. Specify Evaluation
requirements evaluation rigor of evaluation. Identify inputs to evaluation. Requirements
requirements Identify evaluation performed, or to be Specification
(subclause 6.1) performed by others. Identify the acquisition
process to be followed and how evaluation input
requirements are to be communicated to the
supplier.
Evaluation Specify the Select metrics that correlate to the Evaluation
Requirements evaluation characteristics of the software product. Establish Specification
(subclause 6.2) rating levels. Select the most effective set of
evaluation methods. Establish procedures for
summarizing the results of the evaluation of
different quality characteristics and other
aspects that contribute to the assessment of
quality of a software product in a particular
environment.
Evaluation Design the Prepare an evaluation plan describing the Evaluation Plan
Specification evaluation evaluation methods, and the schedule for
(subclause 6.3) evaluation. Identify the tie points between
evaluation activities and acquisition activities.
Evaluation Plan Execute the Conduct the selected evaluation activities, and Evaluation
evaluation analyze and record the results to determine the Records and
(subclause 6.4) suitability of the software product(s). Analyze the Results
impact of identified deficiencies and options to
regulate the use of the product. Draw
conclusions with respect to the acceptability of
the product and the ultimate decision to buy or
not to buy.
6.1 Step 1 - Establish evaluation requirements
6.1.1 Establish the purpose and scope of the evaluation
Theevaluationprocess shall establish:
a) a set of software quality requirements using the quality model and definitions in ISO/IEC 9126-1, against which
the software product can be objectively evaluated for fitness for use;
b) the appropriate priority which should be placed on the software quality characteristics;
c) a systematic basis for the evaluation appropriate for the integrity level of the application, which involves
establishing requirements as to the level of rigor or detail required in the evaluation activities, as well as the
inputs to, and the outputs from the evaluation process;
NOTE Figure 2 provides an overview of the software product evaluation process. It shows the different views of the inputs to
the evaluation process and resulting outputs from the evaluation process, from an acquirer’s perspective.
d) the acquisition process to be followed and how evaluation input requirements are to be communicated to the
supplier;
NOTE See example of a combined evaluation and acquisition process in Figure C.1 in Annex C.
© ISO/IEC
e) the scope, purpose and objectives of the evaluation by considering:
1) whether the software product will be used for a specific application, for a collection of specific applications,
or for a generic range of applications;
2) whether any evaluation has been done by second or third parties, or whether any evaluation activities are
planned to be performed later.
6.1.2 Specify evaluation requirements
The evaluation requirements specification should identify:
a) users and their goals, tasks, and characteristics, and the user environment for the product;
b) the integrity level of the software application (risk incurred in case of software error) and hence the level of
rigor required for the evaluation process;
c) the regulatory requirements (what documentation is required to provide the regulator (if any) with assurance of
the product's quality) (see compliance in ISO 9126-1);
d) the product boundaries and interfaces, including interfacing requirements (i.e. the type of data passed through
the interface, the data format, the interface access mechanisms, failure/error handling, timing issues, interface
behavior issues, and interface state dependencies and transitions) to the software product (see interoperability
in ISO 9126-1);
e) integration requirements if the product is part of a larger system requiring integration with other components or
products;
f) software quality requirements which include:
1) distinction between mandatory and optional requirements;
2) all assumptions, exceptions, limitations, exclusions, or unresolved issues needed to interpret or
understand the requirements;
3) user requirements for all important quality characteristics and their priority (i.e. if maintainability is
considered important, then identify the specific maintainability requirements);
4) all design and environmental constraints; i.e. functional or performance limitations imposed by the use of
the software product, the level and complexity of integration of the software product with other existing
software, custom software, and the hardware in the user application;
5) all project management constraints; i.e. availability of resources and expertise to perform the evaluation
activities, schedule and budget allowance, possible dependencies or risks, key assumptions, or
assumptions about the evaluation effort itself;
6) rationale for the use of a quality model other than the one defined in ISO/IEC 9126-1.
g) supplier services to be assessed; i.e., support capability, application development capability, and training
capability;
h) special requirements to be assessed; i.e. specific technical feasibility issues or design implementation
questions;
i) evaluation requirements that are consistent with each other (i.e. no conflicting requirements) and consistent
with the integrity level of the application;
j) whether the product will be reused in future applications and the documentation necessary to support any
future evaluations of the product;
k) the acquisition process, and information required from suppliers during tendering;
OD - SE
PR UCT-IN U
VIEW
© ISO/IEC
l) evaluations performed by other second or third parties that could be used to reduce the evaluation effort for the
product.
NOTE The level of detail and completeness of the evaluation requirements specification directly affects the level of
completeness of the evaluation, i.e., an evaluation based on only preliminary requirements cannot be considered a full
evaluation, although the results may be used to direct further work in other phases, anticipate problems, or eliminate certain
software products or suppliers. These would normally be done prior to the main evaluation activities as a part of the appropriate
design or planning activity. Some evaluation work may also be required before the requirements are finalized.
VIEWS OF EVALUATION INPUTS
PRODUCT VIEW
User & Technical Software Quality
Cost
Requirements
Documents
Assessment
Evaluation Process
Quality System
Establish Evaluation Requirements
Operating
History
Engineering
Failure
Process
Specify the Evaluation
Information
Maintenance
Prototyping/Testing
Process
Design the Evaluation
Execute the
Evaluation
Additional Verification
Necessary Level of Confidence Constraints on
That Quality Requirements Met Usage of Product Activities Required
VIEW OF EVALUATION RESULTS
.
Figure 2 — Software product evaluation process overview from acquirer’s perspective
6.2 Step 2 - Specify the evaluation
The evaluation specification should be documented so that the evaluation process can be repeated by
appropriately qualified personnel with repeatable results.
6.2.1 Select metrics
The evaluation specification should identify:
a) the characteristics of the product to be evaluated;
PROCESS VIEW
© ISO/IEC
b) the external metrics that correlate to measurable aspects of quality when the software is used ( Table B.2 in
Annex B shows an example of external metrics and possible acceptance criteria. Notice it is up to the user to
specify actual threshold acceptance numbers based on experience as there are no hard and fast rules for
some of these numbers);
c) the “quality in use” metrics correlating to the user’s view of the quality of the system containing the software
(see example of quality in use metrics in Table B.3 in Annex B);
d) sufficient criteria for metrics to describe their acceptable range (e.g. how much operating history is necessary
to provide a reasonable degree of assurance for a given quality characteristic and a given integrity level (see
Annex D.4 and D.5 for specific details on operating history));
e) any packaged evaluation modules;
f) the level of coverage relative to the evaluation requirements that is necessary after review of any previous
evaluations conducted by others;
g) checklists to be answered by the evaluation;
h) a list of examples which would help to answer the questions;
i) test cases to be used;
j) data to be collected and analyzed, and their format;
k) the specific evaluation methods to be used, which include review or assessment of one or more of:
1) Software product user and technical documentation (including on-line documentation) (see Annex D.1);
2) Software product evaluation based on supplier courses and training (see Annex D.2);
3) Software engineering process, including intermediate software products (see Annex D.3);
4) Product operating history with supplier (see Annex D.4);
5) Product operating history with customers (see Annex D.5);
6) Supplier capability, support, and quality system (see Annex D.6);
7) Prototyping or other evaluation methods (see Annex D.7);
8) Product deficiency lists and related information (usually found on web sites).
l) the methods for assessing the evaluation results;
m) suitable methods of ranking the assessments to allow selection, when selecting a product from among similar
products;
n) any rating schemes useful for comparing more than one software product. The rating scheme may be weighted
in accordance with the priority of the quality characteristics.
6.2.2 Select the evaluation methods
A combination of evaluation methods should be specified to allow selection of the product or to establish the
product's fitness for use. Areas to be evaluated include:
a) whether some of the considerations may be mutually conflicting (e.g. "Are costs of the selected [assessment]
methods within budget?” may not be compatible with "Do the methods collectively address all of the evaluation
requirements?"). In this case it is up to the evaluator to make the necessary tradeoffs based on the
prioritization of evaluation requirements;
© ISO/IEC
NOTE Table B.4 in Annex B shows an example of ranking evaluation methods for software quality characteristics by cost
and effectiveness.
b) whether the evaluation provides adequate coverage or scope in the combination of methods chosen
considering:
1) how to demonstrate that software meets its specifications;
2) overlapping of coverage of methods to provide additional confidence;
3) whether the set of activities, as a whole, provides an acceptable level of assurance that there is complete
coverage for those software quality characteristics of concern;
4) the degree to which the methods complement each other;
5) the effectiveness and objectivity of each method in addressing a variety of characteristics;
6) the variety of distinct approaches among the evaluation methods (e.g. some review, some analysis, and
some testing based methods);
7) taking credit for any evaluation activities on the application that will ultimately be conducted as part of the
overall system development life-cycle;
8) crediting evaluations performed by others;
c) use of “informal” preliminary evaluation activities like reviews or surveys or peer/user anecdotal experience,
trade journal product reviews, accessible product user documentation, or database repositories of product
reviews, in order to narrow the selection of products considered functionally suitable for further evaluation.
6.2.3 Evaluations performed by others
Before crediting evaluations performed by others, the following should be considered:
a) whether the evaluation addressed the evaluation requirements with a degree of rigor consistent with the
integrity level of the application;
b) whether the evaluation report identified the version of the software product being assessed, the extent of the
evaluation, the decision criteria used, and the conclusions reached;
c) whether the evaluation report identified any deficiencies in the software product or the software engineering
process, recommended any corrective action to address those deficiencies, and whether the corrective action
was carried out;
d) whether the evaluator had appropriate expertise, including:
1) experience in performing evaluation and analysis;
2) experience in software quality relative to internationally accepted standards;
3) expertise in software engineering issues;
4) total independence from the supplier.
6.3 Step 3 - Design the evaluation
The product evaluation plan should identify:
a) whether the supplier or third party is willing and able to provide access to the required d
...