ISO/IEC 20741:2017

Systems and software engineering - Guideline for the evaluation and selection of software engineering tools

ISO/IEC 20741:2017 gives guidelines for the evaluation and selection of software engineering tools, covering a partial or full portion of the software engineering life cycle. It establishes processes and activities to be applied for the evaluation of software engineering tools and selecting the most appropriate software engineering tools from several candidates. It establishes, for selected processes, the tasks and activities that can be applied for the evaluation of software engineering tools and selecting the most appropriate software engineering tools from several candidates. It establishes processes that can be applied for the evaluation of software engineering tools and selecting the most appropriate software engineering tools from several candidates. As these processes are generic, organizations can adapt these generic processes to meet organizational needs. The software engineering tool evaluation and selection processes can be viewed in the larger context of the organization's technology adoption process. ISO/IEC 20741:2017 provides the following: a) guidance on identifying organizational requirements for software engineering tools; b) guidance on mapping those requirements to software engineering tool characteristics to be evaluated; c) a process for selecting the most appropriate software engineering tool from several tools, based on measurements of the defined characteristics. NOTE 1 Guidance on mapping those requirements to software engineering tool capabilities to be evaluated is not covered by this document, but is covered by a series of standards for each tool area. Primary users of this document are organizations that intend to adopt software engineering tools to support their software life cycle processes. Software tool suppliers can also use this document to describe characteristics of their software engineering tools. ISO/IEC 20741:2017 is not intended to apply to: a) software engineering frameworks whose purpose is to provide mechanisms for data, control and presentation integration; b) general purpose tools (e.g. word processors, spreadsheets) which can be used in software engineering activities, nor software engineering tools of very narrow scope or specific purpose (e.g. a compiler); c) planning for the implementation of software engineering tools within an organization. NOTE 2 A user of this document can make the best possible selection of a software engineering tool and yet have no guarantee of a successful implementation. The methods described in this document are useful not only for the selection of software engineering tools, but for any project where COTS/FOSS software can be selected instead of engaging in new software development. To follow the guidance provided in this document consists in applying the activities and tasks that are attached to the defined processes to evaluate and select software. Organizations using this document for trade purposes can specify the minimum set of processes and their related activities and tasks, suitable to their given application.

Ingénierie des systèmes et du logiciel — Lignes directrices pour l’évaluation et le choix des outils d’ingénierie logicielle

General Information

• Status: Published
• Publication Date: 29-May-2017
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 4 - Tools and environment
• Current Stage: 9093 - International Standard confirmed
• Start Date: 21-Dec-2022
• Completion Date: 30-Oct-2025

Overview

ISO/IEC 20741:2017 - "Systems and software engineering - Guideline for the evaluation and selection of software engineering tools" provides a generic, adaptable framework for objectively evaluating and selecting software engineering tools that support portions or the full software life cycle. The standard defines processes, activities and measurable characteristics to compare candidate tools and recommend the most appropriate option. It is intentionally focused on evaluation and selection (not implementation), and can be applied to COTS/FOSS tool choices as an alternative to new development.

Key technical topics and requirements

• Evaluation & selection processes: Defines a sequence of generic processes (Preparation, Structuring, Evaluation, Selection) with associated activities and tasks to produce repeatable, impartial results.
• Goal setting and selection criteria: Guidance for identifying organizational requirements and mapping them into measurable selection criteria (characteristics to be evaluated).
• Requirements definition and candidate identification: Methods for collecting tool information and narrowing final candidates for detailed assessment.
• Measurement-based evaluation: Emphasis on quantitative, comparable measurements of defined characteristics and use of selection algorithms to rank candidates.
• Tool characteristics: Aligns with ISO/IEC 25010 quality model; covers functionality, performance efficiency, compatibility, usability, reliability, security, maintainability, portability, plus acquisition, implementation and support indicators.
• Reporting and validation: Templates and content for evaluation reports and recommendations, and steps to validate the selection decision.
• Scope limitations: Not intended for general-purpose office tools, extremely narrow-purpose tools (e.g., a single compiler), software engineering frameworks focused on integration, nor for planning tool implementation. The standard does not guarantee successful implementation - it supports best-practice selection.

Practical applications and intended users

• Primary users: Organizations planning to adopt or replace software engineering tools (e.g., requirements, testing, configuration management, CI/CD).
• Secondary users: Tool suppliers preparing objective descriptions of tool characteristics for procurement and comparison.
• Common use cases:
  • Selecting a requirements management or configuration management tool using a repeatable, auditable process.
  • Comparing multiple COTS/FOSS solutions against organizational quality and integration criteria.
  • Establishing organizational minimum processes for procurement or trade specifications.
• Benefits: Reduces selection risk and cost, supports objective vendor comparisons, and helps integrate tool choices into broader technology adoption workflows.

Related standards

• ISO/IEC 25010 (software product quality model) - quality characteristics mapping
• ISO/IEC 25041 (product evaluation model) - evaluation guidance
• Tool-area standards (examples): ISO/IEC 30130 (software testing tools), ISO/IEC TR 24766 (requirements engineering tools), ISO/IEC TR 18018 (configuration management tools)
• ISO/IEC 15940 - guidance on defining tool-area services

Keywords: ISO/IEC 20741:2017, software engineering tools, tool evaluation, tool selection, selection process, tool characteristics, COTS, FOSS, software lifecycle.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 20741
First edition
2017-05
Systems and software engineering —
Guideline for the evaluation and
selection of software engineering tools
Ingénierie des systèmes et du logiciel — Lignes directrices pour
l’évaluation et le choix des outils d’ingénierie logicielle
Reference number
©
ISO/IEC 2017
© ISO/IEC 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO/IEC 2017 – All rights reserved

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 2
3 Terms and definitions . 2
4 Abbreviated terms . 3
5 Overview of evaluation and selection of software engineering tools .3
5.1 Introduction of the evaluation and selection of software engineering tools. 3
5.2 Framework of the evaluation and selection of software engineering tools . 4
5.3 General process considerations . 4
5.3.1 Sequencing of processes . 4
5.3.2 Reducing cost and risk . 5
6 Preparation process . 5
6.1 Purpose . 5
6.2 Outcomes . 6
6.3 Activities and tasks . 6
6.3.1 Goal setting . 6
6.3.2 Establishing selection criteria . 7
6.3.3 Project planning and control . 7
7 Structuring process . 8
7.1 Purpose . 8
7.2 Outcomes . 8
7.3 Activities and tasks . 9
7.3.1 Requirements definition . 9
7.3.2 Software engineering tool information gathering .10
7.3.3 Identifying final candidate software engineering tools .11
8 Evaluation process .11
8.1 Purpose .11
8.2 Outcomes .12
8.3 Activities and tasks .12
8.3.1 Preparing for evaluation .12
8.3.2 Evaluating software engineering tools .13
8.3.3 Evaluation reporting . .14
9 Software engineering tool selection process .14
9.1 Purpose .14
9.2 Outcomes .15
9.3 Activities and tasks .15
9.3.1 Preparing for selection .15
9.3.2 Applying the selection algorithm .15
9.3.3 Recommending a selection decision .15
9.3.4 Validating the selection decision .15
10 General software tool characteristics .16
10.1 Overview .16
10.2 Characteristics related to software engineering tool usage functionality .16
10.2.1 Overview .16
10.2.2 Software engineering tool operation environment characteristics .16
10.2.3 Software engineering tool integrability characteristics .17
10.2.4 Software engineering tool application characteristics .18
10.3 General quality characteristics.20
10.3.1 Overview .20
© ISO/IEC 2017 – All rights reserved iii

10.3.2 Functional suitability characteristics .20
10.3.3 Performance efficiency characteristics .20
10.3.4 Compatibility characteristics .21
10.3.5 Usability characteristics .21
10.3.6 Reliability characteristics .22
10.3.7 Security characteristics .23
10.3.8 Maintainability characteristics .24
10.3.9 Portability characteristics .25
10.4 General characteristics not related to quality .26
10.4.1 Overview .26
10.4.2 Acquisition process characteristics .26
10.4.3 Implementation characteristics .27
10.4.4 Support indicators characteristics .27
10.4.5 Evaluation or certification characteristics .28
Annex A (informative) Examples of selection algorithms .29
Annex B (informative) Evaluation report contents .32
Bibliography .34
iv © ISO/IEC 2017 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO’s adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following
URL: w w w . i s o .org/ iso/ foreword .html.
This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
© ISO/IEC 2017 – All rights reserved v

Introduction
Within systems and software engineering, software engineering tools represent a major part of the
supporting technologies used to develop and maintain information technology systems. Their selection
is carried out with careful consideration of both the technical and management requirements.
The objective of an evaluation process is to provide quantitative and comparable results of all candidate
alternatives. The final selection can then be based on these results. To be widely useful and accepted,
the software engineering tool evaluation and selection processes are supposed to help both the users
and the suppliers of software engineering tools. The more objective, repeatable, and impartial the
evaluation and selection processes are, the more widely acceptable they are. The information and
guidance outlined in this document are intended to lead to more cost-effective selections of software
engineering tools and to a greater uniformity in how software engineering tool functions and features
are described.
For evaluating and selecting software engineering tools, a set of processes providing a procedure for
evaluation and selection, a list of capabilities providing scope of functional requirements, and a list of
characteristics providing scope of non-functional requirements are needed.
Evaluation and selection of software engineering tools is usually performed within a specific, purpose-
oriented tool area for practical reasons, to manage the scope of evaluation and selection. Examples
of such tool areas are requirements engineering tools and configuration management tools. Lists of
capabilities are tool area specific, but the list of characteristics and the set of evaluation and selection
processes are more generic for all software engineering tool areas.
This document defines a set of processes and a list of characteristics which can be used by all software
engineering tool areas. This document can be used together with any tool area-specific standard which
defines list of capabilities for the tool area.
International standards defining lists of capabilities for specific tool areas have been published, such
as ISO/IEC 30130 for “software testing tools”, ISO/IEC TR 24766 for “requirements engineering tools”,
and ISO/IEC TR 18018 for “configuration management tools”. Lists of capabilities for other tool areas of
software engineering can be developed as a series of standards according to their priority.
It is supposed in this document that tool area is decided before starting the evaluation and selection.
It is recommended that the decision would be based on ISO/IEC 15940 which defines the software
engineering service for each tool area.
This document adopts the general model of software product quality characteristics and sub-
characteristics defined in ISO/IEC 25010 and gives additional guidance how to apply the model when
the software product is a software engineering tool. The document follows also the software product
evaluation model defined in ISO/IEC 25041.
vi © ISO/IEC 2017 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 20741:2017(E)
Systems and software engineering — Guideline for the
evaluation and selection of software engineering tools
1 Scope
This document gives guidelines for the evaluation and selection of software engineering tools, covering
a partial or full portion of the software engineering life cycle.
It establishes processes and activities to be applied for the evaluation of software engineering tools and
selecting the most appropriate software engineering tools from several candidates.
It establishes, for selected processes, the tasks and activities that can be applied for the evaluation of
software engineering tools and selecting the most appropriate software engineering tools from several
candidates.
It establishes processes that can be applied for the evaluation of software engineering tools and
selecting the most appropriate software engineering tools from several candidates.
As these processes are generic, organizations can adapt these generic processes to meet organizational
needs. The software engineering tool evaluation and selection processes can be viewed in the larger
context of the organization’s technology adoption process.
This document provides the following:
a) guidance on identifying organizational requirements for software engineering tools;
b) guidance on mapping those requirements to software engineering tool characteristics to be
evaluated;
c) a process for selecting the most appropriate software engineering tool from several tools, based on
measurements of the defined characteristics.
NOTE 1 Guidance on mapping those requirements to software engineering tool capabilities to be evaluated is
not covered by this document, but is covered by a series of standards for each tool area.
Primary users of this document are organizations that intend to adopt software engineering tools
to support their software life cycle processes. Software tool suppliers can also use this document to
describe characteristics of their software engineering tools.
This document is not intended to apply to:
a) software engineering frameworks whose purpose is to provide mechanisms for data, control and
presentation integration;
b) general purpose tools (e.g. word processors, spreadsheets) which can be used in software
engineering activities, nor software engineering tools of very narrow scope or specific purpose
(e.g. a compiler);
c) planning for the implementation of software engineering tools within an organization.
NOTE 2 A user of this document can make the best possible selection of a software engineering tool and yet
have no guarantee of a successful implementation.
The methods described in this document are useful not only for the selection of software engineering
tools, but for any project where COTS/FOSS software can be selected instead of engaging in new
software development.
© ISO/IEC 2017 – All rights reserved 1

To follow the guidance provided in this document consists in applying the activities and tasks that are
attached to the defined processes to evaluate and select software. Organizations using this document
for trade purposes can specify the minimum set of processes and their related activities and tasks,
suitable to their given application.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http:// www .electropedia .org/
— ISO Online browsing platform: available at http:// www .iso .org/ obp
3.1
atomic sub-characteristic
lowest level sub-characteristics
Note 1 to entry: The highest level evaluation categories are called characteristics. Characteristics are usually
subdivided into sub-characteristics. At the lowest level, when no further subdivision is appropriate, the sub-
characteristics are referred to as atomic sub-characteristics.
3.2
characteristic
aspect of a product by which it can be described and evaluated
Note 1 to entry: A characteristic can be refined into multiple levels of sub-characteristics that bear on its ability
to satisfy stated or implied needs.
3.3
measure (noun)
variable to which a value is assigned as the result of measurement
Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and
indicators.
[SOURCE: ISO/IEC 15939:2007, 2.15, modified — The words “plural form” have been changed to “term”.]
3.4
measure (verb)
make a measurement
[SOURCE: ISO/IEC 25040:2011, 4.39]
3.5
measurement
set of operations having the object of determining a value of a measure
Note 1 to entry: Measurement can include assigning a qualitative category such as the language of a source
program (ADA, C, COBOL, etc.).
[SOURCE: ISO/IEC 15939:2007, 2.17, modified — Note 1 to entry has been changed.]
3.6
software engineering tool
software product that assists software engineers by providing automated support
2 © ISO/IEC 2017 – All rights reserved

3.7
rating
action of mapping the measured value to the appropriate rating level
Note 1 to entry: Used to determine the rating level associated with the software for a specific quality
characteristic.
Note 2 to entry: Rating and rating levels can be applied to characteristics other than quality characteristics.
3.8
rating level
scale point on an ordinal scale which is used to categorize a measurement scale
Note 1 to entry: The rating level enables software to be classified (rated) in accordance with the stated or implied
needs (see 8.2).
Note 2 to entry: Appropriate rating levels can be associated with the different views of quality, i.e. “Users”,
“Managers” or “Developers”.
4 Abbreviated terms
BMT benchmark test
COTS commercial off-the-shelf
FOSS free/open-source software
GUI graphical user interface
5 Overview of evaluation and selection of software engineering tools
5.1 Introduction of the evaluation and selection of software engineering tools
This document defines both a set of processes and a structured set of software engineering tool
characteristics for use in the technical evaluation and the ultimate selection of a software engineering
tool. It follows the software product evaluation model defined in ISO/IEC 25041.
This document adopts the general model of software product quality characteristics and sub-
characteristics defined in ISO/IEC 25010 and extends these when the software product is a software
engineering tool; it provides product characteristics unique to software engineering tools as described
in 10.2 to 10.4. This larger set of characteristics is then organized into three groups; they are
characteristics related to software engineering tool usage functionality, general quality and not related
to quality.
NOTE The capabilities of software engineering tools are defined in a series of standards for each tool area.
The objective of the technical evaluation process is to provide quantitative results on which the final
selection can be based. Measurement assigns numbers (or other ratings) to attributes of entities; a major
activity of evaluation is to obtain these measurements for use in selection. The final selection results
should aim to achieve objectivity, repeatability and impartiality. These objectives and the confidence in
the outcomes depend in part on the resources allocated to the overall evaluation and selection process.
The user of this document is asked to deal with these issues at an early stage.
© ISO/IEC 2017 – All rights reserved 3

5.2 Framework of the evaluation and selection of software engineering tools
This subclause illustrates an overview of the evaluation and selection of software engineering tools
discussed in this document as shown in Figure 1. Evaluation and selection of software engineering
tools includes four major processes:
— preparation process;
— structuring process;
— evaluation process; and
— selection process.
In the process diagrams Figures 1 to 5, round corner rectangle is process/activity, normal rectangle
is outcome of the process/activity and dashed rectangle is a plan which is referred by each process
without reference. Also, a solid arrow shows data flow and a dashed arrow shows control flow of
transition between processes/activities.
Figure 1 — Framework of evaluation and selection of software engineering tools
A key process is the structuring of a set of requirements against which candidate software engineering
tools are to be evaluated and upon which selection decisions are based. The software engineering tool
characteristics defined in 10.2 to 10.4 form the basis for requirements structuring and play a central
role in the overall process.
5.3 General process considerations
There are several considerations that apply to the processes described in this document on a global
basis. The intent is for the user of this document to tailor its application in such a way as to maximize
the probability of a successful evaluation and selection process, and minimize its cost and risk.
5.3.1 Sequencing of processes
This document does not impose the sequence of process activities described above and in the following
clauses. It is up to the organization to select the relevant processes and activities needed to meet its
evaluation and selection goals.
The organization decides which to employ, in what sequence, and with what degree of parallelism. The
sequencing of the processes’ activities is then documented in an evaluation project plan.
4 © ISO/IEC 2017 – All rights reserved

5.3.2 Reducing cost and risk
In general, organizations which apply this document want to minimize the cost of the entire evaluation
and selection process to the extent possible, while maintaining the level of effort necessary to select
the most appropriate software engineering tool(s) for their use. These objectives can be addressed by
minimizing the number of tools evaluated, minimizing the cost of evaluating specific tools, and ensuring
that the formality of the process is appropriate to the organization.
The activities of software engineering tool information gathering and identifying final candidates for
selection (see Clause 9) effectively allow the user of this document to screen the available tools against
the organization’s needs, and eliminate from consideration tools which do not, or are not likely to,
substantially address the organization’s needs.
The organization can be unable to find any tool which appears likely to sufficiently meet its needs. In
such a case, the stated needs themselves should be re-examined, and if they are found to accurately
reflect the organization’s actual requirements for technology improvement, the overall evaluation
and selection process can be abandoned. Similarly, if the final candidate tools appear to be marginal
in addressing the organization’s needs, the level of detail and formality of the subsequent activities
should be made to reflect the risk factor, and the organization should be prepared to not select a tool
if the evaluation process so indicates, as the typical cost of bringing a new tool into operational use is
substantial.
Evaluations of candidate tools might have already been performed and be available to the organization.
Such information can be used to reduce the cost of candidate tool evaluation.
NOTE 1 Previous evaluations which have been performed on a different version of the candidate tool can still
yield useful information. Similarly, evaluations which addressed a different set of organizational needs can still
provide useful information.
This document calls for the development of several plans and reports, and implicitly, for their review
by various personnel within the organization. In addition, activities are required to perform the four
processes outlined.
The format and level of detail of the data products are left to the discretion of the organization, as is the
level of effort necessary to perform the activities.
NOTE 2 Some organizations can be required to limit the scope, detail and formality of the processes to apply
this document within existing resource constraints.
6 Preparation process
6.1 Purpose
In this process, the objective and criteria for tool selection are clarified, and the project plan is defined
for tool evaluation and selection.
A set of software engineering tool selection guidelines is identified and a project plan developed. The
process is shown in Figure 2.
© ISO/IEC 2017 – All rights reserved 5

Figure 2 — Overview of preparation process
6.2 Outcomes
Outcomes resulting from the successful implementation of the preparation process:
a) objective for tool selection;
b) list of selection criteria; and
c) project plan for evaluation and selection tools.
6.3 Activities and tasks
6.3.1 Goal setting
The development of a set of realistic goals is a necessary first activity. In developing goals, both a
rationale for acquisition (why acquire a software engineering tool) and a general policy for acquisition
(what type of tool to acquire and how to do it) should be developed.
NOTE Goal setting activities, including possibly the identification of selection criteria, might have already
been performed as a part of other efforts prior to formally entering the preparation process of evaluation and
selection of software engineering tools.
The following tasks should be performed:
a) develop rationale for acquisition:
1) review the organization’s current software development process, determining its maturity and
areas of concern;
2) review the current state of software engineering tool technology and observe trends for
consideration as future reference technology;
3) compare the organization’s current practices to possible future practices if software
engineering tools are adopted and identify areas of potential benefit;
b) define goals and expectations:
1) set overall goals (e.g. productivity improvement, quality improvement, enhanced process
manageability);
2) define evaluation and selection constraints (e.g. cost, schedule, resources);
6 © ISO/IEC 2017 – All rights reserved

3) quantify and classify expectations (based upon goals);
c) set general policy for acquisition:
1) identify constraints on tool acquisition (e.g. implementation cost, schedule, and other
resources);
2) develop alternate approaches to introducing/augmenting software engineering tool (e.g. buy a
tool, modify an existing tool or develop a new tool);
3) assess the feasibility of the various alternatives in light of organizational readiness, technical
considerations, performance specifications, and resources;
4) identify probable impacts of software engineering tools on the organization, e.g. areas where
training and education, procedure guides, and technical support are needed to effectively
deploy software engineering tool; and
5) the goals and expectations established here are used to guide subsequent activities in the
overall process and, finally, to validate the selection decision.
6.3.2 Establishing selection criteria
Based upon the goals and expectations developed above, selection criteria should be established.
a) Decompose the high level goals into a set of selection criteria to make the (go/no go) selection
decision.
The selection criteria should be objective and quantitative. Each selection criterion should include some
defined threshold specified on which the major go/no go decision is made during selection.
b) Define the relative importance of the selection criteria.
NOTE 1 The relative importance of the selection criteria is used to determine the weights assigned to tool
characteristics, sub-characteristics and/or capabilities for evaluation.
c) Define the level of detail and the nature of the evaluation activities to be performed.
NOTE 2 The nature of the evaluation activities covers the methods used in collecting the data, e.g.
reference, for example, how the data are measured, collected with predefined criteria, or based upon
subjective observation.
d) Define the evaluation/selection scenario to be performed.
NOTE 3 The evaluation/selection scenario is defined as a method such as evaluating in a small team of a real
project, evaluating in an experimental pilot project, or evaluating with catalogue of tools.
6.3.3 Project planning and control
Based upon the goals and selection criteria which have been established for the overall evaluation and
selection process, a project plan should be created and a control mechanism implemented. The plan and
control mechanism should be developed in accordance with the organization’s normal planning and
control process, and it should contain the following.
a) A project team organization with assigned responsibilities.
The skill of the evaluators has an impact on the results of the evaluation and its applicability to the
organization. The evaluation personnel should be selected with this in mind, and the skill level
of evaluators should be a factor in assessing evaluation results. The evaluation team should be
representative of the intended tool user group.
b) A set of operational goals obtained by decomposing the overall goals previously established.
© ISO/IEC 2017 – All rights reserved 7

NOTE For instance, overall goal as “No need to install software tools in each client” is decomposed
operational goals as “Need to run on specified web browser” and “Server side of the tool and its data should
be installed on the server set in own intranet”.
c) A set of selection guidelines: weighted selection criteria, definition of level of detail and special
features, and an evaluation and selection scenario.
d) A schedule of activities and their tasks, along with an estimate of resource requirements and a cost
estimate.
The project plan and the evaluation strategy and cost should be proportionate to the life-cycle cost and
expected benefit of the tool being evaluated.
e) A means of monitoring and controlling the execution of the plan.
f) If developed, the project plan and control mechanism should be updated as the project evolves.
7 Structuring process
7.1 Purpose
In the structuring process, requirements for tools are gathered from tool user organization and the
details of requirements are structured, as the framework for tool selection and evaluation was created
in the preparation process.
Figure 3 — Overview of structuring process
7.2 Outcomes
Outcomes resulting from the successful implementation of the structuring process:
a) organizational information;
b) tool requirements; and
c) structured tool requirements.
8 © ISO/IEC 2017 – All rights reserved

7.3 Activities and tasks
7.3.1 Requirements definition
7.3.1.1 Overview
During requirements definition, the requirements for the software engineering tool are collected and
organized into the software engineering tool characteristics. 10.2 identifies characteristics related to
software engineering tool usage functionality, 10.3 identifies general software quality characteristics,
10.4 identifies a set of characteristics not related to quality, and a series of standards for each tool area
identifies the major software engineering tool specific capabilities.
7.3.1.2 Organizational information gathering
To be able to define a set of detailed requirements to be satisfied by the software engineering tool,
information about the organization should be gathered, including:
a) commitment of the organization to fully fund and implement software engineering tool use;
b) current software engineering environment within the organization, including data describing
current hardware, operating software, tool use, and cloud environment (e.g. IaaS, PaaS and SaaS);
c) types of software development projects undertaken by the organization including size, domain of
application;
d) characteristics and constraints of the target systems for which software is developed;
e) specific expected impacts and improvements of software engineering tool on the organization;
f) requirements from potential tool users and end users;
g) current organizational procurement policies; and
h) current information security policies for software adoption.
This information is necessary to ensure the tool or tools are appropriate for use within the organization,
and that they address organizational needs and needs perceived by their future users.
NOTE This information can be gathered in a number of ways, including surveys and focus groups.
7.3.1.3 Requirements identification
The tool user’s requirements should deal with the question of what the software engineering tool
should do as well as its impact on the existing environment. The following tasks should be performed in
building the list of requirements:
a) analyze the requirements and adjust the level of detail to which requirements are defined and
measured;
b) evaluate the current need for software engineering tools while taking into consideration those
projects where the software engineering tool might initially be used;
c) identify desired methodology;
d) identify portions of the life cycle to be supported (e.g. planning, analysis, design);
e) identify required capabilities of the software engineering tool;
f) identify required characteristics related to software engineering tool usage functionality of the
software engineering tool;
g) identify required general software quality characteristics of the software engineering tool;
© ISO/IEC 2017 – All rights reserved 9

h) identify required characteristics not related to quality of the software engineering tool; and
i) check the consistency of the requirements with the previously established goals.
NOTE 1 These requirements represent the total set of organizational requirements. It is possible that no single
software engineering tool can satisfy all of the requirements, but that individual software engineering tools can
satisfy a sufficient number to justify their use by the organization, which can continue to search for tools to
support remaining requirements.
NOTE 2 When identifying required capabilities, extracting capabilities from a series of standards for each tool
area is suggested. If there are no proper standards, the evaluation and selection project is needed to define the
capabilities by itself.
7.3.1.4 Requirements structuring
The applicability of the user needs identified in 10.2 to 10.4 and a series of standards for each tool
area, and any others which the organization may wish to add, should be defined. The purpose of this
structuring is to organize the requirements in such a way that the evaluation can be proceeded more
effectively. The tasks include:
a) categorize the user requirements in terms of the organization of 10.2 to 10.4 and a series of
standards for each tool area, and decompose them into detailed specifications;
b) select characteristics and capabilities from 10.2 to 10.4 and a series of standards for each tool area
which might be evaluated to determine the extent to which the software engineering tool meets
the detailed specifications;
c) identify weights for the characteristics and/or capabilities.
NOTE The weights are applied to the ratings determined during the evaluation as part of the selection
process and reflect the relative importance of the related selection criterion as determined during the
preparation process.
The assignment of weights is a subjective task which has a fundamental impact on the outcome
of the entire evaluation and selection process. The assignment of weights should reflect both the
organization’s actual requirements and the ability of the organization to evaluate the characteristic.
See Annex B for further discussion.
ISO/IEC 25051 addresses quality requirements applicable to software engineering tools when
considered as software packages and should be consulted as part of the requirements structuring task.
It provides additional guidance on a subset of the quality requirements of ISO/IEC 25010.
7.3.2 Software engineering tool information gathering
A general search of potential software engineering tools to be evaluated is undertaken based upon the
requirements and selection criteria established. The activities of gathering information and identifying
the candidate software engineering tools may require several iterations to quickly and efficiently
identify the most promising tools for further evaluation. For the software engineering tools that appear
most promising for further evaluation, additional and more detailed data that deal with their potential
acquisition are obtained. This additional information can help to quickly eliminate many tools, allowing
attention to be focused on the remaining candidates. Information to be obtained includes:
a) supplier general information (e.g. business history, available support, plans and strategies);
NOTE The word “Supplier” is used for provider of commercial tools and open source tools or cloud
environment services (e.g. PaaS and SaaS).
b) supplier’s specific product development strategy;
c) the tool’s cost (e.g. price, maintenance, modifications, training);
d) the hardware and software required to support tool use;
10 © ISO/IEC 2017 – All rights reserved

e) the hardware and software required to support final application/product use;
f) the
...