ISO/IEC 23396:2020

Systems and software engineering - Capabilities of review tools

This document specifies the capabilities of a tool to support review work. The evaluation and selection of the review tools are performed in accordance with ISO/IEC 20741 which defines the general evaluation selection process and evaluation characteristics. This document defines capabilities specific to review tools in the process. By using these two standards together, it is possible to derive objective and reasonable results of the evaluation and selection of review tools. The review work is based on the process, activities, and tasks defined in ISO/IEC 20246. It is also assumed that the review targets are defined in ISO/IEC 20246. The review work in this document is assumed not to be performed by a 3rd party, but within a project. The review tool capabilities specified in this document harmonize with the review process defined in ISO/IEC 20246. This document does not include automated process, activities, or tasks for conducting reviews such as automated source code checkers defined in ISO/IEC 30130. Issues which are identified in the review are recorded and managed by the tool; but defects found in tests and issues found in general except for reviews are out of the scope of this document.

Ingénierie des systèmes et du logiciel — Capacités des outils d'analyse

General Information

• Status: Published
• Publication Date: 07-Jul-2020
• ICS: 35.080 - Software
• Technical Committee: ISO/IEC JTC 1/SC 7 - Software and systems engineering
• Drafting Committee: ISO/IEC JTC 1/SC 7/WG 4 - Tools and environment
• Current Stage: 9060 - Close of review
• Completion Date: 04-Mar-2031

Overview

ISO/IEC 23396:2020 defines the capabilities of review tools used in systems and software engineering. It specifies what a tool must support to enable review work that follows the review process in ISO/IEC 20246, and it is intended to be used alongside ISO/IEC 20741 (tool evaluation and selection). The standard describes an object model for review entities (for example ReviewFolder, Review, Member, WorkProduct, IssueByReview) and enumerates capability categories that review-support tools should provide.

Key topics and technical requirements

• Object model and entities: Defines core review entities and their attributes to ensure consistent representation of reviews, work products, participants and issues.
• Capability categories: Covers administration, review definition and control, issue management, metrics collection, report output, human communication support, and tool linkage (import/export, integration).
• Issue lifecycle support: Requirements for creating, updating, referencing, highlighting and transitioning issue status, plus confirmation and similar-issue detection aids.
• Review metrics and reporting: Collection of review metrics and standardized report output to support progress monitoring and decision making.
• Collaboration features: Notifications, online chat, messaging, shared views and issue sharing to support distributed review teams.
• Integration and interoperability: Import/export of members, classifications, viewpoints and work products, and exporting identified issues/findings for downstream tools or processes.
• Scope clarifications: Excludes automated checking tools (e.g., automated source-code checkers addressed in ISO/IEC 30130) and is focused on in-project review activities rather than third-party external reviews.

Practical applications - who uses it and why

ISO/IEC 23396:2020 is practical for:

• Tool vendors developing review-support software who need to align features with international expectations.
• Project managers and QA leads selecting and evaluating review tools for consistent, auditable review processes.
• Systems and software engineers who run reviews and need tools that support ISO/IEC 20246 processes (work product reviews).
• Procurement and tool-evaluation teams applying ISO/IEC 20741 to perform objective, standards-based tool selection.

Benefits include improved review quality, consistent issue tracking, better collaboration in large-scale projects, and traceable metrics for process improvement.

Related Standards

• ISO/IEC 20246 - Work product reviews (defines review process, activities and review targets).
• ISO/IEC 20741 - Process and characteristics for evaluation and selection of software engineering tools (used together for tool selection).
• ISO/IEC 30130 - Automated source code checkers (out of scope for ISO/IEC 23396).

ISO/IEC 23396:2020 helps organizations standardize review-tool capabilities, enabling fair tool evaluation, reliable review workflows, and better integration with broader software engineering processes.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 23396
First edition
2020-07
Systems and software engineering —
Capabilities of review tools
Ingénierie des systèmes et du logiciel — Capacités des outils d'analyse
Reference number
©
ISO/IEC 2020
© ISO/IEC 2020
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2020 – All rights reserved

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Object model for review tools . 2
4.1 Overview of the object model . 2
4.2 Object model of review entities . 2
4.3 Category of capabilities for review tools . 3
5 Entities of review tools . 5
5.1 Overview . 5
5.2 ReviewFolder . 5
5.3 Review . 6
5.4 Member . 6
5.5 WorkProduct . 7
5.6 IssueByReview . 7
5.7 PlaceOfIssueByReview . 7
6 Capabilities of review tools . 8
6.1 Overview . 8
6.2 Administration . 8
6.2.1 Overview . 8
6.2.2 Administration of review folder . 8
6.2.3 Review definition . 9
6.2.4 Control of review implementation . 9
6.2.5 Dashboard of review progress .10
6.3 IssueByReview management .10
6.3.1 Creation, deletion, reference, and update of issues .10
6.3.2 Highlighting place of issue .11
6.3.3 Transition of issue status .11
6.3.4 Issue confirmation support .11
6.3.5 Similar issue detection support .11
6.3.6 Issue entry support .12
6.4 Review metrics collection and report output .12
6.4.1 Review metrics collection .12
6.4.2 Report output .12
6.5 Human communication support .13
6.5.1 Overview .13
6.5.2 Notification of events and schedule information of the review .13
6.5.3 Online chat between review participants .13
6.5.4 Messaging with specific review participants .13
6.5.5 Sharing issues among review participants .13
6.5.6 Sharing work product view on a screen among review participants .13
6.6 Linkage with other tools .13
6.6.1 Overview .13
6.6.2 Importing and exporting member definition .14
6.6.3 Importing and exporting classification definition .14
6.6.4 Importing and exporting viewpoints of review .14
6.6.5 Importing work products to be reviewed .14
6.6.6 Exporting identified issues .14
6.6.7 Exporting review findings .14
Annex A (informative) Review tool entities and attributes .15
© ISO/IEC 2020 – All rights reserved iii

Annex B (informative) Review tool entities CRUD matrix .19
Annex C (informative) Review tool capabilities and review activities .20
Annex D (informative) Scope of review tools .22
Annex E (informative) How to use this document with ISO/IEC 20741 .23
Annex F (informative) Use case of a review tool .24
Bibliography .31
iv © ISO/IEC 2020 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see http:// patents .iec .ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/
iso/ foreword .html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html.
© ISO/IEC 2020 – All rights reserved v

Introduction
It is widely recognized that almost all software development organizations need to improve the quality
of software at an appropriate cost and review is one important means. Applying reviews early in the
lifecycle is known to reduce the amount of unnecessary rework in regular projects. This is because
problem detection in upstream requires less cost than detection of problems by performing tests in
downstream processes. By conducting the review in this way, it is possible to detect problems early,
evaluate alternatives, improve organizational and personal processes, and improve work outcomes.
ISO/IEC 20246 specifies a standard process for work product reviews.
Review support tools provide capabilities to improve review work and improve review quality, for
example support for collaboration and communication between reviewers, logging and highlighting
comments made on a work product, and support for review report generation.
Support tools are indispensable in the large-scale development project where the number of items
pointed out in the review exceeds one thousand. Support tools are also actively used in small and
medium-sized projects to improve review quality and efficiency. Indeed, various kinds of “review tools”
are provided. For example, there is a relatively simple capability that allows you to add comments as a
tag using the macro function of document creation. On the other hand, there are tools to support the
progress management of the entire project in the large-scale development in which dozens of reviewers
point out issues and the project manager monitors the status of the review implementation status.
In large organizations, it is very important to select appropriate tools from among various kinds of
review tools. It is essential that the selected tool has a high degree of fairness and is evaluated according
to public standards. For this purpose, ISO/IEC 20741 on the process of evaluation and selection of
software engineering tools was published in 2017. However, ISO/IEC 20741 does not prescribe standard
capabilities specific to reviews because it is generalized without depending on a specific tool field.
This document aims to define the capabilities of review support tools and to select the appropriate
tool in combination with ISO/IEC 20741 for tool evaluation and selection (see Annex E). The review
support tool assumed in this document supports the entire process specified in ISO/IEC 20246. For
example, capabilities which support personal activities such as viewing and pointing out deliverables
are necessary, and capabilities which support group activities such as reporting of situations are
necessary (see Annex D). It is assumed that the check work itself such as the source code check defined
in the ISO/IEC 30130 test tool is not included, and it is assumed that humans are checking.
vi © ISO/IEC 2020 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 23396:2020(E)
Systems and software engineering — Capabilities of
review tools
1 Scope
This document specifies the capabilities of a tool to support review work.
The evaluation and selection of the review tools are performed in accordance with ISO/IEC 20741 which
defines the general evaluation selection process and evaluation characteristics. This document defines
capabilities specific to review tools in the process. By using these two standards together, it is possible
to derive objective and reasonable results of the evaluation and selection of review tools.
The review work is based on the process, activities, and tasks defined in ISO/IEC 20246. It is also
assumed that the review targets are defined in ISO/IEC 20246. The review work in this document is
assumed not to be performed by a 3rd party, but within a project.
The review tool capabilities specified in this document harmonize with the review process defined in
ISO/IEC 20246. This document does not include automated process, activities, or tasks for conducting
reviews such as automated source code checkers defined in ISO/IEC 30130.
Issues which are identified in the review are recorded and managed by the tool; but defects found in
tests and issues found in general except for reviews are out of the scope of this document.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 20246, Software and systems engineering — Work product reviews
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 20246 and the
following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at http:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
entity
data concept that may have attributes and relationships to other entities
[SOURCE: ISO/TR 25100:2012, 2.1.3, modified — NOTE has been removed.]
3.2
review folder
entity (3.1) for binding one or more related reviews, including a list of the reviews and information
common to the reviews
Note 1 to entry: The information common to the reviews can include information on members who can participate
in or organize the reviews, and information on the classification given to the issues identified during the reviews.
© ISO/IEC 2020 – All rights reserved 1

4 Object model for review tools
4.1 Overview of the object model
The overall structure of the object model of review consists of the following elements:
a) Review Process: a set of processes defined by ISO/IEC 20246,
b) Review Entity: a set of elements that represents identifiable information which appears in Review
Process and is described as a class in the object model, and
c) Review Tool: a tool which supports to create, refer to, update, and delete Review Entity.
Figure 1 — Overall structure of object model of review
The object model diagrams, Figure 1 to Figure 3, are described using UML 2 (Unified Modelling
Language 2) (see ISO/IEC 19505-2).
An object model for review tools can be identified to define tool capabilities by its input, process,
and output.
4.2 Object model of review entities
ReviewFolder in this object model (hereinafter referred as ReviewFolder) stands for a review folder.
Review in this object model (hereinafter referred as Review) is an entity representing a single review
in which all information, such as a list of work products to be reviewed, a list of participants, and a list
of issues identified during the review, is recorded.
WorkProduct in this object model (hereinafter referred as WorkProduct) is an entity representing a
work product which is defined in ISO/IEC 20246.
Member in this object model (hereinafter referred as Member) is an entity representing a member who
participates in or organizes the review. It represents all roles defined in ISO/IEC 20246 such as author,
review leader, and reviewer.
A single review is usually carried out on one or more WorkProduct by participants selected from Member.
2 © ISO/IEC 2020 – All rights reserved

IssueByReview in this object model (hereinafter referred as IssueByReview) is an entity representing
an issue which is identified in the review.
NOTE 1 Issue is defined in ISO/IEC 20246.
PlaceOfIssueByReview in this object model (hereinafter referred as PlaceOfIssueByReview) is an entity
representing a place where the issue is identified.
NOTE 2 There are two reasons why detection places of issues are set as independent entities on the model.
The first reason is that the multiplicity of the relationship between WorkProduct and IssueByReview is many
to many. That is, the same IssueByReview may be given to more than one PlaceOfIssueByReview, and multiple
IssueByReview may be given to the same PlaceOfIssueByReview. The second reason is that the capability to
support the discovery of the issue detection place is decisively important on the tool.
Figure 2 — Entities of review
The relationship between the entities described here and the activities defined in ISO/IEC 20246 is
shown in Annex B.
4.3 Category of capabilities for review tools
On the object model of the review entity, capabilities are categorized into groups according to the group
of entities to be handled (see Figure 3).
© ISO/IEC 2020 – All rights reserved 3

Key
(1) A group of entities for capabilities of category "IssueByReview management"
(2) A group of entities for capabilities of category" Review metrics collection and report output
(3) A group of entities for capabilities of category "Human communication report"
(4) A group of entities for capabilities of category "Linkage with other schools"
(5) A group of entities for capabilities of category "Administration"
a
Linking destination tools.
Figure 3 — Category of capabilities
1) IssueByReview management
It is a set of capabilities of managing individual IssueByReview identified during the review of a
WorkProduct. It generates and updates IssueByReview. At the same time, accompanying pointing
PlaceOfIssueByReview is also generated and updated.
This category is required for individual review and issue communication and analysis, such
as supporting the easy detection of issues, recording of issues, recording of the status of issues,
communication of issues, and recording of solutions. Furthermore, it is important that the records
can be used for monitoring the status of each issue and analysing the progress of review as a whole.
2) Review metrics collection and report output
It is a set of capabilities to collect and compile information related to review metrics for a single
review. It creates a review report. Review rate, reviewer progress rate, and reviewer completion
criteria are managed by this group of entities.
4 © ISO/IEC 2020 – All rights reserved

3) Human communication support
It is a set of capabilities to support communication among the review organizer and review
participants, such as messaging, chat, and mail distribution of review holding information.
It is important to support communication between each reviewer who usually performs an
individual review at different places. These capabilities can avoid duplication of issues and promote
new discoveries.
4) Linkage with other tools
It is a set of capabilities to import/export the information of each entity.
In the review, cooperation with multiple external tools is required, such as checking out the target
work product from the configuration management tool, registering a problem that cannot be solved
in the review process to the issue management tool.
5) Administration
It is a set of capabilities used by the administrator of the review, such as defining review folders
and reviews and controlling the implementation of reviews. It generates and updates ReviewFolder
and Review as well as related Member and WorkProduct.
5 Entities of review tools
5.1 Overview
The following six entities to be handled by a review tool presented in 4.2 are defined in this clause:
— ReviewFolder;
— Review;
— Member;
— WorkProduct;
— IssueByReview; and
— PlaceOfIssueByReview.
Detailed examples of each entity and the attributes contained therein are shown in Annex A.
5.2 ReviewFolder
The following attributes are normally recorded:
— a name for identifying the review folder;
— a list of members who may participate in or organize reviews bound by this entity;
— names and values of classifications which are standard vocabularies to be uniformly used for each
issue identified during reviews bound by this entity; and
— a list of reviews.
EXAMPLE The names of the classifications can include “severity” and “priority”. The values of the
classifications can include “high”, “medium”, and “low”.
© ISO/IEC 2020 – All rights reserved 5

5.3 Review
This is information of each review bound by the review folder.
The following attributes are normally recorded:
— a review ID;
— a review name;
— a review type;
— a list of work products to be reviewed;
— a list of members planned to participate in the review;
— an organizer of the review;
— a list of participants in the review;
— a list of issues identified during the review;
— date and time when review started;
— date and time when review ended;
— time spent in the review;
— work product appraisal; and
— review decision.
In addition, the following attributes may be recorded:
— purpose of the review;
— scheduled start date and time of the review;
— scheduled end date and time of the review;
— checklists used in the review;
— scenarios used in the review;
— support information such as standards;
— the review man-hours; and
— viewpoints of the review.
5.4 Member
This is information of members who are involved in the review. These members can be the organizer or
participant of the review.
The following attributes are normally recorded:
— a member ID;
— a member name;
— an affiliation; and
— contact information.
6 © ISO/IEC 2020 – All rights reserved

5.5 WorkProduct
This is information of work products to be reviewed.
The following attributes are normally recorded:
— an identifier of a work product;
— a file path;
— an author;
— number of pages or lines of code; and
— a scope to be reviewed.
NOTE If the changed parts are specified by "a scope to be reviewed", reviewers can concentrate on only the
changed parts.
In addition, the following attributes may be recorded:
— the formats of files of work products such as documents, spreadsheets, presentations, or source
codes; and
— status of review implementation for each page or for each line of code.
5.6 IssueByReview
This is information of issues identified during the review.
The following attributes are normally recorded:
— an issue ID;
— the date and time when the issue was identified;
— originator who identified the issue;
— place on the work product where the issue was found (see 5.7);
— description of the issue;
— classifications of the issue;
— member assigned responsibility for addressing the issue; and
— status of the issue (e.g. “corrected” or “confirmed”).
The following attribute may be also recorded:
— a resolution of the issue.
5.7 PlaceOfIssueByReview
This is information of places on work product where the issues were identified.
The following attributes are normally recorded:
— an identifier of the work product; and
— a physical position in the page in the case of a document or the column position and the row in the
case of source code.
© ISO/IEC 2020 – All rights reserved 7

The following attribute may also be recorded:
— a logical place such as chapter and clause.
6 Capabilities of review tools
6.1 Overview
This clause defines capabilities of review tools which are divided into five categories in 4.3. Figure 4
shows overall categories of capabilities of review tools. These capabilities are used as tool specific
capabilities in ISO/IEC 20741 evaluation and selection process (see Annex E). The following capabilities
are used in the activities of the review process defined in ISO/IEC 20246 (see Annex C). Furthermore,
how each capability is applied in the use case is shown in Annex F.
Figure 4 — Overall categories of capabilities of review tools
NOTE See Annex A for attributes included in the entities shown in the description of each capability below.
6.2 Administration
6.2.1 Overview
This capability consists of capabilities to manage the review folder, capabilities to manage each review-
bound by the review folder, and capabilities to control the implementation of the review.
6.2.2 Administration of review folder
6.2.2.1 Review folder definition
A capability to create ReviewFolder shall be provided in order to manage the definitions of the review
folder. Also, a capability to refer to and update the attributes of ReviewFolder shall be
...