ISO/IEC 30130:2016
(Main)Software engineering — Capabilities of software testing tools
Software engineering — Capabilities of software testing tools
ISO/IEC 30130:2016 defines the framework to which capabilities of software testing tools are allocated in order to identify the capabilities of products being used by any project for software testing. Software testing processes are identified in ISO/IEC/IEEE 29119‑2 and software verification processes are identified in ISO/IEC 12207. This International Standard is fully harmonized with these existing standards in terms of software testing processes. ISO/IEC 30130:2016 focuses on the following areas that the existing ISO/IEC standards do not deal with the following: - categorization of software test entities and software testing tools (Clauses 4 and 5); - characterization of each software testing tool category (Clauses 5 and 6); - mapping of software testing tool capabilities and characteristics (Clauses 6 and 7).
Ingénierie du logiciel — Capacités des outils d'essai de logiciel
General Information
Buy Standard
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 30130
First edition
2016-02-01
Software engineering — Capabilities
of software testing tools
Ingénierie du logiciel — Capacités des outils d’essai de logiciel
Reference number
ISO/IEC 30130:2016(E)
©
ISO/IEC 2016
---------------------- Page: 1 ----------------------
ISO/IEC 30130:2016(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2016 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 30130:2016(E)
Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Object model for software testing tools. 2
4.1 Overview of the object model . 2
4.2 Test target . 3
4.3 Dynamic test execution entity . 3
4.4 Code analysis entity . 4
4.5 Test management entity . 4
5 Category of test entity . 5
5.1 Overview . 5
5.2 Categories of dynamic test execution entities . 5
5.2.1 Input for dynamic test execution . 5
5.2.2 Dynamic test execution . 5
5.2.3 Test data repository . 5
5.2.4 Test environment . 6
5.3 Categories of code analysis entities . 6
5.3.1 Input for code analysis . 6
5.3.2 Code analysis . 6
5.4 Categories of test management entities . 6
5.4.1 Test plan . 6
5.4.2 Test asset . 6
5.4.3 Quality record report . 6
5.4.4 Test completion report . 6
5.4.5 Verification and validation report . 6
5.4.6 Test status report . 6
6 Characteristics of software testing tools . 6
6.1 Overview . 6
6.2 Quality characteristics . 7
6.3 Granularity . 8
6.4 Other aspects of characteristics . 9
7 Capabilities of software testing tools . 9
7.1 Overview . 9
7.2 Dynamic test execution . 9
7.2.1 Input for dynamic test execution . 9
7.2.2 Dynamic Test Execution .10
7.2.3 Test data repository .11
7.2.4 Test Environment .12
7.3 Code analysis .13
7.3.1 Input for code analysis .13
7.3.2 Code analysis .13
7.4 Test management .14
7.4.1 Test plan .14
7.4.2 Test asset .14
7.4.3 Quality record report .14
7.4.4 Test completion report .15
7.4.5 Verification and validation report .15
7.4.6 Test status report .15
7.5 Summary of capabilities with characteristics .16
© ISO/IEC 2016 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 30130:2016(E)
Annex A (informative) Capabilities mapping to test process .19
Annex B (informative) Overview of the approach for this International Standard .25
Bibliography .27
iv © ISO/IEC 2016 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 30130:2016(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 7, Software
and systems engineering.
© ISO/IEC 2016 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 30130:2016(E)
Introduction
This International Standard defines the framework to which capabilities of software testing tools are
allocated in order to identify the capabilities of products being used by any project for software testing.
To develop high-quality software with reasonable time and budget, the use of software testing tools is
required. The increase in the size and complexity of software is complemented by an increase in the
difficulty and complexity of software testing. This created a larger demand for the support of tools in
order to test software effectively and efficiently.
Testing tools are highly diverse due to their contexts of use. Testing itself varies by objective, such as
functional testing or nonfunctional testing, and granularity, such as unit testing or system testing.
Testing tool vendors vary by providing tools with a different function or combination of functions.
And despite vendor provided explanations for the type of testing support functions, there is little
common understanding of these functions. In this environment, it is difficult to utilize a testing tool
that is suitable for a project without common understanding of tool functions, proper acquisition of the
needed tools, and efficient training.
The framework defined by this International Standard consists of objectives of testing, granularity of
software to be tested and capabilities. In Clause 4, an object model for software testing tools as basis for
the framework is defined. In Clause 5, three of the categories in that software testing model (Dynamic
Test Execution, Code Analysis, and Test Management) are specified. In Clause 6, quality characteristics,
granularity, and other aspects of characteristics are defined and in Clause 7, tool capabilities are
mapped onto those categories and characteristics.
vi © ISO/IEC 2016 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 30130:2016(E)
Software engineering — Capabilities of software testing
tools
1 Scope
This International Standard defines the framework to which capabilities of software testing tools are
allocated in order to identify the capabilities of products being used by any project for software testing.
Software testing processes are identified in ISO/IEC/IEEE 29119-2 and software verification processes
are identified in ISO/IEC 12207. This International Standard is fully harmonized with these existing
standards in terms of software testing processes.
This International Standard focuses on the following areas that the existing ISO/IEC standards do not
deal with the following:
— categorization of software test entities and software testing tools (Clauses 4 and 5);
— characterization of each software testing tool category (Clauses 5 and 6);
— mapping of software testing tool capabilities and characteristics (Clauses 6 and 7).
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 25010, Systems and software engineering — Systems and software Quality Requirements and
Evaluation (SQuaRE) — System and software quality models
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
3.1
quality record report
report which is generated through dynamic test execution and code analysis to record test results
and other output
Note 1 to entry: Including Test Result, Static Code Analysis Report, Test Incident Report, and Metrics.
3.2
test target version
specific version of test target which is used for one-time execution of Dynamic Test Execution or
Code Analysis
3.3
testing tool
specific or generic tool which is used for test execution and test management such as test results
recording, test results display, test results interpretation, generation of test data, generation of test
procedure, generation of test scripts, test modelling, etc.
3.4
package
namespace for the grouped elements
© ISO/IEC 2016 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC 30130:2016(E)
4 Object model for software testing tools
4.1 Overview of the object model
A software testing tool is described by its input, process, output and environment. To define software
testing tools, the object model for software testing is identified.
The object model for software testing tools consists of the following elements:
a) test process, which represents processes of dynamic test;
b) code analysis process, which represents processes of code analysis;
c) test entity, which represents entities that appear in the process;
d) test tool, which supports inputs, processes, outputs and test entity.
Test Process comprises multiple subprocesses. Each process has input and output. These input and
output are basically test entities. Test processes are specified in detail by ISO/IEC/IEEE 29119-2.
Code Analysis Process comprises multiple sub processes. Each process has input and output. These
input and output are basically test entities.
Test Entity comprises multiple entities. The entity is referred to as Dynamic Test Execution Entity that
is required or created in Dynamic Test Process. The entity is referred to as Code Analysis Entity that is
required or created in Code Analysis Process. A test entity represents a single performance of a test, in
which the target of the test is tested in various respects. Testing has two modes of execution. Dynamic
Test Execution involves actual execution of the code and Code Analysis involves examination of the
source code of the target.
The testing tools take test entities as input and produce test entities. By producing test entities, testing
tools effectively support the testing process.
The object model diagrams, Figures 1 to 5, are described using UML 2 (ISO/IEC 19505-2).
Test Process includes Dynamic Test Process and Test Management Process.
Code Analysis Process includes Code Analysis Process and Code Analysis Management Process.
Test Entity includes Test Management Entity, Dynamic Test Execution Entity and Code Analysis Entity,
and Test Target.
Test Tool includes Test Execution Tool and Test Management Tool.
2 © ISO/IEC 2016 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 30130:2016(E)
Figure 1 — Object model of software testing
Test Process and Test Tool have indirect relationships which are described in Annex A.
4.2 Test target
Test Target is a set of source code and executable code of software that is in development. It has a
hierarchical structure, and any part of the structure or the whole structure is tested.
Test Target Version is a specific version of the software.
Figure 2 — Object model of the package “Test Target”
4.3 Dynamic test execution entity
Dynamic Test Execution Entity is prepared to identify necessary entities which are used in one-time
dynamic test execution. Test Data, Test Target Version, Test Result, and Expected Result are identified
in this package.
In one-time dynamic test execution, Test Environment is used.
Test Target Version which is used in one-time dynamic test execution is extracted from an appropriate
granularity and version of Test Target.
Test Data which is used in one-time dynamic test execution is designed for Test Case which is derived
from Specification and is stored on Test Data Collections in Test Data Repository.
Specification, Test Case, and Test Target are referred to as input for Dynamic Test Execution.
© ISO/IEC 2016 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO/IEC 30130:2016(E)
Test Data Repository has Test Data which is used by all Dynamic Test Execution and is classified by
Objective and Technique.
Figure 3 — Object model of the package “Dynamic Test Execution Entity”
4.4 Code analysis entity
Code Analysis Entity is prepared to identify necessary entities which are used in one-time code analysis.
Test Target Version and Code Analysis Result are identified in this package.
Test Target Version which is used in one-time code analysis is extracted from an appropriate granularity
and version of Test Target.
Test Target and Check List which are used in Code Analysis are referred to as Input for Code Analysis.
Check List usually does not depend on Specification.
Figure 4 — Object model of the package “Code Analysis Entity”
4.5 Test management entity
Test Management Entity is prepared to identify necessary packages and entities which are used to
manage the entire test processes. Test Plan, Test Asset, Quality Record Report, Test Completion Report
,Verification and Validation Report,and Test Status Report are identified in this package.
4 © ISO/IEC 2016 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 30130:2016(E)
Dynamic Test Execution Documentation and Static Code Analysis Report are referred to as Quality
Record Report.
Dynamic Test Execution Documentation is composed by Metric, Test Result, and Test Incident Report.
Static Code Analysis Report is composed by Metric, Static Code Analysis Report, and Test Incident Report.
Figure 5 — Object model of the package “Test Management Entity”
5 Category of test entity
5.1 Overview
In this Clause, categories of test entities are defined to allocate and specify the capabilities of software
tools. Each category is identified by analyzing objective, life history, and usage of each test entity.
5.2 Categories of dynamic test execution entities
5.2.1 Input for dynamic test execution
Entities in this category are used in Dynamic Test Execution as inputs, such as Specification, Test Case,
and Test Target.
5.2.2 Dynamic test execution
Entities in this category are used in one-time dynamic testing. It comprises Test Data, Test Result,
Expected Result, and Test Target Version.
5.2.3 Test data repository
Entities in this category are used in Dynamic Test Execution. It comprises Test Data Collections, in which
there are a large number of test data for the overall Dynamic Test Execution. Test Data Collections can
be subsets which depend on each objective and techniques.
© ISO/IEC 2016 – All rights reserved 5
---------------------- Page: 11 ----------------------
ISO/IEC 30130:2016(E)
5.2.4 Test environment
An entity, Target Environment, in this category is required to define a condition to evaluate Test
Target Version by one-time dynamic testing, such as hardware, operating system, middleware and test
drivers, etc.
5.3 Categories of code analysis entities
5.3.1 Input for code analysis
Entities in this category are used in Code Analysis as inputs, such as Check List and Test Target.
5.3.2 Code analysis
Entities in this category are used in one-time code analysis. It comprises of Test Target Version and
Code Analysis Result.
5.4 Categories of test management entities
5.4.1 Test plan
This category is used for testing that can be performed given constraints on personnel, material and
budget.
5.4.2 Test asset
Entities in this category is identified as a list of Test Data Collections, Test Target Version, and Check
List, which may be reused (e.g. for regression testing).
5.4.3 Quality record report
The entities in this category are created thorough test executions. It comprises Dynamic Test Execution
Documentation and Static Code Analysis Report.
5.4.4 Test completion report
The entities in this category provides a summary of the testing that was performed. This may be for the
project/programme as a whole or for the particular test sub-process.
5.4.5 Verification and validation report
This category is used for reporting verification and validation items, on which the final version of the
Test Completion Report is based.
5.4.6 Test status report
This category is used for reporting the status of testing that is performed in a specified reporting period.
6 Characteristics of software testing tools
6.1 Overview
Test objectives and test granularity are the key drivers of test scope. Other drivers of test scope, such as
test personnel, cost, and schedule constraints also have an effect on the scope of testing.
6 © ISO/IEC 2016 – All rights reserved
---------------------- Page: 12 ----------------------
ISO/IEC 30130:2016(E)
The most important aspect of characteristics is objectives of the software test, software quality. This
aspect is defined by SQuaRE (ISO/IEC 25010) and top level of the characteristics is explained in 6.2.
The second important aspect of characteristics is a size of component to be tested, granularity. This
aspect is defined in 6.3.
For other aspects of characteristics, the reader is suggested to add those aspects if necessary.
6.2 Quality characteristics
Quality characteristics which a test target shall satisfy are defined as described below by SQuaRE
(ISO/IEC 25010).
a) Functional Suitability
degree to which a product or system provides functions that meet stated and implied needs when
used under specified conditions:
— functional completeness;
— functional correctness;
— functional appropriateness.
b) Reliability
degree to which a system, product or component performs specified functions under specified
conditions for a specified period of time:
— maturity;
— availability;
— fault tolerance;
— recoverability.
c) Usability
degree to which a product or system can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use.
— appropriateness recognizability;
— learnability;
— operability;
— user error protection;
— user interface aesthetics;
— accessibility.
d) Performance Efficiency
performance relative to the amount of resources used under sated conditions:
— time behavior;
— resource utilization;
— capacity.
© ISO/IEC 2016 – All rights reserved 7
---------------------- Page: 13 ----------------------
ISO/IEC 30130:2016(E)
e) Maintainability
degree of effectiveness and efficiency with which a product or system can be modified by the
intended maintainers.
— modularity;
— reusability;
— analysability;
— modifiability;
— testability.
f) Portability
degree of effectiveness and efficiency with which a system, product or component can be
transferred from one hardware, software or other operational or usage environment to another.
— adaptability;
— installability;
— replaceability.
g) Compatibility
degree to which product, system or component can exchange information with other products,
system or components, and/or perform its required functions, while sharing the same hardware or
software environment:
— co-existence;
— interoperability.
h) Security
degree to which a product or system protects information and data so that persons or other
products or systems have the degree of data access appropriate to their types and levels of
authorization.
— confidentiality;
— integrity;
— non-repudiation;
— accountability;
— authenticity.
6.3 Granularity
Granularity which is defined in this international standard is a size of a test target. The size of a test
target through test phases varies from small pieces to a whole system.
a) Smallest Unit
Smallest Unit is defined as a unit for smallest subsets of the test target.
Modules, components, functions, subroutines or other such atomic units of description are often
used as the Smallest Unit.
8 © ISO/IEC 2016 – All rights reserved
---------------------- Page: 14 ----------------------
ISO/IEC 30130:2016(E)
b) Intermediate Unit
Intermediate Unit is defined as a unit for combination of Smallest Units of the test target.
c) Largest Unit
Largest Unit is defined as a unit for whole system of the test target.
NOTE Test levels usually correspond to the granularity, such as unit test, integration test, system test and
acceptance test. This standard covers any test level and test type.
6.4 Other aspects of characteristics
In addition to quality characteristics and granularity, other aspects of characteristics may have impact
on a scope of testing. For example, if a system to be tested is required to be of very high quality, many
iterations of regression testing are required. In
...
INTERNATIONAL ISO/IEC
STANDARD 30130
First edition
Software engineering — Capabilities
of software testing tools
Ingénierie du logiciel — Capacités des outils d’essai de logiciel
PROOF/ÉPREUVE
Reference number
ISO/IEC 30130:2015(E)
©
ISO/IEC 2015
---------------------- Page: 1 ----------------------
ISO/IEC 30130:2015(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2015 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 30130:2015(E)
Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Object model for software testing tools. 2
4.1 Overview of the object model . 2
4.2 Test target . 3
4.3 Dynamic test execution entity . 3
4.4 Code analysis entity . 4
4.5 Test management entity . 4
5 Category of test entity . 5
5.1 Overview . 5
5.2 Categories of dynamic test execution entities . 5
5.2.1 Input for dynamic test execution . 5
5.2.2 Dynamic test execution . 5
5.2.3 Test data repository . 5
5.2.4 Test environment . 6
5.3 Categories of code analysis entities . 6
5.3.1 Input for code analysis . 6
5.3.2 Code analysis . 6
5.4 Categories of test management entities . 6
5.4.1 Test plan . 6
5.4.2 Test asset . 6
5.4.3 Quality record report . 6
5.4.4 Test completion report . 6
5.4.5 Verification and validation report . 6
5.4.6 Test status report . 6
6 Characteristics of software testing tools . 6
6.1 Overview . 6
6.2 Quality characteristics . 7
6.3 Granularity . 8
6.4 Other aspects of characteristics . 9
7 Capabilities of software testing tools . 9
7.1 Overview . 9
7.2 Dynamic test execution . 9
7.2.1 Input for dynamic test execution . 9
7.2.2 Dynamic Test Execution .10
7.2.3 Test data repository .11
7.2.4 Test Environment .12
7.3 Code analysis .13
7.3.1 Input for code analysis .13
7.3.2 Code analysis .13
7.4 Test management .14
7.4.1 Test plan .14
7.4.2 Test asset .14
7.4.3 Quality record report .14
7.4.4 Test completion report .15
7.4.5 Verification and validation report .15
7.4.6 Test status report .15
7.5 Summary of capabilities with characteristics .16
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE iii
---------------------- Page: 3 ----------------------
ISO/IEC 30130:2015(E)
Annex A (informative) Capabilities mapping to test process .20
Annex B (informative) Overview of the approach for this International Standard .26
Bibliography .28
iv PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 30130:2015(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 7, Software
and systems engineering.
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE v
---------------------- Page: 5 ----------------------
ISO/IEC 30130:2015(E)
Introduction
This International Standard defines the framework to which capabilities of software testing tools are
allocated in order to identify the capabilities of products being used by any project for software testing.
To develop high-quality software with reasonable time and budget, the use of software testing tools is
required. The increase in the size and complexity of software is complemented by an increase in the
difficulty and complexity of software testing. This created a larger demand for the support of tools in
order to test software effectively and efficiently.
Testing tools are highly diverse due to their contexts of use. Testing itself varies by objective, such as
functional testing or nonfunctional testing, and granularity, such as unit testing or system testing.
Testing tool vendors vary by providing tools with a different function or combination of functions.
And despite vendor provided explanations for the type of testing support functions, there is little
common understanding of these functions. In this environment, it is difficult to utilize a testing tool
that is suitable for a project without common understanding of tool functions, proper acquisition of the
needed tools, and efficient training.
The framework defined by this International Standard consists of objectives of testing, granularity of
software to be tested and capabilities. In Clause 4, an object model for software testing tools as basis for
the framework is defined. In Clause 5, three of the categories in that software testing model (Dynamic
Test Execution, Code Analysis, and Test Management) are specified. In Clause 6, quality characteristics,
granularity, and other aspects of characteristics are defined and in Clause 7, tool capabilities are
mapped onto those categories and characteristics.
vi PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 30130:2015(E)
Software engineering — Capabilities of software testing
tools
1 Scope
This International Standard defines the framework to which capabilities of software testing tools are
allocated in order to identify the capabilities of products being used by any project for software testing.
Software testing processes are identified in ISO/IEC/IEEE 29119-2 and software verification processes
are identified in ISO/IEC 12207. This International Standard is fully harmonized with these existing
standards in terms of software testing processes.
This International Standard focuses on the following areas that the existing ISO/IEC standards do not
deal with the following:
— categorization of software test entities and software testing tools (Clauses 4 and 5);
— characterization of each software testing tool category (Clauses 5 and 6);
— mapping of software testing tool capabilities and characteristics (Clauses 6 and 7).
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 25010, Systems and software engineering — Systems and software Quality Requirements and
Evaluation (SQuaRE) — System and software quality models
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
3.1
quality record report
report which is generated through dynamic test execution and code analysis to record test results
and other output
Note 1 to entry: Including Test Result, Static Code Analysis Report, Test Incident Report, and Metrics.
3.2
test target version
specific version of test target which is used for one-time execution of Dynamic Test Execution or
Code Analysis
3.3
testing tool
specific or generic tool which is used for test execution and test management such as test results
recording, test results display, test results interpretation, generation of test data, generation of test
procedure, generation of test scripts, test modelling, etc.
3.4
package
namespace for the grouped elements
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE 1
---------------------- Page: 7 ----------------------
ISO/IEC 30130:2015(E)
4 Object model for software testing tools
4.1 Overview of the object model
A software testing tool is described by its input, process, output and environment. To define software
testing tools, the object model for software testing is identified.
The object model for software testing tools consists of the following elements:
a) test process, which represents processes of dynamic test;
b) code analysis process, which represents processes of code analysis;
c) test entity, which represents entities that appear in the process;
d) test tool, which supports inputs, processes, outputs and test entity.
Test Process comprises multiple subprocesses. Each process has input and output. These input and
output are basically test entities. Test processes are specified in detail by ISO/IEC/IEEE 29119-2.
Code Analysis Process comprises multiple sub processes. Each process has input and output. These
input and output are basically test entities.
Test Entity comprises multiple entities. The entity is referred to as Dynamic Test Execution Entity that
is required or created in Dynamic Test Process. The entity is referred to as Code Analysis Entity that is
required or created in Code Analysis Process. A test entity represents a single performance of a test, in
which the target of the test is tested in various respects. Testing has two modes of execution. Dynamic
Test Execution involves actual execution of the code and Code Analysis involves examination of the
source code of the target.
The testing tools take test entities as input and produce test entities. By producing test entities, testing
tools effectively support the testing process.
The object model diagrams, Figures 1 to 5, are described using UML 2 (ISO/IEC 19505-2).
Test Process includes Dynamic Test Process and Test Management Process.
Code Analysis Process includes Code Analysis Process and Code Analysis Management Process.
Test Entity includes Test Management Entity, Dynamic Test Execution Entity and Code Analysis Entity,
and Test Target.
Test Tool includes Test Execution Tool and Test Management Tool.
2 PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 30130:2015(E)
Figure 1 — Object model of software testing
Test Process and Test Tool have indirect relationships which are described in Annex A.
4.2 Test target
Test Target is a set of source code and executable code of software that is in development. It has a
hierarchical structure, and any part of the structure or the whole structure is tested.
Test Target Version is a specific version of the software.
Figure 2 — Object model of the package “Test Target”
4.3 Dynamic test execution entity
Dynamic Test Execution Entity is prepared to identify necessary entities which are used in one-time
dynamic test execution. Test Data, Test Target Version, Test Result, and Expected Result are identified
in this package.
In one-time dynamic test execution, Test Environment is used.
Test Target Version which is used in one-time dynamic test execution is extracted from an appropriate
granularity and version of Test Target.
Test Data which is used in one-time dynamic test execution is designed for Test Case which is derived
from Specification and is stored on Test Data Collections in Test Data Repository.
Specification, Test Case, and Test Target are referred to as input for Dynamic Test Execution.
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE 3
---------------------- Page: 9 ----------------------
ISO/IEC 30130:2015(E)
Test Data Repository has Test Data which is used by all Dynamic Test Execution and is classified by
Objective and Technique.
Figure 3 — Object model of the package “Dynamic Test Execution Entity”
4.4 Code analysis entity
Code Analysis Entity is prepared to identify necessary entities which are used in one-time code analysis.
Test Target Version and Code Analysis Result are identified in this package.
Test Target Version which is used in one-time code analysis is extracted from an appropriate granularity
and version of Test Target.
Test Target and Check List which are used in Code Analysis are referred to as Input for Code Analysis.
Check List usually does not depend on Specification.
Figure 4 — Object model of the package “Code Analysis Entity”
4.5 Test management entity
Test Management Entity is prepared to identify necessary packages and entities which are used to
manage the entire test processes. Test Plan, Test Asset, Quality Record Report, Test Completion Report
,Verification and Validation Report,and Test Status Report are identified in this package.
4 PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 30130:2015(E)
Dynamic Test Execution Documentation and Static Code Analysis Report are referred to as Quality
Record Report.
Dynamic Test Execution Documentation is composed by Metric, Test Result, and Test Incident Report.
Static Code Analysis Report is composed by Metric, Static Code Analysis Report, and Test Incident Report.
Figure 5 — Object model of the package “Test Management Entity”
5 Category of test entity
5.1 Overview
In this Clause, categories of test entities are defined to allocate and specify the capabilities of software
tools. Each category is identified by analyzing objective, life history, and usage of each test entity.
5.2 Categories of dynamic test execution entities
5.2.1 Input for dynamic test execution
Entities in this category are used in Dynamic Test Execution as inputs, such as Specification, Test Case,
and Test Target.
5.2.2 Dynamic test execution
Entities in this category are used in one-time dynamic testing. It comprises Test Data, Test Result,
Expected Result, and Test Target Version.
5.2.3 Test data repository
Entities in this category are used in Dynamic Test Execution. It comprises Test Data Collections, in which
there are a large number of test data for the overall Dynamic Test Execution. Test Data Collections can
be subsets which depend on each objective and techniques.
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE 5
---------------------- Page: 11 ----------------------
ISO/IEC 30130:2015(E)
5.2.4 Test environment
An entity, Target Environment, in this category is required to define a condition to evaluate Test
Target Version by one-time dynamic testing, such as hardware, operating system, middleware and test
drivers, etc.
5.3 Categories of code analysis entities
5.3.1 Input for code analysis
Entities in this category are used in Code Analysis as inputs, such as Check List and Test Target.
5.3.2 Code analysis
Entities in this category are used in one-time code analysis. It comprises of Test Target Version and
Code Analysis Result.
5.4 Categories of test management entities
5.4.1 Test plan
This category is used for testing that can be performed given constraints on personnel, material and
budget.
5.4.2 Test asset
Entities in this category is identified as a list of Test Data Collections, Test Target Version, and Check
List, which may be reused (e.g. for regression testing).
5.4.3 Quality record report
The entities in this category are created thorough test executions. It comprises Dynamic Test Execution
Documentation and Static Code Analysis Report.
5.4.4 Test completion report
The entities in this category provides a summary of the testing that was performed. This may be for the
project/programme as a whole or for the particular test sub-process.
5.4.5 Verification and validation report
This category is used for reporting verification and validation items, on which the final version of the
Test Completion Report is based.
5.4.6 Test status report
This category is used for reporting the status of testing that is performed in a specified reporting period.
6 Characteristics of software testing tools
6.1 Overview
Test objectives and test granularity are the key drivers of test scope. Other drivers of test scope, such as
test personnel, cost, and schedule constraints also have an effect on the scope of testing.
6 PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 12 ----------------------
ISO/IEC 30130:2015(E)
The most important aspect of characteristics is objectives of the software test, software quality. This
aspect is defined by SQuaRE (ISO/IEC 25010) and top level of the characteristics is explained in 6.2.
The second important aspect of characteristics is a size of component to be tested, granularity. This
aspect is defined in 6.3.
For other aspects of characteristics, the reader is suggested to add those aspects if necessary.
6.2 Quality characteristics
Quality characteristics which a test target shall satisfy are defined as described below by SQuaRE
(ISO/IEC 25010).
a) Functional Suitability
degree to which a product or system provides functions that meet stated and implied needs when
used under specified conditions:
— functional completeness;
— functional correctness;
— functional appropriateness.
b) Reliability
degree to which a system, product or component performs specified functions under specified
conditions for a specified period of time:
— maturity;
— availability;
— fault tolerance;
— recoverability.
c) Usability
degree to which a product or system can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use.
— appropriateness recognizability;
— learnability;
— operability;
— user error protection;
— user interface aesthetics;
— accessibility.
d) Performance Efficiency
performance relative to the amount of resources used under sated conditions:
— time behavior;
— resource utilization;
— capacity.
© ISO/IEC 2015 – All rights reserved PROOF/ÉPREUVE 7
---------------------- Page: 13 ----------------------
ISO/IEC 30130:2015(E)
e) Maintainability
degree of effectiveness and efficiency with which a product or system can be modified by the
intended maintainers.
— modularity;
— reusability;
— analysability;
— modifiability;
— testability.
f) Portability
degree of effectiveness and efficiency with which a system, product or component can be
transferred from one hardware, software or other operational or usage environment to another.
— adaptability;
— installability;
— replaceability.
g) Compatibility
degree to which product, system or component can exchange information with other products,
system or components, and/or perform its required functions, while sharing the same hardware or
software environment:
— co-existence;
— interoperability.
h) Security
degree to which a product or system protects information and data so that persons or other
products or systems have the degree of data access appropriate to their types and levels of
authorization.
— confidentiality;
— integrity;
— non-repudiation;
— accountability;
— authenticity.
6.3 Granularity
Granularity which is defined in this international standard is a size of a test target. The size of a test
target through test phases varies from small pieces to a whole system.
a) Smallest Unit
Smallest Unit is defined as a unit for smallest subsets of the test target.
Modules, components, functions, subroutines or other such atomic units of description are often
used as the Smallest Unit.
8 PROOF/ÉPREUVE © ISO/IEC 2015 – All rights reserved
---------------------- Page: 14 ----------------------
ISO/IEC 30130:2015(E)
b) Intermediate Unit
Intermediate Unit is defined as a unit for combination of Smallest Units of the test target.
c) Largest Unit
Largest Unit is defined as a unit for whole system of the test target.
NOTE Test levels usually correspond to the granularity, such as unit test, integration test, system test and
acceptance test. This standard covers any test level and test type.
6.4 Other aspects of characteristics
In addition to quality characteristics and granularity, other aspects of characteristics may have impac
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.