ISO/IEC JTC 1/SC 7/WG 26 - Software testing

This document specifies general concepts in software testing and presents key concepts for the ISO/IEC/IEEE 29119 series.

This document defines test design techniques that can be used during the test design and implementation process that is defined in ISO/IEC/IEEE 29119‑2. Each technique follows the test design and implementation process that is defined in ISO/IEC/IEEE 29119‑2 and shown in Figure 1. This document is intended for, but not limited to, testers, test managers, and developers, particularly those responsible for managing and implementing software testing.

This document specifies software test documentation templates that can be used for any organization, project or testing activity. It describes the test documentation that is an output of the processes specified in ISO/IEC/IEEE 29119-2. This document is applicable to testing in all software development lifecycle models. This document is intended for, but not limited to, testers, test managers, developers, and project managers, particularly those responsible for governing, managing, and implementing software testing.

This document specifies test processes that can be used to govern, manage and implement software testing for any organization, project or testing activity. It comprises generic test process descriptions that define the software testing processes. Supporting informative diagrams describing the processes are also provided. This document is applicable to testing in all software development lifecycle models. This document is intended for, but not limited to, testers, test managers, developers and project managers, particularly those responsible for governing, managing and implementing software testing.

This document provides guidance for the application of ISO/IEC/IEEE 29119 (all parts) in agile life cycles. This document is intended for (and not limited to) testers, test managers, business analysts, product owners, Scrum masters and developers involved in agile projects. The mappings provided in this document are designed to benefit any team or organization that is either moving away from traditional/waterfall life cycles and into agile or vice versa as well as new organizations that are commencing agile as their chosen life cycle. It is designed to be understandable regardless of the reader's familiarity with ISO/IEC/IEEE 29119 (all parts).

This document provides an introduction to AI-based systems. These systems are typically complex (e.g. deep neural nets), are sometimes based on big data, can be poorly specified and can be non-deterministic, which creates new challenges and opportunities for testing them. This document explains those characteristics which are specific to AI-based systems and explains the corresponding difficulties of specifying the acceptance criteria for such systems. This document presents the challenges of testing AI-based systems, the main challenge being the test oracle problem, whereby testers find it difficult to determine expected results for testing and therefore whether tests have passed or failed. It covers testing of these systems across the life cycle and gives guidelines on how AI-based systems in general can be tested using black-box approaches and introduces white-box testing specifically for neural networks. It describes options for the test environments and test scenarios used for testing AI-based systems. In this document an AI-based system is a system that includes at least one AI component.

ISO/IEC 20246:2017 establishes a generic framework for work product reviews that can be referenced and used by all organizations involved in the management, development, test and maintenance of systems and software. It contains a generic process, activities, tasks, review techniques and documentation templates that are applied during the review of a work product. A work product is any artefact produced by a process. This document defines work product reviews that can be used during any phase of the life cycle of any work product. This document is intended for, but not limited to, project managers, development managers, quality managers, test managers, business analysts, developers, testers, customers and all those involved in the development, testing and maintenance of systems and software.

ISO/IEC/IEEE 29119-5:2016 defines an efficient and consistent solution for Keyword-Driven Testing by: giving an introduction to Keyword-Driven Testing; providing a reference approach to implement Keyword-Driven Testing; defining requirements on frameworks for Keyword-Driven Testing to enable testers to share their work items, such as test cases, test data, keywords, or complete test specifications; defining requirements for tools that support Keyword-Driven Testing. These requirements could apply to any tool that supports the Keyword-Driven approach (e.g., test automation, test design and test management tools); defining interfaces and a common data exchange format to ensure that tools from different vendors can exchange their data (e.g. test cases, test data and test results); defining levels of hierarchical keywords, and advising use of hierarchical keywords. This includes describing specific types of keywords (e.g. keywords for navigation or for checking a value) and when to use "flat" structured keywords; providing an initial list of example generic technical (low-level) keywords, such as "inputData" or "checkValue". These keywords can be used to specify test cases on a technical level, and may be combined to create business-level keywords as required. NOTE This standard is applicable to all those who want to create keyword-driven test specifications, create corresponding frameworks, or build test automation based on keywords.

ISO/IEC/IEEE 29119-4:2015 defines test design techniques that can be used during the test design and implementation process that is defined in ISO/IEC/IEEE 29119‑2. ISO/IEC/IEEE 29119-4:2015 is intended for, but not limited to, testers, test managers, and developers, particularly those responsible for managing and implementing software testing.

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing. ISO/IEC/IEEE 29119-2:2013 comprises test process descriptions that define the software testing processes at the organizational level, test management level and dynamic test levels. It supports dynamic testing, functional and non-functional testing, manual and automated testing, and scripted and unscripted testing. The processes defined in ISO/IEC/IEEE 29119-2:2013 can be used in conjuntion with any software development lifecycle model. Since testing is a key approach to risk-mitigation in software development, ISO/IEC/IEEE 29119-2:2013 follows a risk-based approach to testing. Risk-based testing is a common industry approach to strategizing and managing testing. Risk-based testing allows testing to be prioritzed and focused on the most important features and functions.

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing. ISO/IEC/IEEE 29119-3:2013 includes templates and examples of test documentation. The templates are arranged within clauses reflecting the overall test process description structure in ISO/IEC/IEEE 29119-2, i.e. by the test process in which they are being produced. Annex A contains outlines of the contents of each document. Annex B contains mappings of ISO/IEC/IEEE 29119-2. Annex C contains an overview of the examples. Annexes D to S contain examples of the application of the templates. Annex T provides mappings to existing standards. The Bibliography for ISO/IEC/IEEE 29119-3:2013 is at the end of the document. ISO/IEC/IEEE 29119-3:2013 supports dynamic testing, functional and non-functional testing, manual and automated testing, and scripted and unscripted testing. The documentation templates defined in ISO/IEC/IEEE 29119-3:2013 can be used in conjunction with any software development lifecycle model.

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing. ISO/IEC/IEEE 29119-1:2013 facilitates the use of the other ISO/IEC/IEEE 29119 standards by introducing the concepts and vocabulary on which these standards are built, as well as providing examples of its application in practice. ISO/IEC/IEEE 29119-1:2013 is informative, providing a starting point, context, and guidance for the other parts.