ISO/IEC JTC 1/SC 7/WG 6 - Software Product and System Quality

This document defines a quality-in-use model composed of three characteristics (which are further subdivided into sub-characteristics) that can influence stakeholders when products or systems are used in a specified context of use. This model is applicable to the entire spectrum of information system and IT service system, including both computer systems in use and software products in use. This document provides a set of quality characteristics for specifying, measuring, evaluating and improving quality-in-use. In this document, because context of use is specified as prerequisite of quality-in-use, context of use is necessary to be re-specified to change prerequisite when a product or service intend to fulfil to context of use changes. The model can be applied, in particular, by those responsible for specifying and evaluating software product quality, such as developers, acquirers, quality assurance and control staff, and independent evaluators. Activities during product development that can benefit from the use of the quality model can include, but are not limited to: — identifying requirements for information system and IT service system in use; — validating the comprehensiveness of a quality-in-use requirements specification; — identifying information system and IT service system design objectives for quality-in-use; — identifying quality-in-use control criteria as part of overall quality assurance; — identifying acceptance criteria for information system and IT service system or information systems; — establishing measures to address the consequences of using products in specified context-of -use; — presenting evaluation items for ethics considerations when using information system and IT service system; — supporting governance of digitalization activities.

This document defines a product quality model, which is applicable to ICT (information and communication technology) products and software products. The product quality model is composed of nine characteristics (which are further subdivided into subcharacteristics) that relate to quality properties of the products. The characteristics and subcharacteristics provide a reference model for the quality of the products to be specified, measured and evaluated. NOTE 1 In this document, a product refers to an ICT product that is part of an information system. ICT product components include subsystems, software, firmware, hardware, data, communication infrastructure, and other elements that are part of the ICT product. This model can be used for requirements specification and evaluation of the target products’ quality throughout their lifecycle by several stakeholders, including developers, acquirers, quality assurance and control staff and independent evaluators. Activities in the product lifecycle that can benefit from the use of this model include: — eliciting and defining product and information system requirements; — validating the comprehensiveness of requirements definition; — identifying product and information system design objectives, and design necessary process for achieving quality; — identifying product and information system testing objectives; — identifying quality control criteria as the part of quality assurance; — identifying acceptance criteria for a product and/or an information system; — establishing measures of product quality characteristics in support of these activities. NOTE 2 Usage of the quality model for measurement is explained in Annex C.

This document defines quality measures useful for requirements and evaluation of IT service quality in terms of characteristics and sub-characteristics defined in ISO/IEC TS 25011. This document contains a basic set of quality measures for each characteristic and sub-characteristic. This document does not assign ranges of values of the quality measures to rated levels or to grades of compliance. Such values are defined based on the nature of the IT service, and so depends on factors such as category of the IT service or users' needs. Some attributes can have a desirable range of values, which does not depend on specific user needs but generic factors, for example, service downtime. This document includes, in Annex A, considerations for the selection and application of quality measures. The quality measures in this document are primarily intended to be used for quality evaluation and improvement of IT services during or after the development life cycle. The main users of this document are people carrying out quality requirements specification and evaluation activities for IT services as part of the following: — development: including requirements analysis, design, implementation, testing and deployment during the development life cycle; — quality management: monitoring activities of quality assurance and performing quality control of an IT service; — supply: making a contract with the user for supplying an IT service under the terms of a contract; — acquisition: including IT service selection, when acquiring or procuring an IT service from a service provider; — maintenance: improvement of an IT service based on quality measurement. The relationship of this document to domain-specific IT service quality model and its precedence over this document is determined by the user in a specific context of use.

This document provides the framework for quality requirements for systems, software products and data, which includes concept of the quality requirements, and requirements and recommendations for the processes and methods to elicit, define, use and govern them. Intended readers of this document include, but are not limited to: — acquirers: evaluate if the system/software products/data fulfills their value proposition, i.e., meets the expected quality, — developers: design, implement and test the system/software products/data to ensure that it meets the expected quality, — testers: verify and validate that the system/software products/data meets the expected quality, — project managers: plan, monitor and control the achievement of the expected quality, and — independent evaluators: evaluate the system/software products/data with the objective criteria. This document complies with the technical processes defined in ISO/IEC/IEEE 15288, which are relevant for elicitation of stakeholders' quality needs and for defining, analyzing and maintaining quality requirements. In this document, the quality models in ISO/IEC 25010 and ISO/IEC 25012 are used to categorize quality requirements and to provide a basis for quantifying them in terms of quality measures in the quality measure division of ISO/IEC 2502n. This document does not cover specification of the other requirements (such as functional requirements, process requirements, etc.), and prescribes neither any specific quality measure nor any specific development process.

This document provides a framework for developing quality measurement. The contents of this document are as follows: — quality measurement reference model; — relationships among different types of quality measures; — guidelines for selecting quality measures; — guidelines for constructing quality measures; — guidelines for planning and performing measurements; — guidelines for the application of measurement results. It includes considerations for selecting quality measures and quality measure elements (Annex A), assessing the reliability of measurement and the validity of quality measures (Annex B), elements for documenting quality measures (Annex C), normalized measurement function for quality measures (Annex D) and the measurement information model in ISO/IEC/IEEE 15939 (Annex E). This document can be applied for designing, identifying, evaluating and executing the measurement model of system and software product quality, quality in use, data quality and IT service quality. This reference model can be used by developers, acquirers, quality assurance staff and independent evaluators—essentially by people responsible for specifying and evaluating the quality of information and communication technology (ICT) systems and services.

ISO/IEC TS 25011:2017 is applicable to IT services that support the needs of an individual user or a business. IT services can be delivered personally or remotely by people, or by an IT application that could be in a local or remote location (see Annex A). These include two types of IT services: a) services completely automated provided by an IT system; b) services provided by a human using an IT system. ISO/IEC TS 25011:2017 describes the use of two quality models for IT services. a) ISO/IEC TS 25011:2017 defines an IT service quality model composed of eight characteristics (which are further subdivided into sub-characteristics) that relate to properties of the IT service made up from a combination of elements including people, processes, technology, facilities and information. b) ISO/IEC TS 25011:2017 describes how the quality in use model in ISO/IEC 25010 which is composed of five characteristics (some of them are further subdivided into sub-characteristics) can be applied to the outcome when an IT service is used in a particular context of use. This model is applicable to the complete service provision system composed of people, processes, technology, facilities and information. The characteristics and sub-characteristics provide consistent terminologies and check lists for specifying, measuring and evaluating IT service quality. The use of the IT service quality models can help: - IT service providers to identify service quality requirements, and evaluate and improve the quality of the service provided; - customers to specify their requirements for the quality of service, define the acceptance criteria for service, and evaluate the quality of an IT service; and - a third party to evaluate the quality of an IT service.

ISO/IEC 25000:2014 provides guidance for the use of the new series of International Standards named Systems and software Quality Requirements and Evaluation (SQuaRE). The purpose of ISO/IEC 25000:2014 is to provide a general overview of SQuaRE contents, common reference models and definitions, as well as the relationship among the documents, allowing users of the Guide a good understanding of those series of standards, according to their purpose of use. It also contains an explanation of the transition process between the old ISO/IEC 9126 and the ISO/IEC 14598 series and SQuaRE.

ISO/IEC 25051:2014 establishes: quality requirements for Ready to Use Software Product (RUSP); requirements for test documentation for the testing of RUSP, including test plan, test description, and test results; instructions for conformity evaluation of RUSP. It includes also recommendations for safety or business critical RUSP. ISO/IEC 25051:2014 deals only with providing the user with confidence that the RUSP will perform as offered and delivered. It does not deal with the production realization (including activities and intermediate products, e.g. specifications). The quality system of a supplier is outside the scope of ISO/IEC 25051:2014.

ISO/IEC 25021:2012 provides guides to specify Quality Measure Elements (QME) and initial set of QME as examples. QME is a measure defined in terms of a property and the measurement method for quantifying it, including optionally the transformation by a mathematical function. ISO/IEC 25021:2012 is intended to be used throughout the system and software product life cycle used with other documents of the ISO/IEC 250nn SQuaRE series, especially ISO/IEC 25022, ISO/IEC 25023 and ISO/IEC 25024.

ISO/IEC 25010:2011 defines: A quality in use model composed of five characteristics (some of which are further subdivided into subcharacteristics) that relate to the outcome of interaction when a product is used in a particular context of use. This system model is applicable to the complete human-computer system, including both computer systems in use and software products in use. A product quality model composed of eight characteristics (which are further subdivided into subcharacteristics) that relate to static properties of software and dynamic properties of the computer system. The model is applicable to both computer systems and software products. The characteristics defined by both models are relevant to all software products and computer systems. The characteristics and subcharacteristics provide consistent terminology for specifying, measuring and evaluating system and software product quality. They also provide a set of quality characteristics against which stated quality requirements can be compared for completeness. Although the scope of the product quality model is intended to be software and computer systems, many of the characteristics are also relevant to wider systems and services. ISO/IEC 25012 contains a model for data quality that is complementary to this model. The scope of the models excludes purely functional properties, but it does include functional suitability. The scope of application of the quality models includes supporting specification and evaluation of software and software-intensive computer systems from different perspectives by those associated with their acquisition, requirements, development, use, evaluation, support, maintenance, quality assurance and control, and audit. The models can, for example, be used by developers, acquirers, quality assurance and control staff and independent evaluators, particularly those responsible for specifying and evaluating software product quality. Activities during product development that can benefit from the use of the quality models include: identifying software and system requirements; validating the comprehensiveness of a requirements definition; identifying software and system design objectives; identifying software and system testing objectives; identifying quality control criteria as part of quality assurance; identifying acceptance criteria for a software product and/or software-intensive computer system; establishing measures of quality characteristics in support of these activities.

ISO/IEC 25040:2011 contains requirements and recommendations for the evaluation of software product quality and clarifies the general concepts. It provides a process description for evaluating software product quality and states the requirements for the application of this process. The evaluation process can be used for different purposes and approaches. The process can be used for the evaluation of the quality of pre-developed software, commercial-off-the-shelf software or custom software and can be used during or after the development process. ISO/IEC 25040:2011 establishes the relationship of the evaluation reference model to the SQuaRE documents as well as shows how each SQuaRE document should be used during the activities of the evaluation process. It is intended for those responsible for software product evaluation and is appropriate for developers, acquirers and independent evaluators of software products. These three different approaches are detailed in ISO/IEC 14598-3, ISO/IEC 14598-4, and ISO/IEC 14598-5. It is not intended for evaluation of other aspects of software products (such as functional requirements, process requirements, business requirements, etc.).

ISO/IEC 25045:2010 is one of the SQuaRE series of International Standards, which provides a framework for software products quality requirements and evaluation including the requirements for methods of software product measurement and evaluation. ISO/IEC 25045:2010 uses a methodology involving two types of evaluation for recoverability. One part of the method makes use of the disturbance injection methodology and a list of disturbances based on common categories of operational faults and events to evaluate the quality measure of resiliency. The second quality measure is based on a set of questions that is defined for each disturbance to evaluate the quality measure of autonomic recovery index by assessing how well the system detects, analyses, and resolves the disturbance without human intervention. ISO/IEC 25045:2010 is applicable to information systems executing transactions in a system supporting single or multiple concurrent users, where speedy recovery and ease of managing recovery is important to the acquirer, owner/operator, and the developer.

This part of ISO/IEC 14598 defines the structure and content of the documentation to be used to describe an Evaluation Module. Evaluation modules are intended to be used within the context of the ISO/IEC 9126 and the ISO/IEC 14598 multipart standards. This part of ISO/IEC 14598 is intended to be used by experts in evaluation technology such as testing laboratories, research institutes and others when producing new evaluation modules.

This International Standard defines how user oriented performance of computer-based software systems (CBSS) may be measured and rated. A CBSS is a data processing system as it is seen by its users, e.g. by users at various terminals, or as it is seen by operational users and business users at the data processing center. A CBSS includes hardware and all its software (system software and application software) which is needed to realize the data processing functions required by the users or what may influence to the CBSS's time behaviour. This International Standard is applicable for tests of all time constrained systems or system parts. Also a network may be part of a system or may be the main subject of a test. The method defined in this International Standard is not limited to special cases like classic batch or terminal-host systems, e.g. also included are client server systems or, with a broader comprehension of the definition of ?task', real time systems. But the practicability of tests may be limited by the expenditure required to test large environments. This International Standard specifies the key figures of user oriented performance terms and specifies a method of measuring and rating these performance values. The specified performance values are those which describe the execution speed of user orders (tasks), namely the triple of: - execution time, - throughput, - timeliness. The user orders, subsequently called tasks, may be of simple or complex internal structure. A task may be a job, transaction, process or a more complex structure, but with a defined start and end depending on the needs of the evaluator. When evaluating the performance it is possible to use this International Standard for measuring the time behaviour with reference to business transaction completion times in addition to other individual response times. The rating is done with respect to users requirements or by comparing two or more measured systems (types or versions). Intentionally no proposals for measuring internal values, such as: - utilisation values, - mean instruction rates, - path lengths, - cache hit rates, - queuing times, - service times, are given, because the definition of internal values depends on the architecture of the hardware and the software of the system under test. Contrary to this the user oriented performance values which are defined in this International Standard are independent of architecture. The definition of internal performance values can be done independently from the definition of user oriented performance values. They may be used and can be measured in addition to the user oriented performance values. Also the definition of terms for the efficiency with which the user oriented values are produced can be done freely. In addition this International Standard gives guidance on how to establish at a data processing system a stable and reproducible state of operation. This reproducible state may be used to measure other performance values such as the above mentioned internal values. This International Standard focuses on: - application software; - system software; - turn-key systems (i.e. systems consisting of an application software, the system software and the hardware for which it was designed); - general data processing systems. This International Standard specifies the requirements for an emulation (by a technical system - the so-called remote terminal emulator (RTE) - of user interactions with a data processing system. It is the guideline for precisely measuring and rating the user oriented performance values. It provides the guideline for estimating these values with the required accuracy and repeatability of CBSSs with deterministic as well as random behaviour of users. It is also a guidance for implementing a RTE or proving whether it works according to this International Standard. This International Standard provides the guideline to measure and rate the perf

ISO/IEC TR 25021:2007 defines the set of quality measure elements to be used throughout the software product life cycle for the purpose of Software Product Quality Requirement and Evaluation (SQuaRE). While some quality measure elements can be used as standalone quality measures, their main purpose is to be the building blocks for other SQuaRE measurements as described in ISO/IEC TR 9126-2, ISO/IEC TR 9126-3 and ISO/IEC TR 9126-4. ISO/IEC TR 25021:2007 constitutes the link between ISO/IEC 9126 and the subsequent SQuaRE series of standards. ISO/IEC TR 25021:2007 contains informative annexes documenting the cross-reference relationship between the quality measure elements and quality measures, characteristics and subcharacteristics defined in ISO/IEC 9126. The Quality Measurement Division, of which ISO/IEC TR 25021:2007 is a member, also offers examples of quality measures that can be used across the product development life-cycle. These measures are defined in the other documents in the division and correspond to the quality characteristics in a software product quality model such as that described in the future International Standard ISO/IEC 25010 (replacing ISO/IEC 9126-1). ISO/IEC TR 25021:2007 is designed to be used, in particular, with other standards in the SQuaRE series that address quality requirements (ISO/IEC 25030) and product quality evaluation (the future International Standard ISO/IEC 25040).

ISO/IEC 25051:2005 defines quality requirements for COTS software products. A COTS software product includes the product description, the user documentation, and the software contained on a computer sensible media. The quality requirements, functionality, reliability, usability, efficiency, maintainability, portability, and quality in use are consistent with the definitions of SQuaRE. In addition to the quality requirements, ISO/IEC 25051:2005 also defines requirements for test documentation. The test documentation purpose is to demonstrate the conformity of the software with the requirements. The documentation contains a test plan, the description of the tests cases, and the tests results. ISO/IEC 25051:2005 can be used by suppliers, certification bodies, testing laboratories, accreditation bodies, regulatory authorities and acquirers.

ISO/IEC 25000:2005 provides guidance for the use of the new series of International Standards named Software product Quality Requirements and Evaluation (SQuaRE). The purpose of this guide is to provide a general overview of SQuaRE contents, common reference models and definitions, as well as the relationship among the documents, allowing users of this guide a good understanding of those series of International Standards, according to their purpose of use. This document contains an explanation of the transition process between the old ISO/IEC 9126 and the 14598 series and SQuaRE, and also presents information on how to use the ISO/IEC 9126 and 14598 series in their previous form. SQuaRE provides: Terms and definitions, Reference models, General guide, Individual division guides, and Standards for requirements specification, planning and management, measurement and evaluation purposes.

ISO/IEC TR 9126-4:2004 provides quality in use metrics for measuring the attributes defined in ISO/IEC 9126-1. ISO/IEC TR 9126-2 defines external metrics and ISO/IEC TR 9126-3 defines internal metrics for measurement of the subcharacteristics defined in ISO/IEC 9126-1. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use. The metrics listed in ISO/IEC TR 9126-4 are not intended to be an exhaustive set. Developers, evaluators, quality managers and acquirers may select metrics from ISO/IEC TR 9126-4 for defining requirements, evaluating software products, measuring quality aspects and other purposes. ISO/IEC TR 9126-2 is intended to be used together with ISO/IEC 9126-1. ISO/IEC TR 9126-4 contains: -- an explanation of how to apply software quality metrics; -- a basic set of metrics for each characteristic; and -- an example of how to apply metrics during the software product life cycle. It includes as informative annexes a quality in use evaluation process and a reporting format.

ISO/IEC TR 9126-3:2003 provides internal metrics for measuring attributes of six external quality characteristics defined in ISO/IEC 9126-1. ISO/IEC TR 9126-2 defines external metrics, ISO/IEC TR 9126-3:2003 defines internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or the subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use. The metrics listed in ISO/IEC TR 9126-3:2003 are not intended to be an exhaustive set. Developers, evaluators, quality managers, maintainers, suppliers, users and acquirers may select metrics from ISO/IEC TR 9126-3:2003 for defining requirements, evaluating software products, measuring quality aspects and other purposes. Users of ISO/IEC TR 9126-3:2003 can select or modify and apply metrics and measures from ISO/IEC TR 9126-3:2003 or may define application-specific metrics for their individual application domain. For internal metrics view, there are pure internal metrics proposed for reference purposes. ISO/IEC TR 9126-3:2003 is intended to be used together with ISO/IEC 9126-1. ISO/IEC TR 9126-3:2003 contains: an explanation of how to apply software quality metrics; a basic set of metrics for each subcharacteristic; an example of how to apply metrics during the software product life cycle. ISO/IEC TR 9126-3:2003 does not assign ranges of values of these metrics to rated levels or to grades of compliance, because these values are defined for each software product or a part of the software product, by its nature, depending on such factors as category of the software, integrity level and users' needs. Some attributes may have a desirable range of values, which does not depend on specific user needs but depends on generic factors; for example, human cognitive factors.

ISO/IEC TR 9126-2:2003 provides external metrics for measuring attributes of six external quality characteristics defined in ISO/IEC 9126-1. ISO/IEC TR 9126-2:2003 defines external metrics, ISO/IEC TR 9126-3 defines internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or the subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use. The metrics listed in ISO/IEC TR 9126-2:2003 are not intended to be an exhaustive set. Developers, evaluators, quality managers and acquirers may select metrics from ISO/IEC TR 9126-2:2003 for defining requirements, evaluating software products, measuring quality aspects and other purposes. Users of ISO/IEC TR 9126-2:2003 can select or modify and apply metrics and measures from ISO/IEC TR 9126-2:2003 or may define application-specific metrics for their individual application domain. ISO/IEC TR 9126-2:2003 is intended to be used together with ISO/IEC 9126-1. ISO/IEC TR 9126-2:2003 contains an explanation of how to apply software quality metrics, a basic set of metrics for each subcharacteristic and an example of how to apply metrics during the software product life cycle. ISO/IEC TR 9126-2:2003 does not assign ranges of values of these metrics to rated levels or to grades of compliance, because these values are defined for each software product or a part of the software product, by its nature, depending on such factors as category of the software, integrity level and users' needs. Some attributes may have a desirable range of values, which does not depend on specific user needs but depends on generic factors; for example, human cognitive factors.

This part of ISO/IEC 14598 provides requirements and recommendations for the practical implementation of software product evaluation when several parties need to understand, accept and trust evaluation results. In particular, it may be used to apply the concepts described in ISO/IEC 9126. The process described in this part of ISO/IEC 14598 defines the activities needed to analyse evaluation requirements, to specify, design and perform evaluation actions and to conclude the evaluation of any kind of software product. The evaluation process may be used to evaluate already existing products, provided the needed product components are available, or to evaluate products in development. NOTE For the evaluation of a product in development, the evaluation process needs to be synchronized with the software development process and product components are evaluated as they are delivered. This part of ISO/IEC 14598 may be used by - testing laboratory evaluators, when providing software product evaluation services, - software suppliers, when planning evaluation of their products, including evaluation to be carried out by independent testing services, - software acquirers, when requesting evaluation information from a supplier or testing service, - software users when evaluating products or when using evaluation reports provided by testing laboratories, - certification bodies in defining new certification schemes for software products.

Applicable to software packages. Establishes requirements for software packages and instructions on how to test a software package against these requirements. Deals only with software packages as offered and delivered. Does not deal with their production process. The quality system of a supplier is outside the scope of this standard.