SIST-TP CEN/CLC/TR 17602-80-04:2021
(Main)Space product assurance - Software metrication programme definition and implementation
Space product assurance - Software metrication programme definition and implementation
The scope of this Handbook is the software metrication as part of a space project, i.e. a space system, a subsystem including hardware and software, or ultimately a software product. It is intended to complement the EN 16602-80 (equivalent to ECSS-Q-ST-80) with specific guidelines related to use of different software metrics including their collection, analysis and reporting. Tailoring guidelines for the software metrication process are also provided to help to meet specific project requirements.
This Handbook provides recommendations, methods and procedures that can be used for the selection and application of appropriate metrics, but it does not include new requirements w ith respect to those provided by EN 16602-80 (equivalent to ECSS-ST-Q-80).
The scope of this Handbook covers the following topics:
• Specification of the goals and objectives for a metrication programme.
• Identification of criteria for selection of metrics in a specific project / environment (goal driven).
• Planning of metrication in the development life cycle.
• Interface of metrication with engineering processes.
• Data collection aspects (including use of tools).
• Approach to the analysis of the collected data.
• Feedback into the process and product based on the analysis results.
• Continuous improvement of measurement process.
• Use of metrics for process and product improvement.
This Handbook is applicable to all types of software of all major parts of a space system, including the space segment, the launch service segment and the ground segment software.
Raumfahrtproduktsicherung - Softwaremetrikhandbuch
Assurance produit des projets spatiaux - Guide de métrologie du logiciel
Zagotavljanje kakovosti proizvodov v vesoljski tehniki - Definicija in izvajanje programa za merjenje programske opreme
General Information
Standards Content (Sample)
SLOVENSKI STANDARD
01-december-2021
Zagotavljanje kakovosti proizvodov v vesoljski tehniki - Definicija in izvajanje
programa za merjenje programske opreme
Space product assurance - Software metrication programme definition and
implementation
Raumfahrtproduktsicherung - Softwaremetrikhandbuch
Assurance produit des projets spatiaux - Guide de métrologie du logiciel
Ta slovenski standard je istoveten z: CEN/CLC/TR 17602-80-04:2021
ICS:
03.120.99 Drugi standardi v zvezi s Other standards related to
kakovostjo quality
35.080 Programska oprema Software
49.140 Vesoljski sistemi in operacije Space systems and
operations
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
TECHNICAL REPORT
CEN/CLC/TR 17602-80-
RAPPORT TECHNIQUE
TECHNISCHER BERICHT
October 2021
ICS 49.140; 35.240.99
English version
Space product assurance - Software metrication
programme definition and implementation
Assurance produit des projets spatiaux - Guide de Raumfahrtproduktsicherung -
métrologie du logiciel Softwaremetrikhandbuch
This Technical Report was approved by CEN on 13 September 2021. It has been drawn up by the Technical Committee
CEN/CLC/JTC 5.
CEN and CENELEC members are the national standards bodies and national electrotechnical committees of Austria, Belgium,
Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy,
Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of North Macedonia, Romania, Serbia,
Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and United Kingdom.
CEN-CENELEC Management Centre:
Rue de la Science 23, B-1040 Brussels
© 2021 CEN/CENELEC All rights of exploitation in any form and by any means Ref. No. CEN/CLC/TR 17602-80-04:2021 E
reserved worldwide for CEN national Members and for
CENELEC Members.
Table of contents
European Foreword . 5
Introduction . 6
1 Scope . 8
2 References . 9
3 Terms, definitions and abbreviated terms . 12
3.1 Terms from other documents . 12
3.2 Definitions in other clauses of the present HB . 12
3.3 Terms specific to the present document . 12
3.4 Abbreviated terms. 13
4 Overview of the Handbook . 14
4.1 Introduction . 14
4.2 Relation to other ECSS Standards . 15
4.2.1 General . 15
4.2.2 Software engineering . 15
4.2.3 Software product assurance . 15
4.2.4 Project management . 16
5 A reference software quality model . 17
5.1 Introduction . 17
5.2 Reference software quality model . 18
5.3 Tailoring the metrication programme . 23
5.4 Detailed tailoring guidelines . 26
6 Measurement process . 29
6.1 Introduction . 29
6.2 Planning of metrication in the development life cycle . 30
6.2.1 Characterize the project quality requirements and select the metrics
to be collected . 30
6.2.2 Define data collection and analysis procedures . 30
6.2.3 Define criteria for validating the metrics and the measurement
process . 31
6.2.4 Define resources and infrastructure for measurement tasks . 32
6.2.5 Define how reporting will be performed . 32
6.2.6 Review and approve the measurement plan . 32
6.2.7 Provide resources and infrastructure for measurement tasks . 33
6.3 Data collection . 33
6.3.1 Integrate procedures . 33
6.3.2 Collect data . 34
6.4 Data validation . 34
6.5 Data analysis . 34
6.6 Data archiving . 35
6.7 Reporting . 35
6.8 Feedback to the measurement process . 36
6.8.1 Evaluate analysis results and the measurement process . 36
6.8.2 Identify potential improvements . 38
Annex A Definition of the quality model . 39
A.1 General introduction . 39
A.2 Characteristics and sub-characteristics definition. 39
A.2.1 Functionality . 39
A.2.2 Reliability . 40
A.2.3 Maintainability . 40
A.2.4 Reusability . 41
A.2.5 Suitability for safety . 41
A.2.6 Security . 42
A.2.7 Usability . 42
A.2.8 Software development effectiveness . 43
A.3 List of proposed metrics . 43
A.3.1 Introduction . 43
A.3.2 Standard metric template . 44
A.3.3 Detailed description of all metrics . 46
A.4 List of proposed OO metrics . 96
A.4.1 Introduction . 96
A.4.2 Detail description of the proposed OO metrics . 96
Figures
Figure 4-1:Organization of this document . 14
Figure 5-1: Elements of the software quality model . 19
Figure 6-1: Metrication process activities . 29
Figure A-1 : Example of report format for requirements allocation . 47
Figure A-2 : Example of SPR/NCR trend analysis . 52
Figure A-3 : Example of cyclomatic complexity . 60
Figure A-4 : Example of nesting level . 62
Figure A-5 : Sample of requirements stability . 93
Figure A-6 : Sample of RID/action status . 94
Figure A-7 : Sample of V&V progress . 95
Tables
Table 5-1: Proposed reference quality model . 21
Table 5-2: Applicability of the metrics depending on the criticality category . 24
Table 5-3: Target value for metric depending on criticality category . 25
Table A-1 : Example of V&V coverage reporting. 50
Table A-2 : Example of summary of NCR-SPR recorded . 51
Table A-3 : Sample checklist for Suitability of development documentation . 54
Table A-4 : Checklist for process reliability adequacy . 70
Table A-5 : Test coverage requirements . 73
Table A-6 : Sample checklist for reusability checklist . 77
Table A-7 : Sample checklist for safety activities adequacy . 81
Table A-8 : Checklist for security checklist. 83
Table A-9 : Checklist for User manual suitability . 87
European Foreword
This document (CEN/CLC/TR 17602-80-04:2021) has been prepared by Technical Committee
CEN/CLC/JTC 5 “Space”, the secretariat of which is held by DIN.
It is highlighted that this technical report does not contain any requirement but only collection of data
or descriptions and guidelines about how to organize and perform the work in support of EN 17602-
80.
This Technical report (CEN/CLC/TR 17602-80-04:2021) originates from ECSS-Q-HB-80-04A.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN [and/or CENELEC] shall not be held responsible for identifying any or all such
patent rights.
This document has been prepared under a mandate given to CEN by the European Commission and
the European Free Trade Association.
This document has been developed to cover specifically space systems and has therefore precedence
over any TR covering the same scope but with a wider domain of applicability (e.g.: aerospace).
Introduction
This Handbook describes the approach to be taken for the definition and implementation of an
effective and efficient metrication programme for the development of software in a space project.
This Handbook provides guidelines and examples of software metrics that can be used in space
system developments, in line with the requirements defined by [ECSS-E-40] and [ECSS-Q-80], given
guidelines and examples to provide a coherent view of the software metrication programme definition
and implementation.
This Handbook is intended to help customers in formulating their quality requirements and suppliers
in preparing their response and implementing the work. This Handbook is not intended to replace
textbook material on computer science or technology and software metrics, so repeating such material
is avoided in this Handbook. The readers and users of this Handbook are assumed to posse’s general
knowledge of computer science and software engineering.
In space projects, the demand of high quality software to be developed within allocated budget and
time is a priority objective, particularly in presence of dependability requirements. Also the size of the
operational software has increased significantly due to the increase in functionality to allow new
challenging missions and to reply to increasingly sophisticated and demanding user requirements.
The space software development is therefore characterized by:
• High dependability of the software products, in both space and ground segments;
• Stringent schedule and cost estimates that are more and more accurate to enable reliable cost
estimates for the overall project;
• Demand of high quality software, and
• Increasing productivity requirements.
To improve, suppliers should know what can be done better, and also what to look at in order to
understand where lessons learnt can be applied to support the improvement.
Measurements are the only way to quantitatively assess the quality of a process or a product.
In presence of complex software projects reliable measures of both processes and products provide a
powerful tool to software management for keeping the project in track and preserve the intended
quality of the software product. Improvement of both the space software product and its development
processes depends upon improved ability to identify, measure, and control essential parameters that
affect software product and its development processes. This is the goal of any software metrication
programme.
The main reasons for measuring software processes and products are:
To characterize the existing processes and the status of products under development or
operations, in order to gain a better understanding and support the overall verification of
the software, as well as to acquire data/information for future assessments of similar
processes/products.
To evaluate the status of the project to determine its status and possible deviation from
the established plans, and to support identification of actions to bring it back under
control; the evaluation includes an assessment of an achievement of quality goals and an
assessment of the impacts of technology and process improvements on products and
processes.
To predict, with the aim to improve the ability to plan. Measuring for prediction involves
gaining understanding of relationships among processes and products and building
models of these relationships, so that the observed values for some attributes can be used
to predict others. The reason for that is to establish achievable goals for cost, schedule,
and quality—so that appropriate resources can be applied. Predictive measures are also
the basis for extrapolating trends, so estimates for cost, time, and quality can be updated
based on current evidence. Projections and estimates based on historical data also help to
analyse risks and make design/cost tradeoffs.
To improve. Gathering quantitative information helps to identify roadblocks, root causes,
inefficiencies, and other opportunities for improving product quality and process
performance. Measures also help to plan and track improvement efforts.
Measures of current performance give baselines to compare against, so that it can be possible to judge
whether or not the improvement actions are working as intended and what the side effects can be.
Good measures also help to communicate goals and convey reasons for improving. This helps engage
and focus the support of those who work within the processes to make them successful.
Scope
The scope of this Handbook is the software metrication as part of a space project, i.e. a space system, a
subsystem including hardware and software, or ultimately a software product. It is intended to
complement the [ECSS-Q-80] with specific guidelines related to use of different software metrics
including their collection, analysis and reporting. Tailoring guidelines for the software metrication
process are also provided to help to meet specific project requirements.
This Handbook provides recommendations, methods and procedures that can be used for the
selection and application of appropriate metrics, but it does not include new requirements with
respect to those provided by ECSS-ST-Q-80C Standard.
The scope of this Handbook covers the following topics:
• Specification of the goals and objectives for a metrication programme.
• Identification of criteria for selection of metrics in a specific project / environment (goal driven).
• Planning of metrication in the development life cycle.
• Interface of metrication with engineering processes.
• Data collection aspects (including use of tools).
• Approach to the analysis of the collected data.
• Feedback into the process and product based on the analysis results.
• Continuous improvement of measurement process.
• Use of metrics for process and product improvement.
This Handbook is applicable to all types of software of all major parts of a space system, including the
space segment, the launch service segment and the ground segment software.
References
For each document or Standard listed, a mnemonic (used to refer to that source throughout this
document) is proposed in the left side, and then the complete reference is provided in the right one.
ECSS Standards
EN Reference Reference in text Title
EN 16601-00-01 [ECSS-S-ST-00-01] ECSS-S-ST-00-01C, ECSS - Glossary of terms
EN 16602-80 [ECSS-Q-80] ECSS-Q-ST-80C, Space Product Assurance – Software
Product Assurance
EN 16603-40 [ECSS-E-40] ECSS-E-ST-40C, Space Engineering – Software
ISO/IEC Standards
ISO 9126 ISO/IEC 9126 Software engineering - Product quality, Parts 1 to 4 (complete
series)
ISO 9126-1 ISO/IEC 9126-1:2001 Part 1: Quality model
ISO 9126-2 ISO/IEC TR 9126-2:2003 Part 2: External metrics
ISO 9126-3 ISO/IEC TR 9126-3:2003 Part 3: Internal metrics
ISO 9126-4 ISO/IEC TR 9126-4:2004 Part 4: Quality in use metrics
ISO 12207 ISO/IEC 12207:2008 Information Technology - Software life cycle processes.
ISO 14143 ISO/IEC 14143 Information technology - Software measurement, Parts 1 to 6
(complete series)
ISO 14598 ISO/IEC 14598 Software engineering - Product evaluation, Parts 1 to 6 (complete
series)
ISO 14598-1 ISO/IEC 14598-1:1999 Part 1: General overview
ISO 24765 ISO/IEC 24765, Systems and Software Engineering Vocabulary
ISO 15939 ISO/IEC 15939:2007 Software Engineering - Software Measurement Process
ISO 17799 ISO/IEC 17799:2005 Information technology - Code of practice for information
security management
ISO 25000 ISO/IEC 25000:2005 Ed. 1 Software Engineering - Software product Quality
Requirements and Evaluation (SQuaRE) - Guide to SQuaRE
Other ESA documents
SPEC ESTEC Contract No. 12650/97/NL/NB(SC) - SPEC – Software Product Evaluation
and Certification (complete series)
SPEC-I ESTEC Contract No. 20421/06/NL/PA - SPEC Method Improvement
SPEC-I-QM SPEC/QM SPEC Method improvement - Quality Model – Issue 1.F
SPEC-I-TN1 SPEC-I/TN1 SPEC Analysis results - Issue 1.F
SPEC-I -TN2 SPEC-I/TN2 Concept for Space Software Product Evaluation for Conformity
WP4 - part 2- Issue 3.B
SPEC-I –TN3 SPEC-I/TN3 Concept for Space Software Product Evaluation for Conformity
WP4 - part 3- Issue 3.2
SPEC-TN3 SPEC/TN3 Space Domain Specific Software Product Quality Models,
Requirements and Related Evaluation Methods - Issue 3.4
SPEC-TN4.1 SPEC/TN4 part 1 Overview of existing software certification schemes - Issue 2
SPEC-TN4.2 SPEC/TN4 part 2 Concept for Space Software Product Evaluation and
Certification - Issue 2
SPEC-TN4.3 SPEC/TN4 part 3 Concept for Space Software Product Evaluation and
Certification - Issue 2
Reports and articles
CHALMERS Chalmers University of Technology Presentation - Department of Computer
Engineering - Dependability and Security Modelling and Metrics
EADS-ST Astrium-ST- Astrium ST internal documents
FENTON “Software Metrics - A Rigorous & Practical Approach” (second edition)
Norman E. Fenton, Shari Lawrence Pfleeger
NASA-1740 NSS 1740.13 “NASA Software Safety Standard” February 1996
NASA-8719 NASA-STD-8719.13A Software Safety, 15 September 1997
NIAC Common Vulnerability Scoring System - Final Report and Recommendations by
the Council, 12 October 2004
NIST-1 NIST and Federal Computer Security Program Managers Forum IT
Security Metrics Workshop - A Practical Approach to Measuring Information
Security: Measuring Security at the System Level, 21 May 2002
NIST-2 NIST Special Publication 800-55
Security Metrics Guide for Information Technology Systems, July 2003
NIST-3 NIST Special Publication 800-35
Guide to Information Technology Security Services, October 2003
SANS-1 SANS Institute - A Guide to Security Metrics - SANS Security Essentials GSEC
Practical Assignment, Version 1.2, 11 July 2001
SANS-2 SANS Institute - Systems Maintenance Programs - The Forgotten Foundation
and Support of the CIA Triad, GSEC v1.3, 10 January 2002
SEC-FIN VTT Technical Research Centre of Finland Publications 544
Process Approach to Information Security Metrics in Finnish Industry and State
Institutions, 2004
SUN-JAVA SUN Java Coding Conventions
Websites
ISECOM The Institute for Security and Open Methodologies (security)
http://www.isecom.org/securitymetrics.shtml
EDUCAUSE http://www.educause.edu
SEC-METRICS Community website
http://www.securitymetrics.org
SW-METRICS International Software Metrics Organization
http://www.swmetrics.org
SEC-DOCS Security white papers and documents
http://www.securitydocs.com
IEEE http://www.ieee.com
IEEE-CS http://www.computer.org
NIST NIST Computer Security Resource Centre
http://csrc.nist.gov/ispab/
COSMICON The Common Software Measurement International Consortium
http://www.cosmicon.com/
UKSMA The UK Software Metrics Association
http://www.uksma.co.uk/?action=0&what=90
ARMY-METRICS Army Software Metrics Office
http;//www.armysoftwaremetrics.org
PSSM Practical Software & Systems Measurement
http://www.psmsc.com
Terms, definitions and abbreviated terms
3.1 Terms from other documents
For the purpose of this document, the terms and definitions from ECSS-S-ST-00-01 and ECSS-Q-ST-80
apply.
3.2 Definitions in other clauses of the present HB
Subclause A.2 of Annex A includes the definitions for all characteristics/sub-characteristics contained
in the quality model used in this Handbook
3.3 Terms specific to the present document
3.3.1 base measure
measure defined in terms of an attribute and the method for quantifying it.
[ISO 24765]
3.3.2 measure (noun)
variable to which a value is assigned as the result of measurement.
[ISO 24765]
3.3.3 measure (verb)
Make a measurement.
[ISO 24765]
3.3.4 measurement
act or process of assigning a number or category to an entity to describe an attribute of that entity.
NOTE "Category" is used to denote qualitative measures of attributes. For
example, some important attributes of software products, e.g. the
language of a source program (such as ADA, C, COBOL) are
qualitative.
[ISO 24765]
3.3.5 metric
a quantitative measure of the degree to which a system, component, or process possesses a given
attribute.
[ISO 24765]
3.3.6 quality model
defined set of characteristics, and of relationships between them, which provides a framework for
specifying quality requirements and evaluating quality
[ISO 24765]
3.4 Abbreviated terms
For the purpose of this document, the abbreviated terms from [ECSS-S-ST-00-01] and the following
apply:
Abbreviation Meaning
CBO coupling between objects
CM configuration management
DDR detailed design review
DIT depth of inheritance tree
IEC International Electrotechnical Commission
ISO International Organization for Standardization
HOOD Hierarchical Object Oriented Design
LOC lines of code
LCOM lack of cohesion method
MIPS millions of instructions per second
MMI man-machine interface
NASA National Aeronautics and Space Administration
NOC number of children
PPF parametric polymorphic factor
RB requirements baseline
RFD request for deviation
RFW request for waiver
SW PA software product assurance
SPR software problem report
SRR system requirements review
UML Unified Modelling Language
V&V verification and validation
VG cyclomatic complexity (McCabe)
Overview of the Handbook
4.1 Introduction
This subclause contains an introduction of the content of this Handbook, the intended audience and
how to use it.
It introduces the rationale of the need of a metrication programme for space software projects based
on [ECSS-E-40] and [ECSS-Q-80] requirements (especially this one).
The organization of this Handbook is reflected in detail in Figure 4-1. This Handbook is organized in
seven main parts:
• Clause 1. Scope
• Clause 2: Normative references
• Clause 3: Terms, definitions and abbreviated terms.
• Clause 4: Overview of the Handbook
• Clause 5: A reference software quality model
• Clause 6: Measurement process
• Annex A: Definition of the quality model
The annex is provided for information only.
Clause 3
Clause 2
Clause 1
Terms, definitions
Normative references
Scope
and abbreviated
terms
Clause 5
Clause 4 Clause 6
A reference
Measurement
Overview of the
software quality
process
Handbook
model
Annex A
Definition of the
quality model
Figure 4-1:Organization of this document
4.2 Relation to other ECSS Standards
4.2.1 General
This subclause discusses how this Handbook interfaces with other ECSS series, namely the ECSS-Q
series of standards (product assurance), ECSS-E series of standards (engineering) and the ECSS-M
series of standards (management).
4.2.2 Software engineering
The interface of this Handbook to the ECSS-E branch is via [ECSS-E-40]; and in turn, the interface of
[ECSS-E-40] to this Handbook is via the [ECSS-Q-80].
[ECSS-E-40] covers all aspects of space software engineering from requirements definition to
retirement. It defines the scope of the space software engineering processes, including details of the
verification and validation processes, and their interfaces with management and product assurance,
which are addressed in the management (-M) and product assurance (-Q) branches of the ECSS
system.
[ECSS-E-40] also defines the content of the document requirements definitions (DRDs) referenced
inside the Standard.
[ECSS-E-40] is intended to help the customers to formulate their requirements and suppliers to
prepare their responses and to implement the work. In this Standard requirements are defined in
terms of what is to be accomplished, rather than in terms of how to organize and perform the
necessary work. This allows the tailoring process to match the requirements to a particular profile and
circumstances of a project. The goal of the tailoring is to select, modify or add adequately
requirements in order to identify the quality ratio adequate to the actual project peculiarities.
There are no specific requirements in [ECSS-E-40] related to software measurement aspects, except
those related to technical budgets management (e.g. 5.3.8 and 5.8.3.12), and a few requirements related
to test coverage (several clauses of subclause 5.8.3.5). In addition, there are some requirements related
to traceability (especially in subclause 5.8.3), which are connected to some extent to the metrication
aspects of the software life cycle.
In addition, clause 5.4.2.1 of ECCS-E-40 demands that quality requirements are established and
documented, as part of the technical specification. This is also reinforced in clauses 7.1 and 7.2 of
[ECSS-Q-80].
All other aspects of software quality modelling and metrication are specified in the [ECSS-Q-80].
4.2.3 Software product assurance
This subclause contains an analysis of [ECSS-Q-80] requirements related to the definition and
implementation of a metrication programme for space software projects.
[ECSS-Q-80] Standard presents software product assurance requirements to be met in a particular
space project in order to provide confidence to the customer and to the suppliers that developed or
reused software satisfies the requirements throughout the system lifetime. In particular, [ECSS-Q-80]
presents requirements to ensure the software is developed to perform as expected and safely in the
operational environment meeting the quality objectives agreed for the project.
[ECSS-Q-80] Standard is tailored in such a way that software product assurance requirements meet a
particular space project and fulfil the specific objectives. In particular, clause 7 of [ECSS-Q-80] explains
that software quality requirements (including safety and dependability) derives from requirements
defined at system level.
Requirement 5.2.7.1 of [ECSS-Q-80] requires that quality models are used to specify software quality
requirements (in quantitative terms as required in 7.1.2) and that they shall be derived from
requirements at system level (as in § 7.1.1). Requirements 5.2.7.2, 6.2.5.3, 6.2.5.4 and 7.1.5 list quality
characteristics, and process and product metrics that shall be part of the quality model.
Clause 7.1.4 requires that a metrication programme is defined and implemented in order to fulfil those
quality requirements. Metrics are selected, collected and analysed against target values and reported
corrective actions are taken when required (in [ECSS-Q-80] requirements 6.2.5.2, 7.1.4, 6.2.5.5 and
7.1.6).
All aspects related to process metrics are included in clause 6.2.5.
Clause 6.3.4 (coding) is special since it contains some requirements related to the coding standards
and its relationship with the product quality requirements (not all related to process requirements).
4.2.4 Project management
The ECSS-M branch defines the requirements to be applied to the management of space projects.
[ECSS-E-40] and [ECSS-Q-80] describe how the ECSS-M series of standards apply to the management
of software projects. In addition, requirements that cannot be found in the M-branch because they are
specific to software product assurance are defined in [ECSS-Q-80].
A reference software quality model
5.1 Introduction
The current situation in space system and software development is characterized by the increasing
complexity of the end product and growing demands on the cost and time effectiveness as well as
overall quality of the development process. In order to facilitate a continuous monitoring of the
progress of the project it is essential to collect quantifiable evidence both for the product being
developed and the process being performed.
[ECSS-Q-80] (ECSS Standard for software product assurance in space projects), in the same line as
other industry standards, includes some requirements for the collection and analysis of a number of
measures with this objective.
The main objective of the metrication framework as defined in this Handbook is to provide support in
the selection of project specific quality model and metrics in order to facilitate a continuous feedback
as a basis for the critical decisions to be taken throughout the project. In particular, metrication should
enable the following goals:
• Communicate the project status throughout the project organization under the responsibility of
quality assurance.
• Identify early and correct problems.
• Provide a basis for key tradeoffs (from both technical and management points of view) while
taking into account the project constraints such as cost, schedule, quality, and functionality.
• Track specific project objectives.
• Defend and justify project decisions throughout the development life-cycle.
• Provide the supplier and the customer with an indication of the quality status of the software
product
Metrication should be seen as a supporting activity to achieve these goals that complement the other
quality assurance activities. As a stand-alone technique it does not provide suitable feedback, but it
does it in the larger context of quality assurance.
Although the main purpose of the metrication as described in this document is the support of projects
throughout the development processes, metrication is also part of process improvement activities. In
addition to the activities of defining metric programs, capturing and evaluation of metric data which
provide a basis for project monitoring, a systematic approach to metrication-based process
improvement requires the selection of a common subset of metrics and their normalization at
department or company level. The goal is the definition of a comparable set of data covering any type
of software development projects. While this topic is addressed briefly in clause 6, full guidance for
organizational metrication process improvements exceeds the scope of this Handbook.
In the following sections, several aspects of the metrication programme as well as the introduction of
the reference quality model for the space software are presented.
As the presented quality model is to be used as a reference, and is required to be tailored for each
project, several metric tailoring tables are included in this Clause, with the aim of facilitating selection
of metric sets from two different perspectives: relevant software quality requirements and criticality
category (applicability and target values).
5.2 Reference software quality model
As in the ISO standards, the first step in the evaluation of software is to select relevant quality
characteristics, using a quality model that breaks software quality objectives down into different
characteristics. Quality models for software evaluation often represent the totality of software quality
attributes classified in a hierarchical tree structure of characteristics and sub-characteristics. A variety
of quality models is available in the literature (see for example [ISO 9126], [SPEC], and [ISO 25000]).
The quality model defined in this Handbook is derived from these references but focuses on a subset
of the proposed models that cover the requirements in [ECSS-Q-80] and for which a quantitative
evaluation is possible.
This quality model takes into account both process and product related characteristics relevant for
space projects. Main characteristics taken into account are presented in below table 5-1.
The quality model proposed below is to be used as reference. It is expected to be tailored according to
projects needs. It is recommended to select a subset of the metrics proposed below that support SW
product quality characteristics representative for the project and beneficial for customer and supplier
for analysis, design coding, testing, operation and maintenance.
For each of the characteristics defined, one or more quantifiable sub-characteristics are identified that
together provide the quality characteristics that are considered for space projects. The selection of
these characteristics is driven by the objective to provide a streamlined and cost-effective reference
metrication approach for space software development.
The criteria used for the selection of characteristics/sub-characteristics of this quality model are the
following:
• All characteristics / sub-characteristics that occur either in the [ECSS-E-40] or in the [ECSS-Q-80]
are included.
• Many other characteristics / sub-characteristics proposed in [ISO 9126] are also considered, as
authoritative references in the field of quality modelling.
• The quality model is then completed with a few additional characteristics / sub-characteristics
proposed in other sources ([NASA-1740], [PSS-01-212], [EADS-ST], literature on metrics), but
not in the main input references.
In order to reduce the size of the resulting quality model when following above approach, and to
augment its usability, few reduction rules were applied:
• When a given sub-characteristic was found not measurable (i.e. no chance to find suitable
metrics), then it was discarded.
• When a given sub-characteristic, even if measurable, was considered as providing very little or
no added value to the project, then it was also discarded.
• In case that all sub-characteristics associated to a given characteristic were discarded, then the
characteristic itself was also removed from the quality model.
In summary, the selection of the characteristics / sub-characteristics of this reference quality model is
driven by pragmatic criteria (i.e. only a few characteristics, measurable, providing a significant added
value, and specific to the objectives of a metrication program).
In Figure 5-1 the elements composing the software quality model are shown. This quality model takes
into account both process and product related characteristics, therefore, in turn, it includes two
categories of metrics: product metrics, related to measuring the products of the development, and
process metrics, referring to measuring the processes themselves.
NOTE The ISO standards use detailed terminology for different types of
measures: base measure or direct measure (measure functionally
independent of other measures), derived measure or indirect measure
(a function of two or more values of base measures), that are
simplified within this Handbook corresponding to the so-called
(product or process) metrics.
The ISO standards also identify the terms external measures
(product properties during execution) and internal measures
(internal attributes that capture static properties of the product)
also specified in the ISO Standards are simplified within this
Handbook corresponding to the so-called product metrics.
Each process or product metric is calculated by the use of a formula or analysis (measurement)
deduced from combining different base measures from attributes of expected outputs or activities
produced or performed at different stages in the project.
Software quality
requirements
composed of
Process quality
Process quality Product quality
Product quality
characteristic
characteristic characteristic
characteristic
composed of
Product quality sub- Product quality sub-
Product quality sub- Product quality sub-
characteristic
characteristic characteristic
characteristic
composed of
E.g. Requirement
implementation Product metric Product metric
Product metric Product metric
coverage
calculated by
Measurement
Measurement
E.g. X= A/B
formula
formula
composed of
E.g. A = number of
E.g. B = number of
correctly
Basic data Basic data
Basic data Basic data
requirements
implemented
requirements
calculated by
simple simple
simple
simple
E.g. count
E.g. count
measurement measurement
measurement measurement
used by
Life cycle Life cycle
Life cycle Life cycle
E.g. Traceability E.g. Requirement
expected expected
expected expected
matrices Specification document
output
output
output output
Figure 5-1: Elements of the software quality model
Table 5-1 presents the proposed reference quality model.
It is important to perform the software measurements in an efficient way and that the resulting
measures are practical. Many software measurements can conveniently be made with the help of
automatic tools and can be packaged as an evaluation module (see [ISO 14598-6]).
To achieve a continuous monitoring throughout the development life-cycle, it is necessary to identify
the relevant phases for data capture and metric evaluation for the specific metrics. While some metrics
are independent of the development progress, other metrics are only available after certain milestones
have been reached.
In order to provide guidelines for this allocation of measurement activities to the project life cycle, this
Handbook refers to the software processes as defined in [ECSS-E-40]. The following table presenting
the proposed reference quality model indicates when each applicable metric should be collected and
provided.
Annex A provides the full definition of the Quality Model and further guidance with regard to the
allocation of individual quality characteristics and the related measurements to the individual
processes.
Table 5-1: Proposed reference quality model
(Main) Sub Metrics First Frequency
characteristic characteristic provided at
PRODUCT RELATED CHARACTERISTICS
Functionality Completeness Requirement allocation SRR Every Review
Requirement implementation
PDR Every Review
coverage
Requirements completeness
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...