SIST ETS 300 406 E1:2003
(Main)Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology
Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology
Definition of the basic framework for a common ETSI methodology encompassing specification related aspects of the Conformance Assessment process
Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje skladnosti protokola in profila – Metodologija standardiziranja
General Information
Standards Content (Sample)
SLOVENSKI STANDARD
01-december-2003
Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje
skladnosti protokola in profila – Metodologija standardiziranja
Methods for Testing and Specification (MTS); Protocol and profile conformance testing
specifications; Standardization methodology
Ta slovenski standard je istoveten z: ETS 300 406 Edition 1
ICS:
33.020 Telekomunikacije na splošno Telecommunications in
general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
EUROPEAN ETS 300 406
TELECOMMUNICATION April 1995
STANDARD
Source: ETSI TC-MTS Reference: DE/MTS-00012
ICS: 33.020, 33.040.40
Key words: Testing, methodology
Methods for Testing and Specification (MTS);
Protocol and profile conformance testing specifications;
Standardization methodology
ETSI
European Telecommunications Standards Institute
ETSI Secretariat
Postal address: F-06921 Sophia Antipolis CEDEX - FRANCE
Office address: 650 Route des Lucioles - Sophia Antipolis - Valbonne - FRANCE
X.400: c=fr, a=atlas, p=etsi, s=secretariat - Internet: secretariat@etsi.fr
Tel.: +33 92 94 42 00 - Fax: +33 93 65 47 16
Copyright Notification: No part may be reproduced except as authorized by written permission. The copyright and the
foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 1995. All rights reserved.
Page 2
ETS 300 406: April 1995
Whilst every care has been taken in the preparation and publication of this document, errors in content,
typographical or otherwise, may occur. If you have comments concerning its accuracy, please write to
"ETSI Editing and Committee Support Dept." at the address shown on the title page.
Page 3
ETS 300 406: April 1995
Contents
Foreword.7
Introduction .7
1 Scope .9
2 Normative references . 10
3 Definitions and abbreviations. 11
3.1 Definitions. 11
3.2 Terms which can cause confusion. 13
3.3 Abbreviations. 13
4 Introduction to conformance testing . 15
4.1 What is conformance testing? . 15
4.1.1 Context and history of conformance testing . 15
4.1.2 Scope of conformance testing . 15
4.2 Conforming implementation . 16
4.2.1 Conformance requirement. 16
4.2.2 Conformance statement. 16
4.3 Technical objectives of conformance testing. 17
4.4 The need for standardized test specifications. 17
5 Conformance testing applied to base specification or to profile . 18
5.1 What is a base specification?. 18
5.2 What is a profile?. 19
5.3 Examples of profiles. 20
5.3.1 In EWOS, CEN/CENELEC. 20
5.3.2 In ETSI. 21
5.4 Structure of a profile specification. 21
5.5 Conformance to a profile . 22
5.6 Conformance testing applied to base specification. 22
5.7 Conformance testing applied to profile. 23
5.8 Decision criteria: base specification of profile testing?. 23
6 The PICS and other base specification ICS; a companion document for base specifications. 26
6.1 Objective of the PICS (base specification ICS). 26
6.2 What is a PICS proforma (base specification ICS proforma) . 27
6.3 PICS (ICS) proforma and TS . 27
6.4 Semantics of PICS (ICS) proforma status notations. 28
6.5 PICS proforma (base specification ICS proforma) criteria . 30
6.5.1 ISO/IEC 9646 criteria. 30
6.5.2 Additional criteria. 30
6.6 Standardization of a PICS (ICS) proforma. 30
7 TS for a base specification. 31
7.1 The production process . 32
7.2 Synopsis of components and their dependencies. 35
7.3 Checklist of components. 38
7.4 Definitions of the components and criteria . 39
7.4.1 TSS&TP. 39
7.4.1.1 TSS. 39
7.4.1.1.1 General rules and recommendations . 39
Page 4
ETS 300 406: April 1995
7.4.1.1.2 Recommendation concerning the testing of "state
transitions" .40
7.4.1.1.3 Recommendation concerning the "inopportune" groups .41
7.4.1.1.4 Recommendation and rule concerning the "variation"
groups .41
7.4.1.1.5 Rules and recommendations concerning the naming
conventions .42
7.4.1.2 TPs .44
7.4.1.3 CTPs .46
7.4.1.3.1 Why combine TPs?.46
7.4.1.3.2 How to combine TPs.47
7.4.1.3.3 TSS for CTPs. 50
7.4.2 ATM. 51
7.4.3 ATS conventions.51
7.4.4 ATS.52
7.4.5 TCP .52
7.4.6 Partial PIXIT proforma (partial IXIT proforma) .53
7.4.7 PCTR proforma.53
7.4.8 Untestable TPs.53
7.4.9 Abstract selection rules.54
7.4.10 ATS to TP map . 54
7.5 Amendments. 54
7.5.1 TSS & TP amendment .54
7.5.2 ATS amendment.55
8 Profile ICS proforma.55
8.1 Objective of the profile ICS .55
8.2 What is a profile ICS proforma? .56
8.2.1 Profile requirement list profile RL .57
8.2.2 Profile-specific ICS proforma.58
8.3 Profile ICS proforma and TS.58
8.4 Profile ICS and conformance assessment process.58
8.4.1 Profile ICS and static conformance review.58
8.4.2 Profile ICS and test case parameterization, de-selection .58
8.4.3 Profile ICS components attached to test reports.59
9 TS for a profile.59
9.1 Principles of profile testing .59
9.2 A word on the production process .61
9.3 Synopsis of components and their dependencies .62
9.4 Checklist of components .63
9.5 Definitions of the components and criteria .64
9.5.1 Profile-specific test specification.64
9.5.2 PTS-summary .66
9.5.3 SCTR proforma tailored to the profile.66
9.5.4 Profile IXIT XRL. 66
9.5.5 Profile IXIT proforma .66
10 Standardization of test specifications .67
10.1 Physical contents of a conformance TS.67
10.1.1 Structure of a conformance TS.67
10.1.1.1 Conformance TS for a base specification. 67
10.1.1.2 Conformance TS for a profile. 70
10.1.2 Need for an overview document.71
10.1.3 Editing rules for TTCN.71
10.1.4 Copyright of proformas .72
10.1.5 Copyright of TTCN.MP. 73
10.1.6 Conformance clause .73
10.1.7 Case of ATSs having parts in common.73
10.2 Minimal quality criteria .74
Page 5
ETS 300 406: April 1995
10.2.1 Reference to the reference specification . 75
10.2.2 Components of a TS. 75
10.2.3 TTCN notation. 75
10.2.4 Validation. 75
10.2.4.1 Validation of TSS&TP . 75
10.2.4.2 Validation of ATS specification. 75
11 Conformance clause. 76
11.1 Conformance with this ETS. 76
11.2 Conformance with ISO/IEC 9646. 76
Annex A (normative): Normative criteria for the standardization of PICS (ICS) proforma
specifications. 78
A.1 Static versus dynamic requirements in an ICS proforma . 78
A.2 Level of detail of an ICS proforma . 79
A.3 Requirements on numeric values. 81
A.4 References to ICS items. 83
A.5 Scope of qualified optionals. 84
Annex B (informative): Components of a TS for a base specification when several ATSs are
produced. 86
B.1 Why a single ATM has been chosen . 86
B.2 Overview .87
B.3 Synopsis of the components and their dependencies. 87
B.4 Relationships between components . 89
B.5 Additional TSS&TP. 89
B.6 Consequences on the pre-existing ATS. 89
Annex C (informative): Guidance on the combination of TPs. 90
C.1 Tests having the same morphology. 90
C.2 Tests having the same spirit. 92
Annex D (informative): Guidance on the definition of a test strategy. 93
D.1 Test strategy: initial steps . 93
D.1.1 Why do the protocols have to be tested? . 93
D.1.2 What will an implementation to test look like?. 93
D.2 Test strategy: choice of test method. 94
D.2.1 One or several test method(s)?. 95
D.2.2 Technical criteria intrinsic to standardized area. 95
D.2.3 Technical criteria related to implementation conditions. 97
D.2.4 Other criteria . 98
D.2.5 Further clarification on test methods characteristics. 98
D.2.6 Clarification concerning the concept of upper tester . 98
D.3 Test strategy: coverage and costs. 99
Page 6
ETS 300 406: April 1995
D.4 Test strategy: technical refinements. 100
Annex E (informative): Bibliography. 101
History . 102
Page 7
ETS 300 406: April 1995
Foreword
This European Telecommunication Standard (ETS) has been produced by the Methods for Testing and
Specification (MTS) Technical Committee of the European Telecommunications Standards Institute (ETSI).
This ETS provides rules and guidance on how to develop telecommunications conformance testing
specifications with the final objective to standardize them.
ETSI has explored innovative solutions for the long term, and at present, a "proposer" in harmonized
approaches to the pressing requirement of developing testing standards in various areas of
telecommunications. It is also essential that ETSI remains aware of what is going on in the information
technology area outside ETSI, in close relations with its own technical activities.
The production of conformance testing standards in ETSI has already started in various technical areas,
and needs to be continued with the best harmonization of efforts.
ETSI TC-MTS recommends that the TC and STC chairmen in their role of co-ordinators of standard
development ensure that the groups of experts working in conformance testing in their bodies are fully
aware of the status of affairs in this area so that initiatives are co-ordinated properly. Special care should
be devoted to the consistency of the methodology used (test specifications' structure, test notations,
criteria for acceptance, etc.), to the need of not duplicating efforts and to the strategy of improving
convergence for testing at the European and international levels.
With the co-operation of all ETSI members, an initial orientation towards a harmonized approach to
conformance testing standards and practice is desirable and possible. This is why the TC MTS work
program includes the production of a comprehensive methodology guide that will represent a further step
forward in harmonization and consistency.
Proposed transposition dates
Date of latest announcement of this ETS (doa): 31 July 1995
Date of latest publication of new National Standard
or endorsement of this ETS (dop/e): 31 January 1996
Date of withdrawal of any conflicting National Standard (dow): 31 January 1996
Introduction
This ETS is one element of a series of documents produced during 1992 and 1993 by ETSI. These ETSs
will provide a reference library of rules and guidance on how to standardize telecommunications
conformance testing specifications in Europe. These rules and guidance take into account the state of the
art, the international and European standards and reports already published on that matter, the specifics of
test suite editing, and the practices of the ETSI and CEN secretariats for the production of European
standards.
Although this ETS has been produced in order to become a reference document for ETSI, it is designed to
be used in a wider European context, for example, as a technical basis for accreditation and specification
purposes, as well as for European Commission initiated conformance testing programmes, e.g. the
Conformance Testing Services (CTS) programme.
Page 8
ETS 300 406: April 1995
Blank page
Page 9
ETS 300 406: April 1995
1 Scope
This European Telecommunication Standard (ETS) specifies a methodology for developing
telecommunications conformance testing specifications with the final objective to standardize them.
This ETS defines the components to be produced when specifying a European conformance Testing
Standard (TS) in the domain of telecommunications.
The methodology specified in this ETS is, in principle, applicable to domains other than
telecommunications, e.g. Open Systems Interconnection (OSI). However, the limits of such an application
of the methodology outside telecommunications testing are not defined, and each group defining test
specifications are responsible for deciding if any part of this methodology is applicable.
This methodology applies to conformance TSs for protocols, for profiles, for information objects, for
interfaces, and for services.
NOTE 1: The methodology of this ETS is generic, in the sense that it applies to protocols, to
profiles, to information objects, etc. Following the example of ISO/IEC 9646
(references [5] to [12]), the terminology employed is also generic, e.g. Implementation
Conformance Statement (ICS), Implementation eXtra Information for Testing (IXIT).
However, in order to avoid confusing the readers who are familiar with the protocol-
related terminology, the non-generic terms (Protocol Implementation Conformance
Statement (PICS), Protocol Implementation eXtra Information for Testing (PIXIT), etc.)
have been kept in the titles of clauses and subclauses.
The test of physical aspects, such as "layer 1", analogue measurements, Electro-Magnetic Compatibility
(EMC), is not covered by this ETS. However, part of the methodology may be still be applied, e.g. the
concepts of ICS, IXIT, Test Purposes (TPs), and Test Suite Structure (TSS).
This ETS provides the rules for standardization bodies developing telecommunications conformance TSs.
NOTE 2: One objective of this ETS is the use, when relevant, within standardization bodies. For
this purpose, within ETSI, it has the same value as an internal Technical Committee
Reference Technical Report (TCR-TR).
This ETS is applicable to organizations such as Conformance Testing Services (CTS) programmes, pre-
standardization institutions, private institutions, which develop conformance testing specifications with the
objective of contributing them for the purposes of European telecommunications standardization.
This ETS contains two types of information, following a twofold objective:
- it clarifies the principles of the ISO/IEC 9646 (references [5] to [12]) testing framework and
methodology, as well as the concept of profile;
- it defines additional criteria for European telecommunications TSs, whereby it constitutes a
"European telecommunications profile" of the ISO/IEC 9646 (references [5] to [12]) methodology,
containing its own recommendations.
This ETS is consistent with the normative references listed in clause 2. This ETS extends the current
principles contained in ISO/IEC 9646 (references [5] to [12]), in particular in the field of the combination of
TPs, and concerning the choice of a single Abstract Test Suite (ATS).
For conformance testing in the "voluntary domain" (i.e. outside the regulatory domain), for instance by the
suppliers to support their commercial claims for conformance, TSs are normally published as I-ETS, or
ETS, depending on their role and their maturity.
This ETS addresses telecommunications TS methodology for the voluntary domain.
Page 10
ETS 300 406: April 1995
2 Normative references
This ETS incorporates by dated and undated reference, provisions from other publications. These
normative references are cited at the appropriate places in the text and the publications are listed
hereafter. For dated references, subsequent amendments to or revisions of any of these publications apply
to this ETS only when incorporated in it by amendment or revision. For undated references the latest
edition of the publication referred to applies.
[1] ETR 153: "Methods for Testing and Specification (MTS); Guidance on the
production and completion of System Conformance Test Report (SCTR) and
Protocol Conformance Test Report (PCTR) proformas".
[2] ETR 141: "Methods for Testing and Specification (MTS); Protocol and profile
conformance testing specifications; The Tree and Tabular Combined Notation
(TTCN) style guide".
[3] PNE-Rules (1991): "CEN/CENELEC Internal Regulations - IR; Part 3: Rules for
the drafting and presentation of European Standards (PNE-Rules)".
[4] EN 45001 (1989): "General criteria for the operation of testing laboratories".
[5] ISO/IEC 9646-1 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
1: General concepts" (including DAM1 and DAM2, to be published).
[6] ISO/IEC 9646-2 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 2: Abstract Test Suite Specification" (including DAM1 and DAM2, to be
published).
[7] ISO/IEC 9646-3: "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation".
[8] ISO/IEC 9646-3 AM 1: "Information technology - Open Systems Interconnection
- Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation. Amendment 1: TTCN Extensions (concurrent
TTCN)".
NOTE 1: ISO/IEC 9646-3 AM 1 will be included in the next edition of ISO/IEC 9646-3, which will
contain TTCN and concurrent TTCN in the same standard.
[9] ISO/IEC 9646-4 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
4: Test Realisation" (including DAM1 and DAM2, to be published).
[10] ISO/IEC 9646-5 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
5: Requirements on test laboratories and clients for the Conformance
Assessment Process" (including DAM1 and DAM2, to be published).
[11] ISO/IEC 9646-6 IS (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 6: Protocol Profile Test Specification".
[12] ISO/IEC 9646-7 DIS (1993): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
7: Implementation Conformance Statements".
Page 11
ETS 300 406: April 1995
NOTE 2: The clauses of ISO/IEC 9646 (references [5] to [12]) referenced in this ETS are from
edition 2: 1994 (unpublished). Although the contents of ISO/IEC 9646 have changed, it
is useful to know the references to edition 1: 1991. The correspondence is as follows:
Edition 1: 1991 Edition 2: 1994
Part 2, clause 12 Part 2, clause 11
Part 1, annex B Part 2, annex A
3 Definitions and abbreviations
3.1 Definitions
For the purposes of this ETS, all the definitions in ISO/IEC 9646 (references [5] to [12]) apply.
In addition, the following definitions apply, or have been re-formulated for clarification:
Abstract Test Method (ATM) (Abstract Testing Method): Definition of ISO/IEC 9646-1 [5], subclause
3.3.5, applies.
NOTE 1: The ISO definition, complemented with the explanation of ISO/IEC 9646-2 [6],
subclause 11.1.7, properly specifies that the ATM contains "enough detail to enable
Abstract Test Cases (ATCs) to be specified for this test method". This means that the
ATM is a detailed description of the testing architecture. Conversely, the four methods
for Single Party Testing (SPyT) context, Remote, Local, Distributed, Coordinated, are
called ATM categories.
Abstract Test Suite (ATS): Definition of ISO/IEC 9646-1 [5], subclause 3.3.6, applies.
NOTE 2: Confusion should be avoided between the ATS (collection of test cases, written in Tree
and Tabular Combined Notation (TTCN)), and the ATS specification. The ATS is just
one component of the ATS specification.
ATS specification: Definition of ISO/IEC 9646-1 [5], subclause 3.3.7, applies.
NOTE 3: The ATS is just one component of the ATS specification. The list of components of an
ATS specification is specified in ISO/IEC 9646-2 [6], clause 14, and in this ETS clause
7.
ATM category: One of the four families of ATM defined in ISO/IEC 9646-2 [6], for SPyT: Remote, Local,
Distributed, Coordinated.
NOTE 4: The term ATM category is not an ISO/IEC 9646 term. However, it is in line with
ISO/IEC 9646, see ISO/IEC 9646-2 [6] subclause 11.3.1.
Page 12
ETS 300 406: April 1995
base specification: A specification of a protocol, telecommunication service, interface, abstract syntax,
encoding rules, or information object.
NOTE 5: This definition extends the definition in ISO/IEC 9646-1 [5], subclause 3.3.10, to
telecommunication services and interfaces.
NOTE 6: A base specification is anything that can be profiled. A base specification is defined by
opposition to a Profile, which constrains optionalities in one or several base
specifications.
NOTE 7: Confusion should be avoided between base specification (on which a profile can be
based), and reference specification, which is the reference to a testing specification.
base specification ICS: An ICS related to a base specification. A PICS is a particular case of base
specification ICS, applied to base specifications of protocols.
base specification TS: A base specification TS is a conformance TS specified for a base specification.
base standard: A standardized base specification, e.g. as an ITU-T Recommendation, an ISO standard
or an ETS.
component (of a profile): A component of a profile is a single protocol, telecommunication service, or
interface, or a combination of one or more protocols, telecommunication services, or interfaces, with zero
or more information objects upon which a profile is based and which are to be tested in combination.
NOTE 8: Definition of ISO/IEC 9646-6 [11], clause 3, is extended to telecommunication services
and interfaces.
functional subset (of a protocol): A grouping of protocol features (Protocol Data Units (PDUs),
mechanisms), that is specified in a protocol specification, and that realizes a protocol function. The
functional subsets of a protocol do not overlap and their sum covers the total function of the protocol. An
example of a functional subset is a role, like responder and initiator. As far as testing is concerned, a
functional subset can be tested with a single test configuration.
reference specification: It is a standard which specifies a base specification, or a set of base
specifications, or a profile, or a set of profiles, and for conformance against which test specifications are
written.
reference standard: Synonymous to reference specification.
Testing Standard (TS) or conformance TS: A document, or a set of documents, containing a
conformance testing specification.
Test Management Protocol (TMP) specification: A document containing the TMP, and a TMP
implementation statement proforma.
untestable TP: A TP that cannot generate an ATC for a given ATM.
Page 13
ETS 300 406: April 1995
3.2 Terms which can cause confusion
Derivation: The word derivation (and its derivatives) should be used carefully, because it
may have several meanings:
a) a TP is derived from a conformance requirement;
b) a Test Suite Structure & Test Purposes (TSS&TP) is derived from a base specification or from a
profile specification;
c) an ATC is derived from a TP;
d) an ATS is derived from a TSS&TP;
e) an executable test case is derived from an ATC;
f) an Executable Test Suite (ExTS) is derived from an ATS.
Therefore, the meaning should be determined in accordance with each context.
NOTE: The meanings given in e) and f) are not used in this ETS.
Entry: The term Entry (e.g. in a PICS proforma) shall not be used. It is ambiguous and
was dropped from ISO/IEC 9646-7 [12]. Instead, the terms Item, Question,
Status (value), (support) Answer should be used. Their definition is in
ISO/IEC 9646-7 [12], clause 3.
3.3 Abbreviations
For the purposes of this ETS, the following abbreviations apply:
ACSE Association Control Service Element
AOW Asian Oceanian Workshop
ASP Abstract Service Primitive
ATC Abstract Test Case
ATM Abstract Test Method
ATS Abstract Test Suite
CMIP Common Management Information Protocol
CTP Combined Test Purpose
CTR Common Technical Regulation
CTS Conformance Testing Services
DECT Digital European Cordless Telephone
DIS Draft International Standard
EG Expert Group (EWOS term)
EMC Electro-Magnetic Compatibility
ETS European Telecommunication Standard
NOTE 1: The acronym for an Executable Test Suite should be avoided, because it introduces a
confusion with an ETS in ETSI. However, in this ETS, an Executable Test Suite is
abbreviated by "ExTS" (see below).
ETG EWOS Technical Guide
ETR ETSI Technical Report
ExTS Executable Test Suite
NOTE 2: Although possible, the use of "ExTS" is discouraged, and the use of the full expansion
"Executable Test Suite" is recommended.
Page 14
ETS 300 406: April 1995
FTAM File Transfer Access and Management
ICS Implementation Conformance Statement
ISDN Integrated Services Digital Network
IT & T Information Technology and Telecommunication
ITSTC Information Technology Steering Committee
IUT Implementation Under Test
IXIT Implementation eXtra Information for Testing
MHS Message Handling Service
MMS Manufacturing Message Services
MOT Means Of Testing
MPyT Multi-Party Testing methodology
OIW Open Systems Environment Implementors' Workshop
OSI Open Systems Interconnection
PAP Public Access Profile (DECT)
PCO Point of Control and Observation
PCTR Protocol Conformance Test Report
PDU Protocol Data Unit
PETS Parameterized Executable Test Suite
PICS Protocol Implementation Conformance Statement
PIXIT Protocol Implementation eXtra Information for Testing
PSDN Public Switched Data Network
PSTS Profile Specific Test Specification
PTS Profile Test Specification
RL Requirement List
SCS System Conformance Statement
SCTR System Conformance Test Report
SOGITS Senior Official Group for Information Technology Standardization
SOGT Senior Official Group for Telecommunication
SPyT Single Party Testing
SUT System Under Test
TBR Technical Basis for Regulation
TCP Test Co-ordination Procedures
TM-PDU Test Management Protocol Data Unit
TMP Test Management Protocol
TP Test Purpose(s)
TS Testing Standard
TSS Test Suite Structure
TSS&TP Test Suite Structure & Test Purposes
TTCN Tree and Tabular Combined Notation
TTCN.GR TTCN Graphic Rendition
TTCN.MP TTCN Machine Processable
VT Virtual Terminal
XRL eXtra Requirement List
Page 15
ETS 300 406: April 1995
4 Introduction to conformance testing
4.1 What is conformance testing?
4.1.1 Context and history of conformance testing
Although everybody is convinced that testing is an essential development step in a modern industrial
environment, the testing activities can easily become a matter of contention between the tester and the
developer. To prevent any dispute, it has become a good practice, in the Information Technology and
Telecommunication (IT & T) area, to have initial agreements on the test of a product, as early as the
product itself is specified. The developers and the testers specify together, and approve, the test criteria
and the test procedures to be applied at each step.
When the product is to include an implementation of a set of international standards specifying
communications protocols issued by ISO, ITU-T, ETSI, CEN, etc. it becomes possible to specify the test
criteria and procedures with a quality comparable to that of the protocol standards themselves.
Since 1983, there is a world-wide consensus on how to achieve this: develop, in the same open and
voluntary environment as the protocols themselves, a general conformance testing methodology, applying
to all protocols, plus a collection of conformance TSs to cover all protocols.
This is being achieved via:
- the standardization of a common testing methodology, which has reached a good level of stability
and a wide applicability (test of protocols, of objects, generalization of the concept of ICSs, multi-
party testing, profile testing);
- the world-wide and European determination to publish standardized conformance testing
specifications, as the technical basis to certification systems.
The standardization of conformance testing covers the methodology and the test specifications
accompanying the reference standards. It does not cover the test tools.
Conformance testing is also used as a basis for terminal equipment approval, within the framework of the
European Directive 91/263/EEC (commonly known as the Terminal Directive). Parts of the test
specifications are then referenced by regulatory documents, called Common Technical Regulations
(CTRs).
4.1.2 Scope of conformance testing
Conformance testing is concerned with the assessment of the extent to which an implementation or system
conforms to a specification.
It should be emphasized that exhaustive testing is impractical on both technical and economical grounds -
and even theoretically impossible - and that, therefore, conformance testing cannot guarantee
conformance to a specification.
Conformance testing is not concerned with the validation of the protocol specifications, nor with the
assessment of performance, reliability, robustness, unless explicitly stated as a conformance requirement
in the reference specification.
Conformance testing shall not add constraints in addition to standards. Conformance testing means testing
against the conformance requirements (see subclause 4.2.1) in the standards. The publication of
conformance TSs shall not be an indirect means to constrain a technology more than the reference
standards themselves do.
Page 16
ETS 300 406: April 1995
4.2 Conforming implementation
According to ISO/IEC 9646-1 [5], a Conforming Implementation "satisfies both static and dynamic
conformance requirements (see subclause 4.2.1 of this ETS), consistent with the capabilities stated in the
ICS (see subclause 4.2.2 of this ETS)".
It is essential to distinguish between a conformance requirement (part of a standard) and a conformance
statement (accompanying an implementation).
4.2.1 Conformance requirement
A conformance requirement is an elementary piece of a specification, stating what an implementation is
required to do or not to do.
A conforming implementation satisfies the conformance requirements contained in the specification.
ISO/IEC 9646-1 [5], subclause 5.2, clarifies the notion of conformance requirement, by distinguishing:
- mandatory, conditional, or optional conformance requirements;
- positively or negatively stated conformance requirements;
- static and dynamic conformance requirements.
ISO/IEC 9646-1 [5], subclauses 5.3 and 5.4, defines dynamic and static conformance requirements:
- "Dynamic conformance requirement: one of the requirements which specifies what observable
behaviour is permitted by the relevant specification(s) in instances of communications";
- "Static conformance requirement: one of the requirements that specifies the limitations on the
combinations of implemented capabilities permitted in a real open system which is claimed to
conform to the relevant specification(s)".
4.2.2 Conformance statement
A conformance statement is associated with an implementation or a system.
A supplier states in a conformance statement which type of conformance is claimed for an implementation
or system.
A conformance statement is the result of the filling of a dedicated proforma, ICS or System Conformance
Statement (SCS) proforma, by the supplier.
Page 17
ETS 300 406: April 1995
4.3 Technical objectives of conformance testing
For communication products, the chief objective of conformance testing is to increase the probability that
different implementations of the same standard will interoperate.
NOTE 1: In the regulatory sector (i.e. type approval of terminal equipment in the framework of
the European Directive 91/263/EEC), conformance testing aims at ensuring that:
- terminal implementations will not harm the public network;
- terminal implementations will interoperate with the public network to the extent
for the purpose of establishing, modifying, charging for, holding and clearing
connection;
- terminal implementations will interoperate via the network, in justified cases.
Conformance testing will provide a certain degree of confidence that a product implementation has the
required capabilities according to the relevant standards. It will not be exhaustive, due to technical and
economic limitations.
NOTE 2: Clearly, the higher degree of confidence a conformance test will offer, the more
expensive it will be (e.g. the more test cases the test suite will contain). The art of the
conformance test specifier resides in finding the good compromise, so that the test
campaigns provide a reasonable level of confidence at a reasonable cost. Optimization
will be sought: best ratio coverage/cost.
4.4 The need for standardized test specifications
This ETS takes into account the following recommendations from ETSI/EWOS/PT005, endorsed by the
European Commission Senior Official Group for Telecommunication (SOGT) and the Senior Official Group
for Information Technology Standardization (SOGITS) in 1991:
a) "European certification requires mutual recognition, which implies harmonized test reports and
standardization of test specifications." (subclause 2.3.1-1 of the PT005 technical report).
b) "Recognition arrangements (in the European certification system) need the standardization of test
specifications for the demonstration of equivalence on which harmonization is based."
"Common test specifications and harmonized test systems which are developed in accordance with
these common test specifications are therefore the basis for harmonized test services."
(subclause 2.3.1-2 of the PT005 technical report).
c) "Standardization of test specifications is important for accreditation bodies in order to properly
assess that test laboratories use the ExTSs in accordance to the test specification requirements."
(subclause 2
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...