Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology

Definition of the basic framework for a common ETSI methodology encompassing specification related aspects of the Conformance Assessment process

Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje skladnosti protokola in profila – Metodologija standardiziranja

General Information

Status
Published
Publication Date
30-Nov-2003
Current Stage
6060 - National Implementation/Publication (Adopted Project)
Start Date
01-Dec-2003
Due Date
01-Dec-2003
Completion Date
01-Dec-2003

Buy Standard

Standard
SIST ETS 300 406 E1:2003
English language
102 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day

Standards Content (sample)

SLOVENSKI STANDARD
SIST ETS 300 406 E1:2003
01-december-2003
Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje
skladnosti protokola in profila – Metodologija standardiziranja

Methods for Testing and Specification (MTS); Protocol and profile conformance testing

specifications; Standardization methodology
Ta slovenski standard je istoveten z: ETS 300 406 Edition 1
ICS:
33.020 Telekomunikacije na splošno Telecommunications in
general
SIST ETS 300 406 E1:2003 en

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST ETS 300 406 E1:2003
---------------------- Page: 2 ----------------------
SIST ETS 300 406 E1:2003
EUROPEAN ETS 300 406
TELECOMMUNICATION April 1995
STANDARD
Source: ETSI TC-MTS Reference: DE/MTS-00012
ICS: 33.020, 33.040.40
Key words: Testing, methodology
Methods for Testing and Specification (MTS);
Protocol and profile conformance testing specifications;
Standardization methodology
ETSI
European Telecommunications Standards Institute
ETSI Secretariat
Postal address: F-06921 Sophia Antipolis CEDEX - FRANCE
Office address: 650 Route des Lucioles - Sophia Antipolis - Valbonne - FRANCE
X.400: c=fr, a=atlas, p=etsi, s=secretariat - Internet: secretariat@etsi.fr
Tel.: +33 92 94 42 00 - Fax: +33 93 65 47 16

Copyright Notification: No part may be reproduced except as authorized by written permission. The copyright and the

foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 1995. All rights reserved.
---------------------- Page: 3 ----------------------
SIST ETS 300 406 E1:2003
Page 2
ETS 300 406: April 1995

Whilst every care has been taken in the preparation and publication of this document, errors in content,

typographical or otherwise, may occur. If you have comments concerning its accuracy, please write to

"ETSI Editing and Committee Support Dept." at the address shown on the title page.

---------------------- Page: 4 ----------------------
SIST ETS 300 406 E1:2003
Page 3
ETS 300 406: April 1995
Contents

Foreword...........................................................................................................................................7

Introduction ........................................................................................................................................7

1 Scope ......................................................................................................................................9

2 Normative references .............................................................................................................. 10

3 Definitions and abbreviations.................................................................................................... 11

3.1 Definitions................................................................................................................. 11

3.2 Terms which can cause confusion............................................................................... 13

3.3 Abbreviations............................................................................................................ 13

4 Introduction to conformance testing .......................................................................................... 15

4.1 What is conformance testing? .................................................................................... 15

4.1.1 Context and history of conformance testing .............................................. 15

4.1.2 Scope of conformance testing ................................................................. 15

4.2 Conforming implementation ........................................................................................ 16

4.2.1 Conformance requirement....................................................................... 16

4.2.2 Conformance statement.......................................................................... 16

4.3 Technical objectives of conformance testing................................................................ 17

4.4 The need for standardized test specifications.............................................................. 17

5 Conformance testing applied to base specification or to profile ................................................... 18

5.1 What is a base specification?..................................................................................... 18

5.2 What is a profile?...................................................................................................... 19

5.3 Examples of profiles.................................................................................................. 20

5.3.1 In EWOS, CEN/CENELEC...................................................................... 20

5.3.2 In ETSI.................................................................................................. 21

5.4 Structure of a profile specification............................................................................... 21

5.5 Conformance to a profile ........................................................................................... 22

5.6 Conformance testing applied to base specification....................................................... 22

5.7 Conformance testing applied to profile........................................................................ 23

5.8 Decision criteria: base specification of profile testing?.................................................. 23

6 The PICS and other base specification ICS; a companion document for base specifications.......... 26

6.1 Objective of the PICS (base specification ICS)............................................................ 26

6.2 What is a PICS proforma (base specification ICS proforma) ........................................ 27

6.3 PICS (ICS) proforma and TS ..................................................................................... 27

6.4 Semantics of PICS (ICS) proforma status notations..................................................... 28

6.5 PICS proforma (base specification ICS proforma) criteria ............................................ 30

6.5.1 ISO/IEC 9646 criteria............................................................................. 30

6.5.2 Additional criteria.................................................................................... 30

6.6 Standardization of a PICS (ICS) proforma................................................................... 30

7 TS for a base specification....................................................................................................... 31

7.1 The production process ............................................................................................. 32

7.2 Synopsis of components and their dependencies......................................................... 35

7.3 Checklist of components............................................................................................ 38

7.4 Definitions of the components and criteria ................................................................... 39

7.4.1 TSS&TP................................................................................................ 39

7.4.1.1 TSS............................................................................... 39

7.4.1.1.1 General rules and recommendations ................................ 39
---------------------- Page: 5 ----------------------
SIST ETS 300 406 E1:2003
Page 4
ETS 300 406: April 1995
7.4.1.1.2 Recommendation concerning the testing of "state

transitions" .....................................................................40

7.4.1.1.3 Recommendation concerning the "inopportune" groups ......41
7.4.1.1.4 Recommendation and rule concerning the "variation"

groups ...........................................................................41

7.4.1.1.5 Rules and recommendations concerning the naming

conventions ....................................................................42

7.4.1.2 TPs ...............................................................................44

7.4.1.3 CTPs .............................................................................46

7.4.1.3.1 Why combine TPs?.........................................................46

7.4.1.3.2 How to combine TPs.......................................................47

7.4.1.3.3 TSS for CTPs................................................................. 50

7.4.2 ATM...................................................................................................... 51

7.4.3 ATS conventions.....................................................................................51

7.4.4 ATS.......................................................................................................52

7.4.5 TCP ......................................................................................................52

7.4.6 Partial PIXIT proforma (partial IXIT proforma) ..........................................53

7.4.7 PCTR proforma......................................................................................53

7.4.8 Untestable TPs.......................................................................................53

7.4.9 Abstract selection rules...........................................................................54

7.4.10 ATS to TP map ...................................................................................... 54

7.5 Amendments............................................................................................................. 54

7.5.1 TSS & TP amendment ............................................................................54

7.5.2 ATS amendment.....................................................................................55

8 Profile ICS proforma................................................................................................................55

8.1 Objective of the profile ICS ........................................................................................55

8.2 What is a profile ICS proforma? .................................................................................56

8.2.1 Profile requirement list profile RL .............................................................57

8.2.2 Profile-specific ICS proforma...................................................................58

8.3 Profile ICS proforma and TS......................................................................................58

8.4 Profile ICS and conformance assessment process.......................................................58

8.4.1 Profile ICS and static conformance review................................................58

8.4.2 Profile ICS and test case parameterization, de-selection ...........................58

8.4.3 Profile ICS components attached to test reports.......................................59

9 TS for a profile........................................................................................................................59

9.1 Principles of profile testing .........................................................................................59

9.2 A word on the production process ..............................................................................61

9.3 Synopsis of components and their dependencies .........................................................62

9.4 Checklist of components ............................................................................................63

9.5 Definitions of the components and criteria ...................................................................64

9.5.1 Profile-specific test specification..............................................................64

9.5.2 PTS-summary ........................................................................................66

9.5.3 SCTR proforma tailored to the profile.......................................................66

9.5.4 Profile IXIT XRL..................................................................................... 66

9.5.5 Profile IXIT proforma ..............................................................................66

10 Standardization of test specifications ........................................................................................67

10.1 Physical contents of a conformance TS.......................................................................67

10.1.1 Structure of a conformance TS................................................................67

10.1.1.1 Conformance TS for a base specification.......................... 67

10.1.1.2 Conformance TS for a profile........................................... 70

10.1.2 Need for an overview document...............................................................71

10.1.3 Editing rules for TTCN.............................................................................71

10.1.4 Copyright of proformas ...........................................................................72

10.1.5 Copyright of TTCN.MP............................................................................ 73

10.1.6 Conformance clause ...............................................................................73

10.1.7 Case of ATSs having parts in common.....................................................73

10.2 Minimal quality criteria ...............................................................................................74

---------------------- Page: 6 ----------------------
SIST ETS 300 406 E1:2003
Page 5
ETS 300 406: April 1995

10.2.1 Reference to the reference specification .................................................. 75

10.2.2 Components of a TS............................................................................... 75

10.2.3 TTCN notation........................................................................................ 75

10.2.4 Validation............................................................................................... 75

10.2.4.1 Validation of TSS&TP ..................................................... 75

10.2.4.2 Validation of ATS specification......................................... 75

11 Conformance clause................................................................................................................ 76

11.1 Conformance with this ETS........................................................................................ 76

11.2 Conformance with ISO/IEC 9646................................................................................ 76

Annex A (normative): Normative criteria for the standardization of PICS (ICS) proforma

specifications.............................................................................................. 78

A.1 Static versus dynamic requirements in an ICS proforma ............................................................. 78

A.2 Level of detail of an ICS proforma ............................................................................................ 79

A.3 Requirements on numeric values............................................................................................... 81

A.4 References to ICS items.......................................................................................................... 83

A.5 Scope of qualified optionals...................................................................................................... 84

Annex B (informative): Components of a TS for a base specification when several ATSs are

produced.................................................................................................... 86

B.1 Why a single ATM has been chosen ......................................................................................... 86

B.2 Overview ................................................................................................................................87

B.3 Synopsis of the components and their dependencies.................................................................. 87

B.4 Relationships between components .......................................................................................... 89

B.5 Additional TSS&TP.................................................................................................................. 89

B.6 Consequences on the pre-existing ATS..................................................................................... 89

Annex C (informative): Guidance on the combination of TPs............................................................. 90

C.1 Tests having the same morphology........................................................................................... 90

C.2 Tests having the same spirit..................................................................................................... 92

Annex D (informative): Guidance on the definition of a test strategy.................................................. 93

D.1 Test strategy: initial steps ........................................................................................................ 93

D.1.1 Why do the protocols have to be tested? .................................................................... 93

D.1.2 What will an implementation to test look like?.............................................................. 93

D.2 Test strategy: choice of test method......................................................................................... 94

D.2.1 One or several test method(s)?.................................................................................. 95

D.2.2 Technical criteria intrinsic to standardized area............................................................ 95

D.2.3 Technical criteria related to implementation conditions.................................................. 97

D.2.4 Other criteria ............................................................................................................ 98

D.2.5 Further clarification on test methods characteristics..................................................... 98

D.2.6 Clarification concerning the concept of upper tester ..................................................... 98

D.3 Test strategy: coverage and costs............................................................................................ 99

---------------------- Page: 7 ----------------------
SIST ETS 300 406 E1:2003
Page 6
ETS 300 406: April 1995

D.4 Test strategy: technical refinements........................................................................................ 100

Annex E (informative): Bibliography.............................................................................................. 101

History ........................................................................................................................................... 102

---------------------- Page: 8 ----------------------
SIST ETS 300 406 E1:2003
Page 7
ETS 300 406: April 1995
Foreword

This European Telecommunication Standard (ETS) has been produced by the Methods for Testing and

Specification (MTS) Technical Committee of the European Telecommunications Standards Institute (ETSI).

This ETS provides rules and guidance on how to develop telecommunications conformance testing

specifications with the final objective to standardize them.

ETSI has explored innovative solutions for the long term, and at present, a "proposer" in harmonized

approaches to the pressing requirement of developing testing standards in various areas of

telecommunications. It is also essential that ETSI remains aware of what is going on in the information

technology area outside ETSI, in close relations with its own technical activities.

The production of conformance testing standards in ETSI has already started in various technical areas,

and needs to be continued with the best harmonization of efforts.

ETSI TC-MTS recommends that the TC and STC chairmen in their role of co-ordinators of standard

development ensure that the groups of experts working in conformance testing in their bodies are fully

aware of the status of affairs in this area so that initiatives are co-ordinated properly. Special care should

be devoted to the consistency of the methodology used (test specifications' structure, test notations,

criteria for acceptance, etc.), to the need of not duplicating efforts and to the strategy of improving

convergence for testing at the European and international levels.

With the co-operation of all ETSI members, an initial orientation towards a harmonized approach to

conformance testing standards and practice is desirable and possible. This is why the TC MTS work

program includes the production of a comprehensive methodology guide that will represent a further step

forward in harmonization and consistency.
Proposed transposition dates
Date of latest announcement of this ETS (doa): 31 July 1995
Date of latest publication of new National Standard
or endorsement of this ETS (dop/e): 31 January 1996
Date of withdrawal of any conflicting National Standard (dow): 31 January 1996
Introduction

This ETS is one element of a series of documents produced during 1992 and 1993 by ETSI. These ETSs

will provide a reference library of rules and guidance on how to standardize telecommunications

conformance testing specifications in Europe. These rules and guidance take into account the state of the

art, the international and European standards and reports already published on that matter, the specifics of

test suite editing, and the practices of the ETSI and CEN secretariats for the production of European

standards.

Although this ETS has been produced in order to become a reference document for ETSI, it is designed to

be used in a wider European context, for example, as a technical basis for accreditation and specification

purposes, as well as for European Commission initiated conformance testing programmes, e.g. the

Conformance Testing Services (CTS) programme.
---------------------- Page: 9 ----------------------
SIST ETS 300 406 E1:2003
Page 8
ETS 300 406: April 1995
Blank page
---------------------- Page: 10 ----------------------
SIST ETS 300 406 E1:2003
Page 9
ETS 300 406: April 1995
1 Scope

This European Telecommunication Standard (ETS) specifies a methodology for developing

telecommunications conformance testing specifications with the final objective to standardize them.

This ETS defines the components to be produced when specifying a European conformance Testing

Standard (TS) in the domain of telecommunications.

The methodology specified in this ETS is, in principle, applicable to domains other than

telecommunications, e.g. Open Systems Interconnection (OSI). However, the limits of such an application

of the methodology outside telecommunications testing are not defined, and each group defining test

specifications are responsible for deciding if any part of this methodology is applicable.

This methodology applies to conformance TSs for protocols, for profiles, for information objects, for

interfaces, and for services.

NOTE 1: The methodology of this ETS is generic, in the sense that it applies to protocols, to

profiles, to information objects, etc. Following the example of ISO/IEC 9646

(references [5] to [12]), the terminology employed is also generic, e.g. Implementation

Conformance Statement (ICS), Implementation eXtra Information for Testing (IXIT).

However, in order to avoid confusing the readers who are familiar with the protocol-

related terminology, the non-generic terms (Protocol Implementation Conformance

Statement (PICS), Protocol Implementation eXtra Information for Testing (PIXIT), etc.)

have been kept in the titles of clauses and subclauses.

The test of physical aspects, such as "layer 1", analogue measurements, Electro-Magnetic Compatibility

(EMC), is not covered by this ETS. However, part of the methodology may be still be applied, e.g. the

concepts of ICS, IXIT, Test Purposes (TPs), and Test Suite Structure (TSS).

This ETS provides the rules for standardization bodies developing telecommunications conformance TSs.

NOTE 2: One objective of this ETS is the use, when relevant, within standardization bodies. For

this purpose, within ETSI, it has the same value as an internal Technical Committee

Reference Technical Report (TCR-TR).

This ETS is applicable to organizations such as Conformance Testing Services (CTS) programmes, pre-

standardization institutions, private institutions, which develop conformance testing specifications with the

objective of contributing them for the purposes of European telecommunications standardization.

This ETS contains two types of information, following a twofold objective:

- it clarifies the principles of the ISO/IEC 9646 (references [5] to [12]) testing framework and

methodology, as well as the concept of profile;

- it defines additional criteria for European telecommunications TSs, whereby it constitutes a

"European telecommunications profile" of the ISO/IEC 9646 (references [5] to [12]) methodology,

containing its own recommendations.

This ETS is consistent with the normative references listed in clause 2. This ETS extends the current

principles contained in ISO/IEC 9646 (references [5] to [12]), in particular in the field of the combination of

TPs, and concerning the choice of a single Abstract Test Suite (ATS).

For conformance testing in the "voluntary domain" (i.e. outside the regulatory domain), for instance by the

suppliers to support their commercial claims for conformance, TSs are normally published as I-ETS, or

ETS, depending on their role and their maturity.
This ETS addresses telecommunications TS methodology for the voluntary domain.
---------------------- Page: 11 ----------------------
SIST ETS 300 406 E1:2003
Page 10
ETS 300 406: April 1995
2 Normative references

This ETS incorporates by dated and undated reference, provisions from other publications. These

normative references are cited at the appropriate places in the text and the publications are listed

hereafter. For dated references, subsequent amendments to or revisions of any of these publications apply

to this ETS only when incorporated in it by amendment or revision. For undated references the latest

edition of the publication referred to applies.
[1] ETR 153: "Methods for Testing and Specification (MTS); Guidance on the
production and completion of System Conformance Test Report (SCTR) and
Protocol Conformance Test Report (PCTR) proformas".
[2] ETR 141: "Methods for Testing and Specification (MTS); Protocol and profile
conformance testing specifications; The Tree and Tabular Combined Notation
(TTCN) style guide".
[3] PNE-Rules (1991): "CEN/CENELEC Internal Regulations - IR; Part 3: Rules for
the drafting and presentation of European Standards (PNE-Rules)".

[4] EN 45001 (1989): "General criteria for the operation of testing laboratories".

[5] ISO/IEC 9646-1 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
1: General concepts" (including DAM1 and DAM2, to be published).
[6] ISO/IEC 9646-2 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 2: Abstract Test Suite Specification" (including DAM1 and DAM2, to be
published).
[7] ISO/IEC 9646-3: "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation".
[8] ISO/IEC 9646-3 AM 1: "Information technology - Open Systems Interconnection
- Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation. Amendment 1: TTCN Extensions (concurrent
TTCN)".

NOTE 1: ISO/IEC 9646-3 AM 1 will be included in the next edition of ISO/IEC 9646-3, which will

contain TTCN and concurrent TTCN in the same standard.
[9] ISO/IEC 9646-4 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
4: Test Realisation" (including DAM1 and DAM2, to be published).
[10] ISO/IEC 9646-5 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
5: Requirements on test laboratories and clients for the Conformance
Assessment Process" (including DAM1 and DAM2, to be published).
[11] ISO/IEC 9646-6 IS (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 6: Protocol Profile Test Specification".
[12] ISO/IEC 9646-7 DIS (1993): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
7: Implementation Conformance Statements".
---------------------- Page: 12 ----------------------
SIST ETS 300 406 E1:2003
Page 11
ETS 300 406: April 1995

NOTE 2: The clauses of ISO/IEC 9646 (references [5] to [12]) referenced in this ETS are from

edition 2: 1994 (unpublished). Although the contents of ISO/IEC 9646 have changed, it

is useful to know the references to edition 1: 1991. The correspondence is as follows:

Edition 1: 1991 Edition 2: 1994
Part 2, clause 12 Part 2, clause 11
Part 1, annex B Part 2, annex A
3 Definitions and abbreviations
3.1 Definitions

For the purposes of this ETS, all the definitions in ISO/IEC 9646 (references [5] to [12]) apply.

In addition, the following definitions apply, or have been re-formulated for clarification:

Abstract Test Method (ATM) (Abstract Testing Method): Definition of ISO/IEC 9646-1 [5], subclause

3.3.5, applies.

NOTE 1: The ISO definition, complemented with the explanation of ISO/IEC 9646-2 [6],

subclause 11.1.7, properly specifies that the ATM contains "enough detail to enable

Abstract Test Cases (ATCs) to be specified for this test method". This means that the

ATM is a detailed description of the testing architecture. Conversely, the four methods

for Single Party Testing (SPy
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.