ISO/IEC 10641:1993
(Main)Information technology — Computer graphics and image processing — Conformance testing of implementations of graphics standards
Information technology — Computer graphics and image processing — Conformance testing of implementations of graphics standards
Specifies a general framework addressing the following six components: conformance in the standard itself; test requirements document (defining what shall be tested for a computer graphics standard); test specifications document (addressing the test technique and the content of each test); test method (defining the implementation of the test specification document, including the test software); test procedures (defining the application of the test software, which consists of the procedures to be used in conformance testing); the establishment of test services. Is applicable to all standards within the scope of the ISO/IEC JTC1 subcommittee responsible for computer graphics and image processing.
Technologies de l'information — Traitement informatisé de l'image et des graphiques — Essais de conformité de la mise en application des normes graphiques
General Information
Standards Content (Sample)
ISO/IEC
INTERNATIONAL
10641
STANDARD
First edition
1993-07-15
Information technology - Computer
graphics and image processing -
Conformance testing of implementations
of graphics standards
- Traitemen t informa tis6 de /‘image et des
Technologies de I’informa tion
graphiques - Essais de conformit de la mise en application des normes
graphiques
Reference number
lSO/IEC 10641 :I 993(E)
---------------------- Page: 1 ----------------------
0 ISO/IEC
ISO/IEC 10641:1993 (E)
Page
Contents
1
1 Scope .
2
2 Normative references .
4
3 Definitions .
G
...........................................................................
4 Overview
6
4.1 Conformance testing .
(i
4.2 Graphics conformance testing .
7
4.2.1 The impact of registration .
............................ 8
4.2.2 Interfaces in computer graphics conformance testing
1I)
4.3 The testing process .
11
...................................
5 Conformance testing requirements within graphics standards
13
6 Graphics test suite .
13
6.1 Test software .
13
..........................................
6.1.1 Determination of testing domain
14
.................................................
6.1.2 Structure of a test suite
14
..............................................
6.1.3 Maintenance of a test suite
IS
..............................................
6.1.4 Portability of test software
15
.........................................
6.1.5 Language bindings and encodings
IS
6.2 Test documentat& .
15
............................................
6.2.1 Test Requirements document
16
.............................................
6.2.2 Test Specifications document
17
............................................
6.2.3 Test suite and documentation
1X
.................................................................
7 Graphics test service
18
7.1 Procedures and guidelines .
18
7.1.1 Acceptance of a test suite .
18
7.1.2 Establishment of test procedures .
18
7.1.2.1 Testing Control Board .
19
7.1.2.2 Testing Control Board procedures .
19
............................................
7.1.2.3 Applying for testing
20
.............................................
7.1.2.4 Testing procedures
21
.....................................
7.1.2.5 Preparation of the test report
0 ISO/IEC 1993
All rights reserved. No part of this publication may be reproduced or utilized in any form or
by any means, electronic or mechanical, including photocopying and microfilm, without per-
mission in writing from the publisher.
l Case Postale 56 l CH-1211 Gen&e 20 l Switzerland
I SO/I EC Copyright Off ice
Printed in Switzerland
ii
---------------------- Page: 2 ----------------------
0 ISO/IEC ISO/IEC 10641r1993 (E)
................................................
21
7.1.2.6 Confidentiality
...............................................
21
7.1.2.7 Documentation
...........................................
21
7.1.2.8 Archiving of records
...................................................
21
7.1.2.9 Checklists
.................................................
22
7.2 Adoption of a test report format
..............................................................
22
7.3 Issue of licences
......................................................
22
7.4 Maintenance requirements
..............................................................
23
8 Establishing a test service
....................................................
23
8.1 Components of a test service
.................................................
23
8.1.1 The testing laboratory
....................................................
23
8.1.2 Accreditation body
.....................................................
23
81.3 Certification body
.................................................
23
8.1.4 Testing support service
24
8.1.5 CIient .
...................................................
24
8.2 Harmonisation of test services
Annexes
.........................................................
25
A Overview of the testing process
.................................................
26
B Overview of test suite development process
I
. . .
111
---------------------- Page: 3 ----------------------
ISO/IEC 10641:1993 (E)
0 ISO/IEC
Foreword
IS0 (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form
the specialized system for worldwide standardization. National bodies that are members of IS0 or IEC participate in the
development of International Standards through technical committees established by the respective organization to deal
with particular fields of technical activity. IS0 and IEC technical committees collaborate in fields of mutual interest.
Other international organizations, governmental and non-governmental, in liaison with IS0 and IEC, also take part in the
work.
In the field of information technology, IS0 and IEC have established a joint technical committee, ISO/IEC JTC 1. Draft
International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication
as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
International Standard ISO/IEC 10641 was prepared by Joint Technical Committee ISOLIEC JTC 1, Information tech-
nology, Sub-Committee SC 24, Computer graphics and image processing.
Annexes A and B of this International Standard are for information only.
iv
---------------------- Page: 4 ----------------------
ISOAEC 10641:1993 (E)
0 ISO/IEC
Introduction
This International S tandard addresses conform nance testing of implementations of graphics stand‘ards. Conformance testing is
the method used to
determine the adherence or non-adherence of an implementation under test (WI’) to a staklard.
This International Standard specifies an approach for St2 ndards of products that
testing the conformance to computer graphics
claim to implement these standards. It addresses the conformance testing processes for all classes of graphics standards
This International Standard defirtes a general framework of procedures and guidelines for conformance testing, together with
definitions of terms and concepts.
The framework given in this International Standard, together with the Test Requirements document for a particular graphics
standard, provides a description of the procedures to be followed to achieve successful conformance testing of products for
conformance to a particular graphics standard.
The concept of conformance is central to every standard. The aims and benefits of a standard can be realized if there is a
means of testing for conformance.
The main reasons for introducing a document on conformance testing in the area of computer graphics are:
- To promote standards that are developed in a way such that products can be tested for conformance to the standards’
requirements;
- To promote that conformance is addressed in each standard;
- To promote test suites that are appropriately defined for testing products for conformance to all areas of the
standard, and are of high quality;
- To promote test methods for similar standards that are developed in a consistent way;
- To promote conformance testing that is carried out in a consistent way throughout the international graphics
community.
Users of this International Standard include:
- Developers of graphics standards;
- Implementors of graphics standards;
- Developers of graphics test suites;
- Testing laboratories;
- Certification bodies;
- Accreditation bodies.
Annexes A and B contain diagrams illustrating the relationships among the users of this International Standard and the
information shared by them.
---------------------- Page: 5 ----------------------
This page intentionally left blank
---------------------- Page: 6 ----------------------
ISO/IEC 10641:1993(E)
INTERNATIONAL STANDARD
Information technology - Computer graphics and
image processing - Conformance testing of
implementations of graphics standards
1
Scope
This International Standard specifies a general framework for testing conformance to a computer graphics standard. The general
framework described in this International Standard addresses the following six components:
- Conformance in the standard itselfr
- Test Requirements document, defining what shall be tested for a computer graphics standard;
- Test Specifications document, addressing the test technique and the content of each test
- Test method, defining the implementation of the Test Specification document, including the test software;
- Test procedures, defining the application of the test software, which consists of the procedures to be used in
conformance testing;
- The establishment of test services.
This International Standard is applicable to all standards within the scope of the subcommittee within ISO/IEC JTCl responsible
for computer graphics and image processing.
---------------------- Page: 7 ----------------------
0 ISOAEC
ISO/IEC 10641:1993 (E)
2 Normative references
The following st‘andcuds contain provisions which, through reference in this text, constitute provisions of this International
Stand‘ard. At the time of publication, the editions indicated were valid. All stLandcards care subject to revision, ‘and p‘arties to
agreements bczsed on this International Stand,ud
of the standcards indicated below. Members of IEC and IS0 mintain registers of currently valid International Standczrds.
Gruphicul Kernel System (MS) functionul
ISO/IEC 7942: 1985, Informution processing systems - Computer gruphios -
description.
ISO/IEC 86320 1: 1992, Infbrmution technology - Computer gruphics - Metufile for the storuge und transfer of picture description
inf~wnution -Purt 1: Functional specijkution.
ISO/lEC 8632-2: 1992, Inf~wnzution technology - Computer gruphics - Metufile for the storuge und trunsfer of picture description
information -Part 2: Churucter encoding.
ISO/IEC 8632-3: 1992, Informxztion technology - Computer gruphics - Metufile for the storqe und trunsfer of picture description
inf~wnution -Purt 3: Binary encoding.
ISO/IEC 8632-4: 1992, Informution technology - Computer gruphics - Metufile for the storage und trunsfer of picture description
infwmution -Purt 4: Cleur text encoding.
Gruphicul Kernel System (GIG) lunguqe
ISO/IEC 8651-l: 1988, Information processing systems - Computer gruphics -
bindings - Purt I: FORTRAN.
- Gruphicul Kernel System (GKS) lunguuge
ISO/IEC 865 l-2: 1988, Information processing systems - Computer gruphics
bindings - Purt 2: Puscul.
Gruphicul Kernel System (GKS) lunguuge
ISO/IEC 865 l-3: 1988, Information processing systems - Computer gruphics -
bindings - Part 3: Adu.
ISO/IEC 865 l-4: 199 1, Information technology - Computer gruphics - Gruphicul Kernel System @KS) lunguuge bindings - Purt
4: c.
- Gruphicul Kernel System for Three Dimensions
ISO/IEC 8805: 1988, Information processing systems - Computer gruphics
(GKS-3D) functional description.
ISO/IEC 8806-l: 1988, Information processing - Computer gruphics - Gruphicul Kernel Systemfor Three Dimensions (GKS-3D)
language bindings - Part I: FORTRAN.
ISOIIEC 8806,4:-l), Inform&on technology - Computer gruphics - Gruphicul Kernel Systemfbr Three Dimensions (GKS-3D)
lunguuge bindings - Purt 4: C.
ISO/IEC 9592~1:1989/Amd.l: 1992, Informution processing systems - Computer gruphics - Progrummer’s Hierurchicul
Inteructive Gruphics System (PHI(X) - Purt 1: Functionul description Amendment i.
IS()/IEC 9592-2: 1989/Amd. 1: 1992, Information processing systems - Computer gruphics - Progrummer’s Hierurchkul
Interactive C;ruphics System (PHIGS) - Purt 2: Archive file $rmut Amendment I.
ISO/IEC 9592-3: 1989/&n& 1: 1992, Inf~wmtion processing systems - Computer gruphics - Progrummer’s Hierurchicul
Interuc:tive Ifruphir:s System (PHIC;S) - Purt 3: Cleur text encoding of urchive file Amendment 1.
IsO/IEC 9592-4: 1992, Inf(lrmtion proc:essing systems - Computer gruphics - Proq-ummer’s Hierurchicul Inteructive Gruphics
System (PHIGS) - Part 4: Plus Lumiere und Su$k:es, PHIGS PLUS.
1) To be published.
2
---------------------- Page: 8 ----------------------
ISO/IEC 10641:1993 (E)
0 ISO/IEC
ISO/IEC 9593-1: 1990, Informtion prclcessing systems - Computer gruphics - Programmer’s Hierurchicul Inteructive (iruphil:s
System (PHIGS) language bindings - Part 1.’ FORTRAN.
ISOfIEC 9593-3: 1990, Information technology - Computer gruphics - Progrummer’s Hierurchicul Interactive Gruphics System
(PHIGS) lunguuge bindings - Purt 3: Adu.
ISO/IEC 9593-J: 1992, Inftwmution technology - Computer gruphics - Progrummer’s Hierurchicul Interutrtive Gruphics System
(PHIGS) lunguuge bindings - Purt 4: C.
ISO/IEC 9636- 1: 199 1, Information technology - Computer gruphics - Inte@xing techniques j?lr dialogues with gruphicul
devices (WI) - Functionul specijkution - Purt I : Overview, &files and confx-munce.
ISO/IEC 9636-2: 1991, Informution technology - Computer gruphics - Interfclcing techniques for diulogues with gruphicul
devices (WI) -Functionul specification - Part 2: Control.
ISO/IEC 9636-3: 1991, Information technology - Computer gruphics - Interjktring techniques for dialogues with gruphicul
devices (WI) - Functionul specification - P&t 3: Output. -
ISO/IEC 9636-4: 1991, Inf~~rmution technology - Computer gruphics - Inter$ucing techniques for dialogues with gruphicrul
devices (WI) - Functionul specification - Purt 4: Segments.
ISO/IEC 9636-S: 1991, Informution technology - Computer gruphics - Interfking techniques for diulogues with gruphicul
devices (WI) - Functional specificution - P&t 5: Input und echoing.
ISO/IEC 9636-6: 1991, Infi~rmution technology - Computer gruphics - Inte$ucing techniques for dialogues with gruphicul
devices (CGI) - Functional specificution - Purt 6: Raster. -
ISOLIEC 9637-l :-2~,Informution technology - Computer gruphics - Interfking techniques for dialogues with gruphicul devices
(CM) - Data streum binding - Part I: Character encoding.
ISO/IEC 9637-2: 1992, Information technology - Computer gruphics - Inter-fcrcing techniques for dialogues with gruphicul
devices (WI) - Dutu streum binding - Purt 2: Binury encoding
- Computer Gruphics Reference Model.
ISO/IEC 11072: 1992, Information technology - Computer gruphics
ISO/IEC Guide 2: 199 1, Generul terms and their definitions concerning stundurdizution und reluted uctivities.
ISO/IEC Guide 23: 1982, Methods of indicating conformity with stundurds for third-party certijkution systems.
ISO/IEC Guide 25: 1990, Generul requirements for the competence of culibrution und testing luborutories.
ISOfIEC Guide 2811982, General rules for u model third-purty certification system for products.
ISO/IEC Guide 45:198S, Guidelines for the presentution of test results.
2) To be published.
---------------------- Page: 9 ----------------------
ISO/IEC 10641:1993 (E)
0 ISO/IEC
3 Definitions
For the purposes of this International Standard, the following definitions apply.
3.1 accreditation: Formal recognition that a testing laboratory is competent to ccv~y out specific tests or specific types of
tests.
3.2 accreditation body: A body which conducts ‘and administers a laboratory accreditation system and giants accreditation.
3.3 application programmer interface (API) standard: A standard which provides
interface.
3.4 candidate implementation: Implementation that is tested for conform,ulce to a given standard.
3.5 certificate (of conformity): A document attesting that a product or a service is conforming to one or more specific
st,ulda.rds or technicczl specifications.
3.6 certification: Procedure resulting in the issuLvlce of a certificate.
3.7 certification body: Body that conducts certificates of conformity “[ISO/IEC Guide 21”; body that issues certificates of
conformity.
3.8 certification criteria: Criteria that determine whether a certificate is issued.
3.9 client: Anyone requesting conform,ulce testing.
3.10 configuration: An interleaved combination of hardw‘are and software including the host computer, the host operating
system, the device-independent graphics package, the device driver h‘andling graphic‘al input/output on a specific device and
the programming hanguage.
3.11 conformance: Fulfillment by a product, process or service of all requirements specified “[ISO/EC Guide 2,
conformity]“; adherence of an implementation to the requirements of one or more specific standards or technical specifications.
3.12 conformance testing: Test to evaluate the adherence or non-adherence of a ccvldidate implementation to a st‘andard.
3.13 data stream encoding: Specific representation of the syntax of a graphics st‘andard suitable for use over a data
interface.
3.14 essential feature: Features that, when grouped together, perform a minimal set of operations expected by a st‘andard.
3.15 falsification: Test method that attempts to find errors in a c‘andidate implementation to determine if it is incorrect.
3.16 full conformance: The implementation of all required features error-free.
3.17 implemmtation conformance: Description of what is required of implementors in order to have the implementation
conform to a standard.
3.18 implementation under test (IUT): A cculdidate implementation being tested.
3.19 language binding: The progmmmer’s interface to the functions in a graphics system through a programming language.
3.20 metafile standards: Smndards defining graphical elements and corresponding encodings for storage ‘and tr,ulsfer of
graphical information.
---------------------- Page: 10 ----------------------
ISO/IEC 10641:1993 (E)
0 ISOAEC
3.21 minimal conformance: The implementation of all essential features error-free.
3.22 operator: The external object that observes the contents of the display and generates physical input values.
3.23 operator interface: Interface provided by the physical environment to the operator.
3.24 program conformance: Description of what is required of a progr,un in order to conform to a standard.
3.25 test case: Smallest unit of a test program that tests one feature of a candidate implementation.
3.26 test method: Specified technical procedure for performing a test “[ISO/IEC Guide 21”; a defined technical procedure
used to design a test suite for a given stand‘ard.
3.27 test procedures: Defines the procedures to be followed when applying a test suite to a product for the purposes of
conformance testing.
3.28 test program: A program consisting of a set of test cases.
3.29 test report: A document that presents the test results and other information relev‘ant to the tests (e.g., configuration
description ‘and detected errors).
Describes the features ‘and functions defined in a pcuticu1‘a.r stand‘ard to which a
3.30 test requirements document:
c‘andidate implementation shall be tested for conform,ulce.
3.31 test result: Output of a test ase.
3.32 test script: Document describing the test software and its v,uious test cases for operator guidance ‘and decision support.
3.33 test service: Service offered by testing laboratories.
3.34 test specifications document: Describes procedures, tools iand test ceases that
Test Requirements document.
3.35 test software: Set of test progr,uns cznd corresponding documentation that care used for conform,ulce testing.
for
cvld test procedures that check an implementation
3.36 test suite: The combination of test softw‘are, test documentation,
conform,ulce to a standcard.
standards and testing set up for each or type of products to
3.37 testing control board: A bo‘ard of experts in
group
problems raised (as a result of conformance testing.
3.38 testing laboratory: Laboratory that performs tests “[ISO/IEC Guide 21”; a laboratory that measures conform,ulce of
implementations of standards to a group or type of standards.
3.39 testing support service: Orgculization responsible for a p‘articular test service (e.g., maintenance of test suite, licensing).
3.40 validation: Testing for conformance.
3.41 verification: Method to prove the correctness of a candidate implementation against standards or technical specifications.
---------------------- Page: 11 ----------------------
ISOAEC 10641:1993 (E)
0 ISO/IEC
4 Overview
Correct utilization of standards allows applications to be moved among different computers and graphics devices with minimal
ch,ulge. The degree of portability achieved is affected by the degree of support for the sca.me sets of functions, levels, and
implementation dependent features of (an implementation. Conformcvlce testing using test suites encourages implementors to
use the standczrds correctly by checking for deviations ‘and omissions from the standards. When all deviations and omissions
,are eliminated, the implementation conforms to the stand~ard. This greatly diminishes or removes the work involved to achieve
portability between dissimilar systems. In rncvly countries, an implementation of a standard that has obtained a certificate is
favoured for purchase by the public. In addition, the certificate may be mandated for sales into certain markets, such as the
government.
Even before a standard becomes official, there is a strong effort to implement products that (claim to) conform to the proposed
standczrd. This is because even d&t standcards create a strong user dem,uld. For ex‘atnple, vendors started advertising
implementations of ca.n IS0 graphics standard at least one ye(zr before the st‘and‘ard wcas actu‘ally approved by ISO/IEC.
Shortly
after that, these implementations were available on the marketplace. Without defined test methods, some of these packages
be‘ar only superficial resemblance to a correct implementation of the standard. Other implementations might conform to many
aspects of the standard but might still be deficient in subtle La.reas. Thus, test suites that test implementations to determine
conformance to standcuds ,a.re needed.
4.1 Conformance testing
Conform,ulce testing is a way of scrutinizing implementations of a computer standczrd to determine whether or not deviations
from the standard exist. Standards usually contain two main ingredients: semantics and syntax. The semantics is the
functional description that defines precisely what must be done, but not how it is to be done. The syntax defines the
mechanism by which these functions an be accessed. Syntax may consist of verbs in a progr,unming language to access the
function, or (in the c
t,?ke the form of data stream encodings for data interchange st,uld
However, a third ingredient is sometimes overlooked in standards - conformance. The conformance or classification and
designation clause, in conjunction with the rest of the standard, specifies the requirements an implementor shall adhere to in
order to conform to the stand‘ard and sets the groundwork for the development of conformance tests.
A test suite is the combination of test softw‘are, test script, and test procedures, aI of which check an implementation for
conformance to a sta.nd,ud.
The approach usually used in developing test suites for testing conformance of implementations is called falsification testing.
This method uses sample c
A test suite tries to find
errors in the implementation. If errors are found, one can correctly deduce that the implementation does not conform to the
standard. However, the absence of errors does not necesa-ily prove that the implementation is correct. The absence of errors
implies either that the implementation conforms to the st‘and
errors. Thus, f‘alsification testing can determine non-conform,ulce to a st‘andca.rd but ccan never cassure complete conformance.
Falsification testing is the method used to test conformance to graphics standards.
A Test Requirements document describes the features ‘and functions defined in a particukar standard to which ‘an implementation
It is vital to the correctness and completeness of a test suite that the Test
under test (IUT) sh‘all be tested for conform,ulce.
Requirements document be developed in p‘araIle1 with the standczrd. This guarantees communication between the standards
developers and test suite developers, which ensures that features in the standard ,a.re defined in a manner which makes testing
possible. It also ensures that the tests (are more comprehensive because the test developers are directly involved in the
specification of the standczrds and can more thoroughly understand what needs to be tested.
4.2 Graphics conformance testing
Generally, the sem‘antics for standards are specified using the English language in n,urative form. As suitable formal
specification techniques mature ‘and experience in their usage grows, it is desirable that the semantics of standards be specified
using such techniques. This would reduce the probability of errors ,uld ambiguities in the standard. Additionally, if formal
specifications existed, formal proofs of correctness could be developed which prove conformance to the standard. This is
contrasted with falsification testing which can only indicate non-conform(ulce to the standcard.
6
---------------------- Page: 12 ----------------------
ISO/IEC 10641:1993 (E)
0 ISOAEC
The absence of formal specifications hcas a profound effect on conform,ulce testing. Without a method for defining an
umambiguous definition of a stand‘ard (a form~aI specification), there is a d‘anger that test routines may become the definition
of the standdard. This is because (any questions of interpretation of the English hanguage sem‘antics or questions of (ambiguities
must be resolved by the developer of a test suite prior to tr,2nscribing the semMtics into vczlidation softw,ue. The possibility
that the criteria for testing conforn-uance may be b‘ased on subjective interpretation by a test developer necessitates the
establishment of a Testing Control Bo(zrd for each functiomal stand‘ard. This Bo‘ard will oversee the development ‘and
m,?inten,ulce of the tests ‘and h,urdle questions of interpretation.
Syntax in graphics standards may consist of hanguage bindings that allow a progr,unmer to access graphics functions in the
hanguage he is using (e.g., FORTRAN, C, Pascal, Ada). Conform,ulce testing aims to ensure that the functions perform
correctly and also that the hanguage bindings have been implemented correctly. Syntax in data interch‘ange standcards, such as
ISO/IEC 8632-l (CGM), specify data encodings for the CGM functionality. For these stand‘ards, conform,ulce tests czirn to
ensure that these encodings (are implemented correctly.
The development of test suites for graphics standards is unique due to the pictorial nature of graphics output. Tests for
conformance to other standards, such as progmmming language standards, can automatic,2lly compare the desired results with
the actual results ‘and c‘an easily m,?ke a pass/fail determination with respect to conform‘ance of the IUT. However, in the
present method of evaluating conformdance of proposed standard-conforming graphics implementations, visual analysis of
Examples of questions (asked during this ‘analysis
pictorial information is required.
“is the line style recognizable as a series of dashed lines”, etc. If API standards were more precise,
recognizable cas text”,
testing could be more automated ‘and less subjective judgement would be required.
4.2.1 The impact of registration
Conformance tests concentrate strictly on implementation conform,ulce. They test that the implementation meets all mandatory
requirements of a star&-d, as well as optional requirements. For example, ISO/IEC 7942 (GKS) requires that four m‘andatory
linetypes exist for the polyline primitive. Thus the GKS test suite ensures that the four m,uldatory linetypes (are implemented
correctly including the ‘assignment of correct vczlues for each linetype (i.e., linetype 1 = solid, linetype 2 = dashed, etc.).
In addition, ‘an implementor may supply cvly number of linetypes by ‘assigning negative values to them. The additional linetypes
may be specific to his implementation. Optiomally, he may choose to share these linetypes with other implementations so that
users c‘an obtain portability. In this c‘ase positive v(alues are assigned to the linetypes and documented in the IS0 Register of
Graphical Items.3)
3) For the purpose of this International Standard and according to the rules for the designation and operation of registration
authorities in the I
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.