Software and systems engineering — Software testing — Part 5: Keyword-Driven Testing

ISO/IEC/IEEE 29119-5:2016 defines an efficient and consistent solution for Keyword-Driven Testing by: giving an introduction to Keyword-Driven Testing; providing a reference approach to implement Keyword-Driven Testing; defining requirements on frameworks for Keyword-Driven Testing to enable testers to share their work items, such as test cases, test data, keywords, or complete test specifications; defining requirements for tools that support Keyword-Driven Testing. These requirements could apply to any tool that supports the Keyword-Driven approach (e.g., test automation, test design and test management tools); defining interfaces and a common data exchange format to ensure that tools from different vendors can exchange their data (e.g. test cases, test data and test results); defining levels of hierarchical keywords, and advising use of hierarchical keywords. This includes describing specific types of keywords (e.g. keywords for navigation or for checking a value) and when to use "flat" structured keywords; providing an initial list of example generic technical (low-level) keywords, such as "inputData" or "checkValue". These keywords can be used to specify test cases on a technical level, and may be combined to create business-level keywords as required. NOTE This standard is applicable to all those who want to create keyword-driven test specifications, create corresponding frameworks, or build test automation based on keywords.

Ingénierie du logiciel et des systèmes — Essais du logiciel — Partie 5: Essais axés sur des mots-clés

General Information

Status
Published
Publication Date
16-Nov-2016
Current Stage
9093 - International Standard confirmed
Ref Project

Relations

Buy Standard

Standard
ISO/IEC/IEEE 29119-5:2016 - Software and systems engineering -- Software testing
English language
54 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (Sample)

INTERNATIONAL ISO/IEC/
STANDARD IEEE
29119-5
First edition
2016-11-15
Software and systems engineering —
Software testing —
Part 5:
Keyword-Driven Testing
Ingénierie du logiciel et des systèmes — Essais du logiciel —
Partie 5: Essais axés sur des mots-clés
Reference number
ISO/IEC/IEEE 29119-5:2016(E)
©
ISO/IEC 2016
©
IEEE 2016

---------------------- Page: 1 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2016, Published in Switzerland
© IEEE 2016
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without
prior written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the
country of the requester.
ISO copyright office IEC Central Office 3,
Institute of Electrical and Electronics Engineers, Inc
Ch. de Blandonnet 8 • CP 401
rue de Varembé CH- 3 Park Avenue, New York
CH-1214 Vernier, Geneva, Switzerland
NY 10016-5997, USA
1211 Geneva 20
Tel. +41 22 749 01 11
Switzerland
Fax +41 22 749 09 47
E-mail inmail@iec.ch
copyright@iso.org
stds.ipr@ieee.org
Web www.iec.ch
www.iso.org
www.ieee.org
© ISO/IEC 2016 – All rights reserved
ii © IEEE 2016 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
Contents Page
1 Scope .1
2 Conformance .1
2.1 Intended usage .1
2.2 Full conformance .1
2.3 Tailored conformance .2
3 Normative references .2
4 Terms and definitions .2
5 Introduction to Keyword-Driven Testing .4
5.1 Overview .4
5.2 Layers in Keyword-Driven Testing .7
5.2.1 Overview .7
5.2.2 Domain layer .8
5.2.3 Test interface layer .9
5.2.4 Multiple layers .9
5.3 Types of keywords. 10
5.3.1 Simple keywords. 10
5.3.2 Composite keywords . 11
5.3.3 Navigation/interaction (input) and verification (output) . 14
5.3.4 Keywords and test result . 14
5.4 Keywords and Data. 15
6 Application of Keyword-Driven Testing . 16
6.1 Overview . 16
6.2 Identifying keywords . 16
6.3 Composing test cases . 17
6.4 Keywords and data-driven testing . 18
6.5 Modularity and refactoring . 18
6.6 Keyword-Driven Testing in the Test Design Process . 19
6.6.1 Overview . 19
6.6.2 TD1 Identify Feature Sets . 20
6.6.3 TD2 Derive Test Conditions . 20
6.6.4 TD3 Derive Test Coverage Items . 20
6.6.5 TD4 Derive Test Cases . 21
6.6.6 TD5 Assemble Test Sets . 22
6.6.7 TD6 Derive Test Procedures . 22
6.7 Converting non keyword-driven test cases into Keyword-Driven Testing . 22
7 Keyword-Driven Testing Frameworks . 22
7.1 Overview . 22
7.2 Components of a Keyword-Driven Testing framework. 23
7.2.1 Overview . 23
7.2.2 Keyword-driven Editor . 25
7.2.3 Decomposer . 26
7.2.4 Data sequencer . 26
7.2.5 Manual test assistant . 26
7.2.6 Tool bridge . 26
7.2.7 Test execution environment and execution engine . 26
7.2.8 Keyword library . 27
7.2.9 Data . 27
7.2.10 Script repository . 27
7.3 Basic attributes of the Keyword-Driven Testing framework . 27
7.3.1 General information on basic attributes . 27
7.3.2 General attributes . 27
7.3.3 Dedicated keyword-driven editor (tool) . 28
7.3.4 Decomposer and data sequencer . 29
© ISO/IEC 2016 – All rights reserved iii
© IEEE 2016 – All rights reserved

---------------------- Page: 3 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
7.3.5 Manual test assistant (tool) . 29
7.3.6 Tool bridge . 29
7.3.7 Test execution engine . 29
7.3.8 Keyword library . 30
7.3.9 Script repository . 30
7.4 Advanced attributes of frameworks . 30
7.4.1 General information on advanced attributes . 30
7.4.2 General attributes . 30
7.4.3 Dedicated keyword-driven editor (tool) . 31
7.4.4 Decomposer and data sequencer . 31
7.4.5 Manual test assistant . 31
7.4.6 Tool bridge . 31
7.4.7 Test execution environment and execution engine . 32
7.4.8 Keyword library . 33
7.4.9 Test data support . 33
7.4.10 Script repository . 33
8 Data interchange . 33
Annex A (normative) Conventions . 34
Annex B (informative) Benefits and Issues of Keyword-Driven Testing . 35
B.1 General benefits of Keyword-Driven Testing . 35
B.2 Benefits of Keyword-Driven Testing for test automation . 35
B.3 Benefits of Keyword-Driven Testing for manual testing . 36
B.4 Possible issues with Keyword-Driven Testing . 36
Annex C (informative) Getting started with Keyword-Driven Testing . 37
C.1 General . 37
C.2 Identifying Keywords . 37
C.3 Composing test cases . 38
Annex D (informative) Roles and Tasks . 39
D.1 Overview – Roles and Tasks . 39
D.2 Domain expert . 39
D.3 Test designer . 39
D.4 Test automation expert . 40
Annex E (informative) Basic keywords . 41
E.1 Overview . 41
E.2 Basic keywords for a GUI . 41
E.3 Example application of basic keywords . 45
Annex F (informative) Examples. 49
F.1 Overview . 49
F.2 Example: test procedure from ISO/IEC/IEEE 29119-3 . 49
F.3 Example: Test of shopping procedure with low-level keywords . 51
F.4 Example for calculator with low-level keywords . 52
F.5 Example for calculator with domain level keywords . 52
Annex G Bibliography . 54
iv © ISO/IEC 2016 – All rights reserved
© IEEE 2016 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees established
by the respective organization to deal with particular fields of technical activity. ISO and IEC technical
committees collaborate in fields of mutual interest. Other international organizations, governmental and non-
governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO
and IEC have established a joint technical committee, ISO/IEC JTC 1.
IEEE Standards documents are developed within the IEEE Societies and the Standards Coordinating
Committees of the IEEE Standards Association (IEEE-SA) Standards Board. The IEEE develops its standards
through a consensus development process, approved by the American National Standards Institute, which
brings together volunteers representing varied viewpoints and interests to achieve the final product. Volunteers
are not necessarily members of the Institute and serve without compensation. While the IEEE administers the
process and establishes rules to promote fairness in the consensus development process, the IEEE does not
independently evaluate, test, or verify the accuracy of any of the information contained in its standards.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of ISO/IEC JTC 1 is to prepare International Standards. Draft International Standards adopted
by the joint technical committee are circulated to national bodies for voting. Publication as an International
Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is called to the possibility that implementation of this standard may require the use of subject matter
covered by patent rights. By publication of this standard, no position is taken with respect to the existence or
validity of any patent rights in connection therewith. ISO/IEEE is not responsible for identifying essential
patents or patent claims for which a license may be required, for conducting inquiries into the legal validity or
scope of patents or patent claims or determining whether any licensing terms or conditions provided in
connection with submission of a Letter of Assurance or a Patent Statement and Licensing Declaration Form, if
any, or in any licensing agreements are reasonable or non-discriminatory. Users of this standard are expressly
advised that determination of the validity of any patent rights, and the risk of infringement of such rights, is
entirely their own responsibility. Further information may be obtained from ISO or the IEEE Standards
Association.
ISO/IEC/IEEE 29119-5 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering, in cooperation with the Software & Systems
Engineering Standards Committee of the IEEE Computer Society, under the Partner Standards Development
Organization cooperation agreement between ISO and IEEE.
ISO/IEC/IEEE 29119 consists of the following parts, under the general title Software and systems engineering
— Software testing:
 Part 1: Concepts and definitions
 Part 2: Test processes
 Part 3: Test documentation
 Part 4: Test techniques
 Part 5: Keyword-Driven Testing
© ISO/IEC 2016 – All rights reserved v
© IEEE 2016 – All rights reserved

---------------------- Page: 5 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
Introduction
The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-
agreed set of standards for software testing that can be used by any organization when managing or
performing any form of software testing.
This part of ISO/IEC/IEEE 29119 defines a unified approach for describing test cases in a modular way, which
assists with the creation of items like keyword-driven test specifications and test automation frameworks. The
term "keyword" refers to the elements which are, once defined, used to compose test cases, such as with
building blocks. ISO/IEC/IEEE 29119-5 will explain the main concepts and application of Keyword-Driven
Testing. It will also define attributes of frameworks designed to support Keyword-Driven Testing.
The concepts and definitions relating to software testing defined in ISO/IEC/IEEE 29119-1 are also applicable
to ISO/IEC/IEEE 29119-5.
The test process model on which the Keyword-Driven Testing framework is based is defined in ISO/IEC/IEEE
29119-2 Test Processes. It comprises test process descriptions that define the software testing processes at
the organizational level, test management level and dynamic test level. Supporting informative diagrams
describing the processes are also provided in ISO/IEC/IEEE 29119-2. However, ISO/IEC/IEEE 29119-5
describes a specific implementation of the test design and implementation processes of ISO/IEC/IEEE
29119-2; in particular TD4 (Derive Test Cases), TD5 (Assemble Test Sets) and TD6 (Derive Test Procedures)
as here applied to Keyword-Driven Testing.
The templates and examples of test documentation as defined in ISO/IEC/IEEE 29119-3 are also applicable
to ISO/IEC/IEEE 29119-5.
Software test design techniques that can be used during test design are defined in ISO/IEC/IEEE 29119-4
Test Techniques. The application of ISO/IEC/IEEE 29119-4 is assumed when designing test cases that are
then described by keywords according to ISO/IEC/IEEE 29119-5.
This part of ISO/IEC/IEEE 29119-5 has the following structure:
 terms and definitions can be found in clause 4
 an introduction to Keyword-Driven Testing is given in clause 5
 the application of Keyword-Driven Testing is explained in clause 6
 frameworks for Keyword-Driven Testing are described in clause 7
 data interchange is covered in clause 8
 Annex A states naming conventions for keywords
 Annex B names benefits that can be achieved with Keyword-Driven Testing
 Annex C gives advice on how interested parties wanting to use Keyword-Driven Testing can start
 Annex D describes roles that can be used in Keyword-Driven Testing
 Annex E contains examples of basic keywords that can be used to create test cases
 Annex F contains examples for keyword test cases
vi © ISO/IEC 2016 – All rights reserved
© IEEE 2016 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC/IEEE 29119-5:2016(E)
Software and Systems Engineering — Software Testing —
Part 5: Keyword-Driven Testing
1 Scope
This part of ISO/IEC/IEEE 29119 defines an efficient and consistent solution for Keyword-Driven Testing
by:
 giving an introduction to Keyword-Driven Testing;
 providing a reference approach to implement Keyword-Driven Testing;
 defining requirements on frameworks for Keyword-Driven Testing to enable testers to share their work
items, such as test cases, test data, keywords, or complete test specifications;
 defining requirements for tools that support Keyword-Driven Testing. These requirements could apply
to any tool that supports the Keyword-Driven approach (e.g., test automation, test design and test
management tools);
 defining interfaces and a common data exchange format to ensure that tools from different vendors
can exchange their data (e.g. test cases, test data and test results);
 defining levels of hierarchical keywords, and advising use of hierarchical keywords. This includes
describing specific types of keywords (e.g. keywords for navigation or for checking a value) and
when to use "flat" structured keywords;
 providing an initial list of example generic technical (low-level) keywords, such as "inputData" or
"checkValue". These keywords can be used to specify test cases on a technical level, and may be
combined to create business-level keywords as required.
NOTE
This standard is applicable to all those who want to create keyword-driven test specifications, create
corresponding frameworks, or build test automation based on keywords.
2 Conformance
2.1 Intended usage
The requirements in ISO/IEC/IEEE 29119-5 are contained in Clause 7 and in Annex A. ISO/IEC/IEEE 29119-
5 provides requirements on frameworks supporting the application of Keyword-Driven Testing. It is
recognized that particular projects or organizations may not need to use all of the components defined
in this standard. Therefore, implementation of ISO/IEC/IEEE 29119-5 typically involves selecting a set of
components or parts of components suitable for the organization or project. There are two ways that an
organization can claim to conform to the provisions of this standard.
The organization or individual shall assert whether full or tailored conformance to this standard is
claimed.
2.2 Full conformance
Full conformance is achieved by demonstrating that all of the Keyword-Driven Testing requirements (i.e.
shall statements) defined in ISO/IEC/IEEE 29119-5 have been satisfied.
© ISO/IEC 2016 – All rights reserved
1
© IEEE 2016 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
2.3 Tailored conformance
When ISO/IEC/IEEE 29119-5 is used for implementing components of frameworks that do not qualify for full
conformance, the subset of components for which tailored conformance is claimed should be recorded.
Tailored conformance is achieved by demonstrating that all of the requirements (i.e. shall statements) for the
recorded subset of components have been satisfied.
Where tailoring occurs, the justification shall be provided, either directly or by reference, whenever a
requirement defined in clauses 7 and Annex A of ISO/IEC/IEEE 29119-5 is not followed. All tailoring decisions
shall be recorded with their rationale, including the consideration of any applicable risks. Tailoring decisions
shall be agreed to by the relevant stakeholders.
EXAMPLE Tool vendors may in their portfolio provide only part of a keyword-driven test framework, and thus decide
not to implement requirements that are covered by complementary tools (e.g. a vendor only provides an execution engine,
but no keyword driven editor – then the execution engine can still be conforming with the standard).
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document, including any amendments, applies.
ISO/IEC/IEEE 24765, Systems and software engineering – Vocabulary
ISO/IEC/IEEE 29119-1, Software and systems engineering – Software Testing – Part 1: Concepts and
Definitions
ISO/IEC/IEEE 29119-2, Software and systems engineering – Software Testing – Part 2: Test Processes
ISO/IEC/IEEE 29119-3, Software and systems engineering – Software Testing – Part 3: Test Documentation
ISO/IEC/IEEE 29119-4, Software and systems engineering – Software Testing – Part 4: Test Techniques
Other standards useful for the implementation and interpretation of this standard are listed in the bibliography.
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC/IEEE 24765 and the following
apply.
NOTE Use of the terminology of ISO/IEC/IEEE 29119-5 is for ease of reference and is not mandatory for
conformance with ISO/IEC/IEEE 29119-5. The following terms and definitions are provided to assist with the
understanding and readability of ISO/IEC/IEEE 29119-5. Only terms critical to the understanding of ISO/IEC/IEEE 29119-5
are included. This clause is not intended to provide a complete list of testing terms. The Systems and Software
Engineering vocabulary ISO/IEC/IEEE 24765 can be referenced for terms not defined in this standard. ISO/IEC/IEEE
29119-1 can be referenced for terms related to software testing in general. ISO/IEC/IEEE 29119-5 only defines terms
specific to Keyword-Driven Testing.
4.1
domain layer
highest level of abstraction for the test item
Note 1 to entry: Keywords on this level are chosen in a way that is familiar to domain experts.
2 © ISO/IEC 2016 – All rights reserved
© IEEE 2016 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC/IEEE 29119-5:2016(E)
4.2
high-level keyword
keyword that covers complex activities that may be composed from other keywords and is used by domain
experts to assemble keyword test cases

4.3
keyword
one or more words used as a reference to a specific set of actions intended to be performed during the
execution of one or more test cases

Note 1 to entry: The actions include interactions with the User Interface during the test, verification, and specific actions to
set up a test scenario.
Note 2 to entry: Keywords are named using at least one verb.
Note 3 to entry: Composite keywords can be constructed based on other keywords.
4.4
keyword dictionary
keyword library
repository containing a set of keywords reflecting the language and level of abstraction used to write test
cases

4.5
Keyword-Driven Testing
testing using test cases composed from keywords

4.6
Keyword-Driven Testing framework
test framework covering the functional capabilities of a keyword-driven editor, decomposer, data sequencer,
manual test assistant, tool bridge, data and script repositories, a keyword library and the test execution
environment

4.7
keyword execution code
implementation of a keyword that is intended to be executed by a test execution engine

4.8
keyword test case
sequence of keywords and the required values for their associated parameters (as applicable) that are
composed to describe the actions of a test case
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.