Information technology -- Real time locating systems -- Test and evaluation of localization and tracking systems

ISO/IEC 18305:2016 identifies appropriate performance metrics and test & evaluation scenarios for localization and tracking systems, and it provides guidance on how best to present and visualize the T&E results. It focuses primarily on indoor environments.

Technologies de l'information - Systèmes de localisation en temps réel - Essais et évaluation des systèmes de localisation et de suivi

General Information

Status
Published
Publication Date
31-Oct-2016
Current Stage
9020 - International Standard under periodical review
Start Date
15-Oct-2021
Ref Project

Buy Standard

Standard
ISO/IEC 18305:2016 - Information technology -- Real time locating systems -- Test and evaluation of localization and tracking systems
English language
76 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 18305
First edition
2016-11-01
Information technology — Real time
locating systems — Test and evaluation
of localization and tracking systems
Technologies de l’information - Systèmes de localisation en temps réel
- Essais et évaluation des systèmes de localisation et de suivi
Reference number
ISO/IEC 18305:2016(E)
ISO/IEC 2016
---------------------- Page: 1 ----------------------
ISO/IEC 18305:2016(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2016, Published in Switzerland

All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form

or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior

written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of

the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2016 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 18305:2016(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms .............................................................................................................................................................................................. 2

5 LTS taxonomy ........................................................................................................................................................................................................... 3

5.1 Types of location sensors ............................................................................................................................................................... 3

5.1.1 Unimodal systems ........................................................................................................................................................... 4

5.1.2 Multimodal systems ...................................................................................................................................................... 5

5.2 Reliance on pre-existing networking / localization infrastructure ........................................................... 5

5.2.1 LTSs requiring infrastructure ............................................................................................................................... 5

5.2.2 LTSs capable of infrastructure-less operation ........................................................................................ 5

5.2.3 Real-time deployment of nodes facilitating localization ............................................................... 6

5.2.4 Opportunistic use of infrastructure/environment ............................................................................. 6

5.3 Off-line, building-specific training ......................................................................................................................................... 6

5.3.1 LTSs requiring off-line training ........................................................................................................................... 6

5.3.2 LTSs not requiring off-line training ................................................................................................................. 7

5.4 Ultimate consumer(s) of location information ........................................................................................................... 7

5.4.1 Introduction ......................................................................................................................................................................... 7

5.4.2 The ELT ..................................................................................................................................................................................... 8

5.4.3 The tracking authority ................................................................................................................................................ 8

5.4.4 Both the ELT and the tracking authority ..................................................................................................... 8

6 LTS privacy and security considerations ..................................................................................................................................... 8

6.1 Privacy ............................................................................................................................................................................................................ 8

6.2 Security .......................................................................................................................................................................................................... 9

7 T&E methodology taxonomy .................................................................................................................................................................... 9

7.1 System vs. component testing .................................................................................................................................................... 9

7.1.1 System testing .................................................................................................................................................................... 9

7.1.2 Component testing ......................................................................................................................................................... 9

7.2 Knowledge about LTS inner-workings ............................................................................................................................10

7.2.1 T&E designed with full knowledge of LTS inner-workings ......................................................10

7.2.2 Black-box testing ...................................................................... .....................................................................................10

7.3 Repeatability ..........................................................................................................................................................................................10

7.3.1 Repeatable testing .......................................................................................................................................................10

7.3.2 Non-repeatable testing ............................................................................................................................................10

7.4 Test site .......................................................................................................................................................................................................10

7.4.1 Building-wide testing ........................................................................................................................................... .....10

7.4.2 Laboratory testing .......................................................................................................................................................11

7.5 Ground truth ..........................................................................................................................................................................................11

7.5.1 Off-line surveyed test points ...............................................................................................................................11

7.5.2 Reference LTS...................................................................................................................................................................11

8 LTS performance metrics ..........................................................................................................................................................................12

8.1 Introduction ...........................................................................................................................................................................................12

8.2 Floor detection probability .......................................................................................................................................................13

8.3 Zone detection probability ........................................................................................................................................................13

8.4 Means of various errors ...............................................................................................................................................................13

8.5 Covariance matrix of the error vector ..............................................................................................................................14

8.6 Variances of magnitudes of various errors ..................................................................................................................14

8.7 RMS values of various errors ...................................................................................................................................................15

8.8 Absolute mean of the error vector ......................................................................................................................................15

© ISO/IEC 2016 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 18305:2016(E)

8.9 Circular Error 95% (CE95) and Circular Error Probable (CEP) ................................................................15

8.10 Vertical Error 95% (VE95) and Vertical Error Probable (VEP) .................................................................16

8.11 Spherical Error 95% (SE95) and Spherical Error Probable (SEP) ..........................................................16

8.12 Coverage ....................................................................................................................................................................................................16

8.13 Relative accuracy ................................................................................................................................................................................17

8.14 Latency ........................................................................................................................................................................................................17

8.15 Set-up time ..............................................................................................................................................................................................18

8.16 Optional performance metrics ...............................................................................................................................................18

8.16.1 Location-specific accuracy ...................................................................................................................................18

8.16.2 Availability..........................................................................................................................................................................19

9 Optional performance metrics for LTS use in mission critical applications .......................................19

9.1 Introduction ...........................................................................................................................................................................................19

9.2 Susceptibility .........................................................................................................................................................................................20

9.3 Resilience ..................................................................................................................................................................................................20

10 T&E considerations and scenarios .................................................................................................................................................20

10.1 Building types .......................................................................................................................................................................................20

10.1.1 Introduction ......................................................................................................................................................................20

10.1.2 Wooden structure single-family house ......................................................................................................20

10.1.3 Medium-size brick & concrete office building .....................................................................................21

10.1.4 Warehouse/factory .....................................................................................................................................................21

10.1.5 High-rise steel structure.........................................................................................................................................21

10.1.6 Subterranean structure ...........................................................................................................................................21

10.2 Effects of mobility ..............................................................................................................................................................................21

10.2.1 Introduction ......................................................................................................................................................................21

10.2.2 Stationary object/person ......................................................................................................................................22

10.2.3 Walking .................................................................................................................................................................................22

10.2.4 Running .................................................................................................................................................................................22

10.2.5 Backward walking .......................................................................................................................................................22

10.2.6 Sidestepping .....................................................................................................................................................................22

10.2.7 Crawling................................................................................................................................................................................22

10.3 Failure modes and vulnerabilities of location sensors ......................................................................................23

10.4 T&E scenarios .......................................................................................................................................................................................23

11 T&E reporting requirements ................................................................................................................................................................30

11.1 Introduction ...........................................................................................................................................................................................30

11.2 Test place and date ...........................................................................................................................................................................33

11.3 Environmental conditions ..........................................................................................................................................................33

11.4 LTS product tested ............................................................................................................................................................................33

11.5 Equipment used by the LTS .......................................................................................................................................................33

11.6 ELTD features ........................................................................................................................................................................................33

11.7 Location data format .......................................................................................................................................................................34

11.8 Location update rate and system capacity ...................................................................................................................34

11.9 RF emission and interference issues .................................................................................................................................34

11.10 Set-up procedure ................................................................................................................................................................................35

11.11 Building information needed by the LTS .......................................................................................................................35

11.12 LTS GUIs .....................................................................................................................................................................................................36

11.12.1 ELTD GUI ..............................................................................................................................................................................36

11.12.2 Tracking authority GUI ............................................................................................................................................36

11.13 Maintenance ...........................................................................................................................................................................................36

11.14 Floor plans of test buildings .....................................................................................................................................................37

11.15 Characterization of T&E scenarios involving entities in motion ..............................................................37

11.16 Presentation of numerical T&E results ...........................................................................................................................38

11.17 Visualization of T&E results .....................................................................................................................................................43

Annex A (normative) Conversions between local Cartesian and WGS 84 coordinates .................................47

Annex B (informative) Location sensors and their failure modes ......................................................................................64

Bibliography .............................................................................................................................................................................................................................76

iv © ISO/IEC 2016 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 18305:2016(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work. In the field of information technology, ISO and IEC have established a joint technical committee,

ISO/IEC JTC 1.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the meaning of ISO specific terms and expressions related to conformity

assessment, as well as information about ISO’s adherence to the WTO principles in the Technical

Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information.

ISO/IEC 18305 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 31, Automatic identification and data capture techniques.
© ISO/IEC 2016 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 18305:2016(E)
Introduction

There exists a potentially large market for personnel / asset Localization and Tracking Systems (LTSs)

in diverse application domains such as:
— emergency response;
— military;
— law enforcement;
— mining;
— E-911;
— offender tracking;
— personal vehicular navigation;
— smart phones / social networking;
— fleet management;
— asset tracking in factories / warehouses / hospitals;
— tracking the elderly / children; and
— personal navigation in museums / shopping malls.

Some applications of localization and tracking – such as personal navigation, fleet management, and

asset tracking in factories / warehouses / hospitals – are commonly referred to as Location-Based

Services (LBS). The use of LBS alone is expected to grow dramatically by 2020. Yet, lack of standardized

Test and Evaluation (T&E) procedures has been an impediment to market growth for LTSs, because:

i) potential users cannot easily determine whether these systems meet the users’ requirements;

ii) it is hard to interpret T&E results when different metrics and procedures are used to evaluate a

given system or even worse to evaluate different systems; and

iii) the use of disparate minimum performance requirements by various buyers / jurisdictions

forces manufacturers to develop jurisdiction-specific products, thereby raising manufacturing costs.

In contrast with LBS, there are many applications of localization and tracking that are essentially

governmental functions in the sense that the government is the entity that is most concerned about

the effectiveness of solutions for such applications. Examples of these applications include tracking

firefighters entering a burning structure for command and control purposes and to launch a rescue

mission if a firefighter becomes incapacitated, prevention of friendly fire when soldiers or Special

Weapons And Tactics (SWAT) team members enter a building where either hostile forces or armed

individuals threatening public safety have taken refuge, and guidance and navigation for missiles and

precision-guided munitions. Many of these applications have more stringent localization accuracy and

latency requirements than other applications of localization and tracking used by the general public,

such as navigation in museums / shopping malls, tracking the elderly in nursing homes, ensuring

children are not abducted from school grounds, and fleet management for a trucking company.

This document deals with T&E of LTSs. Once standardized T&E procedures have been established,

it is possible to set minimum performance requirements for various applications of localization and

tracking. For example, regulations promulgated by a government agency may require coal mine

operators to have the capability to track the miners on duty within 5 m accuracy during normal mine

operations and 100 m accuracy in the aftermath of a catastrophic incident in the mine, such as an

explosion or a roof collapse. It makes sense to separate the T&E issue from minimum performance

requirements, because the same T&E standard may be applicable to many applications of localization

and tracking, but the minimum performance requirements typically vary from one application to

vi © ISO/IEC 2016 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 18305:2016(E)

another. This document deals with T&E only; it does not set minimum performance requirements for

any localization and tracking applications.
T&E of LTSs is challenging for several reasons:

i) Many systems work in a “networked” fashion. That is, several devices would have to

communicate with each other in order to estimate the location(s) of one or more such devices. Therefore,

the LTS performance is affected by how these devices are situated with respect to each other, i.e. by the

network topology.

ii) The physical environment in which the devices are situated affects communications between

them and functionalities such as ranging or estimating direction of another device and hence LTS

performance. For example, Radio Frequency (RF) communications in a single-family house with a

wooden structure is very different from that in a large high-rise building with a steel and concrete

structure.

iii) Even though it is best to take a “black-box” approach to LTS T&E, one needs to be cognizant of

the failure modes of various location sensors (such as Global Positioning System (GPS), RF ranging, RF

direction of arrival estimation, accelerometer, gyroscope, and altimeter) that “might” be used in an LTS

in order to design a comprehensive T&E procedure.

Yet another difficulty of a different nature is that some systems rely on the availability of a networking

infrastructure, such as a Wi-Fi network, or other devices, such as Radio Frequency IDentification

(RFID) or Real Time Locating System (RTLS) tags, to facilitate localization and tracking in a building or

structure. Some allow deployment of such devices – sometimes called “breadcrumbs” – as users enter a

building. Other systems are designed to function based on the assumption that they cannot get any help

with localization and tracking from the building and breadcrumb deployment is not allowed. Therefore,

the T&E procedure has to account for these possibilities or classes of LTSs.

The main purpose of this document is to develop performance metrics and T&E scenarios for LTSs. LBS

are envisaged in many application domains in both governmental operations and general public usage

scenarios. Therefore, industry, consumers, trade, governments, and distributors are all affected by this

document. Every effort has been made to write this document in such a way that it would be applicable

to as many applications of indoor localization and tracking as possible. This document provides explicit

instructions on how to report the T&E results, i.e. what information to document and what kind of tables

and figures/plots to include to best visualize the results of the T&E effort. LTS T&E is complicated even

once this document has been published, because there has to be a “network deployment” and testing

in at least a few types of buildings. One should not expect that LTS T&E can be done in a laboratory.

Performance results can depend on the particular building(s) used in the T&E procedure, but at least

there will be a standardized way of doing the T&E, and if multiple LTSs are evaluated according to

the standard in the same set of buildings, then the performance results can be compared. Localization

and tracking technology has not yet matured. New systems and approaches will be developed in the

next several years, but the T&E procedure can be standardized regardless of what takes place on the

technology front and it may in fact foster technology development. In the absence of a T&E standard, the

present uncertainties in the LTS market, where it is hard for users to asce
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.