ISO/IEC 18305:2016
(Main)Information technology — Real time locating systems — Test and evaluation of localization and tracking systems
Information technology — Real time locating systems — Test and evaluation of localization and tracking systems
ISO/IEC 18305:2016 identifies appropriate performance metrics and test & evaluation scenarios for localization and tracking systems, and it provides guidance on how best to present and visualize the T&E results. It focuses primarily on indoor environments.
Technologies de l'information - Systèmes de localisation en temps réel - Essais et évaluation des systèmes de localisation et de suivi
General Information
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 18305
First edition
2016-11-01
Information technology — Real time
locating systems — Test and evaluation
of localization and tracking systems
Technologies de l’information - Systèmes de localisation en temps réel
- Essais et évaluation des systèmes de localisation et de suivi
Reference number
©
ISO/IEC 2016
© ISO/IEC 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2016 – All rights reserved
Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 2
5 LTS taxonomy . 3
5.1 Types of location sensors . 3
5.1.1 Unimodal systems . 4
5.1.2 Multimodal systems . 5
5.2 Reliance on pre-existing networking / localization infrastructure . 5
5.2.1 LTSs requiring infrastructure . 5
5.2.2 LTSs capable of infrastructure-less operation . 5
5.2.3 Real-time deployment of nodes facilitating localization . 6
5.2.4 Opportunistic use of infrastructure/environment . 6
5.3 Off-line, building-specific training . 6
5.3.1 LTSs requiring off-line training . 6
5.3.2 LTSs not requiring off-line training . 7
5.4 Ultimate consumer(s) of location information . 7
5.4.1 Introduction . 7
5.4.2 The ELT . 8
5.4.3 The tracking authority . 8
5.4.4 Both the ELT and the tracking authority . 8
6 LTS privacy and security considerations . 8
6.1 Privacy . 8
6.2 Security . 9
7 T&E methodology taxonomy . 9
7.1 System vs. component testing . 9
7.1.1 System testing . 9
7.1.2 Component testing . 9
7.2 Knowledge about LTS inner-workings .10
7.2.1 T&E designed with full knowledge of LTS inner-workings .10
7.2.2 Black-box testing . .10
7.3 Repeatability .10
7.3.1 Repeatable testing .10
7.3.2 Non-repeatable testing .10
7.4 Test site .10
7.4.1 Building-wide testing . .10
7.4.2 Laboratory testing .11
7.5 Ground truth .11
7.5.1 Off-line surveyed test points .11
7.5.2 Reference LTS.11
8 LTS performance metrics .12
8.1 Introduction .12
8.2 Floor detection probability .13
8.3 Zone detection probability .13
8.4 Means of various errors .13
8.5 Covariance matrix of the error vector .14
8.6 Variances of magnitudes of various errors .14
8.7 RMS values of various errors .15
8.8 Absolute mean of the error vector .15
© ISO/IEC 2016 – All rights reserved iii
8.9 Circular Error 95% (CE95) and Circular Error Probable (CEP) .15
8.10 Vertical Error 95% (VE95) and Vertical Error Probable (VEP) .16
8.11 Spherical Error 95% (SE95) and Spherical Error Probable (SEP) .16
8.12 Coverage .16
8.13 Relative accuracy .17
8.14 Latency .17
8.15 Set-up time .18
8.16 Optional performance metrics .18
8.16.1 Location-specific accuracy .18
8.16.2 Availability.19
9 Optional performance metrics for LTS use in mission critical applications .19
9.1 Introduction .19
9.2 Susceptibility .20
9.3 Resilience .20
10 T&E considerations and scenarios .20
10.1 Building types .20
10.1.1 Introduction .20
10.1.2 Wooden structure single-family house .20
10.1.3 Medium-size brick & concrete office building .21
10.1.4 Warehouse/factory .21
10.1.5 High-rise steel structure.21
10.1.6 Subterranean structure .21
10.2 Effects of mobility .21
10.2.1 Introduction .21
10.2.2 Stationary object/person .22
10.2.3 Walking .22
10.2.4 Running .22
10.2.5 Backward walking .22
10.2.6 Sidestepping .22
10.2.7 Crawling.22
10.3 Failure modes and vulnerabilities of location sensors .23
10.4 T&E scenarios .23
11 T&E reporting requirements .30
11.1 Introduction .30
11.2 Test place and date .33
11.3 Environmental conditions .33
11.4 LTS product tested .33
11.5 Equipment used by the LTS .33
11.6 ELTD features .33
11.7 Location data format .34
11.8 Location update rate and system capacity .34
11.9 RF emission and interference issues .34
11.10 Set-up procedure .35
11.11 Building information needed by the LTS .35
11.12 LTS GUIs .36
11.12.1 ELTD GUI .36
11.12.2 Tracking authority GUI .36
11.13 Maintenance .36
11.14 Floor plans of test buildings .37
11.15 Characterization of T&E scenarios involving entities in motion .37
11.16 Presentation of numerical T&E results .38
11.17 Visualization of T&E results .43
Annex A (normative) Conversions between local Cartesian and WGS 84 coordinates .47
Annex B (informative) Location sensors and their failure modes .64
Bibliography .76
iv © ISO/IEC 2016 – All rights reserved
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information.
ISO/IEC 18305 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 31, Automatic identification and data capture techniques.
© ISO/IEC 2016 – All rights reserved v
Introduction
There exists a potentially large market for personnel / asset Localization and Tracking Systems (LTSs)
in diverse application domains such as:
— emergency response;
— military;
— law enforcement;
— mining;
— E-911;
— offender tracking;
— personal vehicular navigation;
— smart phones / social networking;
— fleet management;
— asset tracking in factories / warehouses / hospitals;
— tracking the elderly / children; and
— personal navigation in museums / shopping malls.
Some applications of localization and tracking – such as personal navigation, fleet management, and
asset tracking in factories / warehouses / hospitals – are commonly referred to as Location-Based
Services (LBS). The use of LBS alone is expected to grow dramatically by 2020. Yet, lack of standardized
Test and Evaluation (T&E) procedures has been an impediment to market growth for LTSs, because:
i) potential users cannot easily determine whether these systems meet the users’ requirements;
ii) it is hard to interpret T&E results when different metrics and procedures are used to evaluate a
given system or even worse to evaluate different systems; and
iii) the use of disparate minimum performance requirements by various buyers / jurisdictions
forces manufacturers to develop jurisdiction-specific products, thereby raising manufacturing costs.
In contrast with LBS, there are many applications of localization and tracking that are essentially
governmental functions in the sense that the government is the entity that is most concerned about
the effectiveness of solutions for such applications. Examples of these applications include tracking
firefighters entering a burning structure for command and control purposes and to launch a rescue
mission if a firefighter becomes incapacitated, prevention of friendly fire when soldiers or Special
Weapons And Tactics (SWAT) team members enter a building where either hostile forces or armed
individuals threatening public safety have taken refuge, and guidance and navigation for missiles and
precision-guided munitions. Many of these applications have more stringent localization accuracy and
latency requirements than other applications of localization and tracking used by the general public,
such as navigation in museums / shopping malls, tracking the elderly in nursing homes, ensuring
children are not abducted from school grounds, and fleet management for a trucking company.
This document deals with T&E of LTSs. Once standardized T&E procedures have been established,
it is possible to set minimum performance requirements for various applications of localization and
tracking. For example, regulations promulgated by a government agency may require coal mine
operators to have the capability to track the miners on duty within 5 m accuracy during normal mine
operations and 100 m accuracy in the aftermath of a catastrophic incident in the mine, such as an
explosion or a roof collapse. It makes sense to separate the T&E issue from minimum performance
requirements, because the same T&E standard may be applicable to many applications of localization
and tracking, but the minimum performance requirements typically vary from one application to
vi © ISO/IEC 2016 – All rights reserved
another. This document deals with T&E only; it does not set minimum performance requirements for
any localization and tracking applications.
T&E of LTSs is challenging for several reasons:
i) Many systems work in a “networked” fashion. That is, several devices would have to
communicate with each other in order to estimate the location(s) of one or more such devices. Therefore,
the LTS performance is affected by how these devices are situated with respect to each other, i.e. by the
network topology.
ii) The physical environment in which the devices are situated affects communications between
them and functionalities such as ranging or estimating direction of another device and hence LTS
performance. For example, Radio Frequency (RF) communications in a single-family house with a
wooden structure is very different from that in a large high-rise building with a steel and concrete
structure.
iii) Even though it is best to take a “black-box” approach to LTS T&E, one needs to be cognizant of
the failure modes of various location sensors (such as Global Positioning System (GPS), RF ranging, RF
direction of arrival estimation, accelerometer, gyroscope, and altimeter) that “might” be used in an LTS
in order to design a comprehensive T&E procedure.
Yet another difficulty of a different nature is that some systems rely on the availability of a networking
infrastructure, such as a Wi-Fi network, or other devices, such as Radio Frequency IDentification
(RFID) or Real Time Locating System (RTLS) tags, to facilitate localization and tracking in a building or
structure. Some allow deployment of such devices – sometimes called “breadcrumbs” – as users enter a
building. Other systems are designed to function based on the assumption that they cannot get any help
with localization and tracking from the building and breadcrumb deployment is not allowed. Therefore,
the T&E procedure has to account for these possibilities or classes of LTSs.
The main purpose of this document is to develop performance metrics and T&E scenarios for LTSs. LBS
are envisaged in many application domains in both governmental operations and general public usage
scenarios. Therefore, industry, consumers, trade, governments, and distributors are all affected by this
document. Every effort has been made to write this document in such a way that it would be applicable
to as many applications of indoor localization and tracking as possible. This document provides explicit
instructions on how to report the T&E results, i.e. what information to document and what kind of tables
and figures/plots to include to best visualize the results of the T&E effort. LTS T&E is complicated even
once this document has been published, because there has to be a “network deployment” and testing
in at least a few types of buildings. One should not expect that LTS T&E can be done in a laboratory.
Performance results can depend on the particular building(s) used in the T&E procedure, but at least
there will be a standardized way of doing the T&E, and if multiple LTSs are evaluated according to
the standard in the same set of buildings, then the performance results can be compared. Localization
and tracking technology has not yet matured. New systems and approaches will be developed in the
next several years, but the T&E procedure can be standardized regardless of what takes place on the
technology front and it may in fact foster technology development. In the absence of a T&E standard, the
present uncertainties in the LTS market, where it is hard for users to ascertain whether LTS products
meet their requirements and LTS vendor claims are hard to verify, will continue. Therefore, this is
indeed the right time for development of this document.
Extensions of this standard to other application domains, such as miners trapped in an underground
mine, navigation for submersible vehicles or tiny medical devices moving around inside a human body,
may be the subjects of future standards that will be extensions of this “base” standard.
As a final note, the term “localization and tracking” has been used to denote the types of systems
this document is meant to be applied to. However, this is not the only term in use for referring to
such systems. ISO/IEC JTC 1/SC 31 uses the term RTLS, which also appears in the full name for this
document. SC 31, in its deliberations, considered the use of the term “positioning” for the situations in
which a person/object equipped with an appropriate device, uses that device possibly in conjunction
with others and as part of a system to determine its own location. That is, “positioning” is for self-
awareness. On the other hand, SC 31 regards “locating” as the appropriate term for the situation in
which some other entity needs to determine the location of a person/object remotely. In other words,
© ISO/IEC 2016 – All rights reserved vii
“locating” is for tracking and accountability purposes. There is also the possibility that a system needs
to provide both ”positioning” and “locating” functionalities (see 5.4.4), using the terminology just
defined. “Tracking” is another frequently used term that has a time dimension to it. That is, one needs
to keep track of a person/object’s movements over a period of time. In its simplest form, tracking can be
done by invoking a locating capability periodically over the time interval of interest. However, tracking
can also take into account the mobility characteristics of the person/object being tracked. For example,
it is highly unlikely that a firefighter would move faster than 1 m/s while putting the fire down in a
burning building, and this information can be used to do a better job of estimating the firefighter’s
location at any given time. “Location System” is another term used in the literature. Yet another term,
often encountered in military applications, is “navigation”. In order to navigate a person/object to some
destination point, it is necessary to know the person/object’s starting location at a minimum. In case
of navigating a missile or smart bomb, where missing the target or hitting something else can have
catastrophic consequences, it is necessary to know the missile’s/bomb’s location continuously so that
any deviations from its intended path/course can be corrected. Navigation includes computing a path
to the destination. This path is not always the direct line from the starting location to the destination.
For example, consider navigation in city streets or for providing guidance to a disoriented firefighter
to get out of a burning building. Even though this document does not deal with navigation, it does deal
with that component of navigation that has to do with where a person/object is at a given time.
This document adopted the term “localization” to capture both locating and positioning functionalities,
because the person/object has to be “localized” in either case. It also adopted the term “tracking” to ensure
the standard is not just about a snapshot of person/object’s location, but also addresses its evolution over
time. As a matter of fact, SC 31 has so far focused on purely RF–based systems, but this document considers
systems that may use a variety of sensors for localization and tracking, including Inertial Measurement
Units (IMUs), whose performance is indeed affected by how the person/object is moving.
viii © ISO/IEC 2016 – All rights reserved
INTERNATIONAL STANDARD ISO/IEC 18305:2016(E)
Information technology — Real time locating systems —
Test and evaluation of localization and tracking systems
IMPORTANT — The electronic file of this document contains colours which are considered to be
useful for the correct understanding of the document. Users should therefore consider printing
this document using a colour printer.
1 Scope
This document identifies appropriate performance metrics and test & evaluation scenarios for
localization and tracking systems, and it provides guidance on how best to present and visualize the
T&E results. It focuses primarily on indoor environments.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http://www.electropedia.org/
— ISO Online browsing platform: available at http://www.iso.org/obp
3.1
entity to be localized/tracked
person / autonomous robot that needs to know its location for context-awareness / navigation purposes
or person/object whose location is needed by a tracking authority at a given time instance or over a
time interval
Note 1 to entry: See the abbreviation ELT in Clause 4.
3.2
location sensor
device that measures a physical quantity to facilitate estimating the spatial coordinates of a
person/object in a reference coordinate system
3.3
entity localization/tracking device
equipment carried by a person or affixed to an object comprising one or more location sensors that
facilitates estimating the location of the person/object at a given time instance or over a time interval
Note 1 to entry: to entry: See the abbreviation ELTD in Clause 4.
© ISO/IEC 2016 – All rights reserved 1
4 Abbreviated terms
AOA angle-of-arrival
AP access point
CCD charged coupled device
CDF cumulative distribution function
CE95 circular error 95%
CEP circular error probable
ELT entity to be localized/tracked
ELTD entity localization/tracking device
EMI electromagnetic interference
ERP effective radiated power
GDOP geometric dilution of precision
GNSS global navigation satellite system
GPS global positioning system
IMU inertial measurement unit
INS inertial navigation system
ISM industrial, scientific, and medical
IT information technology
LBS location-based services
LOS line-of-sight
LTS localization and tracking system
MEMS microelectromechanical systems
PDOA phase difference of arrival
PII personally identifiable information
RF radio frequency
RFID radio frequency identification
RMS root mean square
RSS received signal strength
RSSI received signal strength indicator
RTLS real time locating system
SE95 spherical error 95%
2 © ISO/IEC 2016 – All rights reserved
SEP spherical error probable
SLAM self-localization and mapping
T&E test and evaluation
TDOA time difference of arrival
TOA time-of-arrival
TOF time-of-flight
UTM universal transverse Mercator
UV ultraviolet
UWB ultra wideband
VE95 vertical error 95%
VEP vertical error probable
WGS 84 world geodetic system 84
5 LTS taxonomy
5.1 Types of location sensors
GPS has been the dominant technology for outdoor navigation and for tracking entities such as a fleet of
taxicabs or trucks. Inertial Navigation Systems (INS) have been used for navigation purposes for a long
time. These trends preceded the recent flurry of activities in indoor localization and tracking, which
have focused primarily on RF-based methods. Two approaches have played key roles in fuelling the
recent drive in development of LTSs. One is based on processing the signals received by a mobile device
/ smartphone from the base stations of a cellular telephony system. This approach works both indoors
and outdoors, but its localization accuracy is not adequate for many applications. The other is based on
the strength of signals received from Wi-Fi Access Points (APs) that are widely deployed in buildings
/ structures throughout the world. Once again, the localization accuracy of this approach may not be
adequate in certain applications, and not all buildings have Wi-Fi APs.
These efforts were followed by exploring other RF-based methods, particularly for indoor environments
where GPS receivers do not work due to the lack of Line-of-Sight (LOS) RF propagation paths to at least
four GPS satellites. Since about ten years ago, researchers have significantly increased their efforts
to develop various RF techniques for localization. Angle-Of-Arrival (AOA) estimation, even though
it has been around for a long time, has been explored for indoor localization. Time-Of-Arrival (TOA)
estimation has also been around, but it has been the subject of renewed interest due to the advent of
Ultra WideBand (UWB) communications and ranging techniques. Widely used RF technologies such
as Bluetooth, ZigBee, and RFID have been explored for indoor localization and tracking. Each of these
technologies and approaches has its own pros and cons. Over time, it has become abundantly clear
that purely RF-based approaches may not provide the desired localization accuracy or may not meet
all the operational requirements of a particular application. For example, firefighters responding to
a fire cannot assume that Wi-Fi APs or RFID tags/readers are available in the building that could be
used for localization purposes. Therefore, there has been considerable effort lately to look at the use
of other sensors for localization and tracking. Of particular interest and promise are hybrid LTSs that
fuse the data from a number of location sensors to produce accurate location estimates. In this regard,
one can design an LTS that employs a fixed set of location sensors or one that is sufficiently flexible to
take advantage of whatever location sensors that might be available at any given time. For example, as
a mobile platform such as a ground vehicle moves around, it may be able to use the signals from a radio
station or TV tower together with the location of the transmitting antennae from the radio/TV stations
for localization purposes. Such signals are called signals of opportunity.
© ISO/IEC 2016 – All rights reserved 3
Given below is a non-exhaustive list of location sensors:
— RF-based location sensors;
— Received Signal Strength (RSS);
— proximity, including RFID;
— TOA;
— Time-Difference-Of-Arrival (TDOA);
— AOA;
— signals of opportunity;
— range / pseudo-range finder;
— GPS / Global Navigation Satellite System (GNSS);
— differential GNSS;
— accelerometer;
— gyroscope;
— magnetometer;
— IMU;
— pedometer;
— inclinometer;
— altimeter;
— acoustic sensor;
— imager;
— optical;
— infrared; and
— lidar.
More is said about these sensors and their failure modes in Annex B.
5.1.1 Unimodal systems
Some LTSs use only one type of sensor for localization and tracking purposes. An example of such a
system is the widely used Wi-Fi localization system. Such a system uses the Received Signal Strength
Indicator (RSSI) available on Wi-Fi receivers to estimate location. Specifically, the Entity to be
Localized/Tracked (ELT), as a Wi-Fi client, uses RSSI measurements from various APs in the building to
estimate its own location. Alternatively, the APs can collaborate with each other and estimate the ELT
location based on the strengths of the signals they receive from it. Another example would be an LTS
that uses RFID technology only. In one variation of such a system, called Reverse RFID, passive RFID
tags are deployed in the building and the ELT is equipped with an RFID reader that reads all RFID tags
in its vicinity. This information enables the ELT to estimate its own location.
4 © ISO/IEC 2016 – All rights reserved
5.1.2 Multimodal systems
These are systems that use more than one type of location sensor. Such systems are also called hybrid
systems. They use data fusion methods to combine various sensor measurements to arrive at a location
estimate. The fusion process can take place on the ELT or at a designated node in the LTS. There
are situations where no unimodal LTS would meet the requirements of a particular application. For
example, when firefighters respond to a fire in a building, they cannot assume the building has any
infrastructure (Wi-Fi APs, RFID, or other wireless technology) that could help with localization and
tracking. If the firefighters and the incident command wish to have localization and tracking capability,
it would have to be provided by the equipment they bring to the scene. If the building poses challenges to
RF propagation, which would be the case for large buildings made of heavy construction materials like
steel and concrete, then no RF-based method brought to the scene can provide the desired localization
and tracking capability. (Note that firefighters are not fond of a breadcrumb solution either, because
breadcrumbs may be destroyed by fire and are hard to retrieve even if they survive.) This is an example
of a situation where the operational requirements of the application dictate the use of a multimodal LTS
that could use GPS for outdoor tracking and inertial navigation and some form of RF ranging – even if it
is not available all the time – for indoor localization and tracking.
Equipment cost might be another reason for using a multimodal LTS. There are cases where a
multimodal LTS outperforms any unimodal LTS, for a given total system cost. The design of multimodal
or hybrid LTSs is an active area of research and development.
5.2 Reliance on pre-existing networking / localization infrastructure
5.2.1 LTSs requiring infrastructure
A Wi-Fi localization system is an example of such a system, because it requires availability of Wi-Fi APs
in the building.
Another example is LTSs that use RFID technology. There are two ways of using RFID for localization,
the so-called direct way and Reverse RFID. The latter has already been described in 5.1.1. In a Direct
RFID system, RFID readers are deployed throughout the building and the ELTs are equipped with
RFID tags. Once a reader reads a tag, the system knows the tag is in its vicinity. If multiple readers can
read/”see” a tag, then some weighted average of the reader locations would be a reasonable estimate
of th
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...