ASTM E3064-16
(Test Method)Standard Test Method for Evaluating the Performance of Optical Tracking Systems that Measure Six Degrees of Freedom (6DOF) Pose
Standard Test Method for Evaluating the Performance of Optical Tracking Systems that Measure Six Degrees of Freedom (6DOF) Pose
SIGNIFICANCE AND USE
5.1 Optical tracking systems are used in a wide range of fields including: video gaming, filming, neuroscience, biomechanics, flight/medical/industrial training, simulation, robotics, and automotive applications.
5.2 This standard provides a common set of metrics and a test procedure for evaluating the performance of optical tracking systems and may help to drive improvements and innovations of optical tracking systems.
5.3 Potential users often have difficulty comparing optical tracking systems because of the lack of standard performance metrics and test methods, and therefore must rely on the claims of a vendor regarding the system’s performance, capabilities, and suitability for a particular application. This standard makes it possible for a user to assess and compare the performance of candidate optical tracking systems, and allows the user to determine if the measured performance results are within the specifications with regard to the application requirements.
SCOPE
1.1 Purpose—This test method presents metrics and procedures for measuring, analyzing, and reporting the relative pose error of optical tracking systems that compute the pose (that is, position and orientation) of a rigid object while the object is moving.
1.2 Usage—System vendors may use this test method to determine the performance of their Six Degrees of Freedom (6 DOF) optical tracking system which measures pose. This test method also provides a uniform way to report the measurement errors and measurement capability of the system. System users may use this test method to verify that the system’s performance is within the user’s specific requirements and within the system’s rated performance.
1.3 Test Location—The procedures defined in this standard shall be performed in a facility in which the environmental conditions are within the optical tracking system’s rated conditions.
1.4 Test Volume—This standard shall be used for testing an optical tracking system working volumes of 3000 mm long by 2000 mm wide by 2000 mm high, 6000 mm long by 4000 mm wide by 2000 mm high, or 12 000 mm long by 8000 mm wide by 2000 mm high.
1.5 Units—The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E3064 − 16
Standard Test Method for
Evaluating the Performance of Optical Tracking Systems
that Measure Six Degrees of Freedom (6DOF) Pose
This standard is issued under the fixed designation E3064; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope mendations issued by the World Trade Organization Technical
Barriers to Trade (TBT) Committee.
1.1 Purpose—This test method presents metrics and proce-
dures for measuring, analyzing, and reporting the relative pose
2. Referenced Documents
errorofopticaltrackingsystemsthatcomputethepose(thatis,
2.1 ASTM Standards:
position and orientation) of a rigid object while the object is
E2919Test Method for Evaluating the Performance of
moving.
Systems that Measure Static, Six Degrees of Freedom
1.2 Usage—System vendors may use this test method to
(6DOF), Pose
determine the performance of their Six Degrees of Freedom (6
E177Practice for Use of the Terms Precision and Bias in
DOF) optical tracking system which measures pose. This test
ASTM Test Methods
methodalsoprovidesauniformwaytoreportthemeasurement
2.2 ASME Standard:
errorsandmeasurementcapabilityofthesystem.Systemusers
B89.4.19Performance Evaluation of Laser-Based Spherical
may use this test method to verify that the system’s perfor-
Coordinate Measurement Systems
manceiswithintheuser’sspecificrequirementsandwithinthe
2.3 ISO/IEC Standards:
system’s rated performance.
ISO/IEC Guide 99:2007 International Vocabulary of
1.3 Test Location—The procedures defined in this standard
Metrology—Basic and General Concepts and Associated
shall be performed in a facility in which the environmental
Terms (VIM: 2007)
conditions are within the optical tracking system’s rated
ISO/IEC Guide 98–3:2008Uncertainty of measurement—
conditions.
Part 3: Guide to the expression of uncertainty in measure-
1.4 Test Volume—This standard shall be used for testing an
ment (GUM: 1995)
optical tracking system working volumes of 3000 mm long by
IEC 60050-300:2001 International Electro technical
2000 mm wide by 2000 mm high, 6000 mm long by 4000 mm
Vocabulary—Electrical and electronic measurements and
wide by 2000 mm high, or 12 000 mm long by 8000 mm wide
measuring instruments
by 2000 mm high.
JCGM 200:2012International Vocabulary of Metrology Ba-
sic and General Concepts and Associated Terms (VIM),
1.5 Units—The values stated in SI units are to be regarded
3rd edition
asstandard.Nootherunitsofmeasurementareincludedinthis
standard.
3. Terminology
1.6 This standard does not purport to address all of the
3.1 Definitions:
safety concerns, if any, associated with its use. It is the
3.1.1 degrees of freedom, DOF, n—any of the minimum
responsibility of the user of this standard to establish appro-
number of translation or rotation components required to
priate safety, health, and environmental practices and deter-
specify completely the pose of a rigid object. E2919
mine the applicability of regulatory limitations prior to use.
3.1.1.1 Discussion—
1.7 This international standard was developed in accor-
dance with internationally recognized principles on standard-
ization established in the Decision on Principles for the
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Development of International Standards, Guides and Recom-
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
1 3
This test method is under the jurisdiction of ASTM Committee E57 on 3D Available from American Society of Mechanical Engineers (ASME), ASME
ImagingSystemsandisthedirectresponsibilityofSubcommitteeE57.50onOptical International Headquarters, Two Park Ave., New York, NY 10016-5990, http://
Tracking Systems. www.asme.org.
Current edition approved June 1, 2016. Published June 2016. DOI: 10.1520/ Available fromAmerican National Standards Institute (ANSI), 25 W. 43rd St.,
E3064-16 4th Floor, New York, NY 10036, http://www.ansi.org.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3064 − 16
(1)In a 3D space, a rigid object can have at most 6DOF,
three translations and three rotations.
(2)The term “degree of freedom” is also used with regard
to statistical testing. It will be clear from the context in which
it is used whether the term relates to a statistical test or the
rotation/translation aspect of the object.
3.1.2 measurement error, error of measurement, and error,
n—measured quantity value minus a reference quantity value.
(JCGM 200:2012)
3.1.3 metrology bar, n—a rod of a known length having
markers (active or passive) attached to both ends and used to
estimate the errors of an optical tracking system.
3.1.4 optical tracking system, n—atrackingsystemthatuses
measurements obtained from camera images.
3.1.5 pose, n—a 6DOF vector whose components represent
FIG. 1 Drawing of the artifact showing critical dimensions of the
the position and orientation of a rigid object with respect to a
bar length and the maximum hemispherical volume inside which
coordinate frame. E2919
the markers can be placed.
3.1.6 precision, n—the closeness of agreement between
independent test results obtained under stipulated conditions.
5. Significance and Use
E177
5.1 Optical tracking systems are used in a wide range of
3.1.7 rated conditions, n—manufacturer-specified limits on
fields including: video gaming, filming, neuroscience,
environmental, utility, and other conditions within which the
biomechanics, flight/medical/industrial training, simulation,
manufacturer’s performance specifications are guaranteed at
robotics, and automotive applications.
the time of installation of the instrument. ASME B89.4.19
5.2 This standard provides a common set of metrics and a
3.1.8 reference system, n—a measurement instrument or
test procedure for evaluating the performance of optical
system used to generate a reference value or quantity. E2919
tracking systems and may help to drive improvements and
innovations of optical tracking systems.
3.1.9 relative pose, n—change of an object’s pose between
two poses measured in the same coordinate frame. E2919
5.3 Potential users often have difficulty comparing optical
tracking systems because of the lack of standard performance
3.1.10 repeatability, n—precision under repeatability
metricsandtestmethods,andthereforemustrelyontheclaims
conditions. E177
of a vendor regarding the system’s performance, capabilities,
3.1.11 repeatability conditions, n—conditions where inde-
andsuitabilityforaparticularapplication.Thisstandardmakes
pendent test results are obtained with the same method on
itpossibleforausertoassessandcomparetheperformanceof
identicaltestitemsinthesamelaboratorybythesameoperator
candidate optical tracking systems, and allows the user to
using the same equipment within short intervals of time. E177
determine if the measured performance results are within the
3.1.12 tracking system, n—a system that is used for mea- specifications with regard to the application requirements.
suring the pose of moving objects and supplies the data as a
6. Apparatus
timely ordered sequence.
6.1 Artifact:
3.1.13 work volume, n—aphysicalspace,orregionwithina
6.1.1 A 300 mm long bar with markers rigidly attached to
physical space, that defines the bounds within which a rigid
each end of the metrology bar shall be used as the 6DOF
object tracking system is acquiring data. E2919
artifact. The bar shall have stiffness and thermal expansion
characteristics such that the deflection is less than or equal to
4. Summary of Test Method
0.01 mm. For example, the metrology bar shown in Fig. 2
4.1 This test method provides a set of statistically based
satisfies these requirements.
performance metrics and a test procedure to quantitatively
6.1.2 Aconstant relative 6DOF pose is formed between the
evaluate the performance of an optical tracking system.
two clusters of markers located at the ends of the metrology
bar. All markers shall be contained within hemispherical
4.2 The measurement errors include the positional and
volumes with a maximum radius of 100 mm from the ends of
orientation error components. Specifically, the test procedure
the bar (see Fig. 1). Examples of metrology bars that can be
measures the relative pose between two marker sets rigidly
used to evaluate optical tracking systems are shown in Fig. 2
attached to the opposing ends of a fixed-length metrology bar
(Ref (1)) .
as shown in Fig. 1. The relative pose is then decomposed into
positional and angular components. Measurement errors are
calculated from the positional and angular components as the
The boldface numbers in parentheses refer to a list of references at the end of
artifact is moved about the work volume. this standard.
E3064 − 16
FIG. 2 Examples of artifacts for evaluating optical tracking systems having a 300 mm long metrology bar (a) with six passive, reflective
markers, within 100 mm radius from the bar end, on each end, and (b) with a reduced pose ambiguity cuboctahedron, Ref (1), within
100 mm radius from the bar end, on each end.
7. Measurement Procedure metrology bar length as shown in Fig. 3 (b). The distance
between the boundary lines and the limits of the work volume
7.1 Introduction:
shall be at most, one-half of the metrology bar length. The
7.1.1 This section describes the basic procedure for deter-
center of the metrology bar shall traverse the X pattern
mining the pose measurement error of an optical tracking
followed by the Y pattern with artifact orientation #1 as shown
system.
inFig.3(c)inacontinuoussmoothmotion.Thistraversalshall
7.2 Pose Measurement:
be repeated two more times, once with artifact orientation #2
7.2.1 The Xand Yaxesarealignedwiththeworkvolumein
and once with artifact orientation #3. The data from all three
the horizontal plane as shown in Fig. 3, and Z is aligned with
traversals shall be combined into a single data set.
the vertical axis. Move the metrology bar throughout the work
7.2.2 The metrology bar paths and orientations shall be
volumealongtworegularpatterns:(Xpattern)parallel,straight
chosen as described in 7.2.1. The height of the centroid of the
line segments back-and-forth along the X axis with the paths
metrology bar shall remain approximately 1000 mm above the
separatedbyatmost,themetrologybarlengthasshowninFig.
3 (a) and (Y pattern) parallel, straight line segments back-and- bottom of the test volume.
forth along the Y axis with the paths separated by at most, the
FIG. 3 The (a) X pattern and (b) Y pattern are combined to make a single path along which the metrology bar is moved throughout the
work volume. (c) Artifact (shown with axes on bar center) orientations with respect to the path: 1) perpendicular to the path segments
in the plane of motion, 2) perpendicular to the path segments and normal to the plane of motion, and 3) in-line with the path segments
in the plane of motion.
NOTE 1—Example artifact shown in (a) and (b) is oriented with respect to the path as in 1) perpendicular to the path segments in the plane of motion.
E3064 − 16
7.2.3 The centroid of the metrology bar shall be moved at a systemundertest.Therelativeposebetweentheleftobjectand
relatively constant walking speed of 1200 6 700 mm/s. rightobjectattheendsofthemetrologybarismeasuredbythe
reference system measurement as:
8. Pose Measurement Error
R T I T
H 5 5 (5)
F G F G
8.1 This section describes methods for computing pose Left Right
0 1 0 1
measurement errors of an optical tracking system (OTS) using
The coordinate frames associated with the right and left ends
the artifact. For each instance of time t, the optical tracking of the bar are rotationally aligned using the reference sys-
tem. Where R=I, and I is the identity matrix.
system measures the pose of an object as:
ˆ ˆ
8.1.2 The following sections describe two methods for
R t T t
~ ! ~ !
OTS Object OTS Object
ˆ
H ~t! 5 (1)
F G
OTS Object
evaluating the optical tracking system. In 8.2, measurements
0 1
ˆ
are taken relative to the measured relative pose of a test
Here, R t isa3×3 matrix describing the orientation
~ !
OTS Object
artifact, which is measured by a more accurate system, and in
ˆ
of the object and T ~t! is a 3-dimensional vector de-
OTS Object
8.3 the measurements are taken relative to the mean value of
scribing the position of the object in the optical tracking sys-
the collected data.
tem coordinate frame. In our artifact, two objects are consid-
ered corresponding to the left and right ends of the
8.2 Error Statistics using a Reference System:
metrology bar. The optical tracking system measures the
8.2.1 Thissectiondescribesthecomputationofsystemerror
poses of the left and right ends, and the corresponding4×4
statistics relative to a precisely characterized test artifact, such
matrices are defined respectively as:
as the one described in Section 6.
ˆ
8.2.2 The relative measured pose H ~t! (see Eq 3)at
ˆ ˆ Left Right
R ~t! T ~t!
ˆ Left Left
H t 5 and
~ ! time t can be compared to the reference system measured pose
OTS Left F G
0 1
H (see Eq 5). Specifically, the positional error at time t
(2)
Left Right
can be calculated as:
ˆ ˆ
R ~t! T ~t!
Right Right
ˆ
H t 5
~ !
F G ˆ
OTS Right
e 5 ǁT t ǁ 2 ǁTǁ (6)
~ !
0 1 p t 2 2
~ !
and the orientation error at time t can be calculated as:
Because the bar is rigid, the relative pose between the left
object and the right object is constant in time (see Fig. 4)
ˆ ˆ
e 5 θ~t! 20 5 θ~t! (7)
o~t!
and can be defined as:
ˆ
where θ~t! is calculated using Eq 4 andǁǁ denotes the
ˆ ˆ ˆ
H t 5 H H
~ !
Left Right OTS Left OTS Right
2-norm of the vector.
ˆ ˆ ˆ ˆ
R ~t! T ~t! R ~t! T ~t!
Left Left Right Right
8.2.3 The statistics on these errors include: the root mean
F G F G
0 1 0 1
square error, the maximum error, and the percentile error. The
ˆ ˆ
root mean square error is calculated as:
R t T t
~ ! ~ !
5 (3)
F G
0 1
N
The angle of rotation is calculated as:
Root Mean Square Error 5 e (8)
Œ
( t
N
t51
ˆ 2 2 2
~= ! The maximum of the errors is defined as:
θ~t! 5 2* asin qˆ ~t!1qˆ ~t!1qˆ ~t! (4)
x y z
T
Here, (qˆ (t),qˆ (t),qˆ (t),qˆ (t)) is the unit quaternion represen-
w x y z
e 5max e ?, e ?,. e (9)
~ !
max ? 1 ? 2 ? N?
ˆ
tation of R~t!, Ref (2), where qˆ ~t
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.