ASTM E2919-22

Standard Test Method for Evaluating the Performance of Systems that Measure Static, Six Degrees of Freedom (6DOF), Pose

SIGNIFICANCE AND USE
5.1 Pose measurement systems are used in a wide range of fields including manufacturing, material handling, construction, medicine, and aerospace. The use of pose measurement systems could, for example, replace the need to fix the poses of objects of interest by mechanical means.  
5.2 Potential users have difficulty comparing pose measurement systems because of the lack of standard performance specifications and test methods, and must rely on the specifications of a vendor regarding the system’s performance, capabilities, and suitability for a particular application. This standard makes it possible for a user to assess and compare the performance of candidate pose measurement systems, and allows the user to determine if the measured performance results are within the vendor’s claimed specifications with regard to the user’s application. This standard also facilitates the improvement of pose measurement systems by providing a common set of metrics to evaluate system performance.  
5.3 The intent of this test method is to allow a user to determine the performance of a vendor’s system under conditions specific to the user’s application, and to determine whether the system still performs in accordance with the vendor’s specifications under those conditions. The intention of this test method is not to validate a vendor’s claims; although, under specific situations, this test method may be adapted for this purpose.
SCOPE
1.1 Purpose—In this test method, metrics and procedures for collecting and analyzing data to determine the performance of a pose measurement system in computing the pose (position and orientation) of a rigid object are provided.  
1.2 This test method applies to the situation in which both the object and the pose measurement system are static with respect to each other when measurements are performed. Vendors may use this test method to establish the performance limits for their six degrees of freedom (6DOF) pose measurement systems. The vendor may use the procedures described in 9.2 to generate the test statistics, then apply an appropriate margin or scaling factor as desired to generate the performance specifications. This test method also provides a uniform way to report the relative or absolute pose measurement capability of the system, or both, making it possible to compare the performance of different systems.  
1.3 Test Location—The methodology defined in this test method shall be performed in a facility in which the environmental conditions are within the pose measurement system’s rated conditions and meet the user’s requirements.  
1.4 Units—The values stated in SI units are to be regarded as the standard. No other units of measurement are included in this standard.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 14-Dec-2022
• ICS: 17.040.99 - Other standards related to linear and angular measurements
• Technical Committee: E57 - 3D Imaging Systems
• Drafting Committee: E57.23 - Industrial 3D Machine Vision Systems

Relations

• Refers: ASTM E2544-24 - Standard Terminology for Three-Dimensional (3D) Imaging Systems
• Effective Date: 01-Jan-2024
• Refers: ASTM E456-13a(2022)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022
• Refers: ASTM E456-13A(2017)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E456-13A(2017)e3 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E456-13ae3 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13ae1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13ae2 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13a - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Aug-2013
• Refers: ASTM E456-12e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-May-2012
• Refers: ASTM E456-12 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-May-2012
• Refers: ASTM E2544-11a - Standard Terminology for Three-Dimensional (3D) Imaging Systems
• Effective Date: 15-May-2011
• Refers: ASTM E2544-11 - Standard Terminology for Three-Dimensional (3D) Imaging Systems
• Effective Date: 01-Apr-2011
• Refers: ASTM E2544-10 - Standard Terminology for Three-Dimensional (3D) Imaging Systems
• Effective Date: 01-Jul-2010
• Refers: ASTM E2544-09b - Standard Terminology for Three-Dimensional (3D) Imaging Systems
• Effective Date: 01-Jul-2009

Overview

ASTM E2919-22: Standard Test Method for Evaluating the Performance of Systems that Measure Static, Six Degrees of Freedom (6DOF), Pose provides a reliable, repeatable method for assessing the performance of static pose measurement systems. These systems, capable of measuring the position and orientation (pose) of rigid objects across six degrees of freedom, are widely used in industries such as manufacturing, medicine, material handling, construction, and aerospace.

This ASTM standard defines key metrics and procedures for collecting and analyzing data to quantitatively assess a pose measurement system’s ability to measure static objects. By offering a common benchmark, the standard enables users to compare the performance of different 6DOF pose measurement systems and verify claims made by system vendors within application-specific environments.

Key Topics

• Performance Evaluation: Provides specific test methods, including statistical procedures, to determine absolute and relative error in both position and orientation.
• Absolute and Relative Pose Measurement: Assesses both the static pose (position and orientation) and changes in pose between two positions.
• Sampling Requirements: Specifies sample sizes and randomization within the work volume to ensure robust performance assessments.
• Reference Systems: Outlines the need for highly accurate reference systems (such as laser trackers or coordinate measurement machines) for benchmarking.
• Reporting and Documentation: Requires standardized reporting templates that document all key findings, test conditions, and statistical analyses.
• Environmental Conditions: Ensures that testing is conducted under conditions suitable for both the pose measurement system and intended application.
• Statistical Validation: Employs statistical tests (e.g., Z-test, quantile error, maximum permissible error, and precision error tests) to objectively determine conformity with vendor specifications.

Applications

The ASTM E2919-22 standard is crucial for a variety of 6DOF pose measurement system users and vendors by offering:

• Product Benchmarking: Provides a standardized test method for comparing new and existing pose measurement systems in the market, assisting procurement decisions.
• System Verification: Enables end users to objectively verify that a system’s performance aligns with application-specific requirements instead of relying solely on vendor specifications.
• Performance Optimization: Offers manufacturers and developers metrics and methods to evaluate, improve, and communicate system capabilities in static measurement scenarios.
• Industrial Implementation: Essential for fields such as:
  • Robotics and Automation: Accurately verifying robotic arm positioning or spatial orientation.
  • Aerospace and Aviation: Ensuring precise alignment of components and assemblies.
  • Medical Navigation: Supporting surgical navigation tools and precise instrument positioning.
  • Construction and Surveying: Facilitating the accurate placement and alignment of structures or materials.
  • Quality Assurance: Enhancing inspection, calibration, and metrology processes by quantifying system biases and uncertainties.

Related Standards

ASTM E2919-22 references and aligns with other important standards to support consistent terminology, calibration, and uncertainty analysis:

• ASTM E456: Terminology Relating to Quality and Statistics.
• ASTM E2544: Terminology for Three-Dimensional (3D) Imaging Systems.
• ANSI/NCSL Z540.3: Requirements for the Calibration of Measuring and Test Equipment.
• ASME B89.4.19: Performance Evaluation of Laser-Based Spherical Coordinate Measurement Systems.
• ISO/IEC JCGM 100: Guide to the Expression of Uncertainty in Measurement (GUM).
• ISO/IEC JCGM 200: International Vocabulary of Metrology.

By adhering to ASTM E2919-22, users and vendors ensure robust, transparent, and comparable assessments of static 6DOF pose measurement systems. This facilitates safer, more efficient implementation of advanced metrology technologies in diverse industrial and scientific environments.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E2919 − 22
Standard Test Method for
Evaluating the Performance of Systems that Measure Static,
Six Degrees of Freedom (6DOF), Pose
This standard is issued under the fixed designation E2919; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 2. Referenced Documents
1.1 Purpose—In this test method, metrics and procedures 2.1 ASTM Standards:
for collecting and analyzing data to determine the performance E456 Terminology Relating to Quality and Statistics
of a pose measurement system in computing the pose (position E2544 Terminology for Three-Dimensional (3D) Imaging
and orientation) of a rigid object are provided. Systems
2.2 ANSI/NCSL Standard:
1.2 This test method applies to the situation in which both
ANSI/NCSL Z540.3:2006 Requirements for the Calibration
the object and the pose measurement system are static with
of Measuring and Test Equipment
respect to each other when measurements are performed.
2.3 ASME Standard:
Vendors may use this test method to establish the performance
ASME B89.4.19 Performance Evaluation of Laser-Based
limits for their six degrees of freedom (6DOF) pose measure-
Spherical Coordinate Measurement Systems
ment systems. The vendor may use the procedures described in
2.4 ISO/IEC Standards:
9.2 to generate the test statistics, then apply an appropriate
JCGM 100:2008 Evaluation of Measurement Data—Guide
margin or scaling factor as desired to generate the performance
to the Expression of Uncertainty in Measurement (GUM)
specifications. This test method also provides a uniform way to
JCGM 106:2012 Evaluation of measurement data – The role
report the relative or absolute pose measurement capability of
of measurement uncertainty in conformity assessment
the system, or both, making it possible to compare the
JCGM 200:2012 International Vocabulary of Metrology—
performance of different systems.
Basic and General Concepts and Associated Terms (VIM),
1.3 Test Location—The methodology defined in this test
3rd edition
method shall be performed in a facility in which the environ-
IEC 60050-300:2001 International Electrotechnical
mental conditions are within the pose measurement system’s
Vocabulary—Electrical and Electronic Measurements and
rated conditions and meet the user’s requirements.
Measuring Instruments
1.4 Units—The values stated in SI units are to be regarded
3. Terminology
as the standard. No other units of measurement are included in
this standard.
3.1 Definitions from Other Standards:
1.5 This standard does not purport to address all of the 3.1.1 calibration, n—operation that, under specified
conditions, in a first step, establishes a relation between the
safety concerns, if any, associated with its use. It is the
responsibility of the user of this standard to establish appro- quantity values with measurement uncertainties provided by
measurement standards and corresponding indications with
priate safety, health, and environmental practices and deter-
mine the applicability of regulatory limitations prior to use. associated measurement uncertainties and, in a second step,
uses this information to establish a relation for obtaining a
1.6 This international standard was developed in accor-
dance with internationally recognized principles on standard- measurement result from an indication. JCGM 200:2012
ization established in the Decision on Principles for the
Development of International Standards, Guides and Recom-
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
mendations issued by the World Trade Organization Technical
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
Barriers to Trade (TBT) Committee. Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
Available from American National Standards Institute (ANSI), 25 W. 43rd St.,
This test method is under the jurisdiction of ASTM Committee E57 on 3D 4th Floor, New York, NY 10036, http://www.ansi.org.
Imaging Systems and is the direct responsibility of Subcommittee E57.23 on Available from American Society of Mechanical Engineers (ASME), ASME
Industrial 3D Machine Vision Systems. International Headquarters, Three Park Ave., New York, NY 10016-5990, http://
Current edition approved Dec. 15, 2022. Published February 2023. Originally www.asme.org.
approved in 2013. Last previous edition approved in 2014 as E2919 – 14. DOI: Available from American National Standards Institute (ANSI), 25 W. 43rd St.,
10.1520/E2919-22. 4th Floor, New York, NY 10036, http://www.ansi.org.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2919 − 22
3.1.1.1 Discussion— measurement standard with a measured quantity value having
(1) A calibration may be expressed by a statement, calibra- a negligible measurement uncertainty or if a conventional
tion function, calibration diagram, calibration curve, or cali- quantity value is given, in which case the measurement error is
bration table. In some cases, it may consist of an additive or known, and
multiplicative correction of the indication with associated (b) If a measurand is supposed to be represented by a
measurement uncertainty. unique true quantity value or a set of true quantity values of
(2) Calibration should not be confused with either adjust- negligible range, in which case the measurement error is not
ment of a measuring system, often mistakenly called “self- known.
calibration,” or verification of calibration. (2) Measurement error should not be confused with pro-
(3) Often, the first step alone in 3.1.1 is perceived as being duction error or mistake.
calibration.
3.1.5 measurement sample and sample, n—group of obser-
3.1.2 maximum permissible measurement error, maximum vations or test results, taken from a larger collection of
permissible error, and limit of error, n—extreme value of observations or test results, that serves to provide information
measurement error, with respect to a known reference quantity that may be used as a basis for making a decision concerning
value, permitted by specifications or regulations for a given the larger collection. E456
measurement, measuring instrument, or measuring system.
3.1.6 measurement uncertainty, uncertainty of
JCGM 200:2012
measurement, and uncertainty, n—non-negative parameter
3.1.2.1 Discussion—
characterizing the dispersion of the quantity values being
(1) Usually, the terms “maximum permissible errors” or
attributed to a measurand based on the information used.
“limits of error” are used when there are two extreme values.
JCGM 200:2012
(2) The term “tolerance” should not be used to designate
3.1.6.1 Discussion—
“maximum permissible error.”
(1) Measurement uncertainty includes components arising
3.1.3 measurand, n—quantity intended to be measured. from systematic effects, such as components associated with
JCGM 200:2012
corrections and the assigned quantity values of measurement
3.1.3.1 Discussion— standards, as well as the definitional uncertainty. Sometimes
(1) The specification of a measurand requires knowledge of estimated systematic effects are not corrected for but, instead,
the kind of quantity; description of the state of the associated measurement uncertainty components are incorpo-
phenomenon, body, or substance carrying the quantity, includ- rated.
ing any relevant component; and the chemical entities in- (2) The parameter may be, for example, a standard devia-
volved. tion called standard measurement uncertainty (or a specified
(2) In the second edition of the VIM and IEC 60050-300, multiple of it) or the half width of an interval, having a stated
the measurand is defined as the “quantity subject to measure- coverage probability.
ment.” (3) Measurement uncertainty comprises, in general, many
(3) The measurement, including the measuring system and components. Some of these may be evaluated by Type A
the conditions under which the measurement is carried out, evaluation of measurement uncertainty from the statistical
might change the phenomenon, body, or substance such that distribution of the quantity values from series of measurements
the quantity being measured may differ from the measurand as and can be characterized by standard deviations. The other
defined. In this case, adequate correction is necessary. components, which may be evaluated by Type B evaluation of
(a) Example 1—The potential difference between the measurement uncertainty, can also be characterized by stan-
terminals of a battery may decrease when using a voltmeter dard deviations evaluated from probability density functions
with a significant internal conductance to perform the measure- based on experience or other information.
ment. The open-circuit potential difference can be calculated (4) In general, for a given set of information, it is under-
from the internal resistances of the battery and the voltmeter. stood that the measurement uncertainty is associated with a
(b) Example 2—The length of a steel rod in equilibrium stated quantity value attributed to the measurand. A modifica-
with the ambient Celsius temperature of 23°C will be different tion of this value results in a modification of the associated
from the length at the specified temperature of 20°C, which is uncertainty.
the measurand. In this case, a correction is necessary.
3.1.7 precision, n—closeness of agreement between inde-
(4) In chemistry, “analyte,” or the name of a substance or
pendent test results obtained under stipulated conditions. E456
compound, are terms sometimes used for “measurand.” This
3.1.7.1 Discussion—
usage is erroneous because these terms do not refer to
(1) Precision depends on random errors and does not relate
quantities.
to the true value or the specified value.
3.1.4 measurement error, error of measurement, and error, (2) The measure of precision is usually expressed in terms
n—measured quantity value minus a reference quantity value. of imprecision and computed as a standard deviation of the test
JCGM 200:2012 results. Less precision is reflected by a larger standard devia-
3.1.4.1 Discussion— tion.
(1) The concept of “measurement error” can be used both: (3) “Independent test results” means results obtained in a
(a) When there is a single reference quantity value to manner not influenced by any previous result on the same or
refer to, which occurs if a calibration is made by means of a similar test object. Quantitative measures of precision depend
E2919 − 22
critically on the stipulated conditions. Repeatability and repro- (7) Registration between two point clouds is sometimes
ducibility conditions are particular sets of extreme stipulated referred to as cloud-to-cloud registration, between two sets of
conditions. control or survey points as target-to-target, between a point
cloud and a surface as cloud-to-surface, and between two
3.1.8 rated conditions, n—manufacturer-specified limits on
surfaces as surface-to-surface.
environmental, utility, and other conditions within which the
(8) The word alignment is sometimes used as a synony-
manufacturer’s performance specifications are guaranteed at
mous term for registration. However, in the context of this
the time of installation of the instrument. ASME B89.4.19
definition, an alignment is the result of the registration process.
3.1.9 reference quantity value and reference value,
3.1.11 true quantity value, true value of a quantity, and true
n—quantity value used as a basis for comparison with values of
value, n—quantity value consistent with the definition of a
quantities of the same kind. JCGM 200:2012
quantity. JCGM 200:2012
3.1.9.1 Discussion—
3.1.11.1 Discussion—
(1) A reference quantity value can be a true quantity value
(1) In the error approach to describing measurement, a true
of a measurand, in which case it is unknown, or a conventional
quantity value is considered unique and, in practice, unknow-
quantity value, in which case it is known.
able. The uncertainty approach is to recognize that, owing to
(2) A reference quantity value with associated measure-
the inherently incomplete amount of detail in the definition of
ment uncertainty is usually provided with reference to:
a quantity, there is not a single true quantity value but rather a
(a) A material, for example, a certified reference material;
set of true quantity values consistent with the definition.
(b) A device, for example, a stabilized laser;
However, this set of values is, in principle and practice,
(c) A reference measurement procedure; and
unknowable. Other approaches dispense altogether with the
(d) A comparison of measurement standards.
concept of true quantity value and rely on the concept of
metrological compatibility of measurement results for assess-
3.1.10 registration, n—process of determining and applying
ing their validity.
to two or more datasets the transformations that locate each
(2) In the special case of a fundamental constant, the
dataset in a common coordinate system so that the datasets are
quantity is considered to have a single true quantity value.
aligned relative to each other. E2544
(3) When the definitional uncertainty associated with the
3.1.10.1 Discussion—
measurand is considered to be negligible compared to the other
(1) A three-dimensional (3D) imaging system generally
components of the measurement uncertainty, the measurand
collects measurements in its local coordinate system. When the
may be considered to have an “essentially unique” true
same scene or object is measured from more than one position,
quantity value. This is the approach taken by JCGM 100 and
it is necessary to transform the data so that the datasets from
associated documents in which the word “true” is considered to
each position have a common coordinate system.
be redundant.
(2) Sometimes the registration process is performed on two
3.2 Definitions of Terms Specific to This Standard:
or more datasets that do not have regions in common. For
example, when several buildings are measured independently, 3.2.1 absolute pose, n—pose of an object in the coordinate
each dataset may be registered to a global coordinate system
frame of the system under test.
instead of to each other.
3.2.2 degree of freedom, DOF, n—any of the minimum
(3) In the context of this definition, a dataset may be a
number of translation or rotation components required to
mathematical representation of surfaces or may consist of a set
specify completely the pose of a rigid body.
of coordinates, for example, a point cloud, a 3D image, control
3.2.2.1 Discussion—
points, survey points, or reference points from a computer-
(1) In a 3D space, a rigid object can have at most 6DOFs,
aided drafted (CAD) model. Additionally, one of the datasets in
three translation and three rotation.
a registration may be a global coordinate system (as in
(2) The term “degree of freedom” is also used with regard
3.1.10.1(2)).
to statistical testing. It will be clear from the context in which
(4) The process of determining the transformation often
it is used whether the term relates to a statistical test or the
involves the minimization of an error function, such as the sum
rotation/translation aspect of the object.
of the squared distances between features (for example, points,
3.2.3 pose, n—a 6DOF vector whose components represent
lines, curves, and surfaces) in two datasets.
the position and orientation of a rigid object with respect to a
(5) In most cases, the transformations determined from a
coordinate frame.
registration process are rigid body transformations. This means
that the distances between points within a dataset do not
3.2.4 pose measurement system, n—a 3-D imaging system
change after applying the transformations, that is, rotations and
that measures the pose of an object.
translations.
3.2.4.1 Discussion—This system can consist of both hard-
(6) In some cases, the transformations determined from a
ware and software.
registration process are nonrigid body transformations. This
3.2.5 reference system, n—a measurement instrument or
means that the transformation includes a deformation of the
system used to generate a reference value or quantity.
dataset. One purpose of this type of registration is to attempt to
compensate for movement of the measured object or deforma- 3.2.6 relative pose, n—change of an object’s pose between
tion of its shape during the measurement. two poses measured in the same coordinate frame.
E2919 − 22
3.2.7 system under test, SUT, n—measurement instrument or tions specific to the user’s application, and to determine
system used to generate a test value or quantity. whether the system still performs in accordance with the
vendor’s specifications under those conditions. The intention of
3.2.8 work volume, n—physical space, or region within a
this test method is not to validate a vendor’s claims; although,
physical space, that defines the bounds within which a pose
under specific situations, this test method may be adapted for
measurement system is acquiring data.
this purpose.
3.3 Notation:
3.3.1 Mathematical equations throughout this test method
6. Apparatus
use the following notation. Scalar variables are lower-cased
6.1 Reference System:
italicized (for example, x), and scalar constants are upper-case
6.1.1 A reference pose measurement shall be established so
and italicized (for example, N). Vectors are lower-case and
that the error of the measured pose can be evaluated. If
bold faced (for example, t), and matrices are upper-case and
possible, the pose measurement uncertainty associated with the
bold-faced (for example, H). Special characters are used to
RS should be an order of magnitude (ten times) less than the
denote the measurements from the system under test (SUT).
measurement uncertainty associated with the SUT based on the
ˆ
The hat symbol (for example, R) represents a measurement
vendor’s specifications. The RS shall have been calibrated
from the SUT in its own coordinate frame, while the tilde (for
within the vendor-recommended calibration cycle and reported
˜
example, R) represents a measurement from the reference
as described in Section 11. The RS shall have been calibrated
system (RS) coordinate frame transformed to the SUT system
according to an available published standard. For example,
coordinate frame.
laser trackers or coordinate measurement machines that com-
ply with ASME B89 can be used to obtain the reference values.
4. Summary of Test Method
6.2 Test Objects:
4.1 This test method provides a set of test procedures and
6.2.1 Test objects should be rigid bodies chosen based on
statistically based performance metrics to evaluate quantita-
the user’s intended purpose or application. The geometry of the
tively the performance of a 6DOF pose measurement system to
objects should be representative of the user’s application; if the
measure the static pose of an object. It is applicable to the
user has no specific application, simple object geometries
situation in which both the pose measurement system and the
designed to minimize or eliminate pose ambiguities can be
object are static with respect to each other when the measure-
used. See English (1) for an illustrative example of a possible
ments are performed. The test method allows for the evaluation
geometric test object designed to minimize pose ambiguity.
of the absolute and relative pose of an object.
6.2.2 In this test method, no restrictions on the properties of
4.2 The test method involves measuring the pose of a
the selected test objects (for example, material, size,
user-specified object with the SUT and an RS at a minimum of
reflectivity, or texture) are placed; however, user or vendor
32 random locations within the work volume of the SUT. The
restrictions on the test object’s properties may need to be
pose errors, absolute or relative, are calculated based on the
accommodated if using this test method to evaluate the
measurements from the SUT and the RS. Performance of the
performance of the system with regard to the vendor’s speci-
SUT with regard to the vendor’s specifications pertaining to the
fications as they pertain to the user’s application.
user’s application is determined by selecting the appropriate
statistical test or tests as determined by the user. 7. Sampling Size
7.1 The performance evaluation of the SUT is based on the
5. Significance and Use
measurement error of a set of measurement results. The set
5.1 Pose measurement systems are used in a wide range of
consists of randomly sampled data points obtained from within
fields including manufacturing, material handling,
the work volume. Assuming that any single measurement error
construction, medicine, and aerospace. The use of pose mea-
depends only on the pose being measured, and not on the
surement systems could, for example, replace the need to fix
sequence of poses measured, the sample size N ≥ 32, should
the poses of objects of interest by mechanical means.
ensure that the average error approaches a normal distribution
5.2 Potential users have difficulty comparing pose measure- per the Central Limit Theorem.
ment systems because of the lack of standard performance
8. Absolute Pose Error and Relative Pose Error
specifications and test methods, and must rely on the specifi-
cations of a vendor regarding the system’s performance,
8.1 This section describes methods for calculating the ab-
capabilities, and suitability for a particular application. This
solute and relative pose errors. The concepts of absolute and
standard makes it possible for a user to assess and compare the
relative pose error will be explained in greater detail in 8.2 and
performance of candidate pose measurement systems, and
8.3, respectively. These errors form the basis for the test
allows the user to determine if the measured performance
procedure discussed in Section 9, which will then be used for
results are within the vendor’s claimed specifications with
the performance evaluations in Section 10.
regard to the user’s application. This standard also facilitates
8.1.1 Consider an instrument, S, performing pose measure-
the improvement of pose measurement systems by providing a
ments of an object, O, at Pose k = 1, 2, …, N. The pose consists
common set of metrics to evaluate system performance.
5.3 The intent of this test method is to allow a user to
The boldface numbers in parentheses refer to the list of references at the end of
determine the performance of a vendor’s system under condi- this standard.
E2919 − 22
of both orientation and position information. This test method ˜
H 5 H × H (4)
SUT O SUT RS RS O
k k
uses a 3 × 3 rotation matrix to represent rotation and a 3 × 1
vector to represent translation. Methods for transforming other
R t R t
SUT RS SUT RS k k
F G F G
rotation representations into a 3 × 3 rotation matrix represen-
0 1 0 1
tation can be found in Huynh (2).
˜ ˜
R t
8.1.2 The rotation and translation information at Pose k can
k k
F G
be simultaneously represented as a 4 × 4 homogeneous matrix. 0 1
R t
where:
k k
H (1)
F G
S O 5
k
0 1
H = transformation matrix of the coordinate frame of
SUT RS
the RS to the SUT (see Fig. 1), and
8.1.2.1 Here, the 3 × 3 rotation matrix, R , represents the
k
rotation of the object, O, in the coordinate system of S at Pose
˜
R 5 R R (5)
k SUT RS k
k and t represents the 3 × 1 translation vector of the object, O,
k
in the coordinate system of S at Pose k. ˜
t 5 R t 1 t
k SUT RS k SUT RS
8.1.3 In 8.2 and 8.3, methods are described to evaluate the
T
5@x˜ y˜ z˜ #
SUT with respect to a RS. In this test method, the poses of the
k k k
SUT and the RS are fixed relative to each other; therefore, there
are the rotation and translation components of the absolute
is a rigid transformation between them. Here,
pose computed from the measurement results obtained from
ˆ ˆ
the RS at Pose k in the SUT coordinate frame.
R t
k k
ˆ
H (2)
F G
SUT O 5
k
0 1
8.2.3 The absolute pose of an object at Pose k computed
from the SUT is represented as:
represents the object pose in the coordinate frame of the
ˆ ˆ
SUT at Pose k and
R t
k k
ˆ
H 5 (6)
F G
SUT O
k
0 1
R t
k k
H (3)
F G
RS O 5
k
0 1
where:
ˆ
= rotation component of the absolute pose computed
R
represents the object pose in the coordinate frame of the RS
k
from the SUT at Pose k, and
at Pose k. In 8.2, a method is described to calculate the ab-
T
ˆ
solute pose error of the object in a common coordinate = xˆ yˆ zˆ = translation component of the absolute pose
t @ #
k k k
k
frame. In 8.3, a method is described to calculate the relative
computed from the SUT at Pose k.
pose error of the object in which the SUT relative pose is
8.2.4 Using this notation, the rotation measurement error
calculated in the SUT coordinate frame and the RS relative
pose is calculated in the RS coordinate frame.
can be computed using the following procedure:
8.2.4.1 Compute R from H .
SUT RS SUT RS
8.2 Absolute Pose Error:
8.2.1 The absolute pose is defined with respect to the 8.2.4.2 Transform the orientation data obtained from the RS
˜
coordinate frame of the SUT. As a result, the object pose in the into the coordinate frame of the SUT by R 5 R R .
k SUT RS k
coordinate frame of the RS shall be transformed to the
coordinate frame of the SUT. It is assumed that the coordinate
frames of the RS and the SUT are fixed relative to one another
and, therefore, the transformation between their respective
coordinate frames does not change. The RS shall be registered
to the SUT according to the vendor’s specified process. In the
case that the vendor does not provide means for registration,
the selection of methods for transforming the coordinate frame
is left to the user. Note that the registration process contributes
toward the total measurement error (see 9.1.2). Once
transformed, the absolute pose of the object computed from the
measurement results obtained from the RS can be compared
with the absolute pose of the object computed from the
measurement results obtained from the SUT to determine the
rotation measurement error and the translation measurement
error.
FIG. 1 Absolute Pose of Object O at Pose k Computed from the
8.2.2 Here, the absolute pose of an object at Pose k
ˆ
computed from the measurement results obtained from the RS SUT Represented by H and Computed from the RS Repre-
SUT O
k
˜
is represented as: sented by H 5 H × H
SUT O SUT RS RS O
k k
E2919 − 22
T
˜ ˆ
ˆ ˆ 21 ˆ
8.2.4.3 Compute the rotation difference, R 5 R R . Note
H 5 H × H (9)
k k k
O O SUT O SUT O
1 k 1 k
˜ ˆ
that if R and R are identical, then R will equal the identity
k k k
ˆ ˆ ˆ ˆ
matrix.
R t R t
1 1 k k
F G F G
0 1 0 1
8.2.4.4 Compute the rotation measurement error as:
trace~R ! 2 1 ˆ ˆ
k
21 R t
1 k 1 k
0 # e 5 cos ,π (7)
S D
AbsAngle,k
F G
0 1
or
while the relative pose between Pose 1 and Pose k as seen
e = roll(R ) = rotation angle error about the x axis
AbsRoll,k k
by the RS can be defined as:
e = pitch(R ) = rotation angle error about the y axis
AbsPitch,k k
H 5 H × H (10)
e = yaw(R ) = rotation angle error about the z axis O O RS O RS O
1 k 1 k
AbsYaw,k k
R t R t
1 1 k k
as defined in Jazar (3). F G F G
0 1 0 1
8.2.5 The translation measurement errors can be evaluated
R t
as follows:
1 k 1 k
F G
0 1
2 2 2
e 5 = xˆ 2 x˜ 1 yˆ 2 y˜ 1 zˆ 2 z˜ (8)
~ ! ~ ! ~ !
AbsTran,k k k k k k k
8.3.3 The rotation measurement error can be evaluated in
e 5 xˆ 2 x˜
AbsX,k k k the following way:
8.3.3.1 Compute the rotation change as seen by the SUT
e 5 yˆ 2 y˜
AbsY,k k k
ˆ ˆ T ˆ
from Pose 1 to Pose k as the rotation matrix, R 5R R , and
1 k 1 k
T
from Pose 1 to Pose k as seen by the RS as R 5 R R .
e 5 zˆ 2 z˜ 1 k 1 k
AbsZ,k k k
ˆ T
8.3.3.2 Compute the rotation difference matrix, R 5 R R .
k 1 k k
8.3.3.3 Compute the rotation measurement error as:
8.3 Relative Pose Error:
trace R 2 1
~ !
k
8.3.1 The relative pose is defined as the change of an
0 # e 5 cos ,π (11)
S D
RelAngle,k
object’s pose between two poses, j and k, in the same
or
coordinate frame. In this test method, Pose j is the first sample
pose, while Pose k is selected from the remaining set of sample
e = roll(R ) = rotation angle error about the x axis
RelRoll,k k
Poses 2 to N. The relative pose as seen by the SUT is compared
e = pitch(R ) = rotation angle error about the y axis
RelPitch,k k
with the relative pose as seen by the RS (see Fig. 2). The
e = yaw(R ) = rotation angle error about the z axis
RelYaw,k k
relative pose metric consists of two error components: the
rotation measurement error and the translation measurement
as defined in Jazar (3).
error.
8.3.4 Translation measurement error can be evaluated by
8.3.2 The relative pose between Pose 1 and Pose k as seen
calculating:
by the SUT can be defined as:
2 2 2
=
e 5 ~xˆ 2 xˆ ! 1~yˆ 2 yˆ ! 1~zˆ 2 zˆ !
RelTran,k k 1 k 1 k 1
2 2 2
=
2 ~x 2 x ! 1~y 2 y ! 1~z 2 z ! (12)
k 1 k 1 k 1
where:
ˆ T
t 5 @xˆ yˆ zˆ # (13)
k k k k
= translation component of the object at Pose k as seen by the
SUT, and
T
t 5 x y z (14)
@ #
k k k k
= translation component of the object at Pose k as seen by the
RS.
9. Procedure
9.1 Introduction:
9.1.1 In this section, the basic procedure is described to
determine the pose measurement error of a pose measurement
FIG. 2 Relative Pose in which Object O is Moving from Pose 1 to
system. This procedure provides the basis for the evaluation of
Pose k with Respect to the RS, which is Represented by H ,
O O
1 k
a pose measurement system that measures the 6DOF pose of an
ˆ
and the SUT, which is Represented by H , and the Gray Re-
O O
1 k
object by comparing the results from a SUT to the results
gion Represents the Volume in which the Object is Being Moved
from Pose O to O obtained from a RS.
1 k
E2919 − 22
9.1.2 The pose measurement performance can be affected respectively) or relative (Eq 11 and Eq 12, respectively), as per
by many non-system parameters and factors, including those Section 8 for the selected pose of the object as observed by the
listed in Section 11. The performance of a pose measurement SUT.
system can also be affected by other factors such as those listed 9.2.4 Step 4—Perform Steps 2 and 3 for N sample locations
in 9.1.2.1 through 9.1.2.3. These errors should be minimized as within the work volume to generate a collection of measure-
much as possible. ment errors, e , e , …, e .
1 2 N
9.1.2.1 Noise—Active equipment in the same environment 9.2.5 Step 5—Calculate the average measurement error, ē,
as the pose measurement system may create noise that inter- as:
feres (for example, electromagnetic noise) with the perfor-
N
e
mance of the pose measurement system. Environmental factors
k
(
k51
may introduce noise that may also affect the performance of the e¯ 5 (15)
N
pose measurement system.
9.1.2.2 Registration Error—Registration processes contrib-
Compute the variance, s , using:
ute toward the final measurement error, and the magnitude of
N
the registration error may differ depending on the registration
~e 2 e¯ !
( k
k51
method used.
s 5 (16)
N 2 1
9.1.2.3 Vibration—Sensor and object vibration during the
test introduces distortion into the measurement results.
9.2.6 Step 6—Analyze the measurement results, e , ē, and
k
9.1.3 For a given sample pose, both the RS and SUT should s , to determine the performance of the SUT with regard to the
measure the reference object’s pose simultaneously from their
vendor’s specifications pertaining to the user’s application per
respective fixed poses. When testing in conditions where it is Section 10.
not possible for the RS and SUT to measure simultaneously,
10. Performance Evaluation
the reference measurement and the measurement from the SUT
should be obtained as close together in time as possible. The 10.1 This section is specifically for the application of this
SUT, RS, and reference object should not be moved during the test method for performance evaluation pertaining to the user’s
intermittent time span until both measurements have been application. Four performance limits are used in this test
collected. The environmental conditions should be as consis- method for performance evaluation, and a statistical test is
tent as possible and should be within the rated conditions of the described for each in the following sections.
RS and SUT over the entire period of the test.
10.2 Introduction:
9.2 Test Sequence—The basic test procedure consists of 10.2.1 After the data has been collected as specified in
obtaining measurement results from within the work volume of Section 9 and the error associated with each data point
the pose measurement system according to the six steps in calculated as described in Section 8, the results shall be
9.2.1 through 9.2.6. Testers may choose to either measure evaluated. Performance evaluation takes the form of using
randomly the pose of a selected object within a user-specified statistical tests to verify whether the SUT is operating within
subset of the work volume of the SUT (for example, a user’s
the vendor’s claimed performance limits.
application may only require that poses be measured in one or 10.2.2 A vendor’s performance specification is verified if
more subregions of the work volume) or measure randomly the
the performance tests in this standard accept the null hypoth-
pose of the object throughout the entire work volume. The esis with a p-value of greater than 0.95. The analysis is
number of random pose measurements shall be as large as described in statistical terms as a combination of null and
practical for the given SUT and RS considering the cost and alternative hypotheses, written as H and H , respectively. In
0 a
complexity of acquiring pose measurements. The number of Table 1, the four statistical tests used in this test method are
random pose measurements shall not be less than N = 32. described in terms of the null and alternative hypotheses. For
9.2.1 Step 1—Set up the RS and the pose measurement SUT example, Test I, the Average Error Test, applies to the expected
at fixed locations according to the vendors’ specifications. average error, E[ē], and the vendor’s specified performance
9.2.2 Step 2—Randomly select a pose for the object. This limit. If H is true in a statistical sense, then measurement
pose will be measured by the SUT and the RS, and measure- results obtained from the SUT are expected to be less than the
ment results will consist of measured values for position (x, y, vendor’s specified performance limit, δ , so the performance
avg
and z) and orientation (a 3 × 3 rotation matrix, R, see Section specification is accepted. In this case, the SUT is referred to as
8). being within the vendor’s performance specifications.
9.2.3 Step 3—Calculate the measurement errors for the Alternatively, if H is true in a statistical sense, then measure-
a
translation and rotation, either absolute (Eq 7 and Eq 8, ment results obtained from the SUT are not expected to be less
TABLE 1 Statistical Tests for the Analysis of Pose Measurement Systems
Test Test Name Null Hypothesis Alternative Hypothesis
I Average error test H : E[e¯] ≤ δ H : E[e¯] > δ
0 avg a avg
II Quantile error test H : q # δ H : q > δ
0 p quan a p quan
III Maximum permissible error test H : e # δ H : e > δ
0 max max a max max
2 2 2 2
IV Precision error test H : σ # σ H : σ > σ
0 0 a 0
E2919 − 22
than the vendor’s performance specification limit, δ . In this the largest and second largest observations, respectively. The
avg
case, the SUT is referred to as being outside of the vendor’s performance of the SUT is not within the vendor’s specifica-
performance specifications. tions if the following is true:
10.2.3 In Table 1, the four tests used in this test method are
δ 2 e α δ 2 e
max L max L
, → , 0.0526 (19)
listed with their associated performance specifications. The
e 2 e 1 2 α e 2 e
L S L S
vendor’s performance specifications are:
where:
δ = The vendor’s specified performance limit on the
avg
α = 0.05 (see X1.5).
expected average error, E[ē];
10.3.4 Precision Error Test—The performance of the SUT is
δ = The upper bound on the vendor-specified pth quan-
quan
not within the vendor’s specifications if the following is true:
tile of the average error, q ;
p
δ = The maximum average error, e ; and
max max
~N 2 1!s
. χ (20)
σ = The vendor’s specified performance limit on the
2 α,N21
σ
variance of the average error, σ .
where:
10.2.4 In X1.1, a more detailed explanation of performance
χ = value in which the cumulative distribution of the
α,N21
acceptance/rejection with regard to Tests I and II is provided.
Chi-squared PDF (see Refs 4 and 5) with N – 1
In particular, for Test II, if the experiment were repeated many
degrees of freedom has a probability of 1 – α = 0.95.
times, 100 × p percent of the trials will be less than δ . When
quan
p = 0.5, Test II is a statement about the median error. An
11. Report
explanation of how one can determine the appropriate test for
a given application is given in X1.1.
11.1 The following subsections summarize the mandatory
and optional information to be reported. An example form
10.3 Evaluating Performance—This section describes the
layout is provided in Appendix X2.
procedure for determining if the performance of the SUT is
within the vendor’s specifications for δ , δ , δ , and σ .
11.1.1 Mandatory Information:
avg quan max 0
In the following subsections, the processes for determining
11.1.1.1 The following information shall be included in the
whether the performance of the SUT is within the vendor’s
test report:
specifications based on Tests I through IV are summarized.
(1) Testing conditions:
(a) Report author name, company, position, e-mail ad-
10.3.1 Average Error Test:
dress and telephone number.
10.3.1.1 With the assumption that the measurement error is
(b) Report author signature and date signed.
normally distributed (see Appendix X1), using the Z-test, the
(c) Facility name, street address, city, state or province
SUT is not within the vendor’s performance specifications if
and country.
the following is true:
(d) Test date (month/day/year).
e¯ 2 δ
avg
. Z (17) (e) Total time to perform the test.
α
=s ⁄N
(f) Portion of total time for initial set-up (including sensor
warm-up).
where:
(g) System Under Test (SUT) Settings:
e¯ = average measurement error computed using Eq 15,
(i) SUT manufacturer, model number, serial number,
s = sample variance defined in Eq 16, and
(ii) Date calibrated,
Z = value at which the cumulative distribution function for
α
(iii) Operator name,
the standard normal distribution has the value 0.95 (see
(iv) System settings.
Ref 4). Specifically Z = 1.6449.
α
(h) Reference System (RS) Settings:
10.3.1.2 See X1.3 for a more detailed explanation of the
(i) Reference Instrument manufacturer, model number,
value of the Z test.
serial number,
10.3.2 Quantile Error Test:
(ii) Date calibrated and reference to supporting docu-
10.3.2.1 Let T be equal to the number of elements of {e , .,
mentation on file,
e } for which e ≤ δ is true. Using the Sign Test, the
N k quan
(iii) Specified measurement uncertainty,
performance of the SUT is not within the vendor’s specifica-
(iv) Operator name,
tions if the following is true:
(v) System settings.
T # b (18)
(i) Ambient Test Conditions:
N,α
(i) Range of ambient temperature during test (____°C
where:
to ____°C),
b = upper quantile of a binomial Probability Density
N,α
(ii) Maximum rate of ambient temperature change
Function (PDF) with parameters N and α = p.
during test. (____°C per minute),
10.3.2.2 See X1.4 for details on the Sign Test and how b
(iii) Relative ambient humidity during test (____%),
N,α
is calculated. (iv) Any particulate matter in air (y/n) ____,
10.3.3 Maximum Permissible Error Test—Order the obser- (v) Approximate average ambient illumination on the
vations {e , ., e } from smallest to largest and let e and e be object during test (____ lumens),
1 N L S
E2919 − 22
(vi) Primary ambient illumination source type on (3) For all pose measurements:
object (for example, sun, fluorescent, incandescent). (a) Reference pose,
(j) Object Characteristics (be as specific as possible in (b) Measured pose,
order to be able to uniquely identify and reproduce the testing (c) Translation error for each metric used, and
(d) Orientation error for each metric used.
conditions):
(i) Attach a picture of the object, (4) Average errors for each repetition.
(5) Performance evaluation:
(ii) General description of object shape and material(s)
from which it is made, (a) Name of statistical test performed,
(b) Computed value and vendor specified performance
(iii) Minimum enclosing bounding box dimensions in
(m), limit, and
(c) Result—Within the Vendor’s Performance
(iv) Object primary surface feature types (for example,
holes, slots, pillars, or convexities), Specifications, or Not Within the Vendor’s Performance Speci-
fications.
(v) Object surface predominant color(s),
(vi) Object surface qualitative deposited particle (for 11.1.1.2 The report shall be formatted so that hard copies of
example, rust, or dirt) condition (approximate average particle test reports include the page number and total number of pages.
11.1.2 Optional Information—If the absolute pose was
size) in (mm),
(vii) Object qualitative surface moisture condition (dry, evaluated, describe the method (for example, measuring
targets, feature matching) used to register the RS to the SUT.
damp, or wet),
(viii) Other material on surface, if any (such as oil or
12. Precision and Bias
machining fluid or coating)—specify material composition and
approximate average thickness. 12.1 No information can be presented on the precision or
(k) Optional Object Characteristics: bias of the procedure in Test Method E2919 for measuring
(i) Object surface reflectance at the sensor’s wave- 6DOF pose measurement system performance because no
lengths (____% to ____%), particular reference system or reference object is specified. The
(ii) Object surfaces scattering at wavelength(s) em- purpose of Test Method E2919 is to evaluate the vendor’s
ployed by sensor system (____% to ____%), specifications for the performance of its system under the
(iii) Object approximate surface optical absorption and conditions of the user’s application. It is expected that the
precision and bias will vary under different testing conditions.
secondary reflection (if any) at the sensor’s wavelengths (%),
(iv) Object surfaces roughness (Ra) in micrometers or
13. Keywords
specify other standard surface roughness metric (for example
ASME B46.1: “Surface Texture (Surface Roughness, 13.1 absolute pose error; performance evaluation; pose
Waviness, and if an Ra value is not available. measurement system; pose measurement test procedure; rela-
(2) Metrics used—Relative pose error, absolute pose error, tive pose error; 6DOF; static pose measurement performance;
or both.
3D imaging system
APPENDIXES
(Nonmandatory Information)
X1. STATISTICAL TESTS
n
X1.1 The pth-quantile of a continuous and positive random 1
¯
X 5 X (X1.4)
( i
variable X with probability density function f(x) is that value q n
i51
satisfying:
and the variance, σ , is usually estimated by:
q
k 5 f x dx 5 F q (X1.1)
* ~ ! ~ ! n
0 2
2 ¯
s 5 ~X 2 X! (X1.5)
( i
n 2 1
i51
where:
F(q) = distribution function of X.
as the sample variance.
X1.1.1 The mean of X is the value μ satisfying:
X1.1.4 The average error and quantile tests assess whether
`
the observed error (translation or rotation) is significantly less
μ 5 xf x dx (X1.2)
* ~ !
than the vendor’s performance specification. Consider Fig.
X1.1.2 The variance of X is the value σ satisfying:
X1.1 in which the bell-shaped curve represents how the
`
measurement results are assumed to be distributed (thus, the
2 2
σ 5 ~x 2 μ! f~x!dx (X1.3)
*
requirement that the data be approximately normally distrib-
X1.1.3 If random observations X , X , ., X are taken on, X, uted for the average error test to be valid) around the average
1 2 n
then the mean, μ, is usually estimated by the sample average: error E[e]. If δ is located at Point A, then δ is not
avg avg
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2919 − 14 E2919 − 22
Standard Test Method for
Evaluating the Performance of Systems that Measure Static,
Six Degrees of Freedom (6DOF), Pose
This standard is issued under the fixed designation E2919; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 Purpose—In this test method, metrics and procedures for collecting and analyzing data to determine the performance of a pose
measurement system in computing the pose (position and orientation) of a rigid object are provided.
1.2 This test method applies to the situation in which both the object and the pose measurement system are static with respect to
each other when measurements are performed. Vendors may use this test method to establish the performance limits for their six
degrees of freedom (6DOF) pose measurement systems. The vendor may use the procedures described in 9.2 to generate the test
statistics, then apply an appropriate margin or scaling factor as desired to generate the performance specifications. This test method
also provides a uniform way to report the relative or absolute pose measurement capability of the system, or both, making it
possible to compare the performance of different systems.
1.3 Test Location—The methodology defined in this test method shall be performed in a facility in which the environmental
conditions are within the pose measurement system’s rated conditions and meet the user’s requirements.
1.4 Units—The values stated in SI units are to be regarded as the standard. No other units of measurement are included in this
standard.
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.6 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E456 Terminology Relating to Quality and Statistics
E2544 Terminology for Three-Dimensional (3D) Imaging Systems
2.2 ANSI/NCSL Standard:
ANSI/NCSL Z540.3:2006 Requirements for the Calibration of Measuring and Test Equipment
This test method is under the jurisdiction of ASTM Committee E57 on 3D Imaging Systems and is the direct responsibility of Subcommittee E57.23 on Industrial 3D
Machine Vision Systems.
Current edition approved July 1, 2014Dec. 15, 2022. Published August 2014February 2023. Originally approved in 2013. Last previous edition approved in 20132014 as
E2919E2919 – 14.-13. DOI: 10.1520/E2919-14.10.1520/E2919-22.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from American National Standards Institute (ANSI), 25 W. 43rd St., 4th Floor, New York, NY 10036, http://www.ansi.org.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2919 − 22
2.3 ASME Standard:
ASME B89.4.19 Performance Evaluation of Laser-Based Spherical Coordinate Measurement Systems
2.4 ISO/IEC Standards:
JCGM 100:2008 Evaluation of Measurement Data—Guide to the Expression of Uncertainty in Measurement (GUM)
JCGM 106:2012 Evaluation of measurement data – The role of measurement uncertainty in conformity assessment
JCGM 200:2012 International Vocabulary of Metrology—Basic and General Concepts and Associated Terms (VIM), 3rd edition
JCGM 100:2008 Evaluation of Measurement Data—Guide to the Expression of Uncertainty in Measurement (GUM)
IEC 60050-300:2001 International Electrotechnical Vocabulary—Electrical and Electronic Measurements and Measuring
Instruments
3. Terminology
3.1 Definitions from Other Standards:
3.1.1 calibration, n—operation that, under specified conditions, in a first step, establishes a relation between the quantity values
with measurement uncertainties provided by measurement standards and corresponding indications with associated measurement
uncertainties and, in a second step, uses this information to establish a relation for obtaining a measurement result from an
indication. JCGM 200:2012
3.1.1.1 Discussion—
(1) A calibration may be expressed by a statement, calibration function, calibration diagram, calibration curve, or calibration
table. In some cases, it may consist of an additive or multiplicative correction of the indication with associated measurement
uncertainty.
(2) Calibration should not be confused with either adjustment of a measuring system, often mistakenly called “self-calibration,”
or verification of calibration.
(3) Often, the first step alone in 3.1.1 is perceived as being calibration.
3.1.2 maximum permissible measurement error, maximum permissible error, and limit of error, n—extreme value of measurement
error, with respect to a known reference quantity value, permitted by specifications or regulations for a given measurement,
measuring instrument, or measuring system. JCGM 200:2012
3.1.2.1 Discussion—
(1) Usually, the terms “maximum permissible errors” or “limits of error” are used when there are two extreme values.
(2) The term “tolerance” should not be used to designate “maximum permissible error.”
3.1.3 measurand, n—quantity intended to be measured. JCGM 200:2012
3.1.3.1 Discussion—
(1) The specification of a measurand requires knowledge of the kind of quantity; description of the state of the phenomenon,
body, or substance carrying the quantity, including any relevant component; and the chemical entities involved.
(2) In the second edition of the VIM and IEC 60050-300, the measurand is defined as the “quantity subject to measurement.”
(3) The measurement, including the measuring system and the conditions under which the measurement is carried out, might
change the phenomenon, body, or substance such that the quantity being measured may differ from the measurand as defined. In
this case, adequate correction is necessary.
(a) Example 1—The potential difference between the terminals of a battery may decrease when using a voltmeter with a
significant internal conductance to perform the measurement. The open-circuit potential difference can be calculated from the
internal resistances of the battery and the voltmeter.
(b) Example 2—The length of a steel rod in equilibrium with the ambient Celsius temperature of 23°C will be different from
the length at the specified temperature of 20°C, which is the measurand. In this case, a correction is necessary.
(4) In chemistry, “analyte,” or the name of a substance or compound, are terms sometimes used for “measurand.” This usage
is erroneous because these terms do not refer to quantities.
3.1.4 measurement error, error of measurement, and error, n—measured quantity value minus a reference quantity value. JCGM
200:2012
3.1.4.1 Discussion—
(1) The concept of “measurement error” can be used both:
Available from American Society of Mechanical Engineers (ASME), ASME International Headquarters, Three Park Ave., New York, NY 10016-5990, http://
www.asme.org.
Available from American National Standards Institute (ANSI), 25 W. 43rd St., 4th Floor, New York, NY 10036, http://www.ansi.org.
E2919 − 22
(a) When there is a single reference quantity value to refer to, which occurs if a calibration is made by means of a
measurement standard with a measured quantity value having a negligible measurement uncertainty or if a conventional quantity
value is given, in which case the measurement error is known, and
(b) If a measurand is supposed to be represented by a unique true quantity value or a set of true quantity values of negligible
range, in which case the measurement error is not known.
(2) Measurement error should not be confused with production error or mistake.
3.1.5 measurement sample and sample, n—group of observations or test results, taken from a larger collection of observations or
test results, that serves to provide information that may be used as a basis for making a decision concerning the larger collection.
E456
3.1.6 measurement uncertainty, uncertainty of measurement, and uncertainty, n—non-negative parameter characterizing the
dispersion of the quantity values being attributed to a measurand based on the information used. JCGM 200:2012
3.1.6.1 Discussion—
(1) Measurement uncertainty includes components arising from systematic effects, such as components associated with
corrections and the assigned quantity values of measurement standards, as well as the definitional uncertainty. Sometimes
estimated systematic effects are not corrected for but, instead, associated measurement uncertainty components are incorporated.
(2) The parameter may be, for example, a standard deviation called standard measurement uncertainty (or a specified multiple
of it) or the half width of an interval, having a stated coverage probability.
(3) Measurement uncertainty comprises, in general, many components. Some of these may be evaluated by Type A evaluation
of measurement uncertainty from the statistical distribution of the quantity values from series of measurements and can be
characterized by standard deviations. The other components, which may be evaluated by Type B evaluation of measurement
uncertainty, can also be characterized by standard deviations evaluated from probability density functions based on experience or
other information.
(4) In general, for a given set of information, it is understood that the measurement uncertainty is associated with a stated
quantity value attributed to the measurand. A modification of this value results in a modification of the associated uncertainty.
3.1.7 precision, n—closeness of agreement between independent test results obtained under stipulated conditions. E456
3.1.7.1 Discussion—
(1) Precision depends on random errors and does not relate to the true value or the specified value.
(2) The measure of precision is usually expressed in terms of imprecision and computed as a standard deviation of the test
results. Less precision is reflected by a larger standard deviation.
(3) “Independent test results” means results obtained in a manner not influenced by any previous result on the same or similar
test object. Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility
conditions are particular sets of extreme stipulated conditions.
3.1.8 rated conditions, n—manufacturer-specified limits on environmental, utility, and other conditions within which the
manufacturer’s performance specifications are guaranteed at the time of installation of the instrument. ASME B89.4.19
3.1.9 reference quantity value and reference value, n—quantity value used as a basis for comparison with values of quantities of
the same kind. JCGM 200:2012
3.1.9.1 Discussion—
(1) A reference quantity value can be a true quantity value of a measurand, in which case it is unknown, or a conventional
quantity value, in which case it is known.
(2) A reference quantity value with associated measurement uncertainty is usually provided with reference to:
(a) A material, for example, a certified reference material;
(b) A device, for example, a stabilized laser;
(c) A reference measurement procedure; and
(d) A comparison of measurement standards.
3.1.10 registration, n—process of determining and applying to two or more datasets the transformations that locate each dataset
in a common coordinate system so that the datasets are aligned relative to each other. E2544
3.1.10.1 Discussion—
(1) A three-dimensional (3D) imaging system generally collects measurements in its local coordinate system. When the same
scene or object is measured from more than one position, it is necessary to transform the data so that the datasets from each position
have a common coordinate system.
E2919 − 22
(2) Sometimes the registration process is performed on two or more datasets that do not have regions in common. For example,
when several buildings are measured independently, each dataset may be registered to a global coordinate system instead of to each
other.
(3) In the context of this definition, a dataset may be a mathematical representation of surfaces or may consist of a set of
coordinates, for example, a point cloud, a 3D image, control points, survey points, or reference points from a computer-aided
drafted (CAD) model. Additionally, one of the datasets in a registration may be a global coordinate system (as in 3.1.10.1(2)).
(4) The process of determining the transformation often involves the minimization of an error function, such as the sum of the
squared distances between features (for example, points, lines, curves, and surfaces) in two datasets.
(5) In most cases, the transformations determined from a registration process are rigid body transformations. This means that
the distances between points within a dataset do not change after applying the transformations, that is, rotations and translations.
(6) In some cases, the transformations determined from a registration process are nonrigid body transformations. This means
that the transformation includes a deformation of the dataset. One purpose of this type of registration is to attempt to compensate
for movement of the measured object or deformation of its shape during the measurement.
(7) Registration between two point clouds is sometimes referred to as cloud-to-cloud registration, between two sets of control
or survey points as target-to-target, between a point cloud and a surface as cloud-to-surface, and between two surfaces as
surface-to-surface.
(8) The word alignment is sometimes used as a synonymous term for registration. However, in the context of this definition,
an alignment is the result of the registration process.
3.1.11 true quantity value, true value of a quantity, and true value, n—quantity value consistent with the definition of a quantity.
JCGM 200:2012
3.1.11.1 Discussion—
(1) In the error approach to describing measurement, a true quantity value is considered unique and, in practice, unknowable.
The uncertainty approach is to recognize that, owing to the inherently incomplete amount of detail in the definition of a quantity,
there is not a single true quantity value but rather a set of true quantity values consistent with the definition. However, this set of
values is, in principle and practice, unknowable. Other approaches dispense altogether with the concept of true quantity value and
rely on the concept of metrological compatibility of measurement results for assessing their validity.
(2) In the special case of a fundamental constant, the quantity is considered to have a single true quantity value.
(3) When the definitional uncertainty associated with the measurand is considered to be negligible compared to the other
components of the measurement uncertainty, the measurand may be considered to have an “essentially unique” true quantity value.
This is the approach taken by JCGM 100 and associated documents in which the word “true” is considered to be redundant.
3.2 Definitions of Terms Specific to This Standard:
3.2.1 absolute pose, n—pose of an object in the coordinate frame of the system under test.
3.2.2 degree of freedom, DOF, n—any of the minimum number of translation or rotation components required to specify
completely the pose of a rigid body.
3.2.2.1 Discussion—
(1) In a 3D space, a rigid object can have at most 6DOFs, three translation and three rotation.
(2) The term “degree of freedom” is also used with regard to statistical testing. It will be clear from the context in which it
is used whether the term relates to a statistical test or the rotation/translation aspect of the object.
3.2.3 pose, n—a 6DOF vector whose components represent the position and orientation of a rigid object with respect to a
coordinate frame.
3.2.4 pose measurement system, n—a 3-D imaging system that measures the pose of an object.
3.2.4.1 Discussion—
This system can consist of both hardware and software.
3.2.5 reference system, n—a measurement instrument or system used to generate a reference value or quantity.
3.2.6 relative pose, n—change of an object’s pose between two poses measured in the same coordinate frame.
3.2.7 system under test, SUT, n—measurement instrument or system used to generate a test value or quantity.
E2919 − 22
3.2.8 work volume, n—physical space, or region within a physical space, that defines the bounds within which a pose measurement
system is acquiring data.
3.3 Notation:
3.3.1 Mathematical equations throughout this test method use the following notation. Scalar variables are lower-cased italicized
(for example, x), and scalar constants are upper-case and italicized (for example, N). Vectors are lower-case and bold faced (for
example, t), and matrices are upper-case and bold-faced (for example, H). Special characters are used to denote the measurements
ˆ
from the system under test (SUT). The hat symbol (for example, R) represents a measurement from the SUT in its own coordinate
˜
frame, while the tilde (for example, R) represents a measurement from the reference system (RS) coordinate frame transformed
to the SUT system coordinate frame.
4. Summary of Test Method
4.1 This test method provides a set of test procedures and statistically based performance metrics to evaluate quantitatively the
performance of a 6DOF pose measurement system to measure the static pose of an object. It is applicable to the situation in which
both the pose measurement system and the object are static with respect to each other when the measurements are performed. The
test method allows for the evaluation of the absolute and relative pose of an object.
4.2 The test method involves measuring the pose of a user-specified object with the SUT and an RS at a minimum of 32 random
locations within the work volume of the SUT. The pose errors, absolute or relative, are calculated based on the measurements from
the SUT and the RS. Performance of the SUT with regard to the vendor’s specifications pertaining to the user’s application is
determined by selecting the appropriate statistical test or tests as determined by the user.
5. Significance and Use
5.1 Pose measurement systems are used in a wide range of fields including manufacturing, material handling, construction,
medicine, and aerospace. The use of pose measurement systems could, for example, replace the need to fix the poses of objects
of interest by mechanical means.
5.2 Potential users have difficulty comparing pose measurement systems because of the lack of standard performance
specifications and test methods, and must rely on the specifications of a vendor regarding the system’s performance, capabilities,
and suitability for a particular application. This standard makes it possible for a user to assess and compare the performance of
candidate pose measurement systems, and allows the user to determine if the measured performance results are within the vendor’s
claimed specifications with regard to the user’s application. This standard also facilitates the improvement of pose measurement
systems by providing a common set of metrics to evaluate system performance.
5.3 The intent of this test method is to allow a user to determine the performance of a vendor’s system under conditions specific
to the user’s application, and to determine whether the system still performs in accordance with the vendor’s specifications under
those conditions. The intention of this test method is not to validate a vendor’s claims; although, under specific situations, this test
method may be adapted for this purpose.
6. Apparatus
6.1 Reference System:
6.1.1 A reference pose measurement shall be established so that the error of the measured pose can be evaluated. If possible, the
pose measurement uncertainty associated with the RS should be an order of magnitude (ten times) less than the measurement
uncertainty associated with the SUT based on the vendor’s specifications. The RS shall have been calibrated within the
vendor-recommended calibration cycle and reported as described in Section 11. The RS shall have been calibrated according to
an available published standard. For example, laser trackers or coordinate measurement machines that comply with ASME B89
can be used to obtain the reference values.
6.2 Test Objects:
6.2.1 Test objects should be rigid bodies chosen based on the user’s intended purpose or application. The geometry of the objects
E2919 − 22
should be representative of the user’s application; if the user has no specific application, simple object geometries designed to
minimize or eliminate pose ambiguities can be used. See English (1) for an illustrative example of a possible geometric test object
designed to minimize pose ambiguity.
6.2.2 In this test method, no restrictions on the properties of the selected test objects (for example, material, size, reflectivity, or
texture) are placed; however, user or vendor restrictions on the test object’s properties may need to be accommodated if using this
test method to evaluate the performance of the system with regard to the vendor’s specifications as they pertain to the user’s
application.
7. Sampling Size
7.1 The performance evaluation of the SUT is based on the measurement error of a set of measurement results. The set consists
of randomly sampled data points obtained from within the work volume. Assuming that any single measurement error depends only
on the pose being measured, and not on the sequence of poses measured, the sample size N ≥ 32, should ensure that the average
error approaches a normal distribution per the Central Limit Theorem.
8. Absolute Pose Error and Relative Pose Error
8.1 This section describes methods for calculating the absolute and relative pose errors. The concepts of absolute and relative pose
error will be explained in greater detail in 8.2 and 8.3, respectively. These errors form the basis for the test procedure discussed
in Section 9, which will then be used for the performance evaluations in Section 10.
8.1.1 Consider an instrument, S, performing pose measurements of an object, O, at Pose k = 1, 2, …, N. The pose consists of both
orientation and position information. This test method uses a 3 × 3 rotation matrix to represent rotation and a 3 × 1 vector to
represent translation. Methods for transforming other rotation representations into a 3 × 3 rotation matrix representation can be
found in Huynh (2).
8.1.2 The rotation and translation information at Pose k can be simultaneously represented as a 4 × 4 homogeneous matrix.
R t
k k
H (1)
F G
S O 5
k
0 1
8.1.2.1 Here, the 3 × 3 rotation matrix, R , represents the rotation of the object, O, in the coordinate system of S at Pose k and
k
t represents the 3 × 1 translation vector of the object, O, in the coordinate system of S at Pose k.
k
8.1.3 In 8.2 and 8.3, methods are described to evaluate the SUT with respect to a RS. In this test method, the poses of the SUT
and the RS are fixed relative to each other; therefore, there is a rigid transformation between them. Here,
ˆ ˆ
R t
k k
ˆ
H (2)
F G
SUT O 5
k
0 1
represents the object pose in the coordinate frame of the SUT at Pose k and
R t
k k
H (3)
F G
RS O 5
k
0 1
represents the object pose in the coordinate frame of the RS at Pose k. In 8.2, a method is described to calculate the absolute
pose error of the object in a common coordinate frame. In 8.3, a method is described to calculate the relative pose error of the
object in which the SUT relative pose is calculated in the SUT coordinate frame and the RS relative pose is calculated in the
RS coordinate frame.
8.2 Absolute Pose Error:
8.2.1 The absolute pose is defined with respect to the coordinate frame of the SUT. As a result, the object pose in the coordinate
frame of the RS shall be transformed to the coordinate frame of the SUT. It is assumed that the coordinate frames of the RS and
the SUT are fixed relative to one another and, therefore, the transformation between their respective coordinate frames does not
change. The RS shall be registered to the SUT according to the vendor’s specified process. In the case that the vendor does not
provide means for registration, the selection of methods for transforming the coordinate frame is left to the user. Note that the
The boldface numbers in parentheses refer to the list of references at the end of this standard.
E2919 − 22
registration process contributes toward the total measurement error (see 9.1.2). Once transformed, the absolute pose of the object
computed from the measurement results obtained from the RS can be compared with the absolute pose of the object computed from
the measurement results obtained from the SUT to determine the rotation measurement error and the translation measurement error.
8.2.2 Here, the absolute pose of an object at Pose k computed from the measurement results obtained from the RS is represented
as:
˜
H 5 H × H (4)
SUT O SUT RS RS O
k k
R t R t
SUT RS SUT RS k k
F G F G
0 1 0 1
˜ ˜
R t
k k
F G
0 1
where:
H = transformation matrix of the coordinate frame of the RS to the SUT (see Fig. 1), and
SUT RS
˜
R 5 R R (5)
k SUT RS k
˜
t 5 R t 1 t
k SUT RS k SUT RS
T
5@x˜ y˜ z˜ #
k k k
are the rotation and translation components of the absolute pose computed from the measurement results obtained from the
RS at Pose k in the SUT coordinate frame.
8.2.3 The absolute pose of an object at Pose k computed from the SUT is represented as:
ˆ ˆ
R t
k k
ˆ
H 5 (6)
F G
SUT O
k
0 1
where:
ˆ
= rotation component of the absolute pose computed from the SUT at Pose k, and
R
k
T
ˆ
= xˆ yˆ zˆ = translation component of the absolute pose computed from the SUT at Pose k.
t @ #
k k k
k
8.2.4 Using this notation, the rotation measurement error can be computed using the following procedure:
8.2.4.1 Compute R from H .
SUT RS SUT RS
ˆ
FIG. 1 Absolute Pose of Object O at Pose k Computed from the SUT Represented by H and Computed from the RS Represented
SUT O
k
˜
by H 5 H × H
SUT O SUT RS RS O
k k
E2919 − 22
8.2.4.2 8.2.4.2 8.2.4.2 8.2.4.2 Transform the orientation data obtained from the RS into the coordinate frame of the SUT by
˜
R 5 R R .
k SUT RS k
˜ ˆ T ˜ ˆ
8.2.4.3 Compute the rotation difference, R 5R R . Note that if R and R are identical, then R will equal the identity matrix.
k k k k k k
8.2.4.4 Compute the rotation measurement error as:
trace R 2 1
~ !
k
0# e 5 cos ,π (7)
S D
AbsAngle,k
or
e = roll(R ) = rotation angle error about the x axis
AbsRoll,k k
e = pitch(R ) = rotation angle error about the y axis
AbsPitch,k k
e = yaw(R ) = rotation angle error about the z axis
AbsYaw,k k
as defined in Jazar (3).
8.2.5 The translation measurement errors can be evaluated as follows:
2 2 2
e 5= xˆ 2 x˜ 1 yˆ 2 y˜ 1 zˆ 2 z˜ (8)
~ ! ~ ! ~ !
AbsTran,k k k k k k k
e 5 xˆ 2 x˜
AbsX,k k k
e 5 yˆ 2 y˜
AbsY,k k k
e 5 zˆ 2 z˜
AbsZ,k k k
8.3 Relative Pose Error:
8.3.1 The relative pose is defined as the change of an object’s pose between two poses, j and k, in the same coordinate frame. In
this test method, Pose j is the first sample pose, while Pose k is selected from the remaining set of sample Poses 2 to N. The relative
pose as seen by the SUT is compared with the relative pose as seen by the RS (see Fig. 2). The relative pose metric consists of
two error components: the rotation measurement error and the translation measurement error.
8.3.2 The relative pose between Pose 1 and Pose k as seen by the SUT can be defined as:
ˆ ˆ ˆ
H 5 H × H (9)
O O SUT O SUT O
1 k 1 k
ˆ ˆ ˆ ˆ
R t R t
1 1 k k
F G F G
0 1 0 1
FIG. 2 Relative Pose in which Object O is Moving from Pose 1 to Pose k with Respect to the RS, which is Represented by H , and
O O
1 k
the SUT, which is Represented by Hˆ , and the Gray Region Represents the Volume in which the Object is Being Moved from Pose
O O
1 k
O to O
1 k
E2919 − 22
ˆ ˆ
R t
1 k 1 k
F G
0 1
while the relative pose between Pose 1 and Pose k as seen by the RS can be defined as:
H 5 H × H (10)
O O RS O RS O
1 k 1 k
R t R t
1 1 k k
F G F G
0 1 0 1
R t
1 k 1 k
F G
0 1
8.3.3 The rotation measurement error can be evaluated in the following way:
ˆ ˆ T ˆ
8.3.3.1 Compute the rotation change as seen by the SUT from Pose 1 to Pose k as the rotation matrix, R 5R R , and from Pose
1 k 1 k
T
1 to Pose k as seen by the RS as R 5R R .
1 k 1 k
ˆ T
8.3.3.2 Compute the rotation difference matrix, R 5 R R .
k 1 k k
8.3.3.3 Compute the rotation measurement error as:
trace R 2 1
~ !
k
0# e 5 cos ,π (11)
S D
RelAngle,k
or
e = roll(R ) = rotation angle error about the x axis
RelRoll,k k
e = pitch(R ) = rotation angle error about the y axis
RelPitch,k k
e = yaw(R ) = rotation angle error about the z axis
RelYaw,k k
as defined in Jazar (3).
8.3.4 Translation measurement error can be evaluated by calculating:
2 2 2 2 2 2
e 5= xˆ 2 xˆ 1 yˆ 2 yˆ 1 zˆ 2 zˆ 2= x 2 x 1 y 2 y 1 z 2 z (12)
~ ! ~ ! ~ ! ~ ! ~ ! ~ !
RelTran,k k 1 k 1 k 1 k 1 k 1 k 1
where:
ˆ T
t 5 @xˆ yˆ zˆ # (13)
k k k k
= translation component of the object at Pose k as seen by the SUT, and
T
t 5 x y z (14)
@ #
k k k k
= translation component of the object at Pose k as seen by the RS.
9. Procedure
9.1 Introduction:
9.1.1 In this section, the basic procedure is described to determine the pose measurement error of a pose measurement system. This
procedure provides the basis for the evaluation of a pose measurement system that measures the 6DOF pose of an object by
comparing the results from a SUT to the results obtained from a RS.
9.1.2 The pose measurement performance can be affected by many non-system parameters and factors, including those listed in
Section 11. The performance of a pose measurement system can also be affected by other factors such as those listed in 9.1.2.1
through 9.1.2.3. These errors should be minimized as much as possible.
9.1.2.1 Noise—Active equipment in the same environment as the pose measurement system may create noise that interferes (for
example, electromagnetic noise) with the performance of the pose measurement system. Environmental factors may introduce
noise that may also affect the performance of the pose measurement system.
E2919 − 22
9.1.2.2 Registration Error—Registration processes contribute toward the final measurement error, and the magnitude of the
registration error may differ depending on the registration method used.
9.1.2.3 Vibration—Sensor and object vibration during the test introduces distortion into the measurement results.
9.1.3 For a given sample pose, both the RS and SUT should measure the reference object’s pose simultaneously from their
respective fixed poses. When testing in conditions where it is not possible for the RS and SUT to measure simultaneously, the
reference measurement and the measurement from the SUT should be obtained as close together in time as possible. The SUT, RS,
and reference object should not be moved during the intermittent time span until both measurements have been collected. The
environmental conditions should be as consistent as possible and should be within the rated conditions of the RS and SUT over
the entire period of the test.
9.2 Test Sequence—The basic test procedure consists of obtaining measurement results from within the work volume of the pose
measurement system according to the six steps in 9.2.1 through 9.2.6. Testers may choose to either measure randomly the pose
of a selected object within a user-specified subset of the work volume of the SUT (for example, a user’s application may only
require that poses be measured in one or more subregions of the work volume) or measure randomly the pose of the object
throughout the entire work volume. The number of random pose measurements shall be as large as practical for the given SUT
and RS considering the cost and complexity of acquiring pose measurements. The number of random pose measurements shall not
be less than N = 32.
9.2.1 Step 1—Set up the RS and the pose measurement SUT at fixed locations according to the vendors’ specifications.
9.2.2 Step 2—Randomly select a pose for the object. This pose will be measured by the SUT and the RS, and measurement results
will consist of measured values for position (x,y, and z) and orientation (a 3 × 3 rotation matrix, R, see Section 8).
9.2.3 Step 3—Calculate the measurement errors for the translation and rotation, either absolute (Eq 7 and Eq 8, respectively) or
relative (Eq 11 and Eq 12, respectively), as per Section 8 for the selected pose of the object as observed by the SUT.
9.2.4 Step 4—Perform Steps 2 and 3 for N sample locations within the work volume to generate a collection of measurement
errors, e , e , …, e .
1 2 N
9.2.5 Step 5—Calculate the average measurement error, ē, as:
N
e
( k
k51
e¯ 5 (15)
N
Compute the variance, s , using:
N
e 2 e¯
~ !
( k
k51
s 5 (16)
N 2 1
9.2.6 Step 6—Analyze the measurement results, e ,ē, and s , to determine the performance of the SUT with regard to the vendor’s
k
specifications pertaining to the user’s application per Section 10.
10. Performance Evaluation
10.1 This section is specifically for the application of this test method for performance evaluation pertaining to the user’s
application. Four performance limits are used in this test method for performance evaluation, and a statistical test is described for
each in the following sections.
10.2 Introduction:
10.2.1 After the data has been collected as specified in Section 9 and the error associated with each data point calculated as
described in Section 8, the results shall be evaluated. Performance evaluation takes the form of using statistical tests to verify
whether the SUT is operating within the vendor’s claimed performance limits.
E2919 − 22
10.2.2 A vendor’s performance specification is verified if the performance tests in this standard accept the null hypothesis with
a p-value of greater than 0.95. The analysis is described in statistical terms as a combination of null and alternative hypotheses,
written as H and H , respectively. In Table 1, the four statistical tests used in this test method are described in terms of the null
0 a
and alternative hypotheses. For example, Test I, the Average Error Test, applies to the expected average error, E[ē], and the
vendor’s specified performance limit. If H is true in a statistical sense, then measurement results obtained from the SUT are
expected to be less than the vendor’s specified performance limit, δ , so the performance specification is accepted. In this case,
avg
the SUT is referred to as being within the vendor’s performance specifications. Alternatively, if H is true in a statistical sense, then
a
measurement results obtained from the SUT are not expected to be less than the vendor’s performance specification limit, δ . In
avg
this case, the SUT is referred to as being outside of the vendor’s performance specifications.
10.2.3 In Table 1, the four tests used in this test method are listed with their associated performance specifications. The vendor’s
performance specifications are:
δ = The vendor’s specified performance limit on the expected average error, E[ē];
avg
δ = The upper bound on the vendor-specified pth quantile of the average error, q ;
quan p
δ = The maximum average error, e ; and
max max
σ = The vendor’s specified performance limit on the variance of the average error, σ .
10.2.4 In X1.1, a more detailed explanation of performance acceptance/rejection with regard to Tests I and II is provided. In
particular, for Test II, if the experiment were repeated many times, 100 × p percent of the trials will be less than δ . When p
quan
= 0.5, Test II is a statement about the median error. An explanation of how one can determine the appropriate test for a given
application is given in X1.1.
10.3 Evaluating Performance—This section describes the procedure for determining if the performance of the SUT is within the
vendor’s specifications for δ , δ , δ , and σ . In the following subsections, the processes for determining whether the
avg quan max 0
performance of the SUT is within the vendor’s specifications based on Tests I through IV are summarized.
10.3.1 Average Error Test:
10.3.1.1 With the assumption that the measurement error is normally distributed (see Appendix X1), using the Z-test, the SUT is
not within the vendor’s performance specifications if the following is true:
e¯ 2δ
avg
.Z (17)
α
=s ⁄N
where:
e¯ = average measurement error computed using Eq 15,
s = sample variance defined in Eq 16, and
Z = value at which the cumulative distribution function for the standard normal distribution has the value 0.95 (see Ref 4).
α
Specifically Z = 1.6449.
α
10.3.1.2 See X1.3 for a more detailed explanation of the value of the Z test.
10.3.2 Quantile Error Test:
10.3.2.1 Let T be equal to the number of elements of {e , ., e } for which e ≤ δ is true. Using the Sign Test, the performance
1 N k quan
of the SUT is not within the vendor’s specifications if the following is true:
T # b (18)
N,α
TABLE 1 Statistical Tests for the Analysis of Pose Measurement Systems
Test Test Name Null Hypothesis Alternative Hypothesis
I Average error test H : E[e¯] ≤ δ H : E[e¯] > δ
0 avg a avg
II Quantile error test H : q # δ H : q > δ
0 p quan a p quan
III Maximum permissible error test H : e # δ H : e > δ
0 max max a max max
2 2 2 2
IV Precision error test
H : σ # σ H : σ > σ
0 0 a 0
E2919 − 22
where:
b = upper quantile of a binomial Probability Density Function (PDF) with parameters N and α = p.
N,α
10.3.2.2 See X1.4 for details on the Sign Test and how b is calculated.
N,α
10.3.3 Maximum Permissible Error Test—Order the observations {e , ., e } from smallest to largest and let e and e be the
1 N L S
largest and second largest observations, respectively. The performance of the SUT is not within the vendor’s specifications if the
following is true:
δ 2 e α δ 2 e
max L max L
, → ,0.0526 (19)
e 2 e 12α e 2 e
L S L S
where:
α = 0.05 (see X1.5).
10.3.4 Precision Error Test—The performance of the SUT is not within the vendor’s specifications if the following is true:
~N 2 1!s
.χ (20)
2 α,N21
σ
where:
χ = value in which the cumulative distribution of the Chi-squared PDF (see Refs 4 and 5) with N – 1 degrees of freedom has
α,N21
a probability of 1 – α = 0.95.
11. Report
11.1 The following subsections summarize the mandatory and optional information to be reported. An example form layout is
provided in Appendix X2.
11.1.1 Mandatory Information:
11.1.1.1 The following information shall be included in the test report:
(1) Testing conditions:
(a) Report author name, company, position, e-mail address and telephone number.
(b) Report author signature and date signed.
(c) Facility name, street address, city, state or province and country.
(d) Test date (month/day/year).
(e) Total time to perform the test.
(f) Portion of total time for initial set-up (including sensor warm-up).
(g) System Under Test (SUT) Settings:
(i) SUT manufacturer, model number, serial number,
(ii) Date calibrated,
(iii) Operator name,
(iv) System settings.
(h) Reference System (RS) Settings:
(i) Reference Instrument manufacturer, model number, serial number,
(ii) Date calibrated and reference to supporting documentation on file,
(iii) Specified measurement uncertainty,
(iv) Operator name,
(v) System settings.
(i) Ambient Test Conditions:
(i) Range of ambient temperature during test (____°C to ____°C),
(ii) Maximum rate of ambient temperature change during test. (____°C per minute),
(iii) Relative ambient humidity during test (____%),
(iv) Any particulate matter in air (y/n) ____,
(v) Approximate average ambient illumination on the object during test (____ lumens),
(vi) Primary ambient illumination source type on object (for example, sun, fluorescent, incandescent).
E2919 − 22
(j) Object Characteristics (be as specific as possible in order to be able to uniquely identify and reproduce the testing
conditions):
(i) Attach a picture of the object,
(ii) General description of object shape and material(s) from which it is made,
(iii) Minimum enclosing bounding box dimensions in (m),
(iv) Object primary surface feature types (for example, holes, slots, pillars, or convexities),
(v) Object surface predominant color(s),
(vi) Object surface qualitative deposited particle (for example, rust, or dirt) condition (approximate average particle size)
in (mm),
(vii) Object qualitative surface moisture condition (dry, damp, or wet),
(viii) Other material on surface, if any (such as oil or machining fluid or coating)—specify material composition and
approximate average thickness.
(k) Optional Object Characteristics:
(i) Object surface reflectance at the sensor’s wavelengths (____% to ____%),
(ii) Object surfaces scattering at wavelength(s) employed by sensor system (____% to ____%),
(iii) Object approximate surface optical absorption and secondary reflection (if any) at the sensor’s wavelengths (%),
(iv) Object surfaces roughness (Ra) in micrometers or specify other standard surface roughness metric (for example ASME
B46.1: “Surface Texture (Surface Roughness, Waviness, and if an Ra value is not available.
(2) Metrics used—Relative pose error, absolute pose error, or both.
(3) For all pose measurements:
(a) Reference pose,
(b) Measured pose,
(c) Translation error for each metric used, and
(d) Orientation error for each metric used.
(4) Average errors for each repetition.
(5) Performance evaluation:
(a) Name of statistical test performed,
(b) Computed value and vendor specified performance limit, and
(c) Result—Within the Vendor’s Performance Specifications, or Not Within the Vendor’s Performance Specifications.
11.1.1.2 The report shall be formatted so that hard copies of test reports include the page number and total number of pages.
11.1.2 Optional Information—If the absolute pose was evaluated, describe the method (for example, measuring targets, feature
matching) used to register the RS to the SUT.
12. Precision and Bias
12.1 No information can be presented on the precision or bias of the procedure in Test Method E2919 for measuring 6DOF pose
measurement system performance because no particular reference system or reference object is specified. The purpose of Test
Method E2919 is to evaluate the vendor’s specifications for the performance of its system under the conditions of the user’s
application. It is expected that the precision and bias will vary under different testing conditions.
13. Keywords
13.1 absolute pose error; performance evaluation; pose measurement system; pose measurement test procedure; relative pose
error; 6DOF; static pose measurement performance; 3D imaging system
E2919 − 22
APPENDIXES
(Nonmandatory Information)
X1. STATISTICAL TESTS
X1.1 The pth-quantile of a continuous and positive random variable X with probability density function f(x) is that value q
satisfying:
q
k 5 f~x!dx 5 F~q! (X1.1)
*
where:
F(q) = distribution function of X.
X1.1.1 The mean of X is the value μ satisfying:
`
μ 5 xf x dx (X1.2)
* ~ !
X1.1.2 The variance of X is the value σ satisfying:
`
2 2
σ 5 x 2 μ f x dx (X1.3)
* ~ ! ~ !
X1.1.3 If random observations X , X , ., X are taken on, X, then the mean, μ, is usually estimated by the sample average:
1 2 n
n
¯
X 5 X (X1.4)
( i
n
i51
and the variance, σ , is usually estimated by:
n
2 ¯
~ !
s 5 X 2 X (X1.5)
( i
n 2 1
i51
as the s
...