ASTM E3311/E3311M-22

Standard Test Method for Evaluating Ground Robot Capabilities and Remote Operator Proficiency: Obstacles: Variable Height Rails

SIGNIFICANCE AND USE
5.1 This test method is part of an overall suite of related test methods that provide repeatable measures of robotic system mobility and remote operator proficiency. The variable height rail obstacle challenges robotic system locomotion, suspension systems to maintain traction, rollover tendencies, high-centering tendencies, self-righting (if necessary), chassis shape variability (if available), and remote situational awareness by the operator. As such, the variable height rail obstacle can be used to represent obstacles in the environment, such as railroad tracks, curbs, and debris.  
5.2 The scale of the apparatus can vary to provide different constraints representative of typical obstacle spacing in the intended deployment environment. For example, the three configurations can be representative of repeatable complexity for unobstructed obstacles (open configuration), relatively open parking lots with spaces between cars (rectangular confinement configuration), or within bus, train, or plane aisles, or dwellings with hallways and doorways (square confinement configuration).  
5.3 The test apparatuses are low cost and easy to fabricate so they can be widely replicated. The procedure is also simple to conduct. This eases comparisons across various testing locations and dates to determine best-in-class systems and operators.  
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. The variable height rail obstacle can also be embedded into operational training scenarios to measure degradation due to uncontrolled variables in lighting, weather, radio communications, GPS accuracy, etc.  
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with existing capability limits.  
5.6 Training—This test method can be used to focus...
SCOPE
1.1 This test method is intended for remotely operated ground robots operating in complex, unstructured, and often hazardous environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of a robot to negotiate an obstacle in the form of variable height rail. This test method is one of several related obstacle tests that can be used to evaluate overall system capabilities.  
1.2 The robotic system includes a remote operator in control of most functionality, so an onboard camera and remote operator display are typically required. This test method can be used to evaluate assistive or autonomous behaviors intended to improve the effectiveness or efficiency of remotely operated systems.  
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various mission requirements.  
1.4 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions can be implemented.  
1.5 Units—The International System of Units (a.k.a. SI Units) and U.S. Customary Units (a.k.a. Imperial Units) are used throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of units to enable use of readily available materials in different countries. The differences between the stated dimensions in each system of units are insignificant for the purposes of comparing test method results, so each system of units is separately considered standard within this test method.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.7 This international standard was developed in accordance wit...

General Information

• Status: Published
• Publication Date: 30-Apr-2022
• ICS: 25.040.30 - Industrial robots. Manipulators
• Technical Committee: E54 - Homeland Security Applications
• Drafting Committee: E54.09 - Response Robots

Relations

• Refers: ASTM E2592-16 - Standard Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force Equipment Caches
• Effective Date: 01-Jan-2016
• Refers: ASTM E2592-07 - Standard Practice for Evaluating Cache Packaged Weight and Volume of Robots for Urban Search and Rescue
• Effective Date: 01-Oct-2007
• Refers: ASTM E2521-07a - Standard Terminology for Urban Search and Rescue Robotic Operations
• Effective Date: 01-Aug-2007
• Refers: ASTM E2521-07 - Standard Terminology for Urban Search and Rescue Robotic Operations
• Effective Date: 01-Feb-2007

Overview

ASTM E3311/E3311M-22: Standard Test Method for Evaluating Ground Robot Capabilities and Remote Operator Proficiency: Obstacles: Variable Height Rails defines a repeatable methodology to measure the mobility of ground robots and the proficiency of their remote operators in the presence of obstacles similar to those encountered in real-world environments. Developed by ASTM Committee E54 on Homeland Security Applications, this standard is part of a comprehensive suite of test methods aimed at enhancing the consistency and comparability of response robot evaluations.

The core of this standard is a test method centered around “variable height rail” obstacles, which simulate environmental features such as railroad tracks, curbs, and debris. Through standardized apparatuses and test procedures, organizations can assess and compare robotic system mobility, suspension performance, operator proficiency, and system reliability under varying operational scenarios.

Key Topics

• Variable Height Rail Obstacles: Challenges robot locomotion, suspension system traction, rollover and high-centering tendencies, and operator situational awareness.
• Test Apparatus Design: Scalable and easy-to-fabricate obstacle courses with open, rectangular, or square confinement configurations to mimic diverse operational settings.
• Remote Operator Proficiency: Shapes operator training by evaluating control skills using onboard cameras and remote displays, including testing autonomous or assistive behaviors.
• Performance Metrics: Includes completeness score (successful repetitions), reliability (successful attempts/total attempts), and efficiency (repetitions per minute).
• Environmental Representation: Apparatus scale and setup can simulate environments from open spaces to cluttered interiors, supporting targeted assessment relevant to intended robot deployments.
• Safety Provisions: Highlights requirements for emergency stops, personal protective equipment, and procedures to ensure safe operation and minimize hazards.

Applications

ASTM E3311/E3311M-22 offers practical value across several fields:

• Procurement and Product Evaluation: Supports informed purchasing decisions and system acceptance testing by providing objective, comparable test results. Vendors and users can validate robots against predefined performance thresholds aligned with specific mission needs.
• Training and Skill Assessment: Enables organizations to measure, track, and improve remote operator proficiency. Regular, repeatable testing helps maintain perishable skills and compare performance across operators, teams, or deployment regions.
• Research and Development: Guides manufacturers and researchers to identify performance trade-offs, inspire innovation, and refine robot design for improved obstacle negotiation and operator support.
• Operational Readiness: Embeds variable rail obstacles within broader training or readiness exercises to quantify impact from variables such as lighting, communications, or environmental degradation.
• Global Applicability: Standardized use of both SI and U.S. customary units allows for worldwide adoption, accounting for differences in materials and construction methods.

Related Standards

For a holistic approach to ground robot evaluation and interoperability, consider these associated standards and resources:

• ASTM E2521 - Terminology for Evaluating Response Robot Capabilities
• ASTM E2592 - Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force Equipment Caches
• NIST SP 1011–I–2.0 - Autonomy Levels for Unmanned Systems (ALFUS) Framework

Keywords: ground robot test, robot obstacle course, operator proficiency, ASTM robotics standard, variable height rail, robot mobility evaluation, mobile robot performance, response robot operations, robot procurement standards, operator training, obstacle negotiation

This structured and repeatable evaluation method helps users ensure ground robots and their operators are capable of meeting the demands of complex, dynamic, and hazardous environments, supporting safer, more effective deployment in critical applications.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation:E3311/E3311M −22
Standard Test Method for
Evaluating Ground Robot Capabilities and Remote Operator
Proficiency: Obstacles: Variable Height Rails
ThisstandardisissuedunderthefixeddesignationE3311/E3311M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
Theroboticscommunityneedswaystomeasurewhetheraparticularrobotiscapableofperforming
specific missions in complex, unstructured, and often hazardous, environments. These missions
require various combinations of elemental robot capabilities. Each capability can be represented as a
test method with an associated apparatus to provide tangible challenges for various mission
requirements and performance metrics to communicate results. These test methods can then be
combined and sequenced to evaluate essential robot capabilities and remote operator proficiencies
necessary to successfully perform intended missions.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations.These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Obstacles suite of test methods.
1. Scope usedtoevaluateassistiveorautonomousbehaviorsintendedto
improve the effectiveness or efficiency of remotely operated
1.1 This test method is intended for remotely operated
systems.
ground robots operating in complex, unstructured, and often
hazardous environments. It specifies the apparatuses, 1.3 Different user communities can set their own thresholds
of acceptable performance within this test method for various
procedures, and performance metrics necessary to measure the
capability of a robot to negotiate an obstacle in the form of mission requirements.
variable height rail. This test method is one of several related
1.4 Performing Location—This test method may be per-
obstacle tests that can be used to evaluate overall system
formed anywhere the specified apparatuses and environmental
capabilities.
conditions can be implemented.
1.2 Theroboticsystemincludesaremoteoperatorincontrol
1.5 Units—The International System of Units (a.k.a. SI
of most functionality, so an onboard camera and remote
Units) and U.S. Customary Units (a.k.a. Imperial Units) are
operatordisplayaretypicallyrequired.Thistestmethodcanbe
used throughout this document. They are not mathematical
conversions. Rather, they are approximate equivalents in each
system of units to enable use of readily available materials in
This test method is under the jurisdiction of ASTM Committee E54 on
Homeland Security Applications and is the direct responsibility of Subcommittee
different countries. The differences between the stated dimen-
E54.09 on Response Robots.
sions in each system of units are insignificant for the purposes
Current edition approved May 1, 2022. Published May 2022. Originally
of comparing test method results, so each system of units is
approved in 2021. Last previous edition approved in 2021 as E3311/E3311M–21.
DOI: 10.1520/E3311_E3311M-22. separately considered standard within this test method.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3311/E3311M−22
1.6 This standard does not purport to address all of the 3.2 Thefollowingtermsareusedinthistestmethodandare
safety concerns, if any, associated with its use. It is the defined in ALFUS Framework Volume I:3: autonomous,
responsibility of the user of this standard to establish appro- autonomy, level of autonomy, operator control unit (OCU),and
priate safety, health, and environmental practices and deter- semi-autonomous.
mine the applicability of regulatory limitations prior to use.
3.3 Definitions of Terms Specific to This Standard:
1.7 This international standard was developed in accor-
3.3.1 apparatus clearance width (W),n—a specification for
dance with internationally recognized principles on standard-
the apparatus dimensions chosen from one of four possible
ization established in the Decision on Principles for the
measurements, based on the intended robot deployment envi-
Development of International Standards, Guides and Recom-
ronment:
mendations issued by the World Trade Organization Technical
240 cm 6 2.5 cm tolerance [96 in. 6 1 in. tolerance], such
Barriers to Trade (TBT) Committee.
as open and outdoor public spaces;
120 cm 6 2.5 cm tolerance [48 in. 6 1 in. tolerance], such
2. Referenced Documents
as indoor spaces in accessibility-compliant buildings;
2.1 ASTM Standards:
60 cm 6 1.3 cm tolerance [24 in. 6 0.5 in. tolerance],
E2521Terminology for Evaluating Response Robot Capa-
residences and aisles of public transportation; or
bilities
30 cm 6 1.3 cm tolerance [12 in. 6 0.5 in. tolerance],
E2592Practice for Evaluating Response Robot Capabilities:
cluttered indoor spaces, ductwork, and voids in collapsed
Logistics: Packaging for Urban Search and Rescue Task
structures.
Force Equipment Caches
3.3.1.1 Discussion—The measures for these scales are
2.2 Other Standard:
nominalanddonotrepresentthemeasurementofthenarrowest
NIST Special Publication 1011–I–2.0Autonomy Levels for
point in the apparatus through which the robot should pass.
Unmanned Systems (ALFUS) Framework, Volume 1:
Consult the Section 6 (Apparatus) for the overall measure-
Terminology, Version 2.04
ments and dimensions of the apparatus at each scale.
3.3.2 subfloor, n—an underlayment of OSB or similar ma-
3. Terminology
terial with dimensional lumber borders used to affix multiple
3.1 Definitions—The following terms are used in this test
subfloors to one another and can contain apparatus elements,
method and are defined in Terminology E2521: abstain,
such as terrains or obstacles.
administrator or test administrator, emergency response robot
3.3.3 variable height rail, n—a solid piece of dimensional
or response robot, fault condition, operator, operator station,
lumber that is sized to fit inside a subfloor at a 45° angle.
remote control, repetition, robot, teleoperation, test event or
event, test form, test sponsor, test suite, testing target or target,
4. Summary of Test Method
testing task or task, and trial or test trial.
4.1 This test method is performed by a remote operator, out
of sight and sound of the robot, while controlling the robot
withinthetestapparatus.Therobottraversesthroughadefined
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
area to negotiate the variable height rail obstacle with or
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
without walls for confinement (see Fig. 1). This test method
Standards volume information, refer to the standard’s Document Summary page on
requires the robot to overcome challenges such as pitch, roll,
the ASTM website.
Available from National Institute of Standards and Technology (NIST), 100
traction, and control of variable chassis shape and articulators.
Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov.
FIG. 1 (A) View of the Variable Height Rail; Shown Heights: 10 cm [4 in.], 20 cm [8 in.], 30 cm [12 in.]
E3311/E3311M−22
FIG. 1 (B) View of the Variable Height Rail Showing the Open, Rectangular, and Square Confinement Approach/exit Areas and
Example Robot Traversal Paths (continued)
4.2 Therobottraversesa pathasshowninFig.1.Therobot areas. The open configuration is representative of operating in
starts on theA-side of the apparatus, crosses to the B-side into unobstructed areas. The rectangular confinement and square
the nearest approach area, traverses over the variable height confinement configurations use walls around the approach/exit
rail obstacle to the exit area on the opposite end of the areas. The walls are used to define the robot’s path and are
apparatus, and then crosses to the A-side to complete each representative of operating in a confined environment. The
repetition. All repetitions alternate directions through the square configuration has half of the available area as the
apparatus. rectangular configuration.
4.3 Therobottraversesthepathinoneoftwooperationally- 4.5 Potential Faults Include:
relevant driving orientations: unrestricted or forward/reverse. 4.5.1 Any contact by the robot with the apparatus that
Unrestrictedallowstherobottotraversethepathinanydriving requires adjustment or repair to return the apparatus to the
orientation throughout the test. Forward/reverse requires the initial condition; and
robot to alternate, for each repetition, driving in forward and 4.5.2 Anyvisual,audible,orphysicalinteractionthatassists
reverse. As repetitions also alternate directions through the either the robot or the remote operator.
apparatus, this means that the robot shall not rotate between
4.6 Testtrialsshallproduceenoughsuccessfulrepetitionsto
endingonerepetitionandstartingthenext.Resultingdatafrom
demonstrate the reliability of the system capability or the
thetwodrivingorientationsarenotcomparabletooneanother.
remote operator proficiency to the desired level of statistical
4.4 There are three apparatus configurations: open, rectan- significance (see Section 9). A complete trial of 10 to 30
gular confinement, and square confinement.Inthe open repetitions should take 10 to 30 min to complete. When
configuration, no walls are used around the approach/exit measuring system capabilities, it is important to allow enough
E3311/E3311M−22
time to capture a complete trial with an expert operator. When andmeasurethereliabilityofsystemsperformingspecifictasks
measuringoperatorproficiency,itisimportanttolimitthetime within an overall mission sequence. Combining or sequencing
ofthetrialsothatexpertoperatorshaveampletimetoperform multiple test methods can guide manufacturers toward imple-
a statistically significant set of repetitions while novice opera- menting the combinations of capabilities necessary to perform
tors are not excessively fatigued. There are three metrics to essential mission tasks.
considerwhencalculatingtheresultsofatesttrial.Theyshould
6. Apparatus
be considered in the following order of importance: complete-
6.1 The apparatus consists of subfloors, walls (only for
ness score, reliability, and efficiency. The results from open,
rectangular confinement and square confinement
rectangular confinement, and square confinement configura-
configurations), and the variable height rail obstacle (see Fig.
tions are not comparable because they represent different
2). The main apparatus dimension to consider is the apparatus
difficulties and clearances.
clearance width (W) for the robot, which can be set to 240 cm
[96 in.] with 62.5 cm [1 in.] tolerance, 120 cm [48 in).] with
5. Significance and Use
62.5 cm [1 in.] tolerance, 60 cm [24 in.] with 61.3 cm [0.5
5.1 Thistestmethodispartofanoverallsuiteofrelatedtest
in.] tolerance, or 30 cm [12 in.] with 61.3 cm [0.5 in.]
methods that provide repeatable measures of robotic system
tolerance. The dimension chosen for W should represent the
mobility and remote operator proficiency. The variable height
intended deployment environment or should be based on the
rail obstacle challenges robotic system locomotion, suspension
size of the robot, or both, (that is, the robot shall be able to
systems to maintain traction, rollover tendencies, high-
maneuverwithintheselecteddimensionsoftheapparatus).All
centeringtendencies,self-righting(ifnecessary),chassisshape
apparatus dimensions scale proportionally with W (see Fig. 3
variability (if available), and remote situational awareness by
and Fig. 4). For example, the width of the variable height rail
the operator. As such, the variable height rail obstacle can be
obstacle apparatus is 1W, and the length of the variable height
usedtorepresentobstaclesintheenvironment,suchasrailroad
rail obstacle apparatus is either 3W (square confinement
tracks, curbs, and debris.
configuration) or 5W (rectangular confinement and open con-
5.2 The scale of the apparatus can vary to provide different figurations). The equipment required to perform this test
constraints representative of typical obstacle spacing in the
methodincludestheapparatusandatimer.Resultingdatafrom
intended deployment environment. For example, the three a specific minimum clearance width of the apparatus is not
configurations can be representative of repeatable complexity
comparable to data from other apparatuses with different
for unobstructed obstacles (open configuration), relatively minimum clearance widths.
open parking lots with spaces between cars (rectangular
6.2 The apparatus consists of two symmetrical approach/
confinementconfiguration),orwithinbus,train,orplaneaisles,
exitareasoneithersideofavariableheightrailobstacle.There
or dwellings with hallways and doorways (square confinement
are three configurations of the apparatus: open, rectangular
configuration).
confinement, and square confinement (see Fig. 3). The selec-
tion of apparatus configuration should correspond to intended
5.3 The test apparatuses are low cost and easy to fabricate
so they can be widely replicated. The procedure is also simple deployment environment.The open configuration does not use
wallsintheapproachareasoneithersideofthevariableheight
to conduct. This eases comparisons across various testing
locations and dates to determine best-in-class systems and rail obstacle, allowing for unobstructed robot movement. The
approach areas in the rectangular confinement configuration
operators.
measure 2W by 1W and are bounded by walls taller than the
5.4 Evaluation—This test method can be used in a con-
robot to obstruct robot movement. The approach areas in the
trolled environment to measure baseline capabilities. The
square confinement configuration measure 1W by 1W and are
variable height rail obstacle can also be embedded into
bounded by walls taller than the robot to further obstruct robot
operational training scenarios to measure degradation due to
movement. Resulting data from a specific configuration of the
uncontrolled variables in lighting, weather, radio
apparatus is not comparable to data from other apparatuses
communications, GPS accuracy, etc.
with different configurations.
5.5 Procurement—This test method can be used to identify
6.3 Variable Height Rail—The variable height rail obstacle
inherent capability trade-offs in systems, make informed pur-
is constructed of one or more pieces of dimensional lumber
chasing decisions, and verify performance during acceptance
attached to the top of the subfloor at a 45° angle. The variable
testing. This aligns requirement specifications and user expec-
height rail is contained within a 1W length of the apparatus
tations with existing capability limits.
(see Fig. 5).The thickness of the variable height rail is relative
5.6 Training—This test method can be used to focus opera-
to the scale of the apparatus (see Table 1). The overall height
tortrainingasarepeatablepracticetaskorasanembeddedtask
of the variable height rail is adjustable by stacking multiple
within training scenarios. The resulting measures of remote
piecesofdimensionallumberontopofeachother.(SeeFig.5.)
operator proficiency enable tracking of perishable skills over Duetovariationsindimensionallumberitmaybenecessaryto
time, along with comparisons of performance across squads,
includeadditionallumbertoensureproperoverallheightofthe
regions, or national averages.
variable diagonal rail.
5.7 Innovation—This test method can be used to inspire 6.4 Subfloor—The subfloor’s surface is constructed of OSB
technical innovation, demonstrate break-through capabilities, or similar material and rails are dimensional lumber that
E3311/E3311M−22
FIG. 2 View of a Test Apparatus with Labeled Components
FIG. 3 Testing Apparatus is Scalable to Represent Different Environments
surround the border of the subfloor. Each subfloor is 2W by 6.5 Walls to Confine the Robot Path—The walls placed
1W. A gap in the rails halfway along the side measuring 2W within the rectangular confinement and square confinement
will allow a containment wall to be inserted. (See Fig. 6.) configurations provide physical and visual guidance for the
E331
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E3311/E3311M − 21 E3311/E3311M − 22
Standard Test Method for
Evaluating Response Robot Mobility Using Variable
Diagonal Rail ObstaclesGround Robot Capabilities and
Remote Operator Proficiency: Obstacles: Variable Height
Rails
This standard is issued under the fixed designation E3311/E3311M; the number immediately following the designation indicates the year
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous, environments. These missions
require various combinations of elemental robot capabilities. Each capability can be represented as a
test method with an associated apparatus to provide tangible challenges for various mission
requirements and performance metrics to communicate results. These test methods can then be
combined and sequenced to evaluate essential robot capabilities and remote operator proficiencies
necessary to successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the MobilityObstacles suite of test methods.
1. Scope
1.1 This test method is intended for remotely operated ground robots operating in complex, unstructured, and often hazardous
environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of a robot to
negotiate an obstacle in the form of variable diagonalheight rail. This test method is one of several related mobilityobstacle tests
that can be used to evaluate overall system capabilities.
1.2 The robotic system includes a remote operator in control of most functionality, so an onboard camera and remote operator
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved Nov. 15, 2021May 1, 2022. Published December 2021May 2022. Originally approved in 2021. Last previous edition approved in 2021 as
E3311/E3311M – 21. DOI: 10.1520/E3311_E3311M-21.10.1520/E3311_E3311M-22.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3311/E3311M − 22
display are typically required. This test method can be used to evaluate assistive or autonomous behaviors intended to improve the
effectiveness or efficiency of remotely operated systems.
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various mission
requirements.
1.4 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions
can be implemented.
1.5 Units—The International System of Units (a.k.a. SI Units) and U.S. Customary Units (a.k.a. Imperial Units) are used
throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of
units to enable use of readily available materials in different countries. The differences between the stated dimensions in each
system of units are insignificant for the purposes of comparing test method results, so each system of units is separately considered
standard within this test method.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E2521 Terminology for Evaluating Response Robot Capabilities
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
2.2 Other Standard:
NIST Special Publication 1011–I–2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework, Volume 1: Terminology,
Version 2.04
3. Terminology
3.1 Definitions—The following terms are used in this test method and are defined in Terminology E2521: abstain, administrator
or test administrator, emergency response robot or response robot, fault condition, operator, operator station, remote control,
repetition, robot, teleoperation, test event or event, test form, test sponsor, test suite, testing target or target, testing task or task,
and trial or test trial.
3.2 The following terms are used in this test method and are defined in ALFUS Framework Volume I:3: autonomous, autonomy,
level of autonomy, operator control unit (OCU), and semi-autonomous.
3.3 Definitions of Terms Specific to This Standard:
3.3.1 apparatus clearance width (W),n—a specification for the apparatus dimensions chosen from one of four possible
measurements, based on the intended robot deployment environment:
240 cm 6 2.5 cm tolerance [96 in. 6 1 in. tolerance], such as open and outdoor public spaces;
120 cm 6 2.5 cm tolerance [48 in. 6 1 in. tolerance], such as indoor spaces in accessibility-compliant buildings;
60 cm 6 1.3 cm tolerance [24 in. 6 0.5 in. tolerance], residences and aisles of public transportation; or
30 cm 6 1.3 cm tolerance [12 in. 6 0.5 in. tolerance], cluttered indoor spaces, ductwork, and voids in collapsed structures.
3.3.1.1 Discussion—
The measures for these scales are nominal and do not represent the measurement of the narrowest point in the apparatus through
which the robot should pass. Consult the Section 6 (Apparatus) for the overall measurements and dimensions of the apparatus at
each scale.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov.
E3311/E3311M − 22
3.3.2 subfloor, n—an underlayment of OSB or similar material with dimensional lumber borders used to affix multiple subfloors
to one another and can contain apparatus elements, such as terrains or obstacles.
3.3.3 variable diagonalheight rail, n—a solid piece of dimensional lumber that is sized to fit inside a subfloor at a 45° angle.
4. Summary of Test Method
4.1 This test method is performed by a remote operator that cannot see or hear the operator, out of sight and sound of the robot,
while controlling the robot within the test apparatus. The robot traverses through a defined area to negotiate the variable
diagonalheight rail obstacle with or without walls for confinement (see Fig. 1). This test method requires the robot to overcome
challenges such as pitch, roll, traction, and control of variable chassis shape and articulators.
4.2 The robot traverses a path as shown in Fig. 1. The robot starts on the A-side of the apparatus, crosses to the B-side into the
nearest approach area, traverses over the variable diagonalheight rail obstacle to the exit area on the opposite end of the apparatus,
and then crosses to the A-side to complete each repetition. All repetitions alternate directions through the apparatus.
4.3 The robot traverses the path in one of two operationally-relevant driving orientations: unrestricted or forward/
reverse.Unrestricted allows the robot to traverse the path in any driving orientation throughout the test. Forward/reverse requires
the robot to alternate, for each repetition, driving in forward and reverse. As repetitions also alternate directions through the
apparatus, this means that the robot shall not rotate between ending one repetition and starting the next. Resulting data from the
two driving orientations are not comparable to one another.
4.4 There are three apparatus configurations: open,rectangular confinement, and square confinement. In the open configuration,
no walls are used around the approach/exit areas. The open configuration is representative of operating in unobstructed areas. The
rectangular confinement and square confinement configurations use walls around the approach/exit areas. The walls are used to
define the robot’s path and are representative of operating in a confined environment. The square configuration has half of the
available area as the rectangular configuration.
4.5 Potential Faults Include:
4.5.1 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial condition;
and
4.5.2 Any visual, audible, or physical interaction that assists either the robot or the remote operator.
4.6 Test trials shall produce enough successful repetitions to demonstrate the reliability of the system capabilities testing requires
FIG. 1 (A) View of the Variable DiagonalHeight Rail; Shown Heights;Heights: 10 cm [4 in.], 20 cm [8 in.], 30 cm [12 in.]
E3311/E3311M − 22
FIG. 1 (B) View of the Variable DiagonalHeight Rail Showing the Open, Rectangular, and Square Confinement Approach/exit Areas and
Example Robot Traversal Paths (continued)
capability or the remote operator proficiency to the desired level of statistical significance (see Section 9enough time for an expert
operator to ). A complete trial of 10 to 30 repetitions should take 10 to 30 min to complete. When measuring system capabilities,
it is important to allow enough time to capture a complete trial with an expert operator. When measuring operator proficiency, it
is important to limit the time of the trial so that expert operators have ample time to perform a statistically significant set of
repetitions, which is typically 10 to 30 repetitions. The total number of repetitions to perform is based on the desired percent of
reliability and confidence in the results.repetitions while novice operators are not excessively fatigued. There are three metrics to
consider when calculating the results of a test trial. They should be considered in the following order of importance: completeness
score, reliability, and efficiency. The results from open, rectangular confinement, and square confinement configurations are not
comparable because they represent different difficulties and clearances.
4.6.1 The resulting rate of successful repetitions can be directly compared to the resulting rate of any other trial no matter the
duration of each trial. The average rate is equal to the number of successful repetitions divided by the elapsed time of the trial.
The average rate is expressed in repetitions per minute.
4.6.2 Time limited trials can be used to evaluate system capabilities or operator proficiency, or both, when a more tightly scheduled
test plan is necessary. Time limited trials can also reduce excessive fatigue for novices that typically take longer to complete the
tasks while learning. Time limited trials should be long enough to enable proficient operators to perform at least 10 successful
repetitions to establish their reliability with some level of confidence. The resulting average rate of success is comparable with
trials of different durations. Time limited trials of equal duration can always be compared directly to one another using the number
of successfully completed repetitions.
E3311/E3311M − 22
4.6.3 There are three metrics to consider when calculating the results of a test trial. They should be considered in the following
order of importance: completeness score, reliability, and efficiency. The results from open, rectangular confinement, and square
confinement configurations are not comparable because they represent different difficulties and clearances.
5. Significance and Use
5.1 This test method is part of an overall suite of related test methods that provide repeatable measures of robotic system mobility
and remote operator proficiency. The variable diagonalheight rail obstacle challenges robotic system locomotion, suspension
systems to maintain traction, rollover tendencies, high-centering tendencies, self-righting (if necessary), chassis shape variability
(if available), and remote situational awareness by the operator. As such, the variable diagonalheight rail obstacle can be used to
represent obstacles in the environment, such as railroad tracks, curbs, and debris.
5.2 The scale of the apparatus can vary to provide different constraints representative of typical obstacle spacing in the intended
deployment environment. For example, the three configurations can be representative of repeatable complexity for unobstructed
obstacles (open configuration), relatively open parking lots with spaces between cars (rectangular confinement configuration), or
within bus, train, or plane aisles, or dwellings with hallways and doorways (square confinement configuration).
5.3 The test apparatuses are low cost and easy to fabricate so they can be widely replicated. The procedure is also simple to
conduct. This eases comparisons across various testing locations and dates to determine best-in-class systems and operators.
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. The variable
diagonalheight rail obstacle can also be embedded into operational training scenarios to measure degradation due to uncontrolled
variables in lighting, weather, radio communications, GPS accuracy, etc.
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing
decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with
existing capability limits.
5.6 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within
training scenarios. The resulting measures of remote operator proficiency enable tracking of perishable skills over time, along with
comparisons of performance across squads, regions, or national averages.
5.7 Innovation—This test method can be used to inspire technical innovation, demonstrate break-through capabilities, and measure
the reliability of systems performing specific tasks within an overall mission sequence. Combining or sequencing multiple test
methods can guide manufacturers toward implementing the combinations of capabilities necessary to perform essential mission
tasks.
6. Apparatus
6.1 The apparatus consists of subfloors, walls (only for rectangular confinement and square confinement configurations), and the
variable diagonalheight rail obstacle (see Fig. 2). The main apparatus dimension to consider is the minimumapparatus clearance
width (W) for the robot. The minimum clearance width should be chosen to robot, which can be set to 240 cm [96 in.] with 62.5
cm [1 in.] tolerance, 120 cm [48 in).] with 62.5 cm [1 in.] tolerance, 60 cm [24 in.] with 61.3 cm [0.5 in.] tolerance, or 30 cm
[12 in.] with 61.3 cm [0.5 in.] tolerance. The dimension chosen for W should represent the intended deployment environment or
should be based on the size of the robot, or both. The minimum clearance width is typically set to 120 cm [4 ft], 60 cm [2 ft], or
30 cm [1 ft] to efficiently use available construction materials, although other apparatus sizes can be used both, (that is, the robot
shall be able to maneuver within the selected dimensions of the apparatus). All apparatus dimensions scale proportionally with W
(see Fig. 3). All apparatus dimensions scale proportionally with the minimum clearance width (see and Fig. 4). For example, the
width of the variable diagonalheight rail obstacle apparatus is 1W, and the length of the variable diagonalheight rail obstacle
apparatus is either 3W (square confinement configuration) or 5W (rectangular confinement and open configurations). The
equipment required to perform this test method includes the apparatus and a timer. Resulting data from a specific minimum
clearance width of the apparatus is not comparable to data from other apparatuses with different minimum clearance widths.
6.2 The apparatus consists of two symmetrical approach/exit areas on either side of a variable diagonalheight rail obstacle. There
are three configurations of the apparatus: open, rectangular confinement, and square confinement (see Fig. 3). The selection of
E3311/E3311M − 22
FIG. 2 View of a Test Apparatus with Labeled Components
FIG. 3 Testing Apparatus is Scalable to Represent Different Environments
apparatus configuration should correspond to intended deployment environment. The open configuration does not use walls in the
approach areas on either side of the variable diagonalheight rail obstacle, allowing for unobstructed robot movement. The approach
areas in the rectangular confinement configuration measure 2W by 1W and are bounded by walls taller than the robot to obstruct
robot movement. The approach areas in the square confinement configuration measure 1W by 1W and are bounded by walls taller
E3311/E3311M − 22
FIG. 4 Top View of a Test Apparatus Showing the Dimensions and Labeled Open, Rectangular Confinement, and Square Confinement
Approach/exit Areas
than the robot to further obstruct robot movement. Resulting data from a specific configuration of the apparatus is not comparable
to data from
...