ASTM E2828/E2828M-20

Standard Test Method for Evaluating Response Robot Mobility Using Symmetric Stepfields Terrains

SIGNIFICANCE AND USE
5.1 This test method is part of an overall suite of related test methods that provide repeatable measures of robotic system mobility and remote operator proficiency. This symmetric stepfield terrain specifically challenges robotic system locomotion, suspension systems to maintain traction, rollover tendencies, self-righting in complex terrain (if necessary), chassis shape variability (if available), and remote situational awareness by the operator. As such, it can be used to represent modest outdoor terrain complexity or indoor debris within confined areas.  
5.2 The overall size of the terrain apparatus can vary to provide different constraints depending on the typical obstacle spacing of the intended deployment environment. For example, the terrain with containment walls can be sized to represent repeatable complexity within bus, train, or plane aisles; dwellings with hallways and doorways; relatively open parking lots with spaces between cars; or unobstructed terrains.  
5.3 The test apparatuses are low cost and easy to fabricate so they can be widely replicated. The procedure is also simple to conduct. This eases comparisons across various testing locations and dates to determine best-in-class systems and operators.  
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. It can also be embedded into operational training scenarios to measure degradation due to uncontrolled variables in lighting, weather, radio communications, GPS accuracy, etc.  
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with existing capability limits.  
5.6 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within training scenarios. The resulting measures of remote opera...
SCOPE
1.1 This test method is intended for remotely operated ground robots operating in complex, unstructured, and often hazardous environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of a robot to traverse complex terrains in the form of symmetric stepfields. This test method is one of several related mobility tests that can be used to evaluate overall system capabilities.  
1.2 The robotic system includes a remote operator in control of all functionality, so an onboard camera and remote operator display are typically required. Assistive features or autonomous behaviors that improve the effectiveness or efficiency of the overall system are encouraged.  
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various mission requirements.  
1.4 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions can be implemented.  
1.5 Units—The International System of Units (SI Units) and U.S. Customary Units (Imperial Units) are used throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of units to enable use of readily available materials in different countries. This avoids excessive purchasing and fabrication costs. The differences between the stated dimensions in each system of units are insignificant for the purposes of comparing test method results, so each system of units is separately considered standard within this test method.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.7 This international standard was developed in ...

General Information

• Status: Published
• Publication Date: 29-Feb-2020
• ICS: 13.200 - Accident and disaster control
• ICS: 25.040.30 - Industrial robots. Manipulators
• Technical Committee: E54 - Homeland Security Applications
• Drafting Committee: E54.09 - Response Robots

Relations

• Replaces: ASTM E2828-11 - Standard Test Method for Evaluating Emergency Response Robot Capabilities: Mobility: Confined Area Terrains: Symmetric Stepfields
• Effective Date: 01-Mar-2020
• Refers: ASTM E2521-07a - Standard Terminology for Urban Search and Rescue Robotic Operations
• Effective Date: 01-Aug-2007
• Refers: ASTM E2521-07 - Standard Terminology for Urban Search and Rescue Robotic Operations
• Effective Date: 01-Feb-2007
• Referred By: ASTM E3380/E3380M-23 - Standard Test Method for Evaluating Ground Response Robot Endurance Using Reproducible Terrains
• Effective Date: 01-Mar-2020
• Referred By: ASTM E2853/E2853M-22 - Standard Test Method for Evaluating Ground Response Robot Capabilities: Search Tasks
• Effective Date: 01-Mar-2020

Overview

ASTM E2828/E2828M-20 is an international standard developed by ASTM International that defines a test method for evaluating the mobility of response robots using symmetric stepfield terrains. This standard is part of a suite of methods designed to provide repeatable, measurable benchmarks for remotely operated ground robots deployed in complex and hazardous environments. Specifically, it focuses on assessing robotic locomotion, operator proficiency, and core system functionalities when traversing terrains that replicate modest outdoor obstacles or indoor debris in confined areas. The method enables stakeholders to quantitatively evaluate robot performance for procurement, operator training, and system development.

Key Topics

• Robotic Mobility Assessment: This test method challenges robots to traverse specially fabricated symmetric stepfield terrains, simulating real-world debris and obstacles found in emergency and disaster environments.
• Operator Proficiency: Since remotely operated robots are evaluated, the standard also measures the operator’s effectiveness in navigating complex paths and maintaining situational awareness using onboard cameras and remote controls.
• System Capability Measurement: Metrics are provided for evaluating traction, rollover tendencies, self-righting features, and chassis adaptability, all within repeatable test conditions.
• Standardized Apparatus: The symmetric stepfield terrain apparatus is scalable and can represent environments such as hallways, vehicle aisles, open lots, or confined, debris-filled spaces. It is designed for easy fabrication and replication across different sites.
• Versatile Applications: The standard addresses both baseline capability testing in controlled settings and operational scenario embedding, enabling evaluation under varied environmental stressors such as low light, weather variables, and degraded communication.
• Procurement and Training: Provides objective data for purchasing decisions and ensures operator skills are measurable and maintainable through standardized practice scenarios.
• International Relevance: Uses both SI and U.S. customary units to encourage widespread adoption and minimize fabrication costs globally.

Applications

• Emergency Response: Core use case for evaluating robots intended for urban search and rescue, hazardous materials handling, and disaster response operations.
• Robot Procurement: Enables agencies to compare robot models using objective, repeatable mobility metrics, ensuring the best fit for specific operational needs.
• Operator Training & Certification: Offers repeatable tasks for operator skill development, performance tracking, and maintaining proficiency.
• Product Development & Testing: Assists robotics manufacturers and researchers in benchmarking designs, validating new mobility features, and supporting iterative development.
• Performance Verification: Integral in acceptance testing to ensure delivered systems meet contract requirements and user expectations.
• Scenario-based Evaluation: Allows embedding the test method into broader operational exercises to measure system and operator responses under realistic conditions.

Related Standards

• ASTM E2521: Terminology for Evaluating Response Robot Capabilities - provides definitions for terms used in response robot assessment.
• ASTM E2592: Practice for Evaluating Response Robot Capabilities: Logistics - covers packaging and logistical considerations.
• NIST SP 1011-I-2.0 (ALFUS Framework): Definitions related to autonomy levels for unmanned systems, relevant for semi-autonomous and assistive robot behaviors.
• National Response Framework (DHS): Supports alignment with national guidelines for robotics in emergency response.
• Other ASTM Mobility Standards: This standard is part of a broader suite including tests for different terrain complexities, aiding comprehensive mobility evaluation.

---

By adhering to ASTM E2828/E2828M-20, organizations can ensure consistent, actionable measurements for ground response robot mobility, leading to more effective, safer, and better-prepared emergency operations. This standard fosters innovation and reliable benchmarking in the rapidly-evolving field of emergency robotics.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation:E2828/E2828M −20
Standard Test Method for
Evaluating Response Robot Mobility Using Symmetric
Stepfields Terrains
ThisstandardisissuedunderthefixeddesignationE2828/E2828M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specificmissionsincomplex,unstructured,andoftenhazardousenvironments.Thesemissionsrequire
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Mobility Suite of test methods.
1. Scope mous behaviors that improve the effectiveness or efficiency of
the overall system are encouraged.
1.1 This test method is intended for remotely operated
ground robots operating in complex, unstructured, and often 1.3 Different user communities can set their own thresholds
hazardous environments. It specifies the apparatuses, of acceptable performance within this test method for various
procedures, and performance metrics necessary to measure the mission requirements.
capability of a robot to traverse complex terrains in the form of
1.4 Performing Location—This test method may be per-
symmetric stepfields. This test method is one of several related
formed anywhere the specified apparatuses and environmental
mobility tests that can be used to evaluate overall system
conditions can be implemented.
capabilities.
1.5 Units—The International System of Units (SI Units) and
1.2 Theroboticsystemincludesaremoteoperatorincontrol
U.S.CustomaryUnits(ImperialUnits)areusedthroughoutthis
of all functionality, so an onboard camera and remote operator
document. They are not mathematical conversions. Rather,
display are typically required. Assistive features or autono-
they are approximate equivalents in each system of units to
enable use of readily available materials in different countries.
This avoids excessive purchasing and fabrication costs. The
This test method is under the jurisdiction of ASTM Committee E54 on
Homeland Security Applications and is the direct responsibility of Subcommittee
differences between the stated dimensions in each system of
E54.09 on Response Robots.
units are insignificant for the purposes of comparing test
Current edition approved March 1, 2020. Published April 2020. Originally
methodresults,soeachsystemofunitsisseparatelyconsidered
approved in 2011. Last previous edition approved in 2011 as E2828/E2828M – 11.
DOI: 10.1520/E2828_E2828M-20. standard within this test method.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2828/E2828M−20
FIG. 1Overview of the Symmetric Stepfield Terrain Apparatus
1.6 This standard does not purport to address all of the or response robot, fault condition, operator, operator station,
safety concerns, if any, associated with its use. It is the remote control, repetition, robot, teleoperation, test event or
responsibility of the user of this standard to establish appro- event, test form, test sponsor, test suite, testing target or target,
priate safety, health, and environmental practices and deter- testing task or task, and trial or test trial.
mine the applicability of regulatory limitations prior to use.
3.2 The following terms are used in this test method and are
1.7 This international standard was developed in accor-
defined in ALFUS Framework Volume I:3: autonomous,
dance with internationally recognized principles on standard-
autonomy, level of autonomy, operator control unit (OCU),and
ization established in the Decision on Principles for the
semi-autonomous.
Development of International Standards, Guides and Recom-
mendations issued by the World Trade Organization Technical
4. Summary of Test Method
Barriers to Trade (TBT) Committee.
4.1 This test method is performed by a remote operator
2. Referenced Documents
controlling the robot out of sight and sound of robot within the
test apparatus. The robot follows one of two defined paths in
2.1 ASTM Standards:
thespecifiedterrainrequiringtherobottoovercomechallenges
E2521 Terminology for Evaluating Response Robot Capa-
including pitch, roll, traction, and turning on uneven surfaces
bilities
within open or confined spaces.
2.2 Other Standards:
National Response Framework U.S. Department of Home-
4.2 The Figure-8 Path (forward) is a continuous forward
land Security
path through the terrain with alternating left and right turns to
NIST Special Publication 1011-I-2.0 Autonomy Levels for
avoid barriers. It can be used to demonstrate terrain traversal
Unmanned Systems (ALFUS) Framework, Volume I:
over long distances within a relatively small apparatus. The
Terminology, Version 2.04
continuous traverse is shown as the white path (see Fig. 1 and
Fig. 2).
3. Terminology
4.3 The Zig-Zag Path (forward/reverse) is an end-to-end
3.1 Definitions—The following terms are used in this test
path that requires forward and reverse traversal through the
method and are defined in Terminology E2521: abstain,
terrain with alternating left and right turns to avoid barriers.
administrator or test administrator, emergency response robot
This can be used to demonstrate traversal of the terrain within
confined spaces. The down-range traverse, shown as the white
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
path, is performed in a forward orientation and the up-range
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
traverse, shown as the black path, is performed in reverse (see
Standards volume information, refer to the standard’s Document Summary page on
Fig. 1 and Fig. 3).
the ASTM website.
Available from Federal Emergency Management Agency (FEMA), P.O. Box
4.4 The robot starts on one side or the other of a lane full of
10055, Hyattsville, MD 20782-8055, http://www.fema.gov.
fabricated symmetric stepfield terrain at a chosen scale. The
Available from National Institute of Standards and Technology (NIST), 100
Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov. robot follows either the figure-8 path (forward) or the zig-zag
E2828/E2828M−20
FIG. 2Top View Showing the Figure-8 Path (Forward) Defined by the Barriers
FIG. 3Top View Showing the Zig-Zag Path (Forward/Reverse) Defined by the Barriers
path (forward/reverse) between the two barriers. The figure-8 an expert operator. When measuring operator proficiency, it is
path (forward) repetition is completed when the robot crosses important to limit the time of the trial so that novice and expert
the start/end centerline of the lane without a fault after operators are similarly fatigued.
approximately following the white path. The zig-zag path
4.7 There are three metrics to consider when calculating the
(forward/reverse) repetition is completed when the robot
results of a test trial. They should be considered in the
crosses the start/end centerline without a fault after approxi-
following order of importance: completeness score, reliability,
mately following the white and black paths.
and efficiency.The results from the figure-8 path (forward) and
4.5 Potential Faults Include: the zig-zag path (forward/reverse) are not comparable because
4.5.1 Any contact by the robot with the apparatus that they measure different capabilities. The results from different
requires adjustment or repair to return the apparatus to the scales of test apparatus are also not comparable because they
initial condition; represent different clearances and distances.
4.5.2 Any visual, audible, or physical interaction that assists
either the robot or the remote operator; 5. Significance and Use
4.5.3 Leaving the apparatus during the trial.
5.1 This test method is part of an overall suite of related test
4.6 Test trials shall produce enough successful repetitions to methods that provide repeatable measures of robotic system
demonstrate the reliability of the system capability or the mobility and remote operator proficiency. This symmetric
remote operator proficiency. A complete trial of 10 to 30 stepfield terrain specifically challenges robotic system
repetitions in either one of the defined paths should take 10 to locomotion, suspension systems to maintain traction, rollover
30 min to complete. When measuring system capabilities, it is tendencies, self-righting in complex terrain (if necessary),
important to allow enough time to capture a complete trial with chassis shape variability (if available), and remote situational
E2828/E2828M−20
FIG. 4Both Paths are Scalable to Represent Different Environments
awareness by the operator.As such, it can be used to represent apparatus dimension to consider is the minimum clearance
modest outdoor terrain complexity or indoor debris within width (W) for the robot throughout the specified path (see Fig.
confined areas. 4). The minimum clear width should be chosen to represent
typical obstacle spacings of the intended deployment environ-
5.2 The overall size of the terrain apparatus can vary to
ment. The minimum clearance width is typically set to 30 cm
provide different constraints depending on the typical obstacle
[1 ft], 60 cm [2 ft], 120 cm [4 ft], or 240 cm [8 ft] to efficiently
spacingoftheintendeddeploymentenvironment.Forexample,
use available construction materials, although other apparatus
the terrain with containment walls can be sized to represent
sizes can be used. All apparatus dimensions scale proportion-
repeatable complexity within bus, train, or plane aisles; dwell-
ally with the minimum clearance width (see Fig. 5). For
ings with hallways and doorways; relatively open parking lots
example, the overall width of the terrain lane is 2W, and the
with spaces between cars; or unobstructed terrains.
overall length of the terrain lane is at least 6W. It can be longer
5.3 The test apparatuses are low cost and easy to fabricate
for larger robots needing more space to maneuver around the
so they can be widely replicated. The procedure is also simple
barriers while staying on the terrain. When choosing a specific
to conduct. This eases comparisons across various testing
minimum clearance width for the apparatus, note the resulting
locations and dates to determine best-in-class systems and
data is not comparable to other apparatuses with different
operators.
minimum clearance widths.
5.4 Evaluation—This test method can be used in a con-
6.2 Stepfield Terrain—Thesymmetricstepfieldterrainforms
trolled environment to measure baseline capabilities. It can
diagonal hills and valleys. The terrain is discretized into step
also be embedded into operational training scenarios to mea-
tops with square dimensions of ⁄6 A and elevation changes of
sure degradation due to uncontrolled variables in lighting,
⁄12A.Thismakessymmetricstepfieldterrainseasytofabricate
weather, radio communications, GPS accuracy, etc.
with typical square lumber posts (see Fig. 6).Apparatuses with
5.5 Procurement—This test method can be used to identify A = 120 cm [48 in.] can be made with clusters of four 10 cm
inherent capability trade-offs in systems, make informed pur- [4 in.] square posts cut to appropriate lengths to make each
chasing decisions, and verify performance during acceptance step. Apparatuses with A = 60 cm [24 in.] can be made with
testing. This aligns requirement specifications and user expec- single 10 cm [4 in.] square post cut to appropriate lengths to
tations with existing capability limits. make each step. Apparatuses with A = 30 cm [12 in.] can be
made with single 5 cm [2 in.] square post cut to appropriate
5.6 Training—This test method can be used to focus opera-
lengths to make each step (see Fig. 7). The fabricated elements
tortrainingasarepeatablepracticetaskorasanembeddedtask
of the symmetric stepfield terrain are an array of square lumber
within training scenarios. The resulting measures of remote
posts cut to various unit lengths depending on the minimum
operator proficiency enable tracking of perishable skills over
clearance width of the overall apparatus. The posts stand
time, along with comparisons of performance across squads,
uprightonendtoformdiagonalhillfeaturesthatare5unitstall
regions, or national averages.
at their peaks.
5.7 Innovation—This test method can be used to inspire
6.3 Barriers to Define the Robot Path—The barriers placed
technical innovation, demonstrate break-through capabilities,
within the terrain must provide visual guidance for the remote
andmeasurethereliabilityofsystemsperformingspecifictasks
robot operator to correctly traverse the defined figure-8 path
within an overall mission sequence. Combining or sequencing
(forward) or zig-zag path (forward/reverse). The barrier can be
multiple test methods can guide manufacturers toward imple-
made from any solid or porous material that provides visual
menting the combinations of capabilities necessary to perform
guidance. They should be sturdy and easily repaired or
essential mission tasks.
replaced after contact with the robot. The barrier’s overall
thickness shall remain less than 5 % of the minimum clearance
6. Apparatus
width and the length shall equal W.
6.1 The equipment required to perform this test method
includes a symmetric stepfield terrain, barriers to define the 6.4 Containment Structure—The symmetric stepfield terrain
robot path, a containment structure, and a timer. The main posts need to be contained so they do not move relative to one
E2828/E2828M−20
FIG. 5Top View of a Test Apparatus Showing Dimensions Scale Proportionally to the Minimum Clearance Width (W)
FIG. 6Details of a Symmetric Stepfield
another (see Fig. 9). The minimum containment is an under- familiarize themselves with the locations of all emergency
layment with a lumber border about half the height of the stops prior to conducting trials.
tallest posts. Walls can be fabricated to contain the robot as
7.2 Emergency stop systems shall be engaged prior to
well as the terrain. This provides an extra level of difficulty for
approaching a remotely operated robot. Constant communica-
the robot. It can also provide a safety barrier for nearby
tion is essential between the robot and the operator until the
personnel within a test facility. The fabricated wood walls are
robot is safely within the test apparatus and people are either
typically supported with arches over top. Shipping containers
outside the apparatus or at a safe distance.The remote operator
can also enclose test methods and turn a parking lot into a test
may not be aware that someone is interacting with the robot
facility. Apparatuses with minimum clearance width W = 120
when they start to drive, actuate a manipulator, or move the
cm [4 ft] can be slightly undersized to fit into a standard
robot in some other way.Avoid standing directly in front of the
shipping container which has an interior width less than 2400
robot, behind the robot, or within reach of the manipulator arm
cm[8ft].Thecontainerwallsshouldbelinedwithwoodpanels
unless the robot is completely deactivated.
to cover the corrugated steel and have enough thickness to fill
7.3 Safety equipment such as a belay shall be used from a
any gaps between the wall and the terrain.
safe distance to prevent robot damage if necessary. Interven-
6.5 Other Devices—A timer is used to measure the elapsed
tion by hand to try to stop a robot from falling or flipping over
time of the trial. It provides a deterministic indication of trial
is to be prohibited. The belay shall be required for this. Any
start and end times to minimize uncertainty. It can count-up or
interaction with the robot, including tightening the belay to
count-down but should have a settable duration in minutes. A
save the robot, is considered a fault for scoring purposes.
stopwatchcana
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2828 − 11 E2828/E2828M − 20
Standard Test Method for
Evaluating Emergency Response Robot Capabilities:
Mobility: Confined Area Terrains: Mobility Using Symmetric
Stepfields Terrains
This standard is issued under the fixed designation E2828;E2828/E2828M; the number immediately following the designation indicates
the year of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last
reapproval. A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous environments. These missions require
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Mobility Suite of test methods.
1. Scope
1.1 Purpose: This test method is intended for remotely operated ground robots operating in complex, unstructured, and often
hazardous environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of
a robot to traverse complex terrains in the form of symmetric stepfields. This test method is one of several related mobility tests
that can be used to evaluate overall system capabilities.
1.1.1 The purpose of this test method, as a part of a suite of mobility test methods, is to quantitatively evaluate a teleoperated
ground robot’s (see Terminology E2521) capability of traversing complex terrain composed of symmetric stepfields in confined
areas.
1.1.2 Robots shall possess a certain set of mobility capabilities, including negotiating complex terrains, to suit critical operations
such as emergency responses. A part of the complexity is that the environments often pose constraints to robotic mobility to various
degrees. This test method specifies apparatuses to standardize a confined areas terrain that is composed of symmetric stepfields and
that notionally represents types of terrains containing extensive discontinuities, existent in emergency response and other
environments. This test method also specifies procedures and metrics to standardize testing using the apparatus.
1.1.3 The test apparatuses are scalable to provide a range of lateral dimensions to constrain the robotic mobility during task
performance. Fig. 1 shows three apparatus sizes to test robots intended for different emergency response scenarios.
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved July 1, 2011March 1, 2020. Published December 2011April 2020. Originally approved in 2011. Last previous edition approved in 2011 as
E2828/E2828M – 11. DOI: 10.1520/E2828-11.10.1520/E2828_E2828M-20.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2828/E2828M − 20
FIG. 1 Mobility: Confined Area Terrains: Symmetric Stepfields ApparatusesOverview of the Symmetric Stepfield Terrain Apparatus
1.1.4 Emergency response ground robots shall be able to handle many types of obstacles and terrains. The required mobility
capabilities include traversing gaps, hurdles, stairs, slopes, various types of floor surfaces or terrains, and confined passageways.
Yet additional mobility requirements include sustained speeds and towing capabilities. Standard test methods are required to
evaluate whether candidate robots meet these requirements.
1.1.5 ASTM Task Group E54.08.01 on Robotics specifies a mobility test suite, which consists of a set of test methods for
evaluating these mobility capability requirements. This confined area terrain with symmetric stepfields is a part of the mobility test
suite. Fig. 2 shows examples of other confined area terrains, along with the traversing paths. The apparatuses associated with the
test methods challenge specific robot capabilities in repeatable ways to facilitate comparison of different robot models as well as
particular configurations of similar robot models.
1.1.6 The test methods quantify elemental mobility capabilities necessary for ground robot intended for emergency response
applications. As such, users of this standard can use either the entire suite or a subset based on their particular performance
requirements. Users are also allowed to weight particular test methods or particular metrics within a test method differently based
on their specific performance requirements. The testing results should collectively represent an emergency response ground robot’s
overall mobility performance as required. This performance data can be used to guide procurement specifications and acceptance
testing for robots intended for emergency response applications.
NOTE 1—Additional test methods within the suite are anticipated to be developed to address additional or advanced robotic mobility capability
requirements, including newly identified requirements and even for new application domains.
1.2 The robotic system includes a remote operator in control of all functionality, so an onboard camera and remote operator
display are typically required. Assistive features or autonomous behaviors that improve the effectiveness or efficiency of the overall
system are encouraged.
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various
mission requirements.
1.4 Performing Location—This test method shallmay be performed in a testing laboratory or the field where the specified
apparatusanywhere the specified apparatuses and environmental conditions are implemented. can be implemented.
1.5 Units—The values stated in SI units are to be regarded as the standard. The values given in parentheses are not precise
mathematical conversions to inch-pound units. They are close approximate equivalents for the purpose of specifying material
dimensions or quantities that are readily available to avoid excessive fabrication costs of test apparatuses while maintaining
repeatability and reproducibility of the test method results. These values given in parentheses are provided for information only
and are not considered standard.International System of Units (SI Units) and U.S. Customary Units (Imperial Units) are used
throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of
units to enable use of readily available materials in different countries. This avoids excessive purchasing and fabrication costs. The
differences between the stated dimensions in each system of units are insignificant for the purposes of comparing test method
results, so each system of units is separately considered standard within this test method.
E2828/E2828M − 20
FIG. 2 Three Confined Area Terrain Apparatuses in the Mobility Test Suite with Increasing Complexity: The Continuous Pitch/Roll
Ramps Terrain Is Shown on the Left; The Crossing Pitch/Roll Ramps Terrain Is Shown at the Center; The Symmetric Stepfields Terrain
Is Shown on the RightTop View Showing the Figure-8 Path (Forward) Defined by the Barriers
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory
limitations prior to use.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E2521 Terminology for Evaluating Response Robot Capabilities
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
2.2 Additional Documents:Other Standards:
National Response Framework U.S. Department of Homeland Security
NIST Special Publication 1011-I-2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework, Volume I: Terminology,
Version 2.04
3. Terminology
3.1 Definitions—The following terms are used in this test method and are defined in Terminology E2521: listsabstain,
administrator additionalor definitions relevanttest administrator, to thisemergency response robot testor method.response
robot,fault condition,operator,operator station,remote control,repetition,robot,teleoperation,test event or event,test form,test
sponsor,test suite,testing target or target,testing task or task, and trial or test trial.
3.2 Definitions of Terms Specific to This Standard:
3.2.1 abstain, v—prior to starting a particular test method, the robot manufacturer or designated operator shall choose to enter
the test or abstain. Any abstention shall be granted before the test begins. The test form shall be clearly marked as such, indicating
that the manufacturer acknowledges the omission of the performance data while the test method was available at the test time.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from Federal Emergency Management Agency (FEMA), P.O. Box 10055, Hyattsville, MD 20782-8055, http://www.fema.gov.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov.
3.2.1.1 Discussion—
E2828/E2828M − 20
Abstentions may occur when the robot configuration is neither designed nor equipped to perform the tasks as specified in the test
method. Practice within the test apparatus prior to testing should allow for establishing the applicability of the test method for the
given robot.
3.2.2 administrator, n—person who conducts the test—The administrator shall ensure the readiness of the apparatus, the test
form, and any required measuring devices such as stopwatch and light meter; the administrator shall ensure that the specified or
required environmental conditions are met; the administrator shall notify the operator when the safety belay is available and ensure
that the operator has either decided not to use it or assigned a person to handle it properly; and the administrator shall call the
operator to start and end the test and record the performance data and any notable observations during the test.
3.2.3 emergency response robot, or response robot, n—a robot deployed to perform operational tasks in an emergency response
situation.
3.2.3.1 Discussion—
A response robot is a deployable device intended to perform operational tasks at operational tempos during emergency responses.
It is designed to serve as an extension of the operator forgaining improved remote situational awareness and for projecting her/his
intent through the equipped capabilities. It is designed to reduce risk to the operator while improving effectiveness and efficiency
of the mission. The desired features of a response robot include: rapid deployment; remote operation from an appropriate standoff
distance; mobility in complex environments; sufficiently hardened against harsh environments; reliable and field serviceable;
durable or cost effectively disposable, or both; and equipped with operational safeguards.
3.2.4 fault condition, n—during the performance of the task(s) as specified by the test method, a certain condition may occur
that renders the task execution to be failed and such a condition is called a fault condition. Fault conditions result in a loss of credit
for the partially completed repetition. The test time continues until the operator determines that she/he can not continue and notifies
the administrator. The administrator shall, then, pause the test time and add a time-stamped note on the test form indicating the
reason for the fault condition.
3.2.4.1 Discussion—
Fault conditions include robotic system malfunction such as de-tracking, task execution problems, and excessive deviation from
a specified path or uncontrolled behaviors and other safety violations which require administrative intervention.
3.2.5 human-scale, adj—used to indicate that the objects, terrains, or tasks specified in this test method are in a scale consistent
with the environments and structures typically negotiated by humans, although possibly compromised or collapsed enough to limit
human access. Also, that the response robots considered in this context are in a volumetric and weight scale appropriate for
operation within these environments.
3.2.5.1 Discussion—
No precise size and weight ranges are specified for this term. The test apparatus constrains the environment in which the tasks are
performed. Such constraints, in turn, limit the types of robots to be considered applicable to emergency response operations.
3.2.6 operator, n—person who controls the robot to perform the tasks as specified in the test method; she/he shall ensure the
readiness of all the applicable subsystems of the robot; she/he through a designated second shall be responsible for the use of a
safety belay; and she/he shall also determine whether to abstain the test.
3.2.7 operator station, n—apparatus for hosting the operator and her/his operator control unit (OCU, see NIST Special
Publication 1011-I-2.0) to teleoperate (see Terminology E2521) the robot; the operator station shall be positioned in such a manner
as to insulate the operator from the sights and sounds generated at the test apparatuses.
3.2.8 repetition, n—robot’s completion of the task as specified in the test method and readiness for repeating the same task when
required.
3.2.8.1 Discussion—
In a traversing task, the entire mobility mechanism shall be behind the START point before the traverse and shall pass the END
point to complete a repetition. A test method can specify returning to the START point to complete the task. Multiple repetitions,
performed in the same test condition, may be used to establish the robot performance of a particular test method to a certain degree
of statistical significance as specified by the testing sponsor.
E2828/E2828M − 20
FIG. 3 Mobility: Confined Area Terrains: Symmetric Stepfield Apparatuses (Perspective View)Top View Showing the Zig-Zag Path
(Forward/Reverse) Defined by the Barriers
3.2.9 stepfield terrain element, n—discontinuous terrain type using 10- by 10-cm (4- by 4-in.) posts with heights of 10, 20, 30,
40, and 50 cm (4, 8, 12, 16, and 20 in.); the posts are arranged in specified topologies, which, in turn, dictate the levels of
complexity of the resulting terrain. The material used to build these elements shall be strong enough to enable the participating
robots to execute the tasks.
3.2.10 test event or event, n—a set of testing activities that are planned and organized by the test sponsor and to be held at the
designated test site(s).
3.2.11 test form, n—form corresponding to a test method that contains fields for recording the testing results and the associated
information.
3.2.12 test sponsor, n—an organization or individual that commissions a particular test event and receives the corresponding test
results.
3.2.13 test suite, n—designed collection of test methods that are used, collectively, to evaluate the performance of a robot’s
particular subsystem or functionality, including mobility, manipulation, sensors, energy/power, communications, human-robot
interaction (HRI), logistics, safety, and aerial or aquatic maneuvering.
3.2.14 testing task, or task, n—a set of activities specified in a test method for testing robots and the operators to perform in
order for the performance to be evaluated according to the corresponding metric(s). A test method may specify multiple tasks.
3.2 The following terms are used in this test method and are defined in ALFUS Framework Volume I:3: autonomous,autonomy,
level of autonomy,operator control unit (OCU), and semi-autonomous.
4. Summary of Test Method
4.1 This test method is performed by a remote operator controlling the robot out of sight and sound of robot within the test
apparatus. The robot follows one of two defined paths in the specified terrain requiring the robot to overcome challenges including
pitch, roll, traction, and turning on uneven surfaces within open or confined spaces.
4.2 The Figure-8 Path (forward) is a continuous forward path through the terrain with alternating left and right turns to avoid
barriers. It can be used to demonstrate terrain traversal over long distances within a relatively small apparatus. The continuous
traverse is shown as the white path (see Fig. 1 and Fig. 2).
4.3 The Zig-Zag Path (forward/reverse) The task for this test method, symmetric stepfield terrain traversing, is defined as the
robot traversing from the START point along the specified path which ends back at the START point, thus enabling continuous
repetitions. The default path shall be a figure-eight, also known as a continuous “S,” around two pylons installed in the test course
as described in Section is an end-to-end path that requires forward and reverse traversal through the terrain with alternating left
and right turns to avoid barriers. This can be used to demonstrate traversal of the terrain within confined spaces. The down-range
traverse, shown as the white path, is performed in a forward orientation and the up-range traverse, shown as 6. The START and
END points are the same, located beside the first pylon upon enteringthe black path, is performed in reverse (see Fig. 1 the gate.
See and Fig. 3 for an illustration.).
4.4 The robot’s traversing capability of this type of terrain is defined as the robot’s ability to complete the task and the associated
effective speed. Further, the test sponsor can specify the statistical reliability and confidence levels of such a capability and, thus,
dictate the number of successful task performance repetitions that is required. In such a case, the average effective speed shall be
E2828/E2828M − 20
used, instead, as the robot’s capability.robot starts on one side or the other of a lane full of fabricated symmetric stepfield terrain
at a chosen scale. The robot follows either the figure-8 path (forward) or the zig-zag path (forward/reverse) between the two
barriers. The figure-8 path (forward) repetition is completed when the robot crosses the start/end centerline of the lane without a
fault after approximately following the white path. The zig-zag path (forward/reverse) repetition is completed when the robot
crosses the start/end centerline without a fault after approximately following the white and black paths.
4.5 Potential Faults Include:
4.5.1 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial
condition;
4.5.2 Any visual, audible, or physical interaction that assists either the robot or the remote operator;
4.5.3 Leaving the apparatus during the trial.
4.6 Teleoperation shall be used from the operator station specified by the administrator to test the robots using an OCU provided
by the operator. The operator station shall be positioned and implemented in such a manner so as to insulate the operator from the
sights and sounds generated at the test apparatus.Test trials shall produce enough successful repetitions to demonstrate the
reliability of the system capability or the remote operator proficiency. A complete trial of 10 to 30 repetitions in either one of the
defined paths should take 10 to 30 min to complete. When measuring system capabilities, it is important to allow enough time to
capture a complete trial with an expert operator. When measuring operator proficiency, it is important to limit the time of the trial
so that novice and expert operators are similarly fatigued.
4.7 The operator is allowed to practice before the test. She/he is also allowed to abstain from the test before it is started. Once
the test begins, there shall be no verbal communication between the operator and the administrator regarding the performance
There are three metrics to consider when calculating the results of a test repetition other than instructions on when to start and
notifications of faults and any safety related conditions. The operator shall have the full responsibility to determine whether and
when the robot has completed a repetition and notify the administrator accordingly. However, it is the administrator’s authority to
judge the completeness of the repetition.trial. They should be considered in the following order of importance: completeness score,
reliability, and efficiency. The results from the figure-8 path (forward) and the zig-zag path (forward/reverse) are not comparable
because they measure different capabilities. The results from different scales of test apparatus are also not comparable because they
represent different clearances and distances.
NOTE 2—Practice within the test apparatus could help establish the applicability of the robot for the given test method. It allows the operator to gain
familiarity with the standard apparatus and environmental conditions. It also helps the test administrator to establish the initial apparatus setting for the
test when applicable.
4.5 The test sponsor has the authority to select the size for the specified confined area apparatus. The test sponsor also has the
authority to select the test methods that constitute the test event, to select one or more test site(s) at which the test methods are
implemented, to determine the corresponding statistical reliability and confidence levels of the results for each of the test methods,
and to establish the participation rules including the testing schedules and the test environmental conditions.
5. Significance and Use
5.1 A main purpose of using robots in emergency response operations is to enhance the safety and effectiveness of emergency
responders operating in hazardous or inaccessible environments. The testing results of the candidate robot shall describe, in a
statistically significant way, how reliably the robot is able to traverse the specified types of terrains and thus provide emergency
responders sufficiently high levels of confidence to determine the applicability of the robot.This test method is part of an overall
suite of related test methods that provide repeatable measures of robotic system mobility and remote operator proficiency. This
symmetric stepfield terrain specifically challenges robotic system locomotion, suspension systems to maintain traction, rollover
tendencies, self-righting in complex terrain (if necessary), chassis shape variability (if available), and remote situational awareness
by the operator. As such, it can be used to represent modest outdoor terrain complexity or indoor debris within confined areas.
5.2 This test method addresses robot performance requirements expressed by emergency responders and representatives from
other interested organizations. The performance data captured within this test method are indicative of the testing robot’s
capabilities. Having available a roster of successfully tested robots with associated performance data to guide procurement and
deployment decisions for emergency responders is consistent with the guideline of “Governments at all levels have a responsibility
to develop detailed, robust, all-hazards response plans” as stated in National Response Framework.The overall size of the terrain
apparatus can vary to provide different constraints depending on the typical obstacle spacing of the intended deployment
environment. For example, the terrain with containment walls can be sized to represent repeatable complexity within bus, train,
or plane aisles; dwellings with hallways and doorways; relatively open parking lots with spaces between cars; or unobstructed
terrains.
5.3 The test apparatuses are scalable to constrain robot maneuverability during task performance for a range of robot sizes in
confined areas associated with emergency response operations. Variants of the apparatus provide minimum lateral clearance of 2.4
m (8 ft) for robots expected to operate around environments such as cluttered city streets, parking lots, and building lobbies;
minimum lateral clearance of 1.2 m (4 ft) for robots expected to operate in and around environments such as large buildings,
E2828/E2828M − 20
FIG. 4 Mobility: Confined Area Terrains: Symmetric Stepfield Apparatuses (Projection View)Both Paths are Scalable to Represent Differ-
ent Environments
stairwells, and urban sidewalks; minimum lateral clearance of 0.6 m (2 ft) for robots expected to operate within environments such
as dwellings and workspaces, buses and airplanes, and semi-collapsed structures; minimum lateral clearance of less than 0.6 m (2
ft) with a minimum vertical clearance adjustable from 0.6 m (2 ft) to 10 cm (4 in.) for robots expected to deploy through breeches
and operate within sub-human size confined spaces voids in collapsed structures.low cost and easy to fabricate so they can be
widely replicated. The procedure is also simple to conduct. This eases comparisons across various testing locations and dates to
determine best-in-class systems and operators.
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. It can also be
embedded into operational training scenarios to measure degradation due to uncontrolled variables in lighting, weather, radio
communications, GPS accuracy, etc.
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing
decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with
existing capability limits.
5.6 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within
training scenarios. The resulting measures of remote operator proficiency enable tracking of perishable skills over time, along with
comparisons of performance across squads, regions, or national averages.
5.7 Innovation—The standard apparatus is specified to be easily fabricated to facilitate self-evaluation by robot developers and
provide practice tasks for emergency responders that exercise robot actuators, sensors, and operator interfaces. The standard
apparatus can also be used to support operator training and establish operator proficiency. This test method can be used to inspire
technical innovation, demonstrate break-through capabilities, and measure the reliability of systems performing specific tasks
within an overall mission sequence. Combining or sequencing multiple test methods can guide manufacturers toward implementing
the combinations of capabilities necessary to perform essential mission tasks.
5.5 Although the test method was developed first for emergency response robots, it may be applicable to other operational
domains.
6. Apparatus
6.1 The equipment required to perform this test method includes a symmetric stepfield terrain, barriers to define the robot path,
a containment structure, and a timer. The main apparatus dimension to consider is the minimum clearance width (W) for the robot
throughout the specified path (see Fig. 4). The minimum clear width should be chosen to represent typical obstacle spacings of
the intended deployment environment. The minimum clearance width is typically set to 30 cm [1 ft], 60 cm [2 ft], 120 cm [4 ft],
or 240 cm [8 ft] to efficiently use available construction materials, although other apparatus sizes can be used. All apparatus
dimensions scale proportionally with the minimum clearance width (see Fig. 5). For example, the overall width of the terrain lane
is 2W, and the overall length of the terrain lane is at least 6W. It can be longer for larger robots needing more space to maneuver
around the barriers while staying on the terrain. When choosing a specific minimum clearance width for the apparatus, note the
resulting data is not comparable to other apparatuses with different minimum clearance widths.
6.2 Stepfield Terrain—The test apparatuses specify three scaled sizes of confined areas fully covered with symmetric stepfield
terrain elements. The three sizes are 7.2 m (24 ft) long by 4.8 m (16 ft), 2.4 mforms diagonal hills and valleys. The terrain is
1 1
discretized into step tops with square dimensions of ⁄6 (8 ft), or 1.2 m (4A and elevation changes of ⁄12 ft) wide. Two pylons define
the figure-eight path. They are posted at the 2.4- andA. This makes symmetric stepfield terrains easy to fabricate with typical square
lumber posts (see Fig. 64.8-m (8- and 16-ft) distances from either end and along the centerline between the two sidewalls. The
clusters of posts form minimally 20 cm (8 in.) step surfaces that increase or decrease in elevation by 10 cm (4 in.) increments
between 10 and 50 cm (4 and 20 in.). The resulting topology has diagonal ridges and valleys with extensive discontinuities that
cause constant orientation complexities for robots. The terrain is surrounded with a). Apparatuses with A = 120 cm [48 in.] can be
E2828/E2828M − 20
FIG. 6 ExampleDetails of a Test Form (Blank)Symmetric Stepfield
made with clusters of four 10 cm [4 in.] square posts cut to appropriate lengths to make each step. Apparatuses with A = 60 cm
[24 in.] can be made with single 10 cm [4 in.] square post cut to appropriate lengths to make each step. Apparatuses with A = 30 cm
[12 in.] can be made with single 5 cm [2 in.] square post cut to appropriate lengths to make each step (see Fig. 71.2 m (4 ft) tall
wall. A gate opens in the front to allow robot entry. See). The fabricated elements of the symmetric stepfield terrain are an array
of square lumber posts cut to various unit lengths depending Figs. 3-5. Each repetition for the figure eight path is nominally
considered to be 15 m long.on the minimum clearance width of the overall apparatus. The posts stand upright 5.3 specifies the
scalability of the apparatus.on end to form diagonal hill features that are 5 units tall at their peaks.
NOTE 3—The walls are typically made of OSB.
6.3 Barriers to Define the Robot Path—The barriers placed within the terrain must provide visual guidance for the remote robot
operator to correctly traverse the defined figure-8 path (forward) or zig-zag path (forward/reverse). The barrier can be made from
any solid or porous material that provides visual guidance. They should be sturdy and easily repaired or replaced after contact with
the robot. The barrier’s overall thickness shall remain less than 5 % of the minimum clearance width and the length shall equal
W.
6.4 Containment Structure—Various test conditions such The symmetric stepfield terrain posts need to be contained so they do
not move relative to one another (see Fig. 9as apparatus surface types and conditions, including wetness and friction levels,
temperature, types of lighting, smoke, humidity, and rain shall be facilitated when the test sponsor requires. For example, for a test
run in the dark environment, a light meter shall be used to read 0.1 lux or less. The darkness shall be re-measured when the lighting
condition might have changed. The actual readings of these conditions should be recorded on the test form.). The minimum
containment is an underlayment with a lumber border about half the height of the tallest posts. Walls can be fabricated to contain
the robot as well as the terrain. This provides an extra level of difficulty for the robot. It can also provide a safety barrier for nearby
personnel within a test facility. The fabricated wood walls are typically supported with arches over top. Shipping containers can
also enclose test methods and turn a parking lot into a test facility. Apparatuses with minimum clearance width W = 120 cm [4
ft] can be slightly undersized to fit into a standard shipping container which has an interior width less than 2400 cm [8 ft]. The
container walls should be lined with wood panels to cover the corrugated steel and have enough thickness to fill any gaps between
the wall and the terrain.
NOTE 4—The testing apparatus can be implemented in a standard International Standards Organization (ISO) shipping container, in which some of the
test conditions can be furnished. To achieve the specified darkness, first turn off all the lighting sources inside and entirely cover the entrance with
light-blocking drapes. The darkness is specified as 0.1 lux due to the implementation costs concerns for the apparatuses and due to the fact that robotic
cameras are less sensitive than human eyes, such that any darkness below 0.1 lux would not make a difference in the cameras’ functioning. It is recognized
that the environments in real applications may be darker than the specified test condition.
6.5 Other Devices—A stopwatch shall be provided timer is used to measure the timing performance.elapsed time of the trial.
It provides a deterministic indication of trial start and end times to minimize uncertainty. It can count-up or count-down but should
have a settable duration in minutes. A stopwatch can also be used. A light meter is necessary if testing in lighted and dark
environments. A lighted environment is considered >150 lx and dark environment is considered <0.1 lx.
7. Hazards
7.1 Functional emergency stop systems are essential for safe remote or autonomous robot operation. The emergency stop on the
operator control unit shall be clearly marked and accessible. The emergency stop on the robot chassis, if available, should also be
marked. All personnel involved in testing shall familiarize themselves with the locations of all emergency stops prior to conducting
trials.
7.2 Emergency stop systems shall be engaged prior to approaching a remotely operated robot. Constant communication is
essential between the robot and the operator until the robot is safely within the test apparatus and people are either outside the
apparatus or at a safe distance. The remote operator may not be aware that someone is interacting with the robot when they start
E2828/E2828M − 20
L
L
L
L
L
L
L
L
L
L
L
L
K 2 2
K 2 2
J 2 23 3
J 2 23 3
I 2 23 3 4 4
I 2 23 3 4 4
H 2 23 3 4 4 5 5
H 2 23 3 4 4 5 5
G 2 23 3 4 4 5 5 4 4
G 2 23 3 4 4 5 5 4 4
F 2 23 3 4 4 5 5 4 4 3 3
F 2 23 3 4 4 5 5 4 4 3 3
E 2 23 3 4 4 5 5 4 4 3 3 2 2
E 2 23 3 4 4 5 5 4 4 3 3 2 2
D 2 23 3 4 4 5 5 4 4 3 3 2 2
D 2 23 3 4 4 5 5 4 4 3 3 2 2
B B
C 2 23 3 4 4 5 5 4 4 3 3 2 2
C 2 23 3 4 4 5 5 4 4 3 3 2 2
B B
2 2
B 2 23 3 4 4 5 5 4 4 3 3 2 2
B 2 23 3 4 4 5 5 4 4 3 3 2 2 2 2
B B
A 2 23 3 4 4 5 5 4 4 3 3 2 2 2 23 3
A 2 23 3 4 4 5 5 4 4 3 3 2 2 2 23 3
B B
2 2 3 3 4 4 5 5 4 4 3 3 2 2 2 23 3 4 4
2 2 3 3 4 4 5 5 4 4 3 3 2 2 2 23 3 4 4
B B
2 23 3 4 4 5 5
3 3 4 45 5 4 4 3 3 2 2
3 3 4 45 5 4 4 3 3 2 2 2 23 3 4 4 5 5
B B
4 4 5 54 4 3 3 2 2 2 23 3 4 4 5 5 4 4
4 4
4 4 5 5 3 3 2 2 2 23 3 4 4 5 5 4 4
B B
5 5 4 43 3 2 2 2 23 3 4 4 5 5 4 4 3 3
5 5 4 43 3 2 2 2 23 3 4 4 5 5 4 4 3 3
B B
2 23 3 4 4 5 5 4 4 3 3 2 2
4 4 3 32 2
4 4 3 32 2 2 23 3 4 4 5 5 4 4 3 3 2 2
B B
3 3 2 2 2 23 3 4 4 5 5 4 4 3 3 2 2
3 3 2 2 2 23 3 4 4 5 5 4 4 3 3 2 2
B B B B
2 23 3 4 4 5 5 4 4 3 3 2 2
2 2
2 2 2 23 3 4 4 5 5 4 4 3 3 2 2
B B B B
2 23 3 4 4 5 5 4 4 3 3 2 2 2 2
2 23 3 4 4 5 5 4 4 3 3 2 2 2 2
B B B B
2 23 3 4 4 5 5 4 4 3 3 2 2 2 23 3
B
2 23 3 4 4 5 5 4 4 3 3 2 2 C
2 23 3
B B D
2 2 3 3 4 4 5 5 4 4 3 3 2 2 2 23 3 4 4
E
F
2 2 3 3 4 4 5 5 4 4 3 3 2 2 2 23 3 4 4 G
B B H
I
3 3 4 45 5 4 4 3 3 2 2 2 23 3 4 4 5 5 2
J
K
3 3 4 45 5 4 4 3 3 2 2 2 23 3 4 4 5 5
2 3
L
B B
M
4 4 5 54 4 3 3 2 2 2 23 3 4 4 5 5 4 4 2 3 4
4 4
4 4 5 5 3 3 2 2 2 23 3 4 4 5 5 4 4 2 3 4 5
B B
5 5 4 43 3 2 2 3
2 23 3 4 4 5 5 4 4 3 3 2 4 5 4
5 5 4 43 3 2 2 2 23 3 4 4 5 5 4 4 3 3 2 3 4 5 4 3
B B
4 4 3 32 2 2 23 3 4 4 5 5 4 4 3 3 2 2 2 3 4 5 4 3 2
4 4 3 32 2 2 3
2 23 3 4 4 5 5 4 4 3 3 2 2 4 5 4 3 2
B B A
3 3 2 2 2 23 3 4 4 5 5 4 4 3 3 2 2 A 2 3 4 5 4 3 2
A
3 3 2 2 2 23 3 4 4 5 5 4 4 3 3 2 2 A 2 3 4 5 4 3 2 2
B B A
2 2 2 23 3 4 4 5 5 4 4 3 3 2 2 B 2 3 4 5 4 3 2 2 3
A
2 2 2 23 3 4 4 5 5 4 4 3 3 2 2 B 2 3 4 4 3 2 2 3 4
B B A
2 23 3 4 4 5 5 4 4 3 3 2 2 C 3 4 5 4 3 2 2 3 4 5
A
2 2 3 4
2 23 3 4 4 5 5 4 4 3 3 2 2 C 4 5 4 3 5 4
B B A
2 23 3 4 4 5 5 4 4 3 3 2 2 D 5 4 3 2 2 3 4 5 4 3
A
2 23 3 4 4 5 5 4 4 3 3 2 2 D 4 3 2 2 3 4 5 4 3 2
A
2 3 3
2 2 3 3 4 4 5 5 4 4 3 3 2 2 E 3 2 4 5 4 2
A A
2 2 3 3 4 4 5 5 4 4 3 3 2 2 E 2 2 3 4 5 4 3 2
A A
3 3 4 45 5 4 4 3 3 2 2 F 2 3 4 5 4 3 2 2
A
A B
C
3 3 4 45 5 4 4 3 3 2 2 F 2 3 4 4 3 2 2 3 D
A E
F
4 4 4 4 3 3 2 3 4 5 4 3 2 2 3 4 G
5 5 2 2 G H
A I
J
K
4 4 5 54 4 3 3 2 2 G 3 4 5 4 3 2 2 3 4 5 L
2 3
M
A
3 4
5 5 4 43 3 2 2 H 4 5 4 3 2 3 4 5 4
2 3 4 5
A
2 3 4 5 4
4 43 3 5 4 3 2 2 3 4 5 4 3
5 5 2 2 H 2 3 3
4 5 4
A
2 3 4 5 4 3 2
4 4 3 32 2 I 4 3 2 2 3 4 5 4 3 2
2 3 4 5 4 3 2
A A
2 3 4 5
4 3 2
A
4 4 3 32 2 I 3 2 2 3 4 5 4 3 2
2 3 4 5 4 3 2 2
A A
2 3 4 5 4 3 2 2 3
J 2 3 4 5 4 3 2 A
3 3 2 2 2 3 2 3 4
2 3 4 5 4 2
A A
3 4 5 4 3 2 2 3 4 5
A
3 3 2 2 J 2 3 4 5 4 3 2
4 5 4 3 2 2 3 4 5 4
A
A
5 2 3 4 5 4 3
4 3 2
A
2 2 K 2 3 4 5 4 3 2
4 3 2 2 3 4 5 4 3 2
A
3 2 2 3 4 5 4 3 2
A A
2 3 4 3 2
2 2 K 4 5 2 3 4 5 3 2
2 4
A A
2 3 4 5 4 3 2 2
A
L 3 4 5 4 3 2 A
2 3 4 5 4 3 2 2 3
A
2 3 5 4 3 2 2 3 4
A
L 4 5 4 3 2
3 4 5 4 3 2 2 3 4 5
A
4 5 4 3 2 2 3 4 5 4
A
L 5 4 3 2 5 4 3 2 2 3 4 5 4 3
A
4 3 2 2 3 4 5 4 3 2
A
4 3 2
L M 3 2 2 3 4 5 4 3 2
A
L 2 2 3 4 5 3 2
A
L 3 2
K 2 3 4 5 4 3 2
A
2 3 4 5 4 3 2
J
L 2 3 5 4 3 2
I 4
3 4 5 4 3 2
H
L
4 5 4 3 2
G
F 5 4 3 2
L E
4 3 2 M
D L
3 2
K
C
L 2 J
I
B H
G
L F
E
D
C
B
L
L
FIG. 7 Example of a Test Form (Filled in with Illustrative Data)Overview of Symmetric Stepfield
Apparatus with A = 120 cm (48 in)
Apparatus with A = 60 cm (24 in)
Apparatus with A = 30 cm (12 in)

E2828/E2828M − 20
FIG. 7 Overview of Symmetric Stepfield (continued)
FIG. 5 Stepfields Topology: Colored boxes Showthe Post Heights; the Numbers in the Boxes, 1 through 5 Indicate the Heights from 10
cm (4 in.) through 50 cm (20 in.); the Darkness of The Colors Also Reflect the HeightsTop View of a Test Apparatus Showing Dimen-
sions Scale Proportionally to the Minimum Clearance Width (W)
to drive, actuate a manipulator, or move the robot in some other way. Avoid standing directly in front of the robot, behind the robot,
or within reach of the manipulator arm unless the robot is completely deactivated.
7.3 Safety equipment such as a belay shall be used from a safe distance to prevent robot damage if necessary. Intervention by
hand to try to stop a robot from falling or flipping over is to be prohibited. The belay shall be required for this. Any interaction
with the robot, including tightening the belay to save the robot, is considered a fault for scoring purposes.
7.4 BesidesTest 1.4, which addresses the human safety and health concerns, users of the standard shall also address the
equipment preservation concerns and human robot coexistence concerns.apparatuses that are intended to challenge robot mobility
can be complex and unstable for humans. Proper footwear and other personal protective equipment shall be worn to mitigate risk.
Caution is required when attending to a robot or carrying equipment within the apparatus.
NOTE 5—The environmental conditions, such as high or low temperatures, excessive moisture, and rough terrain can be stressful to the humans, damage
the robotic components, or cause unexpected robotic motions.
8. Calibration and Standardization
8.1 The robot configuration as tested shall be described in detail on the test form, including all subsystems and components and
their respective features and functionalities. The configuration shall be subjected to all the test suites, as defined in 3.2.13, as
appropriate. Any variation in the configuration shall cause the resulting robot variant to be retested across all the test suites to
provide a consistent and comprehensive representation of the performance. Practice E2592 shall be used to record the robotic
configuration.
8.2 Once a robot begins a test, by starting executing the task as specified in 4.1, the robot shall be teleoperated to perform the
task for the specified number of repetitions through completion without leaving the apparatus. During the process, the robot shall
E2828/E2828M − 20
FIG. 8 Side View Showing Various Barriers
FIG. 9 Various Containment Structures
not be allowed to have the energy/power source replenished nor shall the robot be allowed any human physical intervention,
including adjustment, maintenance, or repair. Any such actions shall be considered a fault condition.
8.3 The metric for this test method is the completeness of the prescribed path successfully traversed for the specified number
of continuous repetitions.
8.4 In addition, the elapsed time for successfully traversing the path, or effective speed in meters per minute, is a performance
proficiency index reflecting the combination of the robot’s capability and efficiency, the OCU’s ease of use, and the operator’s skill
level. Therefore, this temporal aspect is a part of the test and the results shall be recorded on the test form.
NOTE 6—The term “effective” is used because the speed is calculated based on the designed length of the path and not on the actual path of the
traverses, which can deviate from the designed path.
E2828/E2828M − 20
8.5 Although the metric is based on teleoperation, autonomous behaviors are allowed as long as the testing procedure is
followed, with the associated effects reflected in the testing scores. See NIST Special Publication 1011-I-2.0 for the definition of
autonomy.
8.6 The test sponsor has the authority to specify the lighting condition and other environmental variables, which can affect the
test results. All environmental settings shall be noted on the test form.
8.7 A robot’s reliability (R) of performing the specified task at a particular apparatus setting and the associated confidence (C)
shall be established. The required R and C values dictate the required number of successful repetitions and the allowed number
of failures during the test. With a given set of the R and C values, more successes will be needed when more failur
...