ASTM E2566-17a
(Test Method)Standard Test Method for Evaluating Response Robot Sensing: Visual Acuity
Standard Test Method for Evaluating Response Robot Sensing: Visual Acuity
SIGNIFICANCE AND USE
5.1 Various levels of visual acuity are essential when remotely operating robots in unstructured and often hazardous environments. Missions typically include establishing situational awareness, finding available paths, maneuvering through obstacles, identifying objects of interest, and performing detailed inspections. This test method measures robot system far-field and near-field visual acuity which are applicable to virtually every mission. These quantitative measures of performance provide a common language that allows robot users to better understand and express their own requirements and improve the way visual sensing capabilities are specified.
5.2 Multiple cameras could be incorporated into remotely operated robotic systems since a single camera is unlikely to be effective for all aspects of a mission. For example, cameras with zoom lenses are often used for far-field tasks. Cameras with close focus capabilities are often used for near-field tasks. Wide-angle lenses are often used for driving and obstacle avoidance. This test method characterizes each onboard camera to understand overall system capabilities.
5.3 This test method provides a way to unambiguously specify robot requirements in terms of the related measures of visual acuity and field of view. This helps quantify the trade-offs and general usefulness of optical versus digital zoom cameras and fixed versus variable focus lenses. The visual acuity charts can also help provide quantitative measures of performance within other test methods and training scenarios. See Figs. 2-4 for illustrations.
FIG. 2 This Baseline Image is Used for Purposes of Comparisons Below
FIG. 3 Three Images of the Same Scene with the Same Image Resolution. Top Row Shows Field of View Increasing from Left to Right (the image “zooms out”) While Bottom Row Shows Acuity Decreasing (features of the same size become harder to clearly observe)
FIG. 4 Three Images of the Same Scene with the Same Field of View. The Top Row ...
SCOPE
1.1 The purpose of this test method is to specify the apparatuses, procedures, and performance metrics necessary to quantitatively measure a robot’s visual acuity as displayed to a remote operator or vision algorithm. The primary performance metric for this test method shall be a robot’s possession of such a capability with a specified statistical significance level.
1.2 Secondary performance metrics are the robot’s field of view and aspect ratio.
1.3 This test method can also be used to measure the operator proficiency in performing the specified task. The corresponding performance metric may be the number of completed task repetitions per minute over an assigned time period ranging from 10 to 30 minutes.
1.4 This test method is a part of the sensing suite of response robot test methods, but this test method is stand-alone and complete. This test method applies to systems operated remotely from a standoff distance appropriate for the intended mission. The system includes a remote operator in control of all functionality and any assistive features or autonomous behaviors that improve the effectiveness or efficiency of the overall system.
1.5 The apparatus, specified in Section 6, can only test a limited range of a robot’s capabilities. When the robot has been tested through the limit or limits of the apparatus, a note shall be associated with the results indicating that the robot’s actual capability may be outside of the limit or limits imposed by the test apparatus. For example, the robot could exceed the capabilities of the printing process used to create the charts used in the apparatus.
1.6 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions can be implemented.
1.7 Units—The values stated in either SI units or inch-pound units are to be regarded separately as standard. The values stated in each system may not be exact equivalents; the...
General Information
Relations
Buy Standard
Standards Content (Sample)
This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E2566 − 17a
Standard Test Method for
1
Evaluating Response Robot Sensing: Visual Acuity
This standard is issued under the fixed designation E2566; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
Theroboticscommunityneedswaystomeasurewhetheraparticularrobotiscapableofperforming
specific missions in unstructured and often hazardous environments. These missions decompose into
elemental robot tasks that can be represented individually as standard test methods and practices.The
associated test apparatuses and performance metrics provide a tangible language to communicate
various mission requirements. They also enable repeatable testing to establish the reliability of
essential robot capabilities.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
standard test methods and practices for evaluating individual robot capabilities. These standards
facilitate comparisons across diverse models or multiple configurations of a single model. The
standards support robot researchers, manufacturers, and user organizations in different ways.
Researchers use the standards to understand mission requirements, encourage innovation, and
demonstratebreak-throughcapabilities.Manufacturersusethestandardstoevaluatedesigndecisions,
integrate emerging technologies, and harden developed systems. User organizations leverage the
resultingrobotcapabilitiesdatatoguidepurchasingdecisions,aligndeploymentobjectives,andfocus
training with standard measures of operator proficiency. Associated usage guides describe how such
standards can be applied to support these various objectives.
The overall suite of standards addresses critical subsystems of remotely operated response robots,
including maneuvering, mobility, dexterity, sensing, energy, communications, durability, proficiency,
autonomy, logistics, safety, and terminology. This test method is part of the Sensing test suite and
addresses the visual acuity of onboard cameras.
1. Scope completed task repetitions per minute over an assigned time
period ranging from 10 to 30 minutes.
1.1 The purpose of this test method is to specify the
apparatuses,procedures,andperformancemetricsnecessaryto
1.4 This test method is a part of the sensing suite of
quantitatively measure a robot’s visual acuity as displayed to a
responserobottestmethods,butthistestmethodisstand-alone
remote operator or vision algorithm.The primary performance
and complete. This test method applies to systems operated
metricforthistestmethodshallbearobot’spossessionofsuch
remotely from a standoff distance appropriate for the intended
a capability with a specified statistical significance level.
mission.Thesystemincludesaremoteoperatorincontrolofall
1.2 Secondary performance metrics are the robot’s field of functionality and any assistive features or autonomous behav-
view and aspect ratio. iors that improve the effectiveness or efficiency of the overall
system.
1.3 This test method can also be used to measure the
operator proficiency in performing the specified task. The
1.5 The apparatus, specified in Section 6, can only test a
corresponding performance metric may be the number of
limitedrangeofarobot’scapabilities.Whentherobothasbeen
tested through the limit or limits of the apparatus, a note shall
1
be associated with the results indicating that the robot’s actual
This test method is under the jurisdiction of ASTM Committee E54 on
Homeland Security Applications and is the direct responsibility of Subcommittee
capability may be outside of the limit or limits imposed by the
E54.09 on Response Robots.
test apparatus. For example, the robot could exceed the
Current edition approved Sept. 1, 2017. Published November 2017. Originally
capabilities of the printing process used to create the charts
approved in 2008. Last previous edition approved in 2017 as E2566–17. DOI:
10.1520/E2566-17A. used in the apparatus.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
1
---------------------- Page: 1 ----------------------
E2566 − 17a
1.6 Performing Location—This test method may be per- 3.2.1 aspect ratio, n—the ratio of width to height of the
formed anywhere the specified apparatuses and environmental image produced by a camera system.
conditions can be implemented.
3.2.2 camera system, n—a specific camera with its associ-
ated lighting, compression, interface, disp
...
This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2566 − 17 E2566 − 17a
Standard Test Method for
Determining Visual Acuity and Field of View of On-Board
Video Systems for Teleoperation of Robots for Urban
Search and Rescue ApplicationsEvaluating Response Robot
1
Sensing: Visual Acuity
This standard is issued under the fixed designation E2566; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in unstructured and often hazardous environments. These missions decompose into
elemental robot tasks that can be represented individually as standard test methods and practices. The
associated test apparatuses and performance metrics provide a tangible language to communicate
various mission requirements. They also enable repeatable testing to establish the reliability of
essential robot capabilities.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
standard test methods and practices for evaluating individual robot capabilities. These standards
facilitate comparisons across robot models, or across various diverse models or multiple configura-
tions of a particular robot model. They single model. The standards support robot researchers,
manufacturers, and user organizations in different ways. Researchers use them the standards to
understand mission requirements, encourage innovation, and demonstrate break-through capabilities.
Manufacturers use them the standards to evaluate design decisions, integrate emerging technologies,
and harden developed systems. User organizations leverage the resulting robot capabilities data to
guide purchasing, purchasing decisions, align deployment objectives, and focus training with standard
measures of operator proficiency. An associated usage guide describesAssociated usage guides
describe how such standards can be implementedapplied to support these various objectives.
The overall suite of standards addresses critical subsystems of remotely operated response robots,
including maneuvering, mobility, dexterity, sensing, energy, communications, durability, proficiency,
autonomy, logistics, safety, and terminology. This test method is part of the sensingSensing test suite
and addresses the visual acuity of onboard cameras.
1. Scope
1.1 This test method covers the measurement of several key parameters of video systems for remote operations. It is initially
intended for applications of robots for Urban Search and Rescue but is sufficiently general to be used for marine or other remote
platforms. Those parameters are (The purpose of this test method is to specify 1) field of view of the camera system, (the
apparatuses, procedures, 2) visual acuity at far distances with both ambient lighting and lighting on-board the robot, (and
performance metrics necessary to quantitatively3) visual acuity at near distances, again in both light and dark environments, and
( measure a robot’s visual acuity as displayed to a remote operator or vision algorithm. The primary performance4), if available,
visual acuity in both light and dark environments with zoom lens capability. metric for this test method shall be a robot’s possession
of such a capability with a specified statistical significance level.
1.2 Secondary performance metrics are the robot’s field of view and aspect ratio.
1
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved Jan. 1, 2017Sept. 1, 2017. Published February 2017November 2017. Originally approved in 2008. Last previous edition approved in 20082017
as E2566 – 08.E2566 – 17. DOI: 10.1520/E2566-17.10.1520/E2566-17A.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
1
---------------------- Page: 1 ----------------------
E2566 − 17a
1.3 These tests measure only end-to-end capability, that is, they determine the resolution of the images on the display screen
at the operator control unit since that is the important issue for the user. This test method can also be used to measure the operator
proficiency in performing the specified task. The corresponding performance metric may
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.