ASTM E2862-12
(Practice)Standard Practice for Probability of Detection Analysis for Hit/Miss Data
Standard Practice for Probability of Detection Analysis for Hit/Miss Data
SIGNIFICANCE AND USE
The POD analysis method described herein is based on a well-known and well established statistical method. It shall be used to quantify the demonstrated POD for a specific set of examination parameters and known range of discontinuity sizes when the initial response from a nondestructive evaluation inspection system is ultimately binary in nature (that is, hit or miss). This method requires that a relationship between discontinuity size and POD exists and is best described by a generalized linear model with the appropriate link function for binary outcomes.
Prior to performing the analysis it is assumed that the discontinuity of interest is clearly defined; the number and distribution of induced discontinuity sizes in the POD specimen set is known and well-documented; discontinuities in the POD specimen set are unobstructed; the POD examination administration procedure (including data collection method) is well-defined, under control, and unbiased; and the initial response is ultimately binary in nature (that is, hit or miss). The analysis results are only valid if convergence is achieved and the model adequately represents the data.
The POD analysis method described herein is consistent with the analysis method for binary data described in MIL-HDBK-1823A, which is included in several widely utilized POD software packages to perform a POD analysis on hit/miss data. It is also found in statistical software packages that have generalized linear modeling capability. This practice requires that the analyst has access to either POD software or other software with generalized linear modeling capability.
SCOPE
1.1 This practice defines the procedure for performing a statistical analysis on nondestructive testing hit/miss data to determine the demonstrated probability of detection (POD) for a specific set of examination parameters. Topics covered include the standard hit/miss POD curve formulation, validation techniques, and correct interpretation of results.
1.2 The values stated in inch-pound units are to be regarded as standard. The values given in parentheses are mathematical conversions to SI units that are provided for information only and are not considered standard.
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
Designation: E2862 − 12
Standard Practice for
Probability of Detection Analysis for Hit/Miss Data
This standard is issued under the fixed designation E2862; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 3.1.3 false call, n—the perceived detection of a discontinu-
ity that is identified as a find during a POD examination when
1.1 This practice defines the procedure for performing a
no discontinuity actually exists at the inspection site.
statistical analysis on nondestructive testing hit/miss data to
determine the demonstrated probability of detection (POD) for
3.1.4 hit, n—an existing discontinuity that is identified as a
a specific set of examination parameters. Topics covered
find during a POD demonstration examination.
include the standard hit/miss POD curve formulation, valida-
3.1.5 miss, n—an existing discontinuity that is missed dur-
tion techniques, and correct interpretation of results.
ing a POD examination.
1.2 The values stated in inch-pound units are to be regarded
3.1.6 probability of detection, n—the fraction of nominal
as standard. The values given in parentheses are mathematical
discontinuity sizes expected to be found given their existence.
conversions to SI units that are provided for information only
and are not considered standard. 3.2 Symbols:
1.3 This standard does not purport to address all of the 3.2.1 a—discontinuity size.
safety concerns, if any, associated with its use. It is the
3.2.2 a —the discontinuity size that can be detected with
p
responsibility of the user of this standard to establish appro-
probability p.
priate safety and health practices and determine the applica-
3.2.2.1 Discussion—Eachdiscontinuitysizehasanindepen-
bility of regulatory limitations prior to use.
dent probability of being detected and corresponding probabil-
ityofbeingmissed.Forexample,beingabletodetectaspecific
2. Referenced Documents
discontinuity size with probability p does not guarantee that a
2.1 ASTM Standards:
larger size discontinuity will be found.
E1316 Terminology for Nondestructive Examinations
2.2 Department of Defense Handbook: 3.2.3 a —the discontinuity size that can be detected with
p/c
MIL-HDBK-1823A Nondestructive Evaluation System Re- probability p with a statistical confidence level of c.
liability Assessment
3.2.3.1 Discussion—a is calculated by applying a statis-
p/c
tical uncertainty bound to a . The uncertainty bound is a
p
3. Terminology
function the amount of data, the scatter in the data, and the
3.1 Definitions of Terms Specific to This Standard:
specified level of statistical confidence. The resulting value
3.1.1 analyst, n—the person responsible for performing a
represents how large the discontinuity with POD equal to p
POD analysis on hit/miss data resulting from a POD examina-
could be when uncertainty associated with estimating a is
p
tion.
accounted for. Hence a > a . Note that POD is equal to p for
p/c p
both a and a . a isbasedsolelyonthehit/missdataresulting
3.1.2 demonstrated probability of detection, n—the calcu-
p/c p p
from the examination and represents a snapshot in time,
lated POD value resulting from the statistical analysis on the
whereas a accounts for the uncertainty associated with
hit miss data.
p/c
limited sample data.
This practice is under the jurisdiction of ASTM Committee E07 on Nonde-
4. Summary of Practice
structive Testing and is the direct responsibility of Subcommittee E07.10 on
Specialized NDT Methods.
4.1 This practice describes step-by-step the process for
Current edition approved Jan. 15, 2012. Published February 2012. DOI:10.1520/
E2862-12. analyzing nondestructive testing hit/miss data resulting from a
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
POD examination, including minimum requirements for vali-
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
dating the resulting POD curve.
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
4.2 This practice also includes definitions and discussions
Available from Standardization Documents Order Desk, DODSSP, Bldg. 4,
for results of interest (for example, a ) to provide for
Section D, 700 Robbins Ave., Philadelphia, PA 19111-5098, http:// 90/95
dodssp.daps.dla.mil. correct interpretation of results.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2862 − 12
5. Significance and Use 6.3.2 If a discontinuity was missed because it was ob-
structed (such as a clogged discontinuity), the discontinuity
5.1 The POD analysis method described herein is based on
shall be removed from the POD analysis since there was not an
a well-known and well established statistical method. It shall
opportunity for the discontinuity to be found. If a discontinuity
be used to quantify the demonstrated POD for a specific set of
is removed from the analysis, the specific discontinuity and
examination parameters and known range of discontinuity
rationale for removal shall be documented in the final report.
sizes when the initial response from a nondestructive evalua-
6.3.3 POD cannot be modeled as a continuous function of
tion inspection system is ultimately binary in nature (that is, hit
discontinuity size if there is a complete separation of misses
or miss). This method requires that a relationship between
and hits as crack size increases. If a complete separation of
discontinuity size and POD exists and is best described by a
misses and hits is present in the data, the POD examination
generalized linear model with the appropriate link function for
may be re-administered. If this occurs, it shall be documented
binary outcomes.
in the report. If a complete separation of misses and hits occurs
5.2 Prior to performing the analysis it is assumed that the
on a regular basis, the specimen set should be examined for
discontinuity of interest is clearly defined; the number and
suitability as a POD examination specimen set.
distribution of induced discontinuity sizes in the POD speci-
6.3.4 POD cannot be modeled as a continuous function of
men set is known and well-documented; discontinuities in the
discontinuitysizeifallthediscontinuitiesarefoundorifallthe
POD specimen set are unobstructed; the POD examination
discontinuities are missed. If this occurs, the specimen set is
administration procedure (including data collection method) is
inadequate for the POD examination.
well-defined, under control, and unbiased; and the initial
6.4 Theanalystshalluseageneralizedlinearmodelwiththe
responseisultimatelybinaryinnature(thatis,hitormiss).The
appropriate link function to establish the relationship between
analysis results are only valid if convergence is achieved and
POD and discontinuity size. For application to POD, the
the model adequately represents the data.
generalized linear model with discontinuity size as the single
5.3 The POD analysis method described herein is consistent predictor variable is typically expressed as g(y) = b +b •a or
0 1
with the analysis method for binary data described in MIL- g(y) = b +b •ln(a), where a or ln(a) is the continuous
0 1
HDBK-1823A, which is included in several widely utilized predictor variable, b is the intercept, b is the slope, y is the
0 1
POD software packages to perform a POD analysis on hit/miss binary response variable, and g(•) is the function that “links”
data. It is also found in statistical software packages that have the binary response with the predictor variable. If predictor
generalized linear modeling capability. This practice requires variables other than discontinuity size are quantifiable factors,
that the analyst has access to either POD software or other a generalized linear model with more than one predictor may
software with generalized linear modeling capability. be used.
6.5 The analyst shall choose the appropriate link function
6. Procedure
based on how well the model fits the observed data. MIL-
HDBK-1823A discusses four different link functions (Logit,
6.1 The POD analysis objective shall be clearly defined by
Probit, Log-Log, Complementary-LogLog) and describes
the responsible engineer or by the customer.
methods for selecting the appropriate one. In general, the logit
6.1.1 The analyst shall obtain the hit/miss data resulting
and probit link functions have worked well in practice for
from the POD examination, which shall include at a minimum
modeling hit/miss data.
the documented known induced discontinuity sizes, whether or
not the discontinuity was found, and any false calls.
6.6 Only hit/miss data for induced discontinuities shall be
used in the development of the generalized linear model. False
6.2 The analyst shall also obtain specific information about
call data shall not be included in the development of the
the POD examination, which shall include at a minimum the
generalized linear model.
specimen standard geometry (for example, flat panels), speci-
6.7 The analyst shall conduct the analysis using software
menstandardmaterial(forexample,Nickel),examinationdate,
number of inspectors, type of inspection method (for example, that has generalized linear modeling capabilities.
line-of-site Level 3 Fluorescent Penetrant Inspection), and
6.8 After running the analysis, the analyst shall verify that
pertinent comments from the inspector(s) and test administra-
convergence has been achieved.The resulting POD curve shall
tor.
not be used if convergence has not been achieved.
6.3 Prior to performing the analysis, the analyst shall
6.9 After verifying convergence, the analyst shall use at a
conduct a preliminary review of the POD examination proce-
minimum the informal model diagnostic methods listed below
dure and resulting hit/miss data to identify any examination
to assess the reliability of the model and verify that the model
administration or data issues. The analyst shall resolve any
adequately fits the data.
issues prior to conducting the POD analysis. Examples of
6.9.1 If included in the analysis output, the analyst shall
examination administration or data issues and possible resolu-
check the number of iterations it took to meet the convergence
tions are:
criterion. If more than twenty iterations were needed to reach
6.3.1 If problems or interruptions occurred during the POD convergence, the model may not be reliable. A statement
examination that may bias the results, the POD examination indicating that convergence was achieved and the number of
should be re-administered. If this occurs, it shall be docu- iterations needed to achieve convergence shall be included in
mented in the report. the report.
E2862 − 12
6.9.2 The analyst shall visually assess the shape of the POD n 2 x
P 5 11
H J
U
curve. (POD curves tend to be s-shaped.) ~x11!·F
~12α,2x12, 2n22x!
where F is the F-statistics with degrees of
6.9.3 The analyst shall visually assess how well the POD
(1–α, 2x+2, 2n–2x)
freedom (2x+2, 2n–2x) and P[F < F ]=1–α.
(1–α, 2x+2, 2n–2x)
curvefitsthedatabycomparinghowwelltherangeoverwhich
This method is consistent with that used in MIL-HDBK-
the POD curve is rising matches the range over which misses
1823A.
begintooverlapwithandtransitiontohitsasdiscontinuitysize
increases.
7. Report
6.9.4 The analyst should also compare an empirical POD
7.1 AtaminimumthefollowinginformationaboutthePOD
curve to the POD curve based on the generalized linear model.
analysis shall be included in the report.
The empirical POD curve shall be used for validation purposes
7.1.1 The specimen standard geometry (for example, flat
only. It shall not be used as a substitute for a POD curve
panels).
resulting from a hit/miss analysis.
7.1.2 Thespecimenstandardmaterial(forexample,Nickel).
6.9.4.1 To create an empirical POD curve, divide the
discontinuity sizes into bins. For example, (0.010 in., 0.020 7.1.3 Examination date.
in.), (0.020 in., 0.030 in.), …, (0.100 in., 0.110 in.), etc. 7.1.4 Number of inspectors.
((0.0254 cm, 0.0508 cm), (0.0508 cm, 0.0762 cm), …, (0.2540
7.1.5 Type of inspection method (for example, line-of-site
cm, 0.2794 cm), etc.). For each bin, calculate the total number
Level 3 Fluorescent Penetrant Inspection).
of discontinuities contained in the bin and how many were
7.1.6 Any comments from the inspector(s) or test adminis-
detected. Calculate the empirical POD in each bin by dividing
trator.
the number detected in the bin over the total number of
7.1.7 The documented known induced discontinuity sizes.
discontinuities in the bin. Plot the empirical POD versus the
7.1.8 Which discontinuities were found and which were
midpointofthebintoobtaintheempiricalPODcurve.Overlay
missed.
the POD curve based on the generalized linear model on the
7.1.9 Any false calls.
empirical POD curve to assess how well the generalized linear
7.1.10 The selected link function.
model fits the data by how well it matches the empirical POD
7.1.11 The generalized linear model coefficients.
curve.
7.1.12 The variance-covariance matrix (if included in the
6.9.5 If applicable, the analyst shall visually assess the
software output).
shape of the confidence bound on the POD curve. The
7.1.13 A statement indicating that convergence was
confidence bound should roughly follow the same shape as the
achieved.
POD curve. If the confidence bound flares out significantly on
7.1.14 The number of iterations needed to achieve conver-
either or both ends or intersects the x-axis, the confidence
gence if included in the output.
bound should be viewed as suspect and may not be reliable.
7.1.15 A plot of the resulting POD curve and confidence
6.9.6 The analyst should assess the impact of data that
bound (if applicable).
appears to be outlying (for example, an early hit in the small
7.1.16 Specific results of interest as required by the analysis
size range or a late miss in the large size range) by removing
objective (for example, a ).
90/95
the outlying value from the data and re-running the analysis to
7.1.17 A statement about the model diagnostic methods
assess its influence on the shape of the POD curve and
used and conclusions.
confidence bound (if applicable). Both analysis results (with
7.1.18 Any deviations from the POD examination proce-
and without the outlying data) shall be included in the report
dure or standard POD analysis.
along with a discussion of the impact to the POD curve and
7.1.18.1 If the POD examination was re-administered, the
confidence bound (if applicable).
original results and rationale for re-administration shall
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.