ASTM E2862-23

Standard Practice for Probability of Detection Analysis for Hit/Miss Data

SIGNIFICANCE AND USE
5.1 The POD analysis method described herein is based on a well-known and well established statistical regression method. It shall be used to quantify the demonstrated POD for a specific set of examination parameters and known range of discontinuity sizes under the following conditions.  
5.1.1 The initial response from a nondestructive evaluation inspection system is ultimately binary in nature (that is, hit or miss).  
5.1.2 Discontinuity size is the predictor variable and can be accurately quantified.  
5.1.3 A relationship between discontinuity size and POD exists and is best described by a generalized linear model with the appropriate link function for binary outcomes.  
5.2 This practice does not limit the use of a generalized linear model with more than one predictor variable or other types of statistical models if justified as more appropriate for the hit/miss data.  
5.3 If the initial response from a nondestructive evaluation inspection system is measurable and can be classified as a continuous variable (for example, data collected from an Eddy Current inspection system), then Practice E3023 may be more appropriate.  
5.4 Prior to performing the analysis it is assumed that the discontinuity of interest is clearly defined; the number and distribution of induced discontinuity sizes in the POD specimen set is known and well-documented; discontinuities in the POD specimen set are unobstructed; and the POD examination administration procedure (including data collection method) is well-designed, well-defined, under control, and unbiased. The analysis results are only valid if convergence is achieved and the model adequately represents the data.  
5.5 The POD analysis method described herein is consistent with the analysis method for binary data described in MIL-HDBK-1823A, and is included in several widely utilized POD software packages to perform a POD analysis on hit/miss data. It is also found in statistical software packages that have generalized line...
SCOPE
1.1 This practice covers the procedure for performing a statistical analysis on nondestructive testing hit/miss data to determine the demonstrated probability of detection (POD) for a specific set of examination parameters. Topics covered include the standard hit/miss POD curve formulation, validation techniques, and correct interpretation of results.  
1.2 The values stated in inch-pound units are to be regarded as standard. The values given in parentheses are mathematical conversions to SI units that are provided for information only and are not considered standard.  
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 30-Jun-2023
• ICS: 03.120.30 - Application of statistical methods
• ICS: 19.100 - Non-destructive testing
• Technical Committee: E07 - Nondestructive Testing
• Drafting Committee: E07.10 - Specialized NDT Methods

Relations

• Refers: ASTM E1316-24 - Standard Terminology for <?Pub Dtl?>Nondestructive Examinations
• Effective Date: 01-Feb-2024
• Refers: ASTM E3080-23 - Standard Practice for Regression Analysis with a Single Predictor Variable
• Effective Date: 01-Nov-2023
• Refers: ASTM E456-13a(2022)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022
• Refers: ASTM E1316-19b - Standard Terminology for Nondestructive Examinations
• Effective Date: 01-Dec-2019
• Refers: ASTM E3080-19 - Standard Practice for Regression Analysis with a Single Predictor Variable
• Effective Date: 01-Sep-2019
• Refers: ASTM E2586-19e1 - Standard Practice for Calculating and Using Basic Statistics
• Effective Date: 01-Apr-2019
• Refers: ASTM E1316-19 - Standard Terminology for Nondestructive Examinations
• Effective Date: 01-Mar-2019
• Refers: ASTM E1316-18 - Standard Terminology for Nondestructive Examinations
• Effective Date: 01-Jan-2018
• Refers: ASTM E3080-17 - Standard Practice for Regression Analysis
• Effective Date: 01-Nov-2017
• Refers: ASTM E456-13A(2017)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E456-13A(2017)e3 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E1316-17a - Standard Terminology for Nondestructive Examinations
• Effective Date: 15-Jun-2017
• Refers: ASTM E1316-17 - Standard Terminology for Nondestructive Examinations
• Effective Date: 01-Feb-2017
• Refers: ASTM E3080-16 - Standard Practice for Regression Analysis
• Effective Date: 01-Nov-2016
• Refers: ASTM E1316-16a - Standard Terminology for Nondestructive Examinations
• Effective Date: 01-Aug-2016

Overview

ASTM E2862-23: Standard Practice for Probability of Detection Analysis for Hit/Miss Data provides a consistent and rigorous approach for analyzing binary response nondestructive testing (NDT) data. Developed by ASTM International, this standard supports reliable, data-driven assessments of the demonstrated Probability of Detection (POD) for various examination conditions. ASTM E2862-23 is especially relevant where the nondestructive testing system’s responses are inherently binary-classified simply as either a "hit" (detected discontinuity) or a "miss" (undetected discontinuity).

The standard outlines methods to generate, validate, and interpret POD curves using established statistical regression techniques appropriate for binary outcomes. These analyses are essential for understanding and optimizing the effectiveness of NDT inspection systems, enabling organizations to gauge their capability to detect flaws of different sizes with statistical confidence.

Key Topics

• Hit/Miss Data: Focuses on binary inspection results, where each tested discontinuity is recorded as found or missed.
• Predictor Variable: The discontinuity size serves as the primary variable, requiring accurate and well-documented measurements.
• Statistical Modeling: Uses generalized linear models (GLMs) with appropriate link functions (e.g., logit, probit) to relate discontinuity size to probability of detection.
• Validation of Analysis: Stresses proper model selection, verification of model fit, handling of outlier data, and assessment of results using diagnostic checks.
• Reporting: Lists minimum required elements for reports, including model details, specimen set information, POD curve plots, false call rate analysis, and model diagnostics.
• Compliance: Aligns with methods in MIL-HDBK-1823A and recognized international standardization principles.

Applications

ASTM E2862-23 is widely used by industries that rely on nondestructive testing for quality assurance and safety-critical inspections-including aerospace, automotive, energy, construction, and manufacturing. Key practical applications include:

• NDT System Qualification: Demonstrating and documenting the probability that a given inspection technique will detect defects of various sizes under defined conditions.
• Inspection Capability Assessment: Providing quantified evidence for regulatory compliance, supplier audits, or internal quality metrics by linking detection capability directly to discontinuity size.
• POD Study Design and Execution: Supporting robust experiment design, data collection, and statistical assessment for new inspection technologies or process changes.
• Performance Validation: Assisting manufacturers and inspection service providers in meeting customer and regulatory requirements by validating inspection system reliability.
• Procedure Comparison: Evaluating the effect of changes in parameters (equipment, procedures, inspector training) on inspection effectiveness.

Related Standards

ASTM E2862-23 complements and is referenced alongside several other ASTM standards and technical guidelines in nondestructive testing and statistics:

• ASTM E3023 - Practice for Probability of Detection Analysis for Continuous Data (useful when inspection results are not binary).
• MIL-HDBK-1823A - Department of Defense Handbook for NDT System Reliability Assessment.
• ASTM E178 - Practice for Dealing With Outlying Observations.
• ASTM E456 - Terminology Relating to Quality and Statistics.
• ASTM E1316 - Terminology for Nondestructive Examinations.
• ASTM E2586 - Practice for Calculating and Using Basic Statistics.
• ASTM E3080 - Practice for Regression Analysis with a Single Predictor Variable.

Implementing ASTM E2862-23 ensures that NDT hit/miss data are statistically analyzed, results are transparent, and reporting meets international best practices for standardization. This supports evidence-based decision-making, regulatory acceptance, and continuous improvement in nondestructive evaluation methods.

Keywords: ASTM E2862-23, Probability of Detection, POD, hit/miss analysis, NDT, nondestructive testing, binary data, statistical regression, quality assurance, inspection validation.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E2862 − 23
Standard Practice for
Probability of Detection Analysis for Hit/Miss Data
This standard is issued under the fixed designation E2862; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 2.2 Department of Defense Handbook:
MIL-HDBK-1823A Nondestructive Evaluation System Re-
1.1 This practice covers the procedure for performing a
liability Assessment
statistical analysis on nondestructive testing hit/miss data to
determine the demonstrated probability of detection (POD) for
3. Terminology
a specific set of examination parameters. Topics covered
3.1 Definitions—For definitions of terms used in this
include the standard hit/miss POD curve formulation, valida-
practice, refer to Terminology E1316.
tion techniques, and correct interpretation of results.
3.2 Definitions of Terms Specific to This Standard:
1.2 The values stated in inch-pound units are to be regarded
3.2.1 analyst, n—the person responsible for performing a
as standard. The values given in parentheses are mathematical
POD analysis on hit/miss data resulting from a POD examina-
conversions to SI units that are provided for information only
tion.
and are not considered standard.
3.2.2 demonstrated probability of detection, n—the calcu-
1.3 This standard does not purport to address all of the
lated POD value resulting from the statistical analysis of the
safety concerns, if any, associated with its use. It is the
hit/miss data.
responsibility of the user of this standard to establish appro-
priate safety, health, and environmental practices and deter-
3.2.3 false call, n—the perceived detection of a discontinu-
mine the applicability of regulatory limitations prior to use.
ity that is identified as a find during a POD examination when
1.4 This international standard was developed in accor-
no discontinuity actually exists at the inspection site.
dance with internationally recognized principles on standard-
3.2.3.1 Discussion—A synonym for “false call” is “false
ization established in the Decision on Principles for the
positive.”
Development of International Standards, Guides and Recom-
3.2.4 hit, n—an existing discontinuity that is identified as a
mendations issued by the World Trade Organization Technical
find during a POD demonstration examination.
Barriers to Trade (TBT) Committee.
3.2.5 miss, n—an existing discontinuity that is missed dur-
ing a POD examination.
2. Referenced Documents
3.2.6 probability of detection (POD), n—the fraction of
2.1 ASTM Standards:
nominal discontinuity sizes expected to be found given their
E178 Practice for Dealing With Outlying Observations
existence.
E456 Terminology Relating to Quality and Statistics
E1316 Terminology for Nondestructive Examinations 3.3 Symbols:
E2586 Practice for Calculating and Using Basic Statistics
3.3.1 a—discontinuity size.
E3023 Practice for Probability of Detection Analysis for â
3.3.2 a —the discontinuity size that can be detected with
p
Versus a Data
probability p.
E3080 Practice for Regression Analysis with a Single Pre-
3.3.2.1 Discussion—Each discontinuity size has an indepen-
dictor Variable
dent probability of being detected and corresponding probabil-
ity of being missed. For example, being able to detect a specific
discontinuity size with probability p does not guarantee that a
This practice is under the jurisdiction of ASTM Committee E07 on Nonde-
larger size discontinuity will be found.
structive Testing and is the direct responsibility of Subcommittee E07.10 on
3.3.3 a —the discontinuity size that can be detected with
Specialized NDT Methods.
p/c
Current edition approved July 1, 2023. Published August 2023. Originally
probability p with a statistical confidence level of c.
approved in 2012. Last previous edition approved in 2018 as E2862 – 18.
DOI:10.1520/E2862-23.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Available from Standardization Documents Order Desk, DODSSP, Bldg. 4,
Standards volume information, refer to the standard’s Document Summary page on Section D, 700 Robbins Ave., Philadelphia, PA 19111-5098, http://
the ASTM website. dodssp.daps.dla.mil.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2862 − 23
3.3.3.1 Discussion—According to the formula in MIL- 5.4 Prior to performing the analysis it is assumed that the
HDBK-1823A, a is a one-sided upper confidence bound on discontinuity of interest is clearly defined; the number and
p/c
a , a represents how large the true a could be given the
distribution of induced discontinuity sizes in the POD speci-
p p/c p
statistical uncertainty associated with limited sample data.
men set is known and well-documented; discontinuities in the
Hence a > a . Note that POD is equal to p for both a and
POD specimen set are unobstructed; and the POD examination
p/c p p/c
a . a is based solely on the hit/miss data resulting from the
administration procedure (including data collection method) is
p p
examination and represents a snapshot in time, whereas a
p/c well-designed, well-defined, under control, and unbiased. The
accounts for the uncertainty associated with limited sample
analysis results are only valid if convergence is achieved and
data.
the model adequately represents the data.
5.5 The POD analysis method described herein is consistent
4. Summary of Practice
with the analysis method for binary data described in MIL-
4.1 In general, the POD examination process is comprised
HDBK-1823A, and is included in several widely utilized POD
of specimen set design, study design, examination
software packages to perform a POD analysis on hit/miss data.
administration, statistical analysis of examination data, docu-
It is also found in statistical software packages that have
mentation of analysis results, and specimen set maintenance.
generalized linear modeling capability. This practice requires
This practice is focused only on and describes step-by-step the
that the analyst has access to either POD software or other
process for analyzing nondestructive testing hit/miss data
software with generalized linear modeling capability.
resulting from a POD examination and includes minimum
requirements for validating the resulting POD curve and
5.6 This practice does not apply to hit/miss data resulting
documenting the results.
from a POD examination based on the Point Estimate Method
4.2 This practice also includes definitions and discussions (PEM), also referred to as the “29 out of 29” method. (See
for results of interest (for example, a ) to provide for X1.2.4.5 for more detail.)
90/95
correct interpretation of results.
6. Procedure
4.3 Definitions of statistical terminology used in the body of
this practice can be found in Annex A1.
6.1 The POD analysis objective shall be clearly defined by
4.4 A more general discussion of the POD analysis process the responsible engineer or by the customer.
can be found in Appendix X1.
6.2 The analyst shall obtain the hit/miss data resulting from
4.5 An example POD analysis using simulated data can be
the POD examination, which shall include at a minimum the
found in Appendix X2.
documented known induced discontinuity sizes, whether or not
the discontinuity was found, and any false calls.
4.6 A mathematical overview of the underlying model
commonly used with hit/miss data resulting from a POD
6.3 The analyst shall also obtain specific information about
examination can be found in Appendix X3.
the POD examination, which shall include at a minimum the
specimen standard geometry (for example, flat panels), speci-
5. Significance and Use
men standard material (for example, Nickel), examination date,
5.1 The POD analysis method described herein is based on
number of inspectors, type of inspection method (for example,
a well-known and well established statistical regression
line-of-site Level 3 Sensitivity Fluorescent Penetrant
method. It shall be used to quantify the demonstrated POD for
Inspection), and pertinent comments from the inspector(s) and
a specific set of examination parameters and known range of
test administrator.
discontinuity sizes under the following conditions.
6.3.1 In general, the results of an experiment apply to the
5.1.1 The initial response from a nondestructive evaluation
conditions under which the experiment was conducted. Hence,
inspection system is ultimately binary in nature (that is, hit or
the POD analysis results apply to the conditions under which
miss).
the POD examination was conducted.
5.1.2 Discontinuity size is the predictor variable and can be
accurately quantified.
6.4 Prior to performing the analysis, the analyst shall
5.1.3 A relationship between discontinuity size and POD
conduct a preliminary review of the POD examination proce-
exists and is best described by a generalized linear model with
dure and resulting hit/miss data to identify any examination
the appropriate link function for binary outcomes.
administration or data issues. The analyst shall identify and
5.2 This practice does not limit the use of a generalized attempt to resolve any issues prior to conducting the POD
linear model with more than one predictor variable or other analysis. Identified issues and their resolution shall be docu-
types of statistical models if justified as more appropriate for
mented in the report. Examples of issues that could arise and
the hit/miss data. possible resolutions are outlined in the following subsections:
6.4.1 If the examination procedure was poorly designed or
5.3 If the initial response from a nondestructive evaluation
executed, or both, the validity of the resulting data is question-
inspection system is measurable and can be classified as a
continuous variable (for example, data collected from an Eddy able. In this case, the examination procedure design and
execution should be reevaluated. For design guidelines see
Current inspection system), then Practice E3023 may be more
appropriate. MIL-HDBK-1823A.
E2862 − 23
6.4.2 If the examination procedure was properly designed call data shall not be included in the development of the
but problems or interruptions occurred during the POD exami- generalized linear model.
nation that may bias the results, the POD examination should
6.8 The analyst shall conduct the analysis using software
be re-administered.
that has generalized linear modeling capabilities.
6.4.3 Data that appear to be outlying (for example, an early
6.9 After running the analysis, the analyst shall verify that
hit in the small size range or a late miss in the large size range)
convergence has been achieved. The resulting POD curve shall
should be identified and investigated.
not be used if convergence has not been achieved.
6.4.3.1 If a discontinuity was missed because it was ob-
structed (such as a clogged discontinuity), the discontinuity 6.10 If included in the analysis software output, the analyst
shall be removed from the POD analysis since there was not an shall also assess the significance of the predictor variable in the
opportunity for the discontinuity to be found. model. In general, only significant variables are included in a
regression model. (See X1.2.7.1 for details on assessing
6.4.3.2 If a discontinuity is removed from the analysis, the
significance.)
specific discontinuity and rationale for removal shall be docu-
mented in the final report.
6.11 After verifying convergence and assessing the signifi-
6.4.4 POD cannot be modeled as a continuous function of
cance of the predictor variable, the analyst shall use at a
discontinuity size if there is a complete separation of misses
minimum the informal model diagnostic methods listed below
and hits as crack size increases. If a complete separation of
to assess the reliability of the model and verify that the model
misses and hits is present in the data, the POD examination
adequately fits the data.
may be re-administered. If this occurs, it shall be documented
6.11.1 If included in the analysis output, the analyst shall
in the report. If a complete separation of misses and hits occurs
check the number of iterations it took to meet the convergence
on a regular basis, the specimen set should be examined for
criterion. If more than twenty iterations were needed to reach
suitability as a POD examination specimen set.
convergence, the model may not be reliable. A statement
6.4.5 POD cannot be modeled as a continuous function of
indicating that convergence was achieved and the number of
discontinuity size if all the discontinuities are found or if all the
iterations needed to achieve convergence shall be included in
discontinuities are missed. If this occurs, the specimen set is
the report.
inadequate for the POD examination.
6.11.2 The analyst shall visually assess the shape of the
POD curve. (POD curves tend to be s-shaped.)
6.5 The analyst shall use a generalized linear model with the
6.11.3 The analyst shall visually assess how well the POD
appropriate link function to establish the relationship between
curve fits the data by comparing how well the range over which
POD and discontinuity size. For application to POD, the
the POD curve is rising matches the range over which misses
generalized linear model with discontinuity size as the single
begin to overlap with and transition to hits as discontinuity size
predictor variable is typically expressed as g(p) = b + b •a or
0 1
increases.
g(p) = b + b •ln(a), where a or ln(a) is the continuous
0 1
6.11.4 The analyst should also compare an empirical POD
predictor variable, b is the intercept, b is the slope, p is the
0 1
curve to the POD curve based on the generalized linear model.
probability of a response (that is, p=POD), and g is a function
The empirical POD curve shall be used for validation purposes
(commonly referred to as the “link” function) that maps [0, 1]
only. It shall not be used as a substitute for a POD curve
onto the real number line. If predictor variables other than
resulting from a hit/miss analysis.
discontinuity size are quantifiable factors, a generalized linear
6.11.4.1 To create an empirical POD curve, divide the
model with more than one predictor may be used. (For more
discontinuity sizes into bins. For example, (0.010 in.,
detail on GLMs, see Appendix X3.)
0.020 in.), (0.020 in., 0.030 in.), …, (0.100 in., 0.110 in.), etc.
6.6 The analyst shall choose the appropriate link function
((0.0254 cm, 0.0508 cm), (0.0508 cm, 0.0762 cm), …,
based on how well the model fits the observed data. MIL-
(0.2540 cm, 0.2794 cm), etc.). For each bin, calculate the total
HDBK-1823A discusses four different link functions (Logit,
number of discontinuities contained in the bin and how many
Probit, Log-Log, Complementary-LogLog) and describes
were detected. Calculate the empirical POD in each bin by
methods for selecting the appropriate one. In general, the logit
dividing the number detected in the bin over the total number
and probit link functions have worked well in practice for
of discontinuities in the bin. Plot the empirical POD versus the
modeling hit/miss data. (For more detail on the logit and probit
midpoint of the bin to obtain the empirical POD curve. Overlay
link functions, see Appendix X3.)
the POD curve based on the generalized linear model on the
6.6.1 In general, the appropriateness of a selected model is
empirical POD curve to assess how well the generalized linear
determined by the significance of the predictor variable(s), how
model fits the data by how well it matches the empirical POD
well the model fits the observed data, and how well the
curve. For an example, see Table X2.2 and Fig. X2.4 in
underlying assumptions are met. Hence, model selection may
Appendix X2.
be an iterative process as the appropriateness of the link
6.11.5 The analyst should assess the impact of data that
function, the significance of the predictor variable(s),
appear to be outlying observations (for example, an early hit in
goodness-of-fit, and other underlying assumptions are typically
the small size range or a late miss in the large size range) by
assessed after the model has been developed.
removing the outlying value from the data and re-running the
6.7 Only hit/miss data for induced discontinuities shall be analysis to assess its influence on the shape of the POD curve.
used in the development of the generalized linear model. False Both analysis results (with and without the outlying data) shall
E2862 − 23
NOTE 1—Failure to document pertinent information about the specimen
be included in the report along with a discussion of the impact
set, examination design, examination execution, raw data, and analysis
to the POD curve. (See X2.1.7.5 for an example.) This
method may be considered grounds for disputing the validity of the
assessment does not apply to outlying observations resulting
results.
from an obstructed discontinuity which are removed from the
7.1.1 The specimen standard geometry (for example, flat
analysis per 6.4.3.1.
panels).
6.12 If a c % level of confidence is specified by the
7.1.2 The specimen standard material (for example, Nickel).
responsible engineer or the customer, the analyst shall put a
7.1.3 Examination date.
c % lower confidence bound on the POD curve. Methods for
7.1.4 Number of inspectors.
constructing a confidence bound can be found in MIL-HDBK-
7.1.5 Type of inspection method (for example, line-of-sight
1823A as well as statistics text books on generalized linear
Level 3 Fluorescent Penetrant Inspection).
regression.
7.1.6 Any comments from the inspector(s) or test adminis-
6.12.1 The analyst shall visually assess the shape of the
trator.
confidence bound on the POD curve. The confidence bound
7.1.7 The documented known induced discontinuity sizes.
should roughly follow the same shape as the POD curve. If the
7.1.8 Which discontinuities were found and which were
confidence bound flares out significantly on either or both ends
missed.
or intersects the x-axis, the confidence bound should be viewed
7.1.9 Any false calls.
as suspect and may not be reliable.
7.1.10 The selected link function.
6.12.2 The analyst should assess the impact of data that
7.1.11 The generalized linear model coefficients.
appear to be outlying observations by removing the outlying
7.1.12 The variance-covariance matrix (if included in the
value from the data and re-running the analysis to assess its
software output).
influence on the shape of the confidence bound (if applicable).
7.1.13 A statement indicating that convergence was
Both analysis results (with and without the outlying data) shall
achieved.
be included in the report along with a discussion of the impact
7.1.14 The number of iterations needed to achieve conver-
to the confidence bound (if applicable). This assessment may
gence (if included in the software output).
be done in conjunction with the assessment done on the POD
7.1.15 A plot of the resulting POD curve and confidence
curve as described in 6.11.5. This assessment does not apply to
bound (if applicable).
outlying observations resulting from an obstructed discontinu-
7.1.16 Specific results of interest as required by the analysis
ity which are removed from the analysis per 6.4.3.1.
objective (for example, a ).
90/95
6.13 The analyst shall analyze any false call data and shall
7.1.17 A statement about the model diagnostic methods
report the false call rate at the 50 %, 90 %, and 95 % level of
used and conclusions.
statistical confidence. Acceptable false call rates shall be
7.1.18 Any deviations from the POD examination proce-
determined by the responsible engineer or by the customer.
dure or standard POD analysis.
6.13.1 The false call rate shall be defined as the number of
7.1.18.1 If the POD examination was re-administered, the
false calls divided by the number of opportunities in the
original results and rationale for re-administration shall be
specimen set that do not contain a discontinuity.
documented in the report.
6.13.2 What constitutes a false call shall be clearly defined
7.1.18.2 If a discontinuity is removed from the analysis, the
by the responsible engineer or by the customer.
specific discontinuity and rationale for removal shall be docu-
6.13.3 What constitutes an opportunity in the specimen set
mented in the final report.
that does not contain a discontinuity shall be clearly defined by
7.1.18.3 If the impact of outlying data was assessed, the
the responsible engineer or by the customer.
results shall be included in the report along with an explana-
6.13.4 The Clopper-Pearson binomial method for construct-
tion.
ing confidence intervals for proportions should be used to
7.1.19 Summary of false call analysis, including the follow-
calculate the false call rate at the 50 %, 90 % and 95 % level of
ing.
statistical confidence. The Clopper-Pearson upper 100•(1-α)%
7.1.19.1 Definition of what constitutes a false call.
confidence bound for p is:
7.1.19.2 Definition of what constitutes an opportunity in the
n 2 x
specimen set that does not contain a discontinuity.
P 5 11
U H J
~x11!·F
7.1.19.3 False call rate at the 50 %, 90 %, and 95 % level of
~12α, 2x12, 2n22x!
where F is the F-statistics with degrees of
confidence.
(1–α, 2x+2, 2n–2x)
freedom (2x+2, 2n–2x) and P[F < F ]=1–α.
(1–α, 2x+2, 2n–2x)
7.1.20 Name of analyst and company responsible for the
This method is consistent with that used in MIL-HDBK-
POD calculation.
1823A.
7. Report 8. Keywords
7.1 At a minimum the following information about the POD 8.1 hit/miss analysis; penetrant POD; POD; POD analysis;
analysis shall be included in the report. Probability of Detection
E2862 − 23
ANNEX
(Mandatory Information)
A1. TERMINOLOGY
A1.1 Definitions: Poisson. The function relating the mean to the linear combi-
nation of independent variables is called the link function.
A1.1.1 a —the discontinuity size that can be detected with
A1.1.6.2 Discussion—Generalized linear models are the
90 % probability.
basis for the hit/miss POD analysis method described in
A1.1.1.1 Discussion—The value for a resulting from a
MIL-HDBK-1823A. See Appendix X3 for an overview of
POD analysis is a single point estimate of the true value based
GLMs.
on the outcome of the POD examination. It represents the
A1.1.7 independent variable, n—a variable used to predict
typical value and does not account for variability due to
another using an equation. Terminology E456, Practice
sampling or inherent variability in the inspection system,
E3080
which is always present.
A1.1.8 outlying observation, n—an extreme observation in
A1.1.2 a —the discontinuity size that can be detected
90/95
either direction that appears to deviate markedly in value from
with 90 % probability with a statistical confidence level of
other members of the sample in which it appears. Practice
95 %.
E178, Terminology E456
A1.1.2.1 Discussion—The value for a resulting from a
A1.1.9 regression, n—the process of estimating param-
POD analysis is an estimate of the true a based on the
eter(s) of an equation using a set of data. Terminology E456,
outcome of the POD examination. If the examination were
Practice E3080
repeated, the outcome is not expected to be exactly the same.
Hence the estimate of a will not be the same. To account for
A1.1.10 sample, n—a group of observations or test results,
variability due to sampling, a statistical confidence bound with
taken from a larger collection of observations or test results,
a 95 % level of confidence is applied to the estimated value for
which serves to provide information that may be used as a basis
a resulting in an a value. POD is still 90 %. The 95 %
for making a decision concerning the larger collection. Termi-
90 90/95
refers to the ability of the statistical method to capture (or
nology E456, Practice E2586
bound) the true a . That is, if the examination were repeated
A1.1.11 sample size, n—number of observed values in the
over and over under the same conditions, the value for a
90/95
sample. Terminology E456, Practice E2586
will be larger than the true a 95 % of the time. In practice the
A1.1.12 standard error, n—standard deviation of the popu-
POD examination will be conducted once. Using a 95 %
lation of values of a sample statistic in repeated sampling, or an
confidence level implies a 95 % chance that the a value
90/95
estimate of it. Terminology E456, Practice E2586
bounds the true a and a 5 % risk that the true a is actually
90 90
A1.1.12.1 Discussion—If the standard error of a statistic is
larger than the a value.
90/95
estimated, it will itself be a statistic with some variance that
A1.1.3 a —the discontinuity size that can be detected
90/50
depends on the sample size.
with 90 % probability with a statistical confidence level of
A1.1.13 statistical confidence, n—the long run frequency
50 %.
associated with the ability of the statistical method to capture
A1.1.3.1 Discussion—Using a one-sided 50 % confidence
the true value of the parameter of interest.
bound implies a 50 % chance that the a value bounds the
90/50
A1.1.13.1 Discussion—Statistical confidence is a probabil-
true a and a 50 % risk that the true a is actually larger than
90 90
ity statement about the statistical method used to estimate a
the a value. Given this, a is really the same as a .
90/50 90/50 90
parameter of interest—for example, the probability that the
A1.1.4 binary response, n—a response variable with only statistical method has captured the true capability of the
two possible outcomes. inspection system. The opposite of statistical confidence can be
equated to risk. For example, a statistical confidence level of
A1.1.4.1 Discussion—The response from a POD examina-
95 % implies a willingness to accept a 5 % risk of the statistical
tion on a manual fluorescent penetrant inspection system, for
method yielding incorrect results—for example, there is a 5 %
example, is binary. The discontinuity is either found or it is
risk that the wrong conclusion has been drawn about the
missed.
capability of the inspection system.
A1.1.5 dependent variable, n—a variable to be predicted
A1.1.14 statistical confidence bound—a one-sided or two-
using an equation. Terminology E456, Practice E3080
sided bound around a single point estimate representing the
A1.1.6 generalized linear model (GLM), n—a model for a
variability due to sampling.
response variable whose distribution is a member of an
A1.1.14.1 Discussion—According to the formula in MIL-
exponential family where the mean response is predicted by a
HDBK-1823A, a is a one-sided upper confidence bound on
p/c
function of a linear combination of independent variables.
a . a represents how large the true a could be given the
p p/c p
A1.1.6.1 Discussion—The exponential family of distribu- statistical uncertainty associated with limited sample data. In
tions includes, for example, normal, binomial, gamma, and general, a confidence bound is a function of the amount of data,
E2862 − 23
the scatter in the data, and the specified level of confidence. inherent process variability. In order to capture inherent pro-
When the sample size increases, statistical uncertainty de- cess variability, a tolerance bound should be used. As opposed
creases (all else held constant). That is, given an infinite to a confidence bound, a tolerance bound will always differ
amount of data (for example, an infinite number of flaw sizes from the point estimate because process variability cannot be
adequately distributed across a POD specimen set), a will eliminated by increasing the sample size.
p/c
approach a because the statistical uncertainty goes away. It is A1.1.14.2 Discussion—The term “statistical confidence
p
important to note that a statistical confidence bound on a only bound” in this practice is equivalent to the term “confidence
p
accounts for variability due to sampling. It does not account for interval” in Terminology E456 and Practice E2586.
APPENDIXES
(Nonmandatory Information)
X1. POD ANALYSIS PROCESS
X1.1 Fig. X1.1 shows a flowchart of POD Analysis for and hits when the discontinuity sizes are sorted in ascending
hit/miss data. order, then the convergence criteria will not be met. If the
responses are all misses or all hits, then the convergence
X1.2 Additional commentary on the POD analysis process
criteria will not be met.
as illustrated in Fig. X1.1 and its significance.
X1.2.3.3 Examples of examination procedure or data issues,
or both, and possible resolutions can be found in 6.4.
X1.2.1 Define POD Analysis Objective—In general, the
objective of a POD analysis is to determine the relationship
X1.2.4 Select Model:
between discontinuity size and POD. Based on the established
X1.2.4.1 Generalized linear models (GLMs) are the tradi-
relationship, the objective may be to determine the discontinu-
tional statistical models used to describe the relationship
ity size that can be detected with a given probability p and
between continuous variables (such as discontinuity size) and
specified statistical confidence level c, denoted a . It is
binary outcomes (such as hit or miss). For binary outcomes, the
p/c
important for the analyst to have a clear understanding of the
form of a generalized linear model with a single predictor
specific analysis objective prior to performing the analysis.
variable is g(p) = b + b •x, where x is the continuous predictor
0 1
variable, b is the intercept, b is the slope, p is the probability
0 1
X1.2.2 Obtain POD Demonstration Test Data and Exami-
of a response (that is, p=POD), and g is a function (commonly
nation Specifics—In general, the results of an experiment apply
referred to as the “link” function) that maps [0, 1] onto the real
to the conditions under which the experiment was conducted. If
number line. This model is the basis for the hit/miss analysis
the examination procedure was poorly designed or executed, or
method as described in MIL-HDBK-1823A. In general, a
both, the validity of the resulting data is questionable.
generalized linear model is the appropriate statistical model for
X1.2.3 Conduct Preliminary Review of Examination Proce-
relating hit/miss data and flaw size since it restricts POD
dure and Data:
predictions to be between 0 and 1. (For more detail on GLMs,
X1.2.3.1 If an experiment is not properly designed and
see Appendix X3.)
executed, the data collected are subject to question and likely X1.2.4.2 In general, the appropriateness of a selected model
invalid. Invalid data cannot be corrected through a statistical
is determined by the significance of the predictor variable(s),
analysis. Hence, any results from a statistical analysis of how well the mod
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2862 − 18 E2862 − 23
Standard Practice for
Probability of Detection Analysis for Hit/Miss Data
This standard is issued under the fixed designation E2862; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This practice definescovers the procedure for performing a statistical analysis on nondestructive testing hit/miss data to
determine the demonstrated probability of detection (POD) for a specific set of examination parameters. Topics covered include
the standard hit/miss POD curve formulation, validation techniques, and correct interpretation of results.
1.2 The values stated in inch-pound units are to be regarded as standard. The values given in parentheses are mathematical
conversions to SI units that are provided for information only and are not considered standard.
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.4 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E178 Practice for Dealing With Outlying Observations
E456 Terminology Relating to Quality and Statistics
E1316 Terminology for Nondestructive Examinations
E2586 Practice for Calculating and Using Basic Statistics
E3023 Practice for Probability of Detection Analysis for â Versus a Data
E3080 Practice for Regression Analysis with a Single Predictor Variable
2.2 Department of Defense Handbook:
MIL-HDBK-1823A Nondestructive Evaluation System Reliability Assessment
3. Terminology
3.1 Definitions—For definitions of terms used in this practice, refer to Terminology E1316.
3.2 Definitions of Terms Specific to This Standard:
This practice is under the jurisdiction of ASTM Committee E07 on Nondestructive Testing and is the direct responsibility of Subcommittee E07.10 on Specialized NDT
Methods.
Current edition approved Feb. 1, 2018July 1, 2023. Published April 2018August 2023. Originally approved in 2012. Last previous edition approved in 20122018 as
E2862 – 12.E2862 – 18. DOI:10.1520/E2862-18.DOI:10.1520/E2862-23.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from Standardization Documents Order Desk, DODSSP, Bldg. 4, Section D, 700 Robbins Ave., Philadelphia, PA 19111-5098, http://dodssp.daps.dla.mil.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2862 − 23
3.2.1 analyst, n—the person responsible for performing a POD analysis on hit/miss data resulting from a POD examination.
3.2.2 demonstrated probability of detection, n—the calculated POD value resulting from the statistical analysis onof the hit/miss
data.
3.2.3 false call, n—the perceived detection of a discontinuity that is identified as a find during a POD examination when no
discontinuity actually exists at the inspection site.
3.2.3.1 Discussion—
A synonym for “false call” is “false positive.”
3.2.4 hit, n—an existing discontinuity that is identified as a find during a POD demonstration examination.
3.2.5 miss, n—an existing discontinuity that is missed during a POD examination.
3.2.6 probability of detection (POD), n—the fraction of nominal discontinuity sizes expected to be found given their existence.
3.3 Symbols:
3.3.1 a—discontinuity size.
3.3.2 a —the discontinuity size that can be detected with probability p.
p
3.3.2.1 Discussion—
Each discontinuity size has an independent probability of being detected and corresponding probability of being missed. For
example, being able to detect a specific discontinuity size with probability p does not guarantee that a larger size discontinuity will
be found.
3.3.3 a —the discontinuity size that can be detected with probability p with a statistical confidence level of c.
p/c
3.3.3.1 Discussion—
According to the formula in MIL-HDBK-1823A, a is a one-sided upper confidence bound on a , a represents how large the
p/c p p/c
true a could be given the statistical uncertainty associated with limited sample data. Hence a > a . Note that POD is equal to
p p/c p
p for both a and a . a is based solely on the hit/miss data resulting from the examination and represents a snapshot in time,
p/c p p
whereas a accounts for the uncertainty associated with limited sample data.
p/c
4. Summary of Practice
4.1 In general, the POD examination process is comprised of specimen set design, study design, examination administration,
statistical analysis of examination data, documentation of analysis results, and specimen set maintenance. This practice is focused
only on and describes step-by-step the process for analyzing nondestructive testing hit/miss data resulting from a POD examination
and includes minimum requirements for validating the resulting POD curve and documenting the results.
4.2 This practice also includes definitions and discussions for results of interest (for example, a ) to provide for correct
90/95
interpretation of results.
4.3 Definitions of statistical terminology used in the body of this practice can be found in Annex A1.
4.4 A more general discussion of the POD analysis process can be found in Appendix X1.
4.5 An example POD analysis using simulated data can be found in Appendix X2.
4.6 A mathematical overview of the underlying model commonly used with hit/miss data resulting from a POD examination can
be found in Appendix X3.
E2862 − 23
5. Significance and Use
5.1 The POD analysis method described herein is based on a well-known and well established statistical regression method. It
shall be used to quantify the demonstrated POD for a specific set of examination parameters and known range of discontinuity sizes
under the following conditions.
5.1.1 The initial response from a nondestructive evaluation inspection system is ultimately binary in nature (that is, hit or miss).
5.1.2 Discontinuity size is the predictor variable and can be accurately quantified.
5.1.3 A relationship between discontinuity size and POD exists and is best described by a generalized linear model with the
appropriate link function for binary outcomes.
5.2 This practice does not limit the use of a generalized linear model with more than one predictor variable or other types of
statistical models if justified as more appropriate for the hit/miss data.
5.3 If the initial response from a nondestructive evaluation inspection system is measurable and can be classified as a continuous
variable (for example, data collected from an Eddy Current inspection system), then Practice E3023 may be more appropriate.
5.4 Prior to performing the analysis it is assumed that the discontinuity of interest is clearly defined; the number and distribution
of induced discontinuity sizes in the POD specimen set is known and well-documented; discontinuities in the POD specimen set
are unobstructed; and the POD examination administration procedure (including data collection method) is well-designed,
well-defined, under control, and unbiased. The analysis results are only valid if convergence is achieved and the model adequately
represents the data.
5.5 The POD analysis method described herein is consistent with the analysis method for binary data described in
MIL-HDBK-1823A, and is included in several widely utilized POD software packages to perform a POD analysis on hit/miss data.
It is also found in statistical software packages that have generalized linear modeling capability. This practice requires that the
analyst has access to either POD software or other software with generalized linear modeling capability.
5.6 This practice does not apply to hit/miss data resulting from a POD examination based on the Point Estimate Method (PEM),
also referred to as the “29 out of 29” method. (See X1.2.4.5 for more detail.)
6. Procedure
6.1 The POD analysis objective shall be clearly defined by the responsible engineer or by the customer.
6.2 The analyst shall obtain the hit/miss data resulting from the POD examination, which shall include at a minimum the
documented known induced discontinuity sizes, whether or not the discontinuity was found, and any false calls.
6.3 The analyst shall also obtain specific information about the POD examination, which shall include at a minimum the specimen
standard geometry (for example, flat panels), specimen standard material (for example, Nickel), examination date, number of
inspectors, type of inspection method (for example, line-of-site Level 3 Sensitivity Fluorescent Penetrant Inspection), and pertinent
comments from the inspector(s) and test administrator.
6.3.1 In general, the results of an experiment apply to the conditions under which the experiment was conducted. Hence, the POD
analysis results apply to the conditions under which the POD examination was conducted.
6.4 Prior to performing the analysis, the analyst shall conduct a preliminary review of the POD examination procedure and
resulting hit/miss data to identify any examination administration or data issues. The analyst shall identify and attempt to resolve
any issues prior to conducting the POD analysis. Identified issues and their resolution shall be documented in the report. Examples
of issues that could arise and possible resolutions are outlined in the following subsections:
6.4.1 If the examination procedure was poorly designed or executed, or both, the validity of the resulting data is questionable. In
this case, the examination procedure design and execution should be reevaluated. For design guidelines see MIL-HDBK-1823A.
E2862 − 23
6.4.2 If the examination procedure was properly designed but problems or interruptions occurred during the POD examination that
may bias the results, the POD examination should be re-administered.
6.4.3 Data that appear to be outlying (for example, an early hit in the small size range or a late miss in the large size range) should
be identified and investigated.
6.4.3.1 If a discontinuity was missed because it was obstructed (such as a clogged discontinuity), the discontinuity shall be
removed from the POD analysis since there was not an opportunity for the discontinuity to be found.
6.4.3.2 If a discontinuity is removed from the analysis, the specific discontinuity and rationale for removal shall be documented
in the final report.
6.4.4 POD cannot be modeled as a continuous function of discontinuity size if there is a complete separation of misses and hits
as crack size increases. If a complete separation of misses and hits is present in the data, the POD examination may be
re-administered. If this occurs, it shall be documented in the report. If a complete separation of misses and hits occurs on a regular
basis, the specimen set should be examined for suitability as a POD examination specimen set.
6.4.5 POD cannot be modeled as a continuous function of discontinuity size if all the discontinuities are found or if all the
discontinuities are missed. If this occurs, the specimen set is inadequate for the POD examination.
6.5 The analyst shall use a generalized linear model with the appropriate link function to establish the relationship between POD
and discontinuity size. For application to POD, the generalized linear model with discontinuity size as the single predictor variable
is typically expressed as g(p) = b + b •a or g(p) = b + b •ln(a), where a or ln(a) is the continuous predictor variable, b is the
0 1 0 1 0
intercept, b is the slope, p is the probability of a response (that is, p=POD), and g is a function (commonly referred to as the “link”
function) that maps [0, 1] onto the real number line. If predictor variables other than discontinuity size are quantifiable factors, a
generalized linear model with more than one predictor may be used. (For more detail on GLMs, see Appendix X3.)
6.6 The analyst shall choose the appropriate link function based on how well the model fits the observed data. MIL-HDBK-1823A
discusses four different link functions (Logit, Probit, Log-Log, Complementary-LogLog) and describes methods for selecting the
appropriate one. In general, the logit and probit link functions have worked well in practice for modeling hit/miss data. (For more
detail on the logit and probit link functions, see Appendix X3.)
6.6.1 In general, the appropriateness of a selected model is determined by the significance of the predictor variable(s), how well
the model fits the observed data, and how well the underlying assumptions are met. Hence, model selection may be an iterative
process as the appropriateness of the link function, the significance of the predictor variable(s), goodness-of-fit, and other
underlying assumptions are typically assessed after the model has been developed.
6.7 Only hit/miss data for induced discontinuities shall be used in the development of the generalized linear model. False call data
shall not be included in the development of the generalized linear model.
6.8 The analyst shall conduct the analysis using software that has generalized linear modeling capabilities.
6.9 After running the analysis, the analyst shall verify that convergence has been achieved. The resulting POD curve shall not be
used if convergence has not been achieved.
6.10 If included in the analysis software output, the analyst shall also assess the significance of the predictor variable in the model.
In general, only significant variables are included in a regression model. (See X1.2.7.1 for details on assessing significance.)
6.11 After verifying convergence and assessing the significance of the predictor variable, the analyst shall use at a minimum the
informal model diagnostic methods listed below to assess the reliability of the model and verify that the model adequately fits the
data.
6.11.1 If included in the analysis output, the analyst shall check the number of iterations it took to meet the convergence criterion.
If more than twenty iterations were needed to reach convergence, the model may not be reliable. A statement indicating that
convergence was achieved and the number of iterations needed to achieve convergence shall be included in the report.
E2862 − 23
6.11.2 The analyst shall visually assess the shape of the POD curve. (POD curves tend to be s-shaped.)
6.11.3 The analyst shall visually assess how well the POD curve fits the data by comparing how well the range over which the
POD curve is rising matches the range over which misses begin to overlap with and transition to hits as discontinuity size increases.
6.11.4 The analyst should also compare an empirical POD curve to the POD curve based on the generalized linear model. The
empirical POD curve shall be used for validation purposes only. It shall not be used as a substitute for a POD curve resulting from
a hit/miss analysis.
6.11.4.1 To create an empirical POD curve, divide the discontinuity sizes into bins. For example, (0.010 in., 0.020 in.), 0.020 in.),
(0.020 in., 0.030 in.), …, (0.100 in., 0.110 in.), etc. ((0.0254 cm, 0.0508 cm), (0.0508 cm, 0.0762 cm), …, (0.2540 cm, (0.2540 cm,
0.2794 cm), etc.). For each bin, calculate the total number of discontinuities contained in the bin and how many were detected.
Calculate the empirical POD in each bin by dividing the number detected in the bin over the total number of discontinuities in the
bin. Plot the empirical POD versus the midpoint of the bin to obtain the empirical POD curve. Overlay the POD curve based on
the generalized linear model on the empirical POD curve to assess how well the generalized linear model fits the data by how well
it matches the empirical POD curve. For an example, see Table X2.2 and Fig. X2.4 in Appendix X2.
6.11.5 The analyst should assess the impact of data that appear to be outlying observations (for example, an early hit in the small
size range or a late miss in the large size range) by removing the outlying value from the data and re-running the analysis to assess
its influence on the shape of the POD curve. Both analysis results (with and without the outlying data) shall be included in the
report along with a discussion of the impact to the POD curve. (See X2.1.7.5 for an example.) This assessment does not apply to
outlying observations resulting from an obstructed discontinuity which are removed from the analysis per 6.4.3.1.
6.12 If a c%c % level of confidence is specified by the responsible engineer or the customer, the analyst shall put a c%c % lower
confidence bound on the POD curve. Methods for constructing a confidence bound can be found in MIL-HDBK-1823A as well
as statistics text books on generalized linear regression.
6.12.1 The analyst shall visually assess the shape of the confidence bound on the POD curve. The confidence bound should
roughly follow the same shape as the POD curve. If the confidence bound flares out significantly on either or both ends or intersects
the x-axis, the confidence bound should be viewed as suspect and may not be reliable.
6.12.2 The analyst should assess the impact of data that appear to be outlying observations by removing the outlying value from
the data and re-running the analysis to assess its influence on the shape of the confidence bound (if applicable). Both analysis
results (with and without the outlying data) shall be included in the report along with a discussion of the impact to the confidence
bound (if applicable). This assessment may be done in conjunction with the assessment done on the POD curve as described in
6.11.5. This assessment does not apply to outlying observations resulting from an obstructed discontinuity which are removed from
the analysis per 6.4.3.1.
6.13 The analyst shall analyze any false call data and shall report the false call rate at the 50 %, 90 %, and 95 % level of statistical
confidence. Acceptable false call rates shall be determined by the responsible engineer or by the customer.
6.13.1 The false call rate shall be defined as the number of false calls divided by the number of opportunities in the specimen set
that do not contain a discontinuity.
6.13.2 What constitutes a false call shall be clearly defined by the responsible engineer or by the customer.
6.13.3 What constitutes an opportunity in the specimen set that does not contain a discontinuity shall be clearly defined by the
responsible engineer or by the customer.
6.13.4 The Clopper-Pearson binomial method for constructing confidence intervals for proportions should be used to calculate the
false call rate at the 50 %, 90 % and 95 % level of statistical confidence. The Clopper-Pearson upper 100•(1-α)% confidence bound
for p is:
n 2 x
P 5 11
U H J
x11 ·F
~ !
12α, 2x12, 2n22x
~ !
where F is the F-statistics with degrees of freedom (2x+2, 2n–2x) and P[F < F ]=1–α. This method
(1–α, 2x+2, 2n–2x) (1–α, 2x+2, 2n–2x)
is consistent with that used in MIL-HDBK-1823A.
E2862 − 23
7. Report
7.1 At a minimum the following information about the POD analysis shall be included in the report.
NOTE 1—Failure to document pertinent information about the specimen set, examination design, examination execution, raw data, and analysis method
may be considered grounds for disputing the validity of the results.
7.1.1 The specimen standard geometry (for example, flat panels).
7.1.2 The specimen standard material (for example, Nickel).
7.1.3 Examination date.
7.1.4 Number of inspectors.
7.1.5 Type of inspection method (for example, line-of-sight Level 3 Fluorescent Penetrant Inspection).
7.1.6 Any comments from the inspector(s) or test administrator.
7.1.7 The documented known induced discontinuity sizes.
7.1.8 Which discontinuities were found and which were missed.
7.1.9 Any false calls.
7.1.10 The selected link function.
7.1.11 The generalized linear model coefficients.
7.1.12 The variance-covariance matrix (if included in the software output).
7.1.13 A statement indicating that convergence was achieved.
7.1.14 The number of iterations needed to achieve convergence (if included in the software output).
7.1.15 A plot of the resulting POD curve and confidence bound (if applicable).
7.1.16 Specific results of interest as required by the analysis objective (for example, a ).
90/95
7.1.17 A statement about the model diagnostic methods used and conclusions.
7.1.18 Any deviations from the POD examination procedure or standard POD analysis.
7.1.18.1 If the POD examination was re-administered, the original results and rationale for re-administration shall be documented
in the report.
7.1.18.2 If a discontinuity is removed from the analysis, the specific discontinuity and rationale for removal shall be documented
in the final report.
7.1.18.3 If the impact of outlying data was assessed, the results shall be included in the report along with an explanation.
7.1.19 Summary of false call analysis, including the following.
7.1.19.1 Definition of what constitutes a false call.
7.1.19.2 Definition of what constitutes an opportunity in the specimen set that does not contain a discontinuity.
7.1.19.3 False call rate at the 50 %, 90 %, and 95 % level of confidence.
E2862 − 23
7.1.20 Name of analyst and company responsible for the POD calculation.
8. Keywords
8.1 hit/miss analysis; Probability of Detection; penetrant POD; POD; POD analysis; penetrant PODProbability of Detection
ANNEX
(Mandatory Information)
A1. TERMINOLOGY
A1.1 Definitions:
A1.1.1 a —the discontinuity size that can be detected with 90 % probability.
A1.1.1.1 Discussion—The value for a resulting from a POD analysis is a single point estimate of the true value based on the
outcome of the POD examination. It represents the typical value and does not account for variability due to sampling or inherent
variability in the inspection system, which is always present.
A1.1.2 a —the discontinuity size that can be detected with 90 % probability with a statistical confidence level of 95 %.
90/95
A1.1.2.1 Discussion—The value for a resulting from a POD analysis is an estimate of the true a based on the outcome of the
90 90
POD examination. If the examination were repeated, the outcome is not expected to be exactly the same. Hence the estimate of
a will not be the same. To account for variability due to sampling, a statistical confidence bound with a 95 % level of confidence
is applied to the estimated value for a resulting in an a value. POD is still 90 %. The 95 % refers to the ability of the statistical
90 90/95
method to capture (or bound) the true a . That is, if the examination were repeated over and over under the same conditions, the
value for a will be larger than the true a 95 % of the time. In practice the POD examination will be conducted once. Using
90/95 90
a 95 % confidence level implies a 95 % chance that the a value bounds the true a and a 5 % risk that the true a is actually
90/95 90 90
larger than the a value.
90/95
A1.1.3 a —the discontinuity size that can be detected with 90 % probability with a statistical confidence level of 50 %.
90/50
A1.1.3.1 Discussion—Using a one-sided 50 % confidence bound implies a 50 % chance that the a value bounds the true a
90/50 90
and a 50 % risk that the true a is actually larger than the a value. Given this, a is really the same as a .
90 90/50 90/50 90
A1.1.4 binary response, n—a response variable with only two possible outcomes.
A1.1.4.1 Discussion—The response from a POD examination on a manual fluorescent penetrant inspection system, for example,
is binary. The discontinuity is either found or it is missed.
A1.1.5 dependent variable, n—a variable to be predicted using an equation. Terminology E456, Practice E3080
E2862 − 23
A1.1.6 generalized linear model (GLM), n—a model for a response variable whose distribution is a member of an exponential
family where the mean response is predicted by a function of a linear combination of independent variables.
A1.1.6.1 Discussion—The exponential family of distributions includes, for example, normal, binomial, gamma, and Poisson. The
function relating the mean to the linear combination of independent variables is called the link function.
A1.1.6.2 Discussion—Generalized linear models are the basis for the hit/miss POD analysis method described in MIL-HDBK-
1823A. See Appendix X3 for an overview of GLMs.
A1.1.7 independent variable, n—a variable used to predict another using an equation. Terminology E456, Practice E3080
A1.1.8 outlying observation, n—an extreme observation in either direction that appears to deviate markedly in value from other
members of the sample in which it appears. Practice E178, Terminology E456
A1.1.9 regression, n—the process of estimating parameter(s) of an equation using a set of data. Terminology E456, Practice
E3080
A1.1.10 sample, n—a group of observations or test results, taken from a larger collection of observations or test results, which
serves to provide information that may be used as a basis for making a decision concerning the larger collection. Terminology
E456, Practice E2586
A1.1.11 sample size, n—number of observed values in the sample. Terminology E456, Practice E2586
A1.1.12 standard error, n—standard deviation of the population of values of a sample statistic in repeated sampling, or an estimate
of it. Terminology E456, Practice E2586
A1.1.12.1 Discussion—If the standard error of a statistic is estimated, it will itself be a statistic with some variance that depends
on the sample size.
A1.1.13 statistical confidence, n—the long run frequency associated with the ability of the statistical method to capture the true
value of the parameter of interest.
A1.1.13.1 Discussion—Statistical confidence is a probability statement about the statistical method used to estimate a parameter
of interest—for example, the probability that the statistical method has captured the true capability of the inspection system. The
opposite of statistical confidence can be equated to risk. For example, a statistical confidence level of 95 % implies a willingness
to accept a 5 % risk of the statistical method yielding incorrect results—for example, there is a 5 % risk that the wrong conclusion
has been drawn about the capability of the inspection system.
A1.1.14 statistical confidence bound—a one-sided or two-sided bound around a single point estimate representing the variability
due to sampling.
A1.1.14.1 Discussion—According to the formula in MIL-HDBK-1823A, a is a one-sided upper confidence bound on a . a
p/c p p/c
represents how large the true a could be given the statistical uncertainty associated with limited sample data. In general, a
p
E2862 − 23
confidence bound is a function of the amount of data, the scatter in the data, and the specified level of confidence. When the sample
size increases, statistical uncertainty decreases (all else held constant). That is, given an infinite amount of data (for example, an
infinite number of flaw sizes adequately distributed across a POD specimen set), a will approach a because the statistical
p/c p
uncertainty goes away. It is important to note that a statistical confidence bound on a only accounts for variability due to sampling.
p
It does not account for inherent process variability. In order to capture inherent process variability, a tolerance bound should be
used. As opposed to a confidence bound, a tolerance bound will always differ from the point estimate because process variability
cannot be eliminated by increasing the sample size.
A1.1.14.2 Discussion—The term “statistical confidence bound” in this practice is equivalent to the term “confidence interval” in
Terminology E456 and Practice E2586.
APPENDIXES
(Nonmandatory Information)
X1. POD ANALYSIS PROCESS
X1.1 Fig. X1.1 shows a flowchart of POD Analysis for hit/miss data.
X1.2 Additional commentary on the POD analysis process as illustrated in Fig. X1.1 and its significance.
X1.2.1 Define POD Analysis Objective—In general, the objective of a POD analysis is to determine the relationship between
discontinuity size and POD. Based on the established relationship, the objective may be to determine the discontinuity size that
can be detected with a given probability p and specified statistical confidence level c, denoted a . It is important for the analyst
p/c
to have a clear understanding of the specific analysis objective prior to performing the analysis.
X1.2.2 Obtain POD Demonstration Test Data and Examination Specifics—In general, the results of an experiment apply to the
conditions under which the experiment was conducted. If the examination procedure was poorly designed or executed, or both,
the validity of the resulting data is questionable.
X1.2.3 Conduct Preliminary Review of Examination Procedure and Data:
X1.2.3.1 If an experiment is not properly designed and executed, the data collected are subject to question and likely invalid.
Invalid data cannot be corrected through a statistical analysis. Hence, any results from a statistical analysis of invalid data will be
invalid as well.
X1.2.3.2 POD cannot be modeled as a continuous function of discontinuity size if there is a complete separation of misses and
hits as crack size increases or if the responses are all misses or all hits. The model coefficients do not have a closed form solution.
As such, an iterative numerical procedure is required to solve the system of equations from which the estimates of the model
coefficients are derived. The procedure iterates until a convergence criterion is met, at which point estimates of the model
coefficients are obtained from the last iteration. The analysis results are not valid unless the convergence criterion is met. Even if
the analysis software outputs model information, the results shall not be used if the convergence criterion has not been met. Prior
to performing the analysis, a preliminary review of the hit/miss data resulting from the POD examination can reveal whether or
not failure to meet the convergence criteria may be an issue. If there is no overlap between misses and hits when the discontinuity
sizes are sorted in ascending order, then the convergence criteria will not be met. If the responses are all misses or all hits, then
the convergence criteria will not be met.
E2862 − 23
FIG. X1.1 Flowchart of POD Analysis for Hit/Miss Data
X1.2.3.3 Examples of examination procedure or data issues, or both, and possible resolutions can be
...