ASTM E1790-04(2016)e1

Standard Practice for Near Infrared Qualitative Analysis

SIGNIFICANCE AND USE
4.1 NIR spectroscopy is a widely used technique for quantitative analysis, and it is also becoming more widely used for the identification of organic materials, that is, qualitative analysis. In general, however, the concept of qualitative analysis as used in the NIR spectral region differs from that used in the mid-IR spectral region in that NIR qualitative analysis refers to the process of automated comparison of the spectra of unknown materials to the spectra of known materials in order to identify the unknown. This approach constitutes a library search method in which each user generates his own library.  
4.2 Historically, NIR spectroscopy as practiced with classical UV-VIS-NIR instruments using methods similar to those described in Practice E1252 was not considered to be a strong technique for qualitative analysis. Although the positions and intensities of absorption bands in specific wavelength ranges were used to confirm the presence of certain functional groups, the spectra were not considered to be specific enough to allow unequivocal identification of unknown materials. A few important libraries of NIR spectra were developed for qualitative purposes, but the lack of suitable data handling facilities limited the scope of qualitative analysis severely. Furthermore, earlier work was limited almost entirely to liquid samples.  
4.3 Currently, the mid-IR procedure of deducing the structure of an unknown material by method of analysis of the locations, strengths, and positional shifts of individual absorption bands is generally not used in the NIR.  
4.4 With the development of specialized NIR instruments and mathematical algorithms for treating the data, it became possible to obtain a wealth of information from NIR spectra that had hitherto gone unused. While the mathematical algorithms described in this practice can be applied to spectral data in any region, this practice describes their application to the NIR.  
4.5 The application of NIR spectroscopy to...
SCOPE
1.1 This practice covers the use of near-infrared (NIR) spectroscopy for the qualitative analysis of liquids and solids. The practice is written under the assumption that most NIR qualitative analyses will be performed with instruments designed specifically for this region and equipped with computerized data handling algorithms. In principle, however, the practice also applies to work with liquid samples using instruments designed for operation over the ultraviolet (UV), visible, and mid-infrared (IR) regions if suitable data handling capabilities are available. Many Fourier Transform Infrared (FTIR) (normally considered mid-IR instruments) have NIR capability, or at least extended-range beamsplitters that allow operation to 1.2 μm; this practice also applies to data from these instruments.  
1.2 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

General Information

• Status: Published
• Publication Date: 31-Mar-2016
• ICS: 71.040.50 - Physicochemical methods of analysis
• Technical Committee: E13 - Molecular Spectroscopy and Separation Science
• Drafting Committee: E13.11 - Multivariate Analysis

Relations

• Refers: ASTM E131-10 - Standard Terminology Relating to Molecular Spectroscopy
• Effective Date: 01-Mar-2010
• Refers: ASTM E1252-98(2007) - Standard Practice for General Techniques for Obtaining Infrared Spectra for Qualitative Analysis
• Effective Date: 01-Dec-2007
• Refers: ASTM E131-05 - Standard Terminology Relating to Molecular Spectroscopy
• Effective Date: 01-Sep-2005
• Refers: ASTM E1655-04 - Standard Practices for Infrared Multivariate Quantitative Analysis
• Effective Date: 01-Dec-2004
• Refers: ASTM E131-02 - Standard Terminology Relating to Molecular Spectroscopy
• Effective Date: 10-Sep-2002
• Refers: ASTM E131-00a - Standard Terminology Relating to Molecular Spectroscopy
• Effective Date: 10-Sep-2000
• Refers: ASTM E1655-00 - Standard Practices for Infrared Multivariate Quantitative Analysis
• Effective Date: 10-Sep-2000
• Refers: ASTM E1252-98 - Standard Practice for General Techniques for Obtaining Infrared Spectra for Qualitative Analysis
• Effective Date: 10-Mar-1998
• Refers: ASTM E1252-98(2002) - Standard Practice for General Techniques for Obtaining Infrared Spectra for Qualitative Analysis
• Effective Date: 10-Mar-1998

Overview

ASTM E1790-04(2016)e1: Standard Practice for Near Infrared Qualitative Analysis provides guidelines for applying near-infrared (NIR) spectroscopy to the qualitative identification of organic materials in liquids and solids. Developed by ASTM Committee E13, this international standard underlines automated spectral comparison-matching the NIR spectrum of unknown materials with spectra from user-built libraries of known substances. While NIR was historically seen as more suited for quantitative than qualitative analysis, advancements in instrumentation and data algorithms have significantly enhanced its specificity and utility for material identification.

This standard is relevant for laboratories and industries looking to implement NIR qualitative analysis, emphasizing reproducibility, appropriate sample preparations, and robust data handling practices.

Key Topics

• Scope of Application: Provides procedures for applying NIR spectroscopy to both liquids and solids, using instruments specifically designed for NIR or extended-range equipment like FTIR with NIR capabilities.
• Automated Library Search: Focuses on the concept of a user-generated spectral library, utilizing automated comparisons to identify unknowns based on spectral similarity.
• Sample Preparation and Consistency: Highlights the necessity for consistent sample handling (such as grinding solids or fixing optical path length for liquids) across both training (known) and unknown materials to ensure reliable comparison.
• Instrumental Techniques: Supports various NIR measurement methods, including:
  • Transmission
  • Diffuse reflectance
  • Transflectance
  • Fiber optic probes
• Data Analysis Algorithms: Discusses the use of mathematical and chemometric algorithms for spectral classification, including Mahalanobis distance calculations, principal component analysis (PCA), and correlation coefficient methods.
• Limitations and Considerations: Addresses challenges such as low-level contaminants, detection limits, and the need for comprehensive training sets to capture variability within materials.
• SI Units: Specifies all results and measurements using SI units as standard.

Applications

The ASTM E1790-04(2016)e1 standard is widely applicable across several industries for effective material identification and quality control using NIR qualitative analysis:

• Pharmaceuticals: Rapid identification of raw materials and verification of compounds without extensive sample prep.
• Food and Agriculture: Authentication of food products, detection of adulteration, and monitoring ingredient quality.
• Chemicals and Polymers: Verification of feedstocks or finished products, ensuring the consistency of materials.
• Environmental Analysis: Identification of organic contaminants in various matrices.
• Manufacturing Quality Control: On-line or at-line checking of material identity and presence of impurities during production.

By enabling quick, non-destructive, and automated material verification, this NIR qualitative analysis practice enhances workflow efficiency and helps meet regulatory requirements on product identity and quality.

Related Standards

For a comprehensive approach to NIR and molecular spectroscopy, consider the following related ASTM standards:

• ASTM E131: Terminology Relating to Molecular Spectroscopy
• ASTM E1252: Practice for General Techniques for Obtaining Infrared Spectra for Qualitative Analysis
• ASTM E1655: Practices for Infrared Multivariate Quantitative Analysis

These documents provide contextual terminology and complementary analytical practices that support the effective implementation of ASTM E1790 in various laboratory and industrial environments.

---

Keywords: near-infrared spectroscopy, NIR qualitative analysis, ASTM E1790, material identification, spectral library, chemometrics, diffuse reflectance, transmission, sample preparation, quality control.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
´1
Designation: E1790 − 04 (Reapproved 2016)
Standard Practice for
Near Infrared Qualitative Analysis
This standard is issued under the fixed designation E1790; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
ε NOTE—Editorial change was made in Subsection 6.6.3 in April 2016.
1. Scope 3. Terminology
1.1 This practice covers the use of near-infrared (NIR) 3.1 Definitions—For definitions of general terms and sym-
spectroscopy for the qualitative analysis of liquids and solids. bols pertaining to NIR spectroscopy and statistical
The practice is written under the assumption that most NIR computations, refer to Terminology E131.
qualitative analyses will be performed with instruments de- 3.2 Definitions of Terms Specific to This Standard:
signed specifically for this region and equipped with comput-
3.2.1 interactance, n—the phenomenon whereby radiant
erized data handling algorithms. In principle, however, the energy entering the surface of a material is scattered by the
practice also applies to work with liquid samples using
material back to the surface, but at a different portion of the
instruments designed for operation over the ultraviolet (UV), surface.
visible, and mid-infrared (IR) regions if suitable data handling
3.2.1.1 Discussion—This differs from diffuse reflectance,
capabilities are available. Many Fourier Transform Infrared
where the returning radiation exits the same portion of the
(FTIR) (normally considered mid-IR instruments) have NIR
surface of the material as the illuminating radiation entered.
capability, or at least extended-range beamsplitters that allow
3.2.2 training sample (otherwise called a “reference
operation to 1.2 µm; this practice also applies to data from
sample” or “standard”), n—a quantity of material of known
these instruments.
composition or properties, or both, presented to an instrument
1.2 The values stated in SI units are to be regarded as for measurement in order to find relationships between the
standard. No other units of measurement are included in this measurements and the composition or properties, or both, of
standard. the sample.
3.2.2.1 Discussion—This term is typically used in conjunc-
1.3 This standard does not purport to address all of the
tion with computerized methods for ascertaining the relation-
safety concerns, if any, associated with its use. It is the
ships.
responsibility of the user of this standard to establish appro-
Training samples for quantitative analysis (also called
priate safety and health practices and determine the applica-
“calibration samples,” as in Practices E1655) have different
bility of regulatory limitations prior to use.
requirements than training samples used for qualitative
analysis.
2. Referenced Documents
2.1 ASTM Standards:
4. Significance and Use
E131 Terminology Relating to Molecular Spectroscopy
4.1 NIR spectroscopy is a widely used technique for quan-
E1252 Practice for General Techniques for Obtaining Infra-
titative analysis, and it is also becoming more widely used for
red Spectra for Qualitative Analysis
the identification of organic materials, that is, qualitative
E1655 Practices for Infrared Multivariate Quantitative
analysis. In general, however, the concept of qualitative analy-
Analysis
sis as used in the NIR spectral region differs from that used in
the mid-IR spectral region in that NIR qualitative analysis
refers to the process of automated comparison of the spectra of
This practice is under the jurisdiction of ASTM Committee E13 on Molecular
unknown materials to the spectra of known materials in order
Spectroscopy and Separation Science and is the direct responsibility of Subcom-
mittee E13.11 on Multivariate Analysis.
to identify the unknown. This approach constitutes a library
Current edition approved April 1, 2016. Published June 2016. Originally
search method in which each user generates his own library.
approved in 1996. Last previous edition approved in 2010 as E1790 – 04(2010).
DOI: 10.1520/E1790-04R16E01.
4.2 Historically, NIR spectroscopy as practiced with classi-
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
cal UV-VIS-NIR instruments using methods similar to those
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
described in Practice E1252 was not considered to be a strong
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. technique for qualitative analysis. Although the positions and
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
´1
E1790 − 04 (2016)
intensities of absorption bands in specific wavelength ranges 5.1.1 The technique is applicable to liquids, solids, and
were used to confirm the presence of certain functional groups, gases. For analysis of gases, multipath vapor cells capable of
the spectra were not considered to be specific enough to allow
achievingupto100-mpathlengthsmayberequired.Spectraof
unequivocal identification of unknown materials.Afew impor-
vapors and gases may be sensitive to the total sample pressure,
tant libraries of NIR spectra were developed for qualitative
and this has to be determined for each type of sample.
purposes, but the lack of suitable data handling facilities
5.1.2 Unknownsamplestobeidentifiedmaybeprescreened
limited the scope of qualitative analysis severely. Furthermore,
based on criteria other than their NIR spectra (for example,
earlier work was limited almost entirely to liquid samples.
visual inspection). The training samples (that is, the “knowns”
4.3 Currently, the mid-IR procedure of deducing the struc- used to teach the algorithm what different materials look like)
ture of an unknown material by method of analysis of the may also be similarly prescreened and grouped into libraries of
locations, strengths, and positional shifts of individual absorp-
similar materials (for example, liquids and solids). The un-
tion bands is generally not used in the NIR.
known is then compared with only those materials in the
appropriate library. The prescreening will help reduce the
4.4 With the development of specialized NIR instruments
chance of false identification, although care must be taken that
and mathematical algorithms for treating the data, it became
an unknown material not in the library is not identified as a
possible to obtain a wealth of information from NIR spectra
similar material that is in the library.
that had hitherto gone unused. While the mathematical algo-
rithms described in this practice can be applied to spectral data
5.1.3 Measurements may be made by method of
in any region, this practice describes their application to the
transmission, reflection, or any other optical setup suitable for
NIR.
collecting NIR spectra. In practice, only transmission and
diffuse reflection have been in common use.
4.5 The application of NIR spectroscopy to qualitative
analysis in the manner described is relatively new, and proce-
5.1.4 Determination of the relationships between absor-
dures for this application are still evolving. The application of
bances at different wavelengths for a set of materials and
chemometric methods to spectroscopy has limitations, and the
consolidation of these relationships into a set of criteria for
limitations are not all defined yet since the techniques are
identifying those materials requires the use of computerized
relatively new. One area of concern to some scientists is the
learning algorithms. These algorithms can also take into
effect of low-level contaminants. Any analytical methodology
account extraneous variations such as are found, for example,
has its detection limits, and NIR is no different in this regard,
when measurements are made on powdered solids.
but neither would we expect it to be any worse. Since the
5.1.5 Instrumentation is commercially available for making
relatively broad character of NIR bands makes it unlikely that
suitable measurements in the NIR spectral region. Manufac-
a contaminant would not overlap any of the measured
turer’s instructions should be followed to ensure correct
wavelengths, the question would only be one of degree:
operation,optimumaccuracy,andsafetybeforecollectingdata.
whether a given amount of contaminant could be detected.The
5.1.6 NIR spectroscopy has, as one of its paradigms, that
user must be aware of the probable contaminants he is liable to
little or no sample preparation be required. In conformance
run into and account for the possibility of this occurring,
with that paradigm, sample preparation steps in other spectro-
perhaps by including deliberately contaminated samples in the
scopic technologies are replaced with sample presentation
training set.
methodologies in NIR analysis. The most common sample
5. General
presentation methods are the following:
5.1 NIR qualitative analysis is conducted by comparison of
5.1.6.1 Diffuse Reflectance—Solid materials are ground into
NIR absorption spectra of unknown materials with those of
powder (or used as-is, if already in suitably fine powder form)
knownreferencematerials.Sincetheabsorptionbandsofmany
and packed into a cup, which allows the surface of the sample
substances of interest are less distinctive in the NIR than in the
to be illuminated and the reflected radiant power measured.
mid-IR spectral region, the analytical capability of the tech-
5.1.6.2 “Transflectance”—Clear or scattering liquids are
nique relies heavily on the accuracy of the absorption mea-
placed in a cup containing a transparent window with a
surements and the relationship of the relative absorbances at
diffusely reflecting material behind the sample. Any radiant
different wavelengths. Materials to be identified are measured
energy passing through the sample is reflected diffusely by the
by a NIR spectrometer, and the spectral data thus generated are
backingmaterial,sothenetmeasurementisjustlikethediffuse
saved in an auxiliary computer attached to the spectrometer
reflectance measurement of powdered solids.
proper. One of the several algorithms described in Section 6 is
5.1.6.3 Transmission—Liquids or solids are placed in cells
then applied to the data in order to generate classification
with two transparent windows and measured by transmission.
criteria, which can then be applied to data from unknown
5.1.6.4 Fiber Probes—Illuminating and collecting fibers are
samples in order to classify (or identify) them as being the
brought in parallel to the sample. A variety of optical configu-
same as one of the previously seen materials. Good chemical
rations are used to couple the radiant energy from the fibers to
laboratory practice should be followed to help ensure repro-
the sample and back again, in an optical “head” of some sort.
ducible results for each material. The preparation and presen-
tation of samples to the instrument should be consistent within Transmittance, reflectance, and interactance have all been used
at the sample end of the fiber to couple the radiation to the
a library, and unknowns should be treated the same way that
the training samples were. sample.Interactancemeasurementsaresometimesmadebythe
´1
E1790 − 04 (2016)
simple expedient of pressing the end of a fiber bundle materials in a given library, and that procedure should be
containing mixed illuminating and receiving fibers against the specified as part of the method.
sample surface.
5.3.3 The unknowns must also be treated in the same
manner as the training samples. It is particularly important that
5.2 To connect the mathematics with the spectroscopy used,
if the samples must be ground, the unknown samples should be
the procedure can be generally described as follows:
groundtothesameparticlesizeastheknownsamplesincluded
(1) The spectral measurements define some multidimen-
in the library.
sional space. The axes in that space are the absorbances at the
various wavelengths, or some mathematical transformation
6. Algorithms Used
thereof.
(2) Groups of spectra for the same material define some
6.1 This section describes some of the computerized algo-
region in the multidimensional space.
rithms that have been found effective for qualitative analysis in
(3) The analysis involves determining which region the
the NIR spectral region. This section is mainly for reference.
unknown falls in.
Descriptionsofmultivariatemethodsofstatisticaldataanalysis
5.2.1 Problems with this type of analysis include the fol-
tend to be inherently abstract mathematically and resistant to
lowing:insufficientseparationofthegroupsinthemultidimen-
reduction to words. A number of books exist in both the
sional space to allow for classification (indicating insufficient
statistical and chemometric literature that describe methods of
differences among the spectra of the materials involved),
multivariate analysis at varying levels of mathematical abstrac-
inadequate representation of measurement variability within
tion (see, for example, Refs (1-5), a useful starting point but
groups during training (indicating an insufficient number or
far from exhaustive list); most of the algorithms used for NIR
variety of training samples), or poor detection limits for minor
qualitative analysis are relatively straightforward applications
contaminants.
of these methods.
5.2.2 To optimize the methods against these potential prob-
6.1.1 Implementations of these algorithms are available in
lem areas, generation of a method occurs in three stages. In the
standard generic statistical software packages. Software pro-
first, or training stage, known samples are presented to the
grams designed for analysis of spectroscopic data may also
instrument. The data collected are then presented to one of the
contain implementations of these algorithms. In addition, the
various algorithms and are thus used to “train” the algorithm to
manufacturers of modern NIR spectrometers include imple-
recognize the various different materials.
mentations of these algorithms in their proprietary software
5.2.3 In the second, or validation stage, the ability of the
packages that run on the auxiliary computers supplied with the
algorithm to correctly recognize materials not in the training
spectrometers; this approach has the advantage that the soft-
set of samples is tested. Samples measured during the valida-
ware matches the format and nature of the data generated by
tion stage should preferably be in the same phase and physical
the spectrometer. In either case, the details of the algorithms
condition as the known samples were during the training stage.
and their implementations are usually transparent to the user. It
5.2.4 In the third, or use stage, unknown samples are
is the responsibility of the user to ascertain whether any
presented to the instrument, which then compares the data so
particular software package implements the desired algorithm
obtained to the data from the known samples and decides
correctly.
whether the data from the unknown agrees with the data from
6.2 Calculation of Mahalanobis distances has been de-
any of the known materials.The unknown material is classified
scribed (5-9) in the literature directly for application to NIR
as whichever material gives the closest agreement to the data.
spectra. The Mahalanobis distance is a way of measuring
5.2.5 Optionally, the algorithm may provide for the case in
whether a given sample falls within a given region of multi-
whichthedatafromtheunknowndoesnotagreewiththatfrom
dimensional space, since a small distance indicates that the
anyoftheknownssufficientlywelltopermitidentification,and
sample is “close to” the center of the region, and thus within it.
refuse to identify the unknown sample.
The training samples define a region of space so that a
5.3 Samples to be identified during the use stage must be in
multidimensional ellipsoid includes a specified fraction of
the same phase and physical condition as the known samples
these samples; the distance from the center of the region to the
were during the validation stage.
ellipsoid surface (that is, the equivalent of a “diameter”)
5.3.1 Liquids may be run neat or in solution. In either case,
defines the Mahalanobis distance. The Mahalanobis distance is
the optical pathlength of the sample cell should be fixed, be the
calculated from the matrix equation:
same for all liquids to be compared with a given unknown, and
2 t
D 5 ~x 2 x¯ ~i!! M~x 2 x¯ ~i!! (1)
i u u
be specified as part of the method. While an algorithm may be
trainedondataincorporatingvariationsinthesecharacteristics,
where:
greater accuracy will be achieved when extraneous variations
D = Mahalanobis distance of the unknown sample from
i
are reduced. The unknown, of course, should also be run in a th
the center of the ellipsoid for the i member (class of
cell under the same conditions as the training samples. If a
samples) of the library,
solution is used, the amount of dilution should also be
specified.
5.3.2 Some solids may be run as-is if they have one or more
suitably flat surfaces; others may need to be ground. If solid
The boldface numbers in parentheses refer to the list of references at the end of
samples are ground, the same procedure should be used for all this practice.
´1
E1790 − 04 (2016)
6.3.1 The basic steps to performing a PCA-based distance
x = the vector of absorption readings for the unknown
u
measurement are as follows:
sampletobeidentified,takenatdifferentwavelengths,
6.3.1.1 Step 1—A training set, or library of samples, is
x¯(i) = the average of the readings for several different
formed that represents the groups (materials) to be distin-
samples of the type of material representing the ith
guished and so identified. Each group should be represented by
member of the library, and
several samples.
M = matrix inverse of the pooled within-group variance-
6.3.1.2 Step 2—The spectra of the samples or groups are
covariance matrix (described in Appendix X1; see
resolved into principal components. The number of principal
Refs (3, 7) for more details on this and Refs (1-5) for
components necessary for adequate representation of the
more general discussions of the mathematical back-
samples is determined by some measurement of the residual
ground).
variation in the library spectra.
6.2.1 The confidence interval for the Mahalanobis distance
6.3.1.3 Step 3—For each principal component of the PCA-
has been shown to be distributed such that p • D has an F
space group, the mean values and standard deviations are
distribution with k and n-k-1df (9), where p=(n-k-1)/nk, n =
calculated from the sample scores for each member of that
number of spectra and k = number of wavelengths (or
group. Because the principal components are orthogonal, each
frequencies) used.
standard deviation (distance) is in an orthogonal direction.
6.2.2 To train the algorithm, the user should take many
6.3.1.4 Step 4—In order to classify future samples, the
spectra of each standard to introduce the inherent variability of
cross-products of the NIR spectrum of each sample with the
the material into the training data. These readings then define
principal components obtained from the training library are
the region of multidimensional space that is characteristic of
computed. The distance measure from any group is calculated
that group of material; it is important to ensure that the training
using the following equation:
samples do in fact include all of the natural variability of the
material.
D 5 score 2 group /S (2)
~ !
i i i i
6.2.3 Aspecial case of this approach deals with the analysis
where:
of clear (non-scattering) liquids. In this case, pure materials
D = distance along the ith principal component axis
have no inherent variability, so the size of the group, which is
i
from the mean of the scores for that group,
determined by the variability of the samples (and which
score = sample score for the ith principal component,
becomes effectively zero for pure, non-scattering liquids), i
group = mean of the group’s scores for the ith principal
i
collapses to a single point in multidimensional space, that is,
component, and
the “diameter” collapses to practically zero. In this case, the
S = standard deviation of the scores of the i principal
i th
region of acceptance for unknowns is so small that instrument
component for the corresponding group.
noise, or other minor and otherwise unimportant variations of
the measurement conditio
...