Ambient air - Methodology to assess the performance of receptor oriented source apportionment modelling applications for particulate matter

The European Directive on ambient air quality and cleaner air for Europe (2008/50/EC; AQD) identifies different uses for modelling: Assessment, planning, forecast and source apportionment (SA). This document addresses source apportionment modelling and specifies performance tests to check whether given criteria for receptor oriented source apportionment models (RM) are met. The scope of the tests set out in this document is the performance assessment of SA of particulate matter using RM in the context of the European Directives 2004/107/EC and AQD, including the Commission Implementing Decision 2011/850/EU of 12 December 2011. The application of RM does not quantify the spatial origin of particulate matter; hence, this document does not test spatial SA.
This document addresses RM users: practitioners of individual source apportionment studies as well as participants and organizers of source apportionment intercomparison studies. This document is suitable for the evaluation of results of a specific SA modelling system with respect to reference values (a priori known or calculated on the basis of intercomparison participants' values) in the following application areas:
-   Assessment of performance and uncertainties of a modelling system or modelling system set up using the indicators laid down in this document.
-   Testing and comparing different source apportionment outputs in a specific situation (applying an evaluation data set) using the indicators laid down in this document.
-   QA/QC tests every time practitioners run a modelling system.
It should be noted for clarity that the procedures and calculations presented in this document cannot be used to check the performance of a specific SA modelling result without having any a priori reference information about the contributions of sources/source categories.
NOTE   The application of this document implies that the intercomparison is organized and coordinated by an institution with the necessary technical capabilities and independence; the definition of which is beyond the scope of this document.
The principles of RM are summarized in Annex A. An overview of uncertainty sources and recommendations about steps to follow in SA studies are provided in Annex B and Annex C. For further information about SA methodologies, refer to e.g. [1; 2; 3].
There are methodologies different from RM which are widely used to accomplish SA, e.g. source oriented models. These other methodologies cover aspects of SA which are required in the AQD and are not addressed by RM (e.g. allocation of pollutants to geographic emission areas). Performance assessment of such methodologies is out of the scope of this document.

Außenluft - Methodik zur Erfassung der Leistungsfähigkeit von Systemanwendungen zur Quellenzuordnung

Die Europäische Luftqualitätsrichtlinie (Directive on ambient air quality and cleaner air for Europe) (2008/50/EG; AQD) definiert verschiedene Anwendungsgebiete für die Modellierung: Beurteilung, Planung, Vorhersage und Quellzuordnung (engl. source apportionment, SA). Dieses Dokument bezieht sich auf die Quellzuordnungs-Modellierung und spezifiziert Methoden zur Beurteilung, ob festgelegte Gütekriterien für die rezeptororientierten Quellzuordnungsmodelle (Rezeptormodelle, RM) erfüllt werden. Der Anwendungsbereich für die Methoden nach diesem Dokument beschreibt die Leistungsbeurteilung der Quellzuordnung partikulären Materials (PM) mithilfe rezeptororientierter Quellmodellierung (RM) im Kontext der Europäischen Richtlinien 2004/107/EG und AQD einschließlich des Durchführungsbeschlusses der europäischen Kommission 2011/850/EU vom 12. Dezember 2011. Die Anwendung der RM quantifiziert nicht die räumliche Quelle des PM, somit umfasst dieses Dokument keine Prüfung der räumlichen Quellzuordnung.
Dieses Dokument wendet sich an Nutzer der RM: Anwender individueller Quellzuordnungsstudien sowie Teilnehmer und Organisatoren von Quellzuordnungsvergleichsstudien. Dieses Dokument ist für die Auswertung der Ergebnisse eines spezifischen Modellierungssystems zur Quellzuordnung mit Hinsicht auf Referenzwerte (a priori bekannt oder auf Grundlage der Vergleichswerte der Teilnehmer berechnet) in den folgenden Anwendungsbereichen geeignet:
-   Erfassung der Leistungsfähigkeit und Unsicherheiten eines Modellierungssystems oder der Einrichtung eines Modellierungssystems durch Verwendung der in diesem Dokument dargelegten Indikatoren.
-   Prüfen und Vergleichen verschiedener Quellzuordnungsergebnisse in einer spezifischen Situation (Verwenden eines Evaluierungsdatensatzes) mithilfe der in dieser CEN/TS dargelegten Indikatoren.
-   QA/QC-Prüfungen bei jeder Verwendung eines Modellierungssystems durch einen Anwender.
Es sollte zur Verdeutlichung beachtet werden, dass die in diesem Dokument dargestellten Verfahren und Berechnungen nicht für die Prüfung der Leistungsfähigkeit eines spezifischen Quellzuordnungs-Modellierungsergebnisses verwendet werden können, ohne a priori Referenzinformation über die Beiträge der Quellen/Quellenkategorien zu haben.
ANMERKUNG   Die Anwendung dieses Dokuments impliziert, dass die Vergleichsprüfung von einer Institution organisiert und koordiniert ist, die über die nötige technische Ausstattung und Unabhängigkeit verfügt; die Definition einer solchen Institution geht über den Anwendungsbereich dieses Dokuments hinaus.
Die Grundsätze der RM sind in Anhang A zusammengefasst. Eine Übersicht über Messunsicherheitsquellen und Empfehlungen zu Schritten, die in Quellzuordnungsstudien zu befolgen sind, sind in Anhang B und Anhang C dargestellt. Für weitere Informationen zu Quellzuordnungsmethodiken, siehe zum Beispiel [1; 2; 3].
Es gibt neben RM auch andere Methodiken, die häufig zur Quellzuordnung verwendet werden, z. B. quellorientierte Modelle. Diese anderen Methodiken decken Aspekte der Quellzuordnung ab, die von der AQD gefordert und von der RM nicht behandelt werden (z. B. Zuordnung von Schadstoffen zu geographischen Emissionsgebieten). Die Einschätzung der Güte dieser Methodiken liegt außerhalb des Anwendungsbereichs dieses Dokuments.

Air ambiant - Méthode d’évaluation de la performance d’applications d’un système de modélisation de la contribution des sources de particules en suspension de type « récepteur-orienté »

La Directive européenne concernant la qualité de l’air ambiant et un air pur pour l’Europe (2008/50/CE ; AQD) identifie différents usages de la modélisation : évaluation, planification, prévision et contribution des sources (SA). Le présent document concerne la modélisation de la contribution des sources et spécifie les essais de performance permettant de s’assurer que les critères donnés pour les modèles de contribution des sources de type « récepteur-orienté » (RM) sont remplis. Le domaine d’application des essais décrits dans le présent document est l’évaluation de la performance de SA de la matière particulaire à l’aide de RM dans le contexte de la Directive européenne 2004/107/CE et de l’AQD, y compris la Décision d’exécution de la Commission 2011/850/UE du 12 décembre 2011. L’application de RM ne permet pas de quantifier l’origine spatiale de la matière particulaire ; par conséquent, le présent document n’évalue pas la SA spatiale.
Le présent document s’adresse aux utilisateurs de RM et opérateurs d’études de contribution des sources ainsi qu’aux participants et organisateurs de comparaisons interlaboratoires de la contribution des sources. Le présent document est applicable à l’évaluation des résultats d’un système de modélisation SA spécifique par rapport à des valeurs de référence (a priori connues ou calculées d’après les valeurs des participants à la comparaison interlaboratoires) dans les secteurs d’application suivants :
-   Évaluation de la performance et des incertitudes d’un système de modélisation ou d’une configuration du système de modélisation à l’aide des indicateurs énoncés dans le présent document.
-   Essais et comparaison des différents résultats de contribution des sources dans une situation spécifique (application d’un jeu de données d’évaluation) à l’aide des indicateurs énoncés dans le présent document.
-   Essais AQ/CQ à chaque fois qu’un opérateur utilise un système de modélisation.
Il convient de noter que, pour plus de clarté, les modes opératoires et les calculs présentés dans le présent document ne peuvent pas être utilisés pour vérifier la performance d’un résultat de modélisation de SA spécifique sans disposer a priori d’informations de référence sur les contributions des sources/catégories de sources.
NOTE   L’application du présent document implique que la comparaison interlaboratoires soit organisée et coordonnée par un organisme indépendant disposant des compétences techniques nécessaires ; sa définition ne fait pas partie du domaine d’application du présent document.
Les principes des RM sont résumés à l’Annexe A. Une vue d’ensemble des sources d’incertitude et des recommandations relatives aux étapes à respecter lors des études de SA sont fournies aux Annexes B et C. Pour plus d’informations sur les méthodes de SA, voir par exemple [1 ; 2 ; 3].
D’autres méthodes que les RM sont couramment utilisées pour effectuer la SA, par exemple les modèles de type « source-orienté ». Ces autres méthodes couvrent certains aspects de SA qui sont requis dans l’AQD et ne sont pas abordés par les RM (par exemple, attribution de polluants à des zones géographiques d’émission). L’évaluation de la performance de ces méthodes ne fait pas partie du domaine d’application du présent document.

Zunanji zrak - Metodologija za oceno lastnosti aplikacij sistemov za modeliranje porazdelitve virov

General Information

Status
Published
Current Stage
6060 - Definitive text made available (DAV) - Publishing
Due Date
23-Dec-2020
Completion Date
23-Dec-2020

Buy Standard

Technical specification
-TS CEN/TS 17458:2021
English language
29 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day
Technical specification
-TS FprCEN/TS 17458:2019
English language
29 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day

Standards Content (sample)

SLOVENSKI STANDARD
SIST-TS CEN/TS 17458:2021
01-februar-2021
Zunanji zrak - Metodologija za oceno lastnosti aplikacij sistemov za modeliranje
porazdelitve virov
Ambient air - Methodology for the assessment of the performance of source
apportionment modelling system applications

Außenluft - Methodik zur Erfassung der Leistungsfähigkeit von Systemanwendungen zur

Quellenzuordnung

Air ambiant - Méthodologie d'évaluation des performances dapplications de modélisation

de l'attribution des sources et des récepteurs
Ta slovenski standard je istoveten z: CEN/TS 17458:2020
ICS:
13.040.20 Kakovost okoljskega zraka Ambient atmospheres
SIST-TS CEN/TS 17458:2021 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST-TS CEN/TS 17458:2021
---------------------- Page: 2 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458
TECHNICAL SPECIFICATION
SPÉCIFICATION TECHNIQUE
December 2020
TECHNISCHE SPEZIFIKATION
ICS 13.040.01
English Version
Ambient air - Methodology to assess the performance of
receptor oriented source apportionment modelling
applications for particulate matter

Air ambiant - Méthode d'évaluation de la performance Außenluft - Methodik zur Erfassung der

d'applications d'un système de modélisation de la Leistungsfähigkeit von Systemanwendungen zur

contribution des sources de particules en suspension Quellenzuordnung
de type " récepteur-orienté "

This Technical Specification (CEN/TS) was approved by CEN on 24 February 2020 for provisional application.

The period of validity of this CEN/TS is limited initially to three years. After two years the members of CEN will be requested to

submit their comments, particularly on the question whether the CEN/TS can be converted into a European Standard.

CEN members are required to announce the existence of this CEN/TS in the same way as for an EN and to make the CEN/TS

available promptly at national level in an appropriate form. It is permissible to keep conflicting national standards in force (in

parallel to the CEN/TS) until the final decision about the possible conversion of the CEN/TS into an EN is reached.

CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,

Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway,

Poland, Portugal, Republic of North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and

United Kingdom.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION
EUROPÄISCHES KOMITEE FÜR NORMUNG
CEN-CENELEC Management Centre: Rue de la Science 23, B-1040 Brussels

© 2020 CEN All rights of exploitation in any form and by any means reserved Ref. No. CEN/TS 17458:2020 E

worldwide for CEN national Members.
---------------------- Page: 3 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
Contents Page

European foreword ....................................................................................................................................................... 3

1 Scope .................................................................................................................................................................... 4

2 Normative references .................................................................................................................................... 4

3 Terms and definitions ................................................................................................................................... 5

4 Symbols and abbreviations ......................................................................................................................... 7

5 Fundamentals of the evaluation methodology ..................................................................................... 8

6 Arrays of tests ................................................................................................................................................... 9

7 The performance assessment method .................................................................................................... 9

7.1 Evaluation data sets ....................................................................................................................................... 9

7.2 Determination of the SCE reference values and their standard uncertainties ......................... 9

7.2.1 Consensus value from participants .......................................................................................................... 9

7.2.2 Formulation of a consensus value from a synthetic data set ....................................................... 10

8 Performance indicators and other indicators ................................................................................... 10

8.1 Complementary indicators ....................................................................................................................... 10

8.1.1 General ............................................................................................................................................................. 10

8.1.2 Apportioned pollutant mass .................................................................................................................... 10

8.1.3 Number of identified sources .................................................................................................................. 11

8.2 Similarity indicators ................................................................................................................................... 11

8.2.1 General ............................................................................................................................................................. 11

8.2.2 Pearson product-moment correlation coefficient ........................................................................... 12

8.2.3 Standardized Identity Distance (SID, Annex I) .................................................................................. 12

8.2.4 Weighted Distance (WD) ........................................................................................................................... 12

8.3 Performance indicators ............................................................................................................................. 13

8.3.1 General ............................................................................................................................................................. 13

8.3.2 z-scores for the average SCEs ................................................................................................................... 13

8.3.3 RMSE for the SCEs time series ................................................................................................................ 14

8.3.4 Zeta-score for the uncertainty of the SCEs .......................................................................................... 14

Annex A (informative) Principle of receptor oriented models ................................................................. 15

Annex B (informative) Recommended steps for SA studies with receptor oriented models ........ 16

Annex C (informative) Sources of uncertainty in receptor oriented models ...................................... 17

Annex D (informative) Array of tests for individual SA runs (Responsible: Practitioner) ............. 18

Annex E (normative) Array of tests for intercomparisons (Responsible: Coordinator) ................. 19

Annex F (normative) Calculation of the SCE reference values X and X in real-world data

kt k

sets ..................................................................................................................................................................... 20

Annex G (informative) Robust average and standard deviation.............................................................. 21

Annex H (informative) Acceptability range for complementary tests ................................................... 22

Annex I (informative) Geometric meaning of the Standardized Identity Distance ........................... 23

Annex J (normative) Acceptability range for the similarity and performance tests ........................ 25

Annex K (informative) Intercomparison organization and test reports .............................................. 26

Bibliography ................................................................................................................................................................. 28

---------------------- Page: 4 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
European foreword

This document (CEN/TS 17458:2020) has been prepared by Technical Committee CEN/TC 264 “Air

quality”, the secretariat of which is held by DIN.

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. CEN shall not be held responsible for identifying any or all such patent rights.

According to the CEN/CENELEC Internal Regulations, the national standards organisations of the

following countries are bound to announce this Technical Specification: Austria, Belgium, Bulgaria,

Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland,

Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of

North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and the United

Kingdom.
---------------------- Page: 5 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
1 Scope

The European Directive on ambient air quality and cleaner air for Europe (2008/50/EC; AQD) identifies

different uses for modelling: Assessment, planning, forecast and source apportionment (SA). This

document addresses source apportionment modelling and specifies performance tests to check whether

given criteria for receptor oriented source apportionment models (RM) are met. The scope of the tests

set out in this document is the performance assessment of SA of particulate matter using RM in the

context of the European Directives 2004/107/EC and AQD, including the Commission Implementing

Decision 2011/850/EU of 12 December 2011. The application of RM does not quantify the spatial origin

of particulate matter; hence, this document does not test spatial SA.

This document addresses RM users: practitioners of individual source apportionment studies as well as

participants and organizers of source apportionment intercomparison studies. This document is suitable

for the evaluation of results of a specific SA modelling system with respect to reference values (a priori

known or calculated on the basis of intercomparison participants' values) in the following application

areas:

— Assessment of performance and uncertainties of a modelling system or modelling system set up using

the indicators laid down in this document.

— Testing and comparing different source apportionment outputs in a specific situation (applying an

evaluation data set) using the indicators laid down in this document.
— QA/QC tests every time practitioners run a modelling system.

It should be noted for clarity that the procedures and calculations presented in this document cannot be

used to check the performance of a specific SA modelling result without having any a priori reference

information about the contributions of sources/source categories.

NOTE The application of this document implies that the intercomparison is organized and coordinated by an

institution with the necessary technical capabilities and independence; the definition of which is beyond the scope

of this document.

The principles of RM are summarized in Annex A. An overview of uncertainty sources and

recommendations about steps to follow in SA studies are provided in Annex B and Annex C. For further

information about SA methodologies, refer to e.g. [1; 2; 3].

There are methodologies different from RM which are widely used to accomplish SA, e.g. source oriented

models. These other methodologies cover aspects of SA which are required in the AQD and are not

addressed by RM (e.g. allocation of pollutants to geographic emission areas). Performance assessment of

such methodologies is out of the scope of this document.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 13528:2015, Statistical methods for use in proficiency testing by interlaboratory comparison

---------------------- Page: 6 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https://www.iso.org/obp
— IEC Electropedia: available at http://www.electropedia.org/
3.1
candidate modelling system

modelling system or modelling system set up that is being tested in the intercomparison or being applied

in isolation
3.2
candidate source

every source or source category present in the result of a SA candidate modelling system which is tested

with the methodology described in this document
3.3
coordinator

organisation with responsibility for coordinating all activities involved in the operation of an

intercomparison exercise
3.4
contribution-to-species

mass of each single chemical element or compound forming part of a (composite) pollutant apportioned

to a source or source category, expressed as a percentage

Note 1 to entry: Depending on the type of SA modelling applications, this concept could be referred to as:

“contribution by species” in CMB 8.2, “explained variation” in PMF 2, “percentage of species total matrix” in EPA

PMF v3, and “factor fingerprint” in EPA-PMF v5.
3.5
input uncertainty

statistical dispersion of the values that can be reasonably attributed to every single species ambient

concentration in the input matrix of receptor oriented models
3.6
derived source profile

relative chemical composition of the source or source category, resulting from a SA model run, expressed

as parts per unit mass of every chemical species with respect to the total source or source category mass

3.7
factor

underlying latent variable identified by a multivariate analysis SA model (e.g. factor analysis or singular

value decomposition) associated with an air pollution source or source category
3.8
maximum accepted distance

maximum Euclidean distance, on a Cartesian plane, between the point representing the values

(concentration ratios) of a chemical species in two source profiles, one on the abscissa and the other on

the ordinate, and the identity line, which is considered acceptable to indicate the similarity of the two

values
---------------------- Page: 7 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
3.9
output uncertainty

uncertainty attributed to the average source contribution estimate (SCE), expressed in the same units,

for a given time window of a source or source category in a source apportionment application result

3.10
receptor oriented models

models that, using as input the measured concentrations of pollutants at a given site (receptor) in a

defined time window, provide source contribution estimates (SCEs) by applying multivariate analysis

3.11
reference source profile

source profile used as reference for the similarity tests of derived source profiles. In general, the reference

source profiles are chemical profiles obtained from samples collected specifically to characterise the

source or source category and which are available in the literature or in public repositories (e.g.

SPECIATE, SPECIEUROPE)
3.12
reference value

a priori known or calculated source contribution estimate (SCE) attributed to a source or source category,

represented by an evaluation dataset
3.13
results

output obtained by running a candidate modelling system using an evaluation dataset. The source

apportionment results consist of e.g.: source contribution estimates (SCEs) including their output

uncertainty, derived source profiles, source contribution time series and contribution-to-species

3.14
source apportionment

practice of deriving quantitative information about the contribution of sources and/or source categories

to the concentration of pollutants at a given point or area in a defined time window

3.15
source category

group of air pollution sources of the same type that are characterised by similar chemical composition

and/or temporal profile
3.16
source contribution estimate
SCE

amount of the pollutant expressed as mass concentration (e.g. µg m ), attributed to a specific source or

source category in SA model application results
3.17
source of air pollution

any human activity or natural process that causes pollutants to be released into the atmosphere or to be

produced in the atmosphere
---------------------- Page: 8 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
3.18
source profile

relative chemical composition of a (composite) pollutant emitted from an air pollution source expressed

as the ratio of the mass of every single chemical species to the total mass of the pollutant

3.19
source oriented models

models that, starting from emission inventories, meteorological fields and boundary conditions, simulate

the physical and/or chemical atmospheric processes affecting the emitted pollutants

3.20
species

single chemical elements or compounds that compose the studied pollutant when it is a family of

compounds or an aggregate (composite)
3.21
pollutant

collective term meaning the total mass of the substance for which source contribution estimates (SCEs)

are determined using source apportionment methods

Note 1 to entry: It could either be a single chemical compound (e.g. Hg), a family of similar chemical compounds

(e.g. polycyclic aromatic hydrocarbons) or composite: an aggregation of different chemical compounds (e.g.

particulate matter).
4 Symbols and abbreviations
fr tests found-reference tests
ff tests found-found tests

M sum of the SCEs of all the candidate sources k in time step t of a given result

M sum of the time averaged SCEs of all the candidate sources k of a given result

O observed mass concentration of the pollutant in time step t
O time averaged observed mass concentration of the pollutant
Pearson Pearson product-moment correlation coefficient (r )
RM receptor oriented source apportionment model
RMSE uncertainty normalized root mean square error

RMSE root mean square error weighted by the standard deviation of the observations

Pearson product-moment correlation coefficient (Pearson)
SA source apportionment
SCE source contribution estimate (mass concentration)
SID standardized identity distance
WD weighted distance
xkt SCE of candidate source k in time step t
x average SCE of candidate source k
X reference value for the SCE of source k in time step t
---------------------- Page: 9 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
X reference value for the average SCE of source k
u standard uncertainty of X
Xkt kt
u standard uncertainty of X
Xk k
z z-score
σ uncertainty for proficiency test
ζ zeta-score
5 Fundamentals of the evaluation methodology

The goal of the procedures and tests described in this document is to assess the performance of receptor

oriented SA modelling applications. This is done by comparing the results of the tested SA candidate

modelling system using as input pre-defined evaluation data sets with the reference values for the used

data set (details in 7.1 and 7.2).

The evaluation methodology applied to the SA results encompasses three types of tests: complementary

tests, similarity tests and performance tests. The similarity and performance tests are carried out

independently for every candidate source or source category reported in the results. Hereon, the term

“source” includes both source and source category, unless explicitly mentioned.

The complementary tests provide ancillary information about the overall consistency of the SA results.

They do not assess the accuracy of the factor/source contributions’ identification and quantification. The

complementary tests include a) the apportioned pollutant mass test, consisting of the comparison

between the pollutant reference mass (e.g. gravimetric mass) and the sum of the SCEs of all the sources

and b) the comparison among the number of sources in every reported SA result or, if available, with the

reference number of sources (see 8.1).

NOTE For the purposes of this document, the following SCEs are considered: a) the values for every sample or

time step and b) their average for the entire time window represented by the evaluation data set.

The similarity tests assess whether a candidate source in a SA result obtained with a candidate modelling

system can be allocated to a source category. The outcome of these tests is used to select the candidates

for the determination of the reference values (see 7.2.1). To that end, the candidate sources are compared

with reference source profiles (fr tests). In the absence of reference source profiles, all the candidate

sources attributed to a source category are compared among each other (ff tests). The similarity tests

compare the sources based on their chemical profiles, the time series of their SCEs, and their

contribution-to-species. When provided, the uncertainty of the derived source profiles in the reported

results is tested by comparison with the uncertainty of the reference source profiles (see 8.2 on similarity

indicators).

The performance of a candidate SA modelling system is evaluated by comparing the SCEs of the candidate

sources, resulting from runs with a given evaluation data set, with the reference values for that specific

data set. The outcome of the performance test depends on whether the difference between the candidate

and the reference meets a pre-established quality objective (see 8.3).

There are two types of performance tests: a) tests based on the z-scores (ISO 13528:2015), for the

average SCEs of the time window represented in the evaluation data set (x ) and b) tests based on the

uncertainty normalized root mean square error (RMSE ) [4; 5] for the time series of the SCEs (x ).

u kt

Candidates passing the z-score test are considered to have an average SCE for the entire modelled period

comparable with the reference value. Candidates passing the RMSE test are considered to have a SCE

time trend for the modelled period comparable with the reference value. Candidates passing both tests

have a performance which is considered “sufficient” for the purposes of the present document.

---------------------- Page: 10 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
6 Arrays of tests

The tests to apply vary according to the objective of the application: individual run or intercomparison

exercise.

In the individual runs a practitioner executes a source apportionment application using as input a data

set for which the SCE reference values are not available. The array of tests for individual runs is reported

in Annex D.

In intercomparison exercises, more practitioners execute SA models using as input the same evaluation

data set (see 7.1) and the SCE reference values (X) are either available in advance (SA applications using

synthetic evaluation data sets) or are calculated from the participant results. SA applications using

synthetic evaluation data sets are included in this category. The array of tests for intercomparison

exercises are reported in Annex E.
7 The performance assessment method
7.1 Evaluation data sets

The evaluation data sets used as input for the candidate modelling systems are derived from observed

data or are synthetic.

The evaluation data sets consist of a matrix with the concentrations of selected chemical species in the

ambient air measured at a receptor site over a given time-window and with a given time-resolution. The

input uncertainties of the entries in the matrix are provided. The principles for the development of state-

of-the-art data sets to be used as input for RMs are described in [1]. Distributing additional information

about the study site or area (e.g. measured source profiles, pollutant concentrations, meteorological data,

etc.) is optional. Siting information about the monitoring location of the observed data shall be also

distributed.

Synthetic evaluation data sets are created by fixing the contribution of sources in every sample or time

step. The concentration of the chemical species in the synthetic evaluation data sets is mathematically

coherent with the source contributions. To create synthetic evaluation data sets, noise is added using

randomization techniques and attributing an input uncertainty to the entries of the evaluation data set

proportional to the noise e.g. [6].

The result of the performance assessment described in this document is associated with the classification

of the monitoring site location represented in the evaluation data set (e.g. urban background, regional

background, sites close to one specific source) and, therefore, it is not providing evidence about the

performance of the candidate modelling system for different site classifications.

NOTE Estimating the transferability of the performance assessment obtained with one evaluation data set to

classes of sites not present in the evaluation data set is beyond the scope of this document.

7.2 Determination of the SCE reference values and their standard uncertainties
7.2.1 Consensus value from participants

When executing a SA model using real-world evaluation data sets, the true SCEs values are unknown.

Therefore, for this kind of evaluation data set the results reported by participants in intercomparison

exercises are used to calculate the reference values.
---------------------- Page: 11 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)

The reference values for the averaged SCE (X ) and for the SCE in every time step (X ) and their standard

k kt

uncertainties u and u are calculated for every source (k) separately. Such reference values are

Xk Xkt

associated with the evaluation data set of observations that was used as input for the intercomparison.

The best estimators of the reference values and their uncertainties are the robust average and standard

deviation, respectively, of the results that pass the similarity tests and are calculated using the method

described in Annex F and Annex G.
7.2.2 Formulation of a consensus value from a synthetic data set

When the evaluation data set is synthetic, the values of X and u are defined a priori and the

kt Xkt

corresponding reference values X and their uncertainties u are best estimators of the mean and

k Xk
standard deviation of the X population, respectively, for each source (k).
8 Performance indicators and other indicators
8.1 Complementary indicators
8.1.1 General

The application of the complementary tests is outlined in the flow chart in Figure 1. The acceptability

ranges for the complementary tests are reported in Annex H and their use in the arrays of tests is

described in Annex D and Annex E.
Figure 1 — Flow diagram illustrating the complementary tests
8.1.2 Apportioned pollutant mass

This test is applied when the SA aims at estimating all the contributing sources. It aims at comparing the

sum of the mass contributed by the sources with the measured pollutant mass. There are different

methods for the apportioned mass tests:
— direct comparison of the SCEs sum with the pollutant mass,

— analysis of the linear regression parameters between the sum of the SCEs and the mass concentration

of the pollutant in every sample or time step, and
— calculation of the RMSE [6].
---------------------- Page: 12 ----------------------
SIST-TS CEN/TS 17458:2021
CEN/TS 17458:2020 (E)
1 2
M −O
( )
∑ t t
RMSE = (1)

where M is the sum of the modelled SCEs of all the candidate sources in every time step t of a given result,

O is the observed (measured) mass concentration in every time step, σ is the standard deviation of the

t O
observed values and n is the number of considered time steps.
8.1.3 Number of identified sources

The comparison of the number of sources estimated may be accomplished visually using a histogram plot.

A reference number of sources is available only for synthetic evaluation data sets.

8.2 Similarity indicators
8.2.1 General

In the similarity tests, the comparison between sources is accomplished using similarity indicators:

Pearson product-moment correlation coefficient (Pearson) and the standardized identity distance (SID).

In addition, the weighted distance (WD) is used to assess whether the candidates’ declared output

uncertainty of the source profiles in the reported results is coherent with the one of the reference source

profile.

The reference source profiles used for the tests shall be coherent with the site represented in the

evaluation data set and the candidate
...

SLOVENSKI STANDARD
kSIST-TS FprCEN/TS 17458:2019
01-december-2019
[Not translated]
Ambient air - Methodology for the assessment of the performance of source
apportionment modelling system applications

Außenluft - Methodik zur Erfassung der Leistungsfähigkeit von Systemanwendungen zur

Quellenzuordnung

Air ambiant - Méthodologie d'évaluation des performances dapplications de modélisation

de l'attribution des sources et des récepteurs
Ta slovenski standard je istoveten z: FprCEN/TS 17458
ICS:
13.040.20 Kakovost okoljskega zraka Ambient atmospheres
kSIST-TS FprCEN/TS 17458:2019 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
kSIST-TS FprCEN/TS 17458:2019
---------------------- Page: 2 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FINAL DRAFT
TECHNICAL SPECIFICATION
FprCEN/TS 17458
SPÉCIFICATION TECHNIQUE
TECHNISCHE SPEZIFIKATION
October 2019
ICS
English Version
Ambient air - Methodology for the assessment of the
performance of source apportionment modelling system
applications

Air ambiant - Méthodologie d'évaluation des Außenluft - Methodik zur Erfassung der

performances d¿applications de modélisation de Leistungsfähigkeit von Systemanwendungen zur

l'attribution des sources et des récepteurs Quellenzuordnung

This draft Technical Specification is submitted to CEN members for Vote. It has been drawn up by the Technical Committee

CEN/TC 264.

CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,

Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway,

Poland, Portugal, Republic of North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey and

United Kingdom.

Recipients of this draft are invited to submit, with their comments, notification of any relevant patent rights of which they are

aware and to provide supporting documentation.

Warning : This document is not a Technical Specification. It is distributed for review and comments. It is subject to change

without notice and shall not be referred to as a Technical Specification.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION
EUROPÄISCHES KOMITEE FÜR NORMUNG
CEN-CENELEC Management Centre: Rue de la Science 23, B-1040 Brussels

© 2019 CEN All rights of exploitation in any form and by any means reserved Ref. No. FprCEN/TS 17458:2019 E

worldwide for CEN national Members.
---------------------- Page: 3 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
Contents

European foreword ....................................................................................................................................................... 3

1 Scope .................................................................................................................................................................... 4

2 Normative references .................................................................................................................................... 4

3 Terms and definitions ................................................................................................................................... 5

4 Symbols and abbreviations ......................................................................................................................... 7

5 Fundamentals of the evaluation methodology ..................................................................................... 8

6 Arrays of tests ................................................................................................................................................... 9

7 The performance assessment method .................................................................................................... 9

7.1 Evaluation data sets ....................................................................................................................................... 9

7.2 Determination of the SCE reference values and their standard uncertainties ......................... 9

7.2.1 Consensus value from participants .......................................................................................................... 9

7.2.2 Formulation of a consensus value from a synthetic data set ....................................................... 10

8 Performance indicators and other indicators ................................................................................... 10

8.1 Complementary indicators ....................................................................................................................... 10

8.1.1 General ............................................................................................................................................................. 10

8.1.2 Apportioned pollutant mass .................................................................................................................... 10

8.1.3 Number of identified sources .................................................................................................................. 11

8.2 Similarity indicators ................................................................................................................................... 11

8.2.1 General ............................................................................................................................................................. 11

8.2.2 Pearson product-moment correlation coefficient ........................................................................... 11

8.2.3 Standardized Identity Distance (SID, Annex I) .................................................................................. 12

8.2.4 Weighted Distance (WD) ........................................................................................................................... 12

8.3 Performance indicators ............................................................................................................................. 12

8.3.1 General ............................................................................................................................................................. 12

8.3.2 z-scores for the average SCEs ................................................................................................................... 13

8.3.3 RMSE for the SCEs time series ................................................................................................................ 13

8.3.4 Zeta-score for the uncertainty of the SCEs .......................................................................................... 14

Annex A (informative) Principle of receptor oriented models ................................................................. 15

Annex B (informative) Recommended steps for SA studies with receptor oriented models ........ 16

Annex C (informative) Sources of uncertainty in receptor oriented models ...................................... 17

Annex D (informative) Array of tests for individual SA runs (Responsible: Practitioner) ............. 18

Annex E (normative) Array of tests for intercomparisons (Responsible: Coordinator) ................. 19

Annex F (normative) Calculation of the SCE reference values X and X in real-world data

kt k

sets ..................................................................................................................................................................... 20

Annex G (informative) Robust average and standard deviation.............................................................. 21

Annex H (informative) Acceptability range for complementary tests ................................................... 22

Annex I (informative) Geometric meaning of the Standardized Identity Distance ........................... 23

Annex J (normative) Acceptability range for the similarity and performance tests ........................ 25

Annex K (informative) Intercomparison organization and test reports .............................................. 26

Bibliography ................................................................................................................................................................. 28

---------------------- Page: 4 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
European foreword

This document (FprCEN/TS 17458:2019) has been prepared by Technical Committee CEN/TC 264 “Air

quality”, the secretariat of which is held by DIN.
This document is currently submitted to the vote.
---------------------- Page: 5 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
1 Scope

The European Directive on ambient air quality and cleaner air for Europe (2008/50/EC; AQD) identifies

different uses for modelling: Assessment, planning, forecast and source apportionment (SA). This

document addresses source apportionment modelling and specifies performance tests to check whether

given criteria for receptor oriented source apportionment models (RM) are met. The scope of the tests

set out in this document is the performance assessment of SA of particulate matter using RM in the

context of the European Directives 2004/107/EC and AQD, including the Commission Implementing

Decision 2011/850/EU of 12 December 2011. The application of RM does not quantify the spatial origin

of particulate matter; hence, this document does not test spatial SA.

This document addresses RM users: practitioners of individual source apportionment studies as well as

participants and organizers of source apportionment intercomparison studies. This document is suitable

for the evaluation of results of a specific SA modelling system with respect to reference values (a priori

known or calculated on the basis of intercomparison participants' values) in the following application

areas:

— Assessment of performance and uncertainties of a modelling system or modelling system set up using

the indicators laid down in this document.

— Testing and comparing different source apportionment outputs in a specific situation (applying an

evaluation data set) using the indicators laid down in this document.
— QA/QC tests every time practitioners run a modelling system.

It should be noted for clarity that the procedures and calculations presented in this document cannot be

used to check the performance of a specific SA modelling result without having any a priori reference

information about the contributions of sources/source categories.

NOTE The application of this document implies that the intercomparison is organized and coordinated by an

institution with the necessary technical capabilities and independence; the definition of which is beyond the scope

of this document.

The principles of RM are summarized in Annex A. An overview of uncertainty sources and

recommendations about steps to follow in SA studies are provided in Annex B and Annex C. For further

information about SA methodologies, refer to e.g. [1; 2; 3].

There are methodologies different from RM which are widely used to accomplish SA, e.g. source oriented

models. These other methodologies cover aspects of SA which are required in the AQD and are not

addressed by RM (e.g. allocation of pollutants to geographic emission areas). Performance assessment of

such methodologies is out of the scope of this document.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 13528:2015, Statistical methods for use in proficiency testing by interlaboratory comparison

---------------------- Page: 6 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https://www.iso.org/obp
— IEC Electropedia: available at http://www.electropedia.org/
3.1
candidate modelling system

modelling system or modelling system set up that is being tested in the intercomparison or being applied

in isolation
3.2
candidate source

every source or source category present in the result of a SA candidate modelling system which is tested

with the methodology described in this document
3.3
coordinator

organisation with responsibility for coordinating all activities involved in the operation of an

intercomparison exercise
3.4
contribution-to-species

mass of each single chemical element or compound forming part of a (composite) pollutant apportioned

to a source or source category, expressed as a percentage

Note 1 to entry: Depending on the type of SA modelling applications, this concept could be referred to as:

“contribution by species” in CMB 8.2, “explained variation” in PMF 2, “percentage of species total matrix” in EPA

PMF v3, and “factor fingerprint” in EPA-PMF v5.
3.5
input uncertainty

statistical dispersion of the values that can be reasonably attributed to every single species ambient

concentration in the input matrix of receptor oriented models
3.6
derived source profile

relative chemical composition of the source or source category, resulting from a SA model run, expressed

as parts per unit mass of every chemical species with respect to the total source or source category mass

3.7
factor

underlying latent variable identified by a multivariate analysis SA model (e.g. factor analysis or singular

value decomposition) associated with an air pollution source or source category
3.8
maximum accepted distance

maximum Euclidean distance, on a cartesian plane, between the point representing the values

(concentration ratios) of a chemical species in two source profiles, one on the abscissa and the other on

the ordinate, and the identity line, which is considered acceptable to indicate the similarity of the two

values
---------------------- Page: 7 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
3.9
output uncertainty

uncertainty attributed to the average source contribution estimate (SCE), expressed in the same units,

for a given time window of a source or source category in a source apportionment application result

3.10
receptor oriented models

models that, using as input the measured concentrations of pollutants at a given site (receptor) in a

defined time window, provide source contribution estimates (SCEs) by applying multivariate analysis

3.11
reference source profile

source profile used as reference for the similarity tests of derived source profiles. In general, the reference

source profiles are chemical profiles obtained from samples collected specifically to characterise the

source or source category and which are available in the literature or in public repositories (e.g.

SPECIATE, SPECIEUROPE)
3.12
reference value

a priori known or calculated source contribution estimate (SCE) attributed to a source or source category,

represented by an evaluation dataset
3.13
results

output obtained by running a candidate modelling system using an evaluation dataset. The source

apportionment results consist of e.g.: source contribution estimates (SCEs) including their output

uncertainty, derived source profiles, source contribution time series and contribution-to-species

3.14
source apportionment (SA)

practice of deriving quantitative information about the contribution of sources and/or source categories

to the concentration of pollutants at a given point or area in a defined time window

3.15
source category

group of air pollution sources of the same type that are characterised by similar chemical composition

and/or temporal profile
3.16
source contribution estimate (SCE)

amount of the pollutant expressed as mass concentration (e.g. µg m ), attributed to a specific source or

source category in SA model application results
3.17
source of air pollution

any human activity or natural process that causes pollutants to be released into the atmosphere or to be

produced in the atmosphere
3.18
source profile

relative chemical composition of a (composite) pollutant emitted from an air pollution source expressed

as the ratio of the mass of every single chemical species to the total mass of the pollutant

---------------------- Page: 8 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
3.19
source oriented models

models that, starting from emission inventories, meteorological fields and boundary conditions, simulate

the physical and/or chemical atmospheric processes affecting the emitted pollutants

3.20
species

single chemical elements or compounds that compose the studied pollutant when it is a family of

compounds or an aggregate (composite)
3.21
pollutant

collective term meaning the total mass of the substance for which source contribution estimates (SCEs)

are determined using source apportionment methods

Note 1 to entry: It could either be a single chemical compound (e.g. Hg), a family of similar chemical

compounds (e.g. polycyclic aromatic hydrocarbons) or composite: an aggregation of different chemical compounds

(e.g. particulate matter).
4 Symbols and abbreviations
fr tests found-reference tests
ff tests found-found tests

M sum of the SCEs of all the candidate sources k in time step t of a given result

M sum of the time averaged SCEs of all the candidate sources k of a given result

O observed mass concentration of the pollutant in time step t
O time averaged observed mass concentration of the pollutant
Pearson Pearson product-moment correlation coefficient (r )
RM receptor oriented source apportionment model
RMSE uncertainty normalized root mean square error

RMSE root mean square error weighted by the standard deviation of the observations

Pearson product-moment correlation coefficient (Pearson)
SA source apportionment
SCE source contribution estimate (mass concentration)
SID standardized identity distance
WD weighted distance
x SCE of candidate source k in time step t
x average SCE of candidate source k
X reference value for the SCE of source k in time step t
X reference value for the average SCE of source k
u standard uncertainty of X
Xkt kt
u standard uncertainty of X
Xk k
---------------------- Page: 9 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
z z-score
σ uncertainty for proficiency test
ζ zeta-score
5 Fundamentals of the evaluation methodology

The goal of the procedures and tests described in this document is to assess the performance of receptor

oriented SA modelling applications. This is done by comparing the results of the tested SA candidate

modelling system using as input pre-defined evaluation data sets with the reference values for the used

data set (details in 7.1 and 7.2).

The evaluation methodology applied to the SA results encompasses three types of tests: complementary

tests, similarity tests and performance tests. The similarity and performance tests are carried out

independently for every candidate source or source category reported in the results. Hereon, the term

“source” includes both source and source category, unless explicitly mentioned.

The complementary tests provide ancillary information about the overall consistency of the SA results.

They do not assess the accuracy of the factor/source contributions’ identification and quantification. The

complementary tests include a) the apportioned pollutant mass test, consisting of the comparison

between the pollutant reference mass (e.g. gravimetric mass) and the sum of the SCEs of all the sources

and b) the comparison among the number of sources in every reported SA result or, if available, with the

reference number of sources (see 8.1).

NOTE For the purposes of this document, the following SCEs are considered: a) the values for every sample or

time step and b) their average for the entire time window represented by the evaluation data set.

The similarity tests assess whether a candidate source in a SA result obtained with a candidate modelling

system can be allocated to a source category. The outcome of these tests is used to select the candidates

for the determination of the reference values (see 7.2.1). To that end, the candidate sources are compared

with reference source profiles (fr tests). In the absence of reference source profiles, all the candidate

sources attributed to a source category are compared among each other (ff tests). The similarity tests

compare the sources based on their chemical profiles, the time series of their SCEs, and their

contribution-to-species. When provided, the uncertainty of the derived source profiles in the reported

results is tested by comparison with the uncertainty of the reference source profiles (see 8.2 on similarity

indicators).

The performance of a candidate SA modelling system is evaluated by comparing the SCEs of the candidate

sources, resulting from runs with a given evaluation data set, with the reference values for that specific

data set. The outcome of the performance test depends on whether the difference between the candidate

and the reference meets a pre-established quality objective (see 8.3).

There are two types of performance tests: a) tests based on the z-scores (ISO 13528:2015), for the

average SCEs of the time window represented in the evaluation data set (x ) and b) tests based on the

uncertainty normalized root mean square error (RMSE ) [4; 5] for the time series of the SCEs (x ).

u kt

Candidates passing the z-score test are considered to have an average SCE for the entire modelled period

comparable with the reference value. Candidates passing the RMSE test are considered to have a SCE

time trend for the modelled period comparable with the reference value. Candidates passing both tests

have a performance which is considered “sufficient” for the purposes of the present document.

---------------------- Page: 10 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)
6 Arrays of tests

The tests to apply vary according to the objective of the application: individual run or intercomparison

exercise.

In the individual runs a practitioner executes a source apportionment application using as input a data

set for which the SCE reference values are not available. The array of tests for individual runs is reported

in Annex D.

In intercomparison exercises, more practitioners execute SA models using as input the same evaluation

data set (see 7.1) and the SCE reference values (X) are either available in advance (SA applications using

synthetic evaluation data sets) or are calculated from the participant results. SA applications using

synthetic evaluation data sets are included in this category. The array of tests for intercomparison

exercises are reported in Annex E.
7 The performance assessment method
7.1 Evaluation data sets

The evaluation data sets used as input for the candidate modelling systems are derived from observed

data or are synthetic.

The evaluation data sets consist of a matrix with the concentrations of selected chemical species in the

ambient air measured at a receptor site over a given time-window and with a given time-resolution. The

input uncertainties of the entries in the matrix are provided. The principles for the development of state-

of-the-art data sets to be used as input for RMs are described in [1]. Distributing additional information

about the study site or area (e.g. measured source profiles, pollutant concentrations, meteorological data,

etc.) is optional. Siting information about the monitoring location of the observed data shall be also

distributed.

Synthetic evaluation data sets are created by fixing the contribution of sources in every sample or time

step. The concentration of the chemical species in the synthetic evaluation data sets is mathematically

coherent with the source contributions. To create synthetic evaluation data sets, noise is added using

randomization techniques and attributing an input uncertainty to the entries of the evaluation data set

proportional to the noise e.g. [6].

The result of the performance assessment described in this document is associated with the classification

of the monitoring site location represented in the evaluation data set (e.g. urban background, regional

background, sites close to one specific source) and, therefore, it is not providing evidence about the

performance of the candidate modelling system for different site classifications.

NOTE Estimating the transferability of the performance assessment obtained with one evaluation data set to

classes of sites not present in the evaluation data set is beyond the scope of this document.

7.2 Determination of the SCE reference values and their standard uncertainties
7.2.1 Consensus value from participants

When executing a SA model using real-world evaluation data sets, the true SCEs values are unknown.

Therefore, for this kind of evaluation data set the results reported by participants in intercomparison

exercises are used to calculate the reference values.

The reference values for the averaged SCE (X ) and for the SCE in every time step (X ) and their standard

k kt

uncertainties uXk and uXkt are calculated for every source (k) separately. Such reference values are

associated with the evaluation data set of observations that was used as input for the intercomparison.

The best estimators of the reference values and their uncertainties are the robust average and standard

---------------------- Page: 11 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)

deviation, respectively, of the results that pass the similarity tests and are calculated using the method

described in Annex F and Annex G.
7.2.2 Formulation of a consensus value from a synthetic data set

When the evaluation data set is synthetic, the values of X and u are defined a priori and the

kt Xkt

corresponding reference values X and their uncertainties u are best estimators of the mean and

k Xk
standard deviation of the Xkt population, respectively, for each source (k).
8 Performance indicators and other indicators
8.1 Complementary indicators
8.1.1 General

The application of the complementary tests is outlined in the flow chart in Figure 1. The acceptability

ranges for the complementary tests are reported in Annex H and their use in the arrays of tests is

described in Annex D and Annex E.
Figure 1 — Flow diagram illustrating the complementary tests
8.1.2 Apportioned pollutant mass

This test is applied when the SA aims at estimating all the contributing sources. It aims at comparing the

sum of the mass contributed by the sources with the measured pollutant mass. There are different

methods for the apportioned mass tests:
— direct comparison of the SCEs sum with the pollutant mass,

— analysis of the linear regression parameters between the sum of the SCEs and the mass concentration

of the pollutant in every sample or time step, and
— calculation of the RMSE [6].
1 n 2
M −O
( )
∑ t t
RMSE = (1)
---------------------- Page: 12 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)

where M is the sum of the modelled SCEs of all the candidate sources in every time step t of a given result,

O is the observed (measured) mass concentration in every time step, σ is the standard deviation of the

t O
observed values and n is the number of considered time steps.
8.1.3 Number of identified sources

The comparison of the number of sources estimated may be accomplished visually using a histogram plot.

A reference number of sources is available only for synthetic evaluation data sets.

8.2 Similarity indicators
8.2.1 General

In the similarity tests, the comparison between sources is accomplished using similarity indicators:

Pearson product-moment correlation coefficient (Pearson) and the standardized identity distance (SID).

In addition, the weighted distance (WD) is used to assess whether the candidates’ declared output

uncertainty of the source profiles in the reported results is coherent with the one of the reference source

profile.

The reference source profiles used for the tests shall be coherent with the site represented in the

evaluation data set and the candidate sources.

The application of the similarity tests is outlined in the flow chart in Figure 2. The acceptability ranges

for the similarity tests are reported in Annex J and their use in the arrays of tests is described in Annex D

and Annex E. The overall organization of an intercomparison is described in Annex K.

Figure 2 — Flow diagram illustrating the similarity tests
8.2.2 Pearson product-moment correlation coefficient
()x −−xy y
( )
∑ jj
r = (2)
xx−−y y
( ) ( )
∑∑j j
---------------------- Page: 13 ----------------------
kSIST-TS FprCEN/TS 17458:2019
FprCEN/TS 17458:2019 (E)

Where n is the number of items (samples or species) in the sources under comparison x and y [7]. Pearson

values equal or above 0,6 are generally considered acceptable [8]. This indicator is applied to calculate

the similarity between sources on the basis three different types of data: their chemical profiles, their

time tr
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.