prEN 9247

Aerospace series - Programme management - Verification and validation of numerical models and simulations

This document provides an inventory of best practices, shared by actors from the aerospace and defence sector, concerning the verification and validation (V&V) of numerical simulations and models, in order to ensure the credibility of the outputs obtained in a logic of faster development of decision-making support, of reducing the number of physical tests, of shortening development times, of facilitating numerical qualification and certification, etc. These are all the major challenges concerning simulation.
The approach applies to models based on physical equations.
EXAMPLE   Mechanics, acoustics, electrical, electromagnetism, thermal physics for electronics, fluid dynamics, multibody dynamics, multiphysics, optical, signal integrity and power integrity.
The objective is to determine recommendations depending on the challenges of the simulation, in order to adapt the procedures to be applied to ensure the credibility of the simulation. The items being considered are:
-   criticality of the product and the simulation;
-   complexity of the phenomenon or the product;
-   capability, fidelity and maturity of the model;
-   product lifecycle;
-   skills;
-   verification and validation approach, with uncertainties quantification;
-   etc.
This document is organized as follows:
-   terms and definitions;
-   general principles and concepts of simulation V&V:
o   the document’s objectives and added value;
o   state of the art;
o   different uses of simulation depending on the maturity (approximation level) of the model and product lifecycle, linked to the expected fidelity of the model and the simulation outputs;
o   presentation of the different types of models and impacts on criteria and quantities of interest, as well as on requirements;
-   recommended V&V process and activities (linked to the degree of maturity);
-   an example of a simulation plan template;
-   examples for a clearer understanding.
The aim of this document is to complete and reference the information available in the literature. This document takes a generic approach so that it is applicable by most organizations and for different types and domains of simulation.
This document addresses simulation specialists, simulation team managers and other stakeholders involved in the simulation process or decision-making support.
This document provides recommendations for each criticality level, linked also to the level of confidence in the simulation, at each stage of the simulation process.
Modelling and simulation have long been part of product qualification and certification, and the recommendations laid down in this document do not aim to replace the many qualification, certification and analysis processes already proven and established. The practices recommended in this document were specifically developed in response to potential future applications of modelling and simulation which could, in some cases, give it a more prominent role in qualification and certification, thereby reducing programme costs and development times.

Luft- und Raumfahrt - Programm-Management - Überprüfung und Validierung von numerischen Modellen und Simulationen

No Scope available

Série aérospatiale - Management de programme - Vérification et validation des modèles et simulations numériques

Le présent document propose un état des lieux des bonnes pratiques, partagées par les intervenants du secteur aéronautique, spatial et de défense, relatives à la vérification et à la validation (V&V) des simulations et des modèles numériques afin de garantir la crédibilité des résultats obtenus dans une logique de développement plus rapide d'aide à la prise de décision, de réduction des essais physiques, de diminution des délais de développement, de facilitation de la qualification et de la certification numériques… qui représentent les grands enjeux de la simulation.
La démarche s’applique pour les modèles s’appuyant sur les équations de la physique.
EXEMPLES DE PHYSIQUES   Mécanique, acoustique, électrique, électromagnétisme, thermique pour électronique, dynamique des fluides, dynamique multi-corps, multiphysique, optique, intégrité de signal et de puissance.
L'objectif est de déterminer des recommandations en fonction des enjeux de la simulation, afin d'adapter les procédures à appliquer pour garantir la crédibilité de la simulation. Les éléments à considérer sont :
-   la criticité du produit et de la simulation ;
-   la complexité du phénomène ou du produit ;
-   la capacité, la fidélité et la maturité du modèle ;
-   le cycle de vie du produit ;
-   les compétences ;
-   la démarche de vérification et validation avec prise en compte des incertitudes ;
-   ...
Le présent document est organisé selon le schéma suivant :
-   des éléments de terminologie ;
-   les principes et les concepts généraux de la V&V de la simulation :
o   objectifs et valeur ajoutée du document ;
o   état de l’art ;
o   différentes utilisations de la simulation en fonction de la maturité (niveau d’approximation) du modèle et cycle de vie du produit, en lien avec la fidélité attendue du modèle et des résultats de simulation ;
o   présentation de différents types de modèles et les impacts sur les critères et les grandeurs d’intérêt, ainsi que sur les exigences ;
-   le processus et les activités de V&V recommandés (en lien avec le degré de maturité) ;
-   un exemple de trame de plan de simulation ;
-   des exemples afin de faciliter la compréhension.
Le présent document a pour vocation de compléter et de référencer les informations disponibles dans la littérature. Ce document se veut générique pour être applicable par une grande majorité des organismes et pour différents types et domaines de simulation.
Ce document est à destination d'ingénieurs calcul, de responsables de bureau de calcul et autres parties prenantes impliquées dans le processus de simulation ou d'aide à la décision.
Ce document propose des recommandations pour chaque niveau de criticité, en lien également avec le niveau de confiance dans la simulation, à chaque étape du processus de simulation.
La modélisation et la simulation font depuis longtemps partie de la qualification et de la certification des produits et les recommandations du présent document ne visent pas à remplacer les nombreux processus d'analyse, de qualification et de certification déjà éprouvés et établis. Les pratiques recommandées dans ce document ont été spécifiquement développées pour répondre aux futures applications potentielles de la modélisation et de la simulation qui pourraient, dans certains cas, permettre d'accroître son rôle dans la qualification et la certification, avec une réduction correspondante des coûts et des délais des programmes.

Aeronavtika - Vodenje programa - Preverjanje in validacija numeričnih modelov in simulacij

General Information

• Status: Not Published
• Publication Date: 11-Nov-2026
• ICS: 49.020 - Aircraft and space vehicles in general
• Technical Committee: ASD-STAN - Aerospace
• Drafting Committee: ASD-STAN/D 1/WG 11 - System definition and realization
• Current Stage: 4060 - Closure of enquiry - Enquiry
• Start Date: 07-Aug-2025
• Completion Date: 07-Aug-2025
• Ref Project: oSIST prEN 9247:2025 - Aerospace series - Programme management - Verification and validation of numerical models and simulations

Overview

The prEN 9247 standard, titled Aerospace Series - Programme Management - Verification and Validation of Numerical Models and Simulations, is developed by the European Committee for Standardization (CEN) to guide aerospace and defence industry professionals in ensuring the credibility of numerical simulation outputs. This standard compiles best practices for verification and validation (V&V) processes across a wide range of physical behavior models based on equations, covering disciplines such as mechanics, acoustics, electromagnetism, thermal physics, fluid dynamics, multibody dynamics, optics, and signal integrity.

The focus is on improving decision-making support, reducing physical testing, shortening development cycles, and facilitating the numerical qualification and certification of aerospace products. The framework emphasizes adapting verification and validation methods according to the simulation’s criticality, model maturity, product lifecycle stage, and complexity.

---

Key Topics

• Verification and Validation (V&V): Detailed processes to verify calculation codes, validate simulation results, and quantify uncertainties to ensure the reliability and robustness of simulation models.

• Simulation Maturity Levels: Identification of model maturity and fidelity linked to different phases of the product lifecycle, aiding in the appropriate application of simulation outputs.

• Simulation Planning: Guidelines and templates for developing comprehensive simulation plans, including criticality assessments and verification cross-reference matrices.

• Model Types and Their Impact: Distinction between types of physical behavior models and their relevance to criteria and performance indicators, influencing V&V requirements.

• Data Management & Archiving: Best practices for data preservation, including considerations for long-term storage, technological obsolescence, data validation, and configuration management.

• Uncertainty Quantification: Methods for assessing and integrating uncertainties in simulation results to improve confidence in modelling outcomes.

• Role of Simulation in Certification: Recommendations on integrating simulation evidence into product qualification and certification to potentially reduce costs and timelines.

---

Applications

prEN 9247 is designed to be broadly applicable to aerospace and defence organizations engaged in:

• Aerospace Programme Management: Enhancing project timelines by efficiently verifying simulation models, reducing reliance on costly physical tests.

• Simulation Specialists & Teams: Implementing structured V&V protocols for models covering mechanics, acoustics, electromagnetism, thermal behavior, fluid dynamics, multiphysics, optics, and signal integrity.

• Product Lifecycle Management: Employing simulation-driven insights from early design phases through operational stages to optimize product development and lifecycle decisions.

• Numerical Qualification & Certification: Supporting regulatory processes by establishing simulation credibility through structured verification, validation, and documented evidence.

• Decision-Making Support: Providing reliable simulation outputs to assist stakeholders and management in critical engineering decisions.

This standard helps organizations adapt their V&V processes based on the simulation’s criticality, complexity, and intended use, ensuring tailored credibility suitable for each project’s specific requirements.

---

Related Standards

While prEN 9247 specifically addresses V&V of numerical models and simulations for aerospace, it complements other international and regional standards:

• ISO 9001: Quality management systems providing overarching quality assurance principles applicable in aerospace projects.

• ISO/IEC 12207: Systems and software engineering processes relevant to simulation software lifecycle.

• AS9100: Aerospace quality management standard, emphasizing processes that include verification and validation tasks.

• MIL-STD-810: Environmental engineering considerations and laboratory tests supporting robustness validation.

• Guidelines from IEEE and NASA: Best practices on software and numerical model verification and validation in aerospace contexts.

Organizations may integrate prEN 9247 alongside these frameworks to reinforce simulation reliability and meet certification and quality requirements efficiently.

---

Keywords: aerospace numerical simulation, verification and validation, V&V aerospace standard, simulation maturity, model validation aerospace, aerospace programme management, simulation data archiving, aerospace certification simulation, physical model verification, uncertainty quantification in aerospace simulations.

Standards Content (Sample)

SLOVENSKI STANDARD
01-julij-2025
Aeronavtika - Vodenje programa - Preverjanje in validacija numeričnih modelov in
simulacij
Aerospace series - Programme management - Verification and validation of numerical
models and simulations
Luft- und Raumfahrt - Programm-Management - Überprüfung und Validierung von
numerischen Modellen und Simulationen
Série aérospatiale - Management de programme - Vérification et validation des modèles
et simulations numériques
Ta slovenski standard je istoveten z: prEN 9247
ICS:
03.100.01 Organizacija in vodenje Company organization and
podjetja na splošno management in general
49.020 Letala in vesoljska vozila na Aircraft and space vehicles in
splošno general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

DRAFT
EUROPEAN STANDARD
NORME EUROPÉENNE
EUROPÄISCHE NORM
May 2025
ICS 49.020
English Version
Aerospace series - Programme management - Verification
and validation of numerical models and simulations
Série aérospatiale - Management de programme - Luft- und Raumfahrt - Programm-Management -
Vérification et validation des modèles et simulations Überprüfung und Validierung von numerischen
numériques Modellen und Simulationen
This draft European Standard is submitted to CEN members for enquiry. It has been drawn up by the Technical Committee ASD-
STAN.
If this draft becomes a European Standard, CEN members are bound to comply with the CEN/CENELEC Internal Regulations
which stipulate the conditions for giving this European Standard the status of a national standard without any alteration.

This draft European Standard was established by CEN in three official versions (English, French, German). A version in any other
language made by translation under the responsibility of a CEN member into its own language and notified to the CEN-CENELEC
Management Centre has the same status as the official versions.

CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,
Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway,
Poland, Portugal, Republic of North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and
United Kingdom.
Recipients of this draft are invited to submit, with their comments, notification of any relevant patent rights of which they are
aware and to provide supporting documentation.

Warning : This document is not a European Standard. It is distributed for review and comments. It is subject to change without
notice and shall not be referred to as a European Standard.

EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION

EUROPÄISCHES KOMITEE FÜR NORMUNG

CEN-CENELEC Management Centre: Rue de la Science 23, B-1040 Brussels
© 2025 CEN All rights of exploitation in any form and by any means reserved Ref. No. prEN 9247:2025 E
worldwide for CEN national Members.

Contents Page
European foreword . 4
Introduction . 5
1 Scope . 6
2 Normative references . 7
3 Terms and definitions . 7
4 List of acronyms . 13
5 Principles/concepts . 14
5.1 Document objectives and added value. 14
5.2 Simulation-related organization . 14
5.3 Maturity of a model . 14
5.3.1 Maturity levels . 14
5.3.2 Maturity assessment methods . 15
5.3.3 Maturity based on the product lifecycle stages. 17
6 Modelling and simulation . 18
6.1 Assessment of the simulation need . 18
6.2 Selecting the type of simulation, model and tool . 20
6.3 Simulation plan . 22
6.3.1 Objectives of the simulation plan . 22
6.3.2 Identifying the need in terms of simulations . 22
6.3.3 Verification cross-reference matrix (VCRM) . 23
6.3.4 Identification and ranking of physical phenomena . 23
6.3.5 Identification of the types of simulation to be implemented . 24
6.3.6 Determining the criticality of the simulations . 24
6.3.7 Description of the simulation workflow . 25
6.3.8 Finalization of the first version of the simulation plan . 25
6.4 Modelling . 25
6.4.1 Different types of physical behaviour models . 25
6.4.2 Construction of a simulation model . 27
6.4.3 Assumptions . 30
6.5 Input data. 32
7 Verification and validation (V&V) . 34
7.1 General. 34
7.2 Verification . 35
7.2.1 Creation of a model . 35
7.2.2 Verification of the calculation code . 36
7.2.3 Solution verification . 36
7.2.4 Types of errors and recommendations. 36
7.2.5 Verification methods . 37
7.3 Validation . 39
7.3.1 Validation domain . 39
7.3.2 Validation activity . 40
7.3.3 Reference data . 41
7.3.4 Validation criteria . 42
7.3.5 Uncertainties quantification . 42
7.3.6 Deviations between operational, simulation and test environments . 44
7.3.7 Deviations between operational, simulation and test models . 44
8 Interpretation, use and publication of the simulation outputs . 44
8.1 Interpretation of the simulation outputs . 44
8.2 Use of the simulation outputs . 45
8.3 Publication of the simulation outputs . 46
9 Archiving . 46
9.1 Definition and objective . 46
9.2 Challenges involved in long-term storage . 46
9.3 Archived data access scenarios . 46
9.4 Archived data . 47
9.5 Constraints . 47
9.6 Challenges of archiving . 48
9.7 Elements necessary for archiving modelling and simulation data. 48
9.8 Long-term preservation of data and models . 49
9.8.1 Technological obsolescence . 49
9.8.2 Description of the data . 49
9.8.3 Loss of formats . 49
9.8.4 Verification of data to be archived . 50
9.8.5 Validation of archived data . 50
10 Configuration management for the simulation . 50
10.1 Configuration management in short. 50
10.2 Management of simulation data . 51
10.3 Challenges involved in simulation data . 51
10.4 Application of configuration management . 51
10.5 Main differences with PLM . 52
Annex A (informative) Example of a simulation plan template . 53
Annex B (informative) Example of a criticality assessment matrix . 56
Annex C (informative) Example of a data non-regression validation . 58
Annex D (informative) Example of sections in an analysis report . 59
Bibliography . 62

European foreword
This document (prEN 9247:2025) has been prepared by ASD-STAN.
After enquiries and votes carried out in accordance with the rules of this Association, this document has
received the approval of the National Associations and the Official Services of the member countries of
ASD-STAN, prior to its presentation to CEN.
This document is currently submitted to the CEN Enquiry.
Introduction
This document has been drawn up in response to an identified need, which is widely shared among
aerospace and defence industry, to have a document laying down recommendations and best practices
for ensuring the fidelity and robustness of the outputs of physical behaviour simulations.
Through their recommendations, the experts wish to promote to decision-makers the use of
calculations and simulations in all stages of the product lifecycle (from the preliminary project stage to
the use stage).
1 Scope
This document provides an inventory of best practices, shared by actors from the aerospace and
defence sector, concerning the verification and validation (V&V) of numerical simulations and models,
in order to ensure the credibility of the outputs obtained in a logic of faster development of decision-
making support, of reducing the number of physical tests, of shortening development times, of
facilitating numerical qualification and certification, etc. These are all the major challenges concerning
simulation.
The approach applies to models based on physical equations.
EXAMPLE Mechanics, acoustics, electrical, electromagnetism, thermal physics for electronics, fluid dynamics,
multibody dynamics, multiphysics, optical, signal integrity and power integrity.
The objective is to determine recommendations depending on the challenges of the simulation, in order
to adapt the procedures to be applied to ensure the credibility of the simulation. The items being
considered are:
— criticality of the product and the simulation;
— complexity of the phenomenon or the product;
— capability, fidelity and maturity of the model;
— product lifecycle;
— skills;
— verification and validation approach, with uncertainties quantification;
— etc.
This document is organized as follows:
— terms and definitions;
— general principles and concepts of simulation V&V:
o the document’s objectives and added value;
o state of the art;
o different uses of simulation depending on the maturity (approximation level) of the model and
product lifecycle, linked to the expected fidelity of the model and the simulation outputs;
o presentation of the different types of models and impacts on criteria and quantities of interest,
as well as on requirements;
— recommended V&V process and activities (linked to the degree of maturity);
— an example of a simulation plan template;
— examples for a clearer understanding.
The aim of this document is to complete and reference the information available in the literature.
This document takes a generic approach so that it is applicable by most organizations and for different
types and domains of simulation.
This document addresses simulation specialists, simulation team managers and other stakeholders
involved in the simulation process or decision-making support.
This document provides recommendations for each criticality level, linked also to the level of
confidence in the simulation, at each stage of the simulation process.
Modelling and simulation have long been part of product qualification and certification, and the
recommendations laid down in this document do not aim to replace the many qualification, certification
and analysis processes already proven and established. The practices recommended in this document
were specifically developed in response to potential future applications of modelling and simulation
which could, in some cases, give it a more prominent role in qualification and certification, thereby
reducing programme costs and development times.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https://www.iso.org/obp/
— IEC Electropedia: available at https://www.electropedia.org/
3.1
sensitivity analysis
estimation of how the intensity of the variation in the output of a model or simulation can be
apportioned to variations in an input of the model or of the simulation
Note 1 to entry: This analysis allows to determine what model parameters have a significant impact on the
outputs (quantities of interest).
3.2
outputs analysis
any post-processing or interpretation of individual quantities of interest, tables, data files or execution
series resulting from a simulation
3.3
model capability
characteristic that enables a model to answer certain questions or fulfil certain functions
3.4
solver
numerical algorithms describing the simulation process
3.5
factor of safety
multiplying factor to be applied to the values (theoretical, calculated, observed, target) to account for
uncertainties in the methods, calculations, input data and assumptions
Note 1 to entry: The choice of a factor of safety for a program is directly linked to the rationale retained for
designing, dimensioning and testing within the program or it may be required by regulatory authorities.
Note 2 to entry: A factor of safety does not cover analysis or modelling errors, nor poor design or
manufacturing practice.
3.6
boundary condition
physical value at the edge of a product or limits of the domain of interest modelled
3.7
product criticality
index of the severity of an effect on a product combined with the probability of expected frequency of
its occurrence
Note 1 to entry: Criticality = probability × severity of an effect.
[SOURCE: adapted from ISO 13372:2012 – Note 1 added]
3.8
criticality of the simulation
assessment of the impact of the simulation-based decision making on the product, programme
and people
3.9
discretization
replacement of a continuous domain (space) or interval (time) by a finite set of individual and discrete
values which are as close to it as possible
3.10
validated application domain
intersection between the intended application domain and the validation domain
3.11
validation domain
all the conditions, parameters and their intervals that enable shared acceptance of the level of
confidence in the simulation outputs
Note 1 to entry: Outside this domain, the outputs can no longer be used with the same level of confidence.
3.12
environment
at a given moment in time, set of physical, chemical and biological agents that are either natural (such
as temperature and humidity) or induced (such as impact or vibrations) likely to have an effect on the
product or the way it operates throughout its lifecycle
EXAMPLE Examples of environments: operating environment, test environment.
[SOURCE: adapted from the NF X 50–144 series]
3.13
credibility assessment
process by which evidence is collected and documented in order to verify and communicate the
credibility of the models and simulation outputs
Note 1 to entry: The degree of credibility can be provided by:
—  evidence provided by the V&V approach;
—  quantification of uncertainties;
—  the expertise and experience of the modelling and simulation specialists;
—  judgement;
—  decision-makers’ understanding of and agreement with the simulation assumptions.
3.14
uncertainty
characterization of doubt concerning the output or input data of a measurement or calculation
Note 1 to entry: Uncertainty can take the form of an interval about the output of a measurement or a calculation
that encompasses possible values and probably include the actual output attributed to the corresponding quantity
of interest.
Note 2 to entry: Characterization can be qualitative or quantitative (see 3.24).
3.15
margin
difference between the value obtained and the target value
Note 1 to entry: If the margin is positive, it represents the product of interest’s optimization ability.
3.16
maturity of a simulation
capability to represent the actual physical behaviour of the product of interest
3.17
measurement of simulation fidelity
comparative analysis to estimate how faithfully the model represents its corresponding product of
interest
Note 1 to entry: Fidelity measures include: accuracy (representation level), precision, repeatability, resolution
(detail level), scope and sensitivity.
3.18
metadata
information describing the characteristics of a model/simulation to help manage it, understand it, and
aggregate to other models, etc
EXAMPLE 1 Examples of characteristics: definition, scope, objective, representation, credibility.
EXAMPLE 2 Examples of ways of managing a model/simulation: creation, use, archiving, change.
3.19
model
description or representation of an object (product, phenomenon or process) realized in order to be
able to better study it
Note 1 to entry: A model can be constructed from several sub-models (aggregate). Likewise, any data that goes
into a model are considered part of the model.
Note 2 to entry: A model can be conceptual, mathematical, analytical or numerical.
[SOURCE: adapted from NASA-STD-7009A]
3.20
modelling
approach by which a model is built
Note 1 to entry: The elaboration of a model is motivated by a set of precise questions and functions that it shall
help to answer and fulfil.
3.21
analysis report
document describing the simulation need and the numerical model suited to the simulation process
used and the simulation outputs
3.22
simulation plan
document describing the simulation strategy for a given product
Note 1 to entry: It enables forward planning of all the simulations to be realized and identification of the new
processes to be developed and qualified.
3.23
accuracy
closeness between the numerically obtained value and the value considered to be the reference value,
allowing to determine the quality of a numerical output
Note 1 to entry: The lower the difference, the closer the output of the numerical simulation is to the
intended output.
3.24
uncertainty quantification
UQ
approach by which the sources and impact of uncertainty in the model and tests is determined, with an
appropriate level of confidence
Note 1 to entry: Uncertainties are quantified in order to determine how variations in numerical and physical
parameters affect the simulation outputs.
3.25
calibration
adjustment of modelling parameters (physical or numerical) within their validity domain so that the
results converge with experimental or theoretical data
Note 1 to entry: Calibration is not validation.
[SOURCE: adapted from AIAA G-077-1998]
3.26
simulation output
quantity of interest of the simulated model behaviour
EXAMPLE Quantities of interest may be displacements, stresses, the maximum temperature of electronic
components, resistance to hail impact, lifetime, velocity field or flow pressure field.
3.27
robustness of the simulation
stability of the simulation outputs (do not change in a meaningful way) with respect to a set of similar
parameters or variables
3.28
simulation
implementation of a model in order to study the behaviour and performance of a product or a process
Note 1 to entry: A simulation can be analytical or numerical.
3.29
numerical simulation
mathematical and computational method for calculating a quantity of interest with respect to a product
requirement using a given numerical approach
3.30
solution
synonym of output
Note 1 to entry: There are three types of reference solution that are used for verification and validation of
the simulation:
—  the reference analytical solution, the outputs of which come from the analytical resolution of equations;
—  the reference numerical solution, the outputs of which come from the numerical resolution of equations;
—  the reference experimental solution, the outputs of which come from experimental tests.
Note 2 to entry: Each of these three reference solutions can be used to check non-regression of the
simulation software.
3.31
enabling system
system that supports a product of interest during its lifecycle stages
Note 1 to entry: Enabling systems do not necessarily contribute directly to the product of interest’s functions
during operation.
[SOURCE: adapted from ISO/IEC/IEEE 15288 – addition of note 1]
3.32
phenomena identification and ranking table – PIRT
list of physical phenomena influencing the responses of the product of interest
Note 1 to entry: This list is accompanied by a ranking showing the importance of each phenomenon.
[SOURCE: ASME V&V 10:2019]
3.33
validation
approach of determining the degree to which a numerical model is an accurate representation of the
physical product from the perspective of the intended uses of the model
Note 1 to entry: Validation aims to make sure the right equations are solved.
[SOURCE: adapted from ASME V&V 40:2018]
3.34
verification
approach of determining that a numerical model accurately represents the underlying mathematical
model and its reference solutions from the perspective of the intended uses of the model
Note 1 to entry: Verification aims to determine that the equations are rightly solved in terms of accuracy, stability
and performance, from a numerical and computational perspective.
Note 2 to entry: In the case of multiphysics coupling, verification aims to determine that the links and interfaces
between the simulation tools are correctly designed and implemented.
[SOURCE: adapted from ASME V&V 40:2018]
4 List of acronyms
0D zero-dimensional space
1D one-dimensional space
2D two-dimensional space
3D three-dimensional space
ASME American society of mechanical engineers
ASN French nuclear safety authority [autorité de sûreté nucléaire]
CAD computer-aided design
CAS credibility assessment scale
CDR critical design review
CFD computational fluid dynamics
DNS direct numerical simulation
EAS electronic archival system
EASA European union aviation safety agency
EHSA electro-hydro-static actuator
EMA electro-mechanical actuator
ESQMS engineering simulation quality management guidelines
FDA food and drug administration
GAP gap analysis process
IADT inspection, analysis, demonstration, test
ISS in-service support
IVVQ integration, verification, validation and qualification
IVVQP integration, verification, validation and qualification plan
LES large eddy simulation
LOTAR long-term archiving and retrieval
MIL-STD military standard
NAFEMS national agency for finite element methods and standards
NASA national aeronautics and space administration
OAIS open archival information system
PCMM predictive capability maturity model
PIRT phenomena identification ranking table
PLM product lifecycle management
PSD power spectral density
RETEX lessons learned [retour d’expérience]
SOM strength of materials
SPDM simulation process and data management
SPH smoothed particle hydrodynamics
VCRM verification cross-reference matrix
V&V verification and validation
V&V&UQ verification, validation and uncertainty quantification
5 Principles/concepts
5.1 Document objectives and added value
The objective of this document is to provide, in a shared (essential for collaborative work) and
recognized manner, a V&V approach for numerical simulation, based on experience sharing and the use
of best practices (such as modelling, V&V or drafting of the analysis report).
Thus, by following a mature and shared process, the outputs of simulations, whose robustness and
relevance can be better accepted, may be used as reliable justification elements for qualifying or even
certifying a physical product.
What is important is to highlight best practices to be shared along with the rules, in order to shorten
development stages and, consequently, reduce product development costs and risks (while providing
justification for the cost of setting up the simulation). These benefits can be achieved by improving the
credibility of the simulation.
5.2 Simulation-related organization
The model V&V approach covers elements such as:
— selection and validation of the software used;
— development of “in-house” tools for verification of the model (such as meshing) and the quality of
the outputs;
— checklist of verifications performed;
— model design guides;
— staff training and supervision policy;
— validation and authorization workflow for technical reports; or
— archiving policy for models and related technical reports;
If the project involves cooperation between several stakeholders, it is preferable for them all to refer to
the same document, one that is accepted by everyone, recommending a set of best practices. That
means everyone is using the same baseline, and it reinforces mutual confidence in the quality of
the simulations.
5.3 Maturity of a model
5.3.1 Maturity levels
The maturity of a simulation model is the status of a simulation model in its development stage,
enabling identification of its context of use. The maturity of the model is directly linked to knowledge of:
— the product (maturity) and its environment;
— the data (source and quality);
— the physical behaviours to be taken into consideration and their simulation methodologies [lessons
learned (RETEX) from the physics modelled].
There are two aspects to be noted when assessing the maturity of a model:
— the question of interest;
— the development cycle of the product of interest.
Answers to the question of interest differ depending on whether the model is used upstream of the
product development with the aim of performing V&V as early as possible, or later in the product
development process, with fewer uncertainties.
5.3.2 Maturity assessment methods
5.3.2.1 General
Maturity assessment can be addressed from two perspectives:
— from the perspective of the simulation specialist; or
— from the perspective of the person who needs to have confidence in the model or simulation
outputs, which constitute a decision-making support.
Two model maturity assessment methods are presented, one from the perspective of a simulation
specialist (see 5.3.2.2 and 5.3.2.3) and one method based on the second perspective (see 5.3.2.4).
An approach for creating a maturity assessment method is also described (see 5.3.2.5).
Model maturity is expressed as an overall level for the first method (see 5.3.2.2) and according to
maturity criteria for the other methods.
5.3.2.2 Overall maturity assessment
Table 1 gives an example of an assessment of the overall maturity level of a simulation model.
Table 1 — Example of an assessment of the overall maturity level of a simulation model
Maturity levels of Maturity criteria
the simulation
model
1 Geometrical representation, loads and boundary conditions
adapted to analytical calculations.
2 Mesh based on simplified geometries.
Uncertain loads.
Approximate boundary conditions.
Simplified constitutive laws.
3 Mesh based on detailed geometries.
Formally and completely defined loads.
Formally and completely defined boundary conditions.
Simplified constitutive laws.
4 Mesh based on detailed geometries.
Formally and completely defined loads.
Formally and completely defined boundary conditions.
Constitutive laws adapted to the physical phenomena to be
analysed.
5.3.2.3 PCMM method
Another example of model maturity assessment method is the PCMM (predictive capability maturity
model) method. This method assesses the maturity level corresponding to the impact of the simulation
on the product development (from concept engineering through to qualification or certification).
These four maturity levels take into account:
— the impact of the simulation with respect to other contributing evidence in addressing the question
of interest; and
— the consequence of the decision, i.e. the significance of an adverse outcome resulting from an
incorrect decision concerning the question of interest.
Six types of evidence of maturity related to model fidelity (both geometric and physics), verification
(code and solution), validation and analysis of uncertainties, are assessed.
5.3.2.4 CAS method
Another method for assessing the credibility of the simulation outputs is the CAS (credibility
assessment scale) method put forward by NASA (NASA-STD-7009A, Appendix E, 2016). This method is
based on eight independent criteria grouped into three categories (model and simulation development,
model and simulation use, supporting evidence).
Each criterion is assessed against a set of five maturity levels related to data, practices and processes
control (in terms of formalization, documentation, traceability, source, uncertainties, sensitivities) and
experience.
5.3.2.5 Approach for creating a specific method
A maturity assessment method for a simulation model can be established as follows:
a) choose and classify the maturity evidence so as to assess the different aspects of the modelling and
simulation;
b) establish maturity criteria corresponding to the maturity evidence to be provided;
c) specify, for each maturity criterion, the activities that make it possible to progressively increase
their maturity level;
d) define, for each maturity criterion, the target maturity level to be reached, in accordance with the
related risk;
e) describe, for each maturity criterion, how to achieve the proposed maturity objective.
The target maturity level increases in relation to the risk level associated with use of the model. The
maturity level of each criterion contributes to the overall maturity level of the model.
It is recommended that the maturity requirements be adjusted with respect to:
— the model purpose and scope; and
— the consequences of an incorrect decision.
The applicability of the maturity criteria with respect to the model’s purpose and scope shall
be described.
The higher the maturity requirements for a simulation output, the higher the amount of effort and rigor
level applied to V&V, adapted to the risk.
The level of confidence in a simulation output is assessed by maturity criteria and not globally.
5.3.2.6 Aggregation of one or more models in a higher-level model
Models and simulations can be developed from the elementary level (material, component) of a
product-to-product level (subassembly, system). Models can be aggregated in the following ways:
— between one hierarchical level of the product and a higher level, for example in order to
understand the interactions between elements represented at different levels (multiscale
simulation);
— at the same hierarchical level of the product, for example in order to understand interactions
between physical phenomena (multiphysics simulation);
— by combining the first two approaches (multiphysics and multiscale simulation).
The computation time, levels of detail and accuracy (resolution) and fidelity are the main differences
between these models. Elementary-level models and simulations usually have greater resolution and
are more accurate than their higher-level counterparts.
If the simulation purpose requires more accurate solutions, but the implementation of a model is
compromised by the complexity of the problem, it is possible to fulfil this purpose in various stages.
EXAMPLE The structural zoom method makes it possible to refine a small concerned section of the part within
a complex assembly with the geometry of simplified parts.
When one or more models are aggregated in a higher-level model (model hierarchy), in another context
or in the case of coupling between models, the level of each maturity criterion of the lower-level
model(s) ought to be known and traced, to facilitate the assessment of the maturity levels of the higher-
level model. This also raises the question of the propagation of uncertainties.
Each organization should define and share (at least internally) its maturity levels as well as its criteria
for progressing to the next maturity level.
5.3.3 Maturity based on the product lifecycle stages
5.3.3.1 Design and development
In the preliminary project stage, the necessary input data (geometries, loads, etc.) to realize the
simulations are often inaccurate and can change considerably. During this design stage, simulation
plays a risk-elimination role and guides the major design choices. Models are often basic and offer the
possibility of testing several configurations quickly. Model maturity levels are often fairly low overall.
NOTE Representation of the physical model is sufficient when it comes to making a choice, but not for
qualifying the product of interest.
In the development stage, input data becomes more accurate, models are more complex and
incorporate more and more details.
5.3.3.2 Manufacturing
In the manufacturing stage, simulation can be used to assess the consequences of any factory anomalies
on the performance of the product of interest. The models’ degree of detail and the complexity of the
simulations may vary considerably according to expertise to be carried out.
5.3.3.3 Test preparation
Simulation can be used in the test preparation stage, for example to:
— suggest the best way of positioning an instrument;
— ensure the success of a test;
— quantify the deviations between a test and “real life” behaviour (e.g. wind tunnel vs flight).
It is sometimes necessary to represent the product of interest with its enabling system (e.g. test
equipment for this stage in the lifecycle of the product of interest).
During the analysis stage, simulation will provide a way of interpreting the measurement outputs. On-
site and laboratory tests are one way of increasing a model’s maturity level.
5.3.3.4 Qualification/certification
The qualification and/or certification stages often require tests that can be very expensive or
sometimes impossible to realize. During this stage, simulation purpose to represent the real world,
meaning they can help reduce the number of tests necessary for qualification and/or certification of the
products of interest (shift to virtual tests). Models often have a very high maturity level in this stage.
5.3.3.5 In-service support (ISS)
During its operating stage, the product of interest may be subject to different types of damage, the
consequences of which can be analysed using numerical simulation. As for the manufacturing stage, a
model’s degree of detail may depend heavily on the needs of the expertise.
Appropriate models and simulations are set up for each of the stages related to the lifecycle of the
product of interest and based on its development maturity.
During the lifecycle stages of the product of interest, the models shall be reassessed, especially if the
assumptions or expertise needs change (data calculated or observed, etc.).
6 Modelling and simulation
6.1 Assessment of the simulation need
Before performing a simulation (considered as model instancing), its contribution in terms of
knowledge and/or its added value for the programme or for the organization should be identified.
For example, simulations can be used to understand, explain, describe, predict, explore and support
decision-making. These are some examples of the types of use for different stages of the product
lifecycle:
— design and development:
o contribute to the product specification (severity of the requirements and their breakdown);
o guide the product design (choice of architecture);
o analyse the product behaviour to understand the product and its behaviour;
NOTE In some cases, simulation is the only way of study [rare, incomplete, inaccurate or
unreliable data, real target product of interest or elements interacting with the target product of interest
are unavailable, costs lower or development times shorter than those of physical tests (as in the nuclear
industry, for example)].
o study dependencies or the influence of deterministic or random variables [stability, sensitivity,
reliability (e.g. Monte-Carlo method) and robustness analyses] to make it easier to explain a
phenomenon and enable the prediction of what might happen;
o establish critical points by exploring different situations in order to find a robust and optimum
solution;
...