Road vehicles — Objective rating metric for non-ambiguous signals

ISO/TS 18571:2014 provides validation metrics and rating procedures to be used to calculate the level of correlation between two non-ambiguous signals obtained from a physical test and a computational model, and is aimed at vehicle safety applications. The objective comparison of time-history signals of model and test is validated against various loading cases under different types of physical loads such as forces, moments, and accelerations. However, other applications might be possible too, but are not within the scope of ISO/TS 18571:2014.

Véhicules routiers — Mesures pour l'évaluation objective de signaux non ambigus

General Information

Status
Published
Publication Date
04-Aug-2014
Current Stage
9092 - International Standard to be revised
Completion Date
12-Oct-2022
Ref Project

Buy Standard

Technical specification
ISO/TS 18571:2014 - Road vehicles -- Objective rating metric for non-ambiguous signals
English language
62 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (Sample)

TECHNICAL ISO/TS
SPECIFICATION 18571
First edition
2014-08-01
Road vehicles — Objective rating
metric for non-ambiguous signals
Véhicules routiers — Mesures pour l’évaluation objective de signaux
non ambigus
Reference number
ISO/TS 18571:2014(E)
©
ISO 2014

---------------------- Page: 1 ----------------------
ISO/TS 18571:2014(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO 2014
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2014 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/TS 18571:2014(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Symbols and abbreviated terms . 1
5 General data requirements . 4
6 ISO metric . 4
6.1 Calculation of the overall ISO rating . 5
6.2 Corridor score . 5
6.3 Phase, magnitude, and slope scores . 7
7 Meaning of the overall ISO rating .13
8 Pre-processing of the data .13
8.1 Synchronization of the signals .13
8.2 Sampling rate.14
8.3 Filtering .14
8.4 Interval of evaluation .14
9 Limitations .15
9.1 Type of signals .15
9.2 Metric validation .15
9.3 Meaning of the results .15
9.4 Multiple responses .15
Annex A (informative) Case studies .16
Bibliography .61
© ISO 2014 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/TS 18571:2014(E)

Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/TC 22, Road vehicles, Subcommittee SC 10, Impact
test procedures.
iv © ISO 2014 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/TS 18571:2014(E)

Introduction
Computer-aided engineering (CAE) has become a vital tool for product development in the automobile
industry. Various computer programs and models are developed to simulate dynamic systems. To
maximize the use of these models, the validity and predictive capabilities of these models need to be
assessed quantitatively. Model validation is the process of comparing CAE model outputs with test
measurements in order to assess the validity or predictive capabilities of the CAE model for its intended
usage. The fundamental concepts and terminology of model validation have been established mainly by
[1]
standard committees including the American Institute of Aeronautics and Astronautics (AIAA), the
American Society of Mechanical Engineers (ASME) Standards Committees on verification and validation
[2] [3]
of Computational Solid Mechanics and Computational Fluid Dynamics and Heat Transfer, the Defence
[4]
Modelling and Simulation Office (DMSO) of the United States Department of Defence (DoD), the United
[5] [19] [20]
States Department of Energy (DOE), and various other professional societies.
One of the critical tasks to achieve quantitative assessments of models is to develop a validation metric
that has the desirable metric properties to quantify the discrepancy between functional or time history
[6] [16] [17]
responses from both physical test and simulation result of a dynamic system.  Developing
quantitative model validation methods has attracted considerable researchers’ interest in recent
[11] [12] [13] [15] [17] [18] [23] [24] [25] [27]
years.     However, the primary consideration in the selection of an
effective metric should be based on the application requirements. In general, the validation metric is a
quantitative measurement of the degree of agreement between the physical test and simulation result.
[10]
This Technical Specification is the essential excerpt of ISO/TR 16250:2013 which provides
standardized calculations of the correlation between two signals of dynamic systems, and it is validated
against multiple vehicle safety case studies.
© ISO 2014 – All rights reserved v

---------------------- Page: 5 ----------------------
TECHNICAL SPECIFICATION ISO/TS 18571:2014(E)
Road vehicles — Objective rating metric for non-
ambiguous signals
1 Scope
This Technical Specification (TS) provides validation metrics and rating procedures to be used to
calculate the level of correlation between two non-ambiguous signals obtained from a physical test and
a computational model, and is aimed at vehicle safety applications. The objective comparison of time-
history signals of model and test is validated against various loading cases under different types of
physical loads such as forces, moments, and accelerations. However, other applications might be possible
too, but are not within the scope of this Technical Specification.
2 Normative references
There are no normative references used in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
3.1
filtering
smoothing of signals by using standardized algorithms
3.2
goodness or level of correlation
similarity of two signals
3.3
interval of evaluation
time domain that is used to calculate the correlation between two signals
3.4
rating
rating score
calculated value that represents a certain level of correlation (objective rating)
3.5
sampling rate
recording frequency of a signal
3.6
time sample
pair values (e.g. time and amplitude) of a recorded signal
3.7
time-history signal
physical value recorded in a time domain; those signals are non-ambiguous
4 Symbols and abbreviated terms
CAE Computer-aided engineering
© ISO 2014 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO/TS 18571:2014(E)

CORA Correlation and analysis
DTW Dynamic time warping
EEARTH Enhanced error assessment of response time histories
SME Subject matter expert
a Relative half width of the inner corridor
0
b Relative half width of the outer corridor
0
C, C(t) Analysed signal (CAE signal)
ts ts
C , C (i) Truncated and shifted CAE curve
ts+d ts
C Derivative CAE curve, C
ts+w ts
C Warped CAE curve, C
DTW Dynamic time warping distance
DTW (i, j) Cost of the optimal warping path
opt
d Local cost matrix to perform the dynamic time warping
d(i, j) Local cost function to perform the dynamic time warping
dtw[i, j] Cumulative cost matrix
∆t Interval between two time samples
δ Half width of the inner corridor
i
δ (t) Lower/upper bounds of the inner corridor at time, t, (curve)
i
δ Half width of the outer corridor
o
δ (t) Lower/upper bounds of the outer corridor at time, t, (curve)
o
E Magnitude score
M
E Phase score
P
E Slope (topology) score
S
*
Maximum allowable magnitude error
ε
M
*
Maximum allowable percentage of time shift
ε
P
*
Maximum allowable slope error
ε
S
ε Magnitude error
mag
ε Slope error
slope
ts
i Index number of time shifted and truncated CAE curve, C
ts
i Index number of k-th warping path of curve, C
k
2 © ISO 2014 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/TS 18571:2014(E)

w ts
i Index number of warping path of CAE curve, C
ts
j Index number of time shifted and truncated test curve, T
ts
j Index number of k-th warping path of curve, T
k
w ts
j Index number of warping path of test curve, T
k Index number
k Exponent factor for calculating the magnitude score, E
M M
k Exponent factor for calculating the phase score, E
P P
k Exponent factor for calculating the slope score, E
S S
k Exponent factor for calculating the corridor score between the inner and outer
Z
corridors
m Time steps moved to evaluate the phase error
N Total number of sample points (e.g. time steps) between the starting time, t ,
start
and ending time, t
end
N All natural numbers without zero
> 0
ts ts
n Number of data samples of time shifted and truncated curves (C and T )
n Number of data samples of the optimal warping path
w
n Number of time shifts to get ρ
ε E
ρ Maximum cross correlation of all ρ (m) and ρ (m)
E L R
ρ (m) Cross correlation (signal is moved to the left)
L
ρ (m) Cross correlation (signal is moved to the right)
R
R Overall ISO rating
r Rank of the sliding scale of the ISO metric
SC (r) Lower threshold of rank, r
lower
SC (r) Upper threshold of rank, r
upper
T, T(t) Reference signal (test signal)
T Absolute maximum amplitude of the reference signal, T
norm
ts ts
T , T ( j) Truncated and shifted test curve
ts+d ts
T Derivative test curve, T
ts+w ts
T Warped test curve, T
t Time signal (axis of abscissa)
t Ending time of the interval of evaluation
end
t Starting time of the interval of evaluation
start
© ISO 2014 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO/TS 18571:2014(E)

t Time zero of an event (e.g. test, crash, impact etc.)
0
w Warping path
w Weighting factor of the magnitude score, E
M M
w Weighting factor of the phase score, E
P P
w Weighting factor of the slope score, E
S S
w Weighting factor of the corridor score, Z
Z
w The k-th warping path cell
k
Z Corridor score
Z(t) Corridor score at time, t, (curve)
5 General data requirements
The metric described in this Technical Specification requires non-ambiguous curves (e.g. time-history
curves). Furthermore, it is required that the reference curve, T(t), and the evaluated curve, C(t), are both
defined between starting time, t , and ending time, t . Both curves shall have the same number of
start end
sample points, N, with a constant time interval, ∆t, within the evaluation interval.
6 ISO metric
The approach of this Technical Specification is to combine different types of algorithms to get reliable
and robust assessments of the correlation of two signals. The calculated score must provide fair
assessment for poor and for good correlations of two signals. The two most promising metrics are
identified in Reference [10] and they are CORA corridor method and EEARTH. A combined metric based
on the improved CORA corridor method and EEARTH is then proposed for an ISO Technical Specification
which has been fully validated using responses from multiple vehicle passive safety applications.
Figure 1 shows the structure of the overall ISO metric. While the corridor method calculates the
deviation between curves with the help of automatically generated corridors, the EEARTH method
analyses specific curve characteristics such as phase shift, magnitude, and shape. Hence, the ISO metric
consists of the two best available algorithms.
Figure 1 — ISO metric structure
4 © ISO 2014 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/TS 18571:2014(E)

6.1 Calculation of the overall ISO rating
The combination of the four metric ratings (corridor, phase, magnitude, and slope) will provide a single
number, R, for the correlation of the analysed signals which represents the final overall objective rating.
The overall objective rating, R, is calculated by combining the separate sub-ratings of corridor (Z), phase
(E ), magnitude (E ), and slope (E ). Four individual weighting factors are defining the influence of each
P M S
metric on the overall rating [see Formulae (1) and (2)]. The corresponding weighting factors are shown
in Table 1.
Rw=⋅Zw+⋅Ew+⋅Ew+⋅E (1)
ZP PM MS S
www++ +=w 1 (2)
ZP MS
Table 1 — Weighting factors of the ISO sub-ratings
Parameter Value Description
w 0,4 Weighting factor of the corridor score
Z
w 0,2 Weighting factor of the phase score
P
w 0,2 Weighting factor of the magnitude score
M
w 0,2 Weighting factor of the slope score
S
6.2 Corridor score
The corridor metric calculates the deviation between two signals by means of corridor fitting. The two
sets of corridors, the inner and the outer corridors, are defined along the mean curve. If the evaluated
curve, C, is within the inner corridor bounds, a score of “1” is given and if it is outside the outer corridor
bounds, the score is set to “0”. The assessment declines from “1” to “0” between the bounds of inner and
outer corridors resulting in three different rating zones as shown in Figure 2. The compliance with the
corridors is calculated at each specific time, t, and the final corridor score, Z, of a signal is the average of
all scores, Z(t), at specific times, t.
[9]
Figure 2 — Rating zones of the corridor metric (corridors of constant width)
[14]
The philosophy of the ISO approach is to use a narrow inner corridor and a wide outer corridor.
It limits the number of “1” ratings to only good correlations and gives the opportunity to distinguish
between poor and fair correlations. If the outer corridor is too narrow, too many curves of a fair or
moderate correlation would get the same poor rating of “0”, like signals of almost no correlation with
the reference. Basically the width of the corridors can be adjusted in order to reflect the specific signal
characteristic. The width can be constant for the whole duration of the dynamic responses or vary at
the different time intervals. This Technical Specification applies the most common approach of using
[10] [26]
constant corridor widths for the whole duration of the dynamic response.
© ISO 2014 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO/TS 18571:2014(E)

6.2.1 Calculation
The parameters a and b define the relative half widths of the inner and the outer corridors. Both
0 0
shall be between “0” and “1”, and a must be less than b . The absolute half widths of both corridors
0 0
are defined as the product of relative half width and the absolute maximum amplitude, T , of the
norm
reference signal, T. Formula (3) shows the calculation of T and it is calculated within the interval of
norm
evaluation.
TT=maxmin ,max T (3)
() ()
{}
norm
The absolute half width of the inner corridor (absolute distance from the reference signal to the outer
bounds of the inner corridor) is defined by Formula (4). The calculation of the absolute half width of the
outer corridors [see Formula (5)] is similar to that of the inner corridors.
δ =⋅aT 01≤≤a (4)
in00orm
δ =⋅bT 01≤≤baand on00orm 00
Based on these definitions, the lower and upper bounds of the inner corridor are defined by Formula (6)
and the lower and upper bounds of the outer corridor are defined by Formula (7).
δδtT= t ± (6)
() ()
ii
δδtT= t ± (7)
() ()
oo
Formula (8) shows the calculation of the corridor score for the correlation between the reference signal,
T, and the analysed signal, C, at each evaluation time, t. If the absolute difference between the signals, T
and C, is less than the half width of the inner corridor (δ ), then the score is set to “1”. The score is
i
calculated by Formula (8) when the absolute difference between both signals is in between
δδ≤ Tt −Ct ≤ . If the absolute difference between both signals is greater than the half width of
() ()
io
the outer corridor (δ ), then the score is set to “0”. The parameter, k , assesses the location of the analysed
o Z
signal within the outer corridor, and it applies the appropriate penalty on the rating score. A linear (
k = 1 ), quadratic (k = 2 ), cubical (k = 3 ) or any other regression relationship can be defined
Z Z Z
accordingly.

1 if Tt −Ct <δ
() ()
i

k
Z

 
δ − Tt −Ct
() ()
 o
 
Zt = k ∈Ν (8)
() 
Z >0
 
δδ−
oi

 

0 if Tt −Ct >δ
() ( ))
o


The final corridor score, Z, is calculated by averaging all single time step score Z(t) as shown in
Formula (9). The parameter, N, represents the total number of sample points (e.g. time steps) between
starting and ending times of the interval of evaluation.
t
end
Zt
()

tt=
start
Z = (9)
N
One of the advantages of the corridor metric is the simplicity and the clearness of the algorithm. It
reflects criteria which are used intuitively in engineering judgment. Sometimes this simplicity may
be the disadvantage of the method. For example, a small distortion of the phase can lead to a very
[10]
undesirable rating.
6 © ISO 2014 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/TS 18571:2014(E)

[14]
Based on a sensitivity study of CORA and as described in Reference [10], fixed width corridors are
employed and the most appropriate metric parameters are identified as shown in Table 2.
Table 2 — Parameters of the corridor metric
Parameter Value Description
a 0,05 Relative half width of the inner corridor
0
b 0,50 Relative half width of the outer corridor
0
k 2 Transition between ratings of “1” and “0” (progression)
Z
6.2.2 Step by step procedure
First of all, the signals shall be pre-processed as described in Clause 8. After preparing the signals for the
analysis and defining the interval of evaluation, the maximum absolute amplitude, T , of the reference
norm
signal, T, shall be determined within this interval. It is used to calculate the inner and outer corridors.
The actual corridor assessment shall be executed within this defined interval. The total score ranges
between “0” and “1”. A score of “1” does not mean that both signals are identical. Solely their correlation
is mathematically perfect within the defined tolerances.
To summarize, the following step-by-step procedures shall be followed to calculate corridor score.
a) Pre-process both signals according to Clause 8.
b) Calculate T within the interval of evaluation by using the reference signal.
norm
c) Calculate the inner and the outer corridors.
d) Calculate the corridor score, Z(t), at every specific time t within the interval of evaluation.
e) Calculate the total corridor score, Z, based on Z(t) and the number, N, of time samples.
6.3 Phase, magnitude, and slope scores
Phase, magnitude, and slope (or so-called topology) error assessments between the time history curves,
[24] [28]
T and C, are used as objective rating metrics in addition to the corridor metric described earlier.
The enhanced error assessment of response time histories (EEARTH) metric combines these three
[28]
assessments to the global response error. It is defined as the error associated with the complete time
history with equal weight on each point. Quantifying the errors associated with these features of phase,
magnitude, and slope (topology) separately is challenging because there are strong interactions among
them. For example, to quantify the error associated with magnitude, the presence of a phase difference
between the time histories may result in a misleading measurement. A unique feature dynamic time
[22]
warping (DTW) is used to separate the interaction of phase, magnitude, and slope (topology) errors.
It aligns peaks and valleys of the two signals as much as possible by expanding and compressing the
[8]
time axis according to a given cost (distance) function.
The ranges of the three errors are quite different and there is no single rating that can provide a
quantitative assessment alone. Therefore, a numerical optimization method is employed to identify
the appropriate parameters so that the resulted phase, magnitude, and slope sub-ratings can match
[7] [21]
with SME’s ratings closely.  Figure 3 shows the workflow of the procedures and the details of the
algorithms are described in the following subsections.
© ISO 2014 – All rights reserved 7

---------------------- Page: 12 ----------------------
ISO/TS 18571:2014(E)

Figure 3 — Workflow of the calculation of phase, magnitude, and slope scores
6.3.1 Phase score
The phase score, E , is used to measure the phase lag between the two time histories, T and C. The
P
*
maximum allowable percentage of time shift is ε and it is pre-defined. In this step, the initial curve, C,
P
is shifted left then right one step at a time to the original test data, curve (T), and the cross correlation
between the truncated test curve (T), and shifted and truncated curve, C, are calculated until reaching
*
the maximum allowable time shift limits ε ⋅−tt .
()
Pend start
8 © ISO 2014 – All rights reserved

---------------------- Page: 13 ----------------------
ISO/TS 18571:2014(E)

When the initial curve, C, is moved to the left by m time steps, the number of overlapping points of the
two time histories after time shift, mt⋅Δ ,is reduced to n (nN=−m ) and the corresponding cross
correlation value, ρ (m), is calculated by Formula (10).
L
n−1
 
Ct ++mi ⋅ΔΔtC− tT⋅+ti⋅ tT− t
()() () ()() ()
()
∑ startstart
 
i==0
ρ m = (10)
()
L
n−1 n−1
2
2
 
Ct ++mi ⋅ΔΔtC− tT⋅+ ti⋅ t −TTt 
() () () ()
()
start start
∑ ∑
 
 
i=0 i=0
When the initial curve C is moved to the right by m time steps, the number of overlapping points after
time shift, mt⋅Δ , is reduced to n (nN=−m ) and the corresponding cross correlation value, ρ (m), is
R
calculated by Formula (11).
n−1
 
Ct +⋅itΔΔ−Ct ⋅+Tt mi+ ⋅ tT− t
() () () ()
() ()()
∑ startstart
 
i==0
ρ m = (11)
()
R
n−1 n−1
2
2
   
Ct +⋅itΔΔ−Ct ⋅+Tt mi+ ⋅ t −TTt
() () ()() ()
∑ start ∑ start
 
 
i=0 i=0
The maximum cross correlation, ρ , is the maximum of all ρ (m) and ρ (m). The number of the time
E L R
shifting steps that yields the maximum cross correlation, ρ , is defined as the phase error, n . The
E ε
ts
corresponding shifted and truncated CAE curve, C, is recorded as C and the corresponding truncated
ts
test curve, T, is recorded as T .
The phase score, E , is calculated by Formula (12). The best phase score is “1”, which means there is no
P
need to shift the CAE curve to reach the maximum cross correlation between the initial test and CAE
curves. If the time shift, n , is equal to or greater than the maximum allowable time shift threshold
ε
*
ε ⋅N , then the phase score is “0”. In between, the phase score is calculated by a regression method. It
P
is either linear (k = 1 ), quadratic (k = 2 ), or cubical (k = 3 ).
P P P
 10if n =
ε

k
P
*
 
 ε ⋅−Nn
P ε
E =   k ∈ 12,,3 (12)
{}

P P
*
 
ε ⋅N

P
 
 *
0 if nN≥⋅ε
ε P

The pre-defined parameters shown in Table 3 are identical to the definition in Reference [10].
Table 3 — Fixed parameters of the phase score
Parameter Value Description
k 1 Exponent factor for calculating the phase score
P
*
ε 0,2 Maximum allowable percentage of time shift
P
6.3.2 Magnitude score
The magnitude error is a measure of discrepancy in the amplitude of the two time histories. It is defined
as the difference in amplitude of the two time histories when there is no time lag between them. Before
calculating the magnitude error, the difference between the time histories caused by error in phase and
slope (topology) are minimized by using dynamic time warping (DTW).
The definition of DTW is based on the notion of warping path. Let d be the matrix nn× of pair-wise
ts ts
squared distances between samples of C and T . This matrix, d, is called the local cost matrix. The
© ISO 2014 – All rights reserved 9

---------------------- Page: 14 ----------------------
ISO/TS 18571:2014(E)

function used to calculate the value for each cell of the matrix is called local cost function, d(i, j). It is
shown in Formula (13).
2
ts ts
di, jC= iT− j (13)
() () ()
()
Once the local cost matrix is built, the algorithm finds the alignment path which runs through the low-
cost areas on the cumulated cost matrix. A warping path, w, [see Formula (14)] is a sequence of k matrix
cells, w , [see Formula (15)].
k
ww=≤,,ww, nk≤−21n (14)
()
12 k
wi= ,1ji ≥≤and jn (15)
[]
kk kk k
The local cost matrix must meet the following three conditions:
— boundary conditions
w = 11, and wn= ,n , i.e. w starts in the lower left cell and ends in the upper right cell.
[] []
1 k
— continuity
Given wi= , j and wi= , j , then ii−≤ 1 and jj−≤ 1 . This ensures that
[] []
kk−−11 k−1 kk k kk−1 kk−1
the cells of the warping path are a
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.