ISO/IEC 23001-10:2015
(Main)Information technology — MPEG systems technologies — Part 10: Carriage of timed metadata metrics of media in ISO base media file format
Information technology — MPEG systems technologies — Part 10: Carriage of timed metadata metrics of media in ISO base media file format
ISO/IEC 23001-10:2015 defines a storage format for timed metadata metrics. The timed metadata metrics can be associated with other tracks in the ISO Base Media File Format. Typical timed metadata, quality and power consumption information and their metrics, are defined in the specification for carriage in files based on the ISO Base Media File Format (ISO/IEC 14496‑12 and ISO/IEC 15444‑12). The timed metadata can be used for multiple purposes including supporting dynamic adaptive streaming.
Technologies de l'information — Technologies des systèmes MPEG — Partie 10: Transport de métriques de métadonnées de temporisation de supports au format de fichier de support en base ISO
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 23001-10
First edition
2015-09-01
Information technology — MPEG
systems technologies —
Part 10:
Carriage of timed metadata metrics of
media in ISO base media file format
Technologies de l’information — Technologies des systèmes MPEG —
Partie 10: Transport de métriques de métadonnées de temporisation
de supports au format de fichier de support en base ISO
Reference number
ISO/IEC 23001-10:2015(E)
©
ISO/IEC 2015
---------------------- Page: 1 ----------------------
ISO/IEC 23001-10:2015(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2015 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 23001-10:2015(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms, definitions and abbreviated terms . 1
3.1 Terms and definitions . 1
3.2 Abbreviated terms . 2
4 Carriage of Quality Metadata . 2
4.1 Introduction . 2
4.2 Quality Metadata . 2
4.2.1 Definition . 2
4.2.2 Syntax . 2
4.2.3 Semantics . 3
4.3 Quality Metrics . 3
4.3.1 Peak Signal to Noise Ratio (PSNR) . 3
4.3.2 SSIM . 4
4.3.3 MS-SSIM . 5
4.3.4 VQM . 7
4.3.5 PEVQ . 7
4.3.6 MOS . 8
4.3.7 Frame significance (FSIG) . 8
5 Carriage of Green Metadata . 9
5.1 Introduction . 9
5.2 Decoder Power Indication Metadata . 9
5.2.1 Definition . 9
5.2.2 Syntax . 9
5.2.3 Semantics .10
5.3 Display Power Reduction Metadata .10
5.3.1 Display Power Indication Metadata .10
5.3.2 Display Fine Control Metadata .11
Annex A (informative) Eigen Appearance Metric Matrix Specificiation .13
Bibliography .17
© ISO/IEC 2015 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 23001-10:2015(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 29, Coding of
audio, picture, multimedia and hypermedia information.
ISO/IEC 23001 consists of the following parts, under the general title Information technology — MPEG
systems technologies:
— Part 1: Binary MPEG format for XML
— Part 2: Fragment request UNITS
— Part 3: XML IPMP messages
— Part 4: Codec configuration representation
— Part 5: Bitstream Syntax Description Language (BSDL)
— Part 7: Common encryption in ISO base media file format files
— Part 8: Coding-independent code-points
— Part 9: Common encryption for MPEG-2 Transport Streams
— Part 10: Carriage of timed metadata metrics of media in ISO base media file format
— Part 11: Energy-efficient media consumption (green metadata)
iv © ISO/IEC 2015 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 23001-10:2015(E)
Introduction
This part of ISO/IEC 23001 specifies the carriage of timed metadata related to two fields, in files
belonging to the family based on ISO/IEC 14496-12 the ISO base media file format. The two families
of metadata are “green” metadata (related to energy conservation) and quality measurements of the
associated media data (related to video quality metrics).
© ISO/IEC 2015 – All rights reserved v
---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23001-10:2015(E)
Information technology — MPEG systems technologies —
Part 10:
Carriage of timed metadata metrics of media in ISO base
media file format
1 Scope
This part of ISO/IEC 23001 defines a storage format for timed metadata metrics. The timed metadata
metrics can be associated with other tracks in the ISO Base Media File Format. Typical timed metadata,
quality and power consumption information and their metrics, are defined in the specification for
carriage in files based on the ISO Base Media File Format (ISO/IEC 14496-12 and ISO/IEC 15444-12). The
timed metadata can be used for multiple purposes including supporting dynamic adaptive streaming.
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 14496-10, Information technology — Coding of audio-visual objects — Part 10: Advanced Video
Coding
ISO/IEC 23001-11, Information technology — MPEG systems technologies — Part 11: Energy-efficient
media consumption (green metadata)
ISO/IEC 23008-2, Information technology — High efficiency coding and media delivery in heterogeneous
environments — Part 2: High efficiency video coding
ITU-T Recommendation J.144, Objective perceptual video quality measurement techniques for digital cable
television in the presence of a full reference
ITU-T Recommendation J.247, Objective perceptual multimedia video quality measurement in the presence
of a full reference
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 14496-10, and
ISO/IEC 23008-2 apply.
© ISO/IEC 2015 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO/IEC 23001-10:2015(E)
3.2 Abbreviated terms
FSIG Frame SIGnificance
MOS Mean Opinion Score
MSE Mean Signal Error
MS-SSIM Multi-Scale Structural SIMilarity index
PEVQ Perceptual Evaluation of Video Quality
PSNR Peak Signal to Noise Ratio
SSIM Structural SIMilarity Index
VQM Video Quality Metric
4 Carriage of Quality Metadata
4.1 Introduction
If quality metrics are carried in an ISO Base Media File Format, they shall be carried in the metadata
tracks within the ISO Base Media File Format. Different metric types and corresponding storage
formats are identified by their unique code names. This section defines those quality metrics.
The metadata track is linked to the track it describes by means of a ‘cdsc’ (content describes) track
reference.
Codes not defined in this specification are reserved and files must use only codes defined here.
4.2 Quality Metadata
4.2.1 Definition
Sample Entry Type: ‘vqme’
Container: Sample Description Box (‘stsd’)
Mandatory: No
Quantity: 0 or 1
The sample entry for video quality metrics is defined by the QualityMetricsSampleEntry.
The quality metrics sample entry shall contain a QualityMetricsConfigurationBox, describing metrics
that are present in each sample, and the constant field size that is used for the values. The quality
metrics are defined in 4.3.
Each sample is an array of quality values, corresponding one for one to the declared metrics. Each value
is padded by preceding zero bytes, as needed, to the number of bytes indicated by field_size_bytes.
The codecs parameter value for this track as defined in RFC 6381 shall be set to ‘vqme’.The sub-
parameter for the ‘vqme’ codec is a list of the metrics present in the track as indicated by the metrics
code names, joined by “+”, e.g. ‘vqme.psnr+mssm’.
4.2.2 Syntax
aligned(8) class QualityMetricsSampleEntry()
extends MetadataSampleEntry (‘vqme’) {
2 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 23001-10:2015(E)
QualityMetricsConfigurationBox();
}
aligned(8) class QualityMetricsConfigurationBox
extends FullBox(‘vqmC’, version=0, 0){
unsigned int(8) field_size_bytes;
unsigned int(8) metric_count;
for (i = 1 ; i <= metric_count ; i++){
unsigned int(32) metric_code;
}
}
4.2.3 Semantics
field_size_bytes indicates the constant size in byte of the value for a metric in each sample
metric_count the number of metrics for quality values in each sample
metric_code is the code name of the metrics in the sample
4.3 Quality Metrics
4.3.1 Peak Signal to Noise Ratio (PSNR)
4.3.1.1 Definition
PSNR for encoded video sequence is defined based on per-picture mean square error (MSE) differences:
m−1n−1
2
1
MSE = Ii,,jK− ij
() ()
∑∑
mn
i=0 j=0
where I denotes luma plane of the reference m×n picture, K denotes luma plane of the reconstructed
picture, and i,j denote indices enumerating all pixel locations.
The picture-level PSNR is defined as:
2
MAX
I
PSNR =10⋅log
10
MSE
MAX
I
=20⋅log
10
MSE
B
where MAX = 2 −1 where B is the number of bits per sample in pictures.
I
PSNR for a given video sequence is computed as an average of all picture-level PSNR values obtained for
all pictures in the sequence, i.e., for a sequence with N pictures we have
N−1
1
PSNR = PSNR
sequence ∑ picturen()
N
n=0
Only luma component of the video signal is used for PSNR computation.
Note 1 This is the traditional metric referred to as PSNR in the academic literature and in the context of
video compression research.
Note 2 In cases when the spatial resolution of the reference pictures and the reconstructed ones do not
match, reconstructed pictures must be up-sampled to match the spatial resolution of the reference
Note 3 In cases when the pictures of reconstructed video represent only a subset of pictures in the reference
video sequence, reconstructed pictures must be replicated to produce time-aligned reconstructed pictures for
all pictures in the reference sequence.
© ISO/IEC 2015 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO/IEC 23001-10:2015(E)
4.3.1.2 Metric code name
PSNR quality metric values shall be provided as ones under the ‘psnr’ metric code name.
4.3.1.3 Sample storage format
Each PSNR metric value shall be stored as an unsigned 16-bit integer value.
4.3.1.4 Decoding operation
Given stored 16-bit integer value x, the corresponding PSNR value (in dB) is derived as follows
(expressed in floating point):
PSNR = (real) x / 100; with the exception of PSNR = infinity for x=0
4.3.2 SSIM
4.3.2.1 Definition
SSIM for encoded video sequence is defined based on SSIM index map obtained for each picture. Per-
picture SSIM index map is computed as follows:
22μμ +ccσ +
()()
xy 12xy
SSIMx, y =
()
22 22
μμ++ccσσ++
()xy 1 ()xy 2
where
x denotes the 8x8 window in the reference picture;
y denotes the 8x8 window in the reconstructed picture;
μ denotes the average sample value for pixels in x;
x
μ denotes the average sample value for pixels in y;
y
2
σ denotes the average sample value for pixels in x;
x
2
σ
denotes the average sample value for pixels in y
y
σ denotes the covariance computed for pixel values in x and y.
xy
and where
2 2
ck= Lc, = kL
() ()
11 22
are constants computed using
B
kk00,,10,,03 andL=2 -1
12
where B is the number of bits per sample in reference video.
This formula is applied using an 8x8 sliding window, and producing a map of SSIM index values for all
pixel positions within a picture. The overall SSIM index is then computed as the average of index values
in the SSIM map.
This formula is applied only on luma components in each picture.
4 © ISO/IEC 2015 – All rights reserved
==
---------------------- Page: 9 ----------------------
ISO/IEC 23001-10:2015(E)
SSIM for video sequence is computed as an average of all picture-level SSIM values obtained for all
pictures in the sequence, i.e., for a sequence with N pictures we have
N−1
1
SSIM = SSIM
sequence ∑ picturen()
N
n=0
Note 1 This is the traditional metric referred to as SSIM in the academic literature and in the context of video
[1]
compression research.
Note 2 The nominal range of SSIM index values is [-1.1].
Note 3 In cases when the resolution of the reference pictures and the reconstructed ones do not match,
reconstructed pictures must be up-sampled to match the resolution of the reference
Note 4 In cases when the pictures of reconstructed video represent only a subset of pictures in the reference
video sequence, reconstructed pictures must be replicated to produce time-aligned reconstructed pictures for
all pictures in the reference sequence.
4.3.2.2 Metric code name
SSIM quality metric values shall be provided under the ‘ssim’ metric code name.
4.3.2.3 Sample storage format
Each SSIM metric value shall be stored as an unsigned 8-bit integer value.
4.3.2.4 Decoding operation
Given stored 8-bit integer value x, the corresponding SSIM value is derived as follows (expressed in
floating point):
SSIMr=()eal ()x−127 /.128
4.3.3 MS-SSIM
4.3.3.1 Definition
MS-SSIM calculation procedure is described in Figure 1. Taking the reference and distorted image
signals as the input, the system iteratively applies a low-pass filter and downsamples the filtered image
by a factor of 2. The original scale is indexed by j=1 and the highest scale is indexed by j=M, for M-1
levels of iteration. Further details can be found in Reference [2].
Figure 1 — MS-SSIM calculation procedure
© ISO/IEC 2015 – All rights reserved 5
---------------------- Page: 10 ----------------------
ISO/IEC 23001-10:2015(E)
Based on such M scales of processing, MS-SSIM for encoded video sequence is defined as follows:
M
βγ
α
jj
M
MSSSIMx(,yl)(= xy,) cx(,ys)(xy,) ,
∏
Mj j
j==1
where:
2σσ +C
xy 2
is the contrast comparison at scale j (j=1,…M) given by cx(, y)=
cx(, y)
j
j
22
σσ++C
xy 2
σ +C
xy 3
is the structure comparison at scale j (j=1,…,M) given by sx(, y)=
sx(,y)
j
j
σσ +C
xy 3
2μμ +C
xy 1
is the luma comparison (only computed at scale M) given by lx(,y)=
lx(,y)
M
M 22
μμ++C
xy 1
where
x denotes the 8x8 window in the reference picture;
y denotes the 8x8 window in the reconstructed picture;
μ denotes the average sample value for pixels in x;
x
μ denotes the average sample value for pixels in y;
y
2
σ denotes the average sample value for pixels in x;
x
2
σ denotes the average sample value for pixels in y
y
σ denotes the covariance computed for pixel values in x and y.
xy
and where
M
2 2
CK= LC,,= KL CC= /,21αβ==γγand =
() () ()
11 22 32 jj jj∑
j=1
are constants computed using
B
KK00,,10,,03 andL=2 -1
12
where B is the number of bits per sample in reference video.
This formula is applied only on luma components in each picture.
MS-SSIM for video sequence is computed as an average of all picture-level MS-SSIM values obtained for
all pictures in the sequence, i.e., for a sequence with N pictures we have
N−1
1
MSSSIM = MSSSIM
sequence ∑ picturen()
N
n=0
4.3.3.2 Metric code name
MS-SSIM quality metric values shall be provided under the ‘msim’ metric code name.
6 © ISO/IEC 2015 – All rights reserved
==
---------------------- Page: 11 ----------------------
ISO/IEC 23001-10:2015(E)
4.3.3.3 Sample storage format
Each MS-SSIM metric value shall be stored as an unsigned 8-bit integer value.
4.3.3.4 Decoding operation
Given stored 8-bit integer value x, the corresponding MS-SSIM value shall be derived as follows
(expressed in floating point):
MS−=SSIMreal x−127 /;128
()()
4.3.4 VQM
4.3.4.1 Definition
VQM for encoded video sequence is defined as described in ITU-T Recommendation J.144.
4.3.4.2 Metric code name
VQM quality metric values shall be provided under the ‘j144’ metric code name.
4.3.4.3 Sample storage format
Each VQM metric value shall be stored as an unsigned 8-bit integer value.
4.3.4.4 Decoding operation
Given stored 8-bit integer value x, the corresponding VQM score is derived as follows (expressed in
floating point):
VQMr= eal ×/;50
()
4.3.5 PEVQ
4.3.5.1 Definition
PEVQ for encoded video sequence is defined as described in ITU-T Recommendation J.247.
4.3.5.2 Metric code name
PEVQ quality metric values shall be provided as ones carrying ‘j247’ metric code name.
4.3.5.3 Sample storage format
Each PEVQ metric value shall be stored as an unsigned 8-bit integer value.
4.3.5.4 Decoding operation
Given stored 8-bit integer value x, the corresponding PEVQ score is derived as follows (expressed in
floating point):
PEVQ= real ×/;50
()
© ISO/IEC 2015 – All rights reserved 7
---------------------- Page: 12 ----------------------
ISO/IEC 23001-10:2015(E)
4.3.6 MOS
4.3.6.1 Definition
MOS for encoded video sequence is defined as the arithmetic average of result of a set of standard,
[1]
subjective tests where a number of viewers rate the video sequence.
The MOS provides a numerical indication of the perceived quality from the users’ perspective of
received media after compression. The MOS is expressed as a single number in the range 1 to 5, where 1
is the lowest perceived quality, and 5 is the highest perceived quality. It can be obtained with reference
to ITU recommendation ITU-R BT.500-12.
4.3.6.2 Metric code name
MOS quality metric values shall be provided as ones under the ‘mops’ metric code name.
4.3.6.3 Sample storage format
Each MOS metric value shall be stored as an unsigned 8-bit integer value.
4.3.6.4 Decoding operation
Given stored 8-bit integer value x ranging from 0 to 250 (251~255 are reserved), the corresponding
MOS value is derived as follows (expressed in floating point):
MOSc= eilreal ×/;50
()()
ceil(x) is a function which gives the smallest integer not less than x.
4.3.7 Frame significance (FSIG)
4.3.7.1 Definition
FSIG, or frame significance, characterizes the relative importance of frames in a video sequence, and the
sequence level visual impact from various combinations of frame losses, e.g, from dropping a temporal
layer, can be estimated from this frame significance representation.
For a sequence with frames { f , f , ., f }, The Frame Significance (FSIG) for frame f is defined as,
1 2 n k
vd= ff,
()
kk k−1
where d() is the frame difference function of two successive frames in the sequence. It is a differential
[5]
function that captures the rate of change in the sequence, and is computed from the Eigen appearance
[4][5]
metric of the scaled thumbnails of the frames,
T
T
df ,*fS=−fS **fA AS fS− * fk
() () ()
jk jk j
where S is the bi-cubicle smoothing and down-scaling function that brings the frames to the size of
h × w pixels, and the matrix A is a metric of size d x (h x w), where d is the desired dimension of the
metric. The metric A is computed from Eigen appearance modelling of thumbnail frames at size h=12,
w=16, and d=12, its values are provided in the Appendix.
To characterize the QoE impact of different temporal layers in a video sequence, the visual impact
of frame losses are computed from the FSIG in the following fashion. Let the frame loss index be,
8 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 13 ----------------------
ISO/IEC 23001-10:2015(E)
L={l , l , ., l }, where l =1 if there is a frame loss at time stamp k, and l =0, if no frame loss, then the
1 2 n k k
frame losses induced distortion is computed as,
pk +1
()
n
1
−−ak()j
DL = le v
()
∑∑k j
n
k=1 jk=
where p(k) is the last frame played in the sequence before the loss at frame time k. An exponentially
decaying weight function with kernel size a=1 is introduced to model the temporal masking effects for
consecutive frame losses.
4.3.7.2 Metric code name
FSIG quality metric values shall be provided as ones carrying ‘fsig‘ metric code name.
4.3.7.3 Sample storage format
Each FSIG metric value is limited to the max value of 255 and shall be stored as an unsigned 8-bit
integer value.
4.3.7.4 Decoding operation
Given stored 8-bit unsigned integer value x, the corresponding FSIG value is directly decoded.
5 Carriage of Green Metadata
5.1 Introduction
If Green Metadata is carried in an ISO Base Media File Format, it shall be carried in the metadata tracks
within the ISO Base Media File Format. Different Green Metadata types and corresponding storage
formats are identified by their unique sample entry codes.
A metadata track carrying Green metadata is linked to the track it describes by means of a ‘cdsc’
(content describes) track reference.
5.2 Decoder Power Indication Metadata
5.2.1 Definition
Sample Entry Type: ‘depi’
Container: Sample Description Box (‘stsd’)
Mandatory: No
Quantity: 0 or 1
The Decoder-Power Indication Metadata is defined in ISO/IEC 23001-11:2015 Energy-Efficient Media
Consumption (Green Metadata). It provides decoder complexity reduction ratios for the media track to
which the metadata track refers by means of ‘cdsc’ reference.
5.2.2 Syntax
The decoder power indication metadata sample entry shall be as follows.
class DecoderPowerIndicationMetaDataSampleEntry()
extends MetaDataSampleEntry (‘depi‘) {
}
© ISO/IEC 2015 – All rights reserved 9
---------------------- Page: 14 ----------------------
ISO/IEC 23001-10:2015(E)
The Decoder-Power Indication sample shall conform to the following syntax:
aligned(8) class DecoderPowerIndicationMetaDataSample(){
unsigned int(8) Dec_ops_reduction_ratio_from_max;
signed int(16) Dec_ops_reduction_ratio_from_prev;
}
5.2.3 Semantics
Semantics are defined in ISO/IEC 23001-11.
5.3 Display Power Reduction Metadata
The Display-Power Reduction Metadata is defined in ISO/IEC 23001-11:2015 Energy-Efficient Media
Consumption (Green Metadata). The Display Power Reduction Metadata provides frame statistics and
quality indicators for the media track that the metadata track refers to by means of ‘cdsc’ reference.
This Metadata allows the client to attain a specified quality level by scaling frame-buffer pixels and to
reduce power correspondingly by decreasing the display backlight or OLED voltage.
Display-Power Reduction Metadata is of two types:
a) metadata that indicates power saving at different quality levels over the sample duration. This
metadata shall use the ‘dipi’ (display power indication) sample entry type.
b) metadata that allows fine control of the display to achieve power reduction at a specified quality
level. This metadata shall use the ’dfce’ (display fine control) sample entry type.
Static metadata for the display fine control is stored in the sample entry. Dynamic metadata is stored
in the samples.
5.3.1 Display Power Indication Metadata
5.3.1.1 Definition
Sample Entry Type: ‘dipi’
Container: Sample Description Box (‘stsd’)
Mandatory: No
Quantity: 0 or 1
This metadata indicates potential power saving at different quality levels over the sample duration.
10 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 15 ----------------------
ISO/IEC 23001-10:2015(E)
5.3.1.2 Syntax
Display
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.