ISO/IEC TR 23002-8:2021
(Main)Information technology -- MPEG video technologies
Information technology -- MPEG video technologies
This document provides general information about coding efficiency measurement practices for video coding. This document does not provide recommendations for evaluating video quality; it describes the practices that have recently been followed for coding efficiency experiments conducted during work to develop video coding standards.
Technologies de l'information -- Technologies vidéo MPEG
General Information
Buy Standard
Standards Content (Sample)
TECHNICAL ISO/IEC TR
REPORT 23002-8
First edition
2021-05
Information technology — MPEG video
technologies —
Part 8:
Working practices using objective
metrics for evaluation of video coding
efficiency experiments
Reference number
ISO/IEC TR 23002-8:2021(E)
©
ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC TR 23002-8:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC TR 23002-8:2021(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 1
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate) measurements .2
6 The PSNR-based BD-rate concept . 3
7 PSNR-based BD-rate calculation . 4
7.1 General . 4
7.2 Calculation of PSNR for individual frames . 4
7.3 Calculation of sequence PSNR and bit rate numbers for each QP value. 5
7.4 Calculation of sequence BD-rate number . 5
7.5 Consideration of chroma fidelity . 8
7.6 Calculation of aggregate BD-rate value for all sequences . 9
8 BD-rate calculation for HDR material. 9
9 BD-rate calculation for 360° video . 9
Bibliography .11
© ISO/IEC 2021 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC TR 23002-8:2021(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Technical Committee ISO/IEC JTC1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information, in collaboration
with ITU-T (as ITU-T HSTP-VID-WPOM (07/2020)).
A list of all parts in the ISO/IEC 23002 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
TECHNICAL REPORT ISO/IEC TR 23002-8:2021(E)
Information technology — MPEG video technologies —
Part 8:
Working practices using objective metrics for evaluation of
video coding efficiency experiments
1 Scope
This document provides general information about coding efficiency measurement practices for video
coding. This document does not provide recommendations for evaluating video quality; it describes the
practices that have recently been followed for coding efficiency experiments conducted during work to
develop video coding standards.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following term and definition apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
Bjøntegaard delta bit rate
BD-rate
average percentage bit rate difference at equal measured distortion, integrated across a range of bit
rates in the log domain
[1]
Note 1 to entry: The Bjøntegaard delta bit rate measurement method was originally specified in VCEG-M33 .
4 Abbreviated terms
AVC advanced video coding (Rec. ITU-T H.264 | ISO/IEC 14496-10)
BD-rate Bjøntegaard delta bit rate
HDR high dynamic range
HEVC high efficiency video coding (Rec. ITU-T H.265 | ISO/IEC 23008-2)
HLG hybrid log gamma
JCT-VC joint collaborative team on video coding (for development of HEVC)
JVET joint video experts team (for development of VVC)
© ISO/IEC 2021 – All rights reserved 1
---------------------- Page: 5 ----------------------
ISO/IEC TR 23002-8:2021(E)
MPEG moving picture experts group
MS-SSIM multi-scale structural similarity
MSE mean square error
[10] [11]
PQ perceptual quantizer (as defined in SMPTE ST 2084 and Rec. ITU-R BT.2100 )
PSNR peak signal-to-noise ratio
QP quantization parameter
SDR standard dynamic range
SSIM structural similarity
VCEG visual coding experts group
VMAF video multimethod assessment fusion
VVC versatile video coding (Rec. ITU-T H.266 | ISO/IEC 23090-3)
WCG wide colour gamut
WVGA wide video graphics array
Y′C C colour space representation commonly used for video/image distribution, also
B R
written as YUV
YUV colour space representation commonly used for video/image distribution, also written
as Y′C C
B R
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate)
measurements
This document provides general information about coding efficiency measurement practices that
have been used for video coding experiments for the development of video coding standards in the
ITU-T SG 16 VCEG and ISO/IEC JTC 1/SC 29 MPEG communities. Such work has often been conducted
together in the JVET and JCT-VC joint collaborative teams. In particular, the document describes the use
of Bjøntegaard delta bit rate (BD-rate) measurements. It provides a concept-level overview of recent
practices and provides references to other works that describe further details. It includes comments on
why some of the choices were made and indicates situations where caution is needed when interpreting
the results.
For comparing different encodings, often it is helpful to control the encodings so that similar types and
degrees of encoder optimization are applied, except for the aspects to be tested.
When there are large differences between the coding technologies being tested, and especially when
there can be a substantial difference between the resulting subjective quality, subjective testing (i.e.
using humans to measure the visual quality) is the appropriate action. There are also cases where
the quality difference is expected to be primarily a matter of subjective effect – for example, when
measuring the effects of deblocking filters.
The video coding community has typically used formal subjective testing at the call for proposals and
verification testing stages of projects for standardization (i.e. at the beginning and the end of the work).
For measuring smaller effects and where formal subjective testing is not feasible, it is necessary to use
objective measurements. Since objective measurements are collected at multiple operational points,
and to better understand coding behaviour across all these points, what has commonly been used in
[1]
this community is the technique known as the BD-rate (Bjøntegaard delta bit rate) comparison.
2 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC TR 23002-8:2021(E)
Encoding for video distribution is ordinarily performed in the Y′C C domain (YUV). For typical
B R
multimedia applications, it is well known that the human visual system is most sensitive to the fidelity
of the Y component. The Y component also tends to use most of the bit rate, so it is natural to focus
primarily on the Y component. However, it is advised to measure and report the fidelity of all three
components and review the balance between luma and chroma fidelity when interpreting the results.
This practice can help avoid situations where luma gain can be achieved at a significant cost of chroma
fidelity.
To calculate the Bjøntegaard delta bit rate, a distortion metric needs to be used. For standard-dynamic
range video, the distortion metric primarily used in the video coding standardization community
has been the peak signal to noise ratio (PSNR). There are certainly some weaknesses to the PSNR-
based BD-rate measure in terms of its correspondence with human perception of fidelity. Some other
objective distortion metrics which have been asserted to have a better relationship with human
[12] [13]
perception, such as structural similarity (SSIM) index, multi-scale SSIM (MS-SSIM), and video
[14]
multimethod assessment fusion (VMAF), have also been considered with the BD-rate measurement
process. However, the use of PSNR-based BD-rate measures is the most prevalent in the video coding
standardization community. In this document, BD-rate is used to denote PSNR-based BD-rate unless a
different distortion metric is explicitly mentioned.
[15] [16]
This document is based on JVET-Q0826 and JVET-R2016 .
6 The PSNR-based BD-rate concept
When developing a video coding standard, it is important to have a uniform way of reporting the
compression results so that different contributions can be compared against each other.
The PSNR metric is based on the squared error of individual sample values and does not take into
account how the human visual system works. A relevant question is therefore whether the PSNR metric
is a good predictor of subjective quality. The answer depends at least partly on how different the
encoding methods being compared are to each other. If the two methods differ greatly, their artefacts
can be very different, and the perceived subjective quality will depend heavily on which type of artefact
is psychovisually more disturbing. BD-rate measurements are most often used to compare between
two versions of the same video encoder that only differ in that in one of them one tool has been turned
on or has been modified versus the other. In this scenario it is much more likely that the BD-rate score
between these two versions will correlate with a difference in subjective quality. A clear exception is
when tools are considered that are only (or primarily) expected to affect subjective quality, such as
deblocking filters. Here, decisions are almost always based on a subjective test or expert viewing, and
BD-rate numbers are provided more as an assurance that the tool has not caused some unexpected
problem.
An advantage with using PSNR is that it is mathematically simple and therefore straightforward to
optimize for. As an example, if a tool depends on filter coefficients or other parameters, the reference
encoder can search for the parameter value that minimizes mean square error (MSE) and thus optimizes
PSNR, and this type of optimization is often straightforward to analyse and implement. The idea is that
a real encoder can optimize for a different distortion metric that is psychovisually more relevant but
where the parameter search can be a lot more complicated to implement. By choosing PSNR as the
distortion metric in the BD-rate calculations, the work can concentrate on creating coding tools instead
of spending time developing encoder optimizations for advanced distortion metrics. However, it is
also possible to compute BD-rate measurements using other objective distortion metrics or subjective
mean-opinion scores.
For high-dynamic range (HDR)/wide colour gamut (WCG) material and 360° video material, there
are additional aspects that influence the usability of BD-rate calculations; these are addressed in
Clauses 7 and 8, respectively. The JVET common test conditions also specify a separate category for
screen content material (i.e. material that has not been captured by a camera). However, in the context
of standardization development, a need for a special metric instead of PSNR-based BD-rate has not been
identified for this category.
© ISO/IEC 2021 – All rights reserved 3
---------------------- Page: 7 ----------------------
ISO/IEC TR 23002-8:2021(E)
7 PSNR-based BD-rate calculation
7.1 General
There are several steps in the BD-rate calculation process, where the result in each step is calculated
from the result obtained in the previous step:
a) Calculation of PSNR for individual frames.
b) Calculation of per-sequence PSNR and bit rate values for each quantization parameter (QP) value.
The QP value influences the resulting bit rate. Hence, compressing the sequence several times with
different QPs ensures that the final BD-rate measurement will reflect the performance at many
different bit rates.
c) Calculation of per-sequence BD-rate values.
d) Calculation of an aggregate BD-rate value for all sequences.
These steps are further described in subclauses 7.2 through 7.6.
7.2 Calculation of PSNR for individual frames
For an individual frame, the mean square error is calculated between the luma channel decY of the
decoded output image and the luma channel origY of the original image according to Formula (1).
H−1W−1
1
2
MSEY_ = decY()xy,,−origYx()y , (1)
()
∑∑
WH*
y=0 x=0
where
decY xy, and are the luma sample values at position xy, of the decoded and original images
() ()
origYx , y
()
at the same time instance, respectively;
W
is the width of the luma component;
H
is the height of the luma component.
A luma PSNR value for the frame can then be calculated using Formula (2).
2
()255()bitDepth−8
PSNR _Y = *10 log10 , (2)
MSEY_
where
bitDepth = 10 for 10-bit inputs;
denotes a bitwise left-shift operation.
If MSEY_ = 0 , i.e. if the decoded image exactly matches the original image, there is some adjustment
applied to avoid a division by zero. Different implementations can use a different adjustment method.
For example, the HEVC test model (HM) and VVC test model (VTM) software packages set the PSNR _Y
value to 999.99. In this case, the AVC joint test model (JM) and HDRTools software packages impose a
minimum MSE of 1 ÷ (W * H), and another approach could be to impose a minimum MSE of 1÷12, since
that is the MSE that would theoretically result from rounding large numbers to the nearest multiple of
1.
bitDepth
The use of 255 ≪ (bitDepth − 8) instead of 2 − 1 in the numerator of the expression in Formula (2)
is slightly unusual, but it provides a small adjustment so that if the same video content is coded using
bitDepth = 8 or is coded by shifting it up by two bits and using a 10-bit encoder, and when any error is
4 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC TR 23002-8:2021(E)
also jus
...
TECHNICAL ISO/IEC TR
REPORT 23002-8
First edition
Information technology — MPEG video
technologies —
Part 8:
Working practices using objective
metrics for evaluation of video coding
efficiency experiments
PROOF/ÉPREUVE
Reference number
ISO/IEC TR 23002-8:2021(E)
©
ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC TR 23002-8:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC TR 23002-8:2021(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 1
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate) measurements .2
6 The PSNR-based BD-rate concept . 3
7 PSNR-based BD-rate calculation . 4
7.1 General . 4
7.2 Calculation of PSNR for individual frames . 4
7.3 Calculation of sequence PSNR and bit rate numbers for each QP value. 5
7.4 Calculation of sequence BD-rate number . 5
7.5 Consideration of chroma fidelity . 8
7.6 Calculation of aggregate BD-rate value for all sequences . 9
8 BD-rate calculation for HDR material. 9
9 BD-rate calculation for 360° video . 9
Bibliography .11
© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE iii
---------------------- Page: 3 ----------------------
ISO/IEC TR 23002-8:2021(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Technical Committee ISO/IEC JTC1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information, in collaboration
with ITU-T (as ITU-T HSTP-VID-WPOM (07/2020)).
A list of all parts in the ISO/IEC 23002 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
TECHNICAL REPORT ISO/IEC TR 23002-8:2021(E)
Information technology — MPEG video technologies —
Part 8:
Working practices using objective metrics for evaluation of
video coding efficiency experiments
1 Scope
This document provides general information about coding efficiency measurement practices for video
coding. This document does not provide recommendations for evaluating video quality; it describes the
practices that have recently been followed for coding efficiency experiments conducted during work to
develop video coding standards.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following term and definition apply:
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
Bjøntegaard delta bit rate
BD-rate
average percentage bit rate difference at equal measured distortion, integrated across a range of bit
rates in the log domain
[1]
Note 1 to entry: The Bjøntegaard delta bit rate measurement method was originally specified in VCEG-M33 .
4 Abbreviated terms
AVC advanced video coding (Rec. ITU-T H.264 | ISO/IEC 14496-10)
BD-rate Bjøntegaard delta bit rate
HDR high dynamic range
HEVC high efficiency video coding (Rec. ITU-T H.265 | ISO/IEC 23008-2)
HLG hybrid log gamma
JCT-VC joint collaborative team on video coding (for development of HEVC)
JVET joint video experts team (for development of VVC)
© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE 1
---------------------- Page: 5 ----------------------
ISO/IEC TR 23002-8:2021(E)
MPEG moving picture experts group
MS-SSIM multi-scale structural similarity
MSE mean square error
[10] [11]
PQ perceptual quantizer (as defined in SMPTE ST 2084 and Rec. ITU-R BT.2100 )
PSNR peak signal-to-noise ratio
QP quantization parameter
SDR standard dynamic range
SSIM structural similarity
VCEG visual coding experts group
VMAF video multimethod assessment fusion
VVC versatile video coding (Rec. ITU-T H.266 | ISO/IEC 23090-3)
WCG wide colour gamut
WVGA wide video graphics array
Y′C C colour space representation commonly used for video/image distribution, also writ-
B R
ten as YUV
YUV colour space representation commonly used for video/image distribution, also written
as Y′C C
B R
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate)
measurements
This document provides general information about coding efficiency measurement practices that
have been used for video coding experiments for the development of video coding standards in the
ITU-T SG 16 VCEG and ISO/IEC JTC 1/SC 29 MPEG communities. Such work has often been conducted
together in the JVET and JCT-VC joint collaborative teams. In particular, the document describes the use
of Bjøntegaard delta bit rate (BD-rate) measurements. It provides a concept-level overview of recent
practices and provides references to other works that describe further details. It includes comments on
why some of the choices were made and indicates situations where caution is needed when interpreting
the results.
For comparing different encodings, often it is helpful to control the encodings so that similar types and
degrees of encoder optimization are applied, except for the aspects to be tested.
When there are large differences between the coding technologies being tested, and especially when
there can be a substantial difference between the resulting subjective quality, subjective testing (i.e.
using humans to measure the visual quality) is the appropriate action. There are also cases where
the quality difference is expected to be primarily a matter of subjective effect – for example, when
measuring the effects of deblocking filters.
The video coding community has typically used formal subjective testing at the call for proposals and
verification testing stages of projects for standardization (i.e. at the beginning and the end of the work).
For measuring smaller effects and where formal subjective testing is not feasible, it is necessary to use
objective measurements. Since objective measurements are collected at multiple operational points,
and to better understand coding behaviour across all these points, what has commonly been used in
[1]
this community is the technique known as the BD-rate (Bjøntegaard delta bit rate) comparison.
2 PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC TR 23002-8:2021(E)
Encoding for video distribution is ordinarily performed in the Y′C C domain (YUV). For typical
B R
multimedia applications, it is well known that the human visual system is most sensitive to the fidelity
of the Y component. The Y component also tends to use most of the bit rate, so it is natural to focus
primarily on the Y component. However, it is advised to measure and report the fidelity of all three
components and review the balance between luma and chroma fidelity when interpreting the results.
This practice can help avoid situations where luma gain can be achieved at a significant cost of chroma
fidelity.
To calculate the Bjøntegaard delta bit rate, a distortion metric needs to be used. For standard-dynamic
range video, the distortion metric primarily used in the video coding standardization community
has been the peak signal to noise ratio (PSNR). There are certainly some weaknesses to the PSNR-
based BD-rate measure in terms of its correspondence with human perception of fidelity. Some other
objective distortion metrics which have been asserted to have a better relationship with human
[12] [13]
perception, such as structural similarity (SSIM) index, multi-scale SSIM (MS-SSIM), and video
[14]
multimethod assessment fusion (VMAF), have also been considered with the BD-rate measurement
process. However, the use of PSNR-based BD-rate measures is the most prevalent in the video coding
standardization community. In this document, BD-rate is used to denote PSNR-based BD-rate unless a
different distortion metric is explicitly mentioned.
[15] [16]
This document is based on JVET-Q0826 and JVET-R2016 .
6 The PSNR-based BD-rate concept
When developing a video coding standard, it is important to have a uniform way of reporting the
compression results so that different contributions can be compared against each other.
The PSNR metric is based on the squared error of individual sample values and does not take into
account how the human visual system works. A relevant question is therefore whether the PSNR metric
is a good predictor of subjective quality. The answer depends at least partly on how different the
encoding methods being compared are to each other. If the two methods differ greatly, their artefacts
can be very different, and the perceived subjective quality will depend heavily on which type of artefact
is psychovisually more disturbing. BD-rate measurements are most often used to compare between
two versions of the same video encoder that only differ in that in one of them one tool has been turned
on or has been modified versus the other. In this scenario it is much more likely that the BD-rate score
between these two versions will correlate with a difference in subjective quality. A clear exception is
when tools are considered that are only (or primarily) expected to affect subjective quality, such as
deblocking filters. Here, decisions are almost always based on a subjective test or expert viewing, and
BD-rate numbers are provided more as an assurance that the tool has not caused some unexpected
problem.
An advantage with using PSNR is that it is mathematically simple and therefore straightforward to
optimize for. As an example, if a tool depends on filter coefficients or other parameters, the reference
encoder can search for the parameter value that minimizes mean square error (MSE) and thus optimizes
PSNR, and this type of optimization is often straightforward to analyse and implement. The idea is that
a real encoder can optimize for a different distortion metric that is psychovisually more relevant but
where the parameter search can be a lot more complicated to implement. By choosing PSNR as the
distortion metric in the BD-rate calculations, the work can concentrate on creating coding tools instead
of spending time developing encoder optimizations for advanced distortion metrics. However, it is
also possible to compute BD-rate measurements using other objective distortion metrics or subjective
mean-opinion scores.
For high-dynamic range (HDR)/wide colour gamut (WCG) material and 360° video material, there
are additional aspects that influence the usability of BD-rate calculations; these are addressed in
Clauses 7 and 8, respectively. The JVET common test conditions also specify a separate category for
screen content material (i.e. material that has not been captured by a camera). However, in the context
of standardization development, a need for a special metric instead of PSNR-based BD-rate has not been
identified for this category.
© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE 3
---------------------- Page: 7 ----------------------
ISO/IEC TR 23002-8:2021(E)
7 PSNR-based BD-rate calculation
7.1 General
There are several steps in the BD-rate calculation process, where the result in each step is calculated
from the result obtained in the previous step:
a) Calculation of PSNR for individual frames.
b) Calculation of per-sequence PSNR and bit rate values for each quantization parameter (QP) value.
The QP value influences the resulting bit rate. Hence, compressing the sequence several times with
different QPs ensures that the final BD-rate measurement will reflect the performance at many
different bit rates.
c) Calculation of per-sequence BD-rate values.
d) Calculation of an aggregate BD-rate value for all sequences.
These steps are further described in subclauses 7.2 through 7.6.
7.2 Calculation of PSNR for individual frames
For an individual frame, the mean square error is calculated between the luma channel decY of the
decoded output image and the luma channel origY of the original image according to Formula (1).
H−1W−1
1
2
MSEY_ = decY()xy,,−origYx()y , (1)
()
∑∑
WH*
y=0 x=0
where
decY xy, and are the luma sample values at position xy, of the decoded and original images
() ()
origYx , y
()
at the same time instance, respectively;
W
is the width of the luma component;
H
is the height of the luma component.
A luma PSNR value for the frame can then be calculated using Formula (2).
2
()255()bitDepth−8
PSNR _Y = *10 log10 , (2)
MSEY_
where
bitDepth = 10 for 10-bit inputs;
denotes a bitwise left-shift operation.
If MSEY_ = 0 , i.e. if the decoded image exactly matches the original image, there is some adjustment
applied to avoid a division by zero. Different implementations can use a different adjustment method.
For example, the HEVC test model (HM) and VVC test model (VTM) software packages set the PSNR _Y
value to 999.99. In this case, the AVC joint test model (JM) and HDRTools software packages impose a
minimum MSE of 1 ÷ (W * H), and another approach could be to impose a minimum MSE of 1÷12, since
that is the MSE that would theoretically result from rounding large numbers to the nearest multiple of 1.
bitDepth
The use of 255 ≪ (bitDepth − 8) instead of 2 − 1 in the numerator of the expression in Formula (2)
is slightly unusual, but it provides a small adjustment so that if the same video content is coded using
bitDepth = 8 or is coded by shifting it up by two bits and using a 10-bit encoder, and when any error is
also just scaled up accordingly, there will be no difference in the resulting fidelity measurement. The
4 PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserve
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.