Information technology -- MPEG video technologies

This document provides general information about coding efficiency measurement practices for video coding. This document does not provide recommendations for evaluating video quality; it describes the practices that have recently been followed for coding efficiency experiments conducted during work to develop video coding standards.

Technologies de l'information -- Technologies vidéo MPEG

General Information

Status
Published
Publication Date
20-May-2021
Current Stage
5060 - Close of voting Proof returned by Secretariat
Start Date
12-Apr-2021
Completion Date
12-Apr-2021
Ref Project

Buy Standard

Technical report
ISO/IEC TR 23002-8:2021 - Information technology -- MPEG video technologies
English language
12 pages
sale 15% off
Preview
sale 15% off
Preview
Draft
ISO/IEC PRF TR 23002-8:Version 27-mar-2021 - Information technology -- MPEG video technologies
English language
12 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

TECHNICAL ISO/IEC TR
REPORT 23002-8
First edition
2021-05
Information technology — MPEG video
technologies —
Part 8:
Working practices using objective
metrics for evaluation of video coding
efficiency experiments
Reference number
ISO/IEC TR 23002-8:2021(E)
ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC TR 23002-8:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC TR 23002-8:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms .............................................................................................................................................................................................. 1

5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate) measurements .............2

6 The PSNR-based BD-rate concept ....................................................................................................................................................... 3

7 PSNR-based BD-rate calculation .......................................................................................................................................................... 4

7.1 General ........................................................................................................................................................................................................... 4

7.2 Calculation of PSNR for individual frames ...................................................................................................................... 4

7.3 Calculation of sequence PSNR and bit rate numbers for each QP value................................................ 5

7.4 Calculation of sequence BD-rate number ........................................................................................................................ 5

7.5 Consideration of chroma fidelity ............................................................................................................................................. 8

7.6 Calculation of aggregate BD-rate value for all sequences .................................................................................. 9

8 BD-rate calculation for HDR material............................................................................................................................................. 9

9 BD-rate calculation for 360° video .................................................................................................................................................... 9

Bibliography .............................................................................................................................................................................................................................11

© ISO/IEC 2021 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC TR 23002-8:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members

_experts/ refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/

iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.

This document was prepared by Technical Committee ISO/IEC JTC1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information, in collaboration

with ITU-T (as ITU-T HSTP-VID-WPOM (07/2020)).

A list of all parts in the ISO/IEC 23002 series can be found on the ISO and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national

-committees.
iv © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
TECHNICAL REPORT ISO/IEC TR 23002-8:2021(E)
Information technology — MPEG video technologies —
Part 8:
Working practices using objective metrics for evaluation of
video coding efficiency experiments
1 Scope

This document provides general information about coding efficiency measurement practices for video

coding. This document does not provide recommendations for evaluating video quality; it describes the

practices that have recently been followed for coding efficiency experiments conducted during work to

develop video coding standards.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following term and definition apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
Bjøntegaard delta bit rate
BD-rate

average percentage bit rate difference at equal measured distortion, integrated across a range of bit

rates in the log domain
[1]

Note 1 to entry: The Bjøntegaard delta bit rate measurement method was originally specified in VCEG-M33 .

4 Abbreviated terms
AVC advanced video coding (Rec. ITU-T H.264 | ISO/IEC 14496-10)
BD-rate Bjøntegaard delta bit rate
HDR high dynamic range
HEVC high efficiency video coding (Rec. ITU-T H.265 | ISO/IEC 23008-2)
HLG hybrid log gamma
JCT-VC joint collaborative team on video coding (for development of HEVC)
JVET joint video experts team (for development of VVC)
© ISO/IEC 2021 – All rights reserved 1
---------------------- Page: 5 ----------------------
ISO/IEC TR 23002-8:2021(E)
MPEG moving picture experts group
MS-SSIM multi-scale structural similarity
MSE mean square error
[10] [11]
PQ perceptual quantizer (as defined in SMPTE ST 2084 and Rec. ITU-R BT.2100 )
PSNR peak signal-to-noise ratio
QP quantization parameter
SDR standard dynamic range
SSIM structural similarity
VCEG visual coding experts group
VMAF video multimethod assessment fusion
VVC versatile video coding (Rec. ITU-T H.266 | ISO/IEC 23090-3)
WCG wide colour gamut
WVGA wide video graphics array

Y′C C colour space representation commonly used for video/image distribution, also

B R
written as YUV

YUV colour space representation commonly used for video/image distribution, also written

as Y′C C
B R
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate)
measurements

This document provides general information about coding efficiency measurement practices that

have been used for video coding experiments for the development of video coding standards in the

ITU-T SG 16 VCEG and ISO/IEC JTC 1/SC 29 MPEG communities. Such work has often been conducted

together in the JVET and JCT-VC joint collaborative teams. In particular, the document describes the use

of Bjøntegaard delta bit rate (BD-rate) measurements. It provides a concept-level overview of recent

practices and provides references to other works that describe further details. It includes comments on

why some of the choices were made and indicates situations where caution is needed when interpreting

the results.

For comparing different encodings, often it is helpful to control the encodings so that similar types and

degrees of encoder optimization are applied, except for the aspects to be tested.

When there are large differences between the coding technologies being tested, and especially when

there can be a substantial difference between the resulting subjective quality, subjective testing (i.e.

using humans to measure the visual quality) is the appropriate action. There are also cases where

the quality difference is expected to be primarily a matter of subjective effect – for example, when

measuring the effects of deblocking filters.

The video coding community has typically used formal subjective testing at the call for proposals and

verification testing stages of projects for standardization (i.e. at the beginning and the end of the work).

For measuring smaller effects and where formal subjective testing is not feasible, it is necessary to use

objective measurements. Since objective measurements are collected at multiple operational points,

and to better understand coding behaviour across all these points, what has commonly been used in

[1]

this community is the technique known as the BD-rate (Bjøntegaard delta bit rate) comparison.

2 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC TR 23002-8:2021(E)

Encoding for video distribution is ordinarily performed in the Y′C C domain (YUV). For typical

B R

multimedia applications, it is well known that the human visual system is most sensitive to the fidelity

of the Y component. The Y component also tends to use most of the bit rate, so it is natural to focus

primarily on the Y component. However, it is advised to measure and report the fidelity of all three

components and review the balance between luma and chroma fidelity when interpreting the results.

This practice can help avoid situations where luma gain can be achieved at a significant cost of chroma

fidelity.

To calculate the Bjøntegaard delta bit rate, a distortion metric needs to be used. For standard-dynamic

range video, the distortion metric primarily used in the video coding standardization community

has been the peak signal to noise ratio (PSNR). There are certainly some weaknesses to the PSNR-

based BD-rate measure in terms of its correspondence with human perception of fidelity. Some other

objective distortion metrics which have been asserted to have a better relationship with human

[12] [13]

perception, such as structural similarity (SSIM) index, multi-scale SSIM (MS-SSIM), and video

[14]

multimethod assessment fusion (VMAF), have also been considered with the BD-rate measurement

process. However, the use of PSNR-based BD-rate measures is the most prevalent in the video coding

standardization community. In this document, BD-rate is used to denote PSNR-based BD-rate unless a

different distortion metric is explicitly mentioned.
[15] [16]
This document is based on JVET-Q0826 and JVET-R2016 .
6 The PSNR-based BD-rate concept

When developing a video coding standard, it is important to have a uniform way of reporting the

compression results so that different contributions can be compared against each other.

The PSNR metric is based on the squared error of individual sample values and does not take into

account how the human visual system works. A relevant question is therefore whether the PSNR metric

is a good predictor of subjective quality. The answer depends at least partly on how different the

encoding methods being compared are to each other. If the two methods differ greatly, their artefacts

can be very different, and the perceived subjective quality will depend heavily on which type of artefact

is psychovisually more disturbing. BD-rate measurements are most often used to compare between

two versions of the same video encoder that only differ in that in one of them one tool has been turned

on or has been modified versus the other. In this scenario it is much more likely that the BD-rate score

between these two versions will correlate with a difference in subjective quality. A clear exception is

when tools are considered that are only (or primarily) expected to affect subjective quality, such as

deblocking filters. Here, decisions are almost always based on a subjective test or expert viewing, and

BD-rate numbers are provided more as an assurance that the tool has not caused some unexpected

problem.

An advantage with using PSNR is that it is mathematically simple and therefore straightforward to

optimize for. As an example, if a tool depends on filter coefficients or other parameters, the reference

encoder can search for the parameter value that minimizes mean square error (MSE) and thus optimizes

PSNR, and this type of optimization is often straightforward to analyse and implement. The idea is that

a real encoder can optimize for a different distortion metric that is psychovisually more relevant but

where the parameter search can be a lot more complicated to implement. By choosing PSNR as the

distortion metric in the BD-rate calculations, the work can concentrate on creating coding tools instead

of spending time developing encoder optimizations for advanced distortion metrics. However, it is

also possible to compute BD-rate measurements using other objective distortion metrics or subjective

mean-opinion scores.

For high-dynamic range (HDR)/wide colour gamut (WCG) material and 360° video material, there

are additional aspects that influence the usability of BD-rate calculations; these are addressed in

Clauses 7 and 8, respectively. The JVET common test conditions also specify a separate category for

screen content material (i.e. material that has not been captured by a camera). However, in the context

of standardization development, a need for a special metric instead of PSNR-based BD-rate has not been

identified for this category.
© ISO/IEC 2021 – All rights reserved 3
---------------------- Page: 7 ----------------------
ISO/IEC TR 23002-8:2021(E)
7 PSNR-based BD-rate calculation
7.1 General

There are several steps in the BD-rate calculation process, where the result in each step is calculated

from the result obtained in the previous step:
a) Calculation of PSNR for individual frames.

b) Calculation of per-sequence PSNR and bit rate values for each quantization parameter (QP) value.

The QP value influences the resulting bit rate. Hence, compressing the sequence several times with

different QPs ensures that the final BD-rate measurement will reflect the performance at many

different bit rates.
c) Calculation of per-sequence BD-rate values.
d) Calculation of an aggregate BD-rate value for all sequences.
These steps are further described in subclauses 7.2 through 7.6.
7.2 Calculation of PSNR for individual frames

For an individual frame, the mean square error is calculated between the luma channel decY of the

decoded output image and the luma channel origY of the original image according to Formula (1).

H−1W−1
MSEY_ = decY()xy,,−origYx()y , (1)
WH*
y=0 x=0
where

decY xy, and are the luma sample values at position xy, of the decoded and original images

() ()
origYx , y
at the same time instance, respectively;
is the width of the luma component;
is the height of the luma component.
A luma PSNR value for the frame can then be calculated using Formula (2).
 
()255()bitDepth−8
PSNR _Y = *10 log10  , (2)
 
MSEY_
 
where
bitDepth = 10 for 10-bit inputs;
denotes a bitwise left-shift operation.

If MSEY_ = 0 , i.e. if the decoded image exactly matches the original image, there is some adjustment

applied to avoid a division by zero. Different implementations can use a different adjustment method.

For example, the HEVC test model (HM) and VVC test model (VTM) software packages set the PSNR _Y

value to 999.99. In this case, the AVC joint test model (JM) and HDRTools software packages impose a

minimum MSE of 1 ÷ (W * H), and another approach could be to impose a minimum MSE of 1÷12, since

that is the MSE that would theoretically result from rounding large numbers to the nearest multiple of

bitDepth

The use of 255 ≪ (bitDepth − 8) instead of 2 − 1 in the numerator of the expression in Formula (2)

is slightly unusual, but it provides a small adjustment so that if the same video content is coded using

bitDepth = 8 or is coded by shifting it up by two bits and using a 10-bit encoder, and when any error is

4 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC TR 23002-8:2021(E)
also jus
...

TECHNICAL ISO/IEC TR
REPORT 23002-8
First edition
Information technology — MPEG video
technologies —
Part 8:
Working practices using objective
metrics for evaluation of video coding
efficiency experiments
PROOF/ÉPREUVE
Reference number
ISO/IEC TR 23002-8:2021(E)
ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC TR 23002-8:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC TR 23002-8:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms .............................................................................................................................................................................................. 1

5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate) measurements .............2

6 The PSNR-based BD-rate concept ....................................................................................................................................................... 3

7 PSNR-based BD-rate calculation .......................................................................................................................................................... 4

7.1 General ........................................................................................................................................................................................................... 4

7.2 Calculation of PSNR for individual frames ...................................................................................................................... 4

7.3 Calculation of sequence PSNR and bit rate numbers for each QP value................................................ 5

7.4 Calculation of sequence BD-rate number ........................................................................................................................ 5

7.5 Consideration of chroma fidelity ............................................................................................................................................. 8

7.6 Calculation of aggregate BD-rate value for all sequences .................................................................................. 9

8 BD-rate calculation for HDR material............................................................................................................................................. 9

9 BD-rate calculation for 360° video .................................................................................................................................................... 9

Bibliography .............................................................................................................................................................................................................................11

© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE iii
---------------------- Page: 3 ----------------------
ISO/IEC TR 23002-8:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members

_experts/ refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/

iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.

This document was prepared by Technical Committee ISO/IEC JTC1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information, in collaboration

with ITU-T (as ITU-T HSTP-VID-WPOM (07/2020)).

A list of all parts in the ISO/IEC 23002 series can be found on the ISO and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national

-committees.
iv PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
TECHNICAL REPORT ISO/IEC TR 23002-8:2021(E)
Information technology — MPEG video technologies —
Part 8:
Working practices using objective metrics for evaluation of
video coding efficiency experiments
1 Scope

This document provides general information about coding efficiency measurement practices for video

coding. This document does not provide recommendations for evaluating video quality; it describes the

practices that have recently been followed for coding efficiency experiments conducted during work to

develop video coding standards.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following term and definition apply:

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
Bjøntegaard delta bit rate
BD-rate

average percentage bit rate difference at equal measured distortion, integrated across a range of bit

rates in the log domain
[1]

Note 1 to entry: The Bjøntegaard delta bit rate measurement method was originally specified in VCEG-M33 .

4 Abbreviated terms
AVC advanced video coding (Rec. ITU-T H.264 | ISO/IEC 14496-10)
BD-rate Bjøntegaard delta bit rate
HDR high dynamic range
HEVC high efficiency video coding (Rec. ITU-T H.265 | ISO/IEC 23008-2)
HLG hybrid log gamma
JCT-VC joint collaborative team on video coding (for development of HEVC)
JVET joint video experts team (for development of VVC)
© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE 1
---------------------- Page: 5 ----------------------
ISO/IEC TR 23002-8:2021(E)
MPEG moving picture experts group
MS-SSIM multi-scale structural similarity
MSE mean square error
[10] [11]
PQ perceptual quantizer (as defined in SMPTE ST 2084 and Rec. ITU-R BT.2100 )
PSNR peak signal-to-noise ratio
QP quantization parameter
SDR standard dynamic range
SSIM structural similarity
VCEG visual coding experts group
VMAF video multimethod assessment fusion
VVC versatile video coding (Rec. ITU-T H.266 | ISO/IEC 23090-3)
WCG wide colour gamut
WVGA wide video graphics array

Y′C C colour space representation commonly used for video/image distribution, also writ-

B R
ten as YUV

YUV colour space representation commonly used for video/image distribution, also written

as Y′C C
B R
5 Video coding experiments using Bjøntegaard delta bit rate (BD-rate)
measurements

This document provides general information about coding efficiency measurement practices that

have been used for video coding experiments for the development of video coding standards in the

ITU-T SG 16 VCEG and ISO/IEC JTC 1/SC 29 MPEG communities. Such work has often been conducted

together in the JVET and JCT-VC joint collaborative teams. In particular, the document describes the use

of Bjøntegaard delta bit rate (BD-rate) measurements. It provides a concept-level overview of recent

practices and provides references to other works that describe further details. It includes comments on

why some of the choices were made and indicates situations where caution is needed when interpreting

the results.

For comparing different encodings, often it is helpful to control the encodings so that similar types and

degrees of encoder optimization are applied, except for the aspects to be tested.

When there are large differences between the coding technologies being tested, and especially when

there can be a substantial difference between the resulting subjective quality, subjective testing (i.e.

using humans to measure the visual quality) is the appropriate action. There are also cases where

the quality difference is expected to be primarily a matter of subjective effect – for example, when

measuring the effects of deblocking filters.

The video coding community has typically used formal subjective testing at the call for proposals and

verification testing stages of projects for standardization (i.e. at the beginning and the end of the work).

For measuring smaller effects and where formal subjective testing is not feasible, it is necessary to use

objective measurements. Since objective measurements are collected at multiple operational points,

and to better understand coding behaviour across all these points, what has commonly been used in

[1]

this community is the technique known as the BD-rate (Bjøntegaard delta bit rate) comparison.

2 PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC TR 23002-8:2021(E)

Encoding for video distribution is ordinarily performed in the Y′C C domain (YUV). For typical

B R

multimedia applications, it is well known that the human visual system is most sensitive to the fidelity

of the Y component. The Y component also tends to use most of the bit rate, so it is natural to focus

primarily on the Y component. However, it is advised to measure and report the fidelity of all three

components and review the balance between luma and chroma fidelity when interpreting the results.

This practice can help avoid situations where luma gain can be achieved at a significant cost of chroma

fidelity.

To calculate the Bjøntegaard delta bit rate, a distortion metric needs to be used. For standard-dynamic

range video, the distortion metric primarily used in the video coding standardization community

has been the peak signal to noise ratio (PSNR). There are certainly some weaknesses to the PSNR-

based BD-rate measure in terms of its correspondence with human perception of fidelity. Some other

objective distortion metrics which have been asserted to have a better relationship with human

[12] [13]

perception, such as structural similarity (SSIM) index, multi-scale SSIM (MS-SSIM), and video

[14]

multimethod assessment fusion (VMAF), have also been considered with the BD-rate measurement

process. However, the use of PSNR-based BD-rate measures is the most prevalent in the video coding

standardization community. In this document, BD-rate is used to denote PSNR-based BD-rate unless a

different distortion metric is explicitly mentioned.
[15] [16]
This document is based on JVET-Q0826 and JVET-R2016 .
6 The PSNR-based BD-rate concept

When developing a video coding standard, it is important to have a uniform way of reporting the

compression results so that different contributions can be compared against each other.

The PSNR metric is based on the squared error of individual sample values and does not take into

account how the human visual system works. A relevant question is therefore whether the PSNR metric

is a good predictor of subjective quality. The answer depends at least partly on how different the

encoding methods being compared are to each other. If the two methods differ greatly, their artefacts

can be very different, and the perceived subjective quality will depend heavily on which type of artefact

is psychovisually more disturbing. BD-rate measurements are most often used to compare between

two versions of the same video encoder that only differ in that in one of them one tool has been turned

on or has been modified versus the other. In this scenario it is much more likely that the BD-rate score

between these two versions will correlate with a difference in subjective quality. A clear exception is

when tools are considered that are only (or primarily) expected to affect subjective quality, such as

deblocking filters. Here, decisions are almost always based on a subjective test or expert viewing, and

BD-rate numbers are provided more as an assurance that the tool has not caused some unexpected

problem.

An advantage with using PSNR is that it is mathematically simple and therefore straightforward to

optimize for. As an example, if a tool depends on filter coefficients or other parameters, the reference

encoder can search for the parameter value that minimizes mean square error (MSE) and thus optimizes

PSNR, and this type of optimization is often straightforward to analyse and implement. The idea is that

a real encoder can optimize for a different distortion metric that is psychovisually more relevant but

where the parameter search can be a lot more complicated to implement. By choosing PSNR as the

distortion metric in the BD-rate calculations, the work can concentrate on creating coding tools instead

of spending time developing encoder optimizations for advanced distortion metrics. However, it is

also possible to compute BD-rate measurements using other objective distortion metrics or subjective

mean-opinion scores.

For high-dynamic range (HDR)/wide colour gamut (WCG) material and 360° video material, there

are additional aspects that influence the usability of BD-rate calculations; these are addressed in

Clauses 7 and 8, respectively. The JVET common test conditions also specify a separate category for

screen content material (i.e. material that has not been captured by a camera). However, in the context

of standardization development, a need for a special metric instead of PSNR-based BD-rate has not been

identified for this category.
© ISO/IEC 2021 – All rights reserved PROOF/ÉPREUVE 3
---------------------- Page: 7 ----------------------
ISO/IEC TR 23002-8:2021(E)
7 PSNR-based BD-rate calculation
7.1 General

There are several steps in the BD-rate calculation process, where the result in each step is calculated

from the result obtained in the previous step:
a) Calculation of PSNR for individual frames.

b) Calculation of per-sequence PSNR and bit rate values for each quantization parameter (QP) value.

The QP value influences the resulting bit rate. Hence, compressing the sequence several times with

different QPs ensures that the final BD-rate measurement will reflect the performance at many

different bit rates.
c) Calculation of per-sequence BD-rate values.
d) Calculation of an aggregate BD-rate value for all sequences.
These steps are further described in subclauses 7.2 through 7.6.
7.2 Calculation of PSNR for individual frames

For an individual frame, the mean square error is calculated between the luma channel decY of the

decoded output image and the luma channel origY of the original image according to Formula (1).

H−1W−1
MSEY_ = decY()xy,,−origYx()y , (1)
WH*
y=0 x=0
where

decY xy, and are the luma sample values at position xy, of the decoded and original images

() ()
origYx , y
at the same time instance, respectively;
is the width of the luma component;
is the height of the luma component.
A luma PSNR value for the frame can then be calculated using Formula (2).
 
()255()bitDepth−8
PSNR _Y = *10 log10  , (2)
 
MSEY_
 
where
bitDepth = 10 for 10-bit inputs;
denotes a bitwise left-shift operation.

If MSEY_ = 0 , i.e. if the decoded image exactly matches the original image, there is some adjustment

applied to avoid a division by zero. Different implementations can use a different adjustment method.

For example, the HEVC test model (HM) and VVC test model (VTM) software packages set the PSNR _Y

value to 999.99. In this case, the AVC joint test model (JM) and HDRTools software packages impose a

minimum MSE of 1 ÷ (W * H), and another approach could be to impose a minimum MSE of 1÷12, since

that is the MSE that would theoretically result from rounding large numbers to the nearest multiple of 1.

bitDepth

The use of 255 ≪ (bitDepth − 8) instead of 2 − 1 in the numerator of the expression in Formula (2)

is slightly unusual, but it provides a small adjustment so that if the same video content is coded using

bitDepth = 8 or is coded by shifting it up by two bits and using a 10-bit encoder, and when any error is

also just scaled up accordingly, there will be no difference in the resulting fidelity measurement. The

4 PROOF/ÉPREUVE © ISO/IEC 2021 – All rights reserve
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.