Information technology — MPEG systems technologies — Part 14: Partial file format

This document specifies the partial file format, which is a generic format for describing file partially received over lossy communication channels. This format contains the correctly received data, missing block identification, and repair information such as location of the file or high-level original indexing information. This format can be used with any file formats, and provides additional helper tools for formats deriving from ISO/IEC 14496-12.

Technologies de l'information — Technologies des systèmes MPEG — Partie 14: Format de fichier partiel

General Information

Status
Published
Publication Date
09-Jan-2019
Current Stage
6060 - International Standard published
Start Date
10-Jan-2019
Completion Date
10-Jan-2019
Ref Project

RELATIONS

Buy Standard

Standard
ISO/IEC 23001-14:2019 - Information technology -- MPEG systems technologies
English language
20 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 23001-14
First edition
2019-01
Information technology — MPEG
systems technologies —
Part 14:
Partial file format
Technologies de l'information — Technologies des systèmes MPEG —
Partie 14: Format de fichier partiel
Reference number
ISO/IEC 23001-14:2019(E)
ISO/IEC 2019
---------------------- Page: 1 ----------------------
ISO/IEC 23001-14:2019(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2019

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2019 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 23001-14:2019(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms, definitions, symbols, abbreviated terms and conventions ...................................................................1

3.1 Terms and definitions ....................................................................................................................................................................... 1

3.2 Abbreviated terms ............................................................................................................................................................................... 2

4 Partial file organization ................................................................................................................................................................................ 2

4.1 Object structure...................................................................................................................................................................................... 2

4.2 Design consideration ......................................................................................................................................................................... 2

4.2.1 Features ................................................................................................................................................................................... 2

4.2.2 Data layout ............................................................................................................................................................................ 3

4.2.3 Box order ................................................................................................................................................................................ 3

4.2.4 Mixed partial files ............................................................................................................................................................ 4

4.2.5 Processing model ............................................................................................................................................................ 5

5 Box structures ......................................................................................................................................................................................................... 5

5.1 Partial file general boxes ................................................................................................................................................................ 5

5.1.1 Partial file box..................................................................................................................................................................... 5

5.1.2 Partial file header box ................................................................................................................................................. 6

5.1.3 Partial segment box ....................................................................................................................................................... 6

5.1.4 Partial segment header box .................................................................................................................................... 7

5.1.5 Partial data box ................................................................................................................................................................. 8

5.1.6 Partial segment location box ................................................................................................................................. 9

5.1.7 Partial top level entry point box ......................................................................................................................10

5.1.8 Source URL box ..............................................................................................................................................................11

5.2 Partial file ISO base media file helpers ............................................................................................................................12

5.2.1 General...................................................................................................................................................................................12

5.2.2 Box file index box .........................................................................................................................................................12

5.2.3 Box index box ...................................................................................................................................................................13

5.2.4 Data integrity box .........................................................................................................................................................14

5.2.5 Data integrity hash box ...........................................................................................................................................14

5.2.6 Front part box ..................................................................................................................................................................16

Annex A (normative) Partial file MIME type registration ............................................................................................................18

Bibliography .............................................................................................................................................................................................................................20

© ISO/IEC 2019 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 23001-14:2019(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/patents) or the IEC

list of patent declarations received (see http: //patents .iec .ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso

.org/iso/foreword .html.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO 23001 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/members .html.
iv © ISO/IEC 2019 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 23001-14:2019(E)
Introduction

The partial file format is designed to contain files partially received over a lossy link (with unreceived

or corrupted sections), for further processing such as playback or repair. The file structure is object-

oriented; a file can be decomposed into constituent objects very simply, and the structure of the objects

inferred directly from their type. Files conforming to this format may be exchanged without losing

information on losses, allowing redistribution (local gateways) or further repair processes.

The partial file format may be used to document reception of files, regardless of their bitstream format.

For generic cases, it provides ways for file readers to resynchronize their parsing in case of byte losses.

For cases where the documented file derives from ISO/IEC 14496-12, the partial file format provides

additional tools, such as an index of the source file structures and data integrity information.

Annex A of this document defines the associated MIME type for files conformant to this document.

© ISO/IEC 2019 – All rights reserved v
---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23001-14:2019(E)
Information technology — MPEG systems technologies —
Part 14:
Partial file format
1 Scope

This document specifies the partial file format, which is a generic format for describing file partially

received over lossy communication channels. This format contains the correctly received data, missing

block identification, and repair information such as location of the file or high-level original indexing

information. This format can be used with any file formats, and provides additional helper tools for

formats deriving from ISO/IEC 14496-12.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 14496-12, Information technology — Coding of audio-visual objects — Part 12: ISO Base Media

file format

IETF RFC 5905, Network Time Protocol Version 4: Protocol and Algorithms Specification, Mills, D., et al,

June 2010
3 Terms, definitions, symbols, abbreviated terms and conventions
3.1 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https: //www .iso .org/obp
— IEC Electropedia: available at http: //www .electropedia .org/
3.1.1
chunk
addressable set of bytes of a partial segment
3.1.2
indexed box

box in the source file (3.1.5) indexed by a BoxIndexBox in the partial file (3.1.3)

3.1.3
partial file
name of the files conforming to the file format described in this document
3.1.4
partial segment

portion of the partial file (3.1.3) containing a single PartialSegmentBox and identifying its

associated source data
© ISO/IEC 2019 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO/IEC 23001-14:2019(E)
3.1.5
source file
file described and encapsulated by the partial file (3.1.3)
3.2 Abbreviated terms
For the purposes of this document, the following abbreviated terms apply.
MD5 Message Digest 5
FDT File Delivery Table
FEC Forward Error Correction
FLUTE File Delivery over Unidirectional Transport
IANA Internet Assigned Numbers Authority
IETF Internet Engineering Task Force
RFC Request for Comments
SHA Secure Hash Algorithm
4 Partial file organization
4.1 Object structure

A partially received file, hereinafter referred to as partial file, is represented as a sequence of partial

segments, preceded by a header. The header and the partial segments are formed as a series of objects,

called boxes, as defined in ISO/IEC 14496-12; files under this document are object-structured files as

defined in ISO/IEC 14496-12.

All object-structured files conformant to Clause 4 shall contain a FileTypeBox as defined in ISO/IEC

14496-12. The FileTypeBox shall contain a compatible brand of type 'paff' for partial files, and of

type 'pmff' for mixed partial files.

In this document, top-level boxes (boxes not contained in other boxes) are indicated as being at ‘file’

level, with the notation “Container: File”.

The MIME type associated with a file conforming to this specification shall be formatted as defined in

Annex A.
4.2 Design consideration
4.2.1 Features

The partial file format is intended to serve as a storage and exchange format for other file formats

delivered over lossy channels. The format provides the following set of tools:

— Reception data, which provides means to store the received data and document transmission

information such as received or lost byte ranges and whether the corrupted/lost bytes are present

in the file.

— Repair information, such as location of the source file, possible byte offsets in that source, byte stream

position at which a parser can try processing a corrupted file; depending on the communication

channel, this information may be setup by the receiver or through out-of-band means.

2 © ISO/IEC 2019 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 23001-14:2019(E)

— File format specific information, which depends on the type of file stored as a partial file; this

document only defines additional tools for files based on ISO/IEC 14496-12.
4.2.2 Data layout

The partial file format is designed to allow continuous recording and storing of received data in a

flexible way. A partial file is composed of a header providing information valid for the entire recording

followed by any number of segments; all of these segments are described by partial segment structures

and are hence called partial segments, although some can be correctly received. The header does not

contain any data from the source file and contains only information that can usually be inferred from

the parameters of the transmission, and hence does not require editing of the header values upon

finalizing the recording. Each partial segment contains a variable number of received bytes from a given

source, a map of valid byte ranges for these received bytes and optionally some repair information.

Byte maps give offsets relative to the first byte of data in this partial segment, rather than from the

first byte of data in the source file, in order to process partial segments without any knowledge of their

position in the partial file. Each partial segment may describe correctly received data, lost data or a mix

of correctly received and loss data.
4.2.3 Box order

An overall view of the normal encapsulation structure is provided in Table 1. In the event of a conflict

between this table and the prose, the prose prevails. The order of boxes within their container is not

necessarily indicated in the table.

The table shows those boxes that may occur at the top-level in the left-most column; indentation is

used to show possible containment. Not all boxes need to be used in all files; the mandatory boxes are

marked with an asterisk (*). See the description of the individual boxes for a discussion of assumptions

if the optional boxes are not present.

Objects using an extended type may be placed in a wide variety of containers, not just the top level.

The following rules shall be followed for the order of boxes in a partial file:
— The FileTypeBox shall occur first in the file.
— The PartialFileBox, if present, shall occur immediately after the FileTypeBox.
Table 1 — Box types, structure and cross-reference
Box types, structure and cross-reference
ftyp * file type and compatibility
pfil
5.1.1 container for metadata global to file
pfhd * 5.1.2 data about the entire partial file
surl
5.1.8 source URL and mime for the complete source file
ptle 5.1.7 entry points information for the complete source file
fidx
5.2.2 box file index for the complete source file
bidx 5.2.3 box index for a box of the source file
dint
5.2.4 data integrity container for the complete source file
dihd * 5.2.5 data integrity hash
frpa
5.2.6 front part (initial bytes) of an indexed box
dref 5.1.1 data reference box as defined in ISO/IEC 14496-12
pdat
5.1.2 partial segment data
pseg * 5.1.4 partial segment
Key

* Boxes that are mandatory within their container, which itself can be mandatory or optional.

© ISO/IEC 2019 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO/IEC 23001-14:2019(E)
Table 1 (continued)
Box types, structure and cross-reference
5.1.5 partial segment header containing information related to the partial
pshd *
segment
ploc
* 5.1.6 partial segment location
surl 5.1.8 source URL and mime of the source file for the partial segment
ptle
5.1.7 entry point information of the source file for the partial segment
fidx 5.2.2 box file index of the source file for the partial segment
bidx
5.2.3 box index for a box of the source file for the partial segment
dint 5.2.4 data integrity container of the source file for the partial segment
dihd
* 5.2.5 data integrity hash
frpa 5.2.6 front part (initial bytes) of an indexed box
Key

* Boxes that are mandatory within their container, which itself can be mandatory or optional.

4.2.4 Mixed partial files

When the source file uses a format derived from ISO/IEC 14496-12, it can be desirable to encapsulate

only the corrupted top-level boxes of the source file in a partial segment, while leaving the correctly

received top-level boxes untouched. Figure 1 shows the case where certain movie segments have been

received complete, other parts are only partial and there are yet other parts for which it is known that

no information has been received. The incomplete parts map to one or several partial segments.

Figure 1 — Mixed partial file

This design is referred to as mixed partial file. In this configuration, the FileTypeBox shall contain

a compatible brand of type 'pmff'. The FileTypeBox of the source file, whether or not correctly

received, shall not be modified; it shall be encapsulated in one partial segment and stored in a

PartialDataBox, in order to ensure unicity of the FileTypeBox at the container level. Mixed partial

file writers may insert, in the FileTypeBox of the partial file, brands from the source ISOBMFF file

if the requirements of these brands are still met in the mixed partial file. This design allows backward

compatibility with existing ISOBMFF file reader, as they would skip unrecognized boxes but still play

the rest of the presentation.

The mixed partial file format is designed to store partially received fragments of a given ISOBMFF file,

without any inspection or modification of the internal box structure of the source file. This format cannot

be used to indicate, through in-file structures such as SampleTableBox or TrackFragmentBox,

which items or samples are missing or corrupted. Such operation requires reprocessing of the source

file and is possible using dedicated structures defined in ISOBMFF.

NOTE 1 In case an ISOBMFF segment made of multiple movie fragments is corrupted and the initial

SegmentTypeBox is lost, a mixed partial file reader might have to insert a SegmentTypeBox at the appropriate

place for correct processing by higher application layers.

NOTE 2 When ISOBMFF segments are stored in a mixed partial file, track timing can be lost if correctly

received ISOBMFF segments do not contain TrackFragmentBaseMediaDecodeTimeBox.
4 © ISO/IEC 2019 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 23001-14:2019(E)
4.2.5 Processing model

The processing model of partial files or mixed partial files is illustrated in Figure 2. In partial file mode,

there is a “partial file handler” that is used, which in turn delegates to the original file handler. In mixed

partial mode, the original file handler can be extended to handle partial files with their own original

MIME type.
Figure 2 — Processing model of partial files or mixed partial files

The source file shall be identical to the ordered concatenation of the data of each partial segment for

that given source if no losses occurred or if all corrupted blocks were successfully repaired. A mixed

partial file handler may typically optimize this process by not reconstructing the complete source file.

5 Box structures
5.1 Partial file general boxes
5.1.1 Partial file box
5.1.1.1 Definition
Box Type: 'pfil'
Container: File
Mandatory: No
Quantity: Zero or One

This box contains metadata information for the partial file. This metadata can be overridden with each

partial segment.

A PartialFileBox may contain zero or one DataReferenceBox as defined in ISO/IEC 14496-

12. Data references shall only be used to indicate an external location of (part of) the stored data;

consequently, entries of the DataReferenceBox shall not use the flag value 0x000001.

NOTE Usage of data references allows building a partial file describing the recorded result of a transmission

session stored in another format than the partial file format; for example, a partial file can be computed to

describe the current reception state of a torrent session.

When enclosed, the BoxFileIndexBox provides a summary of the box hierarchy of the complete

source file. It may be used to describe only type and size of a file-level box, or the complete or partial

© ISO/IEC 2019 – All rights reserved 5
---------------------- Page: 10 ----------------------
ISO/IEC 23001-14:2019(E)

hierarchy of child boxes of one or several file-level boxes. Each top-level box of the source file will

typically have zero or one associated BoxIndexBoxes in the BoxFileIndexBox.

If the set of boxes in the BoxFileIndexBox is also complete, the corresponding bit in file_flags

should also be set.

If one BoxFileIndexBox is present in the PartialFileBox, it indexes (part of) the complete file

and no other BoxFileIndexBox shall be present in any subsequent PartialSegmentBox.

5.1.1.2 Syntax
aligned(8) class PartialFileBox extends Box(’pfil’) {
Box other_boxes[]
5.1.2 Partial file header box
5.1.2.1 Definition
Box Type: 'pfhd'
Container: PartialFileBox
Mandatory: Yes, if PartialFileBox is present
Quantity: Exactly one

This box contains general information about the partial file data and structures.

The following flag is defined for the PartialFileHeaderBox:

— complete_index: flag value is 0x000001; when set, indicates that the BoxFileIndexBox

contained in this box is complete (the contents are always required to be dense and start at the

beginning, so this means that the box documents all bytes to the end of the original box-structured

file). This flag shall not be set for non-box-structured original files.
5.1.2.2 Syntax
aligned(8) class PartialFileHeaderBox extends FullBox(’pfhd’, 0,
flags) {
5.1.3 Partial segment box
5.1.3.1 Definition
Box Type: 'pseg'
Container: File
Mandatory: Yes
Quantity: One or more

The PartialSegmentBox is a container for metadata describing one or more blocks of received data

from a given source, missing or corrupted bytes, source or indexing information. There are usually

several instances of PartialSegmentBox in a partial file.
6 © ISO/IEC 2019 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 23001-14:2019(E)

A PartialSegmentBox only contains boxes describing the source data; it does not contain

any source data. Each PartialSegmentBox shall contain one PartialSegmentHeaderBox

containing configuration of the partial segment; the associated source data may be located after or

before the PartialSegmentBox, as described by the PartialSegmentLocationBox, and may be

encapsulated in one or more PartialDataBox.

The resulting partial segment (i.e. the PartialSegmentBox and its associated data) shall conform to

the object structure defined in subclause 4.1.

Since reconstruction of the source file is based on information from partial segment, storing a segment

received without errors still requires the presence of a partial segment box describing the received data.

When enclosed, a BoxFileIndexBox provides a summary of the box hierarchy of the source file data

in the current partial segment. It may be used to describe only type and size of a file-level box, or the

complete or partial hierarchy of child boxes of one or several file-level boxes. Each top-level box of the

source file will typically have zero or one associated BoxIndexBoxes in the BoxFileIndexBox.

If the set of boxes in the BoxFileIndexBox box is also complete, the corresponding bit in segment_

flags should also be set.

If one BoxFileIndexBox is present in the PartialFileBox, it indexes (part of) the complete file

and no other BoxFileIndexBox shall be present in any subsequent PartialSegmentBox.

5.1.3.2 Syntax
aligned(8) class PartialSegmentBox extends Box(’pseg’) {
Box other_boxes[]
5.1.4 Partial segment header box
5.1.4.1 Definition
Box Type: 'pshd'
Container: PartialSegmentBox
Mandatory: Yes
Quantity: One per PartialSegmentBox

The PartialSegmentHeaderBox gives general information for the partial segment. The

PartialSegmentHeaderBox shall be placed first in the PartialSegmentBox.
The following flags are defined for the PartialSegmentHeaderBox:

— last_segment: flag value is 0x000001. Presence of this flag indicates that this partial segment

is the last one from the associated source; for a given file location as indicated by SourceURLBox,

there should not be more than one partial segment with last_segment flag set in the

PartialSegmentHeaderBox, and that partial segment should be the last one in the file. Readers

may stop processing the file data from this source once this flag is encountered.

— no_further_repair: flag value is 0x00000002. This flag indicates that the repair operation(s)

repaired the maximum information (no further repair would be beneficial), yet there may still be

corrupted chunks in the content.

— complete_index: flag value is 0x000004; when set, has the same semantics as the complete_

index flag defined for the PartialFileHeaderBox.
© ISO/IEC 2019 – All rights reserved 7
---------------------- Page: 12 ----------------------
ISO/IEC 23001-14:2019(E)

NOTE The last_segment flag can be used to detect end of file reception by file readers processing a

file being recorded by another entity. The no_further_repair flag can be used to avoid repair attempts on

content no longer available.
5.1.4.2 Syntax
aligned(8) class PartialSegmentHeaderBox extends FullBox(’pshd’,
version, flags) {
if (version==0) {
unsigned int(32) source_byte_offset;
} else {
unsigned int(64) source_byte_offset;
}
unsigned int(64) last_repair_time;
5.1.4.3 Semantics

source_byte_offset gives the offset in bytes in the associated file indicated by SourceURLBox

of the first byte of the first chunk in this partial segment; value 0 means that this first byte of the

first chunk is the first byte of the associated source file. The meaning of this field is unspecified if no

SourceURLBox is present in the partial file.

last_repair_time gives the 64-bit NTP timestamp of the last repair process. The value shall be

formatted in accordance with IETF RFC 5905. If 0, that time is unknown. File readers may use this

information to decide whether attempting a repair on the file is useful or not (for example in scenario of

file redistribution in a home network where the access point would have attempte
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.