ISO/IEC 3721:2023

Information technology - Computer graphics, image processing and environmental data representation -Information model for mixed and augmented reality content - Core objects and attributes

This document specifies the information model for representing the mixed and augmented reality (MAR) scene/contents description, namely, information constructs for: a) representing the virtual reality scene graph and structure such that a comprehensive range of mixed and augmented reality contents can also be represented; b) representing physical objects in the mixed and augmented reality scene targeted for augmentation; c) representing physical objects as augmentation to other (virtual or physical) objects in the mixed and augmented reality scene; d) providing ways to spatially associate aforementioned physical objects with the corresponding target objects (virtual or physical) in the mixed and augmented reality scene; e) providing other necessary functionalities and abstractions that will support the dynamic MAR scene description such as event/data mapping, and dynamic augmentation behaviours; f) describing the association between these constructs and the MAR system which is responsible for taking and interpreting this information model and rendering/presenting it out through the MAR display device.

Titre manque

General Information

• Status: Published
• Publication Date: 21-Sep-2023
• ICS: 35.140 - Computer graphics
• Technical Committee: ISO/IEC JTC 1/SC 24 - Computer graphics, image processing and environmental data representation
• Drafting Committee: ISO/IEC JTC 1/SC 24/WG 9 - Augmented reality continuum concepts and reference model
• Current Stage: 6060 - International Standard published
• Start Date: 22-Sep-2023
• Due Date: 14-Dec-2023
• Completion Date: 22-Sep-2023

Overview - What ISO/IEC 3721:2023 covers

ISO/IEC 3721:2023 defines an information model for representing mixed and augmented reality (MAR) content, focusing on the core objects and attributes needed to describe MAR scenes. The standard extends conventional virtual reality scene representations so that content designers and MAR systems can unambiguously represent:

• virtual scene graphs and structure,
• physical (real-world) objects targeted for augmentation,
• physical objects used as augmentations to other objects,
• spatial associations between virtual and physical objects,
• event/data mapping and dynamic behaviour of augmentations,
• the linkage between content constructs and MAR systems/renderers.

This model is intended as a minimal, extensible foundation for interoperable MAR content and future MAR file formats.

Key technical topics and requirements

• Scene graph extension: Augments VR scene structures to include both virtual and real-world object constructs so a comprehensive range of MAR scenarios can be modeled.
• Physical object representation: Defines how to describe real objects within the MAR scene (as targets or augmentations).
• Spatial mapping: Mechanisms to spatially associate physical and virtual objects (e.g., transform groups, mapper nodes).
• Event and data mapping: Objects for mapping system-level events (sensors, trackers) to content-level events and behaviours.
• Dynamic behaviours: Descriptions for time-varying or event-driven augmentation behaviors.
• System association: Attributes and constructs that relate content elements to MAR system components responsible for sensing, tracking and rendering.
• Class/object taxonomy: Core classes such as MARSNode, TransformGroup, Spatial_Mapper, Event_Mapper, MARObject, Behaviour and MetaInfo are specified as building blocks for content models.

Practical applications and who uses it

ISO/IEC 3721:2023 is valuable for:

• MAR/AR content authors and designers creating scene descriptions for cross-platform players and browsers.
• MAR system and engine developers who parse, simulate and render MAR content.
• AR/VR application developers implementing spatial mapping, marker/GNSS-based experiences, or optical see-through glass displays.
• Device manufacturers and integrators aligning sensor, tracker and renderer behavior with content models.
• Standards bodies and file-format creators building interoperable MAR interchange formats.

Typical use cases shown in the standard include marker-based video see-through AR, GNSS/location-based AR, glass/optical see-through display augmentation, screen-based 2D overlays and augmented virtuality workflows.

Related standards

• ISO/IEC 18039 (MAR reference model)
• ISO/IEC 18040 (live actor/entity representation)
• ISO/IEC 19775-1 (X3D VR scene representation)
• Other related work: ISO/IEC 23488, 23490, 18038

Adopting ISO/IEC 3721:2023 helps ensure interoperable, extensible and semantically consistent MAR content across devices and platforms.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 3721
First edition
2023-09
Information technology — Computer
graphics, image processing and
environmental data representation
—Information model for mixed and
augmented reality content — Core
objects and attributes
Reference number
© ISO/IEC 2023
© ISO/IEC 2023
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii
© ISO/IEC 2023 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms, definitions and abbreviated terms . 1
3.1 Terms and definitions . 1
3.2 Abbreviated terms . 3
4 Overview . 4
5 Principles and Requirements . .5
6 MAR content model .6
6.1 Concept . 6
6.2 Use case diagram . 7
6.3 MAR content and its scene structure . 7
[6]
6.4 Major MAR system objects that are associated with the MAR content objects . 8
6.4.1 Sensor . 8
6.4.2 Real World Capturer::Sensor . 9
6.4.3 Tracker::Sensor . 9
6.4.4 Recognizer::Sensor . 10
7 MAR content classes .10
7.1 Overall class/object structure . 10
7.2 MARSNode. 12
7.3 TransformGroup::MARSNode .13
7.3.1 VirtualTG::TransformGroup::MARSNode . 14
7.3.2 RealTG::TransformGroup::MARSNode . 14
7.4 Spatial_Mapper::MARSNode .15
7.5 Event_Mapper::MARSNode . 16
7.6 MARObject::MARSNode . 17
7.6.1 VirtualObject::MARObject::MARSNode . 18
7.6.2 RealObject::MARObject::MARSNode . 19
7.7 Behaviour::MARSNode .22
7.8 MetaInfo::MARSNode . 23
8 Example usage scenarios .24
8.1 Marker based video see-through AR . 24
8.2 Location (GNSS) based video see-through AR . 24
8.3 Marker based AR for glass display . 25
8.4 Location based AR for glass display . 26
8.5 2D screen augmentation over a glass display . 27
8.6 Augmented virtuality with (multiple) live actor extracted and imported live from
a chroma-keying set-up . 27
9 Conformance .28
Bibliography .29
iii
© ISO/IEC 2023 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of
any claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC
had not received notice of (a) patent(s) which may be required to implement this document. However,
implementers are cautioned that this may not represent the latest information, which may be obtained
from the patent database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall
not be held responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 24, Computer graphics, image processing and environmental data representation.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
iv
© ISO/IEC 2023 – All rights reserved

Introduction
Mixed and Augmented Reality (MAR) refers to a spatially coordinated combination of media/
information components that represent, on the one hand the physical real world and its objects and on
the other, those that are virtual, synthetic and computer generated. MAR, as an information medium,
strives to provide rich experience based on realism, presence and augmentation.
In this document, a comprehensive set of information constructs for representing mixed and
augmented reality (MAR) contents is described. This set of components extends the conventional ones
used for representing virtual reality (VR) contents, as MAR environments are technically realized as
virtual environments. The principles and requirements for the extension are laid out and the details
of the component model including (but not limited to) those for representing physical real world
objects, extending the virtual scene graph/structure to that for MAR (with the physical objects), how
to spatially the physical objects into the MAR scene graph, associating these content components to
the MAR system, and other miscellaneous constructs (e.g. event mapping, MAR events/ behaviours,
video backdrops, etc.). This document is designed for the ease, generality and extendibility, and this
is demonstrated with various examples and implementation results. The model will serve as a sound
basis for establishing standard and interoperable file formats MAR contents in the future.
The document also provides definitions for terms as related to these MAR content informational
components and their attributes.
The target audience of this document are mainly MAR system developers and contents designers
interested in specifying MAR contents to be played by an MAR system or browser. The standard will
provide a basis for further application standards or file formats for any virtual and mixed reality
applications and content representation.
The extension will be self-contained in the sense that it is independent from the existing virtual reality
information constructs, focusing only on the mixed and augmented reality aspects.
However, this document only specifies the information model, and neither promotes nor mandate to use
a specific language, file format, algorithm, device, implementation method, and standard. The standard
model is to be considered as the minimal basic model that can be extended for other purposed in actual
implementation,
This document is based on the MAR Reference model (ISO/IEC 18039) that specifies for the contents-
browser/player type reference architecture. The MAR content (in ISO/IEC 18039) is specified as the
input that describes the scene and objects’ behaviours, given to the browser/player which in turn
parses, simulates and renders it to the display. The standard is the information model for the content.
As an extension to the virtual reality based contents or scene structure, this standard is very much
related to the existing standard for VR scene representation such as ISO/IEC 19775-1 (X3D) and other
related on-going standards such as the image-based object/environment representation for VR/MAR
(ISO/IEC 23488) as well. There are also specific object models relevant to this standard such as those
for the live actors and entities (ISO/IEC 18040 and ISO/IEC 23490) and MAR system sensor components
(ISO/IEC 18038).
v
© ISO/IEC 2023 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 3721:2023(E)
Information technology — Computer graphics, image
processing and environmental data representation —
Information model for mixed and augmented reality
content — Core objects and attributes
1 Scope
This document specifies the information model for representing the mixed and augmented reality
(MAR) scene/contents description, namely, information constructs for:
a) representing the virtual reality scene graph and structure such that a comprehensive range of
mixed and augmented reality contents can also be represented;
b) representing physical objects in the mixed and augmented reality scene targeted for augmentation;
c) representing physical objects as augmentation to other (virtual or physical) objects in the mixed
and augmented reality scene;
d) providing ways to spatially associate aforementioned physical objects with the corresponding
target objects (virtual or physical) in the mixed and augmented reality scene;
e) providing other necessary functionalities and abstractions that will support the dynamic MAR
scene description such as event/data mapping, and dynamic augmentation behaviours;
f) describing the association between these constructs and the MAR system which is responsible for
taking and interpreting this information model and rendering/presenting it out through the MAR
display device.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 18039, Information technology — Computer graphics, image processing and environmental data
representation — Mixed and augmented reality (MAR) reference model
ISO/IEC 18040, Information technology — Computer graphics, image processing and environmental data
representation — Live actor and entity representation in mixed and augmented reality (MAR)
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions
For the purposes of this document, the terms and definitions given in from ISO/IEC 18039 and
ISO/IEC 18040 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
© ISO/IEC 2023 – All rights reserved

3.1.1
aggregation
relation among objects/components that specifies a whole-part relationship as defined and used in the
Unified Modelling Language literature
3.1.2
association
relation among objects/components (and their instances) that they can be linked to each other or
combined logically or physically into an aggregation
Note 1 to entry: An aggregation is a specific form of an association as defined and used in the Unified Modelling
Language literature.
3.1.3
behaviour
object or component that describes how certain objects and their attribute values change in time and/or
in response to events
3.1.4
content
MAR content
MAR scene
description of mixed and augmented reality based experience to end-users by publishers or media
producers
3.1.5
component
MAR component
object
MAR object
self-contained computational entity or model that has one or more input channels and/or one or more
output channels
Note 1 to entry: In the context of MAR contents or system, a component has a relevant MAR functionality.
3.1.6
component model
object model
model of a collection of computational objects and their relationships serving a particular purpose or
collective function
3.1.7
event mapping
object or component that maps events produced by the MAR system and maps/relates them to the
corresponding ones in the MAR scene/content space
3.1.8
glass-based
type of an optical see-through device that has the form factor of optical glasses
3.1.9
inheritance
association representing a parent and child relationship
Note 1 to entry: In general, the child object, if not specified otherwise, will inherit the attribute model of the
parent as defined and used in the Unified Modelling Language literature.
3.1.10
node
term referred to an actual implementation or specific format of an object or component
© ISO/IEC 2023 – All rights reserved

3.1.11
optical see-through device
apparatus or device that allows the real/physical world to be seen directly or through optical elements.
in addition, the device allows an overlaid display of graphical (or virtual) elements
3.1.12
real transform group
RTG
transform group representing a group of objects situated at a physical/real spatial location
3.1.13
transform
abstract entity representing a relative spatial relationship (translation, rotation and scaling) with
another coordinate system (or equally transform)
3.1.14
transform group
TG
component or object that represents a group of objects or components sharing a common physical or
virtual coordinate system
3.1.15
video see-through device
apparatus or device that displays the real/physical world live through a video camera feed
3.1.16
virtual transform group
VTG
transform group representing a group of objects situated at a virtual spatial location
3.2 Abbreviated terms
AR augmented reality
AV augmented virtuality
CRT capturer, recognizer, tracker
GNSS global navigation satellite system
LAE live actor and entity
MAR mixed and augmented reality
MAR-RM mixed and augmented reality reference model
MR mixed reality
UI user interface
VR virtual reality
RTG real transform group
TG transform group
VTG virtual transform group
© ISO/IEC 2023 – All rights reserved

4 Overview
MAR refers to the interactive medium that uses and merges the real (or physical) and virtual objects.
[6]
The two representative genres of MAR are augmented reality (AR) where virtual objects are added
on to representation of the real physical world, while in augmented virtuality (AV) real world physical
object representations are added to the virtual environment. The continued innovations and advances
in computer vision, mobile/cloud computing and portable display devices have brought about a renewed
interest in MAR, as a prominent information visualization and interaction medium. MAR not only has
many application areas but also can empower users in daily activities by spatially augmenting useful
information to key interaction objects. Traditionally, MAR services/contents have been developed as a
single application using popular application programming interfaces (APIs). While there are moves to
[7,8]
standardize the API ,there is also a strong and natural move toward the separation of such singleton
applications into contents (in some standard formats) and dedicated players/browsers, similarly to
those for web documents.
X3D already offers a standard and declarative method of specifying dynamic virtual environments
[9]
(ISO/IEC 19775-1) and is in fact being extended for MAR functionalities . Such a service structure (and
standards) can promote the proliferation of the given media and associated content form. Thus, in this
document, a minimal (yet able to cover a comprehensive and reasonable, typical MAR contents) set
of information constructs for representing MAR contents is established. The standard would extend
the conventional constructs used for representing virtual reality contents (such as X3D), as virtual
environments are used for the implementation platform for mixed reality contents as well.
First, the principles and requirements for the extension are laid out and the details of the component
model including (but not limited to) those for representing real world physical objects, extending
the virtual scene graph/structure to that for MAR (with the real objects), how to spatially the real
objects into the MAR scene graph, associating these content components to the MAR system, and other
miscellaneous constructs (e.g. event mapping, MAR events/ behaviours, video backdrops, etc.). The
standard is designed for the ease, generality and extendibility, and this is demonstrated with various
examples and implementation results. The model will serve as a sound basis for establishing standard
and interoperable file formats MAR contents in the future.
Most MAR systems have been implemented as a single application with all the contents and assets
embedded in it, using programming libraries or APIs. The GPS equipped mobile and smart phones
have made ways for location based augmented reality services (e.g. providing guides for commercial
points of interest and tourism). Such a service necessitated the separation of contents (and its format
specification) and the underlying player (to support the notion of one place-many contents). Future
interoperability, reusability and proliferation of MAR contents will hinge on the comprehensible and
extendable content model (and file format). For example, the file format for location based AR service
called the ARML (Augmented Reality Mark-up Language) has been adopted as a standard for the
Open Geospatial Consortium. ARML allows defining geographical points or landmarks of interest and
associate GPS coordinates and simple augmentation contents (e.g. text, logos, and image).
The Web3D consortium has also been considering adding new components for MAR functionality to
X3D, the ISO standard and declarative mark-up file format for representing virtual environments. The
group has specified new and extended nodes to support e.g. video see-through based AR, such as the
[9]
live video background, extended camera sensor nodes . Some of its work is reflected in this standard
as well. As the video augmented content is a popular form of MAR, the MPEG has proposed the ARAF
(AR Application Format) that uses other MPEG standards (like MPEG-4 and MPEG-V) to specify video
based and spatially augmented contents (ISO/IEC 23000-13).
Another class of approach of realizing, extending and specifying for MAR functionality and behaviour
[10]
is through scripting. InstantReality has developed its own extension to X3D and modules (e.g. for
marker and image patch tracking allowing to express MAR contents with scripts through which the
virtual objects (as expressed by the usual X3D formalism) can be associated with the position/pose
tracked real world physical objects. There are several similar web-based MAR systems that can present
1) [11-13]
AR contents with Javascript programming . Despite these approaches, the current state with
1) JavaScript is a registered trademark of Oracle Corporation. This information is given for the convenience of
users of this document and does not constitute an endorsement by ISO or IEC.
© ISO/IEC 2023 – All rights reserved

respect to MAR content representation is still limited in its comprehensiveness (e.g. only covers certain
genre of MAR such as location based or video based), often implementation dependent and proprietary,
and lacks sufficient abstraction and clean modularization requiring lengthy and complicated script
programming.
The MAR Scene or equivalently content description (purple box in the left part of the figure) is an input
to the larger MAR system to be interpreted by it and be rendered to a display for the user consumption.
Figure 1 — The generic MAR system architecture as specified in the ISO/IEC 18039 MAR
reference model
5 Principles and Requirements
The MAR reference model (ISO/IEC 18039) suggests the scope for which a general MAR system and
content shall encompass as shown in Figure 1. Accordingly, the MAR content model should provide
reasonable generality, and be able to express both AR and AV, the two notable genres along the MAR
continuum under a consistent and unified representational framework. Other requirements are as
follows:
— independent of particular sensor/device model, algorithm or implementation platform (such issues
would be absorbed into the browser/player implementation),
— provide virtually any digital information and media, both static and dynamic, as augmentation such
as text, images, videos, animation, HTML document elements, etc.,
— provide useful abstract and declarative constructs for often used content functionalities and
minimize manual scripting or programming,
— make use of existing standard constructs, where possible,
— flexible/extendible to accommodate new future requirements.
To fulfil the final requirement (and others indirectly), this standard will use the component based
approach, similarly to that of the X3D. Components in X3D are collections of objects that perform or
represent similar operations, displays, or functions. Each component may define multiple levels. The
levels indicate the complexity of the object or object features. The component framework is ideal for
introducing and modelling new capabilities that satisfy MAR requirements (regardless of using X3D or
© ISO/IEC 2023 – All rights reserved

not) – new objects can be added or expanded as needed. The components form a structured environment
that ensures consistent behaviour, usability and generality. The MAR reference model (ISO/IEC 18039)
outlines the model architecture of a prototypical MAR system and indirectly points to the informational
needs of an MAR content as an input to the system. In the next clause is described, the kind of new or
expanded MAR components (equivalently functionalities) to be abstracted and represented.
Again it is emphasized that the MAR content is the input to the system (purple box in the left part
of Figure 1) which comprises the entities such as the sensor, capturer, tracker, recognizer, simulator,
renderer, etc. Thus, while the content model may be associated with such system components, the
system components themselves would not be part of the content description or model.
Note that the extension will be self-contained in the sense that it is independent from the existing
virtual reality information constructs, focusing only on the mixed and augmented reality aspects.
6 MAR content model
6.1 Concept
MAR contents are technically realized as virtual reality contents with its scene containing special type
objects that represent the real physical world/objects. In other words, one can think of a MAR scene
that is technically a virtual scene with placeholders for the real world physical objects. The placeholders
may be defined logically or spatially with respect to the virtual scene. These so called special type
placeholder objects may need additional information of how it might be spatially registered into the
implementation virtual scene. It is important to note that such a representation is (and should be)
amenable to representing both AR and AV contents in a unified fashion.
Real world objects (in yellow) are associated with the virtual environment scene graph (in orange), and
in turn the real objects are sensed, tracked, recognized and captured by the MAR system (in blue).
Figure 2 — The concept of extending virtual reality scene structure for MAR contents
Figure 2 illustrates the basic concept in which virtual objects (or the root of the virtual scene) provide
a place for which certain real world physical object (remote or local) would map to in the MAR scene.
This way, the real world physical objects are seamlessly represented within the virtual scene structure.
For example, a virtual object can be spatially defined with respect to a real world physical object (or its
placeholder). Note that the virtual scene may not necessarily be 3D (i.e. could be augmentation onto the
2D content), and real world physical objects can be individually recognized and tracked, or captured
as a whole (e.g. live video) and implanted into the virtual scene. This is accomplished and specified
by association with the MAR System components (e.g. blue boxes in Figure 2). The whole content
© ISO/IEC 2023 – All rights reserved

structure is input to the MAR Simulation Engine (another MAR System component) to be interpreted
and ultimately rendered and displayed to the user. The red arrows in the figure indicate spatial
relationships among the captured/tracked real world physical object with respect to the virtual scene/
objects. Such a scheme is easy to define by simply extending the current scene structure for virtual
environments.
In the following clauses, we describe the component based information model using the Unified
[15]
Modelling Language (UML) like notation, starting with the Use Case Diagram. The use of UML-like
conventional is not meant to be followed strictly, but only to level sufficient for to the purpose of this
document. Then the class-object diagram is described for the most important and essential ones. Several
examples of how the class-object specification can be used to represent a variety of MAR content type
are shown. Finally, the detailed component object specifications are given.
6.2 Use case diagram
Use case diagrams are usually referred to as behavior diagrams used to describe a set of actions
(use cases) that some system or systems (subject) should or can perform in collaboration with one or
more external users of the system (actors). Each use case should provide some observable and valuable
result to the actors or other stakeholders of the system. Figure 3 shows a use case of a user interacting
with the MAR system (e.g. a browser or player) that plays a user selected MAR content. The user selects
a particular content and gives input, if necessary, to drive the content as rendered and displayed by the
MAR system.
Figure 3 — A use case of a user interacting with the MAR system (e.g. a browser or player) that
plays a user selected MAR content
6.3 MAR content and its scene structure
Just like the virtual reality environment or contents, the scene graph or scene tree structure would be
the most natural and suited representation with which the MAR content is represented as a hierarchical
and spatial organization of various types of objects. While the objects are in principle organized by their
spatial relationship ultimately with respect to the assumed root coordinate system, logical association
and aggregation are possible too.
Thus, in this regard all objects, as part of the MAR scene graph, are subclasses of an abstract MARSNode
object, specialized for whatever purposes they may have.
© ISO/IEC 2023 – All rights reserved

Aggregation relation is formed and used among objects to represent their collective spatial positioning
and physical containment among the various objects in the scene. In particular, aggregation relation
will be used to group various objects into a particular TG which also contains the common spatial
information shared among the constituent objects (i.e. transformation – rotation, translation, scaling)
with respect to another TG (or by default with respect to the assumed world root coordinate system).
The TG aggregation also serves to represent the part-of relation in the scene hierarchy.
Association relations may exist among various objects that can refer and relate to each other for
different purposes. The association may be one directional or two directional. For example, associations
between content objects and system objects are typically one directional in the sense that the content
elements seek/retrieve raw data as generated and mapped from the MAR system (i.e. sensors, capturer,
tracker and recognizer).
[6]
6.4 Major MAR system objects that are associated with the MAR content objects
MAR content is an input to the MAR System which interprets the content/scene description, simulates
it and displays it to the user. Among others, the MAR system uses the sensors and other associated
modules such as the tracker, recognizer and real world capturer (see Figure 1) to understand the
real physical environment and extract various information in real time. Such information is used to
associate the real world physical objects to the virtual and form the mixed reality scene. Sensors may
be used simply to realize interactivity of the content as well. Therefore, while the MAR system objects
are not part of the content itself, they have associations to the counterpart content objects. For example,
the tracker may be associated to the spatial_mapper objects (in the content) that uses the tracking
information to position the tracked real world physical object into the MAR scene.
In the specification of this standard, the detailed informational constructs for the non-content MAR
System are excluded such as those for the sensor, real world capturer, tracker and recognizer. We assume
that the constructs exist (in certain system implementation or according to some other standard). We
also assume that the event/data types as produced by these MAR System objects and consumed by the
MAR Content components (MARSNode) are already defined. The definition and functional description
of the main MAR System objects that the content description are associated with are well described
and given in the MAR reference model (ISO/IEC 18039).
6.4.1 Sensor
A sensor is a hardware (and optionally) software component able to measure specific physical
properties. In the context of MAR, a sensor is used to detect, recognize and track the target physical
object to be augmented. In this case, it is called a pure sensor.
A sensor can measure different physical properties, and interpret and convert these observations into
digital signals. The input and output of the sensors are:
— input: real world signals;
— output: sensor observations with or without additional metadata (position, time, etc.).
In particular, the sensor output may feed into the content components. The sensor component is
a physical device characterized by a set of capabilities and parameters. A prototypical object model
is shown in Table 1. Many subclasses of the sensor can exist in the specialized but inherited form. A
[15]
refined standard is being pursued through ISO/IEC 18038 .
Table 1 — A prototypical object specification for Sensor object in the MAR System
Sensor
Data type Attribute/Method name Explanation
string id unique identifier for reference
int type integer code indicating the particular sensor
type
© ISO/IEC 2023 – All rights reserved

6.4.2 Real World Capturer::Sensor
Another use of the Sensor is to capture and stream to the content objects and the execution engine,
as the data representation of the physical world or objects for composing a MAR scene. In such a case,
it is called a Real World Capturer. A typical example is the video camera that captures the real world
as a video to be used as a background in an augmented reality scene. Another example is augmented
virtuality, where a person is filmed in the real world and the corresponding video is embedded into a
virtual world.
Thus, one subclass of Sensor is the Real World Capturer whose output is an audio, video or haptics
stream to be embedded in the MAR scene or analysed by specific hardware or software/content objects.
A prototypical object might look like Table 2:
Table 2 — A prototypical object specification for Real World Capturer in the MAR System
Capturer
Data type Attribute/Method name Explanation
Any* rawData sensed data
bool enabled Boolean value indicating whether to enable this Capturer
Capturer* Capturer() constructor
void filter() function filtering or post-processing the sensed and cap-
tured data
Any* getData() access function for the captured data
6.4.3 Tracker::Sensor
Another two subclasses of the Sensor are the Tracker and the Recognizer. The two are also collectively
called the Context Analyser as well. The Context refers to the condition for which the augmentation
should occur upon its target object. The Tracker and Recognizer provides information regarding this
context.
The Tracker is a hardware or software object that analyses signals from the real world and detects and
measure changes of the properties of the target signals (e.g. pose, orientation, volume).
The input and output of the tracker are:
— input: raw or processed signals representing the physical world and target object specification data
(reference target to be recognized);
— output: instantaneous values of the characteristics (pose, orientation, volume, etc.) of the recognized
target signals and event data.
A prototypical component might look like Table 3:
Table 3 — A prototypical object specification for Tracker object in the MAR System
Tracker
Data type Attribute/Method name Explanation
Any* rawData sensed data (depends on the sensor type)
bool enabled boolean value indicating whether to enable this
Tracker
Tracker* Tracker() constructor
void filter() function filtering or post-processing the sensed
and tracked data
Any* getData() access function for the captured data
© ISO/IEC 2023 – All rights reserved

6.4.4 Recognizer::Sensor
The Recognizer is a hardware or software object that analyses signals from the real world and
produces MAR events and data (Table 4) by comparing with a local or remote target signal (i.e. target
for augmentation).
Recognition can only be based on prior captured target signals. Both the recognizer and tracker can
be configured with a set of target signals provided by or stored in an outside resource (e.g. third
party database server) in a consistent manner with the scene definition, or by the MAR scene/content
description itself.
The input and output of the recognizer are:
— input: raw or processed signals representing the physical world (provided by sensors) and target
object specification data (reference target to be recognized);
— output: at least one event acknowledging the recognition.
Table 4 — A prototypical object specification for Recognizer object in the MAR System
Recognizer
Data type Attribute/Method name Explanation
Any* target target object template information to be recognized
Any* rawData recognized event (depends on the Recognizer type)
bool enabled boolean value indicating whether to enable this
Recognizer
Tracker* Tracker() constructor
void filter() function filtering or post-processing the sensed
and recognized data
Any* getData() access function for the recognized data
7 MAR content classes
7.1 Overall class/object structure
Figures 4, 5, 6 show the overall class diagram as the gross information model for MAR contents.
Figure 4 diagram shows the inheritance relationship among the classes as part of the MAR scene
structure nodes (see 7.1). The Figures 5 and 6 show the association among the classes, defining the
MAR scene structure itself. Explanations of each class description (this clause) follow and the detailed
individual class specifications are given in Clause 8. To be clear, objects are instantiations of the classes
(or subclasses). Some classes will be abstract and have subclasses (or lower level classes) from which
objects can be instantiated for actual deployment in usage.
© ISO/IEC 2023 – All rights reserved

Figure 4 — Inheritance relationship (using the arrow head links) among classes in the MAR
scene structure
Key
aggregation
association
Figure 5 — Association and aggregation relationship among classes in the MAR scene structure
(simplified).
© ISO/IEC 2023 – All rights reserved

Key
aggregation
association
Figure 6 — Association and aggregation relationship among classes in the MAR scene structure
(detailed).
7.2 MARSNode
As indicated before, all MAR content components are subclasses of the abstract MARSNode. MARSNode
represents the superclass that can take a place in the hierarchical (tree) representation of the scene. In
Table 5 is the detailed class specification model with the explanation of the key attributes/methods. All
subsequent class/objects derive from this superclass. In this context, class, class object, object are used
interchangeably without ambiguity.
Table 5 — A prototypical class specification for MARSNode
MARSNode
Access type Data type Attribute/Method name Explanation
private string id unique identifier for reference
private MARSNode[] parent parent nodes (usually, there is only one
parent)
private MARSNode[] childrenNodes list of or array of one or more children nodes,
also of the MARSNode (or its subclass) type
private Cube bounding-box bounding box specification of for the ob-
ject this node represents in the MAR scene
(optional).
public MARSNode MARSCNode() MARSNode constructor
public void init() abstract initializing method for the MARS-
Node class
© ISO/IEC 2023 – All rights reserved

TTabablele 5 5 ((ccoonnttiinnueuedd))
MARSNode
Access type Data type Attribute/Method name Explanation
public string getId() return the string id of this node
public void setId(string id) set the id of this node
public void addChild(MARSCNode child) add a child to
...