Information technology — Computer graphics, image processing and environmental data representation — Live actor and entity representation in mixed and augmented reality (MAR)

This document defines a reference model and base components for representing and controlling a single LAE or multiple LAEs in an MAR scene. It defines concepts, a reference model, system framework, functions and how to integrate a 2D/3D virtual world and LAEs, and their interfaces, in order to provide MAR applications with interfaces of LAEs. It also defines an exchange format necessary for transferring and storing LAE-related data between LAE-based MAR applications. This document specifies the following functionalities: a) definitions for an LAE in MAR; b) representation of an LAE; c) representation of properties of an LAE; d) sensing of an LAE in a physical world; e) integration of an LAE into a 2D/3D virtual scene; f) interaction between an LAE and objects in a 2D/3D virtual scene; g) transmission of information related to an LAE in an MAR scene. This document defines a reference model for LAE representation-based MAR applications to represent and to exchange data related to LAEs in a 2D/3D virtual scene in an MAR scene. It does not define specific physical interfaces necessary for manipulating LAEs, that is, it does not define how specific applications need to implement a specific LAE in an MAR scene, but rather defines common functional interfaces for representing LAEs that can be used interchangeably between MAR applications.

Technologies de l'information — Infographie, traitement de l'image et représentation des données environnementales — Représentation d'acteurs et d'entités réels en réalité mixte et augmentée (MAR)

General Information

Status
Published
Publication Date
28-May-2019
Current Stage
6060 - International Standard published
Start Date
29-May-2019
Completion Date
29-May-2019
Ref Project

Buy Standard

Standard
ISO/IEC 18040:2019 - Information technology -- Computer graphics, image processing and environmental data representation -- Live actor and entity representation in mixed and augmented reality (MAR)
English language
39 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 18040
First edition
2019-05
Information technology — Computer
graphics, image processing and
environmental data representation —
Live actor and entity representation in
mixed and augmented reality (MAR)
Technologies de l'information — Infographie, traitement de l'image
et représentation des données environnementales — Représentation
d'acteurs et d'entités réels en réalité mixte et augmentée (MAR)
Reference number
ISO/IEC 18040:2019(E)
ISO/IEC 2019
---------------------- Page: 1 ----------------------
ISO/IEC 18040:2019(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2019

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2019 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 18040:2019(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms, definitions and abbreviated terms ................................................................................................................................ 1

3.1 Terms and definitions ....................................................................................................................................................................... 1

3.2 Abbreviated terms ............................................................................................................................................................................... 3

4 Concepts of LAE representation in MAR ...................................................................................................................................... 3

4.1 Overview ...................................................................................................................................................................................................... 3

4.2 Components ............................................................................................................................................................................................... 6

4.2.1 General...................................................................................................................................................................................... 6

4.2.2 LAE capturer and sensor .......................................................................................................................................... 7

4.2.3 LAE recognizer .................................................................................................................................................................. 8

4.2.4 LAE tracker ........................................................................................................................................................................... 8

4.2.5 LAE spatial mapper ....................................................................................................................................................... 8

4.2.6 LAE event mapper .......................................................................................................................................................... 8

4.2.7 Renderer ................................................................................................................................................................................. 8

4.2.8 Display and user interface ....................................................................................................................................... 8

4.2.9 Scene representation ................................................................................................................................................... 8

5 LAE capturer and sensor .............................................................................................................................................................................. 9

5.1 Overview ...................................................................................................................................................................................................... 9

5.2 Computational view............................................................................................................................................................................ 9

5.2.1 General...................................................................................................................................................................................... 9

5.2.2 LAE capturer ........................................................................................................................................................................ 9

5.2.3 LAE sensor .................. .................................................... ....................................................................................................10

5.3 Informational view ...........................................................................................................................................................................12

6 Tracker and spatial mapper for an LAE .....................................................................................................................................13

6.1 Overview ...................................................................................................................................................................................................13

6.2 Computational view.........................................................................................................................................................................14

6.3 Informational view ...........................................................................................................................................................................16

6.4 An example of LAE tracking and spatial mapping in MAR .............................................................................17

7 Recognizer and event mapper for an LAE ...............................................................................................................................17

7.1 Overview ...................................................................................................................................................................................................17

7.2 Recognizer ...............................................................................................................................................................................................17

7.3 Event mapper ........................................................................................................................................................................................19

7.4 Event execution ...................................................................................................................................................................................20

7.5 Examples of LAE recognizing and event mapping in MAR.............................................................................21

8 Scene representation for an LAE ......................................................................................................................................................22

8.1 Overview ...................................................................................................................................................................................................22

8.2 Scene description ...............................................................................................................................................................................23

9 Renderer ....................................................................................................................................................................................................................24

9.1 Overview ...................................................................................................................................................................................................24

9.2 Computational view.........................................................................................................................................................................24

9.3 Information view................................................................................................................................................................................25

10 Display and UI .......................................................................................................................................................................................................25

11 Extensions to virtual actor and entity .........................................................................................................................................25

12 System performance ......................................................................................................................................................................................26

13 Safety .............................................................................................................................................................................................................................27

© ISO/IEC 2019 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 18040:2019(E)

14 Conformance ..........................................................................................................................................................................................................28

Annex A (informative) Use case examples ...................................................................................................................................................31

Bibliography .............................................................................................................................................................................................................................39

iv © ISO/IEC 2019 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 18040:2019(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/patents) or the IEC

list of patent declarations received (see http: //patents .iec .ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso

.org/iso/foreword .html.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 24, Computer graphics, image processing and environmental data representation.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/members .html.
© ISO/IEC 2019 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 18040:2019(E)
Introduction

This document defines the scope and key concepts of a representation model for a live actor and

entity (LAE) to be included in a mixed and augmented reality (MAR) world. The relevant terms and

their definitions, and a generalized system architecture, together serve as a reference model for MAR

applications, components, systems, services and specifications. It defines representing and rendering

an LAE in an MAR scene, and interaction interfaces between an LAE and objects in an MAR scene. It

defines a set of principles, concepts and functionalities for an LAE applicable to the complete range of

current and future MAR standards. This reference model establishes the set of required modules and

their minimum functions, the associated information content, and the information models that shall

be provided and/or be supported by a compliant MAR system. It includes (but is not limited to) the

following content:

— an introduction to the mixed and augmented reality standards domain and concepts;

— a representation model for including an LAE in an MAR scene;
— 3D modelling, rendering and simulation of an LAE in an MAR scene;
— attributes of an LAE in an MAR scene;
— sensing representation of an LAE in an MAR scene;
— representation of the interfaces for controlling an LAE in an MAR scene;
— functionalities and base components for controlling an LAE in an MAR scene;
— interactive interfaces between an LAE and an MAR scene;
— interface with other MAR components;
— relationship to other standards;
— use cases.
The objectives of this document are as follows:
— provide a reference model for LAE representation-based MAR applications;
— manage and control an LAE with its properties in an MAR environment;
— integrate an LAE into a 2D and/or 3D virtual scene in an MAR scene;

— achieve interaction of an LAE with a 2D and/or 3D virtual scene in an MAR scene;

— provide an exchange format necessary for transferring and storing data between LAE-based MAR

applications.
This document has the following document structure:
— Clause 4 describes the concepts of LAE-based systems represented in MAR.

— Clause 5 illustrates how a sensor captures an LAE in a physical world and a virtual world.

— Clause 6 describes mechanisms to track the position of an LAE and specifies the role of a spatial

mapper between physical space and the MAR space.

— Clause 7 describes mechanisms to recognize the behaviour of an LAE and specifies an association

or event between an MAR event of an LAE and the condition specified by the MAR content creator.

— Clause 8 describes a scene, which consists of a virtual scene, sensing data, a spatial scene, events,

targets and so on, for an LAE.
vi © ISO/IEC 2019 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 18040:2019(E)

— Clause 9 describes how the MAR scene system renders the scene, LAE mapping, event and so on for

presentation output on a given display device.

— Clause 10 describes types of displays, including monitors, head mounted displays, projectors, haptic

devices and sound output devices for displaying an LAE in an MAR scene.

— Clause 11 identifies and describes virtual LAE, such as virtual 3D model (avatar) and virtual LAE,

such as real human model in an MAR system.

— Clause 12 makes statements regarding any system performance related issues of an LAE in MAR.

— Clause 13 makes statements regarding any operational safety related issues of an LAE in MAR.

— Clause 14 makes statements regarding any conformance related issues of an LAE in MAR.

— Annex A gives examples of representative LAE representation systems in MAR.
© ISO/IEC 2019 – All rights reserved vii
---------------------- Page: 7 ----------------------
INTERNATIONAL STANDARD ISO/IEC 18040:2019(E)
Information technology — Computer graphics, image
processing and environmental data representation — Live
actor and entity representation in mixed and augmented
reality (MAR)
1 Scope

This document defines a reference model and base components for representing and controlling a

single LAE or multiple LAEs in an MAR scene. It defines concepts, a reference model, system framework,

functions and how to integrate a 2D/3D virtual world and LAEs, and their interfaces, in order to provide

MAR applications with interfaces of LAEs. It also defines an exchange format necessary for transferring

and storing LAE-related data between LAE-based MAR applications.
This document specifies the following functionalities:
a) definitions for an LAE in MAR;
b) representation of an LAE;
c) representation of properties of an LAE;
d) sensing of an LAE in a physical world;
e) integration of an LAE into a 2D/3D virtual scene;
f) interaction between an LAE and objects in a 2D/3D virtual scene;
g) transmission of information related to an LAE in an MAR scene.

This document defines a reference model for LAE representation-based MAR applications to represent

and to exchange data related to LAEs in a 2D/3D virtual scene in an MAR scene. It does not define

specific physical interfaces necessary for manipulating LAEs, that is, it does not define how specific

applications need to implement a specific LAE in an MAR scene, but rather defines common functional

interfaces for representing LAEs that can be used interchangeably between MAR applications.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 18039, Information technology — Computer graphics, image processing and environmental data

representation — Mixed and augmented reality (MAR) reference model
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 18039 and the

following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https: //www .iso .org/obp
© ISO/IEC 2019 – All rights reserved 1
---------------------- Page: 8 ----------------------
ISO/IEC 18040:2019(E)
— IEC Electropedia: available at http: //www .electropedia .org/
3.1.1
augmented object
object with augmentation
3.1.2
geographic coordinate system

coordinate system which is provided by sensor devices for defining a location of LAE (3.1.4)

3.1.3
head mounted display
HMD
device which displays stereo views of virtual reality

Note 1 to entry: It has two small displays with lenses and semi-transparent mirrors which can adapt to the left

and right eyes.
3.1.4
live actor and entity
LAE

representation of a living physical or real object, such as a human being, animal or bird, in the MAR

content or system

Note 1 to entry: A live actor can be animated, moved and interacted with virtual objects in an MAR scene by

capturing gesture from a camera. Entity refers to 3D objects and entities that exist in MAR content.

3.1.5
LAE recognizer

MAR component that recognizes the output from an LAE capturer (3.1.6) and an LAE sensor, then

generates MAR events based on conditions indicated by the content creator
3.1.6
LAE capturer

MAR component that captures an LAE (3.1.4) in a virtual world and a physical world, which includes

depth cameras, general cameras, 360° cameras and so on

Note 1 to entry: LAE’s information can be processed by an LAE recognizer and an LAE tracker to extract

background or skeleton.
3.1.7
LAE tracker

MAR component (hardware and software) that analyses signals from LAE capturers (3.1.6) and sensors

and provides some characteristics of a tracked LAE (3.1.4) (for example position, orientation, amplitude,

profile)
3.1.8
physical camera coordinate system

coordinate system which is provided by a camera for capturing LAE(s) (3.1.4) in physical world

3.1.9
physical coordinate system

coordinate system that enables locating an LAE (3.1.4) and is controlled by a geospatial coordinate

system sensing device
2 © ISO/IEC 2019 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 18040:2019(E)
3.1.10
virtual actor and entity
VAE
virtual reality representation of an LAE (3.1.4)

Note 1 to entry: The virtual actor and entity is obtained by a 3D capturing technique and can be reconstructed,

transmitted or compressed in the MAR scene. A virtual actor and entity can be captured in one place or

transmitted to another place in real time using holography technology.
3.1.11
world coordinate system

universal system in computer graphics that allows model coordinate systems to interact with each other

3.2 Abbreviated terms

For the purposes of this document, the abbreviated terms given in ISO/IEC 18039 and the following apply.

DDR dance dance revolution
EID event identifier
FOV field of view
GNSS global navigation satellite system
LAE-MAR live actor and entity representation in mixed and augmented reality
RGB red, green, blue
SDK software development kit
SID sensor identifier
UI user interface
UTM universal transverse mercator
VR virtual reality
4 Concepts of LAE representation in MAR
4.1 Overview

As illustrated in ISO/IEC 18039, MAR represents a continuum which encompasses all domains or

systems that use a combination of reality (for example live video) and virtuality representations (for

[1][2]

example computer graphic objects or scene) as its main presentation medium . Figure 1 illustrates

the MAR that is defined according to a mixture of reality and virtuality representations excluding pure

real environment and pure virtual environment with viewpoints of an LAE representation. The real

environment refers to the physical world environment where an LAE and physical objects are located.

Virtual environment commonly refers to virtual reality, that is the computer-generated realistic images

and hypothetical world that replicate a real environment. Augmented reality refers to the view of the

real-world environment whose elements include LAE and objects that can be augmented by computer-

generated sensory, and augmented virtuality is the virtual environment within which physical world

elements including LAE can be mapped and interacted. In Figure 1, an LAE wears an HMD device to see

the virtual world and interacts directly with virtual objects.
© ISO/IEC 2019 – All rights reserved 3
---------------------- Page: 10 ----------------------
ISO/IEC 18040:2019(E)
Figure 1 — Mixed and augmented reality (MAR)

This clause describes the concepts of LAE representation in an MAR scene based on the MAR reference

model (MAR-RM) of ISO/IEC 18039, which includes objectives, embedding, interaction and functions of

the system for representing an LAE in an MAR scene. In general, an actor is an individual who portrays

a character in a performance. In this case, an actor represents a human captured by a depth camera or

a general camera, which can then perform actions that are embedded into an MAR scene. A 3D object

that exists in an MAR scene and can interact with a live actor is called an entity. The entity can be

moved or interact with an actor’s motion via an event mapper. An LAE in this document is defined as

a representation of a physical living actor and an object in an MAR content or system. For example,

human beings, birds and animals are all represented as LAEs in an MAR scene.

Figure 2 shows the examples of LAE representation in an MAR scene which consists of 2D virtual world

and 3D virtual world that can be described as the following.

Figure 2 a) shows an LAE integrated into a 2D virtual world that is a real or virtual image. The LAE can

be captured from general camera and/or depth camera sensors. This subfigure shows a real-like action

where a man is captured by cameras in a green screen studio and is integrated as an actor into a 2D

virtual world image of the White House.

Figure 2 b) shows multiple LAEs integrated into a 3D virtual world. This scenario can be applicable in

[3][4]

various situations, such as news studios, education services, virtual surgical operations or games .

It supports an integrative combination application of 3D videoconferencing, reality-like communication

features, presentation/application sharing and 3D model display within a mixed environment.

[5]

Figure 2 c) shows an MAR scene constructed by integrating a 3D virtual world and a live actor . The

live actor interacts with objects in the 3D virtual world by using a joystick or by motion captured by a

depth camera. An HMD device is used to display 360° 3D views, including real time and real-like action,

[6]

in the virtual world . The figure shows a man in the studio wearing an HMD through which he sees the

bow sling training field and handling a joystick. By handling the joystick, it appears that he is handling

the arrow and bow sling. As a result, he can shoot at objects in the virtual world.

[7]

Figure 2 d) shows a virtual actor and entity with representation in the physical world . The virtual

LAE and LAE can communicate and interact with each other, for example to have a natural face-to-face

conversation. When combined with MAR, this technology allows an LAE to see, hear and interact with a

virtual LAE in a 3D virtual space, just like a real presentation in physical space.

Figure 2 e) describes the LAE representation as a bird and a dog from which it can be inferred that LAE

can be animals, birds or humans to be represented in MAR scene.
4 © ISO/IEC 2019 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 18040:2019(E)
a) An LAE integrated into a 2D video virtual b) Two LAEs integrated into a
[8]
world after Chroma-keying 3D virtual world
c) An LAE interacting with a d) Virtual representation of
[5] [7]
virtual object in a 3D virtual world an LAE in an MAR scene
© ISO/IEC 2019 – All rights reserved 5
---------------------- Page: 12 ----------------------
ISO/IEC 18040:2019(E)
e) LAE representation as a bird and a dog
Figure 2 — Examples of LAE representation in an MAR scene

Once an LAE in the physical world is integrated into a 3D virtual world, its location, movements and

interactions should be represented precisely in the 3D virtual world. In an MAR application, an LAE

that needs to be embedded in a 3D virtual world shall be defined and then information, such as the

LAE’s location, actions and sensing data from a handled device, shall be able to be transferred between

the physical world and the 3D virtual world, and between MAR applications. This document aims at

defining how an LAE can be managed and controlled with its properties in a 3D virtual world; how

an LAE can be embedded in a 3D virtual world; how an LAE can interact with virtual cameras, virtual

objects and AR content in a 3D virtual world; and how MAR application data can be exchanged in

heterogeneous computing environments.
4.2 Components
4.2.1 General

An LAE in an MAR scene can be captured from the physical world, then represented in a 3D virtual

world, and can interact with cameras, objects and AR content in the 3D virtual world according to an

input of sensing information.

In order to provide a 3D virtual world with the capability of representing an LAE based on the MAR-

RM, the MAR system requires the following functions:

— sensing of an LAE in a physical world from input devices such as a (depth) camera;

— sensing of information for interaction from input sensors;
— recognizing and tracking an LAE in a physical world;
— recognizing and tracking events made by LAEs in a physical world;
— recognizing and tracking events captured by sensors in a physical world;
— represent
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.