Information technology — Media context and control — Part 1: Architecture

This document specifies the architecture of MPEG-V (media context and control) and its three types of associated use cases: — information adaptation from virtual world to real world; — information adaptation from real world to virtual world; — information exchange between virtual worlds.

Technologies de l'information — Contexte et contrôle des medias — Partie 1: Architecture

General Information

Status
Published
Publication Date
01-Sep-2020
Current Stage
6060 - International Standard published
Start Date
02-Sep-2020
Completion Date
02-Sep-2020
Ref Project

RELATIONS

Buy Standard

Standard
ISO/IEC 23005-1:2020 - Information technology -- Media context and control
English language
40 pages
sale 15% off
Preview
sale 15% off
Preview
Draft
ISO/IEC FDIS 23005-1 - Information technology -- Media context and control
English language
40 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 23005-1
Fourth edition
2020-08
Information technology — Media
context and control —
Part 1:
Architecture
Technologies de l'information — Contexte et contrôle des medias —
Partie 1: Architecture
Reference number
ISO/IEC 23005-1:2020(E)
ISO/IEC 2020
---------------------- Page: 1 ----------------------
ISO/IEC 23005-1:2020(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2020

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2020 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 23005-1:2020(E)
Contents Page

Foreword ..........................................................................................................................................................................................................................................v

Introduction ................................................................................................................................................................................................................................vi

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 MPEG-V system architecture .................................................................................................................................................................... 3

5 Use cases ....................................................................................................................................................................................................................... 5

5.1 General ........................................................................................................................................................................................................... 5

5.2 System architecture for information adaptation from virtual world to real world ..................... 5

5.3 System architecture for information adaptation from real world to virtual world ..................... 6

5.4 System architecture for exchanges between virtual worlds ............................................................................ 7

6 Instantiations ........................................................................................................................................................................................................... 8

6.1 Instantiation A: representation of sensory effects (RoSE) ................................................................................ 8

6.1.1 System architecture for representation of sensory effects .................. ........................................ 8

6.1.2 Instantiation A.1: multi-sensorial effects ................................................................................................... 9

6.1.3 Instantiation A.2: motion effects ........................................................................................................................ 9

6.1.4 Instantiation A.3: arrayed light effects .......................................................................................................10

6.2 Instantiation B: natural user interaction with virtual world .......................................................................11

6.2.1 System architecture for natural user interaction with virtual world ..............................11

6.2.2 Examples of sensors .................. ......................................................................................................................... ........11

6.2.3 Instantiation B.1: Full motion control and navigation of avatar/object

with multi-input sources .......................................................................................................................................12

6.2.4 Instantiation B.2: serious gaming for ambient assisted living ..............................................12

6.2.5 Instantiation B.3: gesture recognition using multipoint interaction devices ...........13

6.2.6 Instantiation B.4: avatar facial expression retargeting using smart camera ............13

6.2.7 Instantiation B.5: motion tracking and facial animation with multimodal

interaction ..........................................................................................................................................................................14

6.2.8 Instantiation B.6: serious gaming and training with multimodal interaction .........14

6.2.9 Instantiation B.7: virtual museum guide with embodied conversational agents .15

6.3 Instantiation C: traveling and navigating real and virtual worlds ...........................................................15

6.3.1 System architecture for traveling and navigating real and virtual worlds .................15

6.3.2 Examples of sensors and path finding mechanisms ......................................................................15

6.3.3 Instantiation C.1: virtual travel ........................................................................................................................16

6.3.4 Instantiation C.2: virtual traces of real places .....................................................................................16

6.3.5 Instantiation C.3: virtual tour guides ..........................................................................................................17

6.3.6 Instantiation C.4: unmanned aerial vehicle scenario ....................................................................18

6.4 Instantiation D: interoperable virtual worlds ...........................................................................................................18

6.4.1 System architecture for interoperable virtual worlds ..................................................................18

6.4.2 Instantiation D.1: avatar appearance ..........................................................................................................18

6.4.3 Instantiation D.2: virtual objects ....................................................................................................................18

6.5 Instantiation E: social presence, group decision making and collaboration within

virtual worlds ........................................................................................................................................................................................19

6.5.1 System architecture ....................................................................................................................................................19

6.5.2 Instantiation E.1: social presence ..................................................................................................................19

6.5.3 Instantiation E.2: group decision making in the context of spatial planning ...........20

6.5.4 Instantiation E.3: consumer collaboration in product design processes

along the supply chain .............................................................................................................................................21

6.6 Instantiation F: interactive haptic sensible media ................................................................................................21

6.6.1 System architecture for interactive haptic sensible media ......................................................21

6.6.2 Instantiation F.1: Internet haptic service — YouTube, online chatting .........................22

6.6.3 Instantiation F.2: next-generation classroom — sensation book .......................................23

6.6.4 Instantiation F.3: immersive broadcasting — home shopping, fishing channels .23

© ISO/IEC 2020 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 23005-1:2020(E)
® ®
6.6.5 Instantiation F.4: entertainment — game (Second Life , StarCraft ),

movie theatre ...................................................................................................................................................................23

6.6.6 Instantiation F.5: virtual simulation for training — military task, medical

simulations ........................................................................................................................................................................24

6.7 Instantiation G: bio-sensed information in the virtual world .....................................................................24

6.7.1 System architecture for bio-sensed information in the virtual world ............................24

6.7.2 Instantiation G.1: interactive games sensitive to user’s conditions ..................................24

6.7.3 Instantiation G.2: virtual hospital and health monitoring ........................................................25

6.7.4 Instantiation G.3: mental health for lifestyle management......................................................25

6.7.5 Instantiation G.4: food intake for lifestyle management ............................................................25

6.7.6 Instantiation G.5: cardiovascular rehabilitation for health management ...................26

6.7.7 Instantiation G.6: glucose level/diabetes management for health management .26

6.8 Instantiation H: environmental monitoring with sensors..............................................................................27

6.8.1 General...................................................................................................................................................................................27

6.8.2 System architecture for environmental monitoring ......................................................................27

6.8.3 Instantiation H.1: environmental monitoring system ..................................................................28

6.9 Instantiation I: virtual world interfacing with TV platforms .......................................................................28

6.10 Instantiation J: seamless integration between real and virtual worlds ..............................................29

6.10.1 System architecture for seamless integration between real and virtual worlds ...29

6.10.2 Instantiation J.1: seamless interaction between real and virtual worlds

with integrating virtual and real sensors and actuators ............................................................29

6.11 Instantiation K: hybrid communication .........................................................................................................................31

6.12 Instantiation L: makeup avatar ..............................................................................................................................................33

6.12.1 Spectrum data acquisition ....................................................................................................................................33

6.12.2 Transformation model generation ................................................................................................................35

6.13 Instantiation M: usage scenario for automobile sensors.................................................................................35

6.13.1 Helping auto maintenance/regular inspection ..................................................................................35

6.13.2 Monitoring for eco-friendly driving .............................................................................................................36

6.14 Instantiation N: usage scenario for 3D printing ......................................................................................................37

6.15 Instantiation O: olfactory information in virtual world....................................................................................38

6.15.1 System architecture for olfactory information in virtual world ..........................................38

6.15.2 Instantiation O.1: olfactory signature(fingerprint) with e-nose .........................................38

6.15.3 Instantiation O.2: 4D film with scent effect ...........................................................................................38

6.15.4 Instantiation O.3: healing minds of combat veterans ...................................................................38

6.15.5 Instantiation O.4: advertisement with olfactory information ................................................38

6.15.6 Instantiation O.5: harmful odour monitoring ......................................................................................38

6.16 Instantiation P: virtual panoramic vision in car......................................................................................................39

6.16.1 General...................................................................................................................................................................................39

6.16.2 Instantiation O.6.1: virtual panoramic IVI (in-vehicle information system).............39

6.16.3 Instantiation O.6.2: virtual panoramic black box .............................................................................39

6.17 Instantiation Q: adaptive sound handling ....................................................................................................................39

Bibliography .............................................................................................................................................................................................................................40

iv © ISO/IEC 2020 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 23005-1:2020(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work. In the field of information technology, ISO and IEC have established a joint technical committee,

ISO/IEC JTC 1.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/

is o/ f or ewor d . ht m l .

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

This fourth edition cancels and replaces the third edition (ISO/IEC 23005-1:2016), which has been

technically revised.
The main changes compared to the previous edition are as follows:
— added a new use case for 3D printing;
— added six new use cases for olfactory information in virtual world;
— added two new use cases for virtual panoramic vision in car;
— added a new use case for adaptive sound handling.
A list of all parts in the ISO/IEC 23005 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html.
© ISO/IEC 2020 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 23005-1:2020(E)
Introduction

The ISO/IEC 23005 series provides an architecture and specifies information representation of data

flowing in and out of the real world and virtual worlds.

The data for the real world are communicated through sensors and actuators. The data for virtual

worlds consist of properties of virtual objects and multi-sensorial data embedded in audio-visual

content. The ISO/IEC 23005 series specifies data formats for sensors, actuators, virtual objects and

audio-visual content.

Data captured from the real world can need to be adapted for use in a virtual world and data from

virtual worlds can also need to be adapted for use in the real world. This document does not specify

how the adaptation is carried out but only specifies the interfaces.

Data for sensors are sensor capabilities, sensed data and sensor adaptation preferences.

Data for actuators are sensory device capabilities, sensory device commands and sensory effect

preferences.

Data for virtual objects are characteristics of avatars and virtual world objects.

Data for audio-visual content are sensory effects.

This document contains the tools for exchanging information for interaction devices. To be specific,

it specifies command formats for controlling actuators (e.g. actuator commands for sensory devices)

and data formats for receiving information from sensors (e.g. sensed information from sensors) as

illustrated as the yellow boxes in Figure 1. It also specifies some examples. The adaptation engine is not

within the scope.
vi © ISO/IEC 2020 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 23005-1:2020(E)
Figure 1 — Scope of the data formats for interaction devices

When this document is used, the adaptation engine (RV or VR engine), which is not within the scope

of standardization, performs bi-directional communications using data formats specified in this

document. The adaptation engine can also utilize other tools defined in ISO/IEC 23005-2, which are

user's sensory preferences (USP), sensory device capabilities (SDC), sensor capabilities (SC) and sensor

adaptation preferences (SAP) for fine controlling devices in both real and virtual worlds.

The International Organization for Standardization (ISO) and International Electrotechnical

Commission (IEC) draw attention to the fact that it is claimed that compliance with this document may

involve the use of a patent.

ISO and IEC take no position concerning the evidence, validity and scope of this patent right.

The holder of this patent right has assured ISO and IEC that he/she is willing to negotiate licences under

reasonable and non-discriminatory terms and conditions with applicants throughout the world. In this

respect, the statement of the holder of this patent right is registered with ISO and IEC. Information may

be obtained from the patent database available at www .iso .org/ patents.

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights other than those in the patent database. ISO and IEC shall not be held responsible for

identifying any or all such patent rights.
© ISO/IEC 2020 – All rights reserved vii
---------------------- Page: 7 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23005-1:2020(E)
Information technology — Media context and control —
Part 1:
Architecture
1 Scope

This document specifies the architecture of MPEG-V (media context and control) and its three types of

associated use cases:
— information adaptation from virtual world to real world;
— information adaptation from real world to virtual world;
— information exchange between virtual worlds.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
device command
description of controlling actuators used to generate sensory effects (3.9)
3.2
R→V adaptation
procedure that:

— processes the sensed information (3.3) from the real world in order to be consumed within the

virtual world’s, context;

— takes the sensed information with/without the sensor capabilities from sensors (3.4), the sensor

adaptation preferences (3.5) from users (3.12) and/or the virtual world object characteristics from a

virtual world;

— controls the virtual world (3.13) object characteristics or adapts the sensed information by adapting

the sensed information based on the sensor capabilities and/or the sensor adaptation preferences

3.3
sensed information
information acquired by sensors (3.4)
© ISO/IEC 2020 – All rights reserved 1
---------------------- Page: 8 ----------------------
ISO/IEC 23005-1:2020(E)
3.4
sensor
device by which user (3.12) input or environmental information can be gathered
EXAMPLE Temperature sensor, distance sensor, motion sensor, etc.
3.5
sensor adaptation preferences

description schemes and descriptors to represent (user’s) preferences with respect to adapting sensed

information (3.3)
3.6
sensor capability

representation of the characteristics of sensors in terms of the capability of the given sensor (3.4) such

as accuracy, or sensing range
3.7
sensory device
consumer device by which the corresponding sensory effect (3.9) can be made

Note 1 to entry: Real-world devices can contain any combination of sensors (3.4) and actuators in one device.

3.8
sensory device capability

representation of the characteristics of actuators used to generate sensory effects (3.9) in terms of the

capability of the given actuator
3.9
sensory effect
effect to augment perception by stimulating human senses in a particular scene

EXAMPLE Scent, wind, light, haptic [kinesthetic-force, stiffness, weight, friction, texture, widget (button,

slider, joystick, etc.), tactile: air-jet, suction pressure, thermal, current, vibration, etc. Note that combinations of

tactile display can also provide directional, shape information.]
3.10
sensory effect metadata

metadata that defines the description schemes and descriptors to represent sensory effects (3.9)

3.11
user’s sensory preferences

description schemes and descriptors to represent (user’s) preferences with respect to rendering of

sensory effect (3.9)
3.12
user
end user of the system
3.13
virtual world
digital content, real time or non-real time, of various nature

EXAMPLE On-line virtual world, simulation environment, multi-user game, broadcast multimedia

production, peer-to-peer multimedia production or packaged content like a DVD or game.

3.14
V→R adaptation
procedure that:

— processes the sensory effects (3.9) from the virtual world (3.13) in order to be consumed within the

real world’s context;
2 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 23005-1:2020(E)

— takes sensory effect metadata (3.10) from a virtual world, sensory device (actuator) capabilities

from the sensory devices (actuators), the user’s sensory preferences (3.11) from users (3.12) and/or

the sensed information (3.3) as well as the sensor capabilities from sensors (3.4) as inputs;

— generates the device commands (3.1) by adapting the sensory effects based on the sensed information,

the capabilities and/or the preferences
3.15
VW object characteristics

description schemes and descriptors to represent and describe virtual world objects (from the real

world into the virtual world and vice versa)
4 MPEG-V system architecture

A strong connection (defined by an architecture that provides interoperability through standardization)

between the virtual and the real world is needed to reach simultaneous reactions in both worlds to

changes in the environment and human behaviour. Efficient, effective, intuitive and entertaining

interfaces between users and virtual worlds are of crucial importance for their wide acceptance and

use. To improve the process of creating virtual worlds, a better design methodology and better tools are

indispensable. For fast adoption of virtual world technologies, a better understanding of their internal

economics, rules and regulations is needed. The overall system architecture for the MPEG-V framework

is depicted in Figure 2.
© ISO/IEC 2020 – All rights reserved 3
---------------------- Page: 10 ----------------------
ISO/IEC 23005-1:2020(E)
Figure 2 — System architecture of the MPEG-V framework

The MPEG-V system architecture can be used to serve three different media exchanges. There are

two types of media exchanges occurring between real world and virtual world, i.e. the information

exchange from real world to virtual world and the information exchange from virtual world to real

world. An additional type of media exchanges is the information exchange between virtual worlds. The

three media exchanges are defined as use cases in Clause 5.

Sensory effect metadata, sensory device capability, user’s sensory preferences, device commands,

sensed information, sensor device capability, sensor adaptation preferences and virtual world

object characteristics are within the scope of standardization and are specified in other parts of the

ISO/IEC 23005 series.

On the other side, the V→R adaptation engine, R→V adaptation engine, virtual world as well as devices

(sensors and sensory devices) are left open for industry competition.

Metadata is specified in other parts of the ISO/IEC 23005 series. Sensor device capability, sensory

device capability, sensor adaptation preferences and user’s sensory preferences are specified in

ISO/IEC 23005-2. Sensory effect metadata is specified in ISO/IEC 23005-3. Virtual world object

characteristics is specified in ISO/IEC 23005-4. Device commands and sensed information are specified

in ISO/IEC 23005-5.
4 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 23005-1:2020(E)
5 Use cases
5.1 General

The three types of media exchanges require information adaptations for a targeting world to adapt

information based on capabilities and preferences: information adaptation from virtual world to real

world, information adaptation from real world to virtual world and information adaptation between

virtual worlds.

5.2 System architecture for information adaptation from virtual world to real world

The system architecture for the information adaptation from virtual world to real world is depicted in

Figure 3. It represents V→R adaptation comprising sensory effect metadata, VW object characteristics,

sensory device capability (actuator capability), device commands, user’s sensory preferences and a

V→R adaptation engine which generates output data based on its input data.

Figure 3 — Example of system architecture for information adaptation from virtual world to

real world

A virtual world within the framework is referred to as an entity that acts as the source of the sensory

effect metadata and VW object characteristics such as a broadcaster, content creator/distributor, or

even a service provider. The V→R adaptation engine is an entity that takes the sensory effect metadata,

the sensory device (actuator) capability and the user’s sensory preferences as inputs and generates

the device commands based on those in order to control the consumer devices enabling a worthwhile,

informative experience to the user.

Real-world devices (sensory devices) are entities that act as the sink of the device commands and as

the source of sensory device (actuator) capability. Additionally, entities that provide user’s sensory

preferences towards the RoSE engine are also collectively referred to as real-world devices. Note that

© ISO/IEC 2020 – All rights reserved 5
---------------------- Page: 12 ----------------------
ISO/IEC 23005-1:2020(E)

sensory devices (actuators) are a sub-set of real-world devices including fans, lights, scent devices,

human input devices, such as a TV set with a remote control (e.g. for preferences).

The actual sensory effect metadata provides means for representing so-called sensory effec

...

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 23005-1
ISO/IEC JTC 1/SC 29 Secretariat: JISC
Voting begins on: Voting terminates on:
2018-01-16 2018-04-10
Information technology — Media context and control —
Part 1:
Architecture
Technologies de l'information — Contexte et contrôle des medias —
Partie 1: Architecture
ICS: 35.040.40
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
This document is circulated as received from the committee secretariat.
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 23005-1:2018(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. ISO/IEC 2018
---------------------- Page: 1 ----------------------
ISO/IEC DIS 23005-1:2018(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2018, Published in Switzerland

All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form

or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior

written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of

the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2018 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC DIS 23005-1:2018(E)
Contents Page
Sommaire

1 Scope ..................................................................................................................................................... 1

2 Terms and definitions .......................................................................................................................... 1

• Device Command ................................................................................................................................. 1

• R è V Adaptation ................................................................................................................................ 1

• Sensed Information .............................................................................................................................. 1

• Sensor ................................................................................................................................................... 1

• Sensor Adaptation Preferences .......................................................................................................... 1

• Sensor Capability ................................................................................................................................. 1

• Sensory Device ..................................................................................................................................... 2

• Sensory Device Capability .................................................................................................................. 2

• Sensory Effects .................................................................................................................................... 2

• Sensory Effect Metadata ...................................................................................................................... 2

• User’s Sensory Preferences ................................................................................................................ 2

• User ....................................................................................................................................................... 2

• Virtual World ......................................................................................................................................... 2

• V è R Adaptation ................................................................................................................................ 2

• VW Object Characteristics .................................................................................................................. 2

3 MPEG-V System Architecture ............................................................................................................. 2

4 Use cases .............................................................................................................................................. 5

• Information adaptation from virtual world to real world ................................................................... 5

• System Architecture for information adaptation from virtual world to real world ......................... 5

• Information adaptation from real world to virtual world ................................................................... 6

• System Architecture for information adaptation from real world to virtual world ......................... 6

• Information exchange between virtual worlds .................................................................................. 7

• System Architecture for exchanges between virtual worlds ........................................................... 7

5 Instantiations ........................................................................................................................................ 8

• Instantiation A: Representation of Sensory Effects (RoSE) ............................................................ 8

• System Architecture for Representation of Sensory Effects ........................................................... 8

• Instantiation A.1: Multi-sensorial Effects ........................................................................................... 9

• Instantiation A.2: Motion effects ....................................................................................................... 10

• Instantiation A.3: Arrayed light effects ............................................................................................ 12

• Instantiation B: Natural user interaction with virtual world ........................................................... 13

• System Architecture for Natural user interaction with virtual world ............................................. 13

• Instantiation B.1: Full motion control and navigation of avatar/object with multi-input

sources ................................................................................................................................................ 15

• Instantiation B.2: Serious gaming for ambient assisted living ...................................................... 15

• Instantiation B.3: Gesture recognition using multipoint interaction devices .............................. 16

• Instantiation B.4: Avatar facial expression retargeting using smart camera ............................... 17

• Instantiation B.5: Motion tracking and facial animation with multimodal interaction ................. 18

• Instantiation B.6: Serious gaming and training with multimodal interaction ............................... 19

• Instantiation B.7: Virtual museum guide with embodied conversational agents ........................ 20

• Instantiation C: Traveling and navigating real and virtual worlds ................................................. 20

• System Architecture for traveling and navigating real and virtual worlds ................................... 20

• Instantiation C.1: Virtual travel ......................................................................................................... 22

© ISO/IEC 2018 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC DIS 23005-1:2018(E)

• Instantiation C.2: Virtual traces of real places ................................................................................. 22

• Instantiation C.3: Virtual tour guides ................................................................................................ 23

• Instantiation C.4: Unmanned aerial vehicle scenario ..................................................................... 25

• Instantiation D: Interoperable virtual worlds ................................................................................... 26

• System Architecture for interoperable virtual worlds ..................................................................... 26

• Instantiation D.1: Avatar appearance ............................................................................................... 26

• Instantiation D.2: Virtual objects ....................................................................................................... 26

• Instantiation E: Social presence, group decision-making and collaboration within virtual

worlds .................................................................................................................................................. 28

• System architecture ........................................................................................................................... 28

• Instantiation E.1: Social presence .................................................................................................... 28

• Instantiation E.2: Group decision-making in the context of spatial planning .............................. 29

• Instantiation E.3: Consumer collaboration in product design processes along the supply

chain .................................................................................................................................................... 30

• Instantiation F: Interactive haptic sensible media .......................................................................... 32

• System architecture for interactive haptic sensible media ............................................................ 32

• Instantiation F.1: Internet haptic service - YouTube, online chatting ........................................... 32

• Instantiation F.2: Next generation classroom – sensation book ................................................... 33

• Instantiation F.3: Immersive broadcasting – home shopping, fishing channels ......................... 34

• Instantiation F.4: Entertainment – game (Second Life, Star Craft), movie theater ....................... 34

• Instantiation F.5: Virtual simulation for training – military task, medical simulations ................ 35

• Instantiation G: Bio-sensed information in virtual world ............................................................... 35

• System architecture for bio-sensed information in virtual world .................................................. 35

• Instantiation G.1: Interactive games sensitive to user’s conditions ............................................. 36

• Instantiation G.2: Virtual hospital and health monitoring .............................................................. 36

• Instantiation G.3: Mental health for lifestyle management ............................................................. 37

• Instantiation G.4: Food intake for lifestyle management ................................................................ 38

• Instantiation G.5: Cardiovascular rehabilitation for health management ..................................... 39

• Instantiation G.6: Glucose level / diabetes management for health management ....................... 40

• Instantiation H: Environmental monitoring with sensors ............................................................... 40

• System architecture for environmental monitoring ........................................................................ 40

• Instantiation H.1: Environmental monitoring system ..................................................................... 41

• Instantiation I: Virtual world interfacing with TV platforms ............................................................ 42

• System architecture for virtual world interfacing with TV platform .............................................. 42

• Instantiation I.1: The TV platform as a virtual worlds I/O device ................................................... 43

• Instantiation J: Seamless integration between real and virtual worlds ........................................ 44

• System architecture for seamless integration between real and virtual worlds .......................... 44

• Instantiation J.1: Seamless interaction between real and virtual worlds with integrating

virtual and real sensors and actuators ............................................................................................. 44

• Instantiation K: Hybrid communication ........................................................................................... 46

• Instantiation L: Makeup Avatar ......................................................................................................... 49

• Spectrum data acquisition ................................................................................................................. 49

• Spectrum data combination in a virtual world ................................................................................. 50

• Cosmetic color spectrum metamerism ............................................................................................ 51

• Color reproduction process for a virtual makeup avatar ............................................................... 51

• Transformation model generation .................................................................................................... 52

• Makeup simulation usage example .................................................................................................. 53

• Instantiation M: Usage Scenario for automobile sensors .............................................................. 55

• Helping auto maintenance/regular inspection ................................................................................. 55

• Monitoring for Eco-friendly driving .................................................................................................. 55

• Instantiation N: Usage Scenario for 3D Printing ............................................................................. 56

• Instantiation O: Olfactory information in virtual world ................................................................... 57

• System architecture for olfactory information in virtual world ...................................................... 57

• Instantiation O.1: Olfactory signature(fingerprint) with E-Nose .................................................... 58

• Instantiation O.2: 4D film with scent effect ...................................................................................... 58

• Instantiation O.3: Healing minds of combat veterans ..................................................................... 59

© ISO/IEC 2018 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC DIS 23005-1:2018(E)

• Instantiation O.4: An advertisement with olfactory information .................................................... 59

• Instantiation O.5: Harmful odor monitoring ..................................................................................... 60

• Instantiation O.6: Virtual Panoramic Vision in Car ......................................................................... 60

These scenarios assume that the automobile of the real world contains additional sensors –

RADAR detectors and multiple cameras – outside of cars. In other words, the cars

supporting the virtual panoramic vision equip RADAR detectors and cameras in front and

back of the cars. RADAR detector is an object-detection system that uses radio waves to

determine relative speed, angle of arrival, and distance. The outside cameras of cars

construct a top view images which allow 360⁰ overhead view of a vehicle. Hence the

information acquired from the real-world can be used for safety driving or the

reconstruction of car accidents in virtual world. ............................................................................ 60

• Instantiation O.6.1: Virtual panoramic IVI (In-Vehicle Information system) .................................. 61

• Instantiation P: Adaptive Sound Handling ....................................................................................... 62

Bibliography .................................................................................................................................................... 64

© ISO/IEC 2018 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC DIS 23005-1:2018(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are members of

ISO or IEC participate in the development of International Standards through technical committees

established by the respective organization to deal with particular fields of technical activity. ISO and IEC

technical committees collaborate in fields of mutual interest. Other international organizations, governmental

and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information

technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.

International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.

The main task of the joint technical committee is to prepare International Standards. Draft International

Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as

an International Standard requires approval by at least 75 % of the national bodies casting a vote.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent

rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

ISO/IEC 23005-1 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

This second/third/... edition cancels and replaces the first/second/... edition (), [clause(s) / subclause(s) /

table(s) / figure(s) / annex(es)] of which [has / have] been technically revised.

ISO/IEC 23005 consists of the following parts, under the general title Information technology — Media context

and control:
⎯ Part 1: Architecture
⎯ Part [n]:
⎯ Part [n+1]:
- Part 1: Architecture
- Part 2: Control information
- Part 3: Sensory information
- Part 4: Virtual world object characteristics
- Part 5: Data formats for interaction devices
- Part 6: Common types and tools
- Part 7: Conformance and reference software
© ISO/IEC 2018 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC DIS 23005-1:2018(E)
Introduction

The usage of multimedia content is becoming omnipresent in everyday life, in terms of both consumption and

production. On the one hand, professional content is provided to the end user in high-definition quality,

streamed over heterogeneous networks, and consumed on a variety of different devices. On the other hand,

user-generated content overwhelms the Internet with multimedia assets being uploaded to a wide range of

available Web sites. That is, the transparent access to multimedia content, also referred to as Universal

Multimedia Access (UMA), seems to be technically feasible. However, UMA mainly focuses on the end-user

devices and network connectivity issues, but it is the user who ultimately consumes the content. Hence, the

concept of UMA has been extended to take the user into account, which is generally referred to as Universal

Multimedia Experience (UME).

However, the consumption of multimedia assets can also stimulate senses other than vision or audition, e.g.,

olfaction, mechanoreception, equilibrioception, or thermoception. That is, in addition to the audio-visual

content of, for example, a movie, other senses shall also be stimulated giving the user the sensation of being

part of the particular media which shall result in a worthwhile, informative user experience.

This motivates the annotation of the media resources with metadata as defined in this part of ISO/IEC 23005

that steers appropriate devices capable of stimulating these other senses.

ISO/IEC 23005 (MPEG-V) provides an architecture and specifies associated information representations to

enable the interoperability between virtual worlds, for example, digital content provider of a virtual world,

(serious) gaming, simulation, DVD, and with the real world, for example, sensors, actuators, vision and

rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking,

insurance, travel, real estate, rights management and many others.

Virtual worlds (often referred to as 3D3C for 3D visualization & navigation and the 3C's of community,

creation and commerce) integrate existing and emerging (media) technologies (e.g. instant messaging, video,

3D, VR, AI, chat, voice, etc.) that allow for the support of existing and the development of new kinds of social

networks. The emergence of virtual worlds as platforms for social networking is recognized by businesses as

an important issue for at least two reasons:

a) it offers the power to reshape the way companies interact with their environments (markets,

customers, suppliers, creators, stakeholders, etc.) in a fashion comparable to the Internet;

b) it allows for the development of new (breakthrough) business models, services, applications and

devices.

Each virtual world however has a different culture and audience making use of these specific worlds for a

variety of reasons. These differences in existing metaverses permit users to have unique experiences.

Resistance to real-world commercial encroachment still exists in many virtual worlds where users primarily

seek an escape from real life. Hence, marketers should get to know a virtual world beforehand and the rules

that govern each individual universe.

Although realistic experiences have been achieved via devices such as 3-D audio/visual devices, it is hard to

realize sensory effects only with presentation of audiovisual contents. The addition of sensory effects leads to

even more realistic experiences in the consumption of audiovisual contents. This will lead to the application of

new media for enhanced experiences of users in a more realistic sense.

Such new media will benefit from the standardization of a control and sensory information which can include

sensory effect metadata, sensory device (actuator) capabilities/commands, user’s sensory preferences, and

1) Some examples of virtual worlds are: Second Life (http://secondlife.com/), IMVU (http://www.imvu.com/) and Entropia

Universe (http://www.entropiauniverse.com/).
© ISO/IEC 2018 – All rights reserved vii
---------------------- Page: 7 ----------------------
ISO/IEC DIS 23005-1:2018(E)

various delivery formats. The MPEG-V architecture can be applicable for various business models for which

audiovisual contents can be associated with sensory effects that need to be rendered on appropriate sensory

devices (actuators).

Multi-user online virtual worlds, sometimes called Networked Virtual Environments (NVEs) or massively-

multiplayer online games (MMOGs), have reached mainstream popularity. Although most publications tend to

focus on well-known virtual worlds like World of Warcraft, Second Life, and Lineage, there are hundreds of

popular virtual worlds in active use worldwide, most of which are not known to the general public. These can

be quite different from the above-mentioned titles. To understand current trends and developments, it is useful

to keep in mind that there is large variety in virtual worlds and that they are not all variations on Second Life.

The concept of online virtual worlds started in the late 70s with the creation of the text-based Dungeons &

Dragons world MUD. In the eighties, larger-scale graphical virtual worlds followed, and in the late nineties the

first 3D virtual worlds appeared. Many virtual worlds are not considered games (MMOGs) since there is no

clear objective and/or there are no points to score or levels to achieve. In this report we will use “virtual

worlds” as an umbrella term that includes all possible varieties. See the literature for further discussion of the

distinction between gaming/non-gaming worlds. Often, a virtual world which is not considered to be an MMOG

does contain a wide selection of mini-games or quests, in some way embedded into the world. In this manner

a virtual world acts like a combined graphical portal offering games, commerce, social interactions and other

forms of entertainment. Another way to see the difference: games contain mostly pre-authored stories; in

virtual worlds the users more or less create the stories themselves. The current trend in virtual worlds is to

provide a mix of pre-authored and user-generated stories and content, leading to user-modified content.

Current virtual worlds are graphical and rendered in 2D, 2.5 D (isometric view) or 3D, depending on the

intended effect and technical capabilities of the platform: web-browser, gaming PC, average PC, game

console, mobile phone, and so on.

“Would it not be great if the real world economy could be boosted by the exponential growing economy of the

virtual worlds by connecting the virtual - and real world”; in 2007 the Virtual Economy in Second Life alone

was around 400 MEuro, a factor nine growth from 2006. The connected devices and services in the real world

can represent an economy of a multiple of this virtual world economy.

Virtual worlds have entered our lives, our communication patterns, our culture, and our entertainment never to

leave again. It's not only the teenager active in Second Life and World of Warcraft, the average age of a

gamer is 35 years by now, and it increases every year. This does not even include role-play in the

professional context, also known as serious gaming, inevitable when learning practical skills. Virtual worlds

are in use for entertainment, education, training, obtaining information, social interaction, work, virtual tourism,

reliving the past and forms of art. They augment and interact with our real world and form an important part of

people's lives. Many virtual worlds already exist as games, training systems, social networks and virtual cities

and world models. Virtual worlds will change every aspect of our lives: the way we work, interact, play, travel

and learn. Games will be everywhere and their societal need is very big and will lead to many new products

and require many companies.

Technology improvement, both in hardware and software, forms the basis of this. It is envisaged that the most

important developments will occur in the areas of display technology, graphics, animation, (physical)

simulation, behavior and artificial intelligence, loosely distributed systems and network technology.

The figures in this part of ISO/IEC 23005 have been reproduced here with the permission of Samsung, Sharp

Electronics, ETRI, University of Klagenfurt, Institute of Science and Technology, Myongji University, Institut

national des télécommunications and the partners of the ITEA2 project Metaverse1: Philips, Forthnet S.A.,

Alcatel-Lucent Bell N.V., Innovalia, Alcatel-Lucent France, Technicolor, Orange Labs, DevLab, CBT, Nextel,

Carsa, Avantalia, Ceesa, Virtualware, I&IMS, VicomTECH, E-PYME, CIC Tour Gune, Artefacto, Metaverse

Labs, Technical University Eindhoven, Utrecht University, University of Twente, Stg. EPN, VU Economics &

BA, VU CAMeRA, Ellinogermaniki Agogi, IBBT-SMIT, UPF-MTG, CEA List and Loria/Inria Lorraine.

© ISO/IEC 2018 – All rights reserved
viii
---------------------- Page: 8 ----------------------
DRAFT INTERNATIONAL STANDARD ISO/IEC DIS 23005-1:2018(E)
Information technology — Media context and control —
Part 1:
Architecture
1 Scope

This part of ISO/IEC 23005 specifies the architecture of MPEG-V (media context and control), its three

associated use cases of information adaptation from virtual world to real world, information adaptation from

real world to virtual world, and Information exchange between virtual worlds.
2 Terms and definitions
• Device Command
description of controlling actuators used to generate Sensory Effects.
• R è V Adaptation

procedure that processes the Sensed Information from the real world in order to be consumed within the

virtual world’s context; takes the Sensed Information with/without the Sensor Capabilities from Sensors, the

Sensor Adaptation Preferences from Users, and/or the Virtual World Object Characteristics from a Virtual

world; controls the Virtual World Object Characteristics or adapts the Sensed Information by adapting the

Sensed Information based on the Sensor Capabilities and/or the Sensor Adaptation Preferences.

• Sensed Information
information acquired by sensors.
• Sensor
device by which user input or environmental information can be gathered
EXAMPLES Temperature sensor, distance sensor, motion sensor, etc.
• Sensor Adaptation Preferences
description schemes and descriptors to represent (user’
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.