ISO/IEC 19086-2:2018

Cloud computing — Service level agreement (SLA) framework — Part 2: Metric model

This document establishes common terminology, defines a model for specifying metrics for cloud SLAs, and includes applications of the model with examples. This document establishes a common terminology and approach for specifying metrics. This document is for the benefit of and use for both cloud service providers (CSPs) and cloud service customers (CSCs). This document is intended to complement ISO/IEC 19086-1, ISO/IEC 19086-3 and ISO/IEC 19086-4. This document does not mandate the use of a specific set of metrics for cloud SLAs.

Informatique en nuage — Cadre de travail de l'accord du niveau de service — Partie 2: Modèle métrique

General Information

• Status: Published
• Publication Date: 11-Dec-2018
• ICS: 35.020 - Information technology (IT) in general
• ICS: 35.210 - Cloud computing
• Technical Committee: ISO/IEC JTC 1/SC 38 - Cloud computing and distributed platforms
• Drafting Committee: ISO/IEC JTC 1/SC 38/WG 3 - Cloud Computing Fundamentals (CCF)
• Current Stage: 6060 - International Standard published
• Start Date: 12-Dec-2018
• Completion Date: 12-Dec-2018

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 19086-2
First edition
2018-12
Cloud computing — Service level
agreement (SLA) framework —
Part 2:
Metric model
Informatique en nuage — Cadre de travail de l'accord du niveau de
service —
Partie 2: Modèle métrique
Reference number
ISO/IEC 19086-2:2018(E)
ISO/IEC 2018
---------------------- Page: 1 ----------------------
ISO/IEC 19086-2:2018(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2018

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Fax: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2018 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 19086-2:2018(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Symbols and abbreviated terms ........................................................................................................................................................... 2

5 Conformance ............................................................................................................................................................................................................. 3

6 Metrics overview .................................................................................................................................................................................................. 3

6.1 General ........................................................................................................................................................................................................... 3

6.2 Background ................................................................................................................................................................................................ 3

6.2.1 Choosing a cloud service ........................................................................................................................................... 3

6.2.2 Convert requirements to agreement .............................................................................................................. 4

6.2.3 Ensure the agreement is being met ................................................................................................................. 5

6.3 Metrics ............................................................................................................................................................................................................ 5

6.4 Cloud service metrics (CSMs) .................................................................................................................................................... 5

6.4.1 Major stakeholders ........................................................................................................................................................ 6

6.4.2 CSM usage categories ................................................................................................................................................... 7

7 Metric model overview ...............................................................................................................................................................................10

7.1 General ........................................................................................................................................................................................................10

7.2 Basic concepts ......................................................................................................................................................................................11

7.2.1 Introduction ......................................................................................................................................................................11

7.2.2 Cloud service level objectives and cloud service qualitative objectives .......................11

7.2.3 Metric data format .......................................................................................................................................................12

8 Metric model ..........................................................................................................................................................................................................12

8.1 Metric model development .......................................................................................................................................................12

8.1.1 Metric model specification ...................................................................................................................................13

8.1.2 Use of UML class diagrams and textual descriptions ....................................................................13

8.1.3 Metric model description ......................................................................................................................................13

8.1.4 Extending the metric .................................................................................................................................................14

8.1.5 Metric model details ..................................................................................................................................................14

Annex A (informative) SO and SO evaluation ............................................................................................................................................18

Annex B (informative) Metric — Table form .............................................................................................................................................20

Annex C (informative) CSM examples ...............................................................................................................................................................23

Annex D (normative) XML schema ......................................................................................................................................................................36

Bibliography .............................................................................................................................................................................................................................39

© ISO/IEC 2018 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 19086-2:2018(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

In the field of information technology, ISO and IEC have established a joint technical committee, ISO/

IEC JTC 1

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso

.org/iso/foreword .html.

This document was prepared by Technical Committee ISO/IEC JTC1, Information technology,

Subcommittee SC 38, Cloud Computing and Distributed Platforms.
A list of all parts in the ISO 19086 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/members .html.
iv © ISO/IEC 2018 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 19086-2:2018(E)
Introduction

The measurement of properties of cloud services, especially for the purpose of cloud service level

agreements (SLAs), presents many challenges, which inhibit the uptake of cloud services and inhibit

the overall effectiveness of the cloud services marketplace. Metrics in practice are usually described

using natural languages, typically in ‘plain English’, which is often difficult to understand, compare, and

implement. Such definitions of metrics lead to many problems. Typical concerns include:

— Clarity: The metric definition may be incomplete, ambiguous, illogical, self-contradictory, or not

defined at all. For example, cases exist where ‘availability’ is defined in ways which have little to do

with generally accepted definitions of ‘availability’; where the definition is such that the service can

be unavailable for the majority of the time yet the metric will show 100 % availability; where the

metric requires continuous monitoring, which is actually not possible; or where the provider is able

to determine at its sole discretion what the result is.

— Comparability: It may be impractical or effectively impossible to compare different services

in terms of their promised service levels because of the significant inconsistency in how their

respective metrics and SLOs/SQOs are defined.

— Implementation: It may be impractical or even impossible to measure the metric in practice, and

to determine whether promised service levels have been met or not.

This document has been developed to help address these and similar concerns. It includes technical

content, but the high-level concepts are expected to be understandable by non-technical individuals

who understand the business context for metrics. It provides a metric model that defines the conditions

and rules for performing a measurement and understanding the result.

A metric complying with the model defined by this document addresses the concerns above:

— Clarity: A definition of a metric eliminates the ambiguities which currently exist in natural language

descriptions.

— Comparability: The structured nature of the metric facilitates the comparison of different metrics

and SLOs/SQOs based on a metric.

— Implementation: The structured representation of the information needed to measure a

characteristic facilitates the process of developing measurement tools. Likewise, if the metric is

found not to be implementable, then the metric will need to be revised so that it can be implemented,

and the structure of the technical specification is expected to facilitate this revision process.

The focus of this document is on metrics for cloud SLAs, but it is also usable for cloud service metrics

(CSMs) that are not included in cloud SLAs [such as ones used by cloud service providers (CSPs) for their

internal performance monitoring], and may also be usable for non-CSMs.
© ISO/IEC 2018 – All rights reserved v
---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 19086-2:2018(E)
Cloud computing — Service level agreement (SLA)
framework —
Part 2:
Metric model
1 Scope

This document establishes common terminology, defines a model for specifying metrics for cloud

SLAs, and includes applications of the model with examples. This document establishes a common

terminology and approach for specifying metrics.

This document is for the benefit of and use for both cloud service providers (CSPs) and cloud service

customers (CSCs). This document is intended to complement ISO/IEC 19086-1, ISO/IEC 19086-3 and

ISO/IEC 19086-4.

This document does not mandate the use of a specific set of metrics for cloud SLAs.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 17788, | ITU-T Y.3500, Information technology — Cloud computing — Overview and vocabulary

ISO/IEC 19086-1, Information technology — Cloud computing — Service level agreement (SLA)

framework — Part 1: Overview and concepts

W3C Recommendation 28 October 2004. XML Schema Part 1: Structures Second Edition . ht t p: //w w w .w3

.org/TR/xmlschema -1/

W3C Recommendation 28 October 2004. XML Schema Part 2: Datatypes Second Edition. ht t p: //w w w .w3

.org/TR/xmlschema -2/
3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 17788 and the following

definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— IEC Electropedia: available at http: //www .electropedia .org/
— ISO Online browsing platform: available at http: //www .iso .org/obp
3.1
cloud service characteristic
qualitative or quantitative property of a cloud service
3.2
cloud service metric
metric (3.6) used to assess a cloud service characteristic (3.1)
© ISO/IEC 2018 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO/IEC 19086-2:2018(E)
3.3
cloud service objective

commitment a cloud service provider (CSP) makes for a specific characteristic of a cloud service

Note 1 to entry: The set of SOs is the union of the set of cloud service level objectives (SLOs) and the set of cloud

service qualitative objectives (SQOs).
3.4
measurement
set of operations having the objective of determining a measurement result (3.5)

Note 1 to entry: Based on the definition of measurement in ISO/IEC/IEEE 15939:2017. Also used here to describe

an actual instance of execution of these operations leading to the production of a measurement result instance.

3.5
measurement result

value that expresses a qualitative or quantitative assessment of a cloud service characteristic (3.1)

3.6
metric

standard of measurement that defines the conditions and the rules for performing the measurement

(3.4) and for understanding the measurement result (3.5)

Note 1 to entry: The metric describes what the result of the measurement means, but not how the measurement

implements the metric.

Note 2 to entry: A metric is to be applied in practice within a given context that requires specific properties to be

measured, at a given time(s) for a specific objective.

Note 3 to entry: The metrics model proposed in this document supports the definition of composite metrics,

which can be defined in terms of one or more underlying (reusable) metrics.

[SOURCE: ISO/IEC 19086-1:2016, 3.10, modified — Notes to entry have been modified.]

3.7
unit

real scalar quantity, defined and adopted by convention, with which any other quantity of the same

kind can be compared to express the ratio of the second quantity to the first one as a number

[SOURCE: ISO/IEC 80000-1:2009, modified — NOTES 1 to 5 have not been included.]
4 Symbols and abbreviated terms
CSA Cloud Service Agreement
CSC Cloud Service Customer
CSM Cloud Service Metric
CSP Cloud Service Provider
PII Personally Identifiable Information
SLA Service Level Agreement
SLO Cloud Service Level Objective
SO Cloud Service Objective
SQO Cloud Service Qualitative Objective
2 © ISO/IEC 2018 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 19086-2:2018(E)
5 Conformance

A metric specification is in conformance with this document when the specification of the metric uses

the data-types and relationships described in Clause 8. If the XML namespace at http: //standards .iso

.org/iso -iec/19086/ -2/ed -1/en is used for an XML document representing a metric, that document shall

be valid per the schema in Annex D.
6 Metrics overview
6.1 General

This document describes metrics, metrics usage, and defines a model for the consistent specification of

metrics for cloud SLAs. The underlying model and template is important for those defining metrics and

implementing measurement systems based on specific metrics.
6.2 Background

Cloud computing is an integral part of IT, yet it is still difficult to define the properties and performance

of a cloud service. A CSC purchases cloud services for one overarching reason – to achieve organization

goals. To this end, it is important for a CSC to understand the properties of a cloud service, to understand

the service capabilities for these properties, and to understand if it will meet the CSC’s requirements

necessary to achieve the CSC’s organization goals. Likewise it is important to a CSP to be able to

communicate the properties and performance of a cloud service so CSCs will be able to determine

whether any given service meets their requirements.

Typical categories of cloud service characteristics that the CSC might be interested in include:

performance, availability, information security, accessibility, cloud service support, termination of

service, governance, service changes, service reliability, attestations/certifications, data management,

and PII protection. A description of each along with associated SLOs/SQOs is included in ISO/IEC 19086-

1 and ISO/IEC 19086-4.
The cloud procurement process may be split into three basic aspects:
a) selecting a cloud service that meets the CSC’s requirements;

b) agreement (CSA, which includes the cloud SLA) between the CSP and the CSC on the properties and

performance of the cloud service;

c) operational management, where the operation of the cloud service is monitored to ensure the service

is operating within the constraints specified within the cloud SLA/CSA.
6.2.1 Choosing a cloud service

Currently, CSPs create their own methods to define the representation of a cloud service characteristic,

and therefore influence the understanding of the characteristic itself. This makes comparing cloud

service characteristics across CSPs (and sometimes within a CSP) difficult or impossible. The cloud

service characteristics are often described using ambiguous text descriptions and are not only

difficult to compare, but difficult to understand. This makes transferring a set of requirements into an

agreement between the CSC and CSP difficult; resulting in an agreement that might not meet the needs

of either party.

As described in ISO/IEC 19086-1, a commitment written into a cloud SLA takes the form of either an

SLO or an SQO. SLOs are quantitative commitments for representations of cloud service characteristics,

while SQOs are qualitative commitments for representations of cloud service characteristics.

© ISO/IEC 2018 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO/IEC 19086-2:2018(E)
A text description of availability might look something like:
— Commitment

The cloud service will be available 99,9 % of the time in a given billing cycle. If we fail to meet this

commitment, you will be eligible to receive a credit to your account.
— Unavailability

Unavailability means: a) The cloud service provides no response, or b) the cloud service returns a

server error response to a valid user request during two or more consecutive one minute intervals,

or c) the cloud service fails to deliver an average download time for a reference document of one

second or less as measured by a third party. Unavailability due to scheduled maintenance is

excluded from these conditions and does not contribute towards unavailability calculations.

This is a good example to consider since it is fundamental to all cloud services (CSCs want the service

to be available for their use). Although a number (percentage) is given for availability it is not clear how

this number is calculated. A definition of unavailability is provided, but it is not clear how the availability

percentage is calculated from unavailability. Another provider could use the concept of availability and

calculate it differently, making comparison of an SO or measured service levels impossible.

Figure 1 — Side by side comparison of availability commitment from two different providers

Figure 1 shows a CSC comparing availability for two services. While the characteristic (availability)

and commitment level (99,99 %) seem identical; the text defining uptime for the two services is

fundamentally different. Cloud service A uses a time based description for availability while cloud

service B uses a transaction based description for availability. The CSC is therefore unable to evaluate

how these two services differ.
6.2.2 Convert requirements to agreement

Once a CSC has chosen a service that meets their requirements, the CSP will work with the CSC to come

to an agreement on what exactly will be provided and at what level. The requirements will be codified

into a set of SLOs and SQOs written in a cloud SLA. As with the service capability descriptions used

during the decision making process, the SLOs/SQOs are currently described using natural language

and take different forms depending on the CSP. This not only adds ambiguity to the process, it adds

additional time and complexity to the process as each SO must be thoroughly reviewed and assessed in

each case.
4 © ISO/IEC 2018 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 19086-2:2018(E)
6.2.3 Ensure the agreement is being met

Once an agreement is in place and the cloud service is provisioned for the CSC, the CSC users will start

using the service or services. During operation, both the CSP and the CSC will monitor the services

to ensure that it is operating as expected. The CSP will provide tools to measure the cloud service

characteristics of the service and/or provide the data to the CSC. The CSC will compare these service level

measurement results to the SO specified in the agreement. Due to the ambiguous and inconsistent nature

of how SLOs/SQOs and metrics are currently described, it is difficult for the CSC to have confidence that

these measurement results are calculated in the same manner as defined in the cloud SLA.

6.3 Metrics

A metric, or standard of measurement, helps to clarify cloud service characteristics by providing details

about how a cloud service characteristic is measured. A metric provides a definition (e.g. expression,

unit, rules, parameters) of the measurement of the characteristic, and therefore, provides knowledge

about the characteristic itself. It provides the necessary information for repeatability, reproducibility,

and comparability of measurements and measurement results.
Figure 2 — Measurement process

The metric describes a cloud service characteristic and the details (parameters, data, rules,

expressions, additional details) necessary to use it. For example, an “availability” metric will define

the practical aspects of how to perform the measurement necessary to calculate availability, how to

measure downtime, exclusion rules, etc. The measurement result is a value that results from making

a measurement that follows a given metric. The measurement result is an estimate of a characteristic

that is being observed. As shown in Figure 2, metric defines the rules so a measurement can be made

in a repeatable, comparable manner. A measurement produces a measurement result that, combined

with the information in the metric, provides knowledge about the characteristic. Characteristics

of cloud services are almost never exactly known. Instead, an approximation of the characteristic is

estimated (based on one or more measurements) with some understanding of the uncertainty between

the approximation and the actual value of the characteristic. This uncertainty is directly tied to the

measurement process used. In other words, a metric is a standard set of rules that allow a measurement

process that follows the rules to generate repeatable, comparable estimates of a characteristic.

Since the metric provides both understanding of the characteristic and the information necessary to

make repeatable and comparable measurements, it can be used not only for the measurement process,

but in the conceptual understanding of a given characteristic.

What level of detail is included in a metric is up to the stakeholders. A metric may be only the basic

equations needed to calculate a measurement result or it may include the detailed measurement

process itself.
6.4 Cloud service metrics (CSMs)

Example scenarios for cloud services include the application of an availability metric for a SLO

commitment of 99 % in a cloud SLA or the application of an accessibility metric for a service quality

objective of “high” in a decision process scenario. In this manner CSMs help CSPs communicate the

© ISO/IEC 2018 – All rights reserved 5
---------------------- Page: 10 ----------------------
ISO/IEC 19086-2:2018(E)

properties of their cloud services, help CSCs and CSPs agree on what will be provided, and allow cloud

service features to be measured to ensure the agreement is met.

Cloud services can use metrics in many different ways. Metrics can be used at different layers of a

cloud service (e.g. hardware layers, logic layers, governance layers or service layers). They can also be

used at different stages of the cloud service life cycle (e.g. selection, procurement, operation, audit and

termination).

Although the metric model described in Clause 8 is designed for general use, in this document the focus

is on metrics used in cloud SLAs.

The definition and usage of appropriate metrics with their underlying measures are essential

components of the cloud SLA and the SLOs/SQOs, which are constituents of the cloud SLA. Within a

cloud SLA, the metrics are used to set the boundaries and margins of errors the provider of the service

commits to deliver, and their limitations. Standardized metrics and metric templates for cloud SLAs

makes it easier and quicker to develop cloud SLAs and the included SLOs/SQOs. Once the cloud SLA is

in place, metrics could be used at runtime to measure the services levels and determine if the service is

meeting the commitments in the cloud SLA.

Metrics for cloud services have several uses for CSCs and CSPs including but not limited to:

— Metrics can be used by CSPs to describe the performance of a cloud service.

— Metrics help CSCs to compare offerings from different CSPs (when CSPs uses the same metrics to

describe the performance of a cloud service).
— Metrics can be used to describe SLOs and SQOs within a cloud SLA.

— Metrics can be used by CSCs to determine if the CSP is meeting their commitments as described in

the cloud SLA and to claim remedies if the commitments are not being met.
6.4.1 Major stakeholders

This subclause describes stakeholder types that have an interest in cloud SLA metrics. Each stakeholder

type represents a different set of interests and related concerns. Each stakeholder type uses metrics in

a different way to address those interests.
Attorney

Negotiates agreements for cloud services between CSP and CSC. Uses this document as reference for

specifying metrics in cloud SLAs.
Auditors

Evaluate cloud services for performance against a set of requirements and commitments. Use ISO/

IEC 19086-2 based metrics to define and measure cloud service characteristics that relate to the

requirements. A specific type of auditor may also use this document to evaluate the measurement

system itself. Needs to understand the metric used to measure a cloud service characteristic.

Certification authority
Develop certification criteria for cloud services. Use metrics based on this doc
...