ISO/IEC TR 20913:2016

Information technology - Data centres - Guidelines on holistic investigation methodology for data centre key performance indicators

ISO/IEC TR 20913:2016 describes backgrounds, motivation, and general concept of holistic methodology for data centre key performance indicators (KPIs) to investigate the status of KPIs. It discusses the usefulness of holistic investigation methodology in terms of aggregating a KPI across different contexts, aggregation of two or more KPIs within a single context, aggregation of two or more KPIs across multiple contexts, and aggregation of the multiple KPIs into a single indicator. This document presents a conventional spider web chart-based data centre KPIs status observation method and a control chart method including upper bound and lower bound of the operational status of KPIs. This document presents SWOT analysis results for both methodologies. The methods described in this document are aimed at the self-monitoring of a data centre, not comparison among data centres. Specifically, ISO/IEC TR 20913:2016 a) describes backgrounds, motivation, and general concept of holistic investigation methodology for data centre KPIs, b) analyses the usefulness of holistic investigation methodology for aggregating KPIs, c) describes a spider web chart-based KPIs status observation method and a control chart extending spider web chart to observe the operational status of KPIs, d) describes alternative and/or additional methods of representing dissimilar KPIs to track holistic resource effectiveness of the data centre, and e) presents SWOT analysis results for holistic investigation methods described in this document.

Technologies de l'information — Centres de données — Lignes directrices relatives à la méthodologie de recherche holistique pour les indicateurs de performance clé du centre de données

General Information

• Status: Published
• Publication Date: 03-Nov-2016
• ICS: 35.020 - Information technology (IT) in general
• Technical Committee: ISO/IEC JTC 1/SC 39 - Sustainability, IT and data centres
• Drafting Committee: ISO/IEC JTC 1/SC 39 - Sustainability, IT and data centres
• Current Stage: 6060 - International Standard published
• Start Date: 04-Nov-2016
• Due Date: 12-Jan-2018
• Completion Date: 12-Jan-2018

Overview

ISO/IEC TR 20913:2016 provides guidelines for a holistic investigation methodology to observe and analyse data centre key performance indicators (KPIs). Rather than prescribing a single metric, this Technical Report focuses on visual and analytical methods - notably spider web (radar) charts and an extended control chart approach - to show the state and trends of multiple KPIs together. The guidance is intended for internal, self-monitoring of a single data centre’s resource effectiveness (energy, water and related resources), not for comparing different data centres.

Key topics and guidance

• Holistic investigation concept: rationale and motivation for viewing multiple KPIs together to evaluate overall data centre performance.
• Types of KPI aggregation:
  • Aggregating a single KPI across different contexts
  • Aggregating two or more KPIs within one context
  • Aggregating KPIs across multiple contexts
  • Considerations for combining multiple KPIs into a single indicator (and limits of doing so)
• Spider web chart methodology:
  • Principles for axis selection, normalization/scaling, inversion of axes (when higher/lower values indicate better performance), and KPI ordering
  • Visualisation of multiple temporal measurements on one chart for trend spotting
• Control chart extension:
  • Adding upper and lower operational bounds to spider web displays to highlight out-of-bounds KPI behaviour
  • Use for monitoring operational stability and energy-efficiency deviations
• Considerations and limitations: normalization issues, units/dimensions mismatch, possible masking of individual KPI problems when aggregating, and interpretation challenges
• SWOT analysis: strengths, weaknesses, opportunities and threats of the presented holistic methods

Practical applications

• Monitor and visualise multiple data centre KPIs (e.g., PUE, utilization ratios, renewable energy factors) in a single view
• Detect trends and operational anomalies via control-bound monitoring
• Support continuous improvement and internal reporting for facility managers and sustainability teams
• Facilitate cross-discipline communication (IT, facilities, sustainability) through clear visual summaries

Who should use this standard

• Data centre administrators and operators
• Sustainability and energy-efficiency managers
• Facility and infrastructure engineers
• IT performance analysts seeking a multi-KPI visualisation and monitoring approach

Related standards

• ISO/IEC 30134 series (data centre KPIs and definitions)
• ISO/IEC JTC 1/SC 39 (sustainability for and by information technology)

Keywords: ISO/IEC TR 20913:2016, data centre KPIs, holistic investigation, spider web chart, control chart, KPI aggregation, data centre resource effectiveness, PUE, KPI visualization.

Standards Content (Sample)

TECHNICAL ISO/IEC TR
REPORT 20913
First edition
2016-11-15
Information technology — Data
centres — Guidelines on holistic
investigation methodology for data
centre key performance indicators
Technologies de l’information — Centres de données — Lignes
directrices relatives à la méthodologie de recherche holistique pour
les indicateurs de performance clé du centre de données
Reference number
©
ISO/IEC 2016
© ISO/IEC 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO/IEC 2016 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms, definitions and abbreviated terms . 1
3.1 Terms and definitions . 1
3.2 Abbreviated terms . 2
4 Background and motivation . 2
4.1 General concept of holistic investigation method . 2
4.2 Usefulness of spider web chart methods for visualizing data centre KPIs . 3
4.3 Usefulness of aggregating data centre KPIs . 4
5 Spider web chart-based KPIs status observation method . 4
5.1 Principles for constructing a spider web chart using KPIs . 5
5.1.1 Selection of axis on a spider web chart . 5
5.1.2 Presentation of KPIs on axes . 5
5.2 Example of a holistic approach . 5
5.3 Example of holistic approach of data centre by use of a spider web chart . 6
6 Control chart method extending a basic spider web chart to observe the
operational status .11
6.1 Motivation for control chart method for energy efficiency monitoring .11
6.2 Control chart approach for energy efficiency monitoring .11
7 Considerations for applying holistic investigation methods .14
8 SWOT analysis results for holistic investigation methods .14
Bibliography .16
© ISO/IEC 2016 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information.
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 39,
Sustainability for and by Information Technology.
iv © ISO/IEC 2016 – All rights reserved

Introduction
The ISO/IEC 30134 series defines key performance indicators (KPIs) for data centre resource
effectiveness. There are many aspects to be considered in order to improve data centre resource
effectiveness. As for resources, it may include not only energy, but also water and other natural
resources. As for data centre components, they include air conditioning, power supply, servers, storages,
and network equipment. However, it is difficult to include all aspects into one KPI, so multiple KPIs
are under development, which measure each aspects of resource effectiveness improvement. Resource
effectiveness improvement in each aspect will be performed by measuring each KPI. On the other hand,
there is a need to observe the state and trend of data centre as a whole, or holistically, by monitoring
multiple KPIs in a single view. Analysis of the KPIs from the overall perspective is also referred to
as a holistic investigation method. This document describes a spider web chart-based method and
control chart method extending the functionality of the conventional spider web chart for viewing and
analysing KPIs for data centre resource effectiveness. It also investigates considerations for applying
holistic investigation methods to resource effectiveness evaluation of multiple data centre KPIs. The
usefulness and applicability of holistic methods are discussed using a SWOT analysis. The methods
described in this document are intended for analysis and continuous improvement of a specific data
centre and not for comparing different data centres.
© ISO/IEC 2016 – All rights reserved v

TECHNICAL REPORT ISO/IEC TR 20913:2016(E)
Information technology — Data centres — Guidelines on
holistic investigation methodology for data centre key
performance indicators
1 Scope
This document describes backgrounds, motivation, and general concept of holistic methodology for data
centre key performance indicators (KPIs) to investigate the status of KPIs. It discusses the usefulness of
holistic investigation methodology in terms of aggregating a KPI across different contexts, aggregation
of two or more KPIs within a single context, aggregation of two or more KPIs across multiple contexts,
and aggregation of the multiple KPIs into a single indicator. This document presents a conventional
spider web chart-based data centre KPIs status observation method and a control chart method
including upper bound and lower bound of the operational status of KPIs. This document presents
SWOT analysis results for both methodologies. The methods described in this document are aimed at
the self-monitoring of a data centre, not comparison among data centres.
Specifically, this document
a) describes backgrounds, motivation, and general concept of holistic investigation methodology for
data centre KPIs,
b) analyses the usefulness of holistic investigation methodology for aggregating KPIs,
c) describes a spider web chart-based KPIs status observation method and a control chart extending
spider web chart to observe the operational status of KPIs,
d) describes alternative and/or additional methods of representing dissimilar KPIs to track holistic
resource effectiveness of the data centre, and
e) presents SWOT analysis results for holistic investigation methods described in this document.
2 Normative references
There are no normative references in this document.
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http://www.electropedia.org/
— ISO Online browsing platform: available at http://www.iso.org/obp
3.1.1
holistic investigation method
data centre resource effectiveness investigation method considering multiple key performance
indicators
© ISO/IEC 2016 – All rights reserved 1

3.1.2
spider web chart
chart that consists of multiple performance indicators which are set in a circle like a spider web
3.2 Abbreviated terms
IT Information Technology
ITEEsv IT Equipment Energy Efficiency for Servers
ITEUsv IT Equipment Utilization for Servers
KPI Key Performance Indicator
PUE Power Usage Effectiveness
REF Renewable Energy Factor
SWOT Strength Weakness Opportunity Threat
4 Background and motivation
4.1 General concept of holistic investigation method
Improving the resource effectiveness and carbon footprint of a data centre requires the monitoring
and analysis of multiple KPIs. ISO/IEC JTC 1/SC 39 has determined that it is impractical to aggregate
multiple KPIs to determine the overall energy effectiveness of a data centre. There is a need to observe
the state and trend of multiple KPIs in a single view.
With any performance indicator, it is necessary to understand the expected upper and lower limits and
general behaviour of the performance indicator. There are typically two approaches that are applicable
to holistic investigation of data centre KPIs:
— Engineering/modeling method: This method has been used to establish baseline performance. This
methodology requires the development of an optimized economic and engineering model based
on creating an idealized benchmark specific to each utility — incorporating the topology, demand
patterns, and population density of the service territory. Typical limitations of this approach are as
follows: the engineering models that support it can be very complicated, and the structure of the
underlying components relationships can be obscured through a set of assumed coefficients used in
the optimization process.
— Performance benchmarking method: This method includes a set of specific performance
measurement indicators, such as volume billed per worker, consumed energy per product, quality
of service (continuity, water quality, complaints), coverage, and key financial data. Usually, these
indicators are presented in ratio form to control the scale of operations. These partial measures are
generally available and provide the simplest way to perform comparisons: trends direct attention
to potential problem areas.
Among the methods mentioned above, the performance benchmarking method is useful for evaluating
the resource efficiency of data centres because ISO/IEC JTC 1/SC 39 is offering a selection of energy
effectiveness KPIs. The performance benchmarking method may be further categorized into two types:
performance indicator-based methods and chart-based methods.
— Performance indicator-based methods: In this category, the performance of the target is evaluated
by developing performance indicators for the target. For example, Hz for CPU and bytes for storage
are typical performance indicators. This category allows accurate performance evaluation and
comparison among targets, if the performance indicators are defined. Typical limitation of this
approach is that it is difficult to compare the evaluation results if performance indicators belong to
different dimensions with different units.
2 © ISO/IEC 2016 – All rights reserved

— Chart-based methods: This category depicts the target’s performance by using chart methods, such
as pie, bar, line, and spider web, etc. This category is useful for evaluating performance by displaying
multiple performance indicators, making analysis easier.
Since the chart-based approach supports multiple performance indicators simultaneously, it is
appropriate for a holistic method. The spider web chart in particular is well suited for the display and
analysis of multiple KPIs. A spider web chart is useful for displaying multiple KPIs in a single chart. It
is also useful for displaying multiple measurement values of several KPIs in a single chart, for example,
temporal measurement values of several KPIs. Thus, this document focuses on the spider web chart-
based holistic KPI investigation methods. It is noted that the chart-based approach, especially spider
web chart, has typical issues for applying a KPI investigation, such as scaling and normalization of KPI
values, KPIs with different dimensions, ordering of KPIs in the chart, graphical interpretation of the
chart, and so on. These typical issues are discussed in Clause 6 in detail.
4.2 Usefulness of spider web chart methods for visualizing data centre KPIs
The spider web chart consists of a bundle of performance indicators which are set in a circle. The
indicators are usually normalized from zero to one, one indicating the highest possible performance,
but unnormalized indicators may be utilized. Individual axes may need to be inverted in order for the
different indicators to correlate. It is clear that the quality of the spider web charts depends on the
validity, reliability, and comprehensiveness of the performance indicators. It is known that the spider
web chart has strength on visualizing the status of performance indicators.
Regarding visualization capability, spider web charts provide a synoptic description of multiple
performance measures and make trade-offs between performance measures visible. Figure 1 shows a
spider web chart consisting of three sets of performance measurements and five performance indices.
In the figure, the values of each index are originally measured and unnormalized ones, and the farther
from centre of the chart implies the better. Each green, blue, and red polygon connecting measurement
values of five index shows a single observation of the five indices, respectively. Using the chart, it is
possible to visually compare the performance achievement among multiple performance measurements
and indicators.
Figure 1 — Example of a spider web chart consisting three performance measurements (green,
blue, and red)
© ISO/IEC 2016 – All rights reserved 3

Due to the advantages, spider web charts are popularly used to assess the performance of various
evaluation objectives and to present a visual comparison of performance in various fields, especially
business management. As discussed in this clause, the visualization capability of a spider web chart
can help data centre administrators to monitor the specified performance KPIs of the data centre and
their changes so that they can improve the efficiency of the data centre. For example, by regularly
constructing the spider web chart showing the state of each KPI, the data centre administrator can
effectively monitor the temporal behaviour of the KPIs. Further, the spider web chart has general
advantages for assessing data centre KPIs rather than conventional charts such as bar chart when
assessing the multiple measurement values of multiple KPIs. The advantages of a spider web chart are
discussed in Clause 6 in detail.
4.3 Usefulness of aggregating data centre KPIs
The key objective of aggregating multiple performance indicators into a single indicator is to represent
the overall achievement of each indicator as a single and integrated output. However, there are well-
known problems with aggregating heterogeneous performance indicators. Each indicator can have a
different dimension and scale, so aggregating multiple indicators by normalizing their original values
can lose the characteristics of each indicator. Additionally, the aggregation process can cause a serious
problem to be masked or a minor issue can be overstated depending on how the individual indicators
are scaled. Further, depending on the dimension of indicators, it may be inappropriate to aggregate
multiple indicators into a single indicator.
However, if all indicators are measured with the same dimension, aggregating multiple indicators into
a single indicator may be useful. For example, if an indicator measures the operational achievement
ratio of a KPI and its operational target value, the achievement ratio explains whether the data centre
is operated effectively according to the operational target value. The operational target value of a KPI
indicates the intended threshold for the KPI. Assume that a KPI measures the utilization ratio of IT
server equipment and the administrator of the IT server sets upper bound and lower bound of the KPI
as the operational target values. If the measured KPI value exceeds the upper bound of the KPI, the
administrator may consider to install more IT servers in order to reduce the utilization ratio of IT servers.
Whereas the administrator may consider to consolidate underutilized IT servers if the measured KPI
value is below the lower bound of the KPI. Thus, by integrating the achievement ratios of each KPI into
a single value, the data centre administrator can easily determine whether the data centre is operated
as planned. It should be noted that observing the aggregated number of the measured KPI values may
overlook the detailed characteristics. For example, by looking at the aggregate number of measured
KPI values for server utilization, the overuse of one server may be masked by the underuse of another.
Careful review of the individual server data for such events should be conducted to avoid data masking
issues that may occur during KPI aggregation. However, even in this case, the relative importance (e.g.
weighting) of each indicator is obscured. Thus, the aggregated overall operational achievement helps
with management of the temporal changes in data centre operational efficiency. For example, let us
assume that a data centre regularly examines the values of the overall operational achievement. At some
time, if the overall operational achievement of the data centre is below the threshold for the data centre,
the statuses of each KPI will be investigated and KPIs of which achievement is less than threshold could
be managed by an administrator. Once the overall operational achievement value of the data centre
exceeds the threshold, only the overall operational achievement value may be used to regularly manage
the data centre.
5 Spider web chart-based KPIs status observation method
A holistic approach enables awareness of the effect of changes made to the data centre specific from an
overall viewpoint by use of various efficiency metrics. A holistic approach helps the operator keep in
mind the effects on all metrics simultaneously.
4 © ISO/IEC 2016 – All rights reserved

5.1 Principles for constructing a spider web chart using KPIs
5.1.1 Selection of axis on a spider web chart
To start a holistic approach, a data centre should define its whole scope or whole boundary to measure.
This scope might vary by each data centre, since each data centre’s functionality or service is different.
This scope defines the holistic view, within which KPIs are chosen. Then, it is most important to select
an appropriate combination of axes or KPIs used in the spider web chart. “Holistic approach” implies
that data centre should consider all ways within the scope to achieve data centre energy reduction, CO
reduction, or improvement of sustainability. It is desirable for the combination of these axes to reflect
all available ways within the scope to improve data centre energy efficiency. Also, it is desirable that the
axes do not have much inter-relationship or much inter-dependency. In order to select the appropriate
combination of axes, there is a good principle called the “MECE” principle.
The “MECE” principle, pronounced “me see,” mutually exclusive and collectively exhaustive, is a
grouping principle for separating a set of items into subsets. In the holistic approach, when choosing a
combination of KPIs, KPIs should be mutually exclusive and collectively exhaustive as much as possible.
Two KPIs are mutually exclusive if they do not share the same ways of improvement. Mutually
exclusive principle eliminates overlapping of the aims of two KPIs and reducing inter-relationship or
inter-dependency of KPIs.
A combination of KPIs is jointly or collectively exhaustive if all methods of improvement are taken
into account by at least one of KPIs because they encompass the entire range of possible ways within
the scope. Collectively exhaustive principle guarantees that the data centre considers all ways of
improvement within the scope when selecting multiple KPIs, or these KPIs can provide holistic view.
5.1.2 Presentation of KPIs on axes
A spider web chart is constructed using a set of equally angular axes on a two-dimensional chart. Each
axis represents one KPI. The data length of an axis is generally proportional to the magnitude of the
KPI for the data point relative to the maximum magnitude of the KPI across all data points. A line is
drawn connecting the data values for each axis. This gives a spider web chart. Since the spider web
chart consists of multiple axes representing KPIs for data centre and originally measured values of each
KPI are mapped into the designated axis, there exist guidelines for presenting KPIs on axes of the spider
web chart.
The basic principle is that the larger value in an axis implies the higher efficiency. The following are a
set of guidelines for presenting KPIs on axes.
— The values in each axis are originally measured values of the designated KPI for the axis.
— The values in each axis are presented as the farther value from the centre of the chart is considered
as the more efficient.
— When the lower value of a KPI indicates the higher efficiency, the value in an axis of the KPI can be
inverted.
— Depending on a KPI, the values of an axis can be normalized. The normalization can be performed
by various ways depending on the characteristics of the KPI, for example, uniform values between
0 and 1, uniform values between a and b, nonlinear values, and so on.
— The range of the normalized values of the KPI can be determined by a data centre considering the
characteristics of the data centre, such as region, type, and so on.
5.2 Example of a holistic approach
To better understand the way in which a holistic approach is meant to improve energy efficiency in data
centres, it is useful to consider the metaphor of automobile fuel efficiency. We can consider a “scope”
of automobile fuel efficiency as a whole society. Then, the improved fuel efficiency has not come about
© ISO/IEC 2016 – All rights reserved 5

solely through the efforts of car manufacturers. Rather, progress has been made by involving automobile
manufacturers, component manufacturers, materi
...