ETSI TR 104 004 V1.1.1 (2024-09)

Environmental Engineering (EE); Processor power management functionality of servers

DTR/EE-EEPS60

General Information

• Status: Not Published
• Technical Committee: EE EEPS - EE Eco Environmental Product Standards
• Current Stage: 12 - Citation in the OJ (auto-insert)
• Due Date: 30-Sep-2024
• Completion Date: 19-Sep-2024

Standards Content (Sample)

TECHNICAL REPORT
Environmental Engineering (EE);
Processor power management functionality of servers

2 ETSI TR 104 004 V1.1.1 (2024-09)

Reference
DTR/EE-EEPS60
Keywords
server
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from the
ETSI Search & Browse Standards application.
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format on ETSI deliver.
Users should be aware that the present document may be revised or have its status changed,
this information is available in the Milestones listing.
If you find errors in the present document, please send your comments to
the relevant service listed under Committee Support Staff.
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure (CVD) program.
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2024.
All rights reserved.
ETSI
3 ETSI TR 104 004 V1.1.1 (2024-09)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Executive summary . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 7
3.1 Terms . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 CPU Power Management . 8
4.1 Overview of CPU P-states and C-states . 8
4.2 Impact of P-state and C-state enablement on server power demand . 9
4.3 Latency impacts resulting from power management enablement. 13
4.4 Conclusion/Recommendations . 16
Annex A: Detailed discussion of Hybrid ssj performance and power measurements and key
indicators for server configurations with power management turned on and off . 17
History . 19

ETSI
4 ETSI TR 104 004 V1.1.1 (2024-09)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its
Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Environmental Engineering (EE).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
5 ETSI TR 104 004 V1.1.1 (2024-09)
Executive summary
CPU Power Management functions (P-states and C-states or their equivalent) offer hardware available, software
initiated functions to reduce CPU voltage and frequency when workload demands are low or absent to reduce server
energy use. Depending on the amount of time a server operates below 25 % utilization or is idle, with longer periods
generating higher energy savings, power demand can be reduced by up to 50 % with the full implementation of P-states

and C-states. These power demand reductions improve SERT overall efficiency scores by up to 30 %. Power
management functions come at a cost of reduced server performance and increased response time (exit latency). Data
from SERT measurements on a single configuration with power management turned on resulted in performance
reductions at the 100 % utilization level in the Hybrid ssj worklet of 6 % for an EPYC™ CPU and 14 % for a Power9™
CPU compared to power management off. For response time, the literature shows that P-state transition times and
C-state exit latencies cause response delays that are problematic for some workloads and applications. As an example, ®
Operating System (OS) managed power management profiles can interfere with virtualization programs like VMware
impeding performance rates on the virtualized images. Power management can be a benefit or a problem depending on
the data centre operations, workloads and applications. While the implementation of a power management profile can
be beneficial in many instances, the optimum P-state and C-state settings and the choice of controlling software - BIOS,
hypervisor or operating system - will depend on the specific use case. In some cases, such as high-speed financial
trading, network providers or high-performance computing, power management functions will need to be turned off to
ensure the performance and response times demanded by those workloads.
Introduction
Data centre energy consumption and IT equipment efficiency are increasingly the focal point of data centre customers
and operators, non-governmental organizations and government regulators with an interest in data centre operations.
There are a host of options and approaches to reduce data centre energy consumption that can focus on the individual
IT equipment, the data centre IT system, or the facility equipment. As servers are the majority of the IT equipment in a
data centre and have a validated energy efficiency metric, the SPEC SERT suite, server energy efficiency requirements
have attracted the most attention of regulatory efforts to improve data centre energy efficiency.
Server processor power management features, which can moderate power demand based on factors such as CPU
utilization, provide a means for server manufacturers and/or data centre operators to reduce server power consumption
and improve server energy efficiency. In many CPU architectures, the primary means of server processor power
management is through implementation of CPU P-states and C-states. Enablement of power management states can
reduce performance and increase response time. The present document will focus on the characteristics, benefits and
limitations of CPU power management features in deployment situations.
Server power management features are also available for system fans, memory, storage, Graphic Processing Unit and
I/O components. Their combined implementation introduces greater complexity and requires a higher level of
integration with the server's platform firmware and operating system. Component power management is beyond the
scope of the present document. More specifically the present document will provide a brief overview of CPU P-states
and C-states, SERT test data detailing the power demand reductions achieved by different power management
implementations, the SERT measured active state energy efficiency improvements, and the performance and exit
latency impacts of server power management features. The present document only addresses the current state of these
technologies.
ETSI
6 ETSI TR 104 004 V1.1.1 (2024-09)
1 Scope
The present document is focused on addressing the characterization of the process power management functionality of
servers.
The processor power management of servers is limited to those within scope of Commission
Regulation (EU) 2019/424 [i.1].
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Commission Regulation (EU) 2019/424 of 15 March 2019 laying down ecodesign requirements
for servers and data storage products pursuant to Directive 2009/125/EC of the European
Parliament and of the Council and amending Commission Regulation (EU) No 617/2013.
[i.2] C. Chou, L. N. Bhuyan and D. Wong: "μDPM: Dynamic Power Management for the Microsecond
Era", 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA),
Washington, DC, USA, 2019, pp. 120-132, doi: 10.1109/HPCA.2019.00032.
[i.3] I. Curtis: "Hot Chips 2020 Live Blog, Next Gen Intel Xeon Ice Lake-SP", Hot Chips Conference,
August 17, 2020.
[i.4] D. Molka, R. Schone, M. Werner: "Wake-up latencies for processor idle states on current x86
processors", Computer Science - Research and Development, Vol. 30,
doi: 10.1007/s00450-014-0270-z; 2014/04/01.
[i.5] S. Kanev, K. Hazelwood, G. Wei and D. Brooks: "Tradeoffs between power management and tail
latency in warehousescale applications", 2014 IEEE International Symposium on Workload
Characterization (IISWC), Raleigh, NC, 2014, pp. 31-40, doi: 10.1109/IISWC.2014.6983037. ®
[i.6] Broadcom : "Virtual machine application runs slower than expected in ESXi".
[i.7] Lenovo™: "Tuning VMware for Increased Performance - Lenovo ThinkSystem and x86 Servers".
[i.8] Huawei: "Huawei V5 Server Best Practice with VMware ESXi System 03".
[i.9] HPE: "Workload profiles and performance options | UEFI System Utilities User Guide for
HPE Compute servers".
[i.10] The Green Grid: "SERT™ Active Efficiency: Demonstrating How SERT™ Active Efficiency
Testing Includes Server Idle".
ETSI
7 ETSI TR 104 004 V1.1.1 (2024-09)
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
Basic Input/Output System (BIOS): firmware that provides basic boot capabilities for a platform
C-state: processor power consumption and thermal management state within the global working state
® ®
Hybrid ssj: one of the 7 CPU worklets used in SPEC SERT suite
P-state: power consumption and capability state within the active/executing states, C0 for processors
NOTE: Performance states allow Operating System direct Power Management (OSPM) to make tradeoffs
between performance and energy conservation.
power management profile: combination of P-state and C-state settings designed to reduce power consumption while
addressing the specific operating needs of a customer environment or workload
Server Efficiency Rating Tool (SERT): performance and power software measurement tool created by the SPEC
benchmark standards consortium
NOTE 1: SERT was specifically designed for use in government sponsored server energy efficiency programs.
NOTE 2: SERT has components that run on the system under test and a controller system, and interfaces with a
power analyser connected between the electrical socket and server power supply.
NOTE 3: Detailed performance and power data is collected while running server worklets at different load level,
and these measurements are combined into an overall weighted server energy efficiency score.
worklet: parts of a workload consisting of specific code sequences which are executed during testing
workload: group of worklets which share common attributes and are combined into an overall result
NOTE: SERT includes CPU, Memory and Storage workloads.
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
BIOS Basic Input/Output System
CPU Central Processing Unit
DPM Dynamic Performance Mode
HDD Hard Disk Drive
OEM Original Equipment Manufacturer
OS Operating System
OSPM Operating System direct Power Management
QPS Queries Per Second
SERT Server Efficiency Rating Tool
SKU Stock Keeping Unit
SPS Static Power Savings
ETSI
8 ETSI TR 104 004 V1.1.1 (2024-09)
4 CPU Power Management
4.1 Overview of CPU P-states and C-states
CPU power management is executed through implementation of P-states and C-states. P-states modulate the power
consumption of the CPU as a function of workload demand. An accepted method for implementing P-states is to reduce
the voltage and frequency of the individual cores or groups of cores while the processor is executing instructions.
P-states consist of predetermined operating frequencies and associated operating voltages which are selected to match
the processor work capability to the current workload applied to the server. C-states significantly reduce the processor
voltage during periods where the processor has no work to perform. During no work periods, C-states sequentially
reduce the power demand in time defined steps.
For the purposes of the present document, a specific power management profile is a combination of P-state and C-state
settings designed to reduce power consumption while addressing the specific operating needs of a customer
environment or workload. Power management profiles implemented in production servers may contain any number of
processor or system power management settings. The primary method of reducing power during idle periods is turning
off portions of the CPU which is realized with a penalty of increasingly longer latencies before the processor can begin
executing instructions (Figure 8).
® ®
Figure 1 shows the power and frequency curve for different levels of P-state and C-state in an Intel Xeon x86
processor. AMD x86 processors have similar P-states and C-states with power levels specific to their architecture.
® ™ ®
Servers using the IBM POWER9 processor and Linux operating system have a similar profile, while IBM
POWER9 and PowerVM systems have a different profile. See Annex A for details. ®
NOTE 1: Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

NOTE: This is for illustration only. Actual processor power will vary by processor SKU.

Figure 1: Processor Power Graph representing the available power states
for each individual core for typical processor architectures
ETSI
9 ETSI TR 104 004 V1.1.1 (2024-09)
Since P-states and C-states are independent features, their usage policies can be combined in an infinite number of
combinations. Server manufacturers typically provide 2 or more standard power management policy implementations
(profiles) which can be selected in the OS or the BIOS: maximum performance, maximum power savings and one or
more intermediate implementations designed to balance power savings against increased response time and latency.
Data centre operators can, if they desire, 'tune' their P-state/C-state policies to fit their specific workload profile(s)
through a testing routine similar to performance optimization tuning for their workload(s).
™
NOTE 2: The Green Grid Server published a report in March 2018, "SERT Active Efficiency: Demonstrating
™
How SERT Active Efficiency Testing Includes Server Idle" [i.10], which can be consulted for a more

detailed discussion of the CPU power management function capabilities a
...