ETSI TS 103 586 V1.1.1 (2019-04)

Environmental Engineering (EE); Liquid cooling solutions for Information and Communication Technology ( ICT) infrastructure equipment

DTS/EE-0166

General Information

• Status: Published
• Publication Date: 04-Apr-2019
• Technical Committee: EE 1 - EE Environmental Conditions
• Current Stage: 12 - Completion
• Due Date: 31-Mar-2019
• Completion Date: 05-Apr-2019

Standards Content (Sample)

ETSI TS 103 586 V1.1.1 (2019-04)






TECHNICAL SPECIFICATION
Environmental Engineering (EE);
Liquid cooling solutions for Information and
Communication Technology (ICT) infrastructure equipment

---------------------- Page: 1 ----------------------
2 ETSI TS 103 586 V1.1.1 (2019-04)



Reference
DTS/EE-0166
Keywords
cooling capacity, energy efficiency
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2019.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TS 103 586 V1.1.1 (2019-04)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Introduction . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 6
3.1 Terms . 6
3.2 Symbols . 7
3.3 Abbreviations . 7
4 ICT equipment liquid cooling requirements and energy efficiency . 7
4.1 Introduction . 7
4.2 Cooling requirements for equipment . 7
4.3 Liquid cooled equipment energy efficiency . 8
5 Specifications for liquid cooling solutions . 8
5.1 General requirements . 8
5.2 Liquid flow rate range vs. dissipated power . 9
5.3 Temperature of touchable parts . 9
5.4 Max pressure drop per liquid flow rate . 9
5.5 Max pressure drop per air flow rate. 9
5.6 Pipe threads . 9
5.7 Coolants and cooling distribution units . 9
5.8 Max pressure and tightness . 9
5.9 Liquid connectors positions . 9
5.10 Accessibility in case of cooling with heat exchanger . 10
6 Benchmark methods to evaluate cooling system efficiency and energy efficiency . 10
Annex A (informative): Cooling principles and impact on reliability and energy consumption
. 12
A.1 Air cooling principles and limitations . 12
A.2 Reliability issues . 13
A.3 Energy consumption . 13
A.4 Heat reuse possibilities . 14
Annex B (informative): Cooling implementation options . 15
B.1 Example of liquid cooling at the cabinet level . 15
B.2 Example of liquid cooling at the component level . 18
B.3 Example of liquid cooling by immersion . 20
B.4 Example of topology of the cooling distribution at the room and building level . 21
History . 23

ETSI

---------------------- Page: 3 ----------------------
4 ETSI TS 103 586 V1.1.1 (2019-04)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Environmental Engineering (EE).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Electrical energy supplied to ICT equipment, and more generally to electronic equipment, is nearly totally converted
into heat by resistive losses, leading to temperature increase of the equipment itself and its surrounding environment.
Except for very low power (ICT end-user equipment), ICT equipment should be cooled to ensure reliable operation and
an acceptable lifetime. Air-cooling is up to now dominating in the telecommunication industry. ETSI
EN 300 019 series [i.2] specify environmental conditions for different types of locations, to ensure proper operation of
air cooled telecommunication equipment.
With the emergence of high density racks and cabinets, thermal loads above 7 kW become widely used while density
increase remains on-going. These high loads cabinets lead also to thermal management issues at the room level. More
than ever, separation of hot and cold aisles is necessary and moreover, prevention of hot spots when high and medium
or low loads are mixed in the same room is hard to achieve.
Liquid cooling solutions provide opportunities to solve efficiently these problems and to reduce significantly cooling
energy consumption and, thus, overall ICT energy consumption. Moreover, such technologies can lead to improved
temperature control at the component level and consequently, better reliability. Thanks to higher cooling capacity, ICT
equipment can be more compact leading thus, to space savings. At last, heat reuse can be also considered with very high
efficiency optimizing this way, ICT energy efficiency.
ETSI

---------------------- Page: 4 ----------------------
5 ETSI TS 103 586 V1.1.1 (2019-04)
1 Scope
The present document covers following applications:
• Liquid cooling at the cabinet/rack level.
• Liquid cooling at the product level.
• Liquid cooling via immersion in dielectric liquid.
The present document specifies the following items:
• Liquid circulation layout (connection of multiple units).
• Liquid flow rate range vs. dissipated power.
• Max pressure drop per liquid flow rate.
• Max pressure drop per air flow rate.
• External pipe diameter range and pipe threads.
• Valves requirements.
• Coolants and cooling distribution unites.
• Max pressure and tightness.
Furthermore, the present document provides:
• Benchmark methods to evaluated different cooling system efficiency.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI EN 300 019-1-3 (V2.4.1): "Environmental Engineering (EE); Environmental conditions and
environmental tests for telecommunications equipment; Part 1-3: Classification of environmental
conditions; Stationary use at weatherprotected locations".
[2] ETSI EN 300 019-1-4 (V2.2.1): "Environmental Engineering (EE); Environmental conditions and
environmental tests for telecommunications equipment; Part 1-4: Classification of environmental
conditions; Stationary use at non-weatherprotected locations".
[3] ISO 228-1: " Pipe threads where pressure-tight joints are not made on the threads --
Part 1: Dimensions, tolerances and designation".
[4] BS EN 805:2000: "Water supply. Requirements for systems and components outside buildings".
NOTE: Available at https://shop.bsigroup.com/ProductDetail/?pid=000000000019983094.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI TS 103 586 V1.1.1 (2019-04)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] CENELEC EN 60950-1: "Information technology equipment - Safety; Part 1: General
requirements".
[i.2] ETSI EN 300 019 (all parts): "Environmental Engineering (EE); Environmental conditions and
environmental tests for telecommunications equipment".
[i.3] IEC 62368-1: "Audio/video, information and communication technology equipment -
Part 1: Safety requirements".
[i.4] ETSI ES 203 474 (V1.1.1): "Environmental Engineering (EE); Interfacing of renewable energy or
distributed power sources to 400 VDC distribution systems powering Information and
Communication Technology (ICT) equipment".
NOTE: Available at http://portal.etsi.org/webapp/ewp/copy_file.asp?wki_id=43366.
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
cabinet: free-standing and self-supporting enclosure for housing electrical and/or electronic equipment
component: part or sub-part of an equipment that dissipates heat and needs to be cooled
Cooling Distribution Unit (CDU): unit used to separate or isolate the ICT equipment cooling loop from the facilities
cooling loop, consisting of a liquid to liquid heat exchanger with at least one pump, temperature and pressure controls
cooling efficiency: ability of a given cooling system to lower equipment temperature towards the cooling fluid
temperature
ICT equipment: information and communication equipment (e.g. switch, transmitter, router, server and peripheral
devices) used in telecommunication centres, data-centres and customer premises (see ETSI ES 203 474 [i.4])
NOTE 1: It is integrated in a rack or cabinet
NOTE 2: If the liquid cooling system is provided by the supplier, it will be considered herein that this system is part
of the ICT equipment. Thus, for an equipment with liquid cooling system at the component level (cold
plate), the boundary of the ICT equipment will be the rack/cabinet. For an ICT equipment cooled by a
rear door heat exchanger, the boundary of the ICT equipment will be the cabinet including the heat
exchanger. For a system cooled by immersion, the boundary will be the tank and its control system.
heat exchanger: device used to transfer heat from one fluid to another liquid cooling system
NOTE: System that controls or influence the temperature if a liquid in order to use it to cool component or
equipment or hot air issuing equipment.
pPUE: ratio between the energy consumption of the equipment plus the cooling system, divided by the energy
consumption of the cooling system alone
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TS 103 586 V1.1.1 (2019-04)
rack: free-standing or fixed structure for housing electrical and/or electronic equipment
3.2 Symbols
For the purposes of the present document, the following symbols apply:
Cp Specific heat (J/kg/°C)
dP Pressure drop (Pa)
P Electrical power consumed by the equipment (W)
Qm Mass flow rate (kg/s)
Qv Liquid volume flow rate (l/min)
3
ρ Liquid density (kg/m )
T Temperature (°C)
T Ambient Temperature surrounding the equipment (°C)
amb
T External Temperature outside the building or outdoor cabinet (°C)
ext
T Temperature at the inlet of the liquid cooling system, at the main liquid connector of the
in
equipment
Tout Temperature at the outlet of the liquid cooling system, at the main liquid connector of the
equipment
∆T Temperature difference (°C)
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
CDU Cooling Distribution Unit
CTE Coefficient of thermal expansion
HEX Heat exchanger
ICT Information and Communication Technology
IT Information technology
PCB Printed circuit board
PUE Power usage effectiveness
4 ICT equipment liquid cooling requirements and
energy efficiency
4.1 Introduction
In the present clause, the liquid cooling requirements and energy efficiencies of equipment are defined.
4.2 Cooling requirements for equipment
Liquid cooled equipment for non-weather protected locations shall be compliant with ETSI EN 300 019-1-4 [2] and
shall be compliant with any of the liquid inlet temperature class defined in table 1.
Liquid cooled equipment for weather protected locations shall be compliant with ETSI EN 300 019-1-3 [1] and shall be
compliant with any of the liquid inlet temperature class defined in table 1.
Table 1: Classes defining liquid inlet temperature range
and relevant minimum percentage of heat to water
Type of liquid cooling system Liquid inlet temperature range Minimum Percentage of heat to water
Rear door heat exchanger (A1) +10 °C to +25 °C 80 %
Cold plate at the component level (A2) +10 °C to +40°C 70 %
Immersion system (A3) +10 °C to +50 °C 80 %

ETSI

---------------------- Page: 7 ----------------------
8 ETSI TS 103 586 V1.1.1 (2019-04)
Liquid inlet temperature measurement T shall be considered at the position where operators shall provide liquid
liq-in
connection to the equipment.
Ratio of heat removed by liquid shall be computed with the following formula:
Heat ratio = Qm x Cp x (T - T ) / Total power dissipated by the equipment
liq liq liq_out liq-in
Liquid output temperature measurement T shall be considered at the position where operators provide connection
liq-out
for liquid return from the equipment.
If a CDU is provided, it shall be considered as a part of the equipment, and heat losses of this piece of equipment will
be taken into account.
Liquid cooling at the cabinet level is a technology that can lead to class A1 cooling performances (example is described
in clause B.1).
Liquid cooling at the component level is a technology that can lead to class A2 cooling performances (example is
described in clause B.2).
Liquid cooling by immersion is a technology that can lead to class A3 cooling performances (example is described in
clause B.3).
4.3 Liquid cooled equipment energy efficiency
Energy efficiency targets shall be measured in the following normal conditions:
• External (Outdoor) temperature T =45 °C.
ext
• Ambient (Room) temperature T = 25 °C.
amb
The equipment power consumption shall be considered at its maximal value.
The key indicator shall represent the impact of the cooling energy on the whole equipment energy consumption.
Partial PUE (Power Usage Effectiveness) can be used:
Equipment power consumption + Cooling energy consumption
=
Equipment power consumption

Cooling energy consumption shall take into account internal elements required to cool the equipment in the above
mentioned normal conditions (pumps, fans, control system).
Table 2: Energy efficiency classes
Cooling pPUE classes pPUE
Class B1 ≤ 1,01
Class B2 1,01 < pPUE ≤ 1,05
Class B3 1,05 < pPUE ≤ 1,10
Class B4 > 1,10

5 Specifications for liquid cooling solutions
5.1 General requirements
Liquid cooled equipment shall be compliant with ETSI EN 300 019-1-3 [1] or ETSI EN 300 019-1-4 [2] depending on
their locations.
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TS 103 586 V1.1.1 (2019-04)
5.2 Liquid flow rate range vs. dissipated power
Liquid flow rate (in l/min) and dissipated power are linked by the following steady state power balance:
 Q = (60 000 x P) / ( ρ x Cp x (T - T ))
V liq_out liq-in
5.3 Temperature of touchable parts
For safety purpose, the temperature of touchable part will be compliant with the applicable safety standards
(e.g. CENELEC EN 60950-1 [i.1] or IEC 62368-1 [i.3]).
5.4 Max pressure drop per liquid flow rate
Pressure drop per liquid flow rate shall not be higher than:
dP = 25 x Qv ²
liquid liquid
Qv is the liquid volume flow rate expressed in l/min.
5.5 Max pressure drop per air flow rate
If the cooling system consists in transferring heat from air flow to liquid flow (examples in clause B.1), pressure drop
per air flow rate shall not be higher than:
-6
dP = 1,3 x 10 x Qv ²
air air
Qv is the air volume flow rate expressed in l/min.
5.6 Pipe threads
If pipe threads are used, they shall be compliant with ISO 228-1 [3].
5.7 Coolants and cooling distribution units
If the equipment is cooled internally by another closed loop fluid than the liquid used at the room and building level
(e.g. oil, low pressure two phase fluid, very pure water), the supplier shall provide CDU(s) with at least N+1 system-
level pump redundancy to adapt to room where water cooling is available (where N is the number of pumps needed to
provide the nominal total flow rate).
The liquid cooling system shall not create hazard in terms of product safety. For this scope the relevant ICT safety
standards apply (e.g. IEC 62368-1 [i.3]). Liquid lifetime shall be at least 10 years, unless restrictions from National
Regulation apply.
5.8 Max pressure and tightness
The equipment shall be tested at a pressure level of three times of the nominal pressure with the method described in
BS EN 805 [4].
To ensure proper operation of the whole cooling system, commissioning shall be made at full load.
5.9 Liquid connectors positions
For equipment installed on raised floors, fluid connections shall be provided at the bottom of the rack/cabinet. For
equipment installed on slab floors, fluid connections shall be provided at either the top or the bottom of the rack/cabinet.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI TS 103 586 V1.1.1 (2019-04)
5.10 Accessibility in case of cooling with heat exchanger
If the HEX is not a part of the cabinet, it shall be easily moved to gain access to equipment for servicing. If the HEX is
a part of the cabinet (for example figure B.1c), it shall be easily removed or be mounted on a door to gain access to
equipment for servicing.
6 Benchmark methods to evaluate cooling system
efficiency and energy efficiency
To evaluate the cooling system, the equipment shall be installed in a climatic chamber with the ability to control the
ambient temperature with ±1 °C accuracy.
The following instrumentation is required.
Hydraulic
connectors
Tin
ICT Cooling
system
Equipment
Tout
F
Flowmeter
Energy
E

meter
E Energy

meter
Climatic room
Power cable
Power cable

Figure 1: Experimental setup to evaluate energy efficiency
Power dissipation shall be computed in the following way.
Main power measurement shall be performed with an energy meter for AC, and voltmeter and ammeter for DC,
measurements. Power values shall be based on supplier data if direct measurement is not possible. If the measured value
is not steady, a mean value over 5 minutes shall be computed.
Temperature of the liquid shall be measured at the input and output with a calibrated thermocouple whose junction will
be placed at the centre of the duct:
Thermocouple
Duct

Figure 2: Liquid temperature measurement
A flow meter with 5 % accuracy shall be used.
Ducts shall be thermally insulated.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI TS 103 586 V1.1.1 (2019-04)
A cooling system allowing control of liquid temperature at equipment inlet shall be used (examples: a chiller, an
external air to water heat exchanger, etc.).
For all the performed tests, cooling system ability to keep internal components at temperature below their limits shall be
check with relevant measurements.
Percentage of heat on water shall be computed, for liquid inlet temperatures at minimum value and at maximal value
(see table 1). Comparison shall be made with targets described in table 1, third column.
pPUE shall also be computed for each liquid inlet temperature value.
The above measurements shall be made at +25 °C of ambient temperature.
ETSI

---------------------- Page: 11 ----------------------
12 ETSI TS 103 586 V1.1.1 (2019-04)
Annex A (informative):
Cooling principles and impact on reliability and energy
consumption
A.1 Air cooling principles and limitations
Like with any other single phase cooling fluid, air cooling consists in exchanging heat from the sources, which are
electronic components to particles of fluid by convection, directly (component without heat sink for example) or
indirectly (component with heat sinks or integrated in a closed sheet for example).
Two situations can occur:
• Fluid movements due to particles temperature which leads to differences of density (i.e. natural convection).
These kinds of heat exchanges are only sufficient for low power components and low power density (density
as to be understood here at the equipment level).
• Fluid movements are induced by fans or blowers. Nearly all ICT equipment use this cooling technique called
forced convection. Most of the time, heat sinks or heat pipes are necessary to increase heat exchange between
air and components.
To understand limitations of air as a cooling fluid, thermo-physical properties are detailed and explained.
The air has the following thermal properties:
• Specific heat: C = 1 005 J/kg/°C @20 °C
air
• Thermal conductivity: k = 0,0257 W/m/°C @20 °C (air is an efficient heat insulator)
air
3
• Density: ρair = 1,205 kg/m @20 °C
To compare, the properties of water are:
• Specific heat: C = 4 183 J/kg/°C @20 °C
air
• Thermal conductivity: k = 0,58 W/m/°C @20 °C
air
3
• Density: ρ = 1 000 kg/m @20 °C
air
3
The product ρ C indicates the energy stored by 1 m when the temperature rise is 1 °C:
3 3
• For air: ρ C = 1 211,10 J/m
air air
6 3
• For water: ρ .C = 4 183,10 J/m
water water
3
These physical data indicate that air is not efficient for storing and transporting heat (liquids have 10 higher heat
capacity compared with gases). To cool high power systems, huge air flow rates are therefore needed, that leads to high
energy consumption (and high acoustic noise disturbances).
The thermal conductivity of air (which is a well-known thermal insulator) is also 22 times lower.
Despite these physical constraints, air cooling has been and is still widely used in electronics but in some cases,
efficiency cannot be achieved as the power density per square meter is too high (more than 20 kw/m²).
Finally, as a result, heat transfer coefficients are at least ten times higher with fluids. As a consequence, to cool high
heat densities, air cooling will require high flow rates and much lower fluid temperature. The consequences are high
cooling energy consumption, and huge acoustic noise. Thus, to improve ICT energy efficiency, an obvious way is to
switch from air cooling to liquid cooling.
ETSI

---------------------- Page: 12 ----------------------
13 ETSI TS 103 586 V1.1.1 (2019-04)
Water is only used herein as an example. Liquid cooling solutions can be developed with several fluids, among which:
• Water.
• Water mixed with antifreeze and other additives.
• Dielectric fluids (oils, phase change solutions).
• Refrigerants.
A.2 Reliability issues
ICT equipment reliability is linked with local (near the component) operating temperature and its temporal variations.
As components power densities increas
...