Tier 3 Data Center Cooling System Design

THEODOSIS MOUMIADIS FEBRUARY 15, 2019

Photo: T.Moumiadis, some rights reserved

Following the series of posts related to the Data Centers Cooling system design and based
on the operation requirements developed and described in previous post, let's now dive into
design details of a Tier III rated system according to Uptime Institute requirements.

The HVAC design of datacenters that follow the above requirements can be managed by
assuring design, construction and operation compliance with one of the most industry
recognized standards.

This is the Uptime Institute Tier classification.

The Tier classification of a data center can take four rates, namely:

Tier I, Tier II, Tier III or Tier IV.

The Tier class can be unique for a data center and describes criteria to differentiate four
classifications of site infrastructure topology based on increasing levels (as numbers
increase I to IV) of redundant capacity components and distribution paths.

Scope of this article is to give the HVAC design engineer initial guidelines on how the
topology of the DC cooling system should be structured in order to comply with the UI Tier
standards requirements. The standard of reference is Tier Standard: Topology [2] which can
be requested for download from here.

Carry on with reading this post if interested on a Tier III data center requirements or
check Tier I & II Cooling System design or Tier IV Cooling System design respectively.

Reference Standards and regulations

Guidelines, requirements and design criteria in use below are in accordance with the
following references:
 ASHRAE – Design Considerations for Datacom Equipment centers / 2 nd ed. 2009,
 Uptime Institute – Data Center Site Infrastructure, Tier Standard: Topology / 2018,

Tier III – Design Criteria

According to Tier Standard: Topology [2] / clause 2.3.1, the fundamental requirements for a
Tier III – Concurrently Maintainable Site Infrastructure are:

(a) A Concurrently Maintainable data center has redundant capacity components and
multiple independent distribution paths serving the critical environment. For the electrical
power backbone and mechanical distribution path, only one distribution path is required to
serve the critical environment at any time.
The electrical power backbone is defined as the electrical power distribution path from the
output of the on-site power production system (e.g., engine generator) to the input of the IT
UPS and the power distribution path that serves the critical mechanical equipment.
The mechanical distribution path is the distribution path for moving heat from the critical
space to the outdoor environment. For example:

 Chilled water piping,

 Condenser water piping,
 Refrigerant piping.

(b) All IT equipment is dual powered and installed properly to be compatible with the
topology of the site’s architecture. Transfer devices, such as point-of-use switches, must be
incorporated for critical environment that does not meet this requirement.

(c) Twelve hours of on-site fuel storage for ‘N’ capacity.

A few important points

Before dive into details of the cooling system design requirements in compliance to Tier
criteria, keep first in mind a few more important points from the aforementioned standard
[2].

 Every data center subsystem and system must be consistently deployed with
the same site uptime objective to satisfy the distinctive Tier requirements.

This standard requirement makes clear that certain Tier rating requirements shall be
applicable to all mechanical, electrical and building systems that serve the IT space. So
whatever we will discuss below about the cooling system for a Tier III data center design
are equally applicable and essential for the on-site power production, UPS and storage
equipment, fuel tanks and water storage (evaporative cooling) systems as well.

 The Tier topology rating for an entire site is constrained by the rating of the
weakest subsystem that will impact site operation. For example, a site with a
robust Tier IV UPS configuration combined with a Tier II chilled water system
yield a Tier II site rating.

So even if the team is quite careful and especially concerned about the cooling system
design, but fail to implement the same criteria into the electrical or fuel supply systems, the
overall rating of the data center will be lower than the expected.

Design of Tier III Cooling system

The cooling system design of a Tier III rated data center shall comply with the following
requirements:

Redundant Capacity components – meaning that there is need for redundancy (backup)
of any equipment of the cooling system including and not limited to components like:

 CRAC / CRAH units,

 Chillers, chilled water pumps,
 Cooling towers, condensing pumps,
 AHUs
 Split type DX cooling units,
 Makeup water storage tanks and pumps.

This is what we call “N+1” units configuration or in simple terms the total capacity of all
equipment units to be used is equal to the datacenter cooling demand plus at least one
more unit.

Multiple independent distribution paths – meaning that the following distribution

networks:

 Chilled water piping,

 Condensing water piping,
 Refrigerant copper piping for DX cooling systems,
 Makeup water piping,
shall be designed in a way that the transferred medium (water, fuel, refrigerant) is able to
reach any system unit (CRAC, AHU, Chiller, Cooling tower) from multiple paths. This should
not be confused in any way to the well known ‘loop’ system design. Multiple independent
distribution paths shall be deployed through two chilled water systems where all the terminal
units are connected.

Refer to fig. 01 below where a ‘single distribution’ and a ‘redundant distribution’ chilled water
piping networks serving the same computer installation are compared.

Fig.01 – the illustrations describe the case of a chilled water system

supplying N+1 CRAC units through a (a) two pipe chilled water system
constitute a single non-redundant distribution path and (b) four pipe
chilled water system, which actually is a group of two independent
distribution paths, always one active at a time. Case (b) is a piping
system that can be part of a Tier III cooling installation.

Concurrently Maintainable – meaning Each and Every capacity component (CRAC,

AHU, Chiller, Cooling Tower, pump) and distribution path element (valves, filters, check
valves, regulating valves, meters, instruments, sensors), can be taken out of service for
maintenance, repair or replacement without impacting the Critical Environment or IT
processes.

 A Tier III Concurrently Maintainable data center is an easily and always possible
to expand, without disruption, data center.
A concurrently maintainable cooling system shall be designed so that:

All the capacity components are redundant through an N+1 configuration. So that if for
example the cooling demand of an IT space is 300kW, the implementation of the system
shall include 6 CRAC units of 60kW each in a 5+1 arrangement. There will always be one
unit redundant. The same happens for AHUs, pumps, chillers cooling towers etc.

On the other hand, the piping network shall has all the required accessories so that each
and every part or fitting will be possible to get isolated and removed or maintained without
affecting the chilled water flow to the rest. The key point to design a concurrently
maintainable piping system is the selection of valves positioning. Let’s explore together of
how isolation valves shall be provided in a water piping system in order to achieve
concurrent maintainability.

Consider the pumps system of Fig.02. These pumps serve a piping network where the flow
demand is 200m3/h. The design provides a set of three pumps each one sized with a
maximum flow capacity of 100m 3/h. It comes out that we have a 2+1 configuration where
two pumps continually run and the third one is backup. According to Tier 3 requirements,
the pump as a capacity component is redundant. Also the concurrent maintainability of each
and every component in the system is achieved by providing two valves in the piping
section between each pump.

In that case for example assume that valve A is broken and has to be maintained or even
removed. The team has only to turn off valves B, C and D and without any disruption to the
data center cooling system, that element (valve A) can easily be maintained or replaced.
The same happens with all the valves and elements of this arrangement. Make your checks
and confirm the concept! Use this arrangement on any of your future pump sets design.
  Fig.02 – A set of three pumps in an N+1 configuration that serve a
redundant distribution path (water flow and return is served from both
sides). The provision of two valves between each section assures a
concurrently maintainable system. A pump system arrangement which is
fully compliable with Tier 3 requirements and is applicable to
chilled, condense and make up water systems.

The same concept can also be extended to any piping system as shown in Fig.03 below.
The arrangement of valves is in a way that any part of the system can be isolated and
maintained, without any downtime of the cooling system. Again try to check each and every
component and element and you will realize that there is always a set of valves that can
achieve isolation of that component or element. Be careful with the example since it is not a
Tier 3 system. The piping network cannot be considered as a multiple independent
distribution path. It is just a concurrently maintainable system.
Fig.03 – Provision of isolation valves in a single distribution
piping network (loop design). They are arranged in a way that
allows concurrent maintainability of each and every capacity
component or system element.

How to apply these requirements into the design

These requirements can be applied in several cooling type systems like shown in the
schematic representations below. A Tier III cooling system can use split type air
conditioning systems with refrigerant (Fig.04), or a water type system with CRAC units
inside the IT space (Fig.05) or even more advanced solutions with chilled water and air
handling units (AHU) that blow air within the IT spaces (Fig.06).
These systems configurations are quite common in the data center industry where
redundancy of the 'capacity components' combined with the proper design of distribution
paths achieve concurrent maintainability and system reliability that gives the system Tier III
rate.

For your understanding explore the schematics below and try to identify all the principles
discussed before related to capacity components, distribution paths and system
maintainability.

Fig. 04 - A Tier III split system CRAC units developed with DX type
systems make use of 'N+1' number of CRAC units in a configuration
of N units running and at least 1 redundant. The respective
external condensing units and refrigerant pipes all together
combine a system.
Fig. 05 - A Tier III water system CRAC unit contain single or more
running 'capacity components' all in a 'N+1' configuration. This
includes a combination of water chiller, cooling tower, water pumps
all together running at the same time and supplying chilled water
to 'N' number of running CRAC units. Also the system has at least
one more piece of redundant component for every type of capacity
equipment. So there is a redundant water chiller, cooling tower,
pumps and CRAC unit. Two 2-pipe chilled water systems connect all
the chiller and CRAC units as 'redundant distribution paths'. There
are provisions of valves in a way that two independent piping
networks serve the system in a way that only one is active. Each
and every component or element of the system can be maintained or
replaced without system shutdown.

Fig. 06 - A Tier III water system Air Handling unit  contain single

or more running 'capacity components' all in a 'N+1' configuration.
This includes a combination of water chiller, cooling tower, water
pumps all together running at the same time and supplying chilled
water to 'N' number of running Air Handling units. Also the system
has at least one more piece of redundant component for every type
of capacity equipment. So there is a redundant water chiller,
cooling tower, pumps and AHU. Two 2-pipe chilled water
systems connect all the chiller and CRAC units as 'redundant
distribution paths'. There are provisions of valves in a way that
two independent piping networks serve the system in a way that only
one is active. Each and every component or element of the system
can be maintained or replaced without system shutdown.
Note that in case of water systems there are also requirements for the makeup water
installation and the storage tanks. 

In general a Tier III data center allows all capacity components to be maintained or repaired with
disrupting the operation of the critical environment. The same happens for the distribution paths
(piping) and individual elements (fittings, valves and accessories). The operational risk that this
configuration brings to the computer facility owner is limited but there still exist a risk of disruption
during the scheduled maintenance on redundant components.