WHITE PAPER #18

ASHRAE DATACOM BOOK

SERIES

EDITORS:
DR. MICHAEL PATTERSON, INTEL
RICH JONES, CHATSWORTH PRODUCTS

CONTRIBUTORS:
JOHN BEAN, APC
ROBB JONES, CHATSWORTH PRODUCTS
JOHN WALLERICH, DELL
ROD BEDNAR, EMERSON
WADE VINSON, HEWLETT-PACKARD
PHIL MORRIS, SUN MICROSYSTEMS

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 2

TABLE OF CONTENTS
INTRODUCTION...................................................................................................................................................3
BOOK 1: THERMAL GUIDELINES FOR DATA PROCESSING ENVIRONMENTS..................................................5
BOOK 2: DATACOM EQUIPMENT POWER TRENDS AND COOLING APPLICATIONS.......................................9
BOOK 3: DESIGN CONSIDERATIONS FOR DATACOM EQUIPMENT CENTERS.............................................12
BOOK 4: LIQUID COOLING GUIDELINES FOR DATACOM EQUIPMENT CENTERS.......................................17
BOOK 5: BEST PRACTICES FOR DATACOM FACILITY ENERGY EFFICIENCY................................................19
BOOK 6: STRUCTURAL AND VIBRATION GUIDELINES FOR DATACOM EQUIPMENT CENTERS.................21
BOOK 7: HIGH DENSITY DATA CENTERS - CASE STUDIES AND BEST PRACTICES...................................23

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 3

INTRODUCTION
The IT industry continues to experience higher and higher electronic equipment power densities, resulting
in a significant impact on facility cooling infrastructures. In addition, the mission-critical nature of today’s
computing environment has sensitized businesses to the importance of maintaining the long-term health
of their data centers. These combined effects have made it imperative that data center managers, facilities
operators, and equipment manufacturers be well-aligned to ensure efficient, effective, and reliable operation
of mission-critical computing environments.

The American Society of Heating, Refrigerating, and Air Conditioning Engineers (ASHRAE) Technical
Committee (TC) 9.9, “Mission-Critical Facilities, Technology Spaces, and Electronic Equipment,” has published
a series of environmental books for datacom applications. These books bridge the technology gaps between
IT and facilities, providing a holistic view of data centers and insight into solutions that address current and
future issues. The mission of TC 9.9 is “To be recognized by all areas of the datacom industry as the unbiased
engineering leader in the subject of HVAC and an effective provider of technical information.” The members of
TC 9.9 consist of manufacturers, consultants, and end users from the datacom industry.

The Green Grid’s Technology and Strategy Work Group established its Thermal Management Task Force to
review and summarize recent ASHRAE TC 9.9 books.

This paper presents summaries of the following seven ASHRAE books, in order of publication date:
1. 90431 Thermal Guidelines for Data Processing Environments (2004)
2. 90427 Datacom Equipment Power Trends and Cooling Applications (2005)
3. 90437 Design Considerations for Datacom Equipment Centers (2005)
4. 90430 Liquid Cooling Guidelines for Datacom Equipment Centers (2006)
5. 90438 Best Practices for Datacom Facility Energy Efficiency (2008)
6. 90429 Structural and Vibration Guidelines for Datacom Equipment Centers (2008)
7. 90441 High Density Data Centers - Case Studies and Best Practices (2008)

The Green Grid intends to review and publish summaries on additional ASHRAE Datacom series books as
they become available.

The intent of this paper is to introduce these important publications to IT management, beyond the traditional
facilities audience, and provide a summary overview to enable the reader to efficiently locate the book(s) of
interest. It is important to note that The Green Grid does not necessarily endorse the content of the ASHRAE
publications.

ASHRAE offers book price discounts for volume purchases. The following is the discount schedule that
organizations can use to purchase the book in bulk quantities:

Quantity Discount
Copies 1-9 No discount
Copies 10-99 50% off list price
Copies 100-499 55% off list price
Copies 500 plus 60% off list price

These discounts are for one title only, not a combination of titles in the series.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 4

LOGO IMPRINT PRICING:

Books can be personalized with a logo on the cover. An organization also could choose to imprint something
like “Compliments of XYZ Company” or add more detailed information such as the company’s contact
information. For imprinting, ASHRAE requires a minimum order of 100 copies and a $200 setup fee on orders
of fewer than 500 copies. For orders of 500 copies or more, ASHRAE will provide a requested logo at no cost.

ASHRAE CONTACT:
If organizations wish to place bulk orders at the special discount prices, they should contact Kimberly Gates
in the Publications and Education Department at ASHRAE. Her phone number is 404-636-8400, or you can
send email to: [email protected].

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 5

I. BOOK 1: THERMAL GUIDELINES FOR DATA

PROCESSING ENVIRONMENTS
CHAPTER 1: This book presents guidelines
developed for data center facility operators
and equipment manufacturers on common
environmental conditions. Included are
suggested methods for evaluating and
testing the operational health of the data
center and the environmental characteristics
of a computer system. ASHRAE developed
the book because traditional processes for
sizing cooling resources were not keeping
up with environmental needs for IT reliability
given the increasing IT power density. The
book also is meant to address the significant
amount of over-sizing and over-cooling that
takes place due to misunderstanding the
environmental conditions needed by the
equipment. Current management methods Title Thermal Guidelines for Data
also were forcing cooling resource capital Processing Environments
and operational expenditures to outpace IT Page Count 62
expenditures. Date 2004
Price $43
CHAPTER 2: EQUIPMENT ENVIRONMENT Part numbers 90431 (printed) D-90431
SPECIFICATIONS (electronic - PDF)

Setting the same operating parameters for all types of IT equipment drives facility capital and operational
expenses. This chapter seeks to define various classes of IT and their recommended and allowable humidity
and temperature conditions, with data centers typically categorized as Class 1.

CLASS DEFINITIONS: (Source: ASHRAE-D-94037-20080606)

•Class 1 – Typically a datacom facility with tightly controlled environmental parameters (dew point,
temperature, and relative humidity) and mission-critical operations. Product types typically
designed for this environment include enterprise servers and storage products.
•Class 2 – Typically a datacom space or office or lab environment with some control of
environmental parameters (dew point, temperature, and relative humidity). Product types
typically designed for this environment include small servers, storage products, personal
computers, and workstations.
• Class 3 – Typically an office, home, or transportable environment with little control of
environmental parameters (temperature only). Product types typically designed for this environment
include personal computers, workstations, laptops, and printers.
• Class 4 – Typically a point-of-sale of light industrial or factory environment with weather
protection, sufficient winter heating, and ventilation. Product types typically designed for this
environment include point-of-sale equipment, ruggedized controllers or computers, and PDAs.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 6

• NEBS – According to Telcordia Technologies “GR-63-CORE” (Issue 2, April 2002) and “GR
3028-CORE” (Issue 1, December 2001), typically a telecommunications central office with some
control of environmental parameters (dew point, temperature, and relative humidity). Product types
typically designed for this environment include switches, transport equipment, and routers.

Since Class 3 and 4 environments are primarily not designed for datacom equipment, they are not covered
further in this chapter. Please refer to ASHRAE’s Thermal Guidelines book for further information (ASHRAE
2004h).

Table 2.1 below from the book is a specifications overview, and the book comes with a quick reference card
carrying the same information. Chapter 2 also describes ratings for altitude and discusses acoustic concerns.
The appendices show graphical representations of temperature, humidity, and altitude comparisons of the
various classes. Major IT manufacturers and ASHRAE have agreed on the operating limits, measured at the
equipment inlet. Table 2.1 data was updated at the ASHRAE Summer Annual Meeting in Salt Lake City in June
2008. The new data is shown in Table 2.1a.

ASHRAE Thermal Guidelines for Data Processing Environments

Table 2.1 Equipment Environment Specifications
Class Allowable Recommended Dry Allowable % Recommended %
Dry Bulb (°C) Bulb (°C) Relative Humidity Relative Humidity

1 15 to 32 20 to 25 20 to 80 40 to 55
2 10 to 35 20 to 25 20 to 80 40 to 55
3 5 to 35 NA 8 to 80 NA
4 5 to 40 NA 8 to 80 NA
Max. Altitude = 10,000 feet with dry bulb de-rating applied in some cases.
Courtesy of ASHRAE

TABLE 2.1: ORIGINAL (2004) FACILITY TEMPERATURE AND HUMIDITY MEASUREMENTS ©2004 ASHRAE

Note: ASHRAE has published an update to the operating ranges described in Table 2.1 that can be found at:
http://tc99.ashraetcs.org/documents/ASHRAE_Extended_Environmental_Envelope_Final_Aug_1_2008.pdf

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 7

Key changes in the revised table below are shown in bold.

Class Allowable Recommended Dry Allowable % Recommended % Relative
Dry Bulb (°C) Bulb (°C) Relative Humidity Humidity

1 15 to 32 18 TO 27 20 to 80 5.5 °C DEW POINT TO

15 °C DEW POINT AND
60% RH
2 10 to 35 20 to 25 20 to 80 40 to 55
3 5 to 35 NA 8 to 80 NA
4 5 to 40 NA 8 to 80 NA
Max. Altitude = 10,000 feet with dry bulb de-rating applied in some cases.
Courtesy of ASHRAE

TABLE 2.1A: NEW RECOMMENDED ENVIRONMENTAL GUIDELINES, UPDATED AT THE ASHRAE SUMMER
ANNUAL MEETING IN SALT LAKE CITY IN JUNE 2008 AND APPROVED BY ASHRAE

The objective of the specifications above is to ensure that a facility’s aisle temperature and humidity levels all
are being maintained within the operating conditions recommended for its environment class. Any equipment
inlet temperature and/or humidity measurements found to be outside the recommended operating range for
the class environment should be investigated and corrected.

Even though facility cooling may be sized to accommodate larger than the total IT thermal load, often some of
the IT devices report thermal issues due to local temperature variation in the data center. Chapter 3 defines
temperature locations and humidity measurements—both in IT and at the facility level—to help troubleshoot
this issue. (See the book’s Figure 3.2 below.)

FIGURE 3.2: TEMPERATURE AND HUMIDITY MEASUREMENT LOCATIONS ©2004 ASHRAE

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 8

CHAPTER 4: EQUIPMENT PLACEMENT AND AIRFLOW PATTERNS

An imbalance in airflow throughout the space can cause additional temperature problems, even when facility
cooling equipment is appropriately sized for the thermal load. This chapter provides layout and protocol
suggestions for IT equipment cabinet airflow. It also suggests best practices for facility layout of IT aisles and
air conditioners/handlers, including cabinets with dissimilar airflow patterns, to optimize airflow effectiveness.

CHAPTER 5: EQUIPMENT MANUFACTURERS HEAT AND AIRFLOW REPORTING

This section provides guidelines for IT reporting of power and the resulting heat load and airflow. It is critical
that facility designers size mechanical and electrical systems for actual and future environmental set points.
The chapter suggests a method of obtaining data sheets with typical minimum and full load information from
the IT supplier. This is in contrast to the common but inaccurate method of using power supply name plate
ratings to determine requirements. A sample of a report based on a manufacturer data sheet is shown below
in Figure 3.3 from the book.

FIGURE 3.3: TYPICAL MANUFACTURER’S HEAT AND AIRFLOW TABLE ©2004 ASHRAE

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 9

II. BOOK 2: DATACOM EQUIPMENT POWER TRENDS

AND COOLING APPLICATIONS

A critical aspect of designing and maintaining

all data centers is the ability to efficiently
and effectively plan the IT equipment power
and cooling requirements—both for today
and for growth over time. The book provides
datacom equipment power trend charts that
give IT staff, facilities, architects, engineers,
planners, and operations managers an improved
means of forecasting, designing, maintaining,
and provisioning new and existing datacom
equipment load requirements. It also provides
trends to aid in predicting growth in power costs
due to increased infrastructure load.

Careful planning of datacom power and cooling Title Datacom Equipment Power
facilities is needed because they have a much Trends and Cooling Applications
longer product life cycle than the computer Page Count 122
equipment they support. Datacom/IT equipment Date 2005
typically has a product life cycle of 3 to 5 Price $38
years, whereas a building’s cooling and power Part numbers 90427 (printed) D-90427
distribution systems can last 10 to 25 years. (electronic - PDF)

This discrepancy can lead to inefficiencies due to over- or under-design capacity. Therefore, accurate
capacity forecasting is essential. The design and operation of a building’s power and cooling systems must
accommodate multiple computer equipment deployments and prevent costly and disruptive infrastructure
upgrades, premature obsolescence, and inefficient datacom facility operation.

In 2000, as defined by the Thermal Management Consortium, the Uptime Institute published the broadly
referenced datacom equipment power density trend chart that covers 1992 through 2010. Since that time,
ASHRAE has identified some misapplications of the chart, further clarified some of the chart’s assumptions
and applications, and extended the projections to 2014.

The following is an exemplary chart from the book that projects to 2014 the typical heat load in KW per
rack—based on a 7-square-foot rack—for extreme-density communication equipment, compute servers (1U
blade and custom, high-density communication compute servers along with 2U and greater compute servers),
storage servers, workstations (standalone), and tape storage devices. ASHRAE also felt that new form factor
computers showed higher power densities than more traditional computers. Similarly, communications
equipment branched into two trend lines between high and extreme densities.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 10

FIGURE B-4: NEW POWER TREND CHART (NON-LOG SCALE, KW PER RACK), UPDATED AND EXPANDED BY
ASHRAE ©2005 ASHRAE

In addition to the charts, the book contains key aspects of facility planning. It stresses the importance of
strong collaboration between the building facilities and information technology personnel and of provisioning
for future loads to include system growth components and necessary spatial allocation.

CHAPTER 1: INTRODUCTION
This chapter describes the book’s purpose, which is to provide power trends for datacom (data center
and telecommunication) equipment, describe how those trends affect critical decisions on facility cooling
system infrastructure requirements, and highlight the importance of collaboration between IT and facilities
personnel.

CHAPTER 2: BACKGROUND
This section describes the five key planning aspects of a datacom facility, with an example that illustrates how
the reader can implement the planning process. It also includes a discussion on the use of a power density
metric.

CHAPTER 3: LOAD TRENDS AND THEIR APPLICATION

Chapter 3 contains updated datacom equipment power information, historical and projected power
dissipation trend charts for various classes of equipment, a trend evolution overview grouped by various
datacom equipment types, and a discussion regarding how to apply the load trend charts for capacity and
provisioning when planning a new facility.

CHAPTER 4: AIR COOLING OF COMPUTER EQUIPMENT

This section provides a definition and overview of the various air cooling systems, including underfloor,
overhead distribution, managed supply and return airflow, and local distribution configurations and
equipment.

CHAPTER 5: LIQUID COOLING OF COMPUTER EQUIPMENT

Chapter 5 gives a definition and overview of the various liquid cooling systems associated at the cabinet

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 11

level. It discusses the reasons behind the re-emergence of liquid cooling and supplies details on the types of
liquid used for enhanced heat transfer. The chapter intentionally excludes component and board-level cooling
technologies. The three basic transport liquid strategies discussed in the chapter are:
• Heat rejection by air cooling the heat transport liquid from the electronic equipment
• Heat rejection by liquid cooling the heat transport liquid from the electronic equipment
• Heat rejection by extending the heat transport liquid from the electronic equipment to a remote
location far from the rack/cabinet

This chapter also describes the three types of liquid cooling loops and some of the factors that must be
considered when choosing a cooling methodology.

APPENDICES: The book includes a collection of useful appendices to supplement the chapters.
• Appendix A – Collection of Terms contains a useful list of standard industry terms and high-level
definitions for easy reference.
• Appendix B – Additional Trend Chart Information/Data contains the trend charts in SI units.
It also offers versions of the trend charts without logarithmic scales for power density to provide a
better understanding of the loads escalation. Also included are trend values in a tabular version and
kW per rack and W/ft2 trend graphs.
• Appendix C – Electronics, Semiconductors, Microprocessors, International Roadmap for
Semiconductors provides some semiconductor industry history as well as the semiconductor ITRS
roadmap information.
• Appendix D – A Micro-Macro Overview of Datacom Equipment Packaging provides a high-level
graphical overview of datacom packaging. It also includes a glossary that clarifies terms used by
both the data processing and telecommunications industries.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 12

III. BOOK 3: DESIGN CONSIDERATIONS FOR DATACOM

EQUIPMENT CENTERS
CHAPTER 1: INTRODUCTION
For the purposes of this book, “datacom” refers to
the data and communications industry, whereas
“data center” refers to the portion of a building that
is dedicated to housing data and communications
equipment. Datacom equipment centers are unique
in terms of their increased power and thermal
densities and with regard to their high-availability
requirements. They can be several times greater than
the specifications for office buildings and equipment
support spaces. In addition, these environments
often require special environmental conditions such
as humidification and air purification.

This book is not meant to be a comprehensive guide

to datacom data center design concepts. Rather, it Title Design Considerations for

provides an augmentation to existing data center Datacom Equipment Centers

design standards in order to achieve the basic Page Count 208

standards of datacom equipment centers. Date 2005
Price $38
CHAPTER 2: DESIGN CRITERIA Part numbers 90437 (printed) D-90437
The primary focus of this chapter is to cover all of (electronic - PDF)
the environmental features, such as temperature, humidity, human comfort issues, and air filtration.

A consortium of equipment manufacturers in the industry has established an agreed-upon set of

standardized conditions (Classes 1-4). A definition of these classes can be found in the ASHRAE publication,
Thermal Guidelines of Data Processing Environments (ASHRAE 2004h). Further classifications are defined in
the Network Equipment-Building Systems (NEBS). NEBS-3 is the most current standard.

The rest of Chapter 2 contains valuable graphs and charts that identify operational factors and conditions
for datacom centers based on altitude and environmental conditions that include humidification, filtration
of contaminants, ventilation, vapor containment, pressurization, materials usage, and human safety and
comfort factors.

CHAPTER 3: HVAC LOAD CONSIDERATIONS

This chapter focuses on high-density power and thermal loads, with specifications that allow for electrical
loads of 30kW+/rack. Given the dynamic nature of load calculations, the publication Datacom Equipment
Power Trends and Cooling Applications (ASHRAE 2005i) serves as a useful guide. Specific equipment such as
tape storage, workstations, storage, and compute/server load estimates is provided.

Chapter 3 covers heat implications for lighting, people, envelope (exposure to radiant heat from outside
sources), and transmission heat gain. It also addresses the use of reheat coils to control humidification.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 13

CHAPTER 4: COMPUTER ROOM COOLING OVERVIEW

This chapter covers the basics for cooling infrastructure and designs, yet does not provide specific design
recommendations due to the wide variety of computer room designs and applications. It reviews Computer
Room Air Conditioner (CRAC) and cooling types (Direct Expansion [DX)] versus chilled water) and discusses
CRAC locations in general, yet leaves them to computational fluid dynamics (CFD) analysis for detailed
designs.

Chapter 4 analyzes the use of variable-speed fans, also referred to as variable-air-volume (VAV) equipment,
to minimize operating costs through reduced fan power consumption at less than full design loads. Such
equipment requires careful CFD analysis to apply properly.

Chilled water distribution systems remain the most common method of cooling design because they provide
an efficient mechanism for heat removal and also allow for highly efficient heat rejection systems that utilize
cooling water towers and/or wet-side economizers.

This chapter also goes into more detail on chiller design, piping, humidification systems, controls, and
monitoring.

CHAPTER 5: AIR DISTRIBUTION

Air distribution is perhaps one of the least-understood areas of data center design. Yet it can have a
significant impact on operational efficiency, energy consumption, and cost effectiveness. Due to increased
equipment densities, more sophisticated airflow designs are critical and require advanced CFD tools and
knowledge. OEMs publish recommended temperatures and humidity standards, and local recommendations
are published in the telecom central offices (Telcordia 2001) and data centers (ASHRAE 2004h).

Chapter 5 discusses basic principles of airflow dynamics, such as the implementation of a hot-aisle/cold-
aisle configuration. It also covers various rack flow designs and includes specific recommendations on the
placement of floor tiles and CRACs and ways to eliminate floor leakage of the cold air supply. The chapter
makes recommendations regarding the use of various perforated tile types.

This chapter also compares vertical under floor (VUF) cold air delivery designs to vertical overhead (VOF)
designs. Each design has its unique strengths, weaknesses, and limitations. Supplemental cooling options
are discussed in more detail in Chapter 4 of Datacom Equipment Power Trends and Applications (ASHRAE
2005i). Supplemental systems can be very useful when higher cooling requirements are needed for specific
racks or areas of a data center.

CHAPTER 6: LIQUID COOLING

As mentioned, virtually all equipment is cooled with cold air. However, the means by which heat energy is
rejected can differ. In air cooling, conditioned air supplied to rack and equipment inlets uses convection
cooling to remove heat from the components within the rack. Liquid cooling utilizes conditioned liquids that
are fed to the rack or server to remove heat directly from the rack components. The actual heat removal
from the CPU or other components could still be via cold air, but the heat-rejection medium to the building’s
cooling infrastructure outside the rack is liquid.

Chapter 6 also discusses the thermal properties of water and refrigerants in more detail, including reliability,
availability, and operational cost implications.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 14

CHAPTER 7: ANCILLARY SPACES

Within any data center design, the computer room floor is only one of several areas required for normal data
center operations. Other areas include UPS room, electro-mechanical rooms, battery room, network entrance
room, network operations center (NOC), equipment test and burn-in room, possible rooms for generators, and
storage spaces.

CHAPTER 8: CONTAMINATION
This chapter addresses the sources of particulate matter and other sources of airborne contaminates that
can affect performance and/or mean time before failure (MTBF). These can include anything from dust to
zinc whiskers to chemical contaminants such as gasses and pollutants.

Also covered are the basic characteristics of corrosive gases and more detail regarding methods of removal
and risk mitigation for each contaminant type. The chapter also provides valuable information on how
to detect various contaminants along with Telcordia GR-63-CORE limits for each chemical. The types of
contaminants included are not uncommon, and air quality tests should be a standard part of data center and
facilities operations.

Chapter 8 does a good job covering the basic data center design components of the suppression systems
and designs for fire detection; raised-floor designs; component selection for ceiling panels, floor tiles,
temperature, and humidity controls; air filtration systems and standards; pressurization; IT equipment
installation; operational strategies; and site surveys.

CHAPTER 9: ACOUSTICAL NOISE EMISSIONS

Noise emissions are regulated by OSHA, with different limits applied to different components in a data
center. Until recently, noise levels were not a concern, but with increased power loads, the increased cooling
requirements have driven noise levels up, primarily as a result of fan noise. For guidelines, refer to the
“Sound and Vibration” chapter of the ASHRAE Fundamentals (ASHRAE 2005c). For more information, refer to
ISO 7574, ISO 7779, and ISO 9296 standards. OSHA limits are defined in the USDOL 1991 document and the
EU Directives (EU 2003). Basically, sound pressure levels are not to exceed 90dB(A) for the US and 87dB(A)
for the EU. Most datacom providers set limits closer to 80db(A).

CHAPTER 10: STRUCTURAL AND SEISMIC

One of the major issues for high-density computing environments, including datacom equipment centers, is
weight distribution for floors above ground floor. Chapter 10 provides examples that show how rack loads are
distributed, although professional engineering studies are recommended.

The chapter does a good job covering the four basic load factors: concentrated load, uniform load, ultimate
load, and rolling load. These factors affect building floor loads and floor tile loading. In addition, it covers the
pedestal design element. Lateral stresses are addressed under the seismic loading section, but it should
be mentioned that the use of stringers is not optional: a lack of stringers can result in equipment damage if
enough consecutive tiles are removed.

CHAPTER 11: FIRE DETECTION AND SUPPRESSION

This section defines and discusses various types of sensors, including heat sensors, radiant-energy sensors,
and smoke sensors. In addition, it outlines the various spaces and zones along with assorted interfaces to

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 15

the building maintenance system (BMS) and alert systems. Fire suppression systems have evolved over time
to include carbon dioxide (CO2) systems, which minimize residuals that may affect equipment. Chapter 11
describes and assesses FM-200 and HFC-227. A more advanced extinguishing system is Inergen, designed
to suppress several types of flammable sources. Water-based (wet pipe and dry pipe) systems are discussed,
as well as deluge and pre-action sprinkler systems. The chapter also covers a design called water mist
(depending on the age and level of contaminants and particulate matter), which uses dialectic fluids that
have non-conductive electrical properties.

CHAPTER 12: COMMISSIONING

Commissioning is a process of load testing mission-critical systems following either data center construction
or installation of a major component or subsystem. Refer to the 2003 ASHRAE Handbook – HVAC
Applications, chapter 42, “New Building Commissioning” (ASHRAE 2003d).

There are five levels of commissioning, outlined in the book in Table 12.1:
• Level 1 – product testing prior to shipment to the site
• Level 2 – file component verification upon receipt
• Level 3 – field inspections and certifications
• Level 4 – site acceptance testing
• Level 5 – integrated systems testing

CHAPTER 13: AVAILABILITY AND REDUNDANCY

Datacom equipment centers by design are highly available environments. Their basic design tenants require
high levels of power, network, and equipment redundancy. Regarding HVAC systems, very specific designs are
required to ensure systems can be properly cooled, particularly during site-level failures.

Due to the high-availability requirements of datacom equipment centers, requirements for fault-tolerant
operation have major implications on the facility design and architecture, as well as on its ongoing
management. Consequently, discussions around designs that are compliant with NEBS Tier 1 – Tier 4 are a
central theme in this book.

Failure mode and effects analysis (FMEA) is a precise process for analyzing system failure modes. This
chapter covers well the subject of resource redundancy, with a focus on the cooling infrastructure.

Diversity refers to the use of geographically diverse locations for critical resource delivery such as power and
network. Dual feeds alone are not enough to ensure redundancy. With diversity of location, increased levels
of availability can be achieved. While all of the above methods are beneficial, human error still remains a
great risk.

CHAPTER 14: ENERGY EFFICIENCY

This chapter covers four basic energy usage metrics: environmental criteria, generation, distribution, and
“other measures.” Refer to ASHRAE’s 90.1 (ASHRAE 2004i) for minimum requirements for energy efficiency
for almost all non-residential facilities, including data centers. Chapter 2 of ASHRAE’s Thermal Guidelines for
Data Processing Environments (ASHRAE 2004h) defines temperature and humidity guidelines.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 16

However, the following relaxation of these guidelines, properly implemented, can provide certain energy
efficiency increases:
• Increase thermodynamic efficiency of vapor compression cycles
• Decrease humidification load
• Decrease dehumidification load

This book reviews energy efficiency design considerations related to chilled-water plants, chilled-water pumps,
cooling towers, and controls leading to the optimization of CRAC components, CRAC designs, fans, pumps,
variable-speed drives, humidification, dehumidification, water-side economizers, and air-side economizers.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 17

IV. BOOK 4: LIQUID COOLING GUIDELINES FOR

DATACOM EQUIPMENT CENTERS
This book provides insight to liquid cooling
deployments, from the cooling tower up to and
including liquid-cooled IT equipment. It offers
guidance, general definitions, and descriptions for
various methods of cooling IT equipment using liquid.
Many of the practices described in the book may be
new to readers who were exposed to the previous
generation of liquid-cooled mainframes. Significant
content of this book is dedicated to differentiation
among the various methods adopted for liquid cooling
equipment.

CHAPTER 1: INTRODUCTION
The reader is introduced to the challenges associated
with air cooling and possible benefits derived from
liquid cooling. The chapter offers definitions of what
is and what is not liquid-cooled IT equipment. It also Title Liquid Cooling
provides brief descriptions of the various fluid loops Guidelines for
of datacom equipment cooling systems (DECS), Datacom Equipment
technology cooling systems (TCS), chilled-water Centers
systems (CHWS), and condenser-water systems (CWS). Page Count 100
The CHWS and CWS have relevance beyond liquid- Date 2006
cooled IT equipment since they are typical of most Price $40
chilled water–based cooling systems. Part numbers 90430 (printed)
D-90430 (electronic
The figure below, found in Chapter 1, describes the - PDF)
various components that comprise one scenario for the thermal transport chain of a liquid cooling
architecture.

FIGURE 1.1 LIQUID COOLING SYSTEMS/LOOPS WITHIN A DATA CENTER ©2006 ASHRAE

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 18

CHAPTER 2: FACILITY COOLING EQUIPMENT

This chapter discusses key design consideration for data center cooling systems, including flexibility,
scalability, ease of installation, commissioning, operation, ease of maintenance/troubleshooting, availability,
and reliability. It also offers overviews of chillers, cooling towers, pumps, and energy recovery equipment. The
depth of coverage on these topics is limited but may prove beneficial to those who are not already familiar
with the various components that make up a mechanical chilled-water plant.

CHAPTER 3: FACILITY PIPING DESIGN

Chapter 3 discusses piping primarily from a facility-planning perspective. Subject matter for this chapter is
divided into five sections: Spatial Consideration, Piping Architecture, Piping Arrangements, Water Treatment
and Earthquake Protection. The majority of this chapter is devoted to piping architectures and arrangements.
A comprehensive overview of the various piping schemas deployed within commercial chilled-water
distribution systems is provided. Sections on water treatment and earthquake protection are adequate to
make the reader aware of such considerations but are less comprehensive.

CHAPTER 4: LIQUID COOLING IMPLEMENTATION FOR DATACOM EQUIPMENT

This chapter discusses a variety of physical and thermal arrangements for the various methods of liquid
cooling datacom equipment. Methods covered include traditional air-cooled, hybrid (air- and liquid-cooled),
and completely liquid-cooled. The text is complemented with a variety of figures that clearly illustrate key
attributes of the various methods. The final section of this chapter introduces the coolant distribution unit
(CDU) and clearly establishes its operation and benefits (water volume, temperature, pressure, and quality).

CHAPTER 5: LIQUID COOLING INFRASTRUCTURE REQUIREMENTS FOR CHILLED-WATER SYSTEMS

In Chapter 5, the reader is introduced to the concept of demarcation between the facility side and the IT
side of the liquid cooling solution. Accordingly, this chapter is primarily devoted to the facility side of the
demarcation point. It discusses water-quality issues, including velocity, fouling, water chemistry, and biological
and wetted material compatibility, along with some quantitative tables. It also depicts idealized piping layouts
to promote architectures that facilitate ease of connecting liquid-cooled IT equipment. This chapter also
discusses the effects of various subsystems such as pumps, fans, controls, and heat exchangers on reliability
and availability. A brief overview of a non-chilled-water-based solution for liquid cooling of IT equipment is
given at the end of the chapter. The reader is exposed to the use of refrigerant-based, direct expansion (DX)
as a possible alternative to chilled water to support TCS and DECS.

CHAPTER 6: LIQUID COOLING INFRASTRUCTURE REQUIREMENTS FOR TECHNOLOGY COOLING SYSTEMS

This chapter covers general consideration for TCS, focused primarily on field piping between the CDU and
datacom equipment. Many of the considerations from Chapter 5 are repeated, but with an IT perspective.

APPENDIX: The appendix summarizes key water-quality attributes collected from a number of datacom
facilities. It is noted that all facilities surveyed were able to achieve desired water quality after implementation
of a water-treatment program.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 19

V. BOOK 5: BEST PRACTICES FOR DATACOM FACILITY

ENERGY EFFICIENCY
This book provides detailed information on the
design of datacom facilities to maximize energy
efficiency. It covers many aspects of datacom facility
energy efficiency, including environmental criteria,
mechanical equipment and systems, economizer
cycles, airflow distribution, HVAC controls and energy
management, electrical distribution equipment,
datacom equipment efficiency, liquid cooling, total
cost of ownership (TCO), and emerging technologies.
The appendices discuss facility commissioning,
operations and maintenance, and the telecom facility
experience. Title Best Practices for
Datacom Facility Energy
CHAPTER 1: INTRODUCTION Efficiency
This chapter states the purpose/objective of the Page Count 241
book, defines key terms, and includes a brief Date 2008
overview of the chapters. Price $47
Part numbers 90438 (printed) D-90438
CHAPTER 2: ENVIRONMENTAL CRITERIA (electronic - PDF)
The environmental conditions inside and outside a facility have a significant impact on the choice of an
energy-efficient cooling system. This chapter provides an overview of data center design conditions and
compares these conditions to ASHRAE criteria to demonstrate how following the criteria can result in
increased energy efficiency. It also describes recommendations that have the greatest impact on datacom
facility energy consumption.

CHAPTER 3: MECHANICAL EQUIPMENT AND SYSTEMS

This chapter reviews cooling and humidity control equipment and systems, and it summarizes pertinent
design considerations to aid in the selection of energy-efficient systems. Types of equipment covered include
chillers, CRAC units, humidification equipment, and heat-rejection equipment. Types of systems covered
include air-cooled, water-cooled, and evaporative-cooled systems.

CHAPTER 4: ECONOMIZERS
Chapter 4 discusses economizer cycles and how they can be used for substantial energy savings and
increased TCO in datacom facilities. The chapter also includes discussions of both outdoor air economizer
cycles and water economizer cycles, as well as things to consider when choosing an economizer.

CHAPTER 5: AIRFLOW DISTRIBUTION

The air distribution design in a data center is non-trivial. A substantial percentage of energy costs in an
air-cooled data center is associated with air distribution. This chapter identifies air distribution parameters
affecting energy consumption and discusses ways to increase energy efficiency. Among the topics covered
are best practices methods of airflow delivery, the effect of supply air temperatures, airflow within racks,
leakage, and airflow optimization.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 20

CHAPTER 6: CONTROLS AND ENERGY MANAGEMENT

Control and energy management systems are key components of an energy-efficient data center. This chapter
covers control systems that have an impact on energy-efficient operations. Subjects include control system
architecture, the risks and benefits of using economizers, increasing the chilled-water supply and/or air
temperatures, and developing an efficient part-load operation sequence for the central cooling plant.

CHAPTER 7: ELECTRICAL DISTRIBUTION EQUIPMENT

Studies have identified electrical distribution systems as a significant source of energy loss. This chapter
identifies these losses and discusses approaches that can increase the efficiency of electrical distribution.
The chapter starts with a brief discussion of utility generation and continues with facility distribution,
including an AC and DC overview. Also included are distribution paths, energy efficiency options such as
reduction in amperage and harmonics, and component efficiencies of uninterruptible power supplies (UPSs)
and transformers.

CHAPTER 8: DATACOM EQUIPMENT EFFICIENCY

This chapter discusses many issues related to datacom equipment energy efficiency. Datacom equipment is
continuing to become more energy efficient via more efficient power supplies and voltage regulators, lower-
power processors, and increasingly sophisticated power management. Since datacom equipment needs
electrical distribution and cooling infrastructure to support the resulting heat load, efficiency improvements at
this base level also will reduce energy use in the distribution system and mechanical plant.

CHAPTER 9: LIQUID COOLING

Liquid cooling equipment and installations are becoming more common, primarily in response to increasing
power density and the need to provide a compact cooling infrastructure. This chapter reviews the energy
efficiency and TCO of liquid cooling. It discusses considerations related to open- or closed-cooling
architecture, water, refrigerant or dielectric cooling fluid, and TCO.

CHAPTER 10: TCO

Total cost of ownership is a broad tool that can be used to design datacom facilities in an efficient and cost-
effective manner. This chapter starts by defining TCO and then discusses its applicability for making design
decisions in the datacom industry. It discusses the most important capital and operational components of a
TCO model and provides advice on what to include in models.

CHAPTER 11: EMERGING TECHNOLOGIES AND FUTURE RESEARCH

This chapter presents a listing of some emerging technologies that may have an impact on energy efficiency
in datacom facilities. It also makes suggestions for future research.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 21

VI. BOOK 6: STRUCTURAL AND VIBRATION

GUIDELINES FOR DATACOM EQUIPMENT CENTERS
This book discusses, in a holistic way, the structural
aspects of housing the servers, storage, tape drives,
and network equipment (datacom equipment) within
the datacom equipment center. Datacom equipment
center engineers must consider the structural
and vibration performance of the facility, the
building infrastructure, and the contents (datacom
equipment). As the datacom equipment density
increases (compaction), the datacom power and
cooling infrastructure becomes larger, heavier, and
inherently more structurally challenging.

With the datacom equipment itself becoming

heavier and expectations growing that the facility
will house more datacom equipment, the structural
and vibration performance of the facility is becoming
more and more important. Increased structural Title Structural and Vibration

resistance against high wind, snow, and seismic Guidelines for Datacom

loads may be required. Equipment Centers

Page Count 177

The typical replacement rate of servers, storage, tape Date 2008

drives, and network equipment (datacom equipment) Price $39
is often 3 to 5 years, while the anticipated lifespan Part numbers 904429(printed)
of the supporting mechanical and electrical D-90429 (electronic -
infrastructure is 15 to 20 years. Consequently, the PDF)
datacom building structure may house many vintages of datacom equipment over its lifetime. Structural
design criteria must consider future requirements.

High-performance data center facilities accommodate a variety of complex and sensitive datacom equipment
that is vulnerable to internal and external sources of shock and vibration. Shock and vibration are unwanted
forces found at some level in most datacom facilities. Over time, they can degrade both facilities and
equipment. Isolation design and techniques typically require assumptions as to ground-motion amplitudes,
frequencies, and additional space for the device displacement.

This book describes guidelines for monitoring the level of vibration within the datacom facility and methods
for mitigating vibration’s effect on the datacom equipment. For example, it discusses numerical simulation,
which is one method for evaluating the structural requirements for datacom equipment itself. The book also
examines efforts to resist seismic loads, such as systems that anchor datacom equipment to the datacom
center as well as techniques to stiffen the datacom equipment.

The intended audience for this publication includes both technical and nontechnical readers. Those involved
in the design, construction, commission, operation, implementation, and maintenance of datacom equipment
centers all can benefit from this book. In addition, equipment operators, IT departments, chief information

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 22

officers, and those who develop and\or design electronic, cooling, and other infrastructure equipment will
benefit from its guidelines.

This book is divided into four parts. Part 1 includes an overview of the book and a summary of the best
practices. Part 2, Building Structure, includes chapters on new and existing building structures as well as
types of building structures. In Part 3, Building Infrastructure, raised-access floor systems and vibration
sources and control are discussed in detail. Part 4, Datacom Equipment, includes chapters on shock and
vibration testing, seismic anchorage, and analysis of such systems. For datacom professionals, this book will
provide basic guidelines on which seismic anchorage should be used for datacom equipment. The book’s
appendices provide guidelines for floor loading calculations as well as what type of operational vibration is
expected in a data center.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 23

VII. BOOK 7: HIGH DENSITY DATA CENTERS - CASE

STUDIES AND BEST PRACTICES
This book reviews the seven primary ventilation
schemes commonly used in data centers. The
measured data in this book’s case studies can
be used by data center planners and managers
to do theoretical construction and modeling for
benchmarking against other data centers and to assist
in the planning process. The strategy for using this
book in a new data center is to start a Thermal Profile
around the concepts and recommendations provided
as the data center is being built.

CHAPTER 1: INTRODUCTION
The introduction provides a quick overview and easy-
to-understand diagrams of seven different ventilation
schemes. For each scheme described, there are
several real-world case studies from live data centers Title High Density Data Centers

in which the scheme was tested for airflow, power, - Case Studies and Best

and temperature. The results of each case study are Practices

presented in a uniform format to make comparisons Page Count 194

with the other studies as accurate as possible. Date 2008

Price $43
CHAPTER 2: RAISED-ACCESS FLOOR CASE STUDIES Part numbers 90441 (printed) D-90441
Chapter 2 includes discussions on perimeter CRAC (electronic - PDF)
cooling schemes where CRAC units are located within the data center space and on the outside walls and
where they deliver conditioned air through a raised floor. For each of the case studies presented, this chapter
extensively explains all of the environmental measurements captured, the tools and processes used, and
when estimation by other means was used. According to the book, the key IT equipment health indicator is
the inlet temperature; the majority of the studies included focus on this point. At the end of each case study,
there is a Thermal Profile for the data center that provides the reader with an excellent tool for comparisons
between data centers.

This chapter discusses the following cooling schemes:

• Raised-Access Floor with Air-Handling Units (AHUs) on Subfloor
• Raised-Access Floor Supply/Ceiling Return
• Raised-Access Floor with Heat Exchangers Adjacent to Server Racks
• Cooling within the In-Row Rack
• Raised-Access Floor with Heat Exchangers Over Server Racks
• Raised-Access Floor with Underfloor Supply/Ducted Ceiling Return

CHAPTER 3: NON-RAISED-ACCESS FLOOR CASE STUDIES

These two case studies do not compare the pros and cons of raised-floor versus non-raised-floor cooling
schemes; rather, they offer some best practice suggestions for non-raised-floor environments. Then, for
comparative purposes, readers are encouraged to develop their own profiles, using the performance data

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 24

and suggested practices to enhance the efficiency of their own data center environments. One configuration
utilizes up-flow CRACs with constant-speed AHUs supplied and chilled water from a chiller plant. Another
configuration described is ceiling-mounted fan coil units (FCUs) with chilled water from the chiller plant and
an overhead duct and grill system.

CHAPTER 4: BEST PRACTICES

This chapter covers data center design best practices. To meet IT needs and future growth plans, data center
managers are encouraged to carefully evaluate the available options and to know what questions to ask to
complete a thorough evaluation—before selecting a cooling scheme. The chapter provides recommended
basic questions and today’s best practices to consider when designing a data center.

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010
 PAGE 3

CREDITS
The Technology and Strategy Work Group of The Green Grid would like to acknowledge contributions from the
following individuals in the creation of this document: John Bean, Ron Bednar, Rich Jones, Robb Jones, Phil
Morris, Budy Notohardjono, Mike Patterson, Wade Vinson, and John Wallerich. The Green Grid also wishes to
acknowledge the help of ASHRAE’s TC9.9 in reviewing this document.

ABOUT THE GREEN GRID™

The Green Grid is a global consortium of companies dedicated to advancing energy efficiency in
data centers and computing ecosystems. The Green Grid does not endorse any vendor-specific
products or solutions, and will seek to provide industry-wide recommendations on best practices,
metrics and technologies that will improve overall data center energy efficiencies. Membership is
open to companies interested in data center operational efficiency at the Contributing or General
Member level. General members attend and participate in general meetings of The Green Grid,
review proposals for specifications and have access to specifications for test suites and design
guidelines and IP licensing. Additional benefits for contributor members include participation and
voting rights in committees and working groups. Additional information is available at www.thegreengrid.org

©2010 The Green Grid. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted or stored in any retrieval system of any
nature, without the written permission of the copyright owner.
Version 1.1 Release Date: 18 January 2010