Datacenter Health Check

Terri-Lynn Thayer AVP/CIO, Computing & Information Services EDUCAUSE Enterprise Technology Conference, May 2007

Copyright Terri-Lynn B. Thayer 2007

This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Presentation Overview
Datacenter blood pressure is rising Pre-assessment work Reliability goals Assess existing space, power, cooling, fire suppression, & security Assessment results & options Recommendations for a renovation Key considerations in site selection

Datacenter Blood Pressure is Rising

Growing Demands

Technology Changes

Increased number of servers Increase in breadth of customer base taking in previously distributed computing Storage growth Power/cooling needs new boxes are smaller but they are energy hogs and they are hot, hot, hot (10X power for a fully populated rack and 3-4X power to cool it down) 24 x 7 demands Well publicized disasters - 911, Katrina Many university datacenters today lack standby power generation National trend for R1s to take on more support centrally

Business Resumption Concerns

Research Support

Our Datacenters are Old

Pre-Assessment Work

Capacity Planning and Growth Analysis

Determine a planning horizon Identify services likely to be provided from your datacenter during that time period

Business Continuity and Disaster Recovery Objectives

How long can your University operate without a functioning datacenter? Do you have a cold or hot site? Should you consider a multiple datacenter approach?

More Homework

Research computing support decision is fundamental Review your insurance

Get professional help with the assessment

How much do you have Other requirements of your insurer

What cost/risk profile is your institution comfortable with?

Involve your university facilities engineers Seek advice from outside professionals who are familiar with modern datacenter design and operation

Reliability goals

BRUNS-PAK Data Center Reliability Ranking

Numerical Rankings (1) (2) (3) (4) (5) (6) (7) (8) (9) Terminology Unreliable Partially Isolated, Unreliable Isolated Unreliable Isolated Conditioned Isolated Improved Isolated, Mostly Reliable Reliable Reliable Redundant Ultra-Reliable Summary Definition Shared building power and cooling; no generator Dedicated power system; shared cooling system; unconditioned power; non-redundant air conditioning; no generator Dedicated power and cooling systems; unconditioned power; non-redundant dedicated air conditioning units; no generator Dedicated power and cooling systems; conditioned power; non-redundant dedicated A/C units; no generator Dedicated power and cooling systems; uninterruptible power system; non-redundant dedicated A/C units; no generator Dedicated power and cooling systems; uninterruptible power system; redundant dedicated A/C units; no generator Dedicated power and cooling systems; uninterruptible power system; redundant dedicated A/C units; generator Dedicated power and cooling systems; redundant UPS systems; redundant dedicated A/C units; redundant generators Redundant power train; redundant cooling system; redundant UPS systems; redundant dedicated A/C units; redundant generator systems; redundant fuel system Redundant power train; redundant cooling system, redundant UPS systems, redundant dedicated A/C units; redundant generator systems; redundant fuel system; site hardened for weather and geographic exposures; location minimizes exposure to jurisdictional closure from hazardous spill, terrorism, or similar risks.

(10)

State of the Art

BRUNS-PAK 999 NEW DURHAM ROAD EDISON, NJ 08817 (732) 248-4455 Fax: (732) 248-3644 http://www.bruns-pak.com

Datacenter Evaluation
Space Electrical System Mechanical System Fire Protection System Security

Space

Square footage of conditioned space

Raised floor

Access
Elevators Door size

Machine room layout Furnishings, racks, command center Is the space expandable? Is this a multi purpose facility?

Electrical/Power Considerations

Source and costs Patch Panel/Power Control Units/ Power Distribution Units Standby power

Uninterruptible Power Supply (UPS)

Redundant/non-redundant Battery type (wet vs dry), capacity, and monitoring Type Power and cooling

Generator

Are the systems expandable?

Delicate balance Surge protection, lightning protection, grounding

Other

Mechanical Systems Evaluation

Cooling & humidity control

Chilled water do you have a dedicated chiller? Computer Room Air Conditioner - CRAC units - # and location

Capacity

and reliability

Heat Detectors

Airflow distribution Water sensors

Fire Protection Systems

Detection

Smoke alarms Heat detectors Air sampling Halon system (production banned in 1994) Full flooding clean agent system

Abatement

FM-200, NAF, Inergen etc

Sprinkler system

Wet Pre-action

Security

Physical access to the facility

elevators & doors Caged areas and visitors Multi-purpose facility Door access system

Windows Monitoring

Closed circuit TV

Ancillary Services and Support

Ancillary Services

Other staffing and services which are provided from your datacenter Machine hosting and associated SLAs

Tape storage Secure storage/staging Paper storage Test/setup lab Printers and print support General storage room Break room

others University departments groups external to the University

Assessment Results & Options

Most of us will find that our datacenters are not adequate for the anticipated growth over the next five years Majority will identify power and cooling as the most significant issue

Cooling and the power to cool will be the number one issue

Space constraints will be the runner up

Options
Renovate Build a new datacenter Both of the above Multi-datacenter campus Outsource or Hosting

Recommendations for Renovation

Implement standby power generation capable of supporting both power and cooling Remove ancillary services from machine room and relocate to other spaces Trade off between space and density is a complex issue

High density racking results in significant heat and power provision issues It is general cheaper to provide more space than to keep a small space with high density equipment adequately powered and cooled

Improve Air Flow & Circulation

Provide additional space between racks to promote air circulation Open up plenum space by relocating cabling to overhead trays Increase height of the raised floor if possible Consider new cooling solutions and rack technologies everything old is new again

Reconfigure the layout to implement a double hot aisle/cold aisle configuration Distribute high density racks

Chilled water is far more efficient than cool air for heat removal

Double Hot Aisle/Cold Aisle

hot aisle/cold aisle layout is where cold air is segregated in front of equipment cabinets and hot exhaust air is expelled behind equipment cabinets. This layout eliminates the direct transfer of hot exhaust air from one system into the intake air of another system Double

A CRAC unit is located between two hot aisles

Site Selection for a New Facility

An opportunity to consider cost

Corporate world has moved their datacenters in some cases quite remote from the rest of their operation which allows them to consider
Power

costs Real estate cost Labor costs

Look for a Old Supermarket

Single story Single use facility Slab Few windows Lots of open space around the building Loading docks and delivery truck access

Other Considerations
Voice and data connectivity Flooding and other weather related issues If moving to a multi-datacenter approach as part of a business continuity plan then consideration should be given to put the two datacenters at a sufficient distance to reduce dependence on the same power grid and to minimize weather and other regional disasters

Staffing Implications

Data Center Managers will need to be more skilled in the area of environmental issues, engineering, and server technologies Facilities organizations may need to devote more time and specialization to cooling and power technologies related to the datacenter Managing data center renovation or build projects will be resource intensive and may result in downtime for key services

Go Green

Reduce energy costs (datacenter build may be more expensive) Legislation Environmental concerns and institutional plans to reduce carbon emissions Vendor products

Rack and server cooling technologies CO2 for cooling, DC power systems Solar panels and wind energy Heat recycling

Design Considerations

Data Center

A data center is a facility used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and security devices

Tier Level

Requirements

Single non-redundant distribution path serving the IT equipments Non-redundant capacity components Basic site infrastructure guaranteeing 99.671% availability

Fulfils all Tier 1 requirements Redundant site infrastructure capacity components guaranteeing 99.741% availability

Fulfils all Tier 1 & Tier 2 requirements Multiple independent distribution paths serving the IT equipments All IT equipments must be dual-powered and fully compatible with the topology of a site's architecture Concurrently maintainable site infrastructure guaranteeing 99.982% availability

Fulfils all Tier 1, Tier 2 and Tier 3 requirements All cooling equipment is independently dual-powered, including chillers and Heating, Ventilating and Air Conditioning (HVAC) systems Fault tolerant site infrastructure with electrical power storage and distribution facilities guaranteeing 99.995% availability

Conclusion

Our datacenters are under significant stress If we take a close look we will find that most of us will experience power and cooling problems in the near future. Cooling and the power to cool will be the most substantial issue we face. There are new technologies and best practices which will provide some relief Many of us will build new datacenters over the next five years and we should consider remote locations, outsourcing, and green IT solutions These project will require both significant financial resources as well as IT and Facilities staff time. We may need to employ new skill sets It is highly recommended that you engage professional assistance to evaluate your facility and to assist in renovation and new build designs

Additional Resources

www.stonesoup.org
Past Meetings Spring 2006 meeting Data Center Futures Workshop Presentations