FOM 2.

3:
Best Practices for
Operating a Data Center
Steven Shapiro, P.E., ATD
Mission Critical Practice Lead

1
Data Center World – Certified Vendor Neutral

Each presenter is required to certify that their

presentation will be vendor-neutral.
As an attendee you have a right to enforce
this policy of having no sales pitch within a
session by alerting the speaker if you feel the
session is not being presented in a vendor
neutral fashion. If the issue continues to be a
problem, please alert Data Center World staff
after the session is complete.

2
Agenda
• General Guideline Requirements
• Site Selection
• Building Construction
• Space Planning
• Physical Security
• Mechanical Systems
• Water and Plumbing
• Fire Protection
• Electrical Systems
3
General Guidelines / Requirements
• Uptime and Reliability Standards
 LEED
 Energy Star
 Green Globes
 Uptime Institute Tier Certification: Tier I to IV

4
General Guidelines / Requirements
• Design and Infrastructure Standards
 ANSI/BICSI 002-2014 – Class F0 to F4
 TIA 942 – Tier 1-4
 EN 50600 series 1, 2-1 to 2-6 – Availability Class 1-4
 EN 50173-5
 ISO/IEC 24764
 HIPPA - Health Insurance Portability and Accountability Act
 SOX – Sarbanes Oxley – 2002
 SAS 70 – Type I or II
 Gramm-Leach Bliley Act (GLBA)

5
General Guidelines / Requirements
• Operations Standards
 Uptime Institute: Operational Sustainability
 ISO 9000 - Quality System
 ISO14000 - Environmental Management System
 IS0 27001 - Information Security
 PCI – Payment Card Industry Security Standard
 SAS70 & ISAE 3402 or SSAE16 (USA) - Assurance controls
 AMS-IX - Data Centre Business Continuity Standard
 EN50600-2-6 Management and Operational Information

6
General Guidelines Summary
• Facility reliability must match the requirements
detailed in the business case for the facility
• Total cost of ownership (TCO) approach that will
balance capital and operational expenditures to
your company’s needs. Part of this process is
determining design criteria and performance
characteristics, tailored specifically to your
organization.
• When operations teams are excluded from facility
design, modification and repairs often become
necessary.
• CXA Team participation is essential from
concept to final pull the plug testing.

7
Site Selection
• Economic Incentives
 Tax incentives
 Peak shaving or cogeneration incentives

• Sustainability Opportunities/Incentives
 Green Power availability
 Utility rebates to energy efficient equipment
 Free cooling opportunities

• Data Center “Friendly” Municipalities

 Expedited approval and other helpful processes
 Friendly Noise and Emissions Ordinances
 No Fuel Oil Storage Limitations

8
Site Selection
• Natural Disaster Potential
 Seismic
 Tornado/Hurricane
 Flood
 Tsunami

• Utility Availability/Reliability/Costs
 Power, water, sewer
 Power cost, power cost
and power cost!
 Multiple fiber carriers

9
Site Selection
Stay Away from Hostile Adjacencies…
• Airports/flight paths
• Railway Lines
• Nuclear and other power plants
• Hazardous/explosive materials and facilities
• Residential neighborhoods

But still close enough to…

• Airports
• Freeways
• Emergency Services
• Data Center equipment technicians
• Public transportation

10
Site Selection
Needed Utilities
• Buried and reliable utilities – System
Average Interruption Duration Index (SAIDI)
• Diverse fiber and power availability
• Reliability of power grid
• Sufficient water availability
 City water
 Well water
 Water storage

11
Building Construction
• Fire ratings of areas
• Block or sheet rock walls
• Clear roof
• No roof drains
• No windows
• Hurricane resistant
• Single or multi-story
• Structural steel or reinforced concrete shell
• Thickened existing slabs to support heavier loads.
• Roof to withstand 120 mph winds
• Floor to roof structure
• Roof structure to withstand hanging loads
• Large column – less areas
• Floor loading
• Facility time horizon – expected useful life

12
Space Planning
• Load density
• White space utilization
• True white space square footage
• CRAH galleries/mechanical – electrical corridor
• Tray elevations/power/copper/fiber
• Ceiling or no ceiling
• Floor/no floor
• Hot aisle/cold aisle – containment, blanking panels
• Systems isolation/segregation to redundancy level
• Feeder/piping path isolation
• Cabinet/rack layout – 42U, 72U, 19” wide, 24” wide, 2 post,
w/ wire manager
• POEs with dedicated power/rectifiers
• MPOs, MDF, IDF
• Door sizes
• Equipment delivery path
13
Space Planning
• People and Technician Space Definition
 Test/Assembly Area
 Lab area
 Storage
 Vendor storage/Offices
 Uncrating area
 Dumpsters
• Electronics
• Paper
• Cardboard
• Wood
• Trash

14
Physical Security
• Security Threat Analysis
 Bombs – Bag/Car
 Rockets/Planes
 Direct Attack
 Disgruntled Employee
 Corporate Espionage

• Zones of Protection
 Cabinet
 White Space
 Gray Space and hallways
 Office
 Site
 Perimeter
15
Physical Security
• Crime Prevention Through Environmental Design
(CPTED)
• Segregated POE/Vendor areas
• Devices
• Card Readers
• Biometrics
• Mantraps
• Cameras
• Dogs
• Berms
• EPO
• Escorts
16
Physical Security
• Contractor Badges
 Keys
 Escorts
 Rules

• Copper/fiber secure wireways

• Cages under floor/above ceiling
• Bars in ductwork
• Bollards
• Active Vehicle Barriers
• No Signage on exterior

17
Mechanical Systems
• Energy Efficiency-PUE
• What is PUE/DICE?
• Elevated data center temperature maximizes return air
temperature but accelerates critical equipment failure
So says Los Alamos Heat-to-Failure Study with 10 degree C rise the
failure rate doubles (APC White Paper)
• Containment – Hot/Cold Aisle
• Humidification levels in data center - ASHRAE is opening
window

18
Mechanical Systems
ASHRAE 9.9 and 9.1

Allowable Range

Recommended Range
19
Mechanical Systems
• Air Side Economizer Hours Possible in ASHRAE
Allowable Range

20
Mechanical Systems
Air Side Economizer Hours Possible in ASHRAE
Recommended Range

21
Mechanical Systems
• Variable Speed Server Fans
• EC Plug Fans – No VFD required
• Upflow/Downflow/Freeflow

22
Mechanical Redundancy
• 2 pipe/4 pipe
• N+1, N+20%
• Continuous Cooling

AiNet

23
Mechanical Resiliency
• CBRN – Chemical, Biological, Radiological, Nuclear
filtration/threat
• Water storage
• Redundant piping, loop with dual risers, redundant
coils
• Water treatment
• Fuel storage – 48, 72, 96 hours, above or below, day
tanks, belly tanks
• Fuel treatment – polishing
• Fuel delivery/testing on delivery
• Fuel sampling
24
Cooling Technologies
• Chilled Water
 Air cooled chillers or water cooled chillers
 Economizer
• Water side economizer – Heat Exchanger
• Series/Parallel to chiller
 Cooling Tower - Drycooler-Fluid Cooler-Dry
Towers

25
Cooling Technologies
• DX
 Condenser
 Drycooler
 Pumped Refrigerant
 Air side economizer
 Direct - needs corrosion monitoring systems due to
particulate introduced into the data center
 Indirect – maintains air quality in the data center
 Evaporative – used in direct and indirect airside
economization
 Trimming systems – mechanical cooling required for air
side economizer in certain geographies due to local
weather, DX, chilled water

26
Cooling Techniques
• Outside Air for Pressurization has humidification for
entire facility
• Indoor and outdoor air quality
• MERV and his filters 8 for recirculation and 11 for
outdoor air economizers
• Separate comfort cooling (people spaces) from
critical cooling due to temperature requirements
• Primary Pumping with VSDs
• Primary/Secondary pumping with VSDs
• Reuse of data center waste heat
• In-row cooling
27
Cooling Techniques
• In the aisle – suspended in aisle or in row with cabinets
• Room perimeter/mechanical/CRAC corridor
• Mechanical equipment on UPS – Fans/Chilled Water pumps –
continuous cooling requirements depending on load density
and generator restart times
• Thermal capacity in piping, access flooring, slab – calculate
and test
• Chilled water storage for emergency and peak shaving
• Ice storage for emergency and peak shaving
• Spot Coolers – Move-N-Cool, provide power connections
throughout data center
• Rack level Cooling Doors/Refrigerant/Fan Door

28
Cooling Techniques
• Liquid cooling – bathtub/tank cooling – mineral oil, NOVEC,
etc
• Server level cooling
• Chip Level Cooling
• Data centers must meet an ISO class 8 clean room standard.
• Reheat/Humidification in CRACs - No
• CFD Analysis
• Containment
• Chimneys
• End doors
• Cold Aisle
• Hot Aisle
• Partial containment
• Fire protection issues with containment

29
Containment

30
Controls
• BMS/BAS – smart controllers
• Controls at units (local)
• Master/Slave – remote then local
• VFD/VSD – bypass, harmonic filtration on UPS power and when
on generators
• Smart VFDs match to pump operation.
• Data Center airflow/variable flow based on static pressure
needed
• Rack level monitoring – temperature monitoring, static
pressure monitoring from bottom to top of rack

31
Water/Plumbing
• Conservation:
 Gray water storage/use
 Irrigation water
 Rainwater management

• Plumbing Issues:
 Sewer/Septic – Issues with Cooling Tower Blowdown
 Trap Primers or Plugs – Issues with Gas Suppression
 Well, City Water
 N, 2N supplies

32
Fire Protection
• Clean Agent – Gas – CO2, FM200, Inergen, NOVEC
• Sprinkler – Wet/Dry/Preaction-Single/double interlock
• Diesel Fire Pump
• Mist – Generators/Data Center/Electrical
Rooms/Mechanical Rooms

33
Fire Protection
• ASSD – Air Sampling Smoke Detection – above and
below floor, in ceiling
• Spot detection
• Cross zoned – software
• Photoelectric/ionization/cable
• Rated for underfloor CFM

34
Electrical Systems
• Onsite Cogeneration
 Utility
• N
• 2N
• Voltage
 UPS Configurations
• N
• N+1
• 2N
• 2(N+1)
• Distributed Redundant
• Block Redundant
• Isolated-Redundant
• Stranded Capacity

35
Electrical Systems
• UPS with transformer or transformer free
 Efficiency
 Ground current issues
• UPS Modules with internal redundancy
 Internal redundancy but same single input/output
• UPS System Static Switch
 module level
 system level
• UPS Technology
 Double Conversion
 Offline
 Line interactive
 Delta Conversion
 Rotary
 DRUPS – Diesel Rotary
 Hybrid
 Low Voltage
 Medium Voltage

36
Electrical Systems
• Energy Storage
 Flooded
 VRLA
 Flywheel
 Coupling

• Central UPS vs Distributed UPS

• Maintenance Load Bank Substation – Alternate
Source for MBP

• Permanent Load Bank

 Portable connection for inductive
 For UPS
 For Generator

37
Electrical Systems
• Eco mode
• Good - Efficiency
• Bad – UPS Offline

• ITIC Curve/CBEMA
Curve
• UPS Distribution voltage
• 480V, 120/208V, 415V,
• 3 Phase 3 Wire, 3 Phase 4 Wire
• DC systems and
voltage – 380VDC

38
Electrical Systems
• Battery Monitoring
• Cell
• Jar
• System
• Loading
• Impedance
• Networked

39
Electrical Systems
• Harmonics
• Scalable
• Generator configurations
• Capacity
• Voltage – low, medium
• Prime
• Continuous
• Standby
• Generator distribution
• Single
• Parallel

40
Electrical Systems
• Generator Protection
• Differential protection
• Surge Arrestors
• R-C Snubber Circuits

• Dedicated life safety generator or battery

emergency lighting, fire alarm
• Noise issues

41
Electrical Systems
• Power to the Rack
• 120/208VAC, 415VAC
• PDUs
• RPPs
• Busway
• Branch circuit monitoring
• Above the floor
• Below the Floor
• Short Circuit withstand
• Arc Flash
• Keep both cords live

42
Electrical Systems
• Maintenance PDU
• RPP tie - RPP alternate main
• 4 pole breaker or Neutral contactor for
4 wire transfer
• Transformer Efficiencies and DOE 2016
• Transformer Inrush Current
• Switchgear/Switchboards – Quality –
Resiliency
• Draw out, Bolt on, Plug in CB
• Overcurrent Protection and Code
• Solid State Trip
• Zone Selective Interlock
• Maintenance Mode for
• Reduced Arc Flash
• Fuses
43
Electrical Systems

44
Electrical Systems
• MV Circuit Breakers
• Vacuum - Snubbers
• SF6
• IR
• Manual Camera
• internal fixed in equipment
• Windows

45
Electrical Systems
• EPMS – monitoring, time
stamping – GPS timing
• Arc Flash/Coordination
Issues and PPE
requirements
• Rack level PDUs/MOAs
Multi Outlet Assembly –
power strips
• Smart/Dumb
• Metering
• Short Circuit Rating

46
Electrical Systems
• Green and Energy Savings
• Solar
• Fuel cells
• Micro-turbines
• Wind
• Distributed generation and
load curtailment
• Emissions –
prime/standby/scrubbers
• Generator - lower block heater
temp to save energy
• Demand peak shaving
• Lighting Controls
47
Electrical Systems
• Other Electrical Concerns
• Grounding/bonding/lightning protection
• Surge Suppression devices, SPD, TVSS
• Power Factor Correction
• Portable Generator Connection
• Portable Load Bank Connection
• Load Bank Circuit Breaker
• Load Bank Switchgear
• Generator
• UPS – After the Static Switch

48
Electrical Systems
• Color coded and labeled power paths
• Snow/rain and generators
• Spare Part Storage – Power, UPS, Gens, HVAC
• Concurrent maintainability
• Fault tolerance

49
3 Key Things You Have Learned During this
Session
1. What Codes And Standards Should I
Use To Design And Operate My Data
Center.
2. What Should I Think About When
Looking For A Site.
3. What To Think About When I Decide My
Infrastructure Scheme.

50
Thank you
Steven Shapiro, P.E., ATD
Mission Critical Practice Lead
(914) 420-3213
[email protected]
http://www.linkedin.com/in/stevenshapirope
@stevenshapirope

51