CPSS TRANSACTIONS ON POWER ELECTRONICS AND APPLICATIONS, VOL. 2, NO.

1, MARCH 2017 39

Data Center Challenges and Their Power

Electronics
Philip T. Krein

Abstract—Data centers use more than 1.5% of all electricity savings.

in China and the U.S., with continuing growth. This paper re-
views the power hierarchy levels within modern data centers. It
considers energy consumption and power electronics challenges II. The Hierarchy Levels in a Data Center
across all levels of a data center, including building distribution, Data center power can be treated according to levels of
dc architectures, and conversion down to the board level. Power
function:
electronics plays a central role throughout the hierarchy, and
emerging approaches are described. Strategies that enhance
• Utility level. Independent feeds, cogeneration, and exter-
center energy efficiency, both in terms of overall center opera- nal renewables are a few examples of challenges outside
tion and in terms of computation performance, are discussed. a data center itself.
• Building level. This is the level at which the primary util-
Index Terms—Data centers, dc distribution, dc-dc converters, ity feed enters the facility. Challenges include the choices
server racks, power utilization factor, dc power architecture, dig- between low voltage (e.g. 480 V or 600 V) input and me-
ital power. dium voltage input, and whether the main building feed
should be converted immediately to dc.
• Backup level. Many architectures have uninterruptible
I. Introduction
power supplies (UPS) and fuel-driven generators to pro-

M ODERN data centers are superficially similar to main-

frame computer rooms from the 1960s, but have grown
in scale and in number. They represent a fast-growing load
vide power at the building level. Some distribute backup
at lower levels.
• Rack level. Groups of racks are likely to use broader ther-
component within the electric power grid. In [1], it is reported mal management, but power delivery to individual racks
that in 2015, more than 400,000 data centers in China alone is typical.
were consuming about 1.5% of national electricity supply. In • Board level. Within a blade, power may be managed for
the U.S., consumption in 2014 reached 70 TWh, about 1.8% memory boards, computing motherboards, and other lo-
of total supply [2]. Life-cycle costs of data centers today are cal IT equipment.
often dominated by energy expense [3]. • Chip level. Within a board, individual point-of-load (PoL)
Although there is vast literature about overall data center dc-dc converters provide low-voltage power for chips or
performance, and large-scale industry efforts to enhance chipsets. It is typical to see three to five different voltage
high-level efficiency [3], [4], the attributes associated with values at this level of power management.
power architecture and power electronics across the layers • Internal level. Some ICs, especially multi-core proces-
in a data center are less widely discussed. This paper draws sors have on board energy management.
on [5], with updates across the hierarchy. Opportunities for Each level involves particular power requirements and op-
improvements across the layers are summarized, and work in portunities. Although an optimized system might couple
these areas is reviewed. The impact of power electronics is attributes at several levels to achieve the best results, this pa-
emphasized. per will emphasize the individual levels and the challenges
Based on the recommendation to “follow the power” [6], that each one entails.
it is important to track energy flow in a data center from the
external utility substation through building distribution A. Utility Level
to the individual racks, and then on to the server blades,
memory and computing boards, and finally to the low-level At the utility level, many of the design aspects are linked
information technology (IT) loads. Each level in this hierar- to reliability. Reliability is linked to tiers [7], and at the high-
chy brings its own challenges and opportunities for energy est Tier IV level, multiple independent utility feeds are typi-
cal. A data center might also be part of a microgrid. Some of
Manuscript received March 27, 2017. The work was supported in part the system architecture opportunities for power electronics
by the Grainger Center for Electric Machinery and Electromechanics at the include active switch management of multiple feeds [8] and
University of Illinois. integration into microgrids [9].
Philip T. Krein is with Grainger Center for Electric Machinery and From a thermal perspective, the “energy” in the form of
Electromechanics University of Illinois, Illinois, 61801, USA (e-mail:
[email protected]). information leaving a data center is insignificant, and IT
Digital Object Identifier 10.24295/CPSSTPEA.2017.00005 equipment ultimately converts 100% of incoming energy
40 CPSS TRANSACTIONS ON POWER ELECTRONICS AND APPLICATIONS, VOL. 2, NO. 1, MARCH 2017

into heat. This implies opportunities for combined heat and values in the range of 1.02 to 1.03). The alternative is to
power (CHP or co-gen) utility-level architectures [10]. How- immerse a complete rack or data center itself, possibly with
ever, CHP has only been explored in a limited way for data conventional air-based heat transfer into the surrounding liq-
centers. One issue is that the waste heat from IT devices is at uid. An undersea approach based on this has been announced
relatively low temperature (rarely above 50°C) and therefore by Microsoft [18].
is not useful for steam generation. It can be used in district Building-level emphasis on PUE has encouraged advanced
heat systems [11]. Following some early work [12], the trend HVAC work. Just a few examples include [19], in which en-
has been to emphasize co-gen based on other devices such thalpy wheels are evaluated for data center benefits, and [20],
as microturbines [13]. A more practical situation locates a in which rack-based cooling is proposed. In [21], a compre-
data center near a power plant that can also supply steam for hensive overview of power architectures and interconnections
absorption chillers. to renewables is presented. The work also discusses power
As data center energy use continues to grow, numbers of electronics architectures and cooling alternatives. Much of the
data centers climb, and reliability demands increase, utilities discussion addresses UPS efficiency and operation.
are likely to have a larger role in siting and planning [14]. In the context of power electronics, the discussion and use
This has been typical in the past for large manufacturers, and of direct dc distribution within data centers is a key trend.
data centers are approaching similar load levels. By the time of the workshop [6] held in February 2007, there
were already some demonstration projects and active indus-
B. Building Level try engagement. In [22], architectures are compared in depth
across conventional ac distribution with conversion at the
At the building level, there have been extensive studies that
address the power usage effectiveness (PUE), defined as the rack level, building-level 48 V dc distribution, building-level
ratio of total energy into a facility to total energy consumed 400 V dc distribution, and others. The work identifies 400
by IT equipment within the facility. PUE must be greater than V building-level dc distribution as advantageous. Several
one under any circumstances and can readily exceed two studies, including [23], [24] have reported that dc distribu-
given conventional heating, ventilation, and air-conditioning tion at the building level enhances efficiency and also raises
(HVAC) systems operating in hot environments. Realistically, reliability and availability. It is important to recognize, how-
PUE will exceed one substantially even if HVAC require- ever, that efficiency improvements are on the order of one
ments are minimal, since a useful data center has lighting, percentage point or so with best-practice present designs.
security systems, and other non-IT functional loads. Results from related areas, including mainframe computer
An extreme low in PUE can be achieved with liquid im- systems, supercomputers, and mobile power systems have
mersion cooling in unstaffed data centers. In [15], a PUE bearing on questions of dc distribution. A broad discussion
value of 1.02 is reported for an insulating boiling immersion of the issues and the power conversion requirements can be
cooling approach. However, such a low number strongly found in [25]. This work suggests that dc distribution was
suggests incomplete computation. For example, the power reported to have advantages even as early as the ENIAC
into a data center is delivered to power electronics, and only mainframe system. In [26], architectures and power electron-
indirectly to IT equipment. The loss in power conversion ics designed for electric ship applications are suggested as a
is not really energy delivered to IT loads. Given an overall basis for dc applications in data centers. In [27], a supercom-
power conversion efficiency as high as 95% (unlikely), a puter powered directly from a 380 V dc solar energy bus is
PUE below 1.05 is not physically valid unless power sup- described.
plies are considered to be IT load. This also ignores any One important advantage of building-level dc distribution
energy consumed within uninterruptible supplies and even is that megawatt-scale rectifiers can be designed with much
building switchgear and wiring. higher efficiency that rack-level kilowatt-scale devices [28].
Google reports a much more realistic company-wide PUE Building-level conversion also simplifies reliability design.
of 1.12 [16], and also reports there that Redundancy at the building level is relatively straightfor-
“When measuring our IT equipment power, we include ward, and it is generally accepted that single points of failure
only the servers, storage, and networking equipment. We need to be avoided.
consider everything else overhead power. … Similarly, we The trend toward dc distribution in data centers is part
measure total utility power at the utility side of the substa- of a larger discussion of dc distribution within low-energy
tion, and therefore include substation transformer losses in buildings [29]. Early efforts related to data centers and more
our PUE.” broadly to buildings were conducted by Lawrence Berkeley
The distinction between IT and non-IT equipment is im- National Lab. These activities have culminated in an interna-
portant when PUE values are used to incentivize system im- tional industry group, the Emerge Alliance [30]. The energy
provements. savings are one aspect, even if the amounts are not large [31].
Liquid immersion in data centers can take at least two At present, dc distribution in a data center is not confined to
forms. As in [15], the individual boards or blades can be research. Commercial systems are in place [32].
immersed in coolant. Advanced hydrofluorocarbon liquids It is important to notice that PUE in particular, and also dc
and mineral oils have been used for this [17] (claiming PUE distribution practices, do not by themselves drive power in-
P. T. KREIN et al.: DATA CENTER CHALLENGES AND THEIR POWER ELECTRONICS 41

novation in data centers. Even though PUE is a limited met- structure. Building-level HVAC and lighting are not shown.
ric, it does give linkage to some efficiency improvements, The advantage of an online UPS in this context is high
particularly those associated with HVAC. Starting from rel- power quality and excellent immunity to short-term chang-
atively high PUE numbers (above 1.5), several strategies for es on the grid side. Not shown is a transfer switch that can
energy savings have been suggested by Pouchet [33]: connect in local fuel-driven generation if the utility feed or
• Optimize air flow throughout the building. feeds are lost. The disadvantages are that the UPS must have
• Optimize air flow within each rack. inherently high reliability, and that the multiple conversion
• Extra attention to building outer envelope and room stages involve power loss.
sealing. Offline or “standby” UPS circuits are also in wide use in
• Economizers in the heating, ventilation, and air-condi- data centers. Various hybrid configurations are employed
tioning (HVAC) system. to speed the process of transferring the energy source from
A combination of cooling from ambient air and inexpensive the grid to a battery set [35]. Recent innovations include
energy has driven many data center locations [34]. smaller intermediate converters that support the load during
the transfer interval [36]. This can also be supported with
C. Backup Level passive filters combined with small converters [37]. Grid-in-
teractive standby architectures allow fast source transfer
In conventional practice, a UPS is provided at the build- without having the UPS in the continuous energy conversion
ing level, and represents an important element in overall path [38].
efficiency. The basic electrical distribution scheme in a typ- Notice that in the online UPS case, power is converted at
ical ac data center is shown in Fig. 1. Notice in particular least four times from the ac mains to the IC level. Given es-
the stack-up of power electronics. Just behind the building timates such as a 97% efficient UPS process, a 95% efficient
feeder, an online UPS (also termed a “double conversion” rack-level rectifier, and POL converters at typically 93% or
UPS) provides back-to-back conversion, with an input-side less, overall about 85% of energy, at best, reaches the final
rectifier, regulated battery bus, and output-side inverter. This IT loads. This would limit the PUE to more than 1.17 even
provides high-quality power conditioning and backup. At the without considering HVAC – provided IT power does not
rack level, ac may be connected to individual server blades, include power supply input.
or an active rectifier supports a dc bus for the complete rack, A direct dc architecture, in which the “dc UPS” could be
usually at 48 V. At the board level, a relatively low voltage as simple as a master regulated dc bus, is shown in Fig. 2. In
is used to send power across board traces. The final point- this case, the building rectifier efficiency exceeds 98%, the
of-load (POL) converters supply the digital chip sets. Other rack level might reach 97%, and the POL efficiency is still
POL converters produce supplies for communication links, 93%. The total is about 88%, and the PUE will be more than
disk drives, diagnostic devices, sensors, and other IT infra- 1.13.

Fig. 1. Typical conversion sequence in an ac data center architecture [5].

Given these limitations on power loss and performance, level, and POL level, but the system is simpler and adapts
how, for example, does Google achieve system-wide PUE of better over a wide load range. The effective PUE improves
1.12 [16]? Part of the answer is a distributed UPS architec- by a few points. Many other users have adopted the distrib-
ture [38]-[40]. In the initial 2009 announcement [38], Goo- uted approach.
gle reported the use of small 12 V batteries directly within
each individual server. The result has higher backup perfor- D. Rack Level
mance than a building-level online UPS, since it is distribut-
ed among all computing loads, but avoids power conversion Servers in a data center are organized into conventional rack
layers built into conventional UPS architectures. Strictly mounts, or in high-density blade configurations. A 42-unit
speaking, the overall system efficiency is the same as in Fig. (42U) rack is typical. In an ac data center, each rack might be
2 because there is still conversion at the building level, rack provisioned with three or four three-phase circuit sets, provid-
42 CPSS TRANSACTIONS ON POWER ELECTRONICS AND APPLICATIONS, VOL. 2, NO. 1, MARCH 2017

Fig. 2. Typical conversion sequence in a dc data center architecture.

ing up to 25 kW per rack. In a dc data center, nominal dc volt- Better integration of power and thermal management at
age such as 380 V is delivered directly to each rack, providing the rack level has been discussed as a power saving tool
up to about 30 kW. In each case, power is often delivered [59], [60]. In [59], local rack cooling is linked to computing
directly to server-level power supplies. loads. The intention is to make total system power track the
An important challenge in rack provisioning is to man- computational load. In [60], methods for server power inter-
age various rating stackups. Standards and codes limit the connection within a rack, to reduce redundancy but main-
loading level of any circuit. Servers are likely to have dual tain power reliability, are presented. The gains in energy
redundant supplies. The supplies themselves are oversized, performance from interconnection do not approach those of
and servers do not always operate close to 100% power. a complete series connection, but linkages between system
The combination generates considerable confusion when power and computing power are important. This is linked to
rack-level power provisioning is designed [42]. Worst-case the concept of energy proportional computing, as described
design leads to extreme over-provisioning [43]. by Google [61].
Rack distribution offers several opportunities for advanced
power electronics. For example, supplies within a rack should E. Board Level
be sequenced to avoid combined inrush peaks or other short-
term system stresses. Faults should be managed to avoid Individual server boards often employ one or two power
propagation. Protection at the rack level is especially chal- supplies, known in the data center industry as power sup-
lenging with dc distribution [44]. Supplies must be efficient ply units (PSUs), within the server enclosure. Two are used
over a wide load range, given the limited full power opera- when direct “1+1” redundancy is required. As in [38], many
tion of servers. Several approaches have been proposed to modern designs use battery backup directly at this level. As
broaden the range of high efficiency [45]-[47], but rack-level in [48], there is continuing discussion of 48 V supplies at the
power considerations are not quite the same as board-level board level. Recent results [62] deliver 48 V directly to end-
issues discussed below. use POL converters.
The ongoing discussion about 48 V power for data centers At the board level, power supplies previously had various
[48] mainly considers board-level innovations. However, 48 designations such as “silver,” “gold,” or “platinum,” and
V rack distribution is discussed [22], and direct board 48 V there have been various industry initiatives for improvement
power would eliminate one power conversion level in the [63], [64]. Wide ranges for high efficiency are more import-
chain. ant than full-load efficiency [45], especially in redundant
A more radical rack-level approach is to connect individu- supply architectures in which the maximum output is 50%
al servers in series [49]-[51]. Because communications links unless a failure occurs. In fact, dual redundancy benefits
are electrically isolated, alternative electrical connections from high efficiency in the load range below 50% rather than
can be supported. A series stack of server boards or blades higher.
Most innovations at this level have come from incremen-
eliminates a layer of conversion and substantially increases
tal improvements in power supplies. This is particularly true
system efficiency [52], [53]. An oversimplified version of
because most architectures employ a supply that delivers 12
the approach would connect 32 servers in series, and connect
V, 5 V, 3.3 V, and other voltages similar to those in a desk-
them directly from a 380 V bus. This requires diligent load
top computer. It draws power either from a single-phase ac
balancing, but avoids a conversion step and enhances system
input or a direct 380 V dc bus. Some of the work applies to
efficiency. In [54], it is shown that the approach can raise
rack-level power or to multiple boards (as in a blade enclo-
this part of the power conversion process from a state-of-
sure) [65].
the-art 96% efficiency to more than 99.8% efficiency. Some
of the ideas appeared earlier [55], but the architectures have
much in common with older battery balancing techniques F. Chip Level
[56]-[58]. At the “chip level,” the power province of POL convert-
P. T. KREIN et al.: DATA CENTER CHALLENGES AND THEIR POWER ELECTRONICS 43

ers, high efficiency is challenging. This is because voltages The Intel “Haswell” processor concept was one of the first
are low, currents are high, and additional requirements such large-scale digital loads to incorporate internal power man-
as fast transient response trade off against low losses. Typical agement [83], [84]. As digital supply voltages drop below 1
solutions use two-stage dc-dc converters with an initial 12 V V, on-chip management becomes increasingly important.
to 5 V stage followed by a multi-phase 5 V to 1 V stage [66]. Modern digital chips are likely to comprise many compu-
Other configurations can take advantage of high voltage con- tation cores or internal modules. If these can be connected
version ratios [67]. Some broader concepts are addressed in in other ways, such as in series [49], [55], large voltage
[68], [69], in which the hierarchy issues are considered. savings become possible. For instance, a chip with 61 cores,
Some of the earliest work on POL converters considered connected in series, can operate directly from a 48 V bus at
48 V input [70]-[72], even though implementation is much a nominal 0.79 V per core, eliminating all layers of point of
more recent [62]. In [70], the concept of multi-phase dc-dc load conversion. Such a connection does not eliminate ben-
converters is introduced. In [72], this is extended to a four- efits of DVFS, even though it involves more comprehensive
phase two-stage converter for 2 V output. Comprehensive system-level approaches to power conversion and manage-
work on design based on 48 V POL applications was pre- ment [85]. The series concept applies across many levels,
sented in [73], [74]. The two-stage concept is presented in with the highest benefits for core interconnections [86].
depth in [75]. Recent designs, such as [76], are mature and However, since electrical isolation is rare below the board
ready for applications. level, series connected racks are likely to have more impact
In POL circuits and at the chip level, droop control meth- in the short-term than series connected cores.
ods can be employed for enhanced power management. The internal level is well-suited to certain types of soft-
Voltage droop (termed adaptive voltage positioning) seeks ware-based data center energy management. Individual
to keep POL output impedance resistive, such that increas- cores can be switches on and off, or computing loads can be
ing load drops the supply voltage linearly. This is a helpful actively balanced among cores to best spread thermal stress-
innovation since finite output impedance is unavoidable, es. A potential innovation is cores that report or even predict
and resistive characteristics have advantages given dynamic their power requirements back to on-chip power converters.
disturbances. In digital loads, the effective load is resistive, The converters can anticipate requirements and make dy-
since energy proportional to an internal capacitance Cint is namic adjustments to deliver the best possible performance.
lost during each digital clock cycle. This means the load
power is
III. System-Level Management
2
Pload=fclock Cint V , (1) Even though power distribution and conversion in a data
center are inherently hierarchical processes, in the end the
the same as replacing the digital load with an effective resis- power is consumed by IT loads and combined management
tance Reff. Voltage droop helps the stabilize operation against of hardware and software provides the greatest possible ben-
disturbances. efits. The PUE metric is less useful at this level. There are
Adaptive voltage positioning or scaling can also refer to two aspects to consider:
an adaptive process by which the supply voltage at the chip 1) Is a data center dynamically managed to keep its energy
level is decreased as much as computational performance al- requirements as low as possible?
lows [77]. In an idealized version, a computational load runs 2) What about reduction of energy used per unit of compu-
a test protocol, and requests step-by-step reduction in supply tation?
voltage until excessive error rates are encountered. The chip For the first item, energy proportionality is important, but the
itself might store a map of voltage vs. load to deliver er- complete system architecture also matters. For the second,
ror-free performance at the lowest possible power. the linkage between power electronics and digital design is
Frequency droop for digital loads is not a power supply important.
strategy, but clock frequency adjustment or droop, as in (1), Strategies for keeping energy requirements as low as
can manage load requirements in a dynamic manner. This is possible at the system level tend to be unique to an imple-
part of the industry concept of dynamic voltage and frequen- mentation. For example, for the highest-performing thermal
cy scaling (DVFS) to rapidly adjust the power requirements designs, low energy consumption means good balance. A
of digital loads at the chip level [78]. The term DVFS ap- center operating at 10% of maximum design load in this
pears in more than 1000 papers on IEEExplore. Summaries case functions best if all internal elements also function at
can be found in [79], [80]. 10%. For power electronics, this is not typical, since there
are overhead requirements that begin to affect efficiency as
power decreases. For point-based thermal designs, typical
G. Internal Level
practice is to run racks as close to 100% power as possible,
At the internal level, within a chip, a recent trend is to setting other equipment to an idle state. This works well in
bring dc-dc conversion into the chip itself. The general pow- systems designed for high full-load efficiency, but the high
er-system-on-chip approach has been explored for a long loading can diminish reliability and leads to hot spots and
time [81], [82], but commercial implementation is recent. other potential failure mechanisms.
44 CPSS TRANSACTIONS ON POWER ELECTRONICS AND APPLICATIONS, VOL. 2, NO. 1, MARCH 2017

For reduced energy per computational unit, voltage re- EPRI/PSMA, Tech. Rep. 2007.
[7] U. Müller, K. Strunz, “Integrated reliability modeling for data center
duction reduces this value in a nonlinear manner, but there
infrastructures: A case study,” in Proc. IEEE PES Innovative Smart
are limits. Computation cores operating at near-threshold Grid Technologies Europe (ISGT Europe), 2012.
or sub-threshold conditions use much less power than cores [8] M. Kim, A. Kwasinski, V. Krishnamurthy, “A storage integrated
running at conventional supply levels. However, power con- modular power electronic interface for higher power distribution
availability,” IEEE Trans. Power Electronics, vol. 30, no. 5, pp.
version down to 0.3 V to 0.5 V levels associated with this
2645-2659, May 2015.
reduction are extremely hard to produce; series connections [9] A. Kwasinski, “Power electronic interfaces for ultra available dc
become important, even though they bring more compli- micro-grids,” in Proc. Int’l. Symp. Power Electronics for Distributed
cated management issues. The clock rates also must be re- Generation Systems, 2010, pp. 58-65.
[10] V. Rudolf, et al., “Methodology for EBSILON simulation studies of
duced, trading off performance in a multi-dimensional space.
on-site generation CHP systems for data centre,” in Proc. Int’l. Ener-
As pointed out in [87], the linkages between digital-side gy and Sustainability Conference (IESC), 2016.
“low-power design” and system-level considerations in data [11] F. Huang, J. Lu, J. Zheng and J. Baleynaud, “Feasibility of heat
centers are complicated. recovery for district heating based on cloud computing industrial
park,” in Proc. Int’l. Conf. Renewable Energy Research and Appli-
In the large-scale computing literature, there is a growing
cations (ICRERA), 2015, pp. 287-291.
body of work on system-level data center energy manage- [12] Keith E. Herold, and R. Radermacher, “Integrated power and cool-
ment. In [88], an early summary of management approaches ing systems for data centers,” in Proc. Intersociety Conf. Thermal
is presented. In [89], [90], high-level simulation is used in an Phenomena, 2002, pp. 808-811.
[13] A. Escobar, J. C. Balda, C. A. Busada, and D. Christal, “An indi-
energy management process.
rect matrix converter for CCHP microturbines in data center power
A critical observation is that energy saved at the point of systems,” in Proc. IEEE Int’l. Telecommunications Energy Conf.
end use, especially via reduction in energy required per unit (INTELEC), 2012.
of computation, provides extra dividends throughout a data [14] S. Everitt. (2014, May). Utilities and data centers do not mix – yet.
DatacenterDynamics.[Online]. Available: http://www.datacenterdy-
center. Even if PUE is defined all the way at the point of
namics.com/content-tracks/power-cooling/utilities-and-data-centers-
final chip connection, with a value as low as 1.12, reduced do-not-mix-yet/86565.fullarticle
consumption at the end offers extra benefits. In more con- [15] Two-phase immersion cooling -- A revolution in data center effi-
ventional scenarios, in which PUE does not include power ciency. 3M news article.[Online]. Available: http://multimedia.3m.
com/mws/media/1127920O/2-phase-immersion-coolinga-revolu-
supply losses and impacts, reduced computation energy has
tion-in-data-center-efficiency.pdf.
much larger impact at the input. Dynamic optimization has a [16] Efficiency: How we do it.[Online]. Available: https://www.google.
key role to play in long-term data center operation [91]. com/about/datacenters/efficiency/internal/
[17] D. Varma. Oil submersion cooling for today’s data centers. Green
Revolution Cooling.[Online]. Available: http://www.grcooling.com/
IV. Conclusion wp-content/uploads/2015/06/GRC_WP-CLICK-Oil_Sub_DCc.pdf
[18] B. Cutler, S. Fowers, J. Kramer, and E. Peterson, “Dunking the data
Data centers are large energy consumers, analogous to center,” IEEE Spectrum, Mar. 2017.
heavy manufacturing industries. Both are key economic [19] S. Mok, S. Kumar, R. R. Hutchins, and Y. K. Joshi, “Impact of a
drivers, and energy costs are crucial in both arenas. Energy rotary regenerative heat exchanger on energy efficiency of an air-
cooled data center,” in Proc. IEEE ITHERM Conf., 2016, pp. 1182-
reduction in data centers requires attention to power distribu- 1190.
tion and conversion across all hierarchy levels. A challenging [20] T. Gao, et al., “Innovative server rack design with bottom located
requirement is to combine hardware and software in opera- cooling unit,” in Proc. IEEE ITHERM Conf., 2016, pp. 1172-1181.
tion to achieve system-level optimization. Power electronics [21] X. Wang, M. Li, and Z. Wang, “Research and application of green
power system for new data centers,” in Proc. IEEE Int’l. Telecom-
plays a vital role in enabling intelligent system-level power munications Energy Conference (INTELEC), 2015.
management. [22] A. Pratt, P. Kumar, and T. V. Aldridge, “Evaluation of 400V DC
distribution in telco and data centers to improve energy efficiency,”
in Proc. IEEE Int’l. Telecommunications Energy Conference (IN-
References TELEC), 2007, pp. 32-39.
[23] V. Sithimolada, and P. W. Sauer, “Facility-level DC vs. typical ac
[1] L. Luo.(2015, March). Chinese Government calls for green data distribution for data centers: A comparative reliability study,” in
center catch-up. DatacenterDynamics. [Online]. Available: http:// Proc. TENCON, IEEE Region 10 Conference, 2010, pp. 2102-2107.
www.datacenterdynamics.com/content-tracks/design-build/chinese- [24] A. Kwasinski, “Evaluation of dc voltage levels for integrated infor-
government-calls-for-green-data-center-catch-up/93618.fullarticle mation technology and telecom power architectures,” in Proc. 4th
[2] A. Shehabi, et al., “United States data center energy usage report,” International Telecommunication - Energy special conference, Vien-
Lawrence Berkeley National Laboratory, Berkeley, CA, Tech. Rep. na, Austria, 2009.
LBNL-1005775, Aug. 2016.[Online]. Available: https://eta.lbl.gov/ [25] D. J. Becker, and B. J. Sonnenberg, “DC microgrids in buildings and
publications/united-states-data-center-energy data centers,” in Proc. IEEE Int’l. Telecommunications Energy Con-
[3] S. Chen, S. Irving, and L. Peng, “Operational cost optimization for ference (INTELEC), 2011.
cloud computing data centers using renewable energy,” IEEE Sys- [26] A. Kwasinski, “Advanced power electronics enabled distribution
tems Journal, vol. 10, no. 4, pp. 1447-1458, 2016. architectures: Design, operation, and control,” in Proc. International
[4] M. Dayarathna, Y. Wen, and R. Fan, “Data center energy consump- Conference on Power Electronics/IEEE ECCE Asia, 2011, pp. 1484-
tion modeling: A survey,” IEEE Communications Surveys & Tutori- 1491.
als, vol. 18, no. 1, pp. 732-794, 2016. [27] J. Salazar. (2016, September). New Hikari supercomputer starts
[5] P. T. Krein, “A discussion of data center power challenges across the solar HVDC. Phys.org news.[Online]. Available: https://phys.org/
system,” in Proc. Int’l. Conf. Energy Aware Computing, 2010. news/2016-09-hikari-supercomputer-solar-hvdc.html.
[6] “ Follow the power,’ a report on an energy efficiency workshop,” [28] L. Schrittwieser, J. W. Kolar, and T. B. Soeiro, “99% efficient three-
P. T. KREIN et al.: DATA CENTER CHALLENGES AND THEIR POWER ELECTRONICS 45

phase buck-type SiC MOSFET PFC rectifier minimizing life cycle systems,” IEEE Trans. Power Electronics, vol. 28, no. 4, pp. 1795-
cost in DC data centers,” in Proc. IEEE Int’l. Telecommunications 1806, April 2013.
Energy Conference (INTELEC), 2016. [51] J. McClurg, Y. Zhang, J. Wheeler, and R. Pilawa-Podgurski,
[29] B. Nordman, and K. Christensen, “Dc local power distribution,” “Re-thinking data center power delivery: Regulating series-connect-
IEEE Electrification Magazine, Jun. 2016, pp. 29-36. ed voltage domains in software,” in Proc. IEEE Power and Energy
[30] [Online]. Available: http://www.emergealliance.org/. Conference at Illinois (PECI), 2013, pp. 147-154.
[31] P. Donovan. (2013, September). AC/DC: Highway to data center ef- [52] E. Candan, P. S. Shenoy, and R. C. N. Pilawa-Podgurski, “A se-
ficiency hell.[Online]. Available: http://blog.schneider-electric.com/ ries-stacked power delivery architecture with isolated differential
datacenter/2013/09/09/acdc-highway-data-center-efficiency-hell/. power conversion for data centers,” in Proc. IEEE Int’l. Telecommu-
[32] D. Tuite. (2014, February). 400-V DC distribution in the data center nications Energy Conference (INTELEC), 2014.
gets real. Electronic Design. [Online]. Available: http://electronicde- [53] E. Candan, D. Heeger, P. Shenoy, and R. Pilawa-Podgurski,
sign.com/power/400-v-dc-distribution-data-center-gets-real-0. “Hot-swapping analysis and implementation of series-stacked server
[33] J. Pouchet, “Data center power quality,” in Power Sources Manufac- power delivery architectures,” IEEE Trans. Power Electronics, to be
turers’ Association (PSMA), “Follow the Power — A Report on an published 2017.
Energy Efficiency Workshop from Generator to Integrated Circuit,” [54] E. Candan, P. S. Shenoy, and R. C. N. Pilawa-Podgurski, “A se-
PSMA/EPRI, Tech. Rep., Feb. 2007. ries-stacked power delivery architecture with isolated differential
[34] S. Grundberg, and N. Rolander, “For data center, google goes for the power conversion for data centers,” IEEE Trans. Power Electronics,
cold,” The Wall Street Journal, Sept. 2011. vol. 31, no. 5, pp. 3690-3703, May 2016.
[35] S. Karve, “Three of a kind [UPS topologies],” IEE Review, vol. 46, [55] S. Rajapandian, K. L. Shepard, P. Hazucha, and T. Karnik,
no. 2, pp. 27-31, 2000. “High-voltage power delivery through charge recycling,” IEEE J.
[36] M. Arias, M. M. Hernando, D. G. Lamar, J. Sebastian, and A. Fer- Solid-State Circuits, vol. 41, no. 6, pp. 1400-1410, Jun. 2006.
nandez, “Elimination of the transfer-time effects in line-interactive [56] S. T. Hung, D. C. Hopkins, and C. R. Mosling, “Extension of battery
and passive standby UPSs by means of a small-size inverter,” IEEE life via charge equalization control,” IEEE Trans. Industrial Elec-
Trans. Power Electronics, vol. 27, no. 3, pp. 1468-1478, Mar. 2012. tronics, vol. 40, no. 1, pp. 96-104, Feb. 1993.
[37] M. Arias Perez de Azpeitia, A. Fernandez, D. G. Lamar, M. Rodri- [57] G. L. Brainard, “Nondissipative battery charger equalizer,” U.S. Pat-
guez, and M. M. Hernando, “Simplified voltage-sag filler for line-in- ent 5,479,083, Dec. 26, 1995.
teractive uninterruptible power supplies,” IEEE Trans. Industrial [58] C. Pascual, and P. Krein, “Switched capacitor system for automatic
Electronics, vol. 55, no. 8, pp. 3005-3011, Aug. 2008. series battery equalization,” in Proc. IEEE Appl. Power Electronics
[38] H. Tao, J. L. Duarte, and M. A. M. Hendrix, “Control of grid-interac- Conf. (APEC), Feb 1997, pp. 848-854.
tive inverters as used in small distributed generators,” in Rec. IEEE [59] S. M. Yuan, C. Y. Chiang, M. J. Chen, and Y. L. Chen, “Automatic
Industry Applications Annual Meeting, 2007, pp. 1574-1581. rack cabinet management system for data center,” in Proc. IEEE
[39] S. Shankland. (2009, April). Google uncloaks once-secret server. Int’l. Symp. Consumer Electronics (ISCE), 2013, pp. 181-182.
CNet.[Online]. Available: https://www.cnet.com/news/google-un- [60] W. Kwon, and H. Kim, “Efficient server power supply configuration
cloaks-once-secret-server-10209580/ for cloud computing data center,” in Proc. Int’l. Conf. Computation-
[40] V. Kontorinis, L. E. Zhang, B. Aksanli, J. Sampson, H. Homayoun, E. al Science and Computational Intelligence, 2014, pp. 299-300.
Pettis, D. M. Tullsen, and T. S. Rosing, “Managing distributed UPS [61] L. A. Barroso, and U. Hölzle, “The case for energy-proportional
energy for effective power capping in data centers,” in Proc. 2012 computing,” IEEE Computer, vol. 40, no. 12, pp. 33-37, Dec. 2007.
39th Annual International Symposium on Computer Architecture [62] S. Jiang, X. Li, “Google 48 V power architecture,” in Proc. IEEE
(ISCA), 2012, pp. 488-499. Applied Power Electronics Conf. (APEC), 2017.
[41] S. Alanazi, M. Dabbagh, B. Hamdaoui, M. Guizani, and N. Zorba, [63] C. Calwell, A. Mansoor, and R. Keefe, “Active mode power supply
“Joint resource scheduling and peak power shaving for cloud data efficiency: key issues, measured data and the design competition
centers with distributed uninterruptible power supply,” in IEEE Glo- opportunity,” in Proc. IEEE Applied Power Electronics Conference
becom Workshops, 2016. (APEC), 2004, pp. 323-328.
[42] How many amps are needed for a rack (48U) of servers? (2016, [64] C. Calwell, and A. Mansoor, “AC-DC server power supplies: mak-
June). answered by C. Goolsbee. [Online]. Available: https://www. ing the leap to higher efficiency,” in Proc. IEEE Applied Power
quora.com/How-many-amps-are-needed-for-a-rack-48U-of-servers. Electronics Conference (APEC), 2005, pp. 155-158.
[43] E. M. Landsman, “Scalable data center architecture for on-demand [65] J. Aguillon-Garcia, and G. W. Moon, “A high-efficiency three-phase
power infrastructure,” in Proc. IEEE Applied Power Electronics ZVS PWM converter utilizing a positive double-star active rectifier
Conf. (APEC), 2003, pp. 10-13. stage for server power supply,” IEEE Trans. Industrial Electronics,
[44] K. Tan, X. Song, C. Peng, P. Liu, and A. Q. Huang, “Hierarchical vol. 58, no. 8, pp. 3317-3329, Aug. 2011.
protection architecture for 380V DC data center application,” in [66] Y. Ren, M. Xu, K. Yao, Y. Meng, and F. C. Lee, “Two-stage ap-
Proc. IEEE Energy Conversion Congress and Exposition (ECCE), proach for 12-V voltage regulator,” IEEE Trans. Power Electronics,
2016. vol. 19, no. 6, pp. 1498-1506, 2004.
[45] Power-One, Inc., PFE1100-12-054NA 54 MM 1U PSU in the Serv- [67] J. W. Kimball, J.T. Mossoba, and P.T. Krein, “A stabilizing, high-per-
er Environment. Peritus Power white paper PFE1100-12-054N, formance controller for input series-output parallel converter,” IEEE
pp. 9-10. [Online]. Available: http://plugloadsolutions.com/docu- Trans. Power Electronics, vol 23, no. 3, pp. 1416-1427, 2008.
ments/20101122_1U-54mm_whitepaper_9.pdf. [68] C. Belady, and C. Malone, “Data center power projections to 2014,”
[46] J. W. Kim, D. Y. Kim, C. E. Kim, and G. W. Moon, “A simple in Proc. 10th Intersociety Conf. Thermal and Thermomechanical
switching control technique for improving light load efficiency in Phenomena in Electronics Systems, 2006, pp. 439-444.
a phase-shifted full-bridge converter with a server power system,” [69] X. Wang, M. Chen, C. Lefurgy, and T. W. Keller, “SHIP: Scalable
IEEE Trans. Power Electronics, vol. 29, no. 4, pp. 1562-1566, Apr. hierarchical power control for large-scale data centers,” in Proc. 18th
2014. Int’l. Conf. Parallel Architectures and Compilation Techniques,
[47] Y. Jang, and M. M. Jovanovic, “Light-load efficiency optimization 2009, pp. 91-100.
method,” IEEE Trans. Power Electron., vol. 25, no. 1, pp. 67-74, [70] L. F. Casey, and M. F. Schlecht, “A high-frequency, low volume,
Jan. 2010. point-of-load power supply for distributed power systems,” IEEE
[48] R. Merritt, “Google, intel prep 48V servers,” EE Times, Jan. 21, Trans. Power Electronics, vol. 3, no. 1, pp. 72-82, Jan. 1988.
2016. [71] B. A. Miwa, L. F. Casey, and M. F. Schlecht, “Copper-based hybrid
[49] P. S. Shenoy, S. Zhang, R. A. Abdallah, P. T. Krein, and N. R. Shan- fabrication of a 50 W, 5 MHz 40 V-5 V DC/DC converter,” IEEE
bhag, “Overcoming the power wall: Connecting voltage domains in Trans. Power Electronics, vol. 6, no. 1, pp. 2-10, Jan. 1991.
series,” in Proc. Int’l. Conf. Energy Aware Computing, 2011. [72] P. T. Krein, P. Midya, and U. Ekambaram, “A distributed low-volt-
[50] P. S. Shenoy, and P. T. Krein, “Differential power processing for DC age power converter,” Univ. of Illinois, Urbana, Tech. Rep. UI-
46 CPSS TRANSACTIONS ON POWER ELECTRONICS AND APPLICATIONS, VOL. 2, NO. 1, MARCH 2017

LU-ENG-93–2563, 1993. Symposium on Low Power Electronics and Design, 2011, pp. 109-
[73] M. Ye, P. Xu, B. Yang, and F. C. Lee, “Investigation of topology can- 114.
didates for 48 V VRM,” in Proc. IEEE Applied Power Electronics [88] L. Benini, A. Bogliolo, and G. De Micheli, “A survey of design tech-
Conference (APEC), 2002, pp. 699-705. niques for system-level dynamic power management,” IEEE Trans.
[74] P. Xu, M. Ye, P. L. Wong, and F. C. Lee, “Design of 48 V voltage Very Large Scale Integration (VLSI) Systems, vol. 8, no. 3, pp. 299-
regulator modules with a novel integrated magnetics,” IEEE Trans. 316, June 2000.
Power Electronics, vol. 17, no. 6, pp. 990-998, Nov. 2002. [89] Y. Shi, X. Jiang, and K. Ye, “An energy-efficient scheme for cloud
[75] Y. Ren, M. Xu, K. Yao, and F. C. Lee, “Two-stage 48 V power pod resource provisioning based on CloudSim,” in Proc. IEEE Int’l.
exploration for 64-bit microprocessor,” in Proc. IEEE Applied Pow- Conf. Cluster Computing, 2011, pp. 595-599.
er Electronics Conf. (APEC), 2003, pp. 426-431. [90] D. Meisner, J. Wu, and T. F. Wenisch, “BigHouse: A simulation
[76] C. Fei, M. H. Ahmed, F. C. Lee, and Q. Li, “Two-stage 48 V-12 V/6 infrastructure for data center systems,” in Proc. IEEE Int’l. Symp.
V-1.8 V voltage regulator module with dynamic bus voltage control Performance Analysis of Systems & Software, 2012, pp. 35-45.
for light-load efficiency improvement,” IEEE Trans. Power Elec- [91] C. Li, et al., “Towards sustainable in-situ server systems in the big
tronics, vol. 32, no. 7, pp. 5628-5636, Jul. 2017. data era,” in Proc. ACM/IEEE Ann. Int’l. Symp. Computer Architec-
[77] V. Sharma, A. Thomas, T. Abdelzaher, K. Skadron, and Z. Lu, ture (ISCA), 2015, pp. 14-26.
“Power-aware QoS management in Web servers,” in Proc. IEEE Re-
al-Time Systems Symposium (RTTS), 2003, pp. 63-72.
[78] M. Shojafar, C. Canali, R. Lancellotti, and E. Baccarelli, “Minimizing
computing-plus-communication energy consumptions in virtualized
networked data centers,” in Proc. IEEE Symposium on Computers Philip T. Krein received the B.S. degree in electri-
and Communication (ISCC), 2016, pp. 1137-1144. cal engineering and the A.B. degree in economics
[79] C. Isci, A. Buyuktosunoglu, and M. Martonosi, “Long-term work- and business from Lafayette College, Easton, PA,
load phases: duration predictions and applications to DVFS,” IEEE USA, and the M.S. and Ph.D. degrees in electrical
Micro, vol. 25, no. 5, pp. 39-51, Sept.-Oct. 2005. engineering from the University of Illinois at Ur-
[80] J. Luis Nunez-Yanez, M. Hosseinabady, and A. Beldachi, “Energy bana-Champaign, USA. He was an engineer with
optimization in commercial FPGAs with voltage, frequency and log- Tektronix, Beaverton, OR, USA, and then returned
ic scaling,” IEEE Trans. Computers, vol. 65, no. 5, pp. 1484-1493, to the University of Illinois at Urbana-Champaign.
May 2016. He was a Senior Fulbright Scholar with the Uni-
[81] A. J. Stratakos, S. R. Sanders, and R. W. Brodersen, “A low-voltage versity of Surrey, Guildford, UK, from 1997 to
CMOS DC-DC converter for a portable battery-operated system,” in 1998. From 2003 to 2014 he was a Founder and Director of SolarBridge
Rec., IEEE Power Electronics Specialists Conf. (PESC), 1994, pp. Technologies, Inc., Austin, TX, USA, a developer of long-life integrated
619-626. inverters for solar energy, and now part of Sunpower. He holds the Grainger
[82] M. Ludwig, M. Duffy, T. O’Donnell, P. McCloskey, and S. C. O. Endowed Chair Emeritus in Electric Machinery and Electromechanics
Mathuna, “PCB integrated inductors for low power DC/DC converter,” and is Director of the Grainger Center for Electric Machinery and Electro-
IEEE Trans. Power Electronics, vol. 18, no. 4, pp. 937-945, Jul. 2003. mechanics at the University of Illinois at Urbana-Champaign. He is also
[83] G. Schrom, et al., “A 480-MHz, multi-phase interleaved buck DC- Executive Dean of the Zhejiang University–University of Illinois Institute
DC converter with hysteretic control,” in Proc. IEEE Power Elec- for engineering in Haining, China. He holds 38 U.S. patents with additional
tronics Specialists Conf. (PESC), 2004, pp. 4702-4707. patents pending. His current research interests include all aspects of power
[84] J. Hruska. (2013, May). Intel’s haswell takes a major step forward, electronics, machines, drives, electric transportation, and electrical energy,
integrates voltage regulator. HotHardware. [Online]. Available: with an emphasis on nonlinear control approaches.
http://hothardware.com/news/haswell-takes-a-major-step-for- Dr. Krein is a Registered Professional Engineer in Illinois and Oregon.
ward-integrates-voltage-regulator. In 2001, he helped initiate the International Future Energy Challenge, a
[85] E. Candan, P. S. Shenoy, and R. C. N. Pilawa-Podgurski, “Unregu- major student competition involving fuel cell power conversion and ener-
lated bus operation of server-to-virtual bus differential power pro- gy efficiency. In 2003, he received the IEEE William E. Newell Award in
cessing for data centers,” in Proc. IEEE Applied Power Electronics Power Electronics. He is a past President of the IEEE Power Electronics
Conf. (APEC), 2017, pp. 1632-1639. Society, and served as a member of the IEEE Board of Directors. He is
[86] P. T. Krein, R. H. Campbell, N. R. Shanbhag, “System and method Editor-at-Large of the IEEE Transactions on Power Electronics and an
for improving power conversion for advanced electronic circuits,” Associate Editor of the IEEE Journal of Emerging and Selected Topics in
U.S. Patent 9,116,692, Aug. 25, 2015. Power Electronics. In 2015-2016, he was Chair of the IEEE Transportation
[87] D. Meisner, and T. F. Wenisch, “Does low-power design imply en- Electrification Community. He was elected to the U.S. National Academy
ergy efficiency for data centers?” in Proc. IEEE/ACM International of Engineering in 2016.