Energy and Policy Considerations for Deep Learning in NLP

Emma Strubell Ananya Ganesh Andrew McCallum

College of Information and Computer Sciences
University of Massachusetts Amherst
{strubell, aganesh, mccallum}@cs.umass.edu

Abstract Consumption CO2 e (lbs)

Air travel, 1 passenger, NY↔SF 1984
Recent progress in hardware and methodol-
arXiv:1906.02243v1 [cs.CL] 5 Jun 2019

Human life, avg, 1 year 11,023

ogy for training neural networks has ushered
in a new generation of large networks trained American life, avg, 1 year 36,156
on abundant data. These models have ob- Car, avg incl. fuel, 1 lifetime 126,000
tained notable gains in accuracy across many
NLP tasks. However, these accuracy improve- Training one model (GPU)
ments depend on the availability of exception- NLP pipeline (parsing, SRL) 39
ally large computational resources that neces- w/ tuning & experimentation 78,468
sitate similarly substantial energy consump-
Transformer (big) 192
tion. As a result these models are costly to
train and develop, both financially, due to the w/ neural architecture search 626,155
cost of hardware and electricity or cloud com-
Table 1: Estimated CO2 emissions from training com-
pute time, and environmentally, due to the car-
mon NLP models, compared to familiar consumption.1
bon footprint required to fuel modern tensor
processing hardware. In this paper we bring
this issue to the attention of NLP researchers
NLP models could be trained and developed on
by quantifying the approximate financial and
environmental costs of training a variety of re- a commodity laptop or server, many now require
cently successful neural network models for multiple instances of specialized hardware such as
NLP. Based on these findings, we propose ac- GPUs or TPUs, therefore limiting access to these
tionable recommendations to reduce costs and highly accurate models on the basis of finances.
improve equity in NLP research and practice. Even when these expensive computational re-
sources are available, model training also incurs a
1 Introduction
substantial cost to the environment due to the en-
Advances in techniques and hardware for train- ergy required to power this hardware for weeks or
ing deep neural networks have recently en- months at a time. Though some of this energy may
abled impressive accuracy improvements across come from renewable or carbon credit-offset re-
many fundamental NLP tasks (Bahdanau et al., sources, the high energy demands of these models
2015; Luong et al., 2015; Dozat and Man- are still a concern since (1) energy is not currently
ning, 2017; Vaswani et al., 2017), with the derived from carbon-neural sources in many loca-
most computationally-hungry models obtaining tions, and (2) when renewable energy is available,
the highest scores (Peters et al., 2018; Devlin et al., it is still limited to the equipment we have to pro-
2019; Radford et al., 2019; So et al., 2019). As duce and store it, and energy spent training a neu-
a result, training a state-of-the-art model now re- ral network might better be allocated to heating a
quires substantial computational resources which family’s home. It is estimated that we must cut
demand considerable energy, along with the as- carbon emissions by half over the next decade to
sociated financial and environmental costs. Re- deter escalating rates of natural disaster, and based
search and development of new models multiplies on the estimated CO2 emissions listed in Table 1,
these costs by thousands of times by requiring re- 1
Sources: (1) Air travel and per-capita consump-
training to experiment with model architectures tion: https://bit.ly/2Hw0xWc; (2) car lifetime:
and hyperparameters. Whereas a decade ago most https://bit.ly/2Qbr0w1.
model training and development likely make up Consumer Renew. Gas Coal Nuc.
a substantial portion of the greenhouse gas emis- China 22% 3% 65% 4%
sions attributed to many NLP researchers. Germany 40% 7% 38% 13%
To heighten the awareness of the NLP commu- United States 17% 35% 27% 19%
nity to this issue and promote mindful practice and Amazon-AWS 17% 24% 30% 26%
policy, we characterize the dollar cost and carbon Google 56% 14% 15% 10%
emissions that result from training the neural net- Microsoft 32% 23% 31% 10%
works at the core of many state-of-the-art NLP
models. We do this by estimating the kilowatts Table 2: Percent energy sourced from: Renewable (e.g.
of energy required to train a variety of popular hydro, solar, wind), natural gas, coal and nuclear for
off-the-shelf NLP models, which can be converted the top 3 cloud compute providers (Cook et al., 2017),
to approximate carbon emissions and electricity compared to the United States,4 China5 and Germany
(Burger, 2019).
costs. To estimate the even greater resources re-
quired to transfer an existing model to a new task
or develop new models, we perform a case study We estimate the total time expected for mod-
of the full computational resources required for the els to train to completion using training times and
development and tuning of a recent state-of-the-art hardware reported in the original papers. We then
NLP pipeline (Strubell et al., 2018). We conclude calculate the power consumption in kilowatt-hours
with recommendations to the community based on (kWh) as follows. Let pc be the average power
our findings, namely: (1) Time to retrain and sen- draw (in watts) from all CPU sockets during train-
sitivity to hyperparameters should be reported for ing, let pr be the average power draw from all
NLP machine learning models; (2) academic re- DRAM (main memory) sockets, let pg be the aver-
searchers need equitable access to computational age power draw of a GPU during training, and let
resources; and (3) researchers should prioritize de- g be the number of GPUs used to train. We esti-
veloping efficient models and hardware. mate total power consumption as combined GPU,
CPU and DRAM consumption, then multiply this
2 Methods by Power Usage Effectiveness (PUE), which ac-
counts for the additional energy required to sup-
To quantify the computational and environmen-
port the compute infrastructure (mainly cooling).
tal cost of training deep neural network mod-
We use a PUE coefficient of 1.58, the 2018 global
els for NLP, we perform an analysis of the en-
average for data centers (Ascierto, 2018). Then the
ergy required to train a variety of popular off-
total power pt required at a given instance during
the-shelf NLP models, as well as a case study of
training is given by:
the complete sum of resources required to develop
LISA (Strubell et al., 2018), a state-of-the-art NLP 1.58t(pc + pr + gpg )
pt = (1)
model from EMNLP 2018, including all tuning 1000
and experimentation. The U.S. Environmental Protection Agency (EPA)
We measure energy use as follows. We train the provides average CO2 produced (in pounds per
models described in §2.1 using the default settings kilowatt-hour) for power consumed in the U.S.
provided, and sample GPU and CPU power con- (EPA, 2018), which we use to convert power to
sumption during training. Each model was trained estimated CO2 emissions:
for a maximum of 1 day. We train all models on
a single NVIDIA Titan X GPU, with the excep- CO2 e = 0.954pt (2)
tion of ELMo which was trained on 3 NVIDIA
This conversion takes into account the relative pro-
GTX 1080 Ti GPUs. While training, we repeat-
portions of different energy sources (primarily nat-
edly query the NVIDIA System Management In-
ural gas, coal, nuclear and renewable) consumed
terface2 to sample the GPU power consumption
to produce energy in the United States. Table 2
and report the average over all samples. To sample
lists the relative energy sources for China, Ger-
CPU power consumption, we use Intel’s Running
many and the United States compared to the top
Average Power Limit interface.3
5
U.S. Dept. of Energy: https://bit.ly/2JTbGnI
2 5
nvidia-smi: https://bit.ly/30sGEbi China Electricity Council; trans. China Energy Portal:
3
RAPL power meter: https://bit.ly/2LObQhV https://bit.ly/2QHE5O3
three cloud service providers. The U.S. break- ence. Devlin et al. (2019) report that the BERT
down of energy is comparable to that of the most base model (110M parameters) was trained on 16
popular cloud compute service, Amazon Web Ser- TPU chips for 4 days (96 hours). NVIDIA reports
vices, so we believe this conversion to provide a that they can train a BERT model in 3.3 days (79.2
reasonable estimate of CO2 emissions per kilowatt hours) using 4 DGX-2H servers, totaling 64 Tesla
hour of compute energy used. V100 GPUs (Forster et al., 2019).
GPT-2. This model is the latest edition of
2.1 Models OpenAI’s GPT general-purpose token encoder,
We analyze four models, the computational re- also based on Transformer-style self-attention and
quirements of which we describe below. All mod- trained with a language modeling objective (Rad-
els have code freely available online, which we ford et al., 2019). By training a very large model
used out-of-the-box. For more details on the mod- on massive data, Radford et al. (2019) show high
els themselves, please refer to the original papers. zero-shot performance on question answering and
language modeling benchmarks. The large model
Transformer. The Transformer model (Vaswani
described in Radford et al. (2019) has 1542M pa-
et al., 2017) is an encoder-decoder architecture
rameters and is reported to require 1 week (168
primarily recognized for efficient and accurate ma-
hours) of training on 32 TPUv3 chips. 6
chine translation. The encoder and decoder each
consist of 6 stacked layers of multi-head self-
attention. Vaswani et al. (2017) report that the 3 Related work
Transformer base model (65M parameters) was
trained on 8 NVIDIA P100 GPUs for 12 hours, There is some precedent for work characterizing
and the Transformer big model (213M parame- the computational requirements of training and in-
ters) was trained for 3.5 days (84 hours; 300k ference in modern neural network architectures in
steps). This model is also the basis for recent the computer vision community. Li et al. (2016)
work on neural architecture search (NAS) for ma- present a detailed study of the energy use required
chine translation and language modeling (So et al., for training and inference in popular convolutional
2019), and the NLP pipeline that we study in more models for image classification in computer vi-
detail in §4.2 (Strubell et al., 2018). So et al. sion, including fine-grained analysis comparing
(2019) report that their full architecture search ran different neural network layer types. Canziani
for a total of 979M training steps, and that their et al. (2016) assess image classification model ac-
base model requires 10 hours to train for 300k curacy as a function of model size and gigaflops
steps on one TPUv2 core. This equates to 32,623 required during inference. They also measure av-
hours of TPU or 274,120 hours on 8 P100 GPUs. erage power draw required during inference on
GPUs as a function of batch size. Neither work an-
ELMo. The ELMo model (Peters et al., 2018)
alyzes the recurrent and self-attention models that
is based on stacked LSTMs and provides rich
have become commonplace in NLP, nor do they
word representations in context by pre-training on
extrapolate power to estimates of carbon and dol-
a large amount of data using a language model-
lar cost of training.
ing objective. Replacing context-independent pre-
trained word embeddings with ELMo has been Analysis of hyperparameter tuning has been
shown to increase performance on downstream performed in the context of improved algorithms
tasks such as named entity recognition, semantic for hyperparameter search (Bergstra et al., 2011;
role labeling, and coreference. Peters et al. (2018) Bergstra and Bengio, 2012; Snoek et al., 2012). To
report that ELMo was trained on 3 NVIDIA GTX our knowledge there exists to date no analysis of
1080 GPUs for 2 weeks (336 hours). the computation required for R&D and hyperpa-
rameter tuning of neural network models in NLP.
BERT. The BERT model (Devlin et al., 2019) pro-
vides a Transformer-based architecture for build- 6
Via the authors on Reddit.
ing contextual representations similar to ELMo, 7
GPU lower bound computed using pre-emptible
but trained with a different language modeling ob- P100/V100 U.S. resources priced at $0.43–$0.74/hr, upper
jective. BERT substantially improves accuracy on bound uses on-demand U.S. resources priced at $1.46–
$2.48/hr. We similarly use pre-emptible ($1.46/hr–$2.40/hr)
tasks requiring sentence-level representations such and on-demand ($4.50/hr–$8/hr) pricing as lower and upper
as question answering and natural language infer- bounds for TPU v2/3; cheaper bulk contracts are available.
 Model Hardware Power (W) Hours kWh·PUE CO2 e Cloud compute cost
Transformerbase P100x8 1415.78 12 27 26 $41–$140
Transformerbig P100x8 1515.43 84 201 192 $289–$981
ELMo P100x3 517.66 336 275 262 $433–$1472
BERTbase V100x64 12,041.51 79 1507 1438 $3751–$12,571
BERTbase TPUv2x16 — 96 — — $2074–$6912
NAS P100x8 1515.43 274,120 656,347 626,155 $942,973–$3,201,722
NAS TPUv2x1 — 32,623 — — $44,055–$146,848
GPT-2 TPUv3x32 — 168 — — $12,902–$43,008

Table 3: Estimated cost of training a model in terms of CO2 emissions (lbs) and cloud compute cost (USD).7 Power
and carbon footprint are omitted for TPUs due to lack of public information on power draw for this hardware.

4 Experimental results Estimated cost (USD)

Models Hours Cloud compute Electricity
4.1 Cost of training
1 120 $52–$175 $5
Table 3 lists CO2 emissions and estimated cost of 24 2880 $1238–$4205 $118
training the models described in §2.1. Of note is 4789 239,942 $103k–$350k $9870
that TPUs are more cost-efficient than GPUs on
workloads that make sense for that hardware (e.g. Table 4: Estimated cost in terms of cloud compute and
BERT). We also see that models emit substan- electricity for training: (1) a single model (2) a single
tune and (3) all models trained during R&D.
tial carbon emissions; training BERT on GPU is
roughly equivalent to a trans-American flight. So
et al. (2019) report that NAS achieves a new state- about 60 GPUs running constantly throughout the
of-the-art BLEU score of 29.7 for English to Ger- 6 month duration of the project. Table 4 lists upper
man machine translation, an increase of just 0.1 and lower bounds of the estimated cost in terms
BLEU at the cost of at least $150k in on-demand of Google Cloud compute and raw electricity re-
compute time and non-trivial carbon emissions. quired to develop and deploy this model.9 We see
that while training a single model is relatively in-
4.2 Cost of development: Case study
expensive, the cost of tuning a model for a new
To quantify the computational requirements of dataset, which we estimate here to require 24 jobs,
R&D for a new model we study the logs of or performing the full R&D required to develop
all training required to develop Linguistically- this model, quickly becomes extremely expensive.
Informed Self-Attention (Strubell et al., 2018), a
multi-task model that performs part-of-speech tag- 5 Conclusions
ging, labeled dependency parsing, predicate detec-
tion and semantic role labeling. This model makes Authors should report training time and
for an interesting case study as a representative sensitivity to hyperparameters.
NLP pipeline and as a Best Long Paper at EMNLP. Our experiments suggest that it would be benefi-
Model training associated with the project cial to directly compare different models to per-
spanned a period of 172 days (approx. 6 months). form a cost-benefit (accuracy) analysis. To ad-
During that time 123 small hyperparameter grid dress this, when proposing a model that is meant
searches were performed, resulting in 4789 jobs to be re-trained for downstream use, such as re-
in total. Jobs varied in length ranging from a min- training on a new domain or fine-tuning on a new
imum of 3 minutes, indicating a crash, to a maxi- task, authors should report training time and com-
mum of 9 days, with an average job length of 52 putational resources required, as well as model
hours. All training was done on a combination of sensitivity to hyperparameters. This will enable
NVIDIA Titan X (72%) and M40 (28%) GPUs.8 direct comparison across models, allowing subse-
The sum GPU time required for the project quent consumers of these models to accurately as-
totaled 9998 days (27 years). This averages to sess whether the required computational resources
8 9
We approximate cloud compute cost using P100 pricing. Based on average U.S cost of electricity of $0.12/kWh.
are compatible with their setting. More explicit half the estimated cost to use on-demand cloud
characterization of tuning time could also reveal GPUs. Unlike money spent on cloud compute,
inconsistencies in time spent tuning baseline mod- however, that invested in centralized resources
els compared to proposed contributions. Realiz- would continue to pay off as resources are shared
ing this will require: (1) a standard, hardware- across many projects. A government-funded aca-
independent measurement of training time, such demic compute cloud would provide equitable ac-
as gigaflops required to convergence, and (2) a cess to all researchers.
standard measurement of model sensitivity to data
and hyperparameters, such as variance with re- Researchers should prioritize computationally
spect to hyperparameters searched. efficient hardware and algorithms.
We recommend a concerted effort by industry and
Academic researchers need equitable access to academia to promote research of more computa-
computation resources. tionally efficient algorithms, as well as hardware
that requires less energy. An effort can also be
Recent advances in available compute come at a made in terms of software. There is already a
high price not attainable to all who desire access. precedent for NLP software packages prioritizing
Most of the models studied in this paper were de- efficient models. An additional avenue through
veloped outside academia; recent improvements in which NLP and machine learning software de-
state-of-the-art accuracy are possible thanks to in- velopers could aid in reducing the energy asso-
dustry access to large-scale compute. ciated with model tuning is by providing easy-
Limiting this style of research to industry labs to-use APIs implementing more efficient alterna-
hurts the NLP research community in many ways. tives to brute-force grid search for hyperparameter
First, it stifles creativity. Researchers with good tuning, e.g. random or Bayesian hyperparameter
ideas but without access to large-scale compute search techniques (Bergstra et al., 2011; Bergstra
will simply not be able to execute their ideas, and Bengio, 2012; Snoek et al., 2012). While
instead constrained to focus on different prob- software packages implementing these techniques
lems. Second, it prohibits certain types of re- do exist,10 they are rarely employed in practice
search on the basis of access to financial resources. for tuning NLP models. This is likely because
This even more deeply promotes the already prob- their interoperability with popular deep learning
lematic “rich get richer” cycle of research fund- frameworks such as PyTorch and TensorFlow is
ing, where groups that are already successful and not optimized, i.e. there are not simple exam-
thus well-funded tend to receive more funding ples of how to tune TensorFlow Estimators using
due to their existing accomplishments. Third, the Bayesian search. Integrating these tools into the
prohibitive start-up cost of building in-house re- workflows with which NLP researchers and practi-
sources forces resource-poor groups to rely on tioners are already familiar could have notable im-
cloud compute services such as AWS, Google pact on the cost of developing and tuning in NLP.
Cloud and Microsoft Azure.
While these services provide valuable, flexi- Acknowledgements
ble, and often relatively environmentally friendly
We are grateful to Sherief Farouk and the anony-
compute resources, it is more cost effective for
mous reviewers for helpful feedback on earlier
academic researchers, who often work for non-
drafts. This work was supported in part by the
profit educational institutions and whose research
Centers for Data Science and Intelligent Infor-
is funded by government entities, to pool resources
mation Retrieval, the Chan Zuckerberg Initiative
to build shared compute centers at the level of
under the Scientific Knowledge Base Construc-
funding agencies, such as the U.S. National Sci-
tion project, the IBM Cognitive Horizons Network
ence Foundation. For example, an off-the-shelf
agreement no. W1668553, and National Science
GPU server containing 8 NVIDIA 1080 Ti GPUs
Foundation grant no. IIS-1514053. Any opinions,
and supporting hardware can be purchased for
findings and conclusions or recommendations ex-
approximately $20,000 USD. At that cost, the
pressed in this material are those of the authors and
hardware required to develop the model in our
do not necessarily reflect those of the sponsor.
case study (approximately 58 GPUs for 172 days)
10
would cost $145,000 USD plus electricity, about For example, the Hyperopt Python library.
References Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke
Rhonda Ascierto. 2018. Uptime Institute Global Data Zettlemoyer. 2018. Deep contextualized word rep-
Center Survey. Technical report, Uptime Institute. resentations. In NAACL.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-
gio. 2015. Neural Machine Translation by Jointly Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Learning to Align and Translate. In 3rd Inter- Dario Amodei, and Ilya Sutskever. 2019. Language
national Conference for Learning Representations models are unsupervised multitask learners.
(ICLR), San Diego, California, USA. Jasper Snoek, Hugo Larochelle, and Ryan P Adams.
James Bergstra and Yoshua Bengio. 2012. Random 2012. Practical bayesian optimization of machine
search for hyper-parameter optimization. Journal of learning algorithms. In Advances in neural informa-
Machine Learning Research, 13(Feb):281–305. tion processing systems, pages 2951–2959.

James S Bergstra, Rémi Bardenet, Yoshua Bengio, and David R. So, Chen Liang, and Quoc V. Le. 2019.
Balázs Kégl. 2011. Algorithms for hyper-parameter The evolved transformer. In Proceedings of the
optimization. In Advances in neural information 36th International Conference on Machine Learning
processing systems, pages 2546–2554. (ICML).

Bruno Burger. 2019. Net Public Electricity Generation Emma Strubell, Patrick Verga, Daniel Andor,
in Germany in 2018. Technical report, Fraunhofer David Weiss, and Andrew McCallum. 2018.
Institute for Solar Energy Systems ISE. Linguistically-Informed Self-Attention for Se-
mantic Role Labeling. In Conference on Empir-
Alfredo Canziani, Adam Paszke, and Eugenio Culur- ical Methods in Natural Language Processing
ciello. 2016. An analysis of deep neural network (EMNLP), Brussels, Belgium.
models for practical applications.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Gary Cook, Jude Lee, Tamina Tsai, Ada Kongn, John Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz
Deans, Brian Johnson, Elizabeth Jardim, and Brian Kaiser, and Illia Polosukhin. 2017. Attention is all
Johnson. 2017. Clicking Clean: Who is winning you need. In 31st Conference on Neural Information
the race to build a green internet? Technical report, Processing Systems (NIPS).
Greenpeace.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of
Deep Bidirectional Transformers for Language Un-
derstanding. In NAACL.
Timothy Dozat and Christopher D. Manning. 2017.
Deep biaffine attention for neural dependency pars-
ing. In ICLR.
EPA. 2018. Emissions & Generation Resource Inte-
grated Database (eGRID). Technical report, U.S.
Environmental Protection Agency.
Christopher Forster, Thor Johnsen, Swetha Man-
dava, Sharath Turuvekere Sreenivas, Deyu Fu, Julie
Bernauer, Allison Gray, Sharan Chetlur, and Raul
Puri. 2019. BERT Meets GPUs. Technical report,
NVIDIA AI.
Da Li, Xinbo Chen, Michela Becchi, and Ziliang Zong.
2016. Evaluating the energy efficiency of deep con-
volutional neural networks on cpus and gpus. 2016
IEEE International Conferences on Big Data and
Cloud Computing (BDCloud), Social Computing
and Networking (SocialCom), Sustainable Comput-
ing and Communications (SustainCom) (BDCloud-
SocialCom-SustainCom), pages 477–484.
Thang Luong, Hieu Pham, and Christopher D. Man-
ning. 2015. Effective approaches to attention-based
neural machine translation. In Proceedings of the
2015 Conference on Empirical Methods in Natural
Language Processing, pages 1412–1421. Associa-
tion for Computational Linguistics.