Scientific

Computing
Data Visualization
Information
Technology
Quantum
Computing

JULY 2020 www.computer.org

IEEE
Computer
Society Has
You Covered!
WORLD-CLASS CONFERENCES — Stay
ahead of the curve by attending one of our
200+ globally recognized conferences.

DIGITAL LIBRARY — Easily access over 780k

articles covering world-class peer-reviewed
content in the IEEE Computer Society
Digital Library.

CALLS FOR PAPERS — Discover

opportunities to write and present your
ground-breaking accomplishments.

EDUCATION — Strengthen your resume

with the IEEE Computer Society Course
Catalog and its range of offerings.

ADVANCE YOUR CAREER — Search the

new positions posted in the IEEE Computer
Society Jobs Board.

NETWORK — Make connections that count

by participating in local Region, Section,
and Chapter activities.

Explore all of the member benefits

at www.computer.org today!
 IEEE COMPUTER SOCIETY computer.org

STAFF
Editor Publications Portfolio Managers
Cathy Martin Carrie Clark, Kimberly Sperka

Publications Operations Project Specialist Publisher

Christine Anthony Robin Baldwin

Production & Design Artist Senior Advertising Coordinator

Carmen Flores-Garvey Debbie Sims

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th
Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380;
fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036.
Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage
Paid at New York, New York, and at additional mailing offices. Printed in USA.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in
ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style,
clarity, and space.
Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for
profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-
party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web
servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy.
An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy-
editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications
/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new
collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane,
Piscataway, NJ 08854-4141 or [email protected]. Copyright © 2020 IEEE. All rights reserved.
Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons,
provided the per-copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive,
Danvers, MA 01923.
Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at help@
computer.org and type “unsubscribe ComputingEdge” in your subject line.
IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Intelligent Systems IEEE Pervasive Computing

Jeff Voas, NIST V.S. Subrahmanian, Marc Langheinrich, Università
Dartmouth College della Svizzera italiana
Computing in Science
& Engineering IEEE Internet Computing IEEE Security & Privacy
Lorena A. Barba (Interim), George Pallis, University David Nicol, University
George Washington University of Cyprus of Illinois at
Urbana-Champaign
IEEE Annals of the History IEEE Micro
of Computing Lizy Kurian John, University IEEE Software
Gerardo Con Diaz, University of Texas at Austin Ipek Ozkaya, Software
of California, Davis Engineering Institute
IEEE MultiMedia
IEEE Computer Graphics Shu-Ching Chen, Florida IT Professional
and Applications International University Irena Bojanova, NIST
Torsten Möller,
Universität Wien

2469-7087/20 © 2020 IEEE Published by the IEEE Computer Society July 2020 1
JULY 2020 � VOLUME 6 � NUMBER 7

16
SciPipe—Turning
22
Weather Report: A
30
OpenSpace:
Scientific Site-Specific Artwork Bringing
Workflows Interweaving NASA
into Computer Human Experiences Missions to
Programs and Scientific Data the Public
Physicalization
 Scientific Computing
8 Metamorphic Testing: A Simple Yet Effective Approach
for Testing Scientific Software
UPULEE KANEWALA AND TSONG YUEH CHEN

16 SciPipe—Turning Scientific Workflows into Computer

Programs
SAMUEL LAMPA, MARTIN DAHLÖ, JONATHAN ALVARSSON, AND
OLA SPJUTH

Data Visualization
22 Weather Report: A Site-Specific Artwork Interweaving
Human Experiences and Scientific Data Physicalization
DANIEL F. KEEFE, SETH JOHNSON, ROSS ALTHEIMER, DEUK-GEUN HONG,
ROBERT HUNTER, ANDREA J. JOHNSON, MAURA ROCKCASTLE, MARK
SWACKHAMER, AND AARON WITTKAMPER

30 OpenSpace: Bringing NASA Missions to the Public

ALEXANDER BOCK, CHARLES HANSEN, AND ANDERS YNNERMAN

Information Technology
38 A Manifesto for Energy-Aware Software
ALCIDES FONSECA, RICK KAZMAN, AND PATRICIA LAGO

42 Next Generation IoT: Toward Ubiquitous Autonomous

Cost-Efficient IoT Devices
MOUSTAFA YOUSSEF AND MAHBUB HASSAN

Quantum Computing
46 Powerball and Quantum Supremacy
ERIK P. DEBENEDICTIS

50 Beyond Quantum Supremacy

ERIK P. DEBENEDICTIS

Departments
4 Magazine Roundup
7 Editor’s Note: Advancing Science with Software
56 Conference Calendar

Subscribe to ComputingEdge for free at

www.computer.org/computingedge.
 Magazine Roundup

T he IEEE Computer Society’s lineup of 12 peer-reviewed technical magazines covers cutting-edge topics rang-
ing from software design and computer graphics to Internet computing and security, from scientific appli-
cations and machine intelligence to visualization and microchip design. Here are highlights from recent issues.

In this article from the March/April were researched, implemented,

2020 issue of Computing in Sci- and launched into the market-
Challenges and Opportunities ence & Engineering, the authors’ place, where their intense compe-
in the Detection of Safety- experience shows that it is often tition transformed the 500-year
Critical Cyberphysical Attacks straightforward to first define the tradition of printing and publish-
multiple iterations of tests for per- ing—placing the electronic literacy
Cyberphysical systems (CPSs) forming continuous simulations, on the screens of billions of digital
are increasingly used in various and then keep multiple and even displays, computers, tablets, and
application domains and face the competing metamorphic relations smart phones around the world.
threat of cyberphysical attacks. In open for investigating the test- Read more in this article from the
this article from the March 2020 ing-result patterns. The authors January–March 2020 issue of IEEE
issue of Computer, the authors dis- call this new approach exploratory Annals of the History of Computing.
cuss challenges in detecting these MT, and they report their experi-
attacks. They use power grids and ence of applying it to detect bugs,
surgical robots to clarify their anal- mismatches, and constraints in
ysis, and they use this analysis to automatically calibrating parame- Illustrating Changes in Time-
identify ongoing challenges and ters for the United States Environ- Series Data with Data Video
future research directions. mental Protection Agency’s Storm
Water Management Model. Understanding the changes of
time series is a common task in
many application domains. Con-
Exploratory Metamorphic verting time-series data into vid-
Testing for Scientific Software The Font Wars, Part 1 eos helps an audience with little
or no background knowledge gain
Scientific model developers are The Font Wars were a decades- insights and deep impressions. It
able to verify and validate their long competition in the computer essentially integrates data visual-
software via metamorphic test- industry for dominance in font izations and animations to present
ing (MT), even when the expected technology, viewed as a key suc- the evolution of data expressively.
output of a given test case is not cess factor for personal computing However, it remains challenging to
readily available. The tenet is to platforms. At the heart of the Font create this kind of data video. First,
check whether certain relations Wars was a fundamental question: it is difficult to efficiently detect
hold among the expected outputs What is the best way to turn tradi- important changes and include
of multiple related inputs. Contem- tional printed letter forms into digi- them in the video sequence. Exist-
porary approaches require that the tal fonts for computer screens and ing methods require much man-
relations be defined before tests. printers? Answers to this question ual effort to explore the data and

4 July 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE
 find changes. Second, how these autonomous driving. It is based
changes are emphasized in the on a new system-on-a-chip (SoC)
videos is also worth studying. A Container NATs and Session- that integrates industry-standard
video without emphasis will hinder Oriented Standards: Friends components such as CPUs, ISP,
an audience from noticing those or Foe? and GPU, with custom neural net-
important changes. This article work accelerators. The FSD com-
from the March/April 2020 issue of This article from the November/ puter is capable of processing up
IEEE Computer Graphics and Appli- December 2019 issue of IEEE Inter- to 2,300 frames per second, which
cations presents an approach that net Computing highlights issues is a 21× improvement over Tesla’s
extracts and visualizes important that arise when deploying network previous hardware and at a lower
changes of a time series. address translation middle-boxes cost. When fully utilized, it enables
through containers. The authors a new level of safety and autonomy
focus on Docker as the container on the road. Read more in this arti-
technology of choice and pres- cle from the March/April 2020 issue
Research on Road Traffic ent a thorough analysis of its net- of IEEE Micro.
Situation Awareness System working model, with special atten-
Based on Image Big Data tion to the default bridge network
driver that is used to implement
Road traffic is an important com- network address translation func- Metric Learning-Based
ponent of the national economy tionality. They discuss some unex- Multimodal Audio-Visual
and social life. Promoting intelli- pected shortcomings and elabo- Emotion Recognition
gent and Informa ionization con- rate on the suitability of containers
struction in the field of road traffic for deploying services based on People express their emotions
is conducive to the construction the Interactive Connectivity Estab- through multiple channels, such
of smart cities and the formula- lishment standard protocol. To as visual and audio ones. Conse-
tion of macro-strategies and con- support their findings, they present quently, automatic emotion rec-
struction plans for urban traffic experiments that they conducted ognition can be significantly ben-
development. Aiming at the short- in a real-world operational environ- efited by multimodal learning.
comings of the current road traffic ment, namely a WebRTC service Even though each modality exhib-
system, this article from the Jan- based on the Janus media server. its unique characteristics, multi-
uary/February 2020 issue of IEEE modal learning takes advantage of
Intelligent Systems—on the basis the complementary information of
of combining convolution neu- diverse modalities when measur-
ral networks (CNNs), situational Compute Solution for Tesla’s ing the same instance, resulting in
awareness, databases, and other Full Self-Driving Computer enhanced understanding of emo-
technologies—takes the road traf- tions. Yet, their dependencies and
fic situational awareness system Tesla’s full self-driving (FSD) com- relations are not fully exploited in
as its research object and analyzes puter is the world’s first pur- audio–video emotion recognition.
the information collection, pro- pose-built computer for the Furthermore, learning an effective
cessing, and analysis process. highly demanding workloads of metric through multimodality is a

www.computer.org/computingedge 5
MAGAZINE ROUNDUP

crucial goal for many applications PedibusSmart, SafePath, and Kids- resourcing. In this article from
in machine learning. Therefore, GoGreen. It reports on four years the March/April 2020 issue of IEEE
in this article from the January– of success with more than 1,800 Software, the authors define a
March 2020 issue of IEEE Multi- elementary-age children, their systematical service design con-
Media, the authors propose multi- teachers, and their families. The cept to enable all stakeholders to
modal emotion recognition metric authors further show how (i) dis- achieve better outcomes in co-
learning (MERML), learned jointly appearing, pervasive technology creation activities.
to obtain a discriminative score contributes to successful adop-
and a robust representation in a tion; (ii) properly balancing trust
latent-space for both modalities. and tracking leads to useful, nonin-
The learned metric is efficiently vasive technological support; and Detecting Online Content
used through the radial basis func- (iii) in-classroom, gameful technol- Deception
tion (RBF)-based support vector ogy engages and motivates partici-
machine (SVM) kernel. The eval- pation, with behavior changes per- The surge of deceptive content
uation of the framework shows a sisting over time. (such as fake news) in the past few
significant performance, improv- years has made content decep-
ing the state-of-the-art results on tion an important area of research.
the eNTERFACE and CREMA-D The authors of this article from the
datasets. The Need for New March/April 2020 issue of IT Pro-
Antiphishing Measures fessional identify two main types
against Spear-Phishing of content deception based on
Attacks either fake content or misleading
content. They present a classifi-
CLIMB: A Pervasive Gameful In this article from the March/April cation of deception attacks along
Platform Promoting Child 2020 issue of IEEE Security & Pri- with delivery methods. They also
Independent Mobility vacy, the authors provide extensive discuss defense measures that can
analysis of the unique characteris- detect deception attacks. Finally,
Child independent mobility (CIM) tics of phishing and spear-phishing they highlight some outstanding
refers to the freedom and capa- attacks, argue that spear-phish- challenges in the area of content
bility of children to move about ing attacks cannot be well cap- deception.
their local neighborhoods with- tured by current countermeasures,
out constant direct adult supervi- identify ways forward, and analyze
sion. Our CLIMB project combats an advanced spear-phishing cam-
an observed decline in CIM, offer- paign targeting white-collar work-
ing a pervasive gameful platform ers in 32 countries. Join the IEEE
for home–school mobility com- Computer
posed of three primary compo- Society
nents: the first two using technol-
computer.org/join
ogy to support different levels of Three Phases of Transforming
child independence and the third a Project-Based IT Company
providing an element of continu- into a Lean and Design-Led
ous motivation for positive behav- Digital Service Provider
ior change. This article from the
January–March 2020 issue of IEEE Digital transformation requires
Pervasive Computing describes a continuous review of value
these three novel technologies: creation, value capture, and

6 ComputingEdge July 2020

 Editor’s Note

Advancing Science
with Software

S cientists increasingly use

software to facilitate discov-
eries through simulation and anal-
Also from Computing in Science
& Engineering, “SciPipe—Turning
Scientific Workflows into Com-
people better understand NASA’s
missions and discoveries.
Whether scientific or com-
ysis. Scientific software can enable puter Programs” discusses a tool mercial, software should be as
research that might be imprac- for automating complex scientific energy-efficient as possible. The
tical or impossible using experi- computations involving multiple authors of IEEE Software’s “A Man-
mentation, observation, or theory. programs. ifesto for Energy-Aware Software”
Despite its importance, challenges Scientific data can be chal- argue that software developers
remain in terms of testing, reli- lenging to not only analyze but must consider energy consump-
ability, and reusability. This Com- also convey to the public. This tion when designing programs.
putingEdge issue presents tools ComputingEdge issue includes IEEE Pervasive Computing’s “Next
for helping scientists effectively two examples of creative scien- Generation IoT: Toward Ubiqui-
utilize software in their research. tific data visualizations from IEEE tous Autonomous Cost-Efficient
“Metamorphic Testing: A Sim- Computer Graphics and Appli- IoT Devices” also emphasizes the
ple Yet Effective Approach for cations. In “Weather Report: A need for energy efficiency, specif-
Testing Scientific Software,” from Site-Specific Artwork Interweav- ically in IoT devices.
Computing in Science & Engi- ing Human Experiences and Sci- This issue of ComputingEdge
neering, aims to help scientists entific Data Physicalization,” the concludes with two Computer
who have limited training in soft- authors describe a public art articles on quantum supremacy.
ware development employ sci- installation that educates on cli- “Powerball and Quantum Suprem-
entific software successfully. mate change. In “OpenSpace: acy” teaches quantum concepts
The authors explain that meta- Bringing NASA Missions to the using a lottery metaphor. “Beyond
morphic testing is a good tech- Public,” the authors present a visu- Quantum Supremacy” identifies
nique for testing exploratory soft- alization program for use in inter- advances that are needed to move
ware with inherent uncertainties. active museum displays that helps quantum computing forward.

2469-7087/20 © 2020 IEEE Published by the IEEE Computer Society July 2020 7
 DEPARTMENT: SOFTWARE ENGINEERING
This article originally
appeared in

Metamorphic Testing: vol. 21, no. 1, 2019

A Simple Yet Effective Approach

for Testing Scientific Software
Upulee Kanewala, Montana State University
Tsong Yueh Chen, Swinburne University of Technology

Testing scientific software is a difficult task due to their inherent complexity and the
lack of test oracles. In addition, these software systems are usually developed by end-
user developers who are not normally trained as professional software developers nor
testers. These factors often lead to inadequate testing. Metamorphic testing (MT) is a
simple yet effective testing technique for testing such applications. Even though MT is a
wellknown technique in the software testing community, it is not very well utilized by the
scientific software developers. The objective of this paper is to present MT as an effective
technique for testing scientific software. To this end, we discuss why MT is an appropriate
testing technique for scientists and engineers who are not primarily trained as software
developers. Specifically, how it can be used to conduct systematic and effective testing
on programs that do not have test oracles without requiring additional testing tools.

WHAT MAKES TESTING SCIENTIFIC due to one-off errors,1 compromised performance in

SOFTWARE DIFFICULT? coordinate measuring machines (CMMs) due to soft-
Scientific software is widely used for making criti- ware faults,2 and geoscience software systems pro-
cal decisions in various scientific and engineering ducing seemingly correct yet different results that
domains. For example, simulations are often used are hard to categorize as incorrect. 3 Previous works
in place of physical experiments due to the time also report situations where software faults cause
and cost constraints associated with conducting retractions of published work.4 Testing is the most
physical experiments. Furthermore, decisions made widely used approach for quality assurance of soft-
by these software systems can affect day-to-day ware. But some inherent characteristics in scientific
human life, such as predictions made by climate software make it difficult to conduct systematic test-
models. Thus, it is important to make sure that these ing in these programs. 5
software systems are producing the correct results.
Previous studies have reported many instances ›› Correct answers are often unknown. Typically,
where scientific software systems were affected scientific software is exploratory in nature
by faults such as seismic programs losing precision and due to this, the correct results are often
unknown. If the result is known there would be
no need to develop the software. In such situa-
DOI No. 10.1109/MCSE.2018.2875368 tions, only bounds or ranges of solutions might
Date of publication 18 October 2018; date of current version be available. Typically, in testing, an expected
6 March 2019. output is used to decide test case passing

8 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 SOFTWARE ENGINEERING

FIGURE 1. MT process. ts: source test case, tf : follow-up test case, os: output of the source test case and of: of : output of the
follow-up test case.

or failing. This would make it challenging to input list has two million real numbers. How can we
conduct systematic testing in these programs. know the returned average is correct or not? Though
›› Practically difficult to validate the computed we are not able to validate the computed average in
output. Scientific software often implements this case, we do know some relationships between the
mathematical models that involve complex outputs of some related inputs. For example, consider
calculations. Furthermore, they tend to produce a new list of real numbers, which is a permuted list of
complex outputs. Both these characteristics the original list of real numbers, or which consists of
make it hard to determine the correctness four million real numbers by duplicating the original list
of the produced output of the software. This of real numbers. For either of the new lists of real num-
makes it challenging to use automated test bers, the new average is expected to be the same as
case generation approaches such as random the old average (subject to some round-off error toler-
test generation since the output of such test ance). If the new and old averages are not the same,
cases are difficult to validate. then we know that the program of computing average
›› Inherent uncertainties. Often scientific software has bugs. This is the intuition of MT.
is written to simulate models with inherent In software testing, passing, or failing of a test case
uncertainties. For some of these scientific is decided using a test oracle and it is an essential com-
programs, there may be more than one possible ponent for conducting systematic testing. MT uses
output. This makes it challenging to conduct metamorphic relations (MRs) to determine whether
testing on these programs. a test case has passed or failed. An MR specifies how
›› Choosing suitable tolerances. Scientific the output of the program is expected to change when
software systems often involve complex a specified change is made to the input. The following
floating-point computations. Thus, specifying is the typical process for applying MT to a given pro-
the acceptable tolerance for the expected gram and Figure 1 depicts this process:6
output in test cases is difficult.
›› Incompatible testing tools. Programming 1. MR identification: identifying MRs for the
languages such as FORTRAN are widely used program under test can be done based on the
in the scientific community for developing specification.
scientific software. However, testing tools 2. Source test case creation and execution: com-
are usually developed for languages such as monly used test generation techniques such
JAVA and C++ that are commonly used by as random, structural coverage, or fault-based
the software engineering community. Thus, test input generation can be used. Then the
these testing tools are not effective for testing generated source test cases are executed on
scientific programs. the program under test.
3. Follow-up test case creation: use the MRs
WHAT IS METAMORPHIC identified in Step 1 to transform the source test
TESTING (MT)? case to obtain the follow-up test case.
Consider a program that will accept a list of real num- 4. Follow-up test case execution: Execute the
bers and compute their average. Suppose that the follow-up test case and compare the outputs

www.computer.org/computingedge 9
SOFTWARE ENGINEERING

FIGURE 2. A JUnit test script that uses MT approach to test a matrix multiplication function in the JScience library.

1 2 2 1
of the source and follow-up test cases to verify A = and B = .
3 4 3 4
whether the corresponding MRs are satisfied.
Violation of an MR indicates that the program This source test case is executed on the matrix
under test is faulty. multiplication function under test first. Next, based on
the input relationship specified by the above MR, two
Thus, MT checks whether relationships between follow-up test cases are created, namely, the test case
inputs and outputs of multiple executions were pre- consisting of A and B1 and the test case consisting
served during the program execution and can be used of A and B2. Here, B1 is randomly generated and B2 is
without knowing the correctness of the output for defined as B − B1. Suppose that
individual executions.
Consider a program P that multiplies two matrices
1 6
A and B. Assume that the result of multiplying A with B B1 = .
3 5
is C. Matrix multiplication has the following property:
A × B = A × B1 + A × B2 where B = B1 + B2. We can use this Then,
property as an MR to conduct MT on P. For example,
1 5
Figure 2 shows a test script written using JUnit to con- B2 = B B1 = .
0 1
duct automated testing on the matrix multiplication
function in the JScience Matrix class (http://jscience As shown in Figure 3, these two follow-up test
.org/api/org/jscience/mathematics/vector/Matrix cases are also executed on the matrix multiplication
.html). function under test. Finally, the outputs of the source
The source test case (which consists of A and B) test and the follow-up test cases are validated against
can be provided by the user or can be generated ran- the above MR. As shown with this test script, this MR
domly. For example, assume that the user provided the based testing approach allows to generate follow-up
following simple two matrices: test cases automatically and verify relationships

10 ComputingEdge July 2020

 SOFTWARE ENGINEERING

FIGURE 3. MT of a matrix multiplication program.

between multiple outputs without any manual inter- them to test their programs using their domain
vention. Readers who are interested to know more knowledge. For example, consider a program
about MT may consult the article by Chen et al.7 that computes the sine value of a given angle x.
We can derive the following two MRs for testing
WHY USE MT FOR TESTING the sine function based on its properties: MR1:
SCIENTIFIC SOFTWARE? sin(x’) = sin(x) where x’ = x + 360°. MR2: sin(x’)
›› Scientific software is often written by scientists = −sin(x) where x’ = −x. Due to the constraints
who have the domain knowledge required to on the testing budget, suppose that we can
develop them. However, scientists might lack the conduct testing with only one MR. In such a
knowledge to apply different forms of conven- situation, an electrical engineer will most likely
tional testing methods. But, as evident from the choose MR1 to test her program due to the
example in Figure 2, MT is simple in concept, periodicity of current. But, on the other hand, a
and hence could be easily learned and applied land surveyor may choose MR2 since she usually
without any prior knowledge of software testing works with positive and negative angles to
or without any software testing experience.8 represent clockwise and anticlockwise mea-
›› As shown in the example given in Figure 2, MT is surements of angles.
easy to implement: test scripts could be easily ›› MT provides an effective way to conduct unit
prepared by the scientific software developers to testing for scientific software. One of the
automate the testing process or to incorporate reasons for the lack of unit testing in scientific
MT into existing testing infrastructures such as software is the difficulty in validating the
JUnit. Thus, MT does not require the developers expected output of the unit for a randomly
to buy or maintain additional expensive testing generated test input. In such situations, MT can
tools. be used to conduct automated unit testing by
›› As we discussed in Section “WHAT IS META- means of MRs.
MORPHIC TESTING (MT)?”, many scientific ›› Scientists often conduct testing using a limited
and engineering applications face the test number of test cases with known outputs that
oracle problem. This makes it challenging to they obtain from experiments or analytical
conduct automated systematic testing on these solutions. MT can be used to effectively extend
programs. MT supports automated systematic these limited number of test cases by deriving
testing on such programs. MRs and creating follow-up test cases according
›› MT uses MRs to determine whether test cases to the MRs. These follow-up test cases are most
pass or fail. Often scientific software is devel- likely to execute parts of the program that might
oped by domain experts who know the properties not have been executed with the original set of
of these algorithms the best. Thus, it would be test cases. Thus, MT provides a way to extend
easy for them to derive MRs for testing these existing test cases.
programs. ›› Many scientific software involves elements of
›› Scientific software developers will be able to randomness which makes testing difficult. MT
identify the most effective MRs and prioritize can still be applicable in such situations.

www.computer.org/computingedge 11
SOFTWARE ENGINEERING

SOME EXAMPLES experts the outputs generated by versions of the pro-

gram injected with faults, they were unable to identify
Testing Epidemiological Model that the outputs were produced by a faulty version of
Implementations Using MT the program.
Pullum et al. used MT to verify and validate an epide- Here we report the results of conducting auto-
miological model implementation.9 Such implemen- mated unit testing on the following functions that per-
tations are used to model how diseases are spread form several main calculations in the SAXS program:10
in populations. Thus, it is important to verify and val-
idate these models since they will be used to make ›› calculateDistance (f1): computes the distance
critical decisions during a disease spread. Epidemio- between atoms;
logical model implementations face the oracle prob- ›› findGyrationRadius (f2): computes the gyration
lem because these programs are written to find the radius of groups of atoms;
answer in the first place. Therefore, developing an ora- ›› scatterSample (f3): main function responsible
cle for testing these programs is practically difficult. for scattering.
One of the approaches used to test these models is to
compare the output of the model with data obtained We used the machine learning based MR predic-
from real phenomena. Obviously, such data is limited. tion approach proposed by Kanewala et al.12 to predict
Other approaches used to test this type of programs the likely MRs for these functions. The test inputs were
include comparing the output with results obtained generated randomly. There were no violations of these
from mathematical models and comparing the results predicted MRs when applied to these three functions.
with other simulation models. These techniques are To evaluate the effectiveness of MT for conduct-
not sufficient for conducting systematic and compre- ing unit testing, we created faulty versions, known as
hensive testing on these programs. mutants, of these functions using the μJava (https:
The authors tested an ordinary differential equa- //cs.gmu.edu/_offutt/mujava/) mutation engine. This
tion based epidemiological model and an agent-based mutation engine creates mutants of the program by
epidemiological model using MT. They used the data making a syntactic change in the source code. With
from the 1918 Influenza outbreak to calibrate the MT, we say that a mutant is killed if an MR is violated
models. The authors defined 11 MRs based on mak- when the corresponding source and follow-up test
ing changes to various model parameters and the cases are executed on that mutant. Therefore, the
expected effects that those changes would have on fault detection effectiveness of MT can be measured
the model output. These MRs were defined using by the number of mutants killed during the MT pro-
the authors' domain knowledge about these models. cess. Obviously, the higher the percentage of mutants
Through MT, authors identified an error in the output killed, the more effective MT is in revealing bugs of a
method of the agent-based epidemiological model. program. We use this process to evaluate the fault
detection effectiveness of MT in the functions men-
Using MT to Conduct Automated tioned above.
Unit Testing on a Small Angle X-Ray Table 1 shows the percentage of mutants killed
Scattering (SAXS) Program through MT for individual functions. Overall, 90% of
We used MT to conduct automated unit testing on an the mutants could be killed using MT. The important
open source program written to analyze small angle thing to note here is that the entire unit testing pro-
x-ray scattering data called SAXS.10, 11 This program cess was fully automated starting with MR identifica-
reconstructs macromolecular structures using scat- tion, source test case generation, test execution and
tering patterns obtained from experiments. This pro- further, did not require the domain experts to evalu-
gram was initially tested by running the program on ate the correctness of the test outputs. Though no
a selected set of inputs where the correctness of violations of MRs were detected for SAXS, MT helps
the produced outputs was determined by domain to establish our confidence on the quality of the
experts. However, when we showed the domain SAXS program.

12 ComputingEdge July 2020

 SOFTWARE ENGINEERING

TABLE 1. Mutants detected by predicted MRs. f 1:

Testing a Monte Carlo Simulation calculateDistance, f2: findGyrationRadius, and f3:
Program With MT scatterSample.
Ding and Hu13 used MT for testing a Monte Carlo mod-
f1 f2 f3 Total
eling program that simulates photon propagations in
No. of faulty
biological tissues for the purpose of accurate genera- 19 54 139 212
versions used
tion of reflectance images. The biggest challenge for
testing this program is the lack of test oracles. One No. of faulty versions
19 45 127 191
detected by MT
solution is to compare the results of the Monte Carlo
simulation program to experimental results. But, as % of detected
100 83 91 90
faulty versions
with many scientific software, building the necessary
infrastructure to conduct the relevant physical exper-
iments is time consuming and expensive. For exam-
ple, in this specific case, conducting a physical exper- and Monte Carlo simulations. We are strongly con-
iment would require a laser beam that would produce fident that MT is one of the most appropriate and
a specific number of photons, an environment without cost-effective testing techniques for scientists and
interruptions from other light sources and good reac- engineers.
tive imaging cameras. Thus, the authors used MT to
conduct testing on this program. REFERENCES
The authors identified five MRs for the program 1. L. Hatton, “The T experiments: Errors in scientific
based on domain knowledge and experimental results. software,” IEEE Comput. Sci. Eng., vol. 4, no. 2, pp. 27–38,
They generated tests that cover all the branches and Apr.–Jun. 1997.
functions in the program. Through the violation of one 2. A. J. Abackerli, P. H. Pereira, and N. Calônego, Jr., “A
of the MRs used for testing, the authors discovered case study on testing CMM uncertainty simulation
faults in the program and corrected it. software (VCMM),” J. Brazilian Soc. Mech. Sci. Eng., vol. 32,
They further evaluated the effectiveness of MT pp. 8–14, Mar. 2010.
using mutants. The authors created 150 mutants for 3. L. Hatton and A. Roberts, “How accurate is scientific
the Monte Carlo simulation program and they were software?,” IEEE Trans. Softw. Eng., vol. 20, no. 10,
able to detect 90% (135) of these mutants using MT. pp. 785–797, Oct. 1994.
4. G. Miller, “A scientist’s nightmare: Software problem
SUMMARY leads to five retractions,” Science, vol. 314, no. 5807,
Some characteristics in scientific software, such as pp. 1856–1857, 2006. [Online]. Available: http://www
not knowing the correct answers and inherent uncer- . sciencemag.org/content/314/5807/1856.short
tainties in calculations, make testing them difficult. 5. U. Kanewala and J. M. Bieman, “Testing scientific
MT can be an effective testing technique to test software: A systematic literature review,” Inf. Softw.
these programs. Instead of checking the correctness Technol., vol. 56, no. 10, pp. 1219–1232, 2014.
of individual test outputs, MT checks whether the 6. S. Segura, G. Fraser, A. B. Sánchez, and A. R. Cortés,
changes in the test outputs are according to what is “A survey on metamorphic testing,” IEEE Trans. Softw.
expected by the program with respect to the changes Eng., vol. 42, no. 9, pp. 805–824, 2016. [Online]. Avail-
in the inputs. These relationships between inputs and able: https://doi.org/10.1109/TSE.2016.2532875
the expected changes in the outputs are referred to 7. T. Y. Chen et al., “Metamorphic testing: A review of
as MRs. Scientists, who typically develop this scien- challenges and opportunities,” ACM Comput. Surveys,
tific software, would be in a great position to identify 2017, to be published.
effective MRs because of their domain knowledge 8. T. Y. Chen, F.-C. Kuo, and Z. Q. Zhou, “An effective
and, thus would be able to effectively test their soft- testing method for end-user programmers,” in Proc.
ware using MT. MT has been successfully applied for First Workshop End-User Softw. Eng. (WEUSE 2005),
testing various scientific software including epide- 2005, pp. 21–25. [Online]. Available: http://doi.acm
miological model implementations, SAXS programs, .org/10.1145/1082983.1083236

www.computer.org/computingedge 13
SOFTWARE ENGINEERING

9. L. L. Pullum and O. Ozmen, “ Early results from meta- U.K., the DIC degree from the Imperial College, London, U.K.,
morphic testing of epidemiological models,” in Proc. and the Ph.D. degree from The University of Melbourne, Mel-
ASE/IEEE Int. Conf. BioMed. Comput., Dec. 2012 , bourne, VIC, Australia. Prior to joining Swinburne, he taught
pp. 62– 67. at The University of Hong Kong and The University of Mel-
10. U. Kanewala , A. Lundgren, and J. M. Bieman, “Auto- bourne. He is the Inventor of metamorphic testing and adap-
mated metamorphic testing of scientific software,” tive random testing. His contact address is: Department of
J. C. Carver, N. P. Chue Hong, and G. K. Thiruvathukal, Computer Science and Software Engineering, Swinburne
Eds. Soft. Eng. Sci., Taylor & Francis, 2016 , doi: University of Technology, VIC 3122, Australia. Contact him at
https://doi.org/10.1201/9781315368924. [email protected].
11. [Online]. Available: http://cgi.cs.arizona.edu
/~mstrout/Projects/SAXS/software.php, 2011.
Accessed on: Sep. 11, 2017.
12. U. Kanewala , J. M. Bieman, and A. Ben-Hur, “ Predicting
metamorphic relations for testing scientific software:
A machine learning approach using graph kernels,”
Softw. Testing, Verification Rel., vol. 26, no. 3, pp. 245 –269,
2016, stvr.1594. [Online]. Available at: http://dx.doi.org
/10.1002/stvr.1594
13. J. Ding and X. Hu, “Application of metamorphic
testing monitored by test adequacy in a Monte Carlo
simulation program,” Softw. Qual. J., vol. 25, no. 3,
pp. 841 – 869, 2017. [Online]. Available at: https://doi
.org/10.1007/s11219-016-9337-3

UPULEE KANEWALA is an Assistant Professor with Mon-

Cutting Edge
tana State University, Bozeman, MT, USA. Her research
interests include software testing, metamorphic testing,
stay
and quality assurance of scientific software. She received
the Ph.D. degree in computer science from Colorado State
on the
University, Fort Collins, CO, USA, in 2015, the Master of Sci-
ence degree in computer engineering from Purdue Univer-
sity, West Lafayette, IN, USA, in 2010, and the Bachelor of
Science degree in computer engineering from University of
Peradeniya, Peradeniya, Sri Lanka, in 2007. She has authored
J A N UA RY/ F E B R UA RY 2 016

IEEE Intelligent Systems provides peer-

IEEE

Also in this issue:

AI’s 10 to Watch 56
Real-Time Taxi Dispatching 68
IEEE
JANUARY/FEBRUARY 2016

From Flu Trends to Cybersecurity 84

or coauthored multiple peer-reviewed articles and book P U T T I N G A I I N T O P R A C T I C E

reviewed, cutting-edge articles on the

chapters on metamorphic testing including the first paper on
theory and applications of systems
automatic detection of Metamorphic Relations. Her address
ONLINE BEHAVIORAL ANALYSIS

that perceive, reason, learn, and

is Gianforte School of Computing, 357 Barnard Hall, Montana
State University, Bozeman, MT 59717. Contact her at upulee act intelligently.
VOLUME 31
NUMBER 1

www.computer.org/intelligent

[email protected].
IS-31-01-C1 Cover-1 January 11, 2016 6:06 PM

TSONG YUEH CHEN is a Professor with Swinburne University

of Technology, Melbourne, VIC, Australia. His main research
interest focuses on software testing. He received the B.Sc.
The #1 AI Magazine
www.computer.org/intelligent
IEEE

and M.Phil. degrees from The University of Hong Kong, Hong

Kong, the M.Sc. degree from University of London, London,

14 ComputingEdge July 2020

PURPOSE: The IEEE Computer Society is the world’s largest EXECUTIVE COMMITTEE
association of computing professionals and is the leading provider President: Leila De Floriani
of technical information in the field.
President-Elect: Forrest Shull
MEMBERSHIP: Members receive the monthly magazine Past President: Cecilia Metra
Computer, discounts, and opportunities to serve (all activities First VP: Riccardo Mariani; Second VP: Sy‐Yen Kuo
are led by volunteer members). Membership is open to all IEEE Secretary: Dimitrios Serpanos; Treasurer: David Lomet
members, affiliate society members, and others interested in the VP, Membership & Geographic Activities: Yervant Zorian
computer field. VP, Professional & Educational Activities: Sy-Yen Kuo
VP, Publications: Fabrizio Lombardi
COMPUTER SOCIETY WEBSITE: www.computer.org
VP, Standards Activities: Riccardo Mariani
OMBUDSMAN: Direct unresolved complaints to VP, Technical & Conference Activities: William D. Gropp
[email protected].
2019–2020 IEEE Division VIII Director: Elizabeth L. Burd
CHAPTERS: Regular and student chapters worldwide provide the 2020-2021 IEEE Division V Director: Thomas M. Conte
opportunity to interact with colleagues, hear technical experts, 2020 IEEE Division VIII Director-Elect: Christina M. Schober
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report BOARD OF GOVERNORS
an address change, or obtain more information on any of the Term Expiring 2020: Andy T. Chen, John D. Johnson,
following, email Customer Service at [email protected] or call
Sy-Yen Kuo, David Lomet, Dimitrios Serpanos,
+1 714 821 8380 (international) or our toll-free number, Hayato Yamana
+1 800 272 6657 (US): Term Expiring 2021: M. Brian Blake, Fred Douglis,
• Membership applications Carlos E. Jimenez-Gomez, Ramalatha Marimuthu,
• Publications catalog Erik Jan Marinissen, Kunio Uchiyama
• Draft standards and order forms Term Expiring 2022: Nils Aschenbruck,
• Technical committee list Ernesto Cuadros‐Vargas, David S. Ebert, William Gropp,
• Technical committee application Grace Lewis, Stefano Zanero
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory EXECUTIVE STAFF
• IEEE senior member grade application (requires 10 years Executive Director: Melissa A. Russell
practice and significant performance in five of those 10) Director, Governance & Associate Executive Director:
Anne Marie Kelly
PUBLICATIONS AND ACTIVITIES Director, Finance & Accounting: Sunny Hwang
Director, Information Technology & Services: Sumit Kacker
Computer: The flagship publication of the IEEE Computer Society,
Director, Marketing & Sales: Michelle Tubb
Computer publishes peer-reviewed technical content that covers
Director, Membership Development: Eric Berkowitz
all aspects of computer science, computer engineering,
technology, and applications.
COMPUTER SOCIETY OFFICES
Periodicals: The society publishes 12 magazines and 18 journals. Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
Refer to membership application or request information as noted 20036-4928; Phone: +1 202 371 0101; Fax: +1 202 728 9614;
above. Email: [email protected]
Conference Proceedings & Books: Conference Publishing Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720;
Services publishes more than 275 titles every year. Phone: +1 714 821 8380; Email: [email protected]
Standards Working Groups: More than 150 groups produce IEEE
MEMBERSHIP & PUBLICATION ORDERS
standards used throughout the world.
Phone: +1 800 678 4333; Fax: +1 714 821 4641;
Technical Committees: TCs provide professional interaction in Email: [email protected]
more than 30 technical areas and directly influence computer
engineering conferences and publications. IEEE BOARD OF DIRECTORS
Conferences/Education: The society holds about 200 conferences President: Toshio Fukuda
each year and sponsors many educational activities, including President-Elect: Susan K. “Kathy” Land
computing science accreditation. Past President: José M.F. Moura
Certifications: The society offers three software developer Secretary: Kathleen A. Kramer
credentials. For more information, visit Treasurer: Joseph V. Lillie
www.computer.org/certification. Director & President, IEEE-USA: Jim Conrad
Director & President, Standards Association: Robert S. Fish
BOARD OF GOVERNORS MEETING Director & VP, Educational Activities: Stephen Phillips
Director & VP, Membership & Geographic Activities:
24 – 25 September 2020 in McLean, Virginia, USA Kukjin Chun
Director & VP, Publication Services & Products: Tapan Sarkar
Director & VP, Technical Activities: Kazuhiro Kosuge

revised 1 May 2020

 DEPARTMENT: SCIENTIFIC PROGRAMMING
This article originally
appeared in

SciPipe—Turning Scientific vol. 21, no. 3, 2019

Workflows into Computer

Programs
Samuel Lampa, Martin Dahlö, Jonathan Alvarsson, and Ola Spjuth, Uppsala University

INTRODUCTION might not be obvious before trying to apply them to

Scientific Workflows are becoming increasingly popu- complex tasks.
lar as a way to automate complex scientific computa- At the Department of Pharmaceutical Biosciences,
tions consisting of multiple programs. we have spent the last few years using workflow tools
One of the main motivations behind this develop- to automate machine learning pipelines for predictive
ment is increased robustness and reproducibility of toxicology among other things. We have reviewed the
computational analyses. Chaining together multiple top dozen workflow tools popular in our field of bioin-
programs using plain scripts, as is often the first step formatics. We even tried out Luigi, created by music
in automating a pipeline, can easily become fragile company Spotify, which is popular in industry and far
and error prone due to the manual management of file from a bioinformatics-aimed tool.
paths and program invocations. Also, plain scripts are A recurring theme has been how often tools
not optimal if for some reason you have to cancel a run contain various limits that make them hard to use
and try to restart it from any partially finished steps. It for complex use cases. Machine learning workflows
can be hard to know which output files are properly fin- in particular often lead to highly complex workflows
ished and which are truncated from the cancelled run. because of their common inclusion of cross validation
Last but not least, plain scripts do not by default save and parameter sweeps from hyperparameter optimi-
an execution trace of what was run, such that the full zation. Not only are these workflows complex, but they
procedure used to create a specific output file can be also show some characteristics not always common
clearly presented. These are all aspects that scientific in other domains: The need for dynamic scheduling.
workflows are designed to help with. That is, they need to be able to parametrize and start
Despite many hundreds of scientific workflow tasks based on information obtained during the work-
tools published over the years, there can still be sig- flow run. Somewhat surprisingly, this is a problem in a
nificant challenges when trying to use many of them. majority of workflow tools because of how common
One reason for this is that many workflow tools it is that they have a strict separation between the
have been designed with a very narrow use case in scheduling and execution phases of the workflow run.
mind, often building in assumptions unique to the That is, after a workflow has progressed into its execu-
specific problem domain aimed at, which might make tion phase, it is commonly not possible to schedule and
them less applicable for scientific pipeline needs in start new tasks with parameter values obtained in the
general. current run. At least not without initiating completely
Even among the large numbers of general workflow new workflow runs.
tools, surprisingly many contain various constraints Anyway, after a lot of evaluation, it seemed at the
and assumptions limiting their generality. Often, this time that Luigi1 was the most promising way forward
for us. Later, we learned that Luigi's functional pro-
gramming inspired API design was not quite fit for our
DOI No. 10.1109/MCSE.2019.2907814 needs of dynamic workflow rewiring, which is one of
Date of current version 26 April 2019. the reasons why we later chose to develop the SciPipe

16 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 SCIENTIFIC PROGRAMMING

library (more on that later), but there is an interesting SciPipe7 is designed from the start as a pro-
story to tell about our Luigi phase too. gramming library embedded in the implementation
language (Google's Go, or “Golang”), rather than
Workflows as Computer Programs inventing new textual syntax or graphical tools. It
It turned out that because Luigi was implemented thus leverages the full power and flexibility of the Go
as a programming library, it was flexible enough that programming language for implementing workflow
we could build an alternative API on top of it, which logic. So far, we have not encountered a workflow use
resulted in the SciLuigi helper library. 2 Specifically, case that we have not been able to model with this
SciLuigi enabled us to keep the dependency network approach. Even complex machine learning workflows
definition separate from task definitions—a core prin- with nested branching has been solvable, as exem-
ciple in flow-based programming, as we will explain plified in a recent paper by the authors. 8 Another
shortly—which makes it much easier to reconnect nice side effect of the Go language in particular is
workflows without changing internals of workflow that it compiles to self-contained executable files,
components in complex ways. which makes deployment of most Go programs very
This positive experience from a programming straightforward. SciPipe is open source software
API-based workflow tool helped push a realization that (MIT licensed). A simple “Hello World” style workflow
has grown over a number of years of discussions and example is shown in Figure 2. For more information,
experimenting: Workflows are in the end just a glori- source code and documentation, see the work of
fied version of computer programs. Lampa et al. 6 , 9
It turns out that although many use cases can
be modeled as simple linear sequences of program Flow-Based Programming
invocations depending on each other, not all cases Focuses on Data Flow
are that simple. There are many examples where the Now, there are some differences between most work-
need for logic to control the workflow structure is flow programs and most normal programs. The main
so complex that any attempt at modeling it with a difference can be seen in the strong focus in work-
declarative workflow language ends up implement- flow programs on data flow. When defining how mul-
ing what is already available in existing programming tiple programs depend on each other we are in effect
languages. defining how the data will flow through these pro-
This tendency can also be seen in popular work- grams. There is actually a looming risk for workflow
flow engines building on domain specific languages tool developers to miss this detail and model the work-
(DSLs). Tools that become popular often either have a flow dependency graph as just dependencies between
really flexible DSL from the start, e.g., because the DSL programs and not their inputs and outputs. This can
is implemented in an existing scripting language (e.g., quickly lead to problems because one program typ-
Nextflow, 3 building on the Groovy language), or their ically do not depend on just one other program but
DSLs are step by step becoming increasingly complex, rather specific outputs of possibly multiple upstream
and in the end approaching computer programming programs. That is, data needs to be a first class citizen
languages in their capability (e.g., Cuneiform,4 imple- when defining workflows, or we risk missing important
menting a powerful functional language 5). details that will otherwise be buried in less thought-out
ad hoc code.
SciPipe One paradigm that takes note of this fact is
The above lesson is something we were taking into flow-based programming (FBP).10 Invented at IBM in
account when we set out to design the SciPipe work- the late 1960s and used on large mainframe computers
flow tool from scratch after finding out about sev- at banks and other large institutions, the flow-based
eral limitations with the Luigi/SciLuigi setup we were programming paradigm has seen some resurge in pop-
already using (primarily the need for dynamic schedul- ularity in recent years, possibly driven by the recent
ing and lack of compile time warnings about errors in trends toward multicore CPUs, distributed computing,
workflow connectivity). and message-oriented architectures.

www.computer.org/computingedge 17
SCIENTIFIC PROGRAMMING

FIGURE 1. Flow-based programs can be likened to a factory with processing stations connected with conveyor belts, upon which
data items “flow” through the network of stations and conveyor belts.

Flow-based programming ordains a number of implementations. It thus allows us to create libraries

design principles. The most important one in the con- of reusable components which can be plugged in at
text of dependency definition though is that it models any place in the program network, as long as its in- and
dependencies between processes in an appropriate out-ports are compatible with the in- and out-ports
level of detail; in terms of data inputs and outputs. The they connect to.
data itself are modeled through so-called “information Note that while FBP is often associated with visual
packets” and inputs and outputs as “ports”—a kind programming, that is far from a requirement. In our
of pluggable component between which yet another experience, skipping the visual part and focusing on
concept can be connected; “channels.” Channels a simple programming API has more than fulfilled
have bounded buffers and act as a kind of conveyor our needs, while letting us avoid depending on the
belt between processes, letting processes work on complexity of a visual programming framework. The
information packets from its in-ports asynchronously fact that the Go language provides the most impor-
and sending them on their out-ports (mostly) indepen- tant pieces for this to work (independently running
dently from the processing rate of other processes. go-routines and channels with bounded buffers) cer-
Figure 1 tries to depict this in an artistic way. tainly helps.
The core idea of flow-based programming is how
it draws all of these things together into a declara- Reproducibility in Workflow Programs
tive data flow definition separate from the process We have presented some rationale for writing work-
implementations. This allows easy rewiring of the flows as computer programs, but what about the
data flow without changing a single line of process other aspects important for workflows, such as

18 ComputingEdge July 2020

 SCIENTIFIC PROGRAMMING

FIGURE 2. Simple example workflow implemented with SciPipe. The workflow consists of two processes: “hello” and “world,”
where “hello” writes the word “Hello” to a file, which the “world” process uses as input and appends the word “World,” before
writing the combined output to a new file. Processes are defined from shell commands, where placeholders on the form of
{i:inport|file-extension} and {o:outport|file-extension} are used to define in-ports and out-ports. These placeholders act as
templates that will be replaced with specific file names during the workflow execution, based on the files that are sent on the
processes’ in-ports. Although this is a very simple example, Go programmers will recognize that this workflow definition is also a
simple Go program. More detailed examples are provided on the main SciPipe website. 6

reproducibility? Are not there areas where the tra- nodes in the graph as they pass through them. Since
ditional ways of designing workflows still have its one data item might be the merger of two initially
merits? separate dataflow paths, this is done in a hierarchical
In terms of reproducibility, we argue that maximiz- way. This way of tracing the execution also means that
ing the declarative nature of the workflow definition is the provenance will be captured at the granularity that
not necessarily the key requirement, although that has matters; at the data level. In concrete terms, it means
been a prioritized focus area in many workflow frame- SciPipe can—and does—save a provenance record for
works. Rather, we think what is most important is to be every output file of the workflow.
able to produce an unambiguous record of how each Saving a provenance record only for the full work-
data output from a workflow was produced. flow run risks introducing a distance between the data
In SciPipe we think we have found a simple yet and the workflow run that over time will only grow big-
powerful way to capture provenance, leveraging its ger. By instead saving the provenance information in
low complexity and flow-based programming-based an accompanying file associated with every output file
design. of the workflow, as is done in SciPipe, means it is far
The fact that programs are defined, even program- easier to ensure this information stays associated.
matically, as a graph of processes with dataflow con-
nections on which data items “flow,” made it straight- CONCLUSION
forward to implement a provenance mechanism By providing increased robustness to failure and
around the information packets themselves, letting reproducibility of results, workflows are an estab-
them aggregate information about the processing lished workhorse of computer-aided science as well

www.computer.org/computingedge 19
SCIENTIFIC PROGRAMMING

as an increasingly important part also in data pipe- Forthcoming, GigaScience, 2019.

lines in industry. 8. S. Lampa, J. Alvarsson, S. Arvidsson Mc Shane, A. Berg,
Dominant approaches to workflow authoring, such E. Ahlberg, and O. Spjuth, “Predicting off-target binding
as specialized domain specific languages or graphical profiles with confidence using conformal prediction,”
workflow authoring tools, do not always cater to the Front. Pharmacol., vol. 9, 2018, Art. no. 1256.
full spectrum of workflow needs, such as machine 9. S. Lampa, M. Czygan, and J. Alvarsson, SciPipe source
learning workflows consisting of parameter sweeps code repository at GitHub. Accessed: Mar. 23, 2019.
nested with cross validation and dynamic scheduling. [Online]. Available: https://github.com/scipipe/scipipe
For such complex and dynamic needs, we argue for 10. J. P. Morrison, Flow-Based Programming: A New
the benefits of workflows implemented as computer Approach to Application Development, 2nd ed.
programs, using workflow tools implemented as pro- Charleston: Self-published via CreateSpace, May 2010.
gramming libraries. Furthermore, by using flow-based
programming principles, a declarative while program-
matic API can be presented, while still hiding a lot of SAMUEL LAMPA received the M.Sc. degree in molecular
low-level details under the hood. biotechnology engineering and the Ph.D. degree in pharma-
Being able to provide workflows in the form of com- ceutical bioinformatics from the Department of Pharmaceu-
piled executable files has benefits both for the ease of tical Biosciences, Uppsala University, Uppsala, Sweden, in
deployment and future reproducibility. Workflow tool September 2018. His research interests span intelligent data
implementations are at risk for bit rot or incompatible processing systems, reproducibility, artificial intelligence,
changes well before the mainstream programming and biomimetics in information technology, among other
languages—or processor architectures—are. things. He blogs at bionics.it. Contact him at samuel.lampa
For more information on SciPipe, check out the @farmbio.uu.se.
main website at scipipe.org.
MARTIN DAHLÖ is currently working toward the Ph.D.
REFERENCES degree in bioinformatics at the Department of Pharmaceuti-
1. E. Bernhardsson, E. Freider, and A. Rouhani. spotify/ cal Biosciences. He received the Master of Science degree in
luigi - GitHub. Accessed: March 23, 2019. [Online]. bioinformatics engineering from Uppsala University, Uppsala,
Available: https://github.com/spotify/luigi Sweden, in 2010, and has worked at Uppsala University
2. S. Lampa, J. Alvarsson, and O. Spjuth, “Towards agile since then. More and more of the work has involved working
large-scale predictive modelling in drug discovery closely with the UPPMAX HPC Center with a focus on user
with flow-based programming design principles,” J. efficiency. Contact him at [email protected].
Cheminform., vol. 8, no. 1, p. 67, 2016.
3. P. Di Tommaso, M. Chatzou, E. W. Floden, P. P. Barja, E. JONATHAN ALVARSSON is a researcher in the field of
Palumbo, and C. Notredame, “Nextflow enables repro- Pharmaceutical Bioinformatics, Department of Pharmaceu-
ducible computational workflows,” Nature Biotech, tical Biosciences, Uppsala University, where he also received
vol. 35, no. 4, pp. 316–319, 2017. the PhD degree in 2015. His main research interest has been
4. J. Brandt, M. Bux, and U. Leser, “Cuneiform: A func- in machine learning and molecular fingerprints as well as the
tional language for large scale scientific data analysis,” development of scientific software.
in Proc. EDBT/ICDT Workshops, 2015, pp. 7–16.
5. J. Brandt, W. Reisig, and U. Leser, “Computation seman- OLA SPJUTH is an associate professor at the Department
tics of the functional scientific workflow language of Pharmaceutical Biosciences, Uppsala University. His
cuneiform,” J. Functional Program., vol. 27, p. e22, 2017. main research interests include intelligent systems for drug
6. S. Lampa, SciPipe website. Accessed: Mar. 23, 2019. discovery and chemical safety, including how automated
[Online]. Available: http://scipipe.org high-throughput and high-content molecular and cell profil-
7. S. Lampa, M. Dahlö, J. Alvarsson, and O. Spjuth, ing technologies coupled with AI and predictive modeling
“Scipipe—A workflow library for agile development can enable study of complex phenomena in pharmacology,
of complex and dynamic bioinformatics pipelines,” toxicology, and metabolism.

20 ComputingEdge July 2020

 T
EN
EV
EW

IEEE
N

QUANTUM
WEEK

20

,
LD
20

IE
ER

SA F
U M
B
TO

O OO
C

R
O

R /B
6

R
–1

D
LO E
12

A
O V
C N E
D

REGISTRATION IS OPEN!
qce.quantum.ieee.org
The Future Directions Quantum Initiative invites you to IEEE Quantum Week
2020—the inaugural IEEE International Conference on Quantum Computing
and Engineering (QCE).

Council on Superconductivity
 EDITORS: Bruce Campbell, [email protected]
Francesca Samsel, [email protected]
This article originally
appeared in

DEPARTMENT: ART ON GRAPHICS

vol. 38, no. 4, 2018

Weather Report
A Site-Specific Artwork Interweaving Human Experiences and Scientific
Data Physicalization
Daniel F. Keefe and Seth Johnson, University of Minnesota
Ross Altheimer, TENxTEN Studio
Deuk-Geun Hong, University of Minnesota
Robert Hunter, James Dayton Design, Ltd.
Andrea J. Johnson, University of Minnesota
Maura Rockcastle, TENxTEN Studio
Marc Swackhamer, University of Minnesota
Aaron Wittkamper, Wittkamper / Reiff

Weather Report is a site-specific art installation that entices visitors to examine

climate change at a human scale, both physically and metaphorically. Weather data
are displayed using the balloons as physical pixels that can be touched, part of an effort
to make objective, scientific data graspable by non-scientists. Visitors contrast these
objective weather data with their own data – weather-related memories they enter at
a kiosk to create a subjective weather record from the Twin Cities community.

T
he historic Stone Arch Bridge, once the railroad scientists, created Weather Report in response to
gateway to the city of Minneapolis, crosses the theme of “Climate Chaos | Climate Rising.”1 We iden-
Mississippi River at St. Anthony Falls. Walking tified common threads around the experience we
across this gently curving path on a warm June night, we wished to create: making climate personal and con-
hear the sounds of the Northern Spark art festival. Held necting objective scientific data to subjective human
each year in the Twin Cities, Northern Spark has grown experiences. Then, over months of interdisciplinary
to attract tens of thousands of people who view perfor- design and discussion, we developed, interwove, and
mances, explore temporary installations in the streets revised these threads drawing upon our plurality of
and along the riverfront, and gather for one night to design voices and different technical research interests.
experience art as a community until the sun rises.
Looking down from the bridge on this night in 2016, CONCEPTUAL FRAMEWORK
colored lights flicker and reflect off the water, as a AND EARLY DESIGN
stream of local residents and visitors wind down from Weather Report uses local, human experiences with
the bridge, through Mill Ruins Park, and along the walk- weather as an entry point for discussing the difference
ing path to what people are describing as the “balloon between objective weather data and subjective inter-
tunnel.” More than 800 miniature weather balloons pretations and memories. Although the data are on
are suspended to form two walls that undulate, like the human timescale (45 years of objective data and
air-filled sheets in the night wind. Both the balloons memories), the objective-subjective comparisons the
and the steady stream of visitors perform an animated piece asks visitors to make speak to the broader ques-
dance as the Mississippi River flows by on a parallel tion of how the earth’s climate has shifted over a much
course (see Figure 1). larger period of time, how this is measured and inter-
Our design collective, MINN_LAB, composed preted today, and how rigorous scientific processes
of architects, landscape architects, and computer differ from everyday discussions of weather.

22 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 FIGURE 1. “Balloon pixels” create an illuminated physical manifestation of 4.5 decades of local weather data, installed along the
Mississippi River during the Northern Spark arts festival. Photo credit: Authors, 2016 (Used with permission.)

Scientists often struggle to explain the objective Getting Physical with Balloon Pixels
basis for climate change, what it means to the average Inspired by the role weather balloons play in data col-
citizen, and the grand timescale on which it operates. lection, we came to view the balloon as a simple, phys-
In contrast, our friends and family have no trouble at all ical, relatable manifestation of the scientific process.
explaining everyday human experiences with weather Each weather balloon provides a small data-driven
and extrapolating from these: On the day you were contribution to the larger picture of the science. Rein-
born, there was an amazing blizzard, it took me four terpreting this in the context of an experiential display
hours to shovel the car out of the driveway; that used of data, we reasoned that balloons could function as
to happen all the time, but we don’t have storms like physical “pixels,”, changing appearance in response to
that anymore. Or, just as common, we have never had individual data readings and collectively presenting a
a winter with so many violent storms in a row; grandpa broad picture of the scientific data.
had to buy a generator because the power kept going The balloon-as-pixel concept could have many
off; this year is the worst ever. Weather Report asks interpretations, and this fueled a rich, several
visitors, which of these weather memories is true? month-long period of sketching and ideation within
How do human experiences and memories compare the team. Drawing upon the architectural tradition,
with objective data? How do human timescale data the site for the installation was critical in the design.
points relate to the much larger climate timescale, and One of the prominent features of Mill Ruins Park is
how do scientists objectively measure those data? a walking path that follows the river. Thus, we were

www.computer.org/computingedge 23
ART ON GRAPHICS

FIGURE 2. A tunnel of balloon pixels along the walking path in Mill Ruins Park, Minneapolis, MN. Photo credit: Authors, 2016
(Used with permission.)

inspired to create a walk-though experience—some- ILLUMINATED VISUALS AND

thing viewers could experience and touch as they USER INTERFACE
traversed the path. To stay true to the science, we established a goal to
The method for illuminating or otherwise adjusting make sure that the data-to-visual mapping could be
the balloon pixels in response to data also required considered accurate. However, we also recognized
design. We experimented with balloons on strings that, unlike many data visualizations, the primary goal
with motors and internal LEDs but found that the best of this site-specific artwork is not facilitating analy-
visual results could be achieved by projecting colored sis using specific data points but rather creating an
light onto the balloons. Light is transmitted through experience—a feeling of being aware of and immersed
the balloon, creating the effect of a glowing orb. within the data. This creates the catalyst for discus-
sion that is the real goal of the piece.
Two Walls for Contrasting
Views of Weather Mapping Temperature
The final design arranges the balloons into two walls, Data to Balloon Walls
each constructed from a 12x36 grid of balloon pixels, The objective local weather data were recorded at the
forming a tunnel around the walking path (See Figure US Weather Station KMSP located at the Minneapolis–
2). Within the park, the tunnel is positioned so as to St. Paul International Airport (retrieved via the Wolfram
be visible at a distance from the Stone Arch Bridge or Alpha computational engine). The dataset contains
the hillside looking down to the river and so that light nearly 4.5 decades of hourly readings for temperature,
from the illuminated balloons reflects off the water wind speed, and precipitation (rain and snow) from 12
at night. a.m. on 1 January 1960 to 7 a.m. on 11 June 2016.
Conceptually, the mirrored walls create a strong All of the multi-decade, multi-variable data are dis-
physical basis for visual and body-centric comparisons played (over time) using only the 12x36 pixels available
of the data. Viewers can stand inside the tunnel and for each wall. To accomplish this, the balloon pixels are
point with one hand to a balloon that depicts objective organized into a hierarchical time grid, where each bal-
temperature data for a particular hour, day, month, loon represents a time window, as illustrated in Figure
year, or decade and then point directly across the tun- 3. The base color for each balloon is set by applying a
nel with the other hand to find the corresponding data warm-to-cool color map to the average temperature
point in the subjective record. within the balloon’s time window.

24 ComputingEdge July 2020

 ART ON GRAPHICS

FIGURE 3. The data-to-visual mapping for each wall uses a hierarchical arrangement for time, decades are displayed in the two
leftmost columns, with the current decade highlighted, followed by months of the year with the current month highlighted, fol-
lowed by days of the month, and so on. 7 a.m. on Tuesday, 24 June 2016 is highlighted in this figure.

FIGURE 4. Frames from an animated blue rain effect superimposed over the objective weather wall.

The animated visualization updates at a rate of one Animated Effects for

historical hour each half-second of real-world clock Secondary Data Variables
time with the “current time” signified by a white high- Secondary data variables (rain, snow, wind speed, and
light. This means that the white highlight for the cur- cloud cover) are also included, but these are treated
rent hour in the far right column moves quickly from as discrete weather “events”. When the current time
the top to the bottom of the column every six seconds. reaches an hour that includes one or more weather
When it reaches the end, the hourly-data columns events, a three-second animated weather effect is
each shift to the right by one, paging a new column of added to a queue. These effects are applied to the
twelve hours into the display. Likewise, when the cur- entire display as a semi-transparent overlay, using all
rent time reaches the end of the last day in a month, a of the pixels as illustrated in Figure 4. For a rain event,
new month of data is paged into the display. blue pixels stream down the wall. Snow events create

www.computer.org/computingedge 25
ART ON GRAPHICS

memory is entered by selecting the decade, then year,

then month, then day, and finally hour by touching
the corresponding balloons on a diagram (Figure 5,
top)—in this way, the data entry process teaches visi-
tors about the data mapping used on the walls. Then,
the temperature, rain, snow, wind speed, and cloud
cover are entered using sliders with extremes labeled
in relative, not numeric terms (e.g., the hottest day
vs. the coldest day) (Figure 5, middle). Finally, visitors
act out their weather memory, using the multi-touch
screen to create an animation (Figure 5, bottom).
Similar to the animated weather effects, these
weather memory animations are replayed as semi-
transparent overlays immediately after entering the
memory and again whenever the animation reaches
the time associated with the memory.

Projecting on Balloons:
Some Technical Notes
The balloons are lit from outside the tunnel by four
short-throw projectors and a custom graphics pro-
gram with five coordinated output windows. Calibra-
tion is critical because the site, next to the river and a
hill, requires projecting from extreme angles. The pro-
jected graphics are created by first rendering a regu-
lar grid of 2D colored circles to an off-screen buffer.
Then, this temporary image is texture mapped onto a
deformable four-sided polygon. To calibrate each pro-
jector, a keyboard user interface is used to interac-
tively adjust the coordinates of the corners of the poly-
FIGURE 5. Visitors enter weather memories using a three- gon, skewing it until the colored circles project onto
stage, multi-touch interface. the correct balloons.

THE EXPERIENCE
slower, gentler white pixels wafting down the wall. Thousands of visitors experienced Weather Report at
Cloud-cover events tint the top of the wall light gray, Northern Spark (see Figure 6). People walked through
and wind events cause gray particles to fly across the the tunnel and around the sides. They touched the bal-
wall from one side to the other. loons as they walked. More than 200 people contributed
memories to the subjective weather record. The earli-
The Subjective Weather Record est entry was for 13 Dec. 1961, and there were 22 entries
The subjective dataset contains the same variables for the Halloween Blizzard of 1991. It would have been
but is much sparser. Over the course of the night, vis- ideal to include additional stations or mobile devices
itors fill in this subjective weather record, and the for entering subjective data, as there were often long
visualization interpolates between each data point to lines at the kiosk. Visitors who had a chance to act out
fill in the gaps. their weather memories using their hands and then
The kiosk for entering weather-related memories see the result displayed on the balloons said they felt
uses a multi-touch interface. The specific hour for the as if they were playing a 36-foot-long instrument.

26 ComputingEdge July 2020

 ART ON GRAPHICS

FIGURE 6. Weather Report at Northern Spark 2016. Photo credits: Authors (left, middle), Krista McCullough (right), 2016. (Used
with permission.)

The biggest surprise was in how viewers reacted to data and science in the built environment, described in
the projection. As designers, we treated the projection more detail in a companion paper within the architec-
as simply a behind-the-curtain technology – the tech- ture community.2 Computer scientists have learned to
nology that just happened to provide the best method become a little bit more like architects and now have a
of illuminating the balloons. To our surprise, the projec- foundation for further explorations of embodied expe-
tion beams became interactive play areas, places for riences and physicality in data visualization.
visitors to dance, pose for pictures with data covering Data visualization researchers will connect this
their bodies, and cast shadows on the data that could work to the emerging theme of “data physicalization.”3
be seen from across the river and around the festival. Although Weather Report includes digital components,
Visually, this added another layer of human connection the resulting piece is a decidedly physical, real-world,
and embodied movement to the data-driven visuals. data-driven experience. This is a topic where the
computer science community has much to learn from
CONCLUSIONS architects, designers, and artists. Collaborations like
Interdisciplinary collaborative design processes are this one make it possible to explore concepts of data
both rewarding and challenging. Our experience in physical space at a scale and connecting with the
and results reflect months of discussion, much of it local community in ways that are simply impossible in
devoted to learning to speak the language of each oth- a traditional computer science context.
er’s disciplines. The whole team had to work within the
constraints of an outdoor, dusk-to-dawn festival and a ACKNOWLEDGMENTS
specific site. Each discipline had to stretch to accom- This work was supported in part by a grant from the
modate the interests and expertise of the other. This University of Minnesota Digital Design Center.
process led to a unique result, one that no individual on
the team would have created. It also led to new think- REFERENCES
ing that broadens and refuels our primary disciplines. 1. “Northern Spark Call For Proposals: Climate Chaos |
This is one of the powerful recurrent themes in the Climate Rising,” Northern Lights.mn, 2016; http://www
work highlighted in this Art on Graphics column. In our .mnartists.org/content/northern-spark-call-proposals
case, we will surely continue to collaborate as an inter- -climate-chaos-climate-rising.
disciplinary collective, but even as we return to our pri- 2. M. Swackhamer et al., “ Weather Report: Structuring
mary disciplines we bring new knowledge with us that Data Experience In The Built Environment,” Proceed-
will impact our future work. Architects have learned ings Of Architectural Research Centers Consortium,
to become a little bit more like computer scientists 2017, pp. 102–111.
and now have a foundation for further explorations of 3. Y. Jansen et al., “ Opportunities And Challenges For

www.computer.org/computingedge 27
ART ON GRAPHICS

Data Physicalization,” Proceedings Of The 33rd Annual Design, Ltd. Contact him at [email protected].
ACM Conference On Human Factors In Computing
Systems (CHI 15), 2015, pp. 3227–3236. ANDREA J. JOHNSON is associate director of MS Research
Practices in the School of Architecture at the University of
Minnesota and principal of Andrea J. Johnson Architecture.
DANIEL F. KEEFE is an associate professor in the Depart- Contact her at [email protected].
ment of Computer Science & Engineering at the University of
Minnesota. Contact him at [email protected]. MAURA ROCKCASTLE is a principal and co-founder at TENx-
TEN Studio. Contact her at [email protected].
SETH JOHNSON is a PhD student in the Department of Com-
puter Science & Engineering at the University of Minnesota. MARC SWACKHAMER is a professor and head of school at
Contact him at [email protected]. the School of Architecture at the University of Minnesota.
Contact him at [email protected].
ROSS ALTHEIMER is a principal and cofounder at TENxTEN
Studio. Contact him at [email protected]. AARON WITTKAMPER is founder and design lead at Witt-
kamper Studio and cofounder at Wittkamper / Reiff. Contact
DEUKGEUN HONG is an architectural designer at MSR him at [email protected].
(Meyer, Scherer & Rockcastle, Ltd.) Design. Contact him
at [email protected]. Contact department editor Bruce Campbell at bcampbel01
@risd.edu or department editor Francesca Samsel at figs
ROBERT HUNTER is a senior associate at James Dayton @cat.utexas.edu.

ADVERTISER INFORMATION

Advertising Coordinator Central US, Northwest US, Southeast US, Asia/Pacific:

Eric Kincaid
Debbie Sims Email: [email protected]
Email: [email protected] Phone: +1 214-553-8513 | Fax: +1 888-886-8599
Phone: +1 714-816-2138 | Fax: +1 714-821-4010 Cell: +1 214-673-3742

Midwest US:
Advertising Sales Contacts Dave Jones
Email: [email protected]
Mid-Atlantic US: Phone: +1 708-442-5633 Fax: +1 888-886-8599
Dawn Scoda Cell: +1 708-624-9901
Email: [email protected]
Phone: +1 732-772-0160
Cell: +1 732-685-6068 | Fax: +1 732-772-0164 Jobs Board (West Coast and Asia), Classified Line Ads

Southwest US, California: Heather Bounadies

Mike Hughes Email: [email protected]
Email: [email protected] Phone: +1 623-233-6575
Cell: +1 805-208-5882

Northeast, Europe, the Middle East and Africa: Jobs Board (East Coast and Europe), SE Radio Podcast
David Schissler
Email: [email protected] Marie Thompson
Phone: +1 508-394-4026 Email: [email protected]
Phone: +1 714-813-5094

28 ComputingEdge July 2020

 !

HOST
W
O
N
T ER
IS

2020
G
RE

6–9 Dec. 2020 • San Jose, CA

IEEE INTERNATIONAL SYMPOSIUM

ON HARDWARE-ORIENTED
SECURITY AND TRUST
6–9 Dec. 2020 • San Jose, CA, USA • DoubleTree by Hilton

Join dedicated professionals at the IEEE International Symposium on Hardware Oriented Security
and Trust (HOST) for an in-depth look into hardware-based security research and development.

Key Topics:
• Semiconductor design, test • Cryptography
and failure analysis and cryptanalysis
• Computer architecture • Imaging and microscopy
• Systems security

Discover innovations from outside your sphere of influence at HOST. Learn about new research
that is critical to your future projects. Meet face-to-face with researchers and experts for
inspiration, solutions, and practical ideas you can put to use immediately.

REGISTER NOW: www.hostsymposium.org

 EDITOR: Theresa-Marie Rhyne, [email protected]
This article originally
appeared in
DEPARTMENT: VISUALIZATION VIEWPOINTS
vol. 38, no. 5, 2018

OpenSpace:
Bringing NASA Missions to the Public
Alexander Bock, New York University
Charles Hansen, University of Utah
Anders Ynnerman, Linköping University

This viewpoint presents OpenSpace, an open-source astrovisualization software

project designed to bridge the gap between scientific discoveries and their public
dissemination. A wealth of data exists for space missions from NASA and other
sources. OpenSpace brings together this data and combines it in a range of immersive
settings. Through non-linear storytelling and guided exploration, interactive immersive
experiences help the public to engage with advanced space mission data and models,
and thus be better informed and educated about NASA missions, the solar system and
outer space. We demonstrate this capability by exploring the OSIRIS-Rex mission.

T
he endeavors of space missions have typi- and operations.4–6 OpenSpace is capable of provid-
cally been communicated and demonstrated ing immersive experiences to multiple observers by
through produced movies or animations. NASA leveraging the projection capabilities of modern plan-
missions, such as OSIRIS-REx, are the products of etariums.7 These immersive experiences will help bet-
years of planning and scientific research. Explaining ter inform and educate the general public by allowing
missions to the public in an immersive setting is chal- interactive exploration of NASA missions. OpenSpace
lenging and has only previously been possible with has been developed with partners from the American
produced shows for planetariums or interactively with Museum of Natural History, Linköping University, the
limited mission details and external data.1,2 Our goal is Scientific Computing and Imaging Institute at the
to provide an interactive experience in which the pub- University of Utah, and New York University’s Tandon
lic can see and experience space missions to better School of Engineering.
understand the science, the benefit to mankind, and
the challenges of deep-space missions. SPICE FILES AND NASA MISSIONS
To achieve this, we developed the open source OpenSpace relies on a continuous coordinate sys-
interactive data visualization software OpenSpace3 tem that enables developers to position and ori-
to visualize the entire known universe and portray ent data with extreme accuracy. NASA’s Spacecraft
our ongoing efforts to investigate the cosmos Planet Instrument C-matrix Events (SPICE) obser-
through large-scale, contextualized, multimodal vation geometry system for planetary science mis-
astrovisualization. OpenSpace incorporates the lat- sions, for example, provides the ability to display mis-
est techniques from visualization research and sup- sion planning, data acquisition, and post-mission data
ports interactive presentation of dynamic data from analysis using high-resolution data files called kernels.
observations, simulations, and space mission planning Each mission has a different set of SPICE kernels that

30 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 describe the spatial and temporal aspects of the mis- Orbiter that mapped the Moon, the Mars Reconnais-
sion and its components. For example, in collaboration sance Orbiter retrieving unprecedented details of the
with the Applied Physics Laboratory, the OpenSpace surface of Mars, and the numerous Earth-orbiting
team worked to visualize the Navigation and Ancillary spacecraft providing time-resolved imagery of our
Information Facility’s SPICE-specified instrument tar- own planet. Many of these maps are provided as
geting during New Horizons’ fly-by of Pluto.4 SPICE’s images online through the Web Map Service stan-
guided visualization of planet and spacecraft orbits, dard. Through this standard, images are stored in a
instrument view frustums, and sequenced imagery tree with varying resolution and only images of appro-
over time allow for comparison of observed data with priate resolution are streamed from an online server
model output. While New Horizons was a specific to the client application. Streaming data from a vari-
application, the techniques apply generically across all ety of online sources enables the rapid exploration
space missions with available SPICE data. SPICE com- of datasets from different spacecraft and also pro-
pliance accurately defines positions, orbits, trajecto- vides the ability to access near-real time imagery with-
ries, orientations, and instrument views used in data out the need to download entire image catalogs, thus
collection for Earth and planetary science, heliophys- reducing the latency between data acquisition and
ics, and astrophysics. visualization. NASA’s Global Imagery Browse Services
The ability to read SPICE kernels during mission (GIBS) provides daily updates for many of their image
planning and flight, including actual configurations sources, such as Suomi-NPP’s Visible Infrared Imag-
returned by telemetry during flight, will allow for both ing Radiometer Suite, which provides a full global view
historical reconstruction and conceptual visualization of Earth every day depicting weather and cloud cover-
of missions within OpenSpace. To date, such full SPICE age. Figure 1 shows an image of the three major hur-
visualization has only been available in specialized ricanes that impacted the United States in 2017 and
mission planning software, in produced videos such that was generated from the GIBS dataset and is being
as those created by NASA’s Scientific Visualization inspected by a user in real time. In addition to visible
Studio, or in a limited capability with commercial light measurements, many other mission instruments
software.2 Never before has there been an interactive, are available through this standard, for example cloud
deep level of SPICE capability in planetariums or digital layers, ozone concentration, and temperature mea-
theaters using freely available open source software. surements, thus providing an immense wealth of infor-
Visualizing spacecraft attitude with respect to an mation that can be dissected in the context of the
observation schedule is an important link between observing spacecraft.
mission operations and the science it is designed to
acquire. Visualizing the engineering of how space mis- MULTIUSER IMMERSIVE
sion science is conducted is especially valuable with ENVIRONMENTS
respect to public education. The benefits of multiuser immersive environments
are deeply rooted in the human desire to share experi-
DATA SOURCES ences and the need for a social context in which knowl-
A major product of many space missions is compre- edge can be understood and assessed. Immersive mul-
hensive planetary maps of the bodies in our solar sys- tiuser environments come in many different flavors,
tem. Examples of this include the MESSENGER mis- with different advantages and disadvantages. The
sion that mapped Mercury, the Lunar Reconnaissance available systems range from shared VR environments

www.computer.org/computingedge 31
VISUALIZATION VIEWPOINTS

FIGURE 1. A view of hurricanes impacting the United States on 9 September 2017 based on a NASA’s Global Imagery Browse
Services dataset. From West to East (left to right): hurricanes Katia, Irma, and Jose.

using multiple head mounted displays to immersive high-resolution immersive experiences such as Fig-
environments such as CAVE(s) and large-scale display ure 2 shows. In these environments, user interaction
systems such as dome theaters. One key difference is, however, often limited to one user presenting
between these environments is the number of users and interacting. In dome theaters, this leads to the
who are tracked to provide correct first-person view notion of a mediated immersive experience in which a
of the virtual world being displayed. In multiuser head presenter, often together with a pilot, guides the audi-
mounted systems, all users are tracked. In CAVE(s), ence through a demonstration. As such, this becomes
only one person has the “correct view”. In domes on a powerful tool for immersion because the story and
the other hand, usually no user is tracked. the story teller becomes integrated components of
Our experience shows that multiuser immersion the immersion. OpenSpace was developed primarily
can be facilitated in several ways without the need for dome theater immersion. It benefits greatly from
of tracking users. In dome theaters the size of the the interactive capabilities of the system whereby the
display and the distance to the users make it pos- multiusers perceive the 3D effect of the cosmos as
sible to generate reasonably correct views for most long as the projected image reflects camera motion.
of the users, and the size of the environments adds If it stops, the users suddenly experience a 2D image
the advantage of scalability in number of simultane- projected onto the dome. The 3D illusion makes this
ous users. The same argument also holds for large environment extremely compelling for contextualized
flat displays, such as projection walls, that support astrovisualization.

32 ComputingEdge July 2020

 VISUALIZATION VIEWPOINTS

FIGURE 2. Multitouch table driving a powerwall using a linked OpenSpace session such that the viewpoint on the power wall
is controlled by the touch table. The imagery is the Valles Marineris of Mars obtained from orbiting satellites and streamed to
OpenSpace through the Web Map Service.

Interaction can also provide a path to immersion. to its celestial bodies, any existing object rapidly
Multiuser interaction on touch tables is common becomes exceedingly small. A direct-manipulation
these days and enables immersion via shared input, solution alone becomes non-trivial as it requires 3D
along with content involvement among users sharing points in the scene to constrain the manipulation and
interaction, which can sometimes result in conflict but such points cannot be well tracked in empty space.
can also lead to discussion and user engagement. The multitouch interface is implemented in
OpenSpace using the TUIO library to support a variety
TOUCH TABLE of multitouch devices, although we primarily targeted
AND PROJECTION WALL the multitouch table. Using this interface, the user can
Using large tangible touch surfaces with a multitouch interact with any celestial body in the scene and tra-
navigation interface is more engaging to users than verse it through multitouch gestures. For example, Fig-
a mouse and keyboard, while also enhancing under- ure 1 shows interaction with satellite imagery weather
standing of the navigation control and thus, decreas- data. This allows the user to intuitively zoom into the
ing the learning curve of the system’s user interface.8 geographical area of interest.
Additionally, combining a multitouch interaction model Additionally, OpenSpace provides the capability
together with a screen-space direct-manipulation for- for coupling OpenSpace sessions across devices.
mulation produces a user-friendly interface. We have These could be multiple planetariums, as was done for
integrated the OpenSpace user interface with a com- the New Horizons encounter coupling, among others,
mercial multitouch display table provided by the com- ANMH/Hayden Planetarium with the Norrköping Visu-
pany Interspectral AB9. aliseringscenter C Planetarium for a cross-continent,
Astronomical visualizations have long been an interactive visualization and story-telling of the mis-
interesting application for touch-based interaction. sion while the encounter was taking place. With the
However, due to the scale of the solar system compared multitouch interface, we found it advantageous to

www.computer.org/computingedge 33
VISUALIZATION VIEWPOINTS

FIGURE 3. OSIRIS-REx lift-off from Cape Canaveral on 8 September 2016. The location of the spacecraft at a particular time-
point is determined from NASA’s Spacecraft Planet Instrument C-matrix Events data for the mission.

couple the touch-table interface with a large tiled sample, the mission will map the asteroid and use
display to effectively contextualize NASA missions to its advanced instruments to measure the Yarkovsky
groups of K-12 students and the general public. Figure effect (non-gravitational forces which cause orbital
2 shows the table driving the tiled display wall to seam- deviation) as well as compare close-range observa-
lessly control the OpenSpace session for multiple tions with Earth-based observations.
participants. A YouTube video of the interaction is By using NASA’s SPICE kernels for this mission,
available.10 the reconnaissance campaign mapping the aster-
oid can be visualized and the instrument activation
OSIRIS-REx Mission highlighted as shown in Figure 5. By using the actual
OSIRIS-REx launched on 8 September 2016 at 7:05 PM mission planning data contained in the NASA SPICE
EDT from Cape Canaveral. Figure 3 shows the trail of event file, which contains the mission timeline, a
the spacecraft a few minutes after lift-off. OSIRIS-REx visualization and contextualization of the mission is
orbited the Sun and in September 2017, it used a grav- accomplished. This can be shown in an immersive
ity assist in an Earth fly-by to increase the orbital incli- environment to describe the mission and its scientific
nation to deviate from an orbit in the solar plane to an actions to better inform and educate the public.
orbit matching the orbital plane of asteroid Bennu. Fig-
ure 4 shows the spacecraft’s “slingshot” to the orbital CONCLUSION
plane of Bennu. We have described OpenSpace, a software for the
The OSISRIS-REx mission seeks to gather a sample visualization and demonstration of NASA missions.
from the surface of Bennu, a carbonaceous asteroid The ability to interactively display and communicate
and return the sample to Earth. The sample may pro- missions during mission planning and/or mission
vide information on the formation of life on Earth and activity and/or post-mission data analysis is a power-
the Earth’s oceans, such as whether Bennu contains ful method for public education. One key component is
organics, precious metals, or water. In addition to the the ability to visualize the data acquisition operations

34 ComputingEdge July 2020

 VISUALIZATION VIEWPOINTS

FIGURE 4. On 22 September 2017, OSIRIS-REx used Earth’s gravitational field to slingshot from the solar orbit to the orbital plane
of the asteroid Bennu. Earth is the cyan trail while OSIRIS-REx is the green trail. The celestial bodies and spacecraft locations are
read from NASA’s Spacecraft Planet Instrument C-matrix Events data for the OSIRIS-REx mission.

FIGURE 5. OSIRIS-REx maps Bennu on 25 May 2019 before obtaining the samples. The spacecraft and Bennu are modeled as
geometry. Once the actual images are sent back to Earth, the mapping would appear as textures on the asteroid surface. The
spatial/temporal locations of the asteroid, spacecraft, and its instruments are read from NASA’s Spacecraft Planet Instrument
C-matrix Events data for the mission.

www.computer.org/computingedge 35
VISUALIZATION VIEWPOINTS

from the viewpoint of the spacecraft using SPICE data. Of Public Dissemination In Astronomical Visualization
Immersive environments add to the realism of pre- From What To How,” IEEE Computer Graphics And
senting NASA missions and a variety of immersive Applications, vol. 38, no. 3, 2018, pp. 112–118.
devices are supported; from display walls to touch 4. A. Bock et al., “OpenSpace: Public Dissemination Of
tables to multiuser immersive theaters such as plan- Space Mission Profiles,” 2015 IEEE Scientific Visualiza-
etariums. We believe extensions to educate the public tion Conference (SciVis 15), 2015, pp. 112–118.
on space science, such as explaining the importance 5. E. Axelsson et al., “Dynamic Scene Graph: Enabling
of space weather, will result in better public awareness Scaling, Positioning, And Navigation In The Universe,”
and support of NASA missions. Our viewpoint is that Computer Graphics Forum, vol. 36, no. 3, 2017,
diverse information needs to be aggregated and made pp. 112–118.
available to visualization software that can integrate 6. K. Bladin et al., “Globe Browsing: Contextualized
multiple sources, allowing a coherent visual commu- Spatio-Temporal Planetary Surface Visualization,” IEEE
nication to a wide variety of audiences and space sci- Transactions On Visualization And Computer Graphics,
ence consumers. This requires the flexibility of a num- vol. 24, no. 1, 2018, pp. 112–118.
ber of delivery and interaction mechanisms from the 7. C. Emmart et al., “OpenSpace: From Data Visualiza-
software. tion Research To Planetariums And Classrooms
Worldwide,” American Geophysical Union, Fall General
REFERENCES Assembly, 2016.
1. “The Systems Toolkit,” Analytical Graphics, Inc., other; 8. T. Isenberg, “Interactive Exploration Of
www.agi.com/products/stk. Three-Dimensional Scientific Visualizations On Large
2. S. Klashed et al., “Uniview -- Visualizing The Universe,” Display Surfaces,” Collaboration Meets Interactive
Proceedings Of Eurographics 2010 -- Areas Papers, Spaces, C. Anslow, P. Campos, J. Jorge, Springer, 2016.
2010, pp. 112–118. 9. “Interspectral AB: Commercial Provider Of Touch Table
3. A. Bock et al., “OpenSpace -- Changing The Narrative Technology,” other; www.interspectral.com.
10. J. Bosson, “Touchtable Direct-manipulation With
OpenSpace,” other https://youtu.be/uL-2Rpdl-68, 13
July 2017; https://youtu.be/uL-2Rpdl-68.

Call for Articles ALEXANDER BOCK is a Moore-Sloan Postdoctoral Research

Fellow with New York University’s Center for Data Science
IEEE Software seeks
and the Lead Developer of OpenSpace. Contact him at
practical, readable articles
that will appeal to experts
[email protected].
and nonexperts alike. The
magazine aims to deliver CHARLES HANSEN is a professor of computer science in
reliable information to software the School of Computing and an Associate Director of the
developers and managers to
Scientific Computing and Imaging Institute at the University
help them stay on top of rapid
of Utah. Contact him at [email protected].
technology change. Submissions
must be original and no more
than 4,700 words, including 250 ANDERS YNNERMAN is a professor of scientific visualiza-
words for each table and figure. tion at Linköping University and an adjunct professor at the
University of Utah. He is also the director of the Norrköping
Visualization center and one of the co-founders of the Open­
Author guidelines:
www.computer.org/software/author Space project. Contact him at [email protected].
Further details: [email protected]
www.computer.org/software
Contact department editor Theresa-Marie Rhyne at
[email protected].

36 ComputingEdge July 2020

 Pa p e rs :
Call for c t i o n s
E Tra n s a
IE E
p u te r s
on Com

Publish your work in the IEEE Computer Society’s flagship journal, IEEE
Transactions on Computers (TC). TC is a monthly publication with a wide
distribution to researchers, industry professionals, and educators in the
computing field.

TC seeks original research contributions on areas of current computing

interest, including the following topics:

• Computer architecture • Security and reliability

• Software systems • Machine learning
• Mobile and embedded systems • Quantum computing

All accepted manuscripts are automatically considered for the monthly

featured paper and annual Best Paper Award.

Learn about calls for papers and submission details at

www.computer.org/tc.
 EDITOR: Philippe Kruchten, University of British Columbia, [email protected]
This article originally
appeared in
DEPARTMENT: SOUNDING BOARD
vol. 36, no. 6, 2019

A Manifesto for Energy-Aware

Software
Alcides Fonseca, Rick Kazman, and Patricia Lago

A
ccording to recent estimates, computing need a manifesto? Why now? Although there has been
and communications could account for 20% some attention to this area,3 we believe it has been
of energy usage globally by 2025.1 This trend grossly insufficient given the high stakes involved. The
shows no sign of slowing. The annual growth in power vast majority of practitioners (and researchers) are
consumption of Internet-connected devices is 20%. completely ignorant of energy concerns; they, and the
Data centers alone are now accounting for more than programs they create, are anything but energy aware.4
3% of global emissions. Even if you are not worried
about this trend on the mega scale, you are likely con- THE NINE PRINCIPLES OF
cerned with the power consumption of the devices in ENERGY AWARENESS
your pocket, on your wrist, and in your ears. Energy awareness is a necessary but not sufficient
Software, hardware, and network attributes precondition for energy efficiency. Energy awareness
all contribute to power usage, but little attention is required from all stakeholders, such as end users
has been given to this topic by the information and who may choose product A versus product B based on
communications technology (ICT) community. For energy characteristics. Our goal in this manifesto is to
example, as software engineers, we were never taught call for changes in how we think and what we do. This
to consider, much less manage, the energy consump- will not come for free, but we believe that the cost of
tion of the software systems we created. Despite our inaction is far greater.
lack of awareness and preparation, we are now facing
an undeniable reality: the software community must Public Awareness Is Key for
learn to design for, monitor, and manage the energy Widespread Adoption
usage of software. For this reason, we argue the need We believe that the key to widespread adoption of
for energy-aware software and present a manifesto energy-aware software is to sensitize and empower
describing nine guiding principles. By energy-aware end users. The scary statistics regularly published
software, we mean software that is consciously have proven ineffective so far (the amount of energy
designed and developed to monitor and react to being consumed by ICT, the increasing amount
energy preferences and usage. Energy efficiency is, of energy consumed by cloud providers, the mas-
therefore, one possible (but not the only) response to sive amounts of data being stored in data centers as
being energy aware. opposed to the negligible percentages of data being
This manifesto and the principles it proposes have actually used, and so forth). Neither do the worrisome
arisen from our experience and from the experience of energy-consumption predictions seem to spur us to
more than 100 researchers and practitioners who have action (such as the increasing number of things being
participated in six international workshops on the engi- connected to the Internet or the booming growth in
neering of green and sustainable software.2 Why do we mobile devices and their increasingly sophisticated
applications).
We need to turn these alarming trends into an
DOI No. 10.1109/MS.2019.2924498 opportunity: 1) to sensitize end users to the amount of
Date of current version: 22 October 2019 energy consumed by the software they use and 2) to
create awareness of the fact that software solutions

38 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 SOUNDING BOARD

with similar features may yield very different energy stakeholders. Early in the development process, cli-
profiles. Imagine that we were able to attach “green” ents should be asked about their energy requirements.
labels (like Energy Star ratings) to the apps available These should then be propagated to developers, tes-
in Google Play or the Apple Store. End users of mobile ters, and operations personnel so that they can be
devices could then compare the apps they are seeking taken into account, tested for, and monitored. Results
(such as apps for “meditation” or for “scanning docu- from these phases should be reported back to the
ments”) not only in terms of features, rating, and price; stakeholders, just like other important metrics.
if green labels were added, end users would be empow-
ered to make better-informed decisions based on the Education and Professional
labeled level of energy use. Such labels would force Training Should Cover Energy-
software companies to invest in optimizing the energy Aware Software by Default
impact of their applications (if they wanted to improve To create energy-aware software, we must edu-
their market position), with a resulting positive effect cate the next generation of engineers who need to
on the resources (for example, cloud services and net- acquire the competencies (and provide training for
works) that such applications use. the current-generation workforce, too). Depend-
ing on the target audience and the learning objec-
Incentives for Software Stakeholders tives, educational programs may adopt a centralized
Should Be Provided approach (concentrating the competencies crucial for
We believe that, although some people are altruis- energy-aware systems in one or two courses), distrib-
tic, most people respond to incentives. Therefore, uted (revising traditional courses to include competen-
such incentives should be put in place to encourage cies for energy awareness), or blended (including both
the creation of energy-aware systems, which will, in types of courses across the curriculum). Just as every
turn, lead to energy efficiency. Such incentives would programmer should understand algorithmic complex-
also help raise the consciousness of engineers and ity, energy awareness must become a standard com-
end users. Most stakeholders need to be incentiv- petency of every ICT practitioner and a standard con-
ized to actively pursue software and systems that are sideration for every decision maker and end user.
energy aware. This is a key step to raising the prior-
ity of energy-aware software in our companies and in Broad Adoption Requires
our society. Such incentives, both positive and nega- Attention to Usability
tive, have been used successfully for decades in other To encourage broad adoption of energy-aware sys-
parts of our economy: there are so-called sin taxes to tems, we must pay attention to the usability, from a
discourage drinking and smoking, and there are tax developer perspective, of the tools that we create.
rebates to encourage people to make their homes Best practices for energy-aware software engineering
more energy efficient. should end up being embedded in the tools, packages,
and frameworks we create so that software engineers
Energy-Aware Software do not need to reinvent the wheel. You should not
Engineering Should Be a Priority need a soldering iron or a circuit diagram—or a degree
for Every Stakeholder in electrical engineering—to manage the energy con-
The efficiency of system energy consumption is rele- sumption of an ICT system. Furthermore, it should be
vant for everyone, regardless of their roles in the devel- easy to reuse experience and best practices for engi-
opment process—from end users, concerned with neering energy-aware software.
their battery life, to business owners, concerned with
reducing their electricity costs and from the develop- Energy Awareness Should Be
ers, who should understand the energy impact of their Engineered Throughout the Lifecycle
contributions, to the product owners, who have to Energy awareness can and should be treated like an
decide in what way energy efficiency is a requirement. architectural quality attribute, 5 no different from how
Energy-related information should flow to all we design for, analyze, prototype, and manage other

www.computer.org/computingedge 39
SOUNDING BOARD

qualities in an architecture such as modifiability, per- car battery, it depends on the availability of charging
formance, availability, or security. This means that stations nearby).
architectures are design blueprints with system-wide We believe that software should perform its core
resource-management strategies. For example, functionality while simultaneously ensuring energy
power usage requirements should be explicitly col- awareness. If this happens, end users can count on
lected during requirements gathering, designed for, software applications to be reliable and to promise
and tested for. the best tradeoff between energy and functionality.
In particular, energy awareness and energy effi- To do so, software must be able to detect that its con-
ciency must be designed into a system early in its text has changed and that (possibly) some resources
lifecycle and considered when making major changes have become scarce and flexibly adapt by replacing
to the system. Leaving this consideration until the sys- them with alternatives or downgrading the delivered
tem is already built is a recipe for disaster. Experience functionality.
tells that any quality that you address late in develop-
ment tends to be treated superficially (or at great We Value Measures Over Beliefs (and
cost). If you don’t measure it, you cannot manage it.6 Reliable Trends Over Precision)
This energy awareness will have an associated cost, Energy awareness can be easily ignored early in the
in terms of system complexity, and this cost must be software lifecycle if there are no data to support the
acknowledged and accepted by the ICT community. fact that energy should be a concern. Another com-
mon pitfall is to believe that optimizing for energy effi-
Software Quality Should Not Come at ciency is difficult, if not impossible.7 Measuring energy
the Expense of Energy Awareness consumption is a first step toward energy awareness.
Energy-aware software development does not imply At a minimum, it provides a baseline to compare when
that energy efficiency should be prioritized over other introducing changes into the system. Costly additions,
quality attributes. Being energy aware involves tak- in terms of energy, may need to be revised or even dis-
ing into account the energy consumption of software carded if low power consumption is a priority.
across the software development lifecycle. Energy Different methods for measuring energy con-
awareness cannot be ignored and should be explicitly sumption exist. Because software is often dynamic
considered in tradeoff decisions, even if the final deci- and depends on runtime information, such as the
sion is to prioritize some other quality attribute over it. size of exchanged data, understanding the trend of
Making software development processes explicitly energy efficiency is more important than knowing the
energy aware allows for stakeholders to be informed raw logged values. Just like the Big-O notation in time
of the options chosen regarding the energy consump- complexity, energy efficiency can be seen as another
tion of software. Data about the energy impact of complexity metric.
design choices can be used to inform future decisions
and improve designs with respect to their energy
efficiency. W e believe that energy-aware ICT is inevitable
for many reasons: economic reasons, sustain-
ability reasons, and because users will increasingly
Energy Awareness Demands demand it. To hasten its emergence, we have polled
Dynamic Adaptability the R&D community in a series of workshops and col-
Energy awareness is heavily influenced by the con- lected these nine principles to help focus the emer-
text in which software is being used, both because gence of energy awareness as a true subdiscipline
resource availability varies over time (for example, the of ICT and software engineering. We can see that the
battery load is limited, network connectivity is location successful emergence of this discipline depends on
dependent, or electricity rates change during the day) three foundations: 1) awareness, 2) education and
and because as end users move, their needs change training, and 3) the creation of a body of engineering
(for example, driving the most energy-aware route knowledge. We need all of them to make energy-aware
depends on our location, and if we need to charge our software a reality.

40 ComputingEdge July 2020

 SOUNDING BOARD

REFERENCES 7. C. Pang, A. Hindle, B. Adams, and A. E. Hassan,

1. “‘Tsunami of data’ could consume one fifth of global “What do programmers know about software energy
electricity by 2025,” The Guardian, Dec. 11, 2017. consumption?” IEEE Softw., vol. 33, no. 3, pp. 83–89,
[Online]. Available: https://www.theguardian.com 2016. doi: 10.1109/MS.2015.83.
/environment/2017/dec/11/tsunami-of-data-could
-consume-fifth-global-electricity-by-2025
2. “6th International Workshop on Green and Sustainable ALCIDES FONSECA is an assistant professor
Software.” [Online]. Available: http://greens.cs.vu.nl at LASIGE, Faculdade de Ciências da Universi-
3. C. Becker et al., “Sustainability design and software: dade de Lisboa, Portugal. Contact him at
The Karlskrona manifesto,” in Proc. 37th Int. Conf. [email protected].
Software Engineering, vol. 2, pp. 467–476.
4. G. Pinto and F. Castor, “Energy efficiency: A new RICK KAZMAN is a professor at the Univer-
concern for application software developers,” Comm. sity of Hawaii, Mãnoa, Honolulu, and a visiting
ACM, vol. 60, no. 12, pp. 68–75, Dec. 2017. scientist at the Software Engineering Insti-
5. N. Condori-Fernandez and P. Lago, “Characterizing tute, Pittsburgh, Pennsylvania. Contact him
the contribution of quality requirements to software at [email protected].
sustainability,” J. Syst. Softw., vol. 137, pp. 289–305,
Mar. 2018. doi: 10.1016/j.jss.2017.12.005. PATRICIA LAGO is a professor in software
6. “Peter Drucker,” Wikipedia. Accessed on: Sept. 16, 2019. engineering at the Vrije Universiteit Amster-
[Online]. Available: https://en.wikipedia.org/wiki dam, The Netherlands. Contact her at p.lago
/Peter_Drucker @vu.nl.

IEEE COMPUTER SOCIETY

Call for Papers
Write for the IEEE Computer
Society’s authoritative
computing publications
and conferences.

GET PUBLISHED
www.computer.org/cfp

www.computer.org/computingedge 41
 EDITOR: Gerd Kortuem, [email protected]
This article originally
appeared in
DEPARTMENT: IOT NEWS
vol. 18, no. 4, 2019

Next Generation IoT:

Toward Ubiquitous Autonomous
Cost-Efficient IoT Devices
Moustafa Youssef, Alexandria University
Mahbub Hassan, University of New South Wales

IoT enables a new set of novel applications that promise to transform the physical
world as we know it today. Current IoT applications have demonstrated the feasibility
and promise of this technology, but there is still a large space open for innovation. We
present our vision for the next generation of IoT devices: IoT devices that are autonomous,
in terms of energy and communication, as well as cost-efficient to deploy. We present
enabling-technologies to realize our vision and open research challenges.

AUTONOMOUS IOT DEVICES possible because the new generation of transparent

Iot devices are expected to be deployed everywhere at solar cells absorb and harvest energy from infrared
scale, from well-powered homes to isolated environ- and ultraviolet light, but let the visible light pass
ments and rural areas with limited power supply. For through so we can see through them like a clear glass.
IoT devices to work independently, for an extended Back-scatter communication can also be used
time and with cost-efficiency, it is important for them to further reduce the operational energy of the IoT
to be self-powered and consume minimal energy for device, e.g., by leveraging the ambient RF signals to
their different functionalities. communicate with minimal energy to nearby devices,
Technologies proposed over the years for energy e.g., a gateway deployed in the field.4 Such backscat-
harvesting include RF-, light-, and mechanical/ tering is proving effective for data communication at
vibration-based.1,2 Each technology has its own kilo bps speed while consuming power only at a few
suitable environment for deployment, and they can micro Watts.
be combined together to harvest more energy. New
generations of solar panels, both opaque and trans- VALUE-ADDED SERVICES FOR
parent, are promising technologies for future IoT AUTONOMOUS IOT DEVICES
devices because they allow for full coverage of the Another interesting direction to save the energy of IoT
device body by solar panels, increasing the amount of devices, decrease their deployment cost, and increase
harvested energy. 3 For example, a smartwatch can be their autonomy is to provide value-added services, i.e.,
powered by opaque solar cells on the belt and trans- new applications that leverage the existing hardware
parent cells right on the screen (see Figure 1). This is and signals in novel ways. For example, the device-free
identification concept,5 where an object's presence,6
location,7,8 and features9 can be determined based
DOI No. 10.1109/MPRV.2019.2947974 on analyzing the changes in the ambient RF signals
Date of current version 21 January 2020. can also be used to further sense the environment in

42 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 IOT NEWS

Device
Power
Transparent Photocurrent
Solar
Panel
Gesture
Recog.
Opaque Solar
Panel

FIGURE 1. Smart-watch surface can be over-fitted with both opaque (on the straps) and transparent (on the surface) solar panels
to maximize the harvested energy. The change in the harvested current due to the user’s hand blocking the light arriving at the
watch can be used to detect different gestures.

a sensor-less fashion, i.e., without deploying special spoken words. This allows energy-efficient interac-
sensors on the nodes. This can be leveraged in appli- tions with the future IoT devices in a cost-efficient and
cations such as intrusion detection, activity recogni- user-friendly manner. Moreover, voice and hands-free
tion, and gesture-based device control, among many interaction with the IoT devices will enable persons
others.10 Extending this to different energy-harvesting with vision and hearing impairments to operate future
technologies provides both new opportunities and IoT devices everywhere with ease.
challenges.
For example, the change in the harvested power OPEN CHALLENGES
due to the user interaction with the device can be used Several challenges still need to be addressed to
to infer different user gestures. For instance, a user's realize our vision for the ubiquitous autonomous
hand blocking a solar panel will reduce or eliminate cost-efficient IoT devices. One challenge is to optimize
the energy harvested from light. The analysis of the energy harvesting under different IoT operational sce-
harvested energy level can then be used to estimate narios. For example, if vibrational energy harvesting is
the specific interaction pattern with the device as well used, one must tune the resonance frequency care-
as some of its attributes, e.g., how far away the user's fully to resonate with the vibration frequency of the
hand is from the device, number of repetitions, and device, because that is when the maximum energy is
speed of motion.11 Similarly, the human body affects harvested. However, an IoT device may be subject to
the received ambient RF signal, which can be lever- different levels of vibrations at different times, mak-
aged to estimate the presence of people in the area ing optimal energy harvesting difficult. Material and
of interest based on the changes in the RF-harvested mechanical engineering researchers are working on
power. Along the same line, a vibration-energy har- a new breed of hardware that may solve this problem
vester can be used to estimate the spoken words of in the future by dynamically tuning the resonance fre-
a person.5 What is common about all these ideas is quency of the device. Similarly, if transparent solar
that they use the harvested energy, without using any cells are used on the mobile device screen, the energy
special hardware, to detect how the user interacts output will be low if the transparency is high for a bet-
with the devices, whether through hand gestures or ter quality of viewing experience (Table 1). However,

www.computer.org/computingedge 43
IOT NEWS

Solar cell type Material Current Power

Density Conversion The heterogeneity and scale
Efficiency of IoT devices raise a number of
Opaque Silicon 35 mA/cm2 15–21% new challenges to supporting
Transparent Organic 3.82 mA/cm 2
7–12% sensor-less sensing, including
quantifying the quality of the dif-
TABLE 1. Example on the different solar cell types parameters.11 ferent signals coming from the dif-
ferent heterogeneous IoT devices,
performing multi-person identifi-
the user does not always look at the screen. As such, cation and separation, increasing the SNR, handling
mechanisms are needed that can dynamically adjust shadowing effect, and reducing the calibration effort
the transparency so it can be tuned to the highest (being device and person independent/agnostic).
transparency when the user is looking at the screen Different people in the area of interest will create
and switch to the lowest transparency for the highest several challenges including interference. Interference
energy harvesting at other times. We still do not have can be handled in some application domains by lever-
materials that can fulfill these requirements, but per- aging the redundancy of the IoT devices. For example,
haps techniques such as dynamic reflection control in an RF environment, the IoT devices installed in the
can be used at a separate layer to address this issue.12 area may hear different RF transmitters (e.g., WiFi
To support autonomous communication, we APs) concurrently. The information collected from all
should look for techniques beyond the radio-based devices about the detected APs along with their loca-
backscattering to ensure that IoTs can communicate tions can be fused to separate the different events in
even when there is no WiFi or other ambient radio the area. The idea is that an interfering human may
signals available. For fully autonomous battery-free affect some IoT devices but not all of them. Counting
IoTs, which are powered only through energy harvest- the number of people in the area of interest and sepa-
ing, we need novel communication paradigms that rating their interference signatures is also a potential
can effectively communicate in the face of dynamic direction for research.
and unpredictable power supply. There are new Due to the small amount of harvested energy, the
developments in radio communications that promise SNR of the harvested signal is usually low, making it
to generate extremely low-energy radio pulses in harder for a classification algorithm to detect specific
the terahertz frequency band using graphene.13 The interactions with the device. One way to handle this
energy consumption of such graphene pulses can be is to leverage deep learning algorithms and attempt
as low as only a few atto Jules due to the extremely automatic robust feature selection. Another idea is
short pulse durations on the order of femtoseconds, to try to increase the SNR, e.g., by using active light
which is made possible because of graphene's ability sources to interact with the solar panels to do the ges-
to resonate in an extremely wide frequency band. ture, instead of relying on the change in the ambient
How such graphene-based radio technology can light. Using active light sources has the extra benefit
help achieve communication autonomy for the next of increasing the harvested energy too.
generation battery-free IoTs is a new challenge open Another area to explore is multiservice IoT devices.
for research. The idea is to allow the same deployment of IoT
Unpredictability in power supply also creates chal- devices to perform different services concurrently
lenges for long computation tasks that may have to be such as intrusion detection and gesture recognition.
executed by energy-harvesting IoTs. Any disruption This increases the value of the deployed IoT nodes and
in power supply in the middle of a computation task opens the door for novel applications.
may force the IoT device to start all over again when
the power supply is back. How to jointly coordinate CONCLUSION
computation tasks and energy harvesting within such Future IoT devices will have autonomous opera-
autonomous IoTs require careful design of novel com- tion requiring zero human intervention for mainte-
putational models. nance, allowing for ease of operation at scale and

44 ComputingEdge July 2020

 IOT NEWS

mass adoption. A vision to achieve this goal is through 10. M. Youssef, “CoSDEO 2016 Keynote: A decade later
multitechnology energy harvesting and sensor-less — Challenges: Device-free passive localization for
sensing. The new technologies for energy harvest- wireless environments,” in Proc. 5th IEEE COSDEO
ing through transparent solar panels will allow Workshop, Conjunction IEEE PerCom, 2016, pp. 1–2.
energy-harvesting from the entire surface of vir- 11. 11.D. Ma, et al., “SolarGest: Ubiquitous and energy-free
tually any device. Leveraging the changes in the gesture recognition using solar cells,” in Proc. ACM
energy-harvested signal to sense the environment Mobicom, 2019, pp. 1–15.
as well as to detect the user interaction with the IoT 12. J. Park, J.-H. Kang, S. J. Kim, X. Liu, and M. L. Brong-
device will not only provide a ubiquitous, user-friendly ersma, “Dynamic reflection phase and polarization
way of interacting with IoT devices but will also indi- control in metasurfaces,” Nano Lett., vol. 17, no. 1,
rectly reduce the energy consumption and deploy- pp. 407–413, 2017.
ment cost of the IoT echo-system by providing 13. J. M. Jornet and I. F. Akyildiz, “Femtosecond-long
value-added services without any extra hardware. pulse-based modulation for terahertz band communi-
cation in nanonetworks,” IEEE Trans. Commun., vol. 62,
REFERENCES no. 5, pp. 1742–1754, May 2014.
1. B. Yang and K.-S. Yun, “Piezoelectric shell structures
as wearable energy harvesters for effective power
generation at low-frequency movement,” Sensors MOUSTAFA YOUSSEF is a Professor of computer and sys-
Actuators A, Phys., vol. 188, pp. 427–433, 2012. tems engineering with Alexandria University, Alexandria,
2. S. Kim, et al., “Ambient RF energy-harvesting technolo- Egypt. Contact him at [email protected].
gies for self-sustainable standalone wireless sensor
platforms,” Proc. IEEE, vol. 102, no. 11, pp. 1649–1666, MAHBUB HASSAN is a Professor of computer science and
Nov. 2014. engineering with the University of New South Wales, Syd-
3. M. B. Upama, et al., “High performance semi- ney NSW, Australia. Contact him at mahbub.hassan@unsw
transparent organic solar cells with 5% PCE using .edu.au.
non-patterned MoO3/Ag/MoO3 anode,” Current Appl.
Phys., vol. 17, no. 2, pp. 298–305, 2017.
4. P. Zhang and D. Ganesan, “Enabling bit-by-bit back-
scatter communication in severe energy harvesting
Virtual Computer
environments,” in Proc. NSDI, 2014, pp. 345–357. Society Conferences
5. M. Youssef, M. Mah, and A. Agrawala, “Challenges:
Access valuable learning
Device-free passive localization for wireless environ-
from any location.
ments,” in Proc. 13th Annu. Int. Conf. Mobile Comput.
Netw., Sep. 9–14, 2007.
6. A. E. Kosba, A. M Saeed, and M. Youssef, “Robust WLAN
device-free passive motion detection,” in Proc. IEEE
Wireless Commun. Netw. Conf., 2012, pp. 180–189.
7. H. Abdelnaser, R. Muhammad, I. Sabek, and M. Youssef,
“MonoPHY: Mono-stream-based device-free WLAN
localization via physical layer information,” in Proc. IEEE
Wireless Commun. Netw. Conf., 2013, pp. 4546–4551.
8. A. Eleryan, M. Elsabagh, and M. Youssef, “Synthetic
generation of radio maps for device-free passive View the virtual conference schedule at
localization,” in Proc. IEEE GLOBECOM, 2011, pp. 1–5. https://bit.ly/CSvirtual-conference
9. H. Abdelnasir, K. Harras, and M. Youssef, “UbiBreathe:
A ubiquitous non-invasive WiFi-based breathing
estimator,” in Proc. ACM Mobihoc, 2015, pp. 277–286.

www.computer.org/computingedge 45
 EDITOR: Erik P. DeBenedictis, [email protected]

This article originally

appeared in
DEPARTMENT: REBOOTING COMPUTING
vol. 52, no. 10, 2019

Powerball and
Quantum Supremacy
Erik P. DeBenedictis

In an attempt to communicate better with the nonexpert public, this article explains
recent developments in quantum supremacy in a form that is closer to home: lotteries.

S
ociety’s embrace of positional numbers about information to advance their chosen field with quan-
1,500 years ago may be a road map for today’s tum information.
quantum information revolution. Positional
numbers provide an exponential advantage over the A QUANTUM INFORMATION
unary representations found on prehistoric artifacts. MODEL
The unary representation of values in the millions uses Powerball, where there can be multiple winners of a
millions of symbols, but only log10(millions) = 7 in our single lottery, is an existence proof that a significant
current Hindu–Arabic positional notation. fraction of society can understand statistical trials
One current pre-quantum-supremacy demonstra- with correlations. Let’s see if this can be a starting
tion1 runs essentially the same problem on a one-chip, point for a user-level quantum information model.
n-qubit, n = 72 quantum computer and, for compari- Lottery tickets and qubits both have two phases
son, the world’s largest supercomputer, Oak Ridge’s in their lives. A ticket is characterized by pick numbers
Summit. The most straightforward supercomputer during the first phase of its life. The second phase
code would require an impossibly large 2n entry table, starts with the drawing, where the pick numbers
but researchers have found clever methods of reduc- become irrelevant, and all that matters is whether a
ing the memory requirements. The advantage may be ticket wins or loses. A ticket viewed in isolation either
exponential or nearly so. wins or loses at random, yet tickets with the same
Positional numbers may have been invented by a pick numbers will win or lose at the same time. Lottery
handful of mathematicians, but they are now taught to tickets actually have multiple pick numbers, so there
children. The ability of positional numbers to represent can be complex correlations between tickets that win
large amounts of money allowed economies to grow, and lose.
the ability to represent real numbers through decimal Qubits have a similar life cycle. Quantum comput-
fractions allowed the development of science, and ers actually create qubits in a standard form, which
automatic processing of numbers by computers led physicists call |0>. To create a custom ticket, or qubit,
to civilization-changing applications like the Internet. single-qubit quantum gates effectively modify the pick
If quantum information is to have the same down- numbers. For greater flexibility, two-qubit quantum
stream effect, we will need to teach children destined gates can mix up the pick numbers, although some
to be experts in other fields enough about quantum of these gates increase the size of the pick number
space multiplicatively, for example, a ticket with four
pick numbers and another with eight yielding two
DOI No. 10.1109/MC.2019.2930166 tickets with 32 each. The drawing, which physicists
Date of publication: 24 September 2019 call qubit measurement, yields a result that has a lot of

46 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 randomness but also has correlations due to common
SOME OF THE LEADING PRE-
pick numbers.
QUANTUM SUPREMACY WORK
We teach long division in elementary school, but
VIEWS A QUANTUM COMPUTER
the math required to invent the long division algorithm
AS A MACHINE FOR SIMULATING
is taught only in college and only to math majors. Let’s
LOTTERIES, OR QUBITS.
see if we can explain quantum supremacy and Shor’s
algorithm well enough for somebody to appreciate
and apply the result but not necessarily well enough to
understand how it was invented or to improve it. = 272 = 4.7 × 1021 entries, which is too big even for the
world’s largest supercomputer. Filling the table would
QUANTUM SUPREMACY IN DETAIL be even more challenging. However, some lotteries,
Some of the leading pre-quantum-supremacy work1,2 or quantum gate circuits, will have a much smaller
views a quantum computer as a machine for simulat- number of highly probable winning patterns. Say there
ing lotteries, or qubits. The quantum computer is con- were, for example, a dozen patterns with 1% probabil-
figured with a random qubit gate circuit. Each run ity. If you run a lottery 10,000 times, each of these pat-
starts with n lottery tickets, or qubits, initialized in terns will occur about 100 times. So, 10,000 runs would
standard yet uncorrelated states. Figure 1 shows this identify the highly probable patterns and compute
with simple sequential pick numbers. The tickets are their probability to about one decimal place accuracy.
passed through the series of quantum gates and then Let’s look at specific numbers for one such dem-
measured. Today’s qubit technology is called Noisy onstration.1 A 72-qubit quantum computer is run on
Intermediate-Scale Quantum, which currently has a a random quantum circuit. Decoherence times are
limit of n ≤ 72 and a maximum of around 40 levels of under 1 ms, so an actual run should be about that long.
quantum gates before errors dominate and the result The experimenters then computed the correct
is not useful. probabilities for some of the more probable states on
If you run the quantum computer repeatedly on the
same gate configuration, you will get a sequence of
n-bit patterns representing the winning tickets across Create n Quantum Gates Measure
multiple lotteries. To test the quantum computer, you Simple Generate More n Qubits
Qubits Pick Numbers
might think of running it repeatedly and computing the
probability of each of the 2n patterns. If you were more 1, 2 1, 2, 3, 4 Win
ambitious, you could also compute the same probabili-
3, 4 1, 2, 3, 4 Win
ties on a classical computer. The quantum and classi-
cal probabilities won’t match exactly because today’s 5, 6 5, 8 Lose
qubits are notoriously noisy, but comparing the prob-
7, 8 7, 6 Win
abilities would characterize the quantum computer’s
noise and could be very useful.
There are several problems. As mentioned previ- FIGURE 1. A user-level illustration of a quantum computer.
ously, the table of probabilities for one of today’s Draw: 1, 2, 3, 4, 6, 7. The results (top to bottom) are { Win, Win,
state-of-the-art quantum computers would have 2n Lose, Win } = 11012 = 13.

www.computer.org/computingedge 47
REBOOTING COMPUTING

Create n Quantum Gates Measure

Simple for At Mod N and n Qubits yield factors, and, of course, the hints won’t lead to a
Qubits QFT valid factor if the quantum computer makes a mistake.
n Factoring N requires n = 2 × log2 N lottery tickets,
A At Mod N Hint
Inverse or qubits, each representing a bit of two n-bit numbers
n QFT
t t A and t. Temporary qubits are also required, and there
could be quite a few. The qubits pass individually
through what physicists call Hadamard gates, setting
FIGURE 2. An illustration of Shor’s algorithm. Additional the pick numbers so each has a 50% chance of win-
temporary qubits will be required. QFT: quantum Fourier ning, with all qubits uncorrelated.
transform. The lottery tickets, or qubits, are run through a
circuit that has the same structure as a (reversible)
Boolean logic circuit that computes At mod N. Today’s
Summit. The majority of the research was the develop- logic designers should be able to create the circuit
ment of the (classical) supercomputer code to repro- using O(n3) gates configured into ripple-carry Boolean
duce the results of the 72-qubit quantum computer. To adders without having to reference and textbook. Fast
reduce the memory requirements, the authors outline multiplier circuits3 would be more efficient for large
techniques that can compute an adequate answer n. To provide the status on the state of factoring, the
based on time–space tradeoffs and exploiting the RSA challenge4 of factoring a 768-bit number was
fact that a short decoherence time limits the depth completed in December 2009. This puts the number of
of the circuit and, hence, its complexity. Yet runtimes gates for the At mod N circuit far out of range of today’s
on Summit are still in the hours. The problem is highly quantum computer hardware.
parallel, so runtime isn’t the right measure. But Sum- I’ll summarize the rest of Shor’s factoring algorithm
mit burns about 104 times the energy as the quantum just to show the connection to quantum supremacy
computer doing the same thing. experiments. If N, f(t) = At mod N will be zero for all
Did the researchers1 demonstrate quantum integral values of t but an essentially random function
supremacy? It seems that a one-chip, 72-qubit quan- on the unit circle otherwise. There is a well-known QFT
tum computer can do things that swamp the world’s technique in quantum computing, the inverse of which
largest supercomputer. But is this a useful application essentially computes ei f(t), with some constants. If A
or a contrived benchmark? is a factor of N, this will be the fast Fourier transform
(FFT) of a constant, which is a delta function at zero
SHOR’S ALGORITHM frequency. If not, it will be the FFT of white noise, which
I suggest that Shor’s famous factoring algorithm (Fig- is a constant function. Since the drawing probability is
ure 2) can be appreciated as an application of a lot- the amplitude squared, there will be a vastly increased
tery machine, although the simplified model would probability of the drawing, or qubit measurement,
not have allowed the invention of the system in the yielding a win–lose pattern corresponding to a value
first place nor would it be possible to improve the of A that is a factor of N. This value becomes the hint.
algorithm. This could be useful for the same reasons However, it’s still a lottery, so a run of the quantum
we teach children long division. The lottery machine computer won’t always lead to a factor.
description also shows why Shor’s factoring algorithm,
a practical problem, is too complex for today’s quan-
tum computers.
The quantum hardware for Shor’s algorithm does
W hen I got involved in quantum computing
about 15 years ago, the people in the field
were physicists and algorithms experts. There was
not actually factor numbers but instead gives a hint to no possibility of running an interesting algorithm on
a classical computer about where to look for a factor. the hardware of that era. So, algorithms were devel-
The classical computer does a few arithmetic opera- oped for applications requiring millions of qubits at
tions based on the hint, yielding a candidate factor, the same time that physicists struggled to make one
which is then checked by division. Some hints do not and two qubits and the corresponding gates. Now, we

48 ComputingEdge July 2020

 REBOOTING COMPUTING

have improved hardware capable of running down- REFERENCES

scaled algorithms and yielding a result that, with a bit 1. B. Villalonga et al., Establishing the quantum
of faith, looks like it might work. supremacy Frontier with a 281 Pflop/s simulation. 2019.
Progress will continue, provided high public inter- [Online]. Available: arXiv:1905.00444
est keeps governments funding R&D and quantum 2. A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and
R&D companies’ stock market valuations remain high J. M. Gambetta, Validating quantum computers using
so they have money. For this to happen, the quantum randomized model circuits. 2018. [Online]. Available:
technical community needs a support base that can arXiv:1811.12926
appreciate the results and the potential of quantum 3. R. Rines and I. Chuang, High performance quantum
computers. However, a significant part of the sup- modular multipliers. 2018. [Online]. Available:
port base will not require the understanding of an arXiv:1801.01081
engineer or scientist but need to understand quantum 4. Wikipedia, RSA Factoring Challenge. Accessed on: Aug.
concepts just enough to invest and vote. We just need 19, 2019. [Online]. Available: https://en.wikipedia.org
something more approachable than math and physics /wiki/RSA_Factoring_Challenge
lingo and jokes like 1/√2 |dead> + 1/√2 |alive>.
If this approach catches on, we’ll need to make a
quantum-specific lottery and name it something other ERIK P. DEBENEDICTIS a volunteer supporting the IEEE
than Powerball. The name Powerball implies a specific Computer Society, IEEE Council on Superconductivity, IEEE
set of rules and a drawing method that would not cre- Rebooting Computing, and the IEEE Quantum Initiative. Con-
ate quantum speedup. tact him at [email protected].

IEEE Intelligent Systems solicits nominations for its “Artificial

Intelligence’s 10 to Watch” in 2020. Those selected will appear in
a special section of the magazine.

The 2020 nominees will be those who received their PhDs

in the year 2014 or later and have already made significant
contributions to one or more areas of AI.

A selection committee, chaired by a member of the IEEE

Intelligent Systems Editorial Board, consists of prominent
researchers from different subfields of AI and different
geographic regions of the world.

Nominations from all areas AI are welcome.

AI’S 10 TO WATCH For a list of past winners, and examples of the up-and-coming
researchers we’re looking for, please see 2018 winners here:
CALL FOR bit.ly/2TvG22y

NOMINATIONS
Nominations should include the following:

1. Nominee’s CV and publication list

2. Two reference letters
3. Statement summarizing the candidate’s achievements
in AI (200 words maximum)

Nomination deadline: 10 July 2020

Send nominations to [email protected]

www.computer.org/computingedge 49
 EDITOR: Erik P. DeBenedictis, Zettaflops, LLC, [email protected]

This article originally

appeared in
DEPARTMENT: REBOOTING COMPUTING
vol. 53, no. 2, 2020

Beyond Quantum Supremacy

Erik P. DeBenedictis, Zettaflops, LLC

Demonstrations of quantum supremacy have become convincing enough that it’s

time to think seriously about what comes next.

T
he media tantalizes the public by having single digital multiplier array. In this quantum architecture,
quantum computer chips outperform the the quantum gates naturally perform an entangling
world’s largest computer,1 bringing publicity quantum operation on nearly every cycle, making the
to the participating companies. The demonstrations architecture hard to simulate on a supercomputer
are driven by real progress on analog, quantum, and and ideal for demonstrating quantum supremacy.
scalable logic families and basic automata, that is,
state machines.2 The effort also reveals an emerging BASICS OF QUBIT SIMULATORS
quantum computer architecture flexible enough to Let’s take a look at simulating a quantum computer on
build systems for targeted applications. a supercomputer, because this is the central issue in
Quantum supremacy is the demonstration of a assessing the advantage of the quantum approach.
quantum computer performing a computation that is This discussion will focus on the full Schrödinger-state
infeasible or impossible for a regular computer. As this simulation, although there is a continuum of other
article is being written, Google has a quantum com- methods that trade memory for computation. There
puter chip that indisputably outperforms the world’s are also fundamentally faster simulation methods that
largest supercomputer, Summit, which is located at rely on specific properties of the quantum network,
the Oak Ridge National Laboratory, Tennessee.1 There such as all of the gates being classical. The latter cate-
is just one issue, albeit on just one program. The quan- gory is not helpful here.
tum computer chip runs for 200 s on a problem that Figure 1(a) shows a data structure for a group of k
Google estimates would take Summit 10,000 years— = 3 entangled qubits, where entanglement is the quan-
establishing quantum supremacy. However, IBM says tum property that yields the special computational
it knows how to program Summit to perform the capabilities. Each qubit will have the value of zero or
simulation in 2.5 days—refuting the claim of quantum one when read out. However, entanglement means the
supremacy. 3 In my opinion, both the 10,000-year and entire group will take on each specific pattern of k = 3
2.5-day figures are good enough. zeros and ones with a specific probability. Figure 1(a)
Quantum supremacy requires advances in several represents the group by an array, a, of 2k = 8 rows, each
areas at once. Today’s transmon qubits lose their state row representing the pattern of zeros and ones via the
in 10 s of μs, meaning that the quantum program must binary representation of its index.
run to completion in a few dozen gate delays or cycles. The value in each row of the array includes the
Google’s current chip has 53 qubits, so the “applica- probability that measuring the group of qubits returns
tion program” is essentially a logic diagram of roughly the pattern of zeros and ones in the binary representa-
1,000 quantum gates. Surprisingly, a cogent quantum tion of the index. For a quantum simulator to duplicate
computer architecture is emerging that looks like a unique quantum-information effects, it needs a second
systolic array4 or a solidly packed array of gates like a degree of freedom in each array entry, represented as .
In lieu of a percentage and , a typical quantum simula-
tor uses complex numbers to represent both degrees
DOI No. 10.1109/MC.2019.2958446 of freedom, but we don’t need more detail on this second
Date of current version: 12 February 2020 degree of freedom for this article. There is no reason to

50 July 2020 Published by the IEEE Computer Society  2469-7087/20 © 2020 IEEE
 store the indices, so the state of k
qubits can be represented by an
array of 2k complex numbers.
Let’s look at a two-input quan-
tum gate, say between qubits qm
and qn, m = 0, and n = 2 in Figure 1(a).
Pick a row, perhaps row 2, and then
flip the 2m bit of the index, the 2n
bit, and both bits (highlighted in
yellow). This will yield the group of
four rows 2, 3, 6, and 7, as shown.
Treat the four complex numbers in
the array as a vector, multiply the
vector by a 4 × 4 complex matrix FIGURE 1. (a) A simulation array for three qubits containing 23 = 8 rows. Each entry
that defines the gate, and store the can be considered to have a probability and phase or complex number. (b) A single
result in place. Now repeat until qubit replicated eight times. (c) A four-qubit group, with c resulting from the combi-
every qubit has been updated once. nation of a and b.
In Figure 1(a), the two’s bit is
the only other one in the index,
but Google’s chip has 53 qubits, so there would be 251 secondary storage to increase Summit’s effective
15
= 2 × 10 additional updates. Thinking about Figure memory size. This required some additional gymnas-
1(a) reveals that a two-qubit operation involves a lot tics to circumvent the speed penalty for secondary
of memory and memory bandwidth, with the access storage. Essentially, the memory-access patterns in
pattern ranging from local to global depending on the the previous discussions depend on how qubits are
location of the qubits in the index field. mapped to bits in the array indices. Programmers
To simulate a two-qubit gate on qubits in different frequently manage the stride of vector access and
groups, the groups have to be combined. Figure 2(b) working-set size to improve efficiency. In this case,
shows a second array, b, with just one qubit, or two rows memory is essentially a cache for the disk, and careful
in an array containing its state. To create the state of all attention to the qubit-to-index bit mapping can affect
of the qubits combined, each row in array a is replaced the amount of input–output to the secondary storage.
with the entire array b, concatenating the bits of the
index and multiplying the probabilities. This leads to an THE GAME OF “SWAMP
array whose size is the product of the sizes of the initial THE SIMULATOR”
arrays, as illustrated in Figure 1(c). The largest quantum To demonstrate quantum supremacy with the least
computer chips demonstrated today have 49, 53, 54, hardware, the pure-play strategy is to design the
and 72 qubits, which would require arrays with 249, quantum circuit so the simulator’s data structures
53 54 72
2 , 2 , and 2 entries, which is bigger than the main grow exponentially on every step, quickly swamping
memory on today’s largest supercomputers. the simulator. Start with every qubit in its own group
However, IBM’s rebuttal to the quantum suprem- so that the simulator’s arrays would have 21 = 2 rows.
acy claim3 analyzed the use of solid-state drive Apply a two-qubit gate to every even-numbered qubit

www.computer.org/computingedge 51
REBOOTING COMPUTING

FIGURE 2.The quantum 4-bit ripple-carry adder. The circuit includes a ripple-carry chain on the left, but the circuit must uncom-
pute the carry to maintain reversibility. The inverted “V” structure has a gap in the middle as the carry propagates, hampering the
quantum speed-up. MAJ: majority; UMA: unmajority-add. (This image was derived from Cuccaro et al.5)

and the odd-numbered one that follows it, halving the demonstration will have a hard time doing something
number of groups but increasing their size to 22 = 4 useful. Essentially, computer architecture principles
rows. Then apply two-qubit gates to qubits in differ- require idle time between gate operations, yet leaving
ent groups, yielding four-qubit groups and squaring quantum gates idle will hamper the game of swamp
the array size to 24 = 16 rows. Continuing this would the simulator, so the quantum computer chip will
lead to simulator arrays of 28, 216, and 232 rows. The have a harder time showing a performance advan-
next combination would require arrays with 264 rows, tage over supercomputers. Many proposed quantum
which is too big to be practical. algorithms use quantum numerical data types, such
Unfortunately, the gate-connection pattern for as integers, where the bits have been replaced by
the pure-play strategy is similar to a fast Fourier trans- qubits. The algorithms make use of variables that are
form, requiring long wires and wires that cross over essentially a superposition of numeric values.
each other. Physical-science issues such as noise, the Readers familiar with a classical ripple-carry
loss from long wires, and crosstalk have been analyzed adder know that it consists of a series of stages,
theoretically, and the theory has been tested against each with a carry-in, c in; one bit of the a and b
results from real chips.2 The architectural remedy is a numeric inputs; a carry-out, c out ; and an s output
quantum chip where the qubit interconnection pat- with a bit of the sum. The quantum version in Figure 2
tern has just enough connectivity to enable complete has c in = c k on the lower left of each majority (MAJ)
entanglement before the qubits lose state. The 1D and block and cout = ck+1 on its upper right. The initial c 0
2D qubit arrays with nearest-neighbor interconnections = 0 means no carry, and the vertically propagating
would be scalable without long wires and crossovers, carry chain is essentially the same as a classical
with 1D being simpler. However, the graph diameter of ripple-carry adder. The additional outputs of MAJ are
a 1D qubit array is too large, and today’s qubits would ak ck and ak bk.
lose state before entanglement could be established Quantum circuits must be reversible, so the carry
between all of them. So, the current designs are 2D. signal, c4, must be decomputed back to zero, which
is accomplished by the top-to-bottom sequence of
QUANTUM SUPREMACY unmajority-add (UMA) blocks. The lower right of each
CREATES A BARRIER TO DOING UMA block outputs the reconstructed cin = ck signal
USEFUL THINGS from cout = ck+1, ak ck, and ak bk. The UMA block
The quantum ripple-carry adder in Figure 25 illus- also computes sk, a bit of the sum, and provides a
trates why a chip designed for a quantum-supremacy copy of ak. The copy of ak is also needed to make the

52 ComputingEdge July 2020

 REBOOTING COMPUTING

overall adder reversible. If the circuit is mirrored or run that the issue is really the architecture of the gates,
backward, it will decompute quantum integers s and a not chemistry.
back to the original a and b, effectively implementing
subtraction.
The problem is that the inverted “V” structure in
Figure 2 has a lot of white space, illustrated in red,
A s quantum volume increases further, the range of
problems that demonstrate quantum supremacy
will grow. Eventually, quantum computers will just be
between the prongs of the V. This means that qubits computers and face many of the same issues that cur-
(such as a0 b0) will be idle for many cycles, waiting rent computers face, like word and bus width, appli-
for the carry signal to propagate to the most signifi- cation programming interfaces, error logging, and so
cant bit and back again. forth. I’m suggesting that the time is right for some
Logic diagrams are normally sheets of paper with computer scientists and engineers to consider these
boxes on them representing the resources used in a issues for quantum computation.
design, but in the current state of quantum comput- I’m not suggesting that other paths forward are
ing, white space on the paper counts as a resource as unimportant. We need longer coherence times for
well. This is because white space represents the pas- qubits, quantum error correction, the random inter-
sage of time where the qubits will decohere but the connect, control electronics, and the ability to adapt
simulator data structures will not grow, reducing the to problems as needed. Architecture goes hand in
relative advantage of the quantum computer. Com- hand with these issues to move this area of computa-
puter architects will recognize that logic diagrams for tion forward.
memories, multiplier arrays, systolic arrays, and so on
can be drawn without white space, but many other REFERENCES
functions require it, and there’s not much that can be 1. F. Arute et al., “Quantum supremacy using a program-
done about it. mable superconducting processor,” Nature, vol. 574,
One possible next step is to consider computa- no. 7779, pp. 505–510, 2019.
tional problems that are addressed by something 2. A. Erhard et al., “Characterizing large-scale quantum
short of a solid pack of quantum gates, which might computers via cycle benchmarking,” Nat Commun.,
look like a logic diagram with less than the normal vol. 10, p. 5347, Nov. 2019.
amount of white space. Computer scientists and 3. E. Pednault, J. A. Gunnels, G. Nannicini, L. Horesh, and
engineers understand that many important prob- R. Wisnieff, Leveraging secondary storage to simulate
lems grow sparser at scale, but this is not universally deep 54-qubit sycamore circuits. 2019. [Online]. Avail-
true, and quantum computers have not reached the able: arXiv:1910.09534
humongous scale of classical computers. 4. S. Y. Kung, VLSI Array Processors. Englewood Cliffs, NJ:
The following may be a bit of an oversimplifica- Prentice Hall, 1988, p. 685.
tion, but there is a proposed quantum computer 5. S. A. Cuccaro, T. G. Draper, S. A. Kutin, and D. Petrie
benchmark called quantum volume,6 which essen- Moulton, A new quantum ripple-carry addition circuit.
tially quantifies the maximum number of gate opera- 2004. [Online]. Available: arXiv:quant-ph/0410184
tions that a quantum computer will execute. Google’s 6. A. W. Cross, L. Bishop, S. Sheldon, P. Nation, and J.
recent work shows that 53 qubits are enough for Gambetta, “Validating quantum computers using
quantum supremacy—or at least to create a lot of randomized model circuits,” Phys. Rev. A, vol. 100, no. 3,
discussion about it. However, it assumes that all 53 p. 032328, 2019.
gates work all of the time. If there were twice as many
qubits, say 10 6, or double the runtime, the system
ought to be able to demonstrate quantum supremacy ERIK P. DEBENEDICTIS is a principal with Zettaflops, LLC.
for algorithms that have 50% white space in their He is the editor in chief of IEEE Transactions on Quantum
schematic diagrams. The conventional wisdom is Engineering and a volunteer supporting the IEEE Computer
that chemical simulations will be the applications Society, Council on Superconductivity, and Quantum Initia-
area that fits this next stage best, but I’m suggesting tive. Contact him at [email protected].

www.computer.org/computingedge 53
Evolving Career
Opportunities
Need Your Skills
Explore new options—upload your resume today

Changes in the marketplace shift demands for vital skills and talent. The
IEEE Computer Society Jobs Board is a valuable resource tool to keep job
seekers up to date on the dynamic career opportunities offered by employers.
Take advantage of these special resources for job seekers:

JOB ALERTS TEMPLATES WEBINARS

CAREER RESUMES VIEWED

ADVICE BY TOP EMPLOYERS

No matter what your career level, the IEEE Computer Society Jobs Board
keeps you connected to workplace trends and exciting career prospects.

www.computer.org/jobs
 05.16 04.16 10.16 04.20

Computer
Computer

Computer
Computer
EMERGENCY

APRIL 2016
MAY 2016

OCTOBER 2016

APRIL 2020
RESPONSE

ENERGY-EFFICIENT COMPUTING
EMERGENCY RESPONSE

BIG DATA

COMPLEXITY VERSUS TRUST

IOT STANDARDS, P. 87
LOVE, P. 88 FASTER PATENTING, P. 10
CYBERSECURITY’S FORMAL
ENT TOOLS, P. 95 METHODISTS, P. 102 SKIN IN THE UI GAME, P. 83

Volume 49 Number 4

Volume 49 Number 10
Volume 49 Number 5

www.computer.org/computer www.computer.org/computer
www.computer.org/computer www.computer.org/computer

Volume 53 Number 4
01.20
02.20 vol. 53 no. 4

Computer

Computer
12.19 03.20

Computer
JANUARY 2020
Computer

FEBRUARY 2020
RS OF
ORKING

MARCH 2020
DECEMBER 2019

Technology
Predictions

DIGITAL HEALTH: E-COACHING AND REMOTE MONITORING

DIGITAL HEALTH:
E-COACHING

DATA ANALYSIS AND CYBERPHYSICAL SYSTEMS

AND REMOTE
TECHNOLOGY PREDICTIONS

OUTLOOK

A Half-Century MONITORING
of the Arpanet 14

Outlook DATA ANALY

CYBERP
S

Volume 53 Number 2
Volume 52 Number 12

Volume 53 Number 1

Volume 53 Number 3
vol. 52 no. 10 www.computer.org/computer vol. 52 no. 12 www.computer.org/computer vol. 53 no. 2 www.computer.org/computer
vol. 53 no. 1 www.computer.org/computer

vol. 53 no. 3

CALL FOR SPECIAL ISSUE

PROPOSALS
Computer solicits special issue proposals from lead experts. Proposed themes/issues should address
timely, emerging topics that will be of broad interest to Computer’s readership. Special issues are an
important component of Computer, as they deliver essential research insights and well-developed
perspectives on new and established technologies and computing strategies.

We encourage submissions of high-quality proposals for the 2021 editorial calendar. Of particular
interest are proposals centered on:

• offsite educational and business continuity • technology’s role in disrupted supply chains,
technology challenges, • misinformation and disinformation (fake
• privacy related to personal location tracking information—malicious or non-malicious), and
and surveillance (digital and physical), • cyberwarfare/cyberterrorism
• artificial intelligence and machine learning,

Proposal guidelines are available at:

www.computer.org/csdl/magazine/co/write-for-us/15911

Deadline for proposal submission: 15 September 2020

 Conference Calendar

I EEE Computer Society conferences are valuable forums for learning on broad and dynamically shifting topics
from within the computing profession. With over 200 conferences featuring leading experts and thought lead-
ers, we have an event that is right for you. Questions? Contact [email protected].

AUGUST 21 September 17 October

1 August • ASE (IEEE/ACM Int’l Conf. on • MICRO (IEEE/ACM Int’l Sym-
• JCDL (ACM/IEEE Joint Conf. Automated Sof tware Eng.), posium on Microarchitecture),
on Digital Libraries), virtual Melbourne, Australia Athens, Greece
14 August 24 September 18 October
• SmartIoT (IEEE Int’l Conf. on • BigMM (IEEE Int’l Conf. on Mul- • ICCD (IEEE Int’l Conf. on Com-
Smart Internet of Things), Bei- timedia Big Data), virtual puter Design), Hartford, USA
jing, China 27 September • MODELS (ACM/IEEE Int’l Conf.
16 August • ICSME (IEEE Int’l Conf. on Soft- on Model-Driven Eng. Lan-
• Hot Chips, Palo Alto, USA ware Maintenance and Evolu- guages and Systems), Mon-
17 August tion), Adelaide, Australia treal, Canada
• ACSOS (IEEE Int’l Conf. on 28 September 19 October
Autonomic Computing and • SecDev (IEEE Secure Develop- • DFT (IEEE Int’l Symposium on
Self- Or ganizing S ys tems), ment), virtual Defect and Fault Tolerance
Washington, DC, USA in VLSI and Nanotechnology
19 August OCTOBER Systems), Frascati, Italy
• RTCSA (IEEE Int’l Conf. on 3 October 21 October
Embedded and Real -T ime • PACT (Int’l Conf. on Parallel • FIE (IEEE Frontiers in Educa-
Computing Systems and App- Architectures and Compila- tion Conf.), Uppsala, Sweden
lications), virtual tion Techniques), Atlanta, USA 25 October
31 August 5 October • VIS (IEEE Visualization Conf.),
• RE (IEEE Int’l Requirements • EDOC (IEEE Int’l Enterprise Salt Lake City, USA
Eng. Conf.), Zurich, Switzerland Distributed Object Comput-
ing Conf.), Eindhoven, the NOVEMBER
SEPTEMBER Netherlands 6 November
7 September 12 October • CCEM (IEEE Int’l Conf. on
• EuroS&P (IEEE European Sym- • ISSRE (IEEE Int’l Symposium Cloud Computing in Emerging
posium on Security & Privacy), on Software Reliability Eng.), Markets), Bengaluru, India
virtual virtual • SmartCloud (IEEE Int’l Conf.
14 September 16 October on Smart Cloud), Washington,
• CLUSTER (IEEE Int’l Conf. on • ICEBE (IEEE Int’l Conf. on DC, USA
Cluster Computing), Kobe, e-Business Eng.), Guangzhou, 9 November
Japan China • ICTAI (IEEE Int’l Conf. on Tools

56 July 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE
 with Artificial Intelligence), Hardware-Oriented Security Guangzhou, China
virtual and Trust), San Jose, USA • EUC (IEEE Int’l Conf. on Embed-
• IRC (IEEE Int’l Conf. on Robotic 7 December ded and Ubiquitous Comput-
Computing), Taichung, Taiwan • BDCAT (IEEE/ACM Int’l Conf. ing), Guangzhou, China
• ISMVL (IEEE Int’l Symposium on Big Data Computing, Appli-
on Multiple-Valued Logic), cations and Technologies),
Miyazaki, Japan Leicester, UK 2021
11 November • UCC (IEEE/ACM Int’l Conf. on
• SEC (IEEE/ACM Symposium on Utility and Cloud Computing), JANUARY
Edge Computing), San Jose, Leicester, UK 5 January
USA 9 December • WACV (IEEE Winter Conf. on
15 November • CC (IEEE Int’l Conf. on Conver- Applications of Computer
• SC, Atlanta, USA sational Computing), Irvine, Vision), Waikoloa, USA
16 November USA 17 January
• FOCS (IEEE Symposium on • AIKE (IEEE Int’l Conf. on Arti- • BigComp (IEEE Int’l Conf. on
Foundations of Computer Sci- ficial Intelligence and Knowl- Big Data and Smart Comput-
ence), Durham, USA edge Eng.), Irvine, USA ing), Bangkok, Thailand
• LCN (IEEE Conf. on Local Com- 10 December 27 January
puter Net works), S ydney, • BigData (IEEE Int’l Conf. on Big • IC SC (IEEE Int ’ l Conf. on
Australia Data), virtual Semantic Computing), Laguna
29 November 14 December Hills, USA
• ICDCS (IEEE Int’l Conf. on Dis- • CloudCom (IEEE Int’l Conf. on
tributed Computing Systems), Cloud Computing Technol-
Singapore ogy and Science), Bangkok,
30 November Thailand
• ICHI (IEEE Int’l Conf. on Health- • HPCC (IEEE Int’l Conf. on High
care Informatics), Oldenburg, Performance Computing and
Germany Communications), Cuvu, Fiji
16 December
DECEMBER • BIBM (IEEE Int’l Conf. on Bio- Learn more
2 December informatics and Biomedicine),
about IEEE
• CSDE (IEEE Asia-Pacific Conf. virtual
on Computer Science and Data • HiPC (IEEE Int’l Conf. on High- Computer
Eng.), Gold Coast, Australia Pe r f o r m a n c e C o m p u t in g , Society
• ISM (IEEE Int’l Symposium on
Multimedia), Naples, Italy
Data, and Analytics), virtual
29 December
conferences
6 December • BigDataSE (IEEE Int’l Conf. on computer.org/conferences
• HOST (IEEE Int’l Symposium on Big Data Science and Eng.),
Get Published in the New IEEE Open
Journal of the Computer Society

Submit a paper today to the

premier new open access
journal in computing
and information technology.

Your research will benefit from

the IEEE marketing launch and
5 million unique monthly users
of the IEEE Xplore® Digital Library.
Plus, this journal is fully open
and compliant with funder
mandates, including Plan S.

Submit your paper today!

Visit www.computer.org/oj to learn more.