How Advanced Cloud Technologies can Impact and

Change HPC Environments for Simulation

Marco Mancini Giovanni Aloisio
Advanced Scientific Computing Division (ASC) Supercomputing Center (SCC)
Euro-Mediterranean Center on Climate Change (CMCC) Euro-Mediterranean Center on Climate Change (CMCC)
Lecce, Italy Lecce, Italy
[email protected] [email protected]

EXTENDED ABSTRACT
In the last years, most enterprises and IT organizations have utilization; also, they require a significant amount administrator
adopted virtualization and cloud computing solutions to time to install, monitor and manage. Moreover, the demand for
achieve features such as flexibility, elasticity, fault tolerance, different HPC resources can be fluctuating in time, resulting in
high availability and reliability for their computational, storage periods of underutilization or overload. A dynamic, shared
and networking resource infrastructures. Moreover, recent cloud environment can dramatically improve service levels and
advances in Linux containers [1] and the emergence of resource utilization and eliminate costly, inflexible and static
technologies as Docker [2] are revolutionizing the way of ad-hoc clusters.
developing and deploying web and large scale distributed
applications. Until today, one obstacle to the adoption of virtualization in
HPC and scientific computing has been that traditional
However, those technologies are still not fully explored and hypervisor-based virtualization was not considered as a viable
exploited in the Scientific and High Performance Computing option for HPC due to the overhead costs in compute, I/O and
communities. Indeed, today’s HPC clusters, consisting of network devices, and the subsequent reduction in parallel
compute nodes connected by a high-performance application performance.
interconnection, are typically “static” infrastructures operated
and used by a single organization and typically shared among Recently, with the rise of container-based virtualization
implementations, it is possible to obtain a very low overhead
many research scientists.
leading to near native performance [3], since containers are a
“Traditional” management of HPC clusters introduces lightweight virtualization method for running multiple isolated
several issues. With the proliferation of multicore technology, a Linux systems and offer the flexibility of cloud-type systems
single multicore node can be shared by many different users, in coupled with the performance of bare-metal systems.
order to maximize the overall utilization of the system.
However, without an isolation layer, there are no guarantees In this talk, background on Linux Containers and Docker
will be introduced, showing how container and cloud
that applications from different users will work together in the
same node; indeed, a job might unexpectedly use more computing have the potential to dramatically impact scientific
computing. It will be also shown how virtualization and cloud
resources than it asked for and interfere with other workload
running on the host. In this scenario, the use of virtualization technologies can be exploited to provide researchers with a
reliable, flexible and customized environment for the
and in particular the use of containers (through control groups
and namespaces) could improve the resource sharing allowing development, benchmarking and running scientific and HPC
applications and simulations.
for multiple isolated user-space instances.
Furthermore, traditional HPC clusters run a single, standard In particular, the talk will focus on the main issues that
must be addressed to fully exploit cloud technologies for HPC
OS and the same software stack across all nodes. However,
users may have different requirements in terms of software environments for simulations such as analyze the impact of
container virtualization on HPC application scalability, deploy
packages and configurations. This usage scenario makes it
difficult to deploy newly developed or experimental and integrate of Docker tools within HPC batch schedulers [4],
redesign scientific and HPC applications through Docker
technologies in traditional cluster environments. Hence, the use
of a virtualization layer could facilitate the creation and packaging and runtime environments for developing scientific
applications.
maintenance of multiple environments customized according to
the users’ needs. A “HPC in cloud” prototype is under development at
To meet these different requirements many technical CMCC [5,6]; it will provide advanced compute, storage,
networking and security services allowing self-provisioning of
computing environments create ad hoc clusters for
heterogeneous and customized workloads. These environments elastic and customized clusters based on container
virtualization and cloud management technologies. This
are often inefficient due to their lack of fully hardware

978-1-4673-7813-0/15/$31.00 ©2015 IEEE 667

Authorized licensed use limited to: Nirma University Institute of Technology. Downloaded on October 03,2024 at 07:24:53 UTC from IEEE Xplore. Restrictions apply.
 solution will allow to (I) increase the HPC resources utilization CMCC Strategic Council and Director of the CMCC
with server consolidation and reduced power and cooling costs Supercomputing Center. His expertise concerns high
for heterogeneous workloads, (II) provide quality of service performance computing, grid & cloud computing and
and high availability, (III) dynamic resource management for distributed data management. From 1986 to 1990 he
HPC services and simulations (IV) improve security through contributed to the Caltech Concurrent Computation Program
standardized and automated configurations of compute, storage (C3P) led by Geoffrey C. Fox at the California Institute of
and network assets. Moreover, hybrid solutions will be also Technology, investigating the efficiency of the Hypercube
implemented by connecting CMCC private cloud to public architecture in Real-Time SAR data processing. From 1991 to
cloud providers in order to respond to peak load demands, to
1994, he collaborated with the Center for Advanced
planned maintenance and unplanned outages and guarantee
Computing Research (CACR) led by Paul Messina at the
business continuity for HPC simulations.
California Institute of Technology on High Performance
Moreover, a scientific application development Distributed Computing projects. He also collaborated with
environment, based on Docker tools, is also under development Carl Kesselman of the Information Sciences Institute on the
at CMCC to get benefits such as resource guarantee and use of Computational Grids for the management of large
performance isolation, application lifecycle management, collections of scientific data. He has been a co-founder of the
consistency, repeatability [7], bare-metal performance and fast European Grid Forum (Egrid), which then merged into the
management operations. Global Grid Forum (GGF), now Open Grid Forum (OGF). He
We believe that HPC systems will soon move from typical has been involved into several EU grid projects such as
static ad-hoc clusters to dynamic, shared cloud resources, GridLab, EGEE, IS-ENES1. He has been responsible for
automatically exploiting elasticity to meet workload demands, ENES (European Network for Earth System Modelling) in the
to optimize utilization and maximize throughput, and to EU-FP7 EESI (European Exascale Software Initiative) project,
increase reliability of HPC simulations. Moreover by chairing the Working Group on Weather, Climate and solid
exploiting emergent technologies as Docker, scientific Earth Sciences (WCES). He has also contributed to the IESP
productivity through their life-cycle research activities will be (International Exascale Software Project) exascale roadmap.
improved in such a way that development and testing, He has been the chair of the European panel of experts on
benchmarking and scalability, and experiments can be executed WCES that has contributed to the PRACE strategic document
always in the same isolated and customized environment used "The Scientific Case for HPC in Europe 2015-2020".
for development or adopted by their scientific community. Presently, he is coordinating CMCC activities into several EU
FP7 projects such as EUBrazilCC, IS-ENES2, CLIP-C and the
BIOGRAPHIES G8 ExArch. As CMCC, he is also the coordinator of the
MARCO MANCINI is a Senior Scientist at the Advanced OFIDIA (Operational FIre Danger preventIon plAtform)
Scientific Computing (ASC) Division of the Euro- project, in the context of the European Territorial Cooperation
Mediterranean Center on Climate Change (CMCC), leading Program Greece-Italy 2007-2013. He is responsible for the
the research activities on cloud computing for HPC and big University of Salento (as PRACE Third Party) in the EU-FP7
data analytics environments. His expertise concerns cloud EESI2 project, chairing the WCES Working Group. He is a
computing, Linux containers, high performance computing, member of the ENES HPC Task Force. He is the author of
big data analytics. In 1996 he graduated cum laude in more than 100 papers in referred journals on high performance
Computer Science Engineering at University of Bologna and computing, grid & cloud computing and distributed data
in 2001 he received his Ph.D. in Computer Science and management.
System Engineering from the University of Calabria.
REFERENCES
Afterwards, he was Visiting Researcher at Argonne National
Laboratory (Chicago, US) and University of Aachen [1] “LXD, The Linux container hypervisor”,
http://www.ubuntu.com/cloud/tools/lxd
(Germany), adjunct professor and research fellow at the
[2] “Docker”, http://www.docker.com
Faculty of Engineering at University of Calabria. He has been
[3] W. Felter, A. Ferreira, R. Rajamony, J. Rubio, “An Updated
also a free lance ICT consultant for government and Performance Comparison of Virtual Machines and Linux Containers”,
telecommunication companies. Presently, at CMCC, he is IBM Research Report
involved into national and international projects like: [4] Z. Ding, S. Sanjabi, “Using Docker in High Performance Computing in
EUBrazilCC, OFIDIA, Copernicus Med-MFC. He is author of OpenPOWER Environment”, First Annual OpenPOWER Summit, San
Jose, CA, March 17-19.
several scientific papers in the field of cloud computing, data
[5] M. Mancini, Lightning talk: “Managing Docker Containers with
analytics, high performance computing and scientific OpenNebula”, OpenNebulaConf 2014, Berlin, December 2-4
computing. He is IEEE and IEEE Computer Society member. [6] M. Mancini, “Provisioning and Orchestrating Big Data and HPC
GIOVANNI ALOISIO is Full professor of Information Services with OpenNebula at CMCC”, OpenNebulaConf 2015,
Processing Systems at the Dept. of Innovation Engineering of Barcelona, October 22-24
the University of Salento, Lecce, Italy, where he leads the [7] C. Boettiger, “An introduction to Docker for reproducible research, with
HPC laboratory. Former director of the “Advanced Scientific example from the R environment”, 2015 ACM SIGOPS Operating
Systems Review, Special Issue on Repeatability and Sharing of
Computing” (ASC) Division at the Euro-Mediterranean Center Experimental Artifacts. 49(1), 71-79
on Climate Change (CMCC), he is now a member of the

668

Authorized licensed use limited to: Nirma University Institute of Technology. Downloaded on October 03,2024 at 07:24:53 UTC from IEEE Xplore. Restrictions apply.