Osaka University Creates AI Supercomputer

with Supermicro SuperBlades Utilizing 3rd Gen

Intel® Xeon® Scalable Processors
New Features Enable Faster Results and AI Research

Figure 1: Figure or Caption Style (optional)

Introduction

Osaka University is a world-renowned university known for leading-edge research in

a wide range of disciplines. As a 90-year-old institution, Osaka University serves a
wide range of researchers, students, and external experts, making available a
INDUSTRY
supercomputer system that runs state-of-the-art software on the latest CPUs and
University Research and GPUs.
Industry Collaboration
Challenges
CHALLENGES
The Cybermedia Center at Osaka University offers high-performance systems and
• Faster processing and storage to diverse users, both internal to the university and external researchers. The
reduced time to solution
need for faster processing and a reduced time to solution for a wide range of
• Respond quick for new disciplines was a key factor when Osaka University looked to upgrade their system.
application requests
Osaka University is expanding its research areas, including high-performance data
• Ability to take advantage of analysis, Artificial Intelligence (AI), Deep Learning (DL), as well as more traditional
new CPU instructions
High-Performance Computing (HPC), which required the fastest system to be
constructed in a balanced way for high performance and the lowest TCO. This fact
© Copyright 2021 Supermicro Intel®
 meant that the CPUs had to be fast with an increasing number of cores compared to
previous CPU generations, but communication with other devices such as GPUs and
storage had to be increased as well.

HPC applications always need the latest technologies such as processing speeds and
other system capabilities for research into diverse fields such as genome research,
medical, weather prediction, earth sciences, and manufacturing simulations. With an
increasing demand for performance for a range of innovative technologies, Osaka
University needed a new system that could address the demands of the researchers.
Thus, the SQUID (Supercomputer for Quest to Unsolved Interdisciplinary
Datascience) system was developed to meet these new challenges.

Osaka University offers a wide range of software to its users that is thoroughly tested
SOLUTION
and qualified for new systems. When moving from one generation to the next with
• Supermicro SuperBlade
Intel products, the binary compatibility makes this qualification relatively easy.
• Supermicro GPU systems Although applications may not be entirely optimized for a new instruction set, the
• 3rd Gen Intel® Xeon® application should run without any issues. Re-compiling with updated compilers will
Scalable Processors increase the optimization amounts when a new CPU is introduced, containing new
and more performant instructions. Osaka University expects the FLOPS/Watt to be
significantly improved for reducing the carbon footprint with the new system. Users
BENEFITS will get massive improvement with their application performance by changing the
parallelization tuning to include the latest Intel® Xeon® Scalable processor's
• Faster simulations
instructions.
• AI Workloads with the latest
GPUs and PCI-E 4.0
Solution
• Workload Matching

Osaka University worked closely with NEC Corporation and Supermicro to design and
implement a new supercomputing class system based on a combination of
Supermicro SuperBlade® systems and 4U 8 GPU NVIDIA NVlink ® systems, which is
PRODUCTS
built with the 3rd Gen Intel® Xeon® Scalable Processors. These systems were
• Compute Nodes: benchmarked with the new X12 SuperBlade and exceeded the expectations of the
SuperBlade® 8U system architects and early users. The performance, cost, and power usage of the
SBI-420P blade latest 3rd Gen Intel® Xeon® Scalable Processors have many advantages compared to
previous generations of Intel® Xeon® Scalable Processors.
SBE-820 enclosure, 4
enclosures per rack

InfiniBand HDR 200

• GPU nodes: "Researchers and scientists in Osaka University are now working on their research projects in the
hope of bringing open innovation to our society. Furthermore, the Cybermedia Center at Osaka
SYS-420GP-TNAR
University is in charge of delivering high performance to researchers in Japan to advance
• Front end Nodes:
scientific research as a national joint-use facility. The new supercomputing system named SQUID,
SYS-620U-TNR scheduled to begin operations in May 2021, was designed and implemented to facilitate
researchers and scientists to quest to unsolved interdisciplinary data science. The test drive
conducted on SQUID before its commencement of operation showed us that the Supermicro
servers with Intel CPUs provided more performance than was expected. We are excited to
leverage the latest generation of Intel and Supermicro's solutions for advanced scientific
research projects necessitating HPC (High-Performance Computing) and HPDA (High-
Performance Data Analysis)." - Dr. Susumu Date, Cybermedia Center, Osaka University.

© Copyright 2021 Supermicro Intel®

The SQUID system is a 27 Rack cluster that contains 1520 SuperBlades, 42 nodes of
the 4U 8 GPU NVLink systems, and was integrated and tested by the Supermicro Rack
Integration Services team.
 The systems use the 3rd Gen Intel® Xeon® Scalable Processors, specifically
the Platinum 8368, running at 2.4GHz, 38 Cores, TDP 270W. Though this high
TDP processor, in many cases, can be air-cooled, the most effective cooling
solution for stable computing is to use liquid cooling. Supermicro Direct
Liquid Cooling solution is the most cost-effective method to reduce the
carbon footprint and PUE of their data center.
 The cluster is constructed from Supermicro X12 SuperBlades consisting of
1520 blades (nodes) in 80, 8U enclosures with 3rd Gen Intel Xeon Scalable
Processors for high-performance computing. All of the chassis are integrated
with the direct liquid cooling system.
 A complete rack integrated solution has direct liquid cooling to provide
optimized thermal efficiency for 76 nodes in a 42U rack with InfiniBand HDR
200 connectivity between nodes.
 The Supermicro Rack Integration Services team installed and configured the
systems into 42U racks and tested the systems to ensure multi-rack level
confidence upon delivery.
 The GPU rack for the heterogeneous HPC/HPDA needs also has direct liquid
cooling to its CPU and GPU systems with A100 GPUs and NVSwitch™ modules,
with each rack having 48 GPUs in six SYS-420GP-TNAR servers.
 The 2U Ultra server, SYS-620U-TNR, with direct liquid cooling, is used as the
front end node and is equipped with the Quadro RTX6000 card and hardware
RAID configuration.

The total deployment is 27 complete racks which were all assembled, integrated, and
tested at the Supermicro factory and then directly shipped to Osaka University. Below
is a table that summarizes the different nodes in the Squid supercomputing system.

SuperBlade with DLC GPU System with DLC

© Copyright 2021 Supermicro Intel®

 3rd Gen Intel® Xeon® Memory Per Direct Liquid Cooling
Product Qty # of Sockets
Scalable processors Node Component

Compute node:
3rd Gen Intel® Xeon®
SuperBlade® 8U
1520 blades Scalable processors, DDR4-3200
SBI-420P blade Dual CPU
80 enclosures Platinum 8368, 2.4GHz, 256GB per node
SBE-820 enclosure, 4
38 Core, TDP 270W
enclosures. per rack
3rd Gen Intel® Xeon®
GPU node: Scalable processor, DDR4-3200
42 nodes Dual CPU, GPU, and NVSwitch
SYS-420GP-TNAR Platinum 8368, 2.4GHz, 512GB per node
38 Core, TDP 270W
3rd Gen Intel® Xeon®
Front end node: Scalable processor, DDR4-3200
10 nodes Dual CPU
SYS-620U-TNR Platinum 8368, 2.4GHz, 256GB per node
38 Core, TDP 270W

Benefits

The research community's popularity and scientific needs that Osaka University
serves required that more performance be added to the computing system. For many
researchers, the run queues, which can be thought of as a backlog of applications
waiting for their turn to run with the needed resources for their research, were
growing longer, based on more complex simulations and the absence of GPU specific
workloads. Critical studies that used the latest systems technology were delayed, and
users were looking elsewhere to run their applications.

The arrival of the new systems from Supermicro, based on the 3rd Gen Intel® Xeon®
SUPERMICRO
Scalable Processors, the X12 SuperServer solution includes the latest GPUs and has
Supermicro is a global leader benefited users in the following ways:
in high performance, green  Faster time to completion of simulations. With the new system, more
computing server technology
simulations of similar resolution to be run in a given amount of time, and
and innovation. We provide
our global customers with more research can be accomplished.
application-optimized servers  Increased resolution or fidelity with existing applications. Faster CPUs enable
and workstations customized more complex simulations to be run in a given time frame, with the addition
with blade, storage, and GPU of more physics or finer meshes.
solutions. Our products offer
proven reliability, superior  Workload matching – With the use of the new GPUs, applications in AI, ML,
design, and one of the and DL, has been accelerated significantly, enabling researchers to develop
industry’s broadest array of new algorithms that were not possible previously.
product configurations, to fit
all computational need. The new Supermicro SuperBlade systems containing 3rd Gen Intel® Xeon® Scalable
For more information, visit Processors enable Osaka University to conduct more research much faster than
https://www.supermicro.com previous HPC systems would allow. Entire applications are now able to use the whole
system, employing over 120,000 cores simultaneously. PCI-E 4.0 also enables fast
communication with the latest GPUs, further enhancing and speeding up research
applications.

© Copyright 2021 Supermicro Intel®

 3RD GEN INTEL® XEON® SCALABLE PROCESSORS
3rd Gen Intel® Xeon® Scalable processors, Platinum 8368
PCI-E 4.0
Support for up to 12 TB memory per socket

Further Resources, please visit:

Supermicro X12 Products:

https://www.supermicro.com/en/products/x12?utm_source=Supermicro%20Home
page&utm_medium=Homepage&utm_campaign=X12%20post%20launch

Supermicro SuperBlade Products -

https://www.supermicro.com/en/products/superblade

Supermicro GPU Systems - https://www.supermicro.com/en/products/GPU

Supermicro Rack Integration Services -

https://www.supermicro.com/en/solutions/rack-integration

SQUID system, Cybermedia Center, Osaka University - http://www.hpc.cmc.osaka-

u.ac.jp/en/squid/

© Copyright 2021 Supermicro Intel®