Introduction of CDAC’

Established in March 1988, as a Scientific Society of the Department of Information Technology

(formerly, Dept. of Electronics), Ministry of Communications and Information Technology
(formerly, Ministry of Information Technology), Government of India, The Centre for
Development of Advanced Computing (C-DAC), is primarily an R & D institution involved in
the design, development and deployment of advanced Information Technology (IT) based
solutions.

In a little over a decade since inception, C-DAC has developed and supplied a range of high
performance parallel computers, known as the PARAM series of supercomputers. C-DAC's
development activities in this area have been mission oriented and driven by its mission
objectives, both in technology and application developments.

C-DAC, as a result of its pioneering developments, evolved the Graphics and Intelligence based
Script Technology (GIST), with a view to extend the benefits of Information Technology to the
vast and diversified multilingual population of India. Use of the GIST range of software and
hardware products has led to the proliferation of the use of computers and their applications in all
major Indian languages, with hundreds of thousand of users countrywide.

Over the years, C-DAC has diversified its activities to address requirements in various areas,
consequently, our expertise also extends to other advanced areas of Information Technology,
enabling IT based solutions in areas like Financial and Capital market simulation and modeling,
Network and Internet Software, Healthcare, Real Time Systems, eGovernance, Data
Warehousing, Digital library, Artificial Intelligence and Natural language processing.

ABOUT CTSF
C-DAC is India's national initiative in Advanced
Computing with a mission to deliver state-of-the-art,
open architecture, scalable high performance computers
in the desktop to teraflop range, embodying the
emergent industry-standard building blocks. Towards
fulfillment of this goal, C-DAC has advented the
OpenFrame Architecture which has been realized in its
celebrated PARAM series of High Performance
Computers. PARAM Padma at C-DAC's Tera-Scale
Supercomputing Facility (CTSF) is a result of its third mission project in High Performance
Computing Technology and Applications.

While the need and usefulness of high performance supercomputing in Business as well as
Scientific & Engineering Applications is unquestioned and is growing rapidly, it is not
economically viable and justified to have many such facilities.

Recognizing such a need, C-DAC had earlier set up National PARAM Supercomputing Facility
(NPSF) at Pune, housing its earlier generation PARAM 10000, a 100 Gflop peak computing
power system.

CTSF OBJECTIVES
The primary objectives of CTSF are:

 To provide high performance computing facilities and services for the scientific and
research community and for the enterprise.

 To establish the technological capabilities in high performance computing that have

hitherto been confined only to developed countries.

 To solve some of the grand challenge problems which are the key to economic growth,
environmental understanding and research breakthroughs in science & engineering.

C-DAC's HPCC (High Performance Computing and Communication) initiatives are aimed at
designing, developing and deploying advanced computing systems, tools and technologies that
impact strategically important application areas.

Fostering an environment of innovation and dealing with cutting edge technologies, C-DAC's
PARAM series of supercomputers have been deployed to address diverse applications in science
and engineering, and business computing at various institutions in India and abroad.

C-DAC's commitment to the HPCC initiative has once again manifest as a deliverable through
the design, development and deployment of PARAM Padma, a terascale supercomputing system.

PARAM Padma is C-DAC's next generation high performance scalable computing cluster,
currently with a peak computing power of One Teraflop. The hardware environment is powered
by the Compute Nodes based on the state-of-the-art Power4 RISC processors, using Copper and
SOI technology, in Symmetric Multiprocessor (SMP) configurations. These nodes are connected
through a primary high performance System Area Network, PARAMNet-II, designed and
developed by C-DAC and a Gigabit Ethernet as a backup network.

The PARAM Padma is powered by C-DAC's flexible and scalable HPCC software environment.
The Storage System of PARAM Padma has been designed to provide a primary storage of 5
Terabytes scalable to 22 Terabytes. The network centric storage architecture, based on state-of-
the-art Storage Area Network (SAN) technologies, ensures high performance, scalable and
reliable storage. It uses Fibre Channel Arbitrated Loop (FC-AL) based technology for
interconnecting storage subsystems like Parallel File Servers, NAS Servers, Metadata Servers,
Raid Storage Arrays and Automated Tape Libraries, achieving an I/O performance of upto 2
Gigabytes/Second.

The Secondary backup storage subsystem is scalable from 10 Terabytes to 100 Terabytes with an
automated tape library and support for DLT, SDLT and LTO Ultrium tape drives. It implements
a Hierarchical Storage Management (HSM) technology to optimize the demand on primary
storage and effectively utilize the secondary storage.

The PARAM Padma system is also accessible by users from remote locations.

PARAMNet-II
PARAMNet-II, a low latency, high bandwidth
interconnect, provides data rates of 2.5
Gigabits/sec in full duplex over fiber. The
message latency is as low as 10 µsec.

PARAMNet-II uses a 16 port switch and a

Network Interface Card (NIC) alongwith an
Application Programming Interface i.e. C-DAC's
Virtual Interface Provider Library (C-VIPL). The
non-blocking architecture of the switch allows
multilevel switching for realizing a large cluster. The switch offers very low latency of the order
of 0.5 µsec. Use of an Interval routing scheme and group adaptive routing based on Least
Recently Used (LRU) algorithm, ensures uniform bandwidth distribution. The NIC is based on
C-DAC's Communication Co-Processor-III (CCP-III) chip based on 0.15 micron 1 million gate
technology. Implementation of packetisation & reassembly, flow control, protection mechanism,
address translation and error recovery in CCP-III, results in low latencies and very low
overheads for the CPU.

PARAM Padma has 12 PARAMNet-II switches connected in two level configurations to form a
64-node CLOS network.

PARAMNet-3

PARAMNet-3 is a high performance cluster interconnect developed indigenously by C-DAC.

With this development, C-DAC has joined an elite group of system developers worldwide,
capable of providing a critical high performance networking component for building
supercomputing systems.

The main application of PARAMNet-3 is, as a primary interconnect for PARAM Yuva. Other
application areas identified for its deployment are storage and database applications. It is also an
integral component of other HPC solutions offered by C-DAC.

PARAMNet-3 consists of tightly integrated hardware and software components. The hardware
components consist of Network Interface Cards (NIC) based on C-DAC's fourth generation
communication co-processor "GEMINI", and modular 48-port Packet Routing Switch
"ANVAY". The software component "KSHIPRA" is a lightweight protocol stack designed to
exploit capabilities of hardware and to provide industry standard interfaces to the applications.

Communication Co-processor : Gemini

The Gemini Co-processor is fine tuned for HPC applications over uDAPL and OpenIB software
stacks. It offloads transport layer functionality to hardware for supporting send / receive and
Remote Direct Memory Access (RDMA) channels with RC & UD services, allowing compute
nodes to communicate efficiently. Direct user level hardware access to 4k simultaneous
connection oriented and connectionless hardware endpoints is provided to support applications
of larger problem sizes. Gemini is capable of performing I/O from paged virtual memory with
byte aligned addresses and reliable communication of maximum 4 GByte message length.

Packet Routing Switch : Anvay

The modular, 48-port packet routing switch 'Anvay' provides a near wire speed (10Gbps) packet
routing capability with very low latencies. Anvay switch is based on Spider Line Card and
Backplane subsystems. Each spider card support up to eight ports, with each port working at
10Gbps, full duplex speeds. A fully populated switch supports approximately 1 Terabits / second
of aggregate system throughput. The switch supports multi-level switching, allowing cascading
of switches for supporting large clusters.

Communication Substrate : KSHIPRA

KSHIPRA is a scalable communication substrate. Its a software environment that leverages upon
the RDMA channel to deliver best application level performance. KSHIPRA allows the HPC,
storage and enterprise applications to take advantage of PARAMNet-3's low latency, high
bandwidth capability and drastically cuts down the CPU usage, by using uDAPL transport
mechanism. It also depicts the network as a standard IP network. Thus all TCP/IP applications
can be run seamlessly. MPI, SDP, IPoIB and iSER are included in the protocol set.

KSHIPRA, available as a bundled CD is a complete environment consisting of drivers, libraries,

cluster management tools and documentation.

Specifications

PARAMNet-3 Network Interface Card (NIC)

 Based on GEMINI communication co-processor

 PCI express (x4/x8) full duplex based host interface
 10Gbps Full duplex CX-4 link interface
 16 MBytes onboard memory
 Transport protocol offload engine supporting channel and memory semantics & RC and
UD services
 Proprietary hardware protocol implemented using Dual 32-bit CPUs for low latencies and
reliable communication
 Direct user level access to 4k simultaneous connection oriented or connectionless
hardware endpoints
 Capable of performing I/O from paged virtual memory with byte aligned addresses
 Address Translation table for VA-PA with protection implemented in hardware
 Communicates to host through Interrupt and Completion Queues

PARAMNet-3 Packet Routing Switch : ANVAY

 Modular solution supporting 8"48 ports

 Each port supporting 10 Gbps, full duplex communication over CX-4 cables
  Interval labelling based packet routing
 Wormhole routing for reducing latency
 32Kbyte (ingress) and 16 Kbytes (egress) packet buffers
 Near neighbour communication using dedicated communication path
 Port to port latency < 2 µsec
 Near wire speed performance (1.1 GByte/sec per port)
 Pause/Resume based flow control with back pressure
 Packet payloads up to 4KBytes
 Support for unicast and broadcast
 Fully manageable over Ethernet and RS-232

KSHIPRA Software Stack

 Linux support
 64k end points
 RDMA centric architecture
 uDAPL protocol with kernel bypass
 OpenIB adaptation with IPoIB, SDP and iSER protocols
 Seamless integration of existing TCP based applications
 Communication libraries MVAPICH2, Intel MPI supported
 Targeted for HPC, Storage/Database and enterprise applications

The National PARAM Supercomputing Facility (NPSF) of C-DAC is a result of almost a decade
long effort of R & D in High Performance Computing, Hardware, Systems Software and
Applications development. C-DAC is India's national initiative in Advanced Computing with a
mission to deliver state-of-the-art, open architecture, scalable high performance computers in the
desktop to teraflop range, embodying the emergent industry-standard building blocks. Towards
fulfillment of this goal, C-DAC has advented the OpenFrame Architecture which has been
realized in it's celebrated PARAM series of High Performance Computers.

While the need and usefulness of High performance Supercomputing in Business as well as
Scientific & Engineering Applications is unquestioned and is growing rapidly, it is not
economically viable to have many such facilities. C-DAC has thus established "National
PARAM Supercomputing Facility" to provide Supercomputing facility to various industries and
other institutions that need such a facility to process their diverse applications.
NPSF OBJECTIVES
The broad objectives of NPSF are as follows:

 To explore new industrial applications which demand supercomputing solutions in order

to have international competitiveness of the final designed products.

 To provide facilities to institutions/organizations primarily within India for fulfilling

their computational needs.

 To establish the capabilities in high-technology areas, which are hitherto been confined
to only developed countries.

 To solve some of the demanding problems, which could benefit the society in general to
uplift the standard of living.

PARAM Anant
PARAM Anant is a low-cost supercomputing solution based on C-DAC's
unique OpenFrame architecture for scalable and high-performance computing
that incorporates well-known Cluster of Workstations (COW) and Massively
Parallel Processing (MPP) concepts. The core supercomputing technologies
for the PARAM Anant are derived from C-DAC's 100GF PARAM 10000
installed at the National Param Supercomputing Facility located at Pune, India
which is a culmination of over 10 years of C-DAC's expertise in the
development of supercomputers for scientific and engineering applications.

The entire system including the compute nodes, network elements and the system software are
upgradeable at the system, component and technology levels to suit the needs of the users.

With PARAM Anant, supercomputing is now accessible to the education, research and business
communities at an affordable cost. The distinguishing features of this supercomputing platform
are:

 Based on off-the-shelf components

 Supports cluster of microprocessors (CLUMPS), exploiting shared and distributed
memory
 Uses PARAMNet as the high-bandwidth System Area Network (SAN)
 Switch architecture supports scalability through multistage networks
 MPI layered over active messages to exploit the low latency and high bandwidth of SAN
 C-DAC's HPCC software suite for program development, system management and
software engineering
 Wide range of applications/kernels