What’s so interesting about the new NCAR HPE Supercomputer at Wyoming?

Supercomputers are used for a wide range of computationally intensive tasks. These include weather forecasting, the drug discovery process, nuclear warhead simulations and more. The recently announced new Hewlett Packard Enterprise-Cray EX, at Wyoming will help study phenomena including climate change, severe weather, wildfires and solar flares. A few days ago, Houston-based HPE won a bid to offer the US$35 million to US$40 million machine for a new flagship supercomputing system that will be installed this year at the NCAR-Wyoming Supercomputing Center (NWSC) in west Cheyenne. NCAR stands for National Center for Atmospheric Research. This supercomputer is claimed to rank among the world’s fastest.

This CPU/GPU-powered system, funded by the National Science Foundation, is stated to deliver 3.5x the power of NCAR’s existing HPC system (called Cheyenne) and put it to use in 2022. The system, known for now as NWSC-3, is expected to rank among the world’s 25 fastest supercomputers. Envisaging a capacity to support simulations of weather-related phenomena, this supercomputer will have a theoretical peak performance of 19.87 petaflops. This implies it boasts of a theoretical ability to perform 19.87 quadrillion calculations per second.

Official sources state that it will have a computational power of making 10-year weather forecasts, water supply and drought risk predictions, wildfire management and other studies involving solar plasma flow and its impact on power grids and communications systems. Even, simulating wildfire risks in a region to help forecasts in wooded areas becoming prone to fires, will be one of its capabilities. Here the simulations will be carried out based on data from local winds and air density, soil moisture, and vegetation patterns such as types of grass and leaves.

According to Bill Mannel, vice president and general manager for HPC at HPE, the supercomputer is also meant to help NCAR provide accessible data while informing ecosystem preservation and disaster response decisions.

Features

The supercomputer is said to comprise a total of 2,570 compute nodes. This includes 2,480 CPU-only nodes (each powered by two third-generation AMD Epyc Milan CPUs) and 82 GPU nodes (each powered by one Epyc Milan CPU and four Nvidia A100 “Ampere” GPUs) alongside eight Epyc Rome-based login nodes. So, overall, the system will have 692 TB of memory. Experts argue that the system’s maximum speed per second will be roughly equivalent to each person on Earth completing a math equation every second for an entire month.

Further, NCAR revealed that the GPUs would only constitute 20% of the system’s sustained computing capacity.

It will also feature HPE Slingshot (v11) interconnect in a Dragonfly topology, with 200 Gb/sec bandwidth per port per direction, for congestion control. Also, 60 petabytes of storage for supporting complex workloads in modeling, simulation and AI using the Cray ClusterStor E1000. It will be augmented by an integrated software suite using the HPE Cray Programming Environment and Nvidia-provided HPC software, with AI frameworks, compilers, libraries, models and tools. The system will also be set up to burst to commercial cloud computing services.

The supercomputer will also have HPE Cray Operating System, a tuned version of SUSE Linux; Altair Accelerator Plus scheduler with PBS Professional Workload Manager; support for Docker and Singularity containers, and containers that support the Open Container Initiative standard.

As per the Center’s Director Everette Joseph, the supercomputer will support basic research in ways that will lead to more detailed and useful predictions of the world around us, thereby making our society more resilient to increasingly costly disasters and contributing to improved human health and well-being.

NCAR’s current flagship system – the Cheyenne supercomputer, was launched almost four years ago. Cheyenne contains 4,032 Intel Xeon Broadwell-based nodes, 313 TB of memory and Mellanox EDR InfiniBand networking. Its 4.8 Linpack petaflops (5.3 peak petaflops) placed it 60th on the most recent Top500 list. Cheyenne is over three times faster than its predecessor, named Yellowstone and consumes 40% less electrical power than the new supercomputer.

NWSC says since its inauguration in 2012, over 4,000 users from nearly 625 universities and other institutions in the US and around the world, have used its resources and facilities. The name of this new supercomputer will be finalized from a suggestions contest among school-going children at Wyoming.