Argonne National Laboratory’s former supercomputer, Mira, was an IBM BlueGene/Q system ranked the #3 supercomputer in the world when it was in use. Mira performed at 8.15 petaflops or 8.15 quadrillion floating-point operations per second and could conduct 10 quadrillion calculations per second. Argonne’s goal was to have Mira handle more than 5 billion computations a year when the system was in full production.

Before it was decommissioned in 2019, the Mira supercomputer helped researchers solve problems in a variety of scientific fields – including biochemistry, cosmology, astrophysics, pharmacology, and materials science – and perform simulations and create models of everything from combustion engines to blood flow.

The design/build project, for which IMEG served as Engineer of Record, implemented an earlier study performed by IMEG of the existing infrastructure system to determine expansion needs to support the new water-cooled supercomputing system within the Theory and Computing Science (TCS) Supercomputing Support Facility (SSF). At that time, design for the supercomputer had not yet been completed, so IMEG was tasked with estimating and predicting the needs of the computer as it was being designed. IMEG worked very closely with Argonne subject matter experts and scientists to agree on a path forward, as the design had to be flexible and realistic.

The goal of the project was to provide the computing horsepower and space to save data. The design added 50 computer racks to the existing data center, which contributed 100 KW of heat gain, approximately 90% of which was rejected to water. IMEG’s design added approximately 42 inches of space for chilled water pipes to be installed underneath the computer, along with air cooling to accommodate the remaining 10% along with the corresponding I/O and storage racks. When first launched, Mira was ranked as one of the world’s most energy-efficient supercomputers, thanks to the water-cooling system that provides better and faster cooling than blowing air over the machine.

Specific infrastructure improvements included:

  • Adding two 50,000 CFM recirculating air handling units to accommodate the anticipated increase in air cooled computer load
  • Adding stainless steel medium temperature cooling pumps and piping infrastructure to support the water-cooled computer load
  • A central plant tie-in for the existing low temperature cooling loop, which provides additional capacity and redundancy
  • Humidification for the existing data center
  • Electrical feed and distribution improvements necessary to support the required power
  • Addition of an emergency generator
  • Modifying the VESDA fire suppression system to require two smoke inputs prior to water release.

IMEG also worked with Argonne for budgeting and infrastructure conceptual planning for the next supercomputing system. Since the supercomputers do not exist when planning begins, initial planning is based heavily on assumptions and input from the supercomputer vendor community. Planning begins prior to Argonne knowing which computer vendor they will select. After the vendor is selected, these assumptions are confirmed in an iterative process by the vendor. At each stage, IMEG updates the infrastructure planning and budgeting. This process typically requires several iterations. Construction of the infrastructure improvements begins prior to the computer being fully designed and tested so that the infrastructure is in place and ready for connection when the computers arrive.