Stepping up to Summit

Transcription

1 DEPARTMENT: Leadership Computing Stepping up to Summit Jonathan Hines Oak Ridge National Laboratory Editors: James J. Hack, Michael E. Papka, In November 2014, the Oak Ridge Leadership Computing Facility (OLCF), a US Department of Energy (DOE) Office of Science User Facility located at Oak Ridge National Laboratory (ORNL), announced plans for an ambitious new chapter in scientific computing: the Summit supercomputer, a machine capable of supplying five to ten times more computing power than current leadership-class systems (see Figure 1). Scientists, in turn, could harness this computational behemoth to address longstanding chal- lenges in science and engineering. Fast-forward to 2018 and the OLCF s dream machine is poised to deliver new insights in fields as diverse as biology, astrophysics, and materials science. The system should be completed by early spring, followed by an extensive testing and acceptance period that will lead to early science computational opportunities in the latter part of 2018.The broader user community is slated to gain access to the machine at the start of Researchers who successfully adapt their applications to Summit s GPU-heavy architecture will receive a huge payoff; the IBM Power Systems AC922 is designed with a peak performance of 200 Pflops (200 million billion calculations per second). That s a tremendous leap from the 27Pflops Titan supercomputer, the OLCF s current leadership-class machine, and a substantial step toward the first exascale machine, a system capable of a billion billion calculations per second. Figure 1. The Oak Ridge Leadership Computing Facility s newest supercomputer, Summit, a machine that will supply five to ten times more computing power than current leadership-class systems, will be online in Photo credit: Carlos Jones/Oak Ridge National Laboratory (ORNL). Computing in Science & Engineering 78 Copublished by the IEEE CS and the AIP /18/$ IEEE

2 COMPUTING IN SCIENCE & ENGINEERING Like Titan, Summit will open a door to new ways of simulating and exploring complex systems in the natural world, said James Hack, Director of the National Center for Computational Sciences (NCCS) at ORNL. Our scientific community will see decreased time to solution, along with the ability to increase the complexity of their computational models to explore a wide variety of important phenomena that are beyond the range of conventional experiments. In addition to signifying a remarkable achievement for computer science, the Summit deployment also marks the culmination of a multiyear collaboration between ORNL, high-performance computing (HPC) vendor partners, and the scientific community, among other stakeholders. From facility enhancements to hardware procurement to application development, the project serves as a blueprint for advancing what s possible in scientific discovery. LAYING THE FOUNDATION To accommodate Summit within ORNL s existing footprint, renovations began to the ORNL Computational Sciences building in 2015 (see Figures 2 and 3). Facility enhancements focused on providing the space, power, and cooling necessary to support the pre-exascale system, with an eye toward supporting follow-on systems over the course of the next two decades. Design plans crafted in coordination with the architectural firm Heery International called for facility enhancements to create a 10,000 square foot computer room, infrastructure to deliver an additional 20 MW of power, a cooling plant, and an expanded central energy plant. Over the course of two years, construction crews carried out these plans to meet ORNL s specifications. Figure 2. To accommodate Summit within ORNL s existing footprint, renovations began to the ORNL Computational Sciences building in Photo credit: Jason Richards/ORNL. At peak power, Summit is expected to use around 15 MW. To supply the necessary electricity, personnel installed a subsurface duct bank on the west end of the Computational Sciences building, routing electrical circuits into the building via an overhead conduit bridge that connected to a new transformer platform. Just south of the Summit computer room, crews stood up a new cooling tower capable of delivering a swimming pool s worth of water to Summit s warm-water cooling system. Past OLCF systems have been cooled via chillers that lower the temperature of water to around 42 F an energy-intensive process that accounts for about a quarter of Titan s annual operating budget. Summit s temperature-tolerant hardware and advanced cooling technology allows for the system to be operated at a much higher temperature around 70 F and at a much lower cost. 79

3 LEADERSHIP COMPUTING The technology used to cool Summit, called plate and frame heat exchangers, leverages the tendency of heat to flow from a hot medium to a cold medium. The waste heat from Summit, perhaps 105 F on average, transfers across a thin metal plate to cooler water, which is carried away to high-efficiency cooling towers, said Jim Rogers, NCCS Director of Computing and Facilities. Because Summit s cold-water supply temperature is only about 70 degrees, it s a whole lot easier to maintain that temperature in our environmental conditions. Figure 3. Once construction of the space was complete, installation of the Summit supercomputer began in summer of 2017 with the Mellanox team installing approximately 180 mi of cabling if each of the approximately 5,000 cables were connected end-to-end. Photo credit: Carlos Jones/ORNL. DATA IN DESIGN Building upon the GPU-accelerated architecture pioneered by Titan, Summit s hardware sits at the center of its appeal to innovative research teams. Though the system contains only about a quarter of the nodes of its predecessor (4,600 versus 18,688), the units boast six NVIDIA Volta GPUs and two IBM Power9 CPUs and supply more than 40 Tflops of performance. Guided by IBM s data-centric approach to computing, data movement within and between Summit s nodes is dramatically accelerated. CPU GPU integration facilitated by NVIDIA s NVLink high-bandwidth interconnect gives application developers increased flexibility for tapping into Summit s hierarchical parallelism. Enhanced high-capacity, on-node memory encourages the minimization of data movement, while a dual-rail Mellanox EDR InfiniBand interconnect configured as a full, nonblocking fat-tree supplies fast highways for internode communication. Summit s large, powerful nodes allow applications to achieve very high performance without having to scale across the interconnection network to hundreds of thousands of nodes, said Buddy Bland, OLCF project director. The combination of very large memory per node and the powerful IBM POWER and NVIDIA processors provides an ideal platform for data analysis as well as computation. On the storage side, Summit s IBM-produced GPFS parallel file system provides 250 Pbytes of storage, with an ingest rate of more than 2 Tbytes per second, more than double the rate of Titan s file system. Data transfer between compute and storage is further enhanced by a burst buffer of nonvolatile memory with intake speeds approaching 10 Tbytes per second. In addition to facilitating storage, the buffer also doubles as extended memory for compute. 80

4 COMPUTING IN SCIENCE & ENGINEERING PULLING DOUBLE DUTY Continuing an initiative first piloted in preparation for Titan, the OLCF relaunched its Center for Accelerated Application Readiness (CAAR) in 2015 to help users ready their codes for Summit. Thirteen application teams representing a broad range of computational algorithms, programming approaches, and scientific disciplines were selected. CAAR teams are paired with an OLCF computational liaison and receive technical support from the IBM/NVIDIA Center of Excellence, housed at ORNL. In addition to OLCF expertise, CAAR teams gain access to development platforms that more accurately represent Summit s computing environment. Using the Summit Early Access development platform called Summitdev, a system composed of several dozen nodes with two IBM POWER8 CPUs and four NVIDIA Pascal GPUs, teams got a head start redesigning, porting, and optimizing their software to Summit s architecture. In one early success story, OLCF staff diagnosed and fixed a bottleneck in a new tensor library for three CAAR chemistry applications. The modification boosted the library s runtime performance by as much as 10 times. We want to provide our CAAR teams with the expertise and resources they need to take full advantage of Summit when it comes online, said Tjerk Straatsma, OLCF Scientific Computing group leader. Having access to Summitdev will help get these applications ready for use by our user programs as soon as Summit becomes available and starts delivering science outcomes. Though Summit was designed with traditional modeling and simulation in mind, the IBM system doubles as an artificial intelligence (AI) behemoth. Advancements in machine learning, deep learning, and AI applications driven by the growth of scientific data and the exploitation of GPU parallelism for training neural networks position Summit as one of the most AI-capable machines ever constructed. On Titan, ORNL researchers recently demonstrated the value of scaling evolutionary algorithms to optimize deep neural networks for specific scientific problems, including classification challenges in high-energy physics, biomedicine, and satellite imagery. Other AI-focused projects at the lab have focused on deep learning s potential for accelerating cancer research, fusion power, and materials investigation. With Summit and its more than 27,000 GPUs, each capable of unprecedented levels of machinelearning performance, deep learning researchers expect to take this blossoming technology even further. Because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems. Increasingly, an important part of scientific discovery is about looking at data itself and using that to drive theoretical or scientific understanding, said Arjun Shankar, OLCF Advanced Data and Workflow group leader. We re finding that there are a lot of scientists who are excited about the prospect of using Summit s high-performance GPUs to process data at scale. ENTRY POINT TO EXASCALE Beyond Summit, the OLCF is already making plans to acquire, deploy, and operate one of the first exascale systems, to be named Frontier. Like the Summit project, an exascale system will require creative solutions to practical constraints in engineering, hardware, software, and operation. Getting to capable exascale systems will be a significant task given the constraints on power and space, and the evolution of programming environments. It will also require careful coordination between other DOE efforts, like the Exascale Computing Project, and the leadership computing user facilities, said OLCF Deputy Project Director Justin Whitt. 81

5 LEADERSHIP COMPUTING ACKNOWLEDGMENTS The Oak Ridge Leadership Computing Facility is a Department of Energy Office of Science User Facility supported under contract DE-AC05-00OR This manuscript has been authored by UT-Battelle, LLC, under contract number DE-AC05-00OR22725 with the US Department of Energy. The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. The DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan ( ABOUT THE AUTHOR Jonathan Hines is a science writer at Oak Ridge National Laboratory. He received a bachelor s degree in journalism from Indiana University. Contact him at hinesjd@ornl.gov. 82