Massive Exabyte Storage System to be Built for Energy’s Frontier Supercomputer

The Energy Department will outfit the fastest supercomputer ever built with a first-of-its-kind storage system capable of storing more than an exabyte of data.

For context, fewer than three exabytes made up the world’s entire technological storage capacity in 1986, and a single exabyte contains the same amount of information as a stack of CDs 1,000 miles high.

Produced in partnership with the Oak Ridge National Laboratory and Cray Inc., little is known about the new $50 million storage solution other than it will connect directly to DOE’s Frontier exascale supercomputer—which the agency plans to deliver by 2021—and will advance its machine learning and artificial intelligence capabilities.   

Justin Whitt, ORNL’s Frontier project director told Nextgov the new solution will give scientists the performance and capacity required for traditional modeling and simulation, as well as deep learning and data science.

“We plan for this data storage to be accessible from, not only Frontier, but other types of computational resources that are essential to our workloads," Whitt said. In describing the immense size of the storage solution, Whitt added, “It would take a little over 2.5 billion computers from the 1990s to store an exabyte of hair band music."

In a statement, Cray said the next generation storage solution will be the “largest single high-performance file system in the world,” and comprised of 40 cabinets of storage. The exabyte of total capacity will be provided across two tiers of storage to support random and streaming access of data.

“By delivering a new hybrid storage solution[...], users will be able to drive data of any size, access pattern or scale to feed their converged modeling, simulation and [artificial intelligence] workflows,” John Dinning, Cray’s chief product officer said.

One exabyte of storage is about 1,000 petabytes or 1,000,000,000,000,000,000 bytes. The new system will provide more than four times the capacity and throughput of storage than ORNL’s Summit supercomputer with enough capacity to store more than 200 million high definition movies, though that will not be how it’s put to use. Energy expects the Frontier supercomputer to make innovative leaps and bounds through its ability to conduct more than an exaflop of calculations per second and accelerate innovations around AI.

Typically, traditional high performance computing storage systems are not suited for the usage of AI and machine learning workloads, which generally have a mix of access for small and large data sizes. Cray said the new storage solution will confront that challenge “head on” by providing a blend of storage to support the emerging technologies’ complex access patterns and new means to manage and tier the data.

The company said the storage solution will help enable “seamless scaling of diverse modeling, simulation, analytics and AI workloads running simultaneously on the system.”