Publications

Data Transfer Tool (HPC Annual Report V.3.1)

Lacy, Susan L.; Mcree, Susan R.

Sandia has been developing and supporting data transfer tools for over 20 years and has the expertise to take DOE into the Extreme Scale era. In looking at Exascale and beyond (Extreme Scale Computing), data sets can be thousands of 500TBs in size, a single file can be in the 100TB range, and billions of files are expected. Huge bursts of data need to be transferred, even today. While data archiving is often not thought about, it is an integral part of the full data management path when data is generated on HPC systems. In order to move generated data to its final resting place (data archive) or to transfer between file systems, a capable data transfer tool is required.