High-performance I/O: HDF5 for lattice QCD

Thorsten Kurth; Andrew Pochinsky; Abhinav Sarje; Sergey Syritsyn; André Walker-Loud

High-performance I/O: HDF5 for lattice QCD

Thorsten Kurth
, Andrew Pochinsky
, Abhinav Sarje
, Sergey Syritsyn
, André Walker-Loud

Physics and Astronomy

Lawrence Berkeley National Laboratory
University of California at Berkeley
Massachusetts Institute of Technology
College of William and Mary
Thomas Jefferson National Accelerator Facility

Research output: Contribution to journal › Conference article › peer-review

5 Scopus citations

Abstract

Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-theart high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.

Original language	English
Article number	045
Journal	Proceedings of Science
Volume	Part F130500
State	Published - 2014
Event	32nd International Symposium on Lattice Field Theory, LATTICE 2014 - New York, United States Duration: Jun 23 2014 → Jun 28 2014

Cite this

@article{f43dd939bbbf4fb2b2d4c1c691ca2f83,

title = "High-performance I/O: HDF5 for lattice QCD",

abstract = "Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-theart high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20\% level and leaves room for further improvement by utilizing appropriate dataset chunking.",

author = "Thorsten Kurth and Andrew Pochinsky and Abhinav Sarje and Sergey Syritsyn and Andr{\'e} Walker-Loud",

note = "Publisher Copyright: {\textcopyright} Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence.; 32nd International Symposium on Lattice Field Theory, LATTICE 2014 ; Conference date: 23-06-2014 Through 28-06-2014",

year = "2014",

language = "English",

volume = "Part F130500",

journal = "Proceedings of Science",

issn = "1824-8039",

}

TY - JOUR

T1 - High-performance I/O

T2 - 32nd International Symposium on Lattice Field Theory, LATTICE 2014

AU - Kurth, Thorsten

AU - Pochinsky, Andrew

AU - Sarje, Abhinav

AU - Syritsyn, Sergey

AU - Walker-Loud, André

N1 - Publisher Copyright: © Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence.

PY - 2014

Y1 - 2014

N2 - Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-theart high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.

AB - Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-theart high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.

UR - https://www.scopus.com/pages/publications/85030125984

M3 - Conference article

AN - SCOPUS:85030125984

SN - 1824-8039

VL - Part F130500

JO - Proceedings of Science

JF - Proceedings of Science

M1 - 045

Y2 - 23 June 2014 through 28 June 2014

ER -

High-performance I/O: HDF5 for lattice QCD

Abstract

Other files and links

Fingerprint

Cite this