TY - GEN
T1 - Parallel netCDF
T2 - 2003 ACM/IEEE Conference on Supercomputing, SC 2003
AU - Li, Jianwei
AU - Liao, Wei Keng
AU - Choudhary, Alok
AU - Ross, Robert
AU - Thakur, Rajeev
AU - Gropp, William
AU - Latham, Rob
AU - Siegel, Andrew
AU - Gallagher, Brad
AU - Zingale, Michael
PY - 2003
Y1 - 2003
N2 - Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable, efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimal changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I/O is achieved through MPI-IO, allowing for substantial performance gains through the use of collective I/O optimizations. We compare the implementation strategies and performance with HDF5. Our tests indicate programming convenience and significant I/O performance improvement with this parallel netCDF (PnetCDF) interface.
AB - Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable, efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimal changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I/O is achieved through MPI-IO, allowing for substantial performance gains through the use of collective I/O optimizations. We compare the implementation strategies and performance with HDF5. Our tests indicate programming convenience and significant I/O performance improvement with this parallel netCDF (PnetCDF) interface.
UR - http://www.scopus.com/inward/record.url?scp=84877032908&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84877032908&partnerID=8YFLogxK
U2 - 10.1145/1048935.1050189
DO - 10.1145/1048935.1050189
M3 - Conference contribution
AN - SCOPUS:84877032908
SN - 1581136951
SN - 9781581136951
T3 - Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003
BT - Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003
Y2 - 15 November 2003 through 21 November 2003
ER -