TY - GEN
T1 - Experiences in network and data transfer across large virtual organizations - a retrospective
AU - Benninger, Kathy
AU - Jordan, Chris
AU - Lambert, Michael
AU - Samuel, Tabitha
AU - Liming, Lee
AU - Wheeler, David
N1 - The Extreme Science and Engineering Discovery Environment (XSEDE) project was born of a need to serve the growing national community of academic researchers through a single unified interface to national academic computing resources. Following and building on the success and lessons learned through NSF’s Ter-aGrid programs (2001–2005 and 2005–2012), the XSEDE award was funded by the NSF [17] in 2011. During the term of the first award, from 1 July 2011 to 31 December 2016, XSEDE established itself as the most advanced, powerful, and robust collection of integrated digital research resources and services in the world. XSEDE integrated and coordinated advanced digital services within the national cyberinfrastructure (CI) ecosystem to support contemporary science. This ecosystem continues to involve a highly distributed, yet integrated and coordinated, assemblage of software, supercomputers, visualization systems, storage systems, networks, portals and gateways, collections of data, instruments, and personnel with specific expertise. Due to the substantial successes of the XSEDE project, a follow-on award (often referred to as “XSEDE2” to distinguish it from the initial award) was made to continue the project in 2016 [18]. The mission of XSEDE2 was to “exist to enhance the productivity of a growing community of scholars, researchers, and engineers through access to advanced digital services that support open research by coordinating and adding value to the leading CI resources funded by the NSF and other agencies” [23]. XSEDE2 is governed by the following overarching goals:
This work used the Extreme Science and Engineering Discovery Environment (XSEDE), supported by National Science Foundation grant number ACI-1548562. We also thank the XSEDE senior management team, current and former members of the XSEDE Data Transfer Services, Internet2, and Globus teams for their support and contributions.
PY - 2022/7/8
Y1 - 2022/7/8
N2 - The XSEDE Data Transfer Services (DTS) group focuses on streamlining and improving the data transfer experiences of the national academic research community, while also buttressing and future-proofing the underlying networks that support these transfers. In this paper, the DTS group shares how network and data transfer technologies have evolved over the past six years, with the backdrop of the Distributed Terascale Facility (DTF) and TeraGrid projects that served the national community before the advent of XSEDE. We delve into improvements, challenges, and trends in network and data transfer technologies, and the uses of these technologies in academic institutions across the country, which today translate into 100s of users of CI moving many terabytes each month. We also review the key lessons learned while serving the community in this regard, and what the future holds for academic networking and data transfer.
AB - The XSEDE Data Transfer Services (DTS) group focuses on streamlining and improving the data transfer experiences of the national academic research community, while also buttressing and future-proofing the underlying networks that support these transfers. In this paper, the DTS group shares how network and data transfer technologies have evolved over the past six years, with the backdrop of the Distributed Terascale Facility (DTF) and TeraGrid projects that served the national community before the advent of XSEDE. We delve into improvements, challenges, and trends in network and data transfer technologies, and the uses of these technologies in academic institutions across the country, which today translate into 100s of users of CI moving many terabytes each month. We also review the key lessons learned while serving the community in this regard, and what the future holds for academic networking and data transfer.
KW - data transfer experience
KW - file transfer
KW - monitoring
KW - network performance
UR - http://www.scopus.com/inward/record.url?scp=85135214885&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85135214885&partnerID=8YFLogxK
U2 - 10.1145/3491418.3530763
DO - 10.1145/3491418.3530763
M3 - Conference contribution
AN - SCOPUS:85135214885
T3 - PEARC 2022 Conference Series - Practice and Experience in Advanced Research Computing 2022 - Revolutionary: Computing, Connections, You
BT - PEARC 2022 Conference Series - Practice and Experience in Advanced Research Computing 2022 - Revolutionary
PB - Association for Computing Machinery
T2 - 2022 Conference on Practice and Experience in Advanced Research Computing: Revolutionary: Computing, Connections, You, PEARC 2022
Y2 - 10 July 2022 through 14 July 2022
ER -