A parallel input-output system for resolving spatial data challenges: An agent-based model case study

Eric Shook, Shaowen Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With recent advances in data collection technologies such as remote sensing and global positioning systems, the amount of spatial data being produced has been increasing at a staggering rate. Simultaneously, a shift is being experienced in computing from single-core to multi-core processors. To effectively utilize the computational power afforded by these new generation of processors for serving data-intensive geospatial applications, parallel computing techniques need to be employed. Parallel computing, however, raises new challenges associated with handling the input and output of spatial data in parallel. This paper describes a Parallel Input/Output System (PIOS) to address challenges associated with handling large amounts of diverse spatial data. The PIOS is based on a hierarchical structure that uses a scalable file partitioning strategy and combines data and metadata to enable efficient handling of terabyte-scale data sets in parallel. A spatially-explicit agent-based model is developed as a case study. Computational experiments were conducted on a supercomputer supported by the National Science Foundation. PIOS achieved ten times speedup in parallel input/output time, and was demonstrated to efficiently scale to over one thousand processing cores and handle multiple terabytes of data.

Original languageEnglish (US)
Title of host publicationProceedings of the ACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011
Pages18-25
Number of pages8
DOIs
StatePublished - Dec 19 2011
EventACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011 - Chicago, IL, United States
Duration: Nov 1 2011Nov 1 2011

Publication series

NameProceedings of the ACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011

Other

OtherACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011
CountryUnited States
CityChicago, IL
Period11/1/1111/1/11

    Fingerprint

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Networks and Communications
  • Information Systems

Cite this

Shook, E., & Wang, S. (2011). A parallel input-output system for resolving spatial data challenges: An agent-based model case study. In Proceedings of the ACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011 (pp. 18-25). (Proceedings of the ACM SIGSPATIAL 2nd International Workshop on High Performance and Distributed Geographic Information Systems, ACM SIGSPATIAL HPDGIS 2011). https://doi.org/10.1145/2070770.2070773