Data-Intensive Supercomputing in the Cloud: Global Analytics for Satellite Imagery

Michael S. Warren, Samuel W. Skillman, Rick Chartrand, Tim Kelton, Ryan Keisler, David Raleigh, Matthew J Turk

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present our experiences using cloud computing to support data-intensive analytics on satellite imagery for commercial applications. Drawing from our background in highperformance computing, we draw parallels between the early days of clustered computing systems and the current state of cloud computing and its potential to disrupt the HPC market. Using our own virtual file system layer on top of cloud remote object storage, we demonstrate aggregate read bandwidth of 230 gigabytes per second using 512 Google Compute Engine (GCE) nodes accessing a USA multi-region standard storage bucket. This figure is comparable to the best HPC storage systems in existence. We also present several of our application results, including the identification of field boundaries in Ukraine, and the generation of a global cloud-free base layer from Landsat imagery.

Original languageEnglish (US)
Title of host publicationProceedings of DataCloud 2016
Subtitle of host publication7th International Workshop on Data-Intensive Computing in the Clouds - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages24-31
Number of pages8
ISBN (Electronic)9781509061587
DOIs
StatePublished - Feb 6 2017
Event7th International Workshop on Data-Intensive Computing in the Clouds, DataCloud 2016 - Salt Lake City, United States
Duration: Nov 14 2016 → …

Publication series

NameProceedings of DataCloud 2016: 7th International Workshop on Data-Intensive Computing in the Clouds - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis

Other

Other7th International Workshop on Data-Intensive Computing in the Clouds, DataCloud 2016
CountryUnited States
CitySalt Lake City
Period11/14/16 → …

    Fingerprint

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications

Cite this

Warren, M. S., Skillman, S. W., Chartrand, R., Kelton, T., Keisler, R., Raleigh, D., & Turk, M. J. (2017). Data-Intensive Supercomputing in the Cloud: Global Analytics for Satellite Imagery. In Proceedings of DataCloud 2016: 7th International Workshop on Data-Intensive Computing in the Clouds - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 24-31). [7845278] (Proceedings of DataCloud 2016: 7th International Workshop on Data-Intensive Computing in the Clouds - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DataCloud.2016.007