Streaming satellite data to cloud workflows for on-demand computing of environmental data products

Daniel Zinn, Quinn Hart, Bertram Ludaescher, Yogesh Simmhan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Environmental data arriving constantly from satellites and weather stations are used to compute weather coefficients that are essential for agriculture and viticulture. For example, the reference evapotranspiration (ET0) coefficient, overlaid on regional maps, is provided each day by the California Department of Water Resources to local farmers and turf managers to plan daily water use. Scaling out single-processor compute/data intensive applications operating on realtime data to support more users and higher-resolution data poses data engineering challenges. Cloud computing helps data providers expand resource capacity to meet growing needs besides supporting scientific needs like reprocessing historic data using new models. In this article, we examine migration of a legacy script used for daily ET0 computation by CIMIS to a workflow model that eases deployment to and scaling on the Windows Azure Cloud. Our architecture incorporates a direct streaming model into Cloud virtual machines (VMs) that improves the performance by 130% to 160% for our workflow over using Cloud storage for data staging, used commonly. The streaming workflows achieve runtimes comparable to desktop execution for single VMs and a linear speed-up when using multiple VMs, thus allowing computation of environmental coefficients at a much larger resolution than done presently.

Original languageEnglish (US)
Title of host publication2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
DOIs
StatePublished - Dec 1 2010
Externally publishedYes
Event2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 - New Orleans, LA, United States
Duration: Nov 14 2010Nov 14 2010

Publication series

Name2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

Other

Other2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
CountryUnited States
CityNew Orleans, LA
Period11/14/1011/14/10

ASJC Scopus subject areas

  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Streaming satellite data to cloud workflows for on-demand computing of environmental data products'. Together they form a unique fingerprint.

  • Cite this

    Zinn, D., Hart, Q., Ludaescher, B., & Simmhan, Y. (2010). Streaming satellite data to cloud workflows for on-demand computing of environmental data products. In 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 [5671841] (2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010). https://doi.org/10.1109/WORKS.2010.5671841