Scientific workflow design with data assembly lines

Daniel Zinn, Shawn Bowers, Timothy McPhillips, Bertram Ludäscher

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Despite an increasing interest in scientific workflow technologies in recent years, workflow design remains a challenging, slow, and often error-prone process, thus limiting the speed of further adoption of scientific workflows. Based on practical experience with data-driven workflows, we identify and illustrate a number of recurring scientific workflow design challenges, i.e., parameter-rich functions; data assembly, disassembly, and cohesion; conditional execution; iteration; and, more generally, workflow evolution. In conventional approaches, such challenges usually lead to the introduction of different types of "shims", i.e., intermediary workflow steps that act as adapters between otherwise incorrectly wired components. However, relying heavily on the use of shims leads to brittle (i.e., change-intolerant) workflow designs that are hard to comprehend and maintain. To this end, we present a general workflow design paradigm called virtual data assembly lines (VDAL). In this paper, we show how the VDAL approach can overcome common scientific workflow design challenges and improve workflow designs by exploiting (i) a semistructured, nested data model like XML, (ii) a flexible, statically analyzable configuration mechanism (e.g., an XQuery fragment), and (iii) an underlying virtual assembly line model that is resilient to workflow and data changes. The approach has been implemented as Kepler/COMAD, and applied to improve the design of complex, real-world workflows.

Original languageEnglish (US)
Title of host publicationProceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009
DOIs
StatePublished - Dec 1 2009
Externally publishedYes
Event4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009 - Portland, OR, United States
Duration: Nov 16 2009Nov 16 2009

Publication series

NameProceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009

Other

Other4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009
CountryUnited States
CityPortland, OR
Period11/16/0911/16/09

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Scientific workflow design with data assembly lines'. Together they form a unique fingerprint.

  • Cite this

    Zinn, D., Bowers, S., McPhillips, T., & Ludäscher, B. (2009). Scientific workflow design with data assembly lines. In Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009 [1645178] (Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009). https://doi.org/10.1145/1645164.1645178