Workflow task clustering for best effort systems with Pegasus

Gurmeet Singh, Mei Hui Su, Karan Vahi, Ewa Deelman, Bruce Berriman, John Good, Daniel S. Katz, Gaurang Mehta

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many scientific workflows are composed of fine computational granularity tasks, yet they are composed of thousands of them and are data intensive in nature, thus requiring resources such as the TeraGrid to execute efficiently. In order to improve the performance of such applications, we often employ task clustering techniques to increase the computational granularity of workflow tasks. The goal is to minimize the completion time of the workflow by reducing the impact of queue wait times. In this paper, we examine the performance impact of the clustering techniques using the Pegasus workflow management system. Experiments performed using an astronomy workflow on the NCSA TeraGrid cluster show that clustering can achieve a significant reduction in the workflow completion time (up to 97%).

Original languageEnglish (US)
Title of host publicationProceedings of the 15th ACM Mardi Gras Conference, MG '08
DOIs
StatePublished - Dec 1 2008
Externally publishedYes
Event15th ACM Mardi Gras Conference, MG '08 - Baton Rouge, LA, United States
Duration: Jan 29 2008Feb 3 2008

Publication series

NameACM International Conference Proceeding Series
Volume320

Other

Other15th ACM Mardi Gras Conference, MG '08
CountryUnited States
CityBaton Rouge, LA
Period1/29/082/3/08

Keywords

  • best effort systems
  • queue wait time
  • task clustering
  • workflow clustering

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Workflow task clustering for best effort systems with Pegasus'. Together they form a unique fingerprint.

Cite this