Scientific workflow management and the Kepler system

Bertram Ludäscher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A. Lee, Jing Tao, Yang Zhao

Research output: Contribution to journalArticlepeer-review

Abstract

Many scientific disciplines are now data and information driven, and new scientific knowledge is often gained by scientists putting together data analysis and knowledge discovery 'pipelines'. A related trend is that more and more scientific communities realize the benefits of sharing their data and computational services, and are thus contributing to a distributed data and computational community infrastructure (a.k.a. 'the Grid'). However, this infrastructure is only a means to an end and ideally scientists should not be too concerned with its existence. The goal is for scientists to focus on development and use of what we call scientific workflows. These are networks of analytical steps that may involve, e.g., database access and querying steps, data analysis and mining steps, and many other steps including computationally intensive jobs on high-performance cluster computers. In this paper we describe characteristics of and requirements for scientific workflows as identified in a number of our application projects. We then elaborate on Kepler, a particular scientific workflow system, currently under development across a number of scientific data management projects. We describe some key features of Kepler and its underlying Ptolemy II system, planned extensions, and areas of future research. Kepler is a community-driven, open source project, and we always welcome related projects and new contributors to join.

Original languageEnglish (US)
Pages (from-to)1039-1065
Number of pages27
JournalConcurrency and Computation: Practice and Experience
Volume18
Issue number10
DOIs
StatePublished - Aug 25 2006
Externally publishedYes

Keywords

  • Dataflow networks
  • Grid workflows
  • Problem-solving environments
  • Scientific data management
  • Scientific workflows

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Computer Networks and Communications
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Scientific workflow management and the Kepler system'. Together they form a unique fingerprint.

Cite this