DtCraft: A High-Performance Distributed Execution Engine at Scale

Tsung Wei Huang, Chun Xun Lin, Martin D.F. Wong

Research output: Contribution to journalArticlepeer-review

Abstract

Recent years have seen rapid growth in data-driven distributed systems, such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C++ based distributed execution engine to streamline the development of high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details, such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown the promising performance from single multicore machines to clusters of computers. In a particular semiconductor design problem, we achieved 30 × speedup with 40 nodes and 15 × less development efforts over hand-crafted implementation.

Original languageEnglish (US)
Article number8355904
Pages (from-to)1070-1083
Number of pages14
JournalIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Volume38
Issue number6
DOIs
StatePublished - Jun 2019

Keywords

  • Distributed computing
  • parallel programming

ASJC Scopus subject areas

  • Software
  • Computer Graphics and Computer-Aided Design
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'DtCraft: A High-Performance Distributed Execution Engine at Scale'. Together they form a unique fingerprint.

Cite this