DtCraft: A distributed execution engine for compute-intensive applications

Tsung Wei Huang, Chun Xun Lin, Martin D.F. Wong

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 x less development efforts over hand-crafted implementation.

Original languageEnglish (US)
Title of host publication2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages757-765
Number of pages9
ISBN (Electronic)9781538630938
DOIs
StatePublished - Dec 13 2017
Event36th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017 - Irvine, United States
Duration: Nov 13 2017Nov 16 2017

Publication series

NameIEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD
Volume2017-November
ISSN (Print)1092-3152

Other

Other36th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
CountryUnited States
CityIrvine
Period11/13/1711/16/17

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Fingerprint Dive into the research topics of 'DtCraft: A distributed execution engine for compute-intensive applications'. Together they form a unique fingerprint.

  • Cite this

    Huang, T. W., Lin, C. X., & Wong, M. D. F. (2017). DtCraft: A distributed execution engine for compute-intensive applications. In 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017 (pp. 757-765). (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD; Vol. 2017-November). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCAD.2017.8203853