DtCraft: A distributed execution engine for compute-intensive applications

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 x less development efforts over hand-crafted implementation.

Original languageEnglish (US)
Title of host publication2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages757-765
Number of pages9
ISBN (Electronic)9781538630938
DOIs
StatePublished - Dec 13 2017
Event36th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017 - Irvine, United States
Duration: Nov 13 2017Nov 16 2017

Publication series

NameIEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD
Volume2017-November
ISSN (Print)1092-3152

Other

Other36th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
CountryUnited States
CityIrvine
Period11/13/1711/16/17

Fingerprint

Engines
Concurrency control
Distributed computer systems
Fault tolerance
Electric sparks

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Cite this

Huang, T-W., Lin, C. X., & Wong, M. D. F. (2017). DtCraft: A distributed execution engine for compute-intensive applications. In 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017 (pp. 757-765). (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD; Vol. 2017-November). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCAD.2017.8203853

DtCraft : A distributed execution engine for compute-intensive applications. / Huang, Tsung-Wei; Lin, Chun Xun; Wong, Martin D F.

2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 757-765 (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD; Vol. 2017-November).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Huang, T-W, Lin, CX & Wong, MDF 2017, DtCraft: A distributed execution engine for compute-intensive applications. in 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017. IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD, vol. 2017-November, Institute of Electrical and Electronics Engineers Inc., pp. 757-765, 36th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017, Irvine, United States, 11/13/17. https://doi.org/10.1109/ICCAD.2017.8203853
Huang T-W, Lin CX, Wong MDF. DtCraft: A distributed execution engine for compute-intensive applications. In 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 757-765. (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD). https://doi.org/10.1109/ICCAD.2017.8203853
Huang, Tsung-Wei ; Lin, Chun Xun ; Wong, Martin D F. / DtCraft : A distributed execution engine for compute-intensive applications. 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 757-765 (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD).
@inproceedings{3fd85a6c86074e80a14943384a202ff2,
title = "DtCraft: A distributed execution engine for compute-intensive applications",
abstract = "Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 x less development efforts over hand-crafted implementation.",
author = "Tsung-Wei Huang and Lin, {Chun Xun} and Wong, {Martin D F}",
year = "2017",
month = "12",
day = "13",
doi = "10.1109/ICCAD.2017.8203853",
language = "English (US)",
series = "IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "757--765",
booktitle = "2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017",
address = "United States",

}

TY - GEN

T1 - DtCraft

T2 - A distributed execution engine for compute-intensive applications

AU - Huang, Tsung-Wei

AU - Lin, Chun Xun

AU - Wong, Martin D F

PY - 2017/12/13

Y1 - 2017/12/13

N2 - Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 x less development efforts over hand-crafted implementation.

AB - Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 x less development efforts over hand-crafted implementation.

UR - http://www.scopus.com/inward/record.url?scp=85043521519&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85043521519&partnerID=8YFLogxK

U2 - 10.1109/ICCAD.2017.8203853

DO - 10.1109/ICCAD.2017.8203853

M3 - Conference contribution

AN - SCOPUS:85043521519

T3 - IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD

SP - 757

EP - 765

BT - 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -