A general-purpose distributed programming system using data-parallel streams

Tsung Wei Huang, Chun Xun Lin, Guannan Guo, Martin D.F. Wong

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

Original languageEnglish (US)
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages1360-1363
Number of pages4
ISBN (Electronic)9781450356657
DOIs
StatePublished - Oct 15 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: Oct 22 2018Oct 26 2018

Publication series

NameMM 2018 - Proceedings of the 2018 ACM Multimedia Conference

Other

Other26th ACM Multimedia conference, MM 2018
CountryKorea, Republic of
CitySeoul
Period10/22/1810/26/18

    Fingerprint

Keywords

  • Distributed system
  • Machine learning
  • Stream processing

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Huang, T. W., Lin, C. X., Guo, G., & Wong, M. D. F. (2018). A general-purpose distributed programming system using data-parallel streams. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference (pp. 1360-1363). (MM 2018 - Proceedings of the 2018 ACM Multimedia Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/3240508.3243654