A general-purpose distributed programming system using data-parallel streams

Tsung-Wei Huang, Chun Xun Lin, Guannan Guo, Martin D F Wong

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

Original languageEnglish (US)
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages1360-1363
Number of pages4
ISBN (Electronic)9781450356657
DOIs
StatePublished - Oct 15 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: Oct 22 2018Oct 26 2018

Publication series

NameMM 2018 - Proceedings of the 2018 ACM Multimedia Conference

Other

Other26th ACM Multimedia conference, MM 2018
CountryKorea, Republic of
CitySeoul
Period10/22/1810/26/18

Fingerprint

Computer systems programming
Cluster computing
Concurrency control
Multimedia systems
Containers
Learning systems
Engines
Controllers
Processing
Industry
Linux

Keywords

  • Distributed system
  • Machine learning
  • Stream processing

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Huang, T-W., Lin, C. X., Guo, G., & Wong, M. D. F. (2018). A general-purpose distributed programming system using data-parallel streams. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference (pp. 1360-1363). (MM 2018 - Proceedings of the 2018 ACM Multimedia Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/3240508.3243654

A general-purpose distributed programming system using data-parallel streams. / Huang, Tsung-Wei; Lin, Chun Xun; Guo, Guannan; Wong, Martin D F.

MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2018. p. 1360-1363 (MM 2018 - Proceedings of the 2018 ACM Multimedia Conference).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Huang, T-W, Lin, CX, Guo, G & Wong, MDF 2018, A general-purpose distributed programming system using data-parallel streams. in MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. MM 2018 - Proceedings of the 2018 ACM Multimedia Conference, Association for Computing Machinery, Inc, pp. 1360-1363, 26th ACM Multimedia conference, MM 2018, Seoul, Korea, Republic of, 10/22/18. https://doi.org/10.1145/3240508.3243654
Huang T-W, Lin CX, Guo G, Wong MDF. A general-purpose distributed programming system using data-parallel streams. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc. 2018. p. 1360-1363. (MM 2018 - Proceedings of the 2018 ACM Multimedia Conference). https://doi.org/10.1145/3240508.3243654
Huang, Tsung-Wei ; Lin, Chun Xun ; Guo, Guannan ; Wong, Martin D F. / A general-purpose distributed programming system using data-parallel streams. MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2018. pp. 1360-1363 (MM 2018 - Proceedings of the 2018 ACM Multimedia Conference).
@inproceedings{c33657bed84c4b998ed4fc89ea45cb1a,
title = "A general-purpose distributed programming system using data-parallel streams",
abstract = "In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.",
keywords = "Distributed system, Machine learning, Stream processing",
author = "Tsung-Wei Huang and Lin, {Chun Xun} and Guannan Guo and Wong, {Martin D F}",
year = "2018",
month = "10",
day = "15",
doi = "10.1145/3240508.3243654",
language = "English (US)",
series = "MM 2018 - Proceedings of the 2018 ACM Multimedia Conference",
publisher = "Association for Computing Machinery, Inc",
pages = "1360--1363",
booktitle = "MM 2018 - Proceedings of the 2018 ACM Multimedia Conference",

}

TY - GEN

T1 - A general-purpose distributed programming system using data-parallel streams

AU - Huang, Tsung-Wei

AU - Lin, Chun Xun

AU - Guo, Guannan

AU - Wong, Martin D F

PY - 2018/10/15

Y1 - 2018/10/15

N2 - In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

AB - In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

KW - Distributed system

KW - Machine learning

KW - Stream processing

UR - http://www.scopus.com/inward/record.url?scp=85058211774&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058211774&partnerID=8YFLogxK

U2 - 10.1145/3240508.3243654

DO - 10.1145/3240508.3243654

M3 - Conference contribution

AN - SCOPUS:85058211774

T3 - MM 2018 - Proceedings of the 2018 ACM Multimedia Conference

SP - 1360

EP - 1363

BT - MM 2018 - Proceedings of the 2018 ACM Multimedia Conference

PB - Association for Computing Machinery, Inc

ER -