Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities

D. Doris Xin; L. Litian Ma; J. Jialin Liu; S. Stephen Macke; S. Shuchen Song; A. Aditya Parameswaran

doi:10.1145/3209889.3209896

Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities

D. Doris Xin, L. Litian Ma, J. Jialin Liu, S. Stephen Macke, S. Shuchen Song, A. Aditya Parameswaran

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a "human-in-the-loop" ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of upto 10x on typical iterative workflows against competing systems.

Original language	English (US)
Title of host publication	Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9781450358286
DOIs	https://doi.org/10.1145/3209889.3209896
State	Published - Jun 15 2018
Event	2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference - Houston, United States Duration: Jun 15 2018 → …

Publication series

Name	Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference

Conference

Conference	2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference
Country/Territory	United States
City	Houston
Period	6/15/18 → …

ASJC Scopus subject areas

Sociology and Political Science
Hardware and Architecture
Human-Computer Interaction

Online availability

10.1145/3209889.3209896

Library availability

Discover UIUC Full Text

Cite this

Xin, D. D., Ma, L. L., Liu, J. J., Macke, S. S., Song, S. S., & Parameswaran, A. A. (2018). Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities. In Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference [3209896] (Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/3209889.3209896

Accelerating Human-in-the-loop Machine Learning : Challenges and opportunities. / Xin, D. Doris; Ma, L. Litian; Liu, J. Jialin et al.

Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference. Association for Computing Machinery, Inc, 2018. 3209896 (Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Xin, DD, Ma, LL, Liu, JJ, Macke, SS, Song, SS & Parameswaran, AA 2018, Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities. in Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference., 3209896, Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference, Association for Computing Machinery, Inc, 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference, Houston, United States, 6/15/18. https://doi.org/10.1145/3209889.3209896

Xin DD, Ma LL, Liu JJ, Macke SS, Song SS, Parameswaran AA. Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities. In Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference. Association for Computing Machinery, Inc. 2018. 3209896. (Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference). doi: 10.1145/3209889.3209896

Xin, D. Doris ; Ma, L. Litian ; Liu, J. Jialin et al. / Accelerating Human-in-the-loop Machine Learning : Challenges and opportunities. Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference. Association for Computing Machinery, Inc, 2018. (Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference).

@inproceedings{764f6b7302314d7db64a8d8a8c9988a9,

title = "Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities",

abstract = "Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a {"}human-in-the-loop{"} ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of upto 10x on typical iterative workflows against competing systems.",

author = "Xin, {D. Doris} and Ma, {L. Litian} and Liu, {J. Jialin} and Macke, {S. Stephen} and Song, {S. Shuchen} and Parameswaran, {A. Aditya}",

note = "Publisher Copyright: {\textcopyright} 2018 ACM.; 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference ; Conference date: 15-06-2018",

year = "2018",

month = jun,

day = "15",

doi = "10.1145/3209889.3209896",

language = "English (US)",

series = "Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference",

publisher = "Association for Computing Machinery, Inc",

booktitle = "Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference",

}

TY - GEN

T1 - Accelerating Human-in-the-loop Machine Learning

T2 - 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference

AU - Xin, D. Doris

AU - Ma, L. Litian

AU - Liu, J. Jialin

AU - Macke, S. Stephen

AU - Song, S. Shuchen

AU - Parameswaran, A. Aditya

PY - 2018/6/15

Y1 - 2018/6/15

N2 - Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a "human-in-the-loop" ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of upto 10x on typical iterative workflows against competing systems.

AB - Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a "human-in-the-loop" ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of upto 10x on typical iterative workflows against competing systems.

UR - http://www.scopus.com/inward/record.url?scp=85055423830&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055423830&partnerID=8YFLogxK

U2 - 10.1145/3209889.3209896

DO - 10.1145/3209889.3209896

M3 - Conference contribution

AN - SCOPUS:85055423830

T3 - Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference

BT - Proceedings of the 2nd Workshop on Data Management for End-To-End Machine Learning, DEEM 2018 - In conjunction with the 2018 ACM SIGMOD/PODS Conference

PB - Association for Computing Machinery, Inc

Y2 - 15 June 2018

ER -

Accelerating Human-in-the-loop Machine Learning: Challenges and opportunities

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this