Pandas: Robust Locality-Aware Scheduling with Stochastic Delay Optimality

Qiaomin Xie, Mayank Pundir, Yi Lu, Cristina L. Abad, Roy H. Campbell

Research output: Contribution to journalArticle

Abstract

Data locality is a fundamental problem to data-parallel applications where data-processing tasks consume different amounts of time and resources at different locations. The problem is especially prominent under stressed conditions such as hot spots. While replication based on data popularity relieves hot spots due to contention for a single file, hot spots caused by skewed node popularity, due to contention for files co-located with each other, are more complex, unpredictable, hence more difficult to deal with. We propose Pandas, a light-weight acceleration engine for data-processing tasks that is robust to changes in load and skewness in node popularity. Pandas is a stochastic delay-optimal algorithm. Trace-driven experiments on Hadoop show that Pandas accelerates the data-processing phase of jobs by 11 times with hot spots and 2.4 times without hot spots over existing schedulers. When the difference in processing times due to location is large, such as applicable to the case of memory-locality, the acceleration by Pandas is 22 times.

Original languageEnglish (US)
Pages (from-to)662-675
Number of pages14
JournalIEEE/ACM Transactions on Networking
Volume25
Issue number2
DOIs
StatePublished - Apr 2017

    Fingerprint

Keywords

  • Data processing systems
  • MapReduce
  • hot-spot mitigation
  • locality-aware scheduling

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this