A map-reduce based framework for heterogeneous processing element cluster environments

Yu Shyang Tan, Bu Sung Lee, Bingsheng He, Roy H. Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes.

Original languageEnglish (US)
Title of host publicationProceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012
Pages57-64
Number of pages8
DOIs
StatePublished - Jul 16 2012
Event12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012 - Ottawa, ON, Canada
Duration: May 13 2012May 16 2012

Publication series

NameProceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012

Other

Other12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012
CountryCanada
CityOttawa, ON
Period5/13/125/16/12

Fingerprint

MapReduce
Vertex of a graph
Processing
Distributed computer systems
Configuration
Completion Time
Program processors
Distributed Computing
Framework
Large Data Sets
Queue
Requirements
Demonstrate
Experiment
Experiments

Keywords

  • GPGPU
  • Heterogeneous resource framework
  • MapReduce

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Applied Mathematics

Cite this

Tan, Y. S., Lee, B. S., He, B., & Campbell, R. H. (2012). A map-reduce based framework for heterogeneous processing element cluster environments. In Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012 (pp. 57-64). [6217405] (Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012). https://doi.org/10.1109/CCGrid.2012.35

A map-reduce based framework for heterogeneous processing element cluster environments. / Tan, Yu Shyang; Lee, Bu Sung; He, Bingsheng; Campbell, Roy H.

Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012. 2012. p. 57-64 6217405 (Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Tan, YS, Lee, BS, He, B & Campbell, RH 2012, A map-reduce based framework for heterogeneous processing element cluster environments. in Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012., 6217405, Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012, pp. 57-64, 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012, Ottawa, ON, Canada, 5/13/12. https://doi.org/10.1109/CCGrid.2012.35
Tan YS, Lee BS, He B, Campbell RH. A map-reduce based framework for heterogeneous processing element cluster environments. In Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012. 2012. p. 57-64. 6217405. (Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012). https://doi.org/10.1109/CCGrid.2012.35
Tan, Yu Shyang ; Lee, Bu Sung ; He, Bingsheng ; Campbell, Roy H. / A map-reduce based framework for heterogeneous processing element cluster environments. Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012. 2012. pp. 57-64 (Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012).
@inproceedings{db8d97e26cef43b6a4ed9e8c54d9a903,
title = "A map-reduce based framework for heterogeneous processing element cluster environments",
abstract = "In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes.",
keywords = "GPGPU, Heterogeneous resource framework, MapReduce",
author = "Tan, {Yu Shyang} and Lee, {Bu Sung} and Bingsheng He and Campbell, {Roy H.}",
year = "2012",
month = "7",
day = "16",
doi = "10.1109/CCGrid.2012.35",
language = "English (US)",
isbn = "9780769546919",
series = "Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012",
pages = "57--64",
booktitle = "Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012",

}

TY - GEN

T1 - A map-reduce based framework for heterogeneous processing element cluster environments

AU - Tan, Yu Shyang

AU - Lee, Bu Sung

AU - He, Bingsheng

AU - Campbell, Roy H.

PY - 2012/7/16

Y1 - 2012/7/16

N2 - In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes.

AB - In this paper, we present our design of a Processing Element (PE) Aware MapReduce base framework, Pamar. Pamar is designed for supporting distributed computing on clusters where node PE configurations are asymmetric on different nodes. Pamar's main goal is to allow users to seamlessly utilize different kinds of processing elements (e.g., CPUs or GPUs) collaboratively for large scale data processing. To show proof of concept, we have incorporated our designs into the Hadoop framework and tested it on cluster environments having asymmetric node PE configurations. We demonstrate Pamar's ability to identify PEs available on each node and match-make user jobs with nodes, base on job PE requirements. Pamar allows users to easily parallelize applications across large datasets and at the same time utilizes different PEs for processing different classes of functions efficiently. The experiments show improvement in job queue completion time with Pamar over clusters with asymmetric nodes as compared to clusters with symmetric nodes.

KW - GPGPU

KW - Heterogeneous resource framework

KW - MapReduce

UR - http://www.scopus.com/inward/record.url?scp=84863662127&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863662127&partnerID=8YFLogxK

U2 - 10.1109/CCGrid.2012.35

DO - 10.1109/CCGrid.2012.35

M3 - Conference contribution

AN - SCOPUS:84863662127

SN - 9780769546919

T3 - Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012

SP - 57

EP - 64

BT - Proceedings - 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2012

ER -