Theius: A streaming visualization suite for hadoop clusters

Jon Tedesco, Roman Dudko, Abhishek Sharma, Reza Farivar, Roy Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

As cloud computing clusters continue to grow, maintaining the health of these clusters becomes increasingly challenging. Recent work has studied how we can efficiently monitor the status of machines in these clusters and how we can detect problems or predict them before they occur, yet little work has focused on addressing the bottleneck between when these failures occur and when they are fixed: system administrators. As monitoring and failure detection systems mature, we are able to extract tremendous amounts of information about the status of the system in real time. However, this amount of data is difficult to understand for human beings, especially those inexperienced with the particular cluster. In this paper, we introduce a web-based visualization suite called Theius to allow system administrators to quickly understand the state of the cloud system as a whole. We outline the key features of this visualization tool, and show that it is more intuitive and easy to use than Ganglia, a state-of-the-art visualization tool for clusters. Likewise, we demonstrate that our tool can scale, presenting a use case with our visualization showing a 5000 node cluster. Although our tool is implemented for Hadoop clusters, our contribution is general to any cloud computing system.

Original languageEnglish (US)
Title of host publicationProceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013
Pages177-182
Number of pages6
DOIs
StatePublished - Aug 12 2013
Event1st IEEE International Conference on Cloud Engineering, IC2E 2013 - San Francisco, CA, United States
Duration: Mar 25 2013Mar 28 2013

Publication series

NameProceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013

Other

Other1st IEEE International Conference on Cloud Engineering, IC2E 2013
CountryUnited States
CitySan Francisco, CA
Period3/25/133/28/13

Fingerprint

Visualization
Cloud computing
Health
Monitoring

Keywords

  • Cloud computing
  • Cluster computing
  • Failure detection
  • Failure prediction
  • Hadoop
  • Monitoring
  • Visualization

ASJC Scopus subject areas

  • Software

Cite this

Tedesco, J., Dudko, R., Sharma, A., Farivar, R., & Campbell, R. (2013). Theius: A streaming visualization suite for hadoop clusters. In Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013 (pp. 177-182). [6529282] (Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013). https://doi.org/10.1109/IC2E.2013.36

Theius : A streaming visualization suite for hadoop clusters. / Tedesco, Jon; Dudko, Roman; Sharma, Abhishek; Farivar, Reza; Campbell, Roy.

Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013. 2013. p. 177-182 6529282 (Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Tedesco, J, Dudko, R, Sharma, A, Farivar, R & Campbell, R 2013, Theius: A streaming visualization suite for hadoop clusters. in Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013., 6529282, Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013, pp. 177-182, 1st IEEE International Conference on Cloud Engineering, IC2E 2013, San Francisco, CA, United States, 3/25/13. https://doi.org/10.1109/IC2E.2013.36
Tedesco J, Dudko R, Sharma A, Farivar R, Campbell R. Theius: A streaming visualization suite for hadoop clusters. In Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013. 2013. p. 177-182. 6529282. (Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013). https://doi.org/10.1109/IC2E.2013.36
Tedesco, Jon ; Dudko, Roman ; Sharma, Abhishek ; Farivar, Reza ; Campbell, Roy. / Theius : A streaming visualization suite for hadoop clusters. Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013. 2013. pp. 177-182 (Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013).
@inproceedings{23dd4e9a52b9466182941542ef0513f6,
title = "Theius: A streaming visualization suite for hadoop clusters",
abstract = "As cloud computing clusters continue to grow, maintaining the health of these clusters becomes increasingly challenging. Recent work has studied how we can efficiently monitor the status of machines in these clusters and how we can detect problems or predict them before they occur, yet little work has focused on addressing the bottleneck between when these failures occur and when they are fixed: system administrators. As monitoring and failure detection systems mature, we are able to extract tremendous amounts of information about the status of the system in real time. However, this amount of data is difficult to understand for human beings, especially those inexperienced with the particular cluster. In this paper, we introduce a web-based visualization suite called Theius to allow system administrators to quickly understand the state of the cloud system as a whole. We outline the key features of this visualization tool, and show that it is more intuitive and easy to use than Ganglia, a state-of-the-art visualization tool for clusters. Likewise, we demonstrate that our tool can scale, presenting a use case with our visualization showing a 5000 node cluster. Although our tool is implemented for Hadoop clusters, our contribution is general to any cloud computing system.",
keywords = "Cloud computing, Cluster computing, Failure detection, Failure prediction, Hadoop, Monitoring, Visualization",
author = "Jon Tedesco and Roman Dudko and Abhishek Sharma and Reza Farivar and Roy Campbell",
year = "2013",
month = "8",
day = "12",
doi = "10.1109/IC2E.2013.36",
language = "English (US)",
isbn = "9780769549453",
series = "Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013",
pages = "177--182",
booktitle = "Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013",

}

TY - GEN

T1 - Theius

T2 - A streaming visualization suite for hadoop clusters

AU - Tedesco, Jon

AU - Dudko, Roman

AU - Sharma, Abhishek

AU - Farivar, Reza

AU - Campbell, Roy

PY - 2013/8/12

Y1 - 2013/8/12

N2 - As cloud computing clusters continue to grow, maintaining the health of these clusters becomes increasingly challenging. Recent work has studied how we can efficiently monitor the status of machines in these clusters and how we can detect problems or predict them before they occur, yet little work has focused on addressing the bottleneck between when these failures occur and when they are fixed: system administrators. As monitoring and failure detection systems mature, we are able to extract tremendous amounts of information about the status of the system in real time. However, this amount of data is difficult to understand for human beings, especially those inexperienced with the particular cluster. In this paper, we introduce a web-based visualization suite called Theius to allow system administrators to quickly understand the state of the cloud system as a whole. We outline the key features of this visualization tool, and show that it is more intuitive and easy to use than Ganglia, a state-of-the-art visualization tool for clusters. Likewise, we demonstrate that our tool can scale, presenting a use case with our visualization showing a 5000 node cluster. Although our tool is implemented for Hadoop clusters, our contribution is general to any cloud computing system.

AB - As cloud computing clusters continue to grow, maintaining the health of these clusters becomes increasingly challenging. Recent work has studied how we can efficiently monitor the status of machines in these clusters and how we can detect problems or predict them before they occur, yet little work has focused on addressing the bottleneck between when these failures occur and when they are fixed: system administrators. As monitoring and failure detection systems mature, we are able to extract tremendous amounts of information about the status of the system in real time. However, this amount of data is difficult to understand for human beings, especially those inexperienced with the particular cluster. In this paper, we introduce a web-based visualization suite called Theius to allow system administrators to quickly understand the state of the cloud system as a whole. We outline the key features of this visualization tool, and show that it is more intuitive and easy to use than Ganglia, a state-of-the-art visualization tool for clusters. Likewise, we demonstrate that our tool can scale, presenting a use case with our visualization showing a 5000 node cluster. Although our tool is implemented for Hadoop clusters, our contribution is general to any cloud computing system.

KW - Cloud computing

KW - Cluster computing

KW - Failure detection

KW - Failure prediction

KW - Hadoop

KW - Monitoring

KW - Visualization

UR - http://www.scopus.com/inward/record.url?scp=84881144234&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84881144234&partnerID=8YFLogxK

U2 - 10.1109/IC2E.2013.36

DO - 10.1109/IC2E.2013.36

M3 - Conference contribution

AN - SCOPUS:84881144234

SN - 9780769549453

T3 - Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013

SP - 177

EP - 182

BT - Proceedings of the IEEE International Conference on Cloud Engineering, IC2E 2013

ER -