A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems

Harshvardhan, Brandon West, Adam Fidel, Nancy Marie Amato, Lawrence Rauchwerger

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages799-808
Number of pages10
ISBN (Electronic)9781479986484
DOIs
StatePublished - Jul 17 2015
Externally publishedYes
Event29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015 - Hyderabad, India
Duration: May 25 2015May 29 2015

Publication series

NameProceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015

Other

Other29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015
CountryIndia
CityHyderabad
Period5/25/155/29/15

Keywords

  • Big data
  • out-of-core graph algorithms
  • parallel graph processing

ASJC Scopus subject areas

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems'. Together they form a unique fingerprint.

  • Cite this

    Harshvardhan, West, B., Fidel, A., Amato, N. M., & Rauchwerger, L. (2015). A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems. In Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015 (pp. 799-808). [7161566] (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2015.28