TY - GEN
T1 - A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems
AU - Harshvardhan,
AU - West, Brandon
AU - Fidel, Adam
AU - Amato, Nancy Marie
AU - Rauchwerger, Lawrence
N1 - Publisher Copyright:
© 2015 IEEE.
PY - 2015/7/17
Y1 - 2015/7/17
N2 - With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.
AB - With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.
KW - Big data
KW - out-of-core graph algorithms
KW - parallel graph processing
UR - http://www.scopus.com/inward/record.url?scp=84971469682&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84971469682&partnerID=8YFLogxK
U2 - 10.1109/IPDPS.2015.28
DO - 10.1109/IPDPS.2015.28
M3 - Conference contribution
AN - SCOPUS:84971469682
T3 - Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015
SP - 799
EP - 808
BT - Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015
Y2 - 25 May 2015 through 29 May 2015
ER -