Aluminum: An Asynchronous, GPU-Aware Communication Library Optimized for Large-Scale Training of Deep Neural Networks on HPC Systems

Keyphrases

• HPC Systems 100%
• Communication Library 100%
• Deep Neural Network 100%
• GPU-aware Communication 100%
• Large-scale Training 100%
• Performance Improvement 50%
• Latency 50%
• Multi-GPU 50%
• CUD A 50%
• Strong Scaling 50%
• Communication Cost 50%
• Allreduce 50%
• Asynchrony 50%
• Latency Bound 50%
• Semantic Mismatch 50%
• Weak Scaling 50%
• Deep Network 50%
• Training Deep Neural Networks 50%
• CUDA Stream 50%

Computer Science

• Deep Neural Network 100%
• Graphics Processing Unit 100%
• Open Source 33%
• Communication Cost 33%

Neuroscience

• Neural Network 100%