HogWild++: A new mechanism for decentralized asynchronous stochastic gradient descent

Huan Zhang, Cho Jui Hsieh, Venkatesh Akella

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Stochastic Gradient Descent (SGD) is a popular technique for solving large-scale machine learning problems. In order to parallelize SGD on multi-core machines, asynchronous SGD (Hogwild) has been proposed, where each core updates a global model vector stored in a shared memory simultaneously, without using explicit locks. We show that the scalability of Hogwild on modern multi-socket CPUs is severely limited, especially on NUMA (Non-Uniform Memory Access) system, due to the excessive cache invalidation requests and false sharing. In this paper we propose a novel decentralized asynchronous SGD algorithm called HogWild++ that overcomes these drawbacks and shows almost linear speedup on multi-socket NUMA systems. The main idea in HogWild++ is to replace the global model vector with a set of local model vectors that are shared by a cluster (a set of cores), keep them synchronized through a decentralized token-based protocol that minimizes remote memory access conflicts and ensures convergence. We present the design and experimental evaluation of HogWild++ on a variety of datasets and show that it outperforms state-of-The-Art parallel SGD implementations in terms of efficiency and scalability.

Original languageEnglish (US)
Title of host publicationProceedings - 16th IEEE International Conference on Data Mining, ICDM 2016
EditorsFrancesco Bonchi, Xindong Wu, Ricardo Baeza-Yates, Josep Domingo-Ferrer, Zhi-Hua Zhou
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages629-638
Number of pages10
ISBN (Electronic)9781509054725
DOIs
StatePublished - Jan 31 2017
Externally publishedYes
Event16th IEEE International Conference on Data Mining, ICDM 2016 - Barcelona, Catalonia, Spain
Duration: Dec 12 2016Dec 15 2016

Publication series

NameProceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)1550-4786

Other

Other16th IEEE International Conference on Data Mining, ICDM 2016
Country/TerritorySpain
CityBarcelona, Catalonia
Period12/12/1612/15/16

Keywords

  • Decentralized algorithm
  • Non-uniform memory access (NUMA) architecture
  • Stochastic gradient descent

ASJC Scopus subject areas

  • General Engineering

Fingerprint

Dive into the research topics of 'HogWild++: A new mechanism for decentralized asynchronous stochastic gradient descent'. Together they form a unique fingerprint.

Cite this