Acceleration of deep neural network training with resistive cross-point devices: Design considerations

Tayfun Gokmen, Yurii Vlasov

Research output: Contribution to journalArticle

Abstract

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Original languageEnglish (US)
Article number333
JournalFrontiers in Neuroscience
Volume10
Issue numberJUL
DOIs
StatePublished - Jan 1 2016

Fingerprint

Equipment Design
Equipment and Supplies
Weights and Measures
Microcomputers
Internet
Language
Technology
Power (Psychology)

Keywords

  • Artificial neural networks
  • Deep neural network training
  • Electronic devices
  • Machine learning
  • Materials engineering
  • Memristive devices
  • Nanotechnology
  • Synaptic device

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Acceleration of deep neural network training with resistive cross-point devices : Design considerations. / Gokmen, Tayfun; Vlasov, Yurii.

In: Frontiers in Neuroscience, Vol. 10, No. JUL, 333, 01.01.2016.

Research output: Contribution to journalArticle

@article{809b760618334b0192f2eaec7e3120e2,
title = "Acceleration of deep neural network training with resistive cross-point devices: Design considerations",
abstract = "In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.",
keywords = "Artificial neural networks, Deep neural network training, Electronic devices, Machine learning, Materials engineering, Memristive devices, Nanotechnology, Synaptic device",
author = "Tayfun Gokmen and Yurii Vlasov",
year = "2016",
month = "1",
day = "1",
doi = "10.3389/fnins.2016.00333",
language = "English (US)",
volume = "10",
journal = "Frontiers in Neuroscience",
issn = "1662-4548",
publisher = "Frontiers Research Foundation",
number = "JUL",

}

TY - JOUR

T1 - Acceleration of deep neural network training with resistive cross-point devices

T2 - Design considerations

AU - Gokmen, Tayfun

AU - Vlasov, Yurii

PY - 2016/1/1

Y1 - 2016/1/1

N2 - In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

AB - In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

KW - Artificial neural networks

KW - Deep neural network training

KW - Electronic devices

KW - Machine learning

KW - Materials engineering

KW - Memristive devices

KW - Nanotechnology

KW - Synaptic device

UR - http://www.scopus.com/inward/record.url?scp=84983247214&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84983247214&partnerID=8YFLogxK

U2 - 10.3389/fnins.2016.00333

DO - 10.3389/fnins.2016.00333

M3 - Article

AN - SCOPUS:84983247214

VL - 10

JO - Frontiers in Neuroscience

JF - Frontiers in Neuroscience

SN - 1662-4548

IS - JUL

M1 - 333

ER -