A Formal Characterization of Activation Functions in Deep Neural Networks

Massi Amrouche; Dusan M. Stipanovic

doi:10.1109/TNNLS.2022.3187538

A Formal Characterization of Activation Functions in Deep Neural Networks

Research output: Contribution to journal › Article › peer-review

Abstract

In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.

Original language	English (US)
Pages (from-to)	2153-2166
Number of pages	14
Journal	IEEE Transactions on Neural Networks and Learning Systems
Volume	35
Issue number	2
DOIs	https://doi.org/10.1109/TNNLS.2022.3187538
State	Published - Feb 1 2024

Keywords

Artificial neural networks
deep learning
feed-forward neural networks
partial differential equations (PDEs)

ASJC Scopus subject areas

Software
Artificial Intelligence
Computer Networks and Communications
Computer Science Applications

Online availability

10.1109/TNNLS.2022.3187538

Library availability

Discover UIUC Full Text

Cite this

@article{d93b4c84fb1d4f04ad360600437a8240,

title = "A Formal Characterization of Activation Functions in Deep Neural Networks",

abstract = "In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.",

keywords = "Artificial neural networks, deep learning, feed-forward neural networks, partial differential equations (PDEs)",

author = "Massi Amrouche and Stipanovic, {Dusan M.}",

note = "Publisher Copyright: {\textcopyright} 2012 IEEE.",

year = "2024",

month = feb,

day = "1",

doi = "10.1109/TNNLS.2022.3187538",

language = "English (US)",

volume = "35",

pages = "2153--2166",

journal = "IEEE Transactions on Neural Networks and Learning Systems",

issn = "2162-237X",

publisher = "IEEE Computational Intelligence Society",

number = "2",

}

TY - JOUR

T1 - A Formal Characterization of Activation Functions in Deep Neural Networks

AU - Amrouche, Massi

AU - Stipanovic, Dusan M.

PY - 2024/2/1

Y1 - 2024/2/1

N2 - In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.

AB - In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.

KW - Artificial neural networks

KW - deep learning

KW - feed-forward neural networks

KW - partial differential equations (PDEs)

UR - http://www.scopus.com/inward/record.url?scp=85135242621&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85135242621&partnerID=8YFLogxK

U2 - 10.1109/TNNLS.2022.3187538

DO - 10.1109/TNNLS.2022.3187538

M3 - Article

C2 - 35839202

AN - SCOPUS:85135242621

SN - 2162-237X

VL - 35

SP - 2153

EP - 2166

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

IS - 2

ER -

A Formal Characterization of Activation Functions in Deep Neural Networks

Abstract

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this