Role of Bias Terms in Dot-Product Attention

Mahdi Namazifar, Devamanyu Hazarika, Dilek Hakkani-Tur

Research output: Contribution to journalConference articlepeer-review

Abstract

Dot-product attention is a core module in the present generation of neural network models, particularly transformers, and is being leveraged across numerous areas such as natural language processing and computer vision. This attention module is comprised of three linear transformations, namely query, key, and value linear transformations, each of which has a bias term. In this work, we study the role of these bias terms, and mathematically show that the bias term of the key linear transformation is redundant and could be omitted without any impact on the attention module. Moreover, we argue that the bias term of the value linear transformation has a more prominent role than that of the bias term of the query linear transformation. We empirically verify these findings through multiple experiments on language modeling, natural language understanding, and natural language generation tasks.

Keywords

  • Attention
  • Softmax
  • Transformer

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Role of Bias Terms in Dot-Product Attention'. Together they form a unique fingerprint.

Cite this