Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Ananthan Nambiar; Maeve Heflin; Simon Liu; Sergei Maslov; Mark Hopkins; Anna Ritz

doi:10.1101/2020.06.15.153643

Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Ananthan Nambiar, Maeve Heflin, Simon Liu, Sergei Maslov, Mark Hopkins, Anna Ritz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-Trains task-Agnostic sequence representations. This model is fine-Tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-The-Art approaches for protein family classification while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-Tuning the pre-Trained sequence representations for other protein prediction tasks.

Original language	English (US)
Title of host publication	Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450379649
DOIs	https://doi.org/10.1101/2020.06.15.153643 https://doi.org/10.1145/3388440.3412467
State	Published - Sep 21 2020
Event	11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020 - Virtual, Online, United States Duration: Sep 21 2020 → Sep 24 2020

Publication series

Name	Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020

Conference

Conference	11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020
Country/Territory	United States
City	Virtual, Online
Period	9/21/20 → 9/24/20

Keywords

Neural networks
protein family classification
protein-protein interaction prediction
COVID-19

ASJC Scopus subject areas

Computer Science Applications
Software
Biomedical Engineering
Health Informatics

Online availability

10.1101/2020.06.15.153643License: CC BY
10.1145/3388440.3412467

Library availability

Discover UIUC Full Text

Cite this

Nambiar, A., Heflin, M., Liu, S., Maslov, S., Hopkins, M., & Ritz, A. (2020). Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020 (Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020). Association for Computing Machinery. https://doi.org/10.1101/2020.06.15.153643, https://doi.org/10.1145/3388440.3412467

Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. / Nambiar, Ananthan; Heflin, Maeve; Liu, Simon et al.
Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020. Association for Computing Machinery, 2020. (Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Nambiar, A, Heflin, M, Liu, S, Maslov, S, Hopkins, M & Ritz, A 2020, Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. in Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020. Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020, Association for Computing Machinery, 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020, Virtual, Online, United States, 9/21/20. https://doi.org/10.1101/2020.06.15.153643, https://doi.org/10.1145/3388440.3412467

Nambiar A, Heflin M, Liu S, Maslov S, Hopkins M, Ritz A. Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020. Association for Computing Machinery. 2020. (Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020). doi: 10.1101/2020.06.15.153643, 10.1145/3388440.3412467

Nambiar, Ananthan ; Heflin, Maeve ; Liu, Simon et al. / Transforming the Language of Life : Transformer Neural Networks for Protein Prediction Tasks. Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020. Association for Computing Machinery, 2020. (Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020).

@inproceedings{1acc8648adf84071b78291b7c5a07f29,

title = "Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks",

abstract = "The scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-Trains task-Agnostic sequence representations. This model is fine-Tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-The-Art approaches for protein family classification while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-Tuning the pre-Trained sequence representations for other protein prediction tasks.",

keywords = "Neural networks, protein family classification, protein-protein interaction prediction, COVID-19",

author = "Ananthan Nambiar and Maeve Heflin and Simon Liu and Sergei Maslov and Mark Hopkins and Anna Ritz",

note = "Funding Information: This work has been supported by the National Science Foundation (awards #1750981 and #1725729). This work has also been partially supported by the Google Cloud Platform research credits program (to AR, MH, and AN). AN would like to thank Mark Bedau, Norman Packard and the Reed College Artificial Life Lab for insightful discussions and Desiree Odgers for inspiring the idea of taking a linguistic approach to a biological problem.; 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020 ; Conference date: 21-09-2020 Through 24-09-2020",

year = "2020",

month = sep,

day = "21",

doi = "10.1101/2020.06.15.153643",

language = "English (US)",

series = "Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020",

publisher = "Association for Computing Machinery",

booktitle = "Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020",

address = "United States",

}

TY - GEN

T1 - Transforming the Language of Life

T2 - 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020

AU - Nambiar, Ananthan

AU - Heflin, Maeve

AU - Liu, Simon

AU - Maslov, Sergei

AU - Hopkins, Mark

AU - Ritz, Anna

N1 - Funding Information: This work has been supported by the National Science Foundation (awards #1750981 and #1725729). This work has also been partially supported by the Google Cloud Platform research credits program (to AR, MH, and AN). AN would like to thank Mark Bedau, Norman Packard and the Reed College Artificial Life Lab for insightful discussions and Desiree Odgers for inspiring the idea of taking a linguistic approach to a biological problem.

PY - 2020/9/21

Y1 - 2020/9/21

N2 - The scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-Trains task-Agnostic sequence representations. This model is fine-Tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-The-Art approaches for protein family classification while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-Tuning the pre-Trained sequence representations for other protein prediction tasks.

AB - The scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-Trains task-Agnostic sequence representations. This model is fine-Tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-The-Art approaches for protein family classification while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-Tuning the pre-Trained sequence representations for other protein prediction tasks.

KW - Neural networks

KW - protein family classification

KW - protein-protein interaction prediction

KW - COVID-19

UR - http://www.scopus.com/inward/record.url?scp=85096966907&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85096966907&partnerID=8YFLogxK

U2 - 10.1101/2020.06.15.153643

DO - 10.1101/2020.06.15.153643

M3 - Conference contribution

AN - SCOPUS:85096966907

T3 - Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020

BT - Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020

PB - Association for Computing Machinery

Y2 - 21 September 2020 through 24 September 2020

ER -

Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this