PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

Jianhao Shen; Chenguang Wang; Ye Yuan; Jiawei Han; Heng Ji; Koushik Sen; Ming Zhang; Dawn Song

doi:10.18653/v1/2022.findings-emnlp.281

PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

Jianhao Shen
, Chenguang Wang
, Ye Yuan
, Jiawei Han
, Heng Ji
, Koushik Sen
, Ming Zhang
, Dawn Song

Research output: Contribution to conference › Paper › peer-review

Abstract

This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a “fill-in-the-blank” task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters.

Original language	English (US)
Pages	3862-3876
Number of pages	15
DOIs	https://doi.org/10.18653/v1/2022.findings-emnlp.281
State	Published - 2022
Event	2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: Dec 7 2022 → Dec 11 2022

Conference

Conference	2022 Findings of the Association for Computational Linguistics: EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	12/7/22 → 12/11/22

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems

Online availability

10.18653/v1/2022.findings-emnlp.281

Library availability

Discover UIUC Full Text

Cite this

Shen, J, Wang, C, Yuan, Y, Han, J , Ji, H, Sen, K, Zhang, M & Song, D 2022, 'PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion', Paper presented at 2022 Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, 12/7/22 - 12/11/22 pp. 3862-3876. https://doi.org/10.18653/v1/2022.findings-emnlp.281

@conference{d7a86b0df6684ae98d25768867e6e898,

title = "PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion",

abstract = "This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a {\textquotedblleft}fill-in-the-blank{\textquotedblright} task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1\% of the parameters.",

author = "Jianhao Shen and Chenguang Wang and Ye Yuan and Jiawei Han and Heng Ji and Koushik Sen and Ming Zhang and Dawn Song",

note = "We would like to thank the anonymous reviewers for their suggestions and comments. This paper is partially supported by National Key Research and Development Program of China with Grant No. 2018AAA0101902 and the National Natural Science Foundation of China (NSFC Grant Numbers 62106008 and 62276002). This material is in part based upon work supported by Berkeley DeepDrive and Berkeley Artificial Intelligence Research. The research was also supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and INCAS Program No. HR001121C0165, National Science Foundation IIS-19-56151, IIS-17-41317, and IIS 17-04532.; 2022 Findings of the Association for Computational Linguistics: EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.findings-emnlp.281",

language = "English (US)",

pages = "3862--3876",

}

TY - CONF

T1 - PALT

T2 - 2022 Findings of the Association for Computational Linguistics: EMNLP 2022

AU - Shen, Jianhao

AU - Wang, Chenguang

AU - Yuan, Ye

AU - Han, Jiawei

AU - Ji, Heng

AU - Sen, Koushik

AU - Zhang, Ming

AU - Song, Dawn

N1 - We would like to thank the anonymous reviewers for their suggestions and comments. This paper is partially supported by National Key Research and Development Program of China with Grant No. 2018AAA0101902 and the National Natural Science Foundation of China (NSFC Grant Numbers 62106008 and 62276002). This material is in part based upon work supported by Berkeley DeepDrive and Berkeley Artificial Intelligence Research. The research was also supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and INCAS Program No. HR001121C0165, National Science Foundation IIS-19-56151, IIS-17-41317, and IIS 17-04532.

PY - 2022

Y1 - 2022

N2 - This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a “fill-in-the-blank” task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters.

AB - This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a “fill-in-the-blank” task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters.

UR - https://www.scopus.com/pages/publications/85149889015

UR - https://www.scopus.com/pages/publications/85149889015#tab=citedBy

U2 - 10.18653/v1/2022.findings-emnlp.281

DO - 10.18653/v1/2022.findings-emnlp.281

M3 - Paper

AN - SCOPUS:85149889015

SP - 3862

EP - 3876

Y2 - 7 December 2022 through 11 December 2022

ER -

PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

Abstract

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this