Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT

Bhavya Bhavya, Jinjun Xiong, Cheng Xiang Zhai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose a novel application of prompting Pre-trained Language Models (PLMs) to generate analogies and study how to design effective prompts for two task settings: generating a source concept analogous to a given target concept (aka Analogous Concept Generation or ACG), and generating an explanation of the similarity between a given pair of target concept and source concept (aka Analogous Explanation Generation or AEG). We found that it is feasible to prompt InstructGPT to generate meaningful analogies and the best prompts tend to be precise imperative statements especially with a low temperature setting. We also systematically analyzed the sensitivity of the InstructGPT model to prompt design, temperature, and injected spelling errors, and found that the model is particularly sensitive to certain variations (e.g., questions vs. imperative statements). Further, we conducted human evaluation on 1.4k of the generated analogies and found that the quality of generations varies substantially by model size. The largest InstructGPT model can achieve human-level performance at generating meaningful analogies for a given target while there is still room for improvement on the AEG task.

Original languageEnglish (US)
Title of host publication15th International Natural Language Generation Conference, INLG 2022
EditorsSamira Shaikh, Thiago Castro Ferreira, Amanda Stent
PublisherAssociation for Computational Linguistics (ACL)
Pages298-312
Number of pages15
ISBN (Electronic)9781955917575
DOIs
StatePublished - 2022
Externally publishedYes
Event15th International Natural Language Generation Conference, INLG 2022 - Hybrid, Waterville, United States
Duration: Jul 18 2022Jul 22 2022

Publication series

Name15th International Natural Language Generation Conference, INLG 2022

Conference

Conference15th International Natural Language Generation Conference, INLG 2022
Country/TerritoryUnited States
CityHybrid, Waterville
Period7/18/227/22/22

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Information Systems
  • Software

Fingerprint

Dive into the research topics of 'Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT'. Together they form a unique fingerprint.

Cite this