Investigating the Representation of Open Domain Dialogue Context for Transformer Models

Vishakh Padmakumar, Behnam Hedayatnia, Di Jin, Patrick Lange, Seokhwan Kim, Nanyun Peng, Yang Liu, Dilek Hakkani-Tur

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The bulk of work adapting transformer models to open-domain dialogue represents dialogue context as the concatenated set of turns in natural language. However, it is unclear if this is the best approach. In this work, we investigate this question by means of an empirical controlled experiment varying the dialogue context format from text-only formats (all recent utterances, summaries, selected utterances) as well as variants that are more structurally different (triples, AMR). We compare these formats based on fine-tuned model performance on two downstream tasks---knowledge selection and response generation. We find that simply concatenating the utterances works as a strong baseline in most cases, but is outperformed in longer contexts by a hybrid approach of combining a summary of the context with recent utterances. Through empirical analysis, our work highlights the need to examine the format of context representation and offers recommendations on adapting general-purpose language models to dialogue tasks.
Original languageEnglish (US)
Title of host publicationProceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue
EditorsSvetlana Stoyanchev, Shafiq Joty, David Schlangen, Ondrej Dusek, Casey Kennington, Malihe Alikhani
Place of PublicationPrague, Czechia
PublisherAssociation for Computational Linguistics
Number of pages10
StatePublished - Sep 2023
Externally publishedYes


Dive into the research topics of 'Investigating the Representation of Open Domain Dialogue Context for Transformer Models'. Together they form a unique fingerprint.

Cite this