Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

Research output: Chapter in Book/Report/Conference proceedingConference contribution


This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.
Original languageEnglish (US)
Title of host publicationProceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects
EditorsMarcos Zampieri, Preslav Nakov, Nikola Ljubešić, Jörg Tiedemann, Yves Scherrer, Tommi Jauhiainen
PublisherAssociation for Computational Linguistics
StatePublished - Apr 2021
Externally publishedYes


Dive into the research topics of 'Representations of Language Varieties Are Reliable Given Corpus Similarity Measures'. Together they form a unique fingerprint.

Cite this