Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

Jonathan E Dunn

Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.

Original language	English (US)
Title of host publication	Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects
Editors	Marcos Zampieri, Preslav Nakov, Nikola Ljubešić, Jörg Tiedemann, Yves Scherrer, Tommi Jauhiainen
Publisher	Association for Computational Linguistics
Pages	28-38
State	Published - Apr 2021
Externally published	Yes

Online availability

https://aclanthology.org/2021.vardial-1.4/License: CC BY

Library availability

Discover UIUC Full Text

Cite this

Dunn, J. E. (2021). Representations of Language Varieties Are Reliable Given Corpus Similarity Measures. In M. Zampieri, P. Nakov, N. Ljubešić, J. Tiedemann, Y. Scherrer, & T. Jauhiainen (Eds.), Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects (pp. 28-38). Association for Computational Linguistics. https://aclanthology.org/2021.vardial-1.4/

Representations of Language Varieties Are Reliable Given Corpus Similarity Measures. / Dunn, Jonathan E.
Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects. ed. / Marcos Zampieri; Preslav Nakov; Nikola Ljubešić; Jörg Tiedemann; Yves Scherrer; Tommi Jauhiainen. Association for Computational Linguistics, 2021. p. 28-38.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Dunn, JE 2021, Representations of Language Varieties Are Reliable Given Corpus Similarity Measures. in M Zampieri, P Nakov, N Ljubešić, J Tiedemann, Y Scherrer & T Jauhiainen (eds), Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics, pp. 28-38. <https://aclanthology.org/2021.vardial-1.4/>

@inproceedings{1fbdb0ebb8fe495aa9ded90af67f38b3,

title = "Representations of Language Varieties Are Reliable Given Corpus Similarity Measures",

abstract = "This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.",

author = "Dunn, {Jonathan E}",

year = "2021",

month = apr,

language = "English (US)",

pages = "28--38",

editor = "Marcos Zampieri and Preslav Nakov and Nikola Ljube{\v s}i{\'c} and J{\"o}rg Tiedemann and Yves Scherrer and Tommi Jauhiainen",

booktitle = "Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects",

publisher = "Association for Computational Linguistics",

}

TY - GEN

T1 - Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

AU - Dunn, Jonathan E

PY - 2021/4

Y1 - 2021/4

N2 - This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.

AB - This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.

M3 - Conference contribution

SP - 28

EP - 38

BT - Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects

A2 - Zampieri, Marcos

A2 - Nakov, Preslav

A2 - Ljubešić, Nikola

A2 - Tiedemann, Jörg

A2 - Scherrer, Yves

A2 - Jauhiainen, Tommi

PB - Association for Computational Linguistics

ER -

Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

Abstract

Online availability

Library availability

Fingerprint

Cite this