Language Identification for Austronesian Languages

Jonathan Dunn; Wikke Nijhof

Language Identification for Austronesian Languages

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages.

Original language	English (US)
Title of host publication	Proceedings of the 13th Language Resources and Evaluation Conference
Editors	Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Jan Odijk, Stelios Piperidis
Publisher	European Language Resources Association (ELRA)
Pages	6530-6539
Number of pages	10
ISBN (Electronic)	9791095546726
State	Published - 2022
Externally published	Yes
Event	13th International Conference on Language Resources and Evaluation Conference, LREC 2022 - Marseille, France Duration: Jun 20 2022 → Jun 25 2022

Conference

Conference	13th International Conference on Language Resources and Evaluation Conference, LREC 2022
Country/Territory	France
City	Marseille
Period	6/20/22 → 6/25/22

Keywords

Austronesian languages
code-switching detection
language identification
low-resource languages

ASJC Scopus subject areas

Language and Linguistics
Library and Information Sciences
Linguistics and Language
Education

Online availability

https://aclanthology.org/2022.lrec-1.701License: CC BY-NC

Library availability

Discover UIUC Full Text

Cite this

Dunn, J., & Nijhof, W. (2022). Language Identification for Austronesian Languages. In N. Calzolari, F. Bechet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Eds.), Proceedings of the 13th Language Resources and Evaluation Conference (pp. 6530-6539). European Language Resources Association (ELRA). https://aclanthology.org/2022.lrec-1.701

Language Identification for Austronesian Languages. / Dunn, Jonathan; Nijhof, Wikke.
Proceedings of the 13th Language Resources and Evaluation Conference. ed. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2022. p. 6530-6539.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Dunn, J & Nijhof, W 2022, Language Identification for Austronesian Languages. in N Calzolari, F Bechet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, J Odijk & S Piperidis (eds), Proceedings of the 13th Language Resources and Evaluation Conference. European Language Resources Association (ELRA), pp. 6530-6539, 13th International Conference on Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 6/20/22. <https://aclanthology.org/2022.lrec-1.701>

Dunn, Jonathan ; Nijhof, Wikke. / Language Identification for Austronesian Languages. Proceedings of the 13th Language Resources and Evaluation Conference. editor / Nicoletta Calzolari ; Frederic Bechet ; Philippe Blache ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Jan Odijk ; Stelios Piperidis. European Language Resources Association (ELRA), 2022. pp. 6530-6539

@inproceedings{c5241feb0abb4c84bb47a1c11c4e43a4,

title = "Language Identification for Austronesian Languages",

abstract = "This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages.",

keywords = "Austronesian languages, code-switching detection, language identification, low-resource languages",

author = "Jonathan Dunn and Wikke Nijhof",

note = "Publisher Copyright: {\textcopyright} European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.; 13th International Conference on Language Resources and Evaluation Conference, LREC 2022 ; Conference date: 20-06-2022 Through 25-06-2022",

year = "2022",

language = "English (US)",

pages = "6530--6539",

editor = "Nicoletta Calzolari and Frederic Bechet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Jan Odijk and Stelios Piperidis",

booktitle = "Proceedings of the 13th Language Resources and Evaluation Conference",

publisher = "European Language Resources Association (ELRA)",

}

TY - GEN

T1 - Language Identification for Austronesian Languages

AU - Dunn, Jonathan

AU - Nijhof, Wikke

N1 - Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.

PY - 2022

Y1 - 2022

N2 - This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages.

AB - This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages.

KW - Austronesian languages

KW - code-switching detection

KW - language identification

KW - low-resource languages

UR - http://www.scopus.com/inward/record.url?scp=85144393713&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85144393713&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85144393713

SP - 6530

EP - 6539

BT - Proceedings of the 13th Language Resources and Evaluation Conference

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

T2 - 13th International Conference on Language Resources and Evaluation Conference, LREC 2022

Y2 - 20 June 2022 through 25 June 2022

ER -

Language Identification for Austronesian Languages

Abstract

Conference

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this