Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition

Liming Wang; Junrui Ni; Heting Gao; Jialu Li; Kai Chieh Chang; Xulin Fan; Junkai Wu; Mark Hasegawa-Johnson; Chang D. Yoo

Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition

Liming Wang, Junrui Ni, Heting Gao, Jialu Li, Kai Chieh Chang, Xulin Fan, Junkai Wu, Mark Hasegawa-Johnson, Chang D. Yoo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2signU, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at cactuswiththoughts/UnsupSpeech2Sign.git.

Original language	English (US)
Title of host publication	Findings of the Association for Computational Linguistics, ACL 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	6785-6800
Number of pages	16
ISBN (Electronic)	9781959429623
State	Published - 2023
Event	61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 - Toronto, Canada Duration: Jul 9 2023 → Jul 14 2023

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	61st Annual Meeting of the Association for Computational Linguistics, ACL 2023
Country/Territory	Canada
City	Toronto
Period	7/9/23 → 7/14/23

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

Library availability

Discover UIUC Full Text

Cite this

Wang, L., Ni, J., Gao, H., Li, J., Chang, K. C., Fan, X., Wu, J., Hasegawa-Johnson, M., & Yoo, C. D. (2023). Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition. In Findings of the Association for Computational Linguistics, ACL 2023 (pp. 6785-6800). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL).

Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition. / Wang, Liming; Ni, Junrui; Gao, Heting et al.
Findings of the Association for Computational Linguistics, ACL 2023. Association for Computational Linguistics (ACL), 2023. p. 6785-6800 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Wang, L, Ni, J, Gao, H, Li, J, Chang, KC, Fan, X, Wu, J, Hasegawa-Johnson, M & Yoo, CD 2023, Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition. in Findings of the Association for Computational Linguistics, ACL 2023. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 6785-6800, 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, Toronto, Canada, 7/9/23.

@inproceedings{0b923060f1e544e8a79aa63939ac2fe6,

title = "Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition",

abstract = "Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2signU, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at cactuswiththoughts/UnsupSpeech2Sign.git.",

author = "Liming Wang and Junrui Ni and Heting Gao and Jialu Li and Chang, {Kai Chieh} and Xulin Fan and Junkai Wu and Mark Hasegawa-Johnson and Yoo, {Chang D.}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 ; Conference date: 09-07-2023 Through 14-07-2023",

year = "2023",

language = "English (US)",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "6785--6800",

booktitle = "Findings of the Association for Computational Linguistics, ACL 2023",

}

TY - GEN

T1 - Listen, Decipher and Sign

T2 - 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023

AU - Wang, Liming

AU - Ni, Junrui

AU - Gao, Heting

AU - Li, Jialu

AU - Chang, Kai Chieh

AU - Fan, Xulin

AU - Wu, Junkai

AU - Hasegawa-Johnson, Mark

AU - Yoo, Chang D.

PY - 2023

Y1 - 2023

N2 - Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2signU, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at cactuswiththoughts/UnsupSpeech2Sign.git.

AB - Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2signU, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at cactuswiththoughts/UnsupSpeech2Sign.git.

UR - http://www.scopus.com/inward/record.url?scp=85175446459&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85175446459&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85175446459

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 6785

EP - 6800

BT - Findings of the Association for Computational Linguistics, ACL 2023

PB - Association for Computational Linguistics (ACL)

Y2 - 9 July 2023 through 14 July 2023

ER -

Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition

Abstract

Publication series

Conference

ASJC Scopus subject areas

Library availability

Related links

Fingerprint

Cite this