Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval

Liming Wang; Xinsheng Wang; Mark Hasegawa-Johnson; Odette Scharenborg; Najim Dehak

doi:10.1109/ICASSP39728.2021.9414418

Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval

Liming Wang, Xinsheng Wang, Mark Hasegawa-Johnson, Odette Scharenborg, Najim Dehak

Research output: Contribution to journal › Conference article › peer-review

Abstract

Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.

Original language	English (US)
Pages (from-to)	7603-7607
Number of pages	5
Journal	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume	2021-June
DOIs	https://doi.org/10.1109/ICASSP39728.2021.9414418
State	Published - 2021
Event	2021 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2021 - Virtual, Toronto, Canada Duration: Jun 6 2021 → Jun 11 2021

Keywords

Language acquisition
Low-resource speech technology
Multimodal learning
Spoken term discovery

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Online availability

10.1109/ICASSP39728.2021.9414418

Library availability

Discover UIUC Full Text

Cite this

Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval. / Wang, Liming; Wang, Xinsheng; Hasegawa-Johnson, Mark et al.
In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Vol. 2021-June, 2021, p. 7603-7607.

Research output: Contribution to journal › Conference article › peer-review

@article{97f7de396fa44b079c53f11d43d73718,

title = "Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval",

abstract = "Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.",

keywords = "Language acquisition, Low-resource speech technology, Multimodal learning, Spoken term discovery",

author = "Liming Wang and Xinsheng Wang and Mark Hasegawa-Johnson and Odette Scharenborg and Najim Dehak",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE; 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2021 ; Conference date: 06-06-2021 Through 11-06-2021",

year = "2021",

doi = "10.1109/ICASSP39728.2021.9414418",

language = "English (US)",

volume = "2021-June",

pages = "7603--7607",

journal = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

issn = "1520-6149",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval

AU - Wang, Liming

AU - Wang, Xinsheng

AU - Hasegawa-Johnson, Mark

AU - Scharenborg, Odette

AU - Dehak, Najim

PY - 2021

Y1 - 2021

N2 - Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.

AB - Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.

KW - Language acquisition

KW - Low-resource speech technology

KW - Multimodal learning

KW - Spoken term discovery

UR - http://www.scopus.com/inward/record.url?scp=85112704804&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85112704804&partnerID=8YFLogxK

U2 - 10.1109/ICASSP39728.2021.9414418

DO - 10.1109/ICASSP39728.2021.9414418

M3 - Conference article

AN - SCOPUS:85112704804

SN - 1520-6149

VL - 2021-June

SP - 7603

EP - 7607

JO - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

JF - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

T2 - 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2021

Y2 - 6 June 2021 through 11 June 2021

ER -

Align or attend? Toward more efficient and accurate spoken word discovery using speech-to-image retrieval

Abstract

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this