Counterfactually Fair Automatic Speech Recognition

Leda Sar, Mark Hasegawa-Johnson, Chang D. Yoo

Research output: Contribution to journalArticlepeer-review


Widelyused automatic speech recognition (ASR) systems have been empirically demonstrated in various studies to be unfair, having higher error rates for some groups of users than others. One way to define fairness in ASR is to require that changing the demographic group affiliation of any individual (e.g., changing their gender, age, education or race) should not change the probability distribution across possible speech-to-text transcriptions. In the paradigm of counterfactual fairness, all variables independent of group affiliation (e.g., the text being read by the speaker) remain unchanged, while variables dependent on group affiliation (e.g., the speaker's voice) are counterfactually modified. Hence, we approach the fairness of ASR by training the ASR to minimize change in its outcome probabilities despite a counterfactual change in the individual's demographic attributes. Starting from the individualized counterfactual equal odds criterion, we provide relaxations to it and compare their performances for connectionist temporal classification (CTC) based end-to-end ASR systems. We perform our experiments on the Corpus of Regional African American Language (CORAAL) and the LibriSpeech dataset to accommodate for differences due to gender, age, education, and race. We show that with counterfactual training, we can reduce average character error rates while achieving lower performance gap between demographic groups, and lower error standard deviation among individuals.

Original languageEnglish (US)
Pages (from-to)3515-3525
Number of pages11
JournalIEEE/ACM Transactions on Audio Speech and Language Processing
StatePublished - 2021


  • Automatic speech recognition
  • counterfactual fairness
  • fairness in machine learning
  • speaker adaptation

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Acoustics and Ultrasonics
  • Computational Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Counterfactually Fair Automatic Speech Recognition'. Together they form a unique fingerprint.

Cite this