The Gutenberg-HathiTrust Parallel Corpus:  A Real-World Dataset for Noise Investigation in Uncorrected OCR Texts

Ming Jiang; Yuerong Hu; Glen Cameron Layne-Worthey; Ryan Dubnicek; Boris Capitanu; Deren E. Kudeki; J. Stephen Downie

doi:http://hdl.handle.net/2142/109695

The Gutenberg-HathiTrust Parallel Corpus: A Real-World Dataset for Noise Investigation in Uncorrected OCR Texts

Ming Jiang, Yuerong Hu, Glen Cameron Layne-Worthey, Ryan Dubnicek, Boris Capitanu, Deren E. Kudeki, J. Stephen Downie

Research output: Contribution to conference › Paper › peer-review

Abstract

This paper proposes large-scale parallel corpora of English-language publications for exploring the effects of optical character recognition (OCR) errors in the scanned text of digitized library collections on various corpus-based research. We collected data from: (1) Project Gutenberg (Gutenberg) for a human-proofread clean corpus; and, (2) HathiTrust Digital Library (HathiTrust) for an uncorrected OCR-impacted corpus. Our data is parallel regarding the content. So far as we know, this is the first large-scale benchmark dataset intended to evaluate the effects of text noise in digital libraries. In total, we collected and aligned 19,049 pairs of uncorrected OCR-impacted and human-proofread books in six domains published from 1780 to 1993.

Original language	English (US)
DOIs	https://doi.org/http://hdl.handle.net/2142/109695
State	Published - Mar 17 2021
Event	iConference - Duration: Mar 17 2021 → Mar 31 2021

Conference

Conference	iConference
Period	3/17/21 → 3/31/21

Keywords

Parallel Text Dataset
Optical Character Recognition
Digital Library
Digital Humanities
Data Curation

Online availability

http://hdl.handle.net/2142/109695

Library availability

Discover UIUC Full Text

Cite this

@conference{4d0e5050162946ccacacaf153e0b5778,

title = "The Gutenberg-HathiTrust Parallel Corpus: A Real-World Dataset for Noise Investigation in Uncorrected OCR Texts",

abstract = "This paper proposes large-scale parallel corpora of English-language publications for exploring the effects of optical character recognition (OCR) errors in the scanned text of digitized library collections on various corpus-based research. We collected data from: (1) Project Gutenberg (Gutenberg) for a human-proofread clean corpus; and, (2) HathiTrust Digital Library (HathiTrust) for an uncorrected OCR-impacted corpus. Our data is parallel regarding the content. So far as we know, this is the first large-scale benchmark dataset intended to evaluate the effects of text noise in digital libraries. In total, we collected and aligned 19,049 pairs of uncorrected OCR-impacted and human-proofread books in six domains published from 1780 to 1993.",

keywords = "Parallel Text Dataset, Optical Character Recognition, Digital Library, Digital Humanities, Data Curation",

author = "Ming Jiang and Yuerong Hu and Layne-Worthey, {Glen Cameron} and Ryan Dubnicek and Boris Capitanu and Kudeki, {Deren E.} and Downie, {J. Stephen}",

year = "2021",

month = mar,

day = "17",

doi = "http://hdl.handle.net/2142/109695",

language = "English (US)",

note = "iConference ; Conference date: 17-03-2021 Through 31-03-2021",

}

TY - CONF

T1 - The Gutenberg-HathiTrust Parallel Corpus: A Real-World Dataset for Noise Investigation in Uncorrected OCR Texts

AU - Jiang, Ming

AU - Hu, Yuerong

AU - Layne-Worthey, Glen Cameron

AU - Dubnicek, Ryan

AU - Capitanu, Boris

AU - Kudeki, Deren E.

AU - Downie, J. Stephen

PY - 2021/3/17

Y1 - 2021/3/17

N2 - This paper proposes large-scale parallel corpora of English-language publications for exploring the effects of optical character recognition (OCR) errors in the scanned text of digitized library collections on various corpus-based research. We collected data from: (1) Project Gutenberg (Gutenberg) for a human-proofread clean corpus; and, (2) HathiTrust Digital Library (HathiTrust) for an uncorrected OCR-impacted corpus. Our data is parallel regarding the content. So far as we know, this is the first large-scale benchmark dataset intended to evaluate the effects of text noise in digital libraries. In total, we collected and aligned 19,049 pairs of uncorrected OCR-impacted and human-proofread books in six domains published from 1780 to 1993.

AB - This paper proposes large-scale parallel corpora of English-language publications for exploring the effects of optical character recognition (OCR) errors in the scanned text of digitized library collections on various corpus-based research. We collected data from: (1) Project Gutenberg (Gutenberg) for a human-proofread clean corpus; and, (2) HathiTrust Digital Library (HathiTrust) for an uncorrected OCR-impacted corpus. Our data is parallel regarding the content. So far as we know, this is the first large-scale benchmark dataset intended to evaluate the effects of text noise in digital libraries. In total, we collected and aligned 19,049 pairs of uncorrected OCR-impacted and human-proofread books in six domains published from 1780 to 1993.

KW - Parallel Text Dataset

KW - Optical Character Recognition

KW - Digital Library

KW - Digital Humanities

KW - Data Curation

U2 - http://hdl.handle.net/2142/109695

DO - http://hdl.handle.net/2142/109695

M3 - Paper

T2 - iConference

Y2 - 17 March 2021 through 31 March 2021

ER -