Using Lucene to index and search the digitized 1940 US Census

Research output: Contribution to journalArticlepeer-review

Abstract

An improved approach toward enabling search capabilities over large digitized document archives is described, in which Lucene indices were incorporated in a framework developed to provide automatic searchable access to the 1940 US Census, a collection composed of digitized handwritten forms. As an alternative to trying to recognize the handwritten text in the images, Word Spotting feature vectors are used to describe each cell's content. Instead of querying the system using regular ASCII text, any query is rendered as an image, and a ranked list of matching results is presented to the user. Among other preprocessing steps required by the framework, an index must be compiled to provide fast access to the feature vectors. The advantages and drawbacks of using Lucene to index these vectors instead of other indexing methods are discussed in light of the challenges confronted when dealing with digitized document collections of considerable size.

Original languageEnglish (US)
Pages (from-to)2167-2177
Number of pages11
JournalConcurrency Computation Practice and Experience
Volume26
Issue number13
DOIs
StatePublished - Sep 10 2014

Keywords

  • Lucene
  • approximate similarity search
  • content based retrieval
  • searchable access

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Computer Science Applications
  • Computer Networks and Communications
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Using Lucene to index and search the digitized 1940 US Census'. Together they form a unique fingerprint.

Cite this