WritingHacker: Audio based eavesdropping of handwriting via mobile devices

Tuo Yu, Haiming Jin, Klara Nahrstedt

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

When filling out privacy-related forms in public places such as hospitals or clinics, people usually are not aware that the sound of their handwriting leaks personal information. In this paper, we explore the possibility of eavesdropping on handwriting via nearby mobile devices based on audio signal processing and machine learning. By presenting a proof-ofconcept system, WritingHacker, we show the usage of mobile devices to collect the sound of victims' handwriting, and to extract handwriting-specific features for machine learning based analysis. WritingHacker focuses on the situation where the victim's handwriting follows certain print style. An attacker can keep a mobile device, such as a common smartphone, touching the desk used by the victim to record the audio signals of handwriting. Then the system can provide a word-level estimate for the content of the handwriting. To reduce the impacts of various writing habits and writing locations, the system utilizes the methods of letter clustering and dictionary filtering. Our prototype system's experimental results show that the accuracy of word recognition reaches around 50% - 60% under certain conditions, which reveals the danger of privacy leakage through the sound of handwriting.

Original languageEnglish (US)
Title of host publicationUbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing
PublisherAssociation for Computing Machinery, Inc
Pages463-473
Number of pages11
ISBN (Electronic)9781450344616
DOIs
StatePublished - Sep 12 2016
Event2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016 - Heidelberg, Germany
Duration: Sep 12 2016Sep 16 2016

Publication series

NameUbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Other

Other2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016
CountryGermany
CityHeidelberg
Period9/12/169/16/16

Keywords

  • Audio signals
  • Eavesdropping
  • Handwriting

ASJC Scopus subject areas

  • Information Systems
  • Hardware and Architecture
  • Computer Networks and Communications
  • Software
  • Human-Computer Interaction

Fingerprint Dive into the research topics of 'WritingHacker: Audio based eavesdropping of handwriting via mobile devices'. Together they form a unique fingerprint.

  • Cite this

    Yu, T., Jin, H., & Nahrstedt, K. (2016). WritingHacker: Audio based eavesdropping of handwriting via mobile devices. In UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 463-473). (UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/2971648.2971681