Abstract

When filling out privacy-related forms in public places such as hospitals or clinics, people usually are not aware that the sound of their handwriting leaks personal information. In this paper, we explore the possibility of eavesdropping on handwriting via nearby mobile devices based on audio signal processing and machine learning. By presenting a proof-of-concept system, WritingHacker, we show the usage of mobile devices to collect the sound of victims' handwriting, and to extract handwriting-specific features for machine learning based analysis. WritingHacker focuses on the situation where the victim's handwriting follows certain print style. An attacker can keep a mobile device, such as a common smartphone, touching the desk used by the victim to record the audio signals of handwriting. Then, the system can provide a word-level estimate for the content of the handwriting. To reduce the impacts of various writing habits and writing locations, the system utilizes the methods of letter clustering, dictionary filtering and letter time length based offsetting. Moreover, if the relative position between the device and the handwriting is known, a hand motion tracking method can be further applied to enhance the system's performance. Our prototype system's experimental results show that the accuracy of word recognition reaches around 70 - 80 percent under certain conditions, which reveals the danger of privacy leakage through the sound of handwriting.

Original languageEnglish (US)
Article number8695872
Pages (from-to)1649-1663
Number of pages15
JournalIEEE Transactions on Mobile Computing
Volume19
Issue number7
DOIs
StatePublished - Jul 1 2020

Keywords

  • Eavesdropping
  • handwriting analysis
  • signal processing

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Mobile Devices based Eavesdropping of Handwriting'. Together they form a unique fingerprint.

  • Cite this