Double-cross attacks: Subverting active learning systems

Jose Rodrigo Sanchez Vicarte, Gang Wang, Christopher W. Fletcher

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Active learning is widely used in data labeling services to support real-world machine learning applications. By selecting and labeling the samples that have the highest impact on model retraining, active learning can reduce labeling efforts, and thus reduce cost. In this paper, we present a novel attack called Double Cross, which aims to manipulate data labeling and model training in active learning settings. To perform a double-cross attack, the adversary crafts inputs with a special trigger pattern and sends the triggered inputs to the victim model retraining pipeline. The goals of the triggered inputs are (1) to get selected for labeling and retraining by the victim; (2) to subsequently mislead human annotators into assigning an adversary-selected label; and (3) to change the victim model's behavior after retraining occurs. After retraining, the attack causes the victim to mislabel any samples with this trigger pattern to the adversary-chosen label. At the same time, labeling other samples, without the trigger pattern, is not affected. We develop a trigger generation method that simultaneously achieves these three goals. We evaluate the attack on multiple existing image classifiers and demonstrate that both gray-box and black-box attacks are successful. Furthermore, we perform experiments on a real-world machine learning platform (Amazon SageMaker) to evaluate the attack with human annotators in the loop, to confirm the practicality of the attack. Finally, we discuss the implications of the results and the open research questions moving forward.

Original languageEnglish (US)
Title of host publicationProceedings of the 30th USENIX Security Symposium
PublisherUSENIX Association
Number of pages18
ISBN (Electronic)9781939133243
StatePublished - 2021
Event30th USENIX Security Symposium, USENIX Security 2021 - Virtual, Online
Duration: Aug 11 2021Aug 13 2021

Publication series

NameProceedings of the 30th USENIX Security Symposium


Conference30th USENIX Security Symposium, USENIX Security 2021
CityVirtual, Online

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Safety, Risk, Reliability and Quality


Dive into the research topics of 'Double-cross attacks: Subverting active learning systems'. Together they form a unique fingerprint.

Cite this