A game theoretical framework for inter-process adversarial intervention detection

Muhammed O. Sayin, Hossein Hosseini, Radha Poovendran, Tamer Başar

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In this paper, we propose and analyze a two-level game theoretical framework to detect advanced and persistent threats across processes. The two-level framework adapted facilitates abstraction of the complexity of process level interactions between defense mechanisms and adversaries from easier to interpret and more flexible system-level interaction. At the process-level, program anomaly detection algorithms have already been proposed to detect anomalous program behavior by comparing monitored activities with the predetermined expected behavior. This had led to significant detection performance initially until advanced adversaries modified the attacks to remain undetected. Therefore, we propose defense mechanisms that anticipate the reaction of advanced evaders and seek to maximize the complexity of undetectable attacks at the expense of additional false alarm rate. Furthermore, in the system-level, we propose defense mechanisms to detect adversarial intervention across processes through the assessment of all process activities together in a cohesive way so that the advanced adversaries need to craft their attacks further to remain undetected also at the system-level. This further increases the cost of complexity for the attacker, and correspondingly degrades the motivation to attack. We provide a game theoretical incentive analysis for both defenders and adversaries, and characterize pure and mixed strategy equilibria. We also analyze the coupling between the two levels of the game.

Original languageEnglish (US)
Title of host publicationDecision and Game Theory for Security - 9th International Conference, GameSec 2018, Proceedings
EditorsLinda Bushnell, Radha Poovendran, Tamer Basar
Number of pages22
ISBN (Print)9783030015534
StatePublished - 2018
Event9th International Conference on Decision and Game Theory for Security, GameSec 2018 - Seattle, United States
Duration: Oct 29 2018Oct 31 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11199 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other9th International Conference on Decision and Game Theory for Security, GameSec 2018
Country/TerritoryUnited States


  • Advanced persistent threats
  • Anomaly detection
  • Games-in-games
  • Host-based intrusion detection
  • Mimicry attacks
  • Process monitoring
  • Stackelberg games

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'A game theoretical framework for inter-process adversarial intervention detection'. Together they form a unique fingerprint.

Cite this