A malware detector placement game for intrusion detection

We propose and investigate a game-theoretic approach to the malware filtering and detector placement problem which arises in network security. Our main objective is to develop optimal detector algorithms taking into account attacker strategies and actions. Assuming rational and intelligent attackers, we present a two-person zero-sum non-cooperative Markov security game framework as a basis for modeling the interaction between the attackers who generate malware traffic on a network and a corresponding intrusion detection system (IDS). Thus, we establish a formal model of the detector placement problem based on game theory and derive optimal strategies for both players. In addition, we test the strategies obtained in a realistic agent-based network simulation environment and compare the results of static and dynamic placement scenarios. The obtained IDS strategies and the corresponding simulation results provide interesting insights into how to optimally deploy malware detectors in a network environment.