Detecting traditional packers, decisively

Denis Bueno, Kevin J. Compton, Karem A. Sakallah, Michael Bailey

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Many of the important decidability results in malware analysis are based Turing machine models of computation. We exhibit computational models which use more realistic assumptions about machine and attacker resources. While seminal results such as [1-5] remain true for Turing machines, we show under more realistic assumptions, important tasks are decidable instead of undecidable. Specifically, we show that detecting traditional malware unpacking behavior - in which a payload is decompressed or decrypted and subsequently executed - is decidable under our assumptions. We then examine the issue of dealing with complex but decidable problems. We look for lessons from the hardware verification community, which has been striving to meet the challenge of intractable problems for the past three decades.

Original languageEnglish (US)
Title of host publicationResearch in Attacks, Intrusions, and Defenses - 16th International Symposium, RAID 2013, Proceedings
Number of pages20
StatePublished - 2013
Externally publishedYes
Event16th International Symposium on Research in Attacks, Intrusions, and Defenses, RAID 2013 - Rodney Bay, Saint Lucia
Duration: Oct 23 2013Oct 25 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8145 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other16th International Symposium on Research in Attacks, Intrusions, and Defenses, RAID 2013
Country/TerritorySaint Lucia
CityRodney Bay

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Detecting traditional packers, decisively'. Together they form a unique fingerprint.

Cite this