Data reduction for the scalable automated analysis of distributed darknet traffic

Michael Bailey, Evan Cooke, Farnam Jahanian, Niels Provos, Karl Rosaen, David Watson

Research output: Contribution to conferencePaperpeer-review


Threats to the privacy of users and to the availability of Internet infrastructure are evolving at a tremendous rate. To characterize these emerging threats, researchers must effectively balance monitoring the large number of hosts needed to quickly build confidence in new attacks, while still preserving the detail required to differentiate these attacks. One class of techniques that attempts to achieve this balance involves hybrid systems that combine the scalable monitoring of unused address blocks (or darknets) with forensic honeypots (or honeyfarms). In this paper we examine the properties of individual and distributed dark-nets to determine the effectiveness of building scalable hybrid systems. We show that individual darknets are dominated by a small number of sources repeating the same actions. This enables source-based techniques to be effective at reducing the number of connections to be evaluated by over 90%. We demonstrate that the dominance of locally targeted attack behavior and the limited life of random scanning hosts result in few of these sources being repeated across darknets. To achieve reductions beyond source-based approaches, we look to source-distribution based methods and expand them to include notions of local and global behavior. We show that this approach is effective at reducing the number of events by deploying it in 30 production networks during early 2005. Each of the identified events during this period represented a major globally-scoped attack including the WINS vulnerability scanning, Veritas Backup Agent vulnerability scanning, and the MySQL Worm.

Original languageEnglish (US)
Number of pages14
StatePublished - 2005
Externally publishedYes
Event5th ACM SIGCOMM Conference on Internet Measurement, IMC 2005 - Berkeley, CA, United States
Duration: Oct 19 2005Oct 21 2005


Other5th ACM SIGCOMM Conference on Internet Measurement, IMC 2005
Country/TerritoryUnited States
CityBerkeley, CA

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications


Dive into the research topics of 'Data reduction for the scalable automated analysis of distributed darknet traffic'. Together they form a unique fingerprint.

Cite this