No plan survives contact: Experience with cybercrime measurement

An important mode of empirical security research involves analyzing the behavior, capabilities, and motives of adversaries. By definition, such measurements cannot be conducted in controlled settings and require “engagement” directly with adversaries, their infrastructure or their ecosystem. However, the operational complexities required to successfully carry out such measurements are significant and rarely documented; blacklisting, payment instruments, fraud controls and contact management all represent real challenges in such studies. In this paper, we document our experiences conducting such measurements over five years (covering a range of distinct studies) and distill effective operational practices for others who might conduct similar experiments in the future.