Generating request streams on Big Data using clustered renewal processes

Cristina L. Abad, Mindi Yuan, Chris X. Cai, Yi Lu, Nathan Roberts, Roy H. Campbell

Research output: Contribution to journalArticlepeer-review


The performance evaluation of large file systems, such as storage and media streaming, motivates scalable generation of representative traces. We focus on two key characteristics of traces, popularity and temporal locality. The common practice of using a system-wide distribution obscures per-object behavior, which is important for system evaluation. We propose a model based on delayed renewal processes which, by sampling interarrival times for each object, accurately reproduces popularity and temporal locality for the trace. A lightweight version reduces the dimension of the model with statistical clustering. It is workload-agnostic and object type-aware, suitable for testing emerging workloads and 'what-if' scenarios. We implemented a synthetic trace generator and validated it using: (1) a Big Data storage (HDFS) workload from Yahoo!, (2) a trace from a feature animation company, and (3) a streaming media workload. Two case studies in caching and replicated distributed storage systems show that our traces produce application-level results similar to the real workload. The trace generator is fast and readily scales to a system of 4.3 million files. It outperforms existing models in terms of accurately reproducing the characteristics of the real trace.

Original languageEnglish (US)
Pages (from-to)704-719
Number of pages16
JournalPerformance Evaluation
Issue number10
StatePublished - 2013


  • Big Data
  • HDFS
  • Popularity
  • Storage
  • Temporal locality
  • Workload generation

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Generating request streams on Big Data using clustered renewal processes'. Together they form a unique fingerprint.

Cite this