Network A/B Testing: From sampling to estimation

Huan Gui, Ya Xu, Anmol Bhasin, Jiawei Han

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A/B testing, also known as bucket testing, split testing, or controlled experiment, is a standard way to evaluate user engagement or satisfaction from a new service, feature, or product. It is widely used in online websites, including social network sites such as Facebook, LinkedIn, and Twitter to make data-driven decisions. The goal of A/B testing is to estimate the treatment effects of a new change, which becomes intricate when users are interacting, i.e., the treatment effects of a user may spill over to other users via underlying social connections. When conducting these online controlled experiments, it is a common practice to make the Stable Unit Treatment Value Assumption (SUTVA) that each individual's response is a-ected by their own treatment only. Though this assumption simpli-es the estimation of treatment effects, it does not hold when network interference is present, and may even lead to wrong conclusion. In this paper, we study the problem of network A/B testing in real networks, which have substantially different characteristics from the simulated random networks studied in previous works. We first examine the existence of network effects in a recent online experiment conducted at LinkedIn; Secondly, we propose an effcient and effective estimator for Average Treatment Effect (ATE) considering the interference between users in real online experiments; Finally, we apply our method in both simulations and a real world online experiment. The simulation results show that our estimator achieves better performance with respect to both bias and variance reduction. The real world online experiment not only demonstrates that large-scale network A/B test is feasible but also further validates many of our observations in the simulation studies.

Original languageEnglish (US)
Title of host publicationWWW 2015 - Proceedings of the 24th International Conference on World Wide Web
PublisherAssociation for Computing Machinery, Inc
Pages399-409
Number of pages11
ISBN (Electronic)9781450334693
DOIs
StatePublished - May 18 2015
Event24th International Conference on World Wide Web, WWW 2015 - Florence, Italy
Duration: May 18 2015May 22 2015

Publication series

NameWWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Other

Other24th International Conference on World Wide Web, WWW 2015
CountryItaly
CityFlorence
Period5/18/155/22/15

Keywords

  • A/B testing
  • Balanced graph partition.
  • Controlled experiments
  • Design of experiments
  • Network A/ B testing
  • Network effects
  • Peer effects

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Network A/B Testing: From sampling to estimation'. Together they form a unique fingerprint.

  • Cite this

    Gui, H., Xu, Y., Bhasin, A., & Han, J. (2015). Network A/B Testing: From sampling to estimation. In WWW 2015 - Proceedings of the 24th International Conference on World Wide Web (pp. 399-409). (WWW 2015 - Proceedings of the 24th International Conference on World Wide Web). Association for Computing Machinery, Inc. https://doi.org/10.1145/2736277.2741081