A characteristic study on failures of production distributed data-parallel programs

Sihan Li, Hucheng Zhou, Haoxiang Lin, Tian Xiao, Haibo Lin, Wei Lin, Tao Xie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

SCOPE is adopted by thousands of developers from tens of different product teams in Microsoft Bing for daily web-scale data processing, including index building, search ranking, and advertisement display. A SCOPE job is composed of declarative SQL-like queries and imperative C# user-defined functions (UDFs), which are executed in pipeline by thousands of machines. There are tens of thousands of SCOPE jobs executed on Microsoft clusters per day, while some of them fail after a long execution time and thus waste tremendous resources. Reducing SCOPE failures would save significant resources. This paper presents a comprehensive characteristic study on 200 SCOPE failures/fixes and 50 SCOPE failures with debugging statistics from Microsoft Bing, investigating not only major failure types, failure sources, and fixes, but also current debugging practice. Our major findings include (1) most of the failures (84.5%) are caused by defects in data processing rather than defects in code logic; (2) table-level failures (22.5%) are mainly caused by programmers' mistakes and frequent data-schema changes while row-level failures (62%) are mainly caused by exceptional data; (3) 93% fixes do not change data processing logic; (4) there are 8% failures with root cause not at the failure-exposing stage, making current debugging practice insufficient in this case. Our study results provide valuable guidelines for future development of data-parallel programs. We believe that these guidelines are not limited to SCOPE, but can also be generalized to other similar data-parallel platforms.

Original languageEnglish (US)
Title of host publication2013 35th International Conference on Software Engineering, ICSE 2013 - Proceedings
Pages963-972
Number of pages10
DOIs
StatePublished - 2013
Externally publishedYes
Event2013 35th International Conference on Software Engineering, ICSE 2013 - San Francisco, CA, United States
Duration: May 18 2013May 26 2013

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257

Other

Other2013 35th International Conference on Software Engineering, ICSE 2013
Country/TerritoryUnited States
CitySan Francisco, CA
Period5/18/135/26/13

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'A characteristic study on failures of production distributed data-parallel programs'. Together they form a unique fingerprint.

Cite this