IFixFlakies: A framework for automatically fixing order-dependent flaky tests

August Shi, Wing Lam, Reed Oei, Tao Xie, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.

Original languageEnglish (US)
Title of host publicationESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
EditorsSven Apel, Marlon Dumas, Alessandra Russo, Dietmar Pfahl
PublisherAssociation for Computing Machinery, Inc
Pages545-555
Number of pages11
ISBN (Electronic)9781450355728
DOIs
StatePublished - Aug 12 2019
Event27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019 - Tallinn, Estonia
Duration: Aug 26 2019Aug 30 2019

Publication series

NameESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Conference

Conference27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019
CountryEstonia
CityTallinn
Period8/26/198/30/19

Fingerprint

Testing

Keywords

  • Automated fixing
  • Flaky test
  • Order-dependent test
  • Patch generation

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Cite this

Shi, A., Lam, W., Oei, R., Xie, T., & Marinov, D. (2019). IFixFlakies: A framework for automatically fixing order-dependent flaky tests. In S. Apel, M. Dumas, A. Russo, & D. Pfahl (Eds.), ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 545-555). (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering). Association for Computing Machinery, Inc. https://doi.org/10.1145/3338906.3338925

IFixFlakies : A framework for automatically fixing order-dependent flaky tests. / Shi, August; Lam, Wing; Oei, Reed; Xie, Tao; Marinov, Darko.

ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ed. / Sven Apel; Marlon Dumas; Alessandra Russo; Dietmar Pfahl. Association for Computing Machinery, Inc, 2019. p. 545-555 (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Shi, A, Lam, W, Oei, R, Xie, T & Marinov, D 2019, IFixFlakies: A framework for automatically fixing order-dependent flaky tests. in S Apel, M Dumas, A Russo & D Pfahl (eds), ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, Inc, pp. 545-555, 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, Tallinn, Estonia, 8/26/19. https://doi.org/10.1145/3338906.3338925
Shi A, Lam W, Oei R, Xie T, Marinov D. IFixFlakies: A framework for automatically fixing order-dependent flaky tests. In Apel S, Dumas M, Russo A, Pfahl D, editors, ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, Inc. 2019. p. 545-555. (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering). https://doi.org/10.1145/3338906.3338925
Shi, August ; Lam, Wing ; Oei, Reed ; Xie, Tao ; Marinov, Darko. / IFixFlakies : A framework for automatically fixing order-dependent flaky tests. ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. editor / Sven Apel ; Marlon Dumas ; Alessandra Russo ; Dietmar Pfahl. Association for Computing Machinery, Inc, 2019. pp. 545-555 (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering).
@inproceedings{7894951942ad406c982b91cfc812a806,
title = "IFixFlakies: A framework for automatically fixing order-dependent flaky tests",
abstract = "Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.",
keywords = "Automated fixing, Flaky test, Order-dependent test, Patch generation",
author = "August Shi and Wing Lam and Reed Oei and Tao Xie and Darko Marinov",
year = "2019",
month = "8",
day = "12",
doi = "10.1145/3338906.3338925",
language = "English (US)",
series = "ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering",
publisher = "Association for Computing Machinery, Inc",
pages = "545--555",
editor = "Sven Apel and Marlon Dumas and Alessandra Russo and Dietmar Pfahl",
booktitle = "ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering",

}

TY - GEN

T1 - IFixFlakies

T2 - A framework for automatically fixing order-dependent flaky tests

AU - Shi, August

AU - Lam, Wing

AU - Oei, Reed

AU - Xie, Tao

AU - Marinov, Darko

PY - 2019/8/12

Y1 - 2019/8/12

N2 - Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.

AB - Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.

KW - Automated fixing

KW - Flaky test

KW - Order-dependent test

KW - Patch generation

UR - http://www.scopus.com/inward/record.url?scp=85071919195&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071919195&partnerID=8YFLogxK

U2 - 10.1145/3338906.3338925

DO - 10.1145/3338906.3338925

M3 - Conference contribution

AN - SCOPUS:85071919195

T3 - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

SP - 545

EP - 555

BT - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

A2 - Apel, Sven

A2 - Dumas, Marlon

A2 - Russo, Alessandra

A2 - Pfahl, Dietmar

PB - Association for Computing Machinery, Inc

ER -