Abstract
Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.
| Original language | English (US) |
|---|---|
| Title of host publication | ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering |
| Editors | Sven Apel, Marlon Dumas, Alessandra Russo, Dietmar Pfahl |
| Publisher | Association for Computing Machinery, Inc |
| Pages | 545-555 |
| Number of pages | 11 |
| ISBN (Electronic) | 9781450355728 |
| DOIs | |
| State | Published - Aug 12 2019 |
| Event | 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019 - Tallinn, Estonia Duration: Aug 26 2019 → Aug 30 2019 |
Publication series
| Name | ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering |
|---|
Conference
| Conference | 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019 |
|---|---|
| Country | Estonia |
| City | Tallinn |
| Period | 8/26/19 → 8/30/19 |
Fingerprint
Keywords
- Automated fixing
- Flaky test
- Order-dependent test
- Patch generation
ASJC Scopus subject areas
- Artificial Intelligence
- Software
Cite this
IFixFlakies : A framework for automatically fixing order-dependent flaky tests. / Shi, August; Lam, Wing; Oei, Reed; Xie, Tao; Marinov, Darko.
ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ed. / Sven Apel; Marlon Dumas; Alessandra Russo; Dietmar Pfahl. Association for Computing Machinery, Inc, 2019. p. 545-555 (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - IFixFlakies
T2 - A framework for automatically fixing order-dependent flaky tests
AU - Shi, August
AU - Lam, Wing
AU - Oei, Reed
AU - Xie, Tao
AU - Marinov, Darko
PY - 2019/8/12
Y1 - 2019/8/12
N2 - Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.
AB - Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.
KW - Automated fixing
KW - Flaky test
KW - Order-dependent test
KW - Patch generation
UR - http://www.scopus.com/inward/record.url?scp=85071919195&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85071919195&partnerID=8YFLogxK
U2 - 10.1145/3338906.3338925
DO - 10.1145/3338906.3338925
M3 - Conference contribution
AN - SCOPUS:85071919195
T3 - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
SP - 545
EP - 555
BT - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
A2 - Apel, Sven
A2 - Dumas, Marlon
A2 - Russo, Alessandra
A2 - Pfahl, Dietmar
PB - Association for Computing Machinery, Inc
ER -