A characteristic study of parameterized unit tests in .NET open source projects

Wing Lam, Siwakorn Srisakaokul, Blake Bassett, Peyman Mahdian, Tao Xie, Pratap Lakshman, Jonathan De Halleux

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.

Original languageEnglish (US)
Title of host publication32nd European Conference on Object-Oriented Programming, ECOOP 2018
EditorsTodd Millstein
PublisherSchloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing
ISBN (Electronic)9783959770798
DOIs
StatePublished - Jul 1 2018
Event32nd European Conference on Object-Oriented Programming, ECOOP 2018 - Amsterdam, Netherlands
Duration: Jul 16 2018Jul 21 2018

Publication series

NameLeibniz International Proceedings in Informatics, LIPIcs
Volume109
ISSN (Print)1868-8969

Other

Other32nd European Conference on Object-Oriented Programming, ECOOP 2018
CountryNetherlands
CityAmsterdam
Period7/16/187/21/18

Fingerprint

Testing
Industry

Keywords

  • Automated test generation
  • Parameterized unit testing
  • Unit testing

ASJC Scopus subject areas

  • Software

Cite this

Lam, W., Srisakaokul, S., Bassett, B., Mahdian, P., Xie, T., Lakshman, P., & De Halleux, J. (2018). A characteristic study of parameterized unit tests in .NET open source projects. In T. Millstein (Ed.), 32nd European Conference on Object-Oriented Programming, ECOOP 2018 (Leibniz International Proceedings in Informatics, LIPIcs; Vol. 109). Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing. https://doi.org/10.4230/LIPIcs.ECOOP.2018.5

A characteristic study of parameterized unit tests in .NET open source projects. / Lam, Wing; Srisakaokul, Siwakorn; Bassett, Blake; Mahdian, Peyman; Xie, Tao; Lakshman, Pratap; De Halleux, Jonathan.

32nd European Conference on Object-Oriented Programming, ECOOP 2018. ed. / Todd Millstein. Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2018. (Leibniz International Proceedings in Informatics, LIPIcs; Vol. 109).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lam, W, Srisakaokul, S, Bassett, B, Mahdian, P, Xie, T, Lakshman, P & De Halleux, J 2018, A characteristic study of parameterized unit tests in .NET open source projects. in T Millstein (ed.), 32nd European Conference on Object-Oriented Programming, ECOOP 2018. Leibniz International Proceedings in Informatics, LIPIcs, vol. 109, Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 32nd European Conference on Object-Oriented Programming, ECOOP 2018, Amsterdam, Netherlands, 7/16/18. https://doi.org/10.4230/LIPIcs.ECOOP.2018.5
Lam W, Srisakaokul S, Bassett B, Mahdian P, Xie T, Lakshman P et al. A characteristic study of parameterized unit tests in .NET open source projects. In Millstein T, editor, 32nd European Conference on Object-Oriented Programming, ECOOP 2018. Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing. 2018. (Leibniz International Proceedings in Informatics, LIPIcs). https://doi.org/10.4230/LIPIcs.ECOOP.2018.5
Lam, Wing ; Srisakaokul, Siwakorn ; Bassett, Blake ; Mahdian, Peyman ; Xie, Tao ; Lakshman, Pratap ; De Halleux, Jonathan. / A characteristic study of parameterized unit tests in .NET open source projects. 32nd European Conference on Object-Oriented Programming, ECOOP 2018. editor / Todd Millstein. Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2018. (Leibniz International Proceedings in Informatics, LIPIcs).
@inproceedings{b27750ae03f8463782ae1d0098f488b4,
title = "A characteristic study of parameterized unit tests in .NET open source projects",
abstract = "In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.",
keywords = "Automated test generation, Parameterized unit testing, Unit testing",
author = "Wing Lam and Siwakorn Srisakaokul and Blake Bassett and Peyman Mahdian and Tao Xie and Pratap Lakshman and {De Halleux}, Jonathan",
year = "2018",
month = "7",
day = "1",
doi = "10.4230/LIPIcs.ECOOP.2018.5",
language = "English (US)",
series = "Leibniz International Proceedings in Informatics, LIPIcs",
publisher = "Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing",
editor = "Todd Millstein",
booktitle = "32nd European Conference on Object-Oriented Programming, ECOOP 2018",

}

TY - GEN

T1 - A characteristic study of parameterized unit tests in .NET open source projects

AU - Lam, Wing

AU - Srisakaokul, Siwakorn

AU - Bassett, Blake

AU - Mahdian, Peyman

AU - Xie, Tao

AU - Lakshman, Pratap

AU - De Halleux, Jonathan

PY - 2018/7/1

Y1 - 2018/7/1

N2 - In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.

AB - In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.

KW - Automated test generation

KW - Parameterized unit testing

KW - Unit testing

UR - http://www.scopus.com/inward/record.url?scp=85052014666&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85052014666&partnerID=8YFLogxK

U2 - 10.4230/LIPIcs.ECOOP.2018.5

DO - 10.4230/LIPIcs.ECOOP.2018.5

M3 - Conference contribution

AN - SCOPUS:85052014666

T3 - Leibniz International Proceedings in Informatics, LIPIcs

BT - 32nd European Conference on Object-Oriented Programming, ECOOP 2018

A2 - Millstein, Todd

PB - Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing

ER -