FixEval: Execution-based Evaluation of Program Fixes for Programming Problems

Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, Chris Brown

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FIXEvAL, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixeValoffers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixeValprovides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced.11https://github.com/FixEval/FixEval_official

Original languageEnglish (US)
Title of host publicationProceedings - 2023 IEEE/ACM International Workshop on Automated Program Repair, APR 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages11-18
Number of pages8
ISBN (Electronic)9798350302141
DOIs
StatePublished - 2023
Externally publishedYes
Event3rd IEEE/ACM International Workshop on Automated Program Repair, APR 2023 - Melbourne, Australia
Duration: May 16 2023 → …

Publication series

NameProceedings - 2023 IEEE/ACM International Workshop on Automated Program Repair, APR 2023

Conference

Conference3rd IEEE/ACM International Workshop on Automated Program Repair, APR 2023
Country/TerritoryAustralia
CityMelbourne
Period5/16/23 → …

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality
  • Control and Optimization

Fingerprint

Dive into the research topics of 'FixEval: Execution-based Evaluation of Program Fixes for Programming Problems'. Together they form a unique fingerprint.

Cite this