A new system to test how software agents find and fix bugs

The system creates executable tasks from historical code changes to evaluate software agents.

Researchers built a system to evaluate software agents. This system creates executable regression tasks from historical introducing and fixing commit pairs.

The system extracts candidate tests from fixing-side changes. It uses constrained agents to transplant tests and configure version-specific environments.

An executable project environment is a setting used for investigation. A regression relationship exists when tests fail on an introducing version and pass on a fixing version.

The system constructed 172 validated instances from 522 candidates across 11 Python projects. Researchers evaluated three software agents using ten models and additional reasoning-effort settings.

Baseline combinations achieved success rates of up to 68.7% for repair and 77.1% for regression-test generation. Precise root-line localization remains challenging.

Leading configurations differ by task. Higher task performance does not necessarily yield greater API-cost efficiency.

Key facts

  • The system created 172 validated instances from 522 candidates across 11 Python projects.
  • Repair success rates reached up to 68.7%.
  • Regression-test generation success rates reached up to 77.1%.

All sections

Edition