A new system to test how software agents find and fix bugs
The system creates executable tasks from historical code changes to evaluate software agents.
Researchers built a system to evaluate software agents. This system creates executable regression tasks from historical introducing and fixing commit pairs.
The system extracts candidate tests from fixing-side changes. It uses constrained agents to transplant tests and configure version-specific environments.
An executable project environment is a setting used for investigation. A regression relationship exists when tests fail on an introducing version and pass on a fixing version.
The system constructed 172 validated instances from 522 candidates across 11 Python projects. Researchers evaluated three software agents using ten models and additional reasoning-effort settings.
Baseline combinations achieved success rates of up to 68.7% for repair and 77.1% for regression-test generation. Precise root-line localization remains challenging.
Leading configurations differ by task. Higher task performance does not necessarily yield greater API-cost efficiency.
Key facts
- The system created 172 validated instances from 522 candidates across 11 Python projects.
- Repair success rates reached up to 68.7%.
- Regression-test generation success rates reached up to 77.1%.