Predictors for coding agents struggle to identify why runs fail

Current predictors mostly identify issue difficulty and model ability instead of the run itself.

Trajectory-based failure predictors read a partial trajectory of a run. These tools flag if a run is going right or wrong.

A trajectory is the partial path of a run. A predictor is a tool that flags if a run is going right or wrong.

On SWE-bench Verified trajectories, issue difficulty and model ability dominate predictor performance. A reference score for issue difficulty reaches 0.920.92 AUROC. Trajectory predictors reach about 0.620.62.

Knowing which model ran predicts failure better than trajectory predictors. Beyond difficulty and ability, what predictors gain from the trajectory is statistically equivalent to zero.

Retrying can repair failures caused by a bad run. However, retrying does not rescue failures on an issue beyond the model's ability. Current predictors cannot tell these causes apart.

Failure prediction should explain why a run fails. This should be evaluated on repeated-run benchmarks and in closed-loop, equal-budget settings.

Key facts

  • Issue difficulty reaches a reference score of 0.920.92 AUROC.
  • Trajectory predictors reach about 0.620.62 on SWE-bench Verified trajectories.
  • Predictors mostly predict the issue and the model, not the run.

All sections

Edition