ProteinGeoBench shows differences in structural biology tasks
Frontier agents retrieve structural metadata well but compute geometry from coordinates less reliably.
ProteinGeoBench is a suite of 131 protein-structure tasks. Frontier agents retrieve structural metadata well. They compute geometry from coordinates less reliably.
The study used 4,249 rollouts from four frontier models. It also used 4,486 further trials across 26 models. Three of the four frontier models score between 0.785 and 0.816. The fourth model scores 0.707.
Three of the four frontier models do better on metadata lookup than on numeric calculation. Numeric tasks are 28 of the 131 tasks. Half of the twelve tasks that all four models fail are numeric.
Removing network access lowers scores more than blocking eight structural-database domains. Swapping in a model's own command-line agent changes scores by -0.071 to +0.086. These changes are too small to rule out moderate harness effects.
The experiments use different task subsets. They cannot rank model, harness, access and task composition against one another. Evaluations should report results by task family, the number of runs, the harness and the access conditions.
Key facts
- Three of the four frontier models fall within 0.031 of one another.
- Numeric performance rises with non-numeric performance across 26 models.
- Half of the twelve tasks that all four models fail are numeric tasks.