Program structure can describe and edit code model states
Researchers found that information from a compiler front end can represent and change the hidden states of code models.
Standard static analysis can describe and change the hidden states of code models. This works without retraining the models.
Researchers tested 14 open models, including seven trained for code. They used information from a compiler front end to describe hidden states.
A compiler front end provides data like position in the abstract syntax tree and scope. It also provides definition-use links and syntactic categories.
The researchers replaced the model's own states with a decomposition at every layer. This decomposition combines each token with a structural label.
In 10 of 14 models, this method preserved answers better than a positional baseline. This included six of the seven code models.
Editing the decomposition to rebind a call argument made StarCoder2-15B produce the correct answer in 90 of 100 cases. The positional baseline only succeeded in 19 cases.
Key facts
- The static-analysis decomposition preserves model answers better than positional decomposition in 10 of 14 models.
- Deleting definition-use links removes most of the advantage on the two models tested.
- The Qwen2.5 family is an exception on several measures.