New way to manage memory for real-time tasks on small devices
Researchers found a way to precisely model and optimize memory use during inference.
Real-time inference on small devices requires low activation memory without extra latency. Changing the operator order is a valuable memory lever because it is free in time.
The usual way of handing a better order back to the compiler is unreliable. In PyTorch Inductor, the regenerated program missed its predicted peak in 22 of 54 CPU configurations.
The new method profiles the generated program and models its memory exactly from its own statements. It permutes these statements to find an order.
The researchers use forward and reverse trap lower bounds to know when to stop. These are computable with minimum cuts.
This method certifies the found order is optimal for 94% of held-out wrapper graphs. It also works for all 16 microcontroller-class models studied in under a second.
Selective recomputation with per-operator caps is studied beyond the certified optimum. This task is strongly NP-hard for every fixed cap even at treewidth two.
Key facts
- The method realizes the modeled peak to the byte with bitwise-identical results in all CPU runs.
- Reordering branching networks can pay 10.0% on DeepLabV3.
- In compiled training, the method saves up to 7.6%.
- On ReDimNet speaker networks, selective recomputation lowers the peak by 5.2-6.1%.