A new way to control systems using data and physics

A data-driven Hamiltonian approach provides better performance guarantees and requires fewer samples than standard reinforcement learning.

Researchers proposed a data-driven Hamiltonian (DDH) for neural optimal control. This method serves as an alternative to reinforcement learning (RL).

The DDH uses the Hamilton-Jacobi framework. A Hamiltonian is a tool used to evaluate system dynamics. The value function is a physics-informed neural approximation used for high-dimensional systems.

The DDH requires only minimal prior system information like a Lipschitz constant. This helps quantify and account for uncertainty in regions not covered by data. It can also incorporate additional prior system knowledge to improve performance.

In an LQR problem, the DDH reaches near-optimal cost on every tested dataset. It needs far fewer samples than online RL. On a nonlinear lunar lander benchmark, it matches the performance of SAC with approximately 37×37\times fewer samples.

The DDH achieves an optimality gap to a trajectory-optimization reference more than 3×3\times smaller than CQL and FQL. This framework is promising when available data are limited in quantity or quality.

Key facts

  • The DDH reaches near-optimal cost on random data where offline RL methods remain far from optimal.
  • The DDH matches SAC performance using approximately 37×37\times fewer samples on a lunar lander benchmark.
  • The DDH provides a mechanism for quantifying uncertainty in regions not covered by data.

All sections

Edition