A new way to reduce errors in stochastic gradient Langevin dynamics
The stochastic gradient Leimkuhler-Matthews integrator offers a second order discretization error.
The stochastic gradient Leimkuhler-Matthews (SGLM) integrator has a second order discretization error. It has the same cost as stochastic gradient Langevin dynamics (SGLD).
SGLM uses one stochastic gradient per step. It also reuses each Gaussian vector in consecutive steps. SGLD has a discretization error of first order in the step size .
For a potential strongly convex outside a ball, the mean square error (MSE) of time averages of SGLM over steps is bounded by . Here, bounds the variance of the gradient noise.
The discretization error enters this bound as . In the MSE bound of SGLD, the error enters as . A larger batch reduces only the term in both bounds.
In Bayesian logistic regression on MNIST, SGLM has a smaller estimated MSE than SGLD. The coefficient of in the numerical bias of SGLM is given by the covariance of the gradient noise and the continuous Poisson equation.
Key facts
- SGLM has the same cost as SGLD using one stochastic gradient per step.
- The discretization error for SGLM is while for SGLD it is .
- SGLM showed a smaller estimated MSE than SGLD in Bayesian logistic regression on MNIST.