A new way to reduce errors in stochastic gradient Langevin dynamics

The stochastic gradient Leimkuhler-Matthews integrator offers a second order discretization error.

The stochastic gradient Leimkuhler-Matthews (SGLM) integrator has a second order discretization error. It has the same cost as stochastic gradient Langevin dynamics (SGLD).

SGLM uses one stochastic gradient per step. It also reuses each Gaussian vector in consecutive steps. SGLD has a discretization error of first order in the step size hh.

For a potential strongly convex outside a ball, the mean square error (MSE) of time averages of SGLM over NN steps is bounded by O(1Nh+Σ2h2+h4)\mathcal{O}(\frac{1}{Nh} + \Sigma^2 h^2 + h^4). Here, Σ\Sigma bounds the variance of the gradient noise.

The discretization error enters this bound as h4h^4. In the MSE bound of SGLD, the error enters as h2h^2. A larger batch reduces only the Σ2h2\Sigma^2 h^2 term in both bounds.

In Bayesian logistic regression on MNIST, SGLM has a smaller estimated MSE than SGLD. The coefficient of hh in the numerical bias of SGLM is given by the covariance of the gradient noise and the continuous Poisson equation.

Key facts

  • SGLM has the same cost as SGLD using one stochastic gradient per step.
  • The discretization error for SGLM is h4h^4 while for SGLD it is h2h^2.
  • SGLM showed a smaller estimated MSE than SGLD in Bayesian logistic regression on MNIST.

All sections

Edition