Abstract
While Langevin integrators are widely popular in the study of equilibrium properties of complex systems, it is challenging to estimate the the timestep-induced discretization error: the degree to which the sampled phase space or configuration space probability density departs from the desired target density due to the use of a finite integration timestep. In [1], Sivak et al. introduced a convenient approach to quantifying the a natural measure of distribution error between the sampled density and the target equilibrium density, the KL divergence, in phase space, but did not specifically address the issue of configuration-space properties, which are much more commonly of interest in molecular simulations. Here, we introduce a variant of this near-equilibrium estimator capable of measuring the error in the configuration-space marginal density, validating it against a complex but exact nested Monte Carlo estimator to show that it reproduces the KL divergence with high fidelity. To illustrate its utility, we employ this new near-equilibrium estimator to assess a claim that a recently proposed Langevin integrator introduces extremely small configuration-space density errors up to the stability limit at no extra computational expense. Finally, we show how this approach to quantifying sampling error can be applied to a wide variety of stochastic integrators by following a straightforward procedure to compute the appropriate shadow work, and describe how it can be extended to quantify the error in arbitrary marginal or conditional distributions of interest.
1. Introduction
Langevin integrators are widely employed to simulate the equilibrium thermodynamics and kinetics of microscopic molecular systems. The origin of these methods lies in the Langevin equation [2], which describes the behavior of condensed phase systems subject to random weak collisions with fictitious bath particles at thermal equilibrium,
Here, x and v denote the positions and velocities of all particles in the system in Cartesian coordinates, t is time, M the diagonal mass matrix, and U(x) is the potential energy. The constant σ2 ≡ (2kBTγ) quantifies the rate of heat exchange with the bath, with kBT denoting the thermal energy, γ the collision rate (with dimensions of inverse time), and W(t) a standard multidimensional Wiener process [3,4].
On a digital computer, integration of (1) requires discretizing these equations in time to produce a finite timestep Langevin integrator capable of generating a temporally discrete dynamical trajectory from which equilibrium or dynamical properties can be estimated [5]. A wide variety of schemes have been proposed for this discretization, which we refer to herein as “Langevin integrators” [6–15].
For example, the popular integrator of Bussi and Parrinello [12] (which we will denote OVRVO for reasons that will become clear shortly), also known as velocity Verlet with velocity randomization (VVVR) [16] due to its use of a velocity Verlet integrator core (substeps VRV below) [17], consists of the discrete update equations:
where ak = e−γ(Δt/k) is a constant that depends on timestep Δt and collision rate γ, and k denotes the number of O steps appearing in the integrator. The intermediate timestep subscripts (t + 1/4, t + 1/2, t + 3/4) denote variables used solely for convenience (without physical meaning), and the z denote unit normal random variates.
An alternative splitting championed by Leimkuhler and Matthews termed BAOAB [14,18] (which we will refer to as VRORV) consists of a different set of discrete update equations:
While both discrete time integration schemes reduce to the same stochastic differential equations in the limit that Δt → 0, they can behave quite differently for finite timesteps (Δt > 0), especially for timesteps of practical interest for atomistic molecular simulation.
Langevin integrators introduce sampling bias that grows with the size of the timestep
Langevin integrators intend to sample from the equilibrium density,
where β is the inverse temperature, h(x, v) is a separable reduced (unitless) Hamiltonian h(x, v) = βH(x, v) = u(x) + t(v), u(x) ≡ βU(x) is the reduced potential [19], and t(v) ≡ βT(v) is the reduced kinetic energy. If only configuration-space properties are of interest, as is often the case in molecular simulations, Langevin integrators aim to sample from the marginal density in configuration space, πx,
if the velocities are ignored while the positions are used to estimate expectations or other configuration-dependent properties.
The continuous-time Langevin equations of motion (1) possess the target equilibrium density p (4) as their stationary density, suggesting that, at least in principle, temporal averages along Langevin trajectories can be used to approximate averages with respect to the equilibrium distribution. However, numerical simulations with a finite timestep Δt > 0 will generally sample a slightly different distribution, which we will denote by ρ(x, v), which implicitly depends on timestep Δt. The discrepancy between the distributions ρ and π will grow with Δt at some rate (e.g. 
until numerical simulation becomes unstable altogether. Note that this phenomenon is completely separate from numerical issues introduced by finite-precision arithmetic on digital computers, which introduces roundoff error in mathematical operations; here, we presume that computations can be carried out to arbitrary precision, and analyze only the effects of time-discretization.
In Figure 1, we illustrate a few key behaviors of this stepsize-dependent sampling bias in a simple quartic 1D system. Note that: (1) the numerically sampled distribution deviates from the target distribution, (2) this deviation increases with timestep Δt, and (3) the deviation in phase space (x, v) may be different than the deviation in configuration space only (x).
For a simple 1D system with the quartic potential U(x) = x4, the error in sampled phase space density ρ and its marginal density ρx grows as a function of timestep Δt. However, different Langevin integrators (OVRVO and VRORV shown here) derived from symmetric Strang splittings can lead to drastically different error structures in phase space, which can induce fortuitous cancellation of error in in the marginal distribution under certain circumstances (VRORV). In the top row, we illustrate the definition of the 1D system (left: the potential energy function, U(x) = x4; middle: the equilibrium marginal density over configuration space, πx(x) ∝ e−βU(x); right: the equilibrium joint distribution over phase space π(x, v). In the middle row, we illustrate the increasing discrepancy between the sampled distribution r and the equilibrium distribution π, for both the full phase space and the marginal configuration space, as a function of timestep Δt, for a particular choice of Langevin integrator OVRVO (2). In the bottom row, we illustrate the timestep-dependent error for another integrator VRORV (3).
Many discrete-time integrators can be constructed from the continuous Langevin equations of motion
There are several possible ways to discretize Langevin dynamics in such a way that we recover the exact Langevin equation in the limit Δt → 0, and thus recover the desired equilibrium distribution, but as we shall soon see, these methods can have drastically different behaviors for finite timesteps (Δt > 0). A particularly flexible way to construct numerical integrators for Langevin dynamics is operator splitting, where the Langevin system is commonly split into three components, here labeled V, R, and O because they deal with increments to positions (R), velocities (V), or an Ornstein-Uhlenbeck like process (O)1,
where each component can be solved “exactly” for a small time increment2.
The Strang symmetric operator splitting approach considers splitting the overall propagator 
into a product of propagators for the individual R, V, and O components, defining a family of splittings which we will index by strings indicating the order of appearance of these individual propagators. For example, we use OVRVO to refer to the propagator splitting,
While equality between the true propagator and the Strang splitting is only achieved in the limit that Δt → 0, the great advantage of this approach is that each component propagator, 
and 
has a corresponding finite-time update equation that can be used to implement the action of that operator on a digital computer:
where ak = e−γ(Δt/nO) and ξ ∼ N (0, 1) is a standard normal random variate drawn for each degree of freedom for each O step. The integers nR, nV, and nO denote the number of times each operator appears in the propagator splitting, which is equal to the number of times each corresponding update equation is used in one integrator cycle.
By chaining these operations in the order specified by the splitting string, we can unroll these operations into the sequence of mathematical updates needed to implement one cycle of the integrator for a total time Δt. For VRORV, for example, translating the splitting string into the appropriate sequence of update equations in (8)–(10) produces the equations for one complete integrator timestep:
Different integrators may approximate different properties with varying fidelity
It has been proposed that some integrators of Langevin dynamics (particularly the VRORV aka “BAOAB” integrator of Leimkuhler and Matthews) preserve the configuration distribution with significantly greater fidelity than other equal-cost integration algorithms [14,18,20]. However, as the formal arguments for this “superconvergence” property rely on a high-friction limit, it is unclear how large the friction coefficient needs to be in practice for the argument to apply. Formal descriptions of the error are typically generic, in that they do not provide guidance on precisely which Δt introduces a tolerable amount of bias for a particular system, and they do not provide a way to predict how other choices, such as mass matrix modifications (e.g., hydrogen mass repartitioning) [21–24]), will affect the error for a system of interest.
Until now, assessing whether specific integrators sample the true equilibrium density with greater fidelity than others in specific settings has relied on computing low-dimensional marginal distributions of manually selected observables perceived to be sensitive to configuration-space sampling errors [25], such as radial distribution functions, marginal distributions of internal coordinates [22], or the configurational temperature [14,26,27]. While it is clear that some observables are more sensitive to errors in configuration space density than others (Figure 2), and the error in the observables of interest is paramount for a particular application, this highlights the risk of using the error in a single physical property as a surrogate for judging integrator quality, as the error in other observables of interest may be large despite small error in the test observable.
Here, we illustrate the timestep-dependent discretization error introduced by four integrators on a 1D double-well potential [U(x) ≡ x6 + 2 cos(5(x + 1))]. The top row of 2D contour plots illustrates the difference between the phase-space density ρ(x, v) sampled at the maximum timestep considered (Δt = 0.7, close to the stability limit) and the equilibrium density π(x, v); solid lines indicate positive contours, while dashed lines indicate negative contours. The bottom row of 1D density plots shows timestep-dependent perturbation in the sampled marginal distribution in configuration space, ρx, with the equilibrium distribution πx is depicted as a solid black line. The sampled marginal distributions ρx are shown for increasingly large timesteps, denoted ρΔt, depicted by increasingly light dotted lines, for Δt = 0.3, 0.6,…,0.7 (arbitrary units). Inspecting the contour plots suggests that some integrator splittings (especially VRORV) induce error that fortuitously “cancels out” when the density is marginalized by integrating over v, while the error in other integrator splittings (ORVRO, OVRVO) constructively sums to amplify the error in configuration space.
To evaluate numerical Langevin integrators, there would be great utility in a computable, universal measure of the bias they introduce in specific concrete settings, such that low error in this measure ensures low error in all observables of interest. There is currently no computable measure of the total configuration-sampling bias for concrete choices of integrator parameters and target system.
KL divergence is a natural measure of sampling bias
Controlling the magnitude of the integrator-induced sampling bias is crucial when computing quantitative predictions from simulation. However, because ρ does not have a closed-form, easily computable expression3, it is difficult to quantify the error introduced by a given choice of timestep or integrator. We will show how, for a particularly useful measure of error, we can circumvent this problem and develop a simple, effective approach to measuring error in complex molecular systems.
An ideal measure of the discrepancy between the sampled distribution ρx and the equilibrium distribution πx should be “universal” in the sense that driving that measure to zero implies that error in any expectation also goes to zero. It should also be defined for all densities, and not rely on a system-specific choice of observables. One such measure is the Kullback-Leibler (KL) divergence
The KL divergence is defined and non-negative for any pair of distributions on the same support, and DKL(p‖q) = 0 if and only if p = q.
A near-equilibrium estimator can measure timestep-dependent KL divergence for molecular systems
In [1], Sivak and colleagues demonstrated how to approximate the KL divergence of the sampled distribution ρ from the target distribution π over the full phase-space distribution, DKL(ρ‖π), in terms of a work-like quantity—the shadow work —that is readily computable for a large family of Langevin integrators (Figure 4a). This estimator depends only on the ability to draw samples from π and to measure a suitable work-like quantity. This method was applied in [1] to measure the phase space sampling bias introduced by a particular Langevin integrator (OVRVO) on periodic boxes of TIP3P water [29].
Even for the simple double-well potential considered in Figure 2, configuration-space properties display different sensitivities to sampling error, motivating the use of a “universal” error measure, such as the KL divergence. The top left panel illustrates the double-well potential energy function from Figure 2, and the top center panel shows the resulting marginal equilibrium density, πx, at β = 1. The bottom left panel shows, as a function of Δt, growth in the magnitude of the error in average potential energy, 
which has been used previously as a sensitive measure of sampling error [14]. The bottom center panel shows the error in the apparent free energy difference between the two wells as a function of Δt. Note that the timestep-dependent behavior of these two observables imply different rankings of integrator fidelity that may mislead one into believing error in all observables remains low with increasing timestep. However, as is clear here, just because an integrator introduces low timestep-dependent error in one observable doesn’t mean that the method will introduce low error in another observable: for example, OVRVO preserves the well populations as accurately as VRORV, but introduces the largest errors in the average potential energy. The right column summarizes the growth in timestep-dependent error, as measured by the KL divergence. While all four integrators introduce comparable levels of Δt-dependent error in the phase-space distribution, they induce dramatically different magnitudes of error in the configuration-space marginal.
Simple nonequilibrium protocols can be used in complex molecular systems to rapidly estimate—utilizing the Crooks fluctuation theorem—the KL divergence of sampled Langevin densities from equilibrium. In both panels, the x-axis is the number of steps taken so far in the length-2 T protocol, and 
indicates the average (reduced unitless) shadow work accumulated over T steps of Langevin dynamics, initialized from equilibrium ((x0, v0) ∼ π). (a) The original scheme described in Sivak et al. [1] to measure the KL divergence between the sampled phase space density ρ and the equilibrium phase space density 
is the average shadow work accumulated over T steps of Langevin dynamics, initialized from the integrator’s steady state ((x0, v0) ∼ ρ). (b) The modified scheme introduced here to measure the KL divergence in the configuration-space marginal density between the marginal sampled configuration-space density ρx and marginal equilibrium density 
is the average shadow work accumulated over T steps of Langevin dynamics, where the initial configuration is drawn from the integrator’s steady state, and the initial velocities are drawn from equilibrium (x0 ∼ ρx, v0 ∼ π(v|x0)). We denote this distribution w(x, v) ≡ ρx(x)π(v|x). The top row schematically illustrates “distance from equilibrium”, with y-axis ticks for DKL(π‖π) = 0, DKL(ω‖π) ≤ DKL(ρ‖π). The bottom row illustrates the average work (here, just shadow work) accumulated throughout each protocol.
Since the velocity marginal is usually not of interest, and since some integrators are thought to preserve the configuration marginal of the target distribution with higher fidelity than the phase space joint distribution, we sought to generalize the technique to estimate the KL divergence of the sampled configuration-space marginal, DKL(ρx‖πx). Below, we show how a simple modification of the estimator described in [1] can achieve this goal, and illustrate how this provides a useful tool for measuring the integrator-induced error in configuration-space densities for real molecular systems.
2. Results
2.1. Common Langevin integrator splittings induce comparable timestep-dependent error in phase space
We first applied the original near-equilibrium method of Sivak et al. [1] to measure the timestep-dependent phase-space error introduced by four common Langevin integrators on a molecular mechanics model system (a cluster of 20 TIP3P waters [29] in a harmonic restraining potential), for a range of timesteps Δt between 0.1 and 8 femtoseconds (0.1 fs, 0.5 fs, 1.0 fs,…,7.5 fs, 8.0 fs). All four integrator splitting schemes introduced comparable error in phase space (although we could resolve statistically significant differences among the schemes), as illustrated in Figure 5a. This may be unsurprising, as none of these integrator schemes are known to provide a significant reduction in acceptance rates—which depend in some manner on the induced phase space sampling error—for Metropolized versions of these integrators (Figure A1a; see Appendix A for more details about the relationship between Metropolized acceptance rates and KL divergence). Consistent with the results of [1], the phase-space error appears to scale approximately as 
The results here are reported for a small cluster of rigid TIP3P waters, described in more detail in the Detailed Methods section, and illustrated in the leftmost panel. On the x-axis is the timestep Δt, measured in femtoseconds (fs). On the y-axis is the estimated KL divergence DKL. (a) The error over the joint distribution on DKL(ρ‖π). (b) The error over the configuration-space marginal DKL(ρx‖πx). Each colored curve corresponds to a numerical scheme for Langevin dynamics. The shaded region is the mean ±95% confidence interval.
2.2. A simple modification to the near-equilibrium estimator can compute KL divergence in configuration space
The method in Sivak et al. [1] approximates the KL divergence between the nonequilibrium steady state density ρ and equilibrium density π in terms of two work averages: the work 
required to drive from equilibrium π into the steady state ρ, and the steady-state work 
expended over the same length of time, but starting in ρ (Figure 5a).
We can construct an analogous procedure to estimate the KL divergence of arbitrary marginals of ρ and π, provided we can also sample from an appropriate conditional distribution of π. For example, we are interested in estimating the KL divergence between configuration-space marginals ρx and πx,
To compute these using the machinery of Sivak et al. [1], we replace the second work average 
with 
where the expectation of the shadow work is now computed over a modified density constructed from the nonequilibrium steady-state configuration density but with Maxwell-Boltzmann velocity density,
Practically, this corresponds to drawing samples from the nonequilibrium steady-state ρ and replacing the velocities v with an i.i.d. sample from the Maxwell-Boltzmann distribution π (v|x). The modified procedure depicted schematically in Figure 4b.
2.3. Common Langevin integrators induce substantially different timestep-dependent configuration space error
Figure 5b shows that the measured KL divergence between the configuration-space marginals ρx and πx can be drastically different among the four integrator schemes, and in some cases grow much more slowly than the associated phase-space sampling error (Fig. 5a). In particular, for VRORV, the error in the x-marginal is very nearly zero for the entire range of feasible timesteps, and it can be run at Δt ≈ 6 fs while introducing the same amount of configuration error as other methods at Δt ≈ 2 fs. This is consistent with prior findings [4,14], which showed the VRORV scheme introduces very little error in the average potential energy and multiple other system-specific observables.
We also note that for the OVRVO scheme, DKL(ρx‖πx) ≈ DKL(ρ‖π) over the range of measured timesteps (Fig. 5), consistent with Sivak et al.’s prior findings that estimates of DKL(ρ‖π) tracked well with several measures of configuration-sampling error.
2.4. An exact but expensive estimator of KL divergence validates the near-equilibrium estimate
The accuracy of the near-equilibrium approximation is largely unexplored. While the near-equilibrium approximation introduced by Sivak et al. is computationally and statistically appealing, it is important to validate the accuracy of the approximation over the practical timestep Δt range of relevance to molecular simulation. In particular, it is unknown whether the near-equilibrium approximation produces an over-estimate or under-estimate of the KL divergence, or how accurate the approximation is for high-dimensional systems. Further, it is unknown whether any bias introduced by the approximation is uniform across different numerical methods for Langevin dynamics.
How well does the near-equilibrium estimator approximate the true KL divergence of relevant timestep ranges? The task of validating the near-equilibrium approximation is numerically challenging, since we are unaware of exact estimators for DKL(ρ‖π) that remain tractable in high dimensions.4 In the case of simple fluids, approximate methods are available that express the KL divergence in terms of a series of N-body correlations (as in [31]), typically truncating to 2-body correlation functions (i.e. comparing the radial distribution functions). However, in general we don’t know the effect of truncating the expansion, since the successive terms in the series don’t necessarily have decreasing magnitude.
Thus, we provide an asymptotically exact reference method. First, we will derive an exact expression for the KL divergence between ρ and π in terms of quantities that we can measure, then discuss practical challenges that arise when using this expression, and under what conditions it becomes impractical. We start by writing the KL divergence as an expectation w.r.t. ρ (17), since we can’t evaluate ρ(x, v) pointwise, but we can draw samples (x, v) ∼ ρ.
We note that the inner ratio of nonequilibrium steady-state to equilibrium densities, ρ(x, v)/π(x, v), can be expressed in terms of 
the average of exponentiated nonequilibrium work measured under the application of the time-reversed protocol 
starting from (x, v) (18). L denotes the protocol used to generate ρ from a sample from π —in this case, T applications of the Langevin integrator step kernel; 
denotes the time-reverse of this protocol. Since the protocol we apply to generate ρ from π is time-symmetric for integrators derived from symmetric Strang splittings, we can substitute 
5. In the final step, we substitute a simple Monte Carlo estimator of that average, in terms of work samples wij, where wij is the jth reduced (unitless) work measurement collected from initial condition i. Here, N is the number of initial conditions sampled (i.e. the number of “outer-loop” samples), and M is the number of work samples (i.e., the number of “inner-loop” samples) collected at each initial condition (xi, vi) ∼ ρ.
The required work values wij can be easily computed from simulations. To sample an initial condition i from ρ, we simply run the Langevin integrator of interest for a sufficient number of steps to sample a new uncorrelated configuration from the nonequilibrium steady-state sampled by the integrator. To compute the work accumulated from a given starting condition, we use the notion of shadow work [32]. For numerical methods constructed from symmetric Strang splittings involving the R, V, and O operations described above, we simply need to compute sum of the total energy changes during the deterministic substeps (i.e., the potential energy change during deterministic updates of the position variables, and the kinetic energy change during deterministic updates of the momentum variables). For convenience, we use reduced (unitless) energies and work values throughout, where factors of kBT has been removed, without loss of generality. See Detailed Methods (Section 4.4) for a detailed description on how shadow work can be computed from this family of Langevin integrators in general.
Like the near-equilibrium scheme, this nested scheme can be modified analogously to measure the configuration-space error in isolation, by initializing instead from the distribution ω(x, v) ≡ ρx(x)π(v|x), allowing us to compute DKL[w|π], a quantity that is identical to DKL[ρx‖πx]:
Specifically, to measure the full KL divergence, we sample initial conditions from the Langevin integrator’s steady state: (xi, vi) ∼ ρ. To measure configuration-space-only KL divergence, we draw initial configuration from the integrator’s steady state, and velocities from equilibrium: xi ∼ ρx, vi ∼ π(vi|xi). Note that, for constrained systems, π(v|x) is not independent of x, and care must be taken to eliminate velocity components along constrained degrees of freedom before measuring the contribution of the integrator substep to the shadow work (see Detailed Methods).
We note that the nested plug-in Monte Carlo estimator of the KL divergence is asymptotically exact only when both N (the number of “outer-loop” samples) and M (the number of “inner-loop” samples) go to infinity. In practice, we use a simple adaptive scheme (described in detail in Section 4.7) that draws inner- and outer-loop samples until uncertainty thresholds are met.
We compared the estimates of the KL divergence on the molecular mechanics system introduced in Figure 5, and confirmed that the two estimates are roughly consistent for all four integrator schemes, over a range of feasible timesteps (Figure 6). At large DKL, the near-equilibrium scheme deviates from the nested scheme. However, the nested scheme is likely to be biased in the direction of under-estimating the actual DKL in this region, and the magnitude of this bias increases with the variance of the shadow work distribution. We conclude that the near-equilibrium approximation is empirically reliable for measuring integrator bias on molecular mechanics models for practical timesteps.
In the top row, we validate near-equilibrium estimates of the KL divergence on the full state space (x, v). In the bottom row, we validate near-equilibrium estimates of the KL divergence on configuration space (x) alone. Each column corresponds to a numerical method for Langevin dynamics. The darker band in each plot corresponds to the near-equilibrium estimate ± 95% confidence intervals from asymptotic uncertainty estimate (details in section 4.6). The lighter band corresponds to the nested Monte Carlo estimate ± 95% confidence intervals from bootstrapping (details in section 4.7).
In the second panel, we summarize these results by plotting all near-equilibrium estimates vs. all nested Monte Carlo estimates. The colored dots and bars correspond to the means ± uncertainties used in the earlier panels. The dashed diagonal line shows parity. Grey error dots and error bars correspond to conditions where the nested Monte Carlo estimate reached the computational budget (5 × 104 inner-loop samples) but failed to reach the inner-loop uncertainty threshold, and is thus more biased.
3. Discussion
3.1. The near-equilibrium estimator is a fast and useful way to measure integrator error
Here, we introduced and validated a work-based estimator of the KL divergence over the configuration-space marginal sampled by Langevin dynamics. We demonstrated that we could use this estimator to measure differences between the timestep-dependent configuration-sampling error introduced by each member of a small class of Langevin integrators on molecular mechanics models. Work-based estimators are especially attractive for biomolecular systems, since expectations over work distributions are often tractable when other approaches are not.
Reliable estimates of KL divergence using the work-based estimator considered here require knowledge of the time to reach nonequilibrium steady state. This near-equilibrium approach requires the user select a trajectory length T sufficiently large to reach the nonequilibrium steady state, or else the KL divergence estimate could be substantially biased. Opposing the choice of large T is the variance of the estimate, since the contribution of the steady-state work to the variance of the estimate grows as T. Taken together, this suggests the smallest time T that produces unbiased estimates is optimal. In our calculations, it was sufficient to use a protocol that was twice the average collision time, but in general this choice should be validated for the system under study. One way to do this, for example, is to perform the same computation for T and 2 T and ensure estimates are concordant to within statistical error.
Generating equilibrium samples from π(x, v) can be difficult for large, complex molecular systems. The near-equilibrium method requires access to a large number of independent samples from the equilibrium distribution of the system under study. In this work, we used extra-chance HMC to construct a large cache of independent equilibrium samples, amortizing the cost of equilibrium sampling across the many integrator variants (described in Section 4.3). In cases where we would like to compare a large number of integrator variants on the same system, this can be an acceptable cost, but in other cases, it may be prohibitive. It is unknown whether we can relax this requirement in practice, and use only samples in “local equilibrium,” since integrator error may be dominated by local features of the energy landscape. An alternative would be that, if the primary contributions to the dissipation process that drive integration errors arise from high-frequency motions, adding a weak restraint to parts of the system under study (such as a biological macromolecule) may also allow rapid assessment of integrator schemes in a region of configuration space where estimates can be easily converged. We have not yet tested this hypothesis, but if it is possible to relax the requirement of i.i.d. equilibrium samples and retain accurate estimates, then the method will be much cheaper to apply in difficult settings.
3.2. Future directions
The validation of the near-equilibrium estimate makes it possible to apply the technique to a systematic comparison of sampling bias in integrator variants and biomolecular systems. Although we considered only four Langevin integrators here, this approach can be applied to any stochastic integrator for which the relative path action can be computed (see [33] for examples of how to compute the relative action for stochastic integrators not necessarily derived from operator splitting).
Independently, the work-based estimate for ln(ρ(x)/π(x)) we used in the expensive reference method 2.4 could be useful for other analyses. For example, an estimate of ln(ρ(x)/π(x)) could be used to interpret what features of x are most distorted by integrator bias, e.g. by checking which features of x are most predictive of extreme values of ln(ρ(x)/π(x)).
We also note that nothing about the derivation is specific to the partition between configuration degrees of freedom and velocities. We could also use this method to measure the KL divergence over any subset S of the state variables z = (x, v), provided we can sample from the conditional distribution for the complementary subset S′ of the state variables: π(zS′ |zS). To measure KL divergence over the configuration variables, we need only sample from the conditional distribution of velocities given positions, which is typically tractable. Provided that the required conditional distribution is tractable, this method could also prove useful in contexts other than measuring integrator error.
4. Detailed methods
All code used in this paper, along with a manifest of all conda-installable prerequsites and version numbers needed to run the code, is available at https://github.com/choderalab/integrator-benchmark under the permissive OSI-approved MIT license.
A byproduct of this work is a flexible implementation of Langevin integrators derived from operator splitting for the GPU-accelerated OpenMM molecular simulation framework [34], also available under the MIT license in the openmmtools library: https://github.com/choderalab/openmmtools. This implementation allows the user to specify a Langevin integrator using a splitting string (like OVRVO) and can automatically compute shadow work.
4.1. One-dimensional model system: Double-well
For illustration and to have a model system where the exact DKL was readily computable using histograms, we constructed and analyzed a double-well model in 1D. The potential energy function of this model is U(x) ≡ x6 + 2 cos(5(x + 1)), illustrated in Figure 3. We implemented the four Langevin schemes under study using Numba 0.35.0 [35] for use with 1D toy models. We used a temperature of b = 1, a collision rate of γ = 10, and a mass of m = 10. For these conditions, we found a maximum stable timestep of approximately Δt = 0.7. Histogram-based estimates of the configuration-space density and phase-space density used 100 bins per dimension, where the bin edges were set by bounding box of a trial run at the maximum Δt. The equilibrium density of each bin was computed using numerical quadrature (using the trapezoidal rule, numpy.trapz). The KL divergence between a given ρ and π was then computed using scipy.stats.entropy on the histogram representation.
4.2. Model molecular mechanics system: A harmonically restrained water cluster
As noted, the exact Monte Carlo method involves exponential work averages, resulting in a statistical inefficiency that grows rapidly with both the size of the system and the distance from equilibrium. Since we are interested in identifying whether the near-equilibrium approximation breaks down over the timestep Δt range of interest to molecular simulations, it is important to be able to compute a reliable estimate of DKL far from equilibrium. Thus, we aim to select the smallest system we think will be representative of the geometry of molecular mechanics models generally in order to allow the exact estimate to be computable with reasonable computing resources.
To compare the proposed method with a reference estimator, we needed to select a test system which met the following criteria:
The test system must have interactions typical of solvated molecular mechanics models, so that we would have some justification for generalizing from the results. This rules out 1D systems, for example, and prompted us to search for systems that weren’t alanine dipeptide in vacuum.
The test system must have sufficiently few degrees of freedom that the nested Monte Carlo estimator remains feasible. Because the nested estimator requires converging many exponential averages, the cost of achieving a fixed level of precision grows dramatically with the standard deviation of the steady-state shadow work distribution. The width of this distribution is extensive in system size. Empirically, this ruled out using the first water box we had tried (with approximately 500 rigid TIP3P waters [29], with 3000 degrees of freedom). Practically, there was also a limit to how small it is possible to make a water system with periodic boundary conditions in OpenMM (about 100 waters, or 600 degrees of freedom), which was also infeasible.
The test system must have enough disordered degrees of freedom that the behavior of work averages is typical of larger systems. This was motivated by our observation that it was paradoxically much easier to converge estimates for large disordered systems than it was to converge estimates for the 1D toy system.
To construct a test system that met all of those criteria, we used a WaterCluster test system, which comprises 20 rigid TIP3P waters weakly confined in a central harmonic restraining potential with force constant K = 1 kJ/mol/nm2 applied to all atoms. This test system is available in version 0.14.0 of the openmmtools package [36]. Simulations were performed in double-precision using the Reference platform in OpenMM 7.2 [37] to minimize the potential for introducing significant round-off error due to finite floating point precision.
4.3. Caching equilibrium samples
To enable this study, we attempted to amortize the cost of collecting i.i.d. samples from each test system’s equilibrium distribution π and various integrator-and-Δt-specific distributions ρ. Since there are many different ρ distributions and all are relatively small perturbations of π, we invest initial effort into sampling π exhaustively, and then we draw samples from each integrator-specific ρ by running the integrator of interest from initial conditions (x0, v0) ∼ π.
For each test system, we pre-computed a large collection of K = 1000 equilibrium samples
using Extra-Chance Hamiltonian Monte Carlo (XC-HMC) [38,39], implemented as a CustomIntegrator in OpenMM [37]. In brief, XC-HMC is a strategy to reduce the adverse effects of momentum-flipping on sampling autocorrelation times in HMC. HMC uses leapfrog integration of Hamiltonian dynamics as a proposal mechanism, and accepts or rejects each proposal according to a Metropolis criterion. Whenever the Metropolis test fails, the proposal is rejected and the momentum must be reversed; this is necessary to maintain detailed balance [3,40], but can lead to extremely large autocorrelation times when the acceptance rate is not sufficiently close to 100%, see [39] for empirical examples and further discussion. In “extra-chance” HMC, rather than immediately flipping the momentum whenever the Metropolis criterion fails, the proposal trajectory is instead extended to generate a new proposal, and another (suitably modified) Metropolis criterion is checked. For suitable choices of parameters (length of trajectory proposal, timestep, number of extra chances, length of “extra-chance” trajectories), this strategy can virtually eliminate the effect of momentum-flipping, at the cost of increasing the average length of proposal trajectories. Our initial experiments (on larger systems than reported here) suggested that the cost of collecting uncorrelated samples using Generalized HMC without “extra-chances” was prohibitive, since we needed to make the timestep extremely small (around 0.1–0.25 fs) to keep the acceptance rate sufficiently near 100% that the effect of momentum-flipping was acceptable.
Specifically, we equilibrated for approximately 1 ns (105 XC-HMC iterations, 10 steps per XC-HMC proposal trajectory, 15 extra-chance trajectories per iteration, 1 fs per timestep) from an energy-minimized starting structure. We then saved one sample x(i) every 104 XC-HMC iterations afterwards.
To draw an i.i.d. sample from π(x, v), we draw a sample x uniformly from πcache, and then sample v from the equilibrium distribution of velocities conditioned on x, v ∼ π(x|v). (In the presence of holonomic constraints, the velocity distribution is not independent of the configuration distribution. For example, if bond lengths involving hydrogen atoms are constrained, the velocity of a hydrogen minus the velocity of the bonded heavy atom cannot have any component parallel to the bond.)
To draw an i.i.d. sample from ρ, we draw an i.i.d. sample from π and then simulate Langevin dynamics for a large number of steps. We tested using 1000 steps, (1/γ)/Δt steps, and (2/γ)/Δt steps.
4.4 Computing shadow work for Strang splittings
Here, we demonstrate how to compute the appropriate shadow work for a given discrete-timestep Langevin integration scheme. While there has been a great deal of confusion in the literature about how nonequilibrium work should be computed [41], fortunately, there is an unambiguous mechanical (if tedious) approach to the computation of the appropriate work-like quantity.
Assemble the sequence of all steps and substeps of an integrator cycle into a trajectory Z. For example, for OVRVO, we have
where zn ≡ (xn, vn) are phase space points. Let 
denote the time-reversal of all substeps of Z with all velocities negated,
where 
denotes negating the velocity of phase space point zn.
To compute the reduced, unitless shadow work w[Z], we use the definition of work that satisfies the Crooks fluctuation theorem for starting with a sample from the target equilibrium density π(z0) and taking one integrator cycle step (eqn 4 of [1]):
where the target equilibrium density π(k) = ef−h(z), where f is a log normalizing constant (dimensionless free energy), h(z) ≡ u(x) + t(v) is the reduced Hamiltonian, u(x) the reduced potential, and t(v) the reduced kinetic energy [19]. The quantity 
denotes the probability of generating the time-reversed trajectory 
by starting with 
drawn from the target equilibrium density 
(and not the nonequilibrium steady state) and applying the reverse sequence of integrator operations 
which is identical to the forward sequence of integrator operations Λ because the integrators we consider here are symmetric. Since the Hamiltonian is time-independent, the free energy change Δfeq = 0, and this simplifies to
Computation of the shadow work then proceeds by simple mechanical algebra by computing the log ratio of conditional path probabilities in the last term.
For the family of integrators considered here (symmetric Strang splittings of the propagator, composed of R, V, and O steps), the shadow work has an especially simple form:
where Δh is the total change in reduced Hamiltonian, and Δq is the total change in reduced heat across each of the O substeps. We note that accumulation of the shadow work during integration requires no extra force evaluations, and simply requires knowledge of the potential energy at the beginning and end of the integrator cycle as well as the changes in kinetic energy for each O substep.
For OVRVO, this is
We illustrate how to arrive at this result in detail for OVRVO below.
4.5. Computation of shadow work for OVRVO
For OVRVO, we can represent the forward and reverse of a single integrator cycle diagrammatically as
where 
denotes negating the velocity of phase space point zn.
To compute the conditional path probability P[Z|z0], we write a transition probability density kernel for each substep:
We can write the log ratio of conditional path probabilities as
The probability kernels KV and KR are both deterministic, so as long as we are considering a trajectory Z and its time-reverse 
generated by a symmetric integrator splitting, the ratios involving these kernels are unity.
To compute the ratios involving KO kernels, we note that KO(z0, z1) perturbs the velocity according to the update equation
where ξ is a random variate drawn from the unit normal density, which allows us to solve for the random variate required to propagate from z0 to z1,
where the probability density is given by
We can then rewrite the log ratio of O kernels as
Combining this with (33), this provides the overall work as
4.6. Near-equilibrium estimators for KL divergence
We seek the KL divergence between the stationary distribution produced by a given Langevin integrator, and the desired equilibrium distribution. Sivak and Crooks [42] derived a near-equilibrium estimator, in terms of work averages, for the KL divergence between an arbitrary distribution ρ and the equilibrium distribution π. Sivak, Chodera, and Crooks [1] demonstrated how to apply this estimator to the case where ρ is the biased stationary distribution of a single numerical Langevin integrator (OVRVO). For ρ, the stationary distribution resulting from repeated application of the Langevin integrator, the KL divergence is approximately given by,
Here 
indicates an average over the dynamical ensemble produced by initialization in microstates sampled from density p and subsequent driving by protocol Λ. Note that 
represents an expectation computed with respect to an initial distribution (π, Λ) prepared by first sampling over π followed by the application of protocol Λ, subsequently measuring the shadow work over the application of the protocol Λ again; this is distinct from 
which denotes the expectation where the initial sample is selected from π and the shadow work is measured during the execution of protocol Λ.
In this study, we are especially interested in the configuration-space marginal distribution ρx, so we introduce a distribution w(x, v) ≡ ρx(x)π(v|x) that differs from π only in its x-marginal (so that DKL(ω‖π) = DKL(ρx‖πx)) and compute the near-equilibrium approximation for the KL divergence between ω and π:
4.7. Variance-controlled adaptive estimator for KL divergence
As noted in Section 2.4, the nested Monte Carlo estimator we use as an expensive, but in principle asymptotically exact, estimate of the KL divergence requires converging a separate exponential average 
for every sample (x, v) ∼ ρ or ω. It is obviously impossible to compute this exactly; any practical approach employing finite computational resources can only estimate this quantity to some finite statistical precision, and even then, the logarithm will induce some bias in the computed estimate for finite sample sizes. Here, we take the approach of setting a sensible target statistical error for this inner estimate, arbitrarily selecting 0.01, since we would like to resolve features in DKL larger than this magnitude. Notably, the difficulty in achieving this threshold increases exponentially as the width of the sampled distribution p(w) increases with increasing timestep Δt.
To determine the number of inner-loop samples required to meet this statistical error threshold, we periodically compute an estimate of stdev 
from the available samples using the first term of a Taylor expansion. We compare the estimated standard deviation with the user-defined threshold, and continue to draw samples until we meet the threshold or exceed the sample budget of 5 × 104 samples6. The scaling of computational effort with Δt is shown in Figure 7.
The total computational effort is determined by the total number of trajectories sampled in that condition 
multiplied by the length of each trajectory. Note that the curves are not monotonic, since the number of required trajectories increases superlinearly with Δt, but the number of timesteps in each trajectory decreases linearly with Δt.
Choosing the inner-loop threshold is subtle. If the inner-loop threshold is chosen too large, then the resulting estimate will be very biased (the whole procedure only becomes unbiased in the limit that σinner → 0). If the threshold is chosen too small, then the computational effort becomes prohibitive. Controlling the precision of the inner-loop estimates should also be expected to control their bias, since the bias of the inner-loop estimates is approximately 
(see Section II.B, eqn. 8 in [43]), in the direction of under-estimating the DKL.
To compute and report uncertainty in Figure 6, we use bootstrap resampling, rather than Taylor propagation. The data for each condition is a jagged array of sampled work values, where each row represents an “outer-loop sample” (i.e., a different initial condition (x, v) sampled from ρ (or ω)), and the length of each row is variable, reflecting the number of “inner-loop” samples required to estimate 
to the desired precision. To generate a single bootstrap sample, we resample first the rows uniformly with replacement, and then, within each row, resample the columns uniformly with replacement. The error bands for the “exact” estimator in Figure 6 are computed from 100 bootstrap samples per condition.
Appendix Relation to GHMC acceptance rates
What is the relationship between the bias introduced by an integrator at steady state, and the acceptance rate of the corresponding Metropolized integrator? Specifically, why not just Metropolize VRORV to guarantee samples are drawn appropriately from the equilibrium target density π (x)? Following [3,38], we can construct an exact MCMC method that uses one or more steps of Langevin dynamics as a proposal, by using
as the acceptance criterion. The resulting method is called “GHMC” (Generalized Hybrid Monte Carlo). A natural question arises: if an integrator introduces low configuration-space error, is the rejection rate of the corresponding GHMC method also low?
To answer this question, we estimated the GHMC acceptance rate at all conditions for which we have estimated steady-state DKL. Given a collection of equilibrium samples (described in Section 4.3), we can efficiently estimate the acceptance rate of an MCMC proposal by taking the sample average of the acceptance ratio a over proposals originating from equilibrium, (x0, v0) ∼ π.
We compared the GHMC acceptance rate to the histogram-based DKL estimates for the 1D double-well system in Figure A1. There does not appear to be a consistent relationship between DKL and acceptance rate across the four schemes. Notably, the GHMC reject rate can be extremely “conservative” for splittings such as VRORV.
Since the GHMC rejection rate grows similarly with Δt across all four schemes, but the configuration space KL divergence does not, the GHMC rejection rate can be overly “conservative” for some splittings. Panel (a) shows the growth in the GHMC rejection rate as a function of timestep Δt, for the 1D double-well model considered in Figures 2 and 3. On the x-axis is an evenly spaced grid of 50 timesteps between 0.1 and 0.7. On the y-axis is the estimated rejection rate, which is based on a sample average of the GHMC acceptance criterion. The shaded region is the mean ± 95% confidence interval. Panel (b) compares the GHMC rejection rate vs. the phase-space bias at steady state, over the range of timesteps plotted in panel (a). The y-axis is KL divergence between the phase-space histograms, plotted on a log-scale. Panel (c) compares the GHMC rejection rate vs. the configuration-space bias at steady state, over the range of timesteps plotted in panels (a), (b). The y-axis is the KL divergence between the configuration-space histograms, plotted on a log-scale. Note that in panel (c), we have truncated the leftmost parts of the curves for RVOVR and VRORV rejection rates less than 0.05 and 0.1, respectively, due to noise in histogram estimates of very small DKL(ρx‖πx).
Panel (a) shows the growth in the GHMC rejection rate (1 minus the acceptance rate) as a function of timestep Δt (in femtoseconds), for the water cluster test system illustrated in Figure 5. On the x-axis are timesteps [0.1 fs, 0.5 fs, 1.0 fs,…7.5 fs, 8fs]. On the y-axis is the estimated rejection rate, which is based on a sample average of the GHMC acceptance criterion, over 10000 proposals per condition. The shaded region is the mean ± 95% confidence interval. Panel (b) compares the GHMC rejection rate vs. the phase-space bias at steady state, over the range of timesteps plotted in panel (a). The y-axis is the KL divergence between the phase-space distributions as measured by the near-equilibrium estimate, plotted on a log-scale. Panel (c) compares the GHMC rejection rate vs. the configuration-space bias at steady state, over the range of timesteps plotted in panels (a), (b). Note that in panels (b) and (c), we have truncated at DKL ≤ 10−4, due to noise in near-equilibrium estimates of very small DKL.
In all panels, solid lines and shaded regions denote Gaussian fits, while dots denote histogram estimates. The top row depicts work distributions where initial conditions are sampled from the nonequilibrium steady-state induced by the corresponding integrator and timestep ((x, v) ∼ ρ); these shadow work values are used to measure phase-space error in the near-equilibrium estimates of DKL. The bottom row depicts work distributions where initial conditions are sampled from the w ensemble (x ∼ ρx, v ∼ π(v|x)); these work values are used to estimate configuration-space error in the near-equilibrium estimates of DKL.
Next, we compared the GHMC rejection rate with the near-equilibrium DKL estimates for the water cluster considered in Figure A2. A similar pattern is recapitulated in this molecular mechanics model as in the 1D system—there is not a consistent relationship between configuration-space bias introduced by a Langevin integrator and the rejection rate of its corresponding GHMC method.
This complicates the decision of whether to Metropolize or not. As noted in Section 4.3, incurring even a small rejection rate in GHMC can have a large effect on statistical efficiency, due to the effect of momentum flipping. An open challenge is to construct Metropolis criteria for GHMC that might be less “wasteful” for Langevin splittings that introduce low configuration-space bias.
Appendix Statistics of shadow work distributions
The exact expression for the KL divergence used for validation in Section 2.4 requires estimating the expectation of e−w averaged over p(w). Work distributions for various integrators and timesteps are plotted in Figure A3, and appear to be approximately Gaussian, as can be seen by comparison with Gaussian fits (solid lines). As expected, the width of the work distributions increases with increasing timestep, which can be seen more clearly in Figure A4, which plots the standard deviation of the work distribution as a function of timestep.
While the near-equilibrium estimate will find the difficulty of reaching estimates of a given statistical precision grows linearly with the variance in p(w), the exact estimator must converge expectations of the exponentiated shadow work, which becomes exponentially difficult with increasing variance. This effect is illustrated in Figure A5, where we plot e−w along with the Gaussian fits to these work distributions.
It becomes increasingly difficult to estimate the expectation of e−w with respect to Gaussian fits to work distributions π(w) for the water cluster, as the timestep Δtincreases. The four panels increase in Δt from left to right: note the changing x-axis scales. The solid line is the shadow work distribution p(w) measured at each timestep. The dashed line is e−w. The dash-dotted line is e−w • p(w).
Acknowledgments
The authors gratefully acknowledge members of Ben Leimkuhler’s lab (University of Edinburgh) for stimulating discussions on the nature of errors and efficiencies of Langevin integrators; Grant Rotskoff (University of California, Berkeley) for his input on nonequilibrium shadow work measurement schemes; Charlie Matthews and Brian Radak (University of Chicago) for discussions on geodesic integrators; Jason Wagoner (Stony Brook) for discussions about Metropolized Langevin integrators; and members of the Chodera laboratory for input on various aspects of the implementation and theory. We are also grateful to Peter Eastman (Stanford) for implementing the CustomIntegrator facility within OpenMM that greatly simplifies the implementation of the integrators considered here. We also thank the Entropy editors for their patience. JDC acknowledges support from the Sloan Kettering Institute and NIH grant P30 CA008748 and NIH grant R01 GM121505. JF acknowledges support from NSF grant CHE 1738979. DAS acknowledges support from a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and the Canada Research Chairs program.
Footnotes
↵1 These components are also often called A, B, and O in the literature [4,14].
↵2 The R and O components can be integrated exactly for any time increment h, but V can be integrated only to first order.
↵3 The concept of a shadow Hamiltonian has been used to embed this density in a canonical density context, but the shadow Hamiltonian cannot be directly compute, though some approaches to approximate it via expansion (generally requiring higher-order derivatives than gradients) have been proposed – see [28], Chapter 3 of [4], and references therein.
↵4 Discretization or density estimation are infeasible, due to curse of dimensionality. There are direct estimators of the KL-divergence based on Euclidean nearest-neighbor distances that perform well in some high-dimensional settings (e.g. [30]), but Euclidean distance is an unsuitable metric on molecular configurations.
↵5 Note that applying this methodology to non-symmetric integrators (where the integrator and its time-reverse are not identical) would require modifications to this scheme, as well as the manner in which shadow work is computed.
↵6 This limited the maximum CPU time spent collecting an “outer-loop” sample to approximately 2 hours – conditions where this limit was met are colored grey in the lower panel of Figure 6.
References
