## Abstract

While Langevin integrators are widely popular in the study of equilibrium properties of complex systems, it is challenging to estimate the the timestep-induced discretization error: the degree to which the sampled phase space or configuration space probability density departs from the desired target density due to the use of a finite integration timestep. In [1], Sivak *et al.* introduced a convenient approach to quantifying the a natural measure of distribution error between the sampled density and the target equilibrium density, the KL divergence, in *phase space*, but did not specifically address the issue of *configuration-space properties*, which are much more commonly of interest in molecular simulations. Here, we introduce a variant of this near-equilibrium estimator capable of measuring the error in the configuration-space marginal density, validating it against a complex but exact nested Monte Carlo estimator to show that it reproduces the KL divergence with high fidelity. To illustrate its utility, we employ this new near-equilibrium estimator to assess a claim that a recently proposed Langevin integrator introduces extremely small configuration-space density errors up to the stability limit at no extra computational expense. Finally, we show how this approach to quantifying sampling error can be applied to a wide variety of stochastic integrators by following a straightforward procedure to compute the appropriate shadow work, and describe how it can be extended to quantify the error in arbitrary marginal or conditional distributions of interest.

## 1. Introduction

Langevin integrators are widely employed to simulate the equilibrium thermodynamics and kinetics of microscopic molecular systems. The origin of these methods lies in the Langevin equation [2], which describes the behavior of condensed phase systems subject to random weak collisions with fictitious bath particles at thermal equilibrium,
Here, **x** and **v** denote the positions and velocities of all particles in the system in Cartesian coordinates, *t* is time, *M* the diagonal mass matrix, and *U*(**x**) is the potential energy. The constant *σ*^{2} ≡ (2*k _{B}T_{γ}*) quantifies the rate of heat exchange with the bath, with

*k*denoting the thermal energy,

_{B}T*γ*the collision rate (with dimensions of inverse time), and

*W*(

*t*) a standard multidimensional Wiener process [3,4].

On a digital computer, integration of (1) requires discretizing these equations in time to produce a *finite timestep Langevin integrator* capable of generating a temporally discrete dynamical trajectory from which equilibrium or dynamical properties can be estimated [5]. A wide variety of schemes have been proposed for this discretization, which we refer to herein as “Langevin integrators” [6–15].

For example, the popular integrator of Bussi and Parrinello [12] (which we will denote OVRVO for reasons that will become clear shortly), also known as velocity Verlet with velocity randomization (VVVR) [16] due to its use of a velocity Verlet integrator core (substeps VRV below) [17], consists of the discrete update equations:
where *a _{k}* =

*e*

^{−γ(Δt/k)}is a constant that depends on timestep Δ

*t*and collision rate

*γ*, and

*k*denotes the number of O steps appearing in the integrator. The intermediate timestep subscripts (

*t*+ 1/4,

*t*+ 1/2,

*t*+ 3/4) denote variables used solely for convenience (without physical meaning), and the

*z*denote unit normal random variates.

An alternative splitting championed by Leimkuhler and Matthews termed BAOAB [14,18] (which we will refer to as VRORV) consists of a different set of discrete update equations:
While both discrete time integration schemes reduce to the same stochastic differential equations in the limit that Δ*t* → 0, they can behave quite differently for finite timesteps (Δ*t* > 0), especially for timesteps of practical interest for atomistic molecular simulation.

### Langevin integrators introduce sampling bias that grows with the size of the timestep

Langevin integrators intend to sample from the equilibrium density,
where *β* is the inverse temperature, *h*(**x**, **v**) is a separable reduced (unitless) Hamiltonian *h*(**x**, **v**) = *βH*(**x**, **v**) = *u*(**x**) + *t*(**v**), *u*(**x**) ≡ *βU*(**x**) is the reduced potential [19], and *t*(**v**) ≡ *βT*(**v**) is the reduced kinetic energy. If only configuration-space properties are of interest, as is often the case in molecular simulations, Langevin integrators aim to sample from the marginal density in configuration space, *π_{x}*,
if the velocities are ignored while the positions are used to estimate expectations or other configuration-dependent properties.

The continuous-time Langevin equations of motion (1) possess the target equilibrium density *p* (4) as their stationary density, suggesting that, at least in principle, temporal averages along Langevin trajectories can be used to approximate averages with respect to the equilibrium distribution. However, numerical simulations with a finite timestep Δ*t* > 0 will generally sample a slightly different distribution, which we will denote by *ρ*(**x**, **v**), which implicitly depends on timestep Δ*t*. The discrepancy between the distributions *ρ* and *π* will grow with Δ*t* at some rate (*e.g.* until numerical simulation becomes unstable altogether. Note that this phenomenon is completely separate from numerical issues introduced by finite-precision arithmetic on digital computers, which introduces roundoff error in mathematical operations; here, we presume that computations can be carried out to arbitrary precision, and analyze only the effects of time-discretization.

In Figure 1, we illustrate a few key behaviors of this stepsize-dependent sampling bias in a simple quartic 1D system. Note that: (1) the numerically sampled distribution deviates from the target distribution, (2) this deviation increases with timestep Δ*t*, and (3) the deviation in phase space (**x**, **v**) may be different than the deviation in configuration space only (**x**).

### Many discrete-time integrators can be constructed from the continuous Langevin equations of motion

There are several possible ways to discretize Langevin dynamics in such a way that we recover the exact Langevin equation in the limit Δ*t* → 0, and thus recover the desired equilibrium distribution, but as we shall soon see, these methods can have drastically different behaviors for finite timesteps (Δ*t* > 0). A particularly flexible way to construct numerical integrators for Langevin dynamics is operator splitting, where the Langevin system is commonly split into three components, here labeled V, R, and O because they deal with increments to positions (R), velocities (V), or an Ornstein-Uhlenbeck like process (O)^{1},
where each component can be solved “exactly” for a small time increment^{2}.

The Strang symmetric operator splitting approach considers splitting the overall propagator into a product of propagators for the individual R, V, and O components, defining a family of splittings which we will index by strings indicating the order of appearance of these individual propagators. For example, we use OVRVO to refer to the propagator splitting,
While equality between the true propagator and the Strang splitting is only achieved in the limit that Δ*t* → 0, the great advantage of this approach is that each component propagator, and has a corresponding finite-time update equation that can be used to implement the action of that operator on a digital computer:
where *a _{k}* =

*e*

^{−γ(Δt/nO)}and

*ξ ∼ N*(0, 1) is a standard normal random variate drawn for each degree of freedom for each O step. The integers

*n*

_{R},

*n*

_{V}, and

*n*

_{O}denote the number of times each operator appears in the propagator splitting, which is equal to the number of times each corresponding update equation is used in one integrator cycle.

By chaining these operations in the order specified by the splitting string, we can unroll these operations into the sequence of mathematical updates needed to implement one cycle of the integrator for a total time Δ*t*. For VRORV, for example, translating the splitting string into the appropriate sequence of update equations in (8)–(10) produces the equations for one complete integrator timestep:

### Different integrators may approximate different properties with varying fidelity

It has been proposed that some integrators of Langevin dynamics (particularly the VRORV aka “BAOAB” integrator of Leimkuhler and Matthews) preserve the configuration distribution with significantly greater fidelity than other equal-cost integration algorithms [14,18,20]. However, as the formal arguments for this “superconvergence” property rely on a high-friction limit, it is unclear how large the friction coefficient needs to be in practice for the argument to apply. Formal descriptions of the error are typically generic, in that they do not provide guidance on precisely which Δ*t* introduces a tolerable amount of bias for a particular system, and they do not provide a way to predict how other choices, such as mass matrix modifications (*e.g.*, hydrogen mass repartitioning) [21–24]), will affect the error for a system of interest.

Until now, assessing whether specific integrators sample the true equilibrium density with greater fidelity than others in specific settings has relied on computing low-dimensional marginal distributions of manually selected observables perceived to be sensitive to configuration-space sampling errors [25], such as radial distribution functions, marginal distributions of internal coordinates [22], or the configurational temperature [14,26,27]. While it is clear that some observables are more sensitive to errors in configuration space density than others (Figure 2), and the error in the observables of interest is paramount for a particular application, this highlights the risk of using the error in a single physical property as a surrogate for judging integrator quality, as the error in other observables of interest may be large despite small error in the test observable.

To evaluate numerical Langevin integrators, there would be great utility in a *computable, universal measure* of the bias they introduce in specific concrete settings, such that low error in this measure ensures low error in all observables of interest. There is currently no computable measure of the total configuration-sampling bias for concrete choices of integrator parameters and target system.

### KL divergence is a natural measure of sampling bias

Controlling the magnitude of the integrator-induced sampling bias is crucial when computing quantitative predictions from simulation. However, because *ρ* does not have a closed-form, easily computable expression^{3}, it is difficult to quantify the error introduced by a given choice of timestep or integrator. We will show how, for a particularly useful measure of error, we can circumvent this problem and develop a simple, effective approach to measuring error in complex molecular systems.

An ideal measure of the discrepancy between the sampled distribution *ρ*_{x} and the equilibrium distribution *π*** _{x}** should be “universal” in the sense that driving that measure to zero implies that error in any expectation also goes to zero. It should also be defined for all densities, and not rely on a system-specific choice of observables. One such measure is the Kullback-Leibler (KL) divergence

The KL divergence is defined and non-negative for any pair of distributions on the same support, and DKL(*p‖q*) = 0 if and only if *p* = *q*.

### A near-equilibrium estimator can measure timestep-dependent KL divergence for molecular systems

In [1], Sivak and colleagues demonstrated how to approximate the KL divergence of the sampled distribution *ρ* from the target distribution *π over the full phase-space distribution*, *D*_{KL}(*ρ‖π*), in terms of a work-like quantity—the *shadow work* —that is readily computable for a large family of Langevin integrators (Figure 4a). This estimator depends only on the ability to draw samples from *π* and to measure a suitable work-like quantity. This method was applied in [1] to measure the phase space sampling bias introduced by a particular Langevin integrator (OVRVO) on periodic boxes of TIP3P water [29].

Since the velocity marginal is usually not of interest, and since some integrators are thought to preserve the configuration marginal of the target distribution with higher fidelity than the phase space joint distribution, we sought to generalize the technique to estimate the KL divergence of the sampled configuration-space marginal, *D*_{KL}(*ρ*** _{x}**‖

*π*

**). Below, we show how a simple modification of the estimator described in [1] can achieve this goal, and illustrate how this provides a useful tool for measuring the integrator-induced error in configuration-space densities for real molecular systems.**

_{x}## 2. Results

### 2.1. Common Langevin integrator splittings induce comparable timestep-dependent error in phase space

We first applied the original near-equilibrium method of Sivak *et al.* [1] to measure the timestep-dependent phase-space error introduced by four common Langevin integrators on a molecular mechanics model system (a cluster of 20 TIP3P waters [29] in a harmonic restraining potential), for a range of timesteps Δ*t* between 0.1 and 8 femtoseconds (0.1 fs, 0.5 fs, 1.0 fs,…,7.5 fs, 8.0 fs). All four integrator splitting schemes introduced comparable error in phase space (although we could resolve statistically significant differences among the schemes), as illustrated in Figure 5a. This may be unsurprising, as none of these integrator schemes are known to provide a significant reduction in acceptance rates—which depend in some manner on the induced phase space sampling error—for Metropolized versions of these integrators (Figure A1a; see Appendix A for more details about the relationship between Metropolized acceptance rates and KL divergence). Consistent with the results of [1], the phase-space error appears to scale approximately as

### 2.2. A simple modification to the near-equilibrium estimator can compute KL divergence in configuration space

The method in Sivak *et al.* [1] approximates the KL divergence between the nonequilibrium steady state density *ρ* and equilibrium density *π* in terms of two work averages: the work required to drive from equilibrium *π* into the steady state *ρ*, and the steady-state work expended over the same length of time, but starting in *ρ* (Figure 5a).

We can construct an analogous procedure to estimate the KL divergence of arbitrary marginals of *ρ* and *π*, provided we can also sample from an appropriate conditional distribution of *π*. For example, we are interested in estimating the KL divergence between configuration-space marginals *ρ*** _{x}** and

*π*

**, To compute these using the machinery of Sivak**

_{x}*et al.*[1], we replace the second work average with where the expectation of the shadow work is now computed over a modified density constructed from the nonequilibrium steady-state configuration density but with Maxwell-Boltzmann velocity density, Practically, this corresponds to drawing samples from the nonequilibrium steady-state

*ρ*and replacing the velocities

**v**with an i.i.d. sample from the Maxwell-Boltzmann distribution

*π*(

**v**|

**x**). The modified procedure depicted schematically in Figure 4b.

### 2.3. Common Langevin integrators induce substantially different timestep-dependent configuration space error

Figure 5b shows that the measured KL divergence between the configuration-space marginals *ρ*** _{x}** and

*π*

**can be drastically different among the four integrator schemes, and in some cases grow much more slowly than the associated phase-space sampling error (Fig. 5a). In particular, for VRORV, the error in the**

_{x}**x**-marginal is very nearly zero for the entire range of feasible timesteps, and it can be run at Δ

*t*≈ 6 fs while introducing the same amount of configuration error as other methods at Δ

*t*≈ 2 fs. This is consistent with prior findings [4,14], which showed the VRORV scheme introduces very little error in the average potential energy and multiple other system-specific observables.

We also note that for the OVRVO scheme, *D*_{KL}(*ρ*** _{x}**‖

*π*

**) ≈**

_{x}*D*

_{KL}(

*ρ*‖

*π*) over the range of measured timesteps (Fig. 5), consistent with Sivak et al.’s prior findings that estimates of

*D*

_{KL}(

*ρ*‖

*π*) tracked well with several measures of configuration-sampling error.

### 2.4. An exact but expensive estimator of KL divergence validates the near-equilibrium estimate

The accuracy of the near-equilibrium approximation is largely unexplored. While the near-equilibrium approximation introduced by Sivak *et al.* is computationally and statistically appealing, it is important to validate the accuracy of the approximation over the practical timestep Δ*t* range of relevance to molecular simulation. In particular, it is unknown whether the near-equilibrium approximation produces an over-estimate or under-estimate of the KL divergence, or how accurate the approximation is for high-dimensional systems. Further, it is unknown whether any bias introduced by the approximation is uniform across different numerical methods for Langevin dynamics.

How well does the near-equilibrium estimator approximate the true KL divergence of relevant timestep ranges? The task of validating the near-equilibrium approximation is numerically challenging, since we are unaware of exact estimators for *D*_{KL}(*ρ*‖*π*) that remain tractable in high dimensions.^{4} In the case of simple fluids, approximate methods are available that express the KL divergence in terms of a series of *N*-body correlations (as in [31]), typically truncating to 2-body correlation functions (*i.e.* comparing the radial distribution functions). However, in general we don’t know the effect of truncating the expansion, since the successive terms in the series don’t necessarily have decreasing magnitude.

Thus, we provide an asymptotically exact reference method. First, we will derive an exact expression for the KL divergence between *ρ* and *π* in terms of quantities that we can measure, then discuss practical challenges that arise when using this expression, and under what conditions it becomes impractical. We start by writing the KL divergence as an expectation w.r.t. *ρ* (17), since we can’t evaluate *ρ*(**x**, **v**) pointwise, but we can draw samples (**x**, **v**) ∼ *ρ*.

We note that the inner ratio of nonequilibrium steady-state to equilibrium densities, *ρ*(**x**, **v**)/*π*(**x**, **v**), can be expressed in terms of the average of exponentiated nonequilibrium work measured under the application of the time-reversed protocol starting from (**x**, **v**) (18). L denotes the protocol used to generate *ρ* from a sample from *π* —in this case, *T* applications of the Langevin integrator step kernel; denotes the time-reverse of this protocol. Since the protocol we apply to generate *ρ* from *π* is time-symmetric for integrators derived from symmetric Strang splittings, we can substitute ^{5}. In the final step, we substitute a simple Monte Carlo estimator of that average, in terms of work samples *w _{ij}*, where

*w*is the

_{ij}*j*th reduced (unitless) work measurement collected from initial condition

*i*. Here,

*N*is the number of initial conditions sampled (

*i.e.*the number of “outer-loop” samples), and

*M*is the number of work samples (

*i.e.*, the number of “inner-loop” samples) collected at each initial condition (

**x**

*,*

_{i}**v**

*) ∼*

_{i}*ρ*.

The required work values *w _{ij}* can be easily computed from simulations. To sample an initial condition

*i*from

*ρ*, we simply run the Langevin integrator of interest for a sufficient number of steps to sample a new uncorrelated configuration from the nonequilibrium steady-state sampled by the integrator. To compute the work accumulated from a given starting condition, we use the notion of

*shadow work*[32]. For numerical methods constructed from symmetric Strang splittings involving the R, V, and O operations described above, we simply need to compute sum of the total energy changes during the deterministic substeps (

*i.e.*, the potential energy change during deterministic updates of the position variables, and the kinetic energy change during deterministic updates of the momentum variables). For convenience, we use reduced (unitless) energies and work values throughout, where factors of

*k*has been removed, without loss of generality. See Detailed Methods (Section 4.4) for a detailed description on how shadow work can be computed from this family of Langevin integrators in general.

_{B}*T*Like the near-equilibrium scheme, this nested scheme can be modified analogously to measure the configuration-space error in isolation, by initializing instead from the distribution *ω*(**x**, **v**) ≡ *ρ*** _{x}**(

**x**)

*π*(

**v**|

**x**), allowing us to compute

*D*

_{KL}[

*w|π*], a quantity that is identical to

*D*

_{KL}[

*ρ*

**‖**

_{x}*π*

**]: Specifically, to measure the full KL divergence, we sample initial conditions from the Langevin integrator’s steady state: (**

_{x}**x**

_{i},

**v**

_{i}) ∼

*ρ*. To measure configuration-space-only KL divergence, we draw initial configuration from the integrator’s steady state, and velocities from equilibrium:

**x**

_{i}∼

*ρ*

**,**

_{x}**v**

_{i}∼

*π*(

**v**

_{i}|

**x**

_{i}). Note that, for constrained systems,

*π*(

**v**|

**x**) is not independent of

**x**, and care must be taken to eliminate velocity components along constrained degrees of freedom before measuring the contribution of the integrator substep to the shadow work (see Detailed Methods).

We note that the nested plug-in Monte Carlo estimator of the KL divergence is asymptotically exact only when both *N* (the number of “outer-loop” samples) and *M* (the number of “inner-loop” samples) go to infinity. In practice, we use a simple adaptive scheme (described in detail in Section 4.7) that draws inner- and outer-loop samples until uncertainty thresholds are met.

We compared the estimates of the KL divergence on the molecular mechanics system introduced in Figure 5, and confirmed that the two estimates are roughly consistent for all four integrator schemes, over a range of feasible timesteps (Figure 6). At large *D*_{KL}, the near-equilibrium scheme deviates from the nested scheme. However, the nested scheme is likely to be biased in the direction of *under-estimating* the actual *D*_{KL} in this region, and the magnitude of this bias increases with the variance of the shadow work distribution. We conclude that the near-equilibrium approximation is empirically reliable for measuring integrator bias on molecular mechanics models for practical timesteps.

## 3. Discussion

### 3.1. The near-equilibrium estimator is a fast and useful way to measure integrator error

Here, we introduced and validated a work-based estimator of the KL divergence over the configuration-space marginal sampled by Langevin dynamics. We demonstrated that we could use this estimator to measure differences between the timestep-dependent configuration-sampling error introduced by each member of a small class of Langevin integrators on molecular mechanics models. Work-based estimators are especially attractive for biomolecular systems, since expectations over work distributions are often tractable when other approaches are not.

Reliable estimates of KL divergence using the work-based estimator considered here require knowledge of the time to reach nonequilibrium steady state. This near-equilibrium approach requires the user select a trajectory length *T* sufficiently large to reach the nonequilibrium steady state, or else the KL divergence estimate could be substantially biased. Opposing the choice of large *T* is the variance of the estimate, since the contribution of the steady-state work to the variance of the estimate grows as *T*. Taken together, this suggests the smallest time *T* that produces unbiased estimates is optimal. In our calculations, it was sufficient to use a protocol that was twice the average collision time, but in general this choice should be validated for the system under study. One way to do this, for example, is to perform the same computation for *T* and 2 *T* and ensure estimates are concordant to within statistical error.

Generating equilibrium samples from *π*(**x**, **v**) can be difficult for large, complex molecular systems. The near-equilibrium method requires access to a large number of independent samples from the equilibrium distribution of the system under study. In this work, we used extra-chance HMC to construct a large cache of independent equilibrium samples, amortizing the cost of equilibrium sampling across the many integrator variants (described in Section 4.3). In cases where we would like to compare a large number of integrator variants on the same system, this can be an acceptable cost, but in other cases, it may be prohibitive. It is unknown whether we can relax this requirement in practice, and use only samples in “local equilibrium,” since integrator error may be dominated by local features of the energy landscape. An alternative would be that, if the primary contributions to the dissipation process that drive integration errors arise from high-frequency motions, adding a weak restraint to parts of the system under study (such as a biological macromolecule) may also allow rapid assessment of integrator schemes in a region of configuration space where estimates can be easily converged. We have not yet tested this hypothesis, but if it is possible to relax the requirement of i.i.d. equilibrium samples and retain accurate estimates, then the method will be much cheaper to apply in difficult settings.

### 3.2. Future directions

The validation of the near-equilibrium estimate makes it possible to apply the technique to a systematic comparison of sampling bias in integrator variants and biomolecular systems. Although we considered only four Langevin integrators here, this approach can be applied to any stochastic integrator for which the relative path action can be computed (see [33] for examples of how to compute the relative action for stochastic integrators not necessarily derived from operator splitting).

Independently, the work-based estimate for ln(*ρ*(**x**)/*π*(**x**)) we used in the expensive reference method 2.4 could be useful for other analyses. For example, an estimate of ln(*ρ*(**x**)/*π*(**x**)) could be used to interpret what features of **x** are most distorted by integrator bias, *e.g.* by checking which features of **x** are most predictive of extreme values of ln(*ρ*(**x**)/*π*(**x**)).

We also note that nothing about the derivation is specific to the partition between configuration degrees of freedom and velocities. We could also use this method to measure the KL divergence over any subset *S* of the state variables **z** = (**x**, **v**), provided we can sample from the conditional distribution for the complementary subset *S*^{′} of the state variables: *π*(**z**_{S′} |**z**_{S}). To measure KL divergence over the configuration variables, we need only sample from the conditional distribution of velocities given positions, which is typically tractable. Provided that the required conditional distribution is tractable, this method could also prove useful in contexts other than measuring integrator error.

## 4. Detailed methods

All code used in this paper, along with a manifest of all conda-installable prerequsites and version numbers needed to run the code, is available at https://github.com/choderalab/integrator-benchmark under the permissive OSI-approved MIT license.

A byproduct of this work is a flexible implementation of Langevin integrators derived from operator splitting for the GPU-accelerated OpenMM molecular simulation framework [34], also available under the MIT license in the openmmtools library: https://github.com/choderalab/openmmtools. This implementation allows the user to specify a Langevin integrator using a splitting string (like OVRVO) and can automatically compute shadow work.

### 4.1. One-dimensional model system: Double-well

For illustration and to have a model system where the exact *D*_{KL} was readily computable using histograms, we constructed and analyzed a double-well model in 1D. The potential energy function of this model is *U*(*x*) ≡ *x*^{6} + 2 cos(5(*x* + 1)), illustrated in Figure 3. We implemented the four Langevin schemes under study using Numba 0.35.0 [35] for use with 1D toy models. We used a temperature of *b* = 1, a collision rate of *γ* = 10, and a mass of *m* = 10. For these conditions, we found a maximum stable timestep of approximately Δ*t* = 0.7. Histogram-based estimates of the configuration-space density and phase-space density used 100 bins per dimension, where the bin edges were set by bounding box of a trial run at the maximum Δ*t*. The equilibrium density of each bin was computed using numerical quadrature (using the trapezoidal rule, numpy.trapz). The KL divergence between a given *ρ* and *π* was then computed using scipy.stats.entropy on the histogram representation.

### 4.2. Model molecular mechanics system: A harmonically restrained water cluster

As noted, the exact Monte Carlo method involves exponential work averages, resulting in a statistical inefficiency that grows rapidly with both the size of the system and the distance from equilibrium. Since we are interested in identifying whether the near-equilibrium approximation breaks down over the timestep Δ*t* range of interest to molecular simulations, it is important to be able to compute a reliable estimate of *D*_{KL} far from equilibrium. Thus, we aim to select the smallest system we think will be representative of the geometry of molecular mechanics models generally in order to allow the exact estimate to be computable with reasonable computing resources.

To compare the proposed method with a reference estimator, we needed to select a test system which met the following criteria:

The test system must have

**interactions typical of solvated molecular mechanics models**, so that we would have some justification for generalizing from the results. This rules out 1D systems, for example, and prompted us to search for systems that weren’t alanine dipeptide in vacuum.The test system must have

**sufficiently few degrees of freedom that the nested Monte Carlo estimator remains feasible**. Because the nested estimator requires converging many exponential averages, the cost of achieving a fixed level of precision grows dramatically with the standard deviation of the steady-state shadow work distribution. The width of this distribution is extensive in system size. Empirically, this ruled out using the first water box we had tried (with approximately 500 rigid TIP3P waters [29], with 3000 degrees of freedom). Practically, there was also a limit to how small it is possible to make a water system with periodic boundary conditions in OpenMM (about 100 waters, or 600 degrees of freedom), which was also infeasible.The test system must have

**enough disordered degrees of freedom that the behavior of work averages is typical of larger systems**. This was motivated by our observation that it was paradoxically much easier to converge estimates for large disordered systems than it was to converge estimates for the 1D toy system.

To construct a test system that met all of those criteria, we used a WaterCluster test system, which comprises 20 rigid TIP3P waters weakly confined in a central harmonic restraining potential with force constant *K* = 1 kJ/mol/nm^{2} applied to all atoms. This test system is available in version 0.14.0 of the `openmmtools` package [36]. Simulations were performed in double-precision using the Reference platform in OpenMM 7.2 [37] to minimize the potential for introducing significant round-off error due to finite floating point precision.

### 4.3. Caching equilibrium samples

To enable this study, we attempted to amortize the cost of collecting i.i.d. samples from each test system’s equilibrium distribution *π* and various integrator-and-Δ*t*-specific distributions *ρ*. Since there are many different *ρ* distributions and all are relatively small perturbations of *π*, we invest initial effort into sampling π exhaustively, and then we draw samples from each integrator-specific ρ by running the integrator of interest from initial conditions (**x**_{0}, **v**_{0}) ∼ *π*.

For each test system, we pre-computed a large collection of *K* = 1000 equilibrium samples
using Extra-Chance Hamiltonian Monte Carlo (XC-HMC) [38,39], implemented as a CustomIntegrator in OpenMM [37]. In brief, XC-HMC is a strategy to reduce the adverse effects of momentum-flipping on sampling autocorrelation times in HMC. HMC uses leapfrog integration of Hamiltonian dynamics as a proposal mechanism, and accepts or rejects each proposal according to a Metropolis criterion. Whenever the Metropolis test fails, the proposal is rejected and the momentum must be reversed; this is necessary to maintain detailed balance [3,40], but can lead to extremely large autocorrelation times when the acceptance rate is not sufficiently close to 100%, see [39] for empirical examples and further discussion. In “extra-chance” HMC, rather than immediately flipping the momentum whenever the Metropolis criterion fails, the proposal trajectory is instead extended to generate a new proposal, and another (suitably modified) Metropolis criterion is checked. For suitable choices of parameters (length of trajectory proposal, timestep, number of extra chances, length of “extra-chance” trajectories), this strategy can virtually eliminate the effect of momentum-flipping, at the cost of increasing the average length of proposal trajectories. Our initial experiments (on larger systems than reported here) suggested that the cost of collecting uncorrelated samples using Generalized HMC without “extra-chances” was prohibitive, since we needed to make the timestep extremely small (around 0.1–0.25 fs) to keep the acceptance rate sufficiently near 100% that the effect of momentum-flipping was acceptable.

Specifically, we equilibrated for approximately 1 ns (10^{5} XC-HMC iterations, 10 steps per XC-HMC proposal trajectory, 15 extra-chance trajectories per iteration, 1 fs per timestep) from an energy-minimized starting structure. We then saved one sample **x**^{(i)} every 10^{4} XC-HMC iterations afterwards.

To draw an i.i.d. sample from *π*(**x**, **v**), we draw a sample **x** uniformly from *π*^{cache}, and then sample **v** from the equilibrium distribution of velocities conditioned on **x**, **v** ∼ *π*(**x**|**v**). (In the presence of holonomic constraints, the velocity distribution is not independent of the configuration distribution. For example, if bond lengths involving hydrogen atoms are constrained, the velocity of a hydrogen minus the velocity of the bonded heavy atom cannot have any component parallel to the bond.)

To draw an i.i.d. sample from *ρ*, we draw an i.i.d. sample from *π* and then simulate Langevin dynamics for a large number of steps. We tested using 1000 steps, (1/*γ*)/Δ*t* steps, and (2/*γ*)/Δ*t* steps.

### 4.4 Computing shadow work for Strang splittings

Here, we demonstrate how to compute the appropriate shadow work for a given discrete-timestep Langevin integration scheme. While there has been a great deal of confusion in the literature about how nonequilibrium work should be computed [41], fortunately, there is an unambiguous mechanical (if tedious) approach to the computation of the appropriate work-like quantity.

Assemble the sequence of all steps and substeps of an integrator cycle into a trajectory *Z*. For example, for OVRVO, we have
where *z _{n}* ≡ (

**x**

_{n},

**v**

_{n}) are phase space points. Let denote the time-reversal of all substeps of Z with all velocities negated, where denotes negating the velocity of phase space point

*z*.

_{n}To compute the reduced, unitless shadow work *w*[Z], we use the definition of work that satisfies the Crooks fluctuation theorem for starting with a sample from the target equilibrium density *π*(*z*_{0}) and taking one integrator cycle step (eqn 4 of [1]):
where the target equilibrium density *π*(*k*) = *e*^{f−h(z)}, where *f* is a log normalizing constant (dimensionless free energy), *h*(*z*) ≡ *u*(**x**) + *t*(**v**) is the reduced Hamiltonian, *u*(**x**) the reduced potential, and *t*(**v**) the reduced kinetic energy [19]. The quantity denotes the probability of generating the time-reversed trajectory by starting with drawn from the target equilibrium density (and *not* the nonequilibrium steady state) and applying the reverse sequence of integrator operations which is identical to the forward sequence of integrator operations Λ because the integrators we consider here are symmetric. Since the Hamiltonian is time-independent, the free energy change Δ*f*_{eq} = 0, and this simplifies to

Computation of the shadow work then proceeds by simple mechanical algebra by computing the log ratio of conditional path probabilities in the last term.

For the family of integrators considered here (symmetric Strang splittings of the propagator, composed of R, V, and O steps), the shadow work has an especially simple form:
where Δ*h* is the total change in reduced Hamiltonian, and Δ*q* is the total change in reduced heat across each of the O substeps. We note that accumulation of the shadow work during integration requires no extra force evaluations, and simply requires knowledge of the potential energy at the beginning and end of the integrator cycle as well as the changes in kinetic energy for each O substep.

For OVRVO, this is

We illustrate how to arrive at this result in detail for OVRVO below.

### 4.5. Computation of shadow work for OVRVO

For OVRVO, we can represent the forward and reverse of a single integrator cycle diagrammatically as
where denotes negating the velocity of phase space point *z _{n}*.

To compute the conditional path probability *P*[Z|*z*_{0}], we write a transition probability density kernel for each substep:
We can write the log ratio of conditional path probabilities as
The probability kernels *K*_{V} and *K*_{R} are both deterministic, so as long as we are considering a trajectory *Z* and its time-reverse generated by a symmetric integrator splitting, the ratios involving these kernels are unity.

To compute the ratios involving *K*_{O} kernels, we note that *K*_{O}(*z*_{0}, *z*_{1}) perturbs the velocity according to the update equation
where *ξ* is a random variate drawn from the unit normal density, which allows us to solve for the random variate required to propagate from *z*_{0} to *z*_{1},
where the probability density is given by

We can then rewrite the log ratio of O kernels as

Combining this with (33), this provides the overall work as

### 4.6. Near-equilibrium estimators for KL divergence

We seek the KL divergence between the stationary distribution produced by a given Langevin integrator, and the desired equilibrium distribution. Sivak and Crooks [42] derived a near-equilibrium estimator, in terms of work averages, for the KL divergence between an arbitrary distribution *ρ* and the equilibrium distribution *π*. Sivak, Chodera, and Crooks [1] demonstrated how to apply this estimator to the case where *ρ* is the biased stationary distribution of a single numerical Langevin integrator (OVRVO). For *ρ*, the stationary distribution resulting from repeated application of the Langevin integrator, the KL divergence is approximately given by,
Here indicates an average over the dynamical ensemble produced by initialization in microstates sampled from density *p* and subsequent driving by protocol Λ. Note that represents an expectation computed with respect to an initial distribution (*π*, Λ) prepared by first sampling over *π* followed by the application of protocol Λ, subsequently measuring the shadow work over the application of the protocol Λ again; this is distinct from which denotes the expectation where the initial sample is selected from *π* and the shadow work is measured during the execution of protocol Λ.

In this study, we are especially interested in the configuration-space marginal distribution *ρ*** _{x}**, so we introduce a distribution

*w*(

**x**,

**v**) ≡

*ρ*

**(**

_{x}**x**)

*π*(

**v**|

**x**) that differs from

*π*only in its

**x**-marginal (so that

*D*

_{KL}(

*ω*‖

*π*) =

*D*

_{KL}(

*ρ*

**‖**

_{x}*π*

**)) and compute the near-equilibrium approximation for the KL divergence between**

_{x}*ω*and

*π*:

### 4.7. Variance-controlled adaptive estimator for KL divergence

As noted in Section 2.4, the nested Monte Carlo estimator we use as an expensive, but in principle asymptotically exact, estimate of the KL divergence requires converging a separate exponential average for every sample (**x**, **v**) ∼ *ρ* or *ω*. It is obviously impossible to compute this exactly; any practical approach employing finite computational resources can only estimate this quantity to some finite statistical precision, and even then, the logarithm will induce some bias in the computed estimate for finite sample sizes. Here, we take the approach of setting a sensible target statistical error for this inner estimate, arbitrarily selecting 0.01, since we would like to resolve features in *D*_{KL} larger than this magnitude. Notably, the difficulty in achieving this threshold increases exponentially as the width of the sampled distribution *p*(*w*) increases with increasing timestep Δ*t*.

To determine the number of inner-loop samples required to meet this statistical error threshold, we periodically compute an estimate of stdev from the available samples using the first term of a Taylor expansion. We compare the estimated standard deviation with the user-defined threshold, and continue to draw samples until we meet the threshold or exceed the sample budget of 5 *×* 10^{4} samples^{6}. The scaling of computational effort with Δ*t* is shown in Figure 7.

Choosing the inner-loop threshold is subtle. If the inner-loop threshold is chosen too large, then the resulting estimate will be very biased (the whole procedure only becomes unbiased in the limit that *σ*_{inner} → 0). If the threshold is chosen too small, then the computational effort becomes prohibitive. Controlling the *precision* of the inner-loop estimates should also be expected to control their *bias*, since the bias of the inner-loop estimates is approximately (see Section II.B, eqn. 8 in [43]), in the direction of *under-estimating* the *D*_{KL}.

To compute and report uncertainty in Figure 6, we use bootstrap resampling, rather than Taylor propagation. The data for each condition is a jagged array of sampled work values, where each row represents an “outer-loop sample” (*i.e.*, a different initial condition (**x**, **v**) sampled from *ρ* (or *ω*)), and the length of each row is variable, reflecting the number of “inner-loop” samples required to estimate to the desired precision. To generate a single bootstrap sample, we resample first the rows uniformly with replacement, and then, within each row, resample the columns uniformly with replacement. The error bands for the “exact” estimator in Figure 6 are computed from 100 bootstrap samples per condition.

### Appendix Relation to GHMC acceptance rates

What is the relationship between the bias introduced by an integrator at steady state, and the acceptance rate of the corresponding Metropolized integrator? Specifically, why not just Metropolize VRORV to guarantee samples are drawn appropriately from the equilibrium target density *π* (**x**)? Following [3,38], we can construct an exact MCMC method that uses one or more steps of Langevin dynamics as a proposal, by using
as the acceptance criterion. The resulting method is called “GHMC” (Generalized Hybrid Monte Carlo). A natural question arises: if an integrator introduces low configuration-space error, is the rejection rate of the corresponding GHMC method also low?

To answer this question, we estimated the GHMC acceptance rate at all conditions for which we have estimated steady-state *D*_{KL}. Given a collection of equilibrium samples (described in Section 4.3), we can efficiently estimate the acceptance rate of an MCMC proposal by taking the sample average of the acceptance ratio *a* over proposals originating from equilibrium, (**x**_{0}, **v**_{0}) ∼ *π*.

We compared the GHMC acceptance rate to the histogram-based *D*_{KL} estimates for the 1D double-well system in Figure A1. There does not appear to be a consistent relationship between *D*_{KL} and acceptance rate across the four schemes. Notably, the GHMC reject rate can be extremely “conservative” for splittings such as VRORV.

Next, we compared the GHMC rejection rate with the near-equilibrium *D*_{KL} estimates for the water cluster considered in Figure A2. A similar pattern is recapitulated in this molecular mechanics model as in the 1D system—there is not a consistent relationship between configuration-space bias introduced by a Langevin integrator and the rejection rate of its corresponding GHMC method.

This complicates the decision of whether to Metropolize or not. As noted in Section 4.3, incurring even a small rejection rate in GHMC can have a large effect on statistical efficiency, due to the effect of momentum flipping. An open challenge is to construct Metropolis criteria for GHMC that might be less “wasteful” for Langevin splittings that introduce low configuration-space bias.

### Appendix Statistics of shadow work distributions

The exact expression for the KL divergence used for validation in Section 2.4 requires estimating the expectation of *e*^{−w} averaged over *p*(*w*). Work distributions for various integrators and timesteps are plotted in Figure A3, and appear to be approximately Gaussian, as can be seen by comparison with Gaussian fits (solid lines). As expected, the width of the work distributions increases with increasing timestep, which can be seen more clearly in Figure A4, which plots the standard deviation of the work distribution as a function of timestep.

While the near-equilibrium estimate will find the difficulty of reaching estimates of a given statistical precision grows linearly with the variance in *p*(*w*), the exact estimator must converge expectations of the *exponentiated* shadow work, which becomes exponentially difficult with increasing variance. This effect is illustrated in Figure A5, where we plot *e*^{−w} along with the Gaussian fits to these work distributions.

## Acknowledgments

The authors gratefully acknowledge members of Ben Leimkuhler’s lab (University of Edinburgh) for stimulating discussions on the nature of errors and efficiencies of Langevin integrators; Grant Rotskoff (University of California, Berkeley) for his input on nonequilibrium shadow work measurement schemes; Charlie Matthews and Brian Radak (University of Chicago) for discussions on geodesic integrators; Jason Wagoner (Stony Brook) for discussions about Metropolized Langevin integrators; and members of the Chodera laboratory for input on various aspects of the implementation and theory. We are also grateful to Peter Eastman (Stanford) for implementing the CustomIntegrator facility within OpenMM that greatly simplifies the implementation of the integrators considered here. We also thank the *Entropy* editors for their patience. JDC acknowledges support from the Sloan Kettering Institute and NIH grant P30 CA008748 and NIH grant R01 GM121505. JF acknowledges support from NSF grant CHE 1738979. DAS acknowledges support from a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and the Canada Research Chairs program.

## Footnotes

↵

^{1}These components are also often called A, B, and O in the literature [4,14].↵

^{2}The R and O components can be integrated exactly for any time increment*h*, but V can be integrated only to first order.↵

^{3}The concept of a*shadow Hamiltonian*has been used to embed this density in a canonical density context, but the shadow Hamiltonian cannot be directly compute, though some approaches to approximate it via expansion (generally requiring higher-order derivatives than gradients) have been proposed – see [28], Chapter 3 of [4], and references therein.↵

^{4}Discretization or density estimation are infeasible, due to curse of dimensionality. There are direct estimators of the KL-divergence based on Euclidean nearest-neighbor distances that perform well in some high-dimensional settings (*e.g.*[30]), but Euclidean distance is an unsuitable metric on molecular configurations.↵

^{5}Note that applying this methodology to non-symmetric integrators (where the integrator and its time-reverse are not identical) would require modifications to this scheme, as well as the manner in which shadow work is computed.↵

^{6}This limited the maximum CPU time spent collecting an “outer-loop” sample to approximately 2 hours – conditions where this limit was met are colored grey in the lower panel of Figure 6.