Abstract
Meta-analyses are frequently used to quantify the difference in the average values of two groups (e.g., control and experimental treatment groups), but examine the difference in the variability (variance) of two groups. For such comparisons, the two relatively new effect size statistics, namely the log-transformed ‘variability ratio’ (the ratio of two standard deviations; lnVR) and the log-transformed ‘CV ratio’ (the ratio of two coefficients of variation; lnCVR) are useful. In practice, lnCVR may be of most use because a treatment may affect the mean and the variance simultaneously. We review current, and propose new, estimators for lnCVR and lnVR. We also present methods for use when the two groups are dependent (e.g., for cross-over and pre-test-post-test designs). A simulation study evaluated the performance of these estimators and we make recommendations about which estimators one should use to minimise bias. We also present two worked examples that illustrate the importance of accounting for the dependence of the two groups. We found that the degree to which dependence is accounted for in the sampling variance estimates can impact heterogeneity parameters such as τ2 (i.e., the between-study variance) and I2 (i.e., the proportion of the total variability due to between-study variance), and even the overall effect, and in turn qualitative interpretations. Meta-analytic comparison of the variability between two groups enables us to ask completely new questions and to gain fresh insights from existing datasets. We encourage researchers to take advantage of these convenient new effect size measures for the meta-analysis of variation.
1. INTRODUCTION
Meta-analysis is often used to evaluate studies comparing the average of two groups. These are usually treatment groups in an experiment/trial, one being a concurrent control, but may also represent naturally occurring groups (e.g., different sexes). The standardised mean difference (SMD; also known as Cohen’s d and its associated derivatives), which is the difference between group means divided by the within-study variability, is a commonly-used effect size measure for this purpose1. SMD is popular because it is ‘unitless’, meaning it can be used to compare the results of studies that report outcomes in different units2 A similar unitless effect size measure, which can also be used to compare the means of two groups, is the logarithm of the ratio of the means of the groups. This effect size measure is known as the ratio of means in medicine (ROM3) and the log response ratio in ecology and evolution (lnRR4). Throughout we follow the lnRR notation as this will help to draw parallels with other effect size measures as we progress, but the reader should not be confused with the (logarithm of) risk ratio, which is also sometimes denoted (ln)RR. Surveys have shown that lnRR is the most widely used effect size measure in ecology and evolution5–7 Moreover, SMD and lnRR collectively account for over half of all meta-analyses in ecology6,7 meaning comparisons between group means is the most widespread aim of meta-analysis in this field. SMD also seems to be among the most used standardised effect statistics in the medical and social sciences8.
Two groups may not only differ in terms of their means, but also their variances9. At the most basic level, experimental treatments may directly increase or decrease the total amount of variance in a system due to inter-individual variability in response. Many biological systems also appear to display a mean-variance relationship10–12; most commonly, increasing averages are associated with increasing variances. Perhaps the most well-known example of a biological mean-variance relationship comes from ecology and is known as Taylor’s Law. This ‘law’ has been widely observed, and states that as mean population density increases, variance in population density also increases13,14. Where mean-variance relationships are present, a treatment may indirectly cause groups to have differing variances by altering the mean.
Nakagawa, Poulin, Mengersen, et al.15 proposed a number of methods that allow the user to test for differences in the variance of groups meta-analytically. Among the methods proposed, the logarithm of the ratio of the standard deviations (SDs), named log ‘variability ratio’ (lnVR) and the logarithm of the ratio of the coefficients of variation (CV), termed the log ‘CV ratio’ (lnCVR), are most readily integrated into the standard meta-analytic paradigm; i.e. a contrast-based model using an effect size that corresponds to an effect relative to a concurrent control16,17. Of the two, lnCVR is perhaps the more useful measure where a mean-variance relationship is likely to exist. Nakagawa, Poulin, Mengersen, et al.15 highlight that meta-analysing variation may be used to answer completely novel questions, but it can also be used to provide fresh insights into the topics on which a meta-analysis of means was already conducted. Indeed, lnCVR has already been applied in such diverse fields as ecology18, evolution19, agriculture20, health17, and the social sciences21. It is important to note that lnCVR (and also lnVR) require the same data to calculate as is already needed for computing SMD or lnRR values.
Our aims in this paper are threefold. First, we review existing and propose new estimators for lnCVR and its sampling error variance. These include, for the first time, estimators of the sampling variance when the two groups (treatment and control) are not independent (as may occur, for example, in cross-over trials or in paired, single-subject, or pre-test-post-test designs). Second, we conduct a simulation study to investigate the performance of the different estimators. Finally, we present two case studies using these techniques, and illustrate the importance of accounting for dependence between the two treatment groups in the estimation of sampling variation and other heterogeneity parameters (e.g., τ2, the between-study variance, and I222).
2. METHODS
2.1 Point estimators when groups are independent
Let xT ~ N(μT, σT) and xc ~ N(μC, σc) denote normally distributed random variables with true means (i.e., expected values) given by μT and μC and true standard deviations σT and σC. For independent random samples based on these variables (e.g., representing some outcome of interest measured in a treatment and control group) of size nT and nC, let 
and 
denote the respective sample means and sT and sC the corresponding standard deviations for the two groups. Then comparisons between the means, variances and coefficients of variation for two groups can be made using the lnRR, lnVR and lnCVR effect size measures, respectively. “Naïve” estimators of these effect statistics are:
where ln denotes the natural logarithm and 
and 
denote the coefficients of variation in the treatment and control group, respectively.
While these naïve estimators are consistent and asymptotically unbiased, we can add corrections for the sample size based on a second-order Taylor expansion (also known as, the second order delta method) for each statistic15,23,24. For the lnRR, Lajeunesse23 demonstrated such a correction is important to obtain unbiased estimation especially when sample size is small;
Similarly, for the lnVR, Nakagawa, Poulin, Mengersen, et al.15 proposed:
Combing lnRR2 and lnVR2, one obtains:
Note that Nakagawa, Poulin, Mengersen, et al.15 originally suggested the an estimator of lnCVR that missed the bias correction pertaining to lnRR (i.e. 
). We also note that an alternative estimator of lnCVR could also be obtained based on 
CV, which it has been suggested acts as a ‘rough’ bias correction for the CV (e.g.25). However, this estimator is not recommended here, and it does not perform well (see Supplementary Materials S1, Text S1).
2.2 Dispersion estimators when the two groups are independent
The original estimators of the sampling (error) variance for lnRR4 and lnVR15 are based on the first-order Taylor expansion; they are respectively:
Based on these, for lnCVR Nakagawa, Poulin, Mengersen, et al.15 proposed:
where ρ is the correlation between the log mean and log SD. Nakagawa, Poulin, Mengersen, et al.15 suggested that ρ can be estimated based on the correlation between the log sample mean and log sample SD across the studies included in a meta-analysis. However, in doing so one risks conflating within- and between-study correlation (i.e., the correlation in the bivariate sampling distribution of the sample mean and sample SD could be very different than the correlation of the true means and SDs across studies). In fact, for observations that come from an underlying population distribution that is symmetric (e.g. a normal distribution), the sample mean and variance are uncorrelated26. Thus, for the case considered here where ρ = 0 the equation above simplifies to:
As a better estimator for the sampling variance of lnRR, Lajeunesse23 derived and tested the following sampling variance based on the second-order Taylor expansion:
Similarly, we can derive the following sampling variance for lnVR based on the second-order Taylor expansion as:
Accordingly, the complete estimator of the sampling variance for lnCVR, based on s2(lnRR2) and s2(lnVR2) is:
In the supplementary materials, we propose estimators of the sampling covariance based on the above, which can be used when multiple treatment groups are contrasted with the same control27 (see Supplementary Materials S1, Text S2)
2.3 Point estimators when groups are dependent
Due to experimental design, control and treatment groups are often not independent of one another. A clear example of this dependency is in the case of a cross-over design where the same individuals are subjected to both control and experimental treatments at two different time points. The point estimates given above will perform the same way regardless of whether we are dealing with independent or dependent groups. In cross-over studies, however, nT = nc ≡ n, unless dropouts are included in a pre-post design, in which case we recommend that n = npost (i.e. the sample size in the post-treatment condition) is used. This is because the correlation between pre and post-treatment measurements can only be calculated based on n, which assumes nT = nc (see the next section). We can rewrite the dependent cases of lnRR1 and lnRR2 as:
where subscripts 3 and 4 indicate the naïve estimator and estimator based the second-order Taylor expansion, respectively. Similarly, for lnVR and lnCVR, we have:
2.4 Dispersion estimators when the two groups are dependent
In dependent cases estimates of the sampling variance need to account for the correlation between measurements from the same replicates on the two occasions (i.e. cross-correlation28). Based on the first-order Taylor expansion, the sampling variance for lnRR is:
where rCT is a cross-context correlation value estimated from the two sets of measurements on the same replicate when they are under the control and treatment conditions29 As discussed above for dependent studies nτ = nc ≡ n meaning s2(lnRR3) simplifies to:
If based on the second-order Taylor expansion23, the estimator of the sampling variance for lnRR is:
We can also derive the sampling variance for dependent cases of lnVR based on the first- order Taylor expansion as:
which, where nT = nc = n, simplifies to:
Based on the second-order Taylor expansion, we have the sampling variance for dependent cases of lnVR as:
From the sampling variances for lnRR and lnVR, we have the sampling variance for lnCVR with first- and second-order Taylor expansion as:
Note that, where r is positive the estimated sample variance for a dependent estimator will be smaller than its independent equivalent, but that as r shrinks to 0 the dependent case converges on the independent; e.g. assuming nc = nT, where r > 0, s2(lnCVR3) < s2(lnCVR1), but where r = 0, s2(lnCVR3) = s2(lnCVR1).
3. SIMULATION
3.1 Simulation study design
We simulated a two-group experiment/trial, where a pair of groups is based on nT and nC random samples drawn from populations under an experimental treatment and control conditions. The treatment and control populations have means μT and μC and standard deviations (SDs) σT and σC, respectively. The ith sample in each group, yTi (i = 1 … nT) and yCi (i = 1 … nC) was drawn from a bivariate normal distribution as follows:
Where 
are the population means of the two groups, 
is a variance co-variance matrix specifying the variances of the two groups with pCT giving the degree of correlation among the ith samples in the two groups and all other parameters are as above. When where pCT ≠ 0 the ith data in the two groups are correlated (i.e. dependent or paired samples as in a cross-over design).
In all simulations, μC = 100 and σC = 20, which across the parameters tested ensures positive sample means (required for log transformation). We explored values of μT ranging between μC × e-0.5 and μC × e0.5 and values of σT ranging between σC × e0.5 and σC × e0.5, meaning the ln(μT/ μC) and ln(σT/ σC) is between −0.5 and 0.5. All combinations were explored and where ln(μT / μC) = ln(σT / σC) the coefficient of variance (CV) of the two groups will be identical. We explored nC = 8, 16 and 42, with nC = nT and, with nC < nT (independent case). We also explored pCT = 0 and pCT = 0.8. For each set of parameters, we simulated 100,000 experiments.
Based on the sample means and SDs of each simulated experiment, we calculated lnCVR1 and lnCVR2 for independent cases (ρCT ≠ 0) and lnCVR3 and lnCVR4 for dependent cases (ρCT ≠ 0). We also calculated the sampling variance estimators s2(lnCVR1) and s2(lnCVR2) where ρCT ≠ 0, and s2(lnCVR3) and s2(lnCVR4) where ρCT ≠ 0. We calculated bias in the ith estimator as:
where k is the kth value of K (here 100,000) simulated values of lnCVRi (k = 1… K). This bias can be interpreted as the mean deviation of the ith estimator of lnCVR from the true population value. We calculated bias in sampling variance estimator i as:
where s2(lnCVRi) is the value of the ith sampling variance based on the simulated population statistics and sample sizes and θj is the SD among K simulated effect sizes estimated using estimator j. This bias can be interpreted as the percentage by which the sampling variance estimator deviates from the true value (i.e. 100 = the estimator is twice the true value). We calculated coverage as the proportion of 95% confidence intervals (CIs) that include In 
. For a combination of the jth effect size estimator (lnCVRj) and ith sampling variance s2(lnCVRi), 95% CIs were constructed as:
where lnCVRj is the estimated effect size for the simulated sample, s(lnCVRi) an estimate of the standard error (SE; the square root of the estimated sampling variance), and z0.975 is the function of the 0.975th quantile of a z distribution (approx. 1.96). Simulations and analyses were performed in R v3.5.1;, and using the ‘mvrnorm’ function in the MASS package31. All data and code presented in this manuscript can be found at (https://github.com/AlistairMcNairSenior/lnCVR_Estimators_Sim).
3.2 Simulation results
We begin with the case where the two groups are independent (ρCT = 0). Figure 1 shows bias in the estimated effect as a function of sample size and the log the ratio of the means and SDs in the two groups. Across the diagonal elements of each plot (black-dashed line) the underlying CV of the two populations is identical (even if the means and SDs differ; 
), elements above the line correspond to the CV of the treatment population being greater than that of the control group 
, and elements below the line the opposite 
. lnCVR1 overestimates positive effects and slightly under-estimate negative effects, with bias being most profound where the sample size is small. lnCVR2, on the other hand, displays no systematic bias. Figure 2 shows the results where the sample size of the treatment group is ~25% greater than that of the control group. lnCVR1 showed severe upward bias, especially where the sample size was small, where as lnCVR2 performed with only very minor upward bias, which all but disappeared for larger sample sizes. Given that lnCVR2 was determined to be the most accurate estimator of the effect, we proceeded to explore how lnCVR2 performed in conjunction with different estimators of sampling variance.
Bias in effect size estimators of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (balanced) for the case of independent treatment and control group data (ρCT = 0). Black dashed line indicates no effect (i.e., lnCVR = 0).
The first sampling variance estimator s2(lnCVR1) underestimated the variance among simulated values of lnCVR2, particularly where the sample size was small (Figure 3). Biases for s2(lnCVR2) were minimal, although there was some very slight upward bias for small sample sizes and large positive effects (Figure 3). The coverage of 95% CIs for s2(lnCVR1) and s2(lnCVR2) (paired with lnCVR2) are shown in Figure 4. s2(lnCVR1) generated CIs that were too narrow at smaller sample sizes, whereas again s2(lnCVR2) performed with little bias. At larger sample sizes coverage was much closer to the nominal level (Figure 4), although s2(lnCVR2) still performed more accurately. The same patterns of performance were observed for the case where nC < nT (Supplementary Figures S1 and S2).
Bias in effect size estimators of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (unbalanced) for the case of independent treatment and control group data (ρCT = 0). Black dashed line indicates no effect (i.e., lnCVR = 0).
Bias in sampling variance estimators of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (balanced) for the case of independent treatment and control group data (ρCT = 0). Black dashed line indicates no effect (i.e., lnCVR = 0).
Coverage of 95% CIs based on estimators of the sampling variance of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (balanced) for the case of independent treatment and control group data (ρCT = 0). Black dashed line indicates no effect (i.e., lnCVR = 0).
For the case where treatment and control samples were dependent on one another (ρCT = 0.8) lnCVR4 out-performed lnCVR3, with a pattern identical to that in Figure 1 (Figure S3). With regards the two estimators for dependent sampling variances, s2(lnCVR3) underestimated the variance where as s2(lnCVR4) overestimated the variance (Figure 5). These biases were within a reasonable range for larger samples, but were severe for small samples, and v(lnCVR4) in particular showed extreme upward bias (reaching 60% overestimate) when the SD of the treatment group differed from that of the control group (Figure 5). The CIs generated by v(lnCVR;) had a tendency to be too narrow whereas those generated by s2(lnCVR4) were too wide (Figure 6).
Bias in sampling variance estimators of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (balanced) for the case of dependent treatment and control group data (ρCT = 0.8). Black dashed line indicates no effect (i.e., lnCVR = 0).
Coverage of 95% CIs based on estimators of the sampling variance of lnCVR as a function of the log ratio of population means (x-axis), SDs (y-axis) and sample size (balanced) for the case of dependent treatment and control group data (pCT = 0.8). Black dashed line indicates no effect (i.e., lnCVR = 0).
4. WORKED EXAMPLES
We now provide two examples: one from the field of ecology and the other from the health sciences. All meta-analytic models (random-effects meta-analysis) were fitted using the ‘rma’ function (with default settings) in metafor32
4.1 Example 1: Carbon dioxide levels and plant mass
Curtis, Wang33 performed a meta-analysis of experimental studies that tested for the effects of elevated carbon dioxide (CO2) levels on woody plant mass. Briefly, these studies compared the total biomass (above and below ground) of plants grown under ambient and artificially elevated (~100% increase) CO2 levels. Studies were performed in a range of contexts, including highly controlled (e.g., green houses) and less controlled (e.g., field sites) environments, as well as across temperature, light, water, and soil-fertility levels. Replication was at the level of the locale (e.g., plot/site/greenhouse) at which a treatment was applied, and treatment/control groups may be correlated (i.e., non-independent) if, for example, locales experiencing different treatments are paired spatially or temporally. However, the degree to which such correlations are present was not stated. Aggregating 102 effect sizes (lnRR), Curtis, Wang33 found that the mean biomass of woody plants at a site increases by, on average, 28.8% under elevated CO2 conditions. However, there was evidence that the effect is moderated by the presence of other stressors such as under nutrient- or light-limited conditions.
Here we ask whether elevated CO2 levels also increase among-replicate variability in plant biomass using lnCVR. We tested the sensitivity of the analysis to the assumption that treatment and control groups are uncorrelated. Because we do not know precisely which effect size data come from paired designs, we calculated effect sizes and sampling variance assuming complete independence (0% of effect sizes have correlated groups), varying degrees of partial dependence (a random subset of 20%, 60%, or 80% effect sizes have correlated groups; rCT = 0.8), or complete dependence (100% of effect sizes have correlated groups; rCT = 0.8). For those effect sizes that were assumed to be uncorrelated we used lnCVR2 and s2(lnCVR2), and for those that are correlated lnCVR4 and s2(lnCVR3).
There was evidence for a mean-variance relationship under both elevated and ambient CO2 levels (Figure 7A). The influence of increasing the percentage of effect sizes that are assumed to come from correlated groups on a random-effects meta-analysis is shown in Table 1. There are some qualitative differences in the interpretation of the overall effect, whereby the associated CI spans zero in some cases, but not others (Table 1). In all cases the sign of the overall effect is stable and suggests that elevating CO2 levels on average decreases the CV in biomass among replicates (possibly by somewhere between 100 × (1 − exp(−0.078) = 7.5 to 100 × (1 − exp(−0.116) = 10.9 percent). The effect of increasing the number of studies with correlated groups on the estimated inter-effect size heterogeneity, is however, much more dramatic. As less independence is assumed, the amount of heterogeneity (absolutely, in terms of 
, and relatively, in terms of I2) increases substantially (Table 1), such that when 40% or more of the studies are have used paired designs, Cochran’s Q test yields a significant result.
Association between log sample mean 
and log sample standard deviation (ln s) for treatment (hollow points) and control (solid points) groups in the data from; A) Curtis, Wang33, where the outcome is woody plant biomass under elevated (treatment) vs ambient (control) CO2 levels; and B) Brand-Miller et al. (2003) where the outcome is a measure of glycemia in diabetic individuals on low (treatment) vs high (control) glycemic index diets. Note in (B) measures of glycemia are either fructosamine (black points) or HbA1c (red points) levels, where lower levels indicate better gylcemic control.
Estimates of the overall effect (lnCVR) and heterogeneity statistics from randomeffects meta-analyses of woody plant biomass under elevated vs ambient CO2 levels. Negative estimates indicate lower CV under elevated CO2. Models were refitted based on effect sizes assuming increasing % of the effect sizes contain correlated (dependent) treatment and control group data (rCT = 0.8). LCI and UCI indicate the lower and upper 95% confidence interval bounds. Data from Curtis, Wang33.
4.2 Example 2: Low glycemic index diets and glycemic control in diabetic subjects
Brand-Miller, Petocz, Hayne, Colagiuri34 performed a meta-analysis of studies designed to test the effects of low glycemic index (GI) diets on bio-markers of glycemic control in diabetic (type 1 and 2) individuals. Individuals were given either low or high GI diets, after which glycemia was measured using HbA1c and/or fructosamine levels. These two markers quantify glycemia over longer vs shorter time periods respectively, where lower levels indicate better glycemic control. The studies differed somewhat in the overall GI of the diets used and the duration for which subjects were on the diets. The studies used a mixture of parallel designs where the individuals in each treatment group are completely independent, and cross-over designs where each individual was subject to both treatments. Brand-Miller, Petocz, Hayne, Colagiuri34 acknowledged that for those studies with a cross-over design, there will be a degree of correlation among the treatment and control condition data. They tested the sensitivity of their results to any such correlation by repeating the analyses assuming complete independence (rCT = 0) and also assuming that groups are correlated (rCT = 0.34; based on one of the studies in their primary literature). Their analyses of 14 effect sizes (mean differences, expressed in terms of percent; 11 from studies with cross-over designs) suggested that measures of glycemia are decreased by 6.8 percentage points (improved glycemic control) on low GI diets irrespective of their assumptions about correlations among groups. The authors used a fixed-effect meta-analytic model, and did not present heterogeneity statistics.
We tested whether low GI diets affect inter-individual variability in glycemic control using lnCVR. Unlike example 1, here we do know which studies contain dependent groups (those with cross-over designs), although the strength of the dependence is not precisely known. For independent designs we calculated effect sizes and sampling variances via lnCVR2, and s2(lnCVR2). For those studies using a cross-over design we calculated lnCVR4 and s2(lnCVR3) assuming treatment and control data are correlated with rCT = 0, 0.3, 0.5, and 0.8. Where more than one measure of glycemia was presented from a single study, we primarily use fructosamine levels (this being the more widely reported measure).
We observed a mean-variance relationship amongst both measures of glycemic control within the two treatment groups (Figure 7B). The results of random-effects meta-analyses fitted to the effect sizes are given in Table 2. The analyses estimated that on low-GI diets the CV in biomarkers of glycemic control is on average reduced by between 13% (100 × (1 − exp(−0.135)) and 18% (100 × (1 − exp(−0.177)) compared to high-GI diets. However, as the degree of correlation among data from cross-over trials increased, there was a marginal reduction in the overall effect magnitude and an increase in the associated SE (Table 2); for rCT = 0.5, the overall effect was not statistically significant. With increasing correlation, heterogeneity also increased (Table 2). Where we assumed complete independence (rCT = 0), there was no evidence for heterogeneity, but for rCT = 0.8, we detected inter effect size heterogeneity (Table 2).
Estimates of overall effect (lnCVR) and heterogeneity from random-effects metaanalyses of glycemic control in diabetics on low- vs high-GI diets. Negative estimates indicate lower CV on a low-GI diet. Models were refitted from effect sizes assuming differing strength of correlation (rCT) among repeated measured from the same individuals in cross-over trials. LCI and UCI indicate the lower and upper 95% confidence interval bounds. Data from Brand-Miller, Petocz, Hayne, Colagiuri34.
5. DISCUSSION AND CONCLUSIONS
We recommend that meta-analysts use the following estimator of the lnCVR for independent study designs:
For dependent study designs we recommend the use of the following point estimator:
Under the simulated conditions explored, these estimators exhibited minimal bias, where ‘naïve’ estimators displayed systematic biases, substantially overestimating large positive effects, especially when sample sizes were small. Compared to previous estimators15, this revision contains an additional term, 
, which has also been shown to reduce bias in mean effects estimated via lnRR23. We also recommend that the following estimators for the sampling variance of lnCVR be used for independent and dependent study designs, respectively:
Our simulations demonstrate that the estimator for independent designs performs very well and 95% CIs based on a z distribution give coverage at the nominal level. The estimator for dependent cases slightly underestimates the actual sampling variance in lnCVR, and will generate CIs (based on z or t distributions) that are slightly too narrow. This might be due to the substitution of rCT for the unknown true correlation in the equation for the sampling variance without further account of the additional source of uncertainty this introduces. CIs that are too narrow may be more troublesome in that they can lead to inflated type-1 error rates (a more conservative estimator, s2(lnCVR4), is given above, although this approach may substantially overestimate the sampling variance for small samples). Note that these recommended estimators are now available in the ‘escalc’ function in the development version of metafor (https://github.com/wviechtb/metafor), and will eventually be implemented in the CRAN version.
We used the recommended estimators to evaluate whether: 1) increased CO2 levels affect variation in woody plant biomass, and 2) low-GI diets alter between-individual variation in glycemic control in diabetics. In both cases, we found that the treatments have a tendency to decrease the CV. In both cases the analyses were sensitive to assumptions about the degree to which treatment and control data are correlated. Assuming higher degrees of correlation resulted in small changes in the overall effect (and its standard error). Although these parameters were relatively stable, for estimates with CIs close to zero, changing assumptions about group independence can affect inference. Increasing the degree of correlation dramatically increased the estimated between-effect size heterogeneity, which could change conclusions about the consistency of the reported effects. This trend can be explained by the fact that as more/stronger correlations are assumed the sampling variances associated with the individual effect sizes shrink, effects are assumed to be more precise, and sampling variability therefore becomes less able to explain the variation among the effects. Our results corroborate the points made by Becker28, who introduced an estimator for the sampling variance of SMD for dependent groups.
As is the case with any exercise in data analysis, the most appropriate technique to use will depend on the question being asked. Where the analyst is able to determine with a reasonable degree of certainty that a mean-variance relationship does not exist, lnVR may be a more useful measure of between-group differences in variability than lnCVR. This is because lnCVR risks conflating effects on the SD with effects on the mean. In other instances, the user may be more interested in ascertaining whether a treatment alters the SD irrespective of a mean-variance relationship (e.g., in questions related to power and study design), and again lnVR would be an appropriate choice. However, where mean-variance relationships are present, and the analyst is interested in whether the variation is greater/lower than expected given the mean, lnCVR is useful. For some matters, it may even be common practice for the primary literature to describe variation in terms of CV rather than SD. For instance, in ecology and evolution it is common to present CV when comparing variability amongst species/traits that exist on different scales because CV is a relative measure35. We note that such a practice is not necessarily required for meta-analysis because lnVR is also a relative measure of variation, and as such should also do a good job of correcting for inter-system differences in scale. Nevertheless, where CV is the measure of variability commonly reported in the primary literature, the user may find it intuitive (or even necessary) to use lnCVR.
Nakagawa, Poulin, Mengersen, et al.15 also present alternative arm-based models (and discuss bivariate models) for meta-analysis of variation. The lnCVR metric assumes that changes in the mean are associated with proportional changes in the SD. Arm-based (and bivariate) models are an alternative for meta-analysis which allow the user to circumvent the assumption of proportionality. Arm-based models, however, are not without their critics who argue that these methods are radical departure from established meta-analytic thinking (see16). Like other (contrast-based) effect size measures that reflect the difference between two groups (e.g., the standardized mean difference, log response ratio, log risk/odds ratio or the risk difference), lnCVR readily integrates with our most widespread analytical paradigms, offering a convenient and intuitive method for meta-analysis of variability.
Finally, we finish by reiterating the point made by Nakagawa, Poulin, Mengersen, et al.15, and echoed by subsequent papers using lnCVR in different fields of study17–21. Meta-analysis of variation can tackle entirely new questions and open our eyes to insights that are hidden in datasets. The datasets required to gain these insights already exist because lnCVR is based on the same summary statistics as SMD and lnRR; means, SDs, and sample sizes. We suspect over 50,000 datasets of this sort have already been collected (c.f.36). In this regard it is vital that meta-analytic ‘raw’ data are made available and reusable in the spirit of open and transparent science37,38.
ACKNOWLEDGEMENTS
AMS was supported by an Australian Research Council Discovery Early Career Researcher Award (ARC DECRA: DE180101520). SN was supported by an ARC Discovery Grant (DP180100818). We would like to thank Professor Jennie Brand-Miller for her thoughts on our analysis of glycemia in diabetic individuals.
REFERENCES
