## ABSTRACT

Mendelian Randomization (MR) exploits genetic variants as instrumental variables to estimate the causal effect of an “exposure” trait on an “outcome” trait from observational data. However, the validity of such studies is threatened by population stratification, batch effects, and horizontal pleiotropy. Although a variety of methods have been proposed to partially mitigate those problems, residual biases may still remain, leading to highly statistically significant false positives in large genetic databases. Here, we describe a suite of sensitivity analysis tools for MR that enables investigators to properly quantify the robustness of their findings against these (and other) unobserved validity threats. Specifically, we propose the routine reporting of sensitivity statistics that can be used to readily quantify the robustness of a MR result: (i) the partial *R*^{2} of the genetic instrument with the exposure and the outcome traits; and, (ii) the *robustness value* of both genetic associations. These statistics quantify the *minimal strength* of violations of the MR assumptions that would be necessary to explain away the MR causal effect estimate. We also provide intuitive displays to visualize the sensitivity of the MR estimate to any degree of violation, and formal methods to *bound the worst-case bias* caused by violations in terms of multiples of the observed strength of principal components, batch effects, as well as putative pleiotropic pathways. We demonstrate how these tools can aid researchers in distinguishing robust from fragile findings, by showing that the MR estimate of the causal effect of body mass index (BMI) on diastolic blood pressure is relatively robust, whereas the MR estimate of the causal effect of BMI on Townsend deprivation index is relatively fragile.

**M**any fundamental questions in the social and medical sciences are questions of cause and effect. For instance, what are the social and health consequences of obesity? In practice, however, it is often infeasible or unethical to perform a randomized controlled trial to answer these types of questions. Moreover, observational studies are prone to being biased due to the presence of unmeasured confounders. In such cases, the method of instrumental variables^{1–4} (IVs) may be an appealing alternative, allowing one to infer cause-effect relationships even in the presence of unmeasured confounding between the exposure and the outcome.

Mendelian randomization (MR) exploits genetic variants associated with an “exposure” trait of interest as IVs to investigate whether that exposure has a causal effect on an “outcome” trait of interest^{5–11}. The technique of MR has become a standard tool for inferring causal relationships, with numerous applications published in medical, genetic and epidemiological journals.^{6–14} This growth has been accelerated by the availability of large genetic databases15 and Genome-Wide Association Studies (GWAS) linking many genetic variants to complex phenotypes^{8}. Nevertheless, the validity of MR studies depends on its own set of assumptions, and this rapid growth has not been accompanied with sufficient attention to those assumptions^{16–19}.

In particular, beyond being associated with the exposure, for a genetic variant to be a valid IV it must must satisfy two important and often (though not always^{20–22}) untestable conditions^{6–9}: (i) it must not be itself confounded with the outcome trait; and, (ii) it must affect the outcome trait only through its effect on the exposure trait. These conditions may be violated in several ways due to populational and methodological artifacts, as well as biological mechanisms. Most notably, population stratification^{23–28} and batch effects^{28–30} are well known sources of confounding biases in high-throughput genomic data. Likewise, many genetic variants tend to exert horizontal pleiotropy, meaning they affect the outcome trait through channels other than the exposure trait^{31,32}.

The prevailing method for dealing with population stratification and batch effects in MR is to adjust for genomic principal components and surrogate technical covariates representing genomic batch or assessment centre^{19}. In the case of horizontal pleiotropy, researchers are advised to perform alternative analyses, such as MR-Egger33 or MR-Presso^{34}, that rely on modified identification assumptions. Although these methods have proved useful for partially mitigating these problems, residual biases may still remain^{9,35}. Since those biases are impervious to sample size, they may lead to highly statistically significant false findings with large genomic data, as we demonstrate later via simulations.

Here we build on recent developments of the sensitivity analysis literature in statistics^{36–40} to provide a suite of sensitivity analysis tools for MR studies that quantifies the robustness of inferences to the presence of residual population stratification, batch effects, and horizontal pleiotropy. Specifically, we introduce *robustness values*^{37} (RV) for MR, summarizing the *minimal strength* that residual biases must have (in terms of variance explained of the genetic instrument and of the phenotypes) in order to explain away the MR causal effect estimate. To increase transparency and facilitate the assessment of the credibility of MR studies, we propose the RV to be routinely reported alongside traditional p-values (traditional p-values assume *zero* residual biases). We also provide intuitive sensitivity plots that allow researchers to quickly inspect how their inferences would have changed under biases of any postulated strength. Finally, we show how to place formal *bounds on the worst-case bias* caused by putative unmeasured variables with strength expressed in terms of multiples of the effect of observed variables, thereby facilitating expert judgment regarding the plausibility of such strong violations of the traditional MR assumptions.

We show how these techniques can aid researchers in assessing the degree of robustness of a MR result by examining two findings of highly-cited MR studies using the UK Biobank dataset^{15}–that body mass index (BMI) has a causal effect on diastolic blood pressure (DBP) and Townsend deprivation index (deprivation)^{41–43}. Sensitivity analysis reveals that, while the MR estimate of the causal effect of BMI on DBP is robust to relatively strong residual confounding *and* horizontal pleiotropy, the effect estimate of BMI on deprivation could be nullified by biases as weak as a fraction of current putative pleiotropic pathways, or a fraction of observed batch effects.

## Results

### mr-sensemakr overview—a suite of sensitivity analysis tools for MR

We developed mr-sensemakr, a suite of sensitivity analysis tools for MR that allows researchers to perform robust inferences of causal effect estimates in the presence of violations of the standard MR assumptions. These tools quantify both how much the inferences would have changed under a postulated degree of violation, as well as the minimal strength of violation necessary to overturn a certain conclusion. mr-sensemakr builds on an extension of the “omitted variable bias” framework for regression analysis^{37,38} to the Anderson-Rubin method44 and Fieller’s theorem45 for testing null hypotheses in the IV setting. This approach has a number of benefits, such as: (i) correct test size regardless of instrument strength; (ii) handling multiple confounding or pleiotropic effects acting simultaneously, possibly non-linearly; (iii) providing simple sensitivity statistics for routine reporting; and, (iv) exploiting expert knowledge to bound the maximum strength of biases (see Methods for details).

Let *D* denote the “exposure” trait, *Y* the “outcome” trait, and *Z* the genetic instrument (e.g, a polygenic risk score). Additionally, let ** X** denote a set of

*observed*“control” covariates which account for potential violations of the MR assumptions, such as population stratification (e.g, genetic principal components), batch effects (e.g, batch indicators) and traits that could block putative horizontal pleiotropic pathways

^{19}. Traditional MR analysis assumes that

**is sufficient for making**

*X**Z*a valid instrumental variable for identifying the effect of the exposure trait

*D*on the outcome trait

*Y*. An example for which this is the case is depicted in the directed acyclic graph (DAG) of Figure 1a—in this example there are no pleiotropic pathways, and although there is confounding due to population structure, adjusting for

**(say, genomic principal components and batch indicators) is sufficient for eliminating all biases.**

*X*The problem arises, however, when ** X** does not suffice for making

*Z*a valid instrument; instead, an

*extended*set of control covariates would be necessary to do so, but some of these variables are, unfortunately,

*unobserved*. Figures 1b and 1c illustrate two of such cases. In Figure 1b, although

*X*accounts for part of the confounding biases due to population structure (ps), it cannot account for all of it, and further adjustment for

*W*would be necessary for making Z a valid instrument. In Figure 1c, we have a different type of problem; there, the genetic instrument exerts horizontal pleiotropy (hp) through trait

_{ps}*W*, which needs to be accounted for in a valid MR analysis. In practice, of course, all these residual biases will often be acting simultaneously—we denote by

_{hp}**the**

*W**set*of

*all additional unmeasured variables*that would be necessary for making

*Z*a valid genetic instrument.

In this setting, mr-sensemakr answers the following question: *how strong would the unmeasured variables W have to be such that, if accounted for in the analysis, they would have changed the conclusions of the MR study?* As has been extensively discussed elsewhere

^{7,9,16,19}, MR studies are more reliable to

*test*the presence or direction of a causal effect, rather than to precisely estimate its magnitude. Thus, here we focus on two problematic changes that

**could cause— turning a statistically significant result into an insignificant one; or, leading to unbounded or uninformative confidence intervals due to weak instruments (when using Fieller’s theorem, confidence intervals can be: (i) connected and finite; (ii) the union of two disjoint unbounded intervals; or, (iii) the whole real line; see Methods).**

*W*It can be shown that, given a significance level *α*, the confidence interval for the MR causal effect is unbounded if, and only if, we cannot reject the hypothesis that the *genetic association with the exposure* is zero. Likewise, the MR causal effect estimate is statistically insignificant if, and only if, we cannot reject the hypothesis that the *genetic association with the outcome* is zero (to understand this intuitively, recall that the MR estimate is the ratio of the genetic association with the outcome over the genetic association of the exposure. Note this ratio is zero if the numerator is zero; likewise, the ratio can be arbitrarily large if the denominator can be arbitrarily close to zero). Therefore, the problem of sensitivity analysis of the MR estimate can be reduced to the simpler problem of sensitivity analysis of these two genetic associations.

mr-sensemakr thus performs sensitivity analysis for the MR causal effect estimate by examining how strong ** W** needs to be to explain away

*either*the observed

*genetic association with the exposure or*the observed

*genetic association with the outcome*. It deploys two main tools for assessing the sensitivity of these quantities. First, it computes key

*sensitivity statistics*suited for

*routine reporting*

^{37}, including

The partial

*R*^{2}of the genetic instrument with the (exposure/outcome) trait, revealing the*minimal*share of residual variation thatneeds to explain of the genetic instrument in order to fully eliminate the genetic association with the (exposure/outcome) trait;*W*The

*robustness value*(RV) of the genetic instrument with the (exposure/outcome) trait, revealing the*minimal*share of residual variation (partial*R*^{2}), both of the genetic instrument*and*of the trait, thatneeds to explain in order to make the genetic association with the (exposure/outcome) trait statistically insignificant; and,*W**Bounds*on the*maximum*residual variation explained by unmeasured variablesif they were as strong as: (i) observed principal components; (ii) measured batch effects; and, (iii) observed pleiotropic pathways.*W*

mr-sensemakr also provides sensitivity contour plots^{37} that, given *any* hypothetical strength of ** W** (measured in terms of the partial

*R*

^{2}of

**with the genetic instrument and with the trait), allows researchers to investigate what would have been the result of a significance test of the genetic association with the (exposure/outcome) trait had a**

*W***with such strength been incorporated in the analysis (see Figure 2). Finally, these plots can also include several bounds on the maximum amount of residual variation that**

*W***could explain, both of the genetic instrument and of the (exposure/outcome) trait, if**

*W***were multiple times stronger than observed variables. Next, we apply these tools in a real example that examines the robustness of previous MR findings regarding the causal effect of BMI on blood pressure and deprivation**

*W*^{41–43}.

### mr-sensemakr helps distinguishing robust from fragile findings

Previous studies^{41–43} used MR on the UK Biobank data15 to assess the causal effect of body mass index (BMI) on multiple outcome traits of interest. These MR analyses found a statistically significant effect of BMI on diastolic blood pressure (DBP)^{42} and on Townsend deprivation index (deprivation)—a measure of socioeconomic status.^{41} Following these studies, we filtered the data to only include people with self-reported white British ancestry who were not closely related, leaving a sample size of 291,274 people; the genetic instrument consisted of a polygenic risk score (PRS) derived from 97 sNPs previously found to be associated with BMI, with external weights given by the effect sizes from the GIANT study^{42,46} (see Methods for details).

The first part of Table 1 reports the results of the traditional MR analysis of the effects of BMI both on DBP and on deprivation. As it is usually recommended^{19} and following the original studies, these MR analyses further adjust for: age, gender, 20 leading genomic principal components, assessment center, batch indicators, as well as smoking and drinking status (both are putative pleiotropic pathways, especially for DBP^{47–52}). In consonance with the previous studies, we found that the conventional MR analyses lead to a positive and statistically significant effects of BMI on both traits, at the 5% significance level. The results, however, rely on the assumptions of *zero* residual population stratification, *zero* batch effects and *zero* horizontal pleiotropy, which are unlikely to hold. We thus used mr-sensemakr to investigate the robustness of these findings to potential violations of the standard MR assumptions.

We first examined the robustness of the genetic association with the exposure trait (BMI). Recall that, if confounders are strong enough to explain away the genetic association with the exposure, this can lead to unbounded or uninformative confidence intervals for the MR causal effect estimate—the exercise we are performing here is thus tantamount to assessing the “weak instrument” problem, except that now we are accounting both for sampling uncertainty and potential unmeasured confounders. The results are shown in the section entitled “sensitivity PRs-Exposure” of Table 1. (Note the results are the same both for DBP and deprivation, since the exposure trait, BMI, is the same in both cases.) The first sensitivity measure is the partial *R*^{2} of the PRs with BMI, which amounted to 1.67%. Although this quantity is already reported as a measure of instrument strength in many MR studies^{19}, it is perhaps less known that it is also a measure of its robustness to *extreme* confounding. In particular, this means that, even if the unmeasured variables ** W** explained

*all*left-out variation in BMI, they would still need to account for at least 1.67% of the variance of the genetic instrument, otherwise

**cannot explain away the genetic association with the exposure. Next we obtained a robustness value of 11.88% for the PRs-exposure association. This means that any unmeasured variables**

*W***that explain less than 11.88% of the residual variation, both of the PRs and of BMI, are not strong enough to make the genetic association with the exposure statistically insignificant.**

*W*Next we examined the robustenss of the genetic association with the outcome traits; recall that any unobserved variables capable of explaining away the genetic association with the outcome trait are also capable of explaining away the MR causal effect estimate. The results are shown in the section entitled “sensitivity PRs-Outcome” of Table 1, and here we have two separate results for each trait. specifically, we obtained a partial *R*^{2} of the PRS with DBP of 0.035% and a robustness value 1.47%. This means that, even if unobserved variables explained all variation of DBP, they still need to explain at least 0.035% of the residual variation of the genetic instrument to fully account for the observed PRs-DBP association; moreover, the RV reveals that unobserved variables need to account for at least 1.47% of either the variation of genetic instrument or the variation of DBP to be sufficiently strong to overturn the statistical significance found in the original MR study. Moving to the next trait, the bottom row of Table 1 shows the sensitivity statistics for the effect of BMI on deprivation. Here we found a partial *R*^{2} of 0.002% and a robustness value of 0.08%, revealing that much weaker residual biases would be able to overturn the MR effect estimate of BMI on deprivation.

Confronted with those results, the next step is to make plausibility judgments on whether unobserved variables with the strengths revealed to be problematic can be ruled out. To aid in these plausibility judgments, mr-sensemakr computes bounds on the amount of variance explained by the unmeasured variables ** W** if it were as strong as observed variables. For our running example, these bounds are shown in Table 2; they reveal the maximum partial

*R*

^{2}of unobserved variables

**with the genetic IV and with the traits, if it were as strong as: (i) 20 leading genomic principal components (1 x PCs); (ii) observed batch and centre effects (1 x Batch+Centre); and, finally, (iii) smoking and drinking status (1 x Alc.+Smok.).**

*W*Starting with instrument strength, first note that all bounds on the PRS and BMI columns of Table 2 are (substantially) lower than than the RV of 11.88% for the genetic association with BMI; this means that, even if ** W** were as strong as those variables, this would not be sufficient to result in a “weak instrument” problem. Moreover, since all values of the PRS column are less than the partial

*R*

^{2}of 1.67% of the variantexposure association, even a “worst-case” confounding that explains 100% of the variance of BMI, and as strongly associated with the genetic instrument as the observed variables, cannot account for the observed association of the genetic instrument with the exposure. Moving to statistical significance concerns, similar results hold for the PRS-DBP association. Since the bounds on both columns, for the PRS (column 1) and DBP (column 3), are below the robustness value of 1.47%, Table 2 reveals that biases as strong as the observed variables are not sufficient to make the MR causal effect estimate of BMI on DBP statistically insignificant. However, in stark contrast, note that all bounds on the PRS and deprivation columns are above the RV of 0.08% for deprivation, meaning that unobserved variables

**strong as those could easily overturn the original MR analysis.**

*W*Table 1 forms our proposed minimal reporting for sensitivity analysis in MR studies. Often, when supplemented with bounds such as those of Table 2, these metrics are sufficient to give a broad picture of the robustness of MR findings, as demonstrated above. Researchers, however, can refine their analyses and fully explore the whole range of robustness of their inferences with sensitivity contour plots, placing several different bounds on the strength of confounding multiple times stronger than observed variables. The plots for DBP and deprivation are shown in Figure 3 (see caption of Figure 2 for details on how to read the plot). For DBP, note that neither residual population stratification up to 14x stronger than observed principal components nor residual batch-effects up to 6x stronger than observed batch-effects are sufficient to make the MR estimate statistically insignificant. Likewise, if residual pleiotropy were up to 7x stronger than important observed pleiotropic pathways, such as alcohol and smoking, this is also not sufficient to change the original conclusions. Finally, even if unobserved variables ** W** had the same explanatory power of

*all*the observed variables

*combined*, this again would not change the results for DBP. In contrast, the sensitivity plot for deprivation reveals that the MR causal effect estimate of BMI on deprivation is sensitive to confounding with explanatory power as weak as a fraction (e.g, 0.5) of current observed variables.

Putting these results in context requires assessing the quality of the benchmarks involved. For example, it is not unreasonable to argue that genomic principal components correct for most, or at least a large part, of population structure^{25}, and that it is thus implausible to imagine residual population stratification multiple times stronger than what has been already corrected by observed principal components. Benchmarks for horizontal pleiotropy, on the other hand, require specific knowledge of the aetiology of the disease under study, or of the social process under investigation. In this application, for instance, alcohol consumption is indeed suspected to be an important channel for horizontal pleiotropy in the case of DBP^{47,48}, and smoking also leads to a short-term increase in blood pressure (although its long-term effects are disputed)^{49–51}. Therefore, one could plausibly argue that it is unlikely (although, of course, not impossible) that residual horizontal pleiotropy multiple times as strong as those still remains. The case for deprivation, however, reveals a more fragile finding; not only there is no a priori reason to suspect that alcohol and smoking should be among the strongest pleiotropic pathways, but the bounding exercise shows that residual pleiotropy a fraction as strong as those could overturn the original results.

Overall, the sensitivity analyses suggest that: (i) the genetic association of the instrument (PRs) with the exposure (BMI) is relatively robust, and that instrument strength is unlikely to be an issue; (ii) that it would take substantial residual confounding as well as substantial residual pleiotropic pathways to reverse the original MR finding of the causal effect of BMI on DBP; and that, in contrast, (iii) the previous MR causal effect estimate of BMI on deprivation is fragile, meatning that there is little room for small residual biases, which could easily overturn the original analysis.

### Current proposals for MR “sensitivity analyses” can lead to false positive findings in the presence of small residual biases in large samples

Prevailing proposals for sensitivity analyses of MR studies have focused on replacing traditional instrumental variable assumptions with alternative assumptions about how pleiotropy operates, such as the InSIDE assumption^{33,34}. Although an improvement of traditional MR, under the presence of residual population stratification, batch effects, and certain forms of pleiotropy, such approaches may still lead to statistically significant false findings given large enough samples. Therefore, the sensitivity statistics and exercises we propose here can be a useful complement to those alternative analyses.

To demonstrate this, we performed a simulation study in which the InSIDE assumption is only *slightly* violated through small pleoitropic effects via confounders of the exposure and outcome trait. Our simulation largely follows the same specification of previous work^{34,53,54}, with the following data-generating model:
where *D _{i}* is the exposure trait;

*Y*is the outcome trait;

_{i}*W*is an

_{i}*unobserved*confounder, and

*X*an

_{i}*observed*confounder of

*D*and

_{i}*Y*, both carriers of pleiotropy in a way that violates the InSIDE assumption. The genetic variants

_{i}*G*are drawn independently from a Binomial distribution, Binom(2,1/3); the remaining error terms

_{ij}*U*and

_{i}, ε_{i,W}, ε_{i,X}, ε_{i,D}*ε*are drawn from standard gaussians.

_{i,Y}We set the number of variants *J* = 90, similar to our previous BMI analysis, and consider genetic effects drawn from an uniform distribution from 0.01 to 0.05 for *ϕ _{i}, δ_{i}* and

*β*. The parameters

_{i}*η*and

*γ*give further control to the level of pleiotropy, and here we set both to 0.05. To put this value in context, for the usual simulated sample size considered in previous work (10,000-30,000 individuals), this level of pleitropy is small enough that it does not meaningfully affect type I errors for MR-Egger. Here, however, we simulate larger sample sizes, similar to those found in large genetic databases, ranging from 100,000 to 500,000 individuals.

We investigated the performance of alternative MR methods in a two-sample Mendelian randomization setting, meaning that only summary level data was used in the analyses, and the genetic associations with the exposure trait and the outcome trait were obtained in separate simulated data. Table 3 shows the results of 1,000 simulations of the data generating process for each of the sample sizes, considering two cases: (i) a true null causal effect with *τ* = 0; (ii) and a true positive causal effect of *τ* = 0.1. Note that *X _{i}* and

*W*have similar strengths—a fact that, if known, can be exploited for sensitivity analysis.

_{i}We first focus on the case of a null causal effect. The first three columns of the table shows the proportion of cases in which the null hypothesis of zero effect was rejected, using the three different MR methods: (i) the traditional inverse variance weighted (IVW); (ii) MR-Presso; and, (iii) MR-Egger. Since the true causal effect is zero, these results indicate the proportion of *false positives*. We see that IVW and MR-Presso give similar results with a virtually 100% false positive rate for all sample sizes, and that MR-Egger starts with a false positive rate of 13% for *N* = 100,000, and this rate grows up to 52% at *N* = 500,000. Next, the last three columns show how the sensitivity exercises could help interpreting the results in such cases. Starting with the fourth column, here we have the proportion of false positives if the researcher knew that *X* were among the most important pleiotropic pathways (such as, in our previous example) and that residual pleiotropy could be as strong as *X*. Using the bounding procedure delineated in the previous section, if the researcher accounted for this possibility of confounding as strong as *X*, she would then only falsely conclude that there is an effect roughly around 1% to 3% of the time. Moreover, the last two columns show the *minimum* and the *maximum* robustness values for the association of the genetic instrument with the exposure, over *all* simulations. Note these are the *most extreme* results one could get, and they still always remain roughly below 1%, correctly warning the researcher that residual biases of those magnitudes are capable of overturning those MR findings.

We now turn to the second scenario, in which there is a true positive causal effect of *D* on *Y*. Here all MR methods correctly reject the null hypothesis of zero effect from 86% to 100% of the time. The challenge in this setting, thus, comes not from rejecting the null hypothesis, but from the fact that potential critics of the study could *correctly* be skeptical of the results, and conjecture that the reason why the null was rejected was simply due to residual pleiotropic pathways. To mitigate those concerns, the researcher could again use the bounding procedure, and around 90% to 100% of the time she would conclude that one would still reject the null, even when allowing for residual pleiotropy as strong as that due to the observed *X*. Likewise, the results for the RV show that a researcher would never obtain a robustness value below 1.5%, meaning that, in all cases, the critic would need to argue that biases of *at least* these magnitudes are plausible in order to forcefully dismiss the observed MR finding.

The phenomenon demonstrated in the simulation is simply the well known but often overlooked fact that, with large enough data, *any* residual bias will eventually be statistically significant. It is for that reason that alternative analyses such as MR-Egger and MR-Presso are bound to lead to false positives with large enough genetic databases, unless their modified identification assumptions also hold *exactly*. In contrast, the sensitivity statistics we propose here, such as the partial *R*^{2} and the RV, are directly quantifying the strength of biases needed to overturn a finding—and they will simply converge to their population values as the sample size increases.

## Discussion

We have described a suite of sensitivity analysis tools for performing valid MR inferences under the presence of residual biases of *any* postulated strength. The approach we proposed here starts from the premise that all MR studies will be imperfect in some way or another, but also that a study does not have to be perfect in order to be informative—what matters is not whether certain assumptions hold exactly, but the extent to which certain conclusions are robust to violations of those assumptions, and whether such strong violations are plausible.

We showed how two simple sensitivity statistics, the partial *R*^{2} and the *robustness value*, can be used to easily communicate the *minimum strength* of residual biases necessary to invalidate the results of a MR study. Since researchers are already well advised to report the partial *R*^{2} of the genetic instrument with the exposure trait, routinely reporting the partial *R*^{2} of the genetic instrument with the outcome trait and the robustness value is but a small addition to current practices, and can greatly improve the transparency regarding the robustness of MR findings.

We also showed that, whenever researchers are able to argue that, although not perfect, they have credibly accounted for most of the population structure with genomic principal components, most of possible batch effects with technical dummies, and have measured known important pleiotropic pathways, this knowledge can be leveraged to formally bound the worst possible inferences due to residual biases. Such bounding exercises can be an important piece of the scientific debate when arguing in favor or against the robustness of a certain finding.

Finally, we remind readers that these tools cannot and should not be used to replace expert judgment. On the contrary, the tools described here can aid leveraging certain types of expert knowledge that would have been otherwise neglected, such as judgments regarding the maximum plausible strength of residual biases, or knowledge regarding the relative importance of certain causal pathways. In sum, strong conclusions from Mendelian randomization studies still need to rely on the quality of the research design, and substantive understanding both of the genetic variants as well as the traits under investigation.

## Methods

### Study Design and Participants

#### Study population

The UK Biobank55 is a resource that links genetic data to a variety of physiological and social traits in a cohort of 503,325 British people aged 37-73 years. It has been a valuable resource for estimating causal effects of exposures on a multitude of outcomes using MR^{41–43}. We filtered the data to only include people with self-reported white British ancestry who were not closely related, (e.g. no first, second, or third degree relatives), as defined by pairs of individuals who had a kinship coefficient < (1/2)^{(9/2)} (following^{56}), leaving 291,274 people. We also removed individuals who were not measured for BMI (non-impedence). For our analysis of the Lyall et al.^{42} study, we also excluded patients who responded to a question on whether they were taking anti-hypertensive medication with “don’t know”.

#### Polygenic Risk Score

The Polygenic Risk Scores (PRS) was constructed in the same manner as in Lyall et al.^{42}. This PRS score was derived from 97 SNPs that were genome-wide significantly associated with BMI in the GIANT consortium study^{46}. Two of these SNPs were not directly genotyped in the UK Biobank, and two failed Hardy-Weinberg equilibrium, leaving 93 SNPs to comprise the PRS. The PRS was computed as a weighted score based on these SNPs, with the weights derived from the effect estimated reported by GIANT (*β* per 1-SD unit of BMI)^{42,46}. We used the exact same weights computed by Lyall et al.^{42}.

#### Exposure, outcome and control traits

During the initial visit to the UK Biobank assessment center, height was measured to the nearest centimeter using a Seca 202 device and weight was measured to the nearest 0.1 kilogram using a Tanita BC418MA body composition analyser. These measurements were subsequently used to calculate body mass index (BMI), in kg/m^{2} (field category ID: 21001). The two outcomes of interest were the Townsend deprivation index and diastolic blood pressure. The Townsend deprivation index was calculated using the postcode of the participant at the time of recruitment (field category ID: 189). Diastolic blood pressure was obtained by an automated reading from an Omron blood pressure monitor (field category ID: 4079).

In our analyses, we adjusted for age, sex, assessment centre, genetic batch effects, drinking and smoking status, given by the following variables: “Sex” (field category ID: 31); “Age when attended assessment centre” (field category ID: 21003); “UK Biobank assessment centre” (field category ID: 54); “Genotype measurement batch” (field category ID: 22000); “Smoking status” (field category ID: 20116); “Frequency of drinking alcohol” (field category ID: 20414); “Alcohol intake frequency” (field category ID: 1558).

### Statistical Methods

#### Traditional Mendelian Randomization

Suppose we are interested in assessing the causal effect of an exposure trait *D* on an outcome trait *Y*, by performing a Mendelian Randomization study with a polygenic risk score (PRS) (comprised of a linear combination of SNPs *G _{j}* with weights

*β*) as the putative instrumental variable. Note the weights

_{j}*β*of the PRS could have been obtained either from external data (such as a previous GWAS), or via cross-validation as well as other methods

_{j}^{9}. To give credibility to the study, the researcher considers a

*set*of

*observed*control covariates

**that accounts for potential IV violations of population stratification, batch effects and horizontal pleiotropy**

*X*^{19}. That is,

**consists of,**

*X*Where *X*_{ps} denotes the variables to adjust for population stratification, such as, for instance, genomic principal components; *X*_{batch} denotes variables to adjust for batch effects, for example, dummy variables for the assessment centre and genotype batches; *X*_{hp} denotes *measured* variables which are suspected to be capable of blocking suspected pleoitropic pathways; and, finally, *X*_{ind} are participant characteristics that are usually included in MR, such as the age and sex of the individual.

The traditional MR estimate of the causal effect of *D* on *Y*, here denoted by , would consist of the ratio of the genetic association with the outcome trait, , and the genetic association with the exposure trait, , after adjusting for observed covariates ** X**, namely,

Confidence intervals that have nominal coverage regardless of instrument strength can be obtained via Fieller’s theorem^{45} or via the Anderson-Rubin regression^{44}. These confidence intervals can be of three forms: (i) a connected closed interval [*a,b*]; (ii) the union of disjoint unbounded intervals, (– ∞, *a*] ∪ [*b*, ∞); or, (iii) the whole real line (–∞, ∞).

#### Violation of traditional assumptions

The traditional MR estimate, , however adjusts for ** X** only, and it is unlikely that

**controls for**

*X**all*possible threats to the study validity. Instead, the researcher would have preferred to have also adjusted for additional unobserved variables

**to satisfy the MR assumptions. For instance, we would like to have controlled for the**

*W**true*population indicators

*W*_{ps}instead of its approximation as recovered by the principal components

*X*_{ps}; likewise, the researcher suspects that

*X*_{hp}

*is not enough*to block all pleiotropic pathways, and would have liked to have further adjusted for covariates

*W*_{hp}.

In sum, instead, of performing the MR analysis using ** X** alone, resulting in as our MR estimate, the researcher would have wanted to compute instead
which adjusts for the extended set of covariates {

**}, such that**

*X, W**Z*is a valid instrument for estimating the causal effect of

*D*on

*Y*, conditional on {

**}. Likewise, confidence intervals should have also been computed adjusting for {**

*X, W***}. How would accounting for the omitted variables**

*X, W***have changed our inferences regarding the causal effect of**

*W**D*on

*Y*?

#### The sensitivity analysis of the MR estimate can be reduced to the sensitivity of the genetic associations

We now explain how to perform sensitivity analysis within the Anderson-Rubin (AR) approach^{44}, which as we show is also numerically equivalent to Fieller’s proposal^{45} when considering a single instrumental variable *Z*. Here we take an *exact* algebraic approach—that is, all results here hold both for sample or population estimates.

Let *Y* and *D* denote (*n* × 1) vectors containing the outcome and exposure of interest for each of the *n* observations, respectively. Now let *τ* denote the causal effect of interest, and define a new variable *Y*_{τ0}:= *Y* – *i>τ*_{0}*D*, in which we subtract from *Y* the causal effect of *D*, considering a hypothetical value for *τ*, say, *τ*_{0}. Next consider the following linear regression,

Where *Z* is a (*n* × 1) vector with the genetic instrument; ** X** is a (

*n*×

*p*) matrix of

*observed*covariates, including the constant; and

**is a (**

*W**n*×

*k*) matrix of

*unobserved*covariates the analyst wished to have measured in order to make

*Z*a valid instrument. Here are the OLS coefficient

*estimates*of the regression of

*Y*

_{τ0}on

*Z,*, and its corresponding residual.

**X**,**W**Note that, if *τ* = *τ*_{0} and if *Z* is valid instrument conditional on ** X**,

**, then we must have that , and thus that**

*W**ϕ*

_{τ0}= 0. Following this logic, the AR confidence interval with significance level

*α*is defined as all values of

*τ*

_{0}such that we cannot reject the null hypothesis

*H*

_{0}:

*ϕ*

_{τ0}= 0 at the chosen significance level. More precisely,

Where is the t-value for the null hypothesis *H*_{0}: *ϕ*_{τ0} = 0 and is the critical threshold of the t-distribution for a significance level *α* and df degrees of freedom. This confidence interval can be obtained analytically as a function of the genetic association with the exposure and the genetic association with the outcome, which is now useful to write out explicitly.

By appealing to the Frisch–Waugh–Lovell (FWL) theorem^{57–59}, we can write as,

Where *Y*^{⊥XW} denotes the variable *Y* after removing the components linearly explained by ** X** and

**, and denotes the regression coefficient of**

*W**Z*on

*Y*(the genetic association with the outcome) after adjusting for both

**and denotes the regression coefficient of**

*X**Z*on

*D*(the genetic association with the exposure) after adjusting for

**and**

*X***. Likewise, the estimated variance of can be written as,**

*W*To construct the confidence interval, we need to find all values of *τ*_{0} such that the following inequality holds,

Squaring and rearranging terms we obtain the following quadratic inequality,

These conditions are exactly Fieller’s solution to the confidence interval of the ratio .

Our task has thus simplified to find all values of *τ*_{0} that makes the above quadratic equation, with coefficients *a, b* and *c*, non-positive. But here we have special interest in two specific cases: (i) when the confidence interval for *τ* is unbounded; and, (ii) when the confidence interval for *τ* includes zero.

Let us first consider the case of unbounded confidence intervals. Note this happens when *a* < 0, which means the quadratic curve in Equation 9 will be concave (will have a “∩” shape)—as we increase *τ*_{0} to plus or minus infinity, the inequality is bound to hold and the confidence interval will be unbounded. Also note that *a* < 0 if, and only if,

That is, the confidence interval for *τ* will be unbounded if and only if we cannot reject that the genetic association with the exposure is zero.

We now turn our attention to the null hypothesis of zero effect, that is, *H*_{0}: *τ* = 0. Notice in this case the first two terms of the quadratic equation, *a* and *b*, vanish. What we have left is only the term *c* which will be negative if, and only if,

In other words, the null hypothesis of zero effect for the causal effect is not rejected if, and only if, the null hypothesis of zero association between the instrument *Z* with the outcome *Y* is also not rejected.

We have thus simplified the sensitivity analysis of the MR estimate to the sensitivity analysis of the two genetic associations. If ** W** is strong enough to explain away the genetic association with the exposure, then

**is strong enough to make the the causal effect arbitrarily large in either direction. If**

*W***is strong enough to explain away the genetic association with the outcome trait, than**

*W***is strong enough to explain away the MR estimate. This is summarized in Table 4.**

*W*Since we have reduced the problem of sensitivity analysis of MR to the problem of sensitivity analysis of the genetic associations, we can leverage all tools of Cinelli and Hazlett^{37} for our problem. Here we thus review the main sensitivity analysis results of Cinelli and Hazlett, in the context of the genetic association with the outcome. All results below, of course, also apply to the genetic association with the exposure, by just replacing *Y* with *D* where appropriate.

#### Sensitivity formulas for the genetic associations

Consider first a univariate *W* and let denote the partial *R*^{2} of *W* with the genetic instrument and let denote the partial *R*^{2} of *W* with the outcome trait. Given the observed genetic association and its estimated standard error , adjusting for ** X** alone, the estimate and standard error we would have obtained further adjusting for

*W*can be recovered with

^{37}, and,

Where here now df denote the degrees of freedom of the AR regression actually run. These formulas allow us to investigate how the estimate, standard error, t-values, p-values or confidence intervals would have changed, under a confounder *W* of any postulated strength, as parameterized by and . For a singleton W these formulas are exact, and for multivariate ** W**, it can further be shown that these formulas are conservative, barring an adjustment on the degrees of freedom37 (that is, these are the worse biases a multivariate

**could cause). These formulas form the basis of the contour plots shown in Figure 2.**

*W*#### Bounds on the partial *R*^{2} of *W* based on observed covariates

Where investigators are unable to make direct claims on the strength of *W*, it may be helpful to consider relative claims, by positing, for instance, that *W* is no stronger than some observed covariate *X _{j}*. For that, consider a confounder orthogonal to the observed covariates, ie.,

*W*⊥

**X**and define where

**X**

_{–j}represents the vector of covariates

**X**excluding

*X*. Then the strength of

_{j}*W*can be bounded by

^{37}, where

*η*is a scalar computed from

*k*and . (see Cinelli and Hazlett37 for details).

_{Y}, k_{Z}#### Sensitivity statistics for routine reporting

The previous results allow us to perform sensitivity analysis to confounding of any postulated strength. However, widespread adoption of sensitivity analysis benefits from simple metrics that users can report to quickly summarize the robustness of their results. With that in mind, Cinelli and Hazlett37 introduced two sensitivity statistics for routine reporting: the Robustness Value (RV) and the partial *R*^{2}.

Let *f*:= |*f*_{Y~Z|X}| denote the absolute value of the partial *Cohen’s f* of the genetic instrument with the outcome.^{1} Now also re-scale the critical threshold, , and define . The robustness value RV_{α} is defined as the minimal strength of association that *W* must have, both with the genetic instrument *Z* and the outcome trait *Y*, in order to make the genetic association with the outcome statistically insignificant. This is given by^{37}

Any *W* with both strength of associations below RV_{α} is not sufficiently strong to make the genetic association with the outcome statistically insignificant, and, thus, also not sufficiently strong to make the MR causal effect estimate statistically insignificant. On the other hand, any W with both strength of associations above RV_{α} is sufficiently strong to do so.

Moving to the partial *R*^{2}, in addition to quantifying how much variation of the outcome trait is explained by the genetic instrument, the partial *R*^{2} also tells us how robust the genetic association with the outcome is to an “extreme sensitivity scenario.” More precisely, suppose that the unobserved variable *W* explained *all* residual variance of the outcome trait. Then, for *W* to bring the genetic association to zero, it must explain *at least* as much residual variation of the genetic instrument as the residual variation of the outcome trait that the genetic instrument currently explains^{37}. Mathematically, if *R*_{Y~W|Z,X} = 1, then for *W* to make , we need to have that .

## Acknowledgements

This research was conducted using the UK Biobank Resource under Application 33127.

## Footnotes

↵1 The partial Cohen’s

*f*^{2}can be written as .