Abstract
Hybridization between humans and Neanderthals has resulted in a low level of Neanderthal ancestry scattered across the genomes of many modern-day humans. After hybridization, on average, selection appears to have removed Neanderthal alleles from the human population. Quantifying the strength and causes of this selection against Neanderthal ancestry is key to understanding our relationship to Neanderthals and, more broadly, how populations remain distinct after secondary contact. Here, we develop a novel method for estimating the genome-wide average strength of selection and the density of selected sites using estimates of Neanderthal allele frequency along the genomes of modern-day humans. We confirm that East Asians had somewhat higher initial levels of Neanderthal ancestry than Europeans even after accounting for selection. We find that there are systematically lower levels of initial introgression on the X chromosome, a finding consistent with a strong sex bias in the initial matings between the populations. We find that the bulk of purifying selection against Neanderthal ancestry is best understood as acting on many weakly deleterious alleles. We propose that the majority of these alleles were effectively neutral—and segregating at high frequency—in Neanderthals, but became selected against after entering human populations of much larger effective size. While individually of small effect, these alleles potentially imposed a heavy genetic load on the early-generation human–Neanderthal hybrids. This work suggests that differences in effective population size may play a far more important role in shaping levels of introgression than previously thought.
1 Summary
When modern humans spread out of Africa tens of thousands of years ago, they hybridized with Neanderthals. As a result, a few percent of Neanderthal DNA is present in the genomes of many contemporary non-African human populations, with East Asian genomes containing on average slightly more Neanderthal ancestry than Europeans. Much of Neanderthal DNA in humans appears to be deleterious, and natural selection is acting to remove it. We still do not fully understand why so many Neanderthal-derived alleles are deleterious in humans. It is possible that these alleles were not deleterious in Neanderthals, but rather were hybrid incompatibilities, which became deleterious only once they were introduced to the human population. If so, reproductive barriers had rapidly evolved between Neanderthals and humans. Alternatively, large numbers of unconditionally deleterious, but effectively neutral, alleles may have been segregating in Neanderthals, which after hybridization were selected against in the human population. In this paper, we seek to understand the nature of selection against Neanderthal ancestry in modern-day humans. We confirm that even after accounting for selection, East Asians tend to have more Neanderthal ancestry than Europeans. We find that on average, selection against Neanderthal alleles is weak; it is weaker for autosomal loci than X-linked loci, although our confidence intervals for the X chromosome are rather wide. Lastly, we find evidence for potential sex bias among initial matings between humans and Neanderthals, suggesting that Neanderthal DNA in humans is more likely to have come from Neanderthal males. Overall, our results suggest that Neanderthals over time accumulated many unconditionally weakly deleterious alleles that in their small population were effectively neutral. However, after introgressing into the human population, which has a larger effective population size, those alleles became exposed to purifying selection, as selection is more effective in populations of larger effective size. Therefore, differences between human and Neanderthal effective population sizes appear to have played a key role in shaping our present-day shared ancestry.
Introduction
The recent sequencing of ancient genomic DNA has greatly expanded our knowledge of the relationship to our closest evolutionary cousins, the Neanderthals [Noonan et al., 2006, Green et al., 2010, Reich et al., 2010, Meyer et al., 2012, Pruefer et al., 2014]. Neanderthals, along with Denisovans, were a sister group to modern humans, having likely split from modern humans around 550,000–765,000 years ago [Pruefer et al., 2014]. Genome-wide evidence suggests that modern humans interbred with Neanderthals after humans spread out of Africa, such that nowadays 1.5–2.1% of the autosomal genome of non-African modern human populations derive from Neanderthals [Green et al., 2010]. This admixture dates on average to 47,000–65,000 years ago [Sankararaman et al., 2012], with potentially a second pulse (around the same time) into the ancestors of populations now present in East Asia [Green et al., 2010, Wall et al., 2013, Vernot and Akey, 2014, 2015, Kim and Lohmueller, 2015].
While some introgressed archaic alleles appear to have been adaptive in anatomically modern human (AMH) populations [Khrameeva et al., 2014, Sankararaman et al., 2014, Racimo et al., 2015], on average selection has acted to remove Neanderthal DNA from modern humans. This can be seen from the non-uniform distribution of Neanderthal alleles along the human genome [Vernot and Akey, 2014, Sankararaman et al., 2014]. In particular, regions of high gene density or low recombination rate have low Neanderthal ancestry, which is consistent with selection removing Neanderthal ancestry more efficiently from these regions [Sankararaman et al., 2014]. In addition, the X chromosome has lower levels of Neanderthal ancestry and Neanderthal ancestry is absent from the Y chromosome and mitochondria [Serre et al., 2004, Currat et al., 2004, Sankararaman et al., 2014, Vernot and Akey, 2014, Pruefer et al., 2014, Meyer et al., 2012, Green et al., 2010].
It is less clear why the bulk of Neanderthal alleles would be selected against. Were early-generation hybrids between humans and Neanderthals selected against due to intrinsic genetic incompatibilities? Or was this selection mostly ecological or cultural in nature? If reproductive barriers had already begun to evolve between Neanderthals and AMH, then these two hominids may have been on their way to becoming separate species before they met again [Sankararaman et al., 2014, Gibbons, 2014]. Or, as we propose here, did differences in effective population size and resulting genetic load between humans and Neanderthals shape levels of Neanderthal admixture along the genome?
We set out to estimate the average strength of selection against Neanderthal alleles in AMH. Due to the relatively short divergence time of Neanderthals and AMH, we still share much of our genetic variation with Neanderthals. However, we can recognize alleles of Neanderthal ancestry in humans by aggregating information along the genome using statistical methods [Sankararaman et al., 2014, Vernot and Akey, 2014]. Here, we develop theory to predict the frequency of Neanderthal-derived alleles as a function of the strength of purifying selection at linked exonic sites, recombination, initial introgression proportion, and split time. We fit these predictions to recently published estimates of the frequency of Neanderthal ancestry in modern humans [Sankararaman et al., 2014]. Our results enhance our understanding of how selection shaped the genomic contribution of Neanderthal to our genomes, and shed light on the nature of Neanderthal-human hybridization.
Results
We make use of the estimates from Sankararaman et al. [2014] of the frequency of Neanderthal alleles along the genome in the European (EUR) and East Asian (ASN) samples from the 1000 Genomes Project. We fit these to a model-based prediction of levels of Neanderthal ancestry along the chromosome that takes into account variation in local gene density and recombination rates (Fig 1).
A section of chromosome 1 showing the estimated Neanderthal frequency (pn, black line) for the EUR sample from Sankararaman et al. [2014] and the expected frequency (pt, red line) predicted by our best fitting model. The midpoints of exons are shown as blue bars. Note that the estimated frequency is expected to have much greater variance along the genome than our prediction due to genetic drift. Our prediction refers to the mean around which the deviation due to genetic drift is centered (S2 Text, Figure S2.5).
The model we consider is one of the present-day frequency of Neanderthal alleles along the human chromosomes following a single pulse of admixture t = 2000 generations ago [Sankararaman et al., 2012]. We assume that at the time of introgression a proportion p0 of the autosomal alleles in the human population were drawn at random from the Neanderthal population. This simple model seems justified, since it cannot be distinguished from more complex models, such as continuous and dual-wave admixture models with selection, unless the onset and duration of admixture are known (S1Text).
We assume that, initially, deleterious alleles are fixed in Neanderthals at the time of admixture and that all of these Neanderthal alleles are equally deleterious in the human genomic background (we justify this model in the Methods). We denote the relative fitness of human individuals heterozygous for a deleterious Neanderthal allele by 1 – s. Assuming an initial introgression proportion of a few percent, homozygous carriers of Neanderthal alleles are very rare and can be ignored (S1 Text).
In practice, we do not know the location of the deleterious Neanderthal alleles along the genome, nor could we hope to identify them all as some of their effects may be weak (but perhaps important in aggregate). Therefore, we average over the uncertainty in the locations of these alleles. We assume that each exonic base independently harbors a deleterious Neanderthal allele with probability µ. Building on a long-standing theory on genetic barriers to gene flow [Petry, 1983, Bengtsson, 1985, Barton and Bengtsson, 1986, Gavrilets, 1997, Gavrilets and Cruzan, 1998] at each neutral site ℓ in the genome, we can express the present-day expected frequency of Neanderthal alleles in our admixture model in terms of the initial frequency p0, and a function gℓ of the recombination rates r between ℓ and the neighboring exonic sites under selection, and the parameters s, t, and µ (see equation 5, S2 Text). That is, at locus ℓ, a fraction pt = p0gℓ (r, s, t, µ) of modern humans are expected to carry the Neanderthal allele. The function gℓ() decreases with the time since admixture (t), tighter linkage to potentially deleterious sites, larger selection coefficient (s), and higher density of deleterious exonic sites (µ). If a neutral Neanderthal allele is initially completely unassociated with deleterious alleles, pt would on average be equal to p0. Our model accounts for deleterious alleles that are physically linked to a neutral allele. However, in practice, neutral Neanderthal alleles will initially be associated (i.e. in linkage disequilibrium) with many unlinked deleterious alleles because F1 hybrids inherited half of their genome from Neanderthal parents [Bengtsson, 1985]. Therefore, p0 should be thought of as an effective initial admixture proportion. We will return to this point in the Discussion.
To estimate the parameters of our model (p0, s, and µ), we minimised the residual sum of squared deviations (RSS) between observed frequencies of Neanderthal alleles [Sankararaman et al., 2014] and those predicted by our model (see equation 6 and S2 Text). We assess the uncertainty in our estimates by bootstrapping large contiguous genomic blocks and re-estimating our parameters. We then provide block-wise bootstrap confidence intervals (CI) based on these (Methods and S2 Text). In Fig 2 and 3, we show the RSS surfaces for the parameters p0, s, and µ for autosomal variation in Neanderthal ancestry in the EUR and ASN populations.
The scaled RSS surface (RSSmin − RSS) as a function of s and µ for EUR and ASN autosomal chromosomes. Each value of the RSS is minimized over p0, making this a profile RSS surface. Regions in darker shades of orange represent parameter values of lower scaled RSS. Black circles show bootstrap results of 1000 blockwise bootstrap reestimates, with darker circles corresponding to more common bootstrap estimates.
The scaled RSS surface (RSSmin − RSS) of autosomal chromosomes as a function of the initial admixture proportion p0. Results are shown for a model where only the nearest-neighboring exonic site under selection is considered, and for t = 2000 generations after Neanderthals split from EUR (grey) and ASN (pink) populations. Dots and horizontal lines show the value of p0 that minimizes the RSS and the respective 95% block-bootstrap confidence intervals. The RSS surfaces are shown for values of the selection coefficient (s) and exonic density of selection (µ) given in Table 1.
For autosomal chromosomes, our best estimates for the average strength of selection against deleterious Neanderthal alleles are low in both EUR and ASN (Fig 2), but statistically different from zero (sEUR = 4.1 × 10−4 95% CI [3.4 × 10−4, 5.2 × 10−4], sASN = 3.5 × 10−4, 95% CI [2.6 × 10−4, 5.4 × 10−4]). We obtain similar estimates if we assume that the Neanderthal ancestry in humans has reached its equilibrium frequency or if we account for the effect of multiple selected sites. However, and as expected, the estimated selection coefficients are somewhat lower for those models (S2 Text Table S2.1). Our estimates of the probability of any given exonic site being under selection are similar and low for both samples (µEUR = 8.1 ×10−5, 95% CI [4.1 ×10−5, 1.2 ×10−4], µASN = 6.9 ×10−5, 95% CI [4.1 ×10−5, 1.6 ×10−4]). These estimates correspond to less than 1 in 10, 000 exonic base pairs harboring a deleterious Neanderthal allele, on average. As a result, our estimates of the average selection coefficient against an exonic base pair (the compound parameter µs) are very low, on the order of 10−8 in both samples (Table 1).
Point estimates and 95% bootstrap confidence intervals for the focal parameters. Estimates are based on a minimization of the residual sum of squared deviations (RSS) between observations and a model in which, for each neutral site, only the nearest-neighboring exonic site under selection is considered. Introgression is assumed to have happened t = 2000 generations ago.
Consistent with previous findings [Vernot and Akey, 2015, Kim and Lohmueller, 2015], we infer a higher initial frequency of Neanderthal alleles in the East Asian sample compared to the European sample (p0,EUR = 3.38 × 10−2, 95% CI [3.22 × 10−2, 3.52 × 10−2], p0,ASN = 3.60 × 10−2, 95% CI [3.45 × 10−2, 3.86 × 10−2]), but the 95% bootstrap CI overlap (Fig 3). This occurs because our estimates of the initial frequency of Neanderthal alleles (p0) are mildly confounded with estimates of the strength of selection per exonic base (µs). That is, somewhat similar values of the expected present-day Neanderthal allele frequency can be inferred by simultaneously reducing p0 and µs (Fig 4). This explains why the marginal confidence intervals for p0 overlap for ASN and EUR. However, if µs, the per exonic base fitness cost of Neanderthal introgression, is the same for ASN and EUR (i.e. if we take a vertical slice in Fig 4), the values of p0 for the two samples do no overlap.
The contrast between the inferred parameters for the East Asian (ASN) and European (EUR) samples for the autosomes (A) and both the X and the autosomes (B). Plots show bootstrap estimates of the initial admixture proportion p0 against the estimated exonic density of selection µs, with the empty symbols denoting our minimum RSS estimates. The clear separation of the point clouds for autosomes and the X for both EUR and ASN modern humans suggests that the combination of selection and initial admixture level are likely the reason why the present-day frequency of Neanderthal alleles differs between autosomal and X chromosomes. Note the different scales of the axes in panels A and B.
To verify the fit of our model, we plot the average observed frequency of Neanderthal alleles, binned by gene density per map unit, and compare it to the allele frequency predicted by our model based on the estimated parameter values (Fig 5). There is good agreement between the two, suggesting that our model provides a good description of the relationship between functional density, recombination rates, and levels of Neanderthal introgression. At the scale of 1 cM, the Pearson correlation between observed and predicted levels of autosomal Neanderthal introgression is 0.897 for EUR and 0.710 for ASN (see Table S2.3 in S2 Text for a range of other scales).
Genomic regions with lower exonic density contain higher average Neanderthal allele frequency in both in Europeans (grey circles) and Asians (pink circles). We find a good fit to this pattern under our model (black and red triangles). Ranks are obtained by splitting the genome into 1 cM segments, calculating the number of exonic sites for each segment and sorting the segments into ten bins of equal size. Dashed lines represent 95% blockwise bootstrap confidence intervals. Plots created for different segment sizes look similar (S2 Text).
Our estimated coefficients of selection (s) against deleterious Neanderthal alleles are very low, on the order of the reciprocal of the effective population size of humans. This raises the intriguing possibility that our results are detecting differences in the efficacy of selection between AMH and Neanderthals. Levels of genetic diversity within Neanderthals are consistent with a very low long-term effective population size compared to AMH, i.e. a higher rate of genetic drift. This suggests that weakly deleterious exonic alleles may have been effectively neutral and drifted up in frequency in Neanderthals [Do et al., 2015, Castellano et al., 2014, Lin et al., 2015], only to be slowly selected against after introgressing into modern human populations of larger effective size. To test this hypothesis, we simulated a simple model of a population split between AMH and Neanderthals, using a range of plausible Neanderthal population sizes after the split. In these simulations, the selection coefficients of mutations at exonic sites are drawn from an empirically supported distribution of fitness effects [Boyko et al., 2008]. We track the frequency of deleterious alleles at exonic sites in both AMH and Neanderthals, and compare these frequencies at the time of secondary contact (admixture). We find that at the time of admixture the majority of sites that still harbor a deleterious allele represent fixed or nearly fixed differences between AMH and Neanderthals, with the deleterious allele absent or at low frequency in AMH, but fixed or at high frequency in Neanderthals (Fig S3.1). For plausibly low effective sizes of the Neanderthal population, we find that both the average selection coefficient (s) and the exonic density of fixed deleterious Neanderthal alleles (µ) in the simulations are of the same order as our respective estimates (see Fig S3.2). Therefore, a model in which the bulk of Neanderthal alleles, which are now deleterious in modern humans, simply drifted up in frequency due to the smaller effective population size of Neanderthals seems quite plausible.
We finally turn to the X chromosome, where observed levels of Neanderthal ancestry are strongly reduced compared to autosomes [Sankararaman et al., 2014, Vernot and Akey, 2014]. This reduction could be consistent with the X chromosome playing an important role in the evolution of hybrid incompatibilities at the early stages of speciation [Sankararaman et al., 2014]. However, a range of other phenomena could explain the observed difference between the X and autosomes, including sex-biased hybridization among populations, the absence of recombination in males, as well as differences in the selective regimes [Charlesworth et al., 1987, Vicoso and Charlesworth, 2006, Meisel and Connallon, 2013]. We modified our model to reflect the transmission rules of the X chromosome and the absence of recombination in males. We give the X chromosome its own initial level of introgression (p0,X), different from the auto-somes, which allows us to detect a sex bias in the direction of matings between AHM and Neanderthals. Although our formulae can easily incorporate sex-specific selection coefficients, we keep a single selection coefficient (sX) to reduce the number of parameters. Therefore, sX reflects the average reduction in relative fitness of deleterious Neanderthal alleles across heterozygous females and hemizygous males.
We fit the parameters p0,X, µX, and sX using our modified model to Sankararaman et al. [2014]’s observed levels of admixture on the X chromosome (Table 1 and Supplementary Figures S2.4 and S2.5). Given the smaller amount of data, the inference is more challenging as the parameters are more strongly confounded (for example µX and sX, see Figure S2.4 and S2.5). We therefore focus on the compound parameter µX sX, i.e. the average selection coefficient against an exonic base pair on the X. In Fig 4, we plot a sample of a thousand bootstrap estimates of µX sX for the X, along with analogous estimates of µs for autosomal chromosomes. For the X chromosome, there is also strong confounding between p0,X and µX sX, to a much greater extent than on the autosomes (note the larger spread of the X point clouds). Due to this confounding, our marginal confidence intervals for µX sX and p0,X overlap with their autosomal counterparts (Table 1). However, the plot of p0 and µs bootstrap estimates clearly shows that the X chromosome and autosomes differ in their parameters.
For reasons we do not fully understand, the range of parameter estimates for the X chromosome with strong bootstrap support is much larger for the ASN than for the EUR samples (Fig 4). For the ASN samples, the confidence intervals for µX sX include zero, suggesting there is no strong evidence for selection against introgression on the X. This is consistent with the results of Sankararaman et al. [2014], who found only a weakly significant correlation between the frequency of Neanderthal alleles and gene density on the X chromosome. However, as the ASN confidence intervals for µX sX are large and also overlap with the autosomal estimates, it is difficult to say if selection was stronger or weaker on the X chromosome compared to the autosomes. For the EUR samples, however, the confidence intervals for µX sX do not include zero, which suggests significant evidence for selection against introgression on the X, potentially stronger than that on the autosomes. Note that the selection coefficients on the X (sX, Table 1) are still on the order of one over the effective population size of modern humans, as was the case for the autosomes. Therefore, differences in effective population size between Neanderthals and modern humans, and hence in the efficacy of selection, might well explain observed patterns of introgression on the X as well as on the autosomes. If the exonic density of selection against Neanderthal introgression was indeed stronger on the X, one plausible explanation is the fact that weakly deleterious alleles that are partially recessive would be hidden from selection on the autosomes but revealed on the X in males [Charlesworth et al., 1987, Vicoso and Charlesworth, 2006, Meisel and Connallon, 2013].
Our results are potentially consistent with the notion that the present-day admixture proportion on the X chromosome was influenced not only by stronger purifying selection, but also by a lower initial admixture proportion p0,X (Fig 4). Lower p0,X is consistent with a bias towards matings between Neanderthal males and human females, as compared to the opposite. Based on our point estimates, and if we attribute the difference between the initial admixture frequency between the X and the autosomes (p0,X and p0,A) exclusively to sex-biased hybridization, our result would imply that matings between Neanderthal males and human females were about three times more common than the opposite pairing (S2 Text). However, as mentioned above, there is a high level of uncertainty about our X chromosome point estimates, therefore, we view this finding as provisional.
Discussion
There is growing evidence that selection has on average acted against autosomal Neanderthal alleles in anatomically modern humans (AMH). Our approach represents one of the first attempts to estimate the strength of genome-wide selection against introgression between populations. The method we use is inspired by previous efforts to infer the strength of background selection and selective sweeps from their footprint on linked neutral variation on a genomic scale [Wiehe and Stephan, 1993, McVicker et al., 2009, Sattath et al., 2011, Elyashiv et al., 2014]. We have also developed an approach to estimate selection against on-going maladaptive gene flow using diversity within and among populations (Aeschbacher and Coop, in prep.) that will be useful in extending these findings to a range of taxa. Building on these approaches, more refined models of selection against Neanderthal introgression could be developed. These could extend our results by estimating a distribution of selective effects against Neanderthal alleles, or by estimating parameters separately for various categories of sequence, such as non-coding DNA, functional genes, and other types of polymorphism [e.g. structural variation; Rogers, 2015].
Here, we have shown that observed patterns of Neanderthal ancestry in modern human populations are consistent with genome-wide purifying selection against many weakly deleterious alleles. For simplicity, we allowed selection to act only on exonic sites. It is therefore likely that the effects of nearby functional non-coding regions are subsumed in our estimates of the density (µ) and average strength (s) of purifying selection. Therefore, our findings of weak selection are conservative in the sense that the true strength of selection may be even weaker. We argue that the bulk of selection against Neanderthal ancestry in humans may be best understood as being due to the accumulation of alleles that were effectively neutral in the Neanderthal population, which was of relatively small effective size. However, these alleles started to be purged, by weak purifying selection, after introgressing into the human population, due to its larger effective population size.
Thus, we have shown that it is not necessary to hypothesize many loci harboring intrinsic hybrid incompatibilities, or alleles involved in ecological differences, to explain the bulk of observed patterns of Neanderthal ancestry in AMH. Indeed, given a rather short divergence time between Neanderthals and AMH, it is a priori unlikely that strong hybrid incompatibilities had evolved before the populations interbred. It often takes millions of years for hybrid incompatibilities to evolve in mammals [Fitzpatrick, 2004, Curnoe et al., 2006], and theoretical results suggest that such incompatibilities are expected to accumulate only slowly at first [Orr, 1995, Orr and Turelli, 2001]. While this is a subjective question, our results suggest that genomic data—although clearly showing a signal of selection against introgression-do not strongly support the view that Neanderthals and humans should be viewed as incipient species.
This is not to say that alleles of larger effect, in particular those underlying ecological or behavioral differences, did not exist, but rather that they are not needed to explain the observed relationship between gene density and Neanderthal ancestry. Alleles of large negative effect would have quickly been removed from admixed populations, and would likely have led to extended genomic regions showing a deficit of Neanderthal ancestry [as described by Vernot and Akey, 2014, Sankararaman et al., 2014, Dutheil et al., 2015]. Since our method allows us to model the expected amount of Neanderthal ancestry along the genome accounting for selection, it could serve as a better null model for finding regions that are unusually devoid of Neanderthal ancestry.
We have ignored the possibility of adaptive introgressions from Neanderthals into humans. While a number of fascinating putatively adaptive introgressions have come to light [Racimo et al., 2015], and more will doubtlessly be identified, they will likely make up a tiny fraction of all Neanderthal haplotypes. We therefore think that they can be safely ignored when assessing the long-term deleterious consequences of introgression.
As our results imply, selection against deleterious Neanderthal alleles was very weak on average, such that, after tens of thousands of years since their introduction, these alleles will have only decreased in frequency by 56% on average. Thus, roughly seven thousand loci (≈ µ × 82 million exonic sites) still segregate for deleterious alleles introduced into Eurasian populations via interbreeding with Neanderthals. However, given that the initial frequency of the admixture was very low, we predict that a typical EUR or ASN individual today only carries roughly a hundred of these weak-effect alleles, which may have some impact on genetic load within these populations.
Although selection against each deleterious Neanderthal allele is weak, the early-generation human–Neanderthal hybrids might have suffered a substantial genetic load due to the sheer number of such alleles. The cumulative contribution to fitness of many weakly deleterious alleles strongly depends on the form of fitness interaction among them, but we can still make some educated guesses (the caveats of which we discuss below). If, for instance, the interaction was multiplicative, then an average F1 individual would have experienced a reduction in fitness of 1 − (1 − 4 × 10−4)7000 ≈ 94% compared to modern humans, who lack all but roughly one hundred of these deleterious alleles. This would obviously imply a substantial reduction in fitness, which might even have been increased by a small number of deleterious mutations of larger effect that we have failed to capture. This potentially substantial genetic load has strong implications for the interpretation of our estimate of the effective initial admixture proportion (p0), and, more broadly, for our understanding of those early hybrids and the Neanderthal population. We now discuss these topics in turn.
Our estimate of p0 reflects the initial admixture proportion in the absence of unlinked selected alleles. However, the large number of deleterious unlinked alleles present in the first generation of hybrids violates that assumption, as each of these unlinked alleles also reduces the fitness of hybrids [Bengtsson, 1985]. The initial associations (statistical linkage disequilibrium) among these unlinked alleles will have quickly dissipated by segregation and recombination over subsequent generations. As such, our estimates of p0 are best thought of as an effective admixture proportion to which the frequency of Neanderthal alleles settled down to after the first few generations. The true initial admixture proportion may therefore have been much higher than our current estimates of p0. However, any attempt to correct this is likely very sensitive to assumptions about the form of selection, as we discuss below.
If the predicted drop in hybrid fitness is due to the accumulation of many weakly deleterious alleles in Neanderthals, as supported by our simulations, it also suggests that Neanderthals may have had a very substantial genetic load (> 94% reduction in fitness) compared to AMH [see also Do et al., 2015, Castellano et al., 2014]. It is tempting to conclude that this high load strongly contributed to the low population densities, and the extinction (or at least absorption), of Neanderthals when faced with competition from modern humans. However, this ignores a number of factors. First, selection against this genetic load may well have been soft, i.e. fitness is measured relative to the most fit individual in the local population, and epistasis among these many alleles may not have been multiplicative [Wallace, 1975, Kondrashov, 1995, Charlesworth, 2013]. Therefore, Neanderthals, and potentially early-generation hybrids, may have been shielded from the predicted selective cost of their load. Second, Neanderthals may have evolved a range of compensatory adaptations to cope with this large deleterious load. Finally, Neanderthals may have had a suite of evolved adaptations and cultural practices that offered a range of fitness advantages over AMH at the cold Northern latitudes that they had long inhabited [Weaver, 2009, Churchill, 2014]. These factors also mean that our estimates of the total genetic load of Neanderthals, and indeed the fitness of the early hybrids, are at best provisional. The increasing number of sequenced ancient Neanderthal and human genomes from close to the time of contact [Fu et al., 2014, 2015] will doubtlessly shed more light on these parameters. However, some of these questions may be fundamentally difficult to address from genomic data alone.
Whether or not the many weakly deleterious alleles in Neanderthals were a cause, or a consequence, of the low Neanderthal effective population size, they have had a profound effect on patterning levels of Neanderthal introgression in our genomes. More generally, our results suggest that differences in effective population size and nearly neutral dynamics may be an important determinant of levels of introgression across species and along the genome.
Materials and Methods
Model
We model the allele frequency dynamics at a neutral site ℓ that is linked to a locus under purifying selection after a single pulse of introgression from the Neanderthal population t generations ago. Let S1 and N1 be the introgressed (Neanderthal) alleles at the selected and linked neutral locus, respectively, and S2 and N2 the corresponding resident (human) alleles. The recombination rate between the two loci is r. We assume that allele S1 is deleterious in humans, such that the viability of a heterozygote human is w (S1S2) = 1 − s, while the viability of an S2S2 homozygote is w(S2S2) = 1. We ignore homozygous carriers of allele S1, because they are expected to be very rare, and omitting them does not affect our results substantially (S1 Text). We assume that, prior to admixture, the human population was fixed for alleles S2 and N2, whereas Neanderthals were fixed for alleles S1 and N1. After a single pulse of admixture, the frequency of the introgressing haplotype N1S1 rises from 0 to p0 in the human population.
In S1 Text and S2 Text we study the more generic case where both S1 and S2 are segregating in the Neanderthal population prior to admixture. Fitting this full model to data (S2 Text), we found that it resulted in estimates which implied that the deleterious allele S1 is on average fixed in Neanderthals. This was further supported by our individual-based simulations (S3 Figure S3.1), which show that in a vast majority of realisations, the deleterious allele was either at very low or very high frequency in the Neanderthals immediately prior to introgression. Therefore, we focus only on the simpler model where allele S1 is fixed in Neanderthals, as described above.
The present-day expected frequency of allele N1 in modern humans can be written as
where f (r, s, t) is a function of the recombination rate r between the neutral and the selected site, the selection coefficient s, and the time t in generations since admixture (S1 Text).
For autosomal chromosomes, we find that f is given by
For the non-pseudo-autosomal region of the X chromosome, which does not recombine in males, we obtain
where the factors 2/3 and 1 − 2/3 reflect the fact that, on average, an X-linked allele spends these proportions of time in females and males, respectively. Our results relate to a long-standing theory on genetic barriers to gene flow [Petry, 1983, Bengtsson, 1985, Barton and Bengtsson, 1986, Gavrilets, 1997, Gavrilets and Cruzan, 1998], a central insight of which is that selection can act as a barrier to neutral gene flow. This effect can be modelled as a reduction of the neutral migration rate by the so-called gene flow factor [Bengtsson, 1985], which is a function of the strength of selection and the genetic distance between neutral and selected loci. In a single-pulse admixture model at equilibrium, f is equivalent to the gene flow factor (S1 Text).
Lastly, we introduce a parameter µ to denote the probability that any given exonic base is affected by purifying selection. If µ and s are small, considering only the nearest-neighboring selected exonic site is sufficient to describe the effect of linked selected sites (but see Results and Discussion for the effect of unlinked sites under selection). That is, for small µ, selected sites will be so far apart from the focal neutral site ℓ that the effect of the nearest selected exonic site will dominate over the effects of all the other ones. In S1 Text we provide predictions for the present-day frequency of N1 under a model that accounts for multiple linked selected sites, both for autosomes and the X chromosome. We further assume that an exon of length l bases will contain the selected allele with probability ≈ µl (for µl ≪ 1), and that the selected site is located in the middle of that exon. Lastly, the effects of selection at linked sites will be small if their genetic distance from the neutral site is large compared to the strength of selection (s). In practice, we may therefore limit the computation of equation (1) to exons within a window of a fixed genetic size around the neutral site. We chose windows of size 1 cM around the focal neutral site ℓ. Taken together, these assumptions greatly simplify our computations and allow us to calculate the expected present-day frequency of the Neanderthal allele at each SNP along the genome.
Specifically, consider a genomic window of size 1 cM centered around the focal neutral site ℓ, and denote the total number of exons in this window by . Let the length of the ith nearest exon to the focal locus ℓ be li base pairs. The probability that the ith exon contains the nearest selected site is then
, where the product term is the probability that the selected site is not in any of the i − 1 exons closer to ℓ than exon i. Conditional on the ith exon containing the selected site, the frequency pt of N1 at locus ℓ and time t is computed according to equation (1), with r replaced by ri, the recombination rate between ℓ and the center of exon i. Then, we can write the expected frequency of the neutral Neanderthal allele at site ℓ surrounded by
exons as
where
The last product term accounts for the case where none of the
exons contains a deleterious allele. Equation (5) can be applied to both autosomes and X chromosomes, with f as given in equations (2) and (3), respectively.
Inference procedure
We downloaded recently published estimates of Neanderthal alleles in modern-day humans [Sankararaman et al., 2014], as well as physical and genetic positions of polymorphic sites (SNPs) from the Reich lab website. We use [Sankararaman et al., 2014]’s average marginal probability that an individual carries a Neanderthal allele as our Neanderthal allele frequency, pn, along the human genome. Although pn is also an estimate, we sometimes refer to it as the observed frequency, in contrast to our predicted/expected frequency pt. Sankararaman et al. [2014] performed extensive simulations to demonstrate that these calls were relatively unbiased. We performed separate analyses using estimates of pn for samples originating from Europe (EUR) and East Asia (ASN) (Table 1, Sankararaman et al. [2014]). Although composed of samples from multiple populations, for simplicity we refer to EUR and ASN as two samples or populations. We downloaded a list of exons from the UCSC Genome browser. We matched positions from the GRCh37/hg19 assembly to files containing estimates of pn to calculate distances to exons.
Our inference method relies on minimizing the residual sum of squared differences (RSS) between E[pt, ℓ] and pn,ℓ over all nl autosomal (or X-linked) SNPs for which Sankararaman et al. [2014] provided estimates. Specifically, we minimize
where gℓ(r, s, t, µ) is calculated according to equation (5).
For each population, we first performed a coarse search over a wide parameter space followed by a finer grid search in regions that had the smallest RSS. For each fine grid, we calculated the RSS for a total of 676 (26×26) different combinations of s and µ. We did not perform a grid search for p0. Rather, for each combination of s and µ, we analytically determined the value of p0 that minimizes the RSS as
where gℓ is given in equation (5) and we sum over all nl considered autosomal (X-linked) SNPs. For details, we refer to S2 Text.
We created confidence intervals by calculating 2.5 and 97.5 percentiles from 1000 bootstrapped genomes. We created these chromosome by chromosome as follows. For a given chromosome, for each non-overlapping segment of length 5 cM, and for each of 676 parameter combinations, we first calculated the denominator and the numerator of equation (7) using the number of SNPs in the segments instead of nl. We then resampled these segments (with replacement) to create a bootstrap chromosome of the same length as the original chromosome. Once all appropriate bootstrap chromosomes were created (chromosomes 1–22 in the autosomal case, or the X chromosome otherwise), we obtained for each bootstrap sample the combination of p0, µ, and s that minimises the RSS according to equations (6) and (7).
Individual-based simulations
To test whether selection against alleles introgressed from Neanderthals can be explained by the differences in ancient demography, we simulated the frequency trajectories of deleterious alleles in the Neanderthal and human populations, between the time of the Neanderthal–human split and the time of admixture (S3 Text). We assume that the separation time was 20, 000 generations (~ 600k years) and explore a range of effective population sizes for Neanderthals using a plausible distribution of selection coefficients [Boyko et al., 2008].
For each simulation run, we recorded the frequency of the deleterious allele in Neanderthals and humans immediately prior to admixture. Our simulations show that the majority of deleterious alleles that are still segregating at the end of the simulation are fixed differences (matching the assumption of our method, and the estimates of our more general method). Our simulations include both ancestral variation and new mutations, with the majority of the segregating alleles at the end of the simulations representing differentially sorted ancestral polymorphisms (S3 Figure S3.1, S3 Table S3.1).
In our simulations, the Neanderthal population accumulated more deleterious fixed differences than the human population for alleles with selection coefficients in the range of 10−5 < s < 10−3 (S3 Figure S3.1). Deleterious alleles with very low selection coefficients would not affect Neanderthal introgression levels, e.g. those that are below the nearly neutral boundary in humans, and so we impose a range of lower cutoffs on the distribution of selection coefficients of alleles we consider detectable by our approach (S3 Figure S3.2). In S3 Text we show that, for plausible values of this cutoff, the exonic density of selection and the average selection coefficient of these Neanderthal-specific fixed differences are of the same order of magnitude as our estimates of the autosomal µ and s respectively (S3 Figure S3.2). There is considerable uncertainty about a number of the parameters of this simulation model. However, the agreement between these simulations and parameter estimates from our model suggests that it is quite plausible that nearly neutral alleles make up the bulk of deleterious introgressed Neanderthal alleles.
Acknowledgements
We would like to thank Jeremy Berg, Vince Buffalo, Gideon Bradburd, Yaniv Brandvain, Nancy Chen, Henry Coop, Kristin Lee, Samantha Price, Alisa Sedghifar, Michael Turelli, Tim Weaver, Chenling Xu, and members of the Ross-Ibarra and Schmitt labs at UC Davis for helpful feedback on the work described in this paper. This work was supported by an Advanced Postdoc.Mobility fellowship from the Swiss National Science Foundation P300P3_154613 to SA, and by grants from the National Science Foundation under Grant No. 1353380 to John Willis and GC and the National Institute of General Medical Sciences of the National Institutes of Health under award numbers NIH RO1GM83098 and RO1GM107374 to GC.
Footnotes
↵† Supervised this work jointly.