Abstract
The processes that led to the demise of the Neanderthals and their replacement by modern humans have been the object of speculation, research, and heated debate. Most hypotheses fall into one of two categories: one highlights the role of climate change, epidemics, or other environmental pressures in the Neanderthals’ demise, and the other attributes it to direct or indirect competition with modern humans, who seem to have occupied the same ecological niche. The latter are based on the assumption that modern humans benefited from some selective advantage over Neanderthals that led to the latter’s extinction. We show that a scenario that includes migration and selectively neutral species drift can explain the Neanderthals’ replacement and is in line with the archaeological evidence. Our model offers a parsimonious alternative to those that invoke external factors or selective advantage, and can represent a null hypothesis in assessing such alternatives. We show that for a wide range of parameters this hypothesis cannot be rejected. Moreover, we suggest that although selection and environmental factors may or may not have played a role in the interspecies dynamics of the Neanderthals and modern humans, the eventual outcome of these dynamics, the replacement of the Neanderthals, was the result of the hominid migration dynamics at the end of the middle Paleolithic, namely repeated migration of modern humans from Africa into the Levant and Europe.
Significance statement Multiple factors have been proposed as possible drivers of the extinction of the Neanderthals and their replacement by modern humans circa 40,000 years ago: climate change, epidemics, and – most prominently – a selective advantage, such as superior cognitive capacity of modern humans over Neanderthals. We propose an alternative model that includes only migration of modern humans out of Africa into the Levant and Europe. We show that, given that the two species occupied a similar ecological niche, modern humans were destined to replace the Neanderthals even under a neutral scenario in which neither species has a selective advantage.
Introduction
One of the most intriguing questions concerning the evolution of modern humans is their relationship with other hominid species, particularly in light of recent findings showing that the genomes of modern humans carry the traces of introgression events with Neanderthals and Denisovans (1–5). Although many details of the process are unclear, archaeological and genetic evidence suggests that near the end of the middle Paleolithic, modern humans (henceforth Moderns) migrated out of Africa, where they had evolved and where their population was large, into the Levant and thence to other parts of Eurasia (6–12). As migrating bands of Moderns expanded the species’ range they encountered small populations of other hominid species – Neanderthals, Denisovans, and perhaps others – that seem to have occupied an ecological niche very similar to their own (13–16; see (17) and following commentaries). Archaeological findings point to a period of 10,000 to 15,000 years during which both Moderns and Neanderthals coexisted in the Levant and Europe, including a few thousand years in western Europe and including regional overlap and even recurring replacement of one species by another in particular dwelling sites (18–28; the period of coexistence is discussed in Supplementary Information, section A, SIA). The two species’ temporary coexistence ended in the complete disappearance of Neanderthals by 38,000 years BP ((29); Evidence in support of somewhat later Neanderthal existence is debated. Data regarding Denisovans are scarce, and not discussed in this study). A recent analysis of ancient DNA from an eastern Neanderthal suggests that introgression of Moderns into Neanderthal populations had occurred much earlier, roughly 100,000 BP; that is, the archaeologically-established period of overlap seems to have been preceded by earlier encounters between the two species (30, 31). This should not come as a surprise: Moderns’ remains are found in the Levant as early as 120,000 BP, and the evidence suggests plausible contemporaneous overlap between the two species’ ranges in the Levant for tens of thousands of years, prior to the Moderns’ expansion into Europe (17, 27, 28).
The underlying causes of Neanderthal replacement and its dynamics have attracted much interest and extensive study; hypotheses regarding this process fall into two broad, but not mutually-exclusive, categories. The first highlights environmental factors, such as climate change and epidemics as the causative agents (2, 32–37); The second attributes the Neanderthals’ replacement to direct or indirect competition with Moderns, in which Moderns had some selective advantage, possibly due to a wider dietary breadth, a more efficient mode of subsistence (38–41), advantageous differences in life history (42, 43), or – most prominently – a superior cognitive capacity, potentially reflected in material culture and tool use, symbolic thought as supported by artistic expression, and language (9, 44–49). A recent study has shown that even cultural differences alone, potentially interacting with population size differences, could have provided Moderns with a critical selective edge (50).
Many studies that assign a major role to a selective advantage of Moderns in the Neanderthals’ demise do so based on the premise that such an advantage had to exist in order to explain the latter’s demise, and they focus on determining what the selective advantage could have been. In this study we show that this assumption is unnecessary: selection may have played a role in the Neanderthals’ replacement, but the replacement could also have been the result of selectively neutral demographic processes, a parsimonious alternative that should a-priori be preferred. In fact, our simple model suggests that the migration dynamics mentioned above – recurring migration from Africa into the Levant and Europe – even at a low rate – were sufficient to result in the Neanderthals’ replacement even if neither species had a selective advantage over the other, and regardless of possible differences in population size between the two species. This replacement is found to have been extremely likely even if migration were bidirectional, when the estimated demographic state of affairs at the time is taken into account: a small Neanderthal population in Europe and the Levant, and a larger Modern population in Africa.
Our model plays two roles in the study of the relations between Moderns and Neanderthals. First, it acts as a null model, a parsimonious alternative to models of replacement that invoke selective advantage or environmental factors to explain the replacement. Second, appreciation of the fact that Neanderthals are expected to have been replaced by Moderns regardless of any possible selective advantage to the latter is in itself paramount to our attempts to reconstruct hominin evolution. That is, our finding would be important as a baseline for understanding Neanderthal-Modern dynamics, even if there were clear evidence that selection did play a role in the replacement process.
We first describe our model and present the results of numerical simulations. We then discuss these results in the wider context of archaeological and genetic findings, addressing alternative perspectives.
The model
We suggest that a model of migration and neutral species drift can explain the replacement of Neanderthals by Moderns and is in line with the evidence to date. Our model assumes no selective differences between the two species; that is, the competitive interaction between individuals or groups from the different species is identical to the competitive interaction between individuals or groups within the same species. Thus the two species are equivalent to two non-interbreeding subgroups of a single species. Although the two species did interbreed to some extent (4, 30, 51, 52), for simplicity we do not incorporate introgression into our model. In our model the only trait of interest is the species’ identity of individuals or groups; This formally equates the model with a simple, well-studied scenario: two selectively neutral alleles segregating at a genetic locus (53). Perhaps the most fundamental property of the dynamics in such a scenario (in the absence of mutation) is that random drift will ultimately lead to the fixation of one allele and the extinction of the other. Applied to species, the analogous process has been termed species drift (54, p. 735).
This portrayal of the Neanderthals-Moderns situation is already sufficient to explain why one of the species had to eventually disappear, and is in line with the archaeological evidence that points to a period of co-occurrence of the two species in Europe and the middle-east. However, in order to understand how and why the two species’ history would necessarily result in the Neanderthals’ extinction, we must take into account geographic and demographic aspects of the two species’ populations at the time. To do so, we model Europe and the Levant (deme 1, for simplicity, referred to henceforth as Europe) and Africa (deme 2, Africa) as separate demes with migration between them. The two demes have constant but possibly different hominid carrying capacities.
For realism and simplicity, we consider the dynamics of bands of individuals. That is, the entities whose fate is tracked in our model are small groups of individuals: such a band may die out by chance and be replaced by a propagule from another band (similar to the propagule pool model described in (55); see also (56–58); a propagule should be regarded as a copy of its band of origin). The carrying capacity of bands that reside in deme x (x = 1,2) is the constant Nx. The probability of occurrence of outgoing migration of a propagule from deme x per time step is denoted Mx, and is proportional to a parameter mx and to Nx. The rate of migration is assumed to be small enough that at most a single propagule can migrate per time step; accordingly, if mx * Nx > 1, we set Mx = 1, in which case migration occurs with probability 1 at every time step.
The population dynamics are those of a birth-death process akin to a Moran process with migration: at every time step a band chosen at random (regardless of its species’ identity) dies out, and is randomly replaced by a propagule from one of the other bands in its deme or by a migrant propagule that had arrived from the other deme during the most recent time step. We use the term establishment to describe the case in which a propagule migrated and was chosen to replace a band that died out. The probability of a migrant propagule’s establishment in deme x after arriving from deme y is thus 1/Nx. Only propagules of existing bands migrate; thus migration has no effect on the population in its deme of origin. These dynamics lead to a constant population size in each deme, equal to the carrying capacity of that deme, Nx. Both terms refer henceforth to the number of bands in a deme, except where noted otherwise.
The following set of transition probabilities constitutes a full mathematical description of this model. Let ix denote the number of bands of Moderns in deme x, and denote the probability per time step, at time t, of an increase by 1 of ix,
the probability of decrease by 1 of ix, and
the probability of no change to ix. As indicated by the lower subscript of ix, equations 1-3 describe the dynamics in deme 1, representing Europe, and equations for the dynamics in deme 2, representing Africa, are obtained by replacing every 1 by 2 and 2 by 1 in equations 1-3.



These equations are derived as follows: In equation 1, is the probability that in the current time step, a Neanderthal band in deme 1 died out (otherwise an increase in the Modern’s population in this deme during this time step is impossible). The terms (l − M2) and M2, respectively, represent the probabilities that migration from deme 2 did not occur and that it did occur. The term
is the probability that the propagule chosen to replace the one that died out in deme 1 (Europe) is Modern, given that no migration occurred; thus the number of candidate propagules that can act as a replacement is
and
represent, respectively, the probabilities that the migrant propagule to Europe is Modern, and that a Modern propagule is chosen to replace the band that died out. Another possibility that increases the Modern population in Europe, given that migration had occurred, is represented by
, i.e. the migrant to Europe is Neanderthal, and yet a Modern propagule is chosen to replace the band that died out. Equation 2 is composed of analogous constituents, whose interpretation is analogous to the description above.
This model is similar (but not identical) to the Moran process with mutation that is studied, for example, by Ewens ((53), p. 106), if one of the migration probabilities is zero.
In studying possible dynamics of Neanderthals and Moderns we shall limit the scope of our exploration to conditions in which one of the demes (deme 1, representing Europe) is initially populated only by bands of Neanderthals, and deme 2, representing Africa, is initially populated only by bands of Moderns. We first analyze the case in which migration occurs only from Africa to Europe (deme 2 to deme 1), the scenario that is widely believed to have taken place near to and during the interaction between the Neanderthal and Modern populations, based on the lack of evidence so far that would support Neanderthals’ existence in Africa. We then report simulation results for the case in which migration occurs in both directions.
Results
1. Unidirectional migration from Africa to Europe leads to Neanderthal exclusion
We suppose that all individuals in deme 2 (Africa) are Moderns, and M2 > 0, M1 = 0 (migration occurs only from deme 2 to deme 1, Europe). This results in complete replacement of Neanderthals by Moderns, regardless of the size of M2 or the relative values of N1, N2, the carrying capacities of the two demes. This is because there is constant influx of Moderns into Europe, while within Europe stochastic drift takes place. Thus the process can be viewed as a random walk with a single absorbing boundary: if the frequency of Neanderthal bands in Europe reaches zero, there is no further change, while zero Modern bands in Europe is not an absorbing state due to continued migration from Africa. For the process described by equations (1)-(3) the expected number of time steps for the exclusion process to be realized can be calculated using equations 2.144 and 2.160 in (53), and is not provided here.
Effects of migration rate and carrying capacity
Numerical stochastic simulations of the process described by equations (1)-(3) reveal a number of interesting aspects of the process of species replacement; for example, the relationship of the hominid band carrying capacity in Europe, N1, to the time scale on which the replacement occurs and to the number of successful establishment events of migrating propagules. For a fixed probability of migration, M2, we find that as Europe’s carrying capacity, N1, becomes larger, proportionally more migrations are required before one of the propagules from Africa establishes and ultimately leads to species’ replacement (Fig 1A). The time that it takes for successful establishment and subsequent fixation to occur is nearly proportional to (N1)2 (Figure 1B), because the probability that a migrating propagule will establish is proportional to 1/N1 and the mean time from successful establishment to fixation is proportional to N1 (53). See Supplementary Information, section B (SIB), for analogous results with different migration rates.
(A) Mean number of simulation time steps until complete replacement of Neanderthals by Moderns. Here and in (B) and (C), red error bars denote two standard errors around the mean, and blue bars denote the standard deviation; 500 replicate simulation runs were conducted with each carrying capacity value. The green line is a fit of the means to a quadratic function, demonstrating that the number of time steps to fixation scales with (N1)2. (B) The mean number of migrant establishments in Europe that take place until replacement occurs is linearly related to N1. Here and in (C) the green line is a fit to a linear function. (C) The number of migrant establishments that occur while the Moderns are segregating in Europe on their way to fixation (see text) scales with N1. For all panels, the probability of migration into Europe per time step, M2 = 0.1 (N2 = 100, m2 = 0.001).
Another result concerns the effects of the rate of migration and the European carrying capacity, N1, on the mean number of migration events into Europe that may contribute to the Modern population at the time of fixation; that is, how many migrant propagules might eventually have offspring in the population? When migration is rare (M2 is small) or N1 is small, a single migrating propagule may establish and drift to fixation without any subsequent Modern establishment events taking place during the process. When migration is sufficiently large, it is likely that more than one establishment event occurs before fixation of Moderns, and each of these migrations may contribute to the eventual composition of the population of Moderns in Europe (Figure 1C). To demonstrate this we kept the migration rate constant and ran stochastic simulations of equations (1)-(3) with different population sizes N1 (keeping population size constant and altering migration rate gives similar results). Figure 1C shows that the number of potential contributors to the fixing population rises proportionally to the carrying capacity, N1. This is because the mean time that Moderns segregate in the population until fixation scales with the population size. This pattern bridges what may seem to be a gap between our model’s assumptions of ongoing migration from Africa to Europe, and suggestions that the archaeological record does not support more than a single out-of-Africa event into the Levant and from it to Europe: evidence of a single migration event is to be expected under our model if the rate of migration is low or if Europe’s carrying capacity is small (Figure 1C).
These relationships may be useful in testing our model with the accumulating of empirical evidence concerning the replacement process, because they should leave signatures in the archaeological and genetic records.
The time trajectory of replacement
A potential argument against the sufficiency of a neutral model to explain Neanderthal replacement is that the archaeological evidence within continental Europe seems to point to a clear process of directional selection in which Moderns increase in frequency while Neanderthals disappear, a pattern that might not seem to be in line with a drift explanation. This interpretation may be contested in light of recent re-assessment of archaeological finds that were initially assumed to be associated with Moderns based on their cultural complexity (59–62). Further, even if one accepts that the process was directional, two properties of the demographic processes should be considered: one is that, although neutral, the process we describe is biased by unidirectional migration, which may underlie an increase over time in the Moderns’ frequency, but may be unimportant if the probability of migration is low, particularly if the overall population size in Europe is not small. A second, somewhat less intuitive consideration, is that the trajectory we should expect to have left its mark in the archaeological record according to our model is drawn from a distribution different from the one we are used to attributing to random drift; it is conditional on having reached the point of one species’ fixation. A significant part of such a distribution is composed of trajectories that seem directional, particularly, as seen in Figure 2, in the period approaching final fixation.
Number of bands of Moderns in Europe over time in nine randomly selected simulation runs (A-I), with a carrying capacity of N1 = 500 bands in Europe, and migration probability from Africa to Europe, M2, of 0.1 per time step. Red and green circles denote, respectively, the last time that the Moderns and that the Neanderthals made up at least 10% each of the hominid population in Europe. Many trajectories, particularly in the final phase leading to Modern’s fixation, are highly directional.
The duration of replacement
The replacement of Neanderthals by Moderns seems to have occurred surprisingly fast when compared to archaeological and evolutionary time scales of the two species’ existence. This may be why many scholars assume that the process was necessarily driven by selection, but whether the process should be considered as having been rapid depends on properties of the model assumed, such as the Neanderthal population size at the time, and the expected duration of replacement. Our model, which takes into account major aspects of the two species’ demography at the time but does not include selection, may be regarded as a baseline, or a null model, for such an evaluation. To assess whether this null model can be rejected in favor of a selection scenario, we study the distribution of replacement durations produced by the model and compare it to the replacement duration suggested by empirical evidence. Such an attempt faces a number of obstacles.
First, it is notoriously hard to correlate the time units in evolutionary models with the time span of real life scenarios. This is also the case in our model, which entails the selectively neutral replacement of bands, whose empirical rate is hard to gauge and for which there are no clear estimates. Second, the species’ replacement duration in our model depends critically on the parameter N1, the number of bands in Europe. Estimates of hominid population sizes in Europe near the end of the middle Paleolithic vary over more than an order of magnitude and – according to reconstructions of paleo-climate during this era – are likely to have changed significantly while the replacement was taking place (15). Finally, the way in which replacement duration is estimated may affect the result by more than an order of magnitude. Comparison between the simulations and the archaeologically-supported period of co-existence of the two species should take into account the time point at which species’ co-existence is likely to be evident in the archaeological record. That is, the appropriate duration to be compared should not be the full duration of each model simulation, but the period during which both species are likely to have a demonstrable presence. This would be based on archaeological findings that can be clearly associated with the identity of the species that produced them, and could, for example, be the period from the initial crossing of some frequency threshold by the Moderns until the Neanderthals constitute less than this threshold in the overall European population, or the period between the last crossings of these thresholds (see SIA for a discussion of alternatives).
Following these caveats, we have conducted statistical analyses for a range of combinations of parameters. Because our model serves as a null model relative to models of species replacement that are not neutral (for example, that assume selection), we test, for each combination of parameters, whether our model can be rejected at a significance level of p<0.05. In other words, we attempt to assess, under a range of possible choices and assumptions, whether the time required for species replacement according to our model is within a reasonable range compared to the empirical evidence. Figure 3A presents the ranges of durations of species’ coexistence for various carrying capacities in deme 1 of our model, i.e., Europe, over 500 simulations per carrying capacity. These ranges are split into the 5% fastest replacements (red) and the remaining 95% (green), thus providing the information about the distribution of replacement durations in our neutral model that allows a statistical test. The results are presented in units of the average number of times that each territory that supports a band of hominids changed hands during the replacement process. This unit is used for ease of interpretation, although the model is not spatially explicit, and is calculated as the overall number of events in which a band died out and was replaced by another (of the same or of the other species), divided by the population size, N1. The results are derived for a single rate of migration, of a migrant propagule arriving in Europe every 10 time steps on average, i.e. M2 = 0.1. Figure S1 compares the mean and median replacement durations for different rates of migration, and five possible methods for measuring the time period (SIA). Most methods yield similar results over a wide range of migration rates.
(A) Range of durations of species’ coexistence for various carrying capacities in Europe, N1, in units of average number of events of band replacement per band territory. The range that covers 95% of the results is marked in green, and the 5% of the simulations with the shortest coexistence durations is in red (500 simulation replicates were conducted for each carrying capacity). The mean and median values are marked in black and cyan circles, respectively. Coexistence duration is defined as the period during which both species exist in the population at frequencies above 10%, between the last crossings of this threshold by each of the species, as shown by the red and green circles in the time trajectories of Figure 2. See SIA for more details and discussion of alternative definitions. (B) Tests of the hypothesis of neutral replacement for a range of parameter combinations, assuming a species coexistence duration of 12,000 years: each point in the panel represents the result of a test at significance level p<0.05. The range of parameters for which neutral replacement should be rejected is denoted in red. The range for which the model should not be rejected is marked in green.
To test whether our model of neutral species drift should be rejected, we ask whether the archaeologically-supported duration of species coexistence falls in the range of the 5% shortest coexistence durations or outside of this range (see Figure 3A). In order to compare the simulation results to the period of species’ coexistence in the archaeological record, one must assume a rate at which bands replace one another. This neutral replacement rate is a result of local extinctions due to stochastic (non-directional) environmental factors, stochasticity in reproductive success, or competition among bands that do not have any selective differences. This rate, the characteristic band size, and the hominid carrying capacity of Europe in the middle Paleolithic, are unknowns on which the rejection of our null model depends. Figure 3B depicts whether the model should or should not be rejected at a significance level of p<0.05 for a range of band replacement rates (equivalent to the rate at which territories change hands) and for a range of carrying capacities in Europe, using 12,000 years as the archaeologically-supported period of species’ coexistence. In other words, every point in Figure 3B is the result of a hypothesis test for the combination of parameters depicted along the X and Y axes. Figure 3B is derived from the results presented in Figure 3A. The duration of species coexistence is the time from the last simulation time step in which Moderns comprised 10% of the population in Europe until the last time that they reached 90% of it (see SIA for a discussion of this choice and alternatives). The rates are described in units of the probability of band replacement per generation (25 years). See SIA for analogous results under the assumption of 5,000 years of species’ overlap, and further discussion of estimates of the duration of the species’ coexistence.
Interpretation of figure 3B depends on the assumed band size and Europe’s carrying capacity. For example, for a carrying capacity of 5000 individuals, a slightly higher estimate than the mean population size in Eurasia suggested by Bocquet-Appel et al. (63) during this period, with a band size of 50 individuals (see in this regard (56–58)), our model should be rejected only if one asserts that the rate at which band territories change hands corresponds to a probability of replacement per generation of 0.05 or lower; that is, if a territory changes hands on average less than once in every approximately 500 years. For a carrying capacity of 50,000 individuals (the order of magnitude according to figures suggested by Hassan (64) applied to the known habitation region in Europe in the beginning of the upper Paleolithic) but band size of 500 (in line with the definition of a tribal group in (65), following (66)), the rejection threshold remains the same. Rates of replacement for contemporary traditional populations summarized by Soltis et al. (67) are between 2% and 30%, but refer to group sizes on the order of many hundreds or thousands of individuals, who are sedentary and rely to a great extent on farming of crops and livestock. Bocquet-Appel et al. (15) review numerous population size estimates for the Neanderthal population, based on different methods, and conclude that it was in the range of 5,000-70,000 individuals. Even if one adopts the high end of this range, 70,000 individuals, our null model should be rejected only for the lowest extreme of the range of replacement rates suggested by Soltis (67) when considering – in accord with the accounts reported there – a deme size of 1000 individuals.
In sum, the above analysis suggests that the time scale on which species replacement took place according to the archaeological record is well within the range predicted by our model. Moreover, this null model should be rejected only if one assumes a rate of neutral band replacement or an overall hominid carrying capacity in Europe that are extreme according to ranges that have been suggested for these two variables.
2. Bidirectional migration between Africa and Europe: migration probabilities and carrying capacities strongly influence outcome
Archaeological evidence suggests that the Levant was the southernmost tip of the Neanderthal population, and thus does not support migration of Neanderthals into Africa. However, it is possible that such migrations occurred, and we explore a number of scenarios for this bidirectional migration.
Because the two populations in our model are finite, all scenarios - apart from the case in which there is no migration between the two demes – inevitably end in fixation of one species and extinction of the other. Thus the question is how the probability of each species’ fixation depends on the migration parameters and the carrying capacities of the two demes.
The initial condition in all scenarios that we explore is that deme 1 (Europe) is populated by Neanderthals and deme 2 (Africa) by Moderns. Without loss of generality, we treat only cases in which the population size in deme 2 is equal to or greater than the population size in deme 1, i.e. N2 ≥ N1. Arguments supporting this assumption include:
The effective population size that is inferred based on genetic diversity for Moderns is larger than that of Neanderthals by a factor of 2 or more (see, e.g., (15, 63, 68, 69)).
Population size estimates based on archaeological evidence suggest that Neanderthal population size was significantly smaller than that of the Modern population in Africa (e.g., (15, 63, 68)).
Based on its size and biological productivity, Africa could have supported a much larger hominid population, and this difference is likely to have been greatly magnified during ice ages in which large parts of Europe were uninhabitable (70, 71).
Some accounts of the archaeological record suggest, however, that populations in much of Africa near the transition from the middle to upper Paleolithic were small (e.g. (72)). Contemporaneous bottlenecks and reductions in the Neanderthal population size, potentially driven by glaciation, can consolidate these accounts with the inference from genetic data, which points to a significantly smaller Neanderthal population size than of Moderns.
Symmetric migration
If the parameters of outgoing migration are equal, i.e. m1 = m2, one might expect that the relative probabilities of fixation of the two species would be equal to the ratio of population sizes in the two demes. Somewhat non-intuitively, this is not the case; the species that is initially in the larger of the two demes (deme 2, Africa) has a fixation probability in Europe that is greater than its relative initial population size would suggest. This is because the initial conditions increase this species’ probability of early success in a number of ways: First, to a good approximation, once established, progeny of a migrant from deme x will drift to fixation with probability inversely proportional to Ny. The probability ratio of this occurrence is Nx/Ny for a migrant from x to fix compared to that of a migrant from y. Second, the number of migrants is biased in favor of outgoing migration from the larger deme: we defined the probability of migration in our model as dependent on both the migration parameter and on population size, and so migration occurs more frequently from the larger deme. Third, a migrant from deme x has a probability proportional to 1/Ny of establishing in deme y following its arrival, and thus the probability of establishment following arrival of a migrant from deme x is more likely by a factor of Nx/Ny than the establishment probability in deme x of a migrant from deme y. This potential advantage is corrected for in our simulation by our definition of a time step: at each time step a band in one of the demes dies, and since the choice of band is random, more time steps are realized as dying events in the larger deme; thus migrants from the smaller deme have a smaller probability of establishing, but a proportionally larger number of time steps in which such establishments may occur.
Put together, these effects suggest that the ratio of fixation probabilities, P1/P2, should reflect an advantage to the species from the larger deme that is proportional to the ratio of the population sizes to the power of two. However, these intuitions are exact only when the initial conditions have not changed, and each deme is still composed of a single species. The bias in favor of the species that originated in the larger deme is somewhat attenuated through time, and we should thus expect a ratio of fixation probabilities that reflects an advantage that is somewhat smaller than (Nx/Ny)2 to the species that originates in the larger deme. Figure 4 shows the probability of the fixation of Moderns as a function of the ratio between the two demes’ population sizes.
(A) Probability of Modern’s fixation. (B) Ratio between Moderns’ and Neanderthals’ probabilities of fixation, which is approximated by the square of the inverse of the ratio of population sizes (red) for much of the range of this ratio. It is slightly lower for scenarios in which the Moderns’ population is much larger than the Neanderthals’. For all runs, m1 = m2 = 0.0001, N2 = 500 (Africa), N1 refers to Europe.
Asymmetric migration
If m2 > m1, i.e. the parameter of migration out of Africa is larger, we find that the probability of Moderns’ eventual fixation is greater than that of Neanderthals, as is expected since N2 > N1. This holds true even for some cases in which m2 < m1, as a result of the effects of the larger population size. Although unrealistic given the archaeological findings, study of various ratios between the two migration parameters, in which the parameter of migration from Europe into Africa is greater than its counterpart, i.e. m1 > m2, is of interest as it allows us to explore the effects of population size differences (Figure 5). For a population size ratio of 1:10 between Europe and Africa, we find – in line with the previous analysis – that even with a 10:1 ratio of migration rates, which creates equal probabilities of migration in the two directions, i.e. M1 = M2, the probability of Neanderthal fixation remains very low (~0.1). Only when the migration rate out of Europe (m1) is greater by a factor of 100 than its counterpart (m2) do Neanderthals and Moderns have the same probability of fixation (Figure 5).
The probability of Moderns’ replacement of the Neanderthals for a range of ratios between the migration parameters, m1/m2, with a constant ratio of population sizes. For all runs, N1 = 50, N2 = 500, m2 = 5*10-6.
Discussion
We have shown that a simple selectively neutral model of population dynamics, namely random drift in finite populations with migration, can account for the replacement of the Neanderthals by Moderns that occurred near the transition between the middle and upper Paleolithic. Although a stochastic process, this replacement was certain to occur, even in a selectively neutral setting, given the estimated migration pattern near the onset of the interaction between the two populations, namely repeated migration of small propagules of Moderns out of Africa into the Levant and Europe. Replacement of the Neanderthals was certain to occur even for very low migration rates, as long as migration was unidirectional, regardless of the ratio between the population sizes in the two demes. We have also demonstrated that even if bi-directional migration between Europe and Africa had occurred, Moderns would have been extremely likely to eventually replace Neanderthals, given the estimated differences in population size between the species, in favor of Moderns. This stems from a disproportionate impact of initial population size differences on the probability of eventual fixation.
Our model is a parsimonious alternative to a model in which selection is the major driver in the replacement of the Neanderthals. We show that the time scale on which the replacement occurred according to the archaeological record is within the range of replacement durations predicted by our model for a wide range of parameter values, unless fairly extreme values for demographic parameters, such as Europe’s carrying capacity, are assumed. As demonstrated in SIA, many alternative measures for the duration of the process would yield results of the same order of magnitude. Under a wider range of demographic parameters than we used, as well as incorporation of geographical structure within Europe into the model, our model might produce results that are incompatible with empirical evidence. However, the difference would not be by orders of magnitude, suggesting that a model of selectively neutral replacement could be reasonable under a wide range of demographic and geographical scenarios.
In addition to the duration of the replacement process, it would be desirable to discern other patterns that might distinguish a scenario of neutral species’ replacement from one that is driven by selection. The characteristics of the Moderns’ frequency trajectory over time may provide such a pattern: although many trajectories of neutral fixation seem very directional and do not differ significantly from the near-deterministic trajectory expected under a selection scenario, some trajectories are far from monotone (e.g. Figure 2, panel I) in their eventual ascent towards fixation, and many are characterized by early phases that seem disjoint from the final phases of the fixation process. In these early phases, the Moderns’ frequency reaches intermediate values and then decreases (e.g. panels B, C, D, F, G in Figure 2). Such a pattern is not expected under a selective scenario, which is predicted to produce a near-deterministic trajectory once some minimal frequency threshold is crossed (53). In other words, an increase of Moderns to intermediate frequency, followed by a significant drop, is expected to occur in some, but not in all, cases of neutral replacement, and is extremely unlikely under a selection-driven replacement process. If evidence of a transient Moderns’ intermediate frequency were found, it would argue against the role of (strong) selection in the replacement process.
Such early increases to intermediate frequency are likely to go undetected in the archaeological record for many reasons – its sparseness, the low likelihood of uncovering skeletal remains in a site populated for a short time, and the disconnect from a long-term archaeological context that would help to shed light on the species’ identity. Also, early Moderns in Europe were characterized by material cultures different from those associated with the species during their later, well-established, period of existence there, which may increase the likelihood that the remains would be misclassified as Neanderthal (see also SIA); indeed, a recent detailed analysis of lithic technologies suggests that such misclassification may have occurred, and supports a model of multiple early Modern migrations out of Africa that reached intermediate frequencies in Eurasia (73). Similarly, findings of anatomically modern humans in the Levant in multiple sites from the late middle Paleolithic suggest a probable species’ overlap in this region in considerable numbers over potentially long durations (27). Recent developments in the field of ancient genomics may shed light on this question by providing evidence of early introgression events (30, 31). The probability of detectable successful introgression as a result of any single contact between two species is low, with reasons ranging from the low likelihood of the occurrence of productive sexual contact in the first place, to potential hybrid fitness disadvantage, and drift and possible selection acting to obscure or eliminate the genomic traces of such introgression (52, 69, 74, 75). Thus, observed introgression is likely to be a reflection of a history of substantial interspecies contact, which is most likely to have occurred when both species were in the same region at intermediate frequencies. Evidence of at least one early introgression from Moderns into a subgroup of Neanderthals has recently been found (31), and is estimated to have occurred well before Moderns are identified in the European archaeological record. This lends support – according to the prediction in the previous paragraph – to a reevaluation of the time trajectory of Moderns in Europe and the Levant that is not in line with their having a (strong) selective advantage.
Effectively-neutral replacement could also have occurred under a fairly broad range of conditions, in which selection acts on differences between the two species: under a range of conditions, efficiency of selection is approximately proportional to population size (53). Because of the relatively small population sizes of both species, even if differences between the species had led one to have a selective advantage, this advantage may have had little effect at the population level, leaving the system in a nearly-neutral regime for which our model holds. This would have been the case for a fitness advantage s such that N ⋅ s is near 1, i.e. for an overall fitness advantage of s=10-4 or lower. Because the initial interaction between the groups was probably more localized than in our model and probably occurred between small subgroups of the two populations, the range of selection coefficients for which the species’ interaction would have been within the nearly-neutral regime, at least during part of the demographic process, is realistically even broader, perhaps up to an advantage of s=10 −3 or s=10−2 to one of the species, if the sizes of the interacting populations were on the order of hundreds or thousands of individuals. In other words, even if Neanderthals had a selective advantage, unless it was very large, they are likely to have been eventually replaced by species drift. Introgression between the two species might have mitigated this effect, potentially allowing one or both species to incorporate advantageous alleles from the other species and to radically reduce the selective differences between the two species, even if admixture was limited and even if hybrid lineages initially had some selective disadvantage (75–81).
We do not suggest that there were no differences between the species that had an effect on fitness; on the contrary: morphological and genetic differences between the two species suggests that they differed phenotypically in ways that are highly likely to have affected fitness. Which of these differences conferred a selective advantage is debatable, and how such selective differences in various traits acted jointly to affect the overall fitness will likely remain unknown. Arguments in favor of a selective advantage to either species with regard to sets of traits are compelling, ranging from potential advantage to the Neanderthals stemming from adaptation to local conditions such as climate and pathogens (9, 78), to selective advantage of Moderns due merely to their greater overall population size and proportionally smaller predicted mutational load (69), to differences in morphology, mode of subsistence, cultural aspects such as tool use, and, possibly, cognitive capacity. Such studies are important and insightful in our attempts to unravel the evolutionary history of the two species, but their findings’ interpretation in this context must be careful. Alongside providing support for probable selective advantages to each species over its counterpart with regard to particular traits and thus not providing a decisive conclusion regarding the overall species’ relative fitnesses, many of these studies compare material findings associated with the two species in sites that are not contemporaneous (see, e.g. (82, 83)), a comparison whose findings’ should be interpreted with great caution in our reconstruction attempts of the dynamics of species’ replacement (84). Our study demonstrates that species replacement is expected under a neutral model, in a manner compatible with the replacement that took place, and that neutral processes are able to account for the inter-species interaction regardless of whether selective differences between the two species existed.
An extensive review of the arguments related to the possible selective advantage of Neanderthals over Moderns or vice versa is beyond the scope of this study. We find it important to address one major line of argument in this context, which suggests that Moderns had a cognitive and cultural advantage, potentially in the form of symbolic thought or language, over Neanderthals (9, 44–49, 72, 85). To date, genetic and cranio-morphological comparisons between the species have not produced any unequivocal evidence that would support this argument, which is grounded mostly in the material archaeological record of artifacts and cave drawings that seemingly provide fairly convincing circumstantial evidence: during the period of co-habitation in Europe of the two species and within the first ten millennia that followed it, a demographic and cultural revolution occurred in Europe (86). Population densities increased by a factor of 2-10 in many localities (87), previously uninhabited regions were colonized (88, 89), forms of artistic expression became much more common than before (86), and the repertoire and complexity of tools grew dramatically (72, 90, 91).
Whether Neanderthals were responsible for some of these novelties and whether the revolution was as sudden as initially thought has attracted much discussion (60, 61, 92–98), as has the possibility that Neanderthals and Moderns had significant cultural exchange, suggesting that they were – at the least – comparable, if not on par, in their cognitive abilities (91, 99).
We suggest a number of additional arguments that call for a guarded interpretation of the demographic and cultural shifts as reflecting a selective advantage of Moderns. Undoubtedly, the living conditions of both species changed extensively during this period of time: Moderns’ migration to new localities exposed them to novel challenges, and both species were faced with increased hominin competition and exposure to new ideas and practices together with – potentially – direct competition different from any that preceded it. These may have been accompanied or preceded by significant independent environmental changes (33, 34, 36, 100, 101). In light of these changes, extensive demographic and/or cultural changes were likely to have occurred in both species (98, 102), even if the species did not differ in their cognitive capacities and even if no change in cognitive abilities occurred throughout this period (as has been widely speculated, e.g. (9, 72)). Dramatic punctuated changes occur in many biological systems (54, 103), and are particularly likely to be triggered by extensive changes such as the ones that the Neanderthals and Moderns went through. In previous work we have demonstrated that such sudden change is specifically to be expected in the evolution of culture (104), especially upon exposure to cultural novelties, which can easily trigger innovations by analogy or by combination with existing practices. Such cultural changes can lead to a further rapid, possibly exponential, rise in cultural complexity (104–107), which may in turn prompt demographic change. In other words, we suggest that the increase in cultural complexity that is found in Europe near the replacement of Neanderthals by Moderns may be the result of the Moderns’ geographic expansion and of the two species’ interaction, rather than the cause of the replacement or its driver (see also (28)).
A second relevant observation is that if a cognitive and cultural advantage were a driver of the Moderns’ spread from Africa into the Levant and from there to Europe, one might expect to find cultural continuity between archaeological sites along this route near the transition from the middle to the upper Paleolithic. As has been pointed out and widely discussed, there is no clear-cut evidence for such continuity (see, e.g., (20, 28, 86, 98, 102, 108)). Moreover, the material cultures associated with Moderns and Neanderthals in the Levant during the late middle Paleolithic, in the period preceding the replacement, are indistinguishable from one another (see, e.g., (27, 28)). The appearance of advanced cultural features in Europe and the Levant only after species’ interaction was likely to have taken place is in line with our suggestion that advanced culture is an outcome of this interaction (and see, e.g., (95, 109)). Similarly, if the cultural burst is a result of the species’ interaction, one may expect to find in the emerging cultures some characteristics of the individual cultures that preceded them, including continuity of some local features in some regions. This may be the case in variants of the Ahmarian, Aurignacian and transitional techno-complexes from this period (see, e.g., 89, 109–111, and 112 and ensuing comments). Finally, the “full package” of upper Paleolithic modernity appears in most of the regions that were populated after the Neanderthals’ replacement – Siberia, East Asia, and the Sahul – only between 10,000-20,000 years later, suggesting it may have developed only in particular populations after the replacement had occurred or as it was taking place, and that its role in the replacement and the moderns’ geographic spread was limited (113–118)).
In sum, we do not endorse any particular stance as to whether Moderns did or did not have a cognitive or cultural advantage over Neanderthals, but point out that much of the evidence in support of this claim should be interpreted cautiously.
We suggest that migration dynamics together with events of local dispersal and replacement, in a selectively neutral model, can explain the Neanderthal-Modern interaction and subsequent replacement of the former by the latter, without invoking selection or external environmental factors, and even – under some scenarios – regardless of possible difference in population size between the species. Advanced methods of dating archaeological findings and new methods in ancient genomics are expected to provide more detailed information within the next few years; Combined with models of the two species’ interaction that take into account geographical sub-units, population substructure, and introgression, new empirical data may soon enable us to further elucidate the dynamics that led to the Neanderthals’ replacement and to assess in more detail whether a neutral model such as that we propose is sufficient to explain this process and its outcome.
Acknowledgements
We thank Richard Klein, Erella Hovers, Kenichi Aoki, and Victor Garcia, for insightful comments and suggestions. This research is supported in part by the John Templeton Foundation and the Stanford Center for Computational, Evolutionary, and Human Genomics.
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.
- 8.
- 9.↵
- 10.
- 11.
- 12.↵
- 13.↵
- 14.
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.
- 20.↵
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.↵
- 28.↵
- 29.↵
- 30.↵
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.
- 36.↵
- 37.↵
- 38.↵
- 39.
- 40.
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.
- 46.
- 47.
- 48.
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.
- 77.
- 78.↵
- 79.
- 80.
- 81.↵
- 82.↵
- 83.↵
- 84.↵
- 85.
- 86.↵
- 87.↵
- 88.↵
- 89.↵
- 90.↵
- 91.↵
- 92.↵
- 93.
- 94.
- 95.↵
- 96.
- 97.
- 98.↵
- 99.↵
- 100.↵
- 101.↵
- 102.↵
- 103.↵
- 104.↵
- 105.
- 106.
- 107.↵
- 108.↵
- 109.↵
- 110.
- 111.
- 112.
- 113.↵
- 114.
- 115.
- 116.
- 117.
- 118.↵