## Abstract

Sexually reproducing populations with self-incompatibility bear the cost of limiting potential mates to individuals of a different type. Rare mating types escape this cost since they are unlikely to encounter incompatible partners, leading to the deterministic prediction of continuous invasion by new mutants and an ever increasing number of types. However, rare types are also at an increased risk of being lost by random drift. Calculating the number of mating types that a population can maintain requires consideration of both the deterministic advantages and the stochastic risks. By comparing the relative importance of selection and drift, we show that a population of size *N* can maintain a maximum of approximately *N* ^{1/3} mating types. Although the number of mating types in a population is quite stable, the rare type advantage promotes turnover of types. We derive explicit formulas for both the invasion and turnover probabilities in finite populations. Being vanishingly rare is thus a blessing and a curse associated with both universal compatibility and a high risk of extinction.

## 1. Introduction

Splitting a population of universally compatible gametes into two or more self-incompatible mating types can potentially limit inbreeding depression, control the spread of selfish organelles, and help mate searching [Hoe82, HH92, DXVS05, BLVD^{+}11, HP16]. Although the origins and exact benefits of producing different mating types remain controversial [Per12], the costs are clear: self-incompatibility reduces the number of potential mating partners and increases the potential for subfertility [Fis30, Pow76]. However, if gametes are compatible with any class but their own, new (and therefore extremely rare) mating types do not pay this cost: they can mate with any member of the population. Hence, novel mating type mutants should increase in frequency until all types are equally represented. The deterministic outcome of mating type evolution is the inexorable rise in the number of different types driven by negative-frequency dependent selection for rare mutants, also referred to as the “rare sex advantage” [IS87]. Few studies have considered the flip-side of the rare sex advantage, i.e. rarer types are at much greater risk of being lost from finite populations by random genetic drift. These stochastic extinction events not only hinder invasion of rare mating types, they also avoid the population to consist of too many mating types at low frequencies.

Many different genetic mechanisms of mating type determination have evolved, including those regulated by two unlinked loci like the tetrapolar mating systems of many fungi hypothetically capable of generating hundreds of compatible mating types, and the more widespread single locus incompatibility systems. Here, we focus on a system of mating type determination typified by the yeast *Saccharomyces cerevisiae* where a single haploid locus is responsible. Despite the very strong negative frequency-dependent selection for rare mating types, most single locus haploid incompatibility systems are binary. They produce only two mating types, similar to the two sexes found in many animals despite a population with only two types experiencing the highest rate of incompatible matings [Hur96]. Even in non-binary systems, only a few additional mating types are typically present [Jam15]. For example, heterothallic populations of slime molds, including the social amoeba *Dictyostelium discoideum*, can contain 2-13 different mating types [BSI^{+}10, CH10]. A high diversity of mating types has been reported for certain mushroom-forming fungi: the global population of fairy inkcap *Coprinellus disseminatus* is estimated to have 123 mating types, but the number found in a single population would inevitably be considerably smaller [JSKV06]. There is an upper limit to the number of different mating types that can be maintained by a finite population – a number low enough for the strength of negative frequency-dependent selection to not be over-whelmed by genetic drift. Here, we set out to determine where this threshold lies.

We present a model of haploid self-incompatibility that allows estimation of the expected number of mating types in a population by comparing the deterministic and stochastic dynamics of mating type frequencies. While the evolutionary dynamics of haploid self-incompatibility have rarely been investigated in finite populations, gametophytic self-incompatibility has been subject to many theoretical investigations [Wri39, Fis58, Wri60, Wri64, Ewe64]. These models have been subject to considerable controversy for, among other things, the way they estimate the stationary distribution of alleles [Mor62] and for being difficult to analyze [MW09]. Recently, Constable and Kokko in [CK18] presented a rigorous analysis of haploid incompatibility using a Moran-based approach. While obtaining upper and lower bounds on the number of mating types their conclusions were restricted by similar problems associated with estimating the probability distribution encountered in Wright-Fisher models of gametophytic self-incompatibility. Our approach avoids this controversy altogether by eliminating the need to estimate the probability distribution. Instead, we follow the dynamics of a focal mating type and perform an order analysis of its dynamical structure under stationarity enabling estimation of the expected number of mating types by solving a single inequality. Furthermore, our Moran-based approach allows straightforward calculation of the probabilities of invasion by new types and the turnover of mating types.

## 2. Model

In order to quantify the effects of stochastic drift, mutation and negative frequency-dependence we consider a haploid population of *N* individuals in its ecological equilibrium, i.e. *N* is constant over time. Each individual is of a certain mating type, *M*_{1}, *…, M*_{R} for some positive integer *R*. The total number of individuals of type *M*_{i} is denoted by *X*_{i}. Individuals of type *M*_{i} can only mate with individuals of a different type *M*_{j} ≠ *M*_{i}. Hence, gametes are self-incompatible and can not reproduce asexually.

We implement changes in the population configuration by the Moran process, i.e. generations are overlapping. A transition in the population configuration consists of a birth and a death event in order to maintain a constant population size. The order of events does not affect the overall dynamics and can be exchanged. This can be seen by reordering the terms in the transition rates below. In the birth-step two randomly chosen individuals have the opportunity to mate with each other. If the two individuals have different mating types they give birth to an offspring which randomly inherits the mating type of one of its parents. In the death-step a third randomly chosen individual gets replaced by the newly born offspring thus leaving the population size unchanged. For an illustration of this mechanism see also Figure 1.

These dynamics translate into the following transition rates: the change from *k* to *k* + 1individuals of mating type *M*_{i} happens at rate

The single terms emerge as follows:

The factor 1/2 is due to the offspring inheriting its mating type uniformly at random from one of the two parents. In half of the cases it will inherit the type

*M*_{i}.The

*k*-term is the rate of drawing an individual of type*M*_{i}. The idea is that all of the*N*individuals have a random waiting time (exponential waiting time with rate 1) until they start the reproduction cycle which is independent of all the other individuals. Thus, having*k*type*M*_{i}individuals yields a rate*k*. This might seem to introduce a bias towards more frequent mating types but effectively does not since reproductive success is still coupled to finding a compatible mating partner.(

*N − k*)/(*N −*1) is the probability of drawing a non-*M*_{i}individual from the remaining*N −*1 individuals.The last fraction (

*N − k −*1)/(*N −*2) is the probability of a non-*M*_{i}individual dying and thus being replaced by the offspring. Note, that the parents are excluded from this set of individuals. This assures that there are always at least two mating types in the population.

Arguing analogously, we can write down the decrease rate, i.e. the rate to transition from *k* to *k −* 1 individuals of mating type *M*_{i}:

Again, we can disentangle the transition rate into single terms. The structure of the terms that are summed up is similar to the ones obtained for . Furthermore, the sums are interpreted as follows:

The first sum describes matings which happen between non-

*M*_{i}mating types resulting in a replacement of an*M*_{i}individual.The second sum is the rate at which a mating between a

*M*_{j}and a*M*_{i}individual causes a*M*_{j}offspring to replace another*M*_{i}individual.

Using these transition rates, and , and assuming large population sizes *N* we can derive a stochastic differential equation (also called diffusion approximation - see Appendix A) describing the dynamics of our model. Writing as the frequency of mating type *M*_{i} in the population, the stochastic differential equation reads as
where (*W*_{i})_{i=1,…,R} are independent standard Brownian motions. The functions *C*_{ij}(*x*) and *σ*_{i}(*x*) are polynomials whose exact form is not relevant for the following discussion. The above equation can be read as follows: the first part describes the deterministic dynamics of the system which were already studied in [IS87] and more recently adapted in the framework of asymmetric mating choices in [HP16]. The second term consisting of the sum over the non-*M*_{i} mating types describes the covariances which influence type-*M*_{i} individuals due to the restriction of a constant population size while the last term can be attributed to random births and deaths of type-*M*_{i} individuals, i.e. the dynamical development of the variance of *x*_{i}. These make up the stochastic fluctuations of the finite size population, commonly referred to as genetic drift.

It is worth noting that using a Wright-Fisher implementation of the process, i.e. non-overlapping generations, yields the same dynamics (under an adequate time-rescaling), see for instance [Eth12] on that issue. In the following we choose to use the Moran process since the analysis of invasion and turnover probabilities in Section 3.2 derives more naturally in this framework. The analysis of the mean number of mating types in Section 3.1 can be done analogously assuming Wright-Fisher dynamics.

In Figure 2 we show the temporal evolution of the system with four mating types in a population of (a) 100 and (b) 1000 individuals. The dashed lines represent the corresponding deterministic dynamics which all quickly collapse to the globally attractive equilibrium which is located at (1/4, 1/4, 1/4, 1/4), i.e. a uniform distribution of mating types. As can be seen, the stochastic trajectories remain closer to these lines in larger populations while smaller systems are more prone to random fluctuations. In subfigure (a) eventually, due to a random fluctuation, one of the mating types goes extinct at which point the internal equilibrium shifts to . For a detailed study of the deterministic system we refer to the “Mating kinetics I” model in [IS87].

### 2.1. Including mutations

So far, we have described the dynamics of a population in which no new mating types emerge. Hence, the number of mating types is decreasing over time due to stochastic extinction events as expected in a finite population. These dynamics are illustrated in Figure 3 (a). Eventually, the number of mating types will collapse to two. Now, we proceed by randomly introducing new mating types (mutations) into the population. More formally, at every replacement step a new mating type (or mutant) emerges with probability *u*. In a deterministically modeled setting this would gradually increase the number of mating types since the internal equilibrium at uniform frequencies is attractive. However, in a stochastic model newly arising mutants do not necessarily establish and even if they do the number of mating types is affected by random fluctuations some-times leading to an extinction of a mating type resulting in a turnover of types. These differences between the deterministic and stochastic model are depicted in Figure 3 (b) where the constant increase of mating types in the deterministic system is plotted against the stochastic evolution of the number of mating types. The equations describing the model with mutations are stated in Appendix A.1. Since in the following we restrict ourselves to the case with low mutation rates, for the analysis it is sufficient to consider the previously derived formulas.

## 3. Results

Having defined the model and derived its dynamics we now proceed making theoretical predictions. We start by inferring the expected number of mating types given a population of *N* individuals under low mutation rates. Furthermore, we compute the invasion probability of a newly emerging mating type and subsequently deduce the turnover probability of mating types, i.e. the probability for a resident mating type to get replaced by a newly emerging one.

### 3.1. Expected number of mating types

The maximum number of mating types that a population can support depends on the balance between the deterministic and stochastic dynamics. In a finite population, the number of mating types *R* cannot exceed the total number of individuals *N*. Negative frequency-dependent selection will drive the frequency of each mating type towards the mixed equilibrium where all types are at equal frequency 1*/R*. The global stability of this equilibrium not only attracts the trajectories but also makes it unlikely, though not impossible, for mating types to die out by stochastic fluctuations once they are established.

The likelihood that a mating type is lost through stochastic fluctuations depends on the number of individuals of that mating type in a population (Figure 3a); the stochastic dynamics are therefore most important in small populations (small *N*) with large numbers of mating types (large *R*). When a mating type is lost from a population through genetic drift, the representation of each remaining mating type will increase and the likelihood of losing another mating type will decrease. Eventually, the number of mating types becomes small enough such that the deterministic dynamics ensures high enough representation of each remaining mating type that the risk of extinction becomes negligible. We define this point, where the deterministic dynamics dominate the system, as the expected number of mating types.

New mating types can appear through mutation. In a deterministic model, this causes the number of mating types to increase over time (Figure 3b). However, in a finite population, adding a new mutant will increase the number of mating types in the population above the number where the risk of extinction is negligible, increasing the importance of stochastic fluctuations. Eventually, a mating type will be lost by extinction and the system will return to the expected number of mating types where the stochastic and deterministic dynamics are balanced. When mutants appear infrequently (small *u*), this balance is restored before another mutant appears and the mutation rate will have no effect on the expected number of mating types. However, if the mutation rate exceeds the extinction time then new mating types will accumulate in the population faster than they can be purged by stochastic fluctuations, and the expected number of mating types will increase.

We show that in a finite population of size N with a low mutation rate we would expect a maximum of *N* ^{1/3} different mating types (Figure 4). That is, despite the huge selective advantage enjoyed by rare mutants, a population of 1000 individuals could only support a maximum of 10 different mating types. We obtain this expected value by first calculating the highest order terms in the deterministic and stochastic parts of equation (2.3), and then determine the transition point where the deterministic dynamics are of higher order than the stochastic dynamics causing the population to stabilize.

Let us first consider the deterministic component of equation (2.3) in case *x*_{i} *>* 0, i.e.

We assume that the population is in its internal equilibrium with *R* mating types, i.e. at (1*/R,…,* 1*/R*). Writing *R* in terms of *N* we set *R* = *N*^{α} for *α >* 0 and thus, the frequency of a single mating type can be written as

Now the order of the deterministic component can be written as

Hence, the deterministic dynamics are of order *N*^{−2α}. For the stochastic counterpart of equation (2.3) we first explicitly state the functions *C*_{ij}(*x*) and *σ*_{i}(*x*) which are derived in the Appendix. They read as

Therefore, the order of the stochastic fluctuations is given by

Due to the additivity of independent Brownian motions we find for the order of the stochastic dynamics. Solving (3.1) > (3.2) which means that the deterministic dynamics dominate the system resulting in extinctions to be unlikely, gives

This states that in a population of *N* individuals with a sufficiently low mutation rate *u* we would expect at most *N* ^{1/3} different mating types given our modeling assumptions. This number may vary if we deviate from the neutral assumptions, i.e. no preference of mating partners or other potential mechanisms affecting the mating dynamics. Altering the dynamics causes the deterministic equilibrium to shift such that mating types have equilibrium values closer to zero. Hence, given any kind of non-symmetric selection pressure we would expect even lower numbers of mating types.

Furthermore, since our estimate relies on comparing the leading orders of the stochastic and deterministic component of the system, the result does not provide an exact value. Rather it should be taken as an order of magnitude where the dynamics of the number of mating types are balanced. This is one of the reasons for the simulation results not lying exactly on our prediction (Figure 4a). Additionally, we expect our result to underestimate the actual number of mating types obtained from simulations, at least in large populations. Since we assume low mutation rates and by that neglect the impact of newly arising mating types on the dynamics altogether, we do not account for influx of new mating types. This implies that our prediction only holds in situations in which the system has enough time to balance before new mutations arise. Therefore, especially in large populations where extinction times of mating types are large, our result under-estimates the actually observed number of mating types in the simulations. A possibility to overcome this problem is the estimation of stationary distributions, a method used in previous studies of related systems [Wri39, MW09, CK18]. Still, approximations are necessary in order to make analytical progress and even then a rigorous solution of the quantity of interest is not always accessible. For instance in [CK18] where the same system as here is studied, the authors obtain a upper and lower bound on the expected number of mating types in the case of low mutation rates - the dashed lines in Figure 4.

In conclusion, the simplicity of our method has the advantage of obtaining an explicit prediction of the expected number of mating types while coming at the cost of being restricted to low mutation rates and lacking a rigorous prediction for the mutation-selection balance.

### 3.2. Invasion of new mating types

New mutants, as the sole representatives of their mating type, are subject to both the strongest selection and the greatest risk of being lost by drift. So what is the probability that a novel mating type becomes established in a population? If the rare mutant can avoid immediate extinction, negative frequency-dependence will drive it to higher and higher frequencies, and the risk of extinction will decrease until the mutant reaches a high enough frequency that its survival is guaranteed by the deterministic dynamics (Figure 5a). A small increase in frequency causes a large drop in the risk of extinction but has little effect on the likelihood of encountering an incompatible mate. Even in relatively small populations the risk of extinction will become negligible long before the frequency of the mutant is high enough that the risk of encountering an incompatible mate becomes problematic; the selective advantage at these low frequencies (i.e. below the stochasticity threshold) remains essentially constant.

Computing the establishment probability therefore means calculating the survival probability of the rare mating type at frequencies small enough that the risk of encountering an incompatible mate can be ignored. Given *R* different mating types in the population, we find that the probability of establishment of the (*R* + 1)-th type is (the detailed derivation is given in Appendix B)

The inverse relationship between the establishment probability *φ* and the number of resident mating types can be explained by the higher turnover rates of populations with many mating types, i.e. basically the generation time. Below the stochasticity threshold, all partners encountered by the mutant will be compatible regardless of the number of mating types present in the population. However, the mating success of the resident types is affected by the number of types present. In case of few residential mating types, many pairings will be incompatible, and the turnover rate of the population will be low. Thus, when the number of resident types is low, the extinction risk of the mutant is small. Contrarily, when the number of resident mating types is high, the population turnover rate increases as does the extinction risk of the mutant.

Figure 5b shows a comparison of equation (3.4) with stochastic simulations in a population with *N* = 250 individuals. The accuracy of our prediction is limited to the region where deterministic dynamics dominate the system. Otherwise the intuition that frequency-dependent selection stabilizes the rare mutant fails and no analytical approximation is available.

### 3.3. The turnover probability of mating types

We have shown that finite populations will eventually reach a relatively stable number of mating types. However, the stability of the number of mating types does not imply evolutionary stasis. Resident mating types will be continuously replaced by new mating types in a process referred to as the “turnover of the sexes” (Figure 6; see also [IS87]).

Computation of the probability for such a turnover to happen is straightforward in our setting. Since it can be seen as a two step process, invasion of a mutant type followed by extinction of a resident mating type, the probability is given by

The invasion probability as calculated in the previous section is 1*/R* while the extinction probability of a resident type is given by *R/*(*R*+1) (all mating types behave equally).

An even more interesting quantity to study is the turnover rate, i.e. the approximate time until a resident mating type gets replaced by a new one. Again, the rate at which such a turnover event happens can be split into invasion rate of a new mating type and the extinction rate of a resident mating type. As we noted in Section 3.2, the invasion process needs to be very quick in order to reduce the extinction risk of the new type such that its rate is approximately given by the product of mutation rate and invasion probability. The extinction rate on the other hand is determined by the extinction time for a mating type out of the stable equilibrium. Unfortunately our methodology is insufficient to provide a reasonable estimate of the extinction time. For progress with respect to this rate an estimation of the stationary distribution over the mating type space, similar to the result derived in [CK18], seems to be necessary.

As demonstrated in Figure 5b our estimate of the turnover probability fits the data points derived from stochastic simulations. For populations far below the maximal number of mating types (*R* = 2, 3) the population is too stable such that effectively no turnover takes place. If anything, the new mating type establishes and by that increases the overall number of mating types in the population. This explains the missing data points for *R* = 2, 3.

## 4. Discussion

Even though rare mating types have a significant selective advantage in a sexually reproducing organism with self-incompatible gametes, most species have a very small number of mating types - typically only two [Jam15]. Here, we present an analytical explanation for the discrepancy between the low number of types observed in natural populations and the deterministic prediction. By identifying the point at which the deterministic dynamics – which maintain all types at equal frequency – are balanced by the stochastic dynamics – which can cause rare types to be lost from the population, we show that the maximum number of mating types that can be supported by a finite population is of order *N* ^{1/3}. Accounting for both, deterministic negative frequency-dependence and genetic drift, reveals that the expected number of mating types in real populations is far below the naïve deterministic expectation [BLVD^{+}11, CK18].

At low mutation rates, the number of mating types in a population will be stable. How-ever, this should not be confused with evolutionary stasis. As originally suggested by Iwasa and Sazaki [IS87], the identity of mating types present in a population changes over time. As a first step towards an analytical solution for the rate of turnover, we derive the probability of invasion of a rare mating type. Together with the mutation rate this gives the establishment rate of new mating types which holds as long as the number of mating types is below the previously estimated maximum. Subsequently, for a turnover event to complete a residential mating type needs to die out. Unfortunately, our analysis does not allow estimation of the extinction time of that event. Hence, we are limited to the probability of a turnover event. A analytical prediction for the actual rate of these events to happen is so far lacking and remains an open subject for future investigations.

Our model only accounts for the competing influences of mutation, negative frequency dependent selection, and drift. The number of mating types supported by the balance of these forces should therefore be considered a maximum. Many other factors, not included in our model, could drive the number of types in a population below this limit. Chief among these factors is that our model assumes that newly arising mating types are fully compatible with all resident types. This could be unrealistic if resident types have co-evolved strong interactions. New mutants suffering reduced compatibility will be less likely to invade, limiting the number of types in a population to be below our calculated maximum [Pow76,HP16]. Indeed, in systems where compatibility is determined by highly specific ligand receptor interactions, it seems unlikely that a random mutant would not experience reduced mating efficiency [Hoe87]. Nevertheless, pheromone-receptor mate recognition systems involving multiple ligands do exist. The ciliate *Euplotes raikovi* can at least partially distinguish between self-pheromones and up to eight different non-self pheromones [LPAV16]. Nieuwenhuis et al [NBV^{+}13] have suggested that this problem might be reduced in tetrapolar systems where pheromone-receptor compatibility is uncoupled from heterodimerizing homeodomain proteins, allowing both loci to evolve independently. Indeed, multiallelic haploid incompatibility found in certain bipolar fungi may have evolved in their tetrapolar ancestors [JSKV06]. Of course, the number of possible combinations of compatible homeodomain proteins is almost certainly limited as well [Per12]. Other problems that might limit the number of mating types are reviewed by [BLVD^{+}11]. There are therefore many reasons to suspect that the number of mating types in a population will fall below the maximum allowed by the balance of selection and drift; these constraints might often restrict the number of mating types to two, despite the potentially high costs associated with encountering incompatible mates.

Investigations of the evolution of mutliallelic loci subject to negative frequency dependent selection have long been mired by controversy over the shape of the probability distribution and the often intractably complex nature of previous models. Our analysis avoids the need to assume any distribution, provides a simple analytical solution for the maximum number of alleles in a finite population, and allows computation of the invasion and turnover probabilities of newly arising mating types. This approach could be extended to the investigation of alleles conferring differential fitness, allowing the study of more complex mating systems, meiotic drive, or other multiallelic systems subject to balancing selection including certain gamete recognition proteins [TL10].

## A. Derivation of equation (2.3)

In this section we derive the stochastic differential equation (2.3) describing the dynamics of a focal mating type. For that reason, we will explicitly calculate the infinitesimal change in the expectation, the variance and covariances of the process which translate to the deterministic drift and stochastic diffusion term of the stochastic differential equation. For a more detailed description of these quantities and their relation we refer to [Eth12, Chapter 3].

Let us recall the transition rates for type *M*_{i} individuals. We denote the absolute number of *M*_{i} individuals by *X*_{i} and the corresponding frequency by *x*_{i} = *X*_{i}*/N*. To go from *k* type *M*_{i} individuals to *k* + 1 individuals the rate reads
whereas the rate of decrease is given by

This yields the infinitesimal expected change, i.e. the change in a small time interval *δt*, in relative numbers *x*_{i}
where we have set *R* as the number of present mating types.

Next, we proceed deriving the change in variance in a small time. Neglecting terms of order *O*(*δt*^{2}) we find (for Δ*x*_{i}(*t* + *δt*) = *x*_{i}(*t* + *δt*) *− x*_{i}(*t*))

Lastly, for the covariances we find (again neglecting terms of order (*δt*)^{2})

Collecting all the terms we find the following stochastic diffusion process describing the development of a focal mating type:
where *C*_{ij}(*x*) = *x*_{i}*x*_{j}(2*−x*_{i}*−x*_{j}) and are the variables used in the main text.

### A.1. Including mutations

In a next step we include mutations, the mechanism which generates new mating types. We implement this during the reproduction step meaning that there is a probability that after a successful mating the offspring expresses a completely new mating type. More precisely, we also exclude mutations to already existing mating types, thus assuming a variant of the infinitely many alleles model. Therefore, the increase rate of a focal mating type *M*_{i} transforms to

Additionally, we have an increase rate due to mutations if a mating type *M*_{i} is not present in the population, i.e.

Finally, the decrease rate now has an additional mutation term such that it can be written as

Following the same arguments as in the previous section we again calculate the infinitesimal expected change and variance in the relative frequency of a mating type *M*_{i}. For the change in expectation we find

Similar calculations carried out for the variance yield

It remains to state the exact form of the covariances in the model including mutations. We obtain

Putting together the infinitesimal changes in the expectation, variance and covariance we obtain
where (*W*_{i})_{i=1,…,R} are independent one-dimensional Brownian motions.

## B. Invasion Probability

In order to calculate the probability for a new mating type to establish itself in the population we employ techniques from branching process theory in a constant environment. This is reasonable since we only need to compute the probability for the new mating type to reach a certain threshold frequency from where deterministic dynamics take over and pull the frequency up to the stable internal equilibrium. However, this is only reasonable in the regime in which the deterministic dynamics dominate the stochastic fluctuations, i.e. when the number of mating types is smaller than *N* ^{1/3} - see Section 3.1.

Hence, let us assume that the number of mating types in the population is *R < N* ^{1/3}. A new mating type arises and we can compute its birth and death rates under the assumption that the other *K* mating types are close to their equilibrium frequencies, i.e. . Then we find
since we assume *N* to be large and *k*, the number of individuals having the new mating type, to be negligibly small, i.e. . This is equivalent to the assumption that the individuals of the novel mating type do evolve independently of each other, the basic assumption of a branching process. For the death rate of individuals expressing the new mating type we have

Note, that this approximation also holds if there is more than one rare mutant mating type in the population. One of those new mating types will eventually reach a critical threshold from where deterministic dynamics force it to the new internal equilibrium . However, a rigorous analysis of this situation is beyond the scope of this work. Hence, we restrict ourselves to low mutation rates which allow us to work in a situation where we only have one invading mating type at a time. Then the above transition rates hold until the new mating type hits a low threshold value from where on its survival is guaranteed due to the deterministic dynamics, see also Figure 5a in the main text. To calculate the hitting probability of this (arbitrarily set) threshold we use branching process theory, i.e. we calculate the survival probability of this auxiliary process. Since our system allows for a stable fixed point in the deterministic system the new mating type will, after hitting the threshold, increase the number of mating types.

Let us now proceed computing this invasion or establishment probability which we call *φ*_{R}. As we will see, it is solely dependent on the present number of mating types *R*. Noting, that the number of individuals having the invading mating type is a continuous-time birth-death process we can apply known theorems for computing the survival probability of it, see for instance [All11, Theorem 6.2]. Thus, the extinction probability for a birth-death process with one initial individual is given by the formula

Inserting our rates from above and applying the formula for the geometric series this yields

## C. Simulations

The data used for the estimates in Figure 4 were simulated in the following way. One simulation starting either with 3, 10 or 20 mating types in the corresponding deterministic equilibrium was first given 10, 000 generations to equilibrate. Then we sampled the number of mating types every generation, i.e. after *N* transitions, for in total 1, 000 generations. Each initial condition was run for 50 independent runs such that each symbol is an average of 50, 000 data points.