ABSTRACT
Inbreeding depression resulting from partially recessive deleterious alleles is thought to be the main genetic factor preventing self-fertilizing mutants from spreading in outcrossing hermaphroditic populations. However, deleterious alleles may also generate an advantage to selfers in terms of more efficient purging, while the effects of epistasis among those alleles on inbreeding depression and mating system evolution remain little explored. In this paper, we use a general model of selection to disentangle the effects of different forms of epistasis (additive-by-additive, additive-by-dominance and dominance-by-dominance) on inbreeding depression and on the strength of selection for selfing. Models with fixed epistasis across loci, and models of stabilizing selection acting on quantitative traits (generating distributions of epistasis) are considered as special cases. Besides its effects on inbreeding depression, epistasis may increase the purging advantage associated with selfing (when it is negative on average), while the variance in epistasis favors selfing through the generation of linkage disequilibria that increase mean fitness. Approximations for the strengths of these effects are derived, and compared with individual-based simulation results.
