Abstract
We give recursions for the expected site-frequency spectrum associated with Xi-coalescents, that is exchangeable coalescents which admit simultaneous multiple mergers of ancestral lineages. Xi-coalescents arise, for example, in association with population models of skewed offspring distributions with diploidv, recurrent advantageous mutations, or strong bottlenecks. In contrast, Lambda-coalescents admit multiple mergers of lineages, but at most one such merger each time. Xi-coalescents, as well as Lambda-coalescents, can predict an excess of singletons, compared to the Kingman coalescent. We compare estimates of coalescent parameters when Xi-coalescent models are applied to data obtained from Lambda-coalescents, and vice versa. In general, Xi-coalescents predict fewer singletons than corresponding Lambda-coalescents, but higher count of mutations of ‘size’ larger than singletons. We analyse unfolded site-frequency spectra obtained for nuclear loci of the diploid Atlantic cod, and obtain different coalescent parameter estimates than previously obtained with Lambda-coalescents. Our results provide new inference tools, and suggest that for nuclear population genetic data from diploid or polyploid highly fecund populations who may have skewed offspring distributions, one should not apply Lambda-coalescents, but Xi-coalescents.
