Abstract
A classic problem in population genetics is the characterization of discrete population structure in the presence of continuous patterns of genetic differentiation. Especially when sampling is discontinuous, the use of clustering or assignment methods may incorrectly ascribe differentiation due to continuous processes (e.g., geographic isolation by distance) to discrete processes, such as geographic, ecological, or reproductive barriers between populations. This reflects a shortcoming of current methods for inferring and visualizing population structure when applied to genetic data deriving from geographically distributed populations. Here, we present a statistical framework for the simultaneous inference of continuous and discrete patterns of population structure. The method estimates ancestry proportions for each sample from a set of two-dimensional population layers, and, within each layer, estimates a rate at which relatedness decays with distance. This thereby explicitly addresses the “clines versus clusters” problem in modeling population genetic variation. The method produces useful descriptions of structure in genetic relatedness in situations where separated, geographically distributed populations interact, as after a range expansion or secondary contact. We demonstrate the utility of this approach using simulations and by applying it to empirical datasets of poplars and black bears in North America.
Author summary One of the first steps in the analysis of genetic data, and a principal mission of biology, is to describe and categorize natural variation. A continuous pattern of differentiation (isolation by distance), where individuals found closer together in space are, on average, more genetically similar than individuals sampled farther apart, can confound attempts to categorize natural variation into groups. This is because current statistical methods for assigning individuals to discrete clusters cannot accommodate spatial patterns, and so are forced to use clusters to describe what is in fact continuous variation. As isolation by distance is common in nature, this is a substantial shortcoming of existing methods. In this study, we introduce a new statistical method for categorizing natural genetic variation - one that describes variation as a combination of continuous and discrete patterns. We demonstrate that this method works well and can capture patterns in population genomic data without resorting to splitting populations where they can be described by continuous patterns of variation.
