Theory predicts widespread loss of genetic diversity from drift and inbreeding in trees subjected to habitat fragmentation, yet empirical support of this theory is scarce. We argue that population genetics theory may be misapplied in light of ecological realities that, when recognized, require scrutiny of underlying evolutionary assumptions. One ecological reality is that fragment boundaries often do not represent boundaries for mating populations of trees that benefit from long-distance pollination, sometimes abetted by long-distance seed dispersal. Where fragments do not delineate populations, genetic theory of small populations does not apply. Even in spatially isolated populations, where genetic theory may eventually apply, evolutionary arguments assume that samples from fragmented populations represent trees that have had sufficient time to experience drift, inbreeding, and ultimately inbreeding depression, an unwarranted assumption where stands in fragments are living relicts of largely unrelated predisturbance populations. Genetic degradation may not be as important as ecological degradation for many decades following habitat fragmentation.