Chromosomal rearrangements are mutations contributing to both within and between species variation; however their contribution to fitness is yet to be measured. Here we show that chromosomal rearrangements are pervasive in natural isolates of Schizosaccharomyces pombe and contribute to reproductive isolation. To determine the fitness effects of chromosome structure, we constructed two inversions and eight translocations without changing the coding sequence. We show that chromosomal rearrangements contribute to both reproductive success in meiosis and growth rate in mitosis with a strong genotype by environment interaction. These changes are accompanied by alterations in gene expression. Strikingly, we find several examples leading to antagonistic pleiotropy. Even though chromosomal rearrangements may have a deleterious effect during sexual reproduction, some compensate with a strong growth advantage in mitosis. Our results constitute the first quantification of fitness effects caused by de novo mutations that result in chromosomal rearrangement variation and suggest a mechanism for their maintenance in natural populations.