It is not known whether natural selection can encounter any given phenotype that can be produced by genetic variation. There has been a long-lasting debate about the processes that limit adaptation and, consequently, about how well adapted phenotypes are. Here we examine how development may affect adaptation, by decomposing the genotype-fitness map-the association between each genotype and its fitness-into two: one mapping genotype to phenotype by means of a computational model of organ development, and one mapping phenotype to fitness. In the map of phenotype and fitness, the fitness of each individual is based on the similarity between realized morphology and optimal morphology. We use three different simulations to map phenotype to fitness, and these differ in the way in which similarity is calculated: similarity is calculated for each trait (in terms of each cell position individually), for a large or a small number of phenotypic landmarks (the 'many-traits' and 'few-traits' phenotype-fitness maps), and by measuring the overall surface roughness of morphology (the 'roughness' phenotype-fitness map). Evolution is simulated by applying the genotype-phenotype map and one phenotype-fitness map to each individual in the population, as well as random mutation and drift. We show that the complexity of the genotype-phenotype map prevents substantial adaptation in some of the phenotype-fitness maps: sustained adaptation is only possible using 'roughness' or 'few-traits' phenotype-fitness maps. The results contribute developmental understanding to the long-standing question of which aspects of phenotype can be effectively optimized by natural selection.