It has been argued that the architecture of the genotype-phenotype map determines evolvability, but few studies have attempted to quantify these effects. In this article we use the multilinear epistatic model to study the effects of different forms of epistasis on the response to directional selection. We derive an analytical prediction for the change in the additive genetic variance, and use individual-based simulations to understand the dynamics of evolvability and the evolution of genetic architecture. This shows that the major determinant for the evolution of the additive variance, and thus the evolvability, is directional epistasis. Positive directional epistasis leads to an acceleration of evolvability, while negative directional epistasis leads to canalization. In contrast, pure non-directional epistasis has little effect on the response to selection. One consequence of this is that the classical epistatic variance components, which do not distinguish directional and non-directional effects, are useless as predictors of evolutionary dynamics. The build-up of linkage disequilibrium also has negligible effects. We argue that directional epistasis is likely to have major effects on evolutionary dynamics and should be the focus of empirical studies of epistasis.