Rapid diversification is common among herbivorous insects and is often the result of host shifts, leading to the exploitation of novel food sources. This, in turn, is associated with adaptive evolution of female oviposition behavior and larval feeding biology. Although natural selection is the typical driver of such adaptation, the role of sexual selection is less clear. In theory, sexual selection can either accelerate or impede adaptation. To assess the independent effects of natural and sexual selection on the rate of adaptation, we performed a laboratory natural selection experiment in a herbivorous bruchid beetle (Callosobruchus maculatus). We established replicated selection lines where we varied natural (food type) and sexual (mating system) selection in a 2 x 2 orthogonal design, and propagated our lines for 35 generations. In half of the lines, we induced a host shift whereas the other half was kept on the ancestral host. We experimentally enforced monogamy in half of the lines, whereas the other half remained polygamous. The beetles rapidly adapted to the novel host, which primarily involved increased host acceptance by females and an accelerated rate of larval development. We also found that our mating system treatment affected the rate of adaptation, but that this effect was contingent upon food type. As beetles adapted to the novel host, sexual selection reinforced natural selection whereas populations residing close to their adaptive peak (i.e., those using their ancestral host) exhibited higher fitness in the absence of sexual selection. We discuss our findings in light of current sexual selection theory and suggest that the net evolutionary effect of reproductive competition may critically depend on natural selection. Sexual selection may commonly accelerate adaptation under directional natural selection whereas sexual selection, and the associated load brought by sexual conflict, may tend to depress population fitness under stabilizing natural selection.