The nature of epistasis has important consequences for the evolutionary significance of sex and recombination. Recent efforts to find negative epistasis as a source of negative linkage disequilibrium and associated long-term advantage to sex have yielded little support. Sign epistasis, where the sign of the fitness effects of alleles varies across genetic backgrounds, is responsible for the ruggedness of the fitness landscape, with several unexplored implications for the evolution of sex. Here, we describe fitness landscapes for two sets of strains of the asexual fungus Aspergillus niger involving all combinations of five mutations. We find that approximately 30% of the single-mutation fitness effects are positive despite their negative effect in the wild-type strain and that several local fitness maxima and minima are present. We then compare adaptation of sexual and asexual populations on these empirical fitness landscapes by using simulations. The results show a general disadvantage of sex on these rugged landscapes, caused by the breakdown by recombination of genotypes on fitness peaks. Sex facilitates movement to the global peak only for some parameter values on one landscape, indicating its dependence on the landscape's topography. We discuss possible reasons for the discrepancy between our results and the reports of faster adaptation of sexual populations.