Evolutionary success of bacteria relies on the constant fine-tuning of their mutation rates, which optimizes their adaptability to constantly changing environmental conditions. When adaptation is limited by the mutation supply rate, under some conditions, natural selection favours increased mutation rates by acting on allelic variation of the genetic systems that control fidelity of DNA replication and repair. Mutator alleles are carried to high frequency through hitchhiking with the adaptive mutations they generate. However, when fitness gain no longer counterbalances the fitness loss due to continuous generation of deleterious mutations, natural selection favours reduction of mutation rates. Selection and counter-selection of high mutation rates depends on many factors: the number of mutations required for adaptation, the strength of mutator alleles, bacterial population size, competition with other strains, migration, and spatial and temporal environmental heterogeneity. Such modulations of mutation rates may also play a role in the evolution of antibiotic resistance.