An experiment was performed that measured, for the frontal audio-visual horizon, how accurately listeners could localize a female-voice target amidst four spatially distributed male-voice maskers. To examine whether listeners can make use of a priori knowledge about the configuration of the sources, performance was examined in two conditions: either the masker locations were fixed (in one of five known patterns) or the locations varied from trial to trial. The presence of maskers disrupted speech localization, even after accounting for reduced target detectability. Averaged across all target locations, the rms error in responses decreased by 20% when a priori knowledge about masker locations was available. The effect was even stronger for the target locations that did not coincide with the maskers (error reduction of 36%), while no change in errors was observed for targets coinciding with maskers. The benefits were reduced when the target-to-masker intensity ratio was increased or when the maskers were in a pattern that made it difficult to make use of the a priori information. The results confirm that localization in speech mixtures is modified by the listener's expectations about the spatial arrangement of the sources.