Horizontal sound localization relies on the extraction of binaural acoustic cues by integration of the signals from the two ears at the level of the brainstem. The present experiment was aimed at detecting the sites of binaural integration in the human brainstem using functional magnetic resonance imaging and a binaural difference paradigm, in which the responses to binaural sounds were compared with the sum of the responses to the corresponding monaural sounds. The experiment also included a moving sound condition, which was contrasted against a spectrally and energetically matched stationary sound condition to assess which of the structures that are involved in general binaural processing are specifically specialized in motion processing. The binaural difference contrast revealed a substantial binaural response suppression in the inferior colliculus in the midbrain, the medial geniculate body in the thalamus and the primary auditory cortex. The effect appears to reflect an actual reduction of the underlying activity, probably brought about by binaural inhibition or refractoriness at the level of the superior olivary complex. Whereas all structures up to and including the primary auditory cortex were activated as strongly by the stationary as by the moving sounds, non-primary auditory fields in the planum temporale responded selectively to the moving sounds. These results suggest a hierarchical organization of auditory spatial processing in which the general analysis of binaural information begins as early as the brainstem, while the representation of dynamic binaural cues relies on non-primary auditory fields in the planum temporale.