Directing attention to a behaviorally relevant visual stimulus can overcome the distracting effects of other nearby stimuli. Correspondingly, physiological studies indicate that attention serves to filter distracting stimuli from receptive fields (RFs) in several extrastriate areas. Moreover, a recent study demonstrated that lesions of extrastriate areas V4 and TEO produce impairments in attentional filtering. A critical remaining question concerns why lesions of ventral stream areas cause attentional filtering impairments. To address this question, we tested the effects of restricted area V4 and TEO lesions on both behavioral performance and the responses of downstream neurons in area TE. The lesions impaired behavioral discrimination thresholds and altered neuronal selectivity for target stimuli in the presence of distracters. With attention to the target, but in the absence of V4 and/or TEO inputs, TE neurons responded as though attentional inputs could no longer be used to filter distracters from their RFs. This presumably occurred because top-down attentional signals were no longer able to filter distracters from the RFs of the cells that provide TE with major input. Consistent with this interpretation, increasing the spatial separation between targets and distracters, such that they no longer fell within a typical V4 RF dimension, restored both behavioral performance and neuronal selectivity in the portion of TE RFs affected by the V4 lesion.