The canonical description of the role of the medial temporal lobes (MTLs) in memory is that short-term forms of memory (e.g., working memory [WM]) are spared when the MTL is damaged, but longer term forms of memory are impaired. Tests used to assess this have typically had a heavy verbal component, potentially allowing explicit rehearsal strategies to maintain the WM trace over the memory delay period. Here we test the hypothesis that the MTL is necessary for visual WM when verbal rehearsal strategies are difficult to implement. In three patients with MTL damage we found impairments in spatial, face, and color WM, at delays as short as 4 sec. Impaired memory could not be attributed to memory load or perceptual problems. These findings suggest that the MTLs are critical for accurate visual WM.