Category and letter fluency tasks are commonly used clinically to investigate the semantic and phonological processes central to speech production, but the neural correlates of these processes are difficult to establish with functional neuroimaging because of the relatively unconstrained nature of the tasks. This study investigated whether differential performance on semantic (category) and phonemic (letter) fluency in neurologically normal participants was reflected in regional gray matter density. The participants were 59 highly proficient speakers of 2 languages. Our findings corroborate the importance of the left inferior temporal cortex in semantic relative to phonemic fluency and show this effect to be the same in a first language (L1) and second language (L2). Additionally, we show that the pre-supplementary motor area (pre-SMA) and head of caudate bilaterally are associated with phonemic more than semantic fluency, and this effect is stronger for L2 than L1 in the caudate nuclei. To further validate these structural results, we reanalyzed previously reported functional data and found that pre-SMA and left caudate activation was higher for phonemic than semantic fluency. On the basis of our findings, we also predict that lesions to the pre-SMA and caudate nuclei may have a greater impact on phonemic than semantic fluency, particularly in L2 speakers.