Sentences are easier to memorize than ungrammatical word strings, a phenomenon known as the sentence superiority effect. Yet, it is unclear how higher-order linguistic information facilitates verbal working memory and how this is implemented in the neural system. The goal of the current fMRI study was to specify the brain mechanisms underlying the sentence superiority effect during encoding and during maintenance in working memory by manipulating syntactic structure and working memory load. The encoding of sentence material, as compared with the encoding of ungrammatical word strings, recruited not only inferior frontal (BA 47) and anterior temporal language-related areas but also the medial-temporal lobe, which is not classically reported for language tasks. During maintenance, it was sentence structure as contrasted with ungrammatical word strings that led to activation decrease in Broca's area, SMA, and parietal regions. Furthermore, in Broca's area, an interaction effect revealed a load effect for ungrammatical word strings but not for sentences. The sentence superiority effect, thus, is neurally reflected in a twofold pattern, consisting of increased activation in classical language as well as memory areas during the encoding phase and decreased maintenance-related activation. This pattern reflects how chunking, based on sentential syntactic and semantic information, alleviates rehearsal demands and thus leads to improved working memory performance.