PT  - JOURNAL ARTICLE
AU  - Caucheteux, Charlotte
AU  - Gramfort, Alexandre
AU  - King, Jean-Rémi
TI  - GPT-2’s activations predict the degree of semantic comprehension in the human brain
AID  - 10.1101/2021.04.20.440622
DP  - 2021 Jan 01
TA  - bioRxiv
PG  - 2021.04.20.440622
4099  - http://biorxiv.org/content/early/2021/09/02/2021.04.20.440622.short
4100  - http://biorxiv.org/content/early/2021/09/02/2021.04.20.440622.full
AB  - Language transformers, like GPT-2, have demonstrated remarkable abilities to process text, and now constitute the backbone of deep translation, summarization and dialogue algorithms. However, whether these models encode information that relates to human comprehension remains controversial. Here, we show that the representations of GPT-2 not only map onto the brain responses to spoken stories, but also predict the extent to which subjects understand narratives. To this end, we analyze 101 subjects recorded with functional Magnetic Resonance Imaging while listening to 70 min of short stories. We then fit a linear model to predict brain activity from GPT-2’s activations, and correlate this mapping with subjects’ comprehension scores as assessed for each story. The results show that GPT-2’s brain predictions significantly correlate with semantic comprehension. These effects are bilaterally distributed in the language network and peak with a correlation of R=0.50 in the angular gyrus. Overall, this study paves the way to model narrative comprehension in the brain through the lens of modern language algorithms.Competing Interest StatementThe authors have declared no competing interest.