PT  - JOURNAL ARTICLE
AU  - Ariel Goldstein
AU  - Eric Ham
AU  - Samuel A. Nastase
AU  - Zaid Zada
AU  - Avigail Grinstein-Dabus
AU  - Bobbi Aubrey
AU  - Mariano Schain
AU  - Harshvardhan Gazula
AU  - Amir Feder
AU  - Werner Doyle
AU  - Sasha Devore
AU  - Patricia Dugan
AU  - Daniel Friedman
AU  - Michael Brenner
AU  - Avinatan Hassidim
AU  - Orrin Devinsky
AU  - Adeen Flinker
AU  - Omer Levy
AU  - Uri Hasson
TI  - Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain
AID  - 10.1101/2022.07.11.499562
DP  - 2023 Jan 01
TA  - bioRxiv
PG  - 2022.07.11.499562
4099  - http://biorxiv.org/content/early/2023/02/21/2022.07.11.499562.short
4100  - http://biorxiv.org/content/early/2023/02/21/2022.07.11.499562.full
AB  - Deep language models (DLMs) provide a novel computational paradigm for how the brain processes natural language. Unlike symbolic, rule-based models described in psycholinguistics, DLMs encode words and their context as continuous numerical vectors. These “embeddings” are constructed by a sequence of computations organized in “layers” to ultimately capture surprisingly sophisticated representations of linguistic structures. How does this layered hierarchy map onto the human brain during natural language comprehension? In this study, we used electrocorticography (ECoG) to record neural activity in language areas along the superior temporal gyrus and inferior frontal gyrus while human participants listened to a 30-minute spoken narrative. We supplied this same narrative to a high-performing DLM (GPT2-XL) and extracted the contextual embeddings for each word in the story across all 48 layers of the model. We next trained a set of linear encoding models to predict the temporally-evolving neural activity from the embeddings at each layer. We found a striking correspondence between the layer-by-layer sequence of embeddings from GPT2-XL and the temporal sequence of neural activity in language areas. In addition, we found evidence for the gradual accumulation of recurrent information along the linguistic processing hierarchy. However, we also noticed additional neural processes in the brain, but not in DLMs, during the processing of surprising (unpredictable) words. These findings point to a connection between human language processing and DLMs where the layer-by-layer accumulation of contextual information in DLM embeddings matches the temporal dynamics of neural activity in high-order language areas.Competing Interest StatementThe authors have declared no competing interest.