RT Journal Article
SR Electronic
T1 Thinking ahead: prediction in context as a keystone of language in humans and machines
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2020.12.02.403477
DO 10.1101/2020.12.02.403477
A1 Ariel Goldstein
A1 Zaid Zada
A1 Eliav Buchnik
A1 Mariano Schain
A1 Amy Price
A1 Bobbi Aubrey
A1 Samuel A. Nastase
A1 Amir Feder
A1 Dotan Emanuel
A1 Alon Cohen
A1 Aren Jansen
A1 Harshvardhan Gazula
A1 Gina Choe
A1 Aditi Rao
A1 Catherine Kim
A1 Colton Casto
A1 Fanda Lora
A1 Adeen Flinker
A1 Sasha Devore
A1 Werner Doyle
A1 Daniel Friedman
A1 Patricia Dugan
A1 Avinatan Hassidim
A1 Michael Brenner
A1 Yossi Matias
A1 Kenneth A. Norman
A1 Orrin Devinsky
A1 Uri Hasson
YR 2020
UL http://biorxiv.org/content/early/2020/12/03/2020.12.02.403477.abstract
AB Departing from classical rule-based linguistic models, advances in deep learning have led to the development of a new family of self-supervised deep language models (DLMs). These models are trained using a simple self-supervised autoregressive objective, which aims to predict the next word in the context of preceding words in real-life corpora. After training, autoregressive DLMs are able to generate new “context-aware” sentences with appropriate syntax and convincing semantics and pragmatics. Here we provide empirical evidence for the deep connection between autoregressive DLMs and the human language faculty using a 30-min spoken narrative and electrocorticographic (ECoG) recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Next, we leverage DLM embeddings to demonstrate that many electrodes spontaneously predict the meaning of upcoming words, even hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that deep language models provide an important step toward creating a biologically feasible computational framework for generative language.Competing Interest StatementThe authors have declared no competing interest.