Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Thinking ahead: prediction in context as a keystone of language in humans and machines

Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, View ORCID ProfileSamuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, View ORCID ProfileFanda Lora, Adeen Flinker, Sasha Devore, Werner Doyle, Daniel Friedman, Patricia Dugan, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, Uri Hasson
doi: https://doi.org/10.1101/2020.12.02.403477
Ariel Goldstein
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: ariel.y.goldstein@gmail.com
Zaid Zada
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eliav Buchnik
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mariano Schain
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Amy Price
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bobbi Aubrey
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Samuel A. Nastase
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Samuel A. Nastase
Amir Feder
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Dotan Emanuel
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alon Cohen
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aren Jansen
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Harshvardhan Gazula
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Gina Choe
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aditi Rao
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Catherine Kim
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Colton Casto
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Fanda Lora
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Fanda Lora
Adeen Flinker
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sasha Devore
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Werner Doyle
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel Friedman
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Patricia Dugan
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Avinatan Hassidim
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Brenner
2Google Research
4School of Engineering and Applied Science, Harvard University, Boston, MA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yossi Matias
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kenneth A. Norman
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Orrin Devinsky
3New York University School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Uri Hasson
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

Departing from classical rule-based linguistic models, advances in deep learning have led to the development of a new family of self-supervised deep language models (DLMs). These models are trained using a simple self-supervised autoregressive objective, which aims to predict the next word in the context of preceding words in real-life corpora. After training, autoregressive DLMs are able to generate new “context-aware” sentences with appropriate syntax and convincing semantics and pragmatics. Here we provide empirical evidence for the deep connection between autoregressive DLMs and the human language faculty using a 30-min spoken narrative and electrocorticographic (ECoG) recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Next, we leverage DLM embeddings to demonstrate that many electrodes spontaneously predict the meaning of upcoming words, even hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that deep language models provide an important step toward creating a biologically feasible computational framework for generative language.

Competing Interest Statement

The authors have declared no competing interest.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.
Back to top
PreviousNext
Posted December 03, 2020.
Download PDF
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Thinking ahead: prediction in context as a keystone of language in humans and machines
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Thinking ahead: prediction in context as a keystone of language in humans and machines
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Fanda Lora, Adeen Flinker, Sasha Devore, Werner Doyle, Daniel Friedman, Patricia Dugan, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, Uri Hasson
bioRxiv 2020.12.02.403477; doi: https://doi.org/10.1101/2020.12.02.403477
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Thinking ahead: prediction in context as a keystone of language in humans and machines
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Fanda Lora, Adeen Flinker, Sasha Devore, Werner Doyle, Daniel Friedman, Patricia Dugan, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, Uri Hasson
bioRxiv 2020.12.02.403477; doi: https://doi.org/10.1101/2020.12.02.403477

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Neuroscience
Subject Areas
All Articles
  • Animal Behavior and Cognition (4395)
  • Biochemistry (9619)
  • Bioengineering (7111)
  • Bioinformatics (24915)
  • Biophysics (12642)
  • Cancer Biology (9980)
  • Cell Biology (14388)
  • Clinical Trials (138)
  • Developmental Biology (7977)
  • Ecology (12135)
  • Epidemiology (2067)
  • Evolutionary Biology (16010)
  • Genetics (10938)
  • Genomics (14764)
  • Immunology (9889)
  • Microbiology (23719)
  • Molecular Biology (9493)
  • Neuroscience (50969)
  • Paleontology (370)
  • Pathology (1544)
  • Pharmacology and Toxicology (2688)
  • Physiology (4031)
  • Plant Biology (8685)
  • Scientific Communication and Education (1512)
  • Synthetic Biology (2403)
  • Systems Biology (6446)
  • Zoology (1346)