Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain

View ORCID ProfileAriel Goldstein, Avigail Dabush, Bobbi Aubrey, Mariano Schain, Samuel A. Nastase, Zaid Zada, Eric Ham, Zhuoqiao Hong, Amir Feder, Harshvardhan Gazula, Eliav Buchnik, Werner Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Michael Brenner, Avinatan Hassidim, Orrin Devinsky, Adeen Flinker, Uri Hasson
doi: https://doi.org/10.1101/2022.03.01.482586
Ariel Goldstein
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Ariel Goldstein
  • For correspondence: ariel.y.goldstein@gmail.com
Avigail Dabush
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bobbi Aubrey
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mariano Schain
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Samuel A. Nastase
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zaid Zada
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eric Ham
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zhuoqiao Hong
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Amir Feder
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Harshvardhan Gazula
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eliav Buchnik
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Werner Doyle
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sasha Devore
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Patricia Dugan
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel Friedman
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Brenner
2Google Research
4School of Engineering and Applied Science, Harvard University, Cambridge, MA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Avinatan Hassidim
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Orrin Devinsky
3New York University Grossman School of Medicine, New York, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Adeen Flinker
3New York University Grossman School of Medicine, New York, NY
5New York University Tandon School of Engineering, Brooklyn, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Uri Hasson
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ
2Google Research
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. Do language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language? To test this hypothesis, we densely recorded the neural activity in the Inferior Frontal Gyrus (IFG, also known as Broca’s area) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derived for each patient a continuous vectorial representation for each word (i.e., a brain embedding). Using stringent, zero-shot mapping, we demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry. This shared geometry allows us to precisely triangulate the position of unseen words in both the brain embedding space (zero-shot encoding) and the DLM contextual embedding space (zero-shot decoding). The continuous brain embedding space provides an alternative computational framework for how natural language is represented in cortical language areas.

Competing Interest Statement

The authors have declared no competing interest.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-ND 4.0 International license.
Back to top
PreviousNext
Posted March 02, 2022.
Download PDF
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain
Ariel Goldstein, Avigail Dabush, Bobbi Aubrey, Mariano Schain, Samuel A. Nastase, Zaid Zada, Eric Ham, Zhuoqiao Hong, Amir Feder, Harshvardhan Gazula, Eliav Buchnik, Werner Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Michael Brenner, Avinatan Hassidim, Orrin Devinsky, Adeen Flinker, Uri Hasson
bioRxiv 2022.03.01.482586; doi: https://doi.org/10.1101/2022.03.01.482586
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain
Ariel Goldstein, Avigail Dabush, Bobbi Aubrey, Mariano Schain, Samuel A. Nastase, Zaid Zada, Eric Ham, Zhuoqiao Hong, Amir Feder, Harshvardhan Gazula, Eliav Buchnik, Werner Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Michael Brenner, Avinatan Hassidim, Orrin Devinsky, Adeen Flinker, Uri Hasson
bioRxiv 2022.03.01.482586; doi: https://doi.org/10.1101/2022.03.01.482586

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Neuroscience
Subject Areas
All Articles
  • Animal Behavior and Cognition (4237)
  • Biochemistry (9155)
  • Bioengineering (6797)
  • Bioinformatics (24052)
  • Biophysics (12149)
  • Cancer Biology (9562)
  • Cell Biology (13814)
  • Clinical Trials (138)
  • Developmental Biology (7653)
  • Ecology (11729)
  • Epidemiology (2066)
  • Evolutionary Biology (15534)
  • Genetics (10663)
  • Genomics (14346)
  • Immunology (9503)
  • Microbiology (22876)
  • Molecular Biology (9113)
  • Neuroscience (49080)
  • Paleontology (357)
  • Pathology (1487)
  • Pharmacology and Toxicology (2576)
  • Physiology (3851)
  • Plant Biology (8347)
  • Scientific Communication and Education (1473)
  • Synthetic Biology (2299)
  • Systems Biology (6202)
  • Zoology (1302)