Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Computational noise in reward-guided learning drives behavioral variability in volatile environments

Charles Findling, Vasilisa Skvortsova, Rémi Dromnelle, Stefano Palminteri, Valentin Wyart
doi: https://doi.org/10.1101/439885
Charles Findling
1Laboratoire de Neurosciences Cognitives, Inserm U960, Département d’Etudes Cognitives, Ecole Normale Supérieure, PSL University, Paris, France
2ENSAE ParisTech, Paris-Saclay University, Palaiseau, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Vasilisa Skvortsova
1Laboratoire de Neurosciences Cognitives, Inserm U960, Département d’Etudes Cognitives, Ecole Normale Supérieure, PSL University, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rémi Dromnelle
1Laboratoire de Neurosciences Cognitives, Inserm U960, Département d’Etudes Cognitives, Ecole Normale Supérieure, PSL University, Paris, France
3Institut des Systèmes Intelligents et de Robotique, CNRS UMR7222, Sorbonne University, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stefano Palminteri
1Laboratoire de Neurosciences Cognitives, Inserm U960, Département d’Etudes Cognitives, Ecole Normale Supérieure, PSL University, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Valentin Wyart
1Laboratoire de Neurosciences Cognitives, Inserm U960, Département d’Etudes Cognitives, Ecole Normale Supérieure, PSL University, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: valentin.wyart@ens.fr
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Preview PDF
Loading

Abstract

When learning the value of actions in volatile environments, humans often make seemingly irrational decisions which fail to maximize expected value. We reasoned that these ‘non-greedy’ decisions, instead of reflecting information seeking during choice, may be caused by computational noise in the learning of action values. Here, using reinforcement learning (RL) models of behavior and multimodal neurophysiological data, we show that the majority of non-greedy decisions stems from this learning noise. The trial-to-trial variability of sequential learning steps and their impact on behavior could be predicted both by BOLD responses to obtained rewards in the dorsal anterior cingulate cortex (dACC) and by phasic pupillary dilation – suggestive of neuromodulatory fluctuations driven by the locus coeruleus-norepinephrine (LC-NE) system. Together, these findings indicate that most of behavioral variability, rather than reflecting human exploration, is due to the limited computational precision of reward-guided learning.

Acknowledgments

We thank C. Summerfield for comments on an earlier version of the manuscript. This work was supported by a starting grant from the European Research Council awarded to V.W. (ERC-StG-759341), a junior researcher grant from the Agence Nationale de la Recherche awarded to V.W. (ANR-14-CE13-0028), and two department-wide grants from the Agence Nationale de la Recherche (ANR-10-LABX-0087 and ANR-10-IDEX-0001-02 PSL). C.F. was supported by a graduate research fellowship from the Direction Générale de l’Armement (2015-60-0041). S.P. was supported by a CNRS-Inserm ATIP-Avenir grant (R16069JS) and a research grant from the Programme Emergence(s) of the City of Paris.

Author contributions

Conceptualization: S.P. and V.W.; Methodology: C.F., V.W., S.P., and V.W.; Formal Analysis: C.F., V.S., and V.W.; Investigation: V.S. and R.D.; Writing – Original Draft: C.F., V.S., and V.W.; Writing – Review & Editing: C.F., V.S., S.P., and V.W.; Supervision: V.W.; Funding Acquisition: V.W.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.
Back to top
PreviousNext
Posted October 11, 2018.
Download PDF

Supplementary Material

Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Computational noise in reward-guided learning drives behavioral variability in volatile environments
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Computational noise in reward-guided learning drives behavioral variability in volatile environments
Charles Findling, Vasilisa Skvortsova, Rémi Dromnelle, Stefano Palminteri, Valentin Wyart
bioRxiv 439885; doi: https://doi.org/10.1101/439885
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Computational noise in reward-guided learning drives behavioral variability in volatile environments
Charles Findling, Vasilisa Skvortsova, Rémi Dromnelle, Stefano Palminteri, Valentin Wyart
bioRxiv 439885; doi: https://doi.org/10.1101/439885

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Neuroscience
Subject Areas
All Articles
  • Animal Behavior and Cognition (3609)
  • Biochemistry (7590)
  • Bioengineering (5533)
  • Bioinformatics (20833)
  • Biophysics (10347)
  • Cancer Biology (7998)
  • Cell Biology (11663)
  • Clinical Trials (138)
  • Developmental Biology (6619)
  • Ecology (10227)
  • Epidemiology (2065)
  • Evolutionary Biology (13648)
  • Genetics (9557)
  • Genomics (12860)
  • Immunology (7932)
  • Microbiology (19576)
  • Molecular Biology (7678)
  • Neuroscience (42194)
  • Paleontology (309)
  • Pathology (1259)
  • Pharmacology and Toxicology (2208)
  • Physiology (3272)
  • Plant Biology (7064)
  • Scientific Communication and Education (1296)
  • Synthetic Biology (1953)
  • Systems Biology (5435)
  • Zoology (1119)