Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Formalizing planning and information search in naturalistic decision-making

Nature Neuroscience volume 24, pages 1051–1064 (2021)Cite this article

Subjects

Abstract

Decisions made by mammals and birds are often temporally extended. They require planning and sampling of decision-relevant information. Our understanding of such decision-making remains in its infancy compared with simpler, forced-choice paradigms. However, recent advances in algorithms supporting planning and information search provide a lens through which we can explain neural and behavioral data in these tasks. We review these advances to obtain a clearer understanding for why planning and curiosity originated in certain species but not others; how activity in the medial temporal lobe, prefrontal and cingulate cortices may support these behaviors; and how planning and information search may complement each other as means to improve future action selection.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Aquatic versus aerial visual scenes and how the corresponding habitats affect the utility of habit- and plan-based action selection during dynamic visually guided behavior.
Fig. 2: As rats approach a choice point, a theta-locked hippocampal representation sweeps ahead of the rat toward potential goals.
Fig. 3: A normative model-based planning account of replay events observed in hippocampal place cells and in simulations of spatial navigation tasks.
Fig. 4: The successor representation allows for rapid revaluation and extraction of components that identify key features of state space structure.
Fig. 5: Unsupervised cell assembly detection to identify neural substrates of cognitive tasks.
Fig. 6: Cognitive planning behaviors can be functionally dissociated in several human fMRI studies.
Fig. 7: Activity in the dACC is associated with information sampling across multiple decision-making studies.

Similar content being viewed by others

References

  1. Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).

    Article  CAS  PubMed  Google Scholar 

  2. Niv, Y. Reinforcement learning in the brain. J. Math. Psychol. 53, 139–154 (2009).

    Article  Google Scholar 

  3. Glimcher, P. W. & Fehr, E. Neuroeconomics: Decision Making and the Brain 2nd edn (Elsevier/Academic Press, 2014).

  4. Mobbs, D., Trimmer, P. C., Blumstein, D. T. & Dayan, P. Foraging for foundations in decision neuroscience: insights from ethology. Nat. Rev. Neurosci. 19, 419–427 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    Article  CAS  PubMed  Google Scholar 

  6. Dolan, R. J. & Dayan, P. Goals and habits in the brain. Neuron 80, 312–325 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Redish, A. D. Vicarious trial and error. Nat. Rev. Neurosci. 17, 147–159 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Jones, J. L. et al. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338, 953–956 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D. & Daw, N. D. Model-based choices involve prospective neural activity. Nat. Neurosci. 18, 767–772 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Schmidt, B., Duin, A. A. & Redish, A. D. Disrupting the medial prefrontal cortex alters hippocampal sequences during deliberative decision making. J. Neurophysiol. 121, 1981–2000 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Wilkinson, A. & Huber, L. Cold-blooded cognition: Reptilian cognitive abilities. in The Oxford Handbook of Comparative Evolutionary Psychology (eds Vonk, J. & Shackelford, T. K) 129–143 (Oxford Univ. Press, 2012).

  12. Burghardt, G. M. Environmental enrichment and cognitive complexity in reptiles and amphibians: concepts, review, and implications for captive populations. Appl. Anim. Behav. Sci. 147, 286–298 (2013).

    Article  Google Scholar 

  13. Broglio, C. et al. Hippocampal pallium and map-like memories through vertebrate evolution. J. Behav. Brain Sci. 05, 109–120 (2015).

    Article  Google Scholar 

  14. MacIver, M. A., Schmitz, L., Mugan, U., Murphey, T. D. & Mobley, C. D. Massive increase in visual range preceded the origin of terrestrial vertebrates. Proc. Natl Acad. Sci. USA 114, E2375–E2384 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Stein, W. E., Berry, C. M., Hernick, L. V. & Mannolini, F. Surprisingly complex community discovered in the mid-Devonian fossil forest at Gilboa. Nature 483, 78–81 (2012).

    Article  CAS  PubMed  Google Scholar 

  16. Mugan, U. & MacIver, M. A. Spatial planning with long visual range benefits escape from visual predators in complex naturalistic environments. Nat. Commun. 11, 3057 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Tolman, E. C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).

    Article  CAS  PubMed  Google Scholar 

  18. Tse, D. et al. Schemas and memory consolidation. Science 316, 76–82 (2007).

    Article  CAS  PubMed  Google Scholar 

  19. Raby, C. R., Alexis, D. M., Dickinson, A. & Clayton, N. S. Planning for the future by western scrub-jays. Nature 445, 919–921 (2007).

    Article  CAS  PubMed  Google Scholar 

  20. Wimpenny, J. H., Weir, A. A., Clayton, L., Rutz, C. & Kacelnik, A. Cognitive processes associated with sequential tool use in New Caledonian crows. PLoS ONE 4, e6471 (2009).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  21. Tanji, J., Shima, K. & Mushiake, H. Concept-based behavioral planning and the lateral prefrontal cortex. Trends Cogn. Sci. 11, 528–534 (2007).

    Article  PubMed  Google Scholar 

  22. Clutton-Brock, T. H. & Harvey, P. H. Primates, brains and ecology. J. Zool. 190, 309–323 (1980).

    Article  Google Scholar 

  23. Conway, C. M. & Christiansen, M. H. Sequential learning in non-human primates. Trends Cogn. Sci. 5, 539–546 (2001).

    Article  PubMed  Google Scholar 

  24. Le Fur, S., Fara, E., Mackaye, H. T., Vignaud, P. & Brunet, M. The mammal assemblage of the hominid site TM266 (Late Miocene, Chad Basin): ecological structure and paleoenvironmental implications. Naturwissenschaften 96, 565–574 (2009).

    Article  CAS  PubMed  Google Scholar 

  25. Dunbar, R. I. M. & Shultz, S. Why are there so many explanations for primate brain evolution? Philos. Trans. R Soc. Lond. B Biol. Sci. https://doi.org/10.1098/rstb.2016.0244 (2017).

  26. Lee, D. & Seo, H. Neural basis of strategic decision making. Trends Neurosci. 39, 40–48 (2016).

    Article  CAS  PubMed  Google Scholar 

  27. Gottlieb, J. & Oudeyer, P. Y. Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19, 758–770 (2018).

    Article  CAS  PubMed  Google Scholar 

  28. Glickman, S. E. & Sroges, R. W. Curiosity in zoo animals. Behaviour 26, 151–188 (1966).

    Article  CAS  PubMed  Google Scholar 

  29. Montgomery, S. H. The relationship between play, brain growth and behavioural flexibility in primates. Anim. Behav. 90, 281–286 (2014).

    Article  Google Scholar 

  30. Wimpenny, J. H., Weir, A. A. & Kacelnik, A. New Caledonian crows use tools for non-foraging activities. Anim. Cogn. 14, 459–464 (2011).

    Article  PubMed  Google Scholar 

  31. Callaway, F. et al. Human planning as optimal information seeking. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/byaqd (2021).

  32. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Mattar, M. G. & Daw, N. D. Prioritized memory access explains planning and hippocampal replay. Nat. Neurosci. 21, 1609–1617 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Keramati, M., Dezfouli, A. & Piray, P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 7, e1002055 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Behrens, T. E. J. et al. What Is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).

    Article  CAS  PubMed  Google Scholar 

  36. Johnson, A. & Redish, A. D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180, 552–567.e25 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Diba, K. & Buzsaki, G. Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10, 1241–1242 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Buzsaki, G. Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning. Hippocampus 25, 1073–1188 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  40. Gupta, A. S., van der Meer, M. A., Touretzky, D. S. & Redish, A. D. Segmentation of spatial experience by hippocampal theta sequences. Nat. Neurosci. 15, 1032–1039 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Zielinski, M. C., Shin, J. D. & Jadhav, S. P. Coherent coding of spatial position mediated by theta oscillations in the hippocampus and prefrontal cortex. J. Neurosci. 39, 4550–4565 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. van der Meer, M. A. & Redish, A. D. Expectancies in decision making, reinforcement learning, and ventral striatum. Front. Neurosci. 4, 6 (2010).

    PubMed  PubMed Central  Google Scholar 

  43. Gardner, R. S. et al. A secondary working memory challenge preserves primary place strategies despite overtraining. Learn. Mem. 20, 648–656 (2013).

    Article  PubMed  Google Scholar 

  44. Steiner, A. P. & Redish, A. D. The road not taken: neural correlates of decision making in orbitofrontal cortex. Front. Neurosci. 6, 131 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  45. Powell, N. J. & Redish, A. D. Complex neural codes in rat prelimbic cortex are stable across days on a spatial decision task. Front. Behav. Neurosci. 8, 120 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  46. Stott, J. J. & Redish, A. D. A functional difference in information processing between orbitofrontal cortex and ventral striatum during decision-making behaviour. Philos. Trans. R. Soc. Lond. B Biol. Sci. https://doi.org/10.1098/rstb.2013.0472 (2014).

  47. Hu, D. & Amsel, A. A simple test of the vicarious trial-and-error hypothesis of hippocampal function. Proc. Natl Acad. Sci. USA 92, 5506–5509 (1995).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Meyer-Mueller, C. et al. Dorsal, but not ventral, hippocampal inactivation alters deliberation in rats. Behav. Brain Res. 390, 112622 (2020).

    Article  PubMed  Google Scholar 

  49. Kreher, M. A. et al. The perirhinal cortex supports spatial intertemporal choice stability. Neurobiol. Learn. Mem. 162, 36–46 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Procyk, E., Tanaka, Y. L. & Joseph, J. P. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).

    Article  CAS  PubMed  Google Scholar 

  51. Stoll, F. M., Fontanier, V. & Procyk, E. Specific frontal neural dynamics contribute to decisions to check. Nat. Commun. 7, 11990 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Singer, A. C. & Frank, L. M. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 64, 910–921 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Papale, A. E., Zielinski, M. C., Frank, L. M., Jadhav, S. P. & Redish, A. D. Interplay between hippocampal sharp-wave-ripple events and vicarious trial and error behaviors in decision making. Neuron 92, 975–982 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Gupta, A. S., van der Meer, M. A., Touretzky, D. S. & Redish, A. D. Hippocampal replay is not a simple function of experience. Neuron 65, 695–705 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Singer, A. C., Carr, M. F., Karlsson, M. P. & Frank, L. M. Hippocampal SWR activity predicts correct decisions during the initial learning of an alternation task. Neuron 77, 1163–1173 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Jadhav, S. P., Kemere, C., German, P. W. & Frank, L. M. Awake hippocampal sharp-wave ripples support spatial memory. Science 336, 1454–1458 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Foster, D. J. & Wilson, M. A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440, 680–683 (2006).

    Article  CAS  PubMed  Google Scholar 

  58. Ambrose, R. E., Pfeiffer, B. E. & Foster, D. J. Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91, 1124–1136 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Davidson, T. J., Kloosterman, F. & Wilson, M. A. Hippocampal replay of extended experience. Neuron 63, 497–507 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  60. Olafsdottir, H. F., Barry, C., Saleem, A. B., Hassabis, D. & Spiers, H. J. Hippocampal place cells construct reward related sequences through unexplored space. eLife 4, e06063 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  61. Miller, K.J. & Venditto, S. J. C. Multi-step planning in the brain. Curr. Opin. Behav. Sci. 38, 29–39 (2021).

    Article  Google Scholar 

  62. Kurth-Nelson, Z., Economides, M., Dolan, R. J. & Dayan, P. Fast sequences of non-spatial state representations in humans. Neuron 91, 194–204 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Momennejad, I., Otto, A. R., Daw, N. D. & Norman, K. A. Offline replay supports planning in human reinforcement learning. eLife https://doi.org/10.7554/eLife.32548 (2018).

  64. Schuck, N. W. & Niv, Y. Sequential replay of nonspatial task states in the human hippocampus. Science https://doi.org/10.1126/science.aaw5181 (2019).

  65. Liu, Y., Dolan, R. J., Kurth-Nelson, Z. & Behrens, T. E. J. Human replay spontaneously reorganizes experience. Cell 178, 640–652.e14 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Liu, Y., Mattar, M. G., Behrens, T. E. J., Daw, N. D. & Dolan, R. J. Experience replay is associated with efficient nonlocal learning. Science 372, eabf1357 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. van Opheusden, B. & Ma, W. J. Tasks for aligning human and machine planning. Curr. Opin. Behav. Sci. 29, 127–133 (2019).

    Article  Google Scholar 

  68. Kemp, C. & Tenenbaum, J. B. The discovery of structural form. Proc. Natl Acad. Sci. USA 105, 10687–10692 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  69. Bengio, Y., Courville, A. & Vincent, P. Representation Learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).

    Article  PubMed  Google Scholar 

  70. Radulescu, A., Niv, Y. & Ballard, I. Holistic reinforcement learning: the role of structure and attention. Trends Cogn. Sci. 23, 278–292 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Schrittwieser, J. et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588, 604–609 (2020).

    Article  CAS  PubMed  Google Scholar 

  72. Baram, A. B., Muller, T. H., Nili, H., Garvert, M. M. & Behrens, T. E. J. Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems. Neuron 109, 713–723.e7 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  73. Schulz, E., Franklin, N. T. & Gershman, S. J. Finding structure in multi-armed bandits. Cogn. Psychol. 119, 101261 (2020).

    Article  PubMed  Google Scholar 

  74. Wu, C. M., Schulz, E., Speekenbrink, M., Nelson, J. D. & Meder, B. Generalization guides human exploration in vast decision spaces. Nat. Hum. Behav. 2, 915–924 (2018).

    Article  PubMed  Google Scholar 

  75. Harlow, H. F. The formation of learning sets. Psychol. Rev. 56, 51–65 (1949).

    Article  CAS  PubMed  Google Scholar 

  76. Wang, J. X. et al. Prefrontal cortex as a meta-reinforcement learning system. Nat. Neurosci. 21, 860–868 (2018).

    Article  CAS  PubMed  Google Scholar 

  77. Browning, P. G., Easton, A. & Gaffan, D. Frontal-temporal disconnection abolishes object discrimination learning set in macaque monkeys. Cereb. Cortex 17, 859–864 (2007).

    Article  PubMed  Google Scholar 

  78. M’Harzi, M. et al. Effects of selective lesions of fimbria-fornix on learning set in the rat. Physiol. Behav. 40, 181–188 (1987).

    Article  PubMed  Google Scholar 

  79. Schuck, N. W., Cai, M. B., Wilson, R. C. & Niv, Y. Human orbitofrontal cortex represents a cognitive map of state space. Neuron 91, 1402–1412 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993).

    Article  Google Scholar 

  81. Singh, S., James, M. R. & Rudary, M. R. Predictive state representations: a new theory for modeling dynamical systems. In Proc. 20th Conference on Uncertainty in Artificial Intelligence 512–519 (AUAI Press, 2004).

  82. Gershman, S. J. The successor representation: its computational logic and neural substrates. J. Neurosci. 38, 7193–7200 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  83. Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. The hippocampus as a predictive map. Nat. Neurosci. 20, 1643–1653 (2017).

    Article  CAS  PubMed  Google Scholar 

  84. Mehta, M. R., Quirk, M. C. & Wilson, M. A. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25, 707–715 (2000).

    Article  CAS  PubMed  Google Scholar 

  85. Momennejad, I. et al. The successor representation in human reinforcement learning. Nat. Hum. Behav. 1, 680–692 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  86. Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J. & Daw, N. D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput. Biol. 13, e1005768 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  87. Gardner, M. P. H., Schoenbaum, G. & Gershman, S. J. Rethinking dopamine as generalized prediction error. Proc. Biol. Sci. https://doi.org/10.1098/rspb.2018.1645 (2018).

  88. Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z. & Graybiel, A. M. Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 437, 1158–1161 (2005).

    Article  CAS  PubMed  Google Scholar 

  89. van der Meer, M. A., Johnson, A., Schmitzer-Torbert, N. C. & Redish, A. D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 67, 25–32 (2010).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  90. Koechlin, E. & Summerfield, C. An information theoretical approach to prefrontal executive function. Trends Cogn. Sci. 11, 229–235 (2007).

    Article  PubMed  Google Scholar 

  91. Brunec, I. K. & Momennejad, I. Predictive representations in hippocampal and prefrontal hierarchies. Preprint at bioRxiv https://doi.org/10.1101/786434 (2019).

  92. Mahadevan, S. & Maggioni, M. Proto-value Functions: a Laplacian framework for learning representation and control in Markov decision processes. J. Mach. Learn. Res. 8, 2169–2231 (2007).

    Google Scholar 

  93. Machado, M. C., Bellemare, M. G. & Bowling, M. Count-based exploration with the successor representation. in Proceedings of the AAAI Conference on Artificial Intelligence 34, 5125–5133 (2020).

    Article  Google Scholar 

  94. Schapiro, A. C., Turk-Browne, N. B., Botvinick, M. M. & Norman, K. A. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philos. Trans. R. Soc. Lond. B Biol. Sci. https://doi.org/10.1098/rstb.2016.0049 (2017)

  95. Whittington, J. C. R. et al. The Tolman–Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation. Cell 183, 1249–1263.e23 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  96. Piray, P. & Daw, N. D. Linear reinforcement learning: flexible reuse of computation in planning, grid fields, and cognitive control. Preprint at bioRxiv https://doi.org/10.1101/856849 (2021).

  97. Russo, E. & Durstewitz, D. Cell assemblies at multiple time scales with arbitrary lag constellations. eLife https://doi.org/10.7554/eLife.19428 (2017).

  98. Durstewitz, D. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLoS Comput. Biol. 13, e1005542 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  99. Pipa, G., Wheeler, D. W., Singer, W. & Nikolic, D. NeuroXidence: reliable and efficient analysis of an excess or deficiency of joint-spike events. J. Comput. Neurosci. 25, 64–88 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  100. Benchenane, K. et al. Coherent theta oscillations and reorganization of spike timing in the hippocampal–prefrontal network upon learning. Neuron 66, 921–936 (2010).

    Article  CAS  PubMed  Google Scholar 

  101. Quaglio, P., Yegenoglu, A., Torre, E., Endres, D. M. & Grun, S. Detection and evaluation of spatio-temporal spike patterns in massively parallel spike train data with SPADE. Front. Comput. Neurosci. 11, 41 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  102. Grossberger, L., Battaglia, F. P. & Vinck, M. Unsupervised clustering of temporal patterns in high-dimensional neuronal ensembles using a novel dissimilarity measure. PLoS Comput. Biol. 14, e1006283 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  103. van de Ven, G. M., Trouche, S., McNamara, C. G., Allen, K. & Dupret, D. Hippocampal offline reactivation consolidates recently formed cell assembly patterns during sharp wave-ripples. Neuron 92, 968–974 (2016).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  104. Gridchyn, I., Schoenenberger, P., O’Neill, J. & Csicsvari, J. Assembly-specific disruption of hippocampal replay leads to selective memory deficit. Neuron 106, 291–300.e6 (2020).

    Article  CAS  PubMed  Google Scholar 

  105. Oettl, L. L. et al. Phasic dopamine reinforces distinct striatal stimulus encoding in the olfactory tubercle driving dopaminergic reward prediction. Nat. Commun. 11, 3460 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  106. Gomperts, S. N., Kloosterman, F. & Wilson, M. A. VTA neurons coordinate with the hippocampal reactivation of spatial experience. eLife https://doi.org/10.7554/eLife.05360 (2015).

  107. Kurth-Nelson, Z. & Redish, A. D. Don’t let me do that!—models of precommitment. Front. Neurosci. 6, 138 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  108. Kolling, N., Scholl, J., Chekroud, A., Trier, H. A. & Rushworth, M. F. S. Prospection, perseverance, and insight in sequential behavior. Neuron 99, 1069–1082.e7 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Goel, V. & Grafman, J. Are the frontal lobes implicated in ‘planning’ functions? Interpreting data from the Tower of Hanoi. Neuropsychologia 33, 623–642 (1995).

    Article  CAS  PubMed  Google Scholar 

  110. Burgess, P. W. Strategy application disorder: the role of the frontal lobes in human multitasking. Psychol. Res. 63, 279–288 (2000).

    Article  CAS  PubMed  Google Scholar 

  111. Holroyd, C. B., Ribas-Fernandes, J. J. F., Shahnazian, D., Silvetti, M. & Verguts, T. Human midcingulate cortex encodes distributed representations of task progress. Proc. Natl Acad. Sci. USA 115, 6398–6403 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  112. Averbeck, B. B., Sohn, J. W. & Lee, D. Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 9, 276–282 (2006).

    Article  CAS  PubMed  Google Scholar 

  113. Collins, A. & Koechlin, E. Reasoning, learning, and creativity: frontal lobe function and human decision-making. PLoS Biol. 10, e1001293 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. Donoso, M., Collins, A. G. & Koechlin, E. Human cognition. Foundations of human reasoning in the prefrontal cortex. Science 344, 1481–1486 (2014).

    Article  CAS  PubMed  Google Scholar 

  115. Juechems, K. et al. A network for computing value equilibrium in the human medial prefrontal cortex. Neuron 101, 977–987.e3 (2019).

    Article  CAS  PubMed  Google Scholar 

  116. Huys, Q. J. et al. Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 8, e1002410 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  117. Lally, N. et al. The neural basis of aversive pavlovian guidance during planning. J. Neurosci. 37, 10215–10229 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  118. Huys, Q. J. et al. Interplay of approximate planning strategies. Proc. Natl Acad. Sci. USA 112, 3098–3103 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  119. Keramati, M., Smittenaar, P., Dolan, R. J. & Dayan, P. Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum. Proc. Natl Acad. Sci. USA 113, 12868–12873 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  120. Gittins, J. C. Bandit processes and dynamic allocation indices. J. R. Stat. Soc.: Ser. B (Methodol.) 41, 148–164 (1979).

    Google Scholar 

  121. Russo, D. J., Roy, B. V., Kazerouni, A., Osband, I. & Wen, Z. A tutorial on Thompson sampling. Found. Trends Mach. Learn. 11, 1–96 (2018).

    Article  Google Scholar 

  122. MacIver, M. A., Patankar, N. A. & Shirgaonkar, A. A. Energy–information trade-offs between movement and sensing. PLoS Comput. Biol. https://doi.org/10.1371/journal.pcbi.1000769 (2010).

  123. Chen, C., Murphey, T. D. & MacIver, M. A. Tuning movement for sensing in an uncertain world. eLife 9, e52371 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  124. Khamassi, M., Quilodran, R., Enel, P., Dominey, P. F. & Procyk, E. Behavioral regulation and the modulation of information coding in the lateral prefrontal and cingulate cortex. Cereb. Cortex 25, 3197–3218 (2015).

    Article  PubMed  Google Scholar 

  125. Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A. & Cohen, J. D. Humans use directed and random exploration to solve the explore–exploit dilemma. J. Exp. Psychol. Gen. 143, 2074–2081 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  126. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  127. Boorman, E. D., Behrens, T. E., Woolrich, M. W. & Rushworth, M. F. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).

    Article  CAS  PubMed  Google Scholar 

  128. Blanchard, T. C. & Gershman, S. J. Pure correlates of exploration and exploitation in the human brain. Cogn. Affect. Behav. Neurosci. 18, 117–126 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  129. Badre, D., Doll, B. B., Long, N. M. & Frank, M. J. Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron 73, 595–607 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  130. Zajkowski, W. K., Kossut, M. & Wilson, R. C. A causal role for right frontopolar cortex in directed, but not random, exploration. eLife https://doi.org/10.7554/eLife.27430 (2017).

  131. Warren, C. M. et al. The effect of atomoxetine on random and directed exploration in humans. PLoS ONE 12, e0176034 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  132. Tervo, D. G. R. et al. Behavioral variability through stochastic choice and its gating by anterior cingulate cortex. Cell 159, 21–32 (2014).

    Article  CAS  PubMed  Google Scholar 

  133. Blanchard, T. C., Hayden, B. Y. & Bromberg-Martin, E. S. Orbitofrontal cortex uses distinct codes for different choice attributes in decisions motivated by curiosity. Neuron 85, 602–614 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  134. White, J. K. et al. A neural network for information seeking. Nat. Commun. 10, 5168 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  135. Hunt, L. T. et al. Triple dissociation of attention and decision computations across prefrontal cortex. Nat. Neurosci. 21, 1471–1481 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  136. Ayala-Orozco, B. et al. Lévy walk patterns in the foraging movements of spider monkeys (Ateles geoffroyi). Behav. Ecol. Sociobiol. 55, 223–230 (2004).

    Article  Google Scholar 

  137. Sims, D. W. et al. Scaling laws of marine predator search behaviour. Nature 451, 1098–1102 (2008).

    Article  CAS  PubMed  Google Scholar 

  138. Viswanathan, G. M., Raposo, E. P. & da Luz, M. G. E. Lévy flights and superdiffusion in the context of biological encounters and random searches. Phys. Life Rev. 5, 133–150 (2008).

    Article  Google Scholar 

  139. McNamee, D. C., Stachenfeld, K. L., Botvinick, M. M. & Gershman, S. J. Flexible modulation of sequence generation in the entorhinal–hippocampal system. Nat. Neurosci. 24, 851–862 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  140. Stella, F., Baracskay, P., O’Neill, J. & Csicsvari, J. Hippocampal reactivation of random trajectories resembling brownian diffusion. Neuron 102, 450–461.e7 (2019).

    Article  CAS  PubMed  Google Scholar 

  141. Wosniack, M. E., Santos, M. C., Raposo, E. P., Viswanathan, G. M. & da Luz, M. G. E. The evolutionary origins of Levy walk foraging. PLoS Comput. Biol. 13, e1005774 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  142. Skaggs, W. E., McNaughton, B. L., Wilson, M. A. & Barnes, C. A. Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences. Hippocampus 6, 149–172 (1996).

    Article  CAS  PubMed  Google Scholar 

  143. Fiebelkorn, I. C., Pinsk, M. A. & Kastner, S. A dynamic interplay within the frontoparietal network underlies rhythmic spatial attention. Neuron 99, 842–853.e8 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  144. Kleinfeld, D., Deschenes, M. & Ulanovsky, N. Whisking, sniffing, and the hippocampal theta-rhythm: a tale of two oscillators. PLoS Biol. 14, e1002385 (2016).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  145. Stewart, N., Hermens, F. & Matthews, W. J. Eye movements in risky choice. J. Behav. Decis. Mak. 29, 116–136 (2016).

    Article  PubMed  Google Scholar 

  146. Hunt, L. T., Rutledge, R. B., Malalasekera, W. M., Kennerley, S. W. & Dolan, R. J. Approach-induced biases in human information sampling. PLoS Biol. 14, e2000638 (2016).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  147. Kobayashi, K., Ravaioli, S., Baranes, A., Woodford, M. & Gottlieb, J. Diverse motives for human curiosity. Nat. Hum. Behav. 3, 587–595 (2019).

    Article  PubMed  Google Scholar 

  148. Hayden, B. Y. Economic choice: the foraging perspective. Curr. Opin. Behav. Sci. 24, 1–6 (2018).

    Article  Google Scholar 

  149. Sweis, B. M. et al. Sensitivity to ‘sunk costs’ in mice, rats, and humans. Science 361, 178–181 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  150. Juechems, K. & Summerfield, C. Where does value come from? Trends Cogn. Sci. 23, 836–850 (2019).

    Article  PubMed  Google Scholar 

  151. Nilsson, D. E. Evolution: an irresistibly clear view of land. Curr. Biol. 27, R715–R717 (2017).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

L.T.H. was supported by a Henry Dale Fellowship from the Royal Society and Wellcome Trust (208789/Z/17/Z). N.D.D. was supported by NIDA R01DA038891 and NSF IIS-1822571, both part of the CRCNS program. M.A.M. was funded by NSF Brain Initiative ECCS-1835389. E.R. was supported by a Ch. and H. Schaller Foundation and the Boehringer Ingelheim Foundation grant ‘Complex Systems’. E.P. and C.R.E.W. are supported by the French National Research Agency within the framework of the labex CORTEX ANR-11-LABX-0042 of Université de Lyon, and grant ANR-19-CE37-0008 NORAD and ANR-18-CE37-0016-01 PREDYCT. E.P. is employed by the Centre National de la Recherche Scientifique. J.S. is funded by a MRC Skills Development Fellowship (MR/N014448/1). N.K. is funded by a fellowship from the BBSRC (BB/R010803/1).

Author information

Authors and Affiliations

  1. Department of Psychiatry, Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK

    L. T. Hunt & N. Kolling

  2. Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA

    N. D. Daw

  3. Department of Experimental Psychology, Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK

    P. Kaanders & J. Scholl

  4. Center for Robotics and Biosystems, Department of Neurobiology, Department of Biomedical Engineering, Department of Mechanical Engineering, Northwestern University, Evanston, IL, USA

    M. A. MacIver & U. Mugan

  5. Univ Lyon, Université Claude Bernard Lyon 1, INSERM, Stem Cell and Brain Research Institute U1208, Bron, France

    E. Procyk & C. R. E. Wilson

  6. Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA

    A. D. Redish

  7. Department of Theoretical Neuroscience, Central Institute of Mental Health, Mannheim, Germany

    E. Russo

  8. Department of Psychiatry and Psychotherapy, University Medical Center, Johannes Gutenberg University, Mainz, Germany

    E. Russo

  9. DeepMind, London, UK

    K. Stachenfeld

Authors
  1. L. T. Hunt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. N. D. Daw
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. Kaanders
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. A. MacIver
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. U. Mugan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. E. Procyk
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. A. D. Redish
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. E. Russo
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. J. Scholl
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. K. Stachenfeld
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. C. R. E. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. N. Kolling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to L. T. Hunt or N. Kolling.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Neuroscience thanks Daeyeol Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hunt, L.T., Daw, N.D., Kaanders, P. et al. Formalizing planning and information search in naturalistic decision-making. Nat Neurosci 24, 1051–1064 (2021). https://doi.org/10.1038/s41593-021-00866-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-021-00866-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing