Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Towards an integration of deep learning and neuroscience

Adam H. Marblestone, Greg Wayne, Konrad P. Kording
doi: https://doi.org/10.1101/058545
Adam H. Marblestone
1MIT Media Lab Cambridge, MA 02139, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Greg Wayne
2Google Deepmind London, EC4A 3TW, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Konrad P. Kording
3Rehabilitation Institute of Chicago Northwestern University Chicago, IL 60611, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

Neuroscience has focused on the detailed implementation of computation, studying neural codes, dynamics and circuits. In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and relatively uniform initial architectures. Two recent developments have emerged within machine learning that create an opportunity to connect these seemingly divergent perspectives. First, structured architectures are used, including dedicated systems for attention, recursion and various forms of short- and long-term memory storage. Second, cost functions and training procedures have become more complex and are varied across layers and over time. Here we think about the brain in terms of these ideas. We hypothesize that (1) the brain optimizes cost functions, (2) the cost functions are diverse and differ across brain locations and over development, and (3) optimization operates within a pre-structured architecture matched to the computational problems posed by behavior. In support of these hypotheses, we argue that a range of implementations of credit assignment through multiple layers of neurons are compatible with our current knowledge of neural circuitry, and that the brain’s specialized systems can be interpreted as enabling efficient optimization for specific problem classes. Such a heterogeneously optimized system, enabled by a series of interacting cost functions, serves to make learning data-efficient and precisely targeted to the needs of the organism. We suggest directions by which neuroscience could seek to refine and test these hypotheses.

1. Introduction

Machine learning and neuroscience speak different languages today. Brain science has discovered a dazzling array of brain areas (Solari and Stoner, 2015), cell types, molecules, cellular states, and mechanisms for computation and information storage. Machine learning, in contrast, has largely focused on instantiations of a single principle: function optimization. It has found that simple optimization objectives, like minimizing classification error, can lead to the formation of rich internal representations and powerful algorithmic capabilities in multilayer and recurrent networks (LeCun et al., 2015; Schmidhuber, 2015). Here we seek to connect these perspectives.

The artificial neural networks now prominent in machine learning were, of course, originally inspired by neuroscience (McCulloch and Pitts, 1943). While neuroscience has continued to play a role (Cox and Dean, 2014), many of the major developments were guided by insights into the mathematics of efficient optimization, rather than neuroscientific findings (Sutskever and Martens, 2013). The field has advanced from simple linear systems (Minsky and Papert, 1972), to nonlinear networks (Haykin, 1994), to deep and recurrent networks (Schmidhuber, 2015; LeCun et al., 2015). Backpropagation of error (Werbos, 1974, 1982; Rumelhart et al., 1986) enabled neural networks to be trained effi-ciently, by providing an efficient means to compute the gradient with respect to the weights of a multi-layer network. Methods of training have improved to include momentum terms, better weight initializations, conjugate gradients and so forth, evolving to the current breed of networks optimized using batch-wise stochastic gradient descent. These developments have little obvious connection to neuroscience.

We will argue here, however, that neuro-science and machine learning are again ripe for convergence. Three aspects of machine learning are particularly important in the context of this paper. First, machine learning has focused on the optimization of cost functions (Figure 1A).

Fig 1:
  • Download figure
  • Open in new tab
Fig 1: Putative differences between conventional and brain-like neural network designs.

A) In conventional deep learning, supervised training is based on externally-supplied, labeled data.

B) In the brain, supervised training of networks can still occur via gradient descent on an error signal, but this error signal must arise from internally generated cost functions. These cost functions are themselves computed by neural modules specified by both genetics and learning. Internally generated cost functions create heuristics that are used to bootstrap more complex learning. For example, an area which recognizes faces might first be trained to detect faces using simple heuristics, like the presence of two dots above a line, and then further trained to discriminate salient facial expressions using representations arising from unsupervised learning and error signals from other brain areas related to social reward processing.

C) Internally generated cost functions and error-driven training of cortical deep networks form part of a larger architecture containing several specialized systems. Although the trainable cortical areas are schematized as feedforward neural networks here, LSTMs or other types of recurrent networks may be a more accurate analogy, and many neuronal and network properties such as spiking, dendritic computation, neuromodulation, adaptation and homeostatic plasticity, timing-dependent plasticity, direct electrical connections, transient synaptic dynamics, excitatory/inhibitory balance, spontaneous oscillatory activity, axonal conduction delays (Izhikevich, 2006) and others, will influence what and how such networks learn.

Second, recent work in machine learning has started to introduce complex cost functions, those that are not uniform across layers and time, and those that arise from interactions between different parts of a network. For example, introducing the objective of temporal coherence for lower layers (non-uniform cost function over space) improves feature learning (Sermanet and Kavukcuoglu, 2013), cost function schedules (non-uniform cost function over time) improve1 generalization (Saxe et al., 2013; Goodfellow et al., 2014b; Gülçehre and Bengio, 2016) and adversarial networks – an example of a cost function arising from internal interactions – allow gradient-based training of generative models (Goodfellow et al., 2014a)2. Networks that are easier to train are being used to provide “hints” to help bootstrap the training of more powerful networks (Romero et al., 2014).

Third, machine learning has also begun to diversify the architectures that are subject to optimization. It has introduced simple memory cells with multiple persistent states (Hochreiter and Schmidhuber, 1997; Chung et al., 2014), more complex elementary units such as “capsules” and other structures (Hinton et al., 2011; Livni et al., 2013; Delalleau and Bengio, 2011; Tang et al., 2012), content addressable (Weston et al., 2014; Graves et al., 2014) and location addressable memories (Graves et al., 2014), as well as pointers (Kurach et al., 2015) and hard-coded arithmetic operations (Neelakantan et al., 2015).

These three ideas have, so far, not received much attention in neuroscience. We thus formulate these ideas as three hypotheses about the brain, examine evidence for them, and sketch how experiments could test them. But first, let us state the hypotheses more precisely.

1.1 Hypothesis 1 – The brain optimizes cost functions

The central hypothesis for linking the two fields is that biological systems, like many machine-learning systems, are able to optimize cost functions. The idea of cost functions means that neurons in a brain area can somehow change their properties, e.g., the properties of their synapses, so that they get better at doing whatever the cost function defines as their role. Human behavior sometimes approaches optimality in a domain, e.g., during movement (Körding, 2007), which suggests that the brain may have learned optimal strategies. Subjects minimize energy consumption of their movement system (Taylor and Faisal, 2011), and minimize risk and damage to their body, while maximizing financial and movement gains. Computationally, we now know that optimization of trajectories gives rise to elegant solutions for very complex motor tasks (Mordatch et al., 2012; Todorov and Jordan, 2002; Harris and Wolpert, 1998). We suggest that cost function optimization occurs much more generally in shaping the internal representations and processes used by the brain. Importantly, we also suggest that this requires the brain to have mechanisms for efficient credit assignment in multilayer and recurrent networks.

1.2 Hypothesis 2 – Cost functions are diverse across areas and change over development

A second realization is that cost functions need not be global. Neurons in different brain areas may optimize different things, e.g., the mean squared error of movements, surprise in a visual stimulus, or the allocation of attention. Importantly, such a cost function could be locally generated. For example, neurons could locally evaluate the quality of their statistical model of their inputs (Figure 1B). Alternatively, cost functions for one area could be generated by another area. Moreover, cost functions may change over time, e.g., guiding young humans to understanding simple visual contrasts early on, and faces a bit later3. This could allow the developing brain to bootstrap more complex knowledge based on simpler knowledge. Cost functions in the brain are likely to be complex and to be arranged to vary across areas and over development.

1.3 Hypothesis 3 – Specialized systems allow efficient solution of key computational problems

A third realization is that structure matters. The patterns of information flow seem fundamentally different across brain areas, suggesting that they solve distinct computational problems. Some brain areas are highly recurrent, perhaps making them predestined for short-term memory storage (Wang, 2012). Some areas contain cell types that can switch between qualitatively different states of activation, such as a persistent firing mode versus a transient firing mode, in response to particular neurotransmitters (Hasselmo, 2006). Other areas, like the thalamus appear to have the information from other areas flowing through them, perhaps allowing them to determine information routing (Sherman, 2005). Areas like the basal ganglia are involved in reinforcement learning and gating of discrete decisions (Sejnowski and Poizner, 2014; Doya, 1999). As every programmer knows, specialized algorithms matter for efficient solutions to computational problems, and the brain is likely to make good use of such specialization (Figure 1C).

These ideas are inspired by recent advances in machine learning, but we also propose that the brain has major differences from any of today’s machine learning techniques. In particular, the world gives us a relatively limited amount of information that we could use for supervised learning (Fodor and Crowther, 2002). There is a huge amount of information available for un-supervised learning, but there is no reason to assume that a generic unsupervised algorithm, no matter how powerful, would learn the precise things that humans need to know, in the order that they need to know it. The evolutionary challenge of making unsupervised learning solve the “right” problems is, therefore, to find a sequence of cost functions that will deterministically build circuits and behaviors according to prescribed developmental stages, so that in the end a relatively small amount of information suffices to produce the right behavior. For example, a developing duck imprints (Tinbergen, 1965) a template of its parent, and then uses that template to generate goal-targets that help it develop other skills like foraging.

Generalizing from this and from other studies (Ullman et al., 2012; Minsky, 1977), we propose that many of the brain’s cost functions arise from such an internal bootstrapping process. Indeed, we propose that biological development and reinforcement learning can, in effect, program the emergence of a sequence of cost functions that precisely anticipates the future needs faced by the brain’s internal subsystems, as well as by the organism as a whole. This type of developmentally programmed bootstrapping generates an internal infrastructure of cost functions which is diverse and complex, while simplifying the learning problems faced by the brain’s internal processes. Beyond simple tasks like familial imprinting, this type of bootstrapping could extend to higher cognition, e.g., internally generated cost functions could train a developing brain to properly access its memory or to organize its actions in ways that will prove to be useful later on. The potential bootstrapping mechanisms that we will consider operate in the context of unsupervised and reinforcement learning, and go well beyond the types of curriculum learning ideas used in today’s machine learning (Bengio et al., 2009).

In the rest of this paper, we will elaborate on these hypotheses. First, we will argue that both local and multi-layer optimization is, perhaps surprisingly, compatible with what we know about the brain. Second, we will argue that cost functions differ across brain areas and change over time and describe how cost functions interacting in an orchestrated way could allow bootstrapping of complex function. Third, we will list a broad set of specialized problems that need to be solved by neural computation, and the brain areas that have structure that seems to be matched to a particular computational problem. We then discuss some implications of the above hypotheses for research approaches in neuroscience and machine learning, and sketch a set of experiments to test these hypotheses. Finally, we discuss this architecture from the perspective of evolution.

2. The brain can optimize cost functions

Much of machine learning is based on efficiently optimizing functions, and, as we will detail below, the ability to use backpropagation of error (Werbos, 1974; Rumelhart et al., 1986) to calculate gradients of arbitrary parametrized functions has been a key breakthrough. In Hypothesis 1, we claim that the brain is also, at least in part4, an optimization machine. But what exactly does it mean to say that the brain can optimize cost functions? After all, many processes can be viewed as optimizations. For example, the laws of physics are often viewed as minimizing an action functional, while evolution optimizes the fitness of replicators over a long timescale. To be clear, our main claims are: that a) the brain has powerful mechanisms for credit assignment during learning that allow it to optimize global functions in multi-layer networks by adjusting the properties of each neuron to contribute to the global outcome, and that b) the brain has mechanisms to specify exactly which cost functions it subjects its networks to, i.e., that the cost functions are highly tunable, shaped by evolution and matched to the animal’s ethological needs. Thus, the brain uses cost functions as a key driving force of its development, much as modern machine learning systems do.

To understand the basis of these claims, we must now delve into the details of how the brain might efficiently perform credit assignment throughout large, multi-layered networks, in order to optimize complex functions. We argue that the brain uses several different types of optimization to solve distinct problems. In some structures, it may use genetic pre-specification of circuits for problems that require only limited learning based on data, or it may exploit local optimization to avoid the need to assign credit through many layers of neurons. It may also use a host of proposed circuit structures that would allow it to actually perform, in effect, backpropagation of errors through a multi-layer network, using biologically realistic mechanisms – a feat that had once been widely believed to be biologically implausible (Crick, 1989; Stork, 1989). Potential such mechanisms include circuits that literally backpropagate error derivatives in the manner of conventional backpropagation, as well as circuits that provide other efficient means of approximating the effects of backpropagation, i.e., of rapidly computing the approximate gradient of a cost function relative to any given connection weight in the network. Lastly, the brain may use algorithms that exploit specific aspects of neurophysiology – such as spike timing dependent plasticity, dendritic computation, local excitatory-inhibitory networks, or other properties – as well as the integrated nature of higher-level brain systems. Such mechanisms promise to allow learning capabilities that go even beyond those of current backpropagation networks.

2.1 Local self-organization and optimization without multi-layer credit assignment

Not all learning requires a general-purpose optimization mechanism like gradient descent5. Many theories of cortex (George and Hawkins, 2009; Kappel et al., 2014) emphasize potential self-organizing and unsupervised learning properties that may obviate the need for multi-layer backpropagation as such. Hebbian plasticity, which adjusts weights according to correlations in pre-synaptic and post-synaptic activity, is well established6. Various versions of Hebbian plasticity (Miller and MacKay, 1994), e.g., with nonlinearities (Brito and Gerstner, 2016), can give rise to different forms of correlation and competition between neurons, leading to the self-organized formation of ocular dominance columns, self-organizing maps and orientation columns (Ferster and Miller, 2003; Miller et al., 1989). Often these types of local self-organization can also be viewed as optimizing a cost function: for example, certain forms of Hebbian plasticity can be viewed as extracting the principal components of the input, which minimizes a reconstruction error (Pehlevan and Chklovskii, 2015).

To generate complex temporal patterns, the brain may also implement other forms of learning that do not require any equivalent of full back-propagation through a multilayer network. For example, “liquid-” (Maass et al., 2002) or “echo-state machines” (Jaeger and Haas, 2004) are randomly connected recurrent networks that form a basis set (also known as a “reservoir”) of random filters, which can be harnessed for learning with tunable readout weights. Variants exhibiting chaotic, spontaneous dynamics can even be trained by feeding back readouts into the network and suppressing the chaotic activity (Sussillo and Abbott, 2009). Learning only the readout layer makes the optimization problem much simpler (indeed, equivalent to regression for supervised learning). Additionally, echo state networks can be trained by reinforcement learning as well as supervised learning (Bush, 2007; Hoerzer et al., 2014). Reservoirs of random nonlinear filters are one interpretation of the diverse, high-dimensional, mixed-selectivity tuning properties of many neurons, e.g. in the prefrontal cortex (Enel et al., 2016). Other variants of learning rules that modify only a fraction of the synapses inside a random network are being developed as models of biological working memory and sequence generation (Rajan et al., 2016).

2.2 Biological implementation of optimization

We argue that the above mechanisms of local self-organization are likely insufficient to account for the brain’s powerful learning performance (Brea and Gerstner, 2016). To elaborate on the need for an efficient means of gradient computation in the brain, we will first place backpropagation into its computational context (Hinton, 1989; Baldi and Sadowski, 2015). Then we will explain how the brain could plausibly implement approximations of gradient descent.

2.2.1 The need for efficient gradient descent in multi-layer networks

The simplest mechanism to perform cost function optimization is sometimes known as the “twiddle” algorithm or, more technically, as “serial perturbation”. This mechanism works by perturbing (i.e., “twiddling”), with a small increment, a single weight in the network, and verifying improvement by measuring whether the cost function has decreased compared to the network’s performance with the weight unperturbed. If improvement is noticeable, the perturbation is used as a direction of change to the weight; otherwise, the weight is changed in the opposite direction (or not changed at all). Serial perturbation is therefore a method of “coordinate descent” on the cost, but it is slow and requires global coordination: each synapse in turn is perturbed while others remain fixed.

Weight perturbation (or parallel perturbation) perturbs all of the weights in the network at once. It is able to optimize small networks to perform tasks but generally suffers from high variance. That is, the measurement of the gradient direction is noisy and changes drastically from perturbation to perturbation because a weight’s influence on the cost is masked by the changes of all other weights, and there is only one scalar feedback signal indicating the change in the cost7. Weight perturbation is dramatically inefficient for large networks. In fact, parallel and serial perturbation learn at approximately the same rate if the time measure counts the number of times the network propagates information from input to output (Werfel et al., 2005).

Some efficiency gain can be achieved by perturbing neural activities instead of synaptic weights, acknowledging the fact that any long-range effect of a synapse is mediated through a neuron. Like weight perturbation and unlike serial perturbation, minimal global coordination is needed: each neuron only needs to receive a feedback signal indicating the global cost. The variance of node perturbation’s gradient estimate is far smaller than that of weight perturbation under the assumptions that either all neurons or all weights, respectively, are perturbed and that they are perturbed at the same frequency. In this case, node perturbation’s variance is proportional to the number of cells in the network, not the number of synapses.

All of these approaches are slow either due to the time needed for serial iteration over all weights or the time needed for averaging over low signal-to-noise ratio gradient estimates. To their credit however, none of these approaches requires more than knowledge of local activities and the single global cost signal. Real neural circuits in the brain have mechanisms (e.g., diffusible neuro-modulators) that appear to code the signals relevant to implementing those algorithms. In many cases, for example in reinforcement learning, the cost function, which is computed based on interaction with an unknown environment, cannot be differentiated directly, and an agent has no choice but to deploy clever twiddling to explore at some level of the system (Williams, 1992).

Backpropagation, in contrast, works by computing the sensitivity of the cost function to each weight based on the layered structure of the system. The derivatives of the cost function with respect to the last layer can be used to compute the derivatives of the cost function with respect to the penultimate layer, and so on, all the way down to the earliest layers8. Backpropagation can be computed rapidly, and for a single input-output pattern, it exhibits no variance in its gradient estimate. The backpropagated gradient has no more noise for a large system than for a small system, so deep and wide architectures with great computational power can be trained efficiently.

2.2.2 Biologically plausible approximations of gradient descent

To permit biological learning with efficiency approaching that of machine learning methods, some provision for more sophisticated gradient propagation may be suspected. Contrary to what was once a common assumption, there are now many proposed “biologically plausible” mechanisms by which a neural circuit could implement optimization algorithms that, like back-propagation, can efficiently make use of the gradient. These include Generalized Recirculation (O’Reilly, 1996), Contrastive Hebbian Learning (Xie and Seung, 2003), random feedback weights together with synaptic homeostasis (Lillicrap et al., 2014; Liao et al., 2015), spike timing dependent plasticity (STDP) with iterative inference and target propagation (Scellier and Bengio, 2016; Bengio et al., 2015a), complex neurons with backpropagating action-potentials (Körding and König, 2000), and others (Balduzzi et al., 2014). While these mechanisms differ in detail, they all invoke feedback connections that carry error phasically. Learning occurs by comparing a prediction with a target, and the prediction error is used to drive top-down changes in bottom-up activity.

As an example, consider O’Reilly’s temporally eXtended Contrastive Attractor Learning (XCAL) algorithm (O’Reilly et al., 2012, 2014b). Suppose we have a multilayer neural network with an input layer, an output layer, and a set of hidden layers in between. O’Reilly showed that the same functionality as backpropagation can be implemented by a bidirectional network with the same weights but symmetric connections. After computing the outputs using the forward connections only, we set the output neurons to the values they should have. The dynamics of the network then cause the hidden layers’ activities to evolve toward a stable attractor state linking input to output. The XCAL algorithm performs a type of local modified Hebbian learning at each synapse in the network during this process (O’Reilly et al., 2012). The XCAL Hebbian learning rule compares the local synaptic activity (pre x post) during the early phase of this settling (before the attractor state is reached) to the final phase (once the attractor state has been reached), and adjusts the weights in a way that should make the early phase reflect the later phase more closely. These contrastive Hebbian learning methods even work when the connection weights are not precisely symmetric (O’Reilly, 1996). XCAL has been implemented in biologically plausible conductance-based neurons and basically implements the backpropagation of error approach.

Approximations to backpropagation could also be enabled by the millisecond-scale timing of of neural activities (O’Reilly et al., 2014b). Spike timing dependent plasticity (STDP) (Markram et al., 1997), for example, is a feature of some neurons in which the sign of the synaptic weight change depends on the precise millisecond-scale relative timing of pre-synaptic and post-synaptic spikes. This is conventionally interpreted as Hebbian plasticity that measures the potential for a causal relationship between the pre-synaptic and post-synaptic spikes: a pre-synaptic spike could have contributed to causing a post-synaptic spike only if it occurs shortly beforehand9. To enable a backpropagation mechanism, Hinton has suggested an alternative interpretation: that neurons could encode the types of error derivatives needed for backpropagation in the temporal derivatives of their firing rates (Hinton, 2007, 2016). STDP then corresponds to a learning rule that is sensitive to these error derivatives (Bengio et al., 2015b; Xie and Seung, 2000). In other words, in an appropriate network context, STDP learning could give rise to a biological implementation of backpropagation10.

Another possible mechanism, by which biological neural networks could approximate back-propagation, is “feedback alignment” (Lillicrap et al., 2014; Liao et al., 2015). There, the feedback pathway in backpropagation, by which error derivatives at a layer are computed from error derivatives at the subsequent layer, is replaced by a set of random feedback connections, with no dependence on the forward weights. Subject to the existence of a synaptic normalization mechanism and approximate sign-concordance between the feedforward and feedback connections (Liao et al., 2015), this mechanism of computing error derivatives works nearly as well as backpropagation on a variety of tasks. In effect, the forward weights are able to adapt to bring the network into a regime in which the random backwards weights actually carry the information that is useful for approximating the gradient. This is a remarkable and surprising finding, and is indicative of the fact that our understanding of gradient descent optimization, and specifically of the mechanisms by which backpropagation itself functions, are still incomplete. In neuroscience, meanwhile, we find feedback connections almost wherever we find feed-forward connections, and their role is the subject of diverse theories (Callaway, 2004; Maass et al., 2007). It should be noted that feedback alignment as such does not specify exactly how neurons represent and make use of the error signals; it only relaxes a constraint on the transport of the error signals. Thus, feedback alignment is more a primitive that can be used in fully biological (approximate) implementations of backpropagation, than a fully biological implementation in its own right. As such, it may be possible to incorporate it into several of the other schemes discussed here.

The above “biological” implementations of backpropagation still lack some key aspects of biological realism. For example, in the brain, neurons tend to be either excitatory or inhibitory but not both, whereas in artificial neural networks a single neuron may send both excitatory and inhibitory signals to its downstream neurons. Fortunately, this constraint is unlikely to limit the functions that can be learned (Parisien et al., 2008; Tripp and Eliasmith, 2016). Other biological considerations, however, need to be looked at in more detail: the highly recurrent nature of biological neural networks, which show rich dynamics in time, and the fact that most neurons in mammalian brains communicate via spikes. We now consider these two issues in turn.

Temporal credit assignment: The biological implementations of backpropagation proposed above, while applicable to feedforward networks, do not give a natural implementation of “backpropagation through time” (BPTT) (Werbos, 1990) for recurrent networks, which is widely used in machine learning for training recurrent networks on sequential processing tasks. BPTT “unfolds” a recurrent network across multiple discrete time steps and then runs backpropagation on the unfolded network to assign credit to particular units at particular time steps11. While the network unfolding procedure of BPTT itself does not seem biologically plausible, to our intuition, it is unclear to what extent temporal credit assignment is truly needed (Ollivier and Charpiat, 2015) for learning particular temporally extended tasks.

If the system is given access to appropriate memory stores and representations (Buonomano and Merzenich, 1995; Gershman et al., 2012, 2014) of temporal context, this could potentially mitigate the need for temporal credit assignment as such – in effect, memory systems could “spatialize” the problem of temporal credit assignment12. For example, memory networks (Weston et al., 2014) store everything by default up to a certain buffer size, eliminating the need to perform credit assignment over the write-to-memory events, such that the network only needs to perform credit assignment over the read-from-memory events. In another example, certain network architectures that are superficially very deep, but which possess particular types of “skip connections”, can actually be seen as ensembles of comparatively shallow networks (Veit et al., 2016); applied in the time domain, this could limit the need to propagate errors far backwards in time. Other, similar specializations or higher-levels of structure could, potentially, further ease the burden on credit assignment.

Can generic recurrent networks perform temporal credit assignment in in a way that is more biologically plausible than BPTT? Indeed, new discoveries are being made about the capacity for supervised learning in continuous-time recurrent networks with more realistic synapses and neural integration properties. In internal FORCE learning (Sussillo and Abbott, 2009), internally generated random fluctuations inside a chaotic recurrent network are adjusted to provide feedback signals that drive weight changes internal to the network while the outputs are clamped to desired patterns. This is made possible by a learning procedure that rapidly adjusts the network output to a state where it is close to the clamped values, and exerts continuous control to keep this difference small throughout the learning process13. This procedure is able to control and exploit the chaotic dynamical patterns that are spontaneously generated by the network.

Werbos has proposed in his “error critic” that an online approximation to BPTT can be achieved by learning to predict the backward-through-time gradient signal (costate) in a manner analogous to the prediction of value functions in reinforcement learning (Si, 2004). This kind of idea was recently applied in (Jaderberg et al., 2016) to allow decoupling of different parts of a network during training and to facilitate back-propagation through time. Broadly, we are only beginning to understand how neural activity can itself represent the time variable (Finnerty and Shadlen, 2015; Xu et al., 2014)14, and how recurrent networks can learn to generate trajectories of population activity over time (Liu and Buonomano, 2009). Moreover, as we discuss below, a number of cortical models also propose means, other than BPTT, by which networks could be trained on sequential prediction tasks, even in an online fashion (Cui et al., 2015; O’Reilly et al., 2014b; Brea et al., 2016). A broad range of ideas can be used to approximate BPTT in more realistic ways.

Spiking networks: It has been difficult to apply gradient descent learning directly to spiking neural networks15,16, although there do exist learning rules for doing so in specific representational contexts and network structures (Bekolay et al., 2013). A number of optimization procedures have been used to generate, indirectly, spiking networks which can perform complex tasks, by performing optimization on a continuous representation of the network dynamics and embedding variables into high-dimensional spaces with many spiking neurons representing each variable (Abbott et al., 2016; DePasquale et al., 2016; Komer and Eliasmith, 2016; Thalmeier et al., 2015). The use of recurrent connections with multiple timescales can remove the need for backpropagation in the direct training of spiking recurrent networks (Bourdoukan and Denève, 2015). Fast connections maintain the network in a state where slow connections have local access to a global error signal. While the biological realism of these methods is still unknown, they all allow connection weights to be learned in spiking networks.

These and other novel learning procedures illustrate the fact that we are only beginning to understand the connections between the temporal dynamics of biologically realistic networks, and mechanisms of temporal and spatial credit assignment. Nevertheless, we argue here that existing evidence suggests that biologically plausible neural networks can solve these problems – in other words, it is possible to efficiently optimize complex functions of temporal history in the context of spiking networks of biologically realistic neurons. In any case, there is little doubt that spiking recurrent networks using realistic population coding schemes can, with an appropriate choice of connection weights, compute complicated, cognitively relevant functions17. The question is how the developing brain efficiently learns such complex functions.

2.3 Other principles for biological learning

The brain has mechanisms and structures that could support learning mechanisms different from typical gradient-based optimization algorithms employed in artificial neural networks.

2.3.1 Exploiting biological neural mechanisms

The complex physiology of individual biological neurons may not only help explain how some form of efficient gradient descent could be implemented within the brain, but also could provide mechanisms for learning that go beyond back-propagation. This suggests that the brain may have discovered mechanisms of credit assignment quite different from those dreamt up by machine learning.

One such biological primitive is dendritic computation, which could impact prospects for learning algorithms in several ways. First, real neurons are highly nonlinear (Antic et al., 2010), with the dendrites of each single neuron implementing18 something computationally similar to a three-layer neural network (Mel, 1992)19. Individual neurons thus should not be regarded as single “nodes” but as multi-component subnetworks. Second, when a neuron spikes, its action potential propagates back from the soma into the dendritic tree. However, it propagates more strongly into the branches of the dendritic tree that have been active (Williams and Stuart, 2000), potentially simplifying the problem of credit assignment (Körding and König, 2000). Third, neurons can have multiple somewhat independent dendritic compartments, as well as a somewhat independent somatic compartment, which means that the neuron should be thought of as storing more than one variable. Thus, there is the possibility for a neuron to store both its activation itself, and the error derivative of a cost function with respect to its activation, as required in backpropagation, and biological implementations of backpropagation based on this principle have been proposed (Körding and König, 2001; Schiess et al., 2016)20. Overall, the implications of dendritic computation for credit assignment in deep networks are only beginning to be considered21. But it is clear that the types of bi-directional, non-linear, multi-variate interactions that are possible inside a single neuron could support gradient descent learning or other powerful optimization mechanisms.

Beyond dendritic computation, diverse mechanisms (Marblestone and Boyden, 2014) like retrograde (post-synaptic to pre-synaptic) signals using cannabinoids (Wilson and Nicoll, 2001), or rapidly-diffusing gases such as nitric oxide (Arancio et al., 1996), are among many that could enable learning rules that go beyond conventional conceptions of backpropagation. Harris has suggested (Harris, 2008; Lewis and Harris, 2014) how slow, retroaxonal (i.e., from the outgoing synapses back to the parent cell body) transport of molecules like neurotrophins could allow neural networks to implement an analog of an exchangeable currency in economics, allowing networks to self-organize to efficiently provide information to downstream “consumer” neurons that are trained via faster and more direct error signals. The existence of these diverse mechanisms may call into question traditional, intuitive notions of “biological plausibility” for learning algorithms.

Another potentially important biological primitive is neuromodulation. The same neuron or circuit can exhibit different input-output responses and plasticity depending on a global circuit state, as reflected by the concentrations of various neuromodulators like dopamine, serotonin, norepinephrine, acetylcholine, and hundreds of different neuropeptides such as opiods (Bargmann and Marder, 2013; Bargmann, 2012). These modulators interact in complex and cell-type-specific ways to influence circuit function. Interactions with glial cells also play a role in neural signaling and neuromodulation, leading to the concept of “tripartite” synapses that include a glial contribution (Perea et al., 2009). Modulation could have many implications for learning. First, modulators can be used to gate synaptic plasticity on and off selectively in different areas and at different times, allowing precise, rapidly updated orchestration of where and when cost functions are applied. Furthermore, it has been argued that a single neural circuit can be thought of as multiple overlapping circuits with modulation switching between them (Bargmann and Marder, 2013; Bargmann, 2012). In a learning context, this could potentially allow sharing of synaptic weight information between overlapping circuits. Dayan (2012) discusses further computational aspects of neuromodulation. Overall, neuromodulation seems to expand the range of possible algorithms that could be used for optimization.

2.3.2 Learning in the cortical sheet

A number of models attempt to explain cortical learning on the basis of specific architectural features of the 6-layered cortical sheet. These models generally agree that a primary function of the cortex is some form of unsupervised learning via prediction (O’Reilly et al., 2014b; Brea et al., 2016)22. Some cortical learning models are explicit attempts to map cortical structure onto the framework of message-passing algorithms for Bayesian inference (George and Hawkins, 2009; Dean, 2005; Lee and Mumford, 2003), while others start with particular aspects of cortical neurophysiology and seek to explain those in terms of a learning function, or in terms of a computational function, e.g., hierarchical clustering (Rodriguez et al., 2004). For example, the nonlinear and dynamical properties of cortical pyramidal neurons – the principal excitatory neuron type in cortex (Shepherd, 2014) – are of particular interest here, especially because these neurons have multiple dendritic zones that are targeted by different kinds of projections, which may allow the pyramidal neuron to make comparisons of top-down and bottom-up inputs23.

Other aspects of the laminar cortical architecture could be crucial to how the brain implements learning. Local inhibitory neurons targeting particular dendritic compartments of the L5 pyramidal could be used to exert precise control over when and how the relevant feedback signals and associative mechanisms are utilized. Notably, local inhibitory networks could also give rise to competition (Petrov et al., 2010) between different representations in the cortex, perhaps allowing one cortical column to suppress others nearby, or perhaps even to send more sophisticated messages to gate the state transitions of its neighbors (Bach and Herger, 2015). Moreover, recurrent connectivity with the thalamus, structured bursts of spiking, and cortical oscillations (not to mention other mechanisms like neuromodulation) could control the storage of information over time, to facilitate learning based on temporal prediction. These concepts begin to suggest preliminary, exploratory models for how the detailed anatomy and physiology of the cortex could be interpreted within a machine-learning framework that goes beyond backpropagation. But these are early days: we still lack detailed structural/molecular and functional maps of even a single local cortical microcircuit.

2.3.3 One-shot learning

Human learning is often one-shot: it can take just a single exposure to a stimulus to never forget it, as well as to generalize from it to new examples. One way of allowing networks to have such properties is what is described by I-theory, in the context of learning invariant representations for object recognition (Anselmi et al., 2015). Instead of training via gradient descent, image templates are stored in the weights of simple-complex cell networks while objects undergo transformations, similar to the use of stored templates in HMAX (Serre et al., 2007). The theories then aim to show that you can invariantly and discriminatively represent objects using a single sample, even of a new class (Anselmi et al., 2015)24.

Additionally, the nervous system may have a way of quickly storing and replaying sequences of events. This would allow the brain to move an item from episodic memory into a long-term memory stored in the weights of a cortical network (Ji and Wilson, 2007), by replaying the memory over and over. This solution effectively uses many iterations of weight updating to fully learn a single item, even if one has only been exposed to it once. Alternatively, the brain could rapidly store an episodic memory and then retrieve it later without the need to perform slow gradient updates, which has proven to be useful for fast reinforcement learning in scenarios with limited available data (Blundell et al., 2016).

Finally, higher-level systems in the brain may be able to implement Bayesian learning of sequential programs, which is a powerful means of one-shot learning (Lake et al., 2015). This type of cognition likely relies on an interaction between multiple brain areas such as the prefrontal cortex and basal ganglia.

These potential substrates of one-shot learning rely on mechanisms other than simple gradient descent. It should be noted, though, that recent architectural advances, including specialized spatial attention and feedback mechanisms (Rezende et al., 2016), as well as specialized memory mechanisms (Santoro et al., 2016), do allow some types of one-shot generalization to be driven by backpropagation-based learning.

2.3.4 Active learning

Human learning is often active and deliberate. It seems likely that, in human learning, actions are chosen so as to generate interesting training examples, and sometimes also to test specific hypotheses. Such ideas of active learning and “child as scientist” go back to Piaget and have been elaborated more recently (Gopnik et al., 2000). We want our learning to be based on maximally informative samples, and active querying of the environment (or of internal subsystems) provides a way route to this.

At some level of organization, of course, it would seem useful for a learning system to develop explicit representations of its uncertainty, since this can be used to guide the system to actively seek the information that would reduce its uncertainty most quickly. Moreover, there are population coding mechanisms that could support explicit probabilistic computations (Ma et al., 2006; Zemel and Dayan, 1997; Gershman and Beck, 2016; Eliasmith and Martens, 2011; Rao, 2004; Sahani and Dayan, 2003). Yet it is unclear to what extent and at what levels the brain uses an explicitly probabilistic framework, or to what extent probabilistic computations are emergent from other learning processes (Orhan and Ma, 2016)25 ,26.

Standard gradient descent does not incorporate any such adaptive sampling mechanism, e.g., it does not deliberately sample data so as to maximally reduce its uncertainty. Interestingly, however, stochastic gradient descent can be used to generate a system that samples adaptively (Bouchard et al., 2015; Alain et al., 2015). In other words, a system can learn, by gradient descent, how to choose its own input data samples in order to learn most quickly from them by gradient descent.

Ideally, the learner learns to choose actions that will lead to the largest improvements in its prediction or data compression performance (Schmidhuber, 2010). In (Schmidhuber, 2010), this is done in the framework of reinforcement learning, and incorporates a mechanisms for the system to measure its own rate of learning. In other words, it is possible to reinforcement-learn a policy for selecting the most interesting inputs to drive learning. Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Guez et al., 2012; Sun et al., 2011).

These approaches achieve optimality in an arbitrary, abstract environment. But of course, evolution may also encode its implicit knowledge of the organism’s natural environment, the behavioral goals of the organism, and the developmental stages and processes which occur inside the organism, as priors or heuristics27 which would further constrain the types of adaptive sampling that are optimal in practice. For example, simple heuristics like seeking certain perceptual signatures of novelty, or more complex heuristics like monitoring situations that other people seem to find interesting, might be good ways to bias sampling of the environment so as to learn more quickly. Other such heuristics might be used to give internal brain systems the types of training data that will be most useful to those particular systems at any given developmental stage.

We are only beginning to understand how active learning might be implemented in the brain. We speculate that multiple mechanisms, specialized to different brain systems and spatio-temporal scales, could be involved. The above examples suggest that at least some such mechanisms could be understood from the perspective of optimizing cost functions.

2.4 Differing biological requirements for supervised and reinforcement learning

We have suggested ways in which the brain could implement learning mechanisms of comparable power to backpropagation. But in many cases, the system may be more limited by the available training signals than by the optimization process itself. In machine learning, one distinguishes supervised learning, reinforcement learning and un-supervised learning, and the training data limitation manifests differently in each case.

Both supervised and reinforcement learning require some form of teaching signal, but the nature of the teaching signal in supervised learning is different from that in reinforcement learning. In supervised learning, the trainer provides the entire vector of errors for the output layer and these are back-propagated to compute the gradient: a locally optimal direction in which to update all of the weights of a potentially multilayer and/or recurrent network. In reinforcement learning, however, the trainer provides a scalar evaluation signal, but this is not sufficient to derive a low-variance gradient. Hence, some form of trial and error twiddling must be used to discover how to increase the evaluation signal. Consequently, reinforcement learning is generally much less efficient than supervised learning.

Reinforcement learning in shallow networks is simple to implement biologically. For reinforcement learning of a deep network to be biologically plausible, however, we need a more powerful learning mechanism, since we are learning based on a more limited evaluation signal than in the supervised case: we do not have the full target pattern to train towards. Nevertheless, approximations of gradient descent can be achieved in this case, and there are cases in which the scalar evaluation signal of reinforcement learning can be used to efficiently update a multi-layer network by gradient descent. The “attention-gated reinforcement learning” (AGREL) networks of (Stnior et al., 2013; Brosch et al., 2015; Roelfsema and van Ooyen, 2005), and variants like KickBack (Balduzzi, 2014), give a way to compute an approximation to the full gradient in a reinforcement learning context using a feedback-based attention mechanism for credit assignment within the multi-layer network. The feedback pathway, together with a diffusible reward signal, together gate plasticity. For networks with more than three layers, this gives rise to a model based on columns containing parallel feedforward and feedback pathways (Roelfsema and van Ooyen, 2005), and for recurrent networks that settle into attractor states it gives a reinforcement-trained version (Brosch et al., 2015) of the Almeida/Pineda recurrent backpropagation algorithm (Pineda, 1987). The process is still not as efficient or generic as backpropagation, but it seems that this form of feedback can make reinforcement learning in multi-layer networks more efficient than a naive node perturbation or weight perturbation approach.

The machine-learning field has recently been tackling the question of credit assignment in deep reinforcement learning. Deep Q-learning (Mnih et al., 2015) demonstrates reinforcement learning in a deep network, wherein most of the network is trained via backpropagation. In regular Q learning, we define a function Q, which estimates the best possible sum of future rewards (the return) if we are in a given state and take a given action. In deep Q learning, this function is approximated by a neural network that, in effect, estimates action-dependent returns in a given state. The network is trained using backpropagation of local errors in Q estimation, using the fact that the return decomposes into the current reward plus the discounted estimate of future return at the next moment. During training, as the agent acts in the environment, a series of loss functions is generated at each step, defining target patterns that can be used as the supervision signal for back-propagation. As Q is a highly nonlinear function of the state, tricks are needed to make deep Q learning efficient and stable, including experience replay and a particular type of mini-batch training. It is also necessary to store the outputs from the previous iteration (or clone the entire network) in evaluating the loss function for the subsequent iteration28.

This process for generating learning targets provides a kind of bridge between reinforcement learning and efficient backpropagation-based gradient descent learning29. Importantly, only temporally local information is needed making the approach relatively compatible with what we know about the nervous system.

Even given these advances, a key remaining issue in reinforcement learning is the problem of long timescales, e.g., learning the many small steps needed to navigate from London to Chicago. Many of the formal guarantees of reinforcement learning (Williams and Baird, 1993), for example, suggest that the difference between an optimal policy and the learned policy becomes increasingly loose as the discount factor shifts to take into account reward at longer timescales. Although the degree of optimality of human behavior is unknown, people routinely engage in adaptive behaviors that can take hours or longer to carry out, by using specialized processes like prospective memory to “remember to remember” relevant variables at the right times, permitting extremely long timescales of coherent action. Machine learning has not yet developed methods to deal with such a wide range of timescales and scopes of hierarchical action. Below we discuss ideas of hierarchical reinforcement learning that may make use of callable procedures and sub-routines, rather than operating explicitly in a time domain.

As we will discuss below, some form of deep reinforcement learning may be used by the brain for purposes beyond optimizing global rewards, including the training of local networks based on diverse internally generated cost functions. Scalar reinforcement-like signals are easy to compute, and easy to deliver to other areas, making them attractive mechanistically. If the brain does employ internally computed scalar reward-like signals as a basis for cost functions, it seems likely that it will have found an efficient means of reinforcement-based training of deep networks, but it is an open question whether an analog of deep Q networks, AGREL, or some other mechanism entirely, is used in the brain for this purpose. Moreover, as we will discuss further below, it is possible that reinforcement-type learning is made more efficient in the context of specialized brain systems like short term memories, replay mechanisms, and hierarchically organized control systems. These specialized systems could reduce reliance on a need for powerful credit assignment mechanisms for reinforcement learning. Finally, if the brain uses a diversity of scalar reward-like signals to implement different cost functions, then it may need to mediate delivery of those signals via a comparable diversity of molecular substrates. The great diversity of neuromodulatory signals, e.g., neuropeptides, in the brain (Bargmann, 2012; Bargmann and Marder, 2013) makes such diversity quite plausible, and moreover, the brain may have found other, as yet unknown, mechanisms of diversifying reward-like signaling pathways and enabling them to act independently of one another.

3. The cost functions are diverse across brain areas and time

In the last section, we argued that the brain can optimize functions. This raises the question of what functions it optimizes. Of course, in the brain, a cost function will itself be created (explicitly or implicitly) by a neural network shaped by the genome. Thus, the cost function used to train a given sub-network in the brain is a key innate property that can be built into the system by evolution. It may be much cheaper in biological terms to specify a cost function that allows the rapid learning of the solution to a problem than to specify the solution itself.

In Hypothesis 2, we proposed that the brain optimizes not a single “end-to-end” cost function, but rather a diversity of internally generated cost functions specific to particular brain functions30. To understand how and why the brain may use a diversity of cost functions, it is important to distinguish the differing types of cost functions that would be needed for supervised, unsupervised and reinforcement learning. We can also seek to identify types of cost functions that the brain may need to generate from a functional perspective, and how each may be implemented as supervised, unsupervised, reinforcement-based or hybrid systems.

3.1 How cost functions may be represented and applied

What additional circuitry is required to actually impose a cost function on an optimizing network? In the most familiar case, supervised learning may rely on computing a vector of errors at the output of a network, which will rely on some comparator circuitry31 to compute the difference between the network outputs and the target values. This difference could then be backpropagated to earlier layers. An alternative way to impose a cost function is to “clamp” the output of the network, forcing it to occupy a desired target state. Such clamping is actually assumed in some of the putative biological implementations of back-propagation described above, such as XCAL and target propagation. Alternatively, as described above, scalar reinforcement signals are attractive as internally-computed cost functions, but using them in deep networks requires special mechanisms for credit assignment.

In unsupervised learning, cost functions may not take the form of externally supplied training or error signals, but rather can be built into the dynamics inherent to the network itself, i.e., there may be no need for a separate circuit to compute and impose a cost function on the network. For example, specific spike-timing-dependent and homeostatic plasticity rules have been shown to give rise to gradient descent on a prediction error in recurrent neural networks (Galtier and Wainrib, 2013). Thus, specific unsupervised objectives could be implemented implicitly through specific local network dynamics32 and plasticity rules inside a network without explicit computation of cost function, nor explicit propagation of error derivatives.

Alternatively, explicit cost functions could be computed, delivered to an optimizing network, and used for unsupervised learning, following a variety of principles being discovered in machine learning (e.g., (Radford et al., 2015; Lotter et al., 2015)). These networks rely on backpropagation as the sole learning rule, and typically find a way to encode the desired cost function into the error derivatives which are backpropagated. For example, prediction errors naturally give rise to error signals for unsupervised learning, as do reconstruction errors in autoencoders, and these error signals can also be augmented with additional penalty or regularization terms that enforce objectives like sparsity or continuity, as described below. Then these error derivatives can be propagated throughout the network via standard back-propagation. In such systems, the objective function and the optimization mechanism can thus be mixed and matched modularly. In the next sections, we elaborate on these and other means of specifying and delivering cost functions in different learning contexts.

3.2 Cost functions for unsupervised learning

There are many objectives that can be optimized in an unsupervised context, to accomplish different kinds of functions or guide a network to form particular kinds of representations.

3.2.1 Matching the statistics of the input data using generative models

In one common form of unsupervised learning, higher brain areas attempt to produce samples that are statistically similar to those actually seen in lower layers. For example, the wake-sleep algorithm (Hinton et al., 1995) requires the sleep mode to sample potential data points whose distribution should then match the observed distribution. Unsupervised pre-training of deep networks is an instance of this (Erhan and Manzagol, 2009), typically making use of a stacked auto-encoder framework. Similarly, in target propagation (Bengio, 2014), a top-down circuit, together with lateral information, has to produce data that directs the local learning of a bottom-up circuit and vice-versa. Ladder autoencoders make use of lateral connections and local noise injection to introduce an unsupervised cost function, based on internal reconstructions, that can be readily combined with supervised cost functions defined on the networks top layer outputs (Valpola, 2015). Compositional generative models generate a scene from discrete combinations of template parts and their transformations (Wang and Yuille, 2014), in effect performing a rendering of a scene based on its structural description. Hinton and colleagues have also proposed cortical “capsules” (Tang et al., 2013, 2012; Hinton et al., 2011) for compositional inverse rendering. The network can thus implement a statistical goal that embodies some understanding of the way that the world produces samples33.

Learning rules for generative models have historically involved local message passing of a form quite different from backpropagation, e.g., in a multi-stage process that first learns one layer at a time and then fine-tunes via the wake-sleep algorithm (Hinton et al., 2006). Message-passing implementations of probabilistic inference have also been proposed as an explanation and generalization of deep convolutional networks (Patel et al., 2015; Chen et al., 2014). Various mappings of such processes onto neural circuitry have been attempted (Sountsov and Miller, 2015; George and Hawkins, 2009; Lee and Yuille, 2011), and related models (Makin et al., 2013, 2016) have been used to account for optimal multi-sensory integration in the brain. Feedback connections tend to terminate in distinct layers of cortex relative to the feedforward ones (Felleman and Van Essen, 1991; Callaway, 2004) making the idea of separate but interacting networks for recognition and generation potentially attractive34. Interestingly, such sub-networks might even be part of the same neuron and map onto “apical” versus “basal” parts of the dendritic tree (Körding and König, 2001; Urbanczik and Senn, 2014).

Generative models can also be trained via backpropagation. Recent advances have shown how to perform variational approximations to Bayesian inference inside backpropagation-based neural networks (Kingma and Welling, 2013), and how to exploit this to create generative models (Eslami et al., 2016; Gregor et al., 2015; Goodfellow et al., 2014a; Radford et al., 2015). Through either explicitly statistical or gradient descent based learning, the brain can thus obtain a probabilistic model that simulates features of the world.

3.2.2 Cost functions that approximate properties of the world

A perceiving system should exploit statistical regularities in the world that are not present in an arbitrary dataset or input distribution. For example, objects are sparse, at least in certain representations: there are far fewer objects than there are potential places in the world, and of all possible objects there is only a small subset visible at any given time. As such, we know that the output of an object recognition system must have sparse activations. Building the assumption of sparseness into simulated systems replicates a number of representational properties of the early visual system (Olshausen and Field, 1997; Rozell et al., 2008), and indeed the original paper on sparse coding obtained sparsity by gradient descent optimization of a cost function (Olshausen and Field, 1996). A range of unsupervised machine learning techniques, such as the sparse autoencoders (Le et al., 2011) used to discover cats in YouTube videos, build sparseness into neural networks. Building in such spatio-temporal sparseness priors should serve as an “inductive bias” (Mitchell, 1980) that can accelerate learning.

But we know much more about the regularities of objects. As young babies, we already know (Bremner et al., 2015) that objects tend to persist over time. The emergence or disappearance of an object from a region of space is a rare event. Moreover, object locations and configurations tend to be coherent in time. We can formulate this prior knowledge as a cost function, for example by penalizing representations which are not temporally continuous. This idea of continuity is used in a great number of artificial neural networks and related models (Mobahi et al., 2009; Wiskott and Sejnowski, 2002; Földiák, 2008). Imposing continuity within certain models gives rise to aspects of the visual system including complex cells (Körding et al., 2004), specific properties of visual invariance (Isik et al., 2012), and even other representational properties such as the existence of place cells (Wyss et al., 2006; Franzius et al., 2007). Unsupervised learning mechanisms that maximize temporal coherence or slowness are increasingly used in machine learning35.

We also know that objects tend to undergo predictable sequences of transformations, and it is possible to build this assumption into unsupervised neural learning systems (George and Hawkins, 2009). The minimization of prediction error explains a number of properties of the nervous system (Friston and Stephan, 2007; Huang and Rao, 2011), and biologically plausible theories are available for how cortex could learn using prediction errors by exploiting temporal differences (O’Reilly et al., 2014b) or top-down feedback (George and Hawkins, 2009). In one implementation, a system can simply predict the next input delivered to the system and can then use the difference between the actual next input and the predicted next input as a full vectorial error signal for supervised gradient descent. Thus, rather than optimization of prediction error being implicitly implemented by the network dynamics, the prediction error is used as an explicit cost function in the manner of supervised learning, leading to error derivatives which can be back-propagated. Then, no special learning rules beyond simple backpropagation are needed. This approach has recently been advanced within machine learning (Lotter et al., 2015, 2016). Recently, combining such prediction-based learning with a specific gating mechanism has been shown to lead to unsupervised learning of disentangled representations (Whitney et al., 2016). Neural networks can also be designed to learn to invert spatial transformations (Jaderberg et al., 2015b). Statistically describing transformations or sequences is thus an unsupervised way of learning representations.

Furthermore, there are multiple modalities of input to the brain. Each sensory modality is primarily connected to one part of the brain36. But higher levels of cortex in each modality are heavily connected to the other modalities. This can enable forms of self-supervised learning: with a developing visual understanding of the world we can predict its sounds, and then test those predictions with the auditory input, and vice versa. The same is true about multiple parts of the same modality: if we understand the left half of the visual field, it tells us an awful lot about the right. Indeed, we can use observations of one part of a visual scene to predict the contents of other parts (van den Oord et al., 2016; Noroozi and Favaro, 2016), and optimize a cost function that reflects the discrepancy. Maximizing mutual information is a natural way of improving learning (Becker and Hinton, 1992; Mohamed and Rezende, 2015), and there are many other ways in which multiple modalities or processing streams could mutually train one another. This way, each modality effectively produces training signals for the others37. Evidence from psychophysics suggests that some kind of training via detection of sensory conflicts may be occurring in children (Nardini et al., 2010).

3.3 Cost functions for supervised learning

In what cases might the brain use supervised learning, given that it requires the system to “al-ready know” the exact target pattern to train towards? One possibility is that the brain can store records of states that led to good outcomes. For example, if a baby reaches for a target and misses, and then tries again and successfully hits the target, then the difference in the neural representations of these two tries reflects the direction in which the system should change. The brain could potentially use a comparator circuit to directly compute this vectorial difference in the neural population codes and then apply this difference vector as an error signal.

Another possibility is that the brain uses supervised learning to implement a form of “chunking”, i.e., a consolidation of something the brain already knows how to do: routines that are initially learned as multi-step, deliberative procedures could be compiled down to more rapid and automatic functions by using supervised learning to train a network to mimic the overall input-output behavior of the original multi-step process. Such a process is assumed to occur in cognitive models like ACT-R (Servan-Schreiber and Anderson, 1990), and methods for compressing the knowledge in neural networks into smaller networks are also being developed (Ba and Caruana, 2014). Thus supervised learning can be used to train a network to do in “one step” what would otherwise require long-range routing and sequential recruitment of multiple systems.

3.4 Repurposing reinforcement learning for diverse internal cost functions

Certain generalized forms of reinforcement learning may be ubiquitous throughout the brain. Such reinforcement signals may be repurposed to optimize diverse internal cost functions. These internal cost functions could be specified at least in part by genetics.

Some brain systems such as in the striatum appear to learn via some form of temporal difference reinforcement learning (Tesauro, 1995; Foster et al., 2000). This is reinforcement learning based on a global value function (O’Reilly et al., 2014a) that predicts total future reward or utility for the agent. Reward-driven signaling is not restricted to the striatum, and is present even in primary visual cortex (Chubykin et al., 2013; Stnior et al., 2013). Remarkably, the reward signaling in primary visual cortex is mediated in part by glial cells (Takata et al., 2011), rather than neurons, and involves the neurotransmitter acetylcholine (Chubykin et al., 2013; Hangya et al., 2015). On the other hand, some studies have suggested that visual cortex learns the basics of invariant object recognition in the absence of reward (Li and Dicarlo, 2012), perhaps using reinforcement only for more refined perceptual learning (Roelfsema et al., 2010).

But beyond these well-known global reward signals, we argue that the basic mechanisms of reinforcement learning may be widely re-purposed to train local networks using a variety of internally generated error signals. These internally generated signals may allow a learning system to go beyond what can be learned via standard un-supervised methods, effectively guiding or steering the system to learn specific features or computations (Ullman et al., 2012).

3.4.1 Cost functions for bootstrapping learning in the human environment

Special, internally-generated signals are needed specifically for learning problems where standard unsupervised methods – based purely on matching the statistics of the world, or on optimizing simple mathematical objectives like temporal continuity or sparsity – will fail to discover properties of the world which are statistically weak in an objective sense but nevertheless have special significance to the organism (Ullman et al., 2012). Indigo bunting birds, for example, learn a template for the constellations of the night sky long before ever leaving the nest to engage in navigation-dependent tasks (Emlen, 1967). This memory template is directly used to determine the direction of flight during migratory periods, a process that is modulated hormonally so that winter and summer flights are reversed. Learning is therefore a multi-phase process in which navigational cues are memorized prior to the acquisition of motor control.

In humans, we suspect that similar multistage bootstrapping processes are arranged to occur. Humans have innate specializations for social learning. We need to be able to read one another’s expressions as indicated with hands and faces. Hands are important because they allow us to learn about the set of actions that can be produced by agents (Ullman et al., 2012). Faces are important because they give us insight into what others are thinking. People have intentions and personalities that differ from one another, and their feelings are important. How could we hack together cost functions, built on simple genetically specifiable mechanisms, to make it easier for a learning system to discover such behaviorally relevant variables?

Some preliminary studies are beginning to suggest specific mechanisms and heuristics that humans may be using to bootstrap more sophisticated knowledge. In a groundbreaking study, Ullman et al. (2012) asked how could we explain hands, to a system that does not already know about them, in a cheap way, without the need for labeled training examples? Hands are common in our visual space and have special roles in the scene: they move objects, collect objects, and caress babies. Building these biases into an area specialized to detect hands could guide the right kind of learning, by providing a downstream learning system with many likely positive examples of hands on the basis of innately-stored, heuristic signatures about how hands tend to look or behave (Ullman et al., 2012). Indeed, an internally supervised learning algorithm containing specialized, hard-coded biases to detect hands, on the basis of their typical motion properties, can be used to bootstrap the training of an image recognition module that learns to recognize hands based on their appearance. Thus, a simple, hard-coded module bootstraps the training of a much more complex algorithm for visual recognition of hands.

Ullman et al. (2012) then further exploits a combination of hand and face detection to bootstrap a predictor for gaze direction, based on the heuristic that faces tend to be looking towards hands. Of course, given a hand detector, it also becomes much easier to train a system for reaching, crawling, and so forth. Efforts are underway in psychology to determine whether the heuristics discovered to be useful computationally are, in fact, being used by human children during learning (Fausey et al., 2016; Yu and Smith, 2013).

Ullman refers to such primitive, inbuilt detectors as innate “proto-concepts” (Ullman et al., 2012). Their broader claim is that such prespecification of mutual supervision signals can make learning the relevant features of the world far easier, by giving an otherwise unsupervised learner the right kinds of hints or heuristic biases at the right times. Here we call these approximate, heuristic cost functions “bootstrap cost functions”. The purpose of the bootstrap cost functions is to reduce the amount of data required to learn a specific feature or task, but at the same time to avoid a need for fully unsupervised learning.

Could the neural circuitry for such a bootstrap hand-detector be pre-specified genetically? The precedent from other organisms is strong: for example, it is famously known that the frog retina contains circuitry sufficient to implement a kind of “bug detector” (Lettvin et al., 1959). Ullman’s hand detector, in fact, operates via a simple local optical flow calculation to detect “mover” events. This type of simple, local calculation could potentially be implemented in genetically-specified and/or spontaneously self-organized neural circuitry in the retina or early dorsal visual areas (Biilthoff et al., 1989), perhaps similarly to the frog’s “bug detector”.

How could we explain faces without any training data? Faces tend to have two dark dots in their upper half, a line in the lower half and tend to be symmetric about a vertical axis. Indeed, we know that babies are very much attracted to things with these generic features of upright faces starting from birth, and that they will acquire face-specific cortical areas38 in their first few years of life if not earlier (McKone et al., 2009). It is easy to define a local rule that produces a kind of crude face detector (e.g., detecting two dots on top of a horizontal line), and indeed some evidence suggests that the brain can rapidly detect faces without even a single feed-forward pass through the ventral visual stream (Crouzet and Thorpe, 2011). The crude detection of human faces used together with statistical learning should be analogous to semi-supervised learning (Sukhbaatar et al., 2014) and could allow identifying faces with high certainty.

Humans have areas devoted to emotional processing, and the brain seems to embody prior knowledge about the structure of emotional expressions and how they relate to causes in the world: emotions should have specific types of strong couplings to various other higher-level variables such as goal-satisfaction, should be expressed through the face, and so on (Skerry and Spelke, 2014; Lyons and Cheries, 2016; Baillargeon et al., 2016; Phillips et al., 2002). What about agency? It makes sense to describe, when dealing with high-level thinking, other beings as optimizers of their own goal functions. It appears that heuristically specified notions of goals and agency are infused into human psychological development from early infancy and that notions of agency are used to bootstrap heuristics for ethical evaluation (Skerry and Spelke, 2014; Hamlin et al., 2007). Algorithms for establishing more complex, innately-important social relationships such as joint attention are under study (Gao et al., 2014), building upon more primitive proto-concepts like face detectors and Ullman’s hand detectors (Ullman et al., 2012). The brain can thus use innate detectors to create cost functions and training procedures to train the next stages of learning. This prior knowledge, encoded into brain structure via evolution, could allow learning signals to come from the right places and to appear developmentally at the right times.

It is intuitive to ask whether this type of bootstrapping poses a kind of “chicken and egg” problem: if the brain already has an inbuilt heuristic hand detector, how can it be used to train a detector that performs any better than those heuristics? After all, isn’t a trained system only as good as its training data? The work of Ullman et al. (2012) illustrates why this is not the case. First, the “innate detector” can be used to train a downstream detector that operates based on different cues: for example, based on the spatial and body context of the hand, rather than its motion. Second, once multiple such pathways of detection come into existence, they can be used to improve each other. In Ullman et al. (2012), appearance, body context, and mover motion are all used to bootstrap off of one another, creating a detector that is better than any of its training heuristics. In effect, the innate detectors are used not as supervision signals per se, but rather to guide or steer the learning process, enabling it to discover features that would otherwise be diffi-cult. If such affordances can be found in other domains, it seems likely that the brain would make extensive use of them to ensure that developing animals learn the precise patterns of perception and behavior needed to ensure their later survival and reproduction.

Thus, generalizing previous ideas (Ullman et al., 2012; Poggio, 2015), we suggest that the brain uses optimization with respect to internally generated heuristic39 detection signals to bootstrap learning of biologically relevant features which would otherwise be missed by an unsupervised learner. In one possible implementation, such bootstrapping may occur via reinforcement learning, using the outputs of the innate detectors as local reinforcement signals, and perhaps using mechanisms similar to (Stnior et al., 2013; Rombouts et al., 2015; Brosch et al., 2015; Roelfsema and van Ooyen, 2005) to perform reinforcement learning through a multi-layer network. It is also possible that the brain could use such internally generated heuristic detectors in other ways, for example to bias the inputs delivered to an unsupervised learning network towards entities of interest to humans via an attentional process (Joscha Bach, personal communication), to bias hippocampal replay (Kumaran et al., 2016) or other aspects of memory access, or to directly train simple classifiers (Ullman et al., 2012).

3.4.2 Cost functions for learning by imitation and through social feedback

It has been widely observed that the capacity for imitation and social learning may be a feature that is uniquely human, and that enables other human traits Ramachandran (2000). Humans need to learn more from the environment by than trial and error can provide for, and more than genetically orchestrated internal bootstrapping signals can effectively guide. Hence, babies spend a long time watching adults, especially adults they are attached to (Meltzoff, 1999), and later use specific kinds of social cues from their parents to shape their development. Babies and children learn about cause and effect through models based on goals, outcomes and agents, not just pure statistical inference. For example, young children make inferences about causality selectively in situations where a human is trying to achieve an outcome (Meltzoff et al., 2012, 2013). Minsky (2006) discusses how we derive not just skills but also goals from our attachment figures, through socially induced emotions like pride and shame. To do all this requires a powerful infrastructure of mental abilities: we must attribute social feedback to particular aspects of our goals or actions, and hence we need to signal to each other positively and negatively, to draw attention to these aspects. Minsky speculates (Minsky, 2006) that the development of such “learning by being told” led to language by selecting for the development of increasingly precise parsing of synatatic structures in relation to our representations of agents and action-plans.

How does this connect with cost functions? The idea of goals is central here, as we need to be able to identify the goals of others, update our own goals based on feedback, and measure the success of actions relative to goals. It has been proposed that human intrinsically use a model based on abstract goal and costs to underpin learning about the social world (Jara-Ettinger et al., 2016). Perhaps we even learn about our “selves” by inferring a model of our own goals and cost functions. Relatedly, machine learning in some settings can infer their cost functions from samples of behavior (Ho and Ermon, 2016).

3.4.3 Cost functions for story generation and understanding

It has been widely noticed in cognitive science and AI that the generation and understanding of stories are crucial to human cognition. Researchers such as Winston have framed story understanding as the key to human-like intelligence (Winston, 2011). Stories consist of a linear sequence of episodes, in which one episode refers to another through cause and effect relationships, with these relationships often involving the implicit goals of agents. Many other cognitive faculties, such as conceptual grounding of language, could conceivably emerge from an underlying internal representation in terms of stories.

Perhaps the ultimate series of bootstrap cost functions would be those which would direct the brain to utilize its learning networks and specialized systems so as to construct representations that are specifically useful as components of stories, to spontaneously chain these representations together, and to update them through experience and communication. How could such cost functions arise? One possibility is that they are bootstrapped through imitation and communication, where a child learns to mimic the storytelling behavior of others. Another possibility is that useful representations and primitives for stories emerge spontaneously from mechanisms for learning state and action chunking in hierarchical reinforcement learning and planning. Yet another is that stories emerge from learned patterns of saliency-directed memory storage and recall (e.g., (Xiong et al., 2016)). In addition, priors that direct the developing child’s brain to learn about and attend to social agency seem to be important for stories.

In this section, we have seen how cost functions can be specified that could lead to the learning of increasingly sophisticated mental abilities in a biologically plausible manner. Importantly, however, cost functions and optimization are not the whole story. To achieve more complex forms of optimization, e.g., for learning to understand complex patterns of cause and effect over long timescales, to plan and reason prospectively, or to effectively coordinate many widely distributed brain resources, the brain seems to invoke specialized, pre-constructed data structures, algorithms and communication systems, which in turn facilitate specific kinds of optimization. Moreover, optimization occurs in a tightly orchestrated multi-stage process, and specialized, pre-structured brain systems need to be invoked to account for this meta-level of control over when, where and how each optimization problem is set up. We now turn to how these pre-specialized systems may orchestrate and facilitate optimization.

4. Optimization occurs in the context of specialized structures

Optimization of initially unstructured “blank slate” networks is not sufficient to generate complex cognition in the brain, we argue, even given a diversity of powerful genetically-specified cost functions and local learning rules, as we have posited above. Instead, in Hypothesis 3, we suggest that specialized, pre-structured architectures are needed for at least two purposes.

First, pre-structured architectures are needed to allow the brain to find efficient solutions to certain types of problems. When we write computer code, there are a broad range of algorithms and data structures employed for different purposes: we may use dynamic programming to solve planning problems, trees to efficiently implement nearest neighbor search, or stacks to implement recursion. Having the right kind of algorithm and data structure in place to solve a problem allows it to be solved effi-ciently, robustly and with a minimum amount of learning or optimization needed. This observation is concordant with the increasing use of pre-specialized architectures and specialized computational components in machine learning (Graves et al., 2014; Weston et al., 2014; Neelakantan et al., 2015). In particular, to enable the learning of efficient computational solutions, the brain may need pre-specialized systems for planning and executing sequential multi-step processes, for accessing memories, and for forming and manipulating compositional and recursive structures40.

Second, the training of optimization modules may need to be coordinated in a complex and dynamic fashion, including delivering the right training signals and activating the right learning rules in the right places and at the right times. To allow this, the brain may need specialized systems for storing and routing data, and for flexibly routing training signals such as target patterns, training data, reinforcement signals, attention signals, and modulatory signals. These mechanisms may need to be at least partially in place in advance of learning.

Looking at the brain, we indeed seem to find highly conserved structures, e.g., cortex, where it is theorized that a similar type of learning and/or computation is happening in multiple places (Douglas and Martin, 2004; Braitenberg and Schutz, 1991). But we also see a large number of specialized structures, including thalamus, hippocampus, basal ganglia and cerebellum (Solari and Stoner, 2011). These structures evolutionarily pre-date (Lee et al., 2015) the cortex, and hence the cortex may have evolved to work in the context of such specialized mechanisms. For example, the cortex may have evolved as a trainable module for which the training is orchestrated by these older structures.

Even within the cortex itself, microcircuitry within different areas may be specialized: tinkered variations on a common ancestral micro-circuit scaffold could potentially allow different cortical areas, such as sensory areas vs. pre-frontal areas, to be configured to adopt a number of qualitatively distinct computational and learning configurations (Marcus et al., 2014a, b; Yuste et al., 2005), even while sharing a common gross physical layout and communication interface. Within cortex, over forty distinct cell types – differing in such aspects as dendritic organization, distribution throughout the six cortical layers, connectivity pattern, gene expression, and electrophysiological properties – have already been found (Zeisel et al., 2015; Markram et al., 2015). Central pattern generator circuits provide an example of the kinds of architectures that can be pre-wired into neural microcircuitry, and may have evolutionary relationships with cortical circuits (Yuste et al., 2005). Thus, while the precise degree of architectural specificity of particular cortical regions is still under debate (Marcus et al., 2014a, b), various mechanisms could offer pre-specified heterogeneity.

In this section, we explore the kinds of computational problems for which specialized structures may be useful, and attempt to map these to putative elements within the brain. Our preliminary sketch of a functional decomposition can be viewed as a summary of suggestions for specialized functions that have been made throughout the computational neuroscience literature, and is influenced strongly by the models of O’Reilly, Eliasmith, Grossberg, Marcus, Hayworth and others (O’Reilly, 2006; Eliasmith et al., 2012; Marcus, 2001; Grossberg, 2013; Hayworth, 2012). The correspondence between these models and actual neural circuitry is, of course, still the subject of extensive debate.

Many of the computational and neural concepts sketched here are preliminary and will need to be made more rigorous through future study. Our knowledge of the functions of particular brain areas, and thus our proposed mappings of certain computations onto neuroanatomy, also remains tentative. Finally, it is still far from established which processes in the brain emerge from optimization of cost functions, which emerge from other forms of self-organization, which are pre-structured through genetics and development, and which rely on an interplay of all these mechanisms41. Our discussion here should therefore be viewed as a sketch of potential directions for further study.

4.1 Structured forms of memory

One of the central elements of computation is memory. Importantly, multiple different kinds of memory are needed (Squire, 2004). For example, we need memory that is stored for a long period of time and that can be retrieved in a number of ways, such as in situations similar to the time when the memory was first stored (con-tent addressable memory). We also need memory that we can keep for a short period of time and that we can rapidly rewrite (working memory). Lastly, we need the kind of implicit memory that we cannot explicitly recall, similar to the kind of memory that is classically learned using gradient descent on errors, i.e., sculpted into the weight matrix of a neural network.

4.1.1 Content addressable memories

Content addressable memories42 are classic models in neuroscience (Hopfield, 1982). Most simply, they allow us to recognize a situation similar to one that we have seen before, and to “fill in” stored patterns based on partial or noisy information, but they may also be put to use as sub-components of many other functions. Recent research has shown that including such memories allows deep networks to learn to solve problems that previously were out of reach, even of LSTM networks that already have a simpler form of local memory and are already capable of learning long-term dependencies (Weston et al., 2014; Graves et al., 2014). Hippocampal area CA3 may act as an auto-associative memory43 capable of content-addressable pattern completion, with pattern separation occurring in the dentate gyrus (Rolls, 2013). If no similar pattern is available, an unfamiliar input will be stored as a new memory (Kumaran et al., 2016). Such systems could permit the retrieval of complete memories from partial cues, enabling networks to perform operations similar to database retrieval or to instantiate lookup tables of historical stimulus-response mappings, among numerous other possibilities.

Of course, memory systems may be organized – through cost function optimization or other mechanisms – into higher-order structures. Cost functions might be used to bias memory representations to adopt particular structures, e.g., to be organized into data structures like like Minskys frames and trans-frames (Minsky, 2006).

4.1.2 Working memory buffers

Cognitive science has long characterized properties of the working memory. Its capacity is somewhat limited, with the old idea being that verbal working memory has a capacity of “seven plus or minus two” (Miller, 1956), while visual working memory has a capacity of four (Luck and Vogel, 1997) (or, other authors defend, one). There are many models of working memory (Wang, 2012; Singh and Eliasmith, 2006; O’Reilly and Frank, 2006; Buschman and Miller, 2014; Warden and Miller, 2007), some of which attribute it to persistent, self-reinforcing patterns of neural activation (Goldman et al., 2003) in the recurrent networks of the prefrontal cortex. Prefrontal working memory appears to be made up of multiple functionally distinct subsystems (Markowitz et al., 2015). Neural models of working memory can store not only scalar variables (Seung, 1998), but also high-dimensional vectors (Eliasmith and Anderson, 2004; Eliasmith et al., 2012) or sequences of vectors (Choo and Eliasmith, 2010). Working memory buffers seem crucial for human-like cognition, e.g., reasoning, as they allow short-term storage while also – in conjunction with other mechanisms – enabling generalization of operations across anything that can fill the buffer.

4.1.3 Storing state in association with saliency

Saliency, or interestingness, measures can be used to tag the importance of a memory (Gonzalez Andino and Grave de Peralta Menendez, 2012). This can allow removal of the boring data from the training set, allowing a mechanism that is more like optimal experimentation. Moreover, saliency can guide memory replay or sampling from generative models, to generate more training data drawn from a distribution useful for learning (Mnih et al., 2015; Ji and Wilson, 2007). Conceivably, hippocampal replay could allow a batch-like training process, similar to how most machine learning systems are trained, rather than requiring all training to occur in an online fashion. Plasticity mechanisms in memory systems which are gated by saliency are starting to be uncovered in neuroscience (Dudman et al., 2007). Importantly, the notions of “saliency” computed by the brain could be quite intricate and multi-faceted, potentially leading to complex schemes by which specific kinds of memories would be tagged for later context-dependent retrieval. As a hypothetical example, representations of both timing and importance associated with memories could perhaps allow retrieval only of important memories that happened within a certain window of time (MacDonald et al., 2011; Kraus et al., 2013; Rubin et al., 2015). Storing and retrieving information selectively based on specific properties of the information itself, or of “tags” appended to that information, is a powerful computational primitive that could enable learning of more complex tasks. Relatedly, we know that certain pathways become associated with certain kinds of memories, e.g., specific pathways for fear-related memory in mice.

4.2 Structured routing systems

To use its information flexibly, the brain needs structured systems for routing data. Such systems need to address multiple temporal and spatial scales, and multiple modalities of control. Thus, there are several different kinds of information routing systems in the brain which operate by different mechanisms and under different constraints.

4.2.1 Attention

If we can focus on one thing at a time, we may be able to allocate more computational resources to processing it, make better use of scarce data to learn about it, and more easily store and retrieve it from memory44. Notably in this context, attention allows improvements in learning: if we can focus on just a single object, instead of an entire scene, we can learn about it more easily using limited data. Formal accounts in a Bayesian framework talk about attention reducing the sample complexity of learning (Chikkerur et al., 2010). Likewise, in models, the processes of applying attention, and of effectively making use of incoming attentional signals to appropriately modulate local circuit activity, can themselves be learned by optimizing cost functions (Mnih et al., 2014; Jaramillo and Pearlmutter, 2004). The right kinds of attention make processing and learning more efficient, and also allow for a kind of programmatic control over multi-step perceptual tasks.

How does the brain determine where to allocate attention, and how is the attentional signal physically mediated? Answering this question is still an active area of neuroscience. Higher-level cortical areas may be specialized in allocating attention. The problem is made complex by the fact that there seem to be many different types of attention – such as object-based, feature-based and spatial attention in vision – that may be mediated by interactions between different brain areas. The frontal eye fields (area FEF), for example, are important in visual attention, specifically for controlling saccades of the eyes to attended locations. Area FEF contains “retinotopic” spatial maps whose activation determines the saccade targets in the visual field. Other prefrontal areas such as the dorsolateral prefrontal cortex and inferior frontal junction are also involved in maintaining representations that specify the targets of certain types of attention. Certain forms of attention may require a complex interaction between brain areas, e.g., to determine targets of attention based on higher-level properties that are represented across multiple areas, like the identity and spatial location of a specific face (Baldauf and Desimone, 2014).

There are many proposed neural mechanisms of attention, including the idea that synchrony plays a role (Baldauf and Desimone, 2014), perhaps by creating resonances that facilitate the transfer of information between synchronously oscillating neural populations in different areas45. Other proposed mechanisms include specific circuits for attention-dependent signal routing (Anderson and Van Essen, 1987; Olshausen et al., 1993). Various forms of attention also have specific neurophysiological signatures, such as enhancements in synchrony among neural spikes and with the ambient local field potential, changes in the sharpness of neural tuning curves, and other properties. These diverse effects and signatures of attention may be consequences of underlying pathways that wire up to particular elements of cortical microcircuits to mediate different attentional effects (Bobier et al., 2014).

4.2.2 Buffers

One possibility is that the brain uses distinct groups of neurons, which we can call “buffers”, to store distinct variables, such as the subject or object in a sentence (Frankland and Greene, 2015). Having memory buffers allows the abstraction of a variable.

Once we establish that the brain has a number of memory buffers, we need ways for those buffers to interact. We need to be able to take a buffer, do a computation on its contents and store the output into another buffer. But if the representations in each of two groups of neurons are learned, and hence are coded differently, how can the brain “copy and paste” information between these groups of neurons? Malsburg argued that such a system of separate buffers is impossible because the neural pattern for “chair” in buffer 1 has nothing in common with the neural pattern for “chair” in buffer 2 – any learning that occurs for the contents of buffer 1 would not automatically be transferable to buffer 2. Various mechanisms have been proposed to allow such transferability, which focus on ways in which all buffers could be trained jointly and then later separated so that they can work independently when they need to46.

4.2.3 Discrete gating of information flow between buffers

Dense connectivity is only achieved locally, but it would be desirable to have a way for any two cortical units to talk to one another, if needed, regardless of their distance from one another, and without introducing crosstalk47. It is therefore critical to be able to dynamically turn on and off the transfer of information between different source and destination regions, in much the manner of a switchboard. Together with attention, such dedicated routing systems can make sure that a brain area receives exactly the information it needs. Such a discrete routing system is, of course, central to cognitive architectures like ACT-R (Anderson, 2007). The key feature of ACT-R is the ability to evaluate the IF clauses of tens of thousands of symbolic rules (called “productions”), in parallel, approximately every 50 milliseconds. Each rule requires equality comparisons between the contents of many constant and variable memory buffers, and the execution of a rule leads to the conditional routing of information from one buffer to another.

What controls which long-range routing operations occur when, i.e., where is the switchboad and what controls it? Several models, including ACT-R, have attributed such parallel rule-based control of routing to the action selection circuitry (Gurney et al., 2001; Terrence Stewart, 2010) of the basal ganglia (BG) (Stocco et al., 2010; O’Reilly and Frank, 2006), and its interaction with working memory buffers in the prefrontal cortex. In conventional models of thalamo-cortico-striatal loops, competing actions of the direct and indirect pathways through the basal ganglia can inhibit or disinhibit an area of motor cortex, thereby gating a motor action48. Models like (Stocco et al., 2010; O’Reilly and Frank, 2006; Terrence Stewart, 2010) propose further that the basal ganglia can gate not just the transfer of information from motor cortex to downstream actuators, but also the transfer of information between cortical areas. To do so, the basal ganglia would dis-inhibit a thalamic relay (Sherman, 2005, 2007) linking two cortical areas. Dopamine-related activity is thought to lead to temporal difference reinforcement learning of such gating policies in the basal ganglia (Frank and Badre, 2012). Beyond the basal ganglia, there are also other, separate pathways involved in action selection, e.g., in the prefrontal cortex (Daw et al., 2006). Thus, multiple systems including basal ganglia and cortex could control the gating of long-range information transfer between cortical areas, with the thalamus perhaps largely constituting the switchboard itself.

How is such routing put to use in a learning context? One possibility is that the basal ganglia acts to orchestrate the training of the cortex. The basal ganglia may exert tight control49 over the cortex, helping to determine when and how it is trained. Indeed, because the basal ganglia pre-dates the cortex evolutionarily, it is possible that the cortex evolved as a flexible, trainable resource that could be harnessed by existing basal ganglia circuitry. All of the main regions and circuits of the basal ganglia are conserved from our common ancestor with the lamprey more than five hundred million years ago. The major part of the basal ganglia even seems to be conserved from our common ancestor with insects (Strausfeld and Hirth, 2013). Thus, in addition to its real-time action selection and routing functions, the basal ganglia may sculpt how the cortex learns.

4.3 Structured state representations to enable efficient algorithms

Certain algorithmic problems benefit greatly from particular types of representation and transformation, such as a grid-like representation of space. In some cases, rather than just waiting for them to emerge via gradient descent optimization of appropriate cost functions, the brain may be pre-structured to facilitate their creation.

4.3.1 Continuous predictive control

We often have to plan and execute complicated sequences of actions on the fly, in response to a new situation. At the lowest level, that of motor control, our body and our immediate environment change all the time. As such, it is important for us to maintain knowledge about this environment in a continuous way. The deviations between our planned movements and those movements that we actually execute continuously provide information about the properties of the environment. Therefore it seems important to have a specialized system, optimized for high-speed continuous processing, that takes all our motor errors and uses them to update a dynamical model of our body and our immediate environment that can predict the delayed sensory results of our motor actions (McKinstry et al., 2006).

It appears that the cerebellum is such a structure, and lesions to it abolish our way of dealing successfully with a changing body. Incidentally, the cerebellum has more connections than the rest of the brain taken together, apparently in a largely feedforward architecture, and the tiny cerebellar granule cells, which may form a randomized high-dimensional input representation (Marr, 1969; Jacobson and Friedrich, 2013), outnumber all other neurons. The brain clearly needs a dedicated way of quickly and continuously correcting movements to minimize errors, without needing to rely on slow and complex association learning in the neocortex in order to do so.

Newer research shows that the cerebellum is involved in a broad range of cognitive problems (Moberget et al., 2014) as well, potentially because they share computational problems with motor control. For example, when subjects estimate time intervals, which are naturally important for movement, it appears that the brain uses the cerebellum even if no movements are involved (Gooch et al., 2010). Even individual cerebellar Purkinjie cells may learn to generate precise timings of their outputs (Johansson et al., 2014). The brain also appears to use inverse models to rapidly predict motor activity that would give rise to a given sensory target (Giret et al., 2014; Hanuschkin et al., 2013). Such mechanisms could be put to use far beyond motor control, in bootstrapping the training of a larger architecture by exploiting continuously changing error signals to update a real-time model of the system state.

4.3.2 Hierarchical control

Importantly, many of the control problems we appear to be solving are hierarchical. We have a spinal cord, which deals with the fast signals coming from our muscles and proprioception. Within neuroscience, it is generally assumed that this system deals with fast feedback loops and that this behavior is learned to optimize its own cost function. The nature of cost functions in motor control is still under debate. In particular, the timescale over which cost functions operate remains unclear: motor optimization may occur via real-time responses to a cost function that is computed and optimized online, or via policy choices that change over time more slowly in response to the cost function (Körding, 2007). Nevertheless, the effect is that central processing in the brain has an effectively simplified physical system to control, e.g., one that is far more linear. So the spinal cord itself already suggests the existence of two levels of a hierarchy, each trained using different cost functions.

However, within the computational motor control literature (see e.g., (DeWolf and Eliasmith, 2011)), this idea can be pushed far further, e.g., with a hierarchy including spinal cord, M1, PMd, frontal, prefrontal areas. A low level may deal with muscles, the next level may deal with getting our limbs to places or moving objects, a next layer may deal with solving simple local problems (e.g., navigating across a room) while the highest levels may deal with us planning our path through life. This factorization of the problem comes with multiple aspects: First, each level can be solved with its own cost functions, and second, every layer has a characteristic timescale. Some levels, e.g., the spinal cord, must run at a high speed. Other levels, e.g., high-level planning, only need to be touched much more rarely. Converting the computationally hard optimal control problem into a hierarchical approximation promises to make it dramatically easier.

Does the brain solve control problems hierarchically? There is evidence that the brain uses such a strategy (Botvinick and Weinstein, 2014; Botvinick et al., 2009), beside neural network demonstrations (Wayne and Abbott, 2014). The brain may use specialized structures at each hierarchical level to ensure that each operates effi-ciently given the nature of its problem space and available training signals. At higher levels, these systems may use an abstract syntax for combining sequences of actions in pursuit of goals (Allen et al., 2010). Subroutines in such processes could be derived by a process of chunking sequences of actions into single actions (Botvinick and Weinstein, 2014; Graybiel, 1998). Some brain areas like Broca’s area, known for its involvement in language, also appear to be specifically involved in processing the hierarchical structure of behavior, as such, as opposed to its detailed temporal structure (Koechlin and Jubault, 2006).

At the highest level of the decision making and control hierarchy, human reward systems reflect changing goals and subgoals, and we are only beginning to understand how goals are actually coded in the brain, how we switch between goals, and how the cost functions used in learning depend on goal state (O’Reilly et al., 2014b; Buschman and Miller, 2014; Pezzulo et al., 2014). Goal hierarchies are beginning to be incorporated into deep learning (Kulkarni et al., 2016).

Given this hierarchical structure, the optimization algorithms can be fine-tuned. For the low levels, there is sheer unlimited training data. For the high levels, a simulation of the world may be simple, with a tractable number of high-level actions to choose from. Finally, each area needs to give reinforcement to other areas, e.g., high levels need to punish lower levels for making planning complicated. Thus this type of architecture can simplify the learning of control problems.

Progress is being made in both neuroscience and machine learning on finding potential mechanisms for this type of hierarchical planning and goal-seeking. This is beginning to reveal mechanisms for chunking goals and actions and for searching and pruning decision trees (Balaguer et al., 2016; Krishnamurthy et al., 2016; Tamar et al., 2016; O’Reilly et al., 2014a; Huys et al., 2015). The study of model-based hierarchical reinforcement learning and prospective optimization (Sejnowski and Poizner, 2014), which concerns the planning and evaluation of nested sequences of actions, implicates a network coupling the dorsolateral prefontral and orbitofrontal cortex, and the ventral and dorsolateral striatum (Botvinick et al., 2009). Hierarchical RL relies on a hierarchical representation of state and action spaces, and it has been suggested that error-driven learning of an optimal such representation in the hippocampus50 gives rise to place and grid cell properties (Stachenfeld, 2014), with goal representations themselves emerging in the amygdala, prefrontal cortex and other areas (O’Reilly et al., 2014a).

The question of how control problems can be successfully divided into component problems remains one of the central questions in neuro-science (Wolpert and Flanagan, 2016) and machine learning (Kulkarni et al., 2016), and the cost functions involved in learning to create such decompositions are still unknown. These considerations may begin to make plausible, however, how the brain could not only achieve its remarkable feats of motor learning – such as generating complex “innate” motor programs, like walking in the newborn gazelle almost immediately after birth – but also the kind of planning that allows a human to prepare a meal or travel from London to Chicago.

4.3.3 Spatial planning

Spatial planning requires solving shortest-path problems subject to constraints. If we want to get from one location to another, there are an arbitrarily large number of simple paths that could be taken. Most naive implementations of such shortest paths problems are grossly inefficient. It appears that, in animals, the hippocampus aids – at least in part through “place cell” and “grid cell” systems – in efficient learning about new environments and in targeted navigation in such environments (Brown et al., 2016). Interestingly, once an environment becomes familiar, it appears that areas of the neocortex can take over the role of navigation (Hasselmo and Stern, 2015).

In some simple models, targeted navigation in the hippocampus is achieved via the dynamics of “bump attractors” or propagating waves in a place cell network with Hebbian plasticity and adaptation (Buzsáki and Moser, 2013; Hopfield, 2009; Ponulak and Hopfield, 2013), which allows the network to effectively chart out a path in the space of place cell representations. Other navigation models make use of the grid cell system. The place cell network may51 take input from a grid cell network that computes precise distances and directions, perhaps by integrating head direction and velocity signals – grid cells fire when the animal is on any node of a regularly spaced hexagonal grid. Different parts of the entorhinal cortex contain grid cells with different grid spacings, and place cells may combine information from multiple such grids in order to build up responses to particular single positions. These systems are highly structured temporally, e.g., containing nested gamma and theta oscillation structures that are phased locked to sequences of place-cell responses, interfering oscillators frequency-shifted by the animal’s motion velocity (Zilli and Hasselmo, 2010), tuned cellular resonances (Buzsáki, 2010; Giocomo et al., 2007), and other neural phenomena that lie far outside a conventional artificial neural network description. It seems that an intricate interplay of spatial and temporal network structures may be essential for encoding sequences of spatiotemporal events across multiple scales, and using them to drive multiple forms of learning, e.g., supporting forward and reverse sequence replay with various temporal compression factors (Buzsáki, 2010).

Higher-level cognitive tasks such as prospective planning appear to share computational sub-problems with path-finding (Hassabis and Maguire, 2009)52. Interaction between hippocampus and prefrontal cortex could perhaps support a more abstract notion of “navigation” in a space of goals and sub-goals. Interestingly, there is preliminary evidence from fMRI that abstract concepts are also represented according to grid-cell-like hexagonal grid structures in humans (Constantinescu et al., 2016), as well as preliminary evidence that social relationships may also be represented through a hippocampal map (Tavares et al., 2015). Having specialized structures for path-finding could thus simplify a variety of computational problems at different levels of abstraction.

4.3.4 Variable binding

Language and reasoning appear to present a problem for neural networks (Minsky, 1991; Hadley, 2009; Marcus, 2001): we seem to be able to apply common grammatical rules to sentences regardless of the content of those sentences, and regardless of whether we have ever seen even remotely similar sentences in the training data. While this is achieved automatically in a computer with fixed registers, location addressable memories, and hard-coded operations, how it could be achieved in a biological brain, or emerge from an optimization algorithm, has been under debate for decades.

As the putative key capability underlying such operations, variable binding has been defined as “the transitory or permanent tying together of two bits of information: a variable (such as an X or Y in algebra, or a placeholder like subject or verb in a sentence) and an arbitrary instantiation of that variable (say, a single number, symbol, vector, or word)” (Marcus et al., 2014a, b). A number of potential biologically plausible binding mechanisms (Hayworth, 2012; Kriete et al., 2013; Eliasmith et al., 2012; Goertzel, 2014) are reviewed in (Marcus et al., 2014a, b). Some, such as vector symbolic architectures53, which were proposed in cognitive science (Eliasmith, 2013; Plate, 1995; Stewart and Eliasmith, 2009), are also being considered in the context of efficiently-trainable artificial neural networks (Danihelka et al., 2016) – in effect, these systems learn how to use variable binding.

Variable binding could potentially emerge from simpler memory systems. For example, the Scrub-Jay can remember the place and time of last visit for hundreds of different locations, e.g., to determine whether high-quality food is currently buried at any given location (Clayton and Dickinson, 1998). It is conceivable that such spatially-grounded memory systems enabled a more general binding mechanism to emerge during evolution, perhaps through integration with routing systems or other content-addressable or working memory systems.

4.3.5 Hierarchical syntax

Fixed, static hierarchies (e.g., the hierarchical organization of cortical areas (Felleman and Van Essen, 1991)) only take us so far: to deal with long chains of arbitrary nested references, we need dynamic hierarchies that can implement recursion on the fly. Human language syntax has a hierarchical structure, which Berwick et al described as “composition of smaller forms like words and phrases into larger ones” (Berwick et al., 2012; Miyagawa et al., 2013). The extent of recursion in human language and thought may be captured by a class of automata known as higher-order pushdown automata, which can be implemented via finite state machines with access to nested stacks (Rodriguez and Granger, 2016). Specific fronto-temporal networks may be involved in representing and generating such hierarchies (Dehaene et al., 2015), e.g., with the hippocampal system playing a key role in implementing some analog of a pushdown stack (Rodriguez and Granger, 2016)54.

Little is known about the underlying circuit mechanisms for such dynamic hierarchies, but it is clear that specific affordances for representing such hierarchies in an efficient way would be beneficial. This may be closely connected with the issue of variable binding, and it is possible that operations similar to pointers could be useful in this context, in both the brain and artificial neural networks (Kriete et al., 2013; Kurach et al., 2015). Augmenting neural networks with a differentiable analog of a push-down stack is another such affordance being pursued in machine learning (Joulin and Mikolov, 2015).

4.3.6 Mental programs and imagination

Humans excel at stitching together sub-actions to form larger actions (Sejnowski and Poizner, 2014; Acuna et al., 2014; Verwey, 1996). Structured, serial, hierarchical probabilistic programs have recently been shown to model aspects of human conceptual representation and compositional learning (Lake et al., 2015). In particular, sequential programs were found to enable one-shot learning of new geometric/visual concepts (Lake et al., 2015). Generative programs have also been proposed in the context of scene understanding (Battaglia et al., 2013). The ability to deal with problems in terms of subproblems is central both in human thought and in many successful algorithms.

One possibility is that the hippocampus supports the rapid construction and learning of sequential programs, e.g., in multi-step planning. An influential idea – known as the “complementary learning systems hypothesis” – is that the hippocampus plays a key role in certain processes where learning must occur quickly on the basis of single episodes, whereas the cortex learns more slowly by aggregating and integrating patterns across large amounts of data (Leibo et al., 2015a; Herd et al., 2013; Blundell et al., 2016; Kumaran et al., 2016). The hippocampus appears to explore, in simulation, possible future trajectories to a goal, even those involving previously unvisited locations (Ólafsdóttir et al., 2015). Hippocampal-prefrontal interaction has been suggested to allow rapid, subconscious evaluation of potential action sequences during decision-making, with the hippocampus in effect simulating the expected outcomes of potential actions that are generated and evaluated in the prefrontal (Wang et al., 2015; Mushiake et al., 2006). The role of the hippocampus in imagination, concept generation (Kumaran et al., 2009), scene construction (Hassabis and Maguire, 2007), mental exploration and goal-directed path planning (Ólafsdóttir et al., 2015; Brown et al., 2016; Hopfield, 2009) suggests that it could help to create generative models to underpin more complex inference such as program induction (Lake et al., 2015) or common-sense world simulation (Battaglia et al., 2013). For example, a sequential, programmatic process, mediated jointly by the basal ganglia, hippocampus and prefrontal cortex might allow one-shot learning of a new concept, as in the sequential computations underlying a process like Bayesian Program Learning (Lake et al., 2015).

Another related possibility is that the cortex itself intrinsically supports the construction and learning of sequential programs (Bach and Herger, 2015). Recurrent neural networks have been used for image generation through a sequential, attention-based process (Gregor et al., 2015), although their correspondence with the brain is unclear55.

4.4 Other specialized structures

Importantly, there are many other specialized structures known in neuroscience, which arguably receive less attention than they deserve, even for those interested in higher cognition. In the above, in addition to the hippocampus, basal ganglia and cortex, we emphasized the key roles of the thalamus in routing, of the cerebellum as a fast and rapidly trainable control and modeling system, of the amygdala and other areas as a potential source of utility functions, of the retina or early visual areas as a means to generate detectors for motion and other features to bootstrap more complex visual learning, and of the frontal eye fields and other areas as a possible source of attention control. We ignored other structures entirely, whose functions are only beginning to be uncovered, such as the claustrum (Crick and Koch, 2005), which has been speculated to be important for rapidly binding together information from many modalities. Our overall understanding of the functional decomposition of brain circuitry still seems very preliminary.

4.5 Relationships with other cognitive frameworks involving specialized systems

A recent analysis (Lake et al., 2016) suggested directions by which to modify and enhance existing neural-net-based machine learning towards more powerful and human-like cognitive capabilities, particularly by introducing new structures and systems which go beyond data-driven optimization. This analysis emphasized that systems should construct generative models of the world that incorporate compositionality (discrete construction from re-usable parts), inductive biases reflecting causality, intuitive physics and intuitive psychology, and the capacity for probabilistic inference over discrete structured models (e.g., structured as graphs, trees, or programs) (Tervo et al., 2016) to harness abstractions and enable transfer learning.

We view these ideas as consistent with and complementary to the framework of cost functions, optimization and specialized systems discussed here. One might seek to understand how optimization and specialized systems could be used to implement some of the mechanisms proposed in (Lake et al., 2016) inside neural networks. Lake et al. (2016) emphasize how incorporating additional structure into trainable neural networks can potentially give rise to systems that use compositional, causal and intuitive inductive biases and that “learn to learn” using structured models and shared data structures. For example, sub-dividing networks into units that can be modularly and dynamically combined, where representations can be copied and routed, may present a path towards improved compositionality and transfer learning (Andreas et al., 2015). The control flow for recombining pre-existing modules and representations could be learned via reinforcement learning (Andreas et al., 2016). How to implement the broad set of mechanisms discussed in (Lake et al., 2016) is a key computational problem, and it remains open at which levels (e.g., cost functions and training procedures vs. specialized computational structures vs. underlying neural primitives) architectural innovations will need to be introduced to capture these phenomena.

Primitives that are more complex than those used in conventional neural networks – for instance, primitives that act as state machines with complex message passing (Bach and Herger, 2015) or networks that intrinsically implement Bayesian inference (George and Hawkins, 2009) – could potentially be useful, and it is plausible that some of these may be found in the brain. Recent findings on the power of generic optimization also do not rule out the idea that the brain may explicitly generate and use particular types of structured representations to constrain its inferences; indeed, the specialized brain systems discussed here might provide a means to enforce such constraints. It might be possible to further map the concepts of Lake et al. (2016) onto neuroscience via an infrastructure of interacting cost functions and specialized brain systems under rich genetic control, coupled to a powerful and generic neurally implemented capacity for optimization. For example, it was recently shown that complex probabilistic population coding and inference can arise automatically from backpropagation-based training of simple neural networks (Orhan and Ma, 2016), without needing to be built in by hand. The nature of the underlying primitives in the brain, on top of which learning can operate, is a key question for neuroscience.

5. Machine learning inspired neuroscience

Hypotheses are primarily useful if they lead to concrete, experimentally testable predictions. As such, we now want to go through the hypotheses and see to which level they can be directly tested, as well as refined, through neuroscience.

5.1 Hypothesis 1– Existence of cost functions

There are multiple general strategies for addressing whether and how the brain optimizes cost functions. A first strategy is based on observing the endpoint of learning. If the brain uses a cost function, and we can guess its identity, then the final state of the brain should be close to optimal for the cost function. We could thus compare (Güçlü and van Gerven, 2015) receptive fields that are optimized in a simulation, according to a particular cost function, with the measured receptive fields. Various techniques exist to carry out such comparisons in fRMI studies, including population receptive field estimation (Güçlü and van Gerven, 2015; Dumoulin and Wandell, 2008) and representational dissimilarity matrices (Kriegeskorte et al., 2008; Khaligh-Razavi and Kriegeskorte, 2014). This strategy is only beginning to be used at the moment, perhaps because it has been difficult to measure the receptive fields or other representational properties across a large population of individual neurons (fMRI operates at a much coarser level), but this situation is beginning to improve technologically with the emergence of large-scale recording methods (Hasselmo, 2015).

A second strategy could directly quantify how well a cost function describes learning. If the dynamics of learning minimize a cost function then the underlying vector field should have a strong gradient descent type component and a weak rotational component, i.e., weight changes will primarily move down the gradient rather than drifting in the nullspace. If we could somehow continuously monitor the synaptic strengths, while externally manipulating them, then we could, in principle, measure the vector field in the space of synaptic weights, and calculate its divergence as well as its rotation. For at least the subset of synapses that are being trained via some approximation to gradient descent, the divergence component should be strong relative to the rotational component. This strategy has not been developed yet due to experimental difficulties with monitoring large numbers of synaptic weights56.

A third strategy is based on perturbations: cost function based learning should undo the effects of perturbations which disrupt optimality, i.e., the system should return to local minima after a perturbation, and indeed perhaps to the same local minimum after a sufficiently small perturbation. If we change synaptic connections, e.g., in the context of a brain machine interface, we should be able to produce a reorganization that can be predicted based on a guess of the relevant cost function. This strategy is starting to be feasible in motor areas.

Lastly, if we knew structurally which cell types and connections mediated the delivery of error signals vs. input data or other types of connections, then we could stimulate specific connections so as to impose a user-defined cost function. In effect, we would use the brain’s own networks as a trainable deep learning substrate, and then study how the network responds to training. Brain machine interfaces can be used to set up specific local learning problems, in which the brain is asked to create certain user-specified representations, and the dynamics of this process can be monitored (Sadtler et al., 2014). Likewise, brain machine interfaces can be used to give the brain access to new datastreams, and to investigate how those datastreams are incorporated into task performance, and whether such incorporation is governed by optimality principles (Dadarlat et al., 2015). In order to do this kind of experiment fully and optimally, we must first understand more about how the system is wired to deliver cost signals. Much of the structure that would be found in connectomic circuit maps, for example, would not just be relevant for short-timescale computing, but also for creating the infrastructure that supports cost functions and their optimization.

Many of the learning mechanisms that we have discussed in this paper make specific predictions about connectivity or dynamics. For example, the “feedback alignment” approach to biological backpropagation suggests that cortical feedback connections should, at some level of neuronal grouping, be largely sign-concordant with the corresponding feedforward connections, although not necessarily of concordant weight (Liao et al., 2015), and feedback alignment also makes predictions for synaptic normalization mechanisms (Liao et al., 2015). The Kickback model for biologically plausible back-propagation has a specific role for NMDA receptors (Balduzzi et al., 2014). Some models that incorporate dendritic coincidence detection for learning temporal sequences predict that a given axon should make only a small number of synapses on a given dendritic segment (Hawkins and Ahmad, 2016). Models that involve STDP learning will make predictions about the dynamics of changing firing rates (Hinton, 2007, 2016; Bengio et al., 2015a; Bengio and Fischer, 2015; Bengio et al., 2015b), as well as about the particular network structures, such as those based on autoencoders or recirculation, in which STDP can give rise to a form of backpropagation.

It is critical to establish the unit of optimization. We want to know the scale of the modules that are trainable by some approximation of gradient descent optimization. How large are the networks which share a given error signal or cost function? On what scales can appropriate training signals be delivered? It could be that the whole brain is optimized end-to-end, in principle. In this case we would expect to find connections that carry training signals from each layer to the preceding ones. On successively smaller scales, optimization could be within a brain area, a microcircuit57, or an individual neuron (Körding and König, 2000, 2001; Mel, 1992; Hawkins and Ahmad, 2016). Importantly, optimization may co-exist across these scales. There may be some slow optimization end-to-end, with stronger optimization within a local area and very efficient algorithms within each cell. Careful experiments should be able to identify the scale of optimization, e.g., by quantifying the extent of learning induced by a local perturbation.

The tightness of the structure-function relationship is the hallmark of molecular and to some extent cellular biology, but in large connection-ist learning systems, this relationship can become difficult to extract: the same initial network can be driven to compute many different functions by subjecting it to different training58 ,59. It can be hard to understand the way a neural network solves its problems.

How could one tell the difference, then, between a gradient-descent trained network vs. untrained or random networks vs. a network that has been trained against a different kind of task? One possibility would be to train artificial neural networks against various candidate cost functions, study the resulting neural tuning properties (Todorov, 2002), and compare them with those found in the circuit of interest (Zipser and Andersen, 1988). This has already been done to aid the interpretation of the neural dynamics underlying decision making in the PFC (Sussillo, 2014), working memory in the posterior parietal cortex (Rajan et al., 2016) and object or action representation in the visual system (Yamins and DiCarlo, 2016b, a; Tacchetti et al., 2016). Some have gone on to suggest a direct correspondence between cortical circuits and optimized, appropriately regularized (Sussillo et al., 2015), recurrent neural networks (Liao and Poggio, 2016). In any case, effective analytical methods to reverse engineer complex machine learning systems (Jonas and Kording, 2016), and methods to reverse engineer biological brains, may have some commonalities.

Does this emphasis on function optimization and trainable substrates mean that we should give up on reverse engineering the brain based on detailed measurements and models of its specific connectivity and dynamics? On the contrary: we should use large-scale brain maps to try to better understand a) how the brain implements optimization, b) where the training signals come from and what cost functions they embody, and c) what structures exist, at different levels of organization, to constrain this optimization to efficiently find solutions to specific kinds of problems. The answers may be influenced by diverse local properties of neurons and networks, such as homeostatic rules of neural structure, gene expression and function (Marder and Goaillard, 2006), the diversity of synapse types, cell-type-specific connectivity (Jiang et al., 2015), patterns of inter-laminar projection, distributions of inhibitory neuron types, dendritic targeting and local dendritic physiology and plasticity (Markram et al., 2015; Bloss et al., 2016; Sandler et al., 2016; Morgan et al., 2016) or local glial networks (Perea et al., 2009). They may also be influenced by the integrated nature of higher-level brain systems, including mechanisms for developmental bootstrapping (Ullman et al., 2012), information routing (Gurney et al., 2001; Stocco et al., 2010), attention (Buschman and Miller, 2010) and hierarchical decision making (Lee et al., 2015). Mapping these systems in detail is of paramount importance to understanding how the brain works, down to the nanoscale dendritic organization of ion channels and up to the real-time global coordination of cortex, striatum and hippocampus, all of which are computationally relevant in the framework we have explicated here. We thus expect that large-scale, multi-resolution brain maps would be useful in testing these framework-level ideas, in inspiring their refinements, and in using them to guide more detailed analysis.

5.2 Hypothesis 2– Biological fine-structure of cost functions

Clearly, we can map differences in structure, dynamics and representation across brain areas. When we find such differences, the question remains as to whether we can interpret these as resulting from differences in the internally-generated cost functions, as opposed to differences in the input data, or from differences that reflect other constraints unrelated to cost functions. If we can directly measure aspects of the cost function in different areas, then we can also compare them across areas. For example, methods from inverse reinforcement learning60 might allow backing out the cost function from observed plasticity (Ng and Russell, 2000).

Moreover, as we begin to understand the “neural correlates” of particular cost functions – perhaps encoded in particular synaptic or neuromodulatory learning rules, genetically-guided local wiring patterns, or patterns of interaction between brain areas – we can also begin to understand when differences in observed neural circuit architecture reflect differences in cost functions.

We expect that, for each distinct learning rule or cost function, there may be specific molecularly identifiable types of cells and/or synapses. Moreover, for each specialized system there may be specific molecularly identifiable developmental programs that tune it or otherwise set its parameters. This would make sense if evolution has needed to tune the parameters of one cost function without impacting others.

How many different types of internal training signals does the brain generate? When thinking about error signals, we are not just talking about dopamine and serotonin, or other classical reward-related pathways. The error signals that may be used to train specific sub-networks in the brain, via some approximation of gradient descent or otherwise, are not necessarily equivalent to reward signals. It is important to distinguish between cost functions that may be used to drive optimization of specific sub-circuits in the brain, and what are referred to as “value functions” or “utility functions”, i.e., functions that predict the agent’s aggregate future reward. In both cases, similar reinforcement learning mechanisms may be used, but the interpretation of the cost functions is different. We have not emphasized global utility functions for the animal here, since they are extensively studied elsewhere (e.g., (O’Reilly et al., 2014a; Bach, 2015)), and since we argue that, though important, they are only a part of the picture, i.e., that the brain is not solely an end-to-end reinforcement trained system.

Progress in brain mapping could soon allow us to classify the types of reward signals in the brain, follow the detailed anatomy and connectivity of reward pathways throughout the brain, and map in detail how reward pathways are integrated into striatal, cortical, hippocampal and cerebellar microcircuits. This program is beginning to be carried out in the fly brain, in which twenty specific types of dopamine neuron project to distinct anatomical compartments of the mushroom body to train distinct odor classifiers operating on a set of high-dimensional odor representations (Aso et al., 2014a, b; Caron et al., 2013; Cohn et al., 2015). It is known that, even within the same system, such as the fly olfactory pathway, some neuronal wiring is highly specific and molecularly programmed (Hong and Luo, 2014; Hattori et al., 2007), while other wiring is effectively random (Caron et al., 2013), and yet other wiring is learned (Aso et al., 2014a). The interplay between such design principles could give rise to many forms of “division of labor” between genetics and learning. Likewise, it is believed that birdsong learning is driven by reinforcement learning using a specialized cost function that relies on comparison with a memorized version of a tutor’s song (Fiete et al., 2007), and also that it involves specialized structures for controlling song variability during learning (Aronov et al., 2011). These detailed pathways underlying the construction of cost functions for vocal learning are beginning to be mapped (Mandelblat-Cerf et al., 2014). Starting with simple systems, it should become possible to map the reward pathways and how they evolved and diversified, which would be a step on the way to understanding how the system learns.

These types of mapping efforts would be a first step towards the ability to create a concrete model of the brain’s optimization architecture. Our discussion here has focused on trying to anticipate, based on known neuroscience knowledge and on approaches becoming successful in machine learning, the kinds of local cost functions that the brain may rely on, and how specialized brain systems may enable efficient solutions to optimization problems. However, this framework-level discussion is not a formal specification, either of the architecture, or of a notion of biologically applied cost function that could be directly measured based on neural data. In order to move towards a more formal specification of the kind of model we are proposing here, it would be useful to map the architecture of the brains reward systems and to identify other biological pathways that may mediate the generation and delivery of error signals. Based on such maps, one could identify regions which are proposed to be subject to a single cost function. Otherwise, the problem of inference of the cost function, e.g., based on neural dynamics becomes ill-posed: one can define a local cost function for an arbitrary dynamics by integrating the trajectory of the system, but this approach in general lacks explanatory power and also, crucially, lacks any circuit-level relationship with the brains actual neural mechanisms of optimization, i.e., such a defined cost function does not necessarily correspond to the cost functions that the biological machinery is actually organized to optimize. Notably, some of the relevant biological pathways mediating cost functions and error signals may involve key biomolecular or gene expression aspects, not just real-time patterns of neural activity.

Another related consideration, in trying to formalize this type approach and to infer cost functions from neural measurements, is that not all neurons in the circuit may be subject to optimization: after all, some neurons may be needed to generate the error signals themselves, or to mediate the optimization process for other neurons, or to perform other unrelated functions. Furthermore, within a given region, there may be multiple sub-circuits subject to different optimization pressures. It is the claim that the brain actually has structured biological machinery to generate, route and apply specific cost functions that gives substance to our proposal, over and above the trivial claim that many kinds of dynamics can be viewed as optimizations, but our knowledge of this machinery is still limited. This is not to mention the difficulties involved in inferring cost functions in the presence of noise or constraints on the dynamics. Thus, one cannot blindly collect the neurons in an arbitrary region, measure their dynamics, and hope to infer their cost function by solving an inverse problem – instead, a rich interplay between structural mapping, dynamic mapping, hypothesis generation, modeling and perturbation is likely to be necessary in order to gain a detailed knowledge of which cost functions the brain uses and how it does so.

5.3 Hypothesis 3– Embedding within a pre-structured architecture

If different brain structures are performing distinct types of computations with a shared goal, then optimization of a joint cost function will take place with different dynamics in each area. If we focus on a higher level task, e.g., maximizing the probability of correctly detecting something, then we should find that basic feature detection circuits should learn when the features were insufficient for detection, that attentional routing structures should learn when a different allocation of attention would have improved detection and that memory structures should learn when items that matter for detection were not remembered. If we assume that multiple structures are participating in a joint computation, which optimizes an overall cost function (but see Hypothesis 2), then an understanding of the computational function of each area leads to a prediction of the measurable plasticity rules.

6. Neuroscience inspired machine learning

Machine learning may be equally transformed by neuroscience. Within the brain, a myriad of subsystems and layers work together to produce an agent that exhibits general intelligence. The brain is able to show intelligent behavior across a broad range of problems using only relatively small amounts of data. As such, progress at understanding the brain promises to improve machine learning. In this section, we review our three hypotheses about the brain and discuss how their elaboration might contribute to more powerful machine learning systems.

6.1 Hypothesis 1– Existence of cost functions

A good practitioner of machine learning should have a broad range of optimization methods at their disposal as different problems ask for different approaches. The brain, we have argued, is an implicit machine learning mechanism which has been evolved over millions of years. Consequently, we should expect the brain to be able to optimize cost functions efficiently, across many domains and kinds of data. Indeed, across different animal phyla, we even see convergent evolution of certain brain structures (Shimizu and Karten, 2013; Güntürkün and Bugnyar, 2016), e.g., the bird brain has no cortex yet has developed homologous structures which – as the linguistic feats of the African Grey Parrot demonstrate – can give rise to quite complex intelligence. It seems reasonable to hope to learn how to do truly general-purpose optimization by looking at the brain.

Indeed, there are multiple kinds of optimization that we may expect to discover by looking at the brain. At the hardware level, the brain clearly manages to optimize functions efficiently despite having slow hardware subject to molecular fluctuations, suggesting directions for improving the hardware of machine learning to be more energy efficient. At the level of learning rules, the brain solves an optimization problem in a highly nonlinear, non-differentiable, temporally stochastic, spiking system with massive numbers of feedback connections, a problem that we arguably still do not know how to efficiently solve for neural networks. At the architectural level, the brain can optimize certain kinds of functions based on very few stimulus presentations, operates over diverse timescales, and clearly uses advanced forms of active learning to infer causal structure in the world.

While we have discussed a range of theories (Hinton, 2007, 2016; Bengio et al., 2015a; Balduzzi et al., 2014; Roelfsema et al., 2010; O’Reilly, 1996; O’Reilly et al., 2014a; Körding and König, 2001; Lillicrap et al., 2014) for how the brain can carry out optimization, these theories are still preliminary. Thus, the first step is to understand whether the brain indeed performs multi-layer credit assignment in a manner that approximates full gradient descent, and if so, how it does this. Either way, we can expect that answer to impact machine learning. If the brain does not do some form of back-propagation, this suggests that machine learning may benefit from understanding the tricks that the brain uses to avoid having to do so. If, on the other hand, the brain does do backpropagation, then the underlying mechanisms clearly can support a very wide range of efficient optimization processes across many domains, including learning from rich temporal data-streams and via unsupervised mechanisms, and the architectures behind this will likely be of long-term value to machine learning61. Moreover, the search for biologically plausible forms of backpropagation has already led to interesting insights, such as the possibility of using random feedback weights (feedback alignment) in backpropagation (Lillicrap et al., 2014), or the unexpected power of internal FORCE learning in chaotic, spontaneously active recurrent networks (Sussillo and Abbott, 2009). This and other findings discussed here suggest that there are still fundamental things we don’t understand about backpropagation – which could potentially lead not only to more biologically plausible ways to train recurrent neural networks, but also to fundamentally simpler and more powerful ones.

6.2 Hypothesis 2– Biological fine-structure of cost functions

A good practitioner of machine learning has access to a broad range of learning techniques and thus implicitly is able to use many different cost functions. Some problems ask for clustering, others for extracting sparse variables, and yet others for prediction quality to be maximized. The brain also needs to be able to deal with many different kinds of datasets. As such, it makes sense for the brain to use a broad range of cost functions appropriate for the diverse set of tasks it has to solve to thrive in this world.

Many of the most notable successes of deep learning, from language modeling (Sutskever et al., 2011), to vision (Krizhevsky et al., 2012), to motor control (Levine et al., 2015), have been driven by end-to-end optimization of single task objectives. We have highlighted cases where machine learning has opened the door to multiplicities of cost functions that shape network modules into specialized roles. We expect that machine learning will increasingly adopt these practices in the future.

In computer vision, we have begun to see researchers re-appropriate neural networks trained for one task (e.g., ImageNet classification) and then deploy them on new tasks other than the ones they were trained for or for which more limited training data is available (Yosinski et al., 2014; Oquab et al., 2014; Noroozi and Favaro, 2016). We imagine this procedure will be generalized, whereby, in series and in parallel, diverse training problems, each with an associated cost function, are used to shape visual representations. For example, visual data streams can be segmented into elements like foreground vs. background, objects that can move of their own accord vs. those that cannot, all using diverse un-supervised criteria (Ullman et al., 2012; Poggio, 2015). Networks so trained can then be shared, augmented, and retrained on new tasks. They can be introduced as front-ends for systems that perform more complex objectives or even serve to produce cost functions for training other circuits (Watter et al., 2015). As a simple example, a network that can discriminate between images of different kinds of architectural structures (pyramid, staircase, etc.) could act as a critic for a building-construction network.

Scientifically, determining the order in which cost functions are engaged in the biological brain will inform machine learning about how to construct systems with intricate and hierarchical behaviors via divide-and-conquer approaches to learning problems, active learning, and more.

6.3 Hypothesis 3– Embedding within a pre-structured architecture

A good practitioner of machine learning should have a broad range of algorithms at their disposal. Some problems are efficiently solved through dynamic programming, others through hashing, and yet others through multi-layer back-propagation. The brain needs to be able to solve a broad range of learning problems without the luxury of being reprogrammed. As such, it makes sense for the brain to have specialized structures that allow it to rapidly learn to approximate a broad range of algorithms.

The first neural networks were simple single-layer systems, either linear or with limited nonlinearities (Rashevsky, 1939). The explosion of neural network research in the 1980s (McClelland et al., 1986) saw the advent of multilayer networks, followed by networks with layer-wise specializations as in convolutional networks (Fukushima, 1980; LeCun and Bengio, 1995). In the last two decades, architectures with specializations for holding variables stable in memory like the LSTM (Hochreiter and Schmidhuber, 1997), the control of content-addressable memory (Weston et al., 2014; Graves et al., 2014), and game playing by reinforcement learning (Mnih et al., 2015) have been developed. These networks, though formerly exotic, are now becoming mainstream algorithms in the toolbox of any deep learning practitioner. There is no sign that progress in developing new varieties of structured architectures is halting, and the heterogeneity and modularity of the brain’s circuitry suggests that diverse, specialized architectures are needed to solve the diverse challenges that confront a behaving animal.

The brain combines a jumble of specialized structures in a way that works. Solving this problem de novo in machine learning promises to be very difficult, making it attractive to be inspired by observations about how the brain does it. An understanding of the breadth of specialized structures, as well as the architecture that combines them, should be quite useful.

7. Did evolution separate cost functions from optimization algorithms?

Deep learning methods have taken the field of machine learning by storm. Driving the success is the separation of the problem of learning into two pieces: (1) An algorithm, backpropagation, that allows efficient distributed optimization, and (2) Approaches to turn any given problem into an optimization problem, by designing a cost function and training procedure which will result in the desired computation. If we want to apply deep learning to a new domain, e.g., playing Jeopardy, we do not need to change the optimization algorithm – we just need to cleverly set up the right cost function. A lot of work in deep learning, perhaps the majority, is now focused on setting up the right cost functions.

We hypothesize that the brain also acquired such a separation between optimization mechanisms and cost functions. If neural circuits, such as in cortex, implement a general-purpose optimization algorithm, then any improvement to that algorithm will improve function across the cortex. At the same time, different cortical areas solve different problems, so tinkering with each area’s cost function is likely to improve its performance. As such, functionally and evolutionarily separating the problems of optimization and cost function generation could allow evolution to produce better computations, faster. For example, common unsupervised mechanisms could be combined with area-specific reinforcement-based or supervised mechanisms and error signals, much as recent advances in machine learning have found natural ways to combine supervised and unsupervised objectives in a single system (Rasmus and Berglund, 2015).

This suggests interesting questions62: When did the division between cost functions and optimization algorithms occur? How is this separation implemented? How did innovations in cost functions and optimization algorithms evolve? And how do our own cost functions and learning algorithms differ from those of other animals?

There are many possibilities for how such a separation might be achieved in the brain. Perhaps the six-layered cortex represents a common optimization algorithm, which in different cortical areas is supplied with different cost functions. This claim is different from the claim that all cortical areas use a single unsupervised learning algorithm and achieve functional specificity by tuning the inputs to that algorithm. In that case, both the optimization mechanism and the implicit unsupervised cost function would be the same across areas (e.g., minimization of prediction error), with only the training data differing between areas, whereas in our suggestion, the optimization mechanism would be the same across areas but the cost function, as well as the training data, would differ. Thus the cost function itself would be like an ancillary input to a cortical area, in addition to its input and output data. Some cortical microcircuits could then, perhaps, compute the cost functions that are to be delivered to other cortical microcircuits. Another possibility is that, within the same circuitry, certain aspects of the wiring and learning rules specify an optimization mechanism and are relatively fixed across areas, while others specify the cost function and are more variable. This latter possibility would be similar to the notion of cortical microcircuits as molecularly and structurally configurable elements, akin to the cells in a field-programmable gate array (FPGA) (Marcus et al., 2014a, b), rather than a homogenous substrate. The biological nature of such a separation, if any exists, remains an open question. For example, individual parts of a neuron may separately deal with optimization and with the specification of the cost function, or different parts of a microcircuit may specialize in this way, or there may be specialized types of cells, some of which deal with signal processing and others with cost functions.

8. Conclusions

Due to the complexity and variability of the brain, pure “bottom up” analysis of neural data faces potential challenges of interpretation (Robinson, 1992; Jonas and Kording, 2016). Theoretical frameworks can potentially be used to constrain the space of hypotheses being evaluated, allowing researchers to first address higher-level principles and structures in the system, and then “zoom in” to address the details. Proposed “top down” frameworks for understanding neural computation include entropy maximization, efficient encoding, faithful approximation of Bayesian inference, minimization of prediction error, attractor dynamics, modularity, the ability to subserve symbolic operations, and many others (Bialek, 2002; Bialek et al., 2006; Friston, 2010; Knill and Pouget, 2004; Marcus, 2001; Pinker, 1999). Interestingly, many of the “top down” frameworks boil down to assuming that the brain simply optimizes a single, given cost function for a single computational architecture. We generalize these proposals assuming both a heterogeneous combination of cost functions unfolding over development, and a diversity of specialized sub-systems.

Much of neuroscience has focused on the search for “the neural code”, i.e., it has asked which stimuli are good at driving activity in individual neurons, regions, or brain areas. But, if the brain is capable of generic optimization of cost functions, then we need to be aware that rather simple cost functions can give rise to complicated stimulus responses. This potentially leads to a different set of questions. Are differing cost functions indeed a useful way to think about the differing functions of brain areas? How does the optimization of cost functions in the brain actually occur, and how is this different from the implementations of gradient descent in artificial neural networks? What additional constraints are present in the circuitry that remain fixed while optimization occurs? How does optimization interact with a structured architecture, and is this architecture similar to what we have sketched? Which computations are wired into the architecture, which emerge through optimization, and which arise from a mixture of those two extremes? To what extent are cost functions explicitly computed in the brain, versus implicit in its local learning rules? Did the brain evolve to separate the mechanisms involved in cost function generation from those involved in the optimization of cost functions, and if so how? What kinds of meta-level learning might the brain apply, to learn when and how to invoke different cost functions or specialized systems, among the diverse options available, to solve a given task? What crucial mechanisms are left out of this framework? A more in-depth dialog between neuroscience and machine learning could help elucidate some of these questions.

Much of machine learning has focused on finding ever faster ways of doing end-to-end gradient descent in neural networks. Neuroscience may inform machine learning at multiple levels. The optimization algorithms in the brain have undergone a couple of hundred million years of evolution. Moreover, the brain may have found ways of using heterogeneous cost functions that interact over development so as to simplify learning problems by guiding and shaping the outcomes of un-supervised learning. Lastly, the specialized structures evolved in the brain may inform us about ways of making learning efficient in a world that requires a broad range of computational problems to be solved over multiple timescales. Looking at the insights from neuroscience may help machine learning move towards general intelligence in a structured heterogeneous world with access to only small amounts of supervised data.

In some ways our proposal is opposite to many popular theories of neural computation. There is not one mechanism of optimization but (potentially) many, not one cost function but a host of them, not one kind of a representation but a representation of whatever is useful, and not one homogeneous structure but a large number of them. All these elements are held together by the optimization of internally generated cost functions, which allows these systems to make good use of one another. Rejecting simple unifying theories is in line with a broad range of previous approaches in AI. For example, Minsky and Papert’s work on the Society of Mind (Minsky, 1988) – and more broadly on ideas of genetically staged and internally bootstrapped development in connectionist systems (Minsky, 1977) – emphasizes the need for a system of internal monitors and critics, specialized communication and storage mechanisms, and a hierarchical organization of simple control systems.

At the time these early works were written, it was not yet clear that gradient-based optimization could give rise to powerful feature representations and behavioral policies. One can view our proposal as a renewed argument against simple end-to-end training and in favor of a heterogeneous approach. In other words, this framework could be viewed as proposing a kind of “society” of cost functions and trainable networks, permitting internal bootstrapping processes reminiscent of the Society of Mind (Minsky, 1988). In this view, intelligence is enabled by many computationally specialized structures, each trained with its own developmentally regulated cost function, where both the structures and the cost functions are themselves optimized by evolution like the hyperparameters in neural networks.

Acknowledgments

We thank Ken Hayworth for key discussions that led to this paper. We thank Ed Boyden, Chris Eliasmith, Gary Marcus, Shimon Ullman, Tomaso Poggio, Josh Tenenbaum, Dario Amodei, Alex Williams, Erik Peterson, Tom Dean, David Sussillo, Matthew Botvinick, Joscha Bach, Mohammad Gheshlaghi Azar, Joshua Glaser, Marco Nardini, Ali Hummos, David Markowitz, David Rolnick, Sam Rodriques, Nick Barry, Matthew Larkum, Walter Senn, Eric Drexler, Vikash Mansinghka, Darcy Wayne, Lyra and Neo Marblestone, and all of the participants of a Kavli Salon on Cortical Computation (Feb/Oct 2015) for helpful comments. We thank Miles Brundage for an excellent Twitter feed of deep learning papers.

References

  1. ↵
    LF Abbott and KI Blum. Functional significance of long-term potentiation for sequence learning and prediction. Cerebral Cortex, 1996. URL http://cercor.oxfordjournals.org/content/6/3/406.short.
  2. ↵
    LF Abbott, B DePasquale, and RM Memmesheimer. Building Functional Networks of Spiking Model Neurons. neurotheory.columbia.edu, 2016. URL http://www.neurotheory.columbia.edu/Larry/SpikingNetworkReview.pdf.
  3. ↵
    DH Ackley, GE Hinton, and TJ Sejnowski. A learning algorithm for Boltzmann machines. Cognitive science, 1985. URL http://www.sciencedirect.com/science/article/pii/S0364021385800124.
  4. ↵
    Daniel E Acuna, Nicholas F Wymbs, Chelsea A Reynolds, Nathalie Picard, Robert S Turner, Peter L Strick, Scott T Grafton, and Konrad P Kording. Multifaceted aspects of chunking enable robust algorithms. Journal of neurophysiology, 112(8): 1849–1856, 2014.
    OpenUrlCrossRefPubMed
  5. ↵
    Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015.
  6. ↵
    K Allen, S Ibara, and A Seymour. Abstract structural representations of goal-directed behavior. Psychological …, 2010. URL http://pss.sagepub.com/content/21/10/1518.short.
  7. ↵
    Charles H Anderson and David C Van Essen. Shifter circuits: a computational strategy for dynamic aspects of visual processing. Proceedings of the National Academy of Sciences, 84(17): 6297–6301, 1987.
    OpenUrlAbstract/FREE Full Text
  8. ↵
    John R Anderson. How Can the Human Mind Occur in the Physical Universe? Oxford University Press, 2007. ISBN 0198043538. URL http://books.google.com/books?id=eYZXEtfplyAC{&}pgis=1.
  9. ↵
    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Deep compositional question answering with neural module networks. arXiv preprint arXiv:1511.02799, 2015.
  10. ↵
    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705, 2016.
  11. ↵
    Alessandra Angelucci, Jonathan B Levitt, Emma JS Walton, Jean-Michel Hupe, Jean Bullier, and Jennifer S Lund. Circuits for local and global signal integration in primary visual cortex. The Journal of Neuroscience, 22(19): 8633–8646, 2002.
    OpenUrlAbstract/FREE Full Text
  12. ↵
    Fabio Anselmi, Joel Z. Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio. Unsupervised learning of invariant representations. Theoretical Computer Science, jun 2015. ISSN 03043975. doi: 10.1016/j.tcs.2015.06.048. URL http://www.sciencedirect.com/science/article/pii/S0304397515005587.
    OpenUrlCrossRef
  13. ↵
    Srdjan D Antic, Wen-Liang Zhou, Anna R Moore, Shaina M Short, and Katerina D Ikonomu. The decade of the dendritic nmda spike. Journal of neuroscience research, 88(14): 2991–3001, 2010.
    OpenUrlCrossRefPubMed
  14. ↵
    O Arancio, M Kiebler, C J Lee, V Lev-Ram, R Y Tsien, E R Kandel, and R D Hawkins. Nitric oxide acts directly in the presynaptic neuron to produce long-term potentiation in cultured hippocampal neurons. Cell, 87(6): 1025–35, dec 1996. ISSN 0092–8674. URL http://www.ncbi.nlm.nih.gov/pubmed/8978607.
    OpenUrlCrossRefPubMedWeb of Science
  15. ↵
    Dmitriy Aronov, Lena Veit, Jesse H Goldberg, and Michale S Fee. Two distinct modes of forebrain circuit dynamics underlie temporal patterning in the vocalizations of young songbirds. The Journal of neuroscience : the official journal of the Society for Neuroscience, 31(45): 16353–68, nov 2011. ISSN 1529–2401. doi: 10.1523/JNEUROSCI.3009-11. 2011.
    OpenUrlAbstract/FREE Full Text
  16. ↵
    Sanjeev Arora, Yingyu Liang, and Tengyu Ma. Why are deep nets reversible: A simple theory, with implications for training. nov 2015. URL http://arxiv.org/abs/1511.05653.
  17. ↵
    F Gregory Ashby, John M Ennis, and Brian J Spiering. A neurobiological theory of automaticity in perceptual categorization. Psychological review, 114(3): 632, 2007.
    OpenUrlCrossRefPubMedWeb of Science
  18. ↵
    F Gregory Ashby, Benjamin O Turner, and Jon C Horvitz. Cortical and basal ganglia contributions to habit learning and automaticity. Trends in cognitive sciences, 14(5): 208–15, may 2010. ISSN 1879–307X. doi: 10.1016/j.tics.2010.02.001. URL http://www.cell.com/article/S1364661310000306/fulltext.
    OpenUrlCrossRefPubMedWeb of Science
  19. ↵
    Yoshinori Aso, Daisuke Hattori, Yang Yu, Rebecca M Johnston, Nirmala A Iyer, Teri-T B Ngo, Heather Dionne, L F Abbott, Richard Axel, Hiromu Tanimoto, and Gerald M Rubin. The neuronal architecture of the mushroom body provides a logic for associative learning. eLife, 3:e04577, jan 2014a. ISSN 2050–084X. doi: 10.7554/eLife.04577. URL http://elifesciences.org/content/3/e04577.abstract.
    OpenUrlCrossRefPubMed
  20. ↵
    Yoshinori Aso, Divya Sitaraman, Toshiharu Ichinose, Karla R Kaun, Katrin Vogt, Ghislain Belliart-Guérin, Pierre-Yves Plaçais, Alice A Robie, Nobuhiro Yamagata, Christopher Schnaitmann, William J Rowell, Rebecca M Johnston, Teri-T B Ngo, Nan Chen, Wyatt Korff, Michael N Nitabach, Ulrike Heberlein, Thomas Preat, Kristin M Branson, Hiromu Tanimoto, and Gerald M Rubin. Mushroom body output neurons encode valence and guide memory-based action selection in Drosophila. eLife, 3:e04580, jan 2014b. ISSN 2050–084X. doi: 10.7554/eLife.04580. URL http://elifesciences.org/content/3/e04580.abstract.
    OpenUrlCrossRefPubMed
  21. ↵
    J Ba and R Caruana. Do deep nets really need to be deep? Advances in neural information processing…, 2014. URL http://papers.nips.cc/paper/5484-do-deep-nets-really-need-to-be-deep.
  22. ↵
    Joscha Bach. Modeling Motivation in MicroPsi 2. Artificial General Intelligence, 2015. URL http://link.springer.com/chapter/10.1007/978–3–319–21365–1{_}1.
  23. ↵
    1. Tarek Besold,
    2. Artur dAvila Garcez,
    3. Gary Marcus, and
    4. Risto Miikkulainen
    Joscha Bach and Priska Herger. Request confirmation networks for neuro-symbolic script execution. In Tarek Besold, Artur dAvila Garcez, Gary Marcus, and Risto Miikkulainen, editors, Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches at NIPS, 2015. URL http://www.neural-symbolic.org/CoCo2015/.
  24. ↵
    Renée Baillargeon, Rose M Scott, and Lin Bian. Psychological reasoning in infancy. Annual review of psychology, 67:159–186, 2016.
    OpenUrlCrossRefPubMed
  25. ↵
    Jan Balaguer, Hugo Spiers, Demis Hassabis, and Christopher Summerfield. Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network. Neuron, 90(4): 893–903, may 2016. ISSN 08966273. doi: 10.1016/j.neuron.2016.03.037. URL http://www.cell.com/article/S0896627316300575/fulltext.
    OpenUrlCrossRef
  26. ↵
    Daniel Baldauf and Robert Desimone. Neural mechanisms of object-based attention. Science (New York, N.Y.), 344(6182): 424–7, apr 2014. ISSN 1095–9203. doi: 10.1126/science.1247003. URL http://www.ncbi.nlm.nih.gov/pubmed/24763592.
    OpenUrlAbstract/FREE Full Text
  27. ↵
    Pierre Baldi and Peter Sadowski. The Ebb and Flow of Deep Learning: a Theory of Local Learning. jun 2015. URL http://arxiv.org/abs/1506.06472.
  28. ↵
    D Balduzzi. Cortical prediction markets. Proceedings of the 2014 international conference on …, 2014. URL http://dl.acm.org/citation.cfm?id=2617449.
  29. ↵
    David Balduzzi, Hastagiri Vanchinathan, and Joachim Buhmann. Kickback cuts Backprop’s red-tape: Biologically plausible credit assignment in neural networks. page 7, nov 2014. URL http://arxiv.org/abs/1411.6191.
  30. ↵
    Cornelia I Bargmann. Beyond the connectome: how neuromodulators shape neural circuits. BioEs-says : news and reviews in molecular, cellular and developmental biology, 34(6): 458–65, jun 2012. ISSN 1521–1878. doi: 10.1002/bies.201100185. URL http://www.ncbi.nlm.nih.gov/pubmed/22396302.
    OpenUrlCrossRefPubMedWeb of Science
  31. ↵
    Cornelia I Bargmann and Eve Marder. From the connectome to brain function. Nature Methods, 10(6): 483–490, may 2013. ISSN 1548–7091. doi: 10.1038/nmeth.2451. URL http://dx.doi.org/10.1038/nmeth.2451.
    OpenUrlCrossRefPubMedWeb of Science
  32. ↵
    Andre Moraes Bastos, Julien Vezoli, Conrado Arturo Bosman, Jan-Mathijs Schoffelen, Robert Oostenveld, Jarrod Robert Dowdall, Peter De Weerd, Henry Kennedy, and Pascal Fries. Visual areas exert feedforward and feedback influences through distinct frequency channels. Neuron, 85(2): 390–401, 2015.
    OpenUrlCrossRefPubMed
  33. ↵
    Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences of the United States of America, 110(45): 18327–32, nov 2013. ISSN 1091–6490. doi: 10.1073/pnas.1306572110. URL http://www.pnas.org/content/110/45/18327.short.
    OpenUrlAbstract/FREE Full Text
  34. ↵
    Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356): 161–163, 1992.
    OpenUrlCrossRefPubMed
  35. ↵
    Trevor Bekolay, Carter Kolbeck, and Chris Eliasmith. Simultaneous unsupervised and supervised learning of cognitive functions in biologically plausible spiking neural networks. In Proceedings of the 35th annual conference of the cognitive science society, pages 169–174, 2013.
  36. ↵
    Yoshua Bengio. How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation. jul 2014. URL http://arxiv.org/abs/1407.7906.
  37. ↵
    Yoshua Bengio and Asja Fischer. Early Inference in Energy-Based Models Approximates Back-Propagation. oct 2015. URL http://arxiv.org/abs/1510.02777.
  38. ↵
    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM, 2009.
  39. ↵
    Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, and Zhouhan Lin. Towards Biologically Plausible Deep Learning. feb 2015a. URL http://arxiv.org/abs/1502.04156.
  40. ↵
    Yoshua Bengio, Thomas Mesnard, Asja Fischer, Saizheng Zhang, and Yuhuai Wu. STDP as presynaptic activity times rate of change of postsynaptic activity. sep 2015b. URL http://arxiv.org/abs/1509.05936.
  41. ↵
    Yoshua Bengio, Benjamin Scellier, Olexa Bilaniuk, Joao Sacramento, and Walter Senn. Feedforward initialization for fast inference of deep generative networks is biologically plausible. arXiv preprint arXiv:1606.01651, 2016.
  42. ↵
    Vladimir K Berezovskii, Jonathan J Nassi, and Richard T Born. Segregation of feedforward and feedback projections in mouse visual cortex. Journal of Comparative Neurology, 519(18): 3672–3683, 2011.
    OpenUrlCrossRefPubMed
  43. ↵
    Robert C Berwick, Gabriël J L Beckers, Kazuo Okanoya, and Johan J Bolhuis. A Bird’s Eye View of Human Language Evolution. Frontiers in evolutionary neuroscience, 4:5, jan 2012. ISSN 1663–070X. doi: 10.3389/fnevo.2012.00005. URL http://journal.frontiersin.org/article/10.3389/fnevo.2012.00005/abstract.
    OpenUrlCrossRef
  44. ↵
    W Bialek. Thinking about the brain. Physics of bio-molecules and cells. Physique des …, 2002. URL http://link.springer.com/chapter/10.1007/3–540–45701–1{_}12.
  45. ↵
    William Bialek, Rob De Ruyter Van Steveninck, and Naftali Tishby. Efficient representation as a design principle for neural coding and computation. In 2006 IEEE International Symposium on Information Theory, pages 659–663. IEEE, jul 2006. ISBN 1–4244–0505-X. doi: 10.1109/ISIT.2006.261867. URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=4036045.
    OpenUrlCrossRef
  46. ↵
    H Biilthoff, J Little, and T Poggio. A parallel algorithm for real-time computation of optical flow. Nature, 337(6207): 549–553, 1989.
    OpenUrlCrossRefPubMed
  47. ↵
    ErikB. Bloss, MarkS. Cembrowski, Bill Karsh, Jennifer Colonell, RichardD. Fetter, and Nelson Spruston. Structured Dendritic Inhibition Supports Branch-Selective Integration in CA1 Pyramidal Cells. Neuron, 89(5): 1016–30, feb 2016. ISSN 08966273. doi: 10.1016/j.neuron.2016.01.029. URL http://www.cell.com/article/S0896627316000544/fulltext.
    OpenUrlCrossRefPubMed
  48. ↵
    Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016.
  49. ↵
    Bruce Bobier, Terrence C Stewart, and Chris Eliasmith. A unifying mechanistic model of selective attention in spiking neurons. PLoS Comput Biol, 10(6):e1003577, 2014.
    OpenUrlCrossRefPubMed
  50. ↵
    Nick Bostrom. Cortical integration: Possible solutions to the binding and linking problems in perception, reasoning and long term memory. 1996. URL http://philpapers.org/rec/BOSCIP.
  51. ↵
    Matthew Botvinick and Ari Weinstein. Model-based hierarchical reinforcement learning and human action control. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 369(1655):20130480–, nov 2014. ISSN 1471–2970. doi: 10.1098/rstb.2013.0480. URL http://rstb.royalsocietypublishing.org/content/369/1655/20130480.
    OpenUrlCrossRefPubMed
  52. ↵
    MM Botvinick, Y Niv, and AC Barto. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 2009. URL http://www.sciencedirect.com/science/article/pii/S0010027708002059.
  53. ↵
    Guillaume Bouchard, Théo Trouillon, Julien Perez, and Adrien Gaidon. Accelerating stochastic gradient descent via online learning to sample. arXiv preprint arXiv:1506.09016, 2015.
  54. ↵
    Ralph Bourdoukan and Sophie Denève. Enforcing balance allows local supervised learning in spiking recurrent networks. In Advances in Neural Information Processing Systems, pages 982–990, 2015.
  55. ↵
    V Braitenberg and A Schutz. Anatomy of the cortex: studies of brain function, 1991.
  56. ↵
    Johanni Brea and Wulfram Gerstner. Does computational neuroscience need new synaptic learning paradigms? Current Opinion in Behavioral Sciences, 11:61–66, 2016.
    OpenUrl
  57. ↵
    Johanni Brea, Alexisz Tamás Gaál, Robert Urbanczik, and Walter Senn. Prospective coding by spiking neurons. PLOS Comput Biol, 12(6):e1005003, 2016.
    OpenUrlCrossRefPubMed
  58. ↵
    J. Gavin Bremner, Alan M. Slater, and Scott P. Johnson. Perception of Object Persistence: The Origins of Object Permanence in Infancy. Child Development Perspectives, 9(1): 7–13, mar 2015. ISSN 17508592. doi: 10.1111/cdep.12098. URL http://doi.wiley.com/10.1111/cdep.12098.
    OpenUrlCrossRef
  59. ↵
    Carlos SN Brito and Wulfram Gerstner. Nonlinear hebbian learning as a unifying principle in receptive field formation. arXiv preprint arXiv:1601.00701, 2016.
  60. ↵
    Tobias Brosch, Heiko Neumann, and Pieter R Roelfsema. Reinforcement Learning of Linking and Tracing Contours in Recurrent Neural Networks. PLoS computational biology, 11(10):e1004489, oct 2015. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1004489.
    OpenUrlCrossRef
  61. ↵
    Thackery I. Brown, Valerie A. Carr, Karen F. LaRocque, Serra E. Favila, Alan M. Gordon, Ben Bowles, Jeremy N. Bailenson, and Anthony D. Wagner. Prospective representation of navigational goals in the human hippocampus. Science, 352(6291): 1323–1326, 2016. ISSN 0036–8075. doi: 10.1126/science.aaf0784. URL http://science.sciencemag.org/content/352/6291/1323.
    OpenUrlAbstract/FREE Full Text
  62. ↵
    Robert M Brownstone, Tuan V Bui, and Nicolas Stifani. Spinal circuits for motor learning. Current opinion in neurobiology, 33:166–173, 2015.
    OpenUrlCrossRefPubMed
  63. ↵
    Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics as sampling: a model for stochastic computation in recurrent networks of spiking neurons. PLoS computational biology, 7(11):e1002211, nov 2011. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1002211.
    OpenUrlCrossRefPubMed
  64. ↵
    D V Buonomano and M M Merzenich. Temporal information transformed into a spatial code by a neural network with realistic properties. Science (New York, N.Y.), 267(5200): 1028–30, feb 1995. ISSN 0036–8075. URL http://www.ncbi.nlm.nih.gov/pubmed/7863330.
    OpenUrl
  65. ↵
    Timothy J Buschman and Earl K Miller. Shifting the spotlight of attention: evidence for discrete computations in cognition. Frontiers in human neuroscience, 4:194, jan 2010. ISSN 1662–5161. doi: 10.3389/fnhum.2010.00194.
    OpenUrlCrossRefPubMed
  66. ↵
    Timothy J Buschman and Earl K Miller. Goal-direction and top-down control. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 369(1655), nov 2014. ISSN 1471–2970. doi: 10.1098/rstb. 2013.0471. URL http://www.ncbi.nlm.nih.gov/pubmed/25267814.
    OpenUrlCrossRef
  67. ↵
    Keith A Bush. An echo state model of non-markovian reinforcement learning. 2007.
  68. ↵
    György Buzsáki. Neural syntax: cell assemblies, synapsembles, and readers. Neuron, 68(3): 362–385, 2010.
    OpenUrlCrossRefPubMedWeb of Science
  69. ↵
    György Buzsáki and Edvard I Moser. Memory, navigation and theta rhythm in the hippocampalentorhinal system. Nature neuroscience, 16(2): 130–138, feb 2013. ISSN 1546–1726. doi: 10.1038/nn.3304. URL http://dx.doi.org/10.1038/nn.3304.
    OpenUrlCrossRefPubMed
  70. ↵
    Denise J Cai, Daniel Aharoni, Tristan Shuman, Justin Shobe, Jeremy Biane, Weilin Song, Brandon Wei, Michael Veshkini, Mimi La-Vu, Jerry Lou, et al. A shared neural ensemble links distinct contextual memories encoded close in time. Nature, 534(7605): 115–118, 2016.
    OpenUrlCrossRefPubMed
  71. ↵
    EM Callaway. Feedforward, feedback and inhibitory connections in primate visual cortex. Neural Networks, 2004. URL http://www.sciencedirect.com/science/article/pii/S0893608004000887.
  72. ↵
    Sophie J C Caron, Vanessa Ruta, L F Abbott, and Richard Axel. Random convergence of olfactory inputs in the Drosophila mushroom body. Nature, 497(7447): 113–7, may 2013. ISSN 1476–4687. doi: 10.1038/nature12063. URL http://dx.doi.org/10.1038/nature12063.
    OpenUrlCrossRefPubMedWeb of Science
  73. ↵
    Liang-Chieh Chen, Alexander G. Schwing, Alan L. Yuille, and Raquel Urtasun. Learning Deep Structured Models. page 11, jul 2014. URL http://arxiv.org/abs/1407.2538.
  74. ↵
    Sharat Chikkerur, Thomas Serre, Cheston Tan, and Tomaso Poggio. What and where: a Bayesian inference theory of attention. Vision research, 50 (22):2233–47, oct 2010. ISSN 1878–5646. doi: 10.1016/j.visres.2010.05.013. URL http://www.ncbi.nlm.nih.gov/pubmed/20493206.
    OpenUrlCrossRefPubMedWeb of Science
  75. ↵
    Choo and Eliasmith. A Spiking Neuron Model of Serial-Order Recall. 32nd Annual Conference of the Cognitive Science Society, 2010. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.353.1190.
  76. ↵
    Alexander A Chubykin, Emma B Roach, Mark F Bear, and Marshall G Hussain Shuler. A cholinergic mechanism for reward timing within primary visual cortex. Neuron, 77(4): 723–35, feb 2013. ISSN 1097–4199. doi: 10.1016/j.neuron.2012.12.039. URL http://www.cell.com/article/S0896627313000470/fulltext.
    OpenUrlCrossRefPubMedWeb of Science
  77. ↵
    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. dec 2014. URL http://arxiv.org/abs/1412.3555.
  78. ↵
    Joseph Cichon and Wen-Biao Gan. Branch-specific dendritic ca2+ spikes cause persistent synaptic plasticity. Nature, 520(7546): 180–185, 2015.
    OpenUrlCrossRefPubMed
  79. ↵
    Nicola S Clayton and Anthony Dickinson. Episodic-like memory during cache recovery by scrub jays. Nature, 395(6699): 272–274, 1998.
    OpenUrlCrossRefPubMedWeb of Science
  80. ↵
    Claudia Clopath and Wulfram Gerstner. Voltage and spike timing interact in stdp–a unified model. Spike-timing dependent plasticity, page 294, 2010.
  81. ↵
    Raphael Cohn, Ianessa Morantte, and Vanessa Ruta. Coordinated and Compartmentalized Neuromodulation Shapes Sensory Processing in Drosophila. Cell, 163(7): 1742–1755, dec 2015. ISSN 00928674. doi: 10.1016/j.cell.2015.11.019. URL http://www.cell.com/article/S0092867415014993/fulltext.
    OpenUrlCrossRefPubMed
  82. ↵
    A Colino and JV Halliwell. Differential modulation of three separate k-conductances in hippocampal ca1 neurons by serotonin. Nature, 328(6125): 73–77, 1987.
    OpenUrlCrossRefPubMedWeb of Science
  83. ↵
    Alexandra O Constantinescu, Jill X OReilly, and Timothy EJ Behrens. Organizing conceptual knowledge in humans with a gridlike code. Science, 352(6292): 1464–1468, 2016.
    OpenUrlAbstract/FREE Full Text
  84. ↵
    David Daniel Cox and Thomas Dean. Neural networks and neuroscience-inspired computer vision. Current biology : CB, 24(18):R921–9, sep 2014. ISSN 1879–0445. doi: 10.1016/j.cub.2014.08.026. URL http://www.sciencedirect.com/science/article/pii/S0960982214010392.
    OpenUrlCrossRefPubMed
  85. ↵
    F Crick. The recent excitement about neural networks. Nature, 337(6203): 129–32, jan 1989. ISSN 0028–0836. doi: 10.1038/337129a0. URL http://dx.doi.org/10.1038/337129a0.
    OpenUrlCrossRefPubMed
  86. ↵
    Francis C Crick and Christof Koch. What is the function of the claustrum? Philosophical Transactions of the Royal Society of London B: Biological Sciences, 360(1458):1271–1279, 2005.
    OpenUrlCrossRefPubMed
  87. ↵
    Sébastien M Crouzet and Simon J Thorpe. Low-level cues and ultra-fast face detection. Frontiers in psychology, 2:342, jan 2011. ISSN 1664–1078. doi: 10.3389/fpsyg.2011.00342.
    OpenUrlCrossRef
  88. ↵
    Yuwei Cui, Chetan Surpur, Subutai Ahmad, and Jeff Hawkins. Continuous online sequence learning with an unsupervised neural network model. dec 2015. URL http://arxiv.org/abs/1512.05463.
  89. ↵
    Maria C Dadarlat, Joseph E O’Doherty, and Philip N Sabes. A learning-based approach to artificial sensory feedback leads to optimal integration. Nature neuroscience, 18(1): 138–144, 2015.
    OpenUrlCrossRefPubMed
  90. ↵
    Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative Long Short-Term Memory. feb 2016. URL http://arxiv.org/abs/1602.03032.
  91. ↵
    Nathaniel D Daw, Yael Niv, and Peter Dayan. Actions, policies, values and the basal ganglia. Recent breakthroughs in basal ganglia research, pages 91–106, 2006.
  92. ↵
    Peter Dayan. Twenty-five lessons from computational neuromodulation. Neuron, 76(1): 240–56, oct 2012. ISSN 1097–4199. doi: 10.1016/j.neuron.2012.09.027. URL http://www.sciencedirect.com/science/article/pii/S0896627312008628.
    OpenUrlCrossRefPubMedWeb of Science
  93. ↵
    T Dean. A computational model of the cerebral cortex. Proceedings of the National Conference on Artificial …, 2005. URL http://www.aaai.org/Papers/AAAI/2005/AAAI05–148.pdf.
  94. ↵
    Stanislas Dehaene, Florent Meyniel, Catherine Wacongne, Liping Wang, and Christophe Pallier. The Neural Representation of Sequences: From Transition Probabilities to Algebraic Patterns and Linguistic Trees. Neuron, 88(1): 2–19, oct 2015. ISSN 08966273. doi: 10.1016/j.neuron.2015.09.019. URL http://www.sciencedirect.com/science/article/pii/S089662731500776X.
    OpenUrlCrossRefPubMed
  95. ↵
    Tessa M Dekker and Marko Nardini. Risky visuo-motor choices during rapid reaching in childhood. Developmental science, 2015.
  96. ↵
    Olivier Delalleau and Yoshua Bengio. Shallow vs. Deep Sum-Product Networks. In Advances in Neural Information Processing Systems, pages 666–674, 2011. URL http://papers.nips.cc/paper/4350-shallow-vs-deep-sum-product-networks.
  97. ↵
    B DePasquale, MM Churchland, and LF Abbott. Using firing-rate dynamics to train recurrent networks of spiking model neurons. arXiv preprint arXiv: …, 2016. URL http://arxiv.org/abs/1601.07620.
  98. ↵
    T DeWolf and C Eliasmith. The neural optimal control hierarchy for motor control. Journal of neural engineering, 8(6): 065009, dec 2011. ISSN 1741–2552. doi: 10.1088/1741-2560/8/6/065009. URL http://www.ncbi.nlm.nih.gov/pubmed/22056418.
    OpenUrlCrossRefPubMed
  99. ↵
    James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3): 415–434, 2012.
    OpenUrlCrossRefPubMedWeb of Science
  100. ↵
    Rodney J Douglas and Kevan A C Martin. Neuronal circuits of the neocortex. Annual review of neuroscience, 27:419–51, jan 2004. ISSN 0147–006X. doi: 10.1146/annurev.neuro.27.070203.144152. URL http://www.ncbi.nlm.nih.gov/pubmed/15217339.
    OpenUrlCrossRefPubMedWeb of Science
  101. ↵
    Kenji Doya. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural networks, 12(7): 961–974, 1999.
    OpenUrlCrossRefPubMedWeb of Science
  102. ↵
    Joshua T Dudman, David Tsay, and Steven A Siegelbaum. A role for synaptic inputs at distal dendrites: instructive signals for hippocampal long-term plasticity. Neuron, 56(5): 866–879, 2007.
    OpenUrlCrossRefPubMedWeb of Science
  103. ↵
    Serge O Dumoulin and Brian A Wandell. Population receptive field estimates in human visual cortex. Neuroimage, 39(2): 647–660, 2008.
    OpenUrlCrossRefPubMedWeb of Science
  104. ↵
    Chris Eliasmith. How to Build a Brain: A Neural Architecture for Biological Cognition. Oxford University Press, 2013. ISBN 0199794545. URL http://books.google.com/books?id=BK0YRJPmuzgC{&}pgis=1.
  105. ↵
    Chris Eliasmith and Charles H. Anderson. Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems. MIT Press, 2004. ISBN 0262550601. URL http://books.google.com/books?id=J6jz9s4kbfIC{&}pgis=1.
  106. ↵
    Chris Eliasmith and James Martens. Normalization for probabilistic inference with neurons. Biological cybernetics, 104(4–5):251–262, 2011.
    OpenUrlPubMed
  107. ↵
    Chris Eliasmith, Terrence C Stewart, Xuan Choo, Trevor Bekolay, Travis DeWolf, Yichuan Tang, Charlie Tang, and Daniel Rasmussen. A large-scale model of the functioning brain. Science (New York, N.Y.), 338(6111): 1202–5, nov 2012. ISSN 1095–9203. doi: 10.1126/science.1225266. URL http://www.sciencemag.org/content/338/6111/1202.
    OpenUrlAbstract/FREE Full Text
  108. ↵
    Stephen T Emlen. Migratory orientation in the indigo bunting, passerina cyanea: part i: evidence for use of celestial cues. The Auk, 84(3): 309–342, 1967.
    OpenUrlCrossRef
  109. ↵
    Pierre Enel, Emmanuel Procyk, René Quilodran, and Peter Ford Dominey. Reservoir computing properties of neural dynamics in prefrontal cortex. PLOS Comput Biol, 12(6):e1004967, 2016.
    OpenUrlCrossRefPubMed
  110. ↵
    D Erhan and PA Manzagol. The difficulty of training deep architectures and the effect of unsupervised pre-training. International …, 2009. URL http://machinelearning.wustl.edu/mlpapers/paper{_}files/AISTATS09{_}ErhanMBBV.pdf.
  111. ↵
    SM Eslami, N Heess, and T Weber. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. arXiv preprint arXiv: …, 2016. URL http://arxiv.org/abs/1603.08575.
  112. ↵
    Caitlin M Fausey, Swapnaa Jayaraman, and Linda B Smith. From faces to hands: Changing visual input in the first two years. Cognition, 152:101–107, 2016.
    OpenUrl
  113. ↵
    Daniel J Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex, 1(1): 1–47, 1991.
    OpenUrlCrossRefPubMedWeb of Science
  114. ↵
    David Ferster and Kenneth D. Miller. Neural Mechanisms of Orientation Selectivity in the Visual Cortex. nov 2003. URL http://www.annualreviews.org/doi/abs/10.1146/annurev.neuro.23.1.441.
  115. ↵
    Eberhard E Fetz. Operant conditioning of cortical unit activity. Science, 163(3870): 955–958, 1969.
    OpenUrlAbstract/FREE Full Text
  116. ↵
    Eberhard E Fetz. Volitional control of neural activity: implications for brain–computer interfaces. The Journal of physiology, 579(3): 571–579, 2007.
    OpenUrlCrossRefPubMedWeb of Science
  117. ↵
    Ila R Fiete and H Sebastian Seung. Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physical review letters, 97(4): 048104, jul 2006. ISSN 0031–9007. doi: 10.1103/PhysRevLett.97.048104. URL http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.97.048104.
    OpenUrlCrossRefPubMed
  118. ↵
    Ila R Fiete, Michale S Fee, and H Sebastian Seung. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. Journal of neurophysiology, 98 (4):2038–57, oct 2007. ISSN 0022–3077. doi: 10.1152/jn.01311.2006. URL http://www.ncbi.nlm.nih.gov/pubmed/17652414.
    OpenUrlCrossRefPubMedWeb of Science
  119. ↵
    Ila R Fiete, Walter Senn, Claude Z H Wang, and Richard H R Hahnloser. Spike-time-dependent plasticity and heterosynaptic competition organize networks to produce long scale-free sequences of neural activity. Neuron, 65(4): 563–76, feb 2010. ISSN 1097–4199. doi: 10.1016/j.neuron. 2010.02.003. URL http://www.ncbi.nlm.nih.gov/pubmed/20188660.
    OpenUrlCrossRefPubMedWeb of Science
  120. ↵
    Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. arXiv preprint arXiv:1603.00448, 2016.
  121. ↵
    GT Finnerty and MN Shadlen. Time in Cortical Circuits. The Journal of …, 2015. URL https://www.jneurosci.org/content/35/41/13912.full.
  122. ↵
    Janet Dean Fodor and Carrie Crowther. Understanding stimulus poverty arguments. The Linguistic Review, 18(1–2):105–145, 2002.
    OpenUrl
  123. ↵
    Peter Földiák. Learning Invariance from Transformation Sequences, mar 2008. URL http://www.mitpressjournals.org/doi/abs/10.1162/neco.1991.3.2.194{#}.VofznO8rKHo.
  124. ↵
    D. J. Foster, R. G. M. Morris, Peter Dayan, and Centre For Neuroscience. Models of Hippocampally Dependent Navigation, Using The Temporal Difference Learning Rule. sep 2000.
  125. ↵
    Julien Fournier, Christian M Müller, and Gilles Laurent. Looking for the roots of cortical sensory computation in three-layered cortices. Current opinion in neurobiology, 31:119–126, 2015.
    OpenUrlCrossRefPubMed
  126. ↵
    Steven L Franconeri, Zenon W Pylyshyn, and Brian J Scholl. A simple proximity heuristic allows tracking of multiple objects through occlusion. Attention, Perception, & Psychophysics, 74(4): 691–702, 2012.
    OpenUrl
  127. ↵
    Michael J Frank and David Badre. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cerebral cortex (New York, N.Y. : 1991), 22(3): 509–26, mar 2012. ISSN 1460–2199. doi: 10.1093/cercor/bhr114.
    OpenUrlCrossRefPubMedWeb of Science
  128. ↵
    Steven M Frankland and Joshua D Greene. An architecture for encoding sentence meaning in left mid-superior temporal cortex. Proceedings of the National Academy of Sciences of the United States of America, 112(37): 11732–11737, aug 2015. ISSN 1091–6490. doi: 10.1073/pnas.1421236112. URL http://www.pnas.org/content/112/37/11732.full.
    OpenUrlAbstract/FREE Full Text
  129. ↵
    Mathias Franzius, Henning Sprekeler, and Laurenz Wiskott. Slowness and sparseness lead to place, head-direction, and spatial-view cells. PLoS computational biology, 3(8):e166, aug 2007. ISSN 1553–7358. doi: 10.1371/journal.pcbi.0030166. URL http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.0030166.
    OpenUrlCrossRefPubMed
  130. ↵
    K Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 2010. URL http://www.nature.com/nrn/journal/v11/n2/abs/nrn2787.html.
  131. ↵
    KJ Friston and KE Stephan. Free-energy and the brain. Synthese, 2007. URL http://link.springer.com/article/10.1007/s11229–007–9237-y.
  132. ↵
    Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4): 193–202, 1980.
    OpenUrlCrossRefPubMedWeb of Science
  133. ↵
    Mathieu N Galtier and Gilles Wainrib. A biological gradient descent for prediction through a combination of stdp and homeostatic plasticity. Neural computation, 25(11): 2815–2832, 2013.
    OpenUrl
  134. ↵
    Tao Gao, Daniel Harari, Joshua Tenenbaum, and Shimon Ullman. When Computer Vision Gazes at Cognition. dec 2014. URL http://arxiv.org/abs/1412.2672.
  135. ↵
    Ian Gemp and Sridhar Mahadevan. Modeling context in cognition using variational inequalities. 2015.
  136. ↵
    Dileep George and Jeff Hawkins. Towards a mathematical theory of cortical micro-circuits. PLoS computational biology, 5(10):e1000532, oct 2009. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1000532.
    OpenUrlCrossRefPubMed
  137. ↵
    Samuel J Gershman and Jeffrey M Beck. Complex probabilistic inference: From cognition to neural computation. 2016.
  138. ↵
    Samuel J Gershman, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg. The successor representation and temporal context. Neural Computation, 24(6): 1553–1568, 2012.
    OpenUrlCrossRefPubMed
  139. ↵
    Samuel J Gershman, Ahmed A Moustafa, and Elliot A Ludvig. Time representation in reinforcement learning models of the basal ganglia. 2014.
  140. ↵
    Iain Murray Zoubin Ghahramani. A note on the evidence and bayesian occams razor. 2005.
  141. ↵
    Lisa M Giocomo, Eric A Zilli, Erik Fransén, and Michael E Hasselmo. Temporal frequency of sub-threshold oscillations scales with entorhinal grid cell field spacing. Science, 315(5819): 1719–1722, 2007.
    OpenUrlAbstract/FREE Full Text
  142. ↵
    Nicolas Giret, Joergen Kornfeld, Surya Ganguli, and Richard H R Hahnloser. Evidence for a causal inverse model in an avian cortico-basal ganglia circuit. Proceedings of the National Academy of Sciences of the United States of America, 111(16): 6063–6068, apr 2014. ISSN 1091–6490. doi: 10.1073/pnas.1317087111. URL http://www.pnas.org/content/111/16/6063.full.
    OpenUrlAbstract/FREE Full Text
  143. ↵
    B Goertzel. How might the brain represent complex symbolic knowledge? Neural Networks (IJCNN), 2014 International Joint …, 2014. URL http://ieeexplore.ieee.org/xpls/abs{_}all.jsp?arnumber=6889662.
  144. ↵
    Mark S Goldman, Joseph H Levine, Guy Major, David W Tank, and HS Seung. Robust persistent neural activity in a model integrator with multiple hysteretic dendrites per neuron. Cerebral cortex, 13(11): 1185–1195, 2003.
    OpenUrlCrossRefPubMedWeb of Science
  145. ↵
    S L Gonzalez Andino and R Grave de Peralta Menendez. Coding of saliency by ensemble bursting in the amygdala of primates. Frontiers in behavioral neuroscience, 6:38, jan 2012. ISSN 1662–5153. doi: 10.3389/fnbeh.2012.00038. URL http://journal.frontiersin.org/article/10.3389/fnbeh.2012.00038/abstract.
    OpenUrlCrossRef
  146. ↵
    Cynthia M Gooch, Martin Wiener, Elaine B Wencil, and H Branch Coslett. Interval timing disruptions in subjects with cerebellar lesions. Neuropsychologia, 48(4): 1022–31, mar 2010. ISSN 1873–3514. doi: 10.1016/j.neuropsychologia.2009.11.028.
    OpenUrlCrossRefPubMedWeb of Science
  147. ↵
    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. jun 2014a. URL http://arxiv.org/abs/1406.2661.
  148. ↵
    Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems. dec 2014b. URL http://arxiv.org/abs/1412.6544.
  149. ↵
    Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl. The scientist in the crib: What early learning tells us about the mind. Harper Paperbacks, 2000.
  150. ↵
    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines. ArXiv, oct 2014. URL http://arxiv.org/abs/1410.5401.
  151. ↵
    Ann M Graybiel. The basal ganglia and chunking of action repertoires. Neurobiology of learning and memory, 70(1): 119–136, 1998.
    OpenUrlCrossRefPubMedWeb of Science
  152. ↵
    Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wier-stra. DRAW: A Recurrent Neural Network For Image Generation. feb 2015. URL http://arxiv.org/abs/1502.04623.
  153. ↵
    Sten Grillner, Jeanette Hellgren, Ariane Ménard, Kazuya Saitoh, and Martin A Wikström. Mechanisms for selection of basic motor programs–roles for the striatum and pallidum. Trends in neurosciences, 28(7): 364–70, jul 2005. ISSN 0166–2236. doi: 10.1016/j.tins.2005.05.004. URL http://www.ncbi.nlm.nih.gov/pubmed/15935487.
    OpenUrlCrossRefPubMedWeb of Science
  154. ↵
    Stephen Grossberg. Adaptive Resonance Theory: how a brain learns to consciously attend, learn, and recognize a changing world. Neural networks : the official journal of the International Neural Network Society, 37:1–47, jan 2013. ISSN 1879–2782. doi: 10.1016/j.neunet.2012.09.017. URL http://www.sciencedirect.com/science/article/pii/S0893608012002584.
    OpenUrlCrossRefPubMedWeb of Science
  155. ↵
    Umut Güçlü and Marcel AJ van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. The Journal of Neuroscience, 35(27): 10005–10014, 2015.
    OpenUrlAbstract/FREE Full Text
  156. ↵
    Arthur Guez, David Silver, and Peter Dayan. Effi-cient bayes-adaptive reinforcement learning using sample-based search. In Advances in Neural Information Processing Systems, pages 1025–1033, 2012.
  157. ↵
    Çalar Gülçehre and Yoshua Bengio. Knowledge Matters: Importance of Prior Information for Optimization. Journal of Machine Learning Research, 17(8): 1–32, 2016. URL http://jmlr.org/papers/v17/gulchere16a.html.
    OpenUrl
  158. ↵
    Onur Güntürkün and Thomas Bugnyar. Cognition without cortex. Trends in cognitive sciences, 20 (4):291–303, 2016.
    OpenUrlCrossRefPubMed
  159. ↵
    K Gurney, T J Prescott, and P Redgrave. A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biological cybernetics, 84(6): 401–10, jun 2001. ISSN 0340–1200. URL http://www.ncbi.nlm.nih.gov/pubmed/11417052.
    OpenUrlCrossRefPubMedWeb of Science
  160. ↵
    Robert F Hadley. The problem of rapid variable creation. Neural computation, 21(2): 510–32, mar 2009. ISSN 0899–7667. doi: 10.1162/neco.2008.07-07-572. URL http://www.ncbi.nlm.nih.gov/pubmed/19431268.
    OpenUrlCrossRefPubMedWeb of Science
  161. ↵
    J Kiley Hamlin, Karen Wynn, and Paul Bloom. Social evaluation by preverbal infants. Nature, 450 (7169):557–9, nov 2007. ISSN 1476–4687. doi: 10.1038/nature06288. URL http://dx.doi.org/10.1038/nature06288.
    OpenUrlCrossRefPubMedWeb of Science
  162. ↵
    Balázs Hangya, SachinP. Ranade, Maja Lorenc, and Adam Kepecs. Central Cholinergic Neurons Are Rapidly Recruited by Reinforcement Feedback. Cell, 162(5): 1155–1168, aug 2015. ISSN 00928674. doi: 10.1016/j.cell.2015.07.057. URL http://www.sciencedirect.com/science/article/pii/S0092867415009733.
    OpenUrlCrossRefPubMed
  163. ↵
    A Hanuschkin, S Ganguli, and R H R Hahnloser. A Hebbian learning rule gives rise to mirror neurons and links them to control theoretic inverse models. Frontiers in neural circuits, 7:106, jan 2013. ISSN 1662–5110. doi: 10.3389/fncir.2013.00106.
    OpenUrlCrossRefPubMed
  164. ↵
    Christopher M Harris and Daniel M Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6695): 780–784, 1998.
    OpenUrlCrossRefPubMedWeb of Science
  165. ↵
    KD Harris. Stability of the fittest: organizing learning through retroaxonal signals. Trends in neuro-sciences, 2008. URL http://www.sciencedirect.com/science/article/pii/S0166223608000180.
  166. ↵
    D Hassabis and EA Maguire. The construction system of the brain. … of the Royal …, 2009. URL http://rstb.royalsocietypublishing.org/content/364/1521/1263.short.
  167. ↵
    Demis Hassabis and Eleanor A Maguire. De-constructing episodic memory with construction. Trends in cognitive sciences, 11(7): 299–306, jul 2007. ISSN 1364–6613. doi: 10.1016/j.tics.2007.05.001. URL http://www.sciencedirect.com/science/article/pii/S1364661307001258.
    OpenUrlCrossRefPubMedWeb of Science
  168. ↵
    Michael E Hasselmo. The role of acetylcholine in learning and memory. Current opinion in neurobiology, 16(6): 710–5, dec 2006. ISSN 0959–4388. doi: 10.1016/j.conb.2006.09.002.
    OpenUrlCrossRefPubMedWeb of Science
  169. ↵
    Michael E Hasselmo. If i had a million neurons: Potential tests of cortico-hippocampal theories. Progress in brain research, 219:1–19, 2015.
    OpenUrlCrossRefPubMed
  170. ↵
    Michael E Hasselmo and Chantal E Stern. Current questions on space and time encoding. Hippocampus, 25(6): 744–752, 2015.
    OpenUrlCrossRefPubMed
  171. ↵
    Michael E Hasselmo and Bradley P Wyble. Free recall and recognition in a network model of the hippocampus: simulating effects of scopolamine on human memory function. Behavioural Brain Research, 89(1–2):1–34, dec 1997. ISSN 01664328. doi: 10.1016/S0166-4328(97)00048-X. URL http://www.sciencedirect.com/science/article/pii/S016643289700048X.
    OpenUrlCrossRefPubMedWeb of Science
  172. ↵
    Daisuke Hattori, Ebru Demir, Ho Won Kim, Erika Viragh, S Lawrence Zipursky, and Barry J Dickson. Dscam diversity is essential for neuronal wiring and self-recognition. Nature, 449(7159): 223–227, sep 2007. ISSN 1476–4687. doi: 10.1038/nature06099.
    OpenUrlCrossRefPubMedWeb of Science
  173. ↵
    Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Frontiers in neural circuits, 10, 2016.
  174. ↵
    Jeff Hawkins and Sandra Blakeslee. On Intelligence. Henry Holt and Company, 2007. ISBN 1429900458. URL http://books.google.com/books?id=Qg2dmntfxmQC{&}pgis=1.
  175. ↵
    Akiko Hayashi-Takagi, Sho Yagishita, Mayumi Nakamura, Fukutoshi Shirai, Yi I Wu, Amanda L Loshbaugh, Brian Kuhlman, Klaus M Hahn, and Haruo Kasai. Labelling and optical erasure of synaptic memory traces in the motor cortex. Nature, 2015.
  176. ↵
    Simon S. Haykin. Neural networks: a comprehensive foundation. Macmillan, 1994. ISBN 0023527617. URL https://books.google.com/books/about/Neural{_}networks.html?id=OZJKAQAAIAAJ{&}pgis=1.
  177. ↵
    Kenneth J Hayworth. Dynamically partitionable autoassociative networks as a solution to the neural binding problem. Frontiers in computational neuroscience, 6:73, jan 2012. ISSN 1662–5188. doi: 10.3389/fncom.2012.00073.
    OpenUrlCrossRef
  178. ↵
    Kenneth J Hayworth, Mark D Lescroart, and Irving Biederman. Neural encoding of relative position. Journal of experimental psychology. Human perception and performance, 37(4): 1032–50, aug 2011. ISSN 1939–1277. doi: 10.1037/a0022338. URL http://www.ncbi.nlm.nih.gov/pubmed/21517211.
    OpenUrlCrossRefPubMed
  179. ↵
    Guillaume Hennequin, Tim P Vogels, and Wulfram Gerstner. Optimal control of transient dynamics in balanced networks supports generation of complex movements. Neuron, 82(6): 1394–1406, 2014.
    OpenUrl
  180. ↵
    SA Herd, KA Krueger, and TE Kriete. Strategic cognitive sequencing: a computational cognitive neuroscience approach. Computational …, 2013. URL http://dl.acm.org/citation.cfm?id=2537972.
  181. ↵
    I. Higgins, L. Matthey, X. Glorot, A. Pal, B. Uria, C. Blundell, S. Mohamed, and A. Lerchner. Early Visual Concept Learning with Unsupervised Deep Learning. ArXiv e-prints, June 2016.
  182. ↵
    G Hinton. How to do backpropagation in a brain. Invited talk at the NIPS’2007 Deep Learning Workshop, 2007. URL http://www.cs.utoronto.ca/{˜}hinton/backpropincortex2014.pdf.
  183. ↵
    G Hinton. Can the brain do back-propagation? Invited talk at Stanford University Colloquium on Computer Systems, 2016. URL https://www.youtube.com/watch?v=VIRCybGgHts.
  184. ↵
    GE Hinton. Connectionist learning procedures. Artificial intelligence, 1989. URL http://www.sciencedirect.com/science/article/pii/0004370289900490.
  185. ↵
    GE Hinton and JL McClelland. Learning representations by recirculation. Neural information processing …, 1988.
  186. ↵
    GE Hinton, P Dayan, BJ Frey, and RM Neal. The” wake-sleep” algorithm for unsupervised neural networks. Science, 1995. URL http://www.sciencemag.org/content/268/5214/1158.short.
  187. ↵
    GE Hinton, A Krizhevsky, and SD Wang. Transforming auto-encoders. Artificial Neural Networks and …, 2011. URL http://link.springer.com/chapter/10.1007/978–3–642–21735–7{_}6.
  188. ↵
    Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7): 1527–54, jul 2006. ISSN 0899–7667. doi: 10.1162/neco.2006.18.7.1527. URL http://www.ncbi.nlm.nih.gov/pubmed/16764513.
    OpenUrlCrossRefPubMedWeb of Science
  189. ↵
    Samuel Andrew Hires, Diego A Gutnisky, Jianing Yu, Daniel H O’Connor, and Karel Svoboda. Low-noise encoding of active touch by layer 4 in the somatosensory cortex. eLife, 4, jan 2015. ISSN 2050–084X. doi: 10.7554/eLife.06619.
    OpenUrlCrossRefPubMed
  190. ↵
    Mark H Histed, Amy M Ni, and John HR Maunsell. Insights into cortical mechanisms of behavior from microstimulation experiments. Progress in neuro-biology, 103:115–130, 2013.
    OpenUrlCrossRefPubMed
  191. ↵
    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint arXiv:1606.03476, 2016.
  192. ↵
    S Hochreiter and J Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–80, nov 1997. ISSN 0899–7667. URL http://www.ncbi.nlm.nih.gov/pubmed/9377276.
    OpenUrlCrossRefPubMedWeb of Science
  193. ↵
    Gregor M Hoerzer, Robert Legenstein, and Wolfgang Maass. Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cerebral cortex (New York, N.Y. : 1991), 24(3): 677–90, mar 2014. ISSN 1460–2199. doi: 10.1093/cercor/bhs348. URL http://www.ncbi.nlm.nih.gov/pubmed/23146969.
    OpenUrlCrossRefPubMedWeb of Science
  194. ↵
    Weizhe Hong and Liqun Luo. Genetic control of wiring specificity in the fly olfactory system. Genetics, 196(1): 17–29, jan 2014. ISSN 1943–2631. doi: 10.1534/genetics.113.154336.
    OpenUrlAbstract/FREE Full Text
  195. ↵
    J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(8): 2554–8, apr 1982. ISSN 0027–8424.
    OpenUrlAbstract/FREE Full Text
  196. ↵
    J J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences of the United States of America, 81(10): 3088–92, may 1984. ISSN 0027–8424. URL http://www.pnas.org/content/81/10/3088.abstract.
    OpenUrlAbstract/FREE Full Text
  197. ↵
    J. J. Hopfield. Neurodynamics of mental exploration. Proceedings of the National Academy of Sciences, 107(4): 1648–1653, dec 2009. ISSN 0027–8424. doi: 10.1073/pnas.0913991107. URL http://www.pnas.org/content/107/4/1648.abstract.
    OpenUrlAbstract/FREE Full Text
  198. ↵
    Toshihiko Hosoya, Stephen A Baccus, and Markus Meister. Dynamic predictive coding by the retina. Nature, 436(7047): 71–77, 2005.
    OpenUrlCrossRefPubMedWeb of Science
  199. ↵
    Y Huang and RPN Rao. Predictive coding. Wiley Interdisciplinary Reviews: Cognitive …, 2011. URL http://onlinelibrary.wiley.com/doi/10.1002/wcs.142/pdf.
  200. ↵
    Quentin JM Huys, Níall Lally, Paul Faulkner, Neir Eshel, Erich Seifritz, Samuel J Gershman, Peter Dayan, and Jonathan P Roiser. Interplay of approximate planning strategies. Proceedings of the National Academy of Sciences, 112(10): 3098–3103, 2015.
    OpenUrlAbstract/FREE Full Text
  201. ↵
    Leyla Isik, Joel Z Leibo, and Tomaso Poggio. Learning and disrupting invariance in visual recognition with a temporal association rule. Frontiers in computational neuroscience, 6:37, jan 2012. ISSN 1662–5188. doi: 10.3389/fncom.2012.00037. URL http://journal.frontiersin.org/article/10.3389/fncom.2012.00037/abstract.
    OpenUrlCrossRef
  202. ↵
    Eugene M Izhikevich. Polychronization: computation with spikes. Neural computation, 18(2): 245–282, 2006.
    OpenUrlCrossRefPubMedWeb of Science
  203. ↵
    Eugene M Izhikevich. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral cortex (New York, N.Y. : 1991), 17(10): 2443–52, oct 2007. ISSN 1047–3211. doi: 10.1093/cercor/bhl152. URL http://www.ncbi.nlm.nih.gov/pubmed/17220510.
    OpenUrlCrossRefPubMedWeb of Science
  204. ↵
    Gilad A Jacobson and Rainer W Friedrich. Neural circuits: random design of a higher-order olfactory projection. Current biology : CB, 23(10):R448–51, may 2013. ISSN 1879–0445. doi: 10.1016/j.cub.2013.04.016. URL http://www.sciencedirect.com/science/article/pii/S0960982213004247.
    OpenUrlCrossRef
  205. ↵
    M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. Advances in Neural…, 2015a. URL http://papers.nips.cc/paper/5854-spatial-transformer-networks.
  206. ↵
    M. Jaderberg, W. M. Czarnecki, S. Osindero, O. Vinyals, A. Graves, and K. Kavukcuoglu. Decoupled Neural Interfaces using Synthetic Gradients. ArXiv e-prints, August 2016.
  207. ↵
    Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial Transformer Networks. jun 2015b. URL http://arxiv.org/abs/1506.02025.
  208. ↵
    Herbert Jaeger and Harald Haas. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science (New York, N.Y.), 304(5667): 78–80, apr 2004. ISSN 1095–9203. doi: 10.1126/science.1091277. URL http://www.sciencemag.org/content/304/5667/78.abstract.
    OpenUrlAbstract/FREE Full Text
  209. ↵
    Julian Jara-Ettinger, Hyowon Gweon, Laura E Schulz, and Joshua B Tenenbaum. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in Cognitive Sciences, 20(8): 589–604, 2016.
    OpenUrlCrossRefPubMed
  210. ↵
    Santiago Jaramillo and Barak A Pearlmutter. A normative model of attention: Receptive field modulation. Neurocomputing, 58:613–618, 2004.
    OpenUrl
  211. ↵
    Hueihan Jhuang, Thomas Serre, Lior Wolf, and Tomaso Poggio. A biologically inspired system for action recognition. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pages 1–8. Ieee, 2007.
  212. ↵
    Daoyun Ji and Matthew A Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10(1): 100–7, jan 2007. ISSN 1097–6256. doi: 10.1038/nn1825. URL http://dx.doi.org/10.1038/nn1825.
    OpenUrlCrossRefPubMedWeb of Science
  213. ↵
    X. Jiang, S. Shen, C. R. Cadwell, P. Berens, F. Sinz, A. S. Ecker, S. Patel, and A. S. Tolias. Principles of connectivity among morphologically defined cell types in adult neocortex. Science, 350(6264):aac9462–aac9462, nov 2015. ISSN 0036–8075. doi: 10.1126/science.aac9462. URL http://science.sciencemag.org/content/350/6264/aac9462.abstract.
    OpenUrlAbstract/FREE Full Text
  214. ↵
    Fredrik Johansson, Dan-Anders Jirenhed, Anders Rasmussen, Riccardo Zucca, and Germund Hesslow. Memory trace and timing mechanism localized to cerebellar Purkinje cells. Proceedings of the National Academy of Sciences of the United States of America, 111(41): 14930–4, oct 2014. ISSN 1091–6490. doi: 10.1073/pnas.1415371111. URL http://www.pnas.org/content/111/41/14930.abstract.
    OpenUrlAbstract/FREE Full Text
  215. ↵
    Eric Jonas and Konrad Kording. Could a neuroscientist understand a microprocessor? bioRxiv, 2016. doi: 10.1101/055624. URL http://biorxiv.org/content/early/2016/05/26/055624.
    OpenUrlAbstract/FREE Full Text
  216. ↵
    Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, pages 190–198, 2015.
  217. ↵
    Nir Kalisman, Gilad Silberberg, and Henry Markram. The neocortical microcircuit as a tabula rasa. Proceedings of the National Academy of Sciences of the United States of America, 102(3): 880–5, jan 2005. ISSN 0027–8424. doi: 10.1073/pnas.0407088102.
    OpenUrlAbstract/FREE Full Text
  218. ↵
    Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 17(11): 4302–4311, 1997.
    OpenUrlAbstract/FREE Full Text
  219. ↵
    David Kappel, Bernhard Nessler, and Wolfgang Maass. STDP installs in Winner-Take-All circuits an online approximation to hidden Markov model learning. PLoS computational biology, 10 (3):e1003511, mar 2014. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1003511.
    OpenUrlCrossRef
  220. ↵
    David Kappel, Stefan Habenschuss, Robert Legenstein, and Wolfgang Maass. Network plasticity as bayesian inference. PLoS Comput Biol, 11(11): e1004485, 2015.
    OpenUrlCrossRefPubMed
  221. ↵
    R Kempter, W Gerstner, and J L van Hemmen. Intrinsic stabilization of output rates by spike-based Hebbian learning. Neural computation, 13 (12):2709–41, dec 2001. ISSN 0899–7667. doi: 10.1162/089976601317098501. URL http://www.ncbi.nlm.nih.gov/pubmed/11705408.
    OpenUrlCrossRefPubMedWeb of Science
  222. ↵
    Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not un-supervised, models may explain it cortical representation. PLoS Comput Biol, 10(11): e1003915, 2014.
    OpenUrlCrossRefPubMed
  223. ↵
    Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. dec 2013. URL http://arxiv.org/abs/1312.6114.
  224. ↵
    DC Knill and A Pouget. The Bayesian brain: the role of uncertainty in neural coding and computation. TRENDS in Neurosciences, 2004. URL http://www.sciencedirect.com/science/article/pii/S0166223604003352.
  225. ↵
    Etienne Koechlin and Thomas Jubault. Broca’s area and the hierarchical organization of human behavior. Neuron, 50(6): 963–974, 2006.
    OpenUrlCrossRefPubMedWeb of Science
  226. ↵
    Brent Komer and Chris Eliasmith. A unified theoretical approach for biological cognition and learning. Current Opinion in Behavioral Sciences, 11:14–20, mar 2016. ISSN 23521546. doi: 10.1016/j.cobeha.2016.03.006. URL http://www.sciencedirect.com/science/article/pii/S2352154616300651.
    OpenUrlCrossRef
  227. ↵
    K Körding. Decision theory: what” should” the nervous system do? Science, 2007. URL http://science.sciencemag.org/content/318/5850/606.short.
  228. ↵
    Konrad P Körding, Christoph Kayser, Wolfgang Einhäuser, and Peter König. How are complex cell properties adapted to the statistics of natural stimuli? Journal of neurophysiology, 91(1): 206–12, jan 2004. ISSN 0022–3077. doi: 10.1152/jn.00149. 2003. URL http://jn.physiology.org/content/91/1/206.short.
    OpenUrlCrossRefPubMedWeb of Science
  229. ↵
    K.P Körding and P König. A learning rule for dynamic recruitment and decorrelation. Neural Networks, 13(1): 1–9, jan 2000. ISSN 08936080. doi: 10.1016/S0893-6080(99)00088-X. URL http://www.sciencedirect.com/science/article/pii/S089360809900088X.
    OpenUrlCrossRefPubMed
  230. ↵
    KP Körding and P König. Supervised and unsupervised learning with two sites of synaptic integration. Journal of Computational Neuroscience, 2001. URL http://link.springer.com/article/10.1023/A:1013776130161.
  231. ↵
    Minjoon Kouh and Tomaso Poggio. A canonical neural circuit for cortical nonlinear operations. Neural computation, 20(6): 1427–51, jun 2008. ISSN 0899–7667. doi: 10.1162/neco.2008.02-07-466. URL http://www.ncbi.nlm.nih.gov/pubmed/18254695.
    OpenUrlCrossRefPubMedWeb of Science
  232. ↵
    Benjamin J Kraus, Robert J Robinson, John A White, Howard Eichenbaum, and Michael E Hasselmo. Hippocampal time cells: time versus path integration. Neuron, 78(6): 1090–1101, 2013.
    OpenUrlCrossRefPubMed
  233. ↵
    Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008.
    OpenUrl
  234. ↵
    Trenton Kriete, David C Noelle, Jonathan D Cohen, and Randall C O’Reilly. Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences of the United States of America, 110(41): 16390–16395, oct 2013. ISSN 1091–6490. doi: 10.1073/pnas.1303547110. URL http://www.pnas.org/content/110/41/16390.short.
    OpenUrlAbstract/FREE Full Text
  235. ↵
    Ramnandan Krishnamurthy, Aravind S. Lakshminarayanan, Peeyush Kumar, and Balaraman Ravindran. Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks. page 13, may 2016. URL http://arxiv.org/abs/1605.05359.
  236. ↵
    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
  237. ↵
    Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B. Tenenbaum. Deep Convolutional Inverse Graphics Network. mar 2015. URL http://arxiv.org/abs/1503.03167.
  238. ↵
    Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi, and Joshua B. Tenenbaum. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. page 13, apr 2016. URL http://arxiv.org/abs/1604.06057.
  239. ↵
    Dharshan Kumaran, Jennifer J Summerfield, Demis Hassabis, and Eleanor A Maguire. Tracking the emergence of conceptual knowledge during human decision making. Neuron, 63(6): 889–901, sep 2009. ISSN 1097–4199. doi: 10.1016/j.neuron.2009.07.030. URL http://www.cell.com/article/S0896627309006187/fulltext.
    OpenUrlCrossRefPubMedWeb of Science
  240. ↵
    Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7): 512–534, 2016.
    OpenUrlCrossRefPubMed
  241. ↵
    Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural Random-Access Machines. page 13, nov 2015. URL http://arxiv.org/abs/1511.06392.
  242. ↵
    Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332–1338, dec 2015. doi: 10.1126/science.aab3050. URL http://www.sciencemag.org/content/350/6266/1332.full.
    OpenUrlAbstract/FREE Full Text
  243. ↵
    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. arXiv preprint arXiv:1604.00289, 2016.
  244. ↵
    Matthew Larkum. A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends in neurosciences, 36(3): 141–151, mar 2013. ISSN 1878–108X. doi: 10.1016/j.tins.2012.11.006. URL http://www.ncbi.nlm.nih.gov/pubmed/23273272.
    OpenUrlCrossRefPubMedWeb of Science
  245. ↵
    Quoc V. Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. dec 2011. URL http://arxiv.org/abs/1112.6209.
  246. ↵
    Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 (10):1995, 1995.
    OpenUrl
  247. ↵
    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): 436–444, may 2015. ISSN 0028–0836. doi: 10.1038/nature14539. URL http://dx.doi.org/10.1038/nature14539.
    OpenUrlCrossRefPubMed
  248. ↵
    A M Lee, L-H Tai, A Zador, and L Wilbrecht. Between the primate and ‘reptilian’ brain: Rodent models demonstrate the role of corticostriatal circuits in decision making. Neuroscience, 296:66–74, jun 2015. ISSN 1873–7544. doi: 10.1016/j.neuroscience.2014.12.042. URL http://www.sciencedirect.com/science/article/pii/S0306452214010914.
    OpenUrlCrossRefPubMed
  249. ↵
    Tai Sing Lee and David Mumford. Hierarchical Bayesian inference in the visual cortex. Journal of the Optical Society of America. A, Optics, image science, and vision, 20(7): 1434–48, jul 2003. ISSN 1084–7529. URL http://www.ncbi.nlm.nih.gov/pubmed/12868647.
    OpenUrlCrossRefPubMedWeb of Science
  250. ↵
    TS Lee and AL Yuille. Efficient coding of visual scenes by grouping and segmentation: theoretical predictions and biological evidence. Department of Statistics, UCLA, 2011. URL http://escholarship.org/uc/item/1mc5v1b6.pdf.
  251. ↵
    Robert Legenstein and Wolfgang Maass. Branch-specific plasticity enables self-organization of nonlinear computation in single neurons. The Journal of neuroscience : the official journal of the Society for Neuroscience, 31(30): 10787–802, jul 2011. ISSN 1529–2401. doi: 10.1523/JNEUROSCI.5684-10.2011. URL http://www.jneurosci.org.libproxy.mit.edu/content/31/30/10787.short.
    OpenUrlAbstract/FREE Full Text
  252. ↵
    Joel Z. Leibo, Julien Cornebise, Sergio Gómez, and Demis Hassabis. Approximate Hubel-Wiesel Modules and the Data Structures of Neural Computation. page 13, dec 2015a. URL http://arxiv.org/abs/1512.08457.
  253. ↵
    Joel Z Leibo, Qianli Liao, Fabio Anselmi, and Tomaso Poggio. The invariance hypothesis implies domain-specific regions in visual cortex. PLoS Comput Biol, 11(10):e1004390, 2015b.
    OpenUrlCrossRefPubMed
  254. ↵
    J. Lettvin, H. Maturana, W. McCulloch, and W. Pitts. What the Frog’s Eye Tells the Frog’s Brain. Proceedings of the IRE, 47 (11):1940–1951, nov 1959. ISSN 0096–8390. doi: 10.1109/JRPROC.1959.287207. URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=4065609.
    OpenUrlCrossRef
  255. ↵
    Johannes J Letzkus, Björn M Kampa, and Greg J Stuart. Learning rules for spike timing-dependent plasticity depend on dendritic synapse location. The Journal of neuroscience, 26(41): 10420–10429, 2006.
    OpenUrlAbstract/FREE Full Text
  256. ↵
    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo-motor policies. arXiv preprint arXiv:1504.00702, 2015.
  257. ↵
    Michael S. Lewicki and Terrence J. Sejnowski. Learning Overcomplete Representations. Neural Computation, 12(2): 337–365, feb 2000. ISSN 0899–7667. doi: 10.1162/089976600300015826.
    OpenUrlCrossRefPubMedWeb of Science
  258. ↵
    S. N. Lewis and K. D. Harris. The Neural Marketplace: I. General Formalismand Linear Theory. Technical report, dec 2014. URL http://biorxiv.org/content/early/2014/12/23/013185.abstract.
  259. ↵
    Nuo Li and James J Dicarlo. Neuronal learning of invariant object representation in the ventral visual stream is not dependent on reward. The Journal of neuroscience : the official journal of the Society for Neuroscience, 32(19): 6611–20, may 2012. ISSN 1529–2401. doi: 10.1523/JNEUROSCI.3786-11.2012. URL http://www.jneurosci.org/content/32/19/6611.full.
    OpenUrlAbstract/FREE Full Text
  260. ↵
    Qianli Liao and Tomaso Poggio. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex. apr 2016. URL http://arxiv.org/abs/1604.03640.
  261. ↵
    Qianli Liao, Joel Z. Leibo, and Tomaso Poggio. How Important is Weight Symmetry in Backpropagation? oct 2015. URL http://arxiv.org/abs/1510.05067.
  262. ↵
    Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random feedback weights support learning in deep neural networks. page 14, nov 2014. URL http://arxiv.org/abs/1411.0247.
  263. ↵
    Jian K Liu and Dean V Buonomano. Embedding multiple trajectories in simulated recurrent neural networks in a self-organizing manner. The Journal of neuroscience : the official journal of the Society for Neuroscience, 29(42): 13172–81, oct 2009. ISSN 1529–2401. doi: 10.1523/JNEUROSCI.2358-09.2009. URL http://www.jneurosci.org/content/29/42/13172.short.
    OpenUrlAbstract/FREE Full Text
  264. ↵
    Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. An Algorithm for Training Polynomial Networks. apr 2013. URL http://arxiv.org/abs/1304.7045.
  265. ↵
    William Lotter, Gabriel Kreiman, and David Cox. Unsupervised Learning of Visual Structure using Predictive Generative Networks. nov 2015. URL http://arxiv.org/abs/1511.06380.
  266. ↵
    William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104, 2016.
  267. ↵
    Steven J Luck and Edward K Vogel. The capacity of visual working memory for features and conjunctions. Nature, 390(6657): 279–281, 1997.
    OpenUrlCrossRefPubMedWeb of Science
  268. ↵
    Yan Luo, Xavier Boix, Gemma Roig, Tomaso Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292, 2015.
  269. ↵
    Ashley B Lyons and Erik W Cheries. Inferring social disposition by sound and surface appearance in infancy. Journal of Cognition and Development, (just-accepted), 2016.
  270. ↵
    Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature neuroscience, 9(11): 1432–8, nov 2006. ISSN 1097–6256. doi: 10.1038/nn1790. URL http://dx.doi.org/10.1038/nn1790.
    OpenUrlCrossRefPubMedWeb of Science
  271. Wolfgang Maass. Searching for principles of brain computation.
  272. ↵
    Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural computation, 14 (11):2531–60, nov 2002. ISSN 0899–7667. doi: 10.1162/089976602760407955.
    OpenUrlCrossRefPubMedWeb of Science
  273. ↵
    Wolfgang Maass, Prashant Joshi, and Eduardo D Sontag. Computational aspects of feedback in neural circuits. PLoS computational biology, 3(1):e165, jan 2007. ISSN 1553–7358. doi: 10.1371/journal.pcbi.0020165. URL http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.0020165.
    OpenUrlCrossRefPubMed
  274. ↵
    Christopher J MacDonald, Kyle Q Lepage, Uri T Eden, and Howard Eichenbaum. Hippocampal time cells bridge the gap in memory for discontiguous events. Neuron, 71(4): 737–749, 2011.
    OpenUrlCrossRefPubMedWeb of Science
  275. ↵
    D Maclaurin, D Duvenaud, and RP Adams. Gradient-based hyperparameter optimization through reversible learning. arXiv preprint arXiv:1502.03492, 2015. URL http://arxiv.org/abs/1502.03492.
  276. ↵
    Joseph G Makin, Matthew R Fellows, and Philip N Sabes. Learning multisensory integration and coordinate transformation via density estimation. PLoS Comput Biol, 9(4):e1003035, 2013.
    OpenUrlCrossRefPubMed
  277. ↵
    Joseph G Makin, Benjamin K Dichter, and Philip N Sabes. Recurrent exponential-family harmoniums without backprop-through-time. arXiv preprint arXiv:1605.05799, 2016.
  278. ↵
    Yael Mandelblat-Cerf, Liora Las, Natalia Denisenko, and Michale S Fee. A role for descending auditory cortical projections in songbird vocal learning. eLife, 3:e02152, jan 2014. ISSN 2050–084X. doi: 10.7554/eLife.02152. URL http://elifesciences.org/content/3/e02152.abstract.
    OpenUrlCrossRefPubMed
  279. ↵
    Vikash Mansinghka and Eric Jonas. Building fast bayesian computing machines out of intentionally stochastic, digital parts. arXiv preprint arXiv:1402.4914, 2014.
  280. ↵
    Adam H Marblestone and Edward S Boyden. Designing tools for assumption-proof brain mapping. Neuron, 83(6): 1239–41, sep 2014. ISSN 1097–4199. doi: 10.1016/j.neuron.2014.09.004. URL http://www.cell.com/article/S0896627314007922/fulltext.
    OpenUrlCrossRef
  281. ↵
    Gary Marcus. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press, 2001. ISBN 0262632683. URL http://books.google.com/books?id=7YpuRUlFLm8C{&}pgis=1.
  282. ↵
    Gary Marcus. The birth of the mind: How a tiny number of genes creates the complexities of human thought. Basic Books, 2004.
  283. ↵
    Gary Marcus, Adam Marblestone, and Thomas Dean. Frequently asked question for: the atoms of neural computation. oct 2014a. URL http://arxiv.org/abs/1410.8826.
  284. ↵
    Gary Marcus, Adam Marblestone, and Thomas Dean. The atoms of neural computation. Science, 346(6209): 551–552, 2014b. doi: 10.1126/science.1261661.
    OpenUrlAbstract/FREE Full Text
  285. ↵
    Eve Marder and Jean-Marc Goaillard. Variability, compensation and homeostasis in neuron and network function. Nature reviews. Neuroscience, 7(7): 563–74, jul 2006. ISSN 1471–003X. doi: 10.1038/nrn1949. URL http://dx.doi.org/10.1038/nrn1949.
    OpenUrlCrossRefPubMedWeb of Science
  286. ↵
    David A Markowitz, Clayton E Curtis, and Bijan Pesaran. Multiple component networks support working memory in prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America, 112(35): 11084–11089, aug 2015. ISSN 1091–6490. doi: 10.1073/pnas.1504172112. URL http://www.pnas.org/content/112/35/11084.abstract.
    OpenUrlAbstract/FREE Full Text
  287. ↵
    H Markram, J Lübke, M Frotscher, and B Sakmann. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science (New York, N.Y.), 275(5297): 213–5, jan 1997. ISSN 0036–8075. URL http://www.ncbi.nlm.nih.gov/pubmed/8985014.
    OpenUrlCrossRef
  288. ↵
    Henry Markram, Eilif Muller, Srikanth Ramaswamy, MichaelW. Reimann, Marwan Abdellah, CarlosAguado Sanchez, Anastasia Ailamaki, Lidia Alonso-Nanclares, Nicolas Antille, Selim Arsever, GuyAntoineAtenekeng Kahou, ThomasK. Berger, Ahmet Bilgili, Nenad Buncic, Athanassia Chalimourda, Giuseppe Chindemi, Jean-Denis Courcol, Fabien Delalondre, Vincent Delattre, Shaul Druckmann, Raphael Dumusc, James Dynes, Stefan Eilemann, Eyal Gal, MichaelEmiel Gevaert, Jean-Pierre Ghobril, Albert Gidon, JoeW. Graham, Anirudh Gupta, Valentin Haenel, Etay Hay, Thomas Heinis, JuanB. Hernando, Michael Hines, Lida Kanari, Daniel Keller, John Kenyon, Georges Khazen, Yihwa Kim, JamesG. King, Zoltan Kisvarday, Pramod Kumbhar, Sébastien Lasserre, Jean-Vincent LeBé, BrunoR.C. Magalhães, Angel Merchán-Pérez, Julie Meystre, BenjaminRoy Morrice, Jeffrey Muller, Alberto Muñoz-Céspedes, Shruti Muralidhar, Keerthan Muthurasa, Daniel Nachbaur, TaylorH. Newton, Max Nolte, Aleksandr Ovcharenko, Juan Palacios, Luis Pastor, Rodrigo Perin, Rajnish Ranjan, Imad Riachi, JoséRodrigo Rodríguez, JuanLuis Riquelme, Christian Rössert, Konstantinos Sfyrakis, Ying Shi, JulianC. Shillcock, Gilad Silberberg, Ricardo Silva, Farhan Tauheed, Martin Telefont, Maria Toledo-Rodriguez, Thomas Tränkler, Werner VanGeit, JafetVillafranca Díaz, Richard Walker, Yun Wang, StefanoM. Zaninetta, Javier DeFelipe, SeanL. Hill, Idan Segev, and Felix Schürmann. Reconstruction and Simulation of Neocortical Microcircuitry. Cell, 163(2): 456–492, oct 2015. ISSN 00928674. doi: 10. 1016/j.cell.2015.09.029. URL http://www.cell.com/article/S0092867415011915/fulltext.
    OpenUrlCrossRefPubMed
  289. ↵
    D Marr. A theory of cerebellar cortex. The Journal of physiology, 202(2): 437–70, jun 1969. ISSN 0022–3751.
    OpenUrlCrossRefPubMedWeb of Science
  290. ↵
    J Martens and I Sutskever. Learning recurrent neural networks with hessian-free optimization. Proceedings of the 28th …, 2011. URL http://machinelearning.wustl.edu/mlpapers/paper{_}files/ICML2011Martens{_}532.pdf.
  291. ↵
    Bruce D McCandliss, Laurent Cohen, and Stanislas Dehaene. The visual word form area: expertise for reading in the fusiform gyrus. Trends in cognitive sciences, 7(7): 293–299, 2003.
    OpenUrlCrossRefPubMedWeb of Science
  292. ↵
    James L McClelland, David E Rumelhart, et al. Parallel distributed processing, vol. 1. Cambridge, Ma., MIT Press. Zipser D.(1986), Feature Discovery by Competitive Learning, in DE Rumel hart-JL McClelland (eds), voi, 1:151–163, 1986.
    OpenUrl
  293. ↵
    Warren S. McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5 (4):115–133, dec 1943. ISSN 0007–4985. doi: 10.1007/BF02478259. URL http://link.springer.com/10.1007/BF02478259.
    OpenUrlCrossRef
  294. ↵
    Jeffrey L McKinstry, Gerald M Edelman, and Jeffrey L Krichmar. A cerebellar model for predictive motor control tested in a brain-based device. Proceedings of the National Academy of Sciences of the United States of America, 103(9): 3387–3392, feb 2006. ISSN 0027–8424. doi: 10.1073/pnas.0511281103. URL http://www.pnas.org/content/103/9/3387.full{#}ref-4.
    OpenUrlAbstract/FREE Full Text
  295. ↵
    Elinor McKone, Kate Crookes, Nancy Kanwisher, et al. The cognitive and neural development of face recognition in humans. The cognitive neuro-sciences, 4:467–482, 2009.
    OpenUrl
  296. ↵
    Peter McLeod and Zoltan Dienes. Do fielders know where to go to catch the ball or only how to get there? Journal of Experimental Psychology: Human Perception and Performance, 22(3): 531, 1996.
    OpenUrlCrossRefWeb of Science
  297. ↵
    BW Mel. The clusteron: toward a simple abstraction for a complex neuron. Advances in neural information processing systems, 1992.
  298. ↵
    Andrew N Meltzoff. Born to learn: What infants learn from watching us. The role of early experience in infant development, pages 145–164, 1999.
  299. ↵
    Andrew N Meltzoff, Anna Waismeyer, and Alison Gopnik. Learning about causes from people: observational causal learning in 24-month-old infants. Developmental psychology, 48(5): 1215, 2012.
    OpenUrlCrossRefPubMed
  300. ↵
    Andrew N Meltzoff, Rebecca A Williamson, and Peter J Marshall. 11 developmental perspectives on action science: Lessons from infant imitation and cognitive neuroscience. Action science: Foundations of an emerging discipline, page 281, 2013.
  301. ↵
    George A. Miller. The magical number seven, plus or minus two: some limits on our capacity for processing information. 1956.
  302. ↵
    K. Miller, J. Keller, and M. Stryker. Ocular dominance column development: analysis and simulation. Science, 245(4918): 605–615, aug 1989. ISSN 0036–8075. doi: 10.1126/science.2762813. URL http://www.sciencemag.org/content/245/4918/605.short.
    OpenUrlAbstract/FREE Full Text
  303. ↵
    Kenneth D. Miller and David J. C. MacKay. The Role of Constraints in Hebbian Learning. Neural Computation, 6(1): 100–126, jan 1994. ISSN 0899–7667. doi: 10.1162/neco.1994.6.1.100. URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6797012.
    OpenUrlCrossRefWeb of Science
  304. ↵
    M Minsky. Plain talk about neurodevelopmental epistemology. 1977. URL http://dspace.mit.edu/handle/1721.1/5763.
  305. ↵
    Marvin Minsky. Society Of Mind. Simon and Schuster, 1988. ISBN 0671657135. URL http://books.google.com/books/about/Society{_}Of{_}Mind.html?id=bLDLllfRpdkC{&}pgis=1.
  306. ↵
    Marvin Minsky. The emotion machine. New York: Pantheon, 2006.
  307. ↵
    Marvin L. Minsky. Logical Versus Analogical or Symbolic Versus Connectionist or Neat Versus Scruffy, jun 1991. ISSN 0738–4602. URL http://www.aaai.org/ojs/index.php/aimagazine/article/view/894.
  308. ↵
    Marvin Lee Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. Mit Press, 1972. ISBN 0262630222. URL https://books.google.com/books/about/Perceptrons.html?id=Ow1OAQAAIAAJ{&}pgis=1.
  309. ↵
    Rajiv K. Mishra, Sooyun Kim, Segundo J. Guzman, and Peter Jonas. Symmetric spike timing-dependent plasticity at ca3-ca3 synapses optimizes storage and recall in autoassociative networks. Nat Commun, 7, May 2016. URL http://dx.doi.org/10.1038/ncomms11552.Article.
  310. ↵
    Tom M Mitchell. The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research, Rutgers Univ. New Jersey, 1980.
  311. ↵
    Shigeru Miyagawa, Robert C Berwick, and Kazuo Okanoya. The emergence of hierarchical structure in human language. Frontiers in psychology, 4:71, jan 2013. ISSN 1664–1078. doi: 10.3389/fpsyg.2013.00071.
    OpenUrlCrossRef
  312. ↵
    Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204–2212, 2014.
  313. ↵
    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518 (7540):529–533, feb 2015. ISSN 0028–0836. doi: 10.1038/nature14236. URL http://dx.doi.org/10.1038/nature14236.
    OpenUrlCrossRefPubMed
  314. ↵
    Hossein Mobahi, Ronan Collobert, and Jason Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning - ICML ’09, pages 1–8, New York, New York, USA, jun 2009. ACM Press. ISBN 9781605585161. doi: 10.1145/1553374.1553469. URL http://dl.acm.org/citation.cfm?id=1553374.1553469.
  315. ↵
    Torgeir Moberget, Eva Hilland Gullesen, Stein Andersson, Richard B Ivry, and Tor Endestad. Generalized role for the cerebellum in encoding internal models: evidence from semantic processing. The Journal of neuroscience : the official journal of the Society for Neuroscience, 34(8): 2871–8, feb 2014. ISSN 1529–2401. doi: 10.1523/JNEUROSCI.2264-13.2014. URL http://www.jneurosci.org/content/34/8/2871.short.
    OpenUrlAbstract/FREE Full Text
  316. ↵
    Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. sep 2015. URL http://arxiv.org/abs/1509.08731.
  317. ↵
    Igor Mordatch, Emanuel Todorov, and Zoran Popović. Discovery of complex behaviors through contact-invariant optimization. ACM Transactions on Graphics (TOG), 31(4): 43, 2012.
    OpenUrl
  318. ↵
    Josh Lyskowski Morgan, Daniel Raimund Berger, Arthur Willis Wetzel, and Jeff William Lichtman. The fuzzy logic of network connectivity in mouse visual thalamus. Cell, 165(1): 192–206, 2016.
    OpenUrlCrossRefPubMed
  319. ↵
    Micah M Murray, Mark T Wallace, Céline Cappe, Eric M Rouiller, and Pascal Barone. Cortical and thalamic pathways for multisensory and sensori-motor interplay. 2012.
  320. ↵
    Hajime Mushiake, Naohiro Saito, Kazuhiro Sakamoto, Yasuto Itoyama, and Jun Tanji. Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron, 50(4): 631–41, may 2006. ISSN 0896–6273. doi: 10.1016/j.neuron.2006.03.045. URL http://www.ncbi.nlm.nih.gov/pubmed/16701212.
    OpenUrlCrossRefPubMedWeb of Science
  321. ↵
    Marko Nardini, Rachael Bedford, and Denis Mareschal. Fusion of visual cues is not mandatory in children. Proceedings of the National Academy of Sciences, 107(39): 17041–17046, 2010.
    OpenUrlAbstract/FREE Full Text
  322. ↵
    Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural Programmer: Inducing Latent Programs with Gradient Descent. nov 2015. URL http://arxiv.org/abs/1511.04834.
  323. ↵
    Bernhard Nessler, Michael Pfeiffer, Lars Buesing, and Wolfgang Maass. Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity. PLoS computational biology, 9(4):e1003037, apr 2013. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1003037.
    OpenUrlCrossRef
  324. ↵
    AY Ng and SJ Russell. Algorithms for inverse reinforcement learning. Icml, 2000. URL http://ai.stanford.edu/{˜}ang/papers/icml00-irl.pdf.
  325. ↵
    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. arXiv preprint arXiv:1603.09246, 2016.
  326. ↵
    H Freyja Ólafsdóttir, Caswell Barry, Aman B Saleem, Demis Hassabis, and Hugo J Spiers. Hippocampal place cells construct reward related sequences through unexplored space. eLife, 4: e06063, jun 2015. ISSN 2050–084X. doi: 10.7554/eLife.06063. URL http://elifesciences.org/content/4/e06063.abstract.
    OpenUrlCrossRefPubMed
  327. ↵
    Y Ollivier and G Charpiat. Training recurrent networks online without backtracking. arXiv preprint arXiv:1507.07680, 2015. URL http://arxiv.org/abs/1507.07680.
  328. ↵
    B A Olshausen, C H Anderson, and D C Van Essen. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. The Journal of neuroscience : the official journal of the Society for Neuroscience, 13(11): 4700–19, nov 1993. ISSN 0270–6474. URL http://www.ncbi.nlm.nih.gov/pubmed/8229193.
    OpenUrlAbstract/FREE Full Text
  329. ↵
    Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381 (6583):607–609, jun 1996. ISSN 0028–0836. doi: 10.1038/381607a0. URL http://www.ncbi.nlm.nih.gov/pubmed/8637596.
    OpenUrlCrossRefPubMedWeb of Science
  330. ↵
    Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37(23): 3311–3325, dec 1997. ISSN 00426989. doi: 10.1016/S0042-6989(97)00169-7. URL http://www.sciencedirect.com/science/article/pii/S0042698997001697.
    OpenUrlCrossRefPubMedWeb of Science
  331. ↵
    Bruno A Olshausen and David J Field. What is the other 85% of v1 doing. Problems in Systems Neuroscience, 4(5): 182–211, 2004.
    OpenUrl
  332. ↵
    Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1717–1724, 2014.
  333. ↵
    Randall C. O’Reilly. Biologically Plausible Error-Driven Learning Using Local Activation Differences: The Generalized Recirculation Algorithm. Neural Computation, 8(5): 895–938, jul 1996. ISSN 0899–7667. doi: 10.1162/neco.1996.8.5.895. URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6796552.
    OpenUrlCrossRefWeb of Science
  334. ↵
    Randall C. O’Reilly. Biologically based computational models of high-level cognition. Science (New York, N.Y.), 314(5796): 91–4, oct 2006. ISSN 1095–9203. doi: 10.1126/science.1127242. URL http://www.sciencemag.org/content/314/5796/91.
    OpenUrlAbstract/FREE Full Text
  335. ↵
    Randall C O’Reilly and Michael J Frank. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural computation, 18(2): 283–328, mar 2006. ISSN 0899–7667. doi: 10.1162/089976606775093909. URL http://www.ncbi.nlm.nih.gov/pubmed/16378516.
    OpenUrlCrossRefPubMedWeb of Science
  336. ↵
    Randall C. O’Reilly, Thomas E. Hazy, Jessica Mollick, Prescott Mackie, and Seth Herd. Goal-Driven Cognition in the Brain: A Computational Framework. apr 2014a. URL http://arxiv.org/abs/1404.7591.
  337. ↵
    Randall C. O’Reilly, Dean Wyatte, and John Rohrlich. Learning Through Time in the Thalamocortical Loops. page 37, jul 2014b. URL http://arxiv.org/abs/1407.3432.
  338. ↵
    RC O’Reilly, Y Munakata, MJ Frank, and TE Hazy. Computational cognitive neuro-science. 2012. URL grey.colorado.edu/mediawiki/sites/CompCogNeuro/images/a/a2/Cecn{_}oreilly{_}intro.pdf.
  339. ↵
    A. Emin Orhan and Wei Ji Ma. The Inevitability of Probability: Probabilistic Inference in Generic Neural Networks Trained with Non-Probabilistic Feedback. page 26, jan 2016. URL http://arxiv.org/abs/1601.03060.
  340. ↵
    Lucy M Palmer, Adam S Shai, James E Reeve, Harry L Anderson, Ole Paulsen, and Matthew E Larkum. NMDA spikes enhance action potential generation during sensory input. Nature neuro-science, 17(3): 383–90, mar 2014. ISSN 1546–1726. doi: 10.1038/nn.3646. URL http://dx.doi.org/10.1038/nn.3646.
    OpenUrlCrossRefPubMed
  341. ↵
    C Parisien, CH Anderson, and C Eliasmith. Solving the problem of negative synaptic weights in cortical models. Neural computation, 2008. URL http://ieeexplore.ieee.org/xpls/abs{_}all.jsp?arnumber=6796691.
  342. ↵
    Anitha Pasupathy and Earl K Miller. Different time courses of learning-related activity in the pre-frontal cortex and striatum. Nature, 433(7028): 873–876, feb 2005. ISSN 1476–4687. doi: 10.1038/nature03287. URL http://dx.doi.org/10.1038/nature03287.
    OpenUrlCrossRefPubMedWeb of Science
  343. ↵
    AB Patel, T Nguyen, and RG Baraniuk. A Probabilistic Theory of Deep Learning. arXiv preprint arXiv:1504.00641, 2015. URL http://arxiv.org/abs/1504.00641.
  344. ↵
    Cengiz Pehlevan and Dmitri B Chklovskii. Optimization theory of hebbian/anti-hebbian networks for pca and whitening. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1458–1465. IEEE, 2015.
  345. ↵
    Gertrudis Perea, Marta Navarrete, and Alfonso Araque. Tripartite synapses: astrocytes process and control synaptic information. Trends in neurosciences, 32(8): 421–31, aug 2009. ISSN 1878–108X. doi: 10.1016/j.tins.2009.05.001. URL http://www.ncbi.nlm.nih.gov/pubmed/19615761.
    OpenUrlCrossRefPubMedWeb of Science
  346. ↵
    Alexander A. Petrov, David J. Jilk, and Randall C. O’Reilly. The Leabra architecture: Specialization without modularity. Behavioral and Brain Sciences, 33(04): 286–287, oct 2010. ISSN 0140–525X. doi: 10.1017/S0140525X10001160. URL http://journals.cambridge.org/abstract{_}S0140525X10001160.
    OpenUrlCrossRef
  347. ↵
    Giovanni Pezzulo, Paul F M J Verschure, Christian Balkenius, and Cyriel M A Pennartz. The principles of goal-directed decision-making: from neural mechanisms to computation and robotics. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 369(1655):20130470–, nov 2014. ISSN 1471–2970. doi: 10.1098/rstb.2013.0470. URL http://rstb.royalsocietypublishing.org/content/369/1655/20130470.short.
    OpenUrlCrossRefPubMed
  348. ↵
    Jean-Pascal Pfister and Wulfram Gerstner. Triplets of spikes in a model of spike timing-dependent plasticity. The Journal of neuroscience, 26(38): 9673–9682, 2006.
    OpenUrlAbstract/FREE Full Text
  349. ↵
    Ann T Phillips, Henry M Wellman, and Elizabeth S Spelke. Infants’ ability to connect gaze and emotional expression to intentional action. Cognition, 85(1): 53–78, 2002.
    OpenUrlCrossRefPubMedWeb of Science
  350. ↵
    Filip Piekniewski, Patryk Laurent, Csaba Petre, Micah Richert, Dimitry Fisher, and Todd Hylton. Unsupervised learning from continuous video in a scalable predictive recurrent network. arXiv preprint arXiv:1607.06854, 2016.
  351. ↵
    Fernando J Pineda. Generalization of back-propagation to recurrent neural networks. Physical review letters, 59(19): 2229, 1987.
    OpenUrlCrossRefPubMedWeb of Science
  352. ↵
    S Pinker. How the mind works. Annals of the New York Academy of Sciences, 1999. URL http://onlinelibrary.wiley.com/doi/10.1111/j.1749–6632.1999.tb08538.x/full.
  353. ↵
    T A Plate. Holographic reduced representations. IEEE transactions on neural networks /a publication of the IEEE Neural Networks Council, 6(3): 623–641, jan 1995. ISSN 1045–9227. doi: 10.1109/72.377968. URL http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=377968.
    OpenUrlCrossRefPubMedWeb of Science
  354. ↵
    Tomaso Poggio. What if… 2015. URL http://cbmm.mit.edu/views-reviews/article/what-if.
  355. ↵
    Tomaso Poggio and Emilio Bizzi. Generalization in vision and motor control. Nature, 431(7010): 768–774, 2004.
    OpenUrlCrossRefPubMedWeb of Science
  356. ↵
    Filip Ponulak and John J Hopfield. Rapid, parallel path planning by propagating wavefronts of spiking neural activity. Frontiers in computational neuroscience, 7:98, jan 2013. ISSN 1662–5188. doi: 10.3389/fncom.2013.00098.
    OpenUrlCrossRef
  357. ↵
    Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. nov 2015. URL http://arxiv.org/abs/1511.06434.
  358. ↵
    Kanaka Rajan, Christopher D Harvey, and David W Tank. Recurrent network models of sequence generation and memory. Neuron, 90(1): 128–142, 2016.
    OpenUrl
  359. ↵
    Vilayanur S Ramachandran. Mirror neurons and imitation learning as the driving force behind the great leap forward in human evolution, 2000.
  360. ↵
    Rajesh PN Rao. Bayesian computation in recurrent neural circuits. Neural computation, 16(1): 1–38, 2004.
    OpenUrlCrossRefPubMedWeb of Science
  361. ↵
    Nicolas Rashevsky. Mathematical biophysics: physico-mathematical foundations of biology. Bull. Amer. Math. Soc. 45 (1939), 223–224 DOI: http://dx.doi.org/10.1090/S0002–9904–1939–06963–2 PII, pages 0002–9904, 1939.
    OpenUrl
  362. ↵
    A Rasmus and M Berglund. Semi-Supervised Learning with Ladder Networks. Advances in Neural …, 2015. URL papers.nips.cc/paper/5947-semi-supervised-learning-with-ladder-networks.
  363. ↵
    John H. Reynolds and Robert Desimone. The Role of Neural Mechanisms of Attention in Solving the Binding Problem. Neuron, 24(1): 19–29, sep 1999. ISSN 08966273. doi: 10.1016/S0896-6273(00)80819-3. URL http://www.sciencedirect.com/science/article/pii/S0896627300808193.
    OpenUrlCrossRefPubMedWeb of Science
  364. ↵
    Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. One-Shot Generalization in Deep Generative Models. mar 2016. URL http://arxiv.org/abs/1603.05106.
  365. ↵
    Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585–90, may 2013. ISSN 1476–4687. doi: 10.1038/nature12160.
    OpenUrlCrossRefPubMedWeb of Science
  366. ↵
    DA Robinson. Implications of neural networks for how we think about brain function. Behavioral and brain sciences, 1992. URL http://journals.cambridge.org/abstract{_}S0140525X00072563.
  367. ↵
    A Rodriguez and R Granger. The grammar of mammalian brain capacity. Theoretical Computer Science, 633:100–111, 2016.
    OpenUrl
  368. ↵
    A Rodriguez, J Whitson, and R Granger. Derivation and analysis of basic computational operations of thalamocortical circuits. Journal of cognitive neuroscience, 16(5): 856–877, 2004.
    OpenUrlCrossRefPubMedWeb of Science
  369. ↵
    Pieter R Roelfsema and Arjen van Ooyen. Attention-gated reinforcement learning of internal representations for classification. Neural computation, 17 (10):2176–214, oct 2005. ISSN 0899–7667. doi: 10.1162/0899766054615699. URL http://www.ncbi.nlm.nih.gov/pubmed/16105222.
    OpenUrlCrossRefPubMedWeb of Science
  370. ↵
    PR Roelfsema, A van Ooyen, and T Watanabe. Perceptual learning rules based on reinforcers and attention. Trends in cognitive sciences, 2010. URL http://www.sciencedirect.com/science/article/pii/S1364661309002617.
  371. ↵
    Edmund T Rolls. The mechanisms for pattern completion and pattern separation in the hippocampus. Frontiers in systems neuroscience, 7:74, jan 2013. ISSN 1662–5137. doi: 10.3389/fnsys.2013.00074. URL http://journal.frontiersin.org/article/10.3389/fnsys.2013.00074/abstract.
    OpenUrlCrossRefPubMed
  372. ↵
    Jaldert O Rombouts, Sander M Bohte, and Pieter R Roelfsema. How attention can create synaptic tags for the learning of working memories in sequential tasks. PLoS computational biology, 11(3):e1004060, mar 2015. ISSN 1553–7358. doi: 10.1371/journal.pcbi.1004060. URL http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004060.
    OpenUrlCrossRef
  373. ↵
    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
  374. ↵
    Y Roudi and G Taylor. Learning with hidden variables. Current opinion in neurobiology, 2015. URL http://www.sciencedirect.com/science/article/pii/S0959438815001245.
  375. ↵
    Christopher J Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural computation, 20(10): 2526–63, oct 2008. ISSN 0899–7667. doi: 10.1162/neco.2008.03-07-486. URL http://www.ncbi.nlm.nih.gov/pubmed/18439138.
    OpenUrlCrossRefPubMedWeb of Science
  376. ↵
    Alon Rubin, Nitzan Geva, Liron Sheintuch, and Yaniv Ziv. Hippocampal ensemble dynamics timestamp events in long-term memory. eLife, page e12247, 2015.
  377. ↵
    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J.<