Learning sensory representations with intrinsic plasticity

Abstract

Intrinsic plasticity (IP) refers to a neuron's ability to regulate its firing activity by adapting its intrinsic excitability. Previously, we showed that model neurons combining a model of IP based on information theory with Hebbian synaptic plasticity can adapt their weight vector to discover heavy-tailed directions in the input space. In this paper we show how a network of such units can solve a standard non-linear independent component analysis (ICA) problem. We also present a model for the formation of maps of oriented receptive fields in primary visual cortex and compare our results with those from ICA. Together, our results indicate that intrinsic plasticity that tries to locally maximize information transmission at the level of individual neurons may play an important role for the learning of efficient sensory representations in the cortex.

Introduction

Computational models of unsupervised learning of sensory representations in the brain abound. Frequently, they fall into one of two categories: mechanistic models or functional models. Mechanistic models start from neuroscientific data about the structure of cortical networks and cortical plasticity mechanisms (cell types, connection patterns, plasticity rules, etc.) which are distilled into simplified models. These models are trained on actual sensory data or noise patterns and the learned representations can be compared to neurophysiological observations. If the resulting representations are similar to those found in the brain then this provides evidence that the processes in the brain have been accurately captured, but it does not clarify why the brain operates this way or in what sense the brain's solution may be optimal. An example of a model of this kind is by Linsker [15], where V1-style orientation columns are learned from random prenatal visual noise through Hebbian learning. Later Miller extended this work to learn many of the various map-structures in V1, and used model neurons that were somewhat more plausible [19].

Functional models focus on the abstract computational goal of the problem. For the case of learning sensory representations they start by asking: what is the optimal way to represent sensory stimuli such as natural images, where optimality is usually defined with respect to certain statistical criteria (e.g. sparseness, independence, temporal coherence, etc.) and additional constraints. Algorithms are derived to learn the optimal solution to the problem, which can again be compared to neuroscientific data. If the found solution resembles the biological solution, then this provides evidence that the brain may in fact be trying to optimize a similar objective function. Through what mechanisms the brain may achieve this goal is typically not answered, however. Some examples of such an approach will be given below.

Both mechanistic and functional models have their merits, but for a comprehensive understanding of sensory coding in the cortex we arguably have to develop models that bridge functional and mechanistic levels of description. Such models should explain how the physiological mechanisms contribute to optimizing the system's information processing properties in a meaningful way. In the following, we develop a model that can be viewed as a step in this direction.

A central idea in many functional models of the development of sensory representations is information maximization [1], [3], [16], [20], [22]. According to some formulations of this idea, individual neurons should maximize the entropy of their firing rate distribution. If the firing rate is constrained to lie in a fixed interval between zero and the neuron's maximum firing rate, then entropy maximization means that the neuron should use all its firing rate levels equally often. In order to achieve this, it should spread out its responses in dense regions of the input space and compress responses in sparse regions such that it maps the distribution of its inputs to a uniform distribution of its outputs, maximizing entropy. Biological evidence for this idea comes from Laughlin, who showed that blowfly large monopolar cells have been adapted so that their input/output transfer functions nearly optimally represent the contrast statistics of the blowfly's visual environment [13].

Information maximization may not be the only important objective, however, and energy considerations may also play an important role for sensory coding in the brain, e.g. [14]. In particular, Baddeley et al. found that neurons in different visual cortical areas of cats and monkeys show exponential distributions of their firing rate. They have argued that this maximizes a neuron's information transfer given a fixed energy budget [2]. This is because the exponential distribution has the maximum entropy among all distributions of a positive random variable (the firing rate) with a fixed mean. This and other reasons suggest that sparse representations, where individual units are highly active only rarely, may be an important principle of sensory coding [10].

On the modeling side, Olshausen and Field showed that localized, oriented, and bandpass receptive fields similar to those observed in primary visual cortex (V1) arise when optimizing image reconstruction error subject to lifetime sparseness constraints [21]. They imposed a sparse prior on the contribution of each basis function in a generative model with the intuition that among the space of possible sources of an image, each one is present only rarely. In a closely related approach, Bell and Sejnowski showed that the information maximization principle can be applied to the independent component analysis (ICA) problem. They applied their technique to natural images and also found localized, oriented, and bandpass sources [4].

Most work on the learning of sensory representations has focused on synaptic plasticity as the only mechanism for learning efficient codes. But it is becoming increasingly clear that biological neurons also regulate their pattern of firing by adapting their intrinsic excitability through the modification of voltage-gated channels in their membrane. Such intrinsic plasticity (IP) seems to be a ubiquitous phenomenon in the brain [30]. For example, Desai et al. showed that neurons that had been prevented from spiking for two days increased their response to current injection [6]. Consistent with this finding, it is frequently assumed that IP contributes to the homeostasis of a neuron's mean firing activity. A few computational models do in fact incorporate a mechanism for regulating the mean activity level of a unit by controlling a “threshold” parameter [7], [8], [9]. But it is also plausible that IP may help to optimize the encoding and transmission of information in a more sophisticated fashion. Concretely, it has been speculated that IP may be instrumental in achieving approximately exponential firing rate distributions in cortical neurons [23]. More recently, we have shown that an IP mechanism that drives a neuron to exhibit an exponential firing rate distribution can synergistically interact with Hebbian learning at the synapses. The two processes lead to the discovery of heavy-tailed directions in the input space [24], [26].

In this paper we extend these results to networks of neurons with IP and Hebbian learning. Our specific goal is to explore the potential role of IP for learning efficient map-like representations for sensory stimuli. The model we present in the following attempts to bridge the gap between mechanistic and functional models. On the one hand, it has a clear connection to the idea of information maximization and energy efficient coding [28]. On the other hand, it has a mechanistic formulation that is biologically viable because the learning mechanisms make use of information that is local in time and space. While similar bridges have been attempted before, e.g. [5], [8], our model is distinguished by utilizing an IP model derived from information theory as a mechanism for the learning of efficient sensory representations.