Pattern-information analysis: From stimulus decoding to computational-model testing

Abstract

Pattern-information analysis has become an important new paradigm in functional imaging. Here I review and compare existing approaches with a focus on the question of what we can learn from them in terms of brain theory. The most popular and widespread method is stimulus decoding by response-pattern classification. This approach addresses the question whether activity patterns in a given region carry information about the stimulus category. Pattern classification uses generic models of the stimulus–response relationship that do not mimic brain information processing and treats the stimulus space as categorical—a simplification that is often helpful, but also limiting in terms of the questions that can be addressed. We can address the question whether representations are consistent across different stimulus sets or tasks by cross-decoding, where the classifier is trained with one set of stimuli (or task) and tested with another. Beyond pattern classification, a major new direction is the integration of computational models of brain information processing into pattern-information analysis. This approach enables us to address the question to what extent competing computational models are consistent with the stimulus representations in a brain region. Two methods that test computational models are voxel receptive-field modeling and representational similarity analysis. These methods sample the stimulus (or mental-state) space more richly, estimate a separate response pattern for each stimulus, and can generalize from the stimulus sample to a stimulus population. Computational models that mimic brain information processing predict responses from stimuli. The reverse transform can be modeled to reconstruct stimuli from responses. Stimulus reconstruction is a challenging feat of engineering, but the implications of the results for brain theory are not always clear. Exploratory pattern analyses complement the confirmatory approaches mentioned so far and can reveal strong, unexpected effects that might be missed when testing only a restricted set of predefined hypotheses.

Introduction

Perceptual, cognitive, and motor representations are thought to reside in neuronal population codes (e.g. Averbeck et al., 2006). This provides a straightforward motivation for pattern-information analysis in functional imaging (e.g. Haxby et al., 2001, Cox and Savoy, 2003, Carlson et al., 2003, Mitchell et al., 2004, Kriegeskorte, 2004, Kamitani and Tong, 2005, Haynes and Rees, 2006, Norman et al., 2006, Kriegeskorte et al., 2006, Mur et al., 2009, Kriegeskorte et al., 2009b) and in cell recording (Hung et al., 2005, Kiani et al., 2007): to elucidate (more fully than single-unit or single-voxel analyses can) what information is present in a given brain region. In this commentary, I review our current toolbox of pattern-information approaches with a focus on what we can learn from them about brain function. The paper is divided into four sections, each of which discusses a different approach to pattern-information analysis.

The first section discusses methods that test for information about a particular stimulus dimension in brain response patterns (hypothesis-driven goal 1). This approach includes pattern-classifier decoding, the most popular and widespread type of pattern-information analysis. Classifier decoding treats the stimulus space as categorical and “predicts” the stimulus category from the response pattern. More generally, the stimulus space could be treated as continuous, and I argue that the direction, in which the dependency between stimulus and response is modeled (decoding or encoding) is largely inessential to the neuroscientific interpretation (Fig. 1). Most applications have used generic linear models. I argue in favor of linear models on the basis of their stability and interpretability.

The second section discusses methods that test whether a computational model of brain information processing can account for a region's response patterns (hypothesis-driven goal 2). Traditionally, statistical analysis of brain activity uses generic, often linear, models that do not simulate brain information processing (as in goal 1). Brain-experimental results are then related to computational models only at the level of verbal theory. To directly test computational models with brain-activity data, we need to integrate these (typically nonlinear) models into data analysis. Two methods that achieve this are voxel-receptive-field modeling (Dumoulin and Wandell, 2008, Kay et al., 2008, Mitchell et al., 2008) and representational similarity analysis (Kriegeskorte et al., 2008a, Kriegeskorte et al., 2008b). These methods sample the stimulus (or mental-state) space more richly, estimating a separate response pattern for each stimulus and forgoing any predefined stimulus grouping (Fig. 2). As these approaches are just beginning to gain momentum, there are few examples in the literature. I therefore take a different approach in this section and review three studies in detail (Mitchell et al., 2008, Kay et al., 2008, Kriegeskorte et al., 2008a). Voxel-receptive-field modeling predicts response patterns; representational similarity analysis predicts response-pattern dissimilarities, providing alternative statistical tests of the same conceptual claim, namely that a computational model can account for the representational space of a brain region (Fig. 3, Fig. 4).

The shorter third and fourth sections discuss exploratory analysis of population activity patterns and stimulus reconstruction, respectively. These two approaches do not test explicit hypotheses about brain function. Exploratory analysis requires fewer assumptions and can yield unexpected discoveries. It can reveal stimulus–response relationships that explain a lot of variance, but might have been missed in an overly restricted hypothesis-driven approach. Stimulus reconstruction models perceptual processing in reverse, predicting the stimulus from the response pattern (a form of decoding that generalizes to novel stimuli). Reconstruction is a tough engineering challenge and provides an intuitive illustration (the reconstructed stimulus) of the information represented in a region. However, it is not clear how exactly stimulus reconstruction results constrain neuroscientific theory. Fig. 5 compares the entire range of pattern-information approaches discussed in this paper.

For simplicity, this commentary focuses on the relationship between “stimulus” and “response” in considering pattern-information analysis. However, the arguments apply to other scenarios as well, where the mental states investigated are not directly elicited by stimuli (e.g. mental imagery), or where a brain-behavior relationship is analyzed. The application of pattern-information methods to the relationships between brain and behavior and between different brain regions, individuals, and species have recently been discussed elsewhere (Raizada and Kriegeskorte, (2010), Kriegeskorte, 2009).