Subjects

Seventeen subjects (8 female, mean age 25.9 years, range 20–33 years, 2 left-handed) were recruited from the University of Birmingham campus and paid for their participation. All observers had normal or corrected to normal vision, no history of neurological disorders and gave written informed consent. The study was approved by the Science, Technology, Engineering and Mathematics Ethical Review Committee of the University of Birmingham.

Complete data sets (2 experimental runs of EEG alone, 5 experimental runs of simultaneous EEG-fMRI (see below)) were acquired from 13 of the 17 subjects (for two of the four subjects who did not have complete data, incomplete EEG data were recorded outside the MR scanner; for one of the four incomplete psychophysical data were recorded inside the MR scanner; while for the remaining subject the EEG data recorded inside the MR scanner were strongly contaminated by movement artefacts of the reference electrode due to contact with the head-coil). All information theoretic analyses (outside and inside MR scanner EEG data, fMRI data, and EEG-fMRI data) are based on the 13 complete data sets. To identify regions of interest for the information theoretic analyses with maximum detection power, a total of 16 fMRI data sets were included in the GLM-group analysis. One subject's fMRI data were excluded from the GLM analysis because no psychophysical data could be recorded.

Experimental design and paradigm

In a 2×2 factorial within-subject design, observers performed a perceptual decision task, similar to that described in [19], [22], [26] (Figure 2A). On each trial, a visual stimulus depicting either a face or a car was presented in one visual hemifield (left/right eccentricity of stimulus centre 11 degrees of visual angle, stimulus extension 9 degrees of visual angle) for 200 ms and the observer was asked to indicate via a button press whether a face or a car stimulus was presented. For the button presses, observers used their right index and middle finger for the two categories, and the mapping from stimulus category to response button was counterbalanced across observers. The informativeness of the visual stimulus was manipulated by altering the phase coherence of its spatial frequency spectrum resulting in low and high informative trials (see below). On half of the trials, a cueing arrow shown continuously for 1 s prior to the stimulus indicated in which hemifield the stimulus would be presented (Figure 2B). The observers were asked to allocate their spatial attention to the respective hemifield, while maintaining steady central fixation (spatial prioritization condition). On the other half of the trials, the two-headed cueing arrow was uninformative and the stimulus was presented randomly in either hemifield (no spatial prioritization condition). Face and car stimuli were equally distributed over the four experimental conditions. The order of stimulus presentation was randomized. Observers were asked to respond as quickly and accurately as possible with an emphasis on responding as quickly as possible and to maintain stable fixation of the central fixation cross throughout the experiment. Analyses of eye-movement data (Figure S1) obtained during the combined EEG-fMRI data acquisition in six subjects indicated that good fixation was achieved.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Experimental Design and Paradigm.

A. 2×2 factorial experimental design with factors informativeness (high, low) and spatial prioritization (yes, no). On each trial of the experiment, the observer was presented a face or car stimulus, which had been manipulated according to visual informativeness, and the observer was prompted to either spatially prioritize the stimulus display or not. The stimulus category (face or car), which the observer was asked to discriminate, was manipulated orthogonally to the other factors. B. Single experimental trial outline. Prior to the presentation of the stimulus, either a one-headed arrow indicated the hemifield of the subsequent stimulus presentation, or a two-headed arrow was uninformative in this respect. The cueing arrow was shown continuously for 1 s pre-stimulus, the stimulus itself for 200 ms. The observer was asked to respond as quickly and as accurately as possible with no restrictions on the response window. The inter-trial interval was 0–300 ms for the EEG only and 10–12 s for the combined EEG-fMRI recordings.

https://doi.org/10.1371/journal.pone.0033896.g002

For the EEG only recordings outside of the scanner, data from 72 trials for each of the four conditions (half of them face stimuli) were recorded with an inter-trial interval randomized between 0–300 ms. The data acquisition was split into two experimental runs of approximately 10 minutes each. For the combined EEG-fMRI recordings data from 90 trials for each of the four conditions (half of them face stimuli) were recorded with an inter-trial interval discretely randomized between 10 s and 12 s (5 or 6 TRs). This long inter-trial interval was chosen to obtain reliable recordings of single-trial haemodynamic responses. The 90 trials were split into five experimental runs, each lasting approximately 14 minutes. Prior to the EEG recordings the observers also completed two practice runs to familiarise themselves with the task.

Stimuli

The stimulus set consisted of 18 pictures of cars and 18 pictures of faces, similar to the stimulus set used in [19], [22], [26]. The car images were obtained from http://liinc.bme.columbia.edu/mainTemplate.htm?liinc_downloads.htm while the face images were obtained from the Max Planck face database [28]. The image categories were matched for the number of frontal, and left and right lateral views. All images were converted to bitmap format (.bmp) and the corresponding 256×256 matrices saved with 8 bit depth. The two stimulus sets were matched for their mean driving luminance and contrast as assessed by a one-way ANOVA with factor ‘image category’ and levels ‘face’ and ‘car’ (mean driving luminance: F_(1,34) = 0.08, p = 0.78, contrast: F_(1,34) = 0.22, p = 0.64). To manipulate the informativeness of the images, the spatial phase spectra were linearly weighted with a phase spectrum of a Gaussian noise image using the weighted mean phase technique as described in [29]. With the original phase of an image given by , the final phase was computed as follows:(1)where(2)(3)and is the phase of uniform random noise and is the signal-to-noise weighting coefficient. Based on a psychophysical pilot study (Figure S2), stimuli with weighting coefficients (high informativeness) and (low informativeness) were chosen for the experiment in order to elicit reliable differences in the response times for either stimulus class, while still allowing accurate performance of the task.

Data acquisition

EEG data were recorded using a 64 channel MR compatible EEG system (BrainAmp MR Plus, Brain Products, Munich, Germany), which incorporates current limiting resistors of 5 kΩ at the amplifier input and in each electrode. The EEG cap consisted of 62 scalp electrodes distributed according to the 10–20 system [30] and two additional electrodes, one of which was attached approximately 2 cm below the left collarbone for recording the ECG, while the other was attached below the left eye (on the lower orbital portion of the orbicularis oculi muscle) for detection of eyeblink artefacts. Data were sampled at 5000 Hz. Impedance at all recording electrodes was less than 20 kΩ. For simultaneous EEG-fMRI recordings, the EEG data acquisition setup clock was synchronised with the MRI scanner clock using Brain Product's SyncBox, resulting in exactly 10,000 data points per EPI-TR interval (see details of the fMRI sequence below). The EEG set-up was identical for the recordings outside and inside the MR scanner. In the following, the EEG data set recorded outside the MR scanner will be referred to as EEG only, while the EEG data set recorded simultaneously with the fMRI data will be referred to as EEG-fMRI data.

The simultaneous EEG-fMRI experiment was conducted at the Birmingham University Imaging Centre using a 3T Philips Achieva MRI scanner. An initial T1-weighted anatomical scan (1 mm isotropic voxels) and T2*-weighted functional data were collected with an eight channel phased array SENSE head coil. EPI data (gradient echo-pulse sequence) were acquired from 32 slices (3×3×4 mm resolution, TR 2000 ms, TE 35 ms, SENSE factor 2, flip angle 80°). Slices were oriented parallel to the AC-PC axis of the observer's brain and positioned to cover the entire brain space.

Eye-movements were monitored for six observers while performing the task in the MR scanner using an ASL 6000 Eye-tracker (Applied Science Laboratories, Bedford, MA).

Data pre-processing

EEG data acquired outside the MRI scanner were referenced to electrode FCz, partitioned into data acquisition sessions, band-pass filtered from 0.5 to 25 Hz and down-sampled to 500 Hz using Brain Vision Analyzer 2.0 (Brain Products, Munich, Germany). EEG data acquired inside the MR scanner were referenced to electrode FCz, partitioned into data acquisition sessions and the MR gradient and ballistocardiogram artefact removed using Brain Vision Analyzer 2.0, band-pass filtered from 0.5 to 25 Hz, and down-sampled to 500 Hz. To identify artefactual non-cerebral EEG components (i.e. eye-movements, muscular movement, environmental noise and, in the case of the EEG data acquired simultaneously with the EPI data, residual MR and BCG artefacts) a semiautomatic ICA-based procedure was employed [31], [32]. Upon rejection of the artefactual independent components and back-projection of the remaining independent components into channel space, all trials with maximum or minimum amplitudes outside a physiological range of −100 µV to 100 µV were discarded prior to further analyses.

SPM5 [33] was used for fMRI data pre-processing, and included anatomical realignment, slice scan time correction (reference slice 16), re-interpolation to 2×2×2 mm voxels, anatomical normalization to MNI space and spatial smoothing (5 mm FWHM Gaussian kernel).

EEG data analysis

EEG data were analyzed in electrode space using custom-written Matlab code (The Mathworks, Natick, MA). Specifically, event-related potentials were computed using a 100 ms pre-stimulus baseline and 500 ms post-stimulus window. Because the stimuli were presented lateralized and occipito-parieto electrodes were considered of primary interest for the analysis, data for a given trial (left or right hemifield presentation) were allocated to the respective contra-lateral electrode set and collapsed over hemispheres for subsequent analyses. Grand averages of event-related potentials were computed across all trials of a given condition and subjects for pooled electrodes O1, O2, PO3, PO4, PO7 and PO8.

Upon the identification of time-windows of interest based on visual inspection of the grand average EEG data (see Results) and previous studies [19], [22], [27], [34]–[36], single-trial amplitude estimates were extracted from the EEG time-course for five discrete, non-overlapping time-windows of interest covering the entire −100 to 500 ms peri-stimulus period. For the EEG only data, these time-windows consisted of the intervals −100 to 58 ms, 60 to 120 ms, 122 to 154 ms, 156 to 370 ms, and 372 to 500 ms. As the equivalent neuronal and behavioural responses were slightly delayed for the combined EEG-fMRI data acquisition, the corresponding time-windows for the EEG-fMRI data were determined as −100 to 58 ms, 60 to 140 ms, 142 to 188 ms, 190 to 400 ms and 402 to 500 ms. For each time-window, except the third, the maximum amplitude on each single trial was extracted. For the third time-window, which encompassed a negative potential deflection, the minimum amplitude was extracted. These time-domain features were extracted from a set of eight parieto-occipital electrodes (O1/2, PO3/4, PO7/8, P1/2, P3/4, P5/6, P7/8 and TP7/8), whose selection was based on the topography of the grand average event-related potential (see Results). Upon feature extraction and information estimation, the information theoretic results for the EEG marginal features and EEG-fMRI joint features were averaged across these electrodes to yield the final information estimates.

fMRI data analysis

To identify fMRI regions of interest (ROIs), the experimental data of each individual voxel was modelled using the standard univariate GLM approach in SPM5 [33]. A total of 16 experimental regressors were used, corresponding to the 8 stimulus conditions (2 coherence×2 prioritization×2 stimulus category levels) and 2 presentation locations (left and right visual hemifield). Voxel time-courses were modelled in an event-related fashion using regressors obtained by convolving each stimulus onset unit impulse with a canonical haemodynamic response function and its first temporal derivative. Additional nuisance covariates included the realignment parameters to account for residual motion artefacts and session specific means. A mixed-effects analysis was then implemented using a summary statistics approach to allow inferences at the population level [37], [38]: upon estimation of the model parameters for each subject, a subject-specific contrast image for each effect of interest was computed. Contrast vectors for the following effects of interest were used: all stimuli > fixation, left hemifield stimuli > right hemifield stimuli, right hemifield stimuli > left hemifield stimuli, high coherence stimuli > low coherence stimuli, low coherence stimuli > high coherence stimuli and face stimuli > car stimuli. The contrast images were then subjected to a one-sample t-test at the second level (group level).

For the information theoretic analysis filtered and whitened data were extracted from a sphere of 2 mm radius centred on the subject specific peak for the relevant contrast. The subject specific peak for each ROI was uniquely identified by visual inspection as the coordinates of the peak of the significantly activated cluster that was closest to the group mixed effects analysis coordinates. The average deviation across ROIs and observers from the group coordinates was 15 (±1 SEM) mm. Upon time-course extraction, single-trial event-related haemodynamic responses were computed as percent signal change with respect to a baseline comprising two pre-stimulus data points. The single-trial fMRI amplitude feature was then determined as the maximum over the 10 s post-stimulus period. The HRF amplitude was chosen as the only fMRI data feature as a) it was shown to be marginally more informative compared to the other fMRI data features in [11], b) the fMRI data features of [11] did not vary substantially, and c) to simplify the analysis and prohibit exponential growth in the number of EEG-fMRI data feature combinations.

It can be argued that an inter-stimulus interval of 10–12 s is too short to extract single-trial HRF maximum estimates without the use of an HRF deconvolution model. However, for the current stimulation protocol group average BOLD signal responses returned to baseline approximately 8–10 seconds post-stimulus and strong post-stimulus undershoots were not observed (data not shown, but see Figure S3 for a single subject example). Further, previous studies on the effect of the inter-stimulus interval indicate that inter-stimulus intervals can be reduced down to 6 s with very little change in HRF effect size [39], [40]. Given these considerations, extracting the single-trial HRF maximum amplitudes without deconvolution is appropriate in the context of the current study.

Information theoretic EEG-fMRI feature integration

The calculation of information theoretic quantities from EEG-fMRI data features involves the estimation of probability mass functions, which in the current context was accomplished with a histogram approach as discussed below and in more detail in [41]. Here, only those aspects that are specific to the current experimental paradigm will be discussed.

To elucidate the spatiotemporal information representation signature for perceptual decisions, mutual information quantities were formulated relating to external, internal and behavioural state variables. Here, the external and internal variables are equivalent to experimental manipulations, i.e. stimulus variables. Specifically, the interest lies in a) the informativeness of the stimulus (spatial coherence), b) the stimulus category (face or car) and c) the observer's attentional state, parameterized by spatial prioritization. Let S denote the respective stimulus variable. The following quantities were computed with respect to the different response variable features(4)where the variable R indicates either an EEG or fMRI data feature, and(5)where the variable R₁ indicates an EEG and the variable R₂ an fMRI data feature.

Intuitively, the quantity in equation (4) represents the relative distance of the observed stimulus-response joint distribution from its factorized counterpart which embeds the assumption of stimulus-response variable independence for a univariate response feature. Similarly, in equation (5) represents the same distance for a bivariate response feature, here comprising an EEG and an fMRI feature.

Analogously, with respect to the observer's behaviour, interest lies in 1) the subject's response time, and 2) the subject's categorical decision. Let B denote the respective behavioural variable, then(6)where the variable R indicates either an EEG or fMRI data feature, and(7)where the variable R₁ indicates an EEG and the variable R₂ an fMRI data feature. The expressions with respect to the stimulus or the subject's behaviour are obviously analogous. However, it has to be noted that the marginal stimulus distribution is usually determined by the experimenter and uniform, while the marginal behaviour distributions are experimentally observed, resulting in larger experimental uncertainty for the latter.

To estimate the information of a given EEG or fMRI feature or a feature combination about each of the variables of interest, the trials associated with this variable were sorted according to the respective variable categories and collapsed over all other stimulus categories. For example, to estimate with respect to stimulus informativeness, trials were grouped into low stimulus spatial coherence and high stimulus spatial coherence , the joint observed probability distribution estimated and the informativeness of the signal features with respect to assessed. An analogous procedure was carried out for the information about the stimulus category (face vs. car) and about the observer's attentional state (spatial prioritization vs. no spatial prioritization). Figure S3 depicts an empirical example of the single-trial feature distributions for EEG and fMRI.

For the behavioural state variables, a similar procedure was carried out in a slightly modified manner: first, regarding the information about response times, all trials across all conditions were considered, and the joint probability distribution estimated, where b represents the continuous random variable response time. The small number of trials on which observers did not respond within 1 s of stimulus onset (average of 10.8% per subject) were excluded from the analysis to render the estimation of the joint probability distribution less prone to outliers (the histogram grid is adjusted to include the maximum and minimum values of each response variable, hence single outliers can have strong effects on the overall response space partitioning, which is to be avoided).

Finally, with respect to the observer's decision, only low coherence trials were considered in order to decouple the distribution of the observer's decision as much as possible from the physical stimulus category, i.e. was estimated where b represents the distribution of the discrete random variable decision (face vs. car) on low spatial coherence trials. The current experimental paradigm was not optimized to study the informativeness of features with respect to the observer's perceptual state as the high accuracy of performance (see psychophysical results below) indicates that for most of the trials the physical stimulus category and the observer's perception matched. Future studies using near-threshold paradigms [42] might elucidate the informativeness of joint EEG-fMRI signal features about the observer's perceptual state in more detail. For the current study, it follows that the estimation of information about the observer's decision is more error prone compared to the other variables, as it proceeds based on half of the number of trials.

Based on the single-trial signal feature values extracted from the data, the respective probability distributions were estimated non-parametrically using a two-dimensional histogram approach with the number of bins set to , where N_c denotes the number of trials per condition [43], [44]. Entropy and mutual information values were then computed using the respective equations and bias corrected for limited sample sizes using a combination of PT-, shuffling (1000 permutations)- and Gaussian null model (1000 simulations)-correction [41]. For expressions (6) and (7) the estimation of the marginal behavioural variable distributions is required. This entails a three-dimensional histogram analysis with the number of bins set to where denotes the total number of trials evaluated. Given the non-Gaussianity of response time distributions [21] and the binary nature of the perceptual decision, the pure Gaussian null model bias correction employed for stimulus-response signal relationships was not appropriate. Hence, the respective null models for bias correction were based on sampling from three independent random variables (1000 simulations), two of which were Gaussian, representing the response signals, while the third, representing the behavioural variable, was either a Gamma distribution (response time null model) or a Bernoulli distribution (decisional variable null model).

The chosen numbers of response bins are relatively high and non-conservative, i.e. they maximize sensitivity to informative aspects in the data while decreasing specificity (i.e. increasing the risk of false-positives). Bias control procedures were employed to decrease the risk of false positives. However, the uncertainty about the absolute value of information in the current analysis is reflected in the fact that, in the following, only between-feature information comparisons are evaluated (i.e. the analysis focuses on relative information estimates) and no tests are performed for the difference of the information values from zero (i.e. the analysis does not focus on absolute information estimates).

Statistical comparisons of the estimated information quantities were carried out using one- or two-way repeated measures ANOVA models with Greenhouse-Geisser correction when appropriate, i.e. a significant result of Mauchy's test for sphericity followed by pairwise comparisons based on the estimated marginal means (least-significant differences) in SPSS (SPSS Inc, Chicago, IL).

Psychophysical results

Figure 3 depicts the psychophysical results for EEG only and simultaneous EEG-fMRI experiments. In both cases, faster median response times were observed for high informative compared to low informative and spatially prioritized compared to not spatially prioritized stimuli (Figure 3A). Equivalently, response accuracies increased with stimulus informativeness and spatial prioritization (Figure 3B). The observed behavioural pattern was identical between the EEG only and simultaneous EEG-fMRI experiment. However, the MRI scanner environment lead to an overall increase in response times and decrease in performance accuracy (mean response time across conditions: 441 (±22 (Standard Error of the Mean (SEM))) ms EEG vs. 740 (±38 SEM) EEG-fMRI, accuracy across conditions: 90 (±2 SEM) % correct EEG vs. 84 (±2 SEM) % EEG-fMRI). Possible endogenous sources for this baseline shift to longer response times and lower accuracies include the noisy scanner environment, the uncomfortable scanning position, and fatigue, as the simultaneous EEG-fMRI data acquisition always followed the EEG data collection outside the MR environment. Possible exogenous sources include the lower quality of the visual projection as well as potential signal delays due to differences in the response button set-up and fibre optic conduction. Impairment in behavioural performance in psychophysical tasks for inside the MR scanner compared to outside the MR scanner have been reported previously (see [45] for a review).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Psychophysical Results.

A. Response Times. Bars depict the average (mean) median response times across observers, error bars ± standard error of the mean (SEM) B. Response accuracy. Bars depict the average (mean) median response times across observers, error bars ± standard error of the mean (SEM). The light grey bars (EEG) represent the EEG data set recorded outside the MR scanner, the dark grey bars (EEG-fMRI) represent the EEG data set acquired simultaneously with the fMRI data.

https://doi.org/10.1371/journal.pone.0033896.g003

To quantitatively assess the reliability of the experimental manipulation on the behavioural responses, a two-way repeated measures ANOVA with factors stimulus informativeness and spatial prioritization was carried out. For response times on correct response trials, this ANOVA revealed significant main effects of stimulus informativeness (EEG: F_(1,12) = 39.6, p<0.001, EEG-fMRI: F_(1,12) = 17.4, p = 0.001) and spatial prioritization (EEG: F_(1,12) = 12.1, p = 0.005, EEG-fMRI: F_(1,12) = 30.6, p<0.001). No significant interaction was observed (EEG: F_(1,12) = 1.1, p = 0.31, EEG-fMRI: F_(1,12) = 0.1, p = 0.81). For accuracy, significant main effects of stimulus informativeness (EEG: F_(1,12) = 54.9, p<0.001, EEG-fMRI: F_(1,12) = 146.7, p = 0.001) and spatial prioritization (EEG-fMRI: F_(1,12) = 7.8, p = 0.01) were observed, but not for spatial prioritization using the data recorded outside the MR scanner (F_(1,12) = 3.0, p = 0.10). There was no significant interaction (EEG: F_(1,12) = 0.1, p = 0.70, EEG-fMRI: F_(1,12) = 2,9, p = 0.11).

These results indicate that the experimental manipulation reliably evoked differential behavioural responses, while both experimental factors appear to act on independent cognitive substrates as no significant interaction was observed.

Time course analysis of EEG data

To assess the data quality, to identify time-windows of interest, and to select electrode regions relevant for the current study, a traditional event-related potential (ERP) analysis was performed. Figure 4 depicts the grand average EEG time-courses for a set of parieto-occipital electrodes for both EEG only (Figure 4A) and combined EEG-fMRI data (Figure 4B). Given the hemifield presentation of the stimulus, these data were extracted from electrodes O2, PO4 and PO8 for left hemifield trials, from electrodes O1, PO3, and PO7 for right hemifield trials, and collapsed according to the experimental conditions. In line with similar previous studies [19], [22], no substantial potential deflections were observed after 500 ms post-stimulus, hence the focus of the analyses was on the −100 to 500 ms peri-stimulus time window.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. ERP analysis.

A and B. EEG grand average. EEG grand average time-courses for contralateral trials pooled over electrodes electrodes O2, PO4, PO8 (left hemifield trials) and O1, PO3, PO7 (right hemifield trials) for the EEG only (A) and combined EEG-fMRI (B) data acquisition. The pattern of shaded and non-shaded areas reflects the five non-overlapping time-windows used for the subsequent information theoretic analyses. C and D. EEG grand average topography. Topography plots of the entire EEG electrode set show potentials for time-points 100, 140, 260 and 440 ms post-stimulus. The main positive deflections at all time-points were observed for a set of parieto-occipital electrodes (O2, O1, PO8, PO7, P8, P7, PO4, PO3).

https://doi.org/10.1371/journal.pone.0033896.g004

With respect to the temporal expression of evoked EEG effects, for both data sets and all conditions, early (∼100 ms), and late (∼270 ms) positive potential deflections were identified (Figures 4A and 4B). These positive deflections, referred to as P100 and P300, respectively, were separated by an intermediate negative deflection (∼140 ms, N140), which was more prominent in the EEG only data. The most obvious condition-specific effects were an increase of the P100 amplitude with spatial prioritization and an increase of the P300 amplitude with a decrease of stimulus informativeness. These effects were clearly observed for both the EEG only and, with slightly diminished prominence, for the simultaneous EEG-fMRI data (Figures 4A and 4B). A clear stimulus condition specific effect on the N140 deflection was not observed.

Besides these stimulus evoked effects, we note that the EEG only data exhibited a higher degree of high frequency content than the simultaneous EEG-fMRI data. Identical filter settings were used during data pre-processing, so this temporal smoothing effect is likely due to the additional processing performed on the EEG-fMRI data (MR and BCG correction, ICA-based residual artefact removal) and the more efficient line-noise shielding of the MR scanner environment. Similar effects of artefact correction on the EEG power spectrum have previously been reported [46].

With respect to the spatial expression of evoked EEG effects, topography plots of the grand mean show that the strongest positive deflections for the identified time-points of interest were observed for posterior parieto-occipital electrodes (Figure 4, lower panels). These positive deflections were clearly visible for both the EEG only and simultaneous EEG-fMRI data. This motivated the joint selection of electrodes O1/2, PO7/8, P7/8, PO3/4, P5/6, TP7/8, P1/2 and P3/4 as spatial region of interest for the subsequent information theoretic analyses of both data sets. While the topography plots for the EEG only and EEG-fMRI data were in general similar, some differences remain. Specifically, we observed a smaller expression of the parietal dipole field at 140/162 ms and a weak leftward lateralization of the dipolar field at 440 ms for the EEG-fMRI compared to the EEG only data. These differences are likely due to residual artefacts in the EEG-fMRI data and possibly between-session effects. However, given the overall similar pattern of evoked effects for both data sets, the employed EEG-fMRI artefact correction has worked satisfactorily for the current purpose of evaluating relative information estimates for different data feature combinations.

In summary, the observed spatiotemporal pattern of peri-stimulus EEG responses motivated the information theoretic analysis of data extracted from parieto-occipital electrodes in five non-overlapping time-windows: 1) The pre- and early post-stimulus baseline, 2) the positive deflection around 100 ms (P100), 3) the negative deflection around 140 ms (N140), 4) the positive deflection around 270 ms (P300) and 5) the remaining time. These time-windows are indicated by the pattern of shaded and unshaded areas underlying the time-courses in Figure 4.

General linear model analysis of fMRI data

To identify regions of interest for the subsequent information theoretic analyses a group level GLM analysis of the fMRI data set was performed. The aim of this analysis was to explicitly identify areas previously implicated in perceptual decision processes [17], [27], [47].

Figure 5 and Table 1 depict the results of the group-level GLM analysis of the fMRI data. Of all possible main effect contrasts, the main effect of prioritization was omitted as no significant activation was detected for this contrast at the group level. This was potentially due to the fact that the task demand (attention allocation) was high for both the prioritized and non-prioritized conditions. For the contrasts of left vs. right and right vs. left hemifield stimulus presentation, lateralized activity was detected in the occipital cortex, while higher level cortices did not display lateralized activity. This provides some validation for the use of single hemispheric signal features for occipital (striate cortex, extra-striate cortex, lateral occipital sulcus) regions of interest in the information theoretic analyses reported below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Group-level fMRI GLM results.

The respective statistical parametric maps (thresholded at p<0.001 (uncorrected, voxel level), extent threshold 15–20 voxels) are overlaid on the group MNI template.

https://doi.org/10.1371/journal.pone.0033896.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Group-level fMRI results.

https://doi.org/10.1371/journal.pone.0033896.t001

The set of regions identified as significantly activated (p<0.001 (cluster level, corrected)) for the high coherence vs. low coherence and low coherence vs. high coherence included superior frontal sulcus, pre-central sulcus, anterior cingulate gyrus, insula and frontal-eye fields, all of which have previously been implicated in the processing of visual perceptual decisions [16]. Additionally, given the known role of the intra-parietal sulcus in cognitive tasks [48], [49], the most active voxels in this region for the low vs. high coherence contrast were also identified and selected, although they were not significantly activated at the voxel level even at p<0.001 (cluster level, corrected). Finally, as observers performed a face vs. car categorization task, face responsive cortex of the fusiform gyrus (fusiform face area (FFA), [50]) was identified using the face vs. car stimulus contrast.

While not all of these regions reached family-wise error corrected statistical significance (Table 1) at the cluster-level, their previous implication in the perceptual decision process motivated their selection. As the motivation of the GLM analysis was to determine ROIs for the IT analysis, a relatively liberal threshold was used to avoid missing informative voxels.

EEG-fMRI integrated information theoretic analysis

Temporal information representation.

Figure 6 displays the information about the external, internal and behavioural state variables for the EEG features of interest (). The columns of the figure represent the different stimulus and behavioural variables of interest (external, internal and behavioural state). For each stimulus/behavioural variable of interest (S/B), the average information estimate across subjects ± SEM is depicted for each of the five time windows of interest, in light grey for EEG only and in dark grey for the EEG-fMRI data sets. Overall, it can be observed that the information estimates for both EEG only and EEG-fMRI data sets, while being lower for the EEG data acquired simultaneously with fMRI data, show similar patterns. The lower information estimates for the EEG data acquired inside the MR scanner reflect its lower single-trial SNR.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Temporal information representation.

The data are ordered columnwise according to the variables of interest, external, internal and behavioural state. For each variable of interest, data from the five time-windows identified based on the grand average are displayed. The light bars represent information estimates from the EEG data set acquired outside the MR environment, while the dark gray bars represent information estimates from the simultaneous EEG-fMRI data recordings. All bars reflect group averages (n = 13) and the error bars indicate the standard error of the mean.

https://doi.org/10.1371/journal.pone.0033896.g006

For both data sets, the earliest time-window around stimulus onset yielded the lowest information estimates across features for most stimulus/behavioural variables, in agreement with the fact that information about the stimulus can only be extracted upon stimulus-presentation. With respect to the external state, the information about the stimulus informativeness increased from 140 ms onwards to reach its maximum in the final time-window. This effect was present in both data sets, although slightly diminished and delayed for the data acquired in the MR environment. For the EEG data acquired outside the MR scanner, a trend for a significant effect of the time-window on the information estimate was observed (F_(4,48) = 2.24, p = 0.07). The information estimate in the 450 ms time-window was marginally different from the information estimate in the 0 ms time-window (p = 0.06) and significantly different from the information estimate in the 100 ms time-window (p = 0.04). Similarly, for the EEG data acquired simultaneously with the fMRI data, a marginally significant main effect of time-window was observed (F_(4,48) = 2.42, p = 0.06), while the comparisons between the fifth and the first, second and third time windows approached statistical significance, or were statistical significant (p = 0.08, p = 0.03, and p = 0.02, respectively).

These results suggest that information about the stimulus informativeness, or equivalently, the difficulty of the decision, was represented rather late in the EEG response. This is in concordance with the observed grand average effect of stimulus informativeness depicted in Figure 4 and previous studies on the task-difficulty and reaction time sensitivity of the P300 deflection [27], [34]. It should be noted that for the current experimental design and analysis strategy, task difficulty, reaction time and P300 amplitude co-vary, and the activity dependent information theoretic analysis discussed above cannot dissociate these three concepts.

The other external stimulus attribute that was manipulated in the experimental paradigm was the stimulus category. Category-selective responses for faces compared to other stimuli have been described previously [19], [36]. In the current study, for both EEG data sets, the largest estimate for represented information about the stimulus-category was observed in the 260 ms time-window, i.e. in the interval of 150 to 370 ms post-stimulus onset. This is in concordance with the maximal discriminative time-windows for a similar stimulus set identified by [19]. This finding is substantiated by a significant effect of time-window on the information estimates for the EEG acquired outside the MR scanner (F_(4,48) = 6.01, p = 0.001) and statistically significant differences for the fourth time window in comparison to all others (p<0.05), except the fifth (p = 0.08). For the EEG data acquired inside the MR scanner, the main effect of time-window was not significant (F_(4,48) = 1.53, p = 0.20). However, the information estimate for the fourth time-window showed marginal statistically significant differences from that in the first and second windows (p = 0.05 and p = 0.06, respectively). The most information about the category of the stimulus was thus observed in a time-window 150–370 ms post-stimulus and declining thereafter.

In comparison to the information about the stimulus, the information estimates about the internal state, i.e. the spatial prioritization of the stimulus, were expressed earlier, with effects from 100 ms post-stimulus onwards. This is in line with the well-known attentional modulation effect on the P100 [35]. For the current data sets, the largest information about the attentional state of the observer was observed in the early time-windows of 100 and 140 ms and then, after a decrease in the fourth time-window, again in the last time-window. For the data recorded outside of the scanner, no statistically significant main effect of time-window was observed, consistent with the observation that the information estimates were similar for the time-windows from 100 ms on. The pairwise comparison between the fifth and the first time-window was marginally significant (p = 0.05), while the others were not. Again, for the EEG data acquired inside the MR scanner no significant main effect of time-window was observed, while the pairwise comparison between the fifth and the first time-window was the most reliable (p = 0.16).

Finally, with respect to behavioural state, for the response time the information estimates increased with time. This effect was significant for the EEG only dataset (F_(4,48) = 5.9, p = 0.001), but not for the EEG data acquired simultaneously with the fMRI data (F_(4,48) = 1.6, p = 0.18). This pattern is reminiscent of that observed for the information about stimulus informativeness, which is to be expected given the longer response times for low informative trials. Unfortunately, these two processes cannot be dissociated in the current paradigm. For the observer's decision, all EEG time-windows appear equally informative. Consistent with this observation one-way ANOVAs for both behavioural variables and EEG data sets indicated no statistically significant effects for both data sets (EEG only: F_(4,48) = 0.40, p = 0.80), EEG-fMRI: F_(4,48) = 0.04, p = 0.99). As the estimated information values for later time windows did not appear particularly different from those at the earliest time-point, it may be that the number of trials was too low or the electrode set chosen not appropriate, to detect a decisional effect on the basis of the marginal EEG data distributions. A possible reason for the observed larger information estimates for the EEG-fMRI data set is of methodological nature: while we aimed to reduce the number of outliers (see Methods), by design, the EEG-fMRI data set is more outlier prone with respect to behavioural variables. For our analysis this has the immediate effect that the partition of the data feature space is coarser than if all data feature combinations cluster in a smaller region of data feature space. However, coarser histogram sampling is known to entail larger estimation biases [51]. Future work on efficient outlier control, e.g. using the diffusion model framework [52], [53] may help to obtain better information estimates about the behavioural state.

In summary, the following picture of temporal representation of information for visual perceptual decisions emerges: in concordance with previous studies, information about the state of the stimulus is represented in the EEG response later than that about the subject's attentional state. With respect to behaviour, later time windows represent more information about the response time, in agreement with their involvement in the representation of uncertainty about the stimulus. The same patterns of results was observed for both data sets, although they were more reliable for the data recorded outside of the scanner. However, it is apparent that the observed single-trial information differences appear larger than those observed on the signals' grand averages. This motivates the future evaluation of novel methodologies for the improvement of EEG quality in combined EEG-fMRI recordings on the single-trial level [54].

Spatial information representation.

Figure 7 displays the information about the external, internal and behavioural state variables for each fMRI region of interest (). The columns of the figure represent the different stimulus and behavioural variables of interest (external, internal and behavioural state, S/B). For each stimulus/behavioural variable of interest, the average information estimate across subjects ± SEM is depicted for each of the regions of interest identified based on the group fMRI-GLM analysis. The regions of interest of each panel are ordered according to an approximate occipital - frontal (or “early sensory - higher cognition”) gradient from left to right, in line with Figure 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. Spatial information representation.

The data are ordered columnwise according to the variables of interest, external, internal and behavioural state. For each variable of interest, information estimates from all regions of interest identified based on the GLM group analysis are displayed. All bars reflect group averages (n = 13) and the error bars indicate the SEM. (V1: Striate Cortex, V2: Extrastriate Cortex, LO: Lateral Occipital Complex, CU: Cuneus, FF: Fusiform Gyrus, PL: Left Intra-Parietal Sulcus, PR: Right Intra-Parietal Sulcus, PC: Left Post-Central Gyrus, AC: Right Anterior Cingulate Cortex, IN: Right Insula, FL: Left Frontal Eye Field, FR: Right Frontal Eye Field, SF: Left Superior Frontal Gyrus).

https://doi.org/10.1371/journal.pone.0033896.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 8. Spatiotemporal information representation.

The data are ordered columnwise according to the variables of interest, external, internal and behavioural state. For each variable of interest, information estimates from all joint distributions of all features of interest combinations, i.e. EEG time-windows × fMRI regions of interest are displayed. The tiles of the surface reflect the group averages. (V1: Striate Cortex, V2: Extrastriate Cortex, LO: Lateral Occipital Complex, CU: Cuneus, FF: Fusiform Gyrus, PL: Left Intra-Parietal Sulcus, PR: Right Intra-Parietal Sulcus, PC: Left Post-Central Gyrus, AC: Right Anterior Cingulate Cortex, IN: Right Insula, FL: Left Frontal Eye Field, FR: Right Frontal Eye Field, SF: Left Superior Frontal Gyrus).

https://doi.org/10.1371/journal.pone.0033896.g008

Overall, the information of relevance for the perceptual decision task employed in this study was spatially distributed across the cortex. Both anterior (higher) and posterior (lower) cortical areas were implicated in the representation of stimulus related information, while information about the internal state was slightly more strongly represented in posterior brain areas.

The areas primarily implicated in the representation of information about the stimulus informativeness or task difficulty comprise a network of frontal (insula, left frontal eye field, pre-central), parietal (left intraparietal sulcus) and occipital-temporal (extrastriate and fusiform) cortex. The most information about the stimulus informativeness was represented in insular cortex, with an estimate approaching a significant difference with respect to the cuneus and lateral-occipital cortex (p = 0.07, p = 0.04, respectively). Insular cortex has been implicated in perceptual decision making previously (e.g. [55]). Moreover, due to the involvement of insular cortex in a wide variety of cognitive processes (e.g. interoception, self-recognition, emotional awareness, time perception, attention, cognitive control and performance monitoring), it has been proposed that insular cortex plays a pivotal role in the neurobiological representation of awareness [56]. In the current study insular cortex appeared to be involved in the representation of task difficulty at the single-trial level, which could potentially be reconciled with this view, in the sense that insular cortex activity differentiates different states of the stimulus-dependent awareness induction.

Nevertheless, the distributed nature of the represented information is substantiated by the absence of an overall main effect of region of interest on the information about stimulus informativeness (F_{(12, 144)} = 0.78, p = 0.67). It should be noted that most areas were selected according to the contrast of high vs. low and low vs. high stimulus coherence, i.e. on the basis of being informative about stimulus coherence in the sense of a GLM contrast.

Categorical information about the stimulus was mainly represented in a network of frontal (superior frontal gyrus, frontal eye fields, pre-central) and parietal (intra-parietal sulcus (IPS)) regions, with some contribution from the fusiform gyrus. This finding is in concordance with the known roles of the IPS and superior frontal gyrus or dorso-lateral prefrontal cortex (DLPFC) in perceptual decisions [16], [17], [24]. While again the information appears to be distributed across regions of interest (F_{(12, 144)} = 1.01, p = 0.43), the pairwise comparison of the information estimate for the right IPS showed significantly higher information estimates in comparison to extrastriate visual (p = 0.03) and anterior cingulate cortex (p = 0.04).

A pair of occipito-parietal regions encompassing the cuneus and right parietal sulcus was most informative about the observer's state, i.e. the spatial prioritization of the stimulus. For the right intra-parietal sulcus, a post-hoc pairwise comparison of the information values across subjects reached marginal significance with respect to the right intra-parietal (p = 0.04), right frontal eye field (p = 0.04) and superior frontal gyrus (p = 0.06). This result is in line with previous studies using fMRI to study spatial attention [49], [57]–[60]. The involvement of a dorsal frontoparietal network of regions implicated in spatial attention in these studies was substantiated by the absence of a significant main effect of region of interest on information estimate (F_{(5.4, 64.7)} = 0.74, p = 0.60).

For the behaviour related information, no significant main effects of region of interest, or pairwise comparisons between regions were observed for response time (F_{(5.0, 60.5)} = 0.24, p = 0.94). The widespread network of regions implicated as being informative regarding response time is consistent with GLM-based fMRI studies [61]. With respect to the subject's decisional variable, the largest information values were observed for the superior frontal and the pre-central gyrus. This implicates a shift of stimulus categorical information towards more frontal regions in comparison to the representation of physical stimulus category discussed above. The pairwise comparison of the superior frontal gyrus with the insular cortex, right intra-parietal, fusiform gyrus and cuneus reached statistical significance (p<0.05) while no overall main effect of region of interest was observed (F_{(4.5, 53.7)} = 1.0, p = 0.38). While it is tempting to speculate that this indicates a more high level cognitive (rather than low level perceptual) determination of the decision on low coherence trials, it should be noted that physical and perceptual stimulus attributes were not completely dissociated in the current study.

In summary, the following picture of spatial representation of information for visual perceptual decisions emerges: with respect to the stimulus, information about differences in stimulus informativeness appears to be represented in a distributed manner throughout the cortical regions studied, while information about the (physical) stimulus category showed a maximum in parietal cortices. Occipito-parietal areas were implicated in the representation of information about the observer's attentional state, while no clear pattern emerged with respect to the speed of the response. On low coherence trials, the superior frontal gyrus was most informative about the observer's decision.