Abstract
The ability to distinguish between commonplace and unusual sensory events is critical for efficient learning and adaptive behaviour. This has been investigated using oddball designs in which sequences of often-appearing (i.e. expected) stimuli are interspersed with rare (i.e. surprising) deviants. Resulting differences in electrophysiological responses following surprising compared to expected stimuli are known as visual mismatch responses (VMRs). VMRs are thought to index co-occurring contributions of stimulus repetition effects, expectation suppression (that occurs when one’s expectations are fulfilled), and expectation violation (i.e. surprise) responses; however, these different effects have been conflated in existing oddball designs. To better isolate and quantify effects of expectation suppression and surprise, we adapted an oddball design based on Fast Periodic Visual Stimulation (FPVS) that controls for stimulus repetition effects. We recorded electroencephalography (EEG) while participants (N=48) viewed stimulation sequences in which a single face identity was periodically presented at 6 Hz. Critically, one of two different face identities (termed oddballs) appeared as every 7th image throughout the sequence. The presentation probabilities of each oddball image within a sequence varied between 10-90%, such that participants could form expectations about which oddball face identity was more likely to appear within each sequence. We also included ‘expectation neutral’ 50% probability sequences, whereby consistently biased expectations would not be formed for either oddball face identity. We found that VMRs indexed surprise responses, and effects of expectation suppression were absent. That is, ERPs were more negative-going at occipitoparietal electrodes for surprising compared to neutral oddballs, but did not differ between expected and neutral oddballs. Surprising oddball-evoked ERPs were also highly similar across the 10-40% appearance probability conditions. Our findings indicate that VMRs which are not accounted for by repetition effects are best described as an all-or-none surprise response, rather than a minimisation of prediction error responses associated with expectation suppression.
Highlights
- We used a recently-developed oddball design that controls for repetition effects
- We found effects of surprise but not expectation suppression on ERPs
- Surprise responses did not vary by stimulus appearance probability
1. Introduction
We are highly adept at learning recurring patterns and statistical regularities that occur in our sensory environments, and we can exploit this knowledge to discriminate between commonplace and unusual sensory events. The computations that underpin this capability likely rely on statistics learned over time-scales ranging from hundreds of milliseconds to several minutes (Ulanovsky et al., 2004; Maheu et al., 2019). Characterising the computations that occur within different brain regions can help us understand how we generate and evaluate internal models of our environment, and how these models are used to facilitate detection of potential rewards (Schultz, 2016) and enact adaptive adjustments to decision-making strategies (Wessel, 2017; Wessel & Huber, 2019).
Detection of novel or unusual events has been investigated using visual oddball designs in which a stimulus is presented in high and low probability contexts. For example, in the high probability context, a critical stimulus A is presented frequently as the ‘standard’ stimulus, interspersed with a rare ‘deviant’ stimulus B (e.g., AABAAAAAAABAAAB). In the low probability context, the standard stimulus A is instead presented as a rare deviant (e.g., BBBBBABBBBBABBB). Comparisons of visual stimulus-evoked event-related potentials (ERPs) recorded using electroencephalography (EEG) reveal more negative-going waveforms evoked by deviants at posterior scalp electrodes from ~150-300 ms following stimulus onset (Czigler et al., 2004; Kimura et al., 2009; Stefanics et al., 2011). This difference in ERP waveforms is known as a visual mismatch response (VMR), or the visual mismatch negativity (vMMN; for recent reviews see Kimura, et al., 2011; Stefanics et al., 2014; Kremlacek et al., 2016). The vMMN is considered to be the visual counterpart of the earlier-discovered auditory MMN (Naatanen et al., 1978). VMRs are broadly theorised to reflect the tracking of patterns and regularities within sensory environments as implemented within the architecture of the visual system (e.g., Kimura et al., 2009).
Multiple phenomena can contribute to VMRs as they are measured using ERPs, each of which relates to different statistical regularities that are present in oddball sequences. The first class of phenomena relates to recent stimulation history, that is, which stimuli have recently been presented to an observer. Effects of immediate stimulus repetition, known as repetition suppression or stimulus-specific adaptation (Desimone, 1996; Movshon & Lennie, 1979), are defined as a reduction in a measure of neural activity (e.g., firing rates, local field potential amplitudes or BOLD signals) following repeated as compared to unrepeated stimuli (Grill-Spector, Henson & Martin, 2006). Repetition effects can partly account for VMRs as a suppression of neural responses evoked by standard stimuli and a lack of such suppression for the deviant stimuli (May & Tiitinen, 2010; Nelken & Ulanovsky, 2007). More recently developed excitatory-inhibitory circuit models also describe enhancement of responses to deviant stimuli that occur when both excitatory and lateral inhibitory responses are reduced following repeated exposure to standards (Dhruv, et al., 2011; Solomon & Kohn, 2014; Kaliukhovich & Vogels, 2016), where effects of disinhibition can occur over later time windows during the stimulus-evoked response than suppressive effects of stimulus repetition (Patterson et al., 2013). There are also systematic effects that depend on the combinations of stimuli that were recently presented to an observer (e.g., the most recently seen 2-5 stimuli, Sawamura et al., 2006; Vinken & Vogels, 2017; for similar effects on auditory evoked responses see Ulanovsky et al., 2004; Maheu et al., 2019).
The second class of phenomena relates to internally generated models of sensory environments that are learned through exposure to regularities in oddball sequences. In oddball sequences, expectations derived from these models are likely to be biased toward the often-appearing standard rather than the rare deviant. Such expectations are fulfilled when the standard is presented, and violated when a deviant appears in the expected standard’s place (Friston, 2005; Stefanics et al., 2014, 2018). There is ample evidence from non-oddball designs that an observer s expectations can influence responses of stimulus-selective visual neurons (Summerfield et al., 2008; Amado et al., 2016; Hall et al., 2018; Feuerriegel et al., 2018a), and that humans can generate neural representations of expected sensory events even before the corresponding stimulus has been presented (Kok et al., 2017; Blom et al., 2020).
Within this prediction-based framework, there are two ways in which an observer’s learned expectations might relate to the standard/deviant ERP differences indexed by VMRs. The first account describes VMRs as a marker of prediction error, indexing the degree of mismatch between the predictions of an observer’s internal model and the actual sensory input (Friston et al., 2005; Garrido et al., 2009; Stefanics et al., 2014, 2018). VMRs are conceptualised as a combination of suppressed neural responses for subjectively expected standard stimuli and enhanced responses to surprising deviant stimuli. Variants of this model also incorporate the notion of precision (Feldman & Friston, 2010), whereby larger differences in the proportions of standard/deviant stimuli (and stronger weighted expectations to see standards) lead to larger prediction errors for deviant stimuli, and larger VMRs.
An alternative view is that VMRs index a surprise response that occurs when expectations have been violated, but not concurrent suppression when expectations are fulfilled. This account is equivalent to models which describe mismatch responses that occur following violations of regularities and recurring patterns in oddball sequences (e.g., Winkler, 2007; Paavilainen et al., 2013). Similar surprise responses also follow novel, unexpected or potentially rewarding stimuli in reward-learning paradigms, and precede the computation of reward prediction errors (reviewed in Schultz, 2016).
It has been remarkably difficult to isolate and quantify effects of immediate stimulus repetition, fulfilled expectations, and surprise using visual oddball sequences. This is because in such designs, the expected stimulus (the standard) is almost always also a repeated stimulus, whereas the surprising deviants are seldom repeated. In addition, repetition and expectation effects are likely to interact, whereby differences between expected and surprising stimuli are absent or diminished for immediately repeated stimuli (e.g., Todorovic & de Lange, 2012; Wacongne et al., 2011; Feuerriegel et al., 2018b). Recently-developed designs have attempted to overcome these limitations by adding sequences in which many different stimulus images (including the deviants and standards) are randomly interspersed within a sequence, known as equiprobable control sequences (e.g., Jacobsen & Schroger, 2001; Amado & Kovacs, 2016; reviewed in Stefanics et al., 2014). In equiprobable sequences, each stimulus image is presented with the same probability as the deviant stimuli in the classical oddball sequences. Expectations cannot be reliably formed for any particular stimulus image in the sequence, so the stimuli are neither expected nor surprising. ERPs evoked by the same stimulus image are then compared across equiprobable and deviant contexts to isolate surprise effects while apparently controlling for immediate repetition effects.
These studies have reported separable contributions of repetition suppression and surprise to VMRs (Amado & Kovacs, 2016; Astikainen, et al., 2008; Czigler et al., 2002; Kimura et al., 2009). However, these designs do not adequately isolate effects of fulfilled expectations from repetition effects, as the expected standard stimuli are repeated frequently whereas the equiprobable control stimuli are not. There are also two additional potential confounds to consider when using this design. Massed repetition of standards is also likely to enhance responses to deviants through reductions in lateral inhibition (Dhruv et al., 2011; Kaliukhovich & Vogels, 2016), and this response enhancement would not systematically occur in equiprobable sequences. The number of stimulus identities is also different across classical oddball sequences (2 identities) and equiprobable sequences (typically ~10 identities). This can produce systematic effects on visual evoked ERPs during the same time range as VMRs (Feuerriegel et al., 2018a), which have been localised to ventral temporal areas of visual cortex (Pajani et al., 2017; Rostalski et al., 2020). It is unclear to what extent these confounds have contributed to existing reports of VMRs identified using equiprobable sequences. This has made it extremely difficult to cleanly isolate and quantify each repetition and expectation effect, which is critical for constraining theoretical models of VMRs.
To overcome these limitations, we recently developed an oddball design based on Fast Periodic Visual Stimulation (FPVS; Dzhelyova & Rossion, 2014a, 2014b; Dzhelyova, et al., 2017; Liu-Shuang, et al., 2014, 2016) that allowed us to isolate and quantify [surprising – expected] ERP differences while controlling for immediate stimulus repetition effects (Feuerriegel et al., 2018b). In FVPS oddball designs, a base stimulus is presented at a rapid, periodic rate (e.g., 6 Hz). Oddball stimuli replace the base stimulus every N stimuli at a fixed periodicity (e.g., every 7 stimuli). Viewers exposed to these stimulation sequences can perfectly predict when an oddball stimulus will appear (e.g., after 6 base rate stimuli have been presented). In sequences in which there are two possible oddball stimulus images, with one image appearing more often than the other, observers can also form expectations about the specific identity of each upcoming oddball stimulus. This leads to oddball stimulus-evoked VMRs that appear with similar topographies and latencies to VMRs found when using equiprobable sequences (Feuerriegel et al., 2018b). As six base face images are presented between each oddball face, the visual system is strongly adapted to the base face image before the presentation of each oddball stimulus in the sequences. This means that the state of adaptation in the visual system is equivalent for expected and surprising oddballs, thereby controlling for effects of immediate repetition and recent stimulation history.
In our previous work we identified [surprising – expected] response differences that were independent of stimulus repetition effects, however we could not isolate effects of surprise from those of fulfilled expectations. In the current study, we presented sets of FPVS oddball sequences and systematically varied the relative proportions of different oddball stimulus images in each sequence type, ranging from 10% to 90%. Critically, this included a 50% ‘expectation neutral’ condition, in which participants would not form strong expectations about which specific oddball stimulus was most likely to appear, as each oddball image appeared with equal probability. To isolate and quantify effects of fulfilled expectations and surprise, we compared ERPs evoked by expected (60-90% probability) and surprising (10-40% probability) oddballs with those evoked by neutral (50% probability) oddballs. We could also test whether neural responses indexed by VMRs varied in accordance with the degree of expectedness or surprise associated with seeing a given oddball stimulus (i.e. the objective oddball stimulus presentation probability).
We expected to find ERP differences for surprising compared to neutral conditions, consistent with surprise effects found in studies using equiprobable sequences (e.g., Kimura et al., 2009; Amado et al., 2016). We did not make explicit predictions regarding the magnitude of expectation suppression effects; however, we expected these to be of smaller magnitude than surprise effects, based on existing work using non-oddball designs (e.g., Amado et al., 2016). We also hypothesised that we would find linearly more negative-going ERP amplitudes for oddballs with lower presentation probabilities (i.e. more negative amplitudes for oddballs which were less probable and more surprising), consistent with the notion of precision-weighted prediction error signalling (Stefanics et al., 2014) and previous observations of graded effects on N2 and P3 ERP component amplitudes in oddball designs (e.g., Duncan-Johnson & Donchin, 1977; Polich & Margala, 1997; Wessel & Huber, 2019).
By controlling for immediate repetition effects, we could also test for adaptation effects that might occur over longer timescales, which may be important to consider in future experimental and computational modelling work. We assessed whether oddball-evoked neural response amplitudes gradually changed over the course of a stimulation sequence, as previously reported when using both classical auditory oddball (Ulanovsky et al., 2004) and FPVS oddball designs (Feuerriegel et al., 2018b).
2. Method
2.1. Participants
Forty-eight people participated in this experiment. Participants were fluent in spoken and written English and had normal or corrected-to-normal vision. Five participants were excluded due to excessively noisy EEG data (i.e. more than 10 electrodes with a high number of large-amplitude artefacts). Forty-three participants were included in the final sample (20 males, age range 18-38 years, mean age 24.4 years, two left-handed). Participants were reimbursed 35 AUD for their participation in the experiment. The study was approved by the Human Research Ethics Committee of the Melbourne School of Psychological Sciences (Ethics ID 1953637).
2.2. Code and Data Availability
Code and data used for running the experiment and performing the analyses described here will be made available at https://osf.io/x8wfv/ at the time of publication.
2.3. Stimuli
Fifteen frontal images of faces (six male, nine female, neutral expression) were taken from the stimulus set in Laguesse and Rossion (2013). We converted the images to greyscale and equated their mean pixel intensity and RMS contrast using the SHINE toolbox (Willenbockel et al., 2010). Stimuli subtended approximately 5.4° × 7.9° visual angle at a viewing distance of 80 cm. We also scaled the resulting face images by 120% to create a separate set of images, which were used as oddball stimuli.
Faces are an ideal stimulus type for use in oddball designs, as they are associated with robust face identity repetition and expectation effects (Summerfield et al., 2008; Grotheer & Kovacs, 2015; Henson, 2016; Feuerriegel et al., 2018b). Moreover, in FPVS oddball designs, faces elicit a complex EEG response to changes of identity which has been well characterised in previous work (Dzhelyova & Rossion, 2014a, 2014b).
2.4. Procedure
Participants sat in a dark room 80 cm in front of a gamma-corrected 24-inch LCD monitor (refresh rate 60 Hz). Stimuli were presented against a grey background matched to the mean pixel intensity of the images. A blue fixation cross was superimposed over the nasion of the face images throughout each sequence. Stimuli were presented using Psychtoolbox (Brainard, 1997; Kleiner et al., 2007) and MATLAB (The Mathworks, Natick MA, USA).
Each stimulation sequence began with the presentation of a fixation cross for two seconds, followed by an 86-second stimulation period, and then another two-second period during which only the fixation cross was visible. During the stimulation period, we presented face images at a rate of 6 Hz using square wave contrast modulation with a 50% duty cycle. At the beginning of each 166.66 ms image cycle, a face image appeared on the screen for 83.33 ms and was then replaced by the grey background for 83.33 ms. During the first two seconds of the stimulation period, the stimulus contrast gradually increased from 0 to 100% (i.e. a fade-in), and gradually decreased to 0% across the last two seconds of the stimulation period (i.e. a fade-out). Between each stimulation sequence participants could take a self-paced break (minimum possible break duration was 10 seconds).
In each sequence, a single identity base face appeared at a rate of 6 Hz (see Figure 1). Oddball faces of the same gender replaced the base face image for every 7th image in the sequence. Oddball faces were always a different identity to the base face and 20% larger in size. Within each sequence, the oddball stimulus could be one of two face identities. The relative presentation proportions of each oddball face identity were manipulated across five sequence types, with expected/surprising oddball appearance probabilities of 90%/10%, 80%/20%, 70%/30%, 60%/40% and 50%/50%. For example, in the 90%/10% sequences one face identity would appear as 90% of oddballs, and the other face identity would be presented as 10% of oddballs. In these sequences, we anticipated that participants would form expectations to see the more common oddball face identity. By contrast, in the 50%/50% sequences, we did not expect participants to form expectations for a given oddball face, as both identities appeared with equal probability. We termed oddball images that were more than 50% to appear as expected (60-90% conditions), those less than 50% likely to appear as surprising (10-40% conditions), and oddballs with 50% appearance probability as neutral.
To aid learning of the relative oddball probabilities, sequences were blocked by sequence type. Blocks of 6 consecutive sequences were presented for all sequence types except the 90/10 sequence type, which comprised a block of 10 sequences to ensure an adequate number of 10% probability oddballs per participant. To avoid abrupt transitions in the extent of the common/rare oddball probability splits across sequence types (e.g., a transition from a 90%/10% sequence to a 50%/50% sequence) half of the participants were first presented with a block of the 90%/10% sequence type, followed by blocks of 80%/20%, 70%/30%, 60%/40% and then 50%/50% sequence types. Block order was reversed for the other half of participants, starting with 50%/50% sequences and ending with 90%/10% sequences. Different sets of faces (one base face identity and two oddball identities) were presented as stimuli for each of the five sequence types. Different faces were used across sequence types so that participants’ expectations to see a specific face identity as the oddball were not transferred across successively presented sequence types. The sets of base/oddball faces allocated to each sequence type were counterbalanced across participants.
A total of 34 sequences of 90 seconds in duration were presented during the experiment. Total testing duration was 51 minutes excluding EEG equipment setup and self-paced breaks.
2.5. Experiment Task
We used a task that was orthogonal to the face stimuli in order to engage participants’ attention throughout the experiment (as done by Rossion & Boremanse, 2011; Liu-Shuang et al., 2014). Participants were instructed to fixate on a blue cross overlaying the face images and press a key on a TESORO Tizona numpad (1,000 Hz polling rate) when the cross changed colour from blue to red (ten colour changes per sequence, 100 ms colour change duration, minimum of two seconds between colour changes). We considered key presses within 1,000 ms of a colour change as correct responses. Numbers of correctly identified targets and mean response times were presented as feedback at the end of each sequence.
2.6. Task Performance Analyses
We calculated mean proportion correct scores and mean response times for correctly detected targets separately for each sequence type. We tested for differences in response times across sequence types using a one-way repeated measures ANOVA with the factor of sequence type (90/10, 80/20, 70/30, 60/40 and 50/50) with Greenhouse-Geisser corrections to account for violations of the sphericity assumption.
2.7. EEG Acquisition and Data Processing
We recorded EEG from 64 active scalp electrodes using a Biosemi Active Two system (Biosemi, the Netherlands). Recordings were grounded using common mode sense and driven right leg electrodes (http://www.biosemi.com/faq/cms&drl.htm). We added six channels to the standard montage: two electrodes placed 1cm from the outer canthi of each eye, two electrodes placed above and below the left eye, and two electrodes over the left and right mastoid processes. EEG was sampled at 512 Hz (DC-coupled with an anti-aliasing filter, −3 dB at 102 Hz). Electrode offsets were kept within ± 50 μV.
We processed EEG data using EEGLab v13.4.4b (Delorme & Makeig, 2004). We identified excessively noisy channels by visual inspection (mean noisy channels per participant was 1.6, median = 1, range 0-5) and we excluded these from average reference calculation and independent components analysis (ICA). We also marked and removed time windows of data which contained large amplitude artefacts. We then re-referenced the data to the average of all channels and removed one extra channel (AFz) to correct for the rank deficiency caused by the average reference (as done by Feuerriegel et al., 2018a). The resulting data were low-pass filtered at 30 Hz (EEGLab Basic Finite Impulse Response Filter New, non-causal zero-phase, −6 dB cutoff frequency 33.75 Hz, transition bandwidth 7.5 Hz). We processed a copy of this dataset in the same way, but additionally applied a 0.1 Hz high-pass filter (same filter type as above, −6 dB cutoff frequency 0.05 Hz, transition bandwidth 0.1 Hz) to improve stationarity for the ICA. ICA was performed on this 0.1 Hz high-pass filtered dataset (RunICA extended algorithm, Jung et al., 2000). We then copied the independent component information to the dataset that had not been high-pass filtered. We identified and removed independent components generated by blinks and saccades according to guidelines in Chaumon et al (2015). After subtracting these components, we high-pass filtered the data at 0.1 Hz and interpolated any bad channels and AFz using the cleaned data (spherical spline interpolation).
The resulting EEG data were segmented from −166 ms to 1,000 ms relative to the onset of each oddball face, and baseline-corrected using the average −166 to 0 ms relative to oddball onset. This baseline window captured the period of one complete stimulus on/off cycle. Epochs containing amplitudes at any scalp electrode that exceeded ±100 μV were removed. Mean numbers of retained epochs per participant ranged from 67 (for 10% probability oddballs) to 606 (for 90% oddballs). Summary statistics related to numbers of epochs per condition are displayed in Supplementary Table S1.
2.8. Identification of VMR Regions of Interest
We first identified electrodes and time windows that would comprise our regions of interest (ROIs). To do this, we averaged oddball-evoked ERPs across epochs within each participant and each stimulus appearance probability condition. We then averaged the resulting ERPs across surprising (10-40% probability) and expected (60-90% probability) oddball types separately. We compared ERPs following expected and surprising oddballs at each time point for each of the 64 scalp electrodes using paired-samples t tests (alpha level 0.05). Subsequent analyses were based on the average ERP amplitudes across channels and time points within each identified ROI.
2.9. ERP Mean Amplitude Analyses
To separately quantify effects of expectation suppression and surprise, we used frequentist and Bayesian paired-samples t-tests (two-tailed) using JASP v0.9.1 (JASP Core Team) to estimate ERP amplitude differences between expected and surprising oddballs compared to neutral (50% probability) oddballs. Bayesian t-tests used a Cauchy distribution with width 0.75 (default settings in JASP). We then compared amplitudes for each individual oddball appearance probability condition to the neutral condition using paired-samples t tests.
As we had previously found linear changes in ERP amplitudes over the course of long-duration (e.g., 90-second) FPVS stimulation sequences (Feuerriegel et al., 2018b), we tested for linear relationships between oddball position in the sequence (ranging from the first oddball to the 70th oddball) and ROI-averaged mean amplitudes. We then performed frequentist and Bayesian one-sample t tests to assess whether the resulting beta values (i.e. the slopes) corresponding to the effect of oddball position differed from zero. In these analyses, beta values larger than zero indicate more positive-going ERP amplitudes for oddballs presented later in a sequence. As we identified systematic effects of oddball position, this predictor was included in the regression analyses described below.
We additionally tested whether VMR amplitudes vary by objective oddball presentation probability, to assess whether VMRs signal information about the degree of expectedness or surprise associated with seeing a particular stimulus. We tested for linear relationships between stimulus appearance probability and ROI-averaged ERP amplitudes for surprising (10-40% probability) oddballs. This was not done for expected oddballs, as there was little evidence of expectation suppression effects in the data. To do this, we fit linear regression models to patterns of single-epoch ROI-averaged amplitudes separately for each participant dataset, using predictors of stimulus appearance probability (ranging from 10-40%) and oddball position in the sequence (ranging from 1-70). We then tested whether beta values differed from zero at the group level using frequentist and Bayesian single-sample t tests as described above. In these analyses, beta values larger than zero indicate more positive-going amplitudes for oddballs with higher presentation probabilities.
3. Results
3.1. Task Performance
Target detection rates for the fixation cross colour change detection task were near ceiling for most participants. Group mean proportion correct scores ranged between 95-97% across sequence types (individual participant data displayed in Supplementary Figure S1A). Group mean RTs for correctly detected targets were also very similar and ranged between 372-381 ms across sequence types (Supplementary Figure S1B). There was no statistically significant effect of sequence type on mean RTs, F(2.6, 107.6) = 2.46, p = 0.076. Overall, these results do not indicate any substantial differences in task performance across sequence types.
3.2. Identification of VMR Regions of Interest
We visually identified two time windows during which there were more negative-going amplitudes following surprising compared to expected oddballs: an early time window (200-350 ms) and a late time window (500-1,000 ms; group-averaged ERPs are depicted in Figure 2A). For both time windows, differences were identified at bilateral occipitoparietal electrodes P7, P8, P9, P10, PO7 and PO8 (for scalp maps of time window-averaged mean amplitudes see Figure 2B). The latency and topography of the early effect closely resembles VMRs observed in our previous FPVS oddball experiment (Feuerriegel et al., 2018b), as well as VMRs reported when using equiprobable control sequences (Kimura et al., 2009; Amado & Kovacs, 2016). In addition, we also observed more positive-going ERPs for surprising stimuli at midline central and parietal electrodes for the early and late time windows (electrodes POz, PO3, Pz, P1, P2, P4, CP3, CPz, CP1, CP2, Cz, FCz, FC2). Analyses using these additional ROIs are presented in the Supplementary Material, and produced almost identical patterns of results to those found using the negative-going [surprising – expected] difference ROIs described above.
Differences between surprising/neutral and expected/neutral conditions are plotted in the bottom panels of Figure 2A. There were clearly-visible differences between surprising and neutral conditions (i.e. surprise effects) during the early and late time windows, but not between expected and neutral conditions. As surprise effects lasted until the end of the 1,000 ms epoch, we were concerned that these effects would systematically influence pre-stimulus baselines for oddballs that follow a surprising oddball. We therefore excluded epochs corresponding to oddballs that were preceded by a surprising oddball (e.g., as done by Kimura et al., 2009). Numbers of useable epochs per condition remained high following epoch selection (see Supplementary Table S2). Note that the averaged ERP waveforms, difference waves and scalp maps displayed in Figure 2 were calculated after excluding epochs that were preceded by a surprising oddball.
3.3. Quantifying Effects of Fulfilled Expectations and Surprise
To isolate and quantify effects of expectation suppression and surprise, we compared mean amplitudes for expected and surprising oddballs with those for neutral oddballs. Mean amplitudes were more negative-going for surprising compared to neutral oddballs for the early time window (p = 0.005, BF10 = 6.78) and the late time window (p < 0.001, BF10 = 1141.24). By contrast, amplitudes did not appear to differ between expected and neutral conditions, with Bayes factors indicating evidence for the null hypothesis (early time window: p = 0.433, BF10 = 0.22, late time window: p = 0.966, BF10 = 0.17).
We next performed analyses at a more granular level by comparing the neutral condition with each other stimulus appearance probability condition, to assess whether this pattern of results was consistent with the condition-averaged data. Scalp maps of mean amplitudes by stimulus appearance probability condition, averaged over early and late time windows for each condition, are displayed in Figure 3A. Scatterplots of mean amplitudes by stimulus probability are displayed in Figures 3B (early time window) and 3C (late time window). Results of paired-samples t tests are displayed in Supplementary Table S3 for the early time window, and Supplementary Table S4 for the late time window. ROI-averaged amplitudes were more negative-going for all surprising oddball conditions across both time windows (p’s < 0.05) except for the 20% appearance probability condition (early time window p = 0.092, late time window p = 0.062). Group mean [surprising – neutral] difference magnitudes were similar across all surprising oddball conditions, ranging from −0.26 to −0.39 μV (early time window) and −0.20 to −0.40 μV (late time window). By contrast, none of the expected oddball conditions evoked significantly more positive or negative amplitudes compared to the neutral condition (all p’s > 0.258) except the 60% appearance probability oddballs, which evoked more negative-going amplitudes during the early time window (p = 0.035). Group mean [expected – neutral] difference magnitudes were much smaller than the observed surprise effects, and ranged from −0.21 to 0.08 μV (early time window) and −0.03 to 0.02 μV (late time window).
3.4. Graded Effects of Stimulus Appearance Probability
After identifying contributions of surprise to VMR amplitudes, we then assessed whether there were graded differences in ROI-averaged amplitudes by oddball presentation probability for surprising (10-40%) oddballs. In other words, we tested whether ERP amplitudes indexed an observer’s degree of surprise, which should differ across the 40% (slightly unlikely) and 10% (very unlikely) oddball types. For the early time window beta values did not significantly differ from zero, and the Bayes factor indicated moderate evidence for the null hypothesis, t(42) = 0.20, p = 0.847, BF10 = 0.17, mean beta value 0.001, CI [-0.01, 0.01]. Here, the mean estimated beta value denotes an average change of only 0.001 μV for each percentage increase in oddball appearance probability. We found similar results for the late time window, t(42) = 1.24, p = 0.221, BF10 = 0.34, mean beta value 0.005, CI [-0.003, 0.01].
3.5. Effects of Oddball Position in a Sequence
We also assessed whether there were systematic effects of the oddball position in the sequence (ranging from 1-70) on ROI-averaged amplitudes, as previously identified when using an FPVS oddball design (Feuerriegel et al., 2018b). For these analyses, we included all oddball presentations and did not exclude epochs that were preceded by a surprising oddball presentation.
For the early time window, beta values for the linear effect of oddball position were significantly above zero, t(42) = 10.19, p < 0.001, BF10 > 10,000, mean beta value 0.01, CI [0.009, 0.013], indicating more positive-going amplitudes for oddballs presented later in the sequence (depicted in Figure 4A) as reported in Feuerriegel et al. (2018b). Effects of oddball position also appeared be highly similar for expected, neutral and surprising oddball types (Figure 4B).
For the late time window, beta values were significantly below zero, t(42) = −3.16, p = 0.003, BF10 = 11.52, mean beta value −0.002, CI [−0.004, −0.001], and the effect was much smaller in magnitude than for the early time window (estimated average change of −0.002 μV for each unit increase in oddball position in the sequences). There were no clearly-visible differences in oddball position effects across surprising, neutral and expected conditions (Figure 4C).
4. Discussion
Using classical oddball designs it has been remarkably difficult to disentangle effects of immediate stimulus repetition, expectation and surprise in relation to visual mismatch responses (VMRs). By instead using a modified oddball design that controls for stimulus repetition effects, we could successfully isolate and quantify the contributions of expectation suppression and surprise. We report that VMRs as measured in visual oddball designs index a surprise response that occurs when an observer’s expectations are violated. However, we did not find evidence of expectation suppression effects, in that oddball stimuli with very high probabilities of appearing (e.g., 90%) evoked ERP responses that were no different to an expectation neutral condition (Bayes factors indicated that any such effects were either absent or very small in magnitude). In addition, we did not find systematic variations in the extent of surprise responses by stimulus appearance probability (i.e. the likelihood of a certain stimulus appearing). In other words, VMRs were of similar magnitude for 40% probability (slightly unlikely) and 10% probability (very unlikely) oddballs. We also identified gradual changes in ERP amplitudes according to whether oddballs were presented relatively early or late within a stimulation sequence, as reported in previous work (Ulanovsky et al., 2004; Feuerriegel et al., 2018b). Our findings indicate that VMRs signal the occurrence of surprising or novel events, but convey little information about the degree of surprise associated with an event.
4.1. VMRs Appear to Index a Surprise Response, but Not Expectation Suppression
We found clear evidence for surprise effects; however, we did not observe expectation suppression. That is, ERP amplitudes were markedly different for surprising compared to neutral oddballs, but no differences were found between expected and neutral oddballs. This indicates that VMRs predominantly index a surprise response, and the relative magnitude of any effects of fulfilled expectations on ERP amplitudes (if they exist) are likely to be very small. These results also favour theoretical accounts that describe VMRs as signifying expectation violations (e.g., Winkler, 2007; Paavilainen et al., 2013), rather than predictive coding-based models which describe VMRs as reflecting both enhanced prediction error responses for deviants and suppressed prediction errors for expected standards (e.g., Friston, 2005; Stefanics et al., 2014, 2018). We believe that our findings extend to VMRs that share broadly similar scalp topographies and response latencies, which have been observed in previous work that used strategies to separate repetition effects from surprise responses (e.g., Astikainen et al., 2008; Kimura et al., 2009; Amado & Kovacs, 2016; File & Czigler, 2019; Feuerriegel et al., 2018b; reviewed in Kremlacek et al., 2016).
Based on our findings, we propose that VMRs in classical oddball sequences reflect contributions of immediate stimulus repetition and recent stimulus exposure (e.g., Ulanovsky et al., 2004; Sawamura et al., 2006; Maheu et al., 2019) that partially overlap in time with expectation violation responses that occur following deviant stimuli (e.g., Kimura et al., 2009; Amado & Kovacs, 2016). Response enhancement for deviants may also result from disinhibition of stimulus-selective visual neurons caused by adaptation to the often-repeated standard stimuli (e.g., Dhruv et al., 2011; Kaliukhovich & Vogels, 2016). Although we separated these effects in our experiment, they are likely to interact in the context of classical oddball sequences. Surprise responses appear to scale with the magnitude of stimulus-evoked responses, which can be suppressed via immediate stimulus repetition (Feuerriegel et al., 2018b) or changing the focus of spatial and feature-based attention away from the critical stimuli (File & Czigler, 2019; Smout et al., 2020). Here, we distinguish between the focus of spatial or feature-based attention and the task-relevance of the stimuli that evoke VMRs, as the critical stimuli in visual oddball designs are often not directly relevant to an experimental task.
Based on this evidence, the surprise responses indexed by VMRs appear to signal the presence of unexpected or novel changes in an observer’s visual environment that are within the focus of attention. VMRs may preferentially signal salient events that are relevant for additional computations associated with statistical learning (Leshinskaya & Thompson-Schill, 2020), detection of potential rewards (Schultz, 2016), belief updating (Bennet et al., 2015) and adaptive changes to decision-making strategies following unexpected events (Wessel et al., 2017). VMRs might even index a process that enables or facilitates these different types of computations, similar to the proposed role of other types of surprise responses that precede computations of expected reward and subjective utility (discussed in Winkler, 2007; Schultz, 2016).
Our findings also suggest that, in some contexts, the notion of expectation suppression (Summerfield et al., 2008; Todorovic & de Lange, 2012; Grotheer & Kovacs, 2015; Pajani et al., 2017) may be a misnomer when used to explain neural response differences between expected and surprising stimulus contexts. Studies using non-oddball designs and probabilistic cueing have similarly reported much larger effects of surprise on fMRI BOLD signals than of fulfilled expectations (Egner et al., 2010; Amado et al., 2016; reviewed in Kovacs & Vogels, 2014). Previous studies using roving stimulus variants of oddball designs have reported expectation-related suppression of prediction error-related neural responses (e.g., Stefanics et al., 2018), however these designs tend to conflate expectation and repetition effects, and the observed neural response reductions could instead be attributed to repetition suppression. Where feasible, we recommend including expectation neutral conditions in future studies to quantify relative effects of expectation suppression and surprise.
4.2. Graded Effects of Stimulus Appearance Probability
Contrary to our hypotheses, surprise responses did not linearly vary with stimulus appearance probability. This indicates that VMRs as isolated in our experiment do not signal the degree to which a stimulus is surprising, but instead reflect an all-or-none fixed-amplitude response that occurs following expectation violations. This pattern does not fit with precision-based expectation suppression accounts of VMRs, which would predict more negative-going amplitudes for stimuli that are less likely to appear (e.g., Stefanics et al., 2018).
Our findings also contrast with previous reports of N2 and P3 ERP component amplitudes scaling with stimulus appearance probability (e.g., Duncan-Johnson & Donchin, 1977; Polich & Margala, 1997; Wessel & Huber, 2019). This is likely because the critical stimuli were task relevant in those studies, but not in our experiment. Task relevance of the critical stimuli may be an important factor that determines whether expectancy-related computations that occur following detection of surprising events (e.g., as specified in Schultz, 2016) are reflected in additional ERP components or other neural activity measures. These additional processes may exhibit a higher degree of sensitivity to differences in contextual event probability than those identified in our study.
4.3. Gradual Changes in ERP Amplitudes During Stimulation Sequences
In addition to the surprise effects described above, we also replicated findings of gradual changes in neural response amplitudes that occur over the course of a stimulation sequence (Ulanovsky et al., 2004; Feuerriegel et al., 2018b). This was indexed by more positive-going ERP amplitudes during the early VMR time window for oddballs presented later in the FPVS stimulation sequences, and slightly more negative-going amplitudes during the late VMR time window. These effects also appeared to be qualitatively similar for expected, neutral and surprising oddballs. The cause of these gradual amplitude changes over time is unclear, but may reflect the consequences of prolonged continuous sensory stimulation (discussed in Ulanovsky et al., 2004). As we had observed these effects in our previous work (Feuerriegel et al., 2018b), we distributed expected and surprising oddballs evenly throughout the stimulation sequences in order to avoid spurious findings that are due to this effect. Given that the magnitudes of these effects were large compared to the surprise effects in our study, we recommend that others test for such gradual changes over time and explicitly account for these in statistical and computational models, particularly when model parameters reflecting sequence learning also change over similar timescales.
4.4. Study Limitations
Our findings should be interpreted with the following caveats in mind. First of all, the timing and topography of the VMRs observed here may be particular to experiments that present faces as stimuli. We used faces as they tend to elicit robust repetition and expectation effects, and the time-course of face oddball-evoked ERPs has been well-characterised in previous work (e.g., Dzhelyova & Rossion, 2014a, Quek & Rossion, 2017). We have also previously used face stimuli to identify VMRs in an FPVS oddball design (Feuerriegel et al., 2018b) which has many hallmark features of VMRs observed when using other stimulus categories (e.g., Kimura et al., 2009; Astikainen et al., 2008; Amado & Kovacs, 2016). However, VMRs do appear to qualitatively differ depending on the stimulus features that are relevant to an observer’s expectations (Amado & Kovacs, 2016; Stefanics et al., 2019; Robinson et al., 2020; Smout et al., 2020), possibly reflecting response amplitude changes in sensory neurons that are selective for those features. It remains to be seen whether visual mismatch responses that correspond to different anatomical locations within the visual hierarchy index different contributions of expectation suppression and surprise.
In addition, we used a limited range of stimulus appearance probabilities ranging from 10% to 90%. Consequently, we cannot rule out graded effects of stimulus appearance probability that might be identified using sequences with more extreme expected/surprising probability splits. For example, Javitt et al. (1998) presented auditory deviants at probabilities ranging between 0.56% and 15%, and found progressively more negative-going mismatch response amplitudes in smaller deviant probability blocks (see also Pincze et al., 2002). Experiments presenting stimuli at very low probabilities should include a familiarisation period to avoid potential confounding effects of stimulus novelty that may differ between expected and surprising stimuli (reviewed in Schomaker & Meeter, 2015; Feuerriegel, 2016).
We also note that other phenomena than those identified here are likely to influence VMR magnitudes, and may operate over a broad range of timescales (see Ulanovsky et al., 2004; Sawamura et al., 2006; Maheu et al., 2019). Our openly-available dataset can be used to test for candidate effects, which can then be replicated in future work. The set of effects identified here can also be used to develop computational models of VMRs, which may account for complex sets of potentially interacting phenomena in a mathematically explicit fashion (e.g., Stefanics et al., 2018), however this is beyond the scope of the current study.
4.5. Conclusion
We report that VMRs index a surprise response that occurs when an observer’s expectations about upcoming sensory events are violated. The magnitude of these surprise responses does not appear to vary with graded changes in stimulus appearance probability. Our findings suggest that VMRs may signal the presence of unexpected changes in the environment, functioning as a precursor to further computations that utilise surprise signals for statistical learning, reward seeking and adaptive changes to decision-making strategies. However, we did not find support for predictive coding accounts of VMRs that describe suppression of prediction error responses when an observer’s expectations are fulfilled.
Acknowledgements
This project was supported by a University of Melbourne Early Career Researcher Grant awarded to D.F., a European Union Horizon 2020 Marie Sklodowska-Curie Individual Fellowship awarded to G.L.Q. (841909), and Australian Research Council (ARC) Discovery Project Grants awarded to H.H. (DP180102268) and S.B. (DP160103353). Funding sources had no role in study design, data collection, analysis or interpretation of results.