Violated predictions enhance the representational fidelity of visual features in perception

Predictive coding theories argue that recent experience establishes expectations that generate prediction errors when violated. In humans, brain imaging studies have revealed unique signatures of violated predictions in sensory cortex, but the perceptual consequences of these effects remain unknown. We had observers perform a dual-report task on the orientation of briefly presented gratings within predictable or random sequences, while we recorded their pupil size as an index of surprise. Observers first made a speeded response to categorize grating orientation (clockwise or counterclockwise from vertical), then reproduced the orientation without time pressure by rotating a bar. This allowed us to separately assess response speed and precision for the same stimuli. Critically, on half the trials, the target orientation deviated from the spatiotemporal structure established by the preceding gratings. Observers responded faster and more accurately to unexpected gratings, and pupillometry provided physiological evidence of observers’ surprise in response to these events. In a second experiment, we cued the spatial location and timing of the grating and found the same pattern of results, demonstrating that unexpected orientation information is sufficient to produce faster and more precise responses, even when the location and timing of the relevant stimuli are fully expected. These findings indicate that unexpected events are prioritized by the visual system both in terms of processing speed and representational fidelity.


INTRODUCTION
Predictive coding theories argue that prediction errors are generated when bottom-up sensory inputs deviate from top-down expectations (Friston, 2005).There is broad consensus that prediction errors are associated with increased neural activation (Alink et al., 2010;Den Ouden et al., 2012;Kok et al., 2012;Meyer & Olson, 2011;Richter et al., 2018;Todorovic et al., 2011), which is typically observed during the initial processing cascade in sensory cortices associated with the unexpected stimulus (Tang et al., 2018(Tang et al., , 2023)), but has also been reported in subcortical regions (Mazancieux et al., 2023).By contrast, it remains unclear how these modulations reshape the pattern activity that represents the unexpected stimulus.
One might hypothesize that expected events should produce more precise representations, because the brain can anticipate incoming sensory signals and provide an optimized neural environment to receive them; for example, by increasing the gain of neurons tuned to the expected stimulus features.Such benefits of anticipation are evident in spatiotemporal cueing (Rohenkohl et al., 2014), and in sequential dependency experiments, where observers respond faster and more accurately to events that comply with a recent history of alternations and repetitions between two states (e.g., reporting on state "A" after exposure to states ABAB) than those that violate this pattern (e.g., reporting on state "A" after exposure to states BABA; Remington, 1969).There is also behavioural evidence that memory recall is more precise for expected visual objects then unexpected ones (Abreo et al., 2023), and that expected stimuli are more likely to be consciously perceived than competing unexpected stimuli during binocular rivalry (Denison et al., 2011;Pinto et al., 2015).By contrast, it seems equally reasonable to hypothesize that unexpected events might produce more precise representations.Unexpected or 'surprising' events are more likely to have harmful consequences than expected ones precisely because they have not been anticipated and thus an appropriate response has not been prepared.On this account, prioritizing the speed and fidelity with which unexpected events are represented may have adaptive value.Similarly, prioritizing unexpected events to effectively update internal predictive models may support adaptive future behaviours, e.g., increased caution after narrowly avoiding an obstruction on the road (Friston, 2009;Soltani & Izquierdo, 2019).Some behavioural evidence for prioritization of unexpected events has been reported in studies of representational momentum (Hubbard, 1994(Hubbard, , 2005) ) and the von Restorff effect (Wallace, 1965), in which incongruent or "distinct" visual stimuli benefit from improved recall.
Neuroimaging studies addressing the question of how prediction errors shape sensory representations have produced inconsistent findings.Some evidence from functional magnetic resonance imaging (fMRI) suggests that neural representations formed by unexpected stimuli are less precise than those of expected stimuli (Kok et al., 2012), while others suggest the opposite (Richter et al., 2022).Evidence from electroencephalography (EEG) in humans and two-photon calcium (Ca 2+ ) imaging in mice indicates that neural representations of unexpected stimuli are more precise than those of expected stimuli (Tang et al., 2018(Tang et al., , 2023)), while magnetencephalography work has suggested the opposite (Kok et al., 2017).
Adding further uncertainty to this question, a recent study failed to find a difference between the neural representations formed by expected and unexpected stimuli (den Ouden et al., 2023).
The hierarchical nature of predictive coding, combined with the multitude of approaches for studying brain function, likely contributes to the inconsistent findings on how prediction errors shape sensory representations.These discrepancies might be due to stimulus and experimental design choices (e.g., some designs may not sufficiently establish sensory predictions), but they might also be explained by biophysical differences between neural recording techniques (e.g., fMRI and EEG) or analytic tools for estimating representational precision from neural activity (e.g., linear discriminant analysis versus inverted encoding) (Brouwer & Heeger, 2009;Harrison et al., 2023;Rideaux et al., 2023).
To test the influence of prediction errors on the representation of visual stimuli, here we sought to measure perception directly, rather than indirectly via neural recordings.In Experiment 1, we had observers (N = 60 adults) perform a dual-report task on the orientation of briefly presented gratings embedded within sequences that were either predictable (consistent clockwise or counterclockwise rotation) or random (see Fig. 1).A similar presentation sequence was used previously to measure prediction errors in single neurons in mouse visual cortex (Tang et al., 2023).Observers first made a speeded binary response to categorize grating orientation (clockwise or counterclockwise from vertical), then reproduced the orientation without time pressure by rotating a bar.This allowed us to separately assess response speed and precision for the same stimulus.Critically, while the target grating orientation was selected at random, the orientations of the preceding gratings were either sequential (rotating at 30º per grating) or random.Thus, in the sequential condition, an expectation was established and subsequently violated by the target grating.Note, although the target grating in the sequential condition violated the sequence on most trials, this remained an unexpected event relative to the multiple preceding gratings in the trial that conformed to the sequence.By contrast, in the random condition observers had no stimulus information on which to establish an expectation of grating orientation (Fig. 1a).To isolate the contribution of the feature-specific violation, we ran a further study (Experiment 2) in which the gratings not only rotated but also traversed an imaginary circle, such that the target grating was either presented at a cued or uncued location.Observers viewed a sequence of gratings and were tasked with making a speeded response (clockwise or counterclockwise from vertical) about the target grating before precisely reproducing its orientation.Critically, while the target grating orientation was selected at random, the orientations of the preceding gratings were either sequential (rotating at 30º per grating; blue sequence) or random (black sequence).Thus, in the sequential condition, an expectation was established and subsequently violated by the target grating, whereas in the random condition there was no expectation.Note, the reduced contrast in the image illustrates the passage of time.b, c) The average (b) (speeded) response times and (c) (reproduction) precision in the random (black) and sequential (blue) conditions.Asterisks indicate significant differences of P<.001.

METHODS
Transparency and openness.The data and analysis code for this study are publicly available at https://osf.io/gj7ut/.
Participants.Sixty self-reported neurotypical human adults (mean±standard deviation age, 24.8±3.9years; 43 females) participated in Experiment 1 and ten in Experiment 2 (mean±standard deviation age, 24.1±2.3years; 8 females).Samples sizes were determined by pilot testing and similar experiments within the laboratory.
Observers were recruited from The University of Queensland, had normal or corrected-to-normal vision (assessed using a standard Snellen eye chart).All but three participants (authors PD, LJ and ZB) were naïve to the aims of the experiment and all gave informed written consent.The experiment was approved by The University of Queensland Human Research Ethics Committee.
Stimuli, task, and procedure.The stimuli comprised sinewave gratings (1 cycle/º, full contrast, phase randomized between trials) presented centrally within a circular aperture (radius 4º), which was smoothed at the edges, on a mid-grey background.A centrally positioned green fixation dot (radius 0.25º) was presented to reduce eye movements.
As illustrated in Figure 1a, trials consisted of 5-12 (counterbalanced across trials) sequentially presented grating stimuli (initial sequence).A variable number of gratings were presented in each trial to motivate observers to attend to the entire sequence.The initial sequence was followed by 2 additional gratings (penultimate and target).The orientations of the gratings in the initial sequence were either randomly selected between 0-180º (random condition) or rotated consistently by ±30º from a random start orientation (sequential condition; rotation direction counterbalanced across trials).The orientations of the penultimate and target gratings were matched between trials of the two conditions, such that for each sequential condition trial there was a random condition trial with the same penultimate and target gratings.In the sequential condition, the penultimate grating orientation was consistent with the rotation of the preceding expectation sequence (i.e., ±30º from the final grating in the sequence), while the target grating was selected at random (1-180º).Each grating was presented for 250 ms, separated by a blank 32 ms inter-stimulus-interval.Following the presentation of the target grating and a 32 ms blank period, a black vertical bar (length 3.2º) was displayed and observers were given 2 s to indicate whether the target grating was oriented clockwise or counterclockwise from vertical using the right or left mouse button, respectively.Upon responding, the black bar was displayed at a random orientation (between 0-180º) and the observers were then required to use the mouse to reproduce the orientation of the target grating (no time limit).Observers were instructed to respond quickly for the two-alternative forced choice (2AFC) task and precisely for the reproduction task.There were 24 trials per block and 16 blocks in total (~60 min), with equal numbers of random and sequential orientation trials interleaved pseudo-randomly across the session.
Experiment 2 was similar to Experiment 1, except that the spatial position of the gratings varied such that the stimulus appeared to traverse an imaginary circle (radius 6º, step size 12.85º, direction counterbalanced across trials) in the center of the screen.On half of the trials, the target grating was presented on a cardinal axis aligned to the center of the screen (i.e., above, below, left, or right).The target axis was randomized between blocks and indicated throughout the experiment by a white dot (radius 0.25º) positioned at 10º from center along the corresponding axis; black dots were presented at the same eccentricity on the three remaining axes.On the other half of the trials, the target grating was presented before it reached the target axis.Thus, observers were aware that gratings that were presented on the target axis were target gratings.We opted for this design, rather than a (simpler) temporal cue such as a warning symbol preceding the final grating, to avoid presenting additional unexpected events.The response bar was presented at the location of the target grating.This was repeated 24 times per block and observers performed between 48 and 60 blocks of trials (~240 min, spread across multiple sessions).
Eye tracking.Pupil size recordings were epoched to between -500 ms to 1500 ms around target grating onset, normalized to each observer's average pupil size across the experiment, and a linear model fit to the 500 ms period prior to stimulus presentation was used to baseline each epoch.Missing time samples (e.g., due to blinks) were interpolated.Pupil data from two observers were not collected due to hardware issues and observers with an average pupil diameter >3 standard deviations from the group mean were omitted from further analyses (n=3).
Using the cluster correction analysis described in the Statistical analyses subsection below, we found a single period when observers' pupil diameter was significantly influenced by our experimental manipulation of expectation.To test the relationship between pupil diameter and behaviour, we calculated the average difference in pupil diameter during this period, for each observer, and correlated this with behavioural performance.

Statistical analyses.
Statistical analyses were performed in MATLAB v2020a using CircStat Toolbox v1.12.0.0 (Berens, 2009).Observers' proportion correct and average response time on the 2AFC (speeded) task was calculated; data from observers who scored <.75 correct across all trials were omitted from further analyses (Experiment 1, n=8; Experiment 2, n=1).Precision on the reproduction task was inferred from the angular standard deviation between the target and response orientations.We only included trials on which the speeded task was answered correctly (91.1%), but this did not change the pattern of results from when all data were included.Repeated measures analysis of variance (ANOVA) and paired t-tests were applied to test for differences between conditions.The robust correlation toolbox (Pernet et al., 2013) was used to compute skipped Pearson correlations (default boxplot rule used to identify outliers) to test the relationship between measurements.
When testing the difference in pupil diameter between conditions across time, a cluster correction was applied to mitigate the inflated risk of false positives associated with conducting multiple tests (Pernet et al., 2013).We began by calculating the t-statistic at each time point.Next, we calculated the summed value of these statistics (separately for positive and negative values) within contiguous temporal clusters of significant values.We then simulated the null distribution of the maximum summed cluster values using permutation (n=1000) of the condition labels, from which we derived the 95% percentile threshold value.Clusters identified in the data with a summed effect-size value less than the threshold were considered spurious and removed.A one-tailed t-test was used to test for increased pupil diameter, as we had (a priori) hypothesised that pupil dilation would increase in "surprising" trials.Two-tailed tests were used for all other comparisons.

RESULTS
As shown in Figure 1b-c, observers responded faster and more precisely to unexpected grating stimuli than to random stimuli (response time, F 1,51 =26.17,P=4.78e -6 ; precision, F 1,51 =34.05,P=3.717e -7 ).We tested the inter-individual relationship between response time and precision and found positive correlations in both random and sequential conditions (random, r=.59, P=4.20e -6 ; sequential, r=.55, P=2.68e -5 ; Fig. 2a, b).This is likely explained by individual differences in factors that influence general task performance, such as fatigue and motivation.To control for these factors and isolate the behavioural changes associated with processing unexpected stimuli, we tested whether the difference in response times associated with unexpected stimuli was related to the difference in precision.We found these changes were positively correlated (r=.459,P=6.28e -4 ; Fig. 2c), suggesting that improvements in response time and precision associated with unexpected stimuli are supported by a common underlying mechanism.between the expected and target orientation) in the random and sequential conditions, binned into eight evenly spaced orientations.Note that while there was no clear expectation in the random condition, the trials in this condition were also grouped according to the angular difference between the penultimate and target grating.Filled circles indicate magnitudes at which there was a significant difference between conditions.
The orientation of the target grating was randomized, which means that the target violated the expectation established by the preceding gratings on most of the sequential condition trials.However, the magnitude of this violation (operationalized as the angular difference between the expected and target orientation) ranged between ±90°; thus, on some trials the target grating effectively satisfied, rather than violated, this expectation.If improved response time and precision are due to violations of expectation, these improvements should vary as a function of violation magnitude.In line with this reasoning, for both response time and precision, we found a main effect of condition (response time, F 1,51 =23.82,P=1.07e -5 ; error, F 1,51 =31.00,P=9.65e -7 ) and an interaction between violation magnitude and condition (response time, F 7,51 =12.19,P=5.44e -14 ; error, F 7,51 =12.65,P=1.59e -14 ; Fig. 2d, e).Pairwise comparisons revealed that response time and precision were worst when expectations were satisfied and best when expectations were violated by >=45°.Indeed, when stimuli were expected (i.e., 0° angular offset in the sequential condition), response time and precision were worse than when there was no clear expectation (response time, t 51 =5.14, P=4.45e -6 ; error, t 51 =5.89, P=3.04e -7 ).Thus, the curve formed by the data from the sequential condition effectively represent the tuning function of observers' predictions.Interestingly, the tuning curve appears to be suppressive, that is, it reflects slower and less precise responses for expected stimuli, consistent with an inhibitory predictive signal.Given that no consistent expectation is generated by random sequences, performance in the random condition should be invariant to the difference between the penultimate and target gratings.However, we observed faster and more precise responses when the penultimate and target gratings were similar.This is likely explained by temporal integration, since the stimuli were presented briefly (250 ms) and in rapid succession (32 ms inter-stimulus interval, ISI).When similar gratings appeared one after another, observers receive a clearer sensory signal from which to encode orientation.We note that this finding was incidental to the purpose of our investigation, and responses for unexpected stimuli were nevertheless faster and more precise when compared with the flat profile of responses in the random condition.
Here and in previous neuroimaging studies of prediction errors (Kok et al., 2012;Tang et al., 2018Tang et al., , 2023)), the spatiotemporal structure of visual stimuli was manipulated to establish and violate expectations.It is unclear, however, whether these stimuli sufficiently establish and violate expectation.Pupil dilation is an involuntary physiological marker of arousal/surprise (Preuschoff et al., 2011).If observers' expectations were violated by the target gratings, we would expect their pupils to dilate in response to these stimuli.For both the sequential and random sequences, we observed increased pupil diameter in response to the target grating, which may be explained by either surprise in response to all stimuli or a reduction in luminance between the blank ISI and the target grating (Fig. 3a, b).Critically, when we examined the difference between random and sequential conditions, pupil diameter was significantly larger following target gratings with larger expectation violations (>±30º from the expected orientation; Fig. 3c) from 563 -1100 ms after target onset.This timing is consistent with previous pupillometry work (Preuschoff et al., 2011), and provides physiological evidence that observers were surprised by gratings that violated the spatiotemporal expectation established in the sequential condition.Critically, there was also a positive relationship between pupil diameter and faster response time in this condition (r=.39,P=.0098; Fig. 3d), suggesting a link between violated expectations and improved performance.The positive relationship between pupil diameter and precision did not reach significance (r=.25, P=.1119; Fig. 3e), which may be because there was a longer delay between the unexpected stimulus and the reproduction response than the speeded response.In addition to previous work showing increased cortical activity in response to unexpected events (Den Ouden et al., 2012;Kok et al., 2012;Meyer & Olson, 2011;Todorovic et al., 2011), recent fMRI work has found increased pupil diameter and increased activity in subcortical regions (including the locus coeruleus), which are associated with neuromodulatory systems (dopamine, serotonin, and acetylcholine) (Mazancieux et al., 2023).Our findings complete the characterization of this coordinated response to unexpected events, by revealing the perceptual consequences of potentially related cortical and subcortical activity, and their relationship with physiological markers of surprise.
Figure 3. Changes in pupil diameter associated with expectation violation.a) Change in pupil size in response to the target grating in the random condition, for trials with smaller and larger than ±30º expectation violation.The inset disk indicates the colours used to show the data for target gratings with orientations that were near to (gray) or far from (black) the expected orientation (white dashed line).b) Same as (a), but for the sequential condition.c) The difference in pupil size between random and sequential condition trials with larger and smaller expectation violations; colour mapping is the same as in (a, b).d, e) The difference in average response time (d) and precision (e) between random and sequential conditions as a function of the difference in pupil size.Note that while there was no clear expectation in the random condition, the trials in this condition were also grouped according to the angular difference between the penultimate and target grating.Shaded regions indicate ±SEM and the horizontal bar in (c) indicates a cluster corrected period of significant differences between random and sequential conditions.Solid and dashed rectangles in (a-c) indicate the target and preceding stimulus presentations, respectively.
Prioritization of unexpected events may be implemented through neuromodulator-mediated engagement of attention systems (Feldman & Friston, 2010;Hohwy, 2012;Sara & Bouret, 2012).To motivate observers to attend to all presented stimuli, we displayed a variable number (5-12) of gratings, such that the identity of the target grating was unknown until the response bar appeared.Thus, on most trials in the sequential condition, both the temporal identity and feature-specific (orientation) properties of the target grating were unexpected.To isolate the contribution of the feature-specific violation, we ran a further study (Experiment 2) in which the gratings not only rotated but also traversed an imaginary circle, such that the target grating was either presented at a cued or uncued location (Fig. 4a).Thus, on half the trials observers were validly cued to the temporal identity and location of the target grating before it was presented.We assessed the influence of cueing by comparing performance between random and sequential sequences on trials where the target identity was cued validly or invalidly (Fig. 4b).For both response time and precision, we found significant effects of target sequence, cue validity, and their interaction (all P<.05).Paired t-tests of the aggregate data (including all violation magnitudes) revealed that the results from the uncued condition replicated those from the main experiment (response time, t 8 =4.85, P=.001; error, t 8 =7.64, P=6.08e -5 ; Fig. 4c, d).Critically, we also found faster and more precise responses in the sequential condition when the target identity was cued validly (response time, t 8 =5.41, P=6.38e -4 ; error, t 8 =6.87, P=1.29e -4 ; Fig. 4e, f).Taken together, these results confirm that unexpected features of stimuli (i.e., orientation) are sufficient to produce faster and more precise responses, even when the location and timing of the relevant stimuli are expected.

DISCUSSION
Our findings reveal the perceptual consequences of prediction errors and indicate that unexpected events in the environment are prioritized by the visual system both in terms of processing speed and representational fidelity.Prioritizing unexpected events likely supports immediate and future adaptive behaviours through multiple operations, e.g., rapid responses to potentially harmful events and precise updating of internal predictive models (Friston, 2009;Soltani & Izquierdo, 2019), and may be operationalized through subcortically regulated attention systems (Feldman & Friston, 2010;Hohwy, 2012;Sara & Bouret, 2012).
The finding that unexpected events are reproduced more precisely is consistent with EEG data from humans (Tang et al., 2018) and Ca 2+ imaging data from mice (Tang et al., 2023), but appears to contradict some previous fMRI data in humans (Kok et al., 2012).These discrepancies could due to biophysical differences between the imaging signals used in these studies.Specifically, blood-oxygen-leveldependent (BOLD) signals may not have the temporal resolution to disambiguate pre-activation (Kok et al., 2017) from prediction error signals.Pre-activation likely biases performance toward expected outcomes and thus may obscure the suppressive effects of prediction errors on these same outcomes.Under this interpretation, our findings could indicate that perception reflects an interaction between prediction and sensory processing input, rather than pre-activation, which may facilitate attentional deployment.
How do our findings fit with previous behavioural studies?Previous work found better perceptual sensitivity for stimuli that complied with spatial and temporal cues (Rohenkohl et al., 2014).However, this may be due to differences in attention prior to presentation of attended and unattended stimuli, rather than the influence of prediction error.That is, cuing attention to a particular location that is either valid or invalid, relative to the spatiotemporal location of the upcoming stimulus, creates an attentional bias between conditions before the expected/unexpected stimulus is presented.By contrast, in our experiments, spatial and temporal attention were equally focused on expected and unexpected stimuli (at least prior to stimulus presentation).Sequential dependency paradigms, which establish expectations differently from those typically used to study prediction errors, show that expected events are responded to faster and more accurately than unexpected events (Remington, 1969).By contrast, here we found that observers responded faster to unexpected than expected stimuli.This suggests distinct mechanisms drive the behavioural consequences of expectancy associated with classic sequential dependency paradigms and those observed here.Indeed, our finding that unexpected events are processed faster provide some support for the claim that the differences observed in sequential dependency paradigms are a result of motor priming, rather than a change in sensory encoding (Jentzsch & Sommer, 2002).Abreo et al. (2023) used a similar design to that employed here (Supplementary Fig. 1a), but concluded that expected gratings were reproduced more precisely, which seems to be in direct conflict with our findings.However, re-analysis of their data revealed that their results may have been driven by temporal integration of the penultimate and target gratings, and that the number of stimuli used by Abreo et al.
In summary, here we resolve inconsistent findings on the influence of prediction errors on sensory encoding by showing that unexpected events are responded to faster and reproduced more precisely than those that are either expected or occur randomly and thus without any explicit expectation.These findings offer empirical evidence for neural prioritization of unexpected events, which support immediate adaptive responses and updating of future predictions.

Figure 1 .
Figure 1.Unexpected stimuli provoke faster and more precise responses.a) An illustration of the trial design.Observers viewed a sequence of gratings and were tasked with making a speeded response (clockwise or counterclockwise from vertical) about the target grating before precisely reproducing its orientation.Critically, while the target grating orientation was selected at random, the orientations of the preceding gratings were either sequential (rotating at 30º per grating; blue sequence) or random (black sequence).Thus, in the sequential condition, an expectation was established and subsequently violated by the target grating, whereas in the random condition there was no expectation.Note, the reduced contrast in the image illustrates the passage of time.b, c) The average (b) (speeded) response times and (c) (reproduction) precision in the random (black) and sequential (blue) conditions.Asterisks indicate significant differences of P<.001.

Figure 2 .
Figure 2. Shared mechanisms and tuning properties for visual predictions.The relationship between response time and precision across observers in the (a) random and (b) sequential conditions.c) The relationship between the difference (random -sequential) in response time and precision across observers.Black lines indicate line-of-best-fit.d) Response time and e) precision as a function of the magnitude of expectation violation (operationalized as the angular differencebetween the expected and target orientation) in the random and sequential conditions, binned into eight evenly spaced orientations.Note that while there was no clear expectation in the random condition, the trials in this condition were also grouped according to the angular difference between the penultimate and target grating.Filled circles indicate magnitudes at which there was a significant difference between conditions.

Figure 4 .
Figure 4. Unexpected stimulus features provoke faster and more precise responses.An illustration of the trial design for Experiment 2, which was the same as in the first experiment, except that stimuli moved around the display such that the target grating was presented either b) before (uncued) or at a predefined location (cued, illustrated by dashed red line).Thus, observers were cued to the identity of the target grating before it was presented in the predictable condition.c) Response time and d) precision as a function of expectation violation, for random and sequential sequences, when the target location was uncued.e,f) Same as (c,d), but when the target location was cued.