Dissociable neural effects of temporal expectations due to passage of time and contextual probability

The human brain is equipped with complex mechanisms to track the changing probability of events in time. While the passage of time itself usually leads to a mounting expectation, context can provide additional information about when events are likely to happen. In this study we dissociate these two sources of temporal expectation in terms of their neural correlates and underlying brain connectivity patterns. We analysed magnetoencephalographic (MEG) data acquired from N=24 healthy participants listening to auditory stimuli. These stimuli could be presented at different temporal intervals but occurred most often at intermediate intervals, forming a contextual probability distribution. Evoked MEG response amplitude was sensitive to both passage of time and contextual probability, albeit at different latencies: the effects of passage of time were observed earlier than the effects of context. The underlying sources of MEG activity were also different across the two types of temporal prediction: the effects of passage of time were localised to early auditory regions and superior temporal gyri, while context was additionally linked to activity in inferior parietal cortices. Finally, these differences were modelled using biophysical (dynamic causal) modelling: passage of time was explained in terms of widespread gain modulation and decreased prediction error signalling at lower levels of the hierarchy, while contextual expectation led to more localised gain modulation and decreased prediction error signalling at higher levels of the hierarchy. These results present a comprehensive account of how independent sources of temporal prediction may be differentially expressed in cortical circuits. HIGHLIGHTS - Predictability of tone onset times affects auditory network connectivity - Foreperiod and distribution of events in time have dissociable neural substrates - Decreased prediction error at different levels of cortical hierarchy

In each trial (see Figure 1A for an outline of the experimental paradigm; for more details see: 144 Todorovic, et al., 2015), a central fixation cross was presented for 2-4 s, followed by a standard 145 tone (1000 Hz) presented twice, with one of five inter-stimulus intervals (ISI) between the two 146 tones (250, 375, 500, 625, or 750 ms). After the offset of the second tone, the fixation cross 147 remained on the screen for 0.5-1 s and was subsequently replaced by a blank screen for 1.5-2 148 s. Occasionally, standard tones were replaced by deviant tones (1200 Hz; 9% trials) and 149 participants were instructed to press a button with the right index finger as soon as they heard 150 a deviant tone. 151

152
In addition to the influence of passage of time on temporal expectations (i.e., the longer the 153 foreperiod after the first tone, the more likely the presentation of the second tone), we also 154 varied the width of the distribution of second tone presentation at each ISI. In blocks with focal 155 temporal expectation, the second tone was played following an ISI of 500 ms in 70% trials, and 156 following other ISIs in 7.5% trials each. As a result, participants could build up a narrow 157 expectation of the second tone being presented 500 ms after the first tone. Conversely, in blocks 158 with distributed temporal expectation, the second tone was played following an ISI of 500 ms 159 in 24% trials, and following other ISIs in 19% trials each. In these blocks, participants could 160 expect the second tone to be presented at any ISI with roughly the same probability. In total, 161 four blocks with distributed temporal expectation (corresponding to 88 tones at the ISI = 500 162 ms and 252 tones at the remaining ISIs), and two blocks with focal temporal expectation 163 (corresponding to 120 tones at ISI = 500 ms and 52 tones at the remaining ISIs) were 164 administered in each participant. The order of blocks was counterbalanced across subjects. 165 Finally, the experimental paradigm included an orthogonal manipulation of temporal attention, 166 whereby participants were instructed to attend to the first or second tone and press a button 167 following a deviant tone presented in the attended position. 168 169

Behavioural analyses 170
We log-transformed the single-trial deviant detection RTs to normalise RT distribution. Trials 172 with RTs outside of the mean±2*SD range were excluded from analysis. Individual 173 participants' mean RTs to deviants presented in the second position, calculated separately for 174 ISIs shorter than, equal to, and longer than 500 ms respectively, were entered into a repeated-175 were recorded to guide eye blink and heart beat artefact rejection, using 10-mm-diameter Ag-185 AgCl surface electrodes. 186 187

Event-related field analysis 188
Continuous data were downsampled from 1200 Hz to 300 Hz, notch-filtered at 50 Hz and high-190 pass filtered at 0.1 Hz using a two-pass Butterworth filter. Artefacts induced by eye blinks were 191 removed by subtracting two principal spatiotemporal modes associated with eye blinks (Ille, et 192 al. 2002). Corrected sensor data were low-pass filtered at 48 Hz. 193 194 Our aim was to compare second tones, that followed the first tone with varying ISIs, to each 195 other. Given that tones at shorter ISIs arrived while the first tone was still being processed, we 196 needed to assess what the evoked field to the second tone at all ISIs would look like in the 197 absence of the first tone. To this end, rather than epoching the data, we modelled the evoked 198 responses using convolution modelling in continuous sensor data estimated for the entire 199 allows for overlapping responses to successive trials, in the same way one would model fMRI 203 time-series. Our principal aim was to estimate the evoked responses to tones presented at 204 different ISIs relative to the first tone. To increase the number of events in each regressor, we 205 pooled over ISIs shorter than 500 ms and longer than 500 ms respectively, resulting in three 206 groups of ISIs (short, middle, long). Thus, the regressors included experimental regressors 207 coding for tone onsets (standard tone in the first position; deviant tone in the first position; 208 standard tone in the second position at ISI lower than, equal, and higher than 500 ms 209 respectively; deviant tone in the second position pooled over all ISIs) and motor responses 210 (button presses) as well as nuisance regressors. Tone regressors were further convolved with a 211 binary regressor coding for temporal attention (on or off the tone of interest). Nuisance 212 regressors included EOG time-series and its temporal derivative, as well as the ECG time-213 series, to further reduce artefacts associated with eye movements and heart blinks.  We identified significant effects of foreperiod and context for two separate clusters (250-500 255 ms, corresponding to the effect of foreperiod; and 425-500 ms, corresponding to the effect of 256 context; see Results 3.2). We then based source reconstruction on a multiple sparse priors 257 algorithm (Friston, et al. 2008) performed on these two time windows. For each time window, 258 the estimated sensor-level evoked responses were first used to calculate contrast time-series, 259 corresponding to the linear approximation to the passage-of-time and the inverted U-shaped 260 approximation to the distribution effect. The resulting contrast time-series were treated as 261 evoked responses whose underlying sources were inferred per participant using the multiple 262 sparse priors source reconstruction. The single-participant 3D source activity maps were 263 entered into a group-level GLM to infer those sources which are differentially sensitive to the 264 temporal expectations based on foreperiod and distribution. Thus, significant effects were 265 inferred per time window using a paired F-test, contrasting the source estimates corresponding 266 to the passage-of-time effect and the distribution effect, after thresholding the 3D (spatial) 267 statistical parametric maps at P < 0.001 (peak-level, uncorrected) and correcting P-values for 268 multiple comparisons based on cluster size at a family-wise error rate at P < 0.05. Sources were 269 assigned anatomical labels using the SPM12 atlas provided by Neuromorphometrics, Inc.  within a source). Specifically, there is a laminar asymmetry in the outputs of each sourcefor 296 instance, superficial pyramidal cells propagate ascending signals to hierarchically higher areas, 297 whereas deep pyramidal cells propagate descending signals to hierarchically lower areas. The 298 output of each source is further modulated by intrinsic connectivity parameters, describing the 299 strength of the self-inhibition of each neural population. The equations describing the dynamics 300 at each source are provided below: 301 The distinct neural populations comprising a canonical microcircuit are indicated by subscripts We next wanted to know whether these clusters can be uniquely linked to foreperiod and 387 temporal distribution expectations respectively, or, conversely, if they represent a mixture of 388 both effects. To this end we repeated the analysis by (1) testing the effect of foreperiod while 389 masking out the effect of onset distribution (masking threshold p<.001, uncorrected) and 390 inferring the significant effects at p<.05, FWE-corrected; (2) testing the effect of onset 391 distribution while masking out the effect of foreperiod (using identical thresholds); (3) testing 392 the conjunction of foreperiod and distribution effects (thresholded at p<.001 and corrected for 393 multiple comparisons at pFWE<.05). Here we found that the effects of foreperiod and context 394 were independent of each other ( Figure 4B). There was no evidence of overlap in the two 395 clusters, as identified in the conjunction analysis (no conjunction cluster significant at a 396 threshold of p<.001, after correcting for multiple comparisons at pFWE<.05). 397   and tone distribution expectation ( Figure 4C; Table 1). This analysis revealed that for a long 418 time window (250-500 ms), in which the effect of foreperiod modulated ERF amplitude, the 419 power of source-level activity in bilateral early auditory cortices (A1) and superior temporal 420 gyri (STG) significantly differed between temporal expectations stemming from foreperiod and tone onset distribution (voxels thresholded at p<.001 and corrected using pFWE<.05). Apart 422 from these two bilateral regions, an analysis of the shorter, later time window (425-500 ms) in 423 which ERF amplitude was modulated by expectation of tone distribution, revealed that bilateral 424 sources including A1, STG, and additionally inferior parietal cortex (IPC) once again 425 significantly differed between the two kinds of temporal expectation. In sum, IPC was uniquely 426 sensitive to expectations of distribution of events in time, while A1 and STG were sensitive to 427 both this distribution and the passage of time. We next used these 3 bilateral regions as a basis 428 for subsequent dynamic causal modelling. The full models (with all connections allowed to be 429   activity reduced as the inter-stimulus intervals got longer (i.e. predictability grew), and 485 distribution of events in time corresponding to a model where evoked activity was minimal at 486 the middle interval (where predictability was greatest) but grew towards the edges of the 487 distribution. We found that these two types of temporal expectation correlated with activity in 488 a similar set of areas, but that the modulations began at different moments in time, and that they 489 affected network connectivity in distinct ways. 490 491 Behaviourally, we found that tones that arrived before the most frequent, middle ISI, were 492 paired with slower response times, confirming that likelihood had an effect on how tones were 493 processed. A recent study with normally distributed tone onsets also found faster responses to ISIs ranging from 250 ms to 750 ms), we first applied convolution modelling to estimate the 498 evoked field to the second tone while controlling for the influence of the first tone. After this 499 step, we were able to include early, middle and late arriving tones into a single search for 500 cortical sources. We found two clusters of activity, one beginning early (starting at 267 ms after 501 stimulus onset) and corresponding to passage of time, and one beginning late (starting at 427 502 ms after stimulus onset) and corresponding to temporal distribution of tone events. Importantly, 503 we found that the neural effects related to these two types of temporal expectation were 504 separable from each other. 505

506
We then localised the effects of temporal expectation to sources corresponding, bilaterally, to 507 the primary auditory cortex (A1), superior temporal gyrus (STG) and inferior parietal cortex 508 (IPC). While A1 and STG were sensitive to both this distribution and the passage of time 509 (corresponding to the early and late clusters of activity in the ERF time-series analysis), IPC 510 was uniquely sensitive to expectations of distribution of events in time (corresponding 511 specifically to the late activity clusters). In previous research, A1 has been implicated not only 512 as a hub for auditory processing, but also as a modality-independent timekeeper (Kanai, et al. It has even been suggested that processing of time has motor origins (Morillon and Baillet 519 2017). Interestingly, we did not observe SMA activity to be involved in our temporal hazard. 520 Potentially this might be because the temporal processing in our study was implicit: although it 521 aided task performance, this was a pitch discrimination task, and not one that involved 522 estimating durations. However, one other study with a tone discrimination task involving 523 similarly distributed tone onset times also found SMA activity (Herbst, Fiedler, and Obleser 524 2018). In that study, SMA was involved in foreperiod tracking in blocks with flat (i.e. 525 equiprobable) onset time distributions but not in blocks with strong contextual expectations of 526 the distribution. It is therefore likely that our search for cortical sources, which was specifically 527 sensitive to areas involved both in passage of time and contextual distribution of events in time, 528 led to SMA being excluded. 529

530
We next looked at how each of these two types of temporal expectation modulated connectivity 531 within and between A1, STG and IPC. We found that both types of temporal expectation 532 affected the strength of connections between regions in the auditory network. We were 533 particularly interested in testing whether changes in connection weights would be in line with 534 predictive coding, namely, corresponding to decreases in feedforward excitation, increases in 535 feedback inhibition and increases in intrinsic inhibition when a tone is more predictable. 536 However, the changes in connectivity profiles that were affected by temporal expectation were 537 more varied, especially when it came to tracking predictability as a function of elapsed time. 538 539 The most prominent common feature between the two types of temporal expectation was how 540 they modulated intrinsic connections in bilateral A1, STG and IPC. With growing tone onset 541 probability, whether due to passage of time or due to their distribution, intrinsic connections 542 consistently led to greater inhibition of neural activity (i.e. lower gain). Gain modulation has 543 indeed been posited as a core feature of predictive processing (Garrido, et al., 2009;544 Auksztulewicz & Friston, 2016). In the context of repetition suppression, when an initially 545 novel (unpredicted) stimulus starts repeating and forming a standard (predicted) stimulus, 546 neural gainmodelled as intrinsic connectionsshows a relative decrease following the first 547 repetitions (when a prediction is being established) and a gradual rebound with later repetitions 548 (when a prediction is fully formed; Garrido, et al., 2009). A similar change in intrinsic 549 connectivity was evident here in the entire network for predictability related to elapsed time, 550 and in A1 and IPC, but not STG, for predictability associated with temporal distribution. A 551 number of previous studies have indicated a role for the STG in temporal processing. For 552 instance, BOLD activity evoked by target onset in both STG and SMA has been shown to reflect 553 the cumulative hazard of a target appearing at a given moment, independent of modality 554 (auditory vs. visual) or the presence of a motor response (Cui, et al., 2009). Importantly, when 555 participants were informed of the exact target onset in that study, BOLD signal no longer 556 correlated with the foreperiod, suggesting that activity in these regions did not simply reflect 557 passage of time, but rather a temporal probability estimate. Another study found that STG (but 558 not A1 or IPC) was involved in estimating relative stimulus duration (Coull, et al., 2008). 559 Therefore, a plausible explanation for the difference in STG gain modulation between the two 560 types of temporal expectation is that a more involved time estimation was necessary for 561 estimating the expected peaks of temporal distribution in our study, which required activity in 562 STG, preventing self-inhibition. 563

564
The added complexity of assessing temporal distribution (relative to passage of time) was also 565 visible in the fact that predictability due to distribution was largely tracked in higher-level 566 connections between IPC and STG. Differences in prediction complexity have previously been 567 linked to hierarchical differences in prediction and prediction error signalling, with 568 hierarchically higher areas integrating information over longer time scales and representing 2008). In the current study, fully in line with predictive coding, tones that were more predictable 571 based on the distribution (i.e. middle ISI tones relative to both early and late tones) led to less 572 feedforward activation from STG to IPC, and more feedback inhibition from IPC to STG. 573 574 Similarly, if the prediction came from monitoring elapsed time, less excitation was fed forward 575 from A1 to STG. The same study that found reduced gain during prediction formation also 576 found a monotonically decreasing feedforward excitation from A1 to STG, as well as a 577 reduction in ERPs, in repeated standard tones (Garrido, et al. 2009). Although there was no 578 manipulation of stimulus timing, the similarity with our experiment lies in stimulus 579 predictability: standard tones are predictable and expected, especially when repeated multiple 580 times, and therefore less forward excitation likely mediates reduced prediction error signalling 581 at lower levels of the cortical hierarchy (Auksztulewicz & Friston, 2016). 582 However, at higher levels of the cortical hierarchy, the changes in connectivity begin to diverge 584 from what predictive coding models would posit. Firstly, more (not less) excitation was fed 585 forward from bilateral STG to IPC after the occurrence of a more predictable tone. Secondly, 586 when it comes to feeding inhibition back from IPC to STG, passage of time led to an opposite 587 effect in the two hemispheres, with inhibition increasing with stronger predictions in the left 588 hemisphere, but decreasing in the right hemisphere, amounting to stronger net activation in the 589 right IPC and weaker activation in the left hemisphere. Thirdly, stronger predictability with 590 passage of time led to an asymmetric modulation of cross-hemispheric connectivity between 591 the left and right IPC, amounting to the right IPC exerting stronger influence on the left IPC 592 than vice versa. While stimulus expectation is most often linked to a reduction in forward 593 signalling, expectation enhancement has also occasionally been observed (Segaert et  The differences between how two types of temporal expectation modulated connectivity 610 suggest that, while stronger predictions can lead to less feedforward excitation and more 611 feedback inhibition, these changes are not passed along the entire cortical hierarchy. In fact, 612 they are more likely to be contained between a smaller number of cortical areas that are directly 613 specific to a certain type of information processing, with, in this case, the more complex 614 predictability related to tone onset distribution, modulating only higher order connections but 615 not influenced by changes in activation coming from the primary auditory cortex. In addition, 616 at the higher levels of the hierarchy the prediction errors were not suppressed by prediction 617 related to passage of time, but instead changed the connectivity pattern in a different, often 618 opposite way. Taken together, our results show demonstrate that different aspects of temporal 619 expectation can be dissociable in terms of the latency of neural responses, the underlying 620 sources and connectivity patterns, leading to a more nuanced view of how prediction and 621 prediction error signalling may be expressed in cortical circuits. 622