Spatial and semantic regularities produce interactive effects in early stages of visual orientation

Learning environmental regularities allows predicting multiple dimensions of future events such as their location and semantic features. However, few studies have examined how multi-dimensional predictions are implemented, and mechanistic accounts are absent. Using eye tracking study, we evaluated whether predictions of object-location and object-category interact during the earliest stages of orientation. We presented stochastic series so that across four conditions, participants could predict either the location of the next image, its semantic category, both dimensions, or neither. Participants observed images in absence of any task. We modeled saccade latencies using ELATER, a rise-to-threshold model that accounts for accumulation rate (AR), variance of AR over trials, and decision threshold. The main findings were: 1) accumulation-rate scaled with the degree of surprise associated with location of target-presentation (confirmatory result); 2) predictability of semantic-category hindered latencies, but only when images were presented at a surprising location, suggesting a bottleneck in implementing joint predictions; 3) saccades to images that satisfied semantic expectations were associated with larger variance of accumulation-rate than saccades to semantically-surprising images, consistent with a richer repertoire of early evaluative processes for semantically-expected images. Joint impacts of location and target-identity regularity were also identified in analyses of anticipatory fixation offsets. The results indicate a strong interaction between the processing of regularities in object location and identity during stimulus-guided saccades, and suggest these regularities also impact anticipatory, non-stimulus-guided processes.

Markov entropy was 0.92 bits/trial for the Regular process and 1.58 bits/trial for the nonReg process. Note that, as described 138 above, such transition matrices independently determined transitions between spatial locations or image-categories, as specified 139 in the different conditions, so that the statistical features of the identity and location streams needed to be tracked separately. 140 From these transition matrices we produced series with 120 trials. In the study, these series were presented according to a The four types of series described above were constructed to generate the 9 types trials types in Figure 1B. Beyond this, 146 they were not used as factors in the design or referred to in contexts of statistical analyses. To evaluate statistical interactions 147 between identity and location knowledge, we analyzed the 9 trial-types in the analysis matrix presented Figure   Stimuli and trial structure 161 The timeline of each trial was as follows (see Figure 2A): a fixation symbol appeared for 400 ms, followed by a post-fixation 166 9 • from screen center. Targets subtended a visual angle of 6.36 degrees in each screen coordinate. Participants were instructed 168 to saccade rapidly to the target and fixation symbol when they appeared (see Figure 2). 169 The target images belonged to one of four categories of RGB  the other images were retrieved from publicly accessible online repositories. When assigning these images to the 120-trial 173 series, the elements were assigned randomly with the additional following constraints: the same image could be presented 174 only once per series, could not be presented in two consecutive series, and could not be presented in the same location as the

179
Participants were instructed to maintain their gaze on the fixation symbol when it was present and to move their gaze quickly 180 toward the presented image. The instructions did not mention image-identity. To maintain participants' alertness, we included 181 catch trials that consisted of black discs with a white line through them that appeared instead of the regular fixation symbol.

182
Catch trials appeared every 23-27 trials following a uniform distribution. Participants were told that catch trials would appear 183 infrequently and that they were to press the mouse button when they saw them. Following each series, participants were 184 presented with performance indicators for that series, which included the number of targets and fixation symbols saccaded to 185 within an allowed spatial and temporal tolerance (see below), the number of correct catch trials and eye blinks, as well as their 186 overall mean performance to that point. This was done to motivate participants to perform well and to provide a buffer between 187 the stochastic contexts of the just completed and the following series.

188
Before beginning the study proper, participants underwent training where they viewed series of 20 trials each, until they 189 were comfortable with the procedure (typically within 2-7 series penalize participants for saccading to the screen center prior to stimulus appearance. Here and in the rest of the text, the term 199 "anticipatory saccades" refers to saccades to the laterilized targets (which were of main interest) rather than saccades to central 200 fixation. A summary of the positive and negative scores was presented at the end of each training session. 201 We detected saccades by adaptively determining speed thresholds relative to saccade onset, offset and peak (Nyström & 204 Holmqvist, 2010). We defined saccade onset (offset) time as the time of the first local minimum with speed below an adaptive 205 threshold, preceding (following) a saccade peak.

206
Anticipatory fixation offsets and trial selection 207 We defined a measure of gaze location on each trial (Gaze Offset) as the mean gaze in the x direction in the last 10 milliseconds 208 prior to target appearance. These Gaze Offset data were the input to the steady-state analysis of gaze location described below.

209
In a subsequent analyses, for convenience, we coded gaze offset as positive if in the same direction of the previous target 210 x-coordinate, as negative otherwise. We refer to this quantity as Anticipatory Fixation Offset (AFO). Since AFOs are small in 211 amplitude (usually < 0.5 • ), to avoid having a few outliers bias the mean AFO value, we analyzed only trials in which AFO 212 was below 3 • (as in Notaro et al., 2019). Other than this constraint, we also analyzed only these trials in which the measured 213 saccades were reactive to the target presentation. This was defined as having a latency above 80 ms (see Fischer & Boch, 1983)   2. We calculated the autocorrelation function up to the maximum lag of 11 trials:

243
The LATER model (Carpenter & Williams, 1995) is based on the experimental finding that while saccade latency (SL) is 244 a variable that has a skewed distribution, the distribution of its reciprocal (promptness) is symmetric and well described by 245 a Gaussian. Consequently, the reciprocal has two free parameters, µ and σ µ describing its normal distribution. In LATER, 246 these parameters are taken to describe a generative decision process, where the sensory evidence is accumulated with a rate µ, 247 constant within a trial, but variable across trials according to the uncertainty σ µ . A decision is taken when the accumulated 248 evidence reaches a threshold, generally set to 1.  There was also a significant effect of Identity, F(2, 39) = 10.47, p < .001. Collapsing across Location, we found that µ 297 was highest in the nonRegular condition, and significantly higher than both the expected (t(39) = 3.36, p = .0035, d = 0.86) 298 and surprising conditions (t(39) = 3.79, p < .001, d = 1.10); see Figure 4B. There was however no difference between the 299 identity-expected and identity-surprising trials, even when examined separately within each level of location predictability (all 300 ps > .1). The effect of identity therefore does not track identity-surprisal but is instead consistent with a non-specific effect of 301 identity regularity, as identified for the analysis of saccade latencies. To summarize, to this point, the data for accumulation rate 302 match those found for the mean saccade latencies.

303
Departing from the findings reported for SL, accumulation rates reflected a significant interaction between the two factors. was surprising. This stood in marked contrast to when targets were presented at nonRegular locations, in which case being able 307 to predict target identity had almost no effect.
308 Figure 4. ELATER-estimated accumulation rates. Panel A. Mean accumulation rate for saccades to expected, surprising, and nonRegular locations (data are collapsed across levels of the Identity factor). Panel B. Mean accumulation rate for expected, surprising or nonRegular locations as function of whether identities were presented within statistically-regular or non-regular series. Two asterisks indicate significant pairwise differences with p < .001 (Bonferroni corrected).
To verify these patterns statistically, we defined an identity-regularity cost (idcost) as the difference between the values of  Figure 4B presents these pair-wise comparisons. . We found identity-regularity costs for saccades made to 323 surprising locations, but these saccades were also slower than saccades to expected or non-predictable locations. Therefore, to 324 determine whether the crucial factor for the impact of identity-regularity was location-suprisal or saccade-latency, we performed 325 for each participant median splits of the saccade latency distribution within each level of location (surprising, expected, or 326 nonRegular). We then evaluated if the identity-regularity cost was larger for the slower than faster saccades: for each condition  engaged in a cognitive process that produced a larger set of outcomes for expected trials than for surprising trials.

353
Variance of Decision threshold 354 We did not find any impact of Location or Identity on the variance of decision threshold (3×3 ANOVA, all Fs < 1, ps > .1).

355
This suggests that our experimental manipulations impact exclusively the evidence accumulation stage.  non-regular conditions, the mean probability of having the next target presented on the alternate side was 66%.

362
We examined whether gaze offsets could present a signature indicative of tracking location regularity, and whether identity 363 regularity impacted gaze offsets as well.

364
Steady-state analysis 365 To determine whether gaze tracked the location transition structure of the experimental series, we first constructed series 366 consisting of one gaze offset measure per trial. We then derived the power spectral densities of these series in the x − axis (see 367 Methods). This evaluates whether there is a recurrence frequency that shows particularly high power. 368 We focused our analysis on the power density at the frequency f tag = 1/3(trials −1 ). As can be seen in the power spectra 369 derived from the target-locations themselves (see Appendix and Appx.1), this is the frequency in which there is a peak when 370 location is regular. As shown in Figure 7A, the power spectra of the measured gaze-offset time series also peaked at f tag , 371 with apparently little impact of whether identity was regular or not. This suggests that anticipatory gaze locations tracked the 372 recurrence cycles of the Markov process that generated the targets' locations.

373
To evaluate these patterns statistically, we considered the frequency tagged response (FT R x at f tag = 1/3(trials −1 ), see 374 Methods), using a two-way ANOVA with two factors (Location and Identity) each with two levels (nonRegular, Regular).

375
The data analyzed are presented in Figure 7B. We found that only Location was significant, F(1, 39) = 12.16, p < .001.

387
This demonstrates that identity regularities modulated anticipatory gaze location prior to target appearance. responses. We found that learning the probabilistic transition structure of location and identity translates into interactive impacts 406 of location and identity predictability. Our findings further suggest that some of these interactive effects occur because learning 407 identity-regularities in and of themselves produce a cognitive load that impacts SL, as seen in the fact that identity regularities 408 negatively impacted saccades to surprising locations, and also impacted on fixation offsets before the appearance of the target. Anticipatory Fixation Offsets (AFOs) during fixation, prior to target presentation, in the direction of the expected target location.

420
In that study transition constraints held between two screen sides, whereas in the current study they held between four specific show that participants attended the series and followed instructions.

427
Identity-regularity impacts saccade latencies via specific and non-specific routes 428 Because we examined responses to predictable and less predictable stimuli in the context of learning we could differentiate 429 between two routes through which series-regularity could impact behavior: specific and non-specific effects. is no indication that learning has been translated into the sort of adaptive reactive behavior that improves stimulus-processing.

448
In contrast, specific effects suggest that individuals are using their knowledge in a way that impacts trial-level assessment and 449 in this way differentiates between expected and surprising events.

450
Non-specific effects of identity-regularity 451 Several indicators showed that participants were sensitive to the transition structure that existed between the identities of 452 the stimuli (here, images drawn from four categories). In the analysis of accumulation-rate, we found a non-specific effect 453 of Identity regularity: there was a main effect of Identity which did not track surprisal, and it was further modulated by a 454 significant interaction between Location and Identity. The interaction was produced because the existence of identity-regularities 455 strongly impacted saccades to surprising locations, but had no impact when targets were presented at expected or non-regular 456 (non-predictable) locations.

457
Because saccades to surprising locations were associated with the slowest latencies, we investigated whether the determining . We performed median splits on saccade-latency distributions for saccades to who found no indication that learning identity sequences interferes with learning of spatial sequences. That study used 531 orthogonal location/identity series to deterministically assign location/identity transitions. When location sequences are 532 deterministic, any subsequent spatial transition is by definition a high probability one. As shown in the current study, the 533 interaction between location and identity information was only found when saccading to surprising locations, in which case 534 regularity in identity series produced even slower responses. Thus, prior conclusions about non-interactivity could be attributed 535 to the use deterministic series that contained only high-probability (certain) transitions, and that did not produce a processing 536 bottleneck. 537

538
Our study shows how identity regularities interact with the saccade generation process. Departing from previous studies, we 539 chose to not include any decision about the image identity, but future studies could modify the paradigm to draw participants' We examined whether statistical regularities in object-location and item-identity produced independent or interactive processes. 549 We used a saccade-to-target procedure and we modeled saccade latencies as decisions about target location (ELATER).

550
Regularities in the identity stream decreased the rate of evidence accumulation when the target location was surprising.

551
Additionally, expected (vs. not expected) identities increased the variance of accumulation rate, indicating a larger range of 552 outcomes in the underlying decision process. Joint impacts of location-regularity and identity-regularity were also identified 553 in analysis of anticipatory fixation offsets. In all, our findings demonstrate a strong interaction between the processing 554 of regularities in object location and identity during stimulus-guided saccades, and suggest these regularities also impact 555 anticipatory, non-stimulus-guided fixations. In the condition with Regular location transitions, on each trial there was a probability of 66% of transitioning to one location, 585 33% probability of transitioning to another, and 0% transition to a third (in addition, repeats were never allowed). To understand 586 if anticipatory gaze offsets were aligned to the location of the most probably future target, we partitioned all the trials into 4 587 bins -right/left × top/bottom -depending on the most probable target location in the next trial. If participants' gaze tracked 588 the most likely future position then on first approximation, gaze location should show greater bias towards the right side when 589 the P = 66% transition is expected to be on the right than when the P = 66% transition is expected to ben the left, and similarly, 590 show greater bias towards the top/bottom of the screen depending of the expected vertical position of the next most probable 591 target. An Analysis of Variance with three factors (horizontal position of next most probable target, vertical position of next 592 most probable target, and screen side of just presented image) did not indicate that anticipatory gaze was biased towards the 593 horizontal position of the next most likely target, since the first factor and its interaction were not significant (F < 1). The only 594 significant effect was the screen side of previous screen F(1, 39) = 80.4 p < 10 −6 , because there was a general shift towards 595 the side opposite to that of the last presented target.