Frequency-tagging visual background information enables multi-target perceptual filling-in to be distinguished from phenomenally matched replay

Perceptual filling-in (PFI) occurs when a physically-present visual target disappears from conscious perception, with its location filled-in by the surrounding visual background. Compared to other visual illusions, these perceptual changes are crisp and simple, and can occur for multiple spatially-separated targets simultaneously. Contrasting neural activity during the presence or absence of PFI may complement other multistable phenomena to reveal the neural correlates of consciousness (NCC). We presented four peripheral targets over a background dynamically flickering at 20 Hz, to entrain neural populations responding to the background. While participants reported on target disappearances/reappearances via button press/release, we tracked neural activity associated with PFI using steady-state visually evoked potentials (SSVEPs) recorded in the electroencephalogram. Behaviorally, we found that as the number of filled-in targets increased, the duration of target disappearances also increased, suggesting faciliatory interactions among targets located in separate visual quadrants. We found background SSVEPs closely correlated with subjective report, and increased with an increasing amount of PFI. Unexpectedly, we found distinct spatiotemporal correlates for the SSVEP harmonics. Prior to PFI, the response at 40 Hz preceded the response at 20 Hz, which we tentatively link to an attentional effect. There was no difference between harmonics for physically removed stimuli. These results demonstrate that PFI can be used to study multi-object faciliatory interactions, and because there are distinct neural correlates for endogenously and exogenously induced changes in consciousness, it is ideally suited to study the NCC. Significance statement Perceptual filling-in (PFI) is a transient illusory disappearance of visual objects from consciousness. By holding the object constant, we can contrast neural activity during periods with and without PFI to isolate the neural correlates of conscious perception. Unlike traditional visual illusions, PFIs are subjectively crisp and simple, and can happen simultaneously at different spatial locations. By frequency-tagging the background display, we demonstrate graded neural correlates for graded changes in consciousness, and provide evidence to differentiate between the perceptual processes evoked during PFI from those evoked by the physical removal of the same peripheral stimuli.


89
In perceptual filling-in (PFI) phenomena, areas of the visual environment that are   Task procedure 167 Each experimental session was composed of 25 trials, 60 seconds per trial. Between 168 the trials, participants were able to take short self-timed breaks, resulting in a total time-on- 169 task of approximately 30 minutes. Before the experiment, participants were instructed to 170 fixate on the central cross, and were informed that they may sometimes experience a visual 171 illusion where any number of peripheral targets may disappear from their field of vision. 172 Participants then completed one practice trial to familiarize themselves with the 173 corresponding button presses required for targets in each of the four visual quadrants. 174 Specifically, they were instructed to press keys 'A', 'Z', 'K', and 'M' on a traditional  Catch periods 193 We introduced catch periods to check if participants were correctly reporting on 194 disappearances. During a catch period 1 to 4 targets were physically removed from the 195 display and replaced with the background through alpha blending. Each catch period lasted 196 from 3.5 to 5 seconds in duration (drawn from a uniform distribution). To mimic the 197 phenomenology of endogenous PFI events, we generated catch periods by linearly ramping 198 6 the luminance contrast of the target up or down over 1.5 seconds. Participants were not 199 informed of the catch periods. 200 These physical catch periods also served as a control condition for comparison with 201 the neural signals evoked by PFI. Within 24 trials, catch events in which one, two, three or 202 four targets were removed each occurred on six trials for each participant. The location of the 203 removed targets in the case of one, two and three targets were randomized. The order of these 204 catch events were also randomized for each experiment. A previous study showed that 205 flickering peripheral targets tend not to disappear in the beginning of trials (Schieting & 206 Spillman, 1987), so each catch event began no sooner than 10 seconds after the beginning of 207 each trial to ensure that catch disappearances remained indistinguishable from PFI. Our own 208 data also confirmed that participants reported much lower PFI in the initial 10 seconds of 209 each trial, with PFI plateauing after approximately 10-15 seconds. We also did not include 210 catches within the last 10 seconds. We note that for 10 of our 29 participants, four-target 211 catch periods did not occur due to a coding error, and instead all four targets remained on 212 screen, resulting in catch periods being presented on 92% of trials overall (over all N=29 213 participants). 216 Initial screening analyses sought to confirm whether participants were able to 217 simultaneously monitor the visibility of multiple peripheral targets using four unique buttons, 218 and perform this task accurately and in compliance with instructions. Due to a keyboard 219 malfunction, button press responses to three and four disappearing targets became 220 indistinguishable in our post-hoc analysis, and have been analysed together henceforth as "3 221 or 4 buttons pressed". In the subsequent analyses where the number of buttons pressed 222 mattered, we proceeded as if three buttons were pressed in these periods. 223 We analyzed button press responses during catch periods to estimate participant 224 attention on task. As catch periods were embedded within a trial, some catch periods occurred 225 when participants had already pressed buttons. Such events are more frequent for those who 226 report more frequent PFI. To estimate this baseline button press rate per individual 227 participant, we performed a bootstrapping analysis with replacement. For a given catch onset 228 in trial T at time S (seconds), we randomly selected a trial T' (T=T' was allowed) and 229 epoched the button press time course over the period of [S-2, S+4] at corresponding catch 230 target locations in trial T'. We repeated this for all trials (T=1...24, except for the 4-catch 231 error mentioned above) to obtain a single bootstrapped set of trials per participant. We then 232 obtained the mean button-press time course across button-locations from each of the 200 233 bootstrap sets to obtain a null distribution of the shuffled button-press time course. We also 234 obtained the mean button-press time course for observed data across button-locations, 235 excluding catch periods when four targets were removed.

236
As the distribution at each time point for both observed and shuffled data was not 237 normally distributed, we first converted the data into z-scores using the logit transformation 238 before calculating the confidence intervals (CI). Then, we used mean z-scores (±1.96 239 standard deviation of z-scores) as the CI for the null distribution of shuffled data within each 240 participant, and observed data across participants. We excluded 3 participants whose mean button-press time course around the actual 242 catch onset failed to exceed the CI within the first two seconds (i.e., [0, S+2]). We defined the 243 catch-onset reaction time as the first time point after which the mean button-press data 244 exceeded the top CI, indicating successful button presses for catch targets. Figure 2a shows 245 the catch response for an example participant retained for analysis. Four further participants 246 were removed from subsequent analyses for failing to experience PFI during most of the 247 experimental session (i.e., only brief events on 1 or 2 trials). For the remaining participants, 248 the mean reaction time to respond to catch onsets, and thus the disappearance of a peripheral 249 target was 0.92 seconds (SD = 0.046). Figure 2b shows the proportion of button press 250 responses for all catch events across participants retained for analysis (N=22). Having identified which participants could successfully indicate target disappearance 264 based on their button press data, we continued to identify and remove any trials from the 265 subsequent analysis in which a catch was not correctly detected. We undertook this procedure 266 to assure that in all retained trials participants paid proper attention on task and reported 267 accurately on PFI . We regarded a catch period as being successfully identified if participants 268 pressed the corresponding button for at least 50% of the allowed response time window. (For 269 multi-target catch periods, we applied the same criteria for each button separately. If any 270 button was not pressed at least 50% of the time, the catch was considered undetected. For 271 four-target catch periods, we analyzed it as if it was a three-target catch period). This window 272 was from the onset of the catch plus 1 second (in consideration of the reaction time delay) to 273 the end of catch. For example, if the catch period under consideration was 3.5 seconds in 274 duration, we defined the allowed time window to be [1, 3.5] seconds from the catch onset.  280 To investigate whether the simultaneous multi-target PFI observed in participant data 281 (e.g. Figure 1b) exceeded that to be expected by chance, we performed a shuffling analysis to 282 create a null distribution. Specifically, we created 1000 shuffled trials for each participant, by

294
All statistical analyses were performed using Matlab (ver: R2016b) and jamovi (ver 295 0.9). We used linear-mixed effect (LME) analysis to examine whether various PFI 296 characteristics (e.g., durations) were affected by the number of simultaneously invisible 297 targets (nPFI; n=0, 1, 2, 3 or 4), including intercepts for participants as a random effect. We  We also performed LME analyses to compare the slopes of observed and shuffled 302 data, when considering the effect of the number of simultaneously invisible targets on PFI 303 characteristics. For this analysis, we fit a linear model (1st order polynomial) to the observed 304 data across participants (N=22), and retained the slope (β) as our observed test statistic. 305 Similarly, we also fit the same linear model to each of n=1000 sets of shuffled data, each of 306 which was computed from the shuffled trials across N=22 participants. We shuffled the trials

348
After examining the topography of log SNR responses, we applied rhythmic proceeded by selecting broadband neural activity to construct reference covariance matrices.

370
Comparing signal to broadband activity has previously been shown to allow the 371 reconstruction of SSVEP signals using RESS (Cohen & Gulbinaite, 2017).

372
After epoching all data using the time-windows -3000 to -100ms and 100ms to 373 3000ms peri button press/release, we then constructed RESS spatial filters per participant, 374 avoiding catch periods. Critically, we performed the above procedure without distinguishing 375 whether targets were disappearing or reappearing due to button press or release in order to 376 reduce the possibility of overfitting. If we were to construct separate filters for periods around 377 the time of target disappearance and reappearance, then any differences between these 378 conditions could be due to differences in the obtained filters, or overfitting of the filters prior 379 to our condition comparisons.   schematic pipeline for this entire procedure is displayed in Figure 3.

418
To quantify the relationship between log SNR and the amount of PFI, we grouped 419 events when the amount of PFI was between 0 and 1, 1 and 2, or greater than 2. A median 420 split based on the amount of PFI resulted in similar data and subsequent conclusions.

544
Our presentation of the results will be structured as follows. First, we confirmed that 545 our overall SSVEP frequency tagging was successful ( Figure 5). Second, we checked if the 546 behavioral reports during catch periods were correlated with neural activity (RESS log SNR, 547 Figure 6). Third, we investigated the behavioral reports during genuine PFI events, and 548 focused on whether or not spatially separated PFI targets interact across visual quadrants 549 (Figure 7). Fourth, we then focused on RESS log SNR during PFI events, testing if the 550 amount of PFI correlated with the strength of frequency-tagged EEG activity induced by our 551 flickering background (Figure 8, Figure 9). Fifth, we devised a SNR reconstruction analysis 552 to estimate the influence of multiple PFI events in close temporal proximity on the RESS log 553 16 SNR ( Figure 10). Sixth and finally, we also found unexpected temporal ( Figure 11) and 554 spatial ( Figure 12) differences between PFI events and catch periods, with respect to the first 555 (1f) and second harmonic (2f) responses (log SNR) to background flicker, which we interpret 556 in our Discussion.

558 559
Successful frequency-tagging of dynamic background in PFI display: 560 We first investigated the log SNR of target (8, 13, 15, 18 Hz) and background (20 Hz)   were rare (only 2-3 events per trial; Figure 6a), when they happened, the event tended to be 642 sustained for a long duration (~2 sec, Figure 6b). As a result, the total duration of 3 or 4 target 643 invisibility (~8 sec per trial, Figure 6c) is comparable to that of 2 target invisibility and longer 644 than that of 1 target invisibility, which happened at the highest rate (8.5 events per trial, 4 645 seconds in total per trial). We formally tested this linear trend by LME analysis and    695 perception 696 After demonstrating that spatially distributed targets were interacting, strongly 697 implying the involvement of high-level neural mechanisms during PFI, we turned to the 698 neural correlates of PFI via EEG analysis of SSVEPs. We first visualized how the changes in 699 PFI were related to changes in the log SNR of background flicker using an event-by-event   that have investigated the neural correlates of bistable perception with a single target, our task 745 design allowed graded changes in consciousness to occur in close temporal proximity (< 1 746 second), and even to overlap (Figure 1b). To account for how much of the log SNR time 747 course could be accounted for by sequential responses, we performed an SNR-reconstruction 748 analysis; we used 75% of training trials to construct reconstruction kernels, and applied these 749 to the remaining 25% of test trials to predict the log SNR time course (Figure 4). We then 750 compared the predicted time course of log SNR with the actual time course around the button 751 press events in the test trials during genuine PFI and during catch periods. Figure 9 visualizes 752 the high quality of prediction for the genuine PFI ( Figure 9 e and g) and the poor predictive 753 quality for catch periods (Figure 9 f and h).

754
To quantify prediction accuracy as the degree of correlation between the predicted 755 and the observed time course, we calculated R 2 between the respective 6-second RESS log 756 SNR around button press/release events during genuine PFI and catch periods. For both 1f 757 and 2f, the predicted SNR was correlated more strongly with genuine PFI than the catch, for 758 both disappearances and reappearances (Table 1)  Next, we continue by analysing the timing of these relative changes during target 771 disappearance and reappearance in more detail, using a cross-point analysis.    were topographically distinct ( Figure 5b). As there is a nascent literature suggesting that 805 SSVEP harmonics may correspond to separate cognitive processes (Kim et al., 2007(Kim et al., , 2011, 806 we next investigated these spatiotemporal differences in more detail. lines) became larger than that during reappearances (dotted lines). This effect occurred from -813 0.67 seconds prior to subjective report (paired t-tests, p cluster <.001). Notably, these effects 814 occurred 1.06 seconds later for catch periods (Figure 11b, from 0.39 seconds, p cluster < .001).

815
For 2f (Figure 10a and b, magenta), the RESS log SNR also became larger during 816 disappearances than reappearances from -.97 seconds prior to report (p cluster <.001), and 817 again, were shifted roughly 1.36 seconds compared to the catch-related time course ( Figure   818 11b, from 0.39 seconds; p cluster <.001).

819
The observed divergence (0.3 seconds) in the crossover time for 1f and 2f seemed 820 quite large given that both 1f and 2f were evoked from the same stimulus, using identical  One potential factor that could have contributed to the difference in the crossover time 842 between 1f and 2f is a difference in the spatial filters used for 1f and 2f within RESS analysis.

843
In fact, when we focused only on the (non-RESS) log SNR from a single electrode (POz), the 844 difference in cross-over times between 1f and 2f was not significant at the group or 845 participant level (data not shown). Given this, we further analyzed whether the spatial 846 characteristics for 1f and 2f were also distinct without using RESS spatial filtering during 847 PFI.

848
Around the catch events, spatial correlations across 64 channels were constant ( Figure   849 12b). However, when targets disappeared during PFI, the spatial correlation between 1f and  We embarked to combine a multi-target perceptual filling-in (PFI) paradigm with 876 frequency-tagged EEG. This combination has revealed novel insights into the mechanisms of 877 PFI phenomena including unexpected asymmetric neural correlates for disappearances and 878 reappearances with respect to its relation with the amount of PFI (Figure 8) and 879 spatiotemporal distinctions between steady-state visual evoked potential (SSVEP) harmonics 880 (1f and 2f background responses, Figure 10 and 11). Here, we discuss these findings focusing 881 on several advantages of our experimental paradigm. The multi-target display also allowed us to have a more objective graded measure of 914 differences in the contents of consciousness (i.e., the amount of PFI), which revealed an 915 asymmetry between the neural correlates of disappearances and reappearances. At this point,

Multi-target PFI to track changes in conscious perception
we have no straightforward explanation for this. One possible explanation is the difference in 917 saliency between PFI disappearances and reappearances, as reappearances can be predicted 918 with higher spatial and temporal accuracy than disappearances. Increased spatial accuracy 919 follows from the fact that reappearances can only occur at locations where a target has 920 already disappeared moments prior. As the duration of PFI is also short compared to the 60-921 second trial (Figure 7), reappearances can also be predicted with greater temporal accuracy 922 than multi-target disappearances. Thus, PFI disappearances may be more unexpected than To better understand the mechanisms of this asymmetry, further studies employing a 930 paradigm that feature multi-target and graded conscious perception will be necessary.

932 933
Insights into PFI mechanisms 934 935 Our results are relevant to two popular models of PFI. The first is an isomorphic thus is rendered invisible.

947
Our results favour the isomorphic model, but not exclusively. We found a slow,  On the other hand, the symbolic model that suggests that filling-in happens in higher- across quadrants may point to a mechanism that facilitates perceptual grouping.

962
Grouping may also interact with attentional mechanisms. Indeed, attendingfg to 963 shared features such as temporal modulation has been shown to enhance the binding of 964 distributed visual regions into a perceptual group (Alais, Blake, & Lee, 1998). As attending to 965 shared features such as colour (Lou, 1999)  Another insight that arose from our application of SSVEP to study PFI regards the 980 difference in spatiotemporal profiles of 1f and 2f responses (Figure 10 and 11). This Here we extend efforts to refine NCC paradigms, by using PFI. Unlike traditional 995 stimuli, PFI has the advantage that perceptual changes can be easily mimicked physically, 996 and that participants can accurately report on multiple changes in consciousness occurring in 997 close temporal proximity without much training. While genuine PFI and physical catch 998 periods were phenomenally similar, we revealed significant differences in their respective 999 neural substrates through our SNR reconstruction analysis, and suggest that these differences there are significant differences in the dependence on the amount of PFI for disappearances,