Abstract
Judgments of the present visual world are affected by what came before. When judgements of visual properties such as orientation are biased in the direction of preceding stimuli, this is called visual serial dependence. Visual serial dependence is thought to arise from mechanisms that support perceptual continuity: because physical properties of an object usually vary smoothly in time, perception might be accurately stabilized by smoothing the perceived features in time. However, mechanisms that support perceptual continuity should be object specific, because the orientation of one object is more similar to its own past than it is to the past of a distinct object. Thus, we tested the perceptual continuity explanation by comparing the magnitude of serial dependence between objects and within objects. Across three experiments, we manipulated objecthood by varying the color, the location, and both the color and the location of Gabor patches. We observed a serial dependence effect in every experiment, but did not observe an effect of objecthood in any experiment. We further observed serial dependence even when the orientations of two successive stimuli were nearly orthogonal. These data are inconsistent with explanations of serial dependence based on visual continuity. We hypothesize that serial dependence arise from a combination of perceptual features and internal response variables, which interact within a common task or decisional context.
The physical world changes gradually, and so consecutive states of the world are correlated. For a perceptual system that must infer the origins of sensory input (Clark, 2013; Friston & Kiebel, 2009), it may be adaptive to infer features of the present world to be similar to the recent past. This would lead perceptual report to exhibit a “serial dependence”, a bias to judge present stimuli to be more similar to recent stimuli than they actually were. Serial dependence has been observed for basic perceptual judgments: reports of Gabor patch orientation are biased toward the orientation of stimuli perceived in the previous seconds (Fischer&Whitney, 2014; Fritsche, Mostert,&de Lange, 2017). Serial dependence has also been observed across multiple levels of visual judgment including numerosity (Cicchini, Anobile,&Burr, 2014), facial identity (Liberman, Fischer,&Whitney, 2014), and attractiveness (Taubert&Alais, 2016; Taubert, Van der Burg,&Alais, 2016, Xia, Leib,&Whitney, 2016).
What is the origin and role of these serial dependence phenomena? One hypothesis is that serial dependence phenomena reflect the operation of a “continuity field”, a fundamental mechanism that leads object representation to vary in a continuous manner over time, thus maintaining perceptual stability (Fischer&Whitney, 2014). Here, we test a prediction of the “continuity field” hypothesis: if serial dependence phenomena arise from a mechanism that maintains perceptual continuity, then serial dependence should be specific to a particular object and its features, and should not transfer between distinct objects.
In order to study serial dependence in perception, researchers present series of stimuli that can vary along a continuous dimension and measure the extent to which stimuli from previous trials can influence the perception of the current stimulus. For example, one study had participants judge the orientations of Gabor patch stimuli that differed in orientation trial-by-trial (Fischer&Whitney, 2014). They found that the average orientation judgment for a Gabor patch on a given trial is biased towards to the Gabor patch’s orientation from the previous trial. They further found that no response was needed to the prior Gabor patch for the bias to appear and the bias appeared when they had participants complete a comparison task, suggesting that the bias was perceptual and not response based. Furthermore, they found that the bias followed attention around a display and dissipated as a function of how far stimuli appeared away from the center of attention. They interpreted these findings as evidence for a serial dependence mechanism that attaches previous perception of items in a given region to the current stimulus, promoting perceptual stability.
The continuity field is defined as the spatial and temporal region in which a given feature is attracted towards that feature of a previous stimulus. The idea of continuity fields as mechanisms for helping maintain perceptual stability has a notable relationship to the notions of feature integration (Treisman&Gelade, 1980) and object files (Kahneman, Treisman,&Gibbs, 1992). Feature integration refers to the idea that, since sensory features are coded in a parallel, distributed (Rogers&Mcclelland, 2014) fashion there must be a mechanism that integrates these independently coded features into individual, multi-feature objects. In testing this idea, Kahneman et al. gave participants a preview display consisting of placeholder boxes with a letter in each of them. The letters then offset before the boxes moved to new locations. Next, a single letter appears within one of the boxes. This letter could be the same letter that appeared within that box previously, the letter that appeared in a different box, or a letter absent from the preview display. Kahneman and colleagues hypothesized that if features are bound into specific object files there should be a larger benefit for identifying letters appearing within the same box twice (object specific preview effect) compared to responding to letters that had appeared in both displays, but in different placeholders (object non-specific preview effect), or letters absent from the preview display. The data supported this hypothesis across a range of conditions. The authors concluded that there is a feature integration mechanism that binds independent features into object files, leading to a cost when an unexpected letter appears in a placeholder.
How does the serial dependence phenomenon relate to object-specific preview effects? In both cases, past perception influences current perception, and in both cases these effects depend on the similarity of past and present stimuli. One difference between the phenomena, as noted by Fischer&Whitney (2014), is that repeating objects leads to more accurate object identification on the second presentation, while serial dependence studies show that current perception is made less accurate by previous experience. This difference is not conclusive, of course, because the object identification studies were based on categorical judgments about the second object, rather than measurements of features along continuous dimensions. Still, Fischer&Whitney (2014) tentatively suggested that the continuity field (used to explain serial dependence phenomena) may provide a basic mechanism necessary for the creation and updating of object files (used to explain object tracking phenomena).
If a continuity field introduces a bias to group and track object features, then it should not bias features of one single object in the same way as two different objects. Features of a single object are expected to be auto-correlated in time, and so perception may be usefully biased by assuming continuity. In contrast, features of distinct objects may be completely uncorrelated in time, and assuming continuity of features across objects can lead to erroneous perception. Thus, if serial dependence phenomena reflect the operation of a continuity field, then these phenomena should not transfer between distinct objects. To test the continuity field explanation, we therefore measured the magnitude of serial dependence while manipulating factors that cause stimuli to be treated as separate objects. If continuity field mechanisms drive serial dependence, then these effects should be limited to individual objects, or at least stronger within objects than across objects. However, if serial dependence is not affected by object membership, then it seems implausible that it arises from a perceptual continuity field, or from mechanisms that support object files or object tracking.
To test how serial dependence varies with object membership, we employed a variation of the Fischer and Whitney (2014) procedure in which stimulus features were manipulated so that sequentially presented stimuli were understood as being part of the same or different objects. In Experiment 1, we presented the Gabor patches in blue or yellow, reasoning that individuals would treat differently colored stimuli as different objects. In Experiment 2, we presented the target stimuli to the left or right of fixation randomly on each trial, reasoning that individuals would treat consecutive stimuli at separate locations as separate objects. Finally, in Experiment 3, we manipulated both stimulus color and location such that one color stimulus would always appear at one location and the second stimulus would always appear at the other location. In all cases, we asked participants to report the Gabor patch’s orientation and measured the extent to which any error was biased towards the orientation of the previous stimulus. To anticipate our findings: we found the serial dependence effect, but did not find any effect of whether the previous stimulus was likely to have been perceived as the same object or a different object.
Experiment 1: Manipulating Stimulus Color
In Experiment 1, a Gabor patch always appeared at the same location (within a block of trials) but its color could switch between consecutive trials. We asked: does the orientation of a stimulus on a given trial affect the orientation judgments of the next stimulus only when they are the same color (and might be the same object) or does it also transfer when the stimuli are two different colors (and are unlikely to be the same object)? If the serial dependence effect is sensitive to objecthood we predict a significantly reduced serial dependence effect when stimulus color switches across trials (Figure 1).
Methods
Participants
Seven undergraduate students from the University of Toronto participated in the experiment and we compensated them $10 per hour of participation. All participants provided informed consent prior to the experiment and reported normal or corrected-to-normal visual acuity and normal color vision. Experiments were done in agreement with the Declaration of Helsinki.
Stimuli and apparatus
Participants completed the experiment using a PC connected a CRT monitor (screen resolution: 1024 x 768; refresh rate: 85 Hz). We used Matlab by Mathworks with the Psychophysics toolbox (Kleiner, Brainard, Pelli, Ingling, Murray,&Broussard, 2007) for stimulus presentation. Stimuli appeared on a gray background. The target Gabor patch stimuli were sine wave gratings with a spatial frequency of 0.33 cycles per degree presented in a 1.5° SD Gaussian contrast envelope. We used white noise patches to decrease aftereffects. The white noise patches were smoothed with a 0.91° Gaussian kernel and windowed in a 1.5° SD Gaussian contrast envelope. The Gabor patches could be blue (RGB: 000, 000, 255) or yellow (RGB: 255, 255, 000). The fixation target was a white dot windowed in a 0.5° SD Gaussian contrast envelope. The response bar stimulus was a white 0.61° white bar windowed in a 1.5° SD Gaussian contrast envelope. The Gabor patch stimuli were presented 6° to the left or right of fixation (manipulated between blocks). Participants responded using the left and right arrow keys on a QWERTY keyboard. We presented the response bar at a random orientation. We had participants use chin and head rests to maintain an approximate viewing distance of 44 cm.
Procedure
We presented the fixation target at the center of the display, where it remained throughout the experiment. We instructed participants to maintain fixation throughout the experiment. After 2000 ms, a Gabor patch of random orientation was presented either 6° to the left or to the right of the fixation stimulus, on the horizontal midline of the display. The Gabor patch remained for 500 ms before offsetting and was followed by the white noise stimulus that remained for 1000 ms. After 250 ms, the response bar appeared at the same location as the Gabor patch stimulus. Using the left and right arrow keys, participants then rotated the response bar to match the orientation at which they perceived the Gabor patch stimulus. Once they were satisfied with the response bar’s orientation, they pressed the spacebar to end the trial.
Design
Participants completed eight blocks of 140 trials (70 yellow Gabor patch trials, 70 blue Gabor patch colors) over the course of three sessions on different days for a total of 1120 trials. Whether the Gabor patch appeared to the left or right of fixation was counterbalanced across trial blocks. The orientations of the Gabor patch and the response bar were determined randomly on each trial: the orientations were sampled independently from a uniform distribution on the range [0, 360) degrees.
Analyses
For each trial, n, of the experiment we defined two categorical predictor variables: one predictor tracked whether the color switched from trial n-1 to trial n; the other predictor reflected the difference in orientation between the stimulus on trial n and the stimulus on trial n-1. The orientation difference (orientation on trial n minus orientation on trial n-1) was binned into one of 12 bins, each of 15° width, with bin centers at: -75°, -60°, -45°, -30°, -15°, 0°, 15°, 30°, 45°, 60°, 75. The dependent variable on trial n was the “response error”, which was the subject-reported orientation on trial n minus the veridical presented orientation on trial n. If some form of serial dependence was present in the data, then the response error on trial n should depend on the stimulus that was presented on trial n-1. Thus, to test for serial dependence, we tested whether the response error was the same across all orientation difference bins. To instantiate the null hypothesis that the distribution of response errors have the same mean across all bins, we used a resampling procedure. For each bin, we (i) generated surrogate data for our bin by randomly permuting across trials the orientation difference value assigned to each trial and (ii) calculated the mean response error for this surrogate data. This surrogate sampling procedure was repeated 10,000 times to produce a distribution of surrogate values of the mean response error, under the null hypothesis that response errors do not vary across bins. If fewer than 5% of the surrogate mean errors in the null distribution were greater than the observed mean error, we took this as evidence that the response errors in that bin were different from the mean response errors across bins. This statistical procedure was performed for every orientation difference bin, using data pooled from color repeat and color switch trials. The analysis was then performed for each orientation bin, separately for color repeat trials and for color switch trials.
To measure whether repeat and switch trials were different from each other, we computed confidence intervals for the mean repeat-switch differences. Confidence intervals were computed separately within each orientation difference bin, using a bootstrapping procedure. For each iteration of this bootstrap sampling process: (i) we generated surrogate switch-repeat data by randomly sampling with replacement N random switch trials and N random repeat trial, where N was the number of color or repeat trials presented in the entire dataset for the experiment, pooling across all trials and subjects, and (ii) we computed the mean difference between the color and repeat values in the surrogate data. This two-step procedure was repeated 10,000 times to generate a distribution of mean differences. The upper limit of a 95% confidence interval was set as the mean difference exceeded by only 250 values in the null distribution, and similarly for the lower limit.
If the serial dependence effect is reduced by switching color, this should manifest as an overall reduction in the magnitude of response error. Under the serial dependence model, the response error is different for clockwise and counter-clockwise orientation differences. Therefore, if the serial dependence effect is reduced by switching color, this reduction in response error should occur in different directions for clockwise orientation difference trials and for counter-clockwise trials. We expected that the switch-repeat differences should be clearest at the orientations where serial dependence effects are largest: for orientation differences between 15 and 45 degrees (Fischer&Whitney, 2014). To directly test this phenomenon at the single-subject level: (i) within each subject, we extracted all data from the -15, -30, and -45 orientation difference bins (clockwise bins) bins and all data from the 15, 30, and 45 orientation difference bins (counter-clockwise bins); (ii) for both clockwise and counter-clockwise bins separately, the response error on the n-th presented color repeat trial was paired with the n-th presented color switch trial within a block; (iii) we computed the difference between all repeat-switch pairs; (iv) separately for the clockwise and counter-clockwise data, we computed 95% confidence intervals on the mean of the switch effects. Confidence intervals on the mean of each distribution were computed by generating a distribution of the means across 10,000 surrogate datasets, each generated by sampling with replacement from the empirical distribution. This process of computing means and confidence intervals on the switch effects was performed separately for each subject and separately for the clockwise and counter-clockwise bins within each subject.
Results
Overall, the mean response error was small (M = 0.83°) with small standard deviation (SD = 14.84°) reflecting accurate performance. As can be seen in the left panel of Figure 1, we observed an overall serial dependence effect. When stimuli on consecutive trials were nearly the same orientation, we observed little response error. As stimulus orientation became increasingly different from the previous stimulus orientation the response error became significantly biased in the direction of the previous stimulus orientation, consistent with the serial dependence effect. This was the case when considering all the data as well as considering only the repeat or switch trials. However, inconsistent with previous serial dependence research, the mean response error did not return back to baseline at either the -75° or 75° orientation bins. This was true when examining all data, and when examining repeat and switch trials only.
Of most relevance to our central research question: switching object color did not reduce the serial dependence effect. We observed no systematic differences due to color repeat/switch (Figure 2). The mean of repeat response error minus switch response error was not different from zero in the -75 (M = 2.27°, p = 0.0948), -60° (M = -0.47°, p = 0.7482), -30° (M = 1.16°, p = 0.3618), 0° (M = 1.38°, p = 0.1024), 15° (M = -0.47°, p = 0.6014), 30° (M = 0.86°, p = 0.3628), 45° (M = -1.55°, p = 0.2388), 60° (M = 0.01°, p = 0.9999), and 75° (M = -0.16°, p = 0.8858) orientation bins. There was a significantly larger serial dependence effect in the switch compared to repeat conditions in the -45° (M = 2.65°, p = 0.0072) and 15° (M = -2.99°, p = 0.0044) orientation bins.
Finally, we checked in individual subjects for an effect of color switching, focusing on the 15°-45° range of inter-trial shifts, for which serial dependence effects are largest. This analysis revealed no significant differences between repeat and switch trials for the clockwise or counterclockwise rotation bins for any participants (Figure 3). For all subjects, and for both the positive and the negative orientation bins, the 95% confidence intervals on the serial dependence effect overlapped for switch and repeat bins.
Discussion
The data from Experiment 1 revealed a serial dependence effect: orientation judgments were systematically biased toward the orientation of the previous stimulus. However, this serial dependence effect was not detectibly affected by whether the stimulus repeated or switched colors between trials. This finding is inconsistent with the notion that serial dependence reflects a mechanism for object-level perceptual continuity. A second, incidental finding, was that the response errors were significantly different from zero even at the ±75° orientation difference bins. This observation stands in contrast to both Fischer and Whitney (2014) and Fritsche and colleagues (2017) who found that the response error returned to zero or turned negative, in some cases. This may indicate that the interleaved color change manipulation is somehow affecting the serial dependence effect. It is unclear why more difference between stimuli would lead the serial dependence mechanism to apply across even more dissimilar stimuli.
Experiment 2: Manipulating Stimulus Location
Although we observed a serial dependence effect in Experiment 1, we also found that it existed even at the most extreme orientation bins, a result that differs from Fischer&Whitney (2014). Therefore, in Experiment 2 we used location rather than color to manipulate objecthood. By removing the color switch, we more closely approximate the Fischer and Whitney (2014) design, and are once again to test for an effect of objecthood. In particular, we used grayscale stimuli that could appear at either side of fixation. Since perceptual objects are addressed by their location (Kahneman, Treisman,&Gibbs, 1992), this manipulation may be more likely to lead to repeat - switch differences. Additionally, when the stimuli repeats at the same location, the design is essentially the same as Experiment 1 from Fischer and Whitney. This allows us to test whether that response errors continue to be biased at the ±75° orientation bins even while stimuli repeat in color and location as in prior studies.
Methods
Participants
Six undergraduate students from the University of Toronto participated in the experiment and we compensated them $10 per hour of participation. All participants provided informed consent prior to the experiment and reported normal or corrected-to-normal visual acuity and normal color vision.
Stimuli and apparatus
The stimuli and apparatus replicated those from Experiment 1, except that all Gabor patches were presented in grey.
Procedure and design
The procedure matched the procedure of Experiment 1 with the following exceptions. The stimuli were greyscale and could appear 7° to the left or right of fixation. Within each trial block of 140 trials the stimulus appeared on the left for 70 trials and the right for 70 trials and participants completed nine trial blocks rather than eight.
Results
Overall, the mean response error was small (M = 1.36°) with small standard deviation (SD = 11.59°) reflecting accurate performance. Once again, we successfully replicated the essential features of the serial dependence effect (Figure 4). When stimuli on consecutive trials were nearly the same orientation, we observed little response error. As stimulus orientation became increasingly different from the previous stimulus orientation the response error became significantly biased in the direction of the previous stimulus orientation. As in Experiment 1, the response error in the repeat condition did not return to the baseline mean response error in the - 75° orientation bin (M = 0.21°, p = 0.0156) though for the 75° orientation bin it approached the baseline error (M = 2.16°, p = 0.0665).
In relation to whether objecthood affects the magnitude of serial dependence: we did not observe any systematic differences due to the location switch (Figures 4). The mean repeat - switch response error was not different from zero in the -75 (M = 2.02°, p = 0.0720), -60° (M = - 2.44 °, p = 0.0632), -45° (M = -1.27 °, p = 0.1410), -30° (M = -1.83°, p = 0.1556), -15° (M = 0.24 °, p = 0.7768), 15° (M = 0.01°, p = 0.9999), 30° (M = -0.74°, p = 0.3676), 45° (M = 1.39°, p = 0.2088), and 60° (M = 0.07°, p = 0.9384) orientation bins. There was a significantly larger serial dependence effect in the repeat compared to switch condition in the -0° (M = 1.45°, p = 0.0104) orientation bin and a larger effect in the switch compared repeat condition in the 75° (M = -1.89°, p = 0.0234) orientation bin.
Finally, we looked at the repeat - switch difference within individual participants in the clockwise and counterclockwise rotation bins separately (using only the 15° - 45° bins within each). As can be seen in Figure 5, response errors in the positive and negative orientation bins were indistinguishable in the repeat and switch conditions for all subjects.
Discussion
As in Experiment 1, we observed a systematic serial dependence effect. We further found that the serial dependence effect was not detectibly different when the stimulus location repeated or switched across trials. In Experiment 1 we found larger effects in the switch condition in the - 45° and -15° orientation bins; in Experiment 2 we found that difference for the 75° bin, but nowhere else. The fact that these differences are small, and of inconsistent direction, combined with the absence of differences for any individual subject (Figure 5), suggest that there is no systematic effect of switching location. Given this, our next step was to test whether an even more salient manipulation of objecthood would reduce the serial dependence effect.
Experiment 3: Manipulating Stimulus Color and Location
While in Experiment 1 we manipulated objecthood by changing the color of the stimulus across trials and in Experiment 2, we did so my manipulating stimulus location. In Experiment 3, we manipulate both factors simultaneously in order to maximize the differences between the two objects and maximize the possibility of objecthood affecting the serial dependence effect. Either a blue or a yellow stimulus was presented on each trial, but a blue stimulus would always appear left of fixation and a yellow stimulus would appear right of fixation. Given the suggestion that objects are “addressed” by their locations (Kahneman, Treisman,&Gibbs, 1992) and color is typically a salient feature (Theeuwes, 1992), this procedure may lead to stronger object differentiation and should prevent the serial dependence effect from transferring between objects.
Methods
Participants
Seven undergraduate students from the University of Toronto participated in the experiment and we compensated them $10 per hour of participation. All participants provided informed consent prior to the experiment and reported normal or corrected-to-normal visual acuity and normal color vision.
Stimuli and apparatus
The stimuli and apparatus replicated those from Experiment 1.
Procedure and design
The procedure matched that of Experiment 1, except for the location of the stimulus. Rather than blue and yellow Gabor patches appearing at the same location on each trial, each color always appeared on one side of the fixation stimulus or the other. That is, for example, the blue stimulus always appeared on the left and the yellow stimulus always appeared on the right. Whether the left/right stimulus was blue/yellow was alternated across trial blocks. Observers completed nine blocks of 140 trials, leading to a total of 1260 trials per observer.
Results
Overall, the mean response error was small (M = 0.11°) with small standard deviation (SD = 11.82°) reflecting accurate performance. As can be seen in the left panel of Figure 6, we again found a serial dependence effect. Once again, the overall response error in the ±75° orientation bins did not return to baseline. This failure to return to baseline was observed when considering all of the data as well as when analyzing only repeat or switch trials separately.
As for the repeat - switch difference, Figure 6 (right panel) illustrates that the repeat - switch response error was not different from zero in the -75 (M = -0.16°, p = 0.8278), -45° (M = 0.97°, p = 0.2422), -30° (M = 1.13°, p = 0.2150), -15° (M = -0.77°, p = 0.3146), 0° (M = -0.44°, p = 0.4466), 15° (M = -0.57°, p = 0.5124), 30° (M = -0.76°, p = 0.3774), 45° (M = -1.36°, p = 0.3774), 60° (M = -0.32 °, p = 0.7206), 75° (M = -0.56°, p = 0.5156) orientation bins. There was a significantly larger serial dependence effect in the repeat compared to switch conditions in the - 60° (M = 2.46°, p = 0.0038) orientation bin.
Finally, we again examined the repeat - switch difference within individual participants in the clockwise and counterclockwise rotation bins separately. As can be seen in Figure 7, this analysis revealed no significant differences between repeat and switch trials for the clockwise or counterclockwise rotation bins for any participants.
Discussion
The results of Experiment 3 were consistent with those of Experiment 1 and Experiment 2. We observed a serial dependence effect that was object non-specific even while those objects were differentiated by both color and space. In addition, the mean response errors remained significantly different from zero even at the ±75° orientation difference bins.
General Discussion
Previous studies have observed serial dependence in perception. These dependences are hypothesized to result from continuity fields: regions of space in which perceived stimulus features are biased in the direction of previous stimuli within that space. In this way, the continuity field can support the continuity of perception over time. However, to be adaptive for visual perception, the biasing effect should be applied only to stimuli that are sufficiently similar to previous stimuli within the field. If features from one object ubiquitously biased the features of a separate object, then the continuity field would impair the accuracy of perception and action. Therefore, we tested whether the continuity field was invariant to object-level similarity by manipulating factors that cause stimuli to be perceived as different perceptual objects.
Our study produced two consistent findings. First: we observed no reliable reduction in serial dependence between pairs of stimuli that should be perceived as different objects. We manipulated color (Experiment 1), location (Experiment 2), and color and location simultaneously (Experiment 3), and for each manipulation we observed that serial dependence was indistinguishable for consecutive stimuli that should be perceived as the same object or as different objects. Second, we found evidence the serial dependence effect arises from an interaction between perceptual features and internal response variables: responses were weakly biased towards the previous stimulus even when that previous stimulus had a very dissimilar orientation from the current stimulus. This was the case in all three experiment, including the repeat condition of Experiment 2 which should closely approximate Experiment 1 from Fischer and Whitney (2014).
If the serial dependence effect is object nonspecific and spatially nonspecific, and if it persists (if only weakly) for consecutive stimuli with very different orientations, this suggests that it arises from interactions between perceptual features and internal response variables. Before we attempt to specify what kind of process this could be, it is useful to review the existing evidence motivating both perceptual and post-perceptual explanations of the serial dependence phenomenon.
One motivation for the current study was that the original explanation for serial dependence emphasized perceptual contributions, via the notion of a continuity field (Fischer&Whitney, 2014). If the visual system instantiates an assumption that stimuli appearing close together in space and time are the same object and should look similar, this would enhance continuity of perception, especially under noisy conditions. The continuity field is meant to reflect the region of space in which the visual system employs this assumption of continuity. So, when a Gabor patch appears at a given location and then offsets, and when another Gabor appears at the same location shortly after, the continuity field will pull the second Gabor’s orientation towards the first’s orientation. It was natural to interpret the continuity field as arising from a perceptual process, because the magnitude of serial dependence depended on the similarity of the present and prior stimulus features. In the experiments reported in this manuscript, we did not observe serial dependence returning to baseline for near-orthogonal stimuli, but we did observe that serial dependence depended on similarity: the strongest effects was observed for 15-45° orientation differences, as in prior studies. Any explanation of serial dependence must account for why the effect varies with stimulus similarity; this can be most naturally accounted for in perceptual explanation.
A second motivation for a perceptual continuity field hypothesis is that serial dependence is reduced when objects pass behind an occluder and reappear at an inconsistent location, compared to when they reappear at a consistent location (Liberman, Zhang,&Whitney, 2016). This finding is consistent with the notion that serial dependence arises from a mechanism supporting object continuity. The authors suggested that little or no serial dependence should be found for highly distinct objects such as those used here.
A third motivation for a perceptual interpretation of serial dependence is that the effect persists even when a motor response is not required. Fischer&Whitney (2014) interleaved trials that did and did not require a motor response. They found that serial dependence was equally large when no motor response was produced for the preceding stimulus. These data do not rule out the influence of response selection (because participants did not know whether they would need to respond until after each stimulus), but it indicates that motor execution is not required for serial dependence.
Although the perceptual interpretation of serial dependence is empirically supported by the three findings noted above, there are also challenges to this interpretation. A post-perceptual interpretation of serial dependence was recently proposed by Fritsche et al. (2017). In their Experiment 3, they alternated between adjustment trials (participants saw a Gabor patch followed by a bar that they adjusted to match the stimulus orientation) with same-different judgment trials (participants saw two stimuli and needed to report whether they were the same or different). Fritsche et al. reasoned that if the serial dependence effects are perceptual, they should appear on the same-different judgments, a reporting method that is more bias free than the method of adjustment (Schneider, 2006). However, they found that orientation perception on same-different trials was actually repulsed from the previous stimulus’s orientation, a spatially specific effect that is consistent with previous research (Campbell&Maffei, 1971; Gibson&Radner, 1935).
In addition, Fritsche et al. found that serial dependence was only observed for sequential judgments that were linked by a common task. They found that prior stimuli in the same-different response task did not influence the subsequent adjustment responses even when a stimulus had appeared in the current stimulus’s location recently. However, adjustment responses (which were not affected by the immediately preceding stimuli from the same-different task) were affected by the more temporally distal previous adjustment response, despite a same-different trial occurring in between the two adjustment trials. This task-selective effect on adjustment responses was spatially non-specific.
Finally, Fritsche et al. (2017) observed that the magnitude of the serial dependence effect was increased when the delay between perception and judgment was increased. This observation is inconsistent with a mechanism that enhances the spatiotemporal continuity of perception. Altogether, the data of Fritsche et al. demonstrate (i) a spatially specific perceptual effect that is consistent with established perceptual phenomena and (ii) a spatially non-specific post-perceptual bias which resembles the original serial dependence effect.
Our current findings are also inconsistent with a perceptual origin for serial dependence. The serial dependence effect was unaffected when stimuli changed color and/or location across trials. If a continuity field biases perception of distinct objects as if they were the same object, this would appear to be disadvantageous for accurately comparing and successfully distinguishing objects. Additionally, our data indicated that serial dependence was observed even for dissimilar consecutive stimuli. Altogether, these findings - spatial non-specificity, object non-specificity, and serial dependence for dissimilar features - are most readily accounted for by a model positing an interaction between perceptual features and internal response variables.
Below we sketch a toy model that formalizes our understanding of the perceptual and response based influences on the serial dependence phenomenon.
We model the behavioral judgment on each trial of our task as a joint function of perceptual features and an internal response threshold (Figure 7). The immediately perceived stimulus features are described by a continuous function, sn(θ), representing the strength of each orientation, θ, in the stimulus presented on trial n. The internal response threshold is likewise a continuous function, rn(θ), describing the internal strength (or response bias) on that trial for each orientation, θ. The behavioral judgment, jn, that is produced on each trial is sampled with probability p(jn = θ) = f(sn(θ), rn(θ))where f is a probability function that increases for increasing s and for increasing r. In the simplest case, f(s(θ), r(θ) ∝ s(θ) + r(θ), with a normalization to ensure that fsums to unity.
We make two key assumptions for our model. First, we assume that when a stimulus of orientation θ* is presented on trial n, then sn(θ) becomes a symmetric distribution (such as a Gaussian or central Cauchy distribution) centered on θ*. Second, we assume that on each trial, the internal response threshold1 is updated as a linear combination of its prior state and the prior stimulus: rn(θ) = a rn-1(θ) + (1- α)sn-1(θ). Here, a is a parameter between 0 and 1 that modulates the timescale of serial dependence. When α = 0, then the response threshold on each trial will be modulated only by the stimulus on the immediately preceding trial; when a > 0, then serial dependence may be observed over many consecutive trials.
Figure 8 illustrates how this model works. On the first trial the internal response threshold, r1(θ) is a uniform function. Upon presentation of a stimulus with true orientation 
, the stimulus activation, s1(θ), becomes a Gaussian distribution centered on 
. The function describing the behavioral response p1(0)is then also approximately Gaussian and centered on 
(panel a), because the response threshold function, r1(θ), is flat. However, for the trial following, the response threshold function, r2(θ), is no longer flat (panel b). It is centered on 
. Therefore, if the stimulus orientation on the second trial, 
, is greater than (clockwise from) the stimulus on the first trial, then the behavioral response function p2(θ) will reflect a combination of the previous stimulus and current stimulus, and will be centered on a value between 
and 
. This shifting of the mode and expectation of p2(0) manifests as a serial dependence effect.
The model sketched above would still need to account for why serial dependence is reduced when an object moves behind an occluder and reappears at an inconsistent location. We propose that the internal response threshold is reset at so-called “event boundaries”, which occur when a surprising event occurs in the environment. This possibility is supported by findings of cognitive effects being eliminated or reversed if the two relevant events are more strongly differentiated in some way (Akylirek, Taffanin,&Hommel, 2008; Milliken, He,&Spadaro,2012; Pfister, Kiesel,&Mechler, 2010) and is also consistent with the literature on event boundaries in learning and memory (e.g. Botvinick, 2012).
Recently, Fritsche Mostert,&de Lange (2017) demonstrated that the serial dependence effect increased as the duration between stimulus presentation and the response increased. They interpreted this finding as evidence against a perceptual explanation of the serial dependence effect as the perceptual evidence is the same in both cases and evidence that during the retention interval the working memory bias becomes increasingly biased towards the previous stimuli (Huang&Sekuler, 2010). Such an interpretation is certainly possible within the current model if it is the case that the contribution of r1(0) increases relative to the contribution of s1(0)as the interval between the stimulus and response increases (Pertzov, Bays, Joseph,&Hussain, 2013; Wei, Wang,&Wang, 2012) which would cause the response threshold to be passed at a orientation further away from the true orientation. Teasing out the effects of time on the representation of stimulus and response variables seems an important subject for future work.
In summary, in three experiments we tested the object specificity of serial dependence. We reasoned that if the serial dependence effect arises from perceptual mechanisms, such as a perceptual continuity field, then it should only transfer between stimuli that could be reasonably inferred to be the same object across time. In contrast to this prediction, we found no evidence of the serial dependence effect being object specific nor spatially specific when objects are differentiated by color, location, or both color and location simultaneously. This non-specificity raises the possibility that the serial dependence effect is not simply perceptual. We sketched a toy model to account for the serial dependence effect via an interaction between the stimulus-driven response activation and trial-to-trial changes in a feature-specific internal response threshold.
This model can account for the serial dependence effect in experiments using a single simple feature dimension such as orientation, but it can also conceivably be extended to more abstract feature spaces (characterizing, say, attractiveness and emotion) and to multi-dimensional stimuli.
While we have concluded that the serial dependence effect results from interactions between perceptual features and internal response variables, it remains possible that serial dependence is also influenced by perceptual mechanisms. If there is a perceptual serial dependence mechanism, however, the visual system must balance the integration of stimuli with the necessity to detect novel stimuli. That is, both the over-integration and the over-differentiation of incoming visual input would be functionally problematic for an organism (see also: Kiyonaga, Scimeca, Bliss,&Whitney, 2017). Future research should look to determine the boundary conditions under which integration and differentiation of features are applied across objects. One possibility is that the serial dependence mechanism is applied to processing that is dominated by the object-oriented ventral stream while a change detection mechanism is applied more to processing along the action-oriented dorsal visual stream (Goodale&Milner, 1992).
Finally, we wish to emphasize that even though the serial dependence phenomenon incorporates task and response variables, this hardly decreases its significance. Phenomena arising at the boundary of perception and action in temporally dynamic contexts are of special relevance for understanding mental function and behavior in a continuously changing world.
Footnotes
↵1 A critical component of this model is that we assume that the response selection process, and not executing the response, that leads to the bias. This assumption is based on a substantial amount of theory and research regarding the interaction of stimulus and response factors (Gozli, Huffman,&Pratt, 2016; Hommel, Mlisseler, Aschersleben,&Prinz, 2001; Kunde, 2001; Schumacher&Hazeltine, 2016).
