The Pupillary Light Response Reflects Visual Working Memory Content

Recent studies have shown that the pupillary light response (PLR) is modulated by higher cognitive functions, presumably through activity in visual sensory brain areas. Here we use the PLR to test the involvement of sensory areas in visual working memory (VWM). In two experiments, participants memorized either bright or dark stimuli. We found that pupils were smaller when a pre-stimulus cue indicated that a bright stimulus should be memorized; this reflects a covert shift of attention during encoding of items into VWM. Crucially, we obtained the same result with a post-stimulus cue, which shows that internal shifts of attention within VWM affect pupil size as well. Strikingly, pupil size reflected VWM content only briefly. This suggests that a shift of attention within VWM momentarily activates an “active” memory representation, but that this representation quickly transforms into a “hidden” state that does not rely on sensory areas.

cognition affects activity in visual sensory brain areas, which is subsequently "read 23 out" by the pupils. 24 For example, in several studies, participants were presented with both dark and 25 bright stimuli. Participants were subsequently cued to attend to either the bright or the 26 dark stimulus, without shifting their gaze (i.e. covert attention). Attending to the 27 bright stimuli resulted in smaller pupils than attending to the dark stimuli (Binda, 28 Pereverzeva, & Murray, 2013;Mathôt, van der Linden, Grainger, & Vitu, 2013;29 Naber, Alvarez, & Nakayama, 2013). Single-cell-recording studies have linked this 30 effect to the frontal eye fields (FEF), a part of the frontal cortex that is associated with 31 covert attention. Microstimulation of FEF results in increased attention to a specific 32 part of the visual field (Moore & Fallah, 2001). Crucially, if the stimulated region 33 corresponds to the location where a bright stimulus appears, the pupil constricts more 34 strongly than if the stimulus appears at a different, unstimulated location (Ebitz & 35 Moore, 2017). A similar effect has been reported for microstimulation of the superior 36 colliculus (SC), a midbrain region that is also associated with visual attention (Wang 37 & Munoz,in press). Taken together, both behavioral and neurophysiological studies 38 have shown that covert visual attention enhances the PLR. 39 A PLR can even be elicited without the physical presence of bright or dark 40 stimuli. In studies of mental imagery, participants were instructed to imagine stimuli 41 that had previously been presented with varying brightness levels. The size of the 42 pupil varied depending on the imagined brightness, with brighter objects resulting in 43 smaller pupils (Laeng & Sulutvedt, 2014). This effect was replicated with mental 44 imagery of real-life scenarios: Imagery of scenes like "a sunny sky" resulted in 45 smaller pupils than imagery of scenes like "a dark room" (Laeng & Sulutvedt, 2014). 46 These results are consistent with the finding that similar visual sensory areas are 47 active during perception and mental imagery of visual objects (Ganis, Thompson, & 48 Kosslyn, 2004). Presumably, the activity in visual sensory areas that is elicited by 49 mental imagery subsequently affects pupil size.

WORKING MEMORY CONTENT REFLECTED IN THE PLR
We conducted a linear mixed effects analysis (LME) on all trials (correct and 118 incorrect; analyzing only correct trials did not qualitatively change the results) with 119 Pupil Size as dependent measure, and two fixed effects, each containing two levels 120 (Brightness: Bright and Dark; Condition: Pre-Cue and Retro-Cue). We include by-121 participant random intercepts and slopes for all fixed effects. This analysis was 122 conducted for each 10 ms time window separately. We considered effects significant 123 if t > 1.96 (cf. Mathot et al., 2014;Mathôt et al., 2013), although we emphasize 124 overall patterns and effect sizes rather than significance of individual data points. 125 There was a significant interaction between Brightness and Condition between 5100 126 ms and 5590 ms, indicating that right at the start of the maintenance phase, the effect 127 of brightness depended on the condition (Pre-Cue or Retro-Cue). Subsequently, we 128 performed two separate LMEs for the two conditions (also run with all trials). 129 For the Pre-Cue Condition, there was a significant effect of Brightness from 130 4700 ms to 6990 ms, meaning that the pupil difference appeared during encoding of 131 the brightness-related stimuli and briefly persisted into the maintenance phase ( Figure  132 1a). Twenty-three participants (out of 30) exhibited the effect in the expected 133 direction (Figure 1b). In general, this means that when participants covertly attended 134 to the white circles on the encode screen, their pupils were smaller than when they 135 attended to the black circles. 136 In the Retro-Cue Condition, there was an effect of Brightness from 4900 ms 137 until 5790 ms (Figure 1c), directly corresponding to the maintenance phase. The 138 effect occurred in the expected direction for 19 participants (out of 30; Figure 1d). 139 This indicates that VWM content is reflected in the PLR not only during encoding but 140 also during maintenance. Phrased differently, shifting attention within VWM 141 representations (that are brightness-related) is reflected in pupil size, such that 142 internally shifting attention toward bright stimuli elicits smaller pupils than internally 143 shifting attention toward dark stimuli. 144 In summary, by using a retro-cue to shift attention within vWM representations, 157 we were able to examine the relationship between maintenance of brightness-related 158 content and pupil size while keeping stimulus encoding constant. Our results clearly 159 show that shifting attention to bright stimuli in vWM results in pupil constriction 160 relative to maintaining dark stimuli. 161 162

Experiment 2 163
In Experiment 2, we investigated if the relationship between the VWM content 164 and the PLR depends on whether VWM representations are in a high-or low-priority 165 8 WORKING MEMORY CONTENT REFLECTED IN THE PLR state, following single-item-template theories that postulate that only a single VWM 166 item can be in a high-priority state at a time, and that only this item is represented in 167 visual sensory areas (Folk & Anderson, 2010;Houtkamp & Roelfsema, 2009;168 Oberauer, 2002;Olivers, Peters, Houtkamp, & Roelfsema, 2011;Zokaei, Manohar, 169 Husain, & Feredoes, 2014), and thus affects the size of the pupil. We investigated this 170 by varying memory load. Participants were either asked to maintain one item or two 171 items during the retention interval. Single-item-template theories would predict that 172 the PLR would reflect VWM content only when participants maintained one item, 173 which was in a high-priority state, and not when participants maintained two items, 174 which would then compete with each other and both take on a low-priority state (cf. 175 Olivers et al., 2011;van Moorselaar, Theeuwes, & Olivers, 2014). 176 We designed two conditions: Set-Size-One and Set-Size-Two. The Set-Size-177 One Condition was an exact replication of the Retro-Cue-Condition ( Figure 2a). In the 178 Set-Size-Two Condition, we presented participants with four stimuli during the 179 encoding phase. The retro cue indicated whether the two circles on the right or on the 180 left were task relevant. During the response phase, participants had to indicate 181 whether the two new circles were both identical to the memorized circles, or if one of 182 them was different (Figure 2c). A Quest adaptive procedure (Watson & Pelli, 1983) 183 controlled for accuracy separately for the Bright and Dark Conditions as well as for 184 the Set-Size-One and Set-Size-Two Conditions to try to keep accuracy at a constant 185 75% in all conditions. In the Set-Size-One Condition, the mean accuracy for the bright 186 trials was 69% and 68% for the dark trials. In the Set-Size-Two Condition, the 187 accuracy was 74% for the bright and 71% for the dark trials. Brightness and Condition from 2100 ms to 2290 ms, which corresponded to the 196 encoding phase. However, no interaction could occur before presentation of the 197 targets, as participants were not aware which brightness would have to be encoded, 198 and hence this effect is necessarily spurious.)

WORKING MEMORY CONTENT REFLECTED IN THE PLR
Despite not finding a significant interaction in the overall model, we analyzed 200 the two Memory Load conditions separately. In the Set-Size-One Condition, we 201 replicated the significant effect of Brightness on pupil size between 5000 ms and 5290 202 ms, meaning that on average, the participants' pupils were smaller when maintaining 203 bright circles as compared to the dark at the beginning of the maintenance phase 204 (Figure 3a). However, it should be noted that this effect was weaker and present for 205 fewer (N = 18, of 30) participants than in Experiment 1. 206 The Set-Size-Two Condition revealed no significant effects of brightness on 207 pupil size (Figure 2c). However, because the effect was qualitatively in the same 208 direction as for the Set-Size-One Condition, and because the interaction between 209

Two trials. d) Shows the average effects of individual participants in the Set-Size-Two 224
Condition calculated in the same way as for the Pre-Cue Condition. In two experiments, we examined whether visual working memory (VWM) 229 content is reflected in the pupillary light response (PLR). Specifically, we wanted to 230 know whether maintaining bright stimuli in VWM is associated with smaller pupils 231 than maintaining dark stimuli. Overall, we showed that VWM content is reflected in 232 the PLR during both encoding and maintenance. Consistent with previous studies 233 ( Binda et al., 2013;Laeng & Sulutvedt, 2014;Mathôt et al., 2013), this shows that the 234 PLR, which was previously thought of as a simple reflex, is controlled by higher 235 cognitive processes, such as working memory. Our results further suggest that VWM 236 involves sensory representations, presumably in visual cortex (Yi et al., 2008), which 237 subsequently trigger pupil responses. 238 A striking aspect of our results is the time course (in the Retro-Cue Condition of 239 Experiment 1, and the Set-Size-One Condition of Experiment 2). The content of 240 VWM affected pupil size only briefly after the presentation of the retro-cue, rather 241 than throughout the entire retention interval. This was unexpected, considering that 242 we anticipated that this effect reflected maintenance of different brightness levels, 243 which should occur during the entire retention interval. However, this finding is 244 consistent with recent studies showing that VWM maintenance is not accompanied by 245 sustained activity in visual sensory brain areas, but rather that such activity is 246 periodical or transient (Rose et al., 2016;Sprague, Ester, & Serences, 2016;247 Sreenivasan, Curtis, & D'Esposito, 2014;Stokes, 2015;Wolff, Jochim, Akyürek, & 248 Stokes, 2017). Our finding that pupil size reflects VWM content only briefly may 249 reflect a transition from an 'active' state (which is reflected in pupil size) to a 'hidden' 250 state (which is not reflected in pupil size). This provides unique new support for the 251 notion of hidden VWM states, which has so far come primarily from decoding 252 analyses in brain imaging; however, decoding studies provide inconclusive evidence 253 for hidden states, because simulations indicate that re-emerging stimulus decodability 254 WORKING MEMORY CONTENT REFLECTED IN THE PLR in neuroimaging data could also reflect sustained neural activity (Schneegans & Bays, 255 2017). 256 We did not find a compelling dissociation between maintenance of one or two 257 brightness-related stimuli. Such a dissociation would be predicted by strong single-258 item-template theories, which hold that there can be only one active item in VWM at 259 a time, and that competition within VWM representations avoids any item from 260 becoming active when multiple items are kept in VWM (van Moorselaar et al., 2014). 261 We found qualitatively similar, but weaker effects on pupil size with a memory load 262 of two items, as compared to one item. Overall, our results suggest that whether a 263 VWM item is in an active or "silent" state depends strongly on time, and at most 264 weakly on memory load. The data was collected by recording the size of the right pupil of all of the 291 participants. The collection was done in a dark room and participants were asked to 292 place their head in a chin rest throughout the experiment. The task was designed with 293 OpenSesame 3.2.0 (Mathôt, Schreij, & Theeuwes, 2012), using the PyGaze plug-ins 294 for eye tracking (Dalmaijer, Mathôt, & Van der Stigchel, 2014). The stimuli were 295 presented on a monitor with LCD display with 60 Hz refresh rate and resolution of 296 1920 x 1080. 297

Procedure and Stimuli 298
Before the experiment started, the eye tracker was calibrated with a five-point 299 calibration procedure. Afterward, participants took part in a task in which they 300 memorized a particular brightness level of black and white circles that appeared on a 301 grey background (62 cd/m 2 ). Participants were instructed to keep their eyes focused 302 on a black fixation dot (2 cd/m 2 ) that was in the center at all times. This was ensured 303 by presenting a drift correction at the beginning of each trial, which paused the 304 experiment unless participants shifted their gaze back to the center. Participants had a 305 13 WORKING MEMORY CONTENT REFLECTED IN THE PLR chance to get familiar with the task during a practice phase (10 trials). Experiment 1 306 was composed of 16 blocks, each lasting 3.47 minutes at most (excluding the duration 307 of the drift corrections); the precise duration depended on the speed of responses. 308 Each block consisted of 16 trials, with half the trials belonging to the Pre-Cue and half 309 to the Retro-Cue Condition presented in random order. 310 In the Pre-Cue condition participants were initially presented with a cue (arrow 311 pointing to the left or right) indicating whether the stimulus on the left or right would 312 be task relevant. Subsequently two stimuli appeared (one black and one white circle), 313 one of which they had to encode. This was followed by a retention interval lasting for 314 4 seconds. During the response phase, participants were presented with a circle of the 315 same or a similar brightness as the one they had memorized. Participants had to report 316 whether the brightness of this circle was the same as, or different from, the one they 317 had memorized. The Retro-Cue Condition was almost identical to the Pre-Cue one, 318 however, the order of the cue and target were reversed. (For durations of individual 319 phases of the trial see Figure 1a and 1c.) 320 The targets for the bright and dark trials were selected from a specified 321 brightness ranges. The bright range extended from 88 cd/m 2 to 96 cd/m 2 , and the dark 322 range extended from 11 cd/m 2 to 19 cd/m 2 . A different response stimulus was brighter 323 on some trials and darker on others. The size of this difference was controlled by a 324 Quest adaptive procedure. It was implemented to control for participants' accuracies, 325 holding them constant at 75% for dark and bright stimuli separately. 326 After participants completed the task, they were asked about the strategies they 327 used throughout the experiment (see supplementary materials at 328 https://osf.io/ejxfa/). 329

Exclusion Criteria 330
For both conditions, trials in which the pupil at baseline was smaller than 2.1 331 mm in diameter or greater than 6.8 mm in diameter (N(trial) = 1) were excluded (as 332 values above these were clear outliers based on a visual inspection of the pupil-333 baseline histogram). Additionally, in the Pre-Cue Condition, trials were excluded if 334 participants horizontal gaze position deviated from the central band (between the 335 targets) position during the presentation of the encode screen (N(trial) = 522). No such 336 exclusion criteria were introduced for the Retro-Cue Condition, considering that 337 participants did not know which stimulus was the target during the presentation of the 338