Neuronal firing in the medial temporal lobe reflects human working memory workload, performance and capacity

The involvement of the medial temporal lobe (MTL) in working memory is controversially discussed. Critically, it is unclear whether and how the MTL supports performance of working memory. We recorded single neuron firing in 13 epilepsy patients while they performed a visual working memory task. The number of colored squares in the stimulus set determined the workload of the trial. We used the subjects’ memory capacity (Cowan’s K) to split them into a low and high capacity group. We found MTL neurons that showed persistent firing during the maintenance period. Firing was higher in the hippocampus for trials with correct compared to incorrect performance. Population firing predicted workload particularly during the maintenance period. Prediction accuracy of single trial activity was strongest for neurons in the entorhinal cortex of low capacity subjects. We provide evidence that low capacity subjects recruit their MTL to cope with an overload of working memory task demands. 1 Significance Humans are highly limited in processing multiple objects over a short period of time. The capacity to retain multiple objects in working memory is typically associated with frontal and parietal lobe functioning, even though medial temporal lobe (MTL) neural architecture seems capable to process such information. However, there are conflicting findings from patient, electrophysiological and neuroimaging studies. Here we show for the first time that correct performance, workload and individual performance differences are reflected in separate mechanisms of neural activity within the MTL during maintenance of visual information in working memory. The data suggest that low capacity subjects use the MTL to process the overload of information.

within the MTL during maintenance of visual information in working memory. The 23 data suggest that low capacity subjects use the MTL to process the overload of 24 information. 25

Abstract 26
The involvement of the medial temporal lobe (MTL) in working memory is 27 controversially discussed. Critically, it is unclear whether and how the MTL supports 28 performance of working memory. We recorded single neuron firing in 13 epilepsy 29 patients while they performed a visual working memory task. The number of colored 30 5 sessions, the median memory capacity was 3 (range 1.9-4.5) (Cowan's K,(hit rate + 128 correct rejection rate -1) *set size), which indicates that the subjects were able to 129 maintain at least 3 squares in memory. Separation of below and above memory 130 capacity set sizes was in practice the same as set size separation for low and high 131 workload. We chose the median capacity to divide subjects into a low and a high 132 capacity group (with 7 and 6 subjects, respectively). The mean response time (RT) 133 for the correct trials (2678 trials) increased with set size (Fig. 1c,118 ms/item,134 permuted repeated-measures ANOVA, F3,45 = 48.47; p = .0002). In sum, these data 135 show that our subjects were able to perform the task and that the difficulty of the task 136 increased with set size. 137

Neuronal firing 138
To find out whether MTL neurons participated in this task, we performed 139 multielectrode recordings of neuronal activity from microelectrodes separated into 140 single-or multiunit activity. We refer here to a putative unit by the term "neuron". We  Figure S1). Fig. 1d shows the microelectrode recording sites 148 projected on a parasagittal plane in the Montreal Neurological Institute (MNI) space. 149 In our first analysis of individual neuron types, we focused on the maintenance 150 period, when stimuli were absent, and the test period after the presentation of the 151 probe array. To reduce noise in the classification, unless stated otherwise, 152 subsequent analyses included only trials with correct responses. 153 First, we identified neurons that fired persistently in the maintenance period. These 154 maintenance neurons were defined by their higher firing rates during the 155 maintenance period than during the fixation period (permutation t-test, p = .0005, Fig.  156 2a, further examples in Supplementary Figure S2). Maintenance units were found in 157 the hippocampus, the amygdala and the entorhinal cortex (Fig. 2b). 158 When comparing high load vs. low load trials, we found a significant number of 159 maintenance neurons that showed an increase in firing with workload in the 6 hippocampus (10/55, permutation test against scrambled labels, p = .002, Fig. 2c) 161 and entorhinal cortex (5/21, permutation test against scrambled labels, p = .002). We 162 also found a significant number of maintenance neurons that showed a decrease in 163 firing rate with workload in the hippocampus (6/55, permutation test against 164 scrambled labels, p = .002). 165 When comparing correct and incorrect trials, we noted that correct performance was 166 associated with a larger number of neurons that showed enhanced persistent firing 167 during maintenance, i.e. that were classified as maintenance neurons. There were 168 more of these hippocampal neurons in correct trials vs incorrect trials ( maintenance support correct performance in low capacity subjects. 185 Next, we identified neurons, which fired specifically after the presentation of the 186 probe array that initiated the test period. These probe neurons were defined based 187 on increased firing only during the presentation of the probe array relative to 188 encoding and maintenance (permutation t-test, Fig. 3a). With a rate of 5.7%, these 189 neurons were rare (Fig. 3b).

Neuronal population analysis 191
We subsequently focused on the neural population firing rate during the periods of 192 the trial. We used demixed principal component analysis (dPCA) (38) to project the 7 firing rates from all neurons onto a low-dimensional component space. To demix the 194 effect of stimulus category, dPCA was informed by the workload of the trials. dPCA 195 clearly distinguished between set sizes 1, 2, 4, and 6 (Supplementary Figure S3). For 196 better visualization we formed a three-dimensional space from three demixed 197 principal components (dPCs 2, 3 and 4, explaining 36% of the variance, permutation 198 test against scrambled data, p = .002, Fig. 4a). The population activity distinguished 199 between set sizes very early during the trial as seen from the projection on the dPC3-200 dPC4-plane, which shows four angles of 90° each, i.e. the optimal balanced 201 distinction between the four set sizes. 202 In this dPCA neural state space, we included the first 15 dPCs that explain 81.4% of 203 the signal variance to analyze the rate of change over time (speed) (Fig. 4b). 204 Multidimensional speed was highest during encoding, when maintenance neurons 205 ramped up their firing. Speed was lowest during maintenance (permutation t-test 206 against encoding and test, p = .0005). Analyzing the pairwise distance between 207 trajectories that corresponded to different set sizes, we found that the distances 208 during maintenance were larger than during encoding (permutation t-test, p = .0005, 209

Decoding of information during the trial 216
We next asked, whether the neuronal activity was related to the individual subject's 217 working memory capacity. We analyzed the firing of each subject's neuronal 218 population on a trial-by-trial basis. Furthermore, we selected subpopulations that 219 consisted of, e.g., only hippocampal maintenance neurons. We trained a decoder on 220 a subset of trials and tested its performance on an independent set of test trials (39). 221 This assessed whether neuronal firing was sufficiently salient to be representative of 222 task demands or performance in single trials. Given the substantial variability in 223 neuronal dynamics during a trial (Fig. 4a-c), we focused on the maintenance period. 224 For the maintenance period, we found that the workload (set size 1, 2 vs. set sizes 4, 225 6) of each trial could be decoded with median decoding accuracy of 75% when all 8 neurons were pooled. (Fig. 4d, permutation test against scrambled data, p = .001). In 227 a leave-one-out analysis, the exclusion of any subject resulted in decoding accuracy 228 in the IQR [64%-75%], which verified that the accuracy was supported by all 229 subjects. When pooling neurons from the three anatomical regions separately, 230 decoding accuracy was significant only for entorhinal cortex neurons (Fig. 4d column 231 5), but not for neurons in the other areas. For the entorhinal cortex units, the 232 workload of each trial could be decoded by units from only low capacity subjects 233 (median decoding accuracy 73%, permutation test against scrambled data, p = .001, 234 Fig. 4d column 9), but not by units from high capacity subjects (median decoding 235 accuracy 58%, permutation test against scrambled data, p > 0.05). For low capacity 236 subjects, the prediction accuracy of entorhinal neurons was significantly better than 237 for high capacity subjects (Fig. 4d column 9 and 11, permutation t-test, p = .0005). 238 This points to a distinct recruitment of neural activity in low capacity subjects while 239 they cope with high load on working memory. 240

Discussion 241
We found changes in MTL neural firing patterns across neurons in the MTL and 242 particularly in the entorhinal cortex and hippocampus. These changes depended on 243 workload (the number of items in the array), capacity (whether the subject can 244 correctly detect changes from a small or large array of items) and performance 245 accuracy (whether the subject responds correctly to a change or no change). 246

MTL supports object separation in working memory 247
All four workload conditions could be separated by neural population firing in the 248 MTL, particularly during maintenance, where the components were stable and 249 distinguishable. Such a mechanism would allow for a reduction of noise between 250 separate items in memory and therefore allow for a reduced chance of errors. This 251 would fit with the hypotheses that the MTL can support object separation in working 252 memory (28, 29, 40). The results confirm and extend on the report that showed 253 workload dependent separation of neural firing patterns during maintenance of letters 254 in working memory (36), but contrasts with other findings in which separation of 255 neural firing patterns during maintenance did not predict load on complex stimuli in 256 working memory (30,31). This suggests that maintaining multiple simple items, 257 including objects and letters, in working memory can be characterised by separation 258 of neural population firing. 259

MTL is activated only in low capacity subjects 260
Workload during maintenance could further be predicted by the trial-to-trial variability 261 of neural firing patterns in the MTL. This was particularly the case in the entorhinal 262 cortex, and specifically in subjects with low working memory capacity. The entorhinal 263 cortex is a gateway to the hippocampus receiving information from the neocortex and 264 directing input via the perforant path to the dental gyrus of the hippocampus and is 265 thus the station where novel information is processed first in working memory (41). 266

Persistent firing and theta oscillations and the entorhinal cortex and hippocampus 267
have been suggested to support processing novel items in working memory (42,43). 268 This is achieved by means of phase locking high frequency neural firing to theta 269 oscillations (40,44,45). Our earlier report showed that verbal workload within 270 capacity limits could be predicted by trial-to-trial variability of neural firing in the 271 hippocampus (36). The current study used memory arrays of objects that were 272 designed to exceed working memory capacity. We suggest that good prediction of 273 workload in the entorhinal cortex in low capacity subjects reflects processing an 274 overload of information and continued memorization of the items during maintenance. 275 High capacity subjects seem more efficient in memorization, so that the entorhinal 276 cortex is less involved during working memory maintenance. 277

Correct performance associated with increased hippocampal neural firing. 278
A small portion of neurons in the hippocampus and entorhinal cortex increased firing 279 during maintenance of objects. Importantly, firing rate was higher for correct than for 280 incorrect trials by maintenance neurons, particularly in the hippocampus. We only 281 found this effect in low capacity subjects. This is a novel finding and shows that 282 persistent hippocampal neural firing is supportive of visual working memory 283 performance, particularly in low capacity subjects. 284

Interpreting persistent firing. 285
We suggest that in low capacity subjects, entorhinal and hippocampal persistent 286 firing is instrumental to maintain multiple objects in memory and to achieve correct 287 performance. Persistent posterior parietal activity is known to support performance at 288 high workload in high capacity subjects, but not in low capacity subjects (3, 46, 47). It 289 is thus tenable that low capacity subjects might use entorhinal and hippocampal 290 activity to compensate for impaired posterior parietal functioning. The findings extend 291 to a BOLD fMRI study with the same task showing a similar pattern for low capacity 292 subjects, except that high capacity subjects further increased activity at high 293 workload and that BOLD fMRI activity could not be univocally be identified as 294 maintenance activity (35). Further, our findings add to other single cell recording 295 studies. A verbal working memory task reported increased firing in a substantial 296 portion (~20%) of maintenance units with workload but performance could not be 297 predicted (36), which might be related to the lack of need to bind between object 298 features (color-location vs. letters) and associated higher capacity for verbal items. 299 Other studies showed that certain cells, so called concept cells, showed preferred 300 neural activity for specific complex pictures and that persistent neural activity of 301 concept cells predict correct behavior of a trial (30,31). In contrast to those studies 302 we found trial-to-trial variability of neural activity predicting workload in the entorhinal 303 cortex, and elevation of neural firing for correct trials in the hippocampus, while both 304 these effects were found particularly in low capacity subjects. This suggests that 305 persistent activity is a relevant marker not only for correct maintenance of items that 306 show category selective responses but also for workload and correct performance 307 during memorization of arbitrary arrays of simple objects. Both manifestations of 308 persistent activity reflected different cognitive phenomena and in different neural 309 structures, suggesting a separation of working memory operations within the MTL. 310

Conclusions 311
Our findings elucidate the role of persistent firing in MTL during human working 312 memory maintenance. We propose that subjects with low working memory capacity 313 recruit the persistent firing in their entorhinal cortex to cope with working memory 314 load and that the persistent firing in their hippocampus supports correct performance. 315 6 Methods 316

Subjects 317
Subjects were patients with epilepsy undergoing pre-operative monitoring with 318 implanted electrodes in the MTL (13 subjects, age 18-56 years, 7 male, Table S1). All 319 subjects had normal or corrected-to-normal vision and were right-handed as 320 confirmed by neurophysiological testing. 321

Task 322
Visual working memory was examined using a change detection task (2, 35), where 323 arrays of colored squares were presented and had to be memorized (Fig. 1a). The 324 set size (low workload: 1, 2 squares high workload: 4, 6 squares, total 192 trials per 325 session) determined working memory workload. In each trial, presentation of a 326 memory array (encoding period, 0.8 s) was followed by a delay (retention interval, 0.9 327 s). After the delay, a probe array was shown, and subjects indicated whether the 328 probe array differed from the memory array. A subject's working memory capacity 329 was estimated using Cowan's K (K = (hit rate + correct rejection rate -1)*N, where N 330 = the number of objects presented (1)). High and low capacity subjects were 331 separated on the basis of median split for Cowan's Kmax (Fig. 1b). 332

Recordings and analysis 333
We recorded with microelectrodes in the patients' medial temporal lobe. Spike sorting 334 was done using Combinato (48). Neurons active during multiple sessions entered the 335 analysis independently for each session. Fig. 1d shows the recording sites. We first 336 identified units that show higher activity during maintenance compared to fixation 337 using permutation tests against scrambled labels. These were termed maintenance 338 units. We also identified test units, which are units that show higher activity during 339 test compared to maintenance. 340

Multidimensional state space 341
To illustrate the firing rates of all neurons over the periods of a trial, we used dPCA 342 as a dimensionality reduction method (38). In contrast to PCA, which explains the 343 variance of components regardless of task conditions, dPCA incorporates the task 344 conditions, i.e., set size in our task, when explaining the variance. We used non-345 overlapping bins of 2 ms for neuronal firing and smoothed the rates in a window of 346 1000 ms with a Gaussian kernel. We z-scored the resulting rates based on the mean 347 of the firing rate during fixation. We computed time-dependent and set size-348 dependent dPCA components and ordered them by explained variance. To create a 349 distribution for multidimensional speed in different trial phases, we ran dPCA analysis 350 500 times, subsampling (20%, with replacement) neural firing rate trajectories for 351 each run. 352

Decoding of information at the trial level 353
We analyzed neuronal firing at the population level on a trial-by-trial basis. We 354 grouped the neurons according to their anatomical region (hippocampus, entorhinal 355 cortex or amygdala). We collected the rates of all neurons and all trials (1 ms bin, 356 1000 ms window). We trained and tested the classifier at every 100 ms using a 357