Abstract
Episodic recall depends upon the reinstatement of cortical activity present during the formation of a memory. We identified dissociable cortical networks via functional connectivity that uniquely reinstated semantic content and temporal context of previously studied stimuli during free recall. Network-specific reinstatement predicted the temporal and semantic organization of recall sequences, demonstrating how specialized cortical systems enable the human brain to target specific memories.
Introduction
Episodic recall allows us to remember the past, bringing back memories from a specific place or time. This type of memory retrieval involves the reinstatement of encoding-related neuronal activity that codes for memory attributes 1 (e.g., a specific person 2 or place 3). Neural reinstatement has been proposed as a mechanism for targeting individual memories during memory search 4,5, and subsequent studies have demonstrated content 2,6,7 and context 3,8 reinstatement preceding memory recall. Thus, these two types of reinstatement may serve to target memories that contain certain content or are placed in specific contexts. However, because studies typically focus on either content or context reinstatement in isolation, we know relatively little about how these types of information reinstate across the brain.
Consistent with the longstanding distinction between episodic and semantic memory 9 neuroimaging and electrophysiological studies suggest that separable cortical systems support memory for the semantic meaning and the spatiotemporal context of experienced events 10–14. Specifically, researchers have a identified a posterior medial (PM) network of regions, including parahippocampal, retrosplenial, and posterior parietal cortices, that activates during the processing of contextual information 15,16. By contrast, other work has identified an anterior temporal (AT) network of regions that appears critical for semantic and conceptual memory 17,18. This network includes the ventral temporal pole and perirhinal cortices 19. Because these two networks interact with the hippocampus during memory formation and retrieval 14,20, we predicted reinstatement within these two systems would reflect either the content or context of retrieved memories.
Neuroimaging studies that measure population-level neuronal activity have demonstrated reinstatement across cortical systems during memory retrieval. For example, content-related patterns of spectral power from intracranial electroencephalography (iEEG) across prefrontal and temporal cortices reinstate prior to memory recall 6,21. Slowly-changing patterns of iEEG power in the temporal lobe, consistent with a representation of temporal context 5, reinstate during retrieval and account for temporal organization of recall sequences 8. Functional MRI studies, which provide greater sampling of cortical systems than invasive recording techniques, have demonstrated contextual reinstatement within the hippocampus 22 and functionally connected PM regions 23. Multiple regions within the PM network, including the angular gyrus and medial prefrontal cortex, reinstate event-specific patterns of activity 24, suggesting this network may be involved in representing the content of memories within a specific episodic context. It is possible that these findings stem from representations of content and context integrated within a single cortical system, coding the full range of attributes in memory. In contrast, the PM and AT networks may independently drive hippocampal-dependent retrieval, serving as distinct cortical pathways to recall.
To examine the contributions of cortico-hippocampal networks to recall behavior, we utilized a computational modeling technique originally developed to predict patterns of brain activity based on the semantic content of individual stimuli 25–27. This method takes advantage of the sensitivity of neural signals to semantic attributes of presented items. By learning how activity in the brain is shaped by semantic attributes, these models can reliably decode the semantic content of stimuli from an observed pattern of brain activity. We extended this technique to develop context-based models that were trained to predict patterns of brain activity based on the temporal context (i.e., at which point in time in the experiment) in which stimuli were presented. By applying these models to iEEG signals recorded while subjects performed a free-recall task, we tested whether content- and context-based memory representations are reinstated within distinct cortical systems: the AT and PM networks.
Results
We recorded iEEG from neurosurgical patients (n = 69) while they performed a free-recall task with items drawn from 25 distinct categories presented in same-category pairs (Fig. 1a). Patients recalled an average of 31% (0.02 SEM) of list items. Both the serial position and category of items influenced their probability of recall. We observed a primacy effect, as evidenced by a recall advantage for items presented in the first (mean 49%, 0.03 SEM; t68 = 9.2, P < 0.0001) and second (mean 38%, 0.03 SEM; t68 = 4.9, P < 0.0001) serial positions (Fig. 1b). To evaluate category-level differences in recall performance, we compared the proportion of items recalled from each category to the average across all items (Fig. 1c). We found patients had better memory for zoo animals (mean 42%, 0.03 SEM; t68 = 4.1, P = 0.0001) and weather (mean 39%, 0.02 SEM; t68 = 3.3, P = 0.001), and worse memory for electronics (mean 22%, 0.02 SEM; t68 = −4.4, P < 0.0001).
In addition to overall recall performance, the serial position and category of items influenced the order in which items were recalled. Recall clustering (e.g., consecutively recalling same-category items) indicates that the retrieval cue used during memory search targets certain properties of the studied items. Because no retrieval cues are provided to the subject during free recall, the cue must be internally generated by reinstatement of information present during the study period. To examine category clustering, we used a word embedding model 28 to derive a vector representation of each studied item. To illustrate the similarity structure derived from the word2vec model, we projected the 300-dimensional word representations onto a three-dimensional space derived using principal component analysis (Fig. 1d). We measured clustering with temporal and categorical factor scores, which indicate whether clustering is above or below chance levels 29(see Fig. 1e for example measures). Recall sequences exhibited both categorical (t68 = 30.7, P < 0.0001) and temporal (t68 = 10.8, P < 0.0001) clustering (Fig. 1f). These clustering effects remained after accounting for potential confounds due to the list structure (see Methods). Adjusted measures of categorical (t68 = 25.6, P < 0.0001) and temporal (t68 = 11.4, P < 0.0001) clustering also indicated significant levels of recall organization (Supplementary Fig. 2). Both recall performance and clustering demonstrated reinstatement of both semantic content and temporal context occurred during memory search.
PM and AT networks contain representations of context and content during encoding
To assess representations of temporal context and semantic content within PM and AT networks, we first defined these networks based on an independent resting-state fMRI analysis (Fig. 2a; see Methods for details) 30. This analysis follows the standard method of correlating the BOLD timeseries between a region of interest and other regions distributed throughout the rest of the brain 31. Regions with connectivity to temporal polar cortex defined the AT network. These regions included inferior lateral prefrontal cortex, anterior temporal cortex, and inferior angular gyrus (Fig. 2a, green). Regions with connectivity to the posterior angular gyrus defined the PM network (Fig. 2a, purple). Cortical regions within this network included the posterior cingulate cortex, precuneus, parahippocampal cortex, and posterior parietal cortex (see Supplementary Fig. 1 for details regarding electrode coverage).
Synchronous low-frequency activity across the brain signals memory processing 32, with theta oscillations responsible for interactions between hippocampus and cortex during memory search 33. Prominent theories propose synchronous interactions coordinate network communication 34, which would support hippocampal-dependent reinstatement of memory content. Thus, we predicted synchronous activity in both PM and AT networks during the free-recall task. We next tested the hypothesis that PM and AT networks defined by resting-state fMRI would predict correlations in low-frequency power during task performance. Previous work has established correspondence between networks defined by resting-state fMRI and iEEG 35,36. However, it is possible that different network structures emerged in iEEG signals during the recall task. For example, these two cortical systems could reconfigure into a single network. We predicted distinct networks to emerge as subjects perform the task, supporting the hypothesis that separable cortical systems are involved in the representation of content and context.
To examine the separability of PM and AT networks, we asked whether theta exhibited greater connectivity within than between the two networks. We measured connectivity by correlating spectral power across trials of the free-recall task. Such trial-by-trial variability in power reflects endogenous fluctuations across neural networks 35,37,38. Because these estimates of connectivity can be spuriously elevated due to proximity of recording sites 36 (see Fig. 3 for evidence of drop off in connectivity with inter-electrode distance), we used a bootstrap matching procedure to control for differences in distance between contacts located within the same network or across both networks (see Methods). Using this approach, low-frequency power was significantly more correlated within the AT network than between the PM and AT network (Fig. 2a). This effect held at the lower (3 Hz, t(64) = 4.97, P < 0.0001), intermediate (5 Hz, t(64) = 3.67, P < 0.0001), and upper range of theta (10 Hz, t(64) = 3.12, P < 0.0001). Within the PM network, we found evidence for synchronous activity near the upper end of theta (10 Hz, t(64) = 2.13, P = 0.04). We also explored whether network-specific coupling occurred at higher frequencies, spanning beta and gamma ranges. We did not identify any higher frequency effects within the AT network. Coupling in the low beta range was enhanced within the PM network (17 Hz, t(64) = 2.70, P = 0.009). These results show that changes in spectral power were coherent within each of these two networks, consistent with their identification as dissociable networks from resting-state fMRI.
Having validated the PM and AT networks in our iEEG recordings, we next sought to evaluate their respective roles in the reinstatement of memories’ semantic content and temporal context. To do this, we trained multivariate models to predict brain activity based on the semantic content and temporal context of items presented at encoding. We modeled changes in spectral power from 3 to 180 Hz at each recording site from either the content- or context-based attributes of each item (Fig. 2b). Using these models, we predicted the pattern of spectral power for each serial position or category. We compared these predictions to held-out items in a cross-validation procedure that tested whether it was possible to decode either the serial position or category from brain activity observed within a given network. In addition, applying these models to patterns of spectral power during memory search allowed us to compare reinstatement across the two networks.
We assessed each model’s decoding performance during the encoding phase of the experiment, as subjects studied list items. We used a bootstrap procedure (n = 1000) to estimate decoding performance from five randomly sampled recording sites within each network (this approach allowed us to control for greater electrode coverage in the AT network, see Fig. 1b and Methods for further details). Applying this technique, we reliably decoded the category of items in both the PM (median AUC = 0.54, Z = 6.9, P < 0.0001) and AT (median AUC = 0.54, Z = 7.1, P =< 0.0001) networks (Fig. 2c), but category decoding did not differ between the two networks (median ∆AUC = 0.002, Z = 0.8, P = 0.45). We also reliably decoded the serial position of items in both the PM (median AUC = 0.71, Z = 7.2, P < 0.0001) and AT (median AUC = 0.70, Z = 7.2, P < 0.0001) networks (Fig. 2d). Here, our serial position model achieved more accurate decoding in the PM than in the AT network (median ∆AUC = 0.002, Z = 2.1, P = 0.04). These bootstrap results provide a lower bound for the true decoding performance for a given network (see Supplementary Fig. 4) and confirm the representation of semantic and temporal information across the PM and AT networks during encoding.
Reinstatement of content and context in PM and AT networks
Having established the ability to decode temporal context (serial position) and semantic content (category) from encoding-period neural activity in the PM and AT networks, we next asked whether these two types of information reinstated during free recall. To measure reinstatement, we tested our category and serial position models on patterns of neural activity in the moments preceding recall (from 900 to 100 msec prior to vocalization, in 20 msec intervals; Fig. 3a). We observed reinstatement of temporal information in the PM network and semantic information in both the PM and AT networks. Temporal context reinstated in the PM network from 600 to 400 msec before overt recall (Fig. 3b, top panel). Content reinstatement was sustained throughout the pre-recall period (Fig. 3b, bottom panel) in both PM and AT networks. Moreover, these two forms of reinstatement differed between the two networks, with greater content reinstatement in the AT network (Fig. 3b; P < 0.05, FWER corrected). These findings demonstrate specificity of contextual reinstatement within the PM network. In contrast to previous studies that relied upon similarity in neural activity over time to identify contextual information 8,22,23,39,40, we demonstrate the ability to recover the serial position of recalled items from reinstatement in the PM network. We also show greater, sustained content reinstatement within the AT network. These unique patterns of reinstatement reveal a dissociation between content and context reinstatement across the PM and AT networks.
Our finding that reinstatement of temporal context is specific to the PM network relies upon chance-level decoding within the AT network. It is possible that by limiting the number of electrodes included in this analysis (which was necessary for comparisons between networks), we impaired our ability to detect reinstatement, particularly if information was represented in a distributed manner. To evaluate this possibility, we examined decoding performance after incorporating additional electrodes into our analysis (Supplementary Fig. 4). Adding electrodes produced more robust category reinstatement effects in both the PM (, P = 0.02) and AT (, P < 0.0001) networks. This analysis also revealed stronger context reinstatement in the PM (, P = 0.009) but not the AT (, P = 0.36) network when incorporating additional features. These findings provide additional support for the specificity of context reinstatement to the PM network and suggest that decoding accuracy was limited by anatomical coverage and the spatial resolution of recordings.
To rule out the possibility that epileptiform activity influenced our ability to detect reinstatement, we examined whether electrodes that were either within the seizure onset zone, exhibited inter-ictal spiking, or located near pathological tissue impacted decoding performance. We repeated our decoding analyses after excluding these electrodes and found a remarkably similar pattern of results (Supplementary Fig. 5). We found no evidence that signals from these electrodes impacted our ability to decode information at encoding (all , P′s > 0.61) or recall (all , P′s > 0.53). As such, reinstatement reflected changes in spectral content associated with physiologically and cognitively normal processes.
Cortical reinstatement has been shown to predict how memory search unfolds 4,21, with cortical representations providing a top-down cue for the memory system to retrieve specific information. As such, it is possible that the PM and AT networks are responsible for targeting stored memories with specific content- or context-based attributes. To test this hypothesis, we examined whether differences in reinstatement across the two networks predicted the tendency of subjects to organize their recall sequences along temporal or categorical dimensions. Consistent with prior work linking cortical reinstatement to memory organization 6,8, variability in the organization of a subject’s memory was predicted by cortical reinstatement during memory search (Fig. 3c). Greater reinstatement of context-based representations within the PM network tracked the tendency to consecutively recall items from nearby serial positions (r67 = 0.26, P = 0.028). Increased reinstatement within the AT network was associated with greater organization based on the categorical structure of the list (r67 = −0.35, P = 0.003). Representations encoded in these two cortical systems can differentially guide memory search, biasing how and what we remember.
Discussion
We report evidence that the reinstatement of content- and context-based information occurs within distinct cortical networks. Furthermore, the quality of reinstatement within a given network was predictive of a subject’s tendency to organize their memories along either temporal or categorical dimensions. These results help resolve conflicting viewpoints on how distinct neural representations contribute to memory search. Episodic retrieval occurs when a memory cue converges on the hippocampal formation, prompting associative recall 41. Multiple cortical structures have been proposed to represent temporal context, which cues episodic recall 42, including lateral prefrontal cortex 5 and parahippocampal inputs to the hippocampus 43. Despite evidence implicating these structures in representing temporal information 8,22,44–47, it has been unclear whether these brain regions are the primary drivers of episodic recall. Our findings of greater reinstatement of context within the PM network and content within the AT network resolve this ambiguity by characterizing the contributions of these cortico-hippocampal networks to memory search.
Our findings complement previous studies of the neural bases of memory search 6,8,21 that have linked neural reinstatement to the organization of human memory. Along with fMRI investigations of temporal coding in the MTL and cortico-hippocampal networks 22,23, these previous studies identified candidate regions for the representation of temporal context based on slow changes in the similarity of neural patterns over time. Here, our modeling approach enabled us to recover the temporal position of items within each list, irrespective of the list in which the item was encoded. This rules out the possibility that context-like signals emerge from the covert retrieval or maintenance of list items throughout study lists, an inherent limitation of prior work. Given the PM network specifically reinstates contextual codes that predict the temporal organization of recall sequences, it is likely that this network is necessary to target memories from specific episodic contexts. Because the present findings are inherently correlational in nature, future studies could provide causal evidence between network reinstatement and recall organization via noninvasive or direct electrical stimulation. Adaptation of recently developed closed-loop stimulation techniques 48,49 may allow stimulation during memory search to bias recall of information from a specific context.
Unlike reinstatement of contextual information that was specific to the PM network, we were able to decode the category of recalled items from patterns of activity in both networks. Converging evidence from fMRI studies suggests that regions within this network 50, including the angular gyrus 24, represent recollected content in an episode-specific manner. Our findings suggest that temporal coding may be a unique feature of this network, in contrast to representations coded in the AT network. Understanding the nature of these temporal representations, including where they originate and how they are integrated with other forms of episodic information remains to be determined.
Our ability to decode the content and context of items in memory reflects multiple neuronal processes. Specifically, the measures of spectral power we examined reflect a combination of broadband and oscillatory processes, including slower (2-4 Hz) and faster (5-10 Hz) theta rhythms that predominate hippocampal networks 51. Broadband shifts in power indicate when memory processing occurs, including concurrent decreases in low frequency and increases in high frequency content 52–54. These broadband shifts are associated with changes in neuronal excitability and correlate with firing rates of locally recorded neurons 55,56. Thus, increases in neuronal firing within attribute-specific cortical regions would support accurate decoding. At the same time, reduction of oscillations and increased asynchronous activity supports information coding within cortex 57,58. Changes in the amplitude of narrowband oscillations would therefore contribute to decoding performance, at timescales constrained by the speed of the oscillation. Understanding the relative contributions of oscillatory and broadband signals to memory reinstatement remains an important question for future research.
Neural models of memory search have suggested that an internal representation of context serves as the primary cue for hippocampal-dependent recall 5. Our findings argue for an alternative account regarding the neural basis of memory search. One possibility is that semantic representations within the AT system can cue memories in a context-independent manner, prompting recall of memories with similar semantic attributes. If content-based representations can guide memory search in this fashion, one would expect individuals to organize their memories based on semantic content, rather than the temporal order in which it was learned. Indeed, subjects who showed greater reinstatement in the AT network exhibited greater semantic and less temporal organization. This hypothesis is further supported by evidence that content-based activity in the inferior temporal cortex can be used as a top-down signal to bias retrieval to targeted memories 7. Along these lines, the contextual information represented across the PM network focuses retrieval to a specific temporal context. The effects observed here may reflect a general property of memory search, wherein cortical reinstatement serves as a mechanism to target memories based on network-specific representations.
Methods
Participants
69 patients (40 male; see Supplementary Table 1 for details) with medication-resistant epilepsy underwent neurosurgical procedures to implant intracranial electrodes (subdural, depth, or both) to determine epileptogenic regions. Data were collected at Dartmouth-Hitchcock Medical Center (Hanover, NH), Emory University Hospital (Atlanta, Georgia), Hospital of the University of Pennsylvania (Philadelphia, PA), Mayo Clinic (Rochester, MN), Thomas Jefferson University Hospital (Philadelphia, PA), Columbia University Medial Center (New York, NY), and University of Texas Southwestern Medical Center (Dallas, TX). Prior to data collection, the research protocol was approved by the institutional review board at each hospital. Informed consent was obtained from either the participant or their guardians.
Free-recall task
Each subject performed a categorized free-recall task in which they studied a list of words with the intention to commit the items to memory. The task was performed at the bedside on a laptop, using PyEPL software 59. Analog pulses were sent to available recording channels to enable alignment of experimental events with the recorded iEEG signal. Word presentation lasted for a duration of 1600 ms, followed by a blank inter-stimulus interval (ISI) of 750 to 1000 ms (see Fig. 1a). Each list contained items from three distinct categories (four items per category), with two same-category items presented consecutively. The total word pool consisted of 25 distinct categories, with individual items selected as prototypical items within each category 60. Presentation of word lists was followed by a 20 s post-encoding delay. Subjects performed an arithmetic task during the delay in order to disrupt memory for end-of-list items. Math problems of the form A+B+C=?? were presented to the participant, with values of A, B, and C set to random single digit integers.
After the delay, a row of asterisks, accompanied by an 800 Hz auditory tone, was presented for a duration of 300 ms to signal the start of the recall period. Subjects were instructed to recall as many words as possible from the most recent list, in any order during the 30 s recall period. Vocal responses were digitally recorded and parsed offline using Penn TotalRecall (http://memory.psych.upenn.edu/TotalRecall). Subjects performed up to 25 lists in a single recall session.
Behavioral analysis
To compute behavioral measures of temporal clustering, we used the temporal factor 29 score. Temporal factor measures the percentile rank of the absolute lag between successive recalls from the full distribution of available lags for items that have yet to be recalled. To measure category clustering, we computed a category factor which assumes that all items within the same category have a distance of zero and items of different categories have a distance of one. These metrics measure the degree to which recall sequences exhibit organization along temporal or categorical dimensions, with random recall sequences falling at the median of the distribution (i.e., 0.5).
We presented same category exemplars in sequential pairs, as in the sequence (A1, A2, B1, B2, C1, C2, B3, B4, …, A4) where Xi is an exemplar of category X. This list structure creates circumstances where random recall sequences produce factor scores that deviate from the expected value of 0.5. Consider the case where a subject recalls four items from a single category in an arbitrary temporal order. After recalling the first item, the subject could either recall the same-category pair (e.g., A2 following A1) or jump to one of the other same-category items (A3 or A4). Because more items are available for recall at long temporal lags, sequences generated with random temporal order produce a lower than expected temporal factor score of 0.45. Measures of category clustering can be similarly biased by the list structure. Consider the case where a subject serially recalls four items starting at an arbitrary position in the list, without regard to semantic information. Because same-category pairs are always recalled consecutively, the expected category factor score for these recall sequences is elevated to 0.64. These examples highlight that factor scores can deviate from 0.5 due to the list structure rather than true recall clustering. To rule out this potential confound, we performed a simulation-based control analysis where we generated null distributions from random recall sequences matched to both the number of items recalled and either the temporal or categorical clustering of the observed sequences. We standardized the observed clustering measures based on these null distributions (n = 1000), which indicate the amount of clustering expected by chance given the list structure. This analysis is sensitive to clustering that is not confounded by the list structure, such as temporal clustering across category boundaries and category clustering across large temporal lags.
Electrophysiological recordings and data processing
iEEG signal was recorded using subdural grids and strips (contacts spaced 10 mm apart) or depth electrodes (contacts spaced 3-10 mm apart) using recording systems at each clinical site. iEEG systems included DeltaMed & XlTek (Natus), Grass Telefactor, and Nihon-Kohden EEG systems. Signals were sampled at 500, 512, 1000, 1024, or 2000 Hz, depending on clinical site. Signals recorded at individual contacts were converted to a bipolar montage by computing the difference in signal between adjacent electrode pairs on each strip, grid, and depth electrode. Bipolar signal was notch filtered at 60 Hz with a fourth order 2 Hz stop-band butterworth notch filter in order to remove the effects of line noise on the iEEG signal.
Anatomical localization
Anatomical localization of electrode placement was accomplished using independent processing pipelines for depth and surface electrodes. Post-implant CT images were coregistered with presurgical T1 and T2 weighted structural scans using Advanced Normalization Tools 61. For patients with MTL depth electrodes, hippocampal subfields and MTL cortices were automatically labeled in a pre-implant, T2-weighted MRI using the automatic segmentation of hippocampal subfields multi-atlas segmentation method 62. Subdural electrodes were localized by reconstructing whole-brain cortical surfaces from pre-implant T1-weighted MRIs using Freesurfer 63, and snapping electrode centroids to the cortical surface using an energy minimization algorithm 64. Reconstructed surfaces were additionally mapped to a population-average surface 65 that we used to assign network membership based on resting state connectivity of cortical regions defined by a multi-modal cortical parcellation 30.
Network assignment and analysis
We assigned recording sites to networks of interest was based on resting state functional connectivity in an independent set of subjects from the Human Connectome Project 30. From the Glasser et al. parcellation, we assigned parcels with high connectivity to the posterior angular gyrus (area PGp) to the PM network and parcels with high connectivity to the ventral temporal pole (area TGv) to the AT network. Assignment was based on partial correlations between each parcel and the seed region, to provide a better estimate of direct network connections. Bipolar pairs were assigned to the nearest network if the bipolar centroid was within 8 mm of a parcel within either network, with the exclusion criteria that they could not be within 4 mm of the other network.
To evaluate the properties of these two cortical networks, we used a bootstrap sampling procedure to randomly sample from bipolar pairs within each network. In our assessment of functional connectivity, we controlled for the effect of distance between recording sites on functional connectivity. As the distance between recording sites is greater between rather than within functional networks (leading to a biased estimate of connectivity within each network), we controlled the distances within each distribution of connections (i.e., within AT, within PM, or between networks). For each subject, we randomly sampled connections between the two networks. Connections within a given network were sampled without replacement to match the distribution of between network connections by minimizing the differences in connection length. The functional connectivity within and between each network was computed as the Pearson product-moment correlation across encoding events in the experiment. This procedure was repeated 1000 times, and the intrinsic connectivity at a given frequency was estimated from the bootstrap distribution. In circumstances where electrode coverage prevented well matched samples subjects were excluded from analysis (N = 4).
A similar bootstrap procedure was used to estimate the decoding accuracy of models trained from a given particular network. Networks with greater electrode coverage are likely to have higher decoding accuracy due to the number of features alone. As a result, we randomly sampled five electrodes from each network prior to estimating the ability to decode content and context information from patterns of brain activity. This sampling procedure was repeated 1000 times per model evaluation, and the average performance across bootstrap distributions was used to indicate model performance for a given subject.
Spectral power
To compute spectral power during word encoding, we applied the Morlet wavelet transform (wave number 5) to all bipolar electrode EEG signals from the onset to the offset of stimulus presentation, across 8 logarithmically spaced frequencies from 3 to 180 Hz. Spectral power during recall was estimated from 900 ms to 100 ms preceding the onset of response vocalization for correct recalls. Recall events were required to be free of vocalization onsets in the preceding 1500 ms. Power estimates were log transformed and down sampled to 50 Hz. To avoid edge artifacts, we included buffers of 1000 ms surrounding events of interest during the computation of spectral power; mirrored buffering was applied to all retrieval-period data. Prior to modeling, all power estimates were standardized using the mean and standard deviation of each session.
Model fitting and testing
We constructed two models to predict stimulus-related patterns of neural activity as a function of either 1) the taxonomic category of presented stimuli or 2) the serial position of each presented item within the experiment. For the content model, we modeled the neural response as a function of 300 intermediate category features computed using the wrd2vec model 28 trained from Google News corpora. Category features were computed by averaging semantic representations across all words presented from a given category. This results in the construction of a high dimensional space that respects the semantic relationships between all of the categories. For the context model, we modeled the neural response to stimuli as a function of serial position within the list, across lists, and across sessions. For subjects in which only a single session was run, the regressor predicting the effect of session on neural activity was excluded.
We fit both the content and context models separately to each neural feature (i.e., spectral power at a given frequency and bipolar pair). Ridge regression was used to identify model parameters that minimized prediction error on the training data, which was randomly assigned using a 5-fold cross-validation procedure, holding out individual words. Within each training fold, we performed an additional 10-fold cross-validation procedure (i.e., nested cross-validation 66) to optimize the regularization coefficient used in each fold. Across the ten folds, we selected the regularization coefficient (from 50 potential parameters log-spaced from 10−2 to 1010) that minimized the mean squared error of the model predictions across the training set. The resulting set of model weights across neural features was used to decode either the serial position or category membership for unseen patterns of brain activity. We computed the probability that the observed brain data belonged to each of the 25 categories (or 12 serial positions) by computing the Pearson product-moment correlation between the observed data and the predicted pattern of brain activity for each model. A softmax function was applied to the resultant evidence for each class, and decoding accuracy was measured using area under the receiver operating characteristic curve (AUC) metric across all held out data.
To quantify reinstatement effects across each network, we applied models trained to predict patterns of brain activity observed during encoding of words to epochs just prior (from 900 ms to 100 ms before vocalization onset). For each sample within this window, we computed the ability of each model to decode either the category or serial position of recalled words in the validation data. The resulting AUC timeseries were smoothed with with a 7 ms FWHM Gaussian kernel prior to statistical analysis for noise reduction.
Statistics
One-tailed tests were used to assess differences versus chance performance in evaluating decoding accuracy and recall organization. Theoretical chance levels (e.g., an AUC of 0.5 one class vs. all others) were further tested by constructing null distributions via permutation of class labels (i.e., category or serial position).
To evalutate the effects of the number of features and potential epileptiform activity on decoding performance, we modeled classifier performance using linear mixed-effects models. Fixed effects included the number of electrodes sampled, and whether the electrode was localized to the seizure onset zone. Intercepts were allowed to vary, treating subject as a random variable. Inference was performed through model comparison using a likelihood ratio test, dropping the effect of interest. Model fits and normality of residuals were confirmed through visual inspection.
We corrected for multiple comparisons (across time and frequencies) using a nonparametric permutation procedure. In our analysis of intrinsic connectivity, we performed a nonparametric one sample t test by constructing a null distribution of the maximum t statistic across frequencies, assuming no difference in intrinsic connectivity for connections within and between networks. This assumption was satisfied by random sign flipping of observed values at the subject level. Significance of observed differences in connectivity strength within versus between networks were compared to a distribution constructed from 2000 random permutations, yielding two tailed significance with PFWE < 0.05.
Multiple comparison correction for network reinstatement followed a similar procedure. We identified significant (PFWE < 0.05) clusters in the moments leading up to recall using threshold-free cluster enhancement (TFCE) 67. The TCFE statistic was computed by taking the original test statistics over the pre-recall period and adjusting by weighting by the height (h) and cluster extent (e): where hk is one of k cluster forming thresholds (hk = h0, h + dh,…, hmax). Height (H) and extent (E) exponents were set to 2 and 0.5 respectively, with the step size in the cluster threshold (dh) set to 0.01. After computing TCFE statistics, we used previously described nonparametric t tests to identify significant clusters based on null distributions (n = 1000).
Given the limited availability of iEEG data, no statistical methods were used to predetermine sample sizes (number of subjects). The number of trials included in the experiment was determined by practical constraints of patient testing at epilepsy monitoring units. The subjects in this study were a subset of those included in a multi-site collaboration to investigate modulation of human memory via direct electrical stimulation. Subjects with at least 5 recording sites located within both the PM and AT networks who performed categorized free recall were included in the present study.
Data Availability
De-identified data is available at http://memory.psych.upenn.edu/Electrophysiological_Data.
Code Availability
Analysis code for model fitting and evaluation is available at http://memory.psych.upenn.edu/Electrophysiological_Data.
Author Contributions
M.J.K. and Y.E. designed the study; J.E.K. analyzed data and drafted the manuscript. J.E.K. and M.J.K. edited the manuscript. J.M.S. performed anatomical localization of depth electrodes. M.R.S., G.W., B.L., R.G., B.C.J., K.A.Z., and S.A.S. recruited subjects and performed clinical duties associated with data collection.
Competing Interests
M.R.S. is the principal investigator (PI) of research contracts with SK Life Science, Takeda, UCB Pharma, Neurelis, Eisai, Medtronic, Engage, and Cavion. M.R.S. serves as a consultant to Medtronic through Jefferson University which receives all compensation for these services. B.C.J. has research contracts with Neuropace, Medtronic, Sunovion, and Eisai. R.E.G. servers as a consultant to Medtronic, a subcontractor on the RAM project. R.E.G. receives compensation for these services; the terms of this arrangement have been reviewed and approved by Emory University in accordance with its conflict of interest policies. The remaining authors declare no competing interests.
Acknowledgments
We thank Medtronic and Blackrock Microsystems for providing neural recording equipment. We also thank Drs. Ethan Solomon and Nora Herweg for providing feedback on this work. This work was supported by the DARPA Restoring Active Memory (RAM) program (Cooperative Agreement N66001-14-2-4032). We are indebted to all patients who have selflessly volunteered their time to participate in our study. The views, opinions, and/or findings contained in this material are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Data were provided in part by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.