Structuring time: The hippocampus constructs sequence memories that generalize temporal relations across experiences

Abstract

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge.

Introduction

Our memories are not veridical records, but constructions of our past¹. When constructing scenarios of the past or future, we often combine specific episodic details with general, semantic knowledge^2–7. For example, we can infer the time when an event took place not only from episodic details but also from associative or contextual information and general knowledge^{8, 9}. To answer the question when you left for work yesterday, you may combine knowledge about usually departing from home around 8:30 a.m. with the specific sequence of events that unfolded – eating breakfast while listening to the 8 a.m. news and arriving at work a few minutes late for the 9 a.m. meeting despite good traffic conditions on your commute. You infer that you left later than usual, at around 8:40 a.m. Thus, constructive mnemonic processes allow you to estimate when this event occurred, even if a specific event time is not part of the original memory^{8, 9}. Event representations in the hippocampal-entorhinal region carry information about sequence relationships^{10, 11}, but whether this goes back to mnemonic construction is unclear. Next to its role in memory for specific sequences, the hippocampal-entorhinal region also generalizes across experiences via the abstraction of structural regularities and the recombination of information across episodes^{12, 13}, suggesting you may use knowledge about comparable mornings to recall your departure time. Here, we ask whether temporal event relations are generalized across sequences that share a similar structure and address the question how mnemonic construction and generalization combine in the hippocampus and in participants’ memory for event times.

In line with its well-established role in episodic memory, the hippocampal-entorhinal region is centrally involved in processing and remembering specific event sequences¹⁰. For instance, learning sequences recruits the hippocampus and entorhinal cortex^{14, 15}, and hippocampal activity increases at event boundaries delineating sequences^{16, 17}. Hippocampal multi-voxel patterns are sensitive to objects shown at learned sequence positions¹⁸, and recent work suggests that the hippocampus incorporates the duration of intervals between elements in sequence representations^{19, 20}. Further, pattern correlations in the hippocampus and entorhinal cortex relate to memory for temporal relations^21–26.

Hippocampal and entorhinal representations of events occurring in sequence reflect the temporal relations of these events. In one experiment, participants learned the spatial and temporal relationships of events encountered in sequence along a route through a virtual city^{21, 27}. After relative to before learning, pattern similarity in the anterior hippocampus elicited by event images reflected the sequence relationships between pairs of events²¹. Pattern similarity correlated negatively with temporal distances, so that events close in time elicited more similar activity patterns relative to events separated by longer intervals. Within the entorhinal cortex, this effect was specific to the anterior-lateral subregion²⁷, consistent with the involvement of this area in precise temporal memory recall^{28, 29}.

However, whether event representations in the anterior hippocampus and anterior-lateral entorhinal cortex reflect temporal distances based on actively constructed event times is unclear. Alternatively, these representations of temporal structure could go back to the order of events. For example, successive events could be linked together, resulting in representations of sequence order, where temporal distances are defined based on the number of associative links between events^30–32. Another possibility is that temporal structure representations arise through elapsing time more passively. For example, the firing of individual entorhinal neurons changes with varying time constants in rodents and non-human primates, allowing time to be decoded from population activity^{33, 34}. Slowly drifting activity patterns could be incorporated into event memories as temporal tags, providing a potential mechanism for temporal memory³⁵. Here, we tested whether event representations reflect temporal relations based on mnemonically constructed event times, even when accounting for event order and objectively elapsing time.

Mnemonic construction enables prospective cognition^{2, 5, 36}. The hippocampal-entorhinal region integrates and recombines episodic details across experiences for future simulation, inferential reasoning and generalization^{5, 12, 13, 37–40}. Work in rodents and humans demonstrates that the hippocampus supports transitive inference, which requires inferring novel relations between stimulus pairs from knowledge about previously learned premise pairs^41–43. Further, it combines separately learned associations, enabling inferences about shared associations^44–50. Recent work suggests a central role for the entorhinal cortex in the abstraction of structural knowledge that is linked to sensory experience in the hippocampus^{12, 51}. Indeed, entorhinal activity patterns reflected structural similarities between choice options in a reinforcement learning task⁵². Furthermore, in an associative inference task, hippocampal activity patterns carried information about the shared internal structure of image triads such that the hippocampal representational geometry was generalized across triads⁵³. Applying abstract structural knowledge enables adaptive behavior through the generalization of relations to novel situations^{12, 51}. Whether representations of temporal relations of events in a sequence are constructed such that they generalize across sequences with a similar structure is unclear.

Knowledge about structural regularities and semantic associations closely interacts with episodic construction^{4, 7, 38}. When estimating the size of studied images, participants’ reconstructions were systematically distorted towards category averages^{54, 55}. For relatively small fruits like strawberries, participants tended to overestimate the studied size, whereas they consistently underestimated sizes of large fruits like pineapples. This resulted in an overall bias towards the category mean of all fruits⁵⁴. Consistent with the notion that learned event structures contribute to event cognition^56–58, external and semantic details are used to furnish past and future scenarios when few episodic details are generated^{59, 60}. When estimating the times of events from a movie, which was terminated prematurely, participants underestimated when events took place for events close to the end of the presented section, possibly due to prior knowledge about the typical structure of movie plots⁶¹. These findings suggest that abstract knowledge about general patterns could systematically distort constructions of specific event times. If, as in the introductory example, you usually leave for work at 8:30 a.m., this may bias the estimate of your departure time on the day you arrived late towards this time.

Here, we combine functional magnetic resonance imaging (fMRI) with a sequence learning task requiring the active, memory-based construction of the times of events forming different sequences. We show that event representations in the anterior hippocampus change through learning to reflect constructed event times rather than sequence order or passively elapsing time. Furthermore, the anterior hippocampus generalizes temporal relations across sequences, and structural knowledge about other sequences systematically biases the construction of specific event times. While within- and across-sequence relations are detected in anatomically overlapping regions of the hippocampus, the mode of representation differs depending on whether events belong to the same sequence or not. In contrast, the anterior-lateral entorhinal cortex uses one shared representational format to map relationships of events from the same and from different sequences.

Results

We asked participants to learn four sequences that consisted of five unique event images each (Figure 1, Supplemental Figure 1). Participants were instructed that each sequence depicted events taking place on a specific day in the life of a family. Their task was to infer the time of each event relative to the temporal reference frame of a virtual clock (Figure 1C). The true virtual times of events were never revealed. Rather, the clock was running hidden from participants. It was uncovered only infrequently between event presentations to briefly show the current virtual time (Supplemental Figure 2, see Methods). Participants had to actively construct event times by combining their experience of objectively elapsing time (real time) with the virtual time cues. Importantly, we manipulated the speed of the hidden clock between sequences so that different amounts of virtual time passed in the same real time intervals. With this paradigm, we partially dissociated the virtual time of events from the event order and real time to test whether actively constructed event times underlie participants’ memory for the temporal structure of the sequences.

Figure 1. Experimental Design.

A. Overview of the experiment. B. In the picture viewing tasks before and after learning, participants saw event images presented in the same random order and using identical stimulus timings. C. The day learning task took place in between the picture viewing tasks. Participants learned four sequences (virtual days) of five events each (Supplemental Figure 1) and inferred when events took place relative to a virtual clock. Left: The virtual clock ran hidden in the background for each sequence and was revealed only once in between successive events. These time cues varied across repetitions of a sequence, but events occurred at consistent points in virtual time. The duration of blank screen periods varied according to the interval between the indicated time and the event time. Thus, participants had to mentally construct event times by combining their experience of elapsing time with the time cues. Top right: The hidden clock ran at a fixed speed relative to real time for a given sequence, but its speed varied between sequences (Supplemental Figure 2). Bottom right: Different time metrics capture the temporal structure of the event sequences. Event relations can be quantified using temporal distances relative to the hidden clock (virtual time), sequence positions (order), and elapsed time in seconds (real time). While these metrics inevitably covary, they are partially dissociated by the clock speed manipulation. Virtual temporal distances can be quantified both within (solid lines) and across sequences (dotted lines). D,E. Participants’ memory of the sequences was tested in two tasks. In the sorting task (D), participants sorted the scenes according to the four different sequences. In the timeline task (E), participants positioned the five event images of a given sequence next to a timeline to indicate constructed event times. B-E.The Sims 3 and screenshots of it are licensed property of Electronic Arts, Inc.

Successful construction of event times

We assessed memory for the sequences using two behavioral tests administered at the end of the experimental session. First, participants sorted all event images according to sequence membership (Figure 1D). The high performance in this task (Figure 2A; 86.43%±16.82% mean±standard deviation of correct sorts) demonstrates accurate memory for which events belonged to the same sequence. Second, to probe constructed event times, we asked participants to position the events of a sequence on a timeline (Figure 1E). Remembered times were highly accurate (Figure 2B-D; 0.91±0.47 mean±standard deviation of average absolute errors in virtual hours). To test whether these constructed event times were driven by the virtual time of events, we regressed remembered times on virtual times with event order and real time as control predictors of no interest. We did so in a summary statistics approach based on multiple regression for each participant, combined with permutation tests, and using a linear mixed effects model (see Methods). The effect of virtual time on constructed event times was significant when controlling for variance accounted for by event order and real time (Figure 2E-F; summary statistics: t27=10.62, p<0.001, d=1.95, 95% CI [1.38, 2.70]; mixed model: χ²(1)=115.95, p<0.001, Supplemental Table 1). Together, these findings demonstrate that participants formed precise memories of the different sequences and accurately constructed event times.

Figure 2. Participants learn the temporal structure of the sequences relative to the virtual clock.

A. Plot shows the percentage of correctly sorted event images in the sorting task. B. Constructed event times were assessed in the timeline task. Responses are shown separately for each of the five events (color coded according to true virtual time) of each of the four sequences (rows). Colored circles with gray outline at the bottom of each row show true event times. C, D. Mean absolute errors in constructed times (in virtual hours) are shown (C) averaged across events and sequences and (D) averaged separately for the five event positions. E. Z-values for the effects of different time metrics from participant-specific multiple regression analyses and permutation tests show that virtual time explained constructed event times with event order and real time in the model as control predictors. F. Likewise, parameter estimates and 95% confidence intervals for the fixed effects of the three time metrics from a linear mixed model indicate that virtual time relates to constructed event times beyond the effects of order and real time. G. Estimated marginal means (model predictions) illustrate the effects of the three time metrics. A-E. Circles are individual participant data; boxplots show median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distributions show probability density function of data points. *** p<0.001

Hippocampal representations of within- sequence relations reflect constructed event times

Before and after learning the event sequences, participants viewed the event images in random order while undergoing fMRI (Figure 1AB). We quantified changes in the similarity of multi-voxel patterns between pairs of events from before to after learning (Figure 3, see Methods). Using two approaches to model-based representational similarity analysis, we tested whether changes in pattern similarity could be explained by the temporal relationships between pairs of events. Temporal distances between events were measured in virtual time, real elapsing time in seconds and as differences in sequence order position (Figure 1C). In the summary statistics approach, we compared the fit of linear models predicting pattern similarity changes from temporal distances to shuffle distributions for each participant and assessed the resulting Z-values on the group level using permutation-based tests. Second, we fit linear mixed effects models to quantify whether sequence relationships explained pattern similarity changes. Rather than performing inferential statistics on one summary statistic per participant, mixed models estimate fixed effects and their interactions using all data points. We used temporal distance measures as fixed effects while capturing within-participant dependencies with random intercepts and random slopes (see Methods). The converging results of these analyses demonstrate that our findings do not depend on the specific statistical methods employed. Based on our previous work^{21, 27}, we focused our analyses on the anterior hippocampus and the anterior-lateral entorhinal cortex (see Methods).

Figure 3. Representational Similarity Analysis Logic.

We quantified the representational similarity of all event pairs before and after learning. Representational change was defined by subtracting pre-learning from post-learning pattern similarity (top row). Using two approaches to model- based representational similarity analysis (RSA, see Methods), we analyzed whether pattern similarity changes reflected the temporal structure of the sequences (bottom left). In the summary statistics approach (middle right), we regressed pattern similarity change on temporal distances between events using participant-specific linear models that were compared to null distributions obtained from shuffling similarity change against temporal distances. The resulting Z-values were used for permutation-based group-level statistics. In the mixed model approach (bottom right), we estimated the influence of temporal distances on pattern similarity change using fixed effects, with random effects accounting for within-subject dependencies. The statistical significance of fixed effects was assessed using likelihood ratio tests against reduced models excluding the fixed effect of interest.

We first tested whether pattern similarity changes in the anterior hippocampus (Figure 4A) could be explained by the virtual temporal distances between event pairs from the same sequence. We observed a positive relationship between similarity changes and temporal distances in both the summary statistics (Figure 4B; t27⁼³.07, p=0.006, d=0.56, 95% CI [0.18, 1.00]; α=0.025, corrected for separate tests of events of the same and different sequences) and the mixed model approach (Figure 4CD; χ²(1)=9.87, p=0.002, Supplemental Table 2). Thus, pattern similarity was higher for event pairs separated by longer temporal distances than for pairs separated by shorter intervals.

Figure 4. Sequence representations in anterior hippocampus reflect constructed event times.

A. The anterior hippocampus region of interest is displayed on the MNI template with voxels outside the field of view shown in lighter shades of gray. Color code denotes probability of a voxel to be included in the mask based on participant-specific ROIs (see Methods). B. The Z- values based on permutation tests of participant-specific linear models assessing the effect of virtual time on pattern similarity change for event pairs from the same sequence were significantly positive. C. Dot plot shows the parameter estimate and 95% confidence interval for the fixed effect of virtual time on hippocampal pattern similarity change for same- sequence events from a linear mixed effects model. D. Estimated marginal means illustrate the positive relationship between virtual temporal distances and pattern similarity change. E. Z-values show the relationship of the different time metrics to representational change based on participant-specific multiple regression analyses. Virtual time predicts pattern similarity change with event order and real time in the model as control predictors of no interest. F,G. Parameter estimates with 95% confidence intervals (F) and estimated marginal means (G) show the fixed effects of the three time metrics from the corresponding mixed model. B,E. Circles are individual participant data; boxplots show median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distributions show probability density function of data points. ** p<0.01; * p<0.05

In contrast to our previous work²¹, participants studied multiple sequences. They might have formed strong associations of same-sequence events on top of inferring each event’s virtual time, potentially altering how temporal distances affected hippocampal pattern similarity. Importantly, the effect of virtual temporal distances on pattern similarity changes remained significant when competing for variance with a control predictor accounting for comparisons of the first and last event of each sequence (Supplemental Figure 3A- C; summary statistics: t27=2.25, p=0.034, d=0.41, 95% CI [0.04, 0.82]; mixed model: χ²(1)=5.36, p=0.021, Supplemental Table 3). Thus, the relationship of hippocampal event representations and temporal distances is not exclusively driven by associations of the events marking the transitions between sequences.

Having established that hippocampal pattern similarity changes relate to temporal distances, we next assessed whether this effect was driven by the constructed event times beyond sequence order and real time. We thus included the two additional time metrics as control predictors in the model. Virtual temporal distances significantly predicted pattern similarity changes even when controlling for the effects of event order and real time in seconds (Figure 4E-F; summary statistics: t27=2.18, p=0.040, d=0.40, 95% CI [0.02, 0.81]; mixed model: χ²(1)=5.92, p=0.015, Supplemental Table 4). Further, the residuals of linear models, in which hippocampal representational change was predicted from order and real time, were related to virtual temporal distances (Supplemental Figure 3D; t27⁼2.23, p=0.03⁴, d=0.41, 95% CI [0.03, 0.82]), demonstrating that virtual time accounts for variance that the other time metrics fail to explain. Together, these data show that hippocampal representations of events from the same sequence changed to reflect actively constructed event times.

The hippocampus generalizes temporal relations across sequences

We next tested whether similarity changes of hippocampal representations of events from different sequences mirrored generalized temporal distances. When comparing pairs of events belonging to different sequences, we observed a significant negative effect of virtual temporal distances on pattern similarity change (Figure 5A, Supplemental Figure 4; summary statistics t27=-2.65, p=0.013, d=-0.49, 95% CI [-0.91, - 0.10]; mixed model: χ²(1)=6.01, p=0.014, Supplemental Table 5; α=0.025, corrected for separate tests of events of the same and different sequences). This indicates that hippocampal representations of events from different sequences changed systematically to reflect generalized temporal relations. Events occurring at similar times relative to the virtual clock, but in different sequences, were represented more similarly than those taking place at more different virtual times .

Figure 5. The anterior hippocampus generalizes temporal relations across sequences.

A. Z-values show results of participant-specific linear models quantifying the effect of virtual time for event pairs from the same sequence (blue, as in Figure 4B) and from different sequences (red). Temporal distance is negatively related to hippocampal representational change for event pairs from different sequences. See Supplemental Figure 4 for mixed model analysis of across-sequence comparisons. The effect of virtual time differs for comparisons within the same sequence or between two different sequences. B. In the corresponding mixed effect model, a significant interaction between virtual time and sequence membership was observed. Dot plot shows fixed effect estimates with 95% confidence intervals. C. Estimated marginal means illustrate the diverging effects of virtual time as a function of sequence membership of event pairs. D. To visualize the interaction effect in A-C, raw pattern similarity change was averaged for events separated by low and high temporal distances, respectively. E. Multidimensional scaling results show low-dimensional embedding of the event sequences. Shapes indicate event order, color shows virtual times of events. The different lines connect the events belonging to the four sequences for illustration. *** p≤0.001; ** p<0.01; * p<0.05

The relationship of temporal distances and representational change differed significantly between events from the same or different sequences (Figure 5A-C, summary statistics: paired t-test t27=3.71, p=0.001, d=1.05, 95% CI [0.29, 1.13]; mixed model: interaction of sequence membership with virtual time χ²(1)=14.37, p<0.001, Supplemental Table 6). The positive correlation of temporal distance and pattern similarity change implies relatively higher similarity for same-sequence events that are far apart than those that are close in time, whereas events close together in time became more similar relative to those separated by larger temporal distances when comparing across sequences (Figure 5D, same-sequence events separated by high versus low temporal distance: t27=2.48, p=0.020, d=0.64, 95% CI [0.08, 0.87]; different-sequence events separated by high versus low temporal distance: t27=-3.26, p=0.002, d=-0.89, 95% CI [-1.03, -0.21], α=0.025, corrected for separate tests of events of the same and different sequences). Thus, the way relational knowledge was represented depended on whether events belonged to the same sequence or not.

To explore how event sequences may be arranged in a low-dimensional representational space to give rise to the effects described above, we generated a distance matrix from the mixed effects model fitted to hippocampal pattern similarity change and subjected it to non-metric multidimensional scaling (see Methods). The resulting configuration (Figure 5E, see Supplemental Figure 4D for a qualitatively similar low- dimensional embedding based on the UMAP algorithm) exhibits a c-shaped pattern for each sequence. This may reflect the pattern similarity increases (low representational distances) for events separated by large virtual temporal distances, which is captured by the positive correlation of pattern similarity and temporal distance for same-sequence events. Events occurring at similar virtual times occupy similar locations, in line with high pattern similarity for events from different sequences that are separated by low temporal distances. Thus, the generalization across sequences results in a comparable configuration for each sequence.

Sequence representations differ between hippocampus and entorhinal cortex

In our second region of interest, the anterior-lateral entorhinal cortex (Figure 6A), the effect of virtual time on representational change did not differ statistically between event pairs from the same or from different sequences (summary statistics: paired t-test t27=0.07, p=0.942). We thus collapsed across comparisons from the same and different sequences and observed a significant effect of virtual temporal distances on entorhinal pattern similarity change (Figure 6B-D; summary statistics: t27=-2.31, p=0.029, d=-0.42, 95% CI [-0.84, -0.05]; mixed model: χ²(1)=4.39, p=0.036, Supplemental Table 7; see Supplemental Figure 5 for separate analyses of events from the same and from different sequences). In line with our previous work²⁷, events close together in time became more similar than those separated by longer temporal intervals (Figure 6E). We further corroborated that the temporal structure of the sequences was represented differently between the anterior-lateral entorhinal cortex and the anterior hippocampus (summary statistics: interaction between region and sequence membership in permutation-based repeated-measures ANOVA F1,27=7.76, p=0.010, η²=0.08, main effect of region F1,27=3.10, p=0.086, η²=0.02, main effect of sequence F1,27=7.41, p=0.012, η²=0.08; mixed model: three-way interaction between virtual time, sequence membership and region of interest χ²(1)=6.31, p=0.012, Supplemental Table 8; see Supplemental Figure 6 for a comparison of the signal-to-noise ratio in these regions). Whereas the hippocampus employed two distinct representational formats for temporal relations depending on whether events belonged to the same sequence or not, we observed consistent negative correlations between representational change and temporal distances when collapsing across all event pairs, but no statistically significant difference between representations of temporal relations from the same or different sequences in the entorhinal cortex.

Figure 6. The anterior-lateral entorhinal cortex uses a shared representational format for relations of events from the same and different sequences.

A. The anterior-lateral entorhinal cortex region of interest is displayed on the MNI template with voxels outside the field of view shown in lighter shades of gray. Color code denotes probability of a voxel to be included based on participant-specific masks (see Methods). B. Z-values for participant-specific RSA model fits show a negative relationship between pattern similarity change and virtual temporal distances when collapsing across all event pairs. C,D. Parameter estimate with 95% confidence intervals (C) and estimated marginal means (D) show the fixed effect of virtual time from the corresponding linear mixed effect model. E. To illustrate the effect in B-D, raw pattern similarity change in the anterior-lateral entorhinal cortex was averaged for events separated by low and high temporal distances. * p<0.05

Anatomical overlap between representations of within-sequence relations and across-sequence generalization

We next asked whether representations of same- sequence relations are distinct from or overlap with the across-sequence generalization of temporal relations. For this purpose and to complement our region-of- interest analyses described above, we performed a similarity change for events from different sequences in these voxels (Figure 7B-D; summary statistics t27=- 2.19, p=0.036, d=-0.40, 95% CI [-0.81, -0.03]; mixed model: χ²(1)=4.13, p=0.042, Supplemental Table 10), demonstrating an overlap between representations of within-sequence relations and their generalization across sequences. Further, we conducted a searchlight analysis looking for negative correlations of temporal searchlight analysis that revealed significant effects of virtual temporal distances on representations of events from the same sequence in the bilateral anterior hippocampus (Figure 7A; peak voxel MNI x=-24, y=-13, z=-20; t=4.53, psvc=0.006, Supplemental Table 9). We used the same-sequence searchlight peak cluster to define a region of interest to test for the independent across-sequence generalization effect (see Methods). Indeed, virtual temporal distances explained pattern distances and pattern similarity change for events from different sequences. We detected clusters in anterior hippocampus that overlapped with the same-sequence searchlight effect (Figure 7EF), though this searchlight generalization effect did not survive corrections for multiple comparisons (peak voxel MNI x=-26, y=-19, z=- 15, t=-3.96, psvc=0.071, Supplemental Table 11). Lastly, we directly searched for brain areas in which pattern similarity change differentially scaled with temporal distances depending on whether events were from the same or different sequences. The two largest clusters in our field of view were located in the left and right anterior hippocampus (peak voxel MNI x=31, y=-16, z=- 21; t=4.25, psvc=0.007, Supplemental Table 12). Taken together, these findings highlight that hippocampal representations carry information about the specific sequence in which events occur, and that these temporal relations are generalized across sequences.

Figure 7. Overlapping representations of within- and across-sequence relations.

A. Searchlight analysis results show a positive relationship between representational change and virtual temporal distances for event pairs from the same sequence in the bilateral anterior hippocampus. Statistical image is thresholded at puncorrected<0.01; voxels within black outline are significant after correction for multiple comparisons using small volume correction. B-D. In the peak cluster from the independent within-sequence searchlight analysis (A), representational change was negatively related to virtual temporal distances between events from different sequences. B. Circles show individual participant Z-values from summary statistics approach; boxplot shows median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distribution shows probability density function of data points. C. Dot plot shows fixed effect estimate with 95% confidence interval from linear mixed model. D. The estimated marginal means from the linear mixed effect model illustrate the negative relationship between virtual temporal distances and representational change. E. Searchlight analysis results show negative relationship between representational change and temporal distances for different-sequence event pairs. Statistical image is thresholded at puncorrected <0.05. F. Within the anterior hippocampus, the effects for events from the same sequence and from two different sequences overlap. Visualization is based on statistical images thresholded at puncorrected <0.05 within small volume correction mask. G. Searchlight analysis results show a bilateral interaction effect in the anterior hippocampus that is defined by a differential relationship of virtual temporal distances and representational change for events from the same and different sequences. Statistical image is thresholded at puncorrected<0.01; voxels within black outline are significant after correction for multiple comparisons using small volume correction. A, E, G. Results are shown on the MNI template with voxels outside the field of view displayed in lighter shades of gray. See Supplemental Figure 7 for additional exploratory results. * p<0.05

Generalized knowledge about other sequences biases event time construction

Having established generalized hippocampal event representations, we explored whether knowledge about the general structure of event times in other sequences influenced the construction of individual event times. For each event, we quantified when it took place relative to the average virtual time of the events at the same sequence position in the other three sequences (Figure 8A; see Methods). We reasoned that the construction of a specific event time could be biased by knowledge about the general pattern of event times at that sequence position. Indeed, we observed positive relationships between the relative time of other events and signed errors in constructed event times as assessed in the timeline task (Figure 8B-E, Supplemental Figure 8A; summary statistics: t27=5.32, p<0.001, d=0.98, 95% CI [0.55, 1.48]; mixed model: χ²(1)=17.90, p<0.001, Supplemental Table 13). This demonstrates that general knowledge about the sequences biased the construction of event times. When events at the same sequence position, but in different sequences, took place late relative to an event, the constructed virtual time for that event tended to be overestimated, and vice versa when the other events occurred relatively early. In an independent group of participants⁶², we replicated this generalization bias (Figure 8F-H, Supplemental Figure 8B; summary statistics: t45=11.30, p<0.001, d=1.64, 95% CI [1.23, 2.13]; mixed model: χ²(1)= 53.74, p<0.001, Supplemental Table 14), confirming the influence of generalized knowledge about the sequences on event time construction. One possibility is that structural knowledge about the sequences biases the construction of specific event times, in particular when uncertainty about the virtual time of events is high. Indeed, we observed a significant negative correlation between how strongly pattern similarity changes in the anterior hippocampus reflected temporal relations between same-sequence events in the searchlight analysis and the strength of the behavioral generalization bias (Figure 8IJ, Spearman r=-0.53, p=0.005; α=0.025 corrected for two comparisons; correlation with across-sequence effect: Spearman r=- 0.19, p=0.322), suggesting that the construction of event times was less biased by time patterns generalized across sequences in those participants with precise representations of within-sequence temporal relations.

Figure 8. Structural knowledge biases construction of event times.

A. The generalization bias quantifies the influence of structural knowledge on the construction of individual event times. For each event, the mean time of events at the same sequence position in the other sequences was calculated to test whether event times were biased towards the relative time of other events. B. The scatterplot illustrates the generalization bias for an example participant. Each circle corresponds to one event and the regression line highlights the relationship between the relative time of other events and the errors in constructed event times. The example participant was chosen to have a median-strength generalization bias. See Supplemental Figure 8 for the entire sample. C-E. The relative time of events from other sequences predicted signed event time construction errors as measured in the timeline task. C. Circles show individual participant Z-values from participant- specific linear models (B); boxplot shows median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distribution shows probability density function of data points. D. Dot plot shows fixed effect estimate with 95% confidence interval from linear mixed model. E. The estimated marginal means from the linear mixed effect model illustrate the positive relationship between the time of other events and constructed event times. When other events took place late relative to a specific event, the time of that event was estimated to be later than when other events were relatively early. F-H. The generalization bias in event time construction through structural knowledge was replicated in an independent sample (n=46) based on Montijn et al.⁶². Data shown as in C-E. I. The behavioral generalization bias (regression coefficients from summary statistics approach) did not correlate significantly with the across-sequence generalization effect in the anterior hippocampus (searchlight peak voxel t-values). J. We observed a significant negative correlation between the same- sequence searchlight effect (peak voxel t-values) and the behavioral generalization bias (regression coefficients from summary statistics approach), suggesting that participants with strong hippocampal representations of the temporal relations between events from the same sequence were less biased by structural knowledge in their construction of event times. Statistics in I and J are based on Spearman correlation.

Discussion

Our findings show that hippocampal event representations change through learning to reflect temporal relations based on mnemonically constructed event times. Converging region of interest and searchlight analyses demonstrate that, on the one hand, the hippocampus forms highly specific representations of temporal relations of the events in a sequence that mirror constructed event times beyond the effects of order and real time. On the other hand, temporal relations are generalized across sequences using a different representational format. In contrast, the similarity of event representations in the anterior- lateral entorhinal cortex scaled with temporal distances for events irrespective of sequence membership. The behavioral data demonstrate that the construction of specific event times is biased by structural knowledge abstracted from different sequences.

In our paradigm, participants combined their experience of passing real time with infrequent cues to mentally construct the times of events relative to a hidden virtual clock. Participants’ responses in a memory test and the similarity structure of multi-voxel pattern representations were explained by the virtual times of events beyond the effects of real time and sequence order. Specifically, our data show that the similarity of event representations in the anterior hippocampus scaled with temporal distances between events, highlighting that it maps relational knowledge derived from mnemonic construction.

One possible reason for why, in contrast to our previous work²¹, we observed positive correlations between temporal distances and hippocampal pattern similarity of same-sequence events could be that in the present task participants encountered multiple sequences. Beyond constructing the time of each event relative to the virtual clock, participants also formed associations between the events of each sequence. Notably, when participants used associative strategies in a recency judgment task, more stable hippocampal voxel patterns were related to accurate order judgments²², whereas the opposite was observed in a task where participants encoded a continuous stream of images from the same category²⁴. These findings might support the view that the formation of associations between elements of a sequence may have altered how pattern similarity changes relate to temporal distances for same-sequence events. A second potential reason could be that events occurring close-by in the same sequence were differentiated by the hippocampus, thereby giving rise to decreases in similarity for temporally close events. While the hippocampus differentiates similar episodes^{47, 63–66}, this interpretation does not directly account for the overall correlation of temporal distances and representational change.

The hippocampus constructed a generalized representation of the sequences. Multi-voxel patterns of events taking place at similar virtual times, but in different sequences, were more similar than those of events occurring at more different points in time. Thus, representations of events from different sequences changed systematically to reflect generalized temporal distances. We further observed anatomical overlap between the clusters of voxels where pattern similarity reflected temporal relations of events from the same or different sequences. However, the way temporal relations shaped hippocampal multi-voxel pattern similarity differed between pairs of events from the same and different sequences. The generalization of temporal distances across sequences in the hippocampus is in line with the contribution of constructive mnemonic processes to flexible cognition via the recombination of elements across experiences and statistical learning^{13, 40, 43, 46, 48, 49, 67, 68}. Abstract relational and conceptual knowledge can be generalized to novel situations that share a similar underlying structure^{12, 38, 51, 53, 56}.

Learning the temporal relations of events in a sequence entails acquiring knowledge about the sequence structure. In navigation, cognitive maps of spatial relations enable flexible behavior⁶⁹. Foundational neuroscientific discoveries uncovered spatially tuned cells in the hippocampal-entorhinal region as potential neural substrates for cognitive maps of space^{70, 71}. Consistent with its involvement in various cognitive domains, the hippocampal-entorhinal region was suggested to form cognitive maps of relational knowledge more generally^{71, 72}. The characteristic activity patterns of spatially tuned cells also encode non-spatial dimensions, suggesting the organization of relational knowledge in cognitive spaces⁷³. In the temporal domain, neurons in the rodent hippocampus fire at specific time points during a delay, resulting in characteristic sequences of activity^74–76. These neurons are often referred to as time cells and have been suggested to support temporal memory⁷⁷. Recent work characterized similar activity patterns also in the human hippocampus and entorhinal cortex during the encoding and retrieval of word lists⁷⁸. Time cell representations can scale to different interval lengths⁷⁹. Speculatively, this could underlie the construction of temporal distances referenced to the hidden clock from similar real-time intervals in our task. In concert with the link between the similarity of event representations and temporal relations between these events^{21, 27, 80}, these findings raise the possibility that the hippocampal-entorhinal region forms cognitive maps reflecting the temporal relational structure of events in a sequence. Events occurring at similar times relative to the hidden clock occupy similar positions on this cognitive map, even if they belong to different sequences. This could underlie the generalization of temporal relations across sequences that we observed that are characterized by negative correlations of pattern similarity and temporal distances.

Structural knowledge and mnemonic construction are intertwined. In two independent samples, we show that general time patterns, abstracted from other sequences, bias the construction of specific event times. When events at the same sequence position, but in other sequences, took place relatively late to the time of an event, the time of that event was remembered to be later than when the other events occurred relatively early. This generalization bias shows that knowledge about events at structurally similar positions contributes to constructive memory for specific events. It is in line with biases resulting from the exploitation of environmental statistics when reconstructing stimulus sizes from memory^{54, 55}, when estimating brief time intervals^{81, 82}, or when discriminating the order of previously presented stimuli⁸³. Likewise, prior knowledge can distort memories for short narratives⁸⁴, spatial associations⁸⁵ and temporal positions⁶¹. Consistent with the suggested role of grid cells in the representation of spatial structure, distortions in mnemonic reconstructions of spatial relations induced through boundary geometry follow predictions from models of grid-cell functioning⁸⁶. Further, recombining information across episodes for associative inference can induce false memories for contextual details^{67, 87}, illustrating that generalization impacts memory for specific associations. In line with the greater reproduction of episodic details by participants whose recall follows the temporal structure of an experience more closely⁸⁸, these findings highlight that structural knowledge and mnemonic construction are intertwined. More broadly, abstract semantic or schematic knowledge may provide a scaffold for the recall of episodic details^{4, 7, 38, 89, 90}. Our findings show that structural knowledge not only facilitates, but also biases constructive memory.

The hippocampus supports constructive memory and generalization in concert with a distributed network of brain regions. In addition to medial temporal lobe structures, the mental simulation of past and future episodic scenarios recruits a core network including medial prefrontal and retrosplenial cortex as well as lateral parietal and temporal areas^{39, 91}. Notably, this network overlaps with areas supporting the recombination of elements and generalization. For example, both the construction of novel experiences based on the combination of multiple elements⁹² and memory integration across episodes⁴⁷ are supported by the medial prefrontal cortex and the hippocampus. In sequence processing, representational similarity is increased for items occupying the same position in different sequences in parahippocampal, retrosplenial and medial prefrontal cortices as well as in the angular gyrus^{18, 93}. Likewise, sequence positions can be decoded from magnetoencephalographic responses elicited by visual stimuli presented in scrambled order⁹⁴. In line with the suggestion that the posterior parietal cortex supports generalization by projecting stimuli onto a low-dimensional manifold⁹⁵, neural magnitude representations that generalize across task contexts have been observed using EEG^{96, 97}. While we did not observe effects outside the hippocampal- entorhinal region that survived corrections for multiple comparisons, we note that, based on our prior hypotheses, we opted for high-resolution coverage of the medial temporal lobe at the cost of reducing the field of view of our MR images. As the events in our task can be conceived of as being arranged along one or multiple, parallel mental number lines, future research could test how the parietal cortex encodes event relations to explore commonalities with and differences to the generalization of event times observed in the hippocampus.

Our paradigm allows a highly-controlled read-out of representational change relative to a pre-learning baseline scan. Events are shown in the same random order before and after learning, ruling out that prior associations or the temporal auto-correlation of the blood-oxygen-level-dependent signal drives our effects. Future studies could extend the paradigm to investigate how hierarchically nested sequences are represented, for example by introducing higher-order relations between sequences – akin to different days being grouped in weeks. The precise temporal dynamics of the generalized hippocampal event representation pose another intriguing question. Based on the report that the temporal organization of memory reactivation relative to the hippocampal theta phase reflects semantic relations between items⁹⁸, a speculative hypothesis is that a theta phase code could also underlie memory for temporal relations of events from the same and different sequences.

In conclusion, our findings show that the similarity of event representations in the hippocampus reflects relations between events that go back to mnemonically constructed event times, highlighting the impact of active mnemonic construction on sequence memory beyond the effects of event order and real elapsing time. Temporal relations are generalized to events from different sequences, in line with hippocampal contributions to the abstraction of structural knowledge and the generalization across episodes. General time patterns abstracted from other sequences systematically influence the construction of specific event times, demonstrating that constructions of specific scenarios build on structural knowledge.

Methods

Participants

31 participants were recruited for this experiment. Participants gave written informed consent prior to participation. All proceedings were approved by the local ethics committee (CMO Regio Arnhem- Nijmegen). One participant aborted the experiment due to feeling claustrophobic when entering the MR scanner. Two participants were excluded from further analysis due to bad memory performance and technical difficulties during data acquisition. Thus, the sample consisted of 28 participants (21 female, age: mean±standard deviation 23.04±3.21 years, range 18- 31 years).

Procedure

Overview

The experiment consisted of four parts (Figure 1A) and lasted approximately 2.5 hours in total. The first three parts were performed inside the MR scanner and comprised a learning task lasting around 50 minutes that was completed in between two blocks of a picture viewing task of around 25 minutes each. The tasks inside the scanner were presented on a rear-projection screen with a resolution of 800x600 pixels and implemented using Presentation (version 16.2, Neurobehavioral Systems). Subsequently, outside of the scanner, participants performed two short memory tasks in front of a computer screen, implemented with custom Matlab code. The tasks are described in more detail below. Data analysis was carried out using FSL (version 5.0.4)⁹⁹ and R (version 3.6.1)¹⁰⁰.

Stimuli

The stimuli (Supplemental Figure 1) used throughout the experiment were created within the life-simulation computer game The Sims 3 (Electronic Arts) by taking screenshots. Each image featured a scene in the life of an affluent family. The main character, the family father, was visible in all scenes. In addition, the mother, son, daughter and family dog appeared in some of the images. All of the depicted events took place within the same family home, but showed activities in a number of different rooms. In an effort to design stimuli with minimal to no indication of day time, the house had constant artificial lighting, but no windows or clocks. The 21 pictures used in this study were selected from an initial set of 35 pictures based on an independent sample rating them as the most ambiguous with regard to the time of day they could take place. One image served as a target image for the picture viewing tasks (see below), while the other 20 event images were randomly assigned to different times and days for every participant.

Picture Viewing Tasks

In the picture viewing tasks (Figure 1B), participants viewed a stream of the event images. Their task was to look at the images attentively and to respond via button press whenever a target picture, which showed the father feeding the family’s dog, was presented (pre- learning: 95.71%±7.90% mean±standard deviation of percentage of hits; 881.34ms±131.43ms mean±standard deviation of average reaction times; post-learning: 95.71%±6.90% mean±standard deviation of percentage of hits; 841.40ms±162.16ms mean±standard deviation of average reaction times). The task consisted of 10 mini-blocks. In each mini- block, the target image and the 20 images, which would later make up the virtual days (see Day Learning Task), were shown in random order. Mini-blocks were separated by breaks of 15 s. Stimulus presentations lasted 2.5 s and were time-locked to fMRI volume acquisition onsets. Scene stimuli within a mini-block were separated by 2 or 3 repetition times (TR), randomly assigned so that both stimulus onset asynchronies occurred equally often.

For each participant, we generated a random stimulus order with the constraint that no scene was consistently presented at early or late positions across mini-blocks. Specifically, we compared sequence positions across mini-blocks between the images using a one-way ANOVA. We discarded randomizations where this ANOVA was statistically significant to exclude biases in presentation order. Crucially, the same, participant-specific random order of stimuli and inter-stimulus intervals was used in both the pre-learning and the post-learning picture viewing task. Thus, any systematic differences in the representational similarity of event pairs between the two picture viewing tasks do not go back to differences in the timing of stimulus presentations or the temporal auto-correlation of the BOLD-signal. Rather, we interpret such changes to be a consequence of the learning task.

Day Learning Task

In this task, 20 of the 21 scenes, which were shown in the picture viewing tasks, were presented repeatedly. This time, however, they were grouped into multiple sequences introduced to participants as “virtual days”. There were four different sequences, each comprising 5 events. Events from the same sequence were always shown in a specific order and with a specific time delay between them. Scenes were on screen for 1.5 s. At the end of each sequence, an image of a moon was shown for 5 s, then the next sequence began. Every sequence was presented 7 times. There were 7 mini-blocks in this task. Within each of these, every sequence was presented once. At the end of a mini-block, a 30-s break followed, then the next block started. The order in which the sequences were presented differed randomly across the 7 mini-blocks.

We instructed participants that the scenes depicted events from the life of a family and that the sequences of event images corresponded to different days in the family’s life. Participants were asked to memorize which events made up the different sequences (Figure 1C). We further instructed them to learn when during the respective sequence each event occurred. Specifically, we asked participants to learn event times relative to a virtual clock. This clock was running hidden from participants and event images were shown whenever the hidden clock reached the specific event time (Figure 1C, Supplemental Figure 2AB). The task was devised such that participants had to rely on their experience of passing real time and mnemonic construction to infer the times of events.

Specifically, to give participants an indication of virtual time, the hidden clock was made visible 6 times for every presentation of a sequence: once before the first event, once in between successive events, and once after the last event. Thus, participants had to focus on their experience of passing real time between these time cues to infer the event times relative to the hidden virtual clock. Importantly, the exposure of the hidden clock occurred at random times for each sequence presentation (Supplemental Figure 2CD), with the constraint that it could not be revealed closer than 2 s to a preceding or subsequent event. Thus, participants saw different time cues in each repetition of a sequence. For example, while a specific event always happened at the same virtual time, e.g. 2:07 p.m., the virtual clock could be exposed at any time before the event, e.g. corresponding to 1:32 p.m. in the first repetition of the sequence, and corresponding to 1:17 p.m. in the second repetition. Because true event times were never revealed, participants could not exclusively rely on associative learning to solve the task. Time cues were visible for 1.5 s, but displayed only the time at the start of exposure, i.e. the displayed time did not change within the duration of its presentation.

In short, participants had to combine their experience- based estimates of passing time with the time cues provided by the exposures of the otherwise hidden clock to infer the time at which each event in each sequence took place. Crucially, we varied the speed of the hidden clock between sequences in an effort to partly dissociate real time (in seconds) from virtual time (in virtual hours). Thus, for two sequences more virtual time passed in a comparable amount of real elapsing time (Figure 1C, Supplemental Figure 2). Correlations between the linearly increasing time metrics are inevitably high (Pearson correlation of virtual time with order r=0.969 and virtual time with real time r=0.975). Still this manipulation allowed us to determine using multiple regression whether virtual time explained constructed event times when competing for variance with real elapsed time and event order and whether hippocampal pattern similarity changes related to temporal distances in virtual time beyond ordinal distances and real time distances. Regression models including collinear predictor variables do not result in biased parameter estimates^{101, 102}.

Sorting Task

The day sorting task (Figure 1D) was performed in front of a computer screen. The 20 event images from the day learning task were presented on the screen in a miniature version. They were arranged in a circle around a central area displaying 4 rectangles. Participants were instructed to drag and drop all events of the same sequence into the same rectangle with a computer mouse. Participants freely chose which rectangle corresponded to which sequence as the sequences were not identifiable by any label and were presented in differing orders across mini-blocks during learning.

Timeline Task

In this task, participants saw a timeline ranging from 6 a.m. to midnight together with miniature versions of the five event images belonging to one sequence (Figure 1E). Participants were instructed to drag and drop the event images next to the timeline so that scene positions reflected the event times they had inferred in the day learning task. To facilitate precise alignment to the timeline, event images were shown with an outward pointing triangle on their left side, on which participants were instructed to base their responses.

MRI Acquisition

MRI data were recorded with a 3T Siemens Skyra scanner (Siemens, Erlangen, Germany). A high- resolution 2D EPI sequence was used for functional scanning (TR=2270 ms, TE=24 ms, 40 slices, distance factor 13%, flip angle 85°, field of view (FOV) 210x210x68 mm, voxel size 1.5 mm isotropic). The field of view (FOV) was aligned to fully cover the medial temporal lobe, parts of ventral frontal cortex and (if possible) calcarine sulcus. Functional images for the two picture viewing tasks and the learning task were acquired in three runs. In addition to these partial- volume acquisitions, 10 scans of a functional whole- brain sequence were also acquired to improve registration during preprocessing. The sequence settings were identical to the functional sequence above, but instead of 40 slices, 120 slices were acquired, leading to a longer TR (6804.1ms). A structural scan was acquired for each participant (TR = 2300 ms; TE = 315 ms; flip angle = 8°; in-plane resolution = 256x256 mm; number of slices = 224, voxel resolution = 0.8x0.8x0.8 mm). Lastly, a gradient field map was acquired (for n = 21 participants only due to time constraints), with a gradient echo sequence (TR = 1020 ms; TE1 = 10 ms; TE2 = 12.46 ms; flip angle = 90°; volume resolution = 3.5x3.5x2 mm; FOV = 224x224 mm).

ROI Definition

Our previous work demonstrates representations reflecting the temporal relations of events in a sequence in the anterior hippocampus²¹ and the anterior-lateral entorhinal cortex²⁷. We thus focused our analyses on these regions. Region of interest (ROI) masks were based on participant-specific FreeSurfer segmentations (version 6.0.0-2), which yielded masks for the entire hippocampus and entorhinal cortex. These were co-registered to participants’ functional space. We defined anterior hippocampus using the Harvard-Oxford atlas mask (thresholded at 50% probability), selecting all voxels anterior to MNI y=-21 based on Poppenk et al.¹⁰³. The resulting anterior hippocampus mask was also co-registered to participants’ functional space and intersected with the participant-specific hippocampal mask from FreeSurfer. The mask for the anterior-lateral entorhinal cortex was based on Navarro Schröder et al.¹⁰⁴. It was co-registered to participants’ functional space and intersected with the entorhinal cortex mask from FreeSurfer.

Data Analysis Behavioral Data Analysis

Sorting Task

For analysis of the sorting task, we took the grouping of event images as provided by the participants and assigned them to the four sequences to ensure maximal overlap between actual and sorted sequence memberships. While the assignment of groupings to sequences is unambiguous when performance is, as in our sample, high, this procedure is potentially liberal at lower performance levels. We then calculated the percentage of correctly sorted event images for each participant, see the raincloud plot¹⁰⁵ in Figure 2A.

Timeline Task

We analyzed how well participants constructed the event times based on the day learning task. We quantified absolute errors across all events (Figure 2C) as well as separately for the five sequence positions (Figure 2D). Further, using two approaches we tested whether virtual time drove participants’ responses rather than the sequence order or objectively elapsing time. For the summary statistics approach, we ran a multiple regression analysis for each participant with virtual time, sequence position (order), and real time since the first event of a day as predictors of responses in the timeline task. To test whether virtual time indeed explained participants’ responses even when competing for variance with order and real time, included in the model as control predictors of no interest, we compared the participant-specific t-values of the resulting regression coefficients against null distributions obtained from shuffling the remembered times against the predictors 10,000 times. We converted the resulting p-values to Z-values and tested these against zero using a permutation-based t-test (two-sided; α=0.05; 10,000 random sign-flips, Figure 2E). As a measure of effect size, we report Cohen’s d with Hedges’ correction and its 95% confidence interval as computed using the effsize-package¹⁰⁶.

Second, we addressed this question using linear mixed effects modeling. Here, we included the three z-scored time metrics as fixed effects. Starting from a maximal random effect structure¹⁰⁷, we simplified the random effects structure to avoid convergence failures and singular fits. The final model included random intercepts and random slopes for virtual time for participants. The model results are visualized by dot plots showing the fixed effect parameters with their 95% confidence intervals (Figure 2F) and marginal effects (Figure 2G) estimated using the ggeffects package¹⁰⁸. To assess the statistical significance (α=0.05) of virtual time above and beyond the effects of order and real time, we compared this full model to a nested model without the fixed effect of virtual time, but including order and real time, using a likelihood ratio test. Supplemental Table 1 provides an overview of the final model and the model comparison.

To explore whether structural knowledge about general time patterns biases the construction of event times, we assessed errors in remembered event times. Specifically, when constructing the time of one specific event, participants could be biased in their response by the times of the events from other sequences at that sequence position. For each event, we quantified the average time of events in the other sequences at the same sequence position (Figure 8A). For example, for the fourth event of the first sequence, we calculated the average time of the fourth events of sequences two, three and four. We then asked whether the deviation between the average time of other events and an event’s true virtual time was systematically related to signed errors in constructed event times. A positive relationship between the relative time of other events and time construction errors indicates that, when other events at the same sequence position are relatively late, participants are biased to construct a later time for a given event than when the other events took place relatively early. In the summary statistics approach, we ran a linear regression for each participant (Figure 8B, Supplemental Figure 8) and tested the resulting coefficients for statistical significance using the permutation-based procedures described above (Figure 8C). The regression coefficients from this approach were used to test for a relationship between the behavioral generalization bias and the hippocampal searchlight effects (see below). Further, we analyzed these data using the linear mixed model approach (Figure 8DE, Supplemental Table 13).

To replicate the results from this exploratory analysis, we conducted the same analysis in an independent group of participants. These participants (n=46) constituted the control groups of a behavioral experiment testing the effect of stress induction on temporal memory⁶². They underwent the same learning task as described above with the only difference being the duration of this learning phase (4 rather than 7 mini- blocks of training). The timeline task was administered on the day after learning. The procedures are described in detail in Montijn et al.⁶². The data from this independent sample are shown in Figure 8F-H and Supplemental Figure 8B.

MRI Preprocessing

Preprocessing was performed using FSL FEAT (version 6.00). Functional scans from the picture viewing tasks and the whole-brain functional scan were submitted to motion correction and high-pass filtering using FSL FEAT. For the two picture viewing tasks, data from each mini-block was preprocessed independently. For those participants with a field map scan, distortion correction was applied to the functional data sets. No spatial smoothing was performed. Functional images from the two picture viewing tasks were then registered to the preprocessed mean image of the whole-brain functional scan. The whole-brain functional images were registered to the individual structural scans. The structural scans were in turn normalized to the MNI template (1-mm resolution). Gray matter segmentation was done on the structural images, and the results were mapped back to the space of the whole-brain functional scan for later use in the analysis.

Representational Similarity Analysis

Representational similarity analysis (RSA)¹⁰⁹ was first implemented separately for the pre- and post-learning picture viewing task. It was carried out in ROIs co- registered to the whole-brain functional image and in searchlight analyses (see below). For the ROI analyses, preprocessed data were intersected with the participant-specific anterior hippocampus and anterolateral entorhinal cortex ROI masks as well as a brain mask obtained during preprocessing (only voxels within the brain mask in all mini-blocks were analyzed) and the gray matter mask. For each voxel within the ROI mask, motion parameters from FSL MCFLIRT were used as predictors in a general linear model (GLM) with the voxel time series as the dependent variable. The residuals of this GLM (i.e. data that could not be explained by motion) were taken to the next analysis step. As the presentation of images in the picture viewing tasks was locked to the onset of a new volume (see above), the second volume after image onset was selected for every trial, effectively covering the time between 2270 and 4540 ms after stimulus onset. Only data for the 20 event images that were shown in the learning task were analyzed; data for the target stimulus were discarded. The similarity between the multi-voxel activity pattern for every event image in every mini-block with the pattern of every other event in every other mini-block was quantified using Pearson correlation coefficients. Thus, comparisons of scenes from the same mini-block were excluded. Next, we calculated mean, Fisher z-transformed correlation coefficients for every pair of events, yielding separate matrices of pattern similarity estimates for the pre- and the post-learning picture viewing tasks (Figure 3).

In order to assess changes in representational similarity between the two picture viewing tasks, we quantified pattern similarity changes as the difference of the respective correlation coefficients for every pair of events between the post-learning picture viewing task and its pre-learning baseline equivalent (Figure 3). Then, we analyzed how these difference values related to temporal relations between events, which we quantified using the absolute distances in virtual time (“virtual time”) between events (Figure 1C, bottom right). We further tested whether the effect of virtual time on anterior hippocampal pattern similarity change persisted when including the absolute difference between sequence positions (“order”) and the interval in seconds between events (“real time”) as control predictors of no interest in the model. We separately tested the effect of virtual time for event pairs from the same or different sequences and used a Bonferroni- corrected α-level of 0.025 for these tests. Time metrics were z-scored within each participant prior to analysis. To implement these tests, we employed two approaches to model-based RSA that are described in detail below.

Summary Statistics Approach

In the summary statistics approach, we used the different time metrics as predictors for pattern similarity change. We set up a GLM with the given variable from the day learning task as a predictor and the pairwise representational change values as the criterion for every participant. The t-values of the resulting model coefficients were then compared to a null distribution obtained from shuffling the dependent variable of the linear model (i.e. pattern similarity change) 10,000 times. This approach to permutation- testing of regression coefficients controls Type I errors even under situations of collinear regressors¹¹⁰. Resulting p-values for each coefficient were transformed to a Z-score. The Z-scores were then used for group-level inferential statistics.

Group-level statistics were carried out using permutation-based procedures. For t-tests, we compared the observed t-values against a surrogate distribution obtained from 10,000 random sign-flips to non-parametrically test against 0 or to assess within- participant differences between conditions (two-sided tests; α=0.05 unless stated otherwise). We report Cohen’s d with Hedges’ correction and its 95% confidence interval as computed using the effsize- package for R. For paired tests, Cohen’s d was calculated using pooled standard deviations and confidence intervals are based on the non-central t- distribution. Permutation-based repeated measures ANOVAs were carried out using the permuco- package¹¹¹ and we report generalized η² as effect sizes computed using the afex-package¹¹².

Linear Mixed Effects

Second, we employed linear mixed models to assess how learned sequence relationships were reflected in pattern similarity change using the lme4 package¹¹³. Mixed models have the advantage of estimating fixed effects and their interactions using all data, rather than performing inferential statistics on just one value per participant. We used the different time metrics as the fixed effects of interest. Factorial predictors (region of interest: anterior hippocampus and anterior-lateral entorhinal cortex; sequence: same vs. different) were deviation-coded. Within-subject dependencies were captured using random effects. Following the recommendation by Barr et al.¹⁰⁷, we always first attempted to fit a model with a maximal random effects structure including random intercepts and random slopes for participants. If these models did not converge or resulted in singular fits, we reduced the random effects structure. We always kept random slopes for the fixed effect of interest in the model to avoid anti-conservativity when testing fixed effects or their interactions^{107, 114}. The mixed effects models were fitted using maximum likelihood estimation.

We assessed the statistical significance of fixed effects of interest using likelihood ratio tests (α=0.05). Specifically, the model including the fixed effect of interest was compared against a nested, reduced model excluding this effect, but with the same random effects structure. Throughout the manuscript we report the results of these model comparisons (χ²-tests with one degree of freedom) and refer to supplemental tables for summaries of the final mixed model parameters. We visualize fixed effect estimates with their 95% confidence intervals as dot plots and further illustrate effects using estimated marginal means¹⁰⁸.

Low-dimensional embedding

We aimed to explore how hippocampal event representations could be embedded in a low- dimensional space to give rise to the positive and negative correlations of pattern similarity change and temporal distances for same-sequence and different- sequence events, respectively. For each pair of events, we generated an expected representational similarity using the mixed model fitted to hippocampal pattern similarity that captures the interaction between virtual temporal distances and sequence membership (c.f. Figure 5 and Supplemental Table 6). We used the fixed effects parameters to generate model predictions. We chose this approach over the raw pattern similarity values to obtain less noisy estimates of the pairwise distances. Using the smacof-package¹¹⁵, the model- predicted similarities were converted to distances and the resulting distance matrix (Supplemental Figure 4C) was subjected to non-metric multidimensional scaling using two dimensions. Because MDS is sensitive to starting values, we ran multidimensional scaling 1000 times with random initial configurations and visualized the resulting configuration with the lowest stress value. We further used the UMAP-package¹¹⁶ to explore a low- dimensional embedding of the same distance matrix based on a different algorithm (Supplemental Figure 4D, n nearest neighbors=6; minimum distance=0.1).

Searchlight Analysis

We further probed how temporal distances between events shaped representational change using searchlight analyses. Using the procedures described above, we calculated pattern similarity change values for search spheres with a radius of 3 voxels around the center voxel. Search spheres were centered on all brain voxels within our field of view. Within a given search sphere, only gray matter voxels were analyzed. Search spheres not containing more than 25 gray matter voxels were discarded. For each search sphere, we implemented linear models to quantify the relationship between representational change and the learned temporal structure. Specifically, we assessed the relationship of pattern similarity change and absolute virtual temporal distances, separately for event pairs from the same sequences and from pairs from different sequences. In a third model, we included all event pairs and tested for an interaction effect of sequence membership (same or different) predictor and virtual temporal distances. The t-values of the respective regressors of interest were stored at the center voxel of a given search sphere.

The resulting t-maps were registered to MNI space for group level statistics and spatially smoothed (FWHM 3mm). Group level statistics were carried out using random sign flipping implemented with FSL Randomise and threshold-free cluster enhancement. We corrected for multiple comparisons using a small volume correction mask including our a priori regions of interest, the anterior hippocampus and the anterior- lateral entorhinal cortex. Further, we used a liberal threshold of puncorrected<0.001 to explore the data for additional effects within our field of view. Exploratory searchlight results are shown in Supplemental Figure 7 and clusters with a minimum extent of 30 voxels are listed in Supplemental Tables 9, 11 and 12.

To test whether within- and across-sequence representations overlap, we defined an ROI based on the within-sequence searchlight analysis. Specifically, voxels belonging to the cluster around the peak voxel, thresholded at p<0.01 uncorrected within our small volume correction mask, were included. The analysis of representational change was then carried out as described for the other ROIs above.

Relationship to behavior

We used the regression coefficients quantifying the strength of the behavioral generalization bias to test for an across-subject relationship with the RSA searchlight effects. For each participant, we extracted the t-value of the across-sequence and the within-sequence searchlight effects from the peak voxel in our a priori regions of interest. We chose this approach because the searchlight analysis provides increased spatial precision relative to the ROI analyses. We used Spearman correlations to test for a relationship of the RSA searchlight effects and the behavioral generalization bias (α=0.025, corrected for two comparisons).

Data & Code Availability

Analysis code and documentation is available on GitHub (https://jacbel.github.io/virtem_code/ovierview). Data to reproduce the statistical analyses reported in this manuscript will be made openly available upon publication.

Author Contributions

J.L.S.B., L.D, N.D.M. and C.F.D. conceived the experiment and analyses. N.D.M. and L.D. developed the tasks and acquired the data. L.D. preprocessed the MRI data. J.L.S.B. analyzed the data and wrote the manuscript with input from C.F.D. All authors discussed the results and contributed to the paper.

Competing interests

The authors declare no competing interests.

Supplemental Figures

Supplemental Figure 1. Overview of the event images used as stimuli.

All scenes were devoid of windows to exclude diurnal cues, such as shadows or light color, and were selected so they would be plausible at any time of day. For each participant, event images were randomly allocated to sequences and event times. Event images were created using the life-simulation computer game The Sims 3 (Electronic Arts). The Sims 3 and screenshots of it are licensed property of Electronic Arts, Inc.

Supplemental Figure 2. Design of the day learning task.

A. Each of the four virtual days consisted of a sequence of five events. Event sequences are shown in virtual time, i.e. relative to the hidden clock. Less virtual time passes within the bottom two sequences because clock speed was manipulated between sequences. B. Event sequences shown in real time relative to the first event. A comparable amount of real time (in seconds) elapses during each event sequence despite different amounts of virtual time passing. C, D. Sequences in virtual and real time as shown in (A) and (B), respectively, but separately for each of the seven repetitions of each sequence during the learning task. Black diamonds indicate the time cues shown to one randomly chosen example participant during the task. Time cues varied across repetitions and differed across participants.

Supplemental Figure 3. The relationship of virtual time and hippocampal pattern similarity change is not driven by the first and last event of a sequence.

A. Z-values from the summary statistics approach show a significant positive effect of virtual time on pattern similarity change in the anterior hippcampus when competing for variance with a control predictor of no interest accounting for variance explained by whether pairs of events were made up from the first and last event of a sequence or not. B, C. Fixed effect estimate with 95% confidence intervals (B) and estimated marginal means (C) visualize the results of the corresponding mixed model. D. We implemented participant-specific regression analyses with order and real time distances as predictors of hippocampal pattern similarity change. The plot shows a significant effect of virtual temporal distances when tested on the residuals of these linear models. Thus, variance that cannot be explained by the other time metrics can be accounted for by virtual temporal distances. This analysis was conducted only using the summary statistics approach because the residuals of a mixed model are more difficult to interpret than those of participant-specific regression analyses using ordinary least squares. A,D. Circles show individual participant Z-values from the summary statistics approach; boxplot shows median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distribution shows probability density function of data points. * p<0.05

Supplemental Figure 4. Virtual time predicts hippocampal pattern similarity change for events from different sequences.

A, B. Fixed effect estimate with 95% confidence interval (A) and estimated marginal means (B) for the effect of virtual time on pattern similarity for events from different sequences are shown to illustrate the across-sequences generalization effect as observed in the linear mixed model analysis. This mixed model complements the summary statistics results shown in Figure 5A. * p<0.05. C. Scaled distance matrix showing distances between event pairs. This matrix was derived from the mixed model capturing the interaction effect of virtual temporal distances and sequence membership on hippocampal pattern similarity change and was used as input for multidimensional scaling and the UMAP algorithm (see Methods). D. Low-dimensional embedding of the twenty events as observed using UMAP. Shapes show the different events and color denotes virtual times of these events. The different lines connect the events belonging to the four sequences for illustration.

Supplemental Figure 5. Relationship of pattern similarity change and temporal distances between events from the same and different sequences in the anterior-lateral entorhinal cortex.

There was no statistically significant difference between correlations of virtual temporal distances and representational change in the anterior-lateral entorhinal cortex depending on whether event pairs were from the same or different sequences. Entorhinal representational change was negatively related to temporal distances between events from the same sequence (summary statistics: t24=-3.54, p=0.002,, d=-0.69, 95% CI [- 1.17, -0.27]; α=0.025, corrected for separate tests of events of the same and different sequences; three outliers excluded based on the boxplot criterion). The relationship between entorhinal pattern similarity change for events from different sequences was not statistically different from zero (summary statistics: t27=-1.60, p=0.122, d=-0.29, 95% CI [-0.69, 0.08]; α=0.025, corrected for separate tests of events of the same and different sequences). Circles show participant-specific Z- values from summary statistics approach; boxplot shows median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distribution shows probability density function of data points. ** p<0.01 after outlier exclusion

Supplemental Figure 6. Temporal signal-to-noise ratio in the anterior hippocampus and the anterior-lateral entorhinal cortex.

A. The temporal signal-to-noise ratio was quantified as the mean unsmoothed signal over time divided by its standard deviation. It was calculated for each voxel and then averaged across voxels in a region of interest. The temporal signal-to- noise ratio was higher in the anterior hippocampus (aHPC) than in the anterior-lateral entorhinal cortex (alEC, summary statistics: t27=12.43, p<0.001, d=1.99, 95% CI [1.65, 3.13]). Circles show individual participant values; boxplot shows median and upper/lower quartile along with whiskers extending to most extreme data point within 1.5 interquartile ranges above/below the upper/lower quartile; black circle with error bars corresponds to mean±S.E.M.; distribution shows probability density function of data points. *** p<0.001

Supplemental Figure 7. Exploratory searchlight results.

A. For same-sequence event pairs, clusters of voxels in which pattern similarity change correlated positively with temporal distances were detected in the frontal pole, frontal medial cortex and left entorhinal cortex (see Supplemental Table 9). B. Pattern similarity change correlated negatively with temporal distances between events from different sequences in the cerebellum and lingual gyrus (see Supplemental Table 11). C. The interaction effect, defined as correlations of temporal distances and pattern similarity change depending on whether pairs of events belonged to the same sequence or not, was observed in the occipital pole, lingual gyrus, frontal pole, temporal fusiform cortex and the intracalcerine sulcus (see Supplemental Table 12). A-C. Statistical images are thresholded at p<0.01 uncorrected for display purposes. No clusters outside the hippocampal-entorhinal region survived corrections for multiple comparisons.

Supplemental Figure 8. Generalization bias in individual participants.

A, B. Each panel shows the data from one participant. Each circle corresponds to one event. The x-axis indicates the average relative time of the events occupying the same sequence position in other sequences. The y-axis shows the signed error of constructed event times as measured in the timeline task. The regression line and its confidence interval are overlaid in red. Positive slopes of the regression line indicate that constructed event times are biased by the average time of events in the other sequences. A shows data from the main sample; B from the replication sample.

Supplemental Tables

Supplemental Table 1

Mixed Model: Virtual time explains constructed times with order and real time in the model

Supplemental Table 2

Mixed Model: Virtual time explains representational change for same-sequence events in the anterior hippocampus

Supplemental Table 3

Mixed Model: Virtual time explains representational change for same-sequence events in the anterior hippocampus when controlling for the effect of first-last event pairs

Supplemental Table 4

Mixed Model: Virtual time explains representational change for same-sequence events in the anterior hippocampus when including order and real time in the model

Supplemental Table 5

Mixed Model: Virtual time explains representational change for different-sequence events in the anterior hippocampus

Supplemental Table 6

Mixed Model: The effect of virtual time differs between same-sequence and different-sequence events in the anterior hippocampus

Supplemental Table 7

Mixed Model: Virtual time explains representational change in the anterior-lateral entorhinal cortex (all events)

Supplemental Table 8

Mixed Model: The effect of virtual time differentially depends on sequence membership in the anterior hippocampus and the anterior-lateral entorhinal cortex

Supplemental Table 9

Searchlight Analysis: Virtual time explains representational change for same-sequence events

Supplemental Table 10

Mixed Model: Virtual time explains representational change for different-sequence events in the peak cluster of the same-sequence searchlight analysis

Supplemental Table 11

Searchlight Analysis: Virtual time explains representational change for different-sequence events

Supplemental Table 12

Searchlight Analysis: Interaction of virtual time and sequence membership

Supplemental Table 13

Mixed Model: Behavioral generalization bias

Supplemental Table 14

Mixed Model: Behavioral generalization bias (replication)

Acknowledgements

The authors would like to thank Ignacio Polti for helpful discussions on the behavioral generalization bias, Iris M. Engelhard for making the data of the replication sample available, and Roland Benoit for helpful comments on a previous version of the manuscript. This work was supported by the Netherlands Organisation for Scientific Research (NWO-Vidi 452-12-009; NWO-MaGW 406-14-114), the European Research Council (ERC-CoG GEOCOG 724836) and the Max Planck Society. C.F.D.’s research is further supported by the Kavli Foundation, the Centre of Excellence scheme of the Research Council of Norway— Centre for Neural Computation (223262/F50), The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, the Jebsen Centre for Alzheimer’s Disease, and the National Infrastructure scheme of the Research Council of Norway—NORBRAIN (197467/F50). N.D.M. was funded by the Netherlands Organization for Scientific Research (NWO 453-15-005, awarded to Iris M. Engelhard) during the acquisition of the data used for the replication of the behavioral generalization effect.

Footnotes

• Major Changes: Theoretical framework was revised. Addition of multidimensional scaling analysis.

• https://jacbel.github.io/virtem_code/overview