Abstract
The hippocampus is traditionally considered to be a system that is specialized for long-term memory. Recent work has challenged this notion by demonstrating that this region can contribute to many domains of cognition beyond long-term memory, including perception and attention. One potential reason why the hippocampus contributes broadly to cognition is that it contains relational representations — representations of multidimensional features of experience — that are useful in many different cognitive domains. Here, we explore the hypothesis that the hippocampus plays a critical role in attention via its relational representations. We compared human participants with hippocampal damage to healthy age- and education-matched individuals on attention tasks that varied in the type and amount of relational processing required. On each trial, participants viewed two images (rooms with paintings). On room relational trials, they judged whether the rooms had the same spatial layout from a different perspective. On art relational trials, they judged whether the paintings could have been painted by the same artist. Control trials placed fewer demands on relational processing: Participants simply had to detect identical paintings or rooms. Patients with hippocampal damage were significantly impaired on the room relational task. Selective hippocampal lesions had comparable effects to more extensive medial temporal lobe damage, suggesting that the hippocampus itself plays a critical role in spatial relational attention. This work provides further evidence that the hippocampus plays a ubiquitous role in cognition by virtue of its relational representations, and highlights that spatial relations may be particularly important.
Introduction
The human hippocampus has largely been studied for its critical role in the medial temporal lobe (MTL) "memory system", a system traditionally thought to be specialized for the formation and retention of long-term declarative memories (Cohen & Eichenbaum, 1993; Eichenbaum & Cohen, 2001; Milner et al., 1998; Squire & Wixted, 2011; Suzuki, 2009). Such a view has been challenged in recent years, with the emergence of a large body of work highlighting the many ways that the hippocampus and MTL contribute to cognition beyond long-term declarative memory (Aly & Turk-Browne, 2018; Nadel & Peterson, 2013; Olsen et al., 2012; Shohamy & Turk-Browne, 2013). This includes studies showing a role for the hippocampus in perception (Lee et al., 2012), working memory (Yonelinas, 2013), implicit memory (Hannula & Greene, 2012), decision making (Shohamy & Daw, 2015), imagination (Schacter et al., 2017), creativity (Rubin et al., 2015), language (Duff & Brown-Schmidt, 2012), and social cognition (Schafer & Schiller, 2018).
In addition to this expanding literature, we have recently discovered that the hippocampus contributes to online attention behavior (Aly & Turk-Browne 2016a; Aly & Turk-Browne 2016b; Cordova, Turk-Browne, & Aly, 2019): It exhibits distinct activity patterns for different attentional states, and the stability of these activity patterns predicts attentional performance. However, this work implicating the human hippocampus in attention comes from fMRI studies, which do not tell us if the hippocampus contributes a necessary function for attention behavior. Here, we sought to determine whether the hippocampus plays a critical role in attention, and to elucidate the nature of its contribution.
To that end, we tested patients with hippocampal damage on a modified version of the “art gallery” task we used previously, in fMRI studies, to demonstrate hippocampal involvement in attention (Aly & Turk-Browne 2016a, 2016b). This task allowed us to test two complementary hypotheses about how hippocampal function might support attention. Given the extensive literature demonstrating that a key aspect of hippocampal function is its relational representations (representations that link multiple features of an experience; Eichenbaum & Cohen, 2014; Konkel & Cohen, 2009; Olsen et al., 2012), one prediction is that the hippocampus may only play a necessary role in tasks that require attention to the relations between features (e.g., Cordova et al., 2019). An alternative hypothesis is that the hippocampus may only contribute to attention behaviors that require spatial representations. Such a prediction would be consistent with our previous fMRI work, in which the hippocampus was more strongly modulated by, and predicted behavior more strongly for, attention tasks that put demands on processing spatial vs. object relations (Aly & Turk-Browne 2016a, 2016b). This prediction is also consistent with models of hippocampal function that emphasize its importance for spatial cognition (Maguire & Mullally, 2013; O’Keefe & Nadel, 1978), as well as findings that hippocampal damage impairs perception of complex scenes but not complex objects (Lee et al., 2005a, 2005b, 2012).
To test these hypotheses, we designed a task that required different kinds of attention across different trials that utilized the same type of stimulus (3D-rendered rooms with paintings). Trials varied in whether they placed a heavy demand on relational processing (“relational” trials) or a lighter demand (“control” trials). They also varied in whether they required attention to spatial features (“room” trials) or object features (“art” trials).
This task has several important components. First, this approach allows us to determine how the hippocampus contributes to goal-directed attention when bottom-up stimulation is held constant, because the same type of stimulus is used across trials in which participants’ behavioral goals are different. Second, stimuli were briefly presented and trial-unique, so that long-term memory was neither required nor beneficial for task performance (e.g., Aly et al., 2013). Finally, on relational trials, low-level visual features were not diagnostic for task performance, increasing demands on the relational representations of the hippocampus (e.g., Hartley et al., 2007).
Together, these task features enabled us to rigorously test alternative theories of whether and how the hippocampus might critically contribute to attention.
Materials and Methods
Participants
Demographics and Recruitment
Patients with medial temporal lobe lesions (n = 7; 1 woman, 6 men; Mage= 41.0 years, Meducation= 17.0 years) were recruited via the New York University Patient Registry for the Study of Perception, Emotion, and Cognition (NYU PROSPEC) and the Department of Neurology at Columbia University Irving Medical Center. These patients had normal or corrected-to-normal vision and normal hearing. Neuropsychological test scores are shown in Table 1 and Table 2.
Healthy adults (n = 14; 9 women, 5 men; Mage= 42.0 years, Meducation= 15.8 years) were recruited via flyers posted around the Columbia University community. These participants reported no neurological or psychiatric illness, had normal or corrected-to-normal vision, and normal hearing. 12 of these individuals completed brief neuropsychological assessments. They scored in the normal range on the Montreal Cognitive Assessment (M = 28.36, SD = 1.22; max score = 30, 26+ is normal), the Mini-Mental State Examination (M = 29.00, SD = 1.41; max score = 30, 24+ is normal), and the Beck’s Depression Inventory (M = 4.50, SD = 3.71; scores below 10 are considered normal; one individual scored 13, indicating a mild mood disturbance).
Patients and healthy adults did not differ in age (t19= 0.12, p = 0.91, 95% CIs = −18.71 – 16.71) or education (t19= 0.93, p = 0.37, 95% CIs = −1.52 – 3.96).
Patient Descriptions
Patient 101 was recruited from the Columbia University Irving Medical Center. He suffered a hypoxic brain injury as a result of a period of asphyxiation and suspected cardiac arrest. An initial MRI revealed no signs of volume loss in the brain, but a follow-up FLAIR scan one year after the hypoxic event revealed hippocampal abnormalities. This was confirmed by a volumetric lesion analysis (see Lesion Analyses and Table 3). Relatively selective volume loss in the hippocampus is typically observed for mild hypoxia (Gadian et al., 2000; Hopkins et al., 1995; Rempel-Clower et al., 1996; Smith et al., 1984).
Patients 102 - 107 were recruited via NYU PROSPEC. They underwent unilateral temporal lobectomies for the treatment of intractable epilepsy. The surgeries were to the right hemisphere for patients 102 and 103, and to the left hemisphere for the remaining patients. The resected tissue was primarily the anterior hippocampus and surrounding medial temporal lobe cortex, as well as some sections of the lateral temporal cortex. Patient 106 additionally had a partial frontal lobe resection. Lesion masks are shown for 5 of the 6 temporal lobectomy patients in Figure 1A. Regions of lesion overlap include anterior hippocampus and medial temporal lobe cortex.
Lesion masks were not available for one patient (104), so representative slices from his MRI scan are shown in Figure 1B. This patient also had damage to the anterior hippocampus and medial temporal lobe cortex.
The study was approved by the Columbia University Institutional Review Board. All participants received monetary compensation ($15/hour for the experiment and for travel time). They gave written informed consent, filled out a demographics form, and completed neuropsychological examinations. These included the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005), the Mini-Mental State Examination (MMSE; Folstein et al., 1975), and Beck’s Depression Inventory (BDI; Beck et al., 1961). The temporal lobectomy patients additionally completed the Wechsler Adult Intelligence Scale (WAIS IV; Wechsler, 2008), the Logical Memory subtest of the Wechsler Memory Scale (WMS IV; Wechsler, 2009), the Warrington Recognition Memory Test – Faces (RMF; Warrington, 1996), the Brief Visuospatial Memory Test (BVMT; Benedict et al., 1996), the Rey Complex Figure Text (RCFT; Osterrieth, 1944; Rey, 1941), and the California Verbal Learning Test (CVLT-II; Delis et al., 2000) as part of a separate, extended neuropsychological evaluation conducted by neuropsychologists through NYU PROSPEC. The hypoxia patient completed the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Randolph, Tierney, Mohr, & Chase, 1998). The hypoxia patient and temporal lobectomy patients completed different neuropsychological tests because they were assessed at different hospitals that had different norms for which tests are conducted.
Lesion Analyses
Temporal Lobectomy Patients
Patient MRIs were first transformed to standard MNI space. Then two neuropsychologists manually traced the lesion masks in FSLview, using a semi-transparent overlap of three images (MNI standard brain, patient’s MRI, lesion mask). FLAIR images were consulted in an adjacent window to provide additional cues with respect to lesion extent. Lesion masks were traced in 3 planes (coronal, sagittal, horizontal), and then reviewed again for corrections. Lesion masks were available for 5 (out of 6) temporal lobectomy patients (Figure 1).
Hypoxic Patient
Volumetric analyses were conducted for patient 101, to characterize the nature and extent of his hippocampal damage (Table 3). Hippocampal and amygdala volumes were compared for this patient and three age- and education-matched adults (patient 101: 19 years old, 12 years of education; healthy adults: Mage = 18, Meducation = 13.7). First, Freesurfer was used to obtain intensity-normalized MRIs (via autorecon1). These images were then converted to NIFTI for use in an FSL (https://fsl.fmrib.ox.ac.uk) pipeline. After brain extraction, FIRST (Patenaude et al., 2011) was applied to automatically segment subcortical structures, including the hippocampus and amygdala. Hippocampal and amygdala segmentations were then manually edited by expert raters to correct imperfections from the automated approach. FAST (Zhang et al., 2001) was then used to obtain gray matter, white matter, and cerebrospinal fluid masks. These were summed in order to measure total intracranial volume. The volumes of the left and right hippocampus and amygdala were then divided by the total intracranial volume to correct for differences in overall head and brain size.
Stimuli
Participants viewed images of rooms with paintings (Figure 2). The rooms each contained multiple pieces of furniture, unique wall angles, and a single painting. The paintings were primarily of outdoor scenes and spaces; some also contained people. A subset of these stimuli has previously been used (Aly & Turk-Browne 2016a, 2016b).
Rooms were created in Sweet Home 3D (http://www.sweethome3d.com/). 80 rooms were created for the experimental stimuli and an additional 10 for practice. For each room, a second version (its “relational room match”) was created with a 30 degree viewpoint rotation (half rotated clockwise and half rotated counterclockwise). This “relational room match” had the same spatial layout of furniture and wall angles, but with altered visual content: wall colors were changed, and furniture was replaced with different exemplars of the same category (e.g., a table was replaced with a different table). An additional 10 rooms and their altered versions were created for a practice run of the task.
Paintings were chosen from the Google Art Project (https://artsandculture.google.com/). 80 artists were selected, and 2 paintings were chosen from each artist. The two paintings by each artist (the first painting and its “relational art match”) were similar in terms of style (e.g., choice and use of color, level of detail, brushstrokes) but not necessarily content. An additional set of paintings (10 artists/20 paintings) were selected for practice. None of the practice stimuli overlapped with the experimental stimuli. All stimuli were presented using Psychophysics Toolbox 3 in Matlab (http://psychtoolbox.org/).
Design and Procedure
The design and procedure were modified from Aly & Turk-Browne (2016a, 2016b). A main difference is that only two images were presented on each trial, rather than five. This was to reduce working memory demands, given the known impairments of hippocampal lesion patients on working memory tasks that require relational representations (e.g., Hannula, Tranel, & Cohen, 2006; Olson et al., 2006; see Yonelinas, 2013 for review).
A stimulus set of 480 unique images was generated such that each of the 160 rooms (80 original rooms and 80 relational room matches) were paired with 3 paintings, all from different artists (likewise, each of the 160 paintings [80 original paintings and 80 relational art matches] were paired with 3 different rooms).
From this stimulus set, 80 image groupings of 6 images each were created (one image grouping for each of the 80 trials in the main experiment). For each trial, one image was selected as the “base image” (the first image presented on that trial). The remaining images were ones that could potentially be presented as the second (comparison) image on that trial (Figure 2): a "relational art match" (a room that contained a different painting that was painted by the same artist as that in the base image); a "relational room match" (a room with the same spatial layout as the base image, from a different perspective); an "identical art match" (a room with an identical painting as the base image), an "identical room match" (a room that was identical to the base image); and a non-matching image (an image with a painting by a different artist and a room with a different layout). An image that was an art match (whether identical or relational) to the base image could not also be a room match (whether identical or relational), and vice versa. The same logic followed for an additional 10 image groupings created for a practice run of the task (10 trials).
The 80 trials were split into two tasks: 40 “art” trials and 40 “room” trials. Half of those trials were “control” trials and half were “relational” trials. Two images were presented on each trial. Control trials involved the presentation of a base image and one of the following: an image with an identical painting ("identical art match"), an image with an identical room ("identical room match"), or a non-matching image. Relational trials involved the presentation of a base image and one of the following: its relational art match (different painting by the same artist), relational room match (room with the same layout from a different perspective), or a non-matching image. These were called relational trials because they were hypothesized to tax the relational representations of the hippocampus (Eichenbaum & Cohen, 2014): representations of multidimensional components of an experience, such as the relationship between visual features or spatial layouts. Conversely, control trials did not require the same amount of relational processing: Instead, accurate performance on these trials can be accomplished by detecting repetitions of particular objects or colors at the same spatial locations.
On each trial, participants first viewed a cue, either “ART” or “ROOM” for 0.5 s (Figure 2A). This cue instructed the participant to attend to either the style of the paintings (ART) or layout of the rooms (ROOM). Following this cue, participants viewed a base image for 2.0 s, followed by a 0.5 s interstimulus interval, and then a second image for 2.0 s. The second image could be either an identical art match, an identical room match, a relational art match, a relational room match, or a non-matching image (Figure 2B). Finally, a probe was presented, either “ART?” or “ROOM?”. The probe stayed on the screen until the participant responded; there were no specific instructions to respond as fast as possible. Participants were instructed to respond “yes” if they thought there was a match in the probed dimension and “no” if they thought there was not a match. Responses were made with the 1 and 2 keys, respectively. Participants were to respond “yes” to an “ART?” probe if the two paintings were identical or if they were painted by the same artist (identical art match or relational art match), and “no” otherwise. Participants were to respond “yes” to a “ROOM?” probe if the two rooms were identical or if they had the same spatial layout from a different perspective (identical room match or relational room match), and “no” otherwise.
For each task (art or room) and trial type (control or relational), the probability that the attentional cue at the beginning of the trial matched the probe at the end was 80% (valid trials). On the remaining 20% of trials, the cue at the beginning of the trial did not match the probe at the end (invalid trials): Participants were told to attend to one feature (e.g., “ART”) and were probed about whether there was a match on the other feature (e.g., “ROOM?”). The purpose of invalid trials was to ensure that attention was engaged by the cue at the beginning of the trial: if so, participants should be better on valid vs. invalid trials (Posner, 1980).
On valid trials, the cued (and probed) match was present 50% of the time, the non-cued (and non-probed) match was present 25% of the time, and a non-matching image was present the remaining 25% of the time (hence, the correct answer was “yes” half the time and “no” half the time).
On invalid trials, the probed (but not cued) match was present 50% of the time, the cued (but not probed) match was present 25% of the time, and a non-matching image was shown the remaining 25% of the time (hence, the correct answer was “yes” half the time and “no” half the time).
The task was blocked: Participants completed 10 trials of a given attentional state before switching to the other (e.g., 10 trials with “ART” cues; 10 trials with “ROOM” cues, and so on). Half of the participants started with art attention and half started with room attention. Relational trials and control trials were intermixed, so that participants could not adopt different strategies during the viewing of the first image of each trial.
Participants first received instructions and were shown examples of all the different match types (identical art match, identical room match, relational art match, relational room match). Next, they completed 10 practice trials. Participants were required to perform at 80% accuracy to continue to the full experiment. Each person who was tested met this criterion and completed the full task. During the practice and full experiment, participants received feedback after a block of trials (“Wow! You are doing amazingly well! Keep it up!”, “You are doing very well! Keep it up!”, “You are doing ok! Keep it up!”, “This task is challenging, but keep trying!”), as well as the percentage of correct responses. For the practice, they received feedback after every 5 trials; for the full experiment, they received feedback every 10 trials.
Statistical Analyses
Analysis of variance (ANOVAs) and follow-up t-tests were conducted in Matlab. All reported p-values are two-tailed, and 95% confidence intervals are reported where appropriate. Effect sizes (partial eta squared [ηp2] for ANOVAs and Cohen’s d and dz for t-tests) were implemented following Lakens (2013). Stimuli, code, and data can be found on GitHub: https://github.com/alylab/artmusePatient.
Results
We analyzed behavioral sensitivity (A’: 1 = perfect, 0.5 = chance; Donaldson, 1992) and response times (RTs) to the "ART?" and "ROOM?" probes. A’ was chosen as the measure of behavioral sensitivity because it is non-parametric, and because it is the measure we have used in prior studies with a similar task (Aly & Turk-Browne 2016a; Aly & Turk-Browne 2016b).
Valid vs. Invalid trials
We first examined whether attention was effectively guided by the cue at the beginning of the trial. If so, participants should be more accurate and faster on valid vs. invalid trials. To that end, we conducted four separate 3-way repeated measures ANOVAs, two for patients and two for healthy participants. The dependent measures were A’ and RTs. The independent variables were condition (relational vs. control trials), attentional state (art vs. room trials), and validity (valid vs. invalid trials).
For A’, there was a significant main effect of condition for healthy participants [F1,13 = 8.68, p = 0.01, ηp2 = 0.40] reflecting better performance on the control vs. relational trials (this effect was marginally significant for patients [F1,6 = 5.83, p = 0.052, ηp2 = 0.49]). There was also a significant condition by attentional state interaction for healthy participants [F1,13 = 7.17, p = 0.019, ηp2 = 0.36]: Collapsing across trial validity, performance was better on control room vs. control art trials, but better on relational art vs. relational room trials. Importantly, there was a significant main effect of validity for both the healthy participants [F1,13 = 34.00, p = 0.0001, ηp2 = 0.72] and the patients [F1,6 = 33.82, p = 0.0012, ηp2 = 0.85], indicating that behavioral sensitivity was better on valid vs. invalid trials. No other main effects or interactions were significant [patients: all other ps > 0.16; healthy participants: all other ps > 0.094]. Moreover, A’ on invalid trials was not different from chance for any trial type, for either healthy participants or patients (all ps > 0.20).
For RT, there was a significant main effect of condition for healthy participants [F1,13 = 12.00, p = 0.0042, ηp2 = 0.48], reflecting faster RTs on control vs. relational trials (Mcontrol = 1.55s; Mrelational = 1.83s). In addition, there was a significant attentional state by validity interaction for healthy participants [F1,13 = 5.73, p = 0.032, ηp2 = 0.31]: Collapsing across condition, response times were faster on valid art vs. valid room trials (Mvalid art = 1.37s; Mvalid room = 1.48s), but faster on invalid room vs. invalid art trials (Minvalid art = 2.51s; Minvalid room = 2.17s). Importantly, there was a significant main effect of validity for both the healthy participants [F1,13 = 25.86, p = 0.0002, ηp2 = 0.67] and the patients [F1,6 = 7.49, p = 0.03, ηp2 = 0.56], indicating faster RTs on valid vs. invalid trials (healthy participants: Mvalid= 1.53s, Minvalid = 2.34s; patients: Mvalid = 1.21s, Minvalid = 2.25s). No other main effects or interactions were significant [patients: all other ps > 0.08; healthy participants: all other ps > 0.25].
Taken together, these results suggest that attention was effectively engaged by the cue at the beginning of the trial: Participants were faster and more accurate on valid vs. invalid trials. Furthermore, performance was not different from chance on invalid trials, indicating a particularly strong manipulation of attention. Having confirmed that attention was effectively modulated, we next focus analyses on valid trials.
Control Trials
We examined A’ and RTs on control trials with 2 (attentional state: art or room) by 2 (group: patient or healthy participant) mixed-model ANOVAs (Figure 3A). For A’, there was no main effect of attentional state [F1,19 = 3.66, p = 0.07, ηp2 = 0.16], no main effect of group [F1,19 = 3.19, p = 0.09, ηp2 = 0.14], and no group by attentional state interaction [F1,19 = 0.08, p = 0.78, ηp2 = 0.004]. Similarly, for RTs, there was no main effect of attentional state [F1,19 = 2.11, p = 0.16, ηp2 = 0.10], no main effect of group [F1,19 = 1.21, p = 0.28, ηp2 = 0.06], and no group by attentional state interaction [F1,19 = 2.63, p = 0.12, ηp2 = 0.12]. Thus, performance was well-matched on the art and room control trials, and there was no statistically significant difference between patients and healthy participants.
Performance on the control trials was quite high overall. Thus, one concern is that the patients may not be impaired because the task was simply too easy — i.e., that there were ceiling effects. We believe this is not a concern for two reasons. First, as noted below, an individual patient did perform significantly worse than age-matched healthy individuals on room control trials (see Case Study: Hypoxia Patient), indicating that this task was sensitive to behavioral impairments. Second, both healthy participants (control art: t13 = 6.75, p = 0.00001, 95% CIs = 0.91 – 0.95; control room: t13 = 4.51, p = 0.0006, 95% CIs = 0.94 – 0.98) and patients (control art: t6 = 3.68, p = 0.01, 95% CIs = 0.83 – 0.97; control room, t6 = 3.02, p = 0.02, 95% CIs = 0.88 – 0.98) performed significantly below perfect performance (i.e., A’ of 1) on the control trials. Thus, although performance on these trials was high, it was not at ceiling.
Relational Trials
We next examined A’ and RTs on relational trials using the same 2 (attentional state: art or room) by 2 (group: patient or healthy participant) mixed-model ANOVAs (Figure 3B). For RTs, there was no main effect of attentional state [F1,19 = 0.21, p = 0.65, ηp2 = 0.01], no main effect of group [F1,19 = 2.63, p = 0.12, ηp2 = 0.12], and no group by attentional state interaction [F1,19 = 0.35, p = 0.56, ηp2 = 0.02]. For A’, there was a main effect of attentional state [F1,19 = 21.85, p = 0.0001, ηp2 = 0.53], a main effect of group [F1,19 = 19.16, p = 0.0003, ηp2 = 0.50], and a significant group by attentional state interaction [F1,19 = 17.80, p = 0.0004, ηp2 = 0.48]. Given this interaction, we conducted follow-up t-tests to compare A’ values for healthy participants and patients on the art and room relational trials.
Patients and healthy participants were not different on art relational trials [t19 = 0.003, p = 0.99, 95% CIs = −0.08 – 0.08, Cohen’s d = 0.001]. However, patients were significantly impaired, relative to healthy participants, on room relational trials [t19 = 5.66, p = 0.00002, 95% CIs = −0.35 – −0.16, Cohen’s d = 2.77]. This selective impairment was not a result of differing task difficulty for the art vs. room relational trials, because healthy participants performed just as well on both trial types [t13 = 0.43, p = 0.67, 95% CIs = −0.08 – 0.06, Cohen’s dz = 0.11]. Patients, however, performed significantly worse on room vs. art relational trials [t6 = 4.70, p = 0.0033, 95% CIs = −0.41 – −0.13, Cohen’s dz = 1.92], and in fact their performance on room relational trials was not different from chance [t6 = 1.64, p = 0.15, 95% CIs = 0.46 – 0.70; chance = 0.5]. Patients were significantly above chance (all ps < 0.0001) on all other trial types (control art, control room, relational art; healthy participants performed above chance on every trial type, all ps < 0.0001).
The above results are based on A’, a measure of behavioral sensitivity that includes both hits and false alarms. Thus, it is unclear from the above results whether the patients’ impairment on room relational trials is a result of a reduced hit rate, an increased false alarm rate, or both. We therefore compared the hit and false alarm rates for patients and healthy participants. Patients had significantly reduced hit rates on room relational trials, relative to healthy participants [t19 = 2.88, p = 0.0095, 95% CIs = −0.54 – −0.09, Cohen’s d = 1.40], but false alarm rates did not differ [t19 = 0.78, p = 0.44, 95% CIs = −0.10 – 0.23, Cohen’s d = 0.39]. There was no difference between patients and healthy participants in hit or false alarm rates on any other trial type (all ps > 0.13).
Taken together, patients were significantly and selectively impaired on room relational trials, and their performance was not different from chance.
Comparison of Control and Relational Trials
Patients showed statistically significant impairment only on room relational trials. However, on control trials, the main effect of group (patients vs. healthy participants) approached significance [F1,19 = 3.19, p = 0.09, ηp2 = 0.14]. We therefore directly compared control and relational trials to test whether patients were significantly more impaired on relational trials, but only when attention was directed to room layouts. To this end, we conducted two 2 (group: patient or healthy participant) by 2 (condition: control or relational trials) mixed-model ANOVAs, one for art trials and one for room trials.
For art trials, there was a main effect of condition [F1,19 = 9.21, p = 0.007, ηp2 = 0.33], indicating better performance on control vs. relational trials. However, there was no main effect of group [F1,19 = 0.39, p = 0.54, ηp2 = 0.02] nor a group by condition interaction [F1,19 = 0.57, p = 0.46, ηp2 = 0.03]. Thus, there was no statistically significant difference between the performance of patients and healthy participants on the art control or art relational trials. For room trials, there was a main effect of condition [F1,19 = 103.36, p < 0.00001, ηp2 = 0.84], indicating better performance on control vs. relational trials. There was also a main effect of group [F1,19 = 27.97, p = 0.00004, ηp2 = 0.59] and a group by condition interaction [F1,19 = 25.37, p = 0.00007, ηp2 = 0.57]. This result indicates that patients were significantly more impaired on room relational trials than room control trials. Indeed, comparison of patients and healthy participants on room control trials revealed no statistically significant difference [t19 = 1.05, p = 0.31, 95% CIs = −0.06 – 0.02, Cohen’s d = 0.51].
Case Study: Hypoxia Patient
Although every patient had damage to the hippocampus, many of them had damage to the surrounding medial temporal lobe cortex as well (Figure 1). To test whether the patients’ impairment on room relational trials can be attributed specifically to hippocampal damage, we conducted a follow-up analysis on the behavioral performance of the hypoxia patient (patient 101), who exhibited selective bilateral hippocampal damage (Table 3). We compared him to healthy participants (n=4) who were matched in age (patient 101: 19 years old, healthy participants: Mage= 20.00 years) and education (patient 101: 12 years, healthy participants: Meducation= 14 years).
To conduct this analysis, we used a method derived by Crawford and Howell (1998; also see Crawford, Garthwaite, & Howell, 2009) that allows for the comparison of a single patient against a sample of healthy participants. Patient 101’s performance (A’) did not differ from that of healthy participants on art trials [art control: t3 = 1.63, p = 0.90, 95% CIs = −0.06 – 0.17; art relational: t3 = 0.49, p = 0.33, 95% CIs = −0.17 – 0.13]. However, he was significantly impaired on both types of room trials [room control: t3 = 3.34, p = 0.02, 95% CIs = −0.22 – −0.002; room relational: t3 = 4.36, p = 0.01, 95% CIs = −0.70 – −0.12]. Nevertheless, his performance on room control trials was well above chance (0.83), while his performance on room relational trials was exactly at chance (0.50). Thus, selective bilateral hippocampal damage impairs spatial attention, and spatial relational attention seems to be more strongly affected.
The temporal lobectomy patients exhibited a similar pattern of results. They were significantly impaired on the room relational task relative to age- and education-matched healthy participants, but performed normally on all other trial types [art control: t14 = 1.59, p = 0.14, 95% CIs = −0.10 – 0.015, Cohen’s d = 0.87; room control: t14 = 0.44, p = 0.67, 95% CIs = −0.05 – 0.03, Cohen’s d = 0.25; art relational: t14 = 0.38, p = 0.71, 95% CIs = −0.08 – 0.11, Cohen’s d = 0.21; room relational: t14 = 4.37, p = 0.0006, 95% CIs = −0.32 – −0.11, Cohen’s d = 2.41]. As for the hypoxia patient, the temporal lobectomy patients were not above chance on room relational trials [t5 = 1.68, p = 0.15, 95% CIs = 0.45 – 0.74; chance = 0.5].
Thus, selective hippocampal damage is sufficient to impair spatial relational attention, and sufficient to bring performance down to chance levels. This suggests that the impairment on room relational trials at the level of the patient group may be a result of hippocampal damage specifically.
Discussion
Summary
We examined whether the hippocampus plays an essential role in attention, and what its contribution might be. We tested individuals with hippocampal damage and healthy age- and education-matched participants on attention tasks that varied in their demands on relational and spatial processing. We found that both spatial and relational representations were important components of hippocampal contributions to attention: Patients with hippocampal damage were selectively impaired on attention tasks that taxed spatial representations, and spatial relational representations in particular.
These results provide the first evidence, to our knowledge, that the hippocampus plays a critical role in online attention behavior. They add to a growing body of literature highlighting the far reach of the hippocampus in cognition, including perception (Lee et al., 2012), working memory (Yonelinas, 2013), implicit memory (Hannula & Greene, 2012), decision making (Shohamy & Daw, 2015), imagination (Schacter et al., 2017), creativity (Rubin et al., 2015), language (Duff & Brown-Schmidt, 2012), and social cognition (Schafer & Schiller, 2018). These findings, and our new results, together pose a strong challenge to theories of hippocampal function that view it as a system that is dedicated for long-term memory (Squire & Wixted, 2011).
Relation to Prior Work
Although we have emphasized the attentional demands of our task, the task also placed demands on perception. Indeed, we believe it is very difficult (or impossible) to study attention separately from perception, because the key behavioral marker of attention is improvements in perceptual behavior. Our findings therefore complement — and extend — studies on perception in hippocampal amnesia (Lee et al., 2012; Yonelinas, 2013). For example, studies of perception often find that patients with hippocampal damage are selectively impaired on tasks that use scenes as stimuli, and not those that use faces, objects, art, or colored shapes (Barense et al., 2005, 2007; Behrmann et al., 2016; Graham et al., 2006; Lee et al., 2005a, 2005b; but see Erez et al., 2013; Warren et al., 2010, 2011; also see Goodrich & Yonelinas, 2016). Among studies finding perceptual impairments for non-scene stimuli with hippocampal damage, a key feature is that the task often requires relational processing (e.g., Warren et al., 2011, 2012). That both spatial and relational processing are important (Aly et al., 2013; Hannula, Tranel, & Cohen, 2006) is supported by findings that scene perception impairments in hippocampal lesion patients are pronounced when the task involves changes in perspective, increasing the demands on relational processing (e.g., Behrmann et al., 2016; Erez et al., 2013; Lee et al., 2005b; also see King et al., 2002).
Here, we find converging evidence that spatial and relational processing are both critical features of hippocampal function: Attention to spatial features was more impaired than attention to artistic features, but only when relational processing demands were high.
However, our results go beyond these perception studies in a number of ways. First, we presented stimuli for a relatively brief amount of time, whereas many studies of perception in patients with hippocampal lesions present stimuli for a longer duration. For example, several studies present images until the participant responds (e.g., Barense et al., 2007; Behrmann et al., 2016; Erez et al., 2013; Graham et al., 2006; Lee et al., 2005b; Hartley et al., 2007; Warren et al. 2011), and/or require individuals to remember a target stimulus across many trials (e.g., Barense et al., 2005; Lee et al., 2005a). Although such paradigms convincingly tax perception and the results from them are difficult to explain solely in terms of long-term memory impairments, these paradigms nevertheless leave room for attention, perception, working memory, and/or long-term memory to interact to contribute to performance. Here, we attempted to minimize the influence of working- and long-memory and tax rapidly evolving attention and perception (e.g., Mullally, Intraub, & Maguire, 2012) by using relatively brief stimulus presentations, very short inter-stimulus intervals, and trial-unique images.
Second, we manipulated participants’ attentional states (e.g., attention to artistic style vs. room layouts) while holding the type of stimulus constant. In contrast, studies of perception in patients with hippocampal damage investigate different kinds of perception by varying the stimulus itself (e.g., presenting paintings vs. scenes on different trials; Lee et al., 2005a).
Finally, we designed the relational tasks so that low-level visual features were not particularly useful for task performance: only abstract relations could be used to accurately guide behavior. In contrast, in some studies of perception in hippocampal amnesics, feature- or item-level information can be sufficient for task performance, even if the intention is to only manipulate relational information (see Aly et al., 2013; Baxter, 2009). The approach that we took for relational trials — preserving relational information across images that varied in low-level visual features (e.g., Hartley et al., 2007) — is complementary to approaches taken in some studies of perception, where relational information is manipulated across images that are otherwise identical in low-level visual features (e.g., Aly et al., 2013; Behrmann et al., 2016; Erez et al., 2013; Lee et al., 2005b).
Limitations of Patient Studies
We have emphasized the hippocampal and medial temporal lobe lesions in the patients involved in the current study. However, it is important to note that temporal lobe epilepsy is associated with abnormalities in, and reduction in gray matter throughout, widespread brain networks (e.g. Bonilha et al., 2004). While temporal lobectomy surgical lesions can be relatively focal, the brains of these patients may have disrupted functioning in regions anatomically or functionally connected to the hippocampus and medial temporal lobe cortex.
Likewise, hippocampal damage (from etiologies beyond temporal lobe epilepsy) can cause broader network abnormalities. Even in cases of rather focal hippocampal atrophy, volumetric changes to the extended hippocampal network, such as the entorhinal cortex and thalamus, can be observed (Argyropoulos et al., 2019). Functional abnormalities have also been noted, e.g., decreases or changes in functional connectivity, including between non-hippocampal regions, and changes in overall activity outside of the hippocampus (Argyropoulos et al., 2019; Henson et al., 2016).
As with any lesion study, it is therefore difficult to determine exactly which aspect of disrupted brain functioning is key for the observed behavioral impairments. However, because all of the patients had hippocampal damage, it is likely that this damage either directly or indirectly (through its effects on other brain regions) is an important determinant of the behavioral deficits. That said, the mechanisms by which these behavioral impairments arise is not clear from the current study. There are at least two possibilities: (1) the hippocampus itself is involved in assessing relational similarities between scenes, or (2) the hippocampus provides important input or output to other regions that are themselves involved in assessing relational similarities between scenes. Functional neuroimaging studies of patients with hippocampal lesions will be informative in this regard: Such studies can illuminate how broader network function might be disrupted in these patients while they are performing the attention tasks used here. This would shed light on how and why hippocampal damage impairs performance. Nevertheless, we can conclude that the hippocampus is critical for spatial relational attention — because lesions to this area impair this form of attention — even if the mechanisms by which this happens are not yet clear.
Finally, it is worth noting that the temporal lobectomy patients had unilateral hippocampal lesions. As a result, some memory functions — as assessed by neuropsychological testing — are only mildly rather than severely impaired. Thus, the attentional deficits these patients exhibit in the current study are notable: One could have predicted that they would not have been impaired due to the remaining, intact hippocampus. The current results therefore suggest that unilateral hippocampal damage may be sufficient for impairments in spatial relational attention. Relational attention tasks such as the one used here may be particularly sensitive to reductions in hippocampal integrity.
Future Directions
The evidence reported here is a valuable contribution to the literature given the difficulty of finding, characterizing, and testing patients with medial temporal lobe damage. Many studies of amnesic patients test only a few individuals, given the rarity of the population of interest. Our sample size is comparable to, or larger than, studies investigating perceptual impairments in patients with medial temporal lobe damage (e.g., Aly et al., 2013; Barense et al., 2005, 2007; Behrmann et al., 2016; Erez et al., 2013; Graham et al., 2006; Hartley et al., 2007; Lee et al., 2005a, 2005b; Warren et al., 2010, 2011, 2012). As for any patient study, however, generalizing our results by testing more patients with medial temporal lobe damage — particularly selective hippocampal lesions — will be important. This is especially the case because the patient with selective hippocampal lesions, unlike the patients with temporal lobectomies, also exhibited a spatial attention impairment on trials that did not place heavy demands on relational processing (the impairment was mild relative to that on spatial relational trials). Thus, one possibility is that bilateral hippocampal damage produces attention impairments when spatial representations are required, even if relational processing demands are minimal.
The patient results reported here converge with our previous fMRI studies, which demonstrated that the hippocampus is more strongly modulated by attention to room vs. art relations, and predicted behavior most strongly for spatial relational attention (Aly & Turk-Browne 2016a, 2016b). The patient and neuroimaging findings therefore converge in showing an important role for the hippocampus in attention, and particularly attention to spatial relations.
One potential caveat to this conclusion is that it is possible that the art relational task was simply not as "relational" as the room relational task. In the room relational task, many types of features had to be bound and compared (e.g., angles and lengths of walls, placement and type of furniture). Conversely, in the art relational task, individuals may have chosen to focus on the choice of colors, the style of the painting (e.g., Impressionism vs. Pointillism), or broad categorical features (e.g., nature scene vs. city scene). Although none of these strategies would be perfect (because choice of color, style, and content were not individually diagnostic of paintings by the same artist), it is nevertheless possible that there was less relational processing on art vs. room trials. For example, individuals may have chosen to treat the paintings as a “unitized” whole rather than in terms of associations between individual features (Mayes, Montaldi, & Migo, 2007). Thus, a key difference between the art and room relational task may be in the amount of relational processing required, in addition to the requirement to attend to object-based vs. spatial features (or, alternatively, smaller vs. larger parts of the image). That said, these tasks were equally difficult for healthy individuals, alleviating the concern that the art relational task was simply less challenging or less complex. The fact that the art and room relational tasks were equally difficult for healthy individuals is also key for eliminating the concern that the patients are simply more impaired on harder tasks.
Nonetheless, to comprehensively demonstrate that the hippocampus is selectively involved in relational attention when such attention taxes spatial representations, other forms of relational attention must be examined (for a similar approach in memory, see Konkel et al., 2008). One promising approach is to investigate whether the hippocampus plays a critical role in attention to temporal relations. In a recent fMRI study (Cordova et al., 2019), we found that the hippocampus is more strongly modulated by attention to temporal vs. size or spatial relations in a rapid, relatively simple stimulus display. A role for the hippocampus in attending to temporal relations would be consistent with a vast literature implicating the hippocampus in temporal and sequential processing (Aly et al., 2018; Barnett et al., 2014; Davachi & DuBrow, 2015; DuBrow & Davachi, 2014; Eichenbaum, 2013; Palombo et al., 2016; Ranganath, 2019; Thavabalasingam et al., 2019). For example, neuropsychological studies indicate that the hippocampus plays an essential role in estimating the temporal duration of events (Palombo et al., 2016; Palombo & Verfaellie, 2017). Thus, one compelling avenue for future research is to determine whether the hippocampus makes a critical contribution to temporal attention (e.g., Cordova et al., 2019; Nobre & van Ede, 2018), which would complement existing studies examining its role in temporal memory.
Conclusion
We find strong evidence that the hippocampus makes a critical contribution to attentional performance: Hippocampal damage impairs attention to spatial relations. Such an impairment was observed in a task that placed no demands on long-term memory, demonstrating the importance of hippocampal function even on the timescale of online visual attention. This evidence joins a growing body of work highlighting the ubiquity of hippocampal contributions to cognition, contributions that may be realized via its flexible, spatial, and relational representations.
Author Contributions
N.R. performed research, analyzed data, wrote the paper. M.M. performed research, contributed analytic tools. S.A. performed research. M.A. designed research, performed research, analyzed data, wrote the paper.
Acknowledgements
We would like to thank all the researchers and clinicians of the New York University Patient Registry for the Study of Perception, Emotion, and Cognition (NYU PROSPEC) for their involvement with patient recruitment and patient data collection. We would also like to thank Frank Provenzano, Jeanelle France, and Evie Sobczak for assistance with structural MRI analyses for patient 101, and Eren Gunseli for obtaining structural MRI scans for healthy individuals. This work was funded by an NSF CAREER Award (BCS-1844241) and a Zuckerman Institute Seed Grant for MR Studies (CU-ZI-MR-S-0001) to M.A.
Footnotes
Conflicts of Interest: None.