Abstract
There is a rich behavioral literature on articulatory rehearsal for verbal stimuli, suggesting that rehearsal may facilitate memory, but few studies have examined the benefits for visual stimuli. Neural delay period studies have largely failed to control for the use of maintenance strategies, which make activity patterns during maintenance difficult to interpret. Forty-four participants completed a modified Sternberg Task with either novel scenes (NS) that contained semantic information or phase-scrambled scenes (SS) that lacked it. Participants were instructed to generate a descriptive label and covertly rehearse (CR) or suppress (AS, i.e., repeat “the”) during the delay period. Artifact-corrected delay period activity was compared as a function of maintenance strategy (CR vs. AS) and stimulus type (NS vs. SS). Performance on the working memory task for NS revealed that CR neither provided a short-nor long-term behavioral advantage on the delayed recognition task for CR. Interestingly, when task difficulty increased with SS, there was both a significant short-term as well as a long-term advantage. Comparison of sensor-level delay activity during the maintenance phase for NS and SS revealed two distinct patterns of neural activity for NS; there was greater amplitude in the beta range in the right parietal and centromedial regions. For SS, across all sensors during CR, the higher amplitude was observed in the upper alpha and beta ranges. The results suggest that rehearsal increased subsequent memory with SS but not NS. Moreover, neural modulation during the delay period depends on both task difficulty and maintenance strategy.
Introduction
Rehearsal and Working Memory
It has long been established that rehearsal benefits memory for verbal stimuli, such as words and numbers. It is often assumed that participants engage in cumulative rehearsal when confronted with a list of verbal stimuli to remember. However, recent research has suggested that rehearsal is less beneficial to memory (Souza and Oberauer, 2018), especially with regards to increasing list size and shorter presentation rates (Tan and Ward, 2008; Souza and Oberauer, 2018). Baddeley (1986) established the idea that rehearsal benefits memory, suggesting that the repetition of the to-be-remembered item will refresh the memory trace via the articulatory process in the phonological loop. Numerous studies support that blocking rehearsal with articulatory suppression, repeating a word such as “the” over and again, decreases performance as compared with when someone rehearses (Baddeley et al., 1984; Baddeley, 2012). Recently, Souza & Oberauer (2018) suggested that rehearsal may only provide a benefit for stimuli that have a simple phonological representation and an additional component like semantic representation. How does a conclusion like this apply to complex visual stimuli?
Complex visual stimuli contain rich details and are easy to provide a semantically meaningful label to (Wright et al., 1990). Combining of visual information with a semantic label may result in a deeper level of encoding because the stimulus is encoded in both the visual and verbal domains (Paivio, 1969; Craik and Lockhart, 1972; Nelson and Reed, 1976; Ensor et al., 2019). It has also been suggested that the addition of the verbal label to a visually encoded stimulus does not improve memory for the stimulus, rather the added benefit of labeling is dependent on whether or not semantic associations are automatically accessed without labeling (Nelson and Reed, 1976). Few studies have addressed whether rehearsal for complex visual stimuli will benefit performance on a subsequent memory test. Research on simple stimuli, which lack semantic representations, have suggested that rehearsal may benefit memory for visual stimuli. But the question remains, does rehearsal benefit memory for complex visual stimuli?
Delay Activity and Working Memory
To understand the mechanisms that support maintenance of encoded information and successful retrieval, it is critical to examine the neural activity during the delay period (Sreenivasan and D’Esposito, 2019). Delay activity is characterized as a period of increased and sustained activation throughout the delay period (Sreenivasan et al., 2014; Sreenivasan and D’Esposito, 2019). This traditional view of delay activity suggests that it is supported by persistent neuronal firing, which represents that information is active until a response is made (Constantinidis et al., 2018). Maintenance strategies can differentially engage brain regions which support those strategies (Weiss and Mueller, 2012). Attentional refreshing involves directing attention inward to selectively keep information active and largely engages attentional mechanisms (Cowan et al., 2005), while rehearsal implicates language areas (Henson et al., 2000), especially with regards to verbal stimuli (Baddeley, 2003). If maintenance strategies are not controlled for, delay period activity is difficult to interpret and may explain the recent challenges to the established patterns (Miller et al., 2018; Sreenivasan and D’Esposito, 2019). Similar to the behavioral rehearsal literature, delay period activity has been examined during maintenance of verbal stimuli and simple visual stimuli, but no study has characterized delay activity during maintenance of complex visual stimuli. Thus, the question remains, how does rehearsal influence delay period activity to support memory for complex visual stimuli?
The purpose of Experiments 1 and 2 is to understand how rehearsal and task difficulty impacts performance on a working memory task using complex visual stimuli. It is hypothesized that controlling for the maintenance strategy, namely rehearsal versus suppressing rehearsal, will result in differences in the behavioral performance and delay period activity. Regardless of task difficulty, it is predicted that rehearsal will provide a behavioral advantage over suppressing rehearsal. The delay activity during rehearsal will be continuous and sustained throughout and correlated with performance.
Materials and Methods
Participants
The study was approved by the Institutional Review Board of the City College of New York Human Research Protection Program. A total of 54 participants signed the informed consent and completed the study. Participants were compensated with either 15 dollars or one extra course credit per hour of participation. The behavioral task was recorded as part of an EEG study and took approximately 2 hours to complete.
Experiment 1
One participant was excluded from Experiment 1 of the study because of failure to follow instructions. The final sample included in Experiment 1 of the study consisted of 29 participants (age = 25.4 (8.1) years, 14 females). For the EEG analysis a total of 6 participants were excluded from Experiment 1 of the study, 5 participants were excluded for noisy EEG recordings or difficulty with data collection, and 1 participant was excluded for failing to follow instructions. The final sample for Experiment 1 of the study consisted of 24 participants (age = 25.8 (8.6) years, range 18-56, 11 females).
Experiment 2
To confirm that ceiling effects were not biasing the findings of Experiment 1, Experiment 2 was conducted. Experiment 2 is a replication of Experiment 1 with different stimuli to reduce the overall performance. Four participants were excluded from Experiment 2 of the study because of computer malfunction while recording the behavioral responses. The final sample for Part 2 of the study consisted of 20 participants (age = 24.8 (9.5) years, range 18-56, 12 females).
Experimental Design and Statistical Analyses
Task
Participants completed a modified version of a Sternberg Task (Figure 1; Sternberg, 1966). The task consisted of 2 working memory tasks (100 trials each) and a delayed recognition task (150 trials). During the working memory task, participants were presented with a fixation cross (1 sec) that indicates the start of the trial, followed by 2 images in succession (2 sec each), a blank screen during the delay period (6 sec), a probe choice (2 sec), which is either one of the earlier presented images or a new image, and a phase-scrambled image (1 sec) that indicates the end of the trial. During presentation of images, participants were instructed to generate a verbal label for the image, and the delay period they were instructed to rehearse covertly (i.e. using their inner voice) the verbal label throughout the entire delay period (termed Rehearsal) or were prevented from actively rehearsing (termed Suppression). For the later condition, participants were discouraged from generating a verbal label and instructed to repeat the word "the" throughout the delay period (Baddeley et al., 1975; Landry and Bartling, 2011). Participants were given examples of labels as well as the rate at which they should rehearse during practice trials before beginning.
Participants completed both the Rehearsal and Suppression conditions in a randomized order. The participant made probe choices on an RB-530 response pad (Cedrus Inc). If a probe matched one of the previously presented encoding set, the participant would press the green (right) button on the response pad. If the probe did not match the encoding set, the participant would press the red (left) button. Participants completed the delayed recognition task approximately 10 minutes after the completion of the working memory tasks. During this short break, participants remained in the lab. The recognition task was a mix of any encoding image from either the Rehearsal and Suppression conditions (40 images from each condition), as well as new images (70 images). During the recognition task, the participant indicated if the image was presented in either of the working memory conditions (Rehearsal or Suppression) or a new image. If they indicated that they saw the image in one of the earlier working memory conditions, they were asked to indicate if they remembered labeling the image and verbally stated the label that was used. The experimenter recorded the verbal label.
Stimuli
In Experiment 1, the stimuli consisted of high-resolution, colored outdoor scenes, which did not contain any people’s faces or words. The images were randomly selected from the SUN database (Xiao et al., 2010) and were resized to 800 by 600 pixels. Experiment 2 employed the same study design as Experiment 1 with phase-scrambled versions of the naturalistic scenes used in Experiment 1 (Figure 2). Importantly, the images contained the same colors and spatial frequencies as the images used in Experiment 1 but lacked in semantic content and were more challenging to generate labels because phase-scrambling removes all semantic content. The images were Fourier phase-scrambled in Matlab v9.1 (R2016b).
Behavioral Analysis
The behavioral data were processed in Python 3.0, and the corresponding figures were created using Seaborn 0.9.0 in Python 3.0. Statistical analysis was conducted using JASP v0.9.0.1. Paired-samples t-tests were used to compare behavioral accuracy between conditions on the WM tasks for both the Image and Scramble Study. Paired-samples t-tests were used to compare behavioral accuracy between the image types.
EEG Processing and Analysis
Continuous 64-channel EEG was collected at a sampling rate of 1 kHz using an active electrode system with actiCHamp system (Brain Products). All electrode impedances were lowered to 25 kOhms or below, per the manufacturer’s specifications. Electrodes with impedance above 25 were interpolated. The raw EEG data was processed in BESA Research v 6.1. Data was re-referenced offline to the average reference. Participants were only included in the EEG analysis if they had at least 50 delay periods that survived the artifact scan (amplitude less than 145 μV). Time-frequency analysis (TFA) was conducted on artifact-corrected delay period epochs (0 to 6000 ms). TFA was bandpass filtered between 4 Hz and 30 Hz and generated with 100 ms/0.5 Hz steps.
TFA absolute amplitude and temporal spectral analysis were generated in BESA Research. TFA absolute amplitude and temporal spectral analysis were compared using paired-samples t-tests with corrections for multiple comparisons in BESA Statistics v 2.0. Additionally, correlations were run between TFA temporal spectral analysis and performance with corrections for multiple comparisons.
Results
Experiment 1 and Discussion
Behavioral
Examination of performance on this WM task revealed that there was no significant difference in performance between rehearsal and suppression (.95 proportion correct vs. .95), t(28) = .70, p = .49, d = .13, suggesting that rehearsal did not provide a short-term behavioral advantage (Figure 3). Similarly, there was no long-term behavioral advantage on the delayed recognition task for rehearsal vs. suppression (.80 proportion correct vs. .78), t(28) = 1.38, p = .18, d = .23.
The behavioral results suggest that complex scenes may not benefit from rehearsal. It is also possible that the task was not difficult enough to benefit from rehearsal.
EEG
Sensor-level changes in absolute amplitudes between the two conditions (n=24 subjects) with corrections for multiple comparisons revealed 100 significant clusters (Table 1, p < .05). A cluster is a group of adjacent bins (sensor (<4 cm distance), time (100 ms), frequency (.50 Hz) bins), in which the difference in absolute amplitude between the two conditions is significantly different from a random permutation distribution (Maris and Oostenveld, 2007). For the rehearsal condition (Figure 4a – P8 electrode - orange clusters), amplitude was greater in the theta and beta range for the left frontal, bilateral fronto-temporal, and central regions, suggesting engagement of the phonological loop (Baddeley, 2003; Hwang et al., 2005) and in the beta range for the right parietal region, throughout the delay period. For the suppression condition (Figure 4b – F1 electrode - blue clusters), the amplitude was greater in the upper alpha and lower beta range in the mid-frontal regions early in the delay, and in the theta and upper alpha range in the midline and centro-frontal, right parietal, and occipital regions later in the delay.
Review of the change in amplitude over time (temporal spectral analysis) suggests that activity is increased and synchronous early in the delay period and begins to decrease later in the delay period. There was no significant difference between temporal spectral analysis during rehearsal versus suppression (p = .08). Additionally, temporal spectral analysis during rehearsal was not significantly correlated with working memory (p = .46) nor with recognition performance (p = .28).
Overall, the significant sensor-level difference in absolute amplitude suggests that participants engaged in different maintenance strategies; however, the change in delay activity over time was similar.
Experiment 2 and Discussion
Behavioral
The results show that when task difficulty increased, there was both a significant short-term advantage of rehearsal (Figure 5) as compared with suppression (.85 proportion correct vs. .78, t(19) = 7.93, p < .001, d = 1.77) as well as a long-term advantage for images from the rehearsal condition as compared with suppression (.71 proportion correct vs. .62, t(19) = 4.58, p < .001, d = 1.02).
The lack of behavioral difference for the complex scenes as compared with the phased-scrambled scenes suggests that the task difficulty explains the performance. These images were more difficult to generate a label because they lacked semantic content; therefore, this eliminated the automatic semantic association.
EEG
It was predicted that rehearsal and suppression would produce similar EEG delay period activity to the Experiment 1 sensor-level analysis since the task was the same.
Sensor-level examination of the absolute amplitude between the two conditions (n = 20 subjects) with corrections for multiple comparisons revealed 15 significant clusters (Table 2, p < .05). Greater amplitude was observed in the upper alpha and beta ranges across all sensors for the rehearsal condition (Figure 6a – P08 – orange cluster), as compared with the suppression condition. The pattern of delay activity appears to be both sustained and continuous throughout the entire delay period, as has been previously reported in the literature (Jensen et al., 2002; Tuladhar et al., 2007; Khader et al., 2010; Berger et al., 2014).
Review of the sensor-level temporal spectral analysis suggests that it is transient in nature, similar to the delay activity observed in Experiment 1. Activity is increased and synchronous in the early part of the delay period and begins to decrease later in the delay period. Comparison of the temporal spectral analysis between the rehearsal and suppression conditions revealed 3 clusters of significantly different activity (Table 3). Additionally, temporal spectral analysis during rehearsal was significantly correlated with working memory performance (Figure 6b; Cluster 1: blue, cluster value = −38801.2, p =.005, Cluster 2: orange, Cluster value = 20445.5, p = .065), but not performance on the recognition task (p = .62).
The significant sensor-level difference in absolute amplitude and temporal spectral analysis suggests that delay activity was modulated by maintenance strategy.
Discussion
Role of Rehearsal in Visual Memory
The role of rehearsal in supporting visual memory remains unclear, especially with regards to whether or not rehearsal benefits complex visual stimuli. Experiments 1 and 2 sought to understand how controlling for rehearsal strategy (rehearsal vs. suppression of rehearsal) influenced the short- and long-term memory for visual stimuli.
Experiment 1 used intact, novel outdoor scenes that contained semantic information (i.e. a beach or a farm) which were intended to elicit stored semantic associations automatically. There was no difference in performance on the short- or long-term memory task with intact scenes, which suggests that complex scenes do not benefit from this type of maintenance strategy. It has been suggested that complex scenes automatically trigger stored semantic associations (Ensor et al., 2019) which provide automatic deeper encoding (Craik and Lockhart, 1972). Consequently, the addition of rehearsing with a generated label offers no more benefit than accessing those stored associations. It is also plausible that task difficulty modulated the benefit of a maintenance strategy like rehearsal. Participants saw two images and within 6 seconds responded to whether or not the image was old or new. It has been established that humans can remember thousands of images after only seeing the images for a brief time (Standing et al., 1970; Standing, 1973; Brady et al., 2008). This ability has been termed the picture superiority effect (Stenberg, 2006) and may account for the fact that generating a semantic label and rehearsing provided no additional benefit. While it has been suggested that the addition of the semantic label provides a dual means of encoding (Paivio, 1969; Nelson and Reed, 1976), these results suggest that the semantic associations are automatically generated without recoding and rehearsal (Nelson and Reed, 1976; Ensor et al., 2019). Thus, it is not surprising that the performance was near ceiling.
Experiment 2 was conducted to increase task difficulty by using phase-scrambled images that lacked semantic content. While an automatic association of a label to a picture results in deeper encoding, this automatic association fails to occur with phase-shambled stimuli; therefore, the process of generating a label during encoding ensures that a deeper level-of-processing occurs (Craik and Lockhart, 1972; Ensor et al., 2019). Performance in Experiment 2 provides support for the assumption that the benefit of rehearsal on complex visual stimuli is modulated by task difficulty. More specifically, when participants generated a semantic label and rehearsed throughout the delay period, they engaged in deeper encoding and elaborative rehearsal (Cermak, 1971; Craik and Lockhart, 1972; Phaf and Wolters, 1993; Ensor et al., 2019). These findings are consistent with the idea that generating a label is only beneficial to visual stimuli when semantic information is not automatically accessed (Nelson and Reed, 1976). Whereas with the suppression condition participants engaged in more shallow encoding, relying solely on the visual information, and did not recode or rehearse; hence, performance was lower.
Delay Activity and Rehearsal
The delay period is a critical time during a working memory task when encoded information is maintained. Experiments 1 and 2 sought to understand how delay activity would change as a function of task difficulty (intact novel scenes vs. phase-scrambled scenes) and maintenance strategy (rehearsal vs. suppression). When intact scenes served as stimuli in Experiment 1 we observed greater activity in the left temporal and bilateral central regions, which suggests the engagement of the phonological loop (Baddeley, 2003; Hwang et al., 2005). Whereas, suppression results in the engagement of more frontal electrodes suggesting greater attentional demand (Camos et al., 2011) as well as greater mental effort (Kopp et al., 2006) involved in inhibiting rehearsal. Engagement of the parietal electrodes for both conditions may indicate storage of the images in a temporary visuospatial store (Baddeley, 2003). Delay activity during a WM task is often associated with the engagement of either the prefrontal cortex or the posterior parietal regions but has been established in studies that often fail to control for maintenance strategy. Activity in the parietal region has been suggested as the storage place for visual information during the maintenance phase of a working memory task. It serves as the buffer in which information lives until it is needed for retrieval, analogous to the verbal information store. Specifically, the lateral posterior parietal cortex could represent the area in the brain in which the generated verbal label is associated with the visually stored picture, consistent with the output hypothesis (Baddeley, 2000; Hutchinson, 2009). Activation in these regions, regardless of the connections with attentional networks, likely does not only reflect attentional processes (Hutchinson et al., 2009). The results of this study confirm that controlling for maintenance strategy, hence controlling for the cognitive domains that are involved, will recruit different brain regions (Sreenivasan and D’Esposito, 2019).
In experiment 2 the patterns of delay activity were different than the patterns observed in Experiment 1. For the rehearsal condition, activity was greater for all sensors, as compared with suppression. The simplest explanation is that differences in delay activity between the two studies can be attributed to differences in the task demands (Sreenivasan and D’Esposito, 2019). The stimuli used phase-scrambled scenes that were difficult to generate a label to. Although difficult-to-label images contain the same visual features as regular scenes (i.e., color and spatial frequency), they lack the automatic semantic associations. The easy-to-label images used in Experiment 1, on the other hand, had a definitive semantic association and a verbal label (Wright et al., 1990; Ensor et al., 2019). The generation of a label in Experiment 2 was more effortful than in Experiment 1, and often shallower in nature (i.e., colors and feature-related) both during the recoding process and rehearsal. Thus, the differential pattern of delay activity, particularly in the frontal regions during rehearsal, represents the process of recoding difficult-to-label images (i.e., engagement of bilateral fronto-temporal regions) and a more attention-demanding rehearsal period (i.e., engagement of centro-frontal regions).
Transient vs. Sustained Delay Activity
Elucidating the pattern of delay activity is the current focus in the working memory literature (Nature, 2019). While it has long been established that sustained activity observed during the maintenance phase when stimuli are no longer being encoded represented both maintenance of encoded information and focusing of attention inward, recent research has suggested that delay activity is more complex (Rose et al., 2016; Miller et al., 2018; Sreenivasan and D’Esposito, 2019). For example, only information in the focus of attention may be reflected in delay activity, while items outside the focus of attention may actually be represented by activity silent mechanisms (Stokes, 2015; Rose et al., 2016). Examination of the change in amplitude over time in both Experiments 1 and 2 suggests that when controlling for maintenance strategy, the pattern of delay activity is actually more transient. There is an early period of increased, synchronous activity (until approximately 3000 ms) followed by a period of desynchronous activity, regardless of stimulus type. This pattern of activity is consistent with recent reports that maintenance is not necessarily supported by persistent delay activity in prefrontal regions (Miller et al., 2018; Sreenivasan and D’Esposito, 2019); instead delay activity may reflect more complex processes going on throughout the cortex and deeper regions. Alternatively, previous reports of sustained delay activity could reflect a maintenance period in which participants did not utilize a particular strategy, rather they focused their attention inward until they were required to produce a response (Cowan et al., 2005). Thus, delay activity is a function of the strategy that is employed to maintain information (Sreenivasan and D’Esposito, 2019) as well as the task difficulty.
Limitations
Participants engaged in covert rehearsal and suppression to reduce the amount of noise introduced into the EEG signals. Thus, task compliance is based on participant confirmation during the recognition task (i.e., reported their generated labels). Additionally, the generated labels were reviewed during the recognition task to confirm compliance and were not systematically analyzed for the depth of encoding. Future studies should include a post-trial component during the working memory tasks to confirm task compliance when covert maintenance strategies are used.
Significance Statement
Rehearsal is a maintenance technique that is purported to benefit memory. Despite decades of research affirming the positive effect of rehearsal on memory, the benefit may be limited to certain types of stimuli. Understanding the neural process that underlie maintenance is also a critical area of research. Interestingly, few studies that examine neural patterns of maintenance, control for the technique used. The present study sought to understand how controlling for maintenance, namely rehearsal versus suppressing rehearsal, would influence delay activity. The results provide evidence that rehearsal and task difficulty both modulate the pattern of delay period activity. Moreover, the results suggest that rehearsal may only benefit complex visual stimuli that lack semantic content.
Conflict of Interest Statement
The authors declare no competing financial interests
Acknowledgements (Funding)
The City University of New York Graduate Center Doctoral Student Research Fund, Round 12