Effects of order on episodic memory of event times

How do people remember the time of an event? The nature of time encoding and decoding is a fundamental open question. Memorizing time of an event may employ two different processes (i) encoding of the absolute time of an event within an episode, (ii) encoding of its relative order compared to other events. Here we study interaction between these two processes. We performed experiments in which one or two items (either words or images) were presented within a certain time interval, after which participants were asked to report the time of occurrence of presented items. The results show that when a single item is presented, the distribution of reported times is quite wide, with the overall bias towards the middle of the interval. When two items are presented, the relative order among them strongly affected the reported time of each of them. Moreover, a Bayesian theory taking into account the memory for the events order is broadly compatible with the experimental data, in particular in terms of the effect of order on absolute time reports. Our results suggest that people do not deduce order from memorized time, instead people’s memory for absolute time of events relies critically on memorized order of the events.


Introduction
Tulving [1,2,3] proposed a distinction between semantic memory (general knowledge) and episodic memory (personal experiences that carry information about time and location). We know very little about how time is encoded in the brain, hence theoretical understanding of episodic memory is difficult. Our introspective experiences and psychological studies indicate that event time can be remembered either in the absolute form (when the event happened) or in the relative form (whether the event happened before or after other specific events) [4]. Absolute time processing is quite reliable for short intervals, such as when catching a ball or playing an instrument, but deteriorates when longer intervals are involved, to the extent that we are often unaware of when some events happened (for example, one may know that Robert Kennedy was assassinated later than his brother was, but may not know when this assassination happened). Most of psychological studies of time memory relate to event duration rather than their absolute occurrence time (see e.g. [5,6]).
In the current contribution, we consider the issue of interactions between absolute and relative time representations in episodic memory. In particular, lower-level absolute event time is a continuous feature while higher-level relative time order between events is of a discrete nature, hence one could expect that as time goes by, the latter could be more reliably stored in memory and hence take precedence in inference process. Interactions between absolute and relative attributes was indeed observed in recent studies of memory for simple visual attributes [7,8]. In particular, reports of relative orientation of stimuli to reference strongly biased subsequent reports of absolute orientation of the stimulus [7]. In [8], multiple stimuli were presented to observers who did not have to explicitly report their relative orientations, only the absolute orientation of each stimulus. Still, these reports were strongly biased by the relative orientations between the stimuli. Moreover, the orientation reports could be predicted quantitatively by retrospective Bayesian inference of absolute orientations if relative orientations were assumed to be treated by the brain as given, i.e. decoding followed reverse hierarchical scheme from complex to simple features, as opposed to direct hierarchy of encoding (see also [9]). These intriguing results raise some fundamental issues on the nature of information encoding and decoding in the brain at the time when stimuli that give rise to perception are withdrawn.
Here we study how generic the retrospective decoding in memory is and whether it can be extended to the domain of event times in episodic memory. To this end, we performed experiments where either single events or pairs of events were presented to participants at different times, after which they had to report their time of appearance. Following [8], we evaluated the interaction between absolute presentation time of each event and relative time, i.e. which event was presented earlier. We observed very strong interference between these two types of information, such that temporal order of events consistently shifted the report of absolute times towards earlier (later) times for events presented first (second). We also developed Bayesian inference scheme for absolute time reports and compared it to our experimental observations.

Experimental design and results
Experimental design is illustrated in the Fig. 1. Initially, we wanted to establish the quality of absolute time encoding of an event, when no other events were present during the same episode. Therefore, in the first experiment participants were exposed to a list of three items (words or images; see Methods for more details). Each trial was divided into 11 time slots of Figure 1: Experimental design. Upper panel: Experiment 1 scheme, three items presented. One item always presented at the beginning of the trial, one always at the end. The duration of a trial was divided into several slots of equal size. Intermediate stimulus was presented in a randomly chosen slot. After the presentation participants have to report the time of one of the three items by moving a green cursor to the corresponding position on a sliding bar. Lower panel: Experiment 2 scheme, four items presented. One item is always presented at the beginning of the trial, another one at the end of the trial, while two intermediate items are presented in random time slots. After presentation participants have to report the time of two intermediate items by moving green cursors to the corresponding positions on two sliding bars. duration 1.5 seconds each. The first and the last item were always presented in the first and last slot, respectively, to delineate the beginning and the end of the trial, while the second item was presented in a randomly chosen intermediate slot. Each item was shown for 1000 ms in the beginning of a slot. The experiment was performed using Amazon Mechanical Turk R . Participants were then requested to report the presentation time of one specific item, by moving a green circle with the mouse to the correct position on a sliding bar. At the beginning of each experiment, 5 training trials were performed where participants received a feedback with the correct timing (location of a circle) presented to them on another bar. Additional 15 trials without feedback were subsequently performed for data analysis. Results obtained in the first experiment are presented in Fig. 2 where reported time distributions for each presentation time of the intermediate item are shown. One can see that reported time distributions are rather wide, except for the very beginning, end and middle of a trial. Moreover, it is interesting to notice that the results are very similar for both words and images.
To test the effect of event order on perception of "absolute" time the second experiment with lists of four items was performed. Participants were requested to report the time of two of them, see Fig. 1.
Before analyzing Experiment 2, it is instructive to form a prediction for the accuracy of relative time order for two intermediate items based on the results of Experiment 1. If we assume that two intermediate items are encoded and reported independently, we can predict from Fig. 2 the probability that for any two presentation times the participants will make a mistake in ordering the items (see Fig. 3A,C). When the two presented items are close to each other the probability to flip the order is higher. However, experimental results do not show this tendency (see Fig. 3B,D). Overall, the accuracy of time ordering was 86% for words and 88% for images as opposed to 75% and 77% as predicted from the result of Experiment 1. These results show that reports of intermediate item presentation times in Experiment 2 are not independent from each other.
To illustrate the effect of order on absolute time representations in Experiment 2, in Fig. 4 we show the reported time distributions for the first and second intermediate items separately ( Fig. 4a and Fig. 4b, respectively). The results clearly show that participants tend to report the first intermediate word and the second one in the first and second half of the trial, respectively. This behaviour is dominant except when both items are presented in consecutive time bins either at the beginning or at the end of the trial.

Bayesian Theory
These results indicate that absolute and relative times are two interactive but distinct aspects of episodic memory. We therefore developed a Bayesian time decoding theory that elaborates the precise nature of this interaction. Define t 1 and t 2 the absolute presentation times of events within a trial, andt 1 (t 2 ) their internal representations at the report time. Define also σ = 1(0) if t 1 < t 2 (t 1 > t 2 ), respectively, to be the relative order between the events, andσ its internal representation. Due to representation errors,σ is not necessarily consistent witht 1 andt 2 .
The likelihood function for the internal variables is given by where we assume that P 1 (t|t) can be evaluated from the report time distribution measured in Experiment 1 (see Fig. 2) and P (σ|t 1 , where P σ is the probability of a mistake in the internal representation of a relative order between events. We assume that once all relevant features (t 1 ,t 2 andσ) are represented in working memory, at the report time the brain performs Bayesian inference of absolute presentation times according to where N t 1 ,t 2 ,σ is the normalization and we assumed that prior distribution of event times is uniform in agreement with the experimental protocol (see Methods). One can see from Eqs. (1) and (3) that as the estimation of presentation order becomes more precise (P σ → 0), it serves as an effective prior for the estimates of the absolute presentation times. Following [8], we assume that reported times for the first and second items (t 1r and t 2r , respectively) are generated as averages over the posterior distribution: Sincet 1 ,t 2 ,σ are distributed according to the likelihood function of Eq. (1), this equation gives rise to the distributions of reported times t 1r , t 2r for given presented times t 1 , t 2 . To generate these distributions, we randomly sampled the reported times in Experiment 1 as a proxy for P 1 (t|t), and used Eq. (2) to generate samples ofσ. From the results of Fig. 3 we fix P σ = 0.13 (average error probability for words and images). Our theoretical predictions are shown in Fig. 5, overlaid with experimental results. The predictions for the average report times agree well with corresponding data (Fig. 5A,B). However, some of experimental report distributions exhibit bimodal shape that is not well captured by the model.

Discussion
In this contribution we showed that absolute time of different events is not reliably represented in memory, while presentation order is. Moreover, the ordinal information strongly effects absolute time reports by shifting reported times according to the presentation order, even though ordinal information itself did not have to be explicitly reported by the participants in our experiments. Experimental results can be reasonably approximated by the Bayesian inference framework. These results are quite analogous to those of [8] in the visual domain. We therefore believe that they reflect a general principle in information processing according to which those aspects of information that are more reliably represented in memory take precedence to less reliably represented aspects and moreover act as Bayesian priors for inferring the latter. In particular, it appears that higher-level features such as ordinal relations between elementary components of complex stimuli that are of a discrete nature are  decoded first and then constrain the decoding of lower-level, continuous features such as absolute time of an event or an absolute orientation of a line. Our study concerned the time intervals in the range of tens of seconds, but we believe that our results also hold for much longer times. The generality of the observed effects should be investigated in future experiments.

Participants, items and Procedure
In total 420 participants, were recruited to perform memory experiments on the online platform Amazon Mechanical Turk R (https://www.mturk.com). Ethics approval was obtained by the IRB (Institutional Review Board) of the Weizmann Institute of Science. Each participant accepted an informed consent form before participation and was receiving 85 cents for approximately 10 min. For 222 participants the presented lists were composed of non-repeating words randomly selected from a pool of 751 words produced by selecting English words [10] and then maintaining only the words with a frequency per million greater than 10, from Medler [11]. For 198 participants the presented lists were composed of non-repeating images (out of 149 possible): visual stimuli consisted of animal pictures [12], houses and body parts (free-copyrights Google images). Examples of the images are shown in Fig. 6. All the images were resized in browser to have width of 600 pixels. The items were presented on the standard Amazon Mechanical Turk R web page for Human Intelligent Task (HIT). Each trial was initiated by the participant by pressing "Start Experiment" button on computer screen. List presentation followed 300 ms of white frame. During a trial, depending on the task, 3 or 4 items where shown in a total time frame of 16.5 seconds. More specifically, the trial was divided into 11 slots of 1.5 seconds each, and an item was shown in one of the slots. The first item was always presented in the first slot, the last item was presented in the last slot. One or two intermediate items (Experiments 1 and 2, respectively) were shown in randomly chosen slots with uniform probability. Each item was shown within HIT frame with black font at onset of slot for 1000 ms followed by empty frame for 500 ms. After the last item, there was a 1000 ms delay before participant performed the task. Experiment 1 -Three items. Participants were presented with three items. One item was always presented at the beginning of the trial, one always at the end and one in a random slot. At the end, participants were requested to report the time of one of the items moving a green circle with the mouse to the correct position on a sliding bar. At the beginning there were training sessions where participants received a feedback with the correct timing (5 trials), followed by 15 trials without feedback. Experiment 2 -Four items. Participants were presented with four items. One item was always presented at the beginning of the trial, one always at the end and the two others in random slots. At the end, participants were requested to report the time of two of the words moving a green circles with the mouse to the correct position on two sliding bars. At the beginning there were training sessions where participants received a feedback with the correct timing (5 trials), followed by 15 trials without feedback.