PT  - JOURNAL ARTICLE
AU  - Christopher J. Bates
AU  - George Alvarez
AU  - Samuel J. Gershman
TI  - Scaling models of visual working memory to natural images
AID  - 10.1101/2023.03.17.533050
DP  - 2023 Jan 01
TA  - bioRxiv
PG  - 2023.03.17.533050
4099  - http://biorxiv.org/content/early/2023/03/18/2023.03.17.533050.short
4100  - http://biorxiv.org/content/early/2023/03/18/2023.03.17.533050.full
AB  - Over the last few decades, psychologists have developed precise quantitative models of human recall performance in visual working memory (VWM) tasks. However, these models are tailored to a particular class of artificial stimulus displays and simple feature reports from participants (e.g., the color or orientation of a simple object). Our work has two aims. The first is to build models that explain people’s memory errors in continuous report tasks with natural images. Here, we use image generation algorithms to generate continuously varying response alternatives that differ from the stimulus image in natural and complex ways, in order to capture the richness of people’s stored representations. The second aim is to determine whether models that do a good job of explaining memory errors with natural images also explain errors in the more heavily studied domain of artificial displays with simple items. We find that: (i) features taken from state-of-the-art deep encoders explain coarse-grained and some fine-grained aspects of trial-by-trial difficulty in natural images, while several reasonable baselines do not; and (ii) deep visual encoders could reproduce set-size effects but overall offered a poorer explanation of human data in the artificial domain. Together, our results suggest that people may rely on distinct cognitive systems or brain areas in artificial versus natural task domains. Moving forward, our approach offers a scalable way to build a more generalized understanding of VWM representations by combining recent advances in both AI and cognitive modeling.Competing Interest StatementThe authors have declared no competing interest.