RT Journal Article
SR Electronic
T1 Self-Supervised Natural Image Reconstruction and Rich Semantic Classification from Brain Activity
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2020.09.06.284794
DO 10.1101/2020.09.06.284794
A1 Guy Gaziv
A1 Roman Beliy
A1 Niv Granot
A1 Assaf Hoogi
A1 Francesca Strappini
A1 Tal Golan
A1 Michal Irani
YR 2020
UL http://biorxiv.org/content/early/2020/09/08/2020.09.06.284794.abstract
AB Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1Competing Interest StatementThe authors have declared no competing interest.