RT Journal Article
SR Electronic
T1 Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2021.02.02.429430
DO 10.1101/2021.02.02.429430
A1 Lynn Le
A1 Luca Ambrogioni
A1 Katja Seeliger
A1 Yağmur Güçlütürk
A1 Marcel van Gerven
A1 Umut Güçlü
YR 2021
UL http://biorxiv.org/content/early/2021/04/29/2021.02.02.429430.abstract
AB Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance imaging data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.Competing Interest StatementThe authors have declared no competing interest.