ABSTRACT
Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether (“invariance”), represented in non-interfering subspaces of population activity (“factorization”) or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters – lighting, background, camera viewpoint, and object pose – in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.
Competing Interest Statement
The authors have declared no competing interest.
Footnotes
The Introduction is expanded to clearly define factorization, relating it to existing concepts in the field such as invariance, decodability, and manifold disentanglement. A new & improved Figure 1 explains the concept of factorization through illustrations and equations. In the Results, we elaborate on how factorization is computed and explain our controls better to help the reader grasp the implications of each result. The figures have been more logically divided to communicate each key finding to the reader. As a result, there are now five total figures instead of just three. The Discussion now better relates our work to that in the field, highlighting the similarities and differences between factorization and other normative principles of visual representations that have been proposed.
