Abstract
What enables humans to readily recognize visual objects is attributed to how the brain extracts, represents, and organizes object information from visual input. To understand this process, we used a deep residual network as a hierarchical computational model of visual categorization to predict cortical representation of natural vision. We trained and tested such a predictive encoding model with functional magnetic resonance imaging data from human subjects watching hours of natural video clips, and verified its ability to predict cortical responses to novel images, objects, and categories. We further used the so trained encoding model to synthesize cortical responses to 64,000 visual objects from 80 categories, revealing hierarchical, distributed, and overlapping cortical representations of categories. Such category representations covered both the ventral and dorsal pathways, reflected multiple levels and domains of visual features, and preserved semantic relationships between categories. In the scale of the entire visual cortex, category representations were modularly organized into three categories: biological objects, non-biological objects, and background scenes. In a smaller scale specific to each module, category representation further revealed sub-modules for finer categorization, e.g. biological objects were categorized into terrestrial animals, aquatic animals, humans, and plants. Such nested spatial and representational hierarchies were attributable to different levels of category information to varying degrees. These findings suggest that increasingly more specific category information is represented by cortical patterns in progressively finer spatial scales – an important principle for the brain to categorize visual objects in various levels of abstraction.
Significance Statement This study uses a deep neural network to model how the brain extracts and represents the information from visual objects for efficient and flexible categorization. Results show that the model can predict widespread cortical responses when humans are viewing natural video clips, and generate patterns of cortical responses to tens of thousands of objects from 80 categories. Analysis of these response patterns reveals that the brain organizes distributed, overlapping, and property-based category representations into a nested hierarchy. Increasingly more specific category information is represented by cortical patterns in progressively finer spatial scales. This nested hierarchy may be a fundamental principle for the brain to categorize visual objects in various levels of specificity.
