New Results

Dynamic dot displays reveal material motion network in the human brain

Alexandra C. Schmid, Huseyin Boyaci, Katja Doerschner
doi: https://doi.org/10.1101/2020.03.09.983593

ABSTRACT

There is growing research interest in the neural mechanisms underlying the recognition of material categories and properties. This research field, however, is relatively more recent and limited compared to investigations of the neural mechanisms underlying object and scene category recognition. Motion is particularly important for the perception of non-rigid materials, but the neural basis of non-rigid material motion remains unexplored. Using fMRI, we investigated which brain regions respond preferentially to material motion versus other types of motion. We introduce a new database of stimuli - dynamic dot materials-that are animations of moving dots that induce vivid percepts of various materials in motion, e.g. flapping cloth, liquid waves, wobbling jelly. Control stimuli were scrambled versions of these same animations and rigid three-dimensional rotating dots. Results showed that isolating material motion properties with dynamic dots (in contrast with other kinds of motion) activates a network of cortical regions in both ventral and dorsal visual pathways, including areas normally associated with the processing of surface properties and shape, and extending to somatosensory and premotor cortices. We suggest that such a widespread preference for material motion is due to strong associations between stimulus properties. For example viewing dots moving in a specific pattern not only elicits percepts of material motion; one perceives a flexible, non-rigid shape, identifies the object as a cloth flapping in the wind, infers the object’s weight under gravity, and anticipates how it would feel to reach out and touch the material. These results are a first important step in mapping out the cortical architecture and dynamics in material-related motion processing.

1. Introduction

Recognizing and estimating the material qualities of objects is an essential part of our visual experience. Perceiving material qualities quickly and correctly is critical for guiding decisions or actions, whether we are deciding what fruit to eat, if a blanket is soft enough, or how we should grip a porcelain cup. Despite the importance of recognizing and estimating the properties of materials, it is still not well understood how the brain accomplishes these tasks. Most neuroscientific studies about material perception have focused on the cortical areas involved in the visual processing of material properties, with nearly all of those studies using static images as stimuli (see Komatsu & Goda, 2018 for a review). Although motion compellingly conveys the properties of non-rigid materials such as stiffness and elasticity (e.g. Schmid & Doerschner 2018; Schmidt et al. 2017), the neural basis of non-rigid material motion remains unexplored. Here we investigate whether specialized dynamic dot animations portraying non-rigid material motion elicit meaningful differences in brain responses when compared to other kinds of motion using functional magnetic resonance imaging (fMRI).

The use of static images when investigating the visual processing of materials is somewhat justified because, although material perception is a multisensory (Baumgartner et al. 2013), dynamic experience (Schmid & Doerschner, 2019), many material qualities can be conveyed through images alone (Paulun et al. 2017; Schmid & Doerschner, 2018; Schmidt et al. 2017; van Assen & Fleming 2016). Material properties that can be conveyed directly through visual information are so called optical properties. These properties, such as a surface’s micro- and meso-structure, or its reflective-, transmissive- and refractive properties, give a material its characteristic visual appearance (e.g. glossy, plastic-y, metallic, etc.). Literature investigating neural mechanisms of material perception has so far heavily focused on how the brain discriminates between different optical appearances (Komatsu & Goda, 2018). Non-optical properties, however, can also be conveyed through images: we can infer the ‘feel’ of soft silk or fur just by looking at an image of it (Baumgartner et al. 2013, Xiao et al. 2016) owing to previously formed associations between the different senses when interacting with materials; overtime, specific visual (or auditory, proprioceptive, or olfactory) information becomes associated with specific tactile information and vice versa. It is possible that via this indirect route (Schmidt et al. 2017) mechanical and tactile material qualities like softness, viscosity, and roughness can be conveyed through optical properties of surfaces and the 3D structure of visual objects (e.g. Baumgartner et al. 2013; Fleming, 2014, 2017; Fleming et al. 2013; Giesel & Zaidi, 2013; Ho et al. 2006; van Assen & Fleming 2016; Xiao et al. 2016).

But not all material properties can be faithfully conveyed through static images, and most mechanical material properties become much more vividly apparent with motion: watching a rubber band stretch, a jelly jitter, and hair bending elicits strong impressions of elasticity, wobbliness, and softness, respectively. In fact, image motion has been shown to provide information about material qualities over and above the information available from 2D images (Doerschner et al. 2011, Schmid & Doerschner 2018; Schmidt et al. 2017). Furthermore, recognizing materials in natural environments likely entails integrating spatiotemporally segregated information into coherent percepts, similar to when detecting animate objects (think of detecting a tiger through long, swaying grass). Given that many brain regions are sensitive to certain kinds of motion structure (for reviews see Erlikhman et al., 2018; Kourtzi et al., 2008; Nishida et al., 2018), it is surprising that only very few studies have investigated the neural mechanisms involved in material perception using dynamic stimuli (but see Okazawa et al. 2012; Kam et al. 2015; see Sun et al. 2016a for binocular stimuli). These studies investigated the neural mechanisms of gloss perception using rigid objects. To date, no studies have examined the cortical processing of nonrigid materials. One possible reason for this is that it is difficult to find adequate control conditions for such complex, dynamic stimuli. Recent improvements in computing power and an increased ability to simulate such complex materials with computer graphics now puts us in a position to tackle this problem.

In this study we developed a new class of stimuli for investigating the neural correlates of material perception, which utilize the fact that mechanical and tactile qualities of materials can be convincingly conveyed through image motion alone (in Schmid & Doerschner, 2018; Bi et al., 2019). Previously these have been called point light stimuli by Schmid & Doerschner (2018), analogous to ‘point light walkers’ that have been used extensively in biological motion research (Johanson, 1973). Similar stimuli have also been called ‘dynamic dot stimuli’ (Bi et al., 2019). Here, we name our new movie database ‘dynamic dot materials’, where specific nonrigid material types are solely depicted through the motion of black dots on a gray background. Investigations into biological motion (e.g. Servos et al., 2012) and structure from motion (SfM, e.g. Orban et al. 1999, Peuskens et al., 2004) suggest that the brain can be very sensitive to certain structured motion. Yet so far, the neural mechanisms of motion perception has largely been investigated separately for material and object recognition (Grill-Spector & Weiner, 2014; Komatsu & Goda, 2018). Here we investigated whether cortical areas exist that show a preference for non-rigid material motion over other types of motion.

Dynamic dot materials not only depict mechanical material properties convincingly, but they also have several advantages over the static images used in previous work. In particular, they:

  • isolate dynamic properties from optical cues;

  • capture non-optical aspects of material qualities, in particular mechanical properties;

  • provide more stimulus control compared to ‘full cue’ videos - they can be motion-scrambled and the behavior of trajectories and speed of individual dots can be manipulated;

  • allow us to investigate whether areas previously associated with materials (e.g. CoS, e.g. Cant & Goodale 2007; Cant & Xu, 2012, 2015, 2016; Gallivan et al. 2014; Eck et al., 2016; Kitada et al. 2014) are indeed specialized for processing visual (optical) properties or whether they represent material properties more generally;

  • allow us to investigate whether areas associated with coherent motion preference (e.g. hMT+/V5 and posterior parietal cortex, e.g. Orban et al. 1999; Peuskens, 2004); show a preference for specific types of coherent motion (non-rigid vs. rigid)

Using fMRI, we investigated which brain regions respond preferentially to these novel dynamic dot materials versus other types of motions and motion scrambled control stimuli. Anticipating our results, we find widespread preferential activation for non-rigid material motion. This suggests that dynamic dot materials are very suitable for mapping the cortical network involved in the perception of material qualities.

2. Materials & methods

2.1. Participants

10 volunteers participated in the experiment (age range: 21-42, 2 males, mean age: 27.3, 1 left handed). All participants had normal or corrected to normal vision and had no known neurological disorders. Participants gave their written consent prior to the MR imaging session. Protocols and procedures were approved by the local ethics committee of Justus Liebig University Giessen and in accordance with the Code of Ethics of the World Medical Association (Deklaration of Helsinki).

2.2. Stimuli

2.2.1. Dynamic dot materials

Stimuli were non-rigid materials, generated with Blender (version 2.7) and Matlab (release 2012a; MathWorks, Natick, MA) and presented using Matlab and the Psychophysics Toolbox (Brainard, 1997). Each dynamic dot material was a 2s animation (48 frames) that depicted the deformation of a specific material under force. Object deformations were simulated using the Particles System physics engines in Blender, with either the fluid dynamics or Molecular addon (for technical specifications we refer the reader to Schmid & Doerschner, 2018). For variety, we simulated non-rigid materials with different mechanical properties under various forces (details shown in Table 1), including cloth flapping in the wind, liquids rippling and waving, breaking materials of high, medium, and low elasticity, as well as non-breaking elastic materials. Upon creation of these different material types we exported the 3D coordinates of each particle at each frame. Using Matlab, we calculated the 2D projection for each particle from a specific viewing angle. For each material only 200 random particles were selected for display, as this medium density yielded best perceptual impressions in previous work (Schmid & Doerschner, 2018), with the remaining particles set to invisible. The particles were sampled from throughout the volume of the substance with the following exception: for liquids the particles were sampled from the surface because sampling from the volume in these cases did not convey the material qualities as convincingly. In total we rendered ten animations (Table 1).

View this table:
Table 1: Simulation and rendering details for each material motion animation.

Animations were 2s clips of materials being deformed in various ways. The first column shows the name of the movie provided in the Supplementary Material. The second column shows whether each material was simulated as linked particles (breaking or unbreaking) or fluid particles (see main text). The third column shows details of how the simulation was set up in Blender. The fourth column shows what part of the simulation was rendered in the final animation.

2.2.2. Scrambled control stimuli

We wanted to find cortical regions that are sensitive to material motion over and above other kinds of motion. That is, we wanted to exclude regions that are sensitive to motion generally but do not show a preference for material motion. Therefore we created “scrambled” control stimuli that were matched in motion energy to the dynamic dot materials (measured as average velocity magnitude, see Supplementary Figure 1). We created a scrambled version of each material stimulus by shuffling the position of each particle on the first frame. All dots in the scrambled motion stimuli had the same velocity magnitude as the material motion; however, the direction of trajectory was rotated by a random (uniform distribution) amount between 0 and 2Π every 2nd frame. Without this rotation the scrambled stimuli would still look like non-rigid materials. The trajectory of a given dot was forced to remain inside the spatial range of the dynamic dot materials (in the 1st frame) by forcing it to change its direction if it is outside the boundary. This was accomplished by rotating the dot trajectory to a different direction. As a consequence, acceleration between material and control stimuli were not matched (See Supplementary Figure 2). Example scrambled control stimuli are shown in Figure 1C.

Figure 1.
Figure 1. Example stimuli and block design.

the functional scans participants were presented with alternating blocks of dynamic dot stimuli, motion scrambled control stimuli and structure from motion (SfM) control stimuli. On the right side of panel A we show selected frames of 4 of the 10 possible dynamic dot material animations that were shown in random order in a block. On the left side of the same panel the timing of presentation during a dynamic dot material block is shown. Panel B depicts an example run, and panel C shows selected frames of 1 of the 10 possible random motion control animations (left) and the SfM control stimuli (right). The timing during a block of these control conditions was identical to that of dynamic dot material blocks.

2.2.3. Structure from Motion (SfM) control stimuli

We also wanted to make sure any preference for material motion within a brain region was not just a preference for global coherent 3D motion, or “object-ness". Therefore, we created 3D structure-from-motion control stimuli. In order to generate these control stimuli, we selected one frame of each material motion movie and then rotated the camera back and forth around the center of the scene. This gave the object an appearance of rotating in depth around the horizontal (or vertical axis). SfM control stimuli are shown in Figure 1C.

For each stimulus, coordinates, movies, and individual movie frames can be found in the database provided with the Supplementary Material (10.5281/zenodo.3820669). [Note that this link will not work until publication. Reviewers can find database here: https://www.dropbox.com/sh/nakqpn022lpaptn/AACQkbsgFuVvUIZIRXnoWXryaPdN0).

2.3. Stimulus display

Visual stimuli were presented on an MR-safe LCD screen placed near the rear end of the scanner bore (Cambridge Research Systems Ltd, Rochester, UK; resolution: 1920 by 1080; refresh rate 120 Hz). Participants viewed the screen through an angled mirror attached to the head-coil while lying supine inside the scanner bore. Total eye-to-screen optical distance was 140 cm, and the screen subtended a visual angle of 28 degrees horizontally at this distance. Stimuli were presented at the center of the screen and approximately subtended 15.75 by 15.75 degrees visual angle.

2.4. Fixation task

To ensure fixation during the scanning session, aid maintaining vigilance and wakefulness, and limit attentional effects, we asked participants to perform a demanding fixation task throughout the functional runs. In this task, participants were required to report a brief reduction in the size of the fixation cross via a button press. The cross shrank by a small amount every 3 seconds with a random (+/− up to 1.5 seconds) time jitter to make it unpredictable when the shrinking would occur.

2.5. MR Image Acquisition

Magnetic resonance images were collected on a 3 Tesla MRI scanner (Magnetom Prisma, Siemens AG, Erlangen, Germany) equipped with a 64-channel head coil in BION imaging center of JLU Giessen. MR sessions contained a structural run and 4-8 functional runs. Structural images were acquired using a T1-weighted 3-D anatomical sequence (sagittal MP-RAGE, Spatial resolution: 1 mm3 isotropic; number of slices: 176). Functional images were acquired while participants viewed the visual stimuli and were acquired with a T2*-weighted gradient-recalled echo-planar imaging (EPI) sequence (TR: 2000 ms; TE: 30 ms; spatial resolution: 3×3×3 mm3; number of slices: 36; slice orientation: parallel to calcarine sulcus). Each participant took part in one scanning session that lasted about an hour and a half. During functional runs participant responses were collected using an MR-safe button box.

2.6. Experimental Design

During the functional runs, different types of motion stimuli (material, scrambled control, SfM control) were presented in alternating blocks. The order of blocks was randomized and counter-balanced. Each block lasted 22 seconds and contained ten short clips (2 s) of animation separated by 200 ms interstimulus interval (ISI). Figure 1B depicts the experimental protocol. There were 3 repeats of each stimulus type (material, scrambled control, SfM control) per run, 4-8 runs in a session, thus 12-24 repeats of each stimulus per participant. One run lasted 286 seconds.

2.7. MR Data Analysis

All MR image preprocessing and further analyses were performed using BrainVoyager QX, except an initial inhomogeneity correction step on T1-weighted images, which was conducted using Freesurfer4 software (http://surfer.nmr.mgh.harvard.edu/). After the initial inhomogeneity correction with Freesurfer4, anatomical images were imported into BrainVoyager for further preprocessing. Preprocessing for the anatomical images included the following steps: another inhomogeneity correction using BrainVoyager (for 8 out 10 participants this led to a better white-gray matter segmentation), aligning the images in AC-PC plane and converting to Talairach space, white-gray matter segmentation. After these steps a 3D cortical mesh was created for each subject and the resulting individual meshes were morphed and aligned using cortical surface information (sulci and gyri). Finally an average mesh was created and inflated. Preprocessing steps on functional images included slice acquisition time correction, motion correction, linear trend removal and high pass filter (temporal). The resulting functional images were coregistered with the anatomical images per participant. Functional data were spatially transformed and projected on the inflated 3D average mesh for further analyses. These analyses included fixed-effects surface based group analyses using the general linear model (GLM). Specifically we performed several separate GLM analyses (material vs. scrambled, material vs. 3D; 3D vs. scrambled) and a conjunction analysis of (material>scrambled) AND (material>3D). Active voxels were identified at qFDR > 0.05 level (FDR corrected for multiple comparisons).

3. Results

Figure 2 shows the results of the whole-brain GLM analyses. The activation maps on the left and on the right show that, overall, there are many cortical areas that have a larger BOLD response to dynamic dot materials (yellow) than to scrambled motion stimuli. Moreover, cortical areas with a material “preference” appear to constitute a superset of those that respond to 3D motion (Figure 2, left panel, showing areas - in orange - that respond more to 3D rigid motion than to scrambled stimuli). The overlap in activation for these two contrasts (material vs. scrambled and 3D vs. scrambled) is perhaps not surprising because 3D rigid motion could simply be a special type of material motion and thus activity to this type of stimuli should be contained within the general material motion network. The fact that rigid motion is a subclass of all possible material motions might also explain why cortical responses to dynamic dot materials are almost always stronger than those to 3D motion (Figure 2, right panel, orange color maps): seeing just one type of material motion is likely to cause a relatively weaker cortical response than the rich set of motions that occur in the dynamic dot condition. We also found these patterns of activation at the individual participant level (Supplementary Figure 3). The average accuracy in the fixation task was 83% across observers, suggesting that participants followed our instructions and fixated at the center of the screen during presentation of the stimuli.

Figure 2.
Figure 2. GLM group analyses.

The left panel shows the results of the GLM contrasting average BOLD responses to dynamic dot materials with those to scrambled motion stimuli (yellow-blue), overlaid together with results of a GLM analysis that contrasts average BOLD responses to 3D rigid motion with those to scrambled stimuli (orange-black). The resulting activity maps suggest that cortical areas that respond strongly to material motion are a superset of those that respond to 3D rigid motion. In the right panel we plot the results of the same dynamic dot materials vs scrambled motion contrast (yellow-blue) together with a GLM contrast of dynamic dot materials versus 3D rigid motion (SfM; orange-black). Here, we see that responses to dynamic dot materials were almost always stronger than those to 3D rigid motion (SfM) stimuli. See text for further details. Overall, we see that lower visual areas tended to respond stronger to the scrambled motion stimuli. This is consistent with the literature (e.g. Murray et al. 2002).

A subsequent conjunction analysis confirmed that there is a widespread network of brain areas whose BOLD response is stronger for dynamic dot stimuli compared to other types of motion (scrambled and 3D rigid motion), including dorsal and ventral visual regions in addition to multisensory, somatosensory, and premotor areas (Figure 3). Only early visual areas respond more strongly to the two control motions (blue squares labelled 1). This is in line with our motion energy measurements of the stimuli (Supplementary Figures 1 and 2) and suggests that higher activation for material versus control motion in other regions is not due to low-level differences in motion energy between the conditions. Broadly, the stronger response for material motion versus scrambled and 3D rigid motion (SfM) encompasses ventral visual regions (grey squares labelled 2); occipito-temporal and -parietal regions (yellow squares labelled), dorsal visual and multisensory regions (orange squares labelled 4), somatosensory and multisensory regions (though not primary somatosensory cortex; red squares labelled 5), superior temporal regions (green squares labelled 6) and pre-motor regions (purple squares labelled 7). We will discuss these results next.

Figure 3.
Figure 3. Conjunction analysis.

Cortical areas that respond stronger (hot colors) and weaker (cool colors) to material motion compared to both 3D motion (SfM) and scrambled motion. Only early visual areas respond more weakly to material motion (1. Blue squares). A large network of areas are more strongly active under the material motion condition. Broadly the network encompasses ventral visual regions (2. Grey squares); occipito-temporal and -parietal regions (3. Yellow squares), dorsal visual and multisensory regions (4. Orange squares), somatosensory and multisensory regions (5. Red squares); superior temporal regions (6. Green squares) and premotor regions (7. Purple squares).

4. DISCUSSION

Dynamic dot materials are a novel class of stimuli that convey the non-optical properties of material qualities purely on the basis of 2D image motion patterns. Here, we introduced a database of various types of non-rigid materials, though other materials (e.g. rigid, breakable substances) can also be rendered convincingly by the means of this technique (e.g. see Schmid & Doerschner, 2018). In their creation, these stimuli are conceptually closely related to “point light walkers” (Johannson, 1973), where the motion of small light sources affixed to different parts of limbs of biological species can elicit a very vivid impression of animacy. Similarly, we “attached” small dots to an otherwise invisible substance and recorded the motion of these dots while the material reacted to a force. How individual materials change their shape in response to a force strongly depends on their mechanical properties, and it is this idiosyncratic change of shape information that appears to convey mechanical qualities of a material. It would be very interesting to pin-point exactly the motion characteristics that elicit a particular material quality (Schmid & Doerschner, 2018; also see Bi et al. 2019), just as it has been done in the field of biological motion where researchers have tried to understand what it is that makes point light walkers look “biological” (Troje, 2013). In fact, all biological motion is also non-rigid motion, so our stimuli could be suitable for teasing apart the contributions of these two factors to neural activity observed while watching biological motion stimuli. Our stimuli could help to discover the perceptual boundary between non-rigid animate and inanimate objects, and to investigate corresponding neural maps and mechanisms (Long, Yu, Konkle, 2018; Grill-Spector & Weiner, 2014). Dynamic dot materials could also be used to investigate whether areas previously associated with materials (e.g. CoS, e.g. Cant & Goodale 2007; Cant & Xu, 2012, 2015, 2016; Gallivan et al. 2014; Eck et al., 2016; Kitada et al. 2014) are indeed specialized for processing visual (optical) properties or whether they represent material properties more generally. Our results so far suggest the latter, but further investigation, for example using a multivariate design and analysis approach, would allow one to better test this (Schmid & Doerschner, 2019). The fact that the location, direction and speed of individual dots in our stimuli can be manipulated, renders investigations of research questions like these more feasible.

4.1 Relation to previous work

Using these novel stimuli in an fMRI experiment we found robust and widespread increased activation across the human brain in response to dynamic dot materials when compared to activation in response to other types of motion stimuli (Figure 3). Regions preferring dynamic dot materials included several areas in occipito-temporal and, -parietal cortices, secondary somatosensory cortex, and premotor regions. Ventral visual areas (labelled 2 in Figure 3) have previously been implicated in the processing of surfaces and textures in static scenes (Cant & Goodale, 2007; Cavina-Pratesi et al. (2010a, 2010b); Hiramatsu et al., 2011). Occipito-temporal and -parietal areas include motion-, object-, face-, and place-selective areas (Grill-Spector & Mallach, 2004). Posterior parietal areas (labelled 4) are sensitive to aspects of 3D shape, structure-from-motion, optic flow, multisensory information, and visuomotor control (Culham, et al., 2006; Erlikhman et al., 2018; Uesaki & Ashida, 2015). Recently it has been found that activity patterns in secondary somatosensory cortex (labelled 5) could reliably discriminate visual properties, such as surface gloss and roughness (Sun et al., 2016b). Greater responses to biological versus scrambled point-light stimuli have been found in superior temporal areas (labelled 6) and premotor areas (labelled 7) (Saygin et al., 2004). Note that the higher responses for material versus the two control motions does not include primary somatosensory nor primary motor cortices, but does encompass secondary somatosensory, multisensory, and premotor areas, in addition to nearly all extrastriate regions that are responsive to visual stimuli.

Given the properties of our stimuli, a widespread cortical preference is perhaps expected: our stimuli are objects (LOC, e.g. Grill-Spector & Malach, 2004), they are non-rigidly moving structures (e.g. hMT+, MST, PPC, e.g. see review by Erlikhman, et al., 2018; STS, e.g. Saygin et al., 2004), they elicit a distinct tactile experience (e.g. Schmid & Doerschner, 2018; Bi et al., 2019), and such tactile experiences of material qualities are often associated with certain optical material qualities (CoS, e.g. Arnott et al., 2008; Podrebarac et al., 2014; Sun et al., 2016b; and see Komatsu & Goda, 2018 for a review). The widespread activation in response to quite sparse stimuli also suggests that material perception must be an inherently distributed process (see Schmid & Doerschner 2019). Consistent with this idea, there is a growing body of literature suggesting that object category representations are grounded in distributed networks (e.g. Kravitz et al., 2011; 2013; Martin, 2016).

Discovering a network of preferentially more active areas during dynamic dot material viewing does not mean that all of these areas must be involved in the recognition and differentiation of materials: to find such finer-tuned responses would require further studies, and our stimuli provide a convenient way to investigate this., as discussed next.

4.2. Advantages of studying cortical responses to visually presented materials

Materials are inherently multidimensional in that they have multiple stimulus properties in multiple perceptual dimensions, and they are inherently multimodal in that their properties can be inferred through multiple modalities. For example, a velvet cloth looks soft (optical properties), moves (visual mechanical motion) and folds (visual 3D shape) in a way that suggests that it is soft, but it also feels soft to the touch (tactile). This multidimensionality and multimodality makes materials the ideal candidate to develop experimental designs that can help to understand computational architecture of the cortical representations involved in recognition. As an example, if a cortical region represents material/object category A based on visual property X but not the same category based on the visual property Y then this suggests that this region encodes the visual property X but not the category. Conversely, if this cortical region has a shared representational structure, i.e. category A is encoded through both visual properties X and Y, then it likely encodes the concept. Thus, investigating neural responses to stimuli, like the dynamic dot materials proposed here, we may be able to tease apart the direct encoding of visual properties from the indirect activation of associated properties (Schmid & Doerschner, 2019). Note that we are referring to a generic kind of association between properties; we did not find a preference for material motion in cortical regions involved in memory or contextual associations, such as the angular gyrus, medial parietal cortex, or anterior parahippocampal cortex (Kravitz et al., 2013; Bar et al., 2008).

5. Conclusion

Dynamic dot materials are a novel class of stimuli that convey non-optical properties of material qualities purely on the basis of 2D image motion patterns. Our results act as a proof of principle that such stimuli can be used for mapping the cortical network involved in the perception of material qualities. From a broader perspective, owing to their inherently multidimensional and multimodal nature, materials are a unique type of stimulus that can help neuroimaging research to advance our understanding of the computational architecture of the cortical representations involved in recognition.

6. Acknowledgements

This work was supported by a Sofja Kovalevskaja Award endowed by the German Federal Ministry of Education; a Marie Sklodowska-Curie Action - Innovative Training Network (MSCA-ITN/ETN) Grant, DyViTo: Dynamics in Vision and Touch - the look and feel of stuff. We thank Chris Baker for helpful comments on earlier versions of this manuscript.

7. References

  1. ↵
    Arnott SR, Cant JS, Dutton GN, Goodale MA (2008) Crinkling and crumpling: an auditory fMRI study of material properties. NeuroImage 43:368–378.
    OpenUrlCrossRefPubMed
  2. ↵
    Bar M, Aminoff E, Schacter, D (2008) Scenes unseen: The parahippocampal cortex intrinsically subserves contextual associations, not scenes of places per se. The Journal of Neuroscience 28(34): 8539–8544. DOI:10.1523/JNEUROSCI.0987-08.2008
    OpenUrlAbstract/FREE Full Text
  3. ↵
    Baumgartner E, Wiebel CB, Gegenfurtner KR (2013) Visual and Haptic Representations of Material Properties. Multisensory Research 26: 429–55. https://doi.org/10.1163/22134808-00002429.
    OpenUrlCrossRefPubMed
  4. Bi, W, Jin, P, Nienborg, H, Xiao, B (2020) Manipulating patterns of dynamic deformation elicits the impression of cloth with varying stiffness. Journal of Vision, 19(5):18:1–18. https://doi.org/10.1167/19.5.18
    OpenUrl
  5. ↵
    Cant JS, and Goodale MA (2007) Attention to Form or Surface Properties Modulates Different Regions of Human Occipitotemporal Cortex. Cerebral Cortex 17, no. 3: 71331. https://doi.org/10.1093/cercor/bhk022.
    OpenUrl
  6. ↵
    Cant JS, Xu Y (2012) Object ensemble processing in human anterior-medial ventral visual cortex. Journal of Neuroscience 32:7685–7700.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    Cant JS, Xu Y (2015) The impact of density and ratio on object-ensemble representation in human anterior-medial ventral visual cortex. Cerebral Cortex 25:4226–4239.
    OpenUrlCrossRefPubMed
  8. ↵
    Cant JS, Xu Y (2016) The contribution of object shape and surface properties to object ensemble representation in anterior-medial ventral visual cortex. Journal of Cognitive Neuroscience 29:398–412.
    OpenUrl
  9. ↵
    Cavina-Pratesi, C Kentridge, RW, Heywood, CA, Milner, AD (2010a) Separate processing of texture and form in the ventral stream: evidence from fMRI and visual agnosia. Cerebral Cortex, 20: 433–446. doi:10.1093/cercor/bhplll
    OpenUrlCrossRefPubMedWeb of Science
  10. ↵
    Cavina-Pratesi, C, Kentridge, RW, Heywood, CA, Milner, AD (2010b) Separate channels for processing form, texture, and colour: evidence from fMRI adaptation and visual object agnosia. Cerebral Cortex, 20: 2319–2332. Doi:10.1093/cercor/bhp298
    OpenUrlCrossRefPubMedWeb of Science
  11. ↵
    Culham, JC, Cavina-Pratesi, C, Singhal, A (2006) The role of parietal cortex in visuomotor control: What have we learned from neuroimaging? Neuropsychologia 44, 2688–2684. doi:10.1016/j.neuropsychologia.2005.11.003
    OpenUrlCrossRef
  12. ↵
    Doerschner K, Fleming RW, Yilmaz O, Schrater PR, Hartung B, Kersten DJ (2011) Visual Motion and the Perception of Surface Material.” Current Biology 21, no. 23: 2010–16. https://doi.org/10.1016/j.cub.2011.10.036
    OpenUrlCrossRefPubMed
  13. ↵
    Eck J, Kaas AL, Mulders JL, Hausfeld L, Kourtzi Z, and Goebel R (2016) The Effect of Task Instruction on Haptic Texture Processing: The Neural Underpinning of Roughness and Spatial Density Perception. Cerebral Cortex 26, no. 1: 384–401. https://doi.org/10.1093/cercor/bhu294
    OpenUrlCrossRefPubMed
  14. Erlikhman G, Gurariy G, Mruczek REB, Caplovitz GP (2016) The Neural Representation of Objects Formed through the Spatiotemporal Integration of Visual Transients. Neuroimage 142 (2016): 67–78. https://doi.org/10.1016/j.neuroimage.2016.03.044
    OpenUrl
  15. ↵
    Fleming RW, Wiebel CB, Gegenfurtner KR (2013) Perceptual Qualities and Material Classes. Journal of Vision 13, no. 8: 1–20. https://doi.org/10.1167/13.8.9
    OpenUrlAbstract/FREE Full Text
  16. ↵
    Fleming RW (2014) Visual Perception of Materials and Their Properties. Vision Research 94: 62–75. https://doi.org/10.1016/j.visres.2013.11.004
    OpenUrlCrossRefPubMed
  17. ↵
    Fleming RW (2017) Material Perception. Annual Review of Vision Science, 3: 365–388. https://doi.org/10.1146/annurev-vision-102016-061429
    OpenUrl
  18. ↵
    Gallivan JP, Cant JS, Goodale MA, Flanagan JR (2014) Representation of Object Weight in Human Ventral Visual Cortex. Current Biology 24, no. 16: 1866–73. https://doi.org/10.1016/j.cub.2014.06.046.
    OpenUrlCrossRefPubMed
  19. ↵
    Giesel M, Zaidi Q (2013) Frequency-Based Heuristics for Material Perception.” Journal of Vision 13, no. 14: 7–7. https://doi.Org/10.1167/13.14.7.
    OpenUrlAbstract/FREE Full Text
  20. ↵
    Grill-Spector K, Weiner KS (2014) The Functional Architecture of the Ventral Temporal Cortex and Its Role in Categorization. Nature Reviews Neuroscience 15, no. 8: 536–48. https://doi.org/10.1038/nrn3747.
    OpenUrlCrossRefPubMed
  21. ↵
    Grill-Spector K, Malach R (2004) The human visual cortex. Annual Review of Neuroscience 27: 649–77. https://doi.org/10.1146/annurev.neuro.27.070203.144220.
    OpenUrlCrossRefPubMedWeb of Science
  22. ↵
    Ho YX, Landy MS, Maloney LT (2006) How Direction of Illumination Affects Visually Perceived Surface Roughness. Journal of Vision 6, no. 5: 8. https://doi.org/10.1167/6.5.8.
    OpenUrlAbstract/FREE Full Text
  23. ↵
    Hiramatsu, C, Goda, N, Komatsu, H (2011) Transformation from image-based to perceptual representation of materials along the human ventral visual pathway. Neuroimage, 57: 482–494. Doi:10.1016/j.neuroimage.2011.04.056
    OpenUrlCrossRefPubMedWeb of Science
  24. Johansson G (1973) Visual perception of biological motion and a model for its analysis. Perception & Psychophysics 14, 201–211. https://doi.org/10.3758/BF03212378
    OpenUrlCrossRefWeb of Science
  25. ↵
    Kam TE, Mannion DJ, Lee SW, Doerschner K, Kersten DJ (2015) Human Visual Cortical Responses to Specular and Matte Motion Flows. Frontiers in Human Neuroscience 9: 579. https://doi.org/10.3389/fnhum.2015.00579.
    OpenUrl
  26. ↵
    Kitada R, Sasaki AT, Okamoto Y, Kochiyama T, Sadato N (2014) Role of the precuneus in the detection of incongruency between tactile and visual texture information: a functional MRI study. Neuropsychologia 64:252–262.
    OpenUrl
  27. ↵
    Komatsu H, Goda N (2018) Neural Mechanisms of Material Perception: Quest on Shitsukan. Neuroscience. https://doi.org/10.1016/J.NEUROSCIENCE.2018.09.001
  28. ↵
    Kourtzi, Z, Krekelberg, B, van Wezel, JA (2008) Linking form and motion in the primate brain. Trends in Cognitive Sciences, 12(6): 230–236.
    OpenUrlCrossRefPubMedWeb of Science
  29. ↵
    Kravitz, DJ, Saleem, KS, Baker, CI, Mishkin, M (2011) A new neural framework for visuospatuial processing. Nature Reviews Neuroscience, 12: 217–230. doi:10.1038/nrn3008
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    Kravitz, DJ, Saleem, KS, Baker, CI, Ungerleider, LG, Mishkin, M (2013). Trends in Cognitive Sciences, 17(1): 26–49. http://dx.doi.org/10.1016/j.tics.2012.10.011
    OpenUrlCrossRefPubMedWeb of Science
  31. Long B, Yu CP, Konkle T (2018) Mid-Level Visual Features Underlie the High-Level Categorical Organization of the Ventral Stream. Proceedings of the National Academy of Sciences 115, no. 38: E9015–24. https://doi.org/10.1073/pnas.1719616115.
    OpenUrlAbstract/FREE Full Text
  32. ↵
    Martin, A. (2016) GRAPES—Grounding Representations in Action, Perception, and Emotion Systems: How Object Properties and Categories Are Represented in the Human Brain.” Psychonomic Bulletin and Review 23, no. 4: 979–90. https://doi.org/10.3758/s13423-015-0842-3.
    OpenUrl
  33. ↵
    Murray SO, Kersten D, Olshausen BA, Schrater P, Woods DL (2002) Shape perception reduces activity in human primary visual cortex. PNAS, 99, 15164–15169.
    OpenUrlAbstract/FREE Full Text
  34. ↵
    Nishida, S, Kawabe, T, Sawayama, M, Fukiage, T (2018). Motion perception: from detection to interpretation. Annual Review of Vision Science, 4(20): 1–23. https://doi.org/10.1146/annurev-vision-091517-
    OpenUrl
  35. ↵
    Okazawa G, Goda N, Komatsu H (2012) Selective Responses to Specular Surfaces in the Macaque Visual Cortex Revealed by FMRI.” Neuroimage 63, no. 3 (2012): 1321–33. https://doi.org/10.1016/j.neuroimage.2012.07.052.
    OpenUrlPubMed
  36. ↵
    Orban GA, Sunaert S, Todd JT, Van Hecke P, Marchal G (1999) Human cortical regions involved in extracting depth from motion. Neuron. Dec;24(4):929–940. DOI: 10.1016/s0896-6273(00)81040-5
    OpenUrlCrossRefPubMedWeb of Science
  37. ↵
    Paulun VC, Schmidt F, van Assen JJR, Fleming RW (2017) Shape, Motion, and Optical Cues to Stiffness of Elastic Objects. Journal of Vision 17, no. 1: 20. https://doi.org/10.1167/17.1.20
    OpenUrlCrossRef
  38. ↵
    Peuskens H, Claeys KG, Todd JT, Norman JF, Van HP, Orban GA (2004) Attention to 3-D Shape, 3-D Motion, and Texture in 3-D Structure from Motion Displays. J.Cogn Neurosci. 16, no. 4: 665–82. https://doi.org/10.1162/089892904323057371
    OpenUrlCrossRefPubMedWeb of Science
  39. ↵
    Podrebarac SK, Goodale MA, Snow JC (2014) Are visual texture-selective areas recruited during haptic texture discrimination? Neuroimage 94:129–137.
    OpenUrl
  40. ↵
    Saygin, AP, Wilson, SM, Hagler, DJJ, Bates, E, Sereno, MI (2004). Point-light biological motion perception activates human premotor cortex. The Journal of Neuroscience, 24(27): 6181–6188. DOI:10.1523/JNEUROSCI.0504-04.2004
    OpenUrlAbstract/FREE Full Text
  41. ↵
    Schmid AC, Doerschner K (2018) Shatter and Splatter: The Contribution of Mechanical and Optical Properties to the Perception of Soft and Hard Breaking Materials. Journal of Vision 18, no. 1:14. https://doi.org/10.1167/18.l.14
    OpenUrlCrossRefPubMed
  42. ↵
    Schmid AC, Doerschner K (2019) Representing stuff in the human brain. Current Opinion in Behavioral Sciences, 30, 178–185. https://doi.org/10.1016/jobeha.2019.10.007
    OpenUrl
  43. ↵
    Schmidt F, Paulun VC, van Assen JJR, Fleming RW (2017) Inferring the stiffness of unfamiliar objects from optical, shape, and motion cues. Journal of Vision, 17(3):18. doi: 10.1167/17.3.18
    OpenUrlCrossRef
  44. Servos P, Osu R, Santi A, Kawato M (2002) The Neural Substrates of Biological Motion Perception: an fMRI Study, Cerebral Cortex, Volume 12, Issue 7, Pages 772–782, https://doi.org/10.1093/cercor/12.7.772
    OpenUrlCrossRefPubMedWeb of Science
  45. ↵
    Sun HC, Di Luca M, Ban H, Muryy A, Fleming RW, Welchman AE (2016a) Differential Processing of Binocular and Monocular Gloss Cues in Human Visual Cortex. Journal of Neurophysiology, http://jn.physiology.org/content/jn/115/6/2779.full.pdf
  46. ↵
    Sun, HC, Welchman, AE, Chang, DHF, Di Luca, M. (2016b). Look but don’t touch: visual cues to surface structure drive somatosensory cortex. Neuroimage, 128: 353–361. http://dx.doi.Org/10.1016/j.neuroimage.2015.12.054
    OpenUrl
  47. ↵
    1. Rutherford, M. D. and
    2. Kuhlmeier, V. A.
    Troje N (2013) What Is Biological Motion? Definition, Stimuli, and Paradigms. In: Rutherford, M. D. and Kuhlmeier, V. A. (eds.) Social Perception: Detection and Interpretation of Animacy, Agency, and Intention. MIT Press. 13 – 36. doi:10.7551/mitpress/9780262019279.003.0002.
    OpenUrlCrossRef
  48. ↵
    Uesaki, M, Ashida, H (2015) Optic-flow selective cortical sensory regions associated with self-reported states of vection. Frontiers in Psychology. 6(775), 1–9. doi:10.3389/fpsyg.2015.00775
    OpenUrlCrossRef
  49. ↵
    van Assen J JR, Fleming R W (2016) Influence of optical material properties on the perception of liquids. Journal of Vision, 16(15), 12. https://doi.org/10.1167/16.15.12
    OpenUrlCrossRefPubMed
  50. ↵
    Xiao B, Bi W, Jia X, Wei H, Adelson EH (2016) Can You See What You Feel? Color and Folding Properties Affect Visual-Tactile Material Discrimination of Fabrics. Journal of Vision 16, no. 3 (February 23, 2016): 34. https://doi.org/10.1167/16.334
    OpenUrl
Back to top
Posted May 21, 2020.
Download PDF

Supplementary Material

Email
Dynamic dot displays reveal material motion network in the human brain
Alexandra C. Schmid, Huseyin Boyaci, Katja Doerschner
bioRxiv 2020.03.09.983593; doi: https://doi.org/10.1101/2020.03.09.983593
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area