Abstract
Crossmodal redundancy increases both the speed and accuracy of communication (Evans & Treisman, 2011). For example, rattlesnakes hold their tail aloft when rattling, ensuring that it is both seen and heard. This combined audio-visual display is harder to miss or misinterpret than either movement or rattling alone. Perceivers’ brains must be sensitive to such crossmodal redundancies in order to take advantage of them. One possible adaptation for this purpose is the use of a single neural code shared by both auditory and visual information. To test for such a shared neural code, we created emotionally expressive animation and music stimuli that were precisely matched on all of their dynamic features. Participants viewed these stimuli during fMRI brain scanning. Using representational similarity analysis (Kriegeskorte & Kievit, 2013), we show that a single model of stimulus features and emotion content fits activity in both auditory and visual brain areas. This code is also used supramodally in posterior superior temporal cortex, and is used to represent both prototypical and mixed emotions (e.g., Happy-Sad). Exploratory analysis revealed that stimulus features and emotion content are represented in unimodal areas even when stimuli are presented in the area’s non-preferred modality. This evidence for a shared neural code is consistent with adaptive signaling accounts of emotion perception, in which perceivers specifically adapted to perceive crossmodal redundancy accrue an evolutionary advantage.
