Skip to main content
SpringerLink
Log in

Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study

  • Original Article
  • Published:
Brain Structure and Function Aims and scope Submit manuscript

Abstract

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L (1998) Identification of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8(3):229–239

    CAS  PubMed  Google Scholar 

  • Andric M, Small SL (2012) gesture’s neural language. Front Psychol 3:99

    PubMed  PubMed Central  Google Scholar 

  • Arnstein D, Cui F, Keysers C, Maurits NM, Gazzola V (2011) µ-suppression during action observation and execution correlates with BOLD in dorsal premotor, inferior parietal, and SI cortices. J Neurosci 31(40):14243–14249

    CAS  PubMed  PubMed Central  Google Scholar 

  • Beauchamp MS (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15(2):145–153

    CAS  PubMed  Google Scholar 

  • Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7(11):1190–1192

    CAS  PubMed  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823

    CAS  PubMed  Google Scholar 

  • Biau E, Soto-Faraco S (2013) Beat gestures modulate auditory integration in speech perception. Brain Lang 124(2):143–152

    PubMed  Google Scholar 

  • Biau E, Soto-Faraco S (2015) Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech. Front Hum Neurosci 9:527

    PubMed  PubMed Central  Google Scholar 

  • Biau E, Torralba M, Fuentemilla L, de Diego Balaguer R, Soto-Faraco S (2015) Speaker’s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations. Cortex 68:76–85

    PubMed  Google Scholar 

  • Braboszcz C, Delorme A (2011) Lost in thoughts: neural markers of low alertness during mind wandering. Neuroimage 54(4):3040–3047

    PubMed  Google Scholar 

  • Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13(2):400–404

    CAS  PubMed  Google Scholar 

  • Buzsáki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304(5679):1926–1929

    Google Scholar 

  • Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16(5):805–816

    PubMed  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657

    CAS  PubMed  Google Scholar 

  • Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Debener S, Ullsperger M, Siegel M, Fiehler K, Von Cramon DY, Engel AK (2005) Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci 25(50):11730–11737

    CAS  PubMed  Google Scholar 

  • Debener S, Ullsperger M, Siegel M, Engel AK (2006) Single-trial EEG–fMRI reveals the dynamics of cognitive function. Trends Cogn Sci 10(12):558–563

    PubMed  Google Scholar 

  • Dick AS, Goldin-Meadow S, Hasson U, Skipper JI, Small SL (2009) Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30(11):3509–3526

    PubMed  PubMed Central  Google Scholar 

  • Dick AS, Goldin-Meadow S, Solodkin A, Small SL (2012a) Gesture in the developing brain. Dev Sci 15(2):165–180

    PubMed  Google Scholar 

  • Dick AS, Mok EH, Beharelle AR, Goldin-Meadow S, Small SL (2012b) Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech. Hum Brain Mapp 35:900–917

    PubMed  PubMed Central  Google Scholar 

  • Dosenbach NU, Fair DA, Miezin FM, Cohen AL, Wenger KK, Dosenbach RA, Raichle ME (2007) Distinct brain networks for adaptive and stable task control in humans. Proc Natl Acad Sci 104(26):11073–11078

    CAS  PubMed  Google Scholar 

  • Drijvers L, Özyürek A, Jensen O (2018) Hearing and seeing meaning in noise: alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension. Hum Brain Mapp 1–13 (in press)

  • Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K (2005) Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci USA 102(49):17798–17803

    CAS  PubMed  Google Scholar 

  • Foxe JJ, Snyder AC (2011) The role of alpha-band brain oscillations as a sensory suppression mechanism during selective attention. Front Psychol 2:154

    PubMed  PubMed Central  Google Scholar 

  • Goldin-Meadow S (2005) Hearing gesture: how our hands help us think. Harvard University Press, Cambridge

    Google Scholar 

  • Goldman RI, Stern JM, Engel J Jr, Cohen MS (2002) Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport 13(18):2487

    PubMed  PubMed Central  Google Scholar 

  • Green A, Straube B, Weis S, Jansen A, Willmes K, Konrad K, Kircher T (2009) Neural integration of iconic and unrelated coverbal gestures: a functional MRI study. Hum Brain Mapp 30(10):3309–3324

    PubMed  Google Scholar 

  • Groppe DM, Urbach TP, Kutas M (2011) Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review. Psychophysiology 48(12):1711–1725

    PubMed  PubMed Central  Google Scholar 

  • He Y, Gebhardt H, Steines M, Sammer G, Kircher T, Nagels A, Straube B (2015) The EEG and fMRI signatures of neural integration: an investigation of meaningful gestures and corresponding speech. Neuropsychologia 72:27–42

    PubMed  Google Scholar 

  • Holle H, Gunter TC, Rüschemeyer S-A, Hennenlotter A, Iacoboni M (2008) Neural correlates of the processing of co-speech gestures. Neuroimage 39(4):2010–2024

    PubMed  Google Scholar 

  • Holle H, Obleser J, Rueschemeyer S-A, Gunter TC (2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49(1):875–884

    PubMed  Google Scholar 

  • Hubbard AL, Wilson SM, Callan DE, Dapretto M (2009) Giving speech a hand: gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30(3):1028–1037

    PubMed  PubMed Central  Google Scholar 

  • Järveläinen J, Schuermann M, Hari R (2004) Activation of the human primary motor cortex during observation of tool use. Neuroimage 23(1):187–192

    PubMed  Google Scholar 

  • Kelly SD, Kravitz C, Hopkins M (2004) Neural correlates of bimodal speech and gesture comprehension. Brain Lang 89(1):253–260

    PubMed  Google Scholar 

  • Kircher T, Straube B, Leube D, Weis S, Sachs O, Willmes K, Green A (2009) Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures. Neuropsychologia 47(1):169–179

    PubMed  Google Scholar 

  • Kita S, Özyürek A (2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking. J Mem Lang 48(1):16–32

    Google Scholar 

  • Laufs H, Kleinschmidt A, Beyerle A, Eger E, Salek-Haddadi A, Preibisch C, Krakow K (2003a) EEG-correlated fMRI of human alpha activity. Neuroimage 19(4):1463–1476

    CAS  PubMed  Google Scholar 

  • Laufs H, Krakow K, Sterzer P, Eger E, Beyerle A, Salek-Haddadi A, Kleinschmidt A (2003b) Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest. Proc Natl Acad Sci 100(19):11053–11058

    CAS  PubMed  Google Scholar 

  • Laufs H, Holt JL, Elfont R, Krams M, Paul JS, Krakow K, Kleinschmidt A (2006) Where the BOLD signal goes when alpha EEG leaves. Neuroimage 31(4):1408–1418

    CAS  PubMed  Google Scholar 

  • Leske S, Tse A, Oosterhof NN, Hartmann T, Müller N, Keil J, Weisz N (2014) The strength of alpha and beta oscillations parametrically scale with the strength of an illusory auditory percept. Neuroimage 88:69–78

    PubMed  Google Scholar 

  • Liu Y, Bengson J, Huang H, Mangun GR, Ding M (2014) Top-down modulation of neural activity in anticipatory visual attention: control mechanisms revealed by simultaneous EEG–fMRI. Cereb Cortex 26(2):517–529

    PubMed  PubMed Central  Google Scholar 

  • Luo H, Liu Z, Poeppel D (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8(8):e1000445

    PubMed  PubMed Central  Google Scholar 

  • Makeig S, Inlow M (1993) Lapse in alertness: coherence of fluctuations in performance and EEG spectrum. Electroencephalogr Clin Neurophysiol 86(1):23–35

    CAS  PubMed  Google Scholar 

  • Maris E, Oostenveld R (2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164(1):177–190

    PubMed  Google Scholar 

  • McNeill D (2006) Gesture: a psycholinguistic approach. In: Brown E, Anderson A (eds) The Encyclopedia of Language and Linguistics. Elsevier, Amsterdam, Boston, pp 58–66

  • McNeill D (2008) Gesture and thought. University of Chicago Press, Chicago

    Google Scholar 

  • Nagels A, Kircher T, Steines M, Straube B (2015) Feeling addressed! The role of body orientation and co-speech gesture in social communication. Hum Brain Mapp 36(5):1925–1936

    PubMed  Google Scholar 

  • Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005) Valid conjunction inference with the minimum statistic. Neuroimage 25(3):653–660

    PubMed  Google Scholar 

  • Obleser J, Weisz N (2012) Suppressed alpha oscillations predict intelligibility of speech and its acoustic details. Cereb Cortex 22(11):2466–2477

    PubMed  Google Scholar 

  • Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9(1):97–113

    CAS  Google Scholar 

  • Oostenveld R, Fries P, Maris E, Schoffelen J-M (2011) FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Intell Neurosci 2011:1–9

    Google Scholar 

  • Özyürek A, Willems RM, Kita S, Hagoort P (2007) On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. J Cogn Neurosci 19(4):605–616

    PubMed  Google Scholar 

  • Perry A, Bentin S (2009) Mirror activity in the human brain while observing hand movements: a comparison between EEG desynchronization in the µ-range and previous fMRI results. Brain Res 1282:126–132

    CAS  PubMed  Google Scholar 

  • Perry A, Stein L, Bentin S (2011) Motor and attentional mechanisms involved in social interaction—evidence from mu and alpha EEG suppression. Neuroimage 58(3):895–904

    PubMed  Google Scholar 

  • Pfurtscheller G, Stancak Jr A, Neuper C (1996) Event-related synchronization (ERS) in the alpha band—an electrophysiological correlate of cortical idling: a review. Int J Psychophysiol 24(1):39–46

    CAS  PubMed  Google Scholar 

  • Posner MI (2008) Measuring alertness. Ann N Y Acad Sci 1129(1):193–199

    PubMed  Google Scholar 

  • Quandt LC, Marshall PJ, Shipley TF, Beilock SL, Goldin-Meadow S (2012) Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: an EEG study of action production and gesture observation. Neuropsychologia 50(12):2745–2751

    PubMed  PubMed Central  Google Scholar 

  • Romero YR, Keil J, Balz J, Niedeggen M, Gallinat J, Senkowski D (2016) Alpha-band oscillations reflect altered multisensory processing of the McGurk illusion in schizophrenia. Front Hum Neurosci 10:41

    Google Scholar 

  • Sadaghiani S, Kleinschmidt A (2016) Brain networks and α-oscillations: structural and functional foundations of cognitive control. Trends Cogn Sci 20(11):805–817

    PubMed  Google Scholar 

  • Sadaghiani S, Scheeringa R, Lehongre K, Morillon B, Giraud A-L, Kleinschmidt A (2010) Intrinsic connectivity networks, alpha oscillations, and tonic alertness: a simultaneous electroencephalography/functional magnetic resonance imaging study. J Neurosci 30(30):10243–10250

    CAS  PubMed  PubMed Central  Google Scholar 

  • Sammer G, Blecker C, Gebhardt H, Bischoff M, Stark R, Morgen K, Vaitl D (2007) Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload. Hum Brain Mapp 28(8):793–803

    PubMed  Google Scholar 

  • Scheeringa R, Petersson KM, Oostenveld R, Norris DG, Hagoort P, Bastiaansen M (2009) Trial-by-trial coupling between EEG and BOLD identifies networks related to alpha and theta EEG power increases during working memory maintenance. Neuroimage 44(3):1224–1238

    PubMed  Google Scholar 

  • Scheeringa R, Fries P, Petersson K-M, Oostenveld R, Grothe I, Norris DG, Bastiaansen M (2011) Neuronal dynamics underlying high-and low-frequency EEG oscillations contribute independently to the human BOLD signal. Neuron 69(3):572–583

    CAS  PubMed  Google Scholar 

  • Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12(3):106–113

    PubMed  PubMed Central  Google Scholar 

  • Seeley WW, Menon V, Schatzberg AF, Keller J, Glover GH, Kenna H, Greicius MD (2007) Dissociable intrinsic connectivity networks for salience processing and executive control. J Neurosci 27(9):2349–2356

    CAS  PubMed  PubMed Central  Google Scholar 

  • Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci 31(8):401–409

    CAS  PubMed  Google Scholar 

  • Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2009) Gestures orchestrate brain networks for language understanding. Curr Biol 19(8):661–667

    CAS  PubMed  PubMed Central  Google Scholar 

  • Slotnick SD, Schacter DL (2004) A sensory signature that distinguishes true from false memories. Nat Neurosci 7(6):664–672

    CAS  PubMed  Google Scholar 

  • Straube B, Green A, Bromberger B, Kircher T (2011) The differentiation of iconic and metaphoric gestures: common and unique integration processes. Hum Brain Mapp 32(4):520–533

    PubMed  Google Scholar 

  • Straube B, He Y, Steines M, Gebhardt H, Kircher T, Sammer G, Nagels A (2013) Supramodal neural processing of abstract information conveyed by speech and gesture. Front Behav Neurosci 7:120

    PubMed  PubMed Central  Google Scholar 

  • Straube B, Green A, Sass K, Kircher T (2014) Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia. Schizophr Bull 40(4):936–944

    PubMed  Google Scholar 

  • Sturm W, Longoni F, Fimm B, Dietrich T, Weis S, Kemna S, Willmes K (2004) Network for auditory intrinsic alertness: a PET study. Neuropsychologia 42(5):563–568

    PubMed  Google Scholar 

  • Thut G, Nietzel A, Brandt SA, Pascual-Leone A (2006) α-Band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. J Neurosci 26(37):9494–9502

    CAS  PubMed  Google Scholar 

  • Tseng H-H, Bossong MG, Modinos G, Chen K-M, McGuire P, Allen P (2015) A systematic review of multisensory cognitive–affective integration in schizophrenia. Neurosci Biobehav Rev 55:444–452

    PubMed  Google Scholar 

  • Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N, Joliot M (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1):273–289

    CAS  PubMed  Google Scholar 

  • Uhlhaas PJ, Singer W (2010) Abnormal neural oscillations and synchrony in schizophrenia. Nat Rev Neurosci 11(2):100–113

    CAS  PubMed  Google Scholar 

  • van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102(4):1181–1186

    PubMed  Google Scholar 

  • Wildgruber D, Ethofer T, Grandjean D, Kreifelts B (2009) A cerebral network model of speech prosody comprehension. Int J Speech Lang Pathol 11(4):277–281

    Google Scholar 

  • Willems RM, Özyürek A, Hagoort P (2007) When language meets action: the neural integration of gesture and speech. Cereb Cortex 17(10):2322–2333

    PubMed  Google Scholar 

  • Willems RM, Oostenveld R, Hagoort P (2008) Early decreases in alpha and gamma band power distinguish linguistic from visual information during spoken sentence comprehension. Brain Res 1219:78–90

    CAS  PubMed  Google Scholar 

  • Willems RM, Özyürek A, Hagoort P (2009) Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47(4):1992–2004

    PubMed  Google Scholar 

  • Wu YC, Coulson S (2010) Gestures modulate speech processing early in utterances. Neuroreport 21(7):522

    CAS  PubMed  PubMed Central  Google Scholar 

  • Zhao W, Riggs K, Schindler I, Holle H (2018) Transcranial magnetic stimulation over left inferior frontal and posterior temporal cortex disrupts gesture–speech integration. J Neurosci 38(8):1891–1900

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This research project is supported by a grant from the ‘Von Behring-Röntgen-Stiftung’ (Project no. 59-0002; 64-0001) and by the ‘Deutsche Forschungsgemeinschaft’ (Project no. DFG: STR 1146/11-2, STR 1146/9-1 and SFB/TRR135 project A3). M. S. is supported by the DFG (project no. STR 1146/4-1). B. S. is supported by the DFG (Project no. STR 1146/8-1).

Author information

Authors and Affiliations

  1. Translational Neuroimaging Lab, Department of Psychiatry and Psychotherapy, Marburg Center for Mind, Brain and Behavior (MCMBB), Philipps-University Marburg, Rudolf-Bultmann-Str. 8, 35039, Marburg, Germany

    Yifei He, Miriam Steines & Benjamin Straube

  2. Department of Psychiatry and Psychotherapy, Marburg Center for Mind, Brain and Behavior (MCMBB), Philipps-University Marburg, Rudolf-Bultmann-Str. 8, 35039, Marburg, Germany

    Jens Sommer & Tilo Kircher

  3. Department of General Linguistics, Johannes-Gutenberg University Mainz, Jakob-Welder-Weg 18, 55128, Mainz, Germany

    Arne Nagels

  4. Cognitive Neuroscience at Centre for Psychiatry, Justus Liebig University Giessen, Am Steg 28, 35385, Giessen, Germany

    Helge Gebhardt & Gebhard Sammer

Authors
  1. Yifei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miriam Steines
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jens Sommer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Helge Gebhardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Arne Nagels
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gebhard Sammer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tilo Kircher
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Benjamin Straube
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yifei He.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 1626 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, Y., Steines, M., Sommer, J. et al. Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study. Brain Struct Funct 223, 3073–3089 (2018). https://doi.org/10.1007/s00429-018-1674-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00429-018-1674-5

Keywords

Search

Navigation