New Results

Speech Production in Intracranial Electroencephalography: iBIDS Dataset

Maxime Verwoert, Maarten C. Ottenhoff, Sophocles Goulis, Albert J. Colon, Louis Wagner, Simon Tousseyn, Johannes P. van Dijk, Pieter L. Kubben, Christian Herff
doi: https://doi.org/10.1101/2022.04.05.487183

ABSTRACT

Speech production is an intricate process involving a large number of muscles and cognitive processes. The neural processes underlying speech production are not completely understood. As speech is a uniquely human ability, it can not be investigated in animal models. High-fidelity human data can only be obtained in clinical settings and is therefore not easily available to all researchers. Here, we provide a dataset of 10 participants reading out individual words while we measured intracranial EEG from a total of 1103 electrodes. The data, with its high temporal resolution and coverage of a large variety of cortical and sub-cortical brain regions, can help in understanding the speech production process better. Simultaneously, the data can be used to test speech decoding and synthesis approaches from neural data to develop speech Brain-Computer Interfaces and speech neuroprostheses.

Background & Summary

Brain-Computer Interfaces (BCIs1) that directly decode speech from neural activity have recently gained large attention as they could provide an intuitive means of communication for patients who lost the ability to speak2–7.

The creation of a speech neuroprosthesis depends on a firm understanding of the speech production process in the brain, the particular timing of brain regions involved and where to best decode them. Despite a number of existing models on the speech production process8, 9, the precise role of all areas involved has yet to be understood. Recent advances highlight that deeper brain structures, such as the hippocampus10–12 and thalamus13, 14, are also involved in language in general and speech production specifically. A dataset providing accessible data for a simple speech production task in cortical and deeper brain structures could help to further understand this intricate process.

Despite the fact that a full understanding of speech production is currently lacking, great advances have been made in the field of speech neuroprostheses recently. The decoding of a textual representation by decoding phonemes15, 16, phonetic17 or articulatory18 features, words19, full sentences20–23 or spotting of speech keywords24 is possible from neural recordings during actual speech production. Results are becoming robust enough for first trials in speech impaired patients7. To facilitate more natural communication, some studies aimed at directly synthesizing an audio waveform of speech from neural data recorded during speech production25–28. Initial results indicate that the decoding of speech processes is possible from imagined speech production from offline data29–31 and in real-time32, 33.

Most of these recent advances employ electrocorticography (ECoG), an invasive recording modality of neural activity that provides high temporal and spatial resolution and high signal-to-noise ratio34. Additionally, ECoG is less affected by movement artifacts than non-invasive measures of neural activity. Other studies have used intracortical microarrays to decode speech35–38 or a neurotrophic electrode39 to synthesize formant frequencies40, 41 from the motor cortex. An alternative measure of intracranial neural activity is stereotactic EEG (sEEG), in which electrode shafts are implanted into the brain through small burr holes42. sEEG is considered to be minimally invasive, as a large craniotomy is not necessary and the infection risk is therefore smaller43. Additionally, the method of implanting the electrodes is very similar to that used in Deep Brain Stimulation (DBS), a method that has been used in the treatment of Parkinson’s Disease for several decades. In DBS, electrodes routinely remain implanted for many years, giving hope for the potential of sEEG for long-term BCIs. Similar to ECoG, sEEG is used in the monitoring of epilogenic zones in the treatment of refractory epilepsy. Between 5 and 15 electrode shafts are typically implanted covering a large variety of cortical and sub-cortical brain areas. Here lies one of the main differences to ECoG: instead of high density coverage of specific regions, sEEG provides sparse sampling of multiple regions. This sparse sampling could provide great potential for various BCI applications44.

Invasive recordings of neural activity are usually obtained during surgeries for seizure localization or glioma resection and are therefore not available to many researchers working on traditional speech decoding or speech synthesis technologies. These researchers are part of an active research community investigating the potential of non-invasive brain measurement technologies for speech neuroprostheses. Techniques include scalp-electroencephalography (EEG)45–51, providing high temporal resolution, especially the Kara One database52 provides the foundation for many studies; Magnetoencephalography53, 54, providing more localized information than EEG; and Functional Near Infrared Spectroscopy55–58, providing localized information of cortical hemoglobin levels. The advances made by this community could also benefit invasive speech neuroprostheses, a dataset that is provided to everyone could be used to evaluate and leverage their approaches.

To facilitate an increased understanding of the speech production process in the brain, including deeper brain structures, and to accelerate the development of speech neuroprostheses, we provide this dataset of 10 participants speaking prompted words aloud while audio and intracranial EEG data are recorded simultaneously (Fig. 1).

Figure 1.
Figure 1.

Intracranial EEG and acoustic data are recorded simultaneously while participants read Dutch words shown on a laptop screen. Traces on the right of the figure represent 30 seconds of iEEG, audio and stimulus data. The colors in the iEEG traces represent different electrode shafts.

Methods

Participants

A total of 10 participants suffering from pharmaco-resistant epilepsy participated in our experiment (mean age 32.4 ±12.6 years; 5 male, 5 female). Patients were implanted with sEEG electrodes as part of the clinical therapy for their epilepsy. Electrode locations were purely determined based on clinical necessity. All participants joined the study on a voluntary basis and gave written informed consent. Experiment design and data recording were approved by the IRBs of both Maastricht Unviversity and Epilepsy Center Kempenhaeghe. Data recording was conducted under the supervision of experienced healthcare staff. All participants were native speakers of Dutch. Participants’ voices were pitch-shifted to ensure anonymity.

Experimental Design

In this study, participants were asked to read aloud words that were shown to them on a laptop screen (Fig. 1). A random word from the dutch IFA corpus59 and the numbers 1 to 10 was shown for 2 seconds during which patients read the word aloud. After each word, a fixation cross was displayed for one second. A total of 100 words was recorded for each participant resulting in a total recording time of 300 seconds.

Data Acquisition

Participants were implanted with platinum-iridium sEEG electrode shafts (Microdeep intracerebral electrodes; Dixi Medical, Beçanson, France) with a diameter of 0.8 mm, a contact length of 2 mm and a inter-contact distance of 1.5 mm. Each electrode shaft contained between 8 and 18 electrode contacts. Neural data was recorded using one or two Micromed SD LTM amplifier(s) (Micromed S.p.A., Treviso, Italy) with 128 channels each. Electrode contacts were referenced to a common white matter contact. Data were recorded at either 1024 Hz or 2048 Hz and subsequently downsampled to 1024 Hz. We used the onboard microphone of the recording notebook (HP Probook) to record audio at 48 kHz. Audio data was subsequently pitch-shifted to ensure our participants’ anonymity using LibRosa60. We used LabStreamingLayer61 to synchronize audio and neural data and the experiment markers.

Anatomical Labeling

Electrode locations (Fig. 2) were detected using the img_pipe Python package62 for anatomical labeling of intracranial electrodes. Within the package, for each participant, a pre-implantation anatomical T1-weighted Magnetic Resonance Imaging (MRI) scan was parcellated using Freesurfer (http://surfer.nmr.mgh.harvard.edu/), a post-implantation Computer Tomography (CT) scan was co-registered to the MRI scan and electrode contacts were manually localized. Their anatomical location label was extracted from the Destrieux atlas63 based parcellation.

Figure 2.
Figure 2.

Electrode locations of each participant in the surface reconstruction of their native anatomical MRI. Each red sphere represents an implanted electrode channel.

By far the most electrodes are located in white matter (40.3%) and unknown areas (12.6%). Unknown areas are contacts that were not able to be labelled through the Freesurfer parcellation, for example a contact located just outside of the cortex. Thereafter, electrodes are predominantly located in the superior temporal sulcus, hippocampus and the inferior parietal gyrus.n See Fig. 3 for a full breakdown of anatomical regions and the number of electrodes implanted in those areas.

Figure 3.
Figure 3.

Number of electrode contacts in cortical and subcortical areas across all participants. Colors indicate participants. Lengths of the bars show the number of electrodes in the specified region. Note the deviant x-axis for the white matter and unknown regions.

Data Records

The dataset is available at https://osf.io/nrgx6/. The raw data files (XDF format) were converted to Neurodata Without Borders (NWB; https://www.nwb.org/) format and organised in the iBIDS64 data structure format using custom Python scripts. The NWB format allows for compact storage of multiple data streams within a single file. It is compatible to the iBIDS structure, a community-driven effort to improve the transparency, reusibility and reproducibility of iEEG data.

The data is structured following the BIDS version 1.7.0 specification (https://bids-specification.readthedocs.io/en/stable/). The root folder contains metadata of the participants (participants.tsv), subject specific data folders (i.e., sub-01) and a derivatives folder. The subject specific folders contain .tsv files with information about the implanted electrode coordinates (_electrodes.tsv), recording montage (_channels.tsv) and event markers (_events.tsv). The _ieeg.nwb file contains three raw data streams as timeseries (iEEG, audio and stimulus). Descriptions of recording aspects and of specific .tsv columns are provided in correspondingly named .json files (i.e., participants.json). The derivatives folder contains the pial surface cortical meshes of the right (_rh_pial.mat) and left (_lh_pial.mat) hemisphere per subject derived from the Freesurfer pipeline. The meshes are provided for plotting utilities rather than the T1-weighted anatomical scan, to retain the anonymity of the participants. The description column in the _channels.tsv file refers to the anatomical labels derived from the Destrieux atlas63.

The iBIDS dataset passed a validation check using the BIDS Validator (https://bids-standard.github.io/bids-validator/) and manual inspection of each datafile.

Technical Validation

We validate the recorded data by demonstrating that a spectral representation of speech can be reconstructed from the neural recordings using a simple linear regression model. This analysis is similar to a previous analysis in ECoG65.

Checking for Acoustic Contamination

Acoustic contamination of neural recordings has been reported by Roussel et al.66. To ensure that the presented dataset does not contain acoustic contamination in the neural timeseries, we apply the method provided by the authors and correlate spectral energy between audio and neural data. We do not find any significant correlations (p > 0.01) on the diagonal of the contamination matrix for any of the participants.

Feature Extraction

We extract the Hilbert envelope of the high-gamma band (70-170 Hz) for each contact using an IIR bandpass filter (filter order 4). To attenuate the first two harmonics of the 50 Hz line noise, we used two IIR bandstop filters (filter order 4). We averaged the envelope over 50 ms windows with a frameshift of 10 ms. To include temporal information into the decoding process, non-overlapping neighboring windows up to 200 ms into the past and future were stacked. Features are normalized to zero mean and unit variance using the mean and standard deviation of the training data. The same transform is then applied to the testing data.

The audio data is first downsampled to 16 kHz. To extract features for the audio data, we subsequently calculated the Short-Term-Fourier-Transform in windows of 50 ms with an frameshift of 10 ms. As the frameshift between neural and audio data is the same, there is a correspondence between audio and neural feature vectors. The resulting spectrogram is then compressed into a log-mel representation67 using 23 triangular filter banks.

Decoding Model

To reduce the dimensionality of our decoding problem, we compress the feature space to the first 50 principal components. Principal components are estimated for each fold individually on the training data. We reconstruct the log-mel spectrogram from the high-gamma features using linear regression models. In these models, the high-gamma feature vector is multiplied with a weight matrix to reconstruct the log-mel spectrogram. The weights are determined using a least-squares approach.

Waveform Reconstruction

The log-mel spectrogram does not contain the phase information anymore and an audio waveform can thus not be reconstructed directly. We utilize the method by Griffin and Lim68 for waveform reconstruction, in which the phase is initialized with noise and then iteratively modified. For a good algorithmic discription of the method, see69.

Results

All results are obtained in a non-shuffled 10-fold cross validation in which 9 folds are used for training and the remaining fold is used for evaluation. This process is repeated until each fold has been used for evaluation exactly once.

Spectrograms can be reconstructed

We evaluate the spectral reconstructions in terms of Pearson correlation coefficient between the spectral coefficients of the original speech and the reconstructed speech. For all 10 participants, speech spectrograms can be reconstructed from the neural data using linear regression (Fig. 4a). Reconstruction results were consistent across all 23 mel-scales spectral coefficients (Fig. 4b). Inspecting the spectrograms further (Fig. 5a), it can be seen that the results are mostly driven by the accurate reconstruction of speech versus silence. The spectral variations within speech are not captured by the linear regression approach. The Pearson correlation is not a perfect evaluation metric as this lack of detail during speech does not have a large impact on the score. We utilize the Pearson correlation here as a better metric has yet to be identified. By providing this open dataset,we hope that researchers developing more advanced metrics, such as the Spectro-Temporal Glimpsing Index (STGI)70 or the extended Short-Time Objective Intelligibility (eSTOI)71, will have the means to address this problem. For better reconstruction of speech, models that are more informed about speech processes (e.g. Unit Selection28) or neural network approaches with enough trainable parameters to produce high-quality speech25, 26, 72–74 might be necessary.

Figure 4.
Figure 4.

Results for the spectral reconstruction. (a) Mean correlation coefficients for each participant across all spectral bins and folds. Reconstruction of the spectrogram is possible for all 10 participants. Whiskers indicate standard deviations. Results of individual folds are illustrated by points. (b) Mean correlation coefficients for each spectral bin. Correlations are stable across all spectral bins. Shaded areas show standard errors.

Figure 5.
Figure 5.

Spectrograms (a) and waveforms (b) of the original (top) and reconstructed (bottom) audio. The example contains five individual words from sub-06. While the linear regression approach captures speech and silent intervals very accurately, the finer spectral dynamics within speech are lost.

Waveforms of Speech can be Reconstructed

Using the method by Griffin and Lim68, we can recreate an audio waveform from the reconstructed spectrograms (Fig. 5b). The timing between original and reconstructed waveforms is very similar, but listening to the audio shows that a substantial part of the audio quality is lost due to the synthesis approach. This is particularly clear when listening to the audio recreated from the original spectrogams (_orig_synthesized.wav). State-of-the-art synthesis approaches, such as WaveGlow75 or WaveNet76, could lead to better quality reconstructions.

Usage Notes

iBIDS data

Scripts to handle the data can be obtained from our github repository.

Data loading and feature extraction

Neural traces, synchronized audio and experimental markers can be loaded using the provided extract_features.py script. High-gamma features are subsequently extracted and aligned to logarithmic mel-scaled spectrograms. Electrode channel names are also loaded.

Spectrogram Reconstruction & Waveform Synthesis

Reconstruction of the spectrogram as well as the resynthesis to an audio waveform is performed in the reconstruction_minimal.py script.

Anatomical data

Electrode locations can be found in the participant folder (_electrodes.tsv) and then be visualized using the cortical meshes (_lh_pial.mat and _rh_pial.mat) within the derivatives folder. These mesh files contain vertices coordinates and triangles, which are described by indices corresponding to vertex numbers.

Code availability

All Python code to re-run the technical validation described in this report can be found on our github. The code relies on the numpy77, scipy78, pynwb79, scikit-learn80 and pandas81 packages.

Author contributions statement

C.H., P.L.K., A.J.C., L.W. and S.T. designed the experiment. M.V., M.C.O. and C.H. recorded the data. M.V. and S.G. localized the electrodes. M.V and C.H. analyzed the data and wrote the manuscript. All authors approve of the manuscript.

Competing interests

The authors declare no competing interests.

Acknowledgements

CH acknowledges funding by the Dutch Research Council (NWO) for the project ‘DEcoding Speech In SEEG (DESIS)’.

References

  1. 1.↵
    Wolpaw, J., Birbaumer, N., McFarland, D., Pfurtscheller, G. & Vaughan, T. Brain–computer interfaces for communication and control. Clin. neurophysiology 113, 767–791 (2002).
    OpenUrl
  2. 2.↵
    Rabbani, Q., Milsap, G. & Crone, N. E. The potential for a speech brain–computer interface using chronic electrocorticog-raphy. Neurotherapeutics 16, 144–165 (2019).
    OpenUrlCrossRef
  3. 3.
    Schultz, T. et al. Biosignal-based spoken communication: A survey. IEEE/ACM Transactions on Audio, Speech Lang. Process. 25, 2257–2271, 10.1109/TASLP.2017.2752365 (2017).
    OpenUrl
  4. 4.
    Chakrabarti, S., Sandberg, H. M., Brumberg, J. S. & Krusienski, D. J. Progress in speech decoding from the electrocor-ticogram. Biomed. Eng. Lett. 5, 10–21 (2015).
    OpenUrl
  5. 5.
    Herff, C. & Schultz, T. Automatic speech recognition from neural signals: a focused review. Front. neuroscience 10 (2016).
  6. 6.
    Bocquelet, F., Hueber, T., Girin, L., Chabardès, S. & Yvert, B. Key considerations in designing a speech brain-computer interface. J. Physiol. 110, 392–401 (2016).
    OpenUrl
  7. 7.↵
    Moses, D. A. et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New Engl. J. Medicine 385, 217–227 (2021).
    OpenUrl
  8. 8.↵
    Tourville, J. A. & Guenther, F. H. The diva model: A neural theory of speech acquisition and production. Lang. cognitive processes 26, 952–981 (2011).
    OpenUrl
  9. 9.↵
    Hickok, G. Computational neuroanatomy of speech production. Nat. Rev. Neurosci. 13, 135 (2012).
    OpenUrlCrossRefPubMed
  10. 10.↵
    Piai, V. et al. Direct brain recordings reveal hippocampal rhythm underpinnings of language processing. Proc. Natl. Acad.Sci. 113, 11366–11371 (2016).
    OpenUrlAbstract/FREE Full Text
  11. 11.
    Duff, M. C. & Brown-Schmidt, S. The hippocampus and the flexible use and processing of language. Front. Human neuroscience 6, 69 (2012).
    OpenUrl
  12. 12.↵
    Covington, N. V. & Duff, M. C. Expanding the language network: Direct contributions from the hippocampus. Trends Cogn. Sci. 20, 869–870 (2016).
    OpenUrlCrossRefPubMed
  13. 13.↵
    Hebb, A. O. & Ojemann, G. A. The thalamus and language revisited. Brain Lang. 126, 99–108 (2013).
    OpenUrlCrossRefPubMed
  14. 14.↵
    Klostermann, F. & Ehlen, F. Functional roles of the thalamus for language capacities. Front. systems neuroscience 7, 32 (2013).
    OpenUrl
  15. 15.↵
    Mugler, E. M. et al. Direct classification of all american english phonemes using signals from functional speech motor cortex. J. neural engineering 11, 035015 (2014).
    OpenUrl
  16. 16.↵
    Ramsey, N. F. et al. Decoding spoken phonemes from sensorimotor cortex with high-density ecog grids. Neuroimage 180, 301–311 (2018).
    OpenUrl
  17. 17.↵
    Lotte, F. et al. Electrocorticographic representations of segmental features in continuous speech. Front. human neuroscience 9 (2015).
  18. 18.↵
    Mugler, E. M. et al. Differential representation of articulatory gestures and phonemes in precentral and inferior frontal gyri. J. Neurosci. 38, 9803–9813 (2018).
    OpenUrlAbstract/FREE Full Text
  19. 19.↵
    Kellis, S. et al. Decoding spoken words using local field potentials recorded from the cortical surface. Journal of Neural Engineering 7 (2010).
  20. 20.↵
    Herff, C. et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. neuroscience 9(2015).
  21. 21.
    Moses, D. A., Mesgarani, N., Leonard, M. K. & Chang, E. F. Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. J. neural engineering 13, 056004 (2016).
    OpenUrl
  22. 22.
    Moses, D. A., Leonard, M. K. & Chang, E. F. Real-time classification of auditory sentences using evoked cortical activity in humans. J. neural engineering 15, 036005 (2018).
    OpenUrl
  23. 23.↵
    Makin, J. G., Moses, D. A. & Chang, E. F. Machine translation of cortical activity to text with an encoder–decoder framework. Tech. Rep., Nature Publishing Group (2020).
  24. 24.↵
    Milsap, G. et al. Keyword spotting using human electrocorticographic recordings. Front. neuroscience 13, 60 (2019).
    OpenUrl
  25. 25.↵
    Anumanchipalli, G. K., Chartier, J. & Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Nature 568, 493–498, 10.1038/s41586-019-1119-1 (2019).
    OpenUrlCrossRefPubMed
  26. 26.↵
    Angrick, M. et al. Speech synthesis from ecog using densely connected 3d convolutional neural networks. J. neural engineering (2019).
  27. 27.
    Wang, R., Wang, Y. & Flinker, A. Reconstructing speech stimuli from human auditory cortex activity using a wavenet approach. In 2018 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 1–6 (IEEE, 2018).
  28. 28.↵
    Herff, C. et al. Generating natural, intelligible speech from brain activity in motor, premotor, and inferior frontal cortices. Front. Neurosci. 13, 1267 (2019).
    OpenUrlCrossRefPubMed
  29. 29.↵
    Martin, S. et al. Decoding spectrotemporal features of overt and covert speech from the human cortex. Front. neuroengi-neering 7 (2014).
  30. 30.
    Martin, S. et al. Word pair classification during imagined speech using direct brain recordings. Sci. reports 6, 25803(2016).
    OpenUrl
  31. 31.↵
    Proix, T. et al. Imagined speech can be decoded from low-and cross-frequency intracranial eeg features. Nat. communica-tions 13, 1–14 (2022).
    OpenUrl
  32. 32.↵
    Angrick, M. et al. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity.Commun. biology 4, 1–10 (2021).
    OpenUrl
  33. 33.↵
    Meng, K. et al. Implementation of a closed-loop bci system for real-time speech synthesis under clinical constraints. In 2022 10th International Winter Conference on Brain-Computer Interface (BCI), 1–6 (IEEE, 2022).
  34. 34.↵
    Parvizi, J. & Kastner, S. Promises and limitations of human intracranial electroencephalography. Nat. neuroscience 21, 474–483 (2018).
    OpenUrlCrossRefPubMed
  35. 35.↵
    Stavisky, S. D. et al. Decoding speech from intracortical multielectrode arrays in dorsal “arm/hand areas” of human motor cortex. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 93–97 (IEEE, 2018).
  36. 36.
    Stavisky, S. D. et al. Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis. Elife 8, e46015 (2019).
    OpenUrlCrossRef
  37. 37.
    Stavisky, S. D. et al. Speech-related dorsal motor cortex activity does not interfere with ibci cursor control. J. Neural Eng. 17, 016049 (2020).
    OpenUrl
  38. 38.↵
    Wilson, G. H. et al. Decoding spoken english phonemes from intracortical electrode arrays in dorsal precentral gyrus. bioRxiv (2020).
  39. 39.↵
    Bartels, J. et al. Neurotrophic electrode: method of assembly and implantation into human motor speech cortex. J. neuroscience methods 174, 168–176 (2008).
    OpenUrlPubMed
  40. 40.↵
    Brumberg, J., Wright, E., Andreasen, D., Guenther, F. & Kennedy, P. Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech motor cortex. Front. Neurosci. 5, 65, 10.3389/fnins.2011.00065 (2011).
    OpenUrlPubMed
  41. 41.↵
    Guenther, F. H. et al. A wireless brain-machine interface for real-time speech synthesis. PloS one 4, e8218 (2009).
    OpenUrlCrossRefPubMed
  42. 42.↵
    van der Loo, L. E. et al. Methodology, outcome, safety and in vivo accuracy in traditional frame-based stereoelectroen-cephalography. Acta neurochirurgica 159, 1733–1746 (2017).
    OpenUrl
  43. 43.↵
    Iida, K. & Otsubo, H. Stereoelectroencephalography: indication and efficacy. Neurol. medico-chirurgica 57, 375–385 (2017).
    OpenUrl
  44. 44.↵
    Herff, C., Krusienski, D. J. & Kubben, P. The potential of stereotactic-eeg for brain-computer interfaces: Current progress and future directions. Front. Neurosci. 14, 123 (2020).
    OpenUrl
  45. 45.↵
    Wang, L., Zhang, X., Zhong, X. & Zhang, Y. Analysis and classification of speech imagery eeg for bci. Biomed. Signal processing control 8, 901–908 (2013).
    OpenUrl
  46. 46.
    Sereshkeh, A. R., Trott, R., Bricout, A. & Chau, T. Online eeg classification of covert speech for brain–computer interfacing. Int. journal neural systems 27, 1750033 (2017).
    OpenUrl
  47. 47.
    García-Salinas, J. S., Villaseñor-Pineda, L., Reyes-García, C. A. & Torres-García, A. A. Transfer learning in imagined speech eeg-based bcis. Biomed. Signal Process. Control. 50, 151–157 (2019).
    OpenUrl
  48. 48.
    Cooney, C., Folli, R. & Coyle, D. Mel frequency cepstral coefficients enhance imagined speech decoding accuracy from eeg. In 2018 29th Irish Signals and Systems Conference (ISSC), 1–7 (IEEE, 2018).
  49. 49.
    Krishna, G., Tran, C., Carnahan, M. & Tewfik, A. Advancing speech recognition with no speech or with noisy speech. In 2019 27th European Signal Processing Conference (EUSIPCO), 1–5 (IEEE, 2019).
  50. 50.
    Sharon, R. A., Narayanan, S. S., Sur, M. & Murthy, A. H. Neural speech decoding during audition, imagination andproduction. IEEE Access 8, 149714–149729 (2020).
    OpenUrl
  51. 51.↵
    Islam, M. M. & Shuvo, M. M. H. Densenet based speech imagery eeg signal classification using gramian angular field. In 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), 149–154, 10.1109/ICAEE48663.2019. 8975572 (2019).
  52. 52.↵
    Zhao, S. & Rudzicz, F. Classifying phonological categories in imagined and articulated speech. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 992–996 (IEEE, 2015).
  53. 53.↵
    Wang, J., Kim, M., Hernandez-Mulero, A. W., Heitzman, D. & Ferrari, P. Towards decoding speech production from single-trial magnetoencephalography (meg) signals. In 2017 IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), 3036–3040 (IEEE, 2017).
  54. 54.↵
    Dash, D., Ferrari, P. & Wang, J. Decoding imagined and spoken phrases from non-invasive neural (meg) signals. Front.Neurosci. (2020).
  55. 55.↵
    Sereshkeh, A. R., Yousefi, R., Wong, A. T. & Chau, T. Online classification of imagined speech using functional near-infrared spectroscopy signals. J. neural engineering 16, 016005 (2018).
    OpenUrl
  56. 56.
    Herff, C., Heger, D., Putze, F., Guan, C. & Schultz, T. Cross-subject classification of speaking modes using fnirs. In International Conference on Neural Information Processing, 417–424 (Springer, 2012).
  57. 57.
    Liu, Y. & Ayaz, H. Speech recognition via fnirs based brain signals. Front. neuroscience 12, 695 (2018).
    OpenUrl
  58. 58.↵
    Rezazadeh Sereshkeh, A., Yousefi, R., Wong, A. T., Rudzicz, F. & Chau, T. Development of a ternary hybrid fnirs-eeg brain–computer interface based on imagined speech. Brain-Computer Interfaces 6, 128–140 (2019).
    OpenUrl
  59. 59.↵
    Van Son, R. J., Binnenpoorte, D., Heuvel, H. v. d. & Pols, L. The ifa corpus: a phonemically segmented dutch” open source” speech database. In 7th European Conference on Speech Communication and Technology.
  60. 60.↵
    McFee, B. et al. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, vol. 8, 18–25 (Citeseer, 2015).
    OpenUrl
  61. 61.↵
    Kothe, C. Lab streaming layer (lsl). https://github.com/sccn/labstreaminglayer 26, 2015 (2014).
    OpenUrl
  62. 62.↵
    Hamilton, L. S., Chang, D. L., Lee, M. B. & Chang, E. F. Semi-automated anatomical labeling and inter-subject warping of high-density intracranial recording electrodes in electrocorticography. Front. Neuroinformatics 11, 62 (2017).
    OpenUrl
  63. 63.↵
    Destrieux, C., Fischl, B., Dale, A. & Halgren, E. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. NeuroImage 53, 1–15, https://doi.org/10.1016/j.neuroimage.2010.06.010 (2010).
    OpenUrlCrossRefPubMedWeb of Science
  64. 64.↵
    Holdgraf, C. et al. Bids-ieeg: an extension to the brain imaging data structure (bids) specification for human intracranial electrophysiology. (2018).
  65. 65.↵
    Herff, C. et al. Towards direct speech synthesis from ECoG: A pilot study. In Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the, 1540–1543 (IEEE, 2016).
  66. 66.↵
    Roussel, P. et al. Observation and assessment of acoustic contamination of electrophysiological brain signals during speech production and sound perception. J. Neural Eng. 17, 056028 (2020).
    OpenUrl
  67. 67.↵
    Stevens, S. S., Volkmann, J. & Newman, E. B. A scale for the measurement of the psychological magnitude pitch. The J. Acoust. Soc. Am. 8, 185–190 (1937).
    OpenUrl
  68. 68.↵
    Griffin, D. & Lim, J. Signal estimation from modified short-time fourier transform. IEEE Transactions on acoustics, speech, signal processing 32, 236–243 (1984).
    OpenUrl
  69. 69.↵
    Bayram, I. An analytic wavelet transform with a flexible time-frequency covering. IEEE Transactions on Signal Process.61, 1131–1142 (2012).
    OpenUrl
  70. 70.↵
    Edraki, A., Chan, W. Y., Jensen, J. & Fogerty, D. A spectro-temporal glimpsing index (stgi) for speech intelligibility prediction. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, 2738–2742 (International Speech Communication Association, 2021).
  71. 71.↵
    Jensen, J. & Taal, C. H. An algorithm for predicting the intelligibility of speech masked by modulated noise maskers. IEEE/ACM Transactions on Audio, Speech, Lang. Process. 24, 2009–2022 (2016).
    OpenUrl
  72. 72.↵
    Angrick, M. et al. Speech synthesis from stereotactic eeg using an electrode shaft dependent multi-input convolutional neural network approach. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 6045–6048 (IEEE, 2021).
  73. 73.
    Kohler, J. et al. Synthesizing speech from intracranial depth electrodes using an encoder-decoder framework. arXiv preprint arxiv:2111.01457 (2021).
  74. 74.↵
    Wang, R. et al. Distributed feedforward and feedback processing across perisylvian cortex supports human speech. bioRxiv (2021).
  75. 75.↵
    Prenger, R., Valle, R. & Catanzaro, B. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3617–3621 (IEEE, 2019).
  76. 76.↵
    Van Den Oord, A. et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).
  77. 77.↵
    Harris, C. R. et al. Array programming with numpy. Nature 585, 357–362 (2020).
    OpenUrlCrossRefPubMed
  78. 78.↵
    Virtanen, P. et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nat. methods 17, 261–272 (2020).
    OpenUrlCrossRefPubMed
  79. 79.↵
    Teeters, J. L. et al. Neurodata without borders: creating a common data format for neurophysiology. Neuron 88, 629–634 (2015).
    OpenUrlCrossRefPubMed
  80. 80.↵
    Pedregosa, F. et al. Scikit-learn: Machine learning in python. J. machine Learn. research 12, 2825–2830 (2011).
    OpenUrl
  81. 81.↵
    McKinney, W. et al. pandas: a foundational python library for data analysis and statistics. Python for high performance scientific computing 14, 1–9 (2011).
    OpenUrl
Back to top
Posted April 08, 2022.
Download PDF
Data/Code
Email
Speech Production in Intracranial Electroencephalography: iBIDS Dataset
Maxime Verwoert, Maarten C. Ottenhoff, Sophocles Goulis, Albert J. Colon, Louis Wagner, Simon Tousseyn, Johannes P. van Dijk, Pieter L. Kubben, Christian Herff
bioRxiv 2022.04.05.487183; doi: https://doi.org/10.1101/2022.04.05.487183
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area