High-resolution 7-Tesla fMRI data on the perception of musical genres – an extension to the studyforrest dataset

Participants

Acquisition of the data described herein was part of a previously published study¹, and took place in close temporal proximity (no more than a few weeks apart). The participants in this data release are identical to those previously reported. They were fully instructed about the nature of the study and were paid a total of 100 EUR for their participation, which included the previously reported data acquisitions, as well as the one described herein. All data acquisitions were jointly approved by the ethics committee of the Otto-von-Guericke-University of Magdeburg, Germany (approval reference 37/13).

Stimulus

All stimuli employed in this study are identical to those used in a previous study [for details refer to 3]. They were five natural, stereo, high-quality music stimuli (6 s duration; 44.1 kHz sampling rate) for each of five different musical genres: 1) Ambient, 2) Roots Country 3) Heavy Metal, 4) 50s Rock’n’Roll, and 5) Symphonic (see Figure 1 for details).

Figure 1. Spectrograms for all 25 stimuli showing structural differences in the time-frequency characteristics of the five musical genres.

Each stimulus was a six second excerpt from the middle of a distinct musical piece. Excerpts were normalized so that their root-mean-square power values were equal, and a 50 ms quarter-sine ramp was applied at the start and end of each excerpt to suppress transients. Most prominent are the differences between music clips with and without vocal components.

Procedures and stimulation setup

The setup for audio-visual presentation was as previously reported¹. Participants listened to the audio using custom-built in-ear headphones, and an LCD projector displayed visual instructions on a rear-projection screen that they saw via a mirror attached to the head coil.

At the start of each recording session, during the preparatory MR scans, participants listened to a series of longer excerpts of musical pieces and songs from the five different genres. During this phase participants were instructed to request adjustments of the stimulus volume in order to guarantee optimal perception of the stimuli against the noise pedestal emitted by the scanner. There was no overlap between the songs presented in this phase and those used as stimuli in the main experiment.

Eight scanning runs followed the initial sound calibration. Each run was started by the participant with a key-press ready signal. There were 25 trials, with five different stimuli (Figure 1) for each of the five genres per run (see Figure 2 for details on the experiment design). At the end of each run participants were given the opportunity for a break of variable length until they indicated readiness for the next run. Most participants started the next run within a minute.

Figure 2. Experiment design.

(A) Trial configuration. The start of each trial was synchronized with the MRI volume acquisition trigger. When the trigger was received the permanently displayed white fixation cross turned green, a 6 s music stimulus was presented, and, immediately afterwards, the fixation cross turned white again. Stimulation was followed by a variable delay (minimum delay 4 s). For the five trials of a genre, a 4 s and 8 s delay occurred once, while the remained three trials included a 6 s delay period. Thereby all trials had 4–8 s of uniform stimulation (no audio, white fixation cross) after each musical stimulus. The order of delays was randomized within a run. During trials with an 8 s second delay participants were presented with a yes/no question four seconds after the end of the music stimulus. The content of the question was randomized and asked for particular features of the stimulus that had just ended (e.g., “Was there a female singer?”, “Did the song have a happy melody?”). Participants had to indicate their response by pressing one of two buttons with the index or middle finger of their right hand corresponding to the response alternative presented on the screen. “Yes” was always mapped to the left side (index finger), “No” always to the right side (middle finger). The question had the purpose of keeping the participants attentive to the stimuli and counteract the effect of increasing familiarity across multiple runs. (B) Run configuration. The 25 stimuli were identical across runs and presented exactly once per run. Order of stimulus genres within each run was counter-balanced using De Bruijn cycles⁶ (alphabet size = 5, counter-balancing level = 2), hence each genre was followed by any other genre equally often and exactly once. Eight unique genre order sequences were generated and used for all participants, while randomizing the order of run sequences across participants. This was done in order to enable the application of the hyperalignment algorithm⁴. Data acquisition for two participants showed anomalies with respect to this procedure (see Table 2 for details).

Stimulus presentation and response logging were implemented using PsychoPy⁷ running on a computer with the (Neuro)Debian operating system⁸.

Functional MRI data acquisition

The acquisition protocol for functional MRI was largely identical with the one previously reported¹, hence only differences and key facts are listed here.

Importantly, the same landmark-based procedure for automatic slice positioning that was used to align the scanner field-of-view between acquisition sessions, was used again to align the field-of-view for this acquisition with the one in the previous study¹. As the exact same alignment target was used, this led to a very similar field-of-view configuration across acquisitions.

Each acquisition run consisted of 153 volumes (repetition time of 2.0 seconds with no inter-volume gaps).

Physiological recordings

The cardiac and respiratory traces were recorded for the full duration of all eight runs. The acquisition setup for physiological was identical with the one previously reported¹.

Behavioral log files

Log files are available as plain text files with comma-separated value markup. All enumerations are zero-based. Each lines represents a trial. Columns for the following information are present: order of run in sequence (run), ID of trial sequence for this run (run_id; see Figure 2B), fMRI volume corresponding to stimulation start (total: volume, in the current run: run_volume), stimulus file name (stim), music genre label (genre), inter-stimulus interval in seconds (delay), flag whether a control question with presented (catch), measured asynchrony between MRI trigger and sound onset in seconds (sound_soa), and time stamp of the corresponding MRI trigger with respect to the start of the experiment in seconds (trigger_ts).

Information on the stimulus timing is also available in per-subject, per-run, per-condition plain-text files in FSL’s EV3 format: one line per stimulation event, three columns with stimulus onset and duration (both in seconds relative to the start of a scan), as well as a third column with an arbitrary intensity weight that is always set to 1.

fMRI data

All functional MRI data were converted from the DICOM format into the NIfTI format for publication using the same procedure as in 1.

fMRI data are available in three different flavours, each stored in an individual 4D image for each run separately. Raw BOLD data are stored in bold.nii.gz. While raw BOLD data are suitable for further analysis, they suffer from severe geometric distortions. BOLD data that have been distortion-corrected⁹ at the scanner console are provided in bold_dico.nii.gz. In addition, distortion-corrected data that have been anatomically aligned to a per-subject BOLD template image are available: bold_bold7Tp1_to_subjbold7Tp1.nii.gz.

Participant motion estimates

Head movement correction was performed with respect to a dedicated reference scan at the start of the recording session within scanner online reconstruction as part of the distortion correction procedure. The associated motion estimates are provided in a whitespace-delimited 6-column text file (translation X, Y, Z in mm, rotation around X, Y, Z in deg) with one row per fMRI volume for each run separately.

Physiological recordings

Physiological data were truncated to start with the first MRI trigger pulse and to end one volume acquisition duration after the last trigger pulse. Data are provided in a four-column (MRI trigger, respiratory trace, cardiac trace and oxygen saturation), space-delimited text file for each run. A log file of the automated conversion procedure is provided in the same directory (conversion.log). Sampling rate for the majority of all participants is 200 Hz (see Table 2 for exceptions).

Audio features

Recent experiments have shown that audio features can be predicted via regression models from fMRI signals to test stimulus coding hypotheses^3,10. To facilitate this activity with the current data we extracted four audio features from down-mixed mono stimuli. Feature extraction used a front-end windowed short-time Fourier transform, with window size 16384 samples (371.52 ms) and hop size 4410 samples (100 ms) yielding 63 overlapping feature vectors per stimulus file. Window parameters were chosen to trade temporal for spectral acuity, yielding frequency samples spaced linearly at 2.69 Hz intervals from 0–22.05 kHz. The four features extracted from this representation are described below.

Mel-Frequency Spectrum (mfs) – 48 dimensions. Motivated by human auditory perception the mel scale organizes frequency by equidistant pitch locations as determined by psychophysical experiments. We used the essentia open source audio processing library¹¹ to extract the mel-frequency spectrum, which yielded energy in mel bands by applying a frequency-domain filterbank¹² to the short-time Fourier spectrum. Frequency-domain filtering consisted of applying equal area overlapping triangular filters to the Fourier spectrum spaced according to the mel scale and normalized such that the sum of coefficients for every filter equals one.

Mel-Frequency Cepstral Coefficients (mfcc) – 48 dimensions. Cepstral features have been widely reported to perform well in speech recognition and music classification systems¹³, where the task is required to be sensitive to timbre. Typically, only the lower 10–20 cepstral coefficients (low quefrency) are retained; these encode the shape of the broad spectral envelope – an acoustic correlate of timbre. However, when sensitivity to timbre is not required, utilizing the upper coefficients (high quefrency), that encode fine spectral structure such as pitch, makes the feature robust to timbral changes¹⁴. We extracted the full set of 48 cepstral coefficients from the mel-frequency spectrum, by mapping the mel spectrum to a decibel amplitude scale and multiplying by the discrete cosine transform (DCT-II) matrix. It is expected that any application would first remove the constant first column and retain either the subsequent 13–20 coefficients or the remaining upper coefficients after those, depending on whether sensitivity or robustness to timbral difference is required. The remaining two features yield such a separation into low and high quefrency spectral components.

Low-Quefrency and High-Quefrency Mel-Frequency Spectrum (lq_mfs, hq_mfs). Although proven to be useful in machine classification tasks, cepstral coefficients are in a different domain than the spectrum. The last two features map selected cepstral coefficients back to the spectrum domain by reconstructing the 48 mel-frequency spectrum bands using the low-quefrency and high-quefrency mfcc coefficients respectively. In each case, the non-selected coefficients were zeroed and the resulting feature mapped back to the spectral domain using the inverse (transposed) DCT-II matrix and then inverting the decibel amplitude scale. These two sets of features represent broad-spectrum information (timbre) and fine-scale spectral structure (pitch) respectively. The product of these two spectra yields the mel-frequency spectrum.

Source code

The source code for descriptive statistics in Figure 1 and Figure 3, as well as the implementation for the analysis presented in Figure 4 is available in a Git repository at https://github.com/psychoinformatics-de/paper-f1000_pandora_data. Source code for the implementation of the stimulation paradigm and audio feature extraction are included in the data release. Additional scripts for data conversion and quality control are available at: https://github.com/hanke/gumpdata.

Figure 3. Summary statistics for head movement estimates across runs and participants.

These estimates indicate relative motion with respect to a dedicated reference scan at the beginning of each scan session. The area shaded in light gray depicts the range across participants, while the medium gray area indicates the 50% percentile around the mean, and the dark gray area shows ± one standard error of the mean. The black line indicates the median estimate. Dashed vertical lines indicate run boundaries where participants had a brief break. The red lines indicate motion estimate time series of outlier participants. An outlier was defined as a participant whose motion estimate exceeded a distance of two standard deviations from the mean across participants for at least one fMRI volume in a run. For a breakdown of detected outliers see Table 2.

Figure 4. Localization of genre-discriminating signals in the brain.

(A) Voxel-wise genre-selectivity label. Random-effects GLM group analysis (n=20) were computed using the FEAT component of FSL¹⁶. Individual contrasts were evaluated for each genre to identify voxels showing a BOLD response to this particular genre that is larger than the average response to all other genres. For all voxel clusters that show a significant difference at the group-level (cluster forming threshold Z=3.1, cluster probability threshold p<0.05) for any genre, the selectivity label was determined by the maximum Z statistic across all genres. No significant selective activation was found for the ambient genre. The majority of all voxels were labeled selective for one of the musical genres where stimuli contained vocals (country, rock’n’roll, heavy metal). Only a small cluster in BA44 R (Broca’s area) was labeled selective for symphonic music, despite the lack of speech content in these stimuli. (B) For comparison, the location of voxel clusters with above-chance classification accuracy for predicting the genre of a music stimulus (colors only indicate individual clusters, not association with particular genres). The associated areas are largely overlapping with the results of the GLM analysis. However, genre-discriminating signals were identified in a number of additional areas. For details on the MVP analysis and cluster statistics see Table 3. Unthresholded maps for GLM and MVP analyses are available at NeuroVault.org¹⁷ collection 308.