Abstract
Neurofeedback (NF) allows to exert self-regulation over specific aspects of one’s own brain function by returning information extracted in real-time from brain activity measures. These measures are usually acquired from a single modality, most commonly EEG or fMRI. EEG-fMRI-neurofeedbaek (EEG-fMRI-NF) is a new NF approach that consists of providing a NF based simultaneously on EEG and fMRI. By exploiting the complementarity of these two modalities, EEG-fMRI-NF opens a spectrum of possibilities for defining bimodal NF targets that could be more ro-bust, flexible and effective than unimodal ones. However facing a greater amount of information, the question arises of how to represent the EEG and fMRI features simultaneously in order to allow the subject to achieve better self-regulation. In this work, we propose that the EEG and fMRI features should be represented in a single bimodal feedback, which we refer to as integrated feedback. We then introduce two integrated feedback strategies for EEG-fMRI-NF and compare their early effects on a motor imagery task with a between-group design. The first group (BI DIM, n=10) was shown a two-dimensional (2D) feedback in which each dimension de-picted the information from one modality. The second group (UNI DIM, n=10) was shown a one-dimensional (1D) feedback that integrated both types of information even further by merging them into one. Online fMRI activations were significantly higher in the UNI DIM group than in the BI DIM group, which suggests that the 1D feedback is easier to control than the 2D feedback. However when looking at posthoc activation levels, the difference of fMRI activation levels between the NF runs and the preliminary motor imagery run without NF was more significant in the 2D group. Moreover, subjects from the BI DIM group produced more specific BOLD activations with a notably stronger activation in the right superior parietal lobule (BI DIM > UNI DIM, p < 0.001, uncorrected). These results suggest that the 2D feedback encourages subjects to explore their strategies to recruit more spe-cific brain patterns. To summarize, our study shows that 1D and 2D integrated feedbacks are effective but also appear to be complementary and could therefore be used in combination in a bimodal NF training program. Altogether, our study paves the way to novel integrated feedback strategies for the development of flexible and effective bimodal NF paradigms making the most of the bimodal information and better suited to clinical applications.
1 Introduction
Neurofeedback is the process of feeding back real-time information to an individual about his/her ongoing brain activity so that he/she can learn to control some aspects of it, hopefully resulting in functional (behavioral, physical, cognitive, emotional) improvements (Arns et al., 2017; Sitaram et al., 2016; Sulzer et al., 2013; Thibault, Lifshitz, Birbaumer, et al., 2015). NF has been investigated for a host of clinical (Birbaumer, Ramos Murguial-day, et al., 2009) and non clinical applications (Arns et al., 2017; Gruzelier, 2014; Sitaram et al., 2016; Sulzer et al., 2013; Thibault, Lifshitz, Birbaumer, et al., 2015; Thibault, Lifshitz, and Raz, 2016). However its effective deployment in the clinical armamentiarium is being held back by the debated evidence about its efficiency, most likely as a result of poor study design and lack of established guidelines and knowledge about the underlying mechanisms of NF (Perronnet, Lécuyer, Lotte, et al., 2016; Thibault, Lifshitz, and Raz, 2016). In recent years, this situation has started to change as increasingly rigorous approaches are becoming the new standard (Stoeckel et al., 2014; Sulzer et al., 2013; Thibault, Lifshitz, and Raz, 2016), and as new studies are delving into the mechanisms (Birbaumer, Ruiz, and Sitaram, 2013; Emmert, Kopel, Sulzer, et al., 2016; Kober, Witte, et al., 2013; Ninaus et al., 2013; Sitaram et al., 2016) as well as the methodological aspects of NF (Emmert, Kopel, Koush, et al., 2017; Krause et al., 2017; Sepulveda et al., 2016; Sorger et al., 2016). However another reason for the debated efficiency of current approaches might be the inherent limitations of single imaging modalities (Biessmann et al., 2011; Fazli, Dahne, et al., 2015). Indeed, most NF approaches rely on the use of a single brain imaging modality such as EEG (Hammond, 2011), fMRI (Sulzer et al., 2013), functional near infra-red spectroscopy (fNIRS) (Kober, Wood, et al., 2014; Mihara et al., 2012) or magnetoencephalography (Buch et al., 2008; Lal et al., 2005; Sudre et al., 2011). Each of these modalities is sensitive to a particular biophysical phenomenon related to the brain activity and comes with technical and physiological limitations (Biessmann et al., 2011). NF studies often report a significant proportion of subjects (usually about 30%) that are not able to self-regulate, (Alkoby et al., 2017). In the brain-computer-interface (BCI) community this phenomenon is known as BCI-deficiency and might originate from non optimal features, flaws from the design (Chavarriaga et al., 2016; Lotte, Larrue, and Miihl, 2013), but also from anatomo-physiological factors that would make some subjects less responsive to certain modalities (Zich et al., 2015). EEG is the most popular NF modality for it is unexpensive and benefits from millisecond temporal resolution. However, its spatial resolution is limited by volume conduction of the head and the number of electrodes. Also source localization from EEG is inaccurate because of the ill-posed inverse problem (Baillet, Mosher, and Leahy, 2001; Grech et al., 2008). fMRI is being increasingly used for NF as it allows to regulate even deeper brain regions with high spatial resolution (Sulzer et al., 2013). However its temporal resolution is limited by the time required to acquire one brain volume (hundreds of milliseconds), and the fact that the hemodynamic response peak is delayed of 4-6s from the neuronal onset and that it acts like a low-pass filter that smears out the neuronal response.
Bimodal EEG-fMRI-neurofeedback (EEG-fMRI-NF) is a new neurofeedback (NF) approach that consists of using information coming simultaneously from EEG and fMRI in real-time to allow the subject to regulate electrophysiological and hemodynamic activities of their brain at the same time (Zotev et al., 2014). The feasibility of this approach was demonstrated by Zotev et al. who hypothesized that it could be more efficient than the unimodal approaches (ibid.). Bimodal EEG-fMRI-NF was compared against unimodal EEG-NF and fMRI-NF in a recent study (Perronnet, Lécuyer, Mano, et al., 2017). This study suggested that EEG-fMRI-NF could indeed be more specific or more engaging than EEG-NF as demonstrated by higher BOLD activations during EEG-fMRI-NF than during EEG-NF. It also highligted that during bimodal EEG-fMRI-NF subjects could happen to regulate more one modality than the other suggesting the existence of specific mechanisms involved when learning to regulate simultaneously hemodynamic and electrophysiological aspects of the brain activity.
EEG and fMRI share mutual information yet also contain important distinct features. However their degree of overlap is hard to predict. In the context of NF, the information coming from EEG and fMRI could therefore benefit from being integrated in order to be used as an efficient feedback. Yet integrating EEG and fMRI data is a real challenge (Biessmann et al., 2011; Fazli, Dahne, et al., 2015; Jorge, Van der Zwaag, and Figueiredo, 2014; Lahat, Adali, and Jutten, 2015). Multimodal data integration methods are categorized as asymmetrical (EEG-informed fMRI, fMRI-informed EEG) and symmetrical (data fusion, model-driven or data-driven) (Biessmann et al., 2011; Jorge, Van der Zwaag, and Figueiredo, 2014; Lahat, Adali, and Jutten, 2015). For NF purpose the integration method should be applicable in real-time. As illustrated by Figure 1, the integration of multimodal data can theoretically be made at different levels: the raw measures level, the features level (high level or multivariate), the NF signal level or the feedback level (Fazli, Dahne, et al., 2015). It is also possible not to integrate EEG and fMRI data and simply show them as two separate feedbacks but we argue that this might be sub-optimal (see below). Integrating EEG and fMRI at the measures level in real-time does not seem feasible due to the considerable amount of information that it would represent. In hybrid BCI, output of different classifiers are usually passed to a meta-classifier (Fazli, Dahne, et al., 2015; Fazli, Mehnert, et al., 2012). In NF it is less common to use a classifier (Huster et al., 2014) and the feature often directly constitutes the NF signal. We argue that for EEG-fMRI-NF, EEG and fMRI integration can be done directly at the feedback level and that this already has strong implication.
A few unimodal NF/BCI studies have investigated the effects of feedback presentation (Darvishi et al., 2017; Jeunet, Vi, et al., 2015; Kaufmann and Williamson, 2011; Krause et al., 2017; Ono et al., 2014; Sollfrank et al., 2016; Stoeckel et al., 2014) which is a central issue in NF/BCI design. In the case of bimodal NF, feedback design might be even more critical as there is more information to display and as the EEG and fMRI bits of information have different spatio-temporal dynamic properties. To our knowledge, no previous work has addressed the question of how to represent the EEG and fMRI information simultaneously and how the bimodal representation would affect NF performance.
In their pioneering work Zotev et al. naturally extended the classical thermometer feedback to bimodal NF by juxtaposing two feedback gauges, one for EEG and one for fMRI (Zotev et al., 2014). Though this has the advantage of clearly and fully representing both features, this could suffer from a few drawbacks. Firstly, it might be suboptimal in term of cognitive load (Gaume et al., 2016; Sweller, Merrienboer, and Paas, 1998) because the subject has to concentrate on two gauges. Secondly, the fact that the representations of both signals are separated seem to imply that there are two targets to reach. Therefore the regulation task might be perceived by the subject as two simultaneous regulation tasks instead of one. Thirdly, it can be misleading when the subject tries to interpret how both features evolve in time, especially when they exhibit inconsistencies. Lastly, it does not fully exploit the possibility of using a NF target defined by the state of both features.
In contrast to representing the EEG and fMRI features with two separate feedbacks, we propose that both features could be represented in an integrated feedback: that is a single feedback having a single NF target characterized by the state of both features. This type of feedback has the advantage of encouraging the subject to perceive the bimodal NF task as a single self-regulation task and offers great flexibility in the definition of bimodal NF targets. In this study, we introduce two integrated feedback strategies (illustrated in figure Figure 2) and compare their early effects on an EEG-fMRI-NF guided motor-imagery task with a between-group design in order to evaluate which strategy is better than the other on a single EEG-fMRI-NF session. The first integrated feedback strategy is a two-dimensional (2D) plot in which each dimension depicts the information from one modality. The second integrated feedback strategy is a one-dimensional (1D) gauge that merges both information into one and therefore has a higher degree of integration than the 2D feedback. The performance of each strategy is evaluated in terms of sensitivity (activation level) and spatial specificity of the motor-imagery related EEG and fMRI activations.
2 Material and methods
The study was conducted at the Neurinfo platform (CHU Pontchaillou, Rennes, France) and was approved by the Institutional Review Board. Twenty right-handed NF-naive healthy volunteers with no prior MI-NF experience (mean age: 35 ± 10.6 years, 10 females) participated in the study. Participants were randomly assigned to the bidimensional (BI_DIM; mean age: 37 ± 14 years, 5 females) or to the uni-dimensional (UNI_DIM; mean age: 33 ± 6.2 years, 5 females) group. Throughout the whole experiment, the participants were lying down in the MR bore and wearing a 64 channel MR-compatible EEG cap.
2.1 Hypotheses
Figure 2 illustrates the separate feedback (following Zotev et al., 2014) as well as the integrated 2D and 1D feedback strategies and summarizes their potential advantages and drawbacks. Our integrated 2D feedback consists of a ball moving on a 2D diamond-shape plot, the left dimension representing the EEG feature and the right dimension representing the fMRI feature. This type of feedback was introduced in our previous work (Perronnet, Lécuyer, Mano, et al., 2017) and we propose here an upgraded version in which the plot background delineates regions that indicate prefered direction of effort, encouraging the subject to regulate EEG and fMRI equitably. Our integrated 1D feedback consists of a ball moving in a gauge, the ball position representing the average of the EEG and fMRI features. The background of the 1D gauge is splitted in four regions to give the subject reference points. Interestingly, the separate feedback and the integrated 2D feedback strategies share common advantages and drawbacks because they both map the bimodal information onto two dimensions. On the good side, they fully represent the EEG and fMRI features and allow to discriminate between both, but as a counterpart the difference of update rates is perceivable and inconsistencies between the two features can be misleading. The integrated 2D feedback is visually more optimal than the separate feedback as subjects only need to look at a single representation. However it might be complex to apprehend as the subjects need to understand how the ball travels in the 2D space. Subjects would therefore probably need more time to get used to this feedback. But the information it conveys is highly meaningful regarding how the subject is regulating both features at the same time. When the ball is on one side it means that the subject is controlling more one feature than the other, but when the ball is on the diagonal it means that the subject is controlling both features equally. The integrated 1D feedback has the advantage of being simpler to comprehend but as a counterpart it is less informative than the two other strategies.
2.2 Experimental protocol
After signing an informed consent form describing the MR environment, the participants were verbally informed about the goal of both the study and the protocol. They were instructed that during the NF runs, they would be presented with a ball moving in two dimensions (for the BI_DIM group) or in a one-dimensional gauge (for the UNI_DIM group) according to the activity in their motor regions as measured with EEG and fMRI (see Figure 2). Participants were told that they would have to bring the ball closer to the darker blue areas by imagining clenching their right-hand. This instruction was reminded in written form on the screen at the beginning of each NF run. More specifically we explained the participants that they would need to perform kinesthetic motor imagery (kMI)(Neuper et al., 2005) of their right-hand in order to control the ball. Kinesthetic motor imagery was defined as trying to feel the sensation of the motion rather than only visualizing it. Participants were informed about the nature of EEG and fMRI signal, and specifically about the 4-6 seconds delay of the hemodynamic response. Additionally, for participants in the BI_DIM group, we explained them that EEG was represented on the left axis while fMRI was represented on the right axis. This implied that when the ball would be on the left side, it would mean that they are controlling more EEG than fMRI, and on the opposite when the ball would be on the right side it would mean that they are controlling fMRI more than EEG. We told them that they should try to control both dimensions, i.e. try to move the ball near the diagonal. These instructions were given verbally at the beginning of the experiment and reminded later if the participant asked for it. Participants were asked not to move at all, especially during the course of a run. Video monitoring of the inside of the MR tube allowed to check for whole-body movements of the participant.
After receiving the instructions and having the EEG cap setup on his/her scalp, the participant was installed in the MR tube where we made sure the electrodes impedances one last time in the supine position. The experimental protocol then consisted of: a structural 3D T1; a preliminary MI run without NF (MI_pre), the data of which was used to calibrate the NF target (see subsection 2.5); three NF runs with a one minute break in between each; a post MI run without NF (MI_post). The five EEG-fMRI functional runs employed a block-design alternating 8 times 20s of rest and 20s of task (see Figure 3).
During rest, the screen displayed a white cross and participants were asked to concentrate on the cross and not on the passed or upcoming task block. During task, the screen displayed the cue “Imagine right” as well as the feedback during NF runs. The feedback consisted of a yellow ball moving in a two-dimensional plot for the BI_DIM group or in a one-dimensional gauge for the UNI DIM group. The participants were instructed to bring the ball closer to the darker blue area by performing kinesthetic motor imagery of their right hand clenching. The EEG feature was defined as the event-related desynchronization (ERD) (Pfurtscheller and Lopes da Silva, 1999) in the [8-30Hz] band of the EEG data filtered with a subject specific spatial filter (see Section 2.5 and 2.4) and was updated every 250ms. The fMRI feature was defined as the mean BOLD in a subject-specific motor region-of-interest (ROI) (see Section 2.5 and 2.4) and was updated at every repetition time (TR=1s). For the UNI_DIM group, the ball position was the average of the EEG and fMRI features (EEG_nf+fMRI_n f)/2. For the BI_DIM group, the right axis depicted the normalized fMRI feature while the left axis depicted the normalized EEG feature. At the end of the experiment, the participants were asked to fill out a questionnaire about their perceived performance, motivation, fatigue, interest and difficulty in performing the NF task. Figure 3 illustrates the experimental protocol.
2.3 Data acquisition
EEG and fMRI data were simultaneously acquired with a 64-channel MR-compatible EEG solution from Brain Products (Brain Products GmbH, Gilching, Germany) and a 3T Verio Siemens scanner (VB17) with a 12 channel head coil. Foam pads were used to restrict head motion. EEG data was sampled at 5kHz with FCz as the reference electrode and AFz as the ground electrode. fMRI acquisitions were performed using echo-planar imaging (EPI) with the following parameters: repetition time (TR) / echo time (TE) = 1000/23ms, FOV = 210 × 210mm2, voxel size = 2 × 2 × 4mm3, matrix size = 105 × 105, 16 slices, flip angle = 90°). Visual instructions and feedback were transmitted using the NordicNeurolab hardware and presented to the participant via an LCD screen and a rear-facing mirror fixed on the coil. As a structural reference for the fMRI analysis, a high resolution 3D T1 MPRAGE sequence was acquired with the following parameters: TR/TI/TE = 1900/900/2.26ms, GRAPPA 2, FOV = 256 × 256mm2 and 176 slabs, voxel size = 1 × 1 × 1/mm3, flip angle = 90°. Our multimodal EEG/fMRI-NF system (Mano et al. 2017) integrates EEG and fMRI data streams via a TCP/IP socket. The EEG data is pre-processed with BrainVision Recview (Brain Products GmbH, Gilching, Germany) software for gradient and ballistocardiogram (BCG) artifact correction (see Section 2.4) and sent to Matlab (The MathWorks, Inc., Natick, Massachussets, United States) for further processing. The fMRI data is pre-processed online for slice-time correction and motion correction with custom Matlab code adapted from SPM8 (FIL, Wellcome Trust Centre for Neuroimaging, UCL, London, UK). EEG and fMRI NF features are then computed and translated as feedback with Psychtoolbox (Kleiner et al., 2007).
2.4 Real-time data processing
During NF runs, online gradient artifact correction and BCG correction of the EEG data were done in BrainVision Recview (Brain Products GmbH, Gilching, Germany) software. The gradient artifact correction in Recview is based on the average artifact subtraction (AAS) method (Allen, Josephs, and Turner, 2000). At the beginning and throughout the length of each experiment, we checked that the signal quality of the ECG channel was good because BCG artifact correction relies on the quality of the ECG channel. We would also reset the template regularly before the start of each run. We used an artifact subtraction template of 2000ms and 4 templates for template drift correction. The data was then down-sampled to 200Hz and low pass filtered at 50 Hz (48 db slope) with a Butterworth filter. The data were subsequently corrected for BCG artifact (Allen, Polizzi, et al., 1998). The pulse model was searched in the first 15 seconds of the data. The pulse detection was based on a moving template matching approach with minimal pulse period of 800ms, minimum correlation threshold of 0.7, and amplitude ratio range from 0.6 to 1.2 relative to the pulse model. For pulse correction, a moving template was computed by averaging the 10 previously detected pulses, and the correction was done on a window length of [-100ms, 700ms] relatively to the R-peak. This corrected data was then sent to Matlab for feature extraction. The corrected data was filtered with the subject specific spatial filter FILT computed during the calibration phase (see Section 2.5). The band power in the [8-30Hz] band was then computed on this filtered data using the periodogram and a 2s window size, and it was normalized with the following ERD-like (Pfurtscheller and Lopes da Silva, 1999) formulae: where bp(t) is the power at time t, is the average power over the previous rest block (values between the fourteen and the nineteen seconds) and reverse = 1 if the selected filter FILT = frest>task or the default filter (laplacian around C3), or reverse = — 1 otherwise. Finally, the EEG feature was smoothed over the last four values, divided by EEGtresh (see subsection 2.5) and translated as visual feedback every 250ms.
The fMRI signal was pre-processed online for motion correction, slice-time correction and then the fMRI NF feature was computed according to the following definition: where Broi(v) (respectively Bbg(v)) is the average BOLD signal in the ROI (respectively in the background (BG)) at volume v, and (respectively is the ROI (respectively BG) baseline obtained by averaging the signal in the ROI (respectively in the BG) from the fourteenth to the nineteenth second (to account for the hemodynamic delay) of the previous rest block. The background was defined as a large slice (slice 6 out of 16) in deeper regions and used to cancel out global changes. Finally the fMRI feature was smoothed over the last three volumes, divided by fMRItresh (see subsection 2.5) and translated as visual feedback every 1s.
2.5 Calibration phase
In order to define subject-specific NF features, right at the end of the MI_pre run, the MI_pre EEG and fMRI data were pre-processed and analyzed to extract a spatial filter FILT and a threshold EEGthresh for the EEG NF feature as well as a BOLD ROI and a threshold fMRIthresh for the fMRI NF feature.
2.5.1 EEG calibration
Right at the end of the MI_pre run, the MI_pre data was pre-processed similarly to what was done in real-time (see Section 2.4) except that the BCG correction was done semi-automatically. Using the Common Spatial Pattern (CSP) method (Ramoser, Müller-Gerking, and Pfurtscheller, 2000), we then computed the pairs of spatial filters that best maximized the difference in [8-30Hz] power between rest and task blocks on 18 channels located over the motor regions (C3, C4, FC1, FC2, CP1, CP2, FC5, FC6, CP5. CP6, C1, C2, FC3, FC4, CP3, CP4, C5, C6). The first filter frest>task of the pair maximizes the power during the rest blocks while the second filter ftask>rest of the pair maximizes the power during the task blocks. If the eigenvalue of frest>task was greater than the inverse of the eigenvalue of ftask>rest (Blankertz et al., 2008), then the subject-specific filter FILT was set to frest>task, otherwise it was set to ftask>rest. In case the CSP filter did not look satisfactory (visual inspection to see if the MI_pre data filtered was correlated with the task), we used a laplacien filter over C3 instead (Nunez et al., 1997). The ERD feature was then computed (see Section 2.4) and the threshold for the EEG NF was set by computing the ERD threshold that was reached at least 30% of the time.
2.5.2 fMRI calibration
MI_pre fMRI data was pre-processed for slice-time correction, spatial realignment and spatial smoothing with a 6mm Gaussian kernel with SPM8. A first-level general linear model (GLM) analysis modeling the task and the rest was then performed. The fMRI ROI was defined by taking a 9 × 9 × 3 box around the maximum of activation (constrained to the left motor area) of the thresholded T-map (task > rest, p < 0.001, k > 10). The fMRI feature was then computed on this MI_pre data (see Section 2.4) and the threshold for the fMRI NF was set by computing the value that was reached at least 30% of the time.
2.6 Offline analysis
2.6.1 EEG analysis
For offline analysis, EEG signal was pre-processed similarly to what was done in realtime (see Section 2.4) except that the BCG correction was done semi-automatically. For each subject and run, we checked that the R-peaks were well detected by adapting the parameters and by correcting or selecting the R-peaks manually when necessary.
To analyze how the participants regulated their EEG NF feature, we re-computed the ERD values on offline pre-processed data filtered with the online FILT as defined in 2.4 except that the baseline was not computed sliding-block-wise, but instead by averaging power values after the first second and before the nineteenth second of all rest blocks. We refer to this feature as “online ERD”.
As the amount of calibration data was limited and as participants had no prior MI training, it is possible that the filter from the calibration was suboptimal. Therefore we also extracted the ERD values on data filtered with a posthoc FILT. We refer to this feature as “posthoc ERD”. The posthoc FILT was computed the same way as the online FILT (see Section 2.5) except that it was computed on the concatenation of MI_pre, NF1, NF1 and NF3 instead of MI_pre only.
For statistical analysis, the ERD values were standardized to z-scores by considering for each subject their mean and standard deviation over MI_pre, NF1, NF2, NF3, MI_post. For each run the standardized ERD values were averaged by considering the values between the first and the nineteenth second of all NF blocks but the first. The mean ERD over NF1, NF2 and NF3 was averaged to get the mean NF ERD . We also considered maxiNFi the best mean ERD over the three NF runs. We refer to the best NF run regarding the EEG feature as maxNFeeg.
2.6.2 fMRI analysis
The fMRI data from each of the five runs (MI_pre, NF1, NF2, NF3, MI_post) was pre-processed and analyzed with AutoMRI (Maumet. 2013), a proprietary software for fMRI analysis automation based on SPM8. Pre-processing included slice-time correction, spatial realignment, co-registration to the 3D T1, followed by spatial smoothing with a 8 mm Gaussian kernel. A first-level and second-level general linear model (GLM) analysis was performed. The first-level GLM included the canonical HRF for the task as well as its temporal and dispersion derivatives. For the second-level GLM analysis, the individual data were normalized to the Montreal Neurological Institute (MNI) template and grouped using a mixed effects linear model. The activation maps were corrected for multiple comparisons using Family-Wise error (FWE) correction (p < 0.05 with cluster size >10 voxels).
To analyze how the participants regulated the BOLD signal in the online ROI, we extracted the ROI percent signal change (PSC) on offline pre-processed data. For each participant and each run, the registered fMRI values were high-pass filtered (100 seconds) to remove the linear drift, averaged in the online ROI and transformed into PSC using the formulae (Broi(v) — m)/m where m is the average of Broi values from the fourteenth to the nineteenth second. We refer to this feature as “online PSC”.
Because NF training affects patterns beyond the one being fed back (Kopel et al., 2016; Wander et al., 2013), the same procedure was done to extract the PSC in a posthoc ROI defined by computing individually an average activation map over NF1, NF2 and NFS and taking a 9 × 9 × 3 box around the maximum of activation (constrained to the left motor area). We refer to this feature as “posthoc PSC”. Finally the PSC values were standardized to z-scores by considering for each subject their mean and standard deviation over MI_pre, NF1, NF2, NF3, MI_post. For each run the standardized PSC values were averaged across the last 16 volumes of all NF blocks but the first. The mean PSC over NF1, NF2 and NF3 was averaged to get the mean NF PSC . We also considered maxi NFi the best mean PSC over the three NF runs. We refer to the best NF run regarding the fMRI feature as maxNFfmri.
2.6.3 Statistical analysis
For each group (UNI_DlM/BI_DIM), each modality (EEG/fMRI) and level of feature (online/posthoc) we conducted non-parametric Friedman tests of the differences among MI_pre, , MI_post, as well as Wilcoxon signed-rank tests (signrank Matlab function) between and MI_pre as well as between maxi NFi and MI_pre with Bonferroni correction (corrected p-value: 0.05/3 conditions = 0.0167). For between group comparison we computed a Wilcoxon test (ranksum Matlab function, equivalent to Mann-Whitney U-test) on . The tests were done both for the online PSC and for the posthoc PSC.
3 Results
GLM analysis of both groups (UNI_DIM + BI_DIM) revealed activations during NF (see Figure 4) in: bilateral premotor cortex (BA 6) including left and right supplementary motor area (SMA), left and right inferior frontal gyrus (pars opercularis rolandic operculum) (BA 44), left and right inferior parietal lobule (IPL), left and right superior parietal lobule (SPL), left and right supramarginal lobule/gyrus (BA 40,BA 2,BA 48), left and right superior parietal (BA 7, BA 5), bilateral mid-cingulate cortex, left and right precuneus (BA 7). Deactivations were observed in right primary motor cortex (M1), left and right angular gyrus (BA 39), right cuneus (BA 18), left and right precuneus, left middle occipital (BA 10) and in the left inferior parietal lobule (BA 19).
GLM analysis of the BI_DIM group during NF revealed activations in (figure 5): Left PMC (BA 6)including SMA, left IPL (BA 40), left SPL (BA 7), right SPL (BA 5, BA 7), right superior occipital (BA 7). Deactivations were observed in right M1, (BA 4), left IPL (BA 19). GLM analysis of the UNI_DIM group during NF revealed activations in (Figure 5): left and right PMC (BA 6) including left and right SMA, left IPL (BA 40), left superior parietal lobule (BA 40), left and right supramarginal lobule (BA 2).
Deactivations were observed in the right angular gyrus (BA 39).
The BI_DIM group showed more activations (p < 0.001, uncorrected) than the UNI_DIM group in the right superior parietal lobule (BA 7).
Friedman tests between MI_pre, and MI_post were significant for posthoc EEG in the BI_DIM group (p=0.045, χ2(2,10) = 6.2) and for posthoc fMRI in the BI_DIM group (p=0.0136, χ2(2,10)= 8.6).
Wilcoxon signed rank tests between MI_pre and maxNF were significant for: online EEG (p=0.0098, signedrank = 52) and online fMRI (p=0.0195, signedrank = 50) in the UNI_DIM group; posthoc EEG (p=0.0020, signedrank =55) and posthoc fMRI (p=0.0137, signedrank =51) in the UNI_DIM group; and for posthoc EEG (p=0.0020, signedrank =55) and posthoc · fMRI (p=0.0020, signedrank =55) in the BI_DIM group. Wilcoxon signed rank tests between MI_pre and NF were significant for: posthoc EEG (p=0.0195, signedrank = 50) in the UNI_DIM group; posthoc EEG (p=0.0273, signedrank = 49) and posthoc fMRI (p=0.0039, signedrank = 54) in the BI_DIM group. Results are summarized in figure 8 and figure 9. During the NF runs the fMRI PSC in the online ROI was significantly higher in the UNI_DIM group than in the BI_DIM group (Wilcoxon: z = 3.0615, ranksum = 146, p = 0.0022).
Questionnaire: In the BI_DIM group 5 participants out of 10 found that the blocks were too short (against one who found them too long in the UNI_DIM group), and 5 participants out of 10 found that the feedback was not a good indicator of their motor imagery (against 0 in the UNI_DIM group).
4 Discussion
In the present study we introduced and evaluated two integrated feedback strategies for EEG-fMRI-NF: a 2D plot in which EEG and fMRI are mapped onto each dimension, and a 1D gauge that integrates both information even more by merging them into one. In contrast to representing the EEG and fMRI features with two separate feedbacks, these integrated feedback strategies represent both information in a single feedback with a single NF target. They have the advantage of relieving the cognitive load of the subject, to represent the task as a single regulation task instead of two and to allow to define a NF target characterized by the state of both signals.
Online and posthoc performance
Overall both strategies allowed participants to up-regulate Mi-related EEG and fMRI patterns, as demonstrated by the higher posthoc EEG and fMRI activation levels during maxNF/NF compared to MI_pre (see Figure 9). The improvement was even more significant on posthoc fMRI in the BI_DIM group.
Online fMRI activation level during NF were significantly higher in the UNI_DIM group than in the BI_DIM group (figure 8) which showed particularly high variability among participants and NF runs. Though the UNI_DIM worked better than the BI_DIM regarding the regulation of the initial (online) targets, their performance was moderate. Indeed, the online activation level improvement with respect to MI_pre was only significant for the UNI_DIM group in the maxNF run (see Figure 8). The loss of performance on the online fMRI activation level during NF with a bi-dimensional feedback was also observed in our previous study (Perronnet, L écuyer, Mano, et al., 2017). Our new results thus highlight the fact that the bi-dimensional feedback is harder to control than the uni-dimensional feedback and that this affects online EEG and fMRI activation levels differently, at least on a single-session basis. We hypothesize that this could be due to the higher complexity of the bi-dimensional feedback. This complexity comes from the fact that it has two degrees of freedom with slightly different update rates (4 Hz and 1 Hz), whose relationship is non-trivial, and one of which is delayed from the other. Subjects therefore need more time to get acquainted with this more complex feedback. By allowing subjects to discriminate between the information coming from EEG and fMRI, the bi-dimensional feedback leads subjects to make interpretations about EEG and fMRI contingency. They might be able to try different strategies and analyze how they affect both dimensions. In particular it can be disturbing when both dimensions seem to present inconsistencies. This could explain why half of the participants in the BI_DIM group reported that the feedback was not a good indicator of their motor imagery. The hypothesis that the bi-dimensional feedback is more complex and therefore requires more habituation time is supported by the fact that half of the participants in the BI_DIM group reported they found the training blocks too short (20 seconds) and by participants comments from the BI_DIM group: “it is hard to know which mental process will favor EEG activity and which one will favor fMRI activity”, “the discrepancy between EEG and fMRI did not help to control the feedback given the small number of trials”, “task blocks could have been longer to allow to test different strategies and observe their effect”. The fact that the loss of performance affected more fMRI than EEG could mean that they focused more on regulating the EEG because feedback from EEG is immediate while feedback from fMRI is delayed. Additionnally this could also be due to the fact that the feedback was moving 4 times faster in the EEG dimension.
Looking at the opposite trend between the online and posthoc activation levels of both groups (i.e. higher online fMRI activation levels for UNI_DIM and higher posthoc fMRI activation elvels for BI_DIM) suggests that participants in the BI_DIM group could have moved further away from their initial MI_pre calibration pattern than participants from the UNI_DIM group. Though the 2D feedback is more complex, it seems to encourage participants to explore mental strategies, interpret their effects on the two feedback dimensions in order to find a strategy that allows to control both dimensions equitably. Training block length might benefit from being adapted to the feedback strategy, with shorter block for the 1D feedback and longer block for the 2D feedback to allow for the exploration and interpretation of inner strategies. The 1D strategy could be well suited during earlier phases of a NF program as it is easier to control, while the 2D strategy could prove valuable in the longer term to reach more specific self-regulation.
Group distribution across the 3 NF runs
Looking at the distribution of online mean activation levels (figure 11) over the three NF runs shows how the two group populations evolved over the course of the training. In the first run, both populations were rather widespread and distributed along the EEG axis which suggests that participants started by exploring EEG. Participants from the BI_DIM group were also slightly distributed along the fMRI axis in the first NF run. In the second NF run, both populations were spread along the fMRI axis, which suggests that participants explored fMRI while keeping EEG at a mean level. In the third run, both populations are spread along the central (0.5) isoline, which suggests that participants adopted a strategy that minimized the errors in both dimensions. Overall the progression look similar in both group but the BI_DIM population is more widespread than UNI_DIM in NF1 and NF2. This higher variability might once again be due to the higher complexity of the feedback to which participants need to get used.
Accurate assessments of strategies, motivation, cognitive state but also of personality traits (such as the sense of agency …) (Jeunet, Lotte, et al., 2018; Jeunet, N’Kaoua, et al., 2015) are key to interpreting the learning curve and the effects of neurofeedback. Our questionnaire inquiring about strategies, fatigue, motivation, perceived performance, and confidence in the feedback gave us valuable qualitative hindsights about how subjects in the BI_DIM group reacted to the complexity of the feedback. In future experiments, especially some involving multiple NF sessions, we plan to use a more thorough questionnaire incorporating behavioral and personality aspects.
Activation maps BOLD activation maps show that during NF both groups significantly activated regions from the motor imagery network including premotor areas and posterior parietal areas (figures 5 and 4), as well as regions that have been shown to be consistently active during NF (Emmert, Kopel, Sulzer, et al., 2016) (mid-cingulate (ACC), supra-marginal (temporo-parietal), dlPFC, premotor,). Subcortical and cerebellar regions activations could not be identified as they were out of the field of view. The BI_DIM group showed more activations (p<0.001, uncorrected) than the UNI_DIM group in the right superior parietal lobule (BA 7). The SPL plays an essential role in many cognitive, perceptive, and motor-related processes (Culham and Kanwisher, 2001; Wang et al., 2015). In particular it has been reported to be activated both during motor execution (ME) and MI (Confalonieri et al., 2012; Fleming, Stinear, and Byblow, 2010; Hétu et al., 2013; Lotze and Halsband, 2006; Raffin et al., 2012; Sharma and Baron, 2013; Solodkin et al., 2004) though greater activation has been observed during MI than ME (Gerardin et al., 2000; Hanakawa et al., 2002). More specifically, the SPL is known to play a role in guiding motor activity in relation to spatial information (Buneo and Andersen, 2006; (Culham and Kanwisher, 2001; Wang et al., 2015) and to be crucial in the generation of mental motor representations (Sirigu et al., 1996). Several studies have demonstrated that impairments to the parietal cortex reduced MI ability (Danckert et al., 2002; McInnes, Friesen, and Boe, 2016; Sirigu et al., 1996). A meta-analysis recently conducted to determine which neurologic disorders/lesions impair or restrict MI ability showed that patients with parietal lobe damage were most impaired (McInnes, Friesen, and Boe, 2016). In MI, the SPL is thought to play a role in facilitating the planning and coordination of imagined movements and/or in indirectly inhibiting M1 through its connection with the SMA (Kasess et al., 2008; McInnes, Friesen, and Boe, 2016; Solodkin et al., 2004). Activations in the SPL have been shown to be more active during visual imagery than during kinaesthetic imagery (Guillot et al., 2009). However we found no significant activation in the occipital regions as would be expected during visual imagery. Therefore it is unlikely that the SPL activation would indicate that participants in the BI_DIM performed a motor imagery that would have been more visual than kinesthetic. The superior parietal cortex has also been demonstrated to be active during generalized neurofeedback when feedback is presented visually (Emmert, Kopel, Sulzer, et al., 2016; Ninaus et al., 2013; Sitaram et al., 2016). However the fact that the SPL was more significantly active in the BI_DIM group than in the UNI_DIM group suggest that it is more than a generalized NF effect. This activation could result from both the overlap of the motor imagery task and the self-regulation process (Sitaram et al., 2016), both of which could be more intense under the bi-dimensional condition.
Though not shown at the group comparison level, the overlay of UNI_DIM activations and BI_DIM activations (see Figure 5) shows that activations in the premotor areas were more widespread and bilateral in the UNI_DIM group while they were more localized and lateralized to the left hemisphere in the BI_DIM group. Also, the BI_DIM group showed significant deactivations in the right primary motor cortex while the UNI_DIM group did not. Overall, our results suggest that the bi-dimensional feedback triggered more specific activations than the uni-dimensional feedback.
Defining bimodal NF targets An integrated feedback allows to reward specific EEG/fMRI pair values and gives flexibility on the definition of the bimodal NF target, depending on the assumed spatio-temporal complementarity of the EEG and fMRI features. In this study, we designed the integrated strategies so that subjects would have to regulate both EEG and fMRI at the same time in order to reach the NF target. This assumes that such a state is possible. Indeed, neuro-vascular studies show that the electrophysiological and hemodynamic activity are correlated (Formaggio et al., 2010; Gonçalves et al., 2006; Murta et al., 2015; Ritter, Moosmann, and Villringer, 2009; Zaidi et al., 2015). For example, a study by (Zaidi et al., 2015) found significant correlations between hemodynamic peak-times of [HbO] and [HbR] signals with the underlying neural activity as measured with intra-cortical electrophysiology in primates, but not for their peak-amplitude. However depending on the type of tasks, the features, and the subjects, this might not necessarily be the case as illustrated in the study by De Vos et al. (De Vos et al., 2013) who reported no correlation between EEG and fMRI of a face processing task. Though it is hard to predict the degree of complementarity and redundancy of the EEG and fMRI features, it might be beneficial to take into consideration the degree of correlation of both features during the calibration phase.
Instead of defining the target on the “intersection” of the EEG and fMRI features, one could think of using a more laxist target defined by their “union”, that is the target would be reached when the EEG target or the fMRI target is reached. Such a target would be easier to reach, therefore potentially less specific, but it might be advantageous in order to limit the user frustration when used at the beginning of a protocol for example. Also the “union” strategy could be used in case the EEG and fMRI features would be hardly redundant. This could happen if the mental process being regulated was more complex and involved for example a cognitive regulation and an emotional regulation aspect each of which would be associated to one of the feature. Moreover in the “union” strategy, one could imagine displaying a secondary reward when the pair of EEG and fMRI features would reach the intersection without “penalizing” the subject when he/she does not control for both.
Artifacts
The strong artifacts affecting the EEG in the MR environment and the difficulty to correct them in real-time constitute the main current limitation of EEG-fMRI-NF. Artifact removal for simultaneous EEG/fMRI traditionnally consists of correcting the gradient and the BCG artifact with the AAS technique. However, this technique is likely to result in residual artifacts. In their pionneering work, Zotev et al. reported that residual MR and CB artifacts contributed up to 50% to the EEG feature after basic real-time signal processing (Zotev et al, 2014). Other sources of artifacts such as the helium pump (Nierhaus et al., 2013), the MR ventilation and motion (Fellner et al., 2016; Jansen et al., 2012) can also seriously affect the EEG data. As the helium pump artifact lies in the gamma range, in our particular case it should not affect our features of interest (alpha/beta). But altogether, these other sources of artifacts can limit the quality of the online EEG-nf signal and therefore also the potential of the EEG-fMRI-NF approach. However, the posthoc CSP patterns of the majority of the participants corresponded to motor imagery patterns which confirms that participants executed the task well and that the data was well pre-processed.
Alternative to AAS such as optimal basis sets (Niazy et al., 2005), independent component analysis (Mantini et al., 2007), reference-layer adaptive subtraction(Chowdhury et al, 2014) and carbon-wire loop (Abbott et al., 2014; Meer et al., 2016) have been proposed in order to better remove gradient and BCG artifacts or to remove the other types of artefacts. However only few methods are currently available for online use (Kr-ishnaswamy et al., 2016; Mayeli et al., 2016; Meer et al., 2016; Steyrl et al., 2017; Wu et al., 2016). In future experiments, using such methods could allow to improve for the EEG-NF signal quality. Interestingly, a recent approach called automated EEG-assisted retrospective motion correction (aE-REMCOR) (Wong et al., 2016)) uses the EEG data in order to estimate head motion and improve fMRI motion correction.
5 Conclusion
Our study introduces new integrated feedback strategies for EEG-fMRI-NF and demonstrates that during a motor imagery task they enable to regulate EEG and fMRI simultaneously, even when EEG and fMRI are integrated in a 1D feedback. Our results also suggest that the 1D feedback is easier to control on a single session while the 2D feedback encourages subjects to explore their strategies to find one that allows to control EEG and fMRI by recruiting more specific brain patterns. Altogether, our study paves the way to novel integrated EEG-fMRI-NF strategies for the development of flexible and effective NF paradigms that could prove useful for clinical applications.
6 Aknowledgements
This work has received a French government support granted to the CominLabs excellence laboratory and managed by the National Research Agency in the “Investing for the Future” program under reference ANR-lO-LABX-07-01. It was also financed by Brittany region under HEMISFER project, and the National Research Agency with the REBEL project and grant ANR-15-CE23-0013-01. MRI data acquisition was supported by the Neurinfo MRI research facility from the University of Rennes I. Neurinfo is granted by the European Union (FEDER), the French State, the Brittany Council, Rennes.