A Novel Performance Metric for Multiclass Subject Invariant Brain Computer Interfaces with Imbalanced Classes

[Finding suitable common feature sets for use in multiclass subject independent brain-computer interface (BCI) classifiers is problematic due to characteristically large inter-subject variation of electroencephalographic signatures. We propose a wrapper search method using a one versus the rest discrete output classifier. Obtaining and evaluating the quality of feature sets requires the development of appropriate classifier metrics. A one versus the rest classifier must be evaluated by a scalar performance metric that provides feedback for the feature search algorithm. However, the one versus the rest discrete classifier is prone to settling into degenerate states for difficult discrimination problems. The chance of occurrence of degeneracy increases with the number of classes, number of subjects and imbalance between the number of samples in the majority and minority classes. This paper proposes a scalar Quality (Q)-factor to compensate for classifier degeneracy and to improve the convergence of the wrapper search. The Q-factor, calculated from the ratio of sensitivity to specificity of the confusion matrix, is applied as a penalty to the accuracy (1-error rate). This method is successfully applied to a multiclass subject independent BCI using 10 untrained subjects performing 4 motor tasks in conjunction with the Sequential Floating Forward Selection feature search algorithm and Support Vector Machine classifiers.]

49 While many approaches focus on subject specific models due to high inter-subject 50 variances present in larger populations, a wide array of studies have shown the 51 viability of subject independent (SI) models [2,7,10-13]. Reducing complexity is an 52 important part of emerging consumer grade EEG devices. Viability of an off the shelf 53 subject independent solution relies on methods capable of being deployed for mobile 54 devices and embedded platforms. While the computational capabilities of these 55 devices are rapidly increasing, power requirements play a key role in feasibility of 56 high computational complexity models. As a result, optimized models based on 57 traditional machine learning techniques gain an advantage for BCI applications.
58 Additionally, while BCI methods focus on interface and output classification, these 59 models can be used to develop a more comprehensive understanding of the inner 60 workings of the human brain for future research applications.

76
The 92 In this paper we propose a novel scalar metric that indicates the successful detection 93 rate and the degree of degeneracy of the classifier. We begin by providing 94 descriptions of the multi-class SIBCI feature space, the degeneracies of the one versus 95 the rest discrete output classifier, and the wrapper feature selection methods. We then 96 provide an outline for our experimental design, feature selection methods and an 97 overview of SFFS. This is followed by details of our proposed solution, and a 98 presentation of our results which contains a comparison of corrected and non-99 corrected data processed by both subject independent and subject specific models.

100
The final section provides an assessment of our results and a conclusion to the paper.

116
N(T) indicates the cardinality of each feature T, and indicates that the subject invariant 117 task may occupy a larger area of the feature space than for any individual subject.

118
Each N(T) increases as more subjects are included in T k . Fig 1A shows     This results a factor of (M-1)/2 more 164 feature subsets than tasks, none of which are uniquely assigned to a specific task.

165
Additional methods are then required to select a single unique feature set for each 166 task.

167
Generally speaking, for each class, there is an optimal discriminant function g i (x), i = so that g i (x)= 0 separates class ω i from all of the others where each classifier should according to the rule: 172    transformation into a scalar value which preserves both the accuracy information 333 while accounting for possible degeneracies in the solution. To achieve this, we 334 devised a metric henceforth dubbed as the Q -factor which is defined as: From elements of A, we can obtain overall accuracy (ACC), Sensitivity (SENS) and      Thus we can better assess our features by confusion matrix's imbalance in addition to 379 accuracy. While a simple accuracy calculation (Fig 3B) Where y is the simple accuracy, and IMB, the degree of imbalance, is given by (12).

388
This accomplishes our objective of moving any F(x,y) with a large value of imbalance 389 into a region of lower relevance and imbalance (Fig 3), clearing out the elements 390 located in Region 2 and moving them into Regions 1, 3 and 4 as shown in Fig 3C. 391 Finally, the Q-factor is given by

417
For comparison, we conducted both Q-factor guided and simple accuracy guided 418 feature searches on subject specific BCI (SSBCI) classifiers using our earlier-419 mentioned 10 subject BCI dataset (Fig 5A-B). The results of that comparison are 420 presented in Table 2. After application of the Q factor to the classifier, the SSBCI 421 results moved out of region 2 (high accuracy -high imbalance) into regions 1, 3, and 422 4. Prior to applying Q to our model, we found that the SFFS search algorithm more 423 frequently became trapped in local minima and was therefore unable to produce a 424 viable feature subset. The total number of convergent SFFS searches that produce 425 feature subsets is indicated in Table 3.

426
After incorporation of the Q factor an increase in converged searches was produced.  Table 3 shows three 450 areas where Q factor improves the performance.

451
First, the number of SFFS searches that produce feature sets has increased in all cases, 452 and more so for SIBCI models. While a small increase in the number of converged 453 wrapper models occurred for the SSBCI, the improvement is not as pronounced.

454
Uncorrected SIBCI wrapper searches encounter more difficulty in convergence due to 455 the lower discrimination of features as described earlier, and thus show larger 456 improvements when using Q-factor.

457
Second, we observe that the imbalance for all models improves after the application 458 of the Q factor. The improvement is most pronounced for SIBCI than for SSBCI, 459 except for SIBCI with CSP. We conjecture this is due to uncorrected CSP models not 460 exhibiting a high degree of imbalance in the first place.