PT  - JOURNAL ARTICLE
AU  - Satoshi Hirose
TI  - Valid and powerful statistical test for decoding accuracy—proposal of Permutation-based Information Prevalence Inference using the <em>i</em>-th order statistic
AID  - 10.1101/578930
DP  - 2019 Jan 01
TA  - bioRxiv
PG  - 578930
4099  - http://biorxiv.org/content/early/2019/03/29/578930.short
4100  - http://biorxiv.org/content/early/2019/03/29/578930.full
AB  - In multivariate pattern analysis, such as fMRI decoding studies using pattern classification, a “second-level” group statistical test is typically performed after “first-level” decoding analyses for individual participants. Neuroscientists often test the mean decoding accuracy across participants against the chance level accuracy (e.g. Student t-test), to verify whether brain activation includes information about the label (i.e., cognitive content). However, because the decoding accuracy, as well as other information-like measures, can never be below the chance level, positive results for such tests only indicate that “at least one person in the population has information about the label in the brain.” Thus, such tests cannot provide inferences about the trend of the majority. In this study, a second-level statistical test procedure is proposed and referred to as the Permutation-based Information Prevalence Inference using the i-th order statistic (iPIPI). This procedure is robust against outliers and has high statistical power to provide an inference regarding major population trends. In iPIPI, the i-th worst samples decoding accuracy (i.e., the i-th order statistic) is compared to the null distribution estimated by the permutation tests, in order to test whether the proportion of the better-than-chance decoding accuracy in the population (information prevalence) is higher than the threshold. Thus, a significant result of the iPIPI can be interpreted as the “majority of the population has information about the label in the brain.” By applying this approach to artificial data, the robustness of iPIPI against outliers was confirmed, along with its high statistical power. In addition, theoretical detail is provided, and the use of this method is demonstrated by reporting a real dataset measured in an fMRI decoding study. The program code for this iPIPI method is available at the following website: http://www2.nict.go.jp/bnc/hirose/iPIPI/index.html.