Abstract
In RNA-seq differential expression analysis, investigators aim to detect those genes with changes in expression level across experimental conditions, despite technical and biological variability in the observations. A common task is to accurately estimate the effect size, often in terms of a logarithmic fold change (LFC) in expression levels. When the counts of reads are low or highly variable in either or both conditions, the maximum likelihood estimates for the LFCs has high variance, leading to large estimates not representative of true differences, and poor ranking of genes by effect size. One approach is to introduce filtering thresholds and pseudocounts to exclude or moderate estimated LFCs. Filtering may result in a loss of genes from the analysis with true differences in expression across conditions, while pseudocounts provide a limited solution that needs to be adapted per dataset. Here, we propose the use of a heavy-tailed Cauchy prior distribution for effect sizes, which avoids the use of filter thresholds or pseudocounts. The proposed method, Approximate Posterior Estimation for GLM, apeglm, has lower bias than previously proposed shrinkage estimators, while still reducing variance for those genes with little information for statistical inference. The apeglm package is available as an R/Bioconductor package at http://bioconductor.org/packages/apeglm, and the methods can be called from within the DESeq2 software.
