RT Journal Article
SR Electronic
T1 An Incrementally Updatable and Scalable System for Large-Scale Sequence Search using LSM Trees
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2021.02.05.429839
DO 10.1101/2021.02.05.429839
A1 Fatemeh Almodaresi
A1 Jamshed Khan
A1 Sergey Madaminov
A1 Prashant Pandey
A1 Michael Ferdman
A1 Rob Johnson
A1 Rob Patro
YR 2021
UL http://biorxiv.org/content/early/2021/02/08/2021.02.05.429839.abstract
AB Motivation In the past few years, researchers have proposed numerous indexing schemes for searching large databases of raw sequencing experiments. Most of these proposed indexes are approximate (i.e. with one-sided errors) in order to save space. Recently, researchers have published exact indexes—Mantis, VariMerge, and Bifrost—that can serve as colored de Bruijn graph representations in addition to serving as k-mer indexes. This new type of index is promising because it has the potential to support more complex analyses than simple searches. However, in order to be useful as indexes for large and growing repositories of raw sequencing data, they must scale to thousands of experiments and support efficient insertion of new data.Results In this paper, we show how to build a scalable and updatable exact sequence-search index. Specifically, we extend Mantis using the Bentley-Saxe transformation to support efficient updates. We demonstrate Mantis’s scalability by constructing an index of ≈ 40K samples from SRA by adding samples one at a time to an initial index of 10K samples.Compared to VariMerge and Bifrost, Mantis is more efficient in terms of index-construction time and memory, query time and memory, and index size. In our benchmarks, VariMerge and Bifrost scaled to only 5K and 80 samples, respectively, while Mantis scaled to more than 39K samples. Queries were over 24× faster in Mantis than in Bifrost (VariMerge does not immediately support general search queries we require). Mantis indexes were about 2.5× smaller than Bifrost’s indexes and about half as big as VariMerge’s indexes.Availability The updatable Mantis implementation is available at https://github.com/splatlab/mantis/tree/mergeMSTs.Contact rob{at}cs.umd.eduSupplementary information Supplementary data are available online.Competing Interest StatementRob Patro is a co-founder of Ocean Genomics Inc.