PT  - JOURNAL ARTICLE
AU  - Martin Steinegger
AU  - Johannes Söding
TI  - Clustering huge protein sequence sets in linear time
AID  - 10.1101/104034
DP  - 2018 Jan 01
TA  - bioRxiv
PG  - 104034
4099  - http://biorxiv.org/content/early/2018/01/05/104034.short
4100  - http://biorxiv.org/content/early/2018/01/05/104034.full
AB  - Metagenomic datasets contain billions of protein sequences that could greatly enhance large-scale functional annotation and structure prediction. Utilizing this enormous resource would require reducing its redundancy by similarity clustering. However, clustering hundreds of million of sequences is impractical using current algorithms because their runtimes scale as the input set size N times the number of clusters K, which is typically of similar order as N, resulting in runtimes that increase almost quadratically with N. We developed Linclust, the first clustering algorithm whose runtime scales as N, independent of K. It can also cluster datasets several times larger than the available main memory. We cluster 1.6 billion metagenomic sequence fragments in 14 hours on a single server to 50% sequence identity, > 1000 times faster than has been possible before. Linclust will help to unlock the great wealth contained in metagenomic and genomic sequence databases. (Open-source software and metaclust database: https://mmseqs.org/).