Abstract
Genomes computationally inferred from large metagenomic data sets are often incomplete and may be missing functionally important content and strain variation. We introduce an information retrieval system for large metagenomic data sets that exploits the sparsity of DNA assembly graphs to efficiently extract subgraphs surround-ing an inferred genome. We apply this system to recover missing content from genome bins and show that substantial genomic se-quence variation is present in a real metagenome. Our software implementation is available at https://github.com/spacegraphcats/spacegraphcats under the 3-Clause BSD License.
Metagenomics is the analysis of microbial communities through shotgun DNA sequencing, which randomly samples many subsequences (reads) from the genomic DNA of each microbe present in the community. A common problem in metagenomics is the reconstruction of individual microbial genomes from the mixture. Typically this is done by first running an assembly algorithm that reconstructs longer linear regions based on a graph of the sampled subsequences (1), and then binning assembled contigs together using composi-tional features and genic content (2, 3). These “metagenome-assembled genomes” can then be analyzed for phylogenetic markers and metabolic function. In recent years, well over 10,000 metagenome-assembled genomes have been published, dramatically expanding our view of microbial life (4–8).
During the process of metagenome assembly and binning, some sequence present in the data may go unbinned due either to a failure to assemble (9, 10) or a failure to bin. The underlying technical reasons for these failures include low coverage, high sequencing error, high strain variation, and/or sequences with insufficient compositional or genic signal. Recent work has particularly focused on the problem of strain confusion, in which high strain variation results in considerable fragmentation of assembled content in mock or synthetic communities (9, 10); the extent and impact of strain confusion in real metagenomes is still unknown but potentially significant (11–13).
Associating missing metagenomic sequence to inferred bins or known genomes is technically challenging. Current ap-proaches to extending bins use manual inspection of sequence read pairs or assembly graphs and do not scale to many genomes (14, 15). In pangenome studies with the goal of inves-tigating strain variants of known genomes, current approaches use mapping or k-mer baiting, in which assembled sequences are used to extract reads or contigs from a metagenome (16–19). These methods fail to recover genomic content that does not overlap with the query, such as large indels or novel genomic islands.
Missing content may in some instances be connected in the assembly graph to known sequence, in which case it is possible to recover it via a graph-based neighborhood query (14, 15). A graph query approach allows the recovery of sequence dis-tinct from but connected to extant genomic regions, as well as sequence from regions that do not assemble well but are graph-proximal to the query (15). However, many graph query algorithms are NP-hard and hence computationally intractable in the general case; compounding the computational challenge, metagenome assembly graphs are frequently large, with mil-lions of nodes, and require 10s to 100s of gigabytes of RAM.
In this paper, we develop and implement a scalable graph query framework that crucially exploits the structural sparsity of compact De Bruijn assembly graphs in order to compute a succinct index data structure in linear time. We use this framework to perform neighborhood queries in large assembly graphs, which enables us to extract the genome of a novel bacterial species, recover missing sequence variation in amino acid sequences for genome bins, and identify missing content for metagenome-assembled genomes. Our methods are assembly-free and avoid error correction and other techniques that may discard strain information. These algorithms are available in an open-source Python software package, spacegraphcats (20).
Results
Dominating sets enable efficient neighborhood queries in large assembly graphs
We design and implement (20) a set of algorithms for efficiently finding content in a metagenome that is close to a query as measured by distance in a compact De Bruijn graph (cDBG) representation of the sequencing data (Figure 1). To accomplish this, we first organize the cDBG into pieces around a set of dominators that are collectively close to all vertices. In this context, the neighborhood of a query is the union of all pieces it overlaps; to enable efficient search, we build an invertible index of the pieces.
Starting from a collection of genomic sequences (a), we form an assembly graph where nodes represent distinct linear subsequences (b). In this assembly graph, known as a compact De Bruijn graph (3), nodes may represent many k-mers. The original genomic sequences correspond to walks in the graph, and shared nodes between the walks represent shared subsequences. We then (c) identify a subset of nodes D called a dominating set so that every node in the assembly graph is at distance at most one from some member of D (marked pink). We further partition the graph into pieces by assigning every node to exactly one of the closest members of D (beige regions in (c) and (d)). For a genomic query Q, the neighborhood of Q in this graph is the union of all pieces which share at least one k-mer with the query. The colorful subsets of the pieces in (d) correspond to the neighborhoods of the queries Q1, Q2.
We compute dominators so that the minimum distance from every vertex in the cDBG to some dominator is at most r (an r-dominating set) using Algorithm 1, which is based on the linear-time approximation algorithm given by Dvořák and Reidl (21). Although finding a minimum r-dominating set is NP-hard (22–24) and an approximation factor below log n is probably impossible (23) in general graphs, our approach guarantees constant-factor approximations in linear running time by exploiting the fact that (compact) De Bruijn graphs have bounded expansion, a special type of sparsity (25). Algo-rithm 1 first annotates the graph to determine the distances between all pairs of vertices at distance at most r (lines 1-3) and then uses these distances to ensure each vertex is close to a dominator. The core of the efficient distance computation is based on distance-truncated transitive fraternal (dtf) augmentations (21) which produce a directed graph in which each arc uv is labeled with ω(uv), the distance from u to v in the original cDBG.
Importantly, our implementation enhances the algorithm in (21) by computing only r − 1 rounds of dtf-augmentations in-stead of 2r − 1. Since augmentation is the computationally most expensive part of the pipeline and the running time depends non-linearly on the number of rounds, this vastly improves this algorithm’s scalability. To maintain approximation guar-antees on the dominating set size with fewer augmentations, we introduce a threshold parameter domThreshold(r) which affects the constant factor in our worst-case bound. We se-lected a threshold (see Appendix) that produces r-dominating sets of comparable size to those computed by the algorithm in (21). Additionally, we found that processing vertices using a minimum in-degree ordering (line 6) was superior to other common orders (e.g. arbitrary, min/max total degree, max in-degree).
After computing an r-dominating set D of G with algorithm 1, Algorithm 2 partitions the vertices of G into pieces so that each piece P [v] contains a connected set of vertices for which v is the closest member of D (Figure 1). Finally, we use minimal perfect hashing (mphfIndex) (26) to compute an invertible index∗ between pieces and their sequence content in the metagenome.
One feature of this approach is that the dominating set and index only need to be computed once for a given metagenome, independent of the number and content of anticipated queries. Queries can then be performed using Algorithm 3 in time that scales linearly with the size of their neighborhood – all genomic content assigned to pieces that contain part of the query.
Our implementations of these algorithms in spacegraphcats can be run on metagenomic data with millions of cDBG nodes; (Table 1) indexing takes under an hour, enabling queries to complete in seconds to minutes (Table 2). See Appendix A for full benchmarking (including cDBG construction). This provides us with a tool to systematically investigate assembly graph neighborhoods.
Number of cDBG nodes |V |, edge density of cDBG |E|/|V |, size of 1-dominating set |D|, average query size (k-mers) 
, and average number of pieces in query neighborhood 
. Queries are the 51 genomes and 23 genome bins fully present in podarV and HuSB1, respectively.
Time and memory usage of spacegraphcats for Algorithms 1-3 on representative metagenome data. The times for Algorithm 3 are averaged over all queries (see Table 1). Statistics reported for Algorithm 2 exclude lines 1-2 of pseudocode. Times are rounded to the nearest tenth of a second; memory is rounded to the nearest Megabyte.
Neighborhood queries enable recovery of relevant unknown genomic content
We first measure how well an inexact query can recover a target genome from a metagenome. For a bench-mark data set, we use the podarV data set (27). This is a “mock” metagenome containing genomes from 65 strains and species of bacteria and archaea, each grown independently and rendered into DNA, then combined and sequenced as a metagenome. This metagenome is a commonly used bench-mark for assembly (10, 28–30).
To evaluate the effectiveness of neighborhood query at re-covering strain variants, we choose three target genomes from podarV– Porphyromonas gingivalis ATCC 33277, Treponema denticola ATCC 35405, and Bacteroides thetaiotaomicron VPI-5482 – that have many taxonomically close relatives in Gen-bank. We then use these relatives to query the podarV mixture and measure the recovery of the target genome. The results, in Figure 2(a), show that graph neighborhood query can recover 35% or more of some target genomes starting from a relative with Jaccard similarity as low as 1%: even a small number of shared k-mers suffice to define a much larger neighborhood that contains related genomes.
Neighborhood queries enable recovery of relevant genomic content. (a) Left Panel: Recovery of each of three target genomes from podarV using queries at a variety of Jaccard distances from the target. Recovery is calculated as containment of target genome in query neighborhood. (b) Right Panel: Recovery of novel Proteiniclasticum content from podarV. Nucleotide content from two of the three known P. ruminis genomes overlapped approximately a megabase of sequence in the query neighborhood, which also contained approximately 2.3 Mbp of unknown sequence; the third known genome, P. ruminis CGMCC, was omitted from the figure as it is 99.7% similar to P. ruminis DSM. Numbers are in thousands of k-mers.
We next apply neighborhood query to retrieve an unknown genome from podarV. Several papers have noted the presence of unexpected sequence in the assemblies of this data, and Awad et al. identify at least two species that differ from the intended mock metagenome contents (10, 28). One species variant has partial matches to several different Fusobacterium nucleatum genomes, while the other incompletely matches to three strains of Proteiniclasticum ruminis.
The complete genomes of these two variants are not in pub-lic databases and, for the Proteiniclasticum variant, cannot be entirely recovered with existing approaches: when we assemble the reads that share k-mers with the available genomes, a marker-based analysis with CheckM estimates that 98.8% of the Fusobacterium variant is recovered, while only 72.96% of the Proteiniclasticum variant is recovered. We therefore try using neighborhood queries to expand our knowledge of the Proteiniclasticum variant.
We perform a neighborhood query into podarV with all three known Proteiniclasticum genomes from GenBank. We then extract the reads overlapping this neighborhood and assemble them with MEGAHIT. The retrieved genome neigh-borhood for Proteiniclasticum contains 2256K novel k-mers (Figure 2(b)). The reads from the query neighborhood assem-ble into a 3.1 Mbp genome bin. The assembly is estimated by CheckM to be 100% complete, with 10.3% contamination. The mean amino acid identity between P. ruminis ML2 and the neighborhood assembly is above 95%, suggesting that this is indeed the genome of the Proteiniclasticum variant, and that neighborhood query retrieves a full draft genome sequence (see Appendix A).
Query neighborhoods in a real metagenome do not always as-semble well
Real metagenomes may differ substantially from mock metagenomes in size, complexity, and content. In par-ticular, real metagenomes may contain a complex mixture of species and strain variants (31) and the performance of assem-bly and binning algorithms on these data sets is challenging to evaluate in the absence of ground truth. One recent compari-son of single-cell genomes and metagenome-assembled genomes in an ocean environment found that up to 40% of single-cell genome content may be missing in metagenome-assembled genomes (13).
We first ask whether genome query neighborhood sizes in a real metagenome differ from mock metagenomes. We examine genomes inferred from the SB1 sample from the Hu et al. (2016) study, in which 6 metagenomic samples taken from various types of oil reservoirs were sequenced, assembled, binned, and computationally analyzed for biochemical function (32). Examining the 23 binned genomes in GenBank originating from the SB1 sample, we compare the HuSB1 neighborhood size distribution with the podarV data set (Figure 3(a)). We see that more genome bins in HuSB1 have 1.5x or larger query neighborhoods than do the genomes in podarV. This suggests the presence of considerably more local neighborhood content in the real metagenome than in the mock metagenome.
We next investigate the metagenomic content in the query neighborhoods. As with the unknown variants in podarV, we use CheckM to estimate genome bin completeness. The estimated bin completeness for many of the query genomes is low (Appendix A). To see if the query neighborhoods contain missing marker genes, we assemble reads from the query neighborhoods using MEGAHIT. However, we find little improvement in the completion metrics (Figure 3(b)).
Investigating further, we find that the query neighborhood assemblies contain only between 4% and 56% of the neighbor-hood k-mer content (Appendix A), suggesting that MEGAHIT is not including many of the reads in the assembly of the query neighborhoods. This could result from high error rates and/or high strain variation in the underlying reads (9, 10).
To attempt to recover more gene content from the assem-blies, we turn to the Plass amino acid assembler (33). Plass implements an overlap-based amino acid assembly approach that is considerably more sensitive than nucleotide assemblers and could be more robust to errors and strain variation (34).
When we apply Plass to the reads from the query neighbor-hoods, we see a further increase in neighborhood completeness (Figure 3(b)). This suggests that the genome bin query neigh-borhoods contain real genes that are accessible to the Plass amino acid assembler. We note that these are unlikely to be false positives, since CheckM uses a highly specific Hidden Markov Model (HMM)-based approach to detecting marker genes (35).
Query neighborhoods in HuSB1 metagenome are large and contain additional marker genes. (a) Left Panel: Neighborhood sizes are larger in HuSB1 than in podarV. Here we query podarV and HuSB1 using each of 64 and 23 genomes fully present in the respective datasets and measure the relative size of its neighborhood—a size of 1 indicates that no additional sequence is present in the neighborhood, while a size of 2 indicates that the retrieved neighborhood is twice the size of the query genome. (b) Right Panel: Query neighborhoods are estimated to be more complete than the original genome bins. We query HuSB1 using each of 23 genomes binned from SB1, and assemble the resulting neighborhoods using MEGAHIT and Plass. The yellow points represent completeness estimates of MEGAHIT-assembled neighborhoods, while green and pink bars represent the additional or missing content in the Plass assemblies respectively.
Some query neighborhoods contain substantial strain variation
If strain variation is contributing to poor nucleotide assembly of marker genes in the query neigborhoods, then Plass should assemble these variants into similar amino acid sequences. Strain variation for unknown genes can be difficult to study due to lack of ground truth, but highly conserved proteins should be readily identifiable.
The gyrA gene encodes an essential DNA topoisomerase that participates in DNA supercoiling and was used by (32) as a phylogenetic marker. In the GenBank bins, we find that 15 of the 23 bins contain at least one gyrA sequence (with 18 gyrA genes total). We therefore use gyrA for an initial analysis of the Plass-assembled neighborhood content for all 23 bins. To avoid confounding effects of random sequencing error in the analysis and increase specificity at the cost of sensitivity, we focus only on high-abundance data: we truncate all reads in the query neighborhoods at any k-mer that appears fewer than five times, and run Plass on these abundance-trimmed reads from each neighborhood. We then search the gene assemblies with a gyrA-derived HMM, align all high-scoring matches, and calculate a pairwise similarity matrix from the resulting alignment.
When we examine all of the high-scoring gyrA protein matches in the hard-trimmed data, we see considerable se-quence variation in some query neighborhoods (Figure 4(a)). Much of this variation is present in fragmented Plass assem-blies; when the underlying nucleotide sequences are retrieved and used to construct a compact De Bruijn graph, the vari-ation is visible as spurs off of a few longer paths (insets in Figure 4(a)). When we count the number of well-supported amino acid variants in isolated positions (i.e. ignoring linkage between variants) we see that ten of the 23 neighborhoods increase in the number of gyrA genes, with four neighborhoods gaining a gyrA where none exists in the bin (Appendix A; see lowest inset in Figure 4(a) for one example). Only one neighborhood, M. bacterium, loses its gyrA genes, due to the stringent k-mer abundance trimming. Collectively, the use of the Plass assembler on genome neighborhoods substantially increases the number of gyrA sequences associated with bins.
Query neighborhoods in HuSB1 contain sequence variants and new genes. (a) Left Panel: gyrA has substantial minor sequence variation in several query neighborhoods. In this multidimensional scaling plot, each point represents a distinct gyrA sequence from the Plass assemblies of four representative query neighborhoods, colored by query binned genome. The triangles represent gyrA sequences originating from the query binned genome, if any are present. The inlays are visualizations of assembly graphs of reads that contain gyrA sequence in each neighborhood. Unitigs are colored by their cluster of origin; matches to gyrA sequences from the bin are highlighted using color from relevant triangle. (b) Right Panel: Genome neighborhoods re-associate annotated functionality to binned genomes. For each of 23 genome bins originating from HuSB1, we find the unbinned content by removing all orthologs found in the binned genomes in (36) and by counting distinct ortholog annotations once. Functional content is distributed throughout pathways present in the binned genomes, and increases functionality associated with binned genomes by approximately 13%.
A multiple sequence alignment and neighbor joining tree of representative gyrA amino acid fragments assembled by Plass from the genome neighborhoods in HuSB1.
We see this same pattern for many genes, including alaS, gyrB, rpb2 domain 6, recA, rplB, and rpsC (Appendix A). This shows that multiple variants of those proteins are present within at least some of the neighborhoods and implies the presence of underlying nucleotide strain variation. This strain variation may be one reason that nucleotide assembly performs poorly: on average, only 19.6% of Plass-assembled proteins are found within the nucleotide assemblies.
Query neighborhoods assembled with Plass contain addi-tional functional content
In addition to capturing variants close to sequences in the bins, we identify many novel genes in the query neighborhoods. We use KEGG to annotate the Plass-assembled amino acid sequences, and subtract any ho-molog already annotated in the genome bin. We also ignore homolog abundance so that each homolog is counted only once per neighborhood.
Novel functional content is distributed throughout path-ways present in the genome bins, and increases functionality associated with binned genomes by approximately 13% (Figure 4(b)). This includes orthologs in biologically relevant pathways such as methane metabolism, which are important for biogeochemical cycling in oil reservoirs (32).
Genes in these neighborhoods contain important metabolic functionality expanding the pathways already identified in (32). We find 40 unique orthologs involved in nitrogen fixation across eight neighborhoods, four of which had no ortholog in the bin. Importantly, we find the ratio of observed orthologs approximately matches that noted in (32), where two thirds of nitrogen fixation functionality is attributable to archaea (29 of 40 orthologs). This is in contrast to most ecological systems where bacteria are the dominant nitrogen fixers (32).
Discussion
Efficient graph algorithms provide novel tools for investigat-ing graph neighborhoods
Recent work has shown that incor-porating the structure of the assembly graph into the analysis of metagenome data can provide a more complete picture of gene content (14, 15). While this has provided evidence that it is useful to analyze sequence that has small graph distance from a query (is in a “neighborhood”), this approach has not been widely adopted due to the lack of available tools and methods. Naïvely, local expansion around many queries in the assembly graph does not scale to these types of analyses due to the overhead associated with searching in a massive graph. The neighborhood index structure described in this work overcomes this computational obstacle and enables rapid exploration of sequence data that is local to a query.
Because a partition into pieces provides an implicit data reduction (the cDBG edge relationships are subsumed by piece membership), the query-independent nature of the index allows many queries to be processed quickly without loading the entire graph into memory. Our approach consequently provides a data exploration framework not available in any other automated software packages.
Exploiting the structural sparsity of cDBGs is a crucial com-ponent of our algorithms. First, it is necessary to use graph structure to obtain a guarantee that Algorithm 2 finds a small number of pieces since the size of a minimum r-dominating set cannot be approximated better than a factor of log n in general graphs† unless NP ⊆ DTIME(nO(log log n)) (23). With-out such a guarantee, we cannot be sure that we are achieving significant data reduction by grouping cDBG vertices into pieces. Being able to do this in linear time also ensures that the indexing and querying can scale to very large data sets. Furthermore, because we utilize a broad structural characteri-zation (bounded expansion) rather than a highly specialized aspect of cDBGs, our methods enable neighborhood-based information retrieval in any domain whose graphs exhibit bounded expansion structure; examples include some infras-tructure, social, and communication networks (21, 37, 38).
Neighborhood queries extend genome bins
In both the podarV and HuSB1 metagenomes, neighborhood queries were able to identify additional content likely belonging to query genomes. In the podarV mock metagenome, we retrieved a potentially complete genome for an unknown strain based on partial matches to known genomes. In the HuSB1 metagenome, we increased the estimated completeness of all but one genome bin – in some cases substantially, e.g. in the case of Microgeno-mates bacterium 39_7 we added an estimated 16.5% to the genome bin. In both cases we rely solely on the structure of the assembly graph to expand the genome bins. We do not make use of sequence composition, contig abundance, or phylogenetic marker genes in our search. Neighborhood search provides an orthogonal set of information for genome-resolved metagenomics that could be used to improve current binning techniques.
Query neighborhoods from real metagenomes contain sub-stantial strain variation that may block assembly
Previous work suggests that metagenome assembly and binning ap-proaches are fragile to strain variation, which would prevent the characterization of some genomes from metagenomes (9, 10). The extent of this problem is unknown, although the major-ity of approaches to genome-resolved metagenomics rely on assembly and thus could be affected.
In this work, we find that some of the sequence missing from genome bins can be retrieved using neighborhood queries. For HuSB1, some genome bins are missing as much as 66.6% of marker genes from the original bins, with more than half of the 22 bins missing 20% or more; this accords well with evidence from a recent comparison of single-cell genomes and metagenome-assembled genomes (13), in which it was found that metagenome-assembled genomes were often missing 20% to 40% of single-cell genomic sequence. Neighborhood query followed by amino acid assembly recovers additional content for all but two of the genome bins; this is likely an underestimate, since Plass may also be failing to assemble some content.
We note that when we bioinformatically analyze the func-tion of the expanded genome content from neighborhood queries, our results are consistent with the previous metabolic analyses by (32), and extend the set of available genes by 13%. This suggests that current approaches to genome bin-ning are specific, and that the main question is sensitivity, which agrees with the conclusions in the recent analysis of Baltic Sea microbial genomes (13).
Neighborhood queries enable a genome-targeted workflow to recover strain variation
The analysis workflow described above starts with genome bins. The genome bins are used as a query into the metagenome assembly graph, following which we extract reads from the query neighborhood. We assemble these reads with the Plass amino acid assembler, and then analyze the assembly for gene content. We show that the Plass assembly contains strain-level heterogeneity at the amino acid level, and that this heterogeneity is supported by underlying nucleotide diversity. Even with stringent error trimming on the underlying reads, we identify at least thirteen novel gyrA sequences in ten genome neighborhoods.
Of note, this workflow explicitly associates the Plass as-sembled proteins with specific genome bins, as opposed to a whole-metagenome Plass assembly which recovers protein sequence from the entire metagenome but does not link those proteins to specific genomes. The binning-based workflow connects the increased sensitivity of Plass assembly to the full suite of tools available for genome-resolved metagenome analysis, including both phylogenomic and metabolic analysis. The spacegraphcats workflow does not separate regions of the graph shared in multiple query neighborhoods; existing strain recovery approaches such as DESMAN or MSPminer could be used for this purpose (16, 19).
One future step could be to characterize unbinned genomic content from metagenomes by looking at Plass-assembled marker genes in the metagenome that do not belong to any bin’s query neighborhood. This would provide an estimate of the extent of metagenome content remaining unbinned.
Conclusions
The assembly-independent approach described in this work provides an alternative window into metagenome content. We extend previous work showing that assembly-based methods are fragile to strain variation, and provide an alternative work-flow that substantially broadens our ability to characterize metagenome content. This first investigation focuses on only one mock and one real data set, but the neighborhood in-dexing approach should be broadly applicable to all shotgun metagenomes.
In this initial investigation of neighborhood indexing, we have focused on using neighborhood queries with a genome bin. We recognize that this approach is of limited use in regions where no genome bin is available; spacegraphcats is flexible and performant enough to support alternative approaches such as querying with k-mers belonging to genes of interest.
Potential applications of spacegraphcats in metagenomics include developing metrics for genome binning quality, an-alyzing pangenome neighborhood structure, exploring r-dominating sets for r > 1, extending analyses to colored De Bruijn graphs, and investigating de novo extraction of genomes based on neighborhood content. We could also ap-ply spacegraphcats to analyze the neighborhood structure of assembly graphs overlayed with physical contact information (from e.g. HiC), which could yield new applications in both metagenomics and genomics (39, 40).
More generally, the graph indexing approach developed here may be applicable well beyond metagenomes and se-quence analysis. The exploitation of bounded expansion to efficiently compute r-dominating sets on large graphs makes this technique applicable to a broad array of problems.
Materials and Methods
Data
We use two data sets: SRR606249 from podarV (41) and SRR1976948 (sample SB1) from hu (36). Each data set was first preprocessed to remove low-abundance k-mers as in (42), using trim-low-abund.py from khmer v2.1.2 (43) with the parame-ters -C 3 -Z 18 -M 20e9 -V -k 31. We build compact De Bruijn graphs using BCALM v2.2.0 (44). Stringent read trimming at low-abundance k-mers was done with trim-low-abund.py from khmer, with the parameters -C 5 -M 20e9 -k 31.
Software
The source code for the index construction and search is available at https://github.com/spacegraphcats/spacegraphcats (20). It is implemented in Python 3 under the 3-Clause BSD License.
Snakemake (45) workflows to reproduce all of the analysis are available at https://github.com/dib-lab/2018-paper-spacegraphcats/, and Jupyter Notebooks to recreate all of the figures are in that same repository (46). The notebooks rely on the numpy, matplotlib, pandas, scipy, and Vega-Lite libraries (47–51).
Benchmarking
We measured time and memory usage for Algo-rithms 1-3 by executing the following targets in the spacegraphcats conf/Snakefile: catlas.csv for rdomset, contigs.fa.gz.mphf for indexPieces, and search for search. We report wall time and maximum resident set size, running under Ubuntu 18.04 on an NSF Jetstream virtual machine with 10 cores and 30 GB of RAM (52, 53). To measure maximum resident set size, we used the memusg script (Jaeho Shin, https://gist.github.com/netj/526585).
Graph denoising
For each data set, we built a compact De Bruijn graph (cDBG) for k=31 by computing the set of unitigs with BCALM (54), removing all vertices of degree one with a mean k-mer abundance of 1.1 or less, and then contracting long degree-two paths when possible.
Neighborhood indexing and search
We use spacegraphcats to build an r-dominating set for the denoised cDBG and index it. We then perform neighborhood queries with spacegraphcats, which produces a set of cDBG nodes and reads that contributed to them. The full list of query genomes for the Proteiniclasticum query is available in Supp Material A, and the NCBI acces-sions for the P. gingivalis, T. denticola, and B. thetaiotamicron queries are in the directory pipeline-base of the paper repos-itory, files strain-gingivalis.txt, strain-denticola.txt, and strain-bacteroides.txt, respectively.
Search results analysis
Query neighborhood size, Jaccard contain-ment, and Jaccard similarity was estimated using modulo hash signatures with a k-mer size of 31 and a scaled factor of 1000, as implemented in sourmash v2.0a9 (55).
Assembly and genome bin analysis
We assembled reads using MEGAHIT v1.1.3 (28) and Plass v2-c7e35 (33), treating the reads as single-ended. Bin completeness was estimated with CheckM 1.0.11, with the –reduced_tree argument (35). Amino acid identity between bins and genomes was calculated using CompareM commit 7cd51276 (https://github.com/dparks1134/CompareM).
Gene targeted analysis
Analysis of specific genes was done with HM-MER v3.2.1 (56). Plass amino acid assemblies were queried with HMMER hmmscan using the PFAM domains listed in Supp Material 8, using a threshold score of 100 (57). Matching sequences were then extracted from the assemblies for further analysis. To overcome problems associated with comparing non-overlapping sequence frag-ments, only sequences that overlapped 125 of the most-overlapped 200 residues of the PFAM domain were retained (all sequences shared a minimum overlap of 50 residues with all other sequences). These sequences were aligned with MAFFT v7.407 with the –auto argument (58). Pairwise similarities were calculated using HMMER where the final value represented the number of identical amino acids in the alignment divided by the number of overlapping residues between the sequences. Pairwise distances were visualized using a multidimensional scaling calculated in R using the cmdscale func-tion. To visualize the assembly graph structure underlying these amino acid assemblies, we use paladin v1.3.1 to map abundance-trimmed reads back to the Plass amino acid assembly, with the -f argument set to 125 to set the minimum ORF length accepted (59). We extracted the reads that mapped to the gene of interest, created an assembly graph using BCALM v2.2.0 (54), and visualized the graph using Bandage v0.8.1 (60).
KEGG Analysis
We annotated the Plass assemblies using Ky-oto Encyclopedia of Genes GhostKOALA v2.0 (61). To as-sign KEGG ortholog function, we used methods implemented at https://github.com/edgraham/GhostKoalaParser release 1.1.
Bin and neighborhood completeness, as estimated by CheckM. “Bin completeness” is the result of running CheckM on the genome sequence from GenBank; MEGAHIT is the result of running CheckM on the MEGAHIT nu-cleotide assembly of the neighborhood reads; Plass is the result of running CheckM on the Plass amino acid assembly of the neighborhood reads. ∆ is the difference between the column and the bin completeness.
Bin and neighborhood gyrA protein content. gyrA count for each bin is the number of gyrA amino acid sequences part of the original bin. gyrA count by Plass is the minimum number of gyrA amino acid sequences supported by at least one position with at least 10 copies of each variant, e.g. “3” indicates that there is at least one position in the multiple sequence alignment of gyrA sequences for that neighborhood that has 3 distinct vari-ants in 10 distinct sequences.
ACKNOWLEDGMENTS
This work is funded in part by the Gor-don and Betty Moore Foundation’s Data-Driven Discovery Initiative through Grants GBMF4551 to C. Titus Brown, GBMF4553 to Jef-frey Heer, and GBMF4560 to Blair D. Sullivan. This work arose from the Barnraising for Data-Intensive Discovery at Mt. Desert Island Biological Lab in May 2016. We thank Erich Schwarz, Mar-tin Steinegger, Johannes Söding, Mark Blaxter, and members of the Data Intensive Biology lab at UC Davis for discussion and feedback.
A. Appendix Approximation guarantee
Let us introduce some notation for the analysis of Algorithm 1. We first partition the vertices of D according to whether they were added in line 10 (denoted by D1) or in line 15 (denoted by D2). Let v1, …, vn be the vertex-order in which they are iterated over in the loop starting at line 6. We will use the notation 
, and 
to represent the states of the respective sets and data structures during the ith iteration of said loop. Let τ := domThreshold(r) be the chosen threshold (we discuss a good value for τ on cDBGs below).
After the for-loop at line 8 has finished,
Proof. The statement trivially holds while 
, so assume otherwise. Let uh ∈ Di be the vertex closest to vi and let h < i of type 2 and we conclude that be the iteration in which uh was added to D (either in line 10 or line 15 of that iteration).
If d := distG(vi, uh) > r, then di[vi] has not been changed yet and is still set to ∞. Otherwise, consider the three possible scenarios promised by the distance-property of dtf-augmentations:
Case 1:
. Then ω(viuh) = d and in iteration h the value of dh[vi] is set to the correct value d in the loop at line 8. By assumption this distance remains minimal until iteration i and hence di[vi] = dh[vi] = d.Case 2:
. Then ω(viuh) = d and in iteration i the value of di [vi] is set to the correct value d in the loop at line 8.Case 3:
with ω(xuh) + ω(xvi) = d. During iteration h the value of dh[x] is set to ω(xuh) in the loop at line 8 and subsequently retrieved in iteration i when di[vi] is set to
We conclude that after the execution of the loop at line 8. di[vi] is set to ∞ if vi is not dominated by Di and is otherwise set to distG(vi, Di), as claimed.
As an immediate consequence, we see the conditional statement at the end of the loop at line 8 accurately determines whether vi is dominated by Di or not. Accordingly, lines 15 of the loop are only executed if vi is not dominated by Di. Another consequence is that all vertices in D1 have large distance to each other:
The set D1 is (r + 1)-scattered in G.
We need one more important property of the algorithm in order to derive the approximation factor.
For every w ∈ G it holds that 
.
Proof. Assume towards a contradiction that τ + 2 such vertices 
exist in 
. Since every such vertex vi, ∈ {i1, … iτ+2}, was added to D in part (2), part (3) of the algorithm was executed during iteration i as well. Thus c[w] was increased in each iteration i and during iteration iτ+1 we have that c[w] ≥ τ + 1 after the increment of c[w]. There-fore part (4) must have been executed for w, including w into D. Hence w ∈ Ds for s > iτ+1 and in particular 
. But then 
was dominated by w at the beginning of iteration iτ+2 since we assumed that 
, thus 
would not have been included in D at step (2). This contradicts our assumption of 
so the claim must hold.
There exists a subset A ⊆ D1 such that A is (2r + 1)-scattered in G and
Proof. We construct an auxiliary graph H with vertices D1 by adding arcs vivj for vi, vj ∈ D1 with i < j whenever distG(vi, vj) ≤ 2r. Let 
be a 2rth dtf-augmentation of G and let us create a digraph 
by orienting every edge uv ∈ H as follows:
If of
, then orient uv in according to the corresponding arc in (if both arcs exists choose an arbitrary orientation),otherwise there exists
with ω2r(u) + ω2r(v) = distG(u, v) ≤ 2r. Orient the edge uv towards that vertex x ∈ {u, v} for which ω2r(x) is larger.
We now argue that 
is small. Consider any vertex 
. Every in-arc 
either is of type 1, of which we have at most 
, or of type 2. Consider a group of in-arcs uiv, 1 ≤ i ≤ l of type 2 that are all present because of a common vertex w. Since 
, we have at most 
such groups. By construction, ω2r(wui) ≤ ω2r(wv) and since both weights sum to less than 2r, this means that ω2r(wui) ≤ r. Lemma 2 now tells us that l ≤ τ + 1. Therefore v has at most 
in-arcs of type 2 and we conclude that
This finally implies that H is 
-degenerate and therefore contains an independent set A ⊆ V (H) of size at least 
. Taken together with the fact that 
) (every vertex added to D1 will cause at most 
many vertices to be added to D2 in the loop at line 11 and D = D1 ∪ D2), we find that
By construction of H we conclude that A is (2r + 1)-scattered in G of the claimed sized.
Since a (2r + 1)-scattered set provides a lower bound for an r-dominating set, we conclude that Algorithm 1 computes a 
-approximation of an optimal r-dominating set, H h which is a constant-factor approximation in graphs of bounded expansion.
In practice one could, depending on the value of 
and 
, compute the optimal value for τ. However, this would necessitate the computation of 2r augmentation, the expensive step we want to avoid. Alternatively, we can choose a ‘good enough’ value for τ that still guarantees a constant-factor approximation while being easy to determine in practice. In the context of cDBGs, we found that τ := (2r)2 yields reliably good results.
Computational Runtimes
See “Benchmarking” in Materials and Methods for benchmarking methods.
The podarV data set was retrieved from the NCBI SRA using accession SRR606249. The full build and indexing of the 103 million error-trimmed reads (10.3 Gbp in total) took approximately 23 minutes and required 12.8 GB of RAM. Loading the indices for search required 4.3 GB of RAM and a search with a 3 Mbp genome took approximately 32 seconds.
The HuSB1 data set was retrieved from the NCBI SRA using accession SRR1976948. The full build and indexing of the 34 million error-trimmed reads (8.5 Gbp in total) required approximately 217 minutes and required 24.4 GB of RAM. Loading the indices for search required 18 GB of RAM and a search with a 3 Mbp genome took approximately 80 seconds.
For data set complexity (number of k-mers, number of cDBG nodes) please see Table 1.
Query genome accession numbers for Proteiniclasticum search. See Table 3.
Amino Acid Identity results for Proteiniclasticum. See Table 4.
Genome bin completeness improvements for HuSB1. See Table 5.
K-mer inclusion of reads by MEGAHIT assemblies. See Table 7.
gyrA alignment. See Figure 5
gyrA by neighborhood. See Table 6.
Footnotes
DM, MPO, FR, and BDS designed and implemented algorithms; CTB, DM, MPO, and FR devel-oped software; CTB and TR conducted biological data analysis; CTB and BDS supervised work. All authors interpreted results, wrote text, created figures, and approved the submitted paper.
↵∗ an invertible function that defines both an index and the corresponding inverted index
↵† That is, graphs about which we make no structural assumptions.
References
