Abstract
The use of null models has been a cornerstone to assess the emergence of many network properties at different levels of organization (micro-, meso- and macroscale). Notwithstanding, the debate around which is the most appropriate randomization procedure for a given problem is far from being over. Within the ecological community, for example, the discussion around whether nestedness is –or is not– a frequent pattern in natural systems, and under which assumptions, remains open. For this particular problem, efforts have been devoted to exploring to what extent current models are vulnerable to statistical errors, or to introduce new models that employ different randomization procedures. However, few or no attention has been devoted to the performance of those null models against other architectures. Here, we show that assessing alternative structures under a single null model may produce ambiguous results, which difficult the comparison regarding the joint emergence of different arrangements within a single network. To this aim, we analyze the statistical significance –in terms of z-scores– of nestedness, modularity, and in-block nestedness scores, employing five different null models on a benchmark of ∼ 2.5 × 104 synthetic bipartite networks with prescribed levels of the mentioned patterns. We show that some null models systematically over- or underestimate the presence of one or another structural pattern. In light of these ambiguities, we introduce an alternative model (termed Corrected Probabilistic model) that reduces the observed biases towards under- and overestimation, and highlight the need for the development of new frameworks that take into account those biases.
1 INTRODUCTION
Almost twenty years ago, research in community ecology was deeply challenged by the evidence that biota interactions, rather than being randomly assembled, manifest clear architectural patterns. In 2003, the pioneer paper of Bascompte et al.1 showed that a significant amount of mutualistic plant-animal networks are organized according to well-defined nested arrangements. Along the same line, Olesen et al.2 unveiled the prevalence of modular configuration among pollination communities with a high (> 150) number of species. In the following years, both nestedness and modularity were recognized to strongly affect some beneficial properties –the former was linked to the promotion of diversity3 and the maximization of abundances4, while the latter was associated to the enhancing of stability5,6 – leading to a great consensus about the crucial role they play in the dynamics of natural systems7,8,9,10,11,12,13,14,15.
The importance of interaction structure in the behavior of ecological communities fueled, consequently, a long series of studies aimed to assess the statistical significance of the emergent patterns. Herein, null models stand up as one of the ubiquitous tools to address this class of problems, in particular referring to nestedness16,17. Paraphrasing Gotelli and Graves18, a null model is a pattern-generating model, designed with respect to some ecological or evolutionary process of interest, which allows for randomization test of ecological data, for instance, a species’ network (as in our case). Certain elements of the network are held constant (constraints), and others are permitted to vary stochastically to create new assemblages. In this way, a null model generates a set of objects that match the original one only in the selected constraints and may be employed as a term of comparison to infer wether the network in question displays nontrivial features, such as the architectural pattern, as a consequence of those constraints, namely as a result of a null hypothesis embodying a particular ecological mechanism.
The selection of these constraints has important effects, and yields many controversial aspects concerning the use of null models. On the one hand, dealing with too loose constraints could misguide us to think that a structural pattern really exists (type I error)19,20. On the other hand, imposing too strict constraints will increase the chance of falsely rejecting the statistical significance of the considered pattern (type II error)16,19,21. For instance, if the null hypothesis refers to species interactions, the first point of view could translate into an ensemble of networks in which degrees are proportionally constrained, matching only on average the marginal total of the original matrix1. The second approach, instead, might fix node degrees by swapping pairs of interactions such that both row and column totals are preserved22. Intuitively, the former algorithms avail themselves of ample room to generate random structures, thus often inducing a type I error. While the latter will tend to generate null matrices that closely resemble the original network and so, in order to get a truly representative ensemble, a very large number of null matrices needs to be produced, something which is not possible in some scenarios and could lead to the generation of a biased sample.
Since the 70s, scholars have struggled to overcome the aforementioned limitations23,24. Later, a myriad of papers have addressed the ambiguities in pattern assessment, both improving existing algorithms, and introducing new ones25,26,27,28,29. Many of these, however, present some limitations that need to be further explored or extended, both on the conceptual and technical side. First, many of the works about null models take into account a reduced number of real world networks30,31,32. Such procedure implies, to some extent, circularity. For example, testing for nestedness via randomization on an empirical network for which we would like to know whether it presents nestedness or not. Some authors have considered large synthetic networks ensembles, e.g., Strona et al.21 employed an ensemble of 103 synthetic networks, but it only considered square networks with up to 50 rows and columns. In general, there is still a lack of a proper synthetic benchmark aimed to build prescribed architectural patterns. The assessment of nestedness (or any other pattern, for that matter), indeed, is often performed over random matrices, and the few papers dealing with effective nested matrices generate them in a contrived way. One exception, at the best of our knowledge, is provided by the work of Beckett et al.33.
Moreover, another pressing issue relates to the structural arrangements under consideration. So far, the debate has focused almost exclusively on nestedness, while modularity, for example, has been mostly left aside with few exceptions34,35. Considering the ecological implications of exhibiting one or another pattern, null models should be balanced enough to discriminate correctly among the different possibilities. Furthermore, note that, in addition to nestedness and modularity, there are other architectural patterns worth being considered, e.g. in-block nestedness, which describes networks compound by weakly interlinked blocks with internal nested character36,37,38. This has recently attracted a lot of attention to be predominant in real mutualistic communities39 and to emerge as a consequence of micro-40,41 and macroscopic42 mechanisms.
In the current manuscript, we first introduce a network generation benchmark which enables the construction of synthetic graphs with varying levels of specified planted structures (from nested, to modular, to in-block nested, to completely random patterns). On top of such benchmark, we analyze the statistical significance –in terms of z-scores– of three rich sets of bipartite matrices with nested, modular and in-block nested configuration. We consider five different null models and detect the presence of ambiguities in assessing the various network descriptors, corresponding to over- (or under-) estimations of the given arrangements. To overcome the observed inconsistencies, we introduce an alternative approach, the Corrected Probabilistic model, that helps to reduce the observed under- and overestimation, and highlights the need for the development of new frameworks that take into account those biases. These contributions are wrapped into a single compact framework, consisting of software repositories that enable the generation of the synthetic networks with the specified planted structures, the calculation of nestedness under different formulations, optimization algorithms for the maximization of modularity and in-block nestedness, and the construction of the reviewed null models.
The article is organized as follows: Section 2.1 provides the definitions for nestedness, modularity, and in-block nestedness. Section 2.2 provides the details of the suitable synthetic network generation benchmark, that is employed in this work. Section 2.3 offers details regarding the empirical data we analyzed. Section 2.4 introduces the five null models that will be studied in this work. Section 3 presents the results of our study in both the synthetic and empirical benchmarks. Finally, the discussion in Section 4 summarizes all the previous and highlights the main open challenges for further developments.
2 MATERIALS AND METHODS
2.1 Relevant structural patterns in Ecological Networks
We present here the three structural arrangements considered in the manuscript: nestedness at the macroscale, modularity and in-block nestedness at the mesoscale. We provide their definition and discuss the related properties, paying particular attention to the measures aimed to quantify them –implemented and available at https://github.com/COSIN3-UOC/N-Q-IBN.
2.1.1 Nestedness
Nestedness, originally conceived in the field of biogeography43,44,45, constitutes a frequent architectural pattern both inside1,13 and outside Ecology46,47,48,49,15. A perfect nested configuration describes a hierarchical organization where the set of neighbors of a node is a subset of the neighborhoods of larger degree nodes. That is, given a graph G and let V be the set of its vertexes, a perfectly nested configuration exists if where kn and Γn indicate the degree of a node n and its neighborhood, respectively1. In terms of adjacency matrixes, this definition translates into the characteristic triangular shape, as that shown in Fig. 1 a.
Even though perfectly nested patterns are extremely rare in real complex networks, many of these exhibit a highly nested haracter. Quantifying nestedness has rendered itself a large amount of definitions50. From an algebraic perspective, the spectral properties of perfectly nested graphs have been studied by mathematicians51,52 facilitating the proposal of the detection method of Staniczenko et al.11, which quantifies nestedness with respect to the maximum eigenvalue of binary and weighted graphs’ adjacency matrices. Ecologists have also dedicated many efforts to evaluate nestedness in real systems. In first place, there are measures based on counting misplaced relations to complete a perfect upper triangular nested structure in the biadjacency matrix, such as the Nested Temperature (NT) measure, introduced by Atmar and Patterson53. To overcome some pitfalls around placement-based measures, Almeida-Neto et al.54 developed overlap metrics, like the Node Overlap and Decreasing Fill (NODF), which considers the amount of common neighbours between every two pairs of nodes in matrix A, alongside with its weighted version55,56,57.
In this work, we stick to a NODF-like descriptor, which for the case of a bipartite network writes as where Nr and Nc represent the number of adjacency matrix rows and columns, respectively; Oij= Σkaikajk, and similarly, Olm,account for the amount of commonly shared neighbours –a.k.a overlap– between pair of species (row or column node pairs); ki = Σk aik corresponds to the degree of node i and quantifies the number of species with whom i is related to; and Θ (·) is the Heaviside step function, that ensures that ij has a positive contribution when ki ≥ kj. Additionally, Oij is conveniently corrected by a null model that discounts the expected change of each species have to share a neighbor38, namely, the expected overlap ⟨Oij⟩ Assuming no correlation between neighboring species of i and j the probability .of sharing a particular neighbour only depends on the degree of i and j and on size of the network, . Hence, the average overlap is . The same argument applies for the other set of species (ki km) /Nr. Note that Eq. 2 follows closely (and often reduces exactly) to NODF (Almeida-Neto et al.) if the term ⟨Oij⟩ is suppressed. We chose to employ this metric to assess nestedness, instead of the original NODF since it also incorporates a term that discounts the expected overlap among pairs of nodes. Notwithstanding, as we will show in Sec.3, our results for the statistical significance of nested networks over the different null models are robust against other existing metrics, i.e. spectral radius, and the original NODF.
2.1.2 Modularity
Modular networks are those built up from weakly interlinked groups (also called communities, modules, compartments, clusters) with high internal connectivity. Fig. 1 b depicts the adjacency matrix of a perfectly modular network, i.e. with no intergroup links. Modular structure is a rather ubiquitous mesoscale architecture58,59,60,61 and appears in several fields, as diverse as biology59 and cognitive science62.
The problem of identifying community structure constitutes itself a sub-area of network science. Over the last decades, scholars have developed a rich collection of algorithms and methodologies to infer these communities from relational data61,63. Probably, the most popular method in network science, and in particular in Ecology, is through the maximization of a fitness function called modularity Q ∈ [0, 1)64. We will also employ this function in the present work. Formally, the modularity measure can be written as where A is the biadjacency matrix, L is the total number of links in the network, is a vector representing the membership ariable of species with entries αi, δ (αi, αj) corresponds to the Kronecker delta, equal to one if nodes i and j belong to the same community (zero otherwise) and kikj /L is a regularisation term to discount the expectation that two nodes are connected by chance considering the degree they have.
2.1.3 In-block nestedness
Over the last decade, nestedness and modularity have concentrated most of the attention in community ecology, as they seem to have deep implications on the dynamics and other ecological properties. Nestedness, indeed, promotes the diversity of species3, but restrains instead the local asymptotic stability9,11, a quality that is maximized by modular organization10,6. This suggests that natural systems, which are often observed to be diverse and stable at the same time, develop jointly nestedness and modularity.
The existence of concurrent nested and modular organizations has been debated in different situations2,49, but empirical34 and later analytical65 evidence proved that macroscale nestedness and modular mesoscale cannot coexist easily65. This apparent incompatibility is due to the fact that the two patterns arise as a consequence of different mechanisms: certain pressures promote the block organization, while others favor the emergence of nested patterns. Actually, the analysis of these mechanisms suggests that the co-occurrence of nestedness and modularity may take place at different scales. Herein, in-block nestedness stands out as a hybrid architecture that results from recasting nestedness at the mesoscale, i.e., within modules37,38.
In-block nestedness depicts a network configuration where weakly connected blocks exhibit an internal nested assembly (see Fig. 1 c), and was first introduced by Lewinson et al.36. This has been found to unfold from an abundance-maximization process4 on top of a niche-structured population40,41, providing a bottom-up perspective on nestedness-modularity, able to trade off stability and diversity in mutualistic communities39. Significantly, in-block nestedness has been shown to be the predominant pattern in a significant amount of real plant-animal communities39, as well as outside of ecology66.
The original approaches37,67,36 to detect nested compartments started first optimizing modularity, then subsequently computing the level of nestedness exclusively for nodes within the detected communities. Although this sequential procedure delivers good results in many situations (since, often, detected modules gather nodes with degree heterogeneity), a specialized fitness function is required for the general case. Thus, as in the case of modularity, in-block nestedness detection is a hard computational problem where the use of heuristic algorithms is mandatory. Using the formulation developed in38, the degree of in-block nestedness of a network ℐ can be computed as where Ci and Cl are the size of the communities where nodes i and l belongs to. Worth highlighting this hybrid structure reframes nestedness, originally a macroscale feature, to the mesoscopic level. In this sense, a perfectly nested structure corresponds to an in-block nested structure with a single community, i.e. Eq. 4 reduces exactly to Eq. 2 when B = 1.
2.2 Synthetic benchmark
We carry out the analysis of the robustness of different types of null models over three sets of synthetic networks with the three patterns of interest. So far, many of the works concerning the study of structured interactions in Ecology proceed by quantifying the degree of a given pattern over networks with a random organization, or nested networks constructed in contrived way. The only exception, to the best of our knowledge, is constituted by Beckett and Williams33, who construct synthetic perfect nested matrices with varying levels of connectance, by employing the unit ball equation, and then iteratively perform rewires to degrade the ideal structure, introducing noise. Thus, one of the main contributions of the current manuscript is the generation of networks with a well-defined structure, as a variation of the model introduced in38, and available at https://github.com/COSIN3-UOC/network-generator. In our framework, the different architectural configurations are controlled by four tunable parameters: the number of communities B ∈ [1, ∞], the inter-community noise μ ∈ [0, 1], the noise outside the perfect nested structure p ∈ [0, 1], and the shape parameter controlling the slimness of the nested profile ξ ∈ [1, ∞].
The construction of an N×N adjacency matrix (the extension to rectangular matrixes is straightforward) with a given structure arts off by fixing the number of blocks. Particularly, we build [B] blocks of size [N/B] and another with the remaining N− [N/B] nodes, where [.] represents the integer part function. The intra-block structure is obtained by first generating a perfect nested configuration, which shape depends on the parameter ξ; as inspired by the unit ball equation. Specifically, a block with NB nodes is constructed by tessellating the [0, 1] × [0, 1] space into squares and adding a link into each entry whose center lies above the curve in Eq. 5. The resulting perfect nested blocks are rarely found in real systems, thus we introduce a noise relying on a dual-step procedure: a fraction p of links lying above the curve in Eq 5 are randomly selected and then distributed across the empty entries of the block. Similarly, we reproduce inter-block perturbations with an additional noise procedure ruled by the parameter μ: for each block, each link is removed with probability pi = μ(B − 1)B and then distributed at random to connect a node of the original block to another of a different one.
Figure 2 shows several synthetic networks related to different values of the model parameters. The first row refers to nested matrices, which can be obtained by setting B = 1 and μ = 0, i.e. considering only one block, and are fully characterized by the choice of ξ and p. From left to right, the former parameter takes increasing values and the resulting nested shape becomes more and more slight. The value of p, instead, decreases from left to right. The situation in which p = 1 yields to a Erdös-Rényi graph. The middle row presents a few examples of modular matrices, exhibiting a different number of blocks B. These are obtained by setting p = 1 in order to get rid of any intra-block structure, while ξ is fixed according the wished edge density. The parameter μ allows to interpolate between a purely modular network (μ = 0), i.e. without inter-blocks links, and an Erdös-Rény one (μ = 1). Finally, the third row portrays in-block nested matrices associated to several parameter choices, paying particular attention to the role of p and μ. In the limit in which p = μ = 0 we have a purely in-block nested configuration, while for p = μ = 1 an Erdös-Rény arises.
The generative process described above may be implemented in terms of edge probabilities, simplifying the process of algorithmic design. The probability of having a link between nodes i and j within a block writes as The first term within square brackets, (1 − p), embodies the probability of not altering a link, while the second, pp r, provides that of recovering a link after removal in the random dispersion of removed links. The Heaviside function restricts them to the noiseless block portion. The third term, pr = pE(N − E + pE) (E edge number) gives the probability of selecting links in the random distribution of removed ones. The probability of an inter-block link is where the numerator takes into account the amount of removed links, while the denominator regards the possible matrix entries where those links can be placed. For further details regarding the network generation model we refer the reader to38.
2.3 Empirical datasets
To explore our hypothesis in real scenarios, we perform experiments on 192 bipartite ecological networks from two papers and two online databases: 35 trophic networks from68,69, 141 mutualistic networks from the Web of Life dataset70, and 16 (3 mutualistic and 13 trophic) networks from the Interaction Web Database http://www.ecologia.ib.usp.br/iwdb hosted by the National Center for Ecological Analysis and Synthesis (University of California). All the networks in our dataset were treated as binary (unweighted links) and we have only kept in our study networks with a minimum of 20 nodes, to ensure that the identified architectural patterns are well defined. In parallel, to reduce computational costs we restricted our analysis to networks below 300 nodes. We highlight, however, that the number of nodes in most of the networks lie between this range, values above or below can be considered outliers, see Section 3.3 below.
2.4 Null Models
The appropriate selection of a null model has been a center of debate in the ecology literature for many decades. The first attempts to estimate the significance of structural patterns in ecological networks date back to the 70s23,24. In this context, scholars introduced approaches relying on the generation of random networks, either by keeping fixed the number of rows (or columns) nodes degrees and allowing the number of columns (or rows) nodes degrees to change23 ; or by generating networks that exactly preserved degrees of both guilds24. Afterward, models that preserves, on average, nodes degrees in both guilds were also introduced71,1. More recently, models based on constructing maximum-entropy ensembles of networks with an expected degree sequence were also exploited to assess statistical significance48,72.
In Gotelli et al.22, the authors considered three basic constraints that can be imposed independently for rows or columns nodes: 1) where the individual nodes’ degrees are exactly preserved, namely fixed (F), 2) where the nodes’ degrees are preserved on average, i.e., the probability of interaction is proportional (P) to the degree, and 3) where all the pairwise interactions between nodes are equally probable (E). Resulting in a total of 32 = 9 possible null models, each one having multiple possible implementations.
Below, we provide details for these three basic classes of null models and describe some of the implementations for each case, with special attention to those that were employed in the present work. It worth highlighting that in all of our implementations, the corresponding constraints were applied for both rows and columns nodes. Therefore, we will talk about EE, FF, and PP models, according the nomenclature employed in16.
2.4.1 Equiprobable-Column or Equiprobable-Row (EE) model
The least constrained scheme reviewed here, EE is a probabilistic binary null model that assumes that all the interactions between pairs of nodes –row-to-column links– are equally probable, and that such connection is given with probability , where L is the total number of connections in the original matrix. This model generates Erdös-Rényi networks with nodes exhibiting a varying number of connections around a well-defined average degree ⟨k⟩, preserving L on average. It is known for its tendency to falsely detect significant nested patterns, i.e., it is prone to type I statistical errors19,73,20.
2.4.2 Fixed-Column Fixed-Row (FF) models
In the other extreme of the spectrum74, the FF model enforces row-nodes’ and column-nodes’ degrees to be strictly preserved. It is thus the most restrictive null model. The usage of this type of model to assess the probability that an observed pattern could be explained by chance alone has been applied since the 70s24. A typical way for generating random matrices under this constraint is through the application of some form of swap algorithm, performing exchanges between pairs of interacting nodes without altering the network’s degree sequence22.
Noteworthy, generating random matrices by exactly preserving the degrees is far from being a trivial task. Swap methods tend to generate repeated matrices that closely resemble the original network. Thus, in order to produce a truly representative set of matrices within the universe of possible matrix configurations, a very large number of matrices need be generated75, which might not be possible in some scenarios. As a consequence, this model is susceptible to type II errors, particularly, when it comes to the assessment of nested patterns16,73. On the other hand, it also tends to generate matrices with more checkerboard units2 than expected by chance25, producing a biased sample. To solve the latter downside, scholars have introduced variations of the swap methods, and alternative methods that are able to generate a uniform sample of null matrices with fixed row and column degrees25,26,27,28,29.
In this work, we employ the Curveball algorithm, introduced in26 as an FF model, as a fast and efficient alternative to generate random matrices with fixed nodes’ degrees guaranteeing a uniform sample76. The algorithm was introduced as a computationally cheap method to perform permutations on binary matrices, producing uniformly distributed random matrices with fixed degrees. The functioning of the algorithm goes as follows: starting from a binary bipartite matrix –e.g., representing species per site or plant-pollinator matrices, in the ecological context–, for each row (column) node, a list is created containing all the column (row) indices with nonzero elements. Then, the algorithm iteratively chooses at random two row lists (columns), identifies the column (row) indices that are present in list, but not in the other, and vice-versa, and performs random “trades” of the column (row) indices. The number of trades for each pair of lists will go from zero to the size of the smallest list of column indices. In this work, we resort to the Python implementation of the algorithm that was provided by the authors in26.
2.4.3 Row-Proportional Column-Proportional (PP) models
Somewhere between the extremes –and limitations– described above74, an intermediate alternative is to generate models that preserve degrees on average, mitigating then, the proneness to type II error that is present in the FF models, and the proneness to type I error present in the EE model. For this type of model, the probability of interaction of a pair of nodes is assigned to be proportional to the nodes’ degrees. From this general principle, there exist different flavors along the loose-strict continuum. Below we introduce three variants employed here.
Bascompte’s probabilistic model
One of the most popular probabilistic models for the assessment of statistically significant structural patterns (particularly nested ones) was introduced by Bascompte et al. in 20031. In this model, the probability of an interaction by a pair of nodes is proportional to the nodes’ degrees of the original matrix as: where ki is the degree of the i−th row, kj is the degree of the j−th column and Nr and Nc are the respective number of rows and columns nodes.
Maximum-entropy exponential random graphs
Introduced in the last decade77, this is a statistical physics framework that facilitates the analytic computation of the expected properties of an ensemble of networks. Under this formulation, the expected values and standard deviation of the network properties are calculated by finding a probability distribution over an ensemble of networks –which, on average preserve the network’s degree sequence–, that maximizes the entropy of the network. This approach can be applied to both uni- and bipartite networks, and has been recently exploited to assess the significance of nestedness structural arrangements in economic78, and ecological72 networks.
With this approach, one aims to find a probability distribution P (G) over the ensemble of graphs G, with G* as the graph corresponding to the real network, that maximizes the Shannon-Gibbs entropy S = ΣG− P (G) ln P (G), and that keeps the average nodes’ degree fixed. This maximization has a solution given by the canonical distribution where Z is the partition function, H(G, θ) = θ C(G) is the graph Hamiltonian, and θ is a vector of Lagrange multipliers resulting from the maximization of the Shannon-Gibbs entropy, under the chosen constraints C, e.g., the graph’s average degree. The next step consists on the calculation of the exact values of the Lagrange multipliers. Following the approach presented in77, these multipliers are determined by imposing that the chosen constraints of the network are found in the ensemble with maximum probability. This is achieved by rewriting the log-likelihood of observing the real network as L(θ) = −H(G *, θ) − ln Z(θ) and maximizing this quantity in order to find the optimal variables θ* that define the ensemble. Once the parameters θ* are found, it is possible to build the matrix containing the average probability of interaction corresponding to our empirical network ⟨G*⟩. Finally, it is possible to derive an analytical expression of the first and second moments of the desired network property. In this work, instead of analytically deriving such statistical properties, we will generate a random ensemble directly from the matrix of average probabilities of interactions.
Corrected Probabilistic Model
Lastly, we now introduce an alternative PP variant that aims to mitigate the possible weaknesses of the models described above. The corrected probabilistic model, in the same spirit as Bascompte’s one, sets the probability of interaction by a pair of nodes to be proportional to the nodes’ degrees of the observed matrix, but includes two corrections terms. These terms help to reduce the discrepancies observed in the probabilistic model at preserving the nodes’ degrees –on average–, while avoiding the production of network instances that are too similar to the original matrix, as in the Maximum-entropy model. In this work, we will test in depth the robustness of this model against type I and type II errors. Overall, in this model, the threshold ruling the existence of a link is given by: where ki is the degree of the i−th row, kj is the degree of the j−th column, ⟨kr⟩ and ⟨kc⟩ are the row and column average degrees, and Nr and Nc are the respective number of rows and columns nodes. The quantity is the sum of the term in equation (8) and a correction term taking into account the probability that links can appear in a row (column) also by sampling over the columns (rows). Specifically, the degree of a given row node is obtained by sampling over this axis (something which is correctly described by Equation (8)), or by sampling over columns: this information is embodied in the term within the square brackets proportional to kj /Nr In this latter case, one has to taken into account the probability that a links already exists because of the first sampling process, and this justifies the term proportional to ⟨k⟩ Of course, the final expression is symmetrical for row-column permutation.
3 RESULTS
In the following, we present the results concerning the test of the five null model classes against the three architectural patterns of interest. For a systematic exploration, Sections 3.1 and 3.2 exploit the synthetic generative model introduced in Sec. 2.2. With it, the construction of nested matrices is performed by keeping fixed the number of blocks (B = 1) and varying values of ξ and p, as in the top row of Fig. 2. For the set of purely modular networks, we fixed ξ = 1.5 and p = 1 and smoothly modified B and μ to cover a wide range of modularity values, e.g. second row of Fig. 2. Finally, the generation of in-block nested networks was carried out setting B = 4, ξ = 1.5 and varying levels of p and μ, e.g. third row of Fig. 2. At the end of the process, we generated around 9000 networks for each one of the three patterns of interest, i.e., ∼ 2.5 × 104 in total. For each generated network, we created 100 random surrogates with each null model. The number of rows and columns were varied within the range [30, 60]. Finally, in Section 3.3, we perform the statistical significance test on a large collection of empirical networks.
3.1 Resemblance of original vs. randomized networks
To understand the implications of each null model approach, we analyze in detail how a synthetic network resembles those of the corresponding null model ensembles. To do so, we first look into the node degrees of a particular matrix, and then perform a comparison over the space of parameters introduced in Sec. 2.2 by recalling the Jaccard distance.
Figure 4 analyzes in detail the properties of the ensemble obtained with the three PP models for a given network with planted nested structure (large matrix on the left). Examples for other types of matrices are available in Fig. S1 of the supplementary material. The top row represents the resulting matrices of interaction probability as a heatmap. Even visually, it is clear how permissive –or strict– each model is with respect to the original network. More in detail, the bottom row presents a scatter plot of the degree sequence of the original matrix against those of the resulting null matrices sets (in average). The error bars correspond to one standard deviation above and below the obtained average degrees. Herein, we observe that the probabilistic model appears to be too permissive, failing to preserve –even on average– the nodes’ original degree. The max-entropy scheme almost exactly preserves the average degree sequences, hence, it might solve the proneness to type I errors that are present in the probabilistic model. The corrected probabilistic model presents a good agreement, with slight deviations for extreme (smallest and largest) degrees. Noteworthy, EE and FF models are not included in this figure –their scatter plots would render a trivial picture.
For a wider look at the ensemble of null matrices, we now consider the parameter space of the network benchmark. Concretely, e explore to what extent the generated null matrices resemble the original synthetic ones for the whole range of parameters of he graph model. To quantify such resemblance, we employ the Jaccard distance Jd, which accounts for the level of dissimilarity between two binary arrays –the original and its random counterparts, in this case. It should be noted that, in the case of binary arrays, Jd = 0 indicates that the matrices are identical.
Top row in Fig. 5 shows the average Jaccard distance ⟨Jd⟩ over the nested ξ −p parameter space for each null model scheme. Similar results for the modular and in-block nested matrices are shown in Fig. S2 of the supplementary material. As expected, the results for the EE and the probabilistic models show high values of the ⟨Jd⟩ over the entire ξ−p parameter space, confirming that the randomized matrices generated by these models are substantially different from the original one. Turning to the FF and Max-entropy models, we observe a wide region of the ξ− p parameter space in which ⟨Jd⟩ drops to intermediate and low values –specially for networks with small noise p, regardless of the slimness of the nested structure ξ These results for FF and Max-entropy confirm that, at least for nested networks, the resulting null matrices are very similar to the original one, since the models only allow for a very little deviation on the degree sequences –none in the FF model. Hence, these models might easily fall into type II errors. The corrected probabilistic model, instead, provides a good trade-off, offering a better fit of the degree sequences, while generating null matrices less similar to the original one, as evinced by the overall intermediate values of the ⟨Jd⟩ over the nested ξ− p parameter space.
From another perspective, the bottom panel of Fig. 5 shows the percentage of matrices in the null model ensembles for which Jd ≥ 0.4 in the nested ξ − p parameter space. Clearly, there are relevant regions for which the FF and the Max-entropy models cannot generate true variants of the original matrix, even with such moderate threshold. Of course, when imposing a more rigid threshold this region becomes wider. In the other extreme, the EE and Probabilistic schemes prove to have ample room to generate largely different networks, no matter the parameter values under study. Somewhere in between, the Corrected Probabilistic model is flexible enough to produce different networks, except in some extreme cases (very sparse networks with low p). These insights are all in good agreement with a recent work developed by Strona et al.74, in which they proved that slightly relaxing the restrictiveness in the preservation of the rows and column totals –quasi-FF matrices– can reduce the tendency to underestimate significant nestedness.
3.2 Statistical significance of structural patterns
We now analyze how consistent the five null models described above are in assessing the statistical significance of nested, modular and in-block nested patterns, for each set of the generated synthetic matrices (192 empirical ecological networks are analyzed in the next section). In each case, we assessed the significance of the three patterns of interest. The results of the analysis are summarized in Table 1, which accounts the % of networks with z ≥2, i.e., an associated p-value around 0.025. For nested networks, we also explore the behavior of the null models under other nestedness metrics: NODF54 and the spectral radius 51,11 normalized with respect to the square root of the number of links in the matrices, , as in32.
For networks with planted nested structure (B = 1, p E [0, 0.5]), the EE and Probabilistic models overestimate systematically he statistical significance for nestedness –no matter the metric. In the other extreme, both the FF and Max-entropy restrictive schemes deliver an underestimation of the amount of truly nested networks. Thus, confirming their proneness to type I and II errors, respectively. Only the corrected probabilistic model renders a balanced assessment, consistent with the parameters used in the generation of this set of nested synthetic matrices. The assessment of the significance for in-block nestedness is generally consistent with the one observed for nestedness, as ℐ reduces to ℐ when B = 1 (recall Eqs. 2 and 4).
More intriguingly, the FF and Max-entropy models identify a large fraction of nested-by-construction networks as being significantly modular (24.2% and 46.37%, respectively). To some extent, this could be explained by the densest nested matrices in the collection38, i.e., low i;, but by no means the collection should be regarded, overall, to be more modular than nested.
Turning to modular-by-construction networks (B > 1, p = 1, μ ∈ [0, 0.5]), we observe that –unlike the previous case– all models render intuitively correct results: these networks are unanimously assessed as non-nested, mostly assessed as modular (except for those with high μ), and around half of the collection is assessed as being in-block nested, which makes sense considering configurations with low p (which are modular and in-block nested).
Finally, networks with planted in-block nestedness (B = 4, p ∈ [0, 0.5], μ ∈ [0, 0.5]) again arise inconsistencies among the randomization procedures. Some of them (EE, FF and Probabilistic) incorrectly assess a large amount of networks as being nested –this is particularly marked in the case of EE.
Considering the results in Table 1, it is clear that nested structures are the most confusing given the outcomes of the different randomization schemes. For this reason, we explore these inconsistencies in further detail in Figure 6 regarding only the nested- by-construction synthetic subset. The figure shows the z-score values for nestedness z..N (top) and modularity zQ (bottom) over the ξ− p parameter space of the generated nested matrices, considering the three PP models. Thumbnails of the resulting generated matrices over the parameter space are shown on the left part of the figure as a visual aid.
The results of Fig. 6 for the probabilistic model confirm the tendency for overestimating the statistical significance of nested patterns, i.e., type I error. The model yields values of z𝒩 ≥2 over the whole parameter space, including the regions where the planted nested structure becomes hardly identifiable (high p). The Max-entropy model, conversely, is not able to assess the significance of nestedness, even in regions where the networks have a clearly defined nested pattern, except for those networks with high ξ The corrected probabilistic model is capable of assessing the significance of the nested pattern over a wide region of the parameter space, for intermediate to low values of noise p –as expected–, and different shape values ξ. Note, however, that the higher the value of ξ is, less noise p is necessary for the model to reach z𝒩 ≤2. This is not surprising, since the ξ parameter affects the network connectance, and sparser networks will require a lower level of noise to distort the planted nested structure. For the sake of completeness, Fig. S3 in the supplementary material shows the results for NODF and, showing equivalent results, in spite of the different nature of the two metrics. The assessment of significance for modularity zQ with the probabilistic and corrected probabilistic models shows small regions of zQ ≥2 that corresponds to high connectance networks (low ξ). In these cases, one can assume that networks are dense enough, such that the optimization algorithm employed for modularity is able to find a partition with B > 1, resulting in values of Q larger than expected38. The Max-entropy model, however, assesses significance for modular patterns over a wide region of the ξ − p parameter space –around 46% of the networks in Table 1 –, strengthening our assumptions regarding the presence of biases towards an overestimation of statistical significance of modular patterns.
Moving on, we know discuss the results for the set of synthetic matrices with modular structure. From Table 1, we can observe that the assessment of significance for nestedness across this set of matrices yields coherent results for the five models. We obtained values of z𝒩 ≤2 consistently throughout the whole set. Such result is not surprising, since by construction, all the networks present a clearly modular planted structure, and as mentioned above, we have evidence that these two patterns may not be structurally compatible65. Results for the assessment of modularity and in-block nestedness from Table 2 are complemented with Figure 7. The figure shows the z-score values for modularity zQ (top) and in-block nestedness zℐ (bottom) over the μ − B parameter space of the generated matrices, with the three PP models. Similarly to Fig. 6, examples of the resulting matrices over the parameter space are shown on the left part of the figure. Again, Table 1 reports an unanimous assessment of modularity across the five models. Top panel in Fig.7 for the PP models confirms these results: the models are able to assess the significance of modularity for a wide range of the parameter space. As expected, the assessment is sensitive to level inter-block noise μ. Although, the Max-entropy model is still able to assess the significance for modularity in regions with high μ values, where the planted modular structure is hardly identifiable anymore. Results for the assessment of in-block nestedness also show consistent results across the five models (see Table 1). For around half of the networks in this set, we obtain a significant level of in-block nestedness. Again, the critical ingredient here is the presence of inter-block noise μ. The result is in agreement with Palazzi et al.65, in which it is proved that both Q and ℐ can coexist, as there are no mutually imposed constraints between them.
Finally, we move on to examine the results for the last set of synthetic networks, with planted in-block nested structure. igure 8 complements the results for this set of networks in Table 1, by showing the z-score values for in-block nestedness zℐ (top) and nestedness z𝒩 (bottom). As in the previous cases, the left part of the figure shows examples of the generated matrices with the PP models over the μ − p parameter space. The three models illustrated here show similar (and expected) results for zℐ: in-block nestedness becomes less significant when the structure is distorted within (higher p) or among (higher μ) blocks. Such coherence contrasts with the assessment of significance for 𝒩, which deserve to be examined in more detail. It is known that global nestedness decreases with the number of blocks in a network, regardless of their internal nested structure65. Consequentially, the Max-entropy and Corrected probabilistic models correctly present non-significant z.𝒩 < 2 throughout the parameter plane, and even for low values of noise (either μ or p). Instead, the probabilistic scheme identifies as nested those matrices for which p < 0.2, regardless of μ, a clear overestimation of nested patterns which occurs, even at higher proportions, in the EE and FF models (see Table 1).
3.3 Empirical Networks
Lastly, we also analyze how consistent the five null models are in assessing the statistical significance of nested, modular, and in-block nested patterns over a set of 192 empirical networks. All networks are treated as binary, and that only networks between 20 and 300 nodes are kept (see Section S2 and Figure S4 in the supplementary material for details). For the sake of consistency and completeness, we explore the behavior of the null models under the three nestedness metrics: 𝒩, NODF, and . We report the percentage of networks with z ≥2 (p-value ∼ 0.025) for each one of our five structural metrics in Table 2.
Assessment of nestedness in the empirical data for the EE and Probabilistic models is consistent across the three different metrics, yielding z-score values z 2 for a large proportion of the analyzed networks. The assessment of significance for in- block nestedness renders equivalent results, showing significant zℐ ≥ for a high percentage of the networks, although in a lower proportion with respect to nestedness. In contrast, only a small fraction of the networks is significantly modular (zQ ≥ 2). Results for the strictest schemes –FF and Max-entropy models–, show, in the case of nestedness, a good agreement with the results in the synthetic matrices (and the literature72), yielding non-significant z-scores for a large majority of the networks. Note that, similar to the synthetic case, the normalized spectral radius , with the FF model is able to assess nestedness for a bigger proportionof the networks. Once again, the corrected probabilistic model seems to provide a more balanced assessment of nested patterns over the different metrics. The small variations in the preservation of the node’s degrees in the corrected probabilistic model allow nestedness to reemerge as significant in a small proportion of the networks.
Moving on, results for modularity in the FF and Max-entropy model render inconsistent results, suggesting once again the esence of biases in the Max-entropy model with regard to the assessment of modular patterns. Finally, we observe that a elevant fraction of the networks exhibit significant in-block nestedness structure even with the most restrictive algorithms, in close agreement with recent studies suggesting the presence of in-block nestedness as a frequent pattern in ecological communities42,40,39.
4 DISCUSSION
The overwhelming evidence that real ecological networks are highly structured, rather than randomly assembled, has sparked much research to explain the emergence and dynamical effects of those interaction patterns. In parallel, efforts have been devoted at determining the statistical significance of such patterns. As a result, the use and design of specialized null models to assess the statistical significance of selected architectures has become a widespread approach within the ecological community. However, to this day, there is still no agreement on which is the most suitable procedure to perform this kind of analysis. The discussion regarding the selection on the level of constraints to build the null matrices, the construction of synthetic benchmarks, or the appropriate null model-metric combination remains an open issue among scholars. Furthermore, there is already abundant evidence suggesting that networks often exhibit hybrid structures and/or at different scales of organization, yet the concurrent assessment of the statistical significance of such complex patterns has not been extensively studied.
In this work, we have first addressed the need to build synthetic networks with total control over their dominant structure. Our generative benchmark allows for the construction of networks with clear defined architectural patterns, transitioning from structured (nested, modular, in-block nested) to random networks in a smooth way. Only then we can generate a rich sample to systematically study of the performance of five popular null models.
With over 25000 networks of varying size, connectance and internal structure, we have observed the existence of ambiguities and biases in the concurrent assessment of network patterns, either over- or underestimating their significance. In particular, our use of an extensive ensemble of networks uncovers the weaknesses of both too restrictive (FF, Max-entropy) and too permissive (EE, Probabilistic) randomizing procedures. At the face of such results, we propose a variation of the Probabilistic model, that mitigates some of the limitations exhibited by the current models –in close agreement with recent research, regarding the convenience of quasi-FF approaches74. Last but not least, our results on empirical networks show an overall good agreement with the results obtained on the synthetic matrices. Indeed, we observe again the presence of ambiguities in the simultaneous assessment of the patterns across models, with the corrected probabilistic model standing out as a reliable alternative to assess the significance of highly structured matrices.
We are convinced that our results can orient the future debate regarding the use of null models in bipartite networks within and beyond the ecological community. Providing a compact framework and off-the-shelf implementations (synthetic benchmark, structural descriptors, optimization algorithms, and null models), our aim is double: on the theoretical side, to continue the research on the underlying assumptions and implicit consequences of the different null models. For example, to discern the apparent bias of the Max-entropy model towards underestimating the presence of nested configurations, while overestimating the presence of modular ones. On the practical side, to encourage researchers to shift the focus towards the simultaneous assessment of the statistical significance of multiple structural patterns at multiple scales –especially given the increasing evidence that empirical systems may display structure at several scales of organization, and frequently in the form of compound structures.
CONFLICT OF INTERESTS
The authors declare that there are no conflicts of interest regarding the publication of this paper.
CODE AVAILABILITY
The corresponding software code to (1) generate synthetic networks with nested, modular and in-block nested structures; (2) the implementation of the metrics and optimization algorithms aimed to characterize nestedness, modularity, and in-block nestedness; and (3) the Python implementation of the five null models employed in this paper are available at the Github repository https://github.com/COSIN3-UOC.
ACKNOWLEDGEMENTS
M.J.P., A.S.-R. and J.B-H. acknowledge the support of the Spanish MICINN project PGC2018-096999-A-I00. M.J.P. acknowledges as well the support of a Postdoctoral research grant from the Universitat Oberta de Catalunya (UOC).
Footnotes
1 In the case of bipartite networks, this definition is valid provided one compares nodes belonging to the same class.
2 2 × 2 submatrices containing 1’s in the main diagonal and 0’s outside, or vice versa
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.↵
- 25.↵
- 26.↵
- 27.↵
- 28.↵
- 29.↵
- 30.↵
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.↵
- 38.↵
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵