Abstract
Nonlinearity is a characteristic of complex biological regulatory networks that has implications ranging from therapy to control. To better understand its nature, we analyzed a suite of published Boolean network models, containing a variety of complex nonlinear interactions, using a probabilistic generalization of Boolean logic that George Boole himself had proposed. The continuous-nature of this formulation made the models amenable to Taylor decomposition that revealed their distinct layers of nonlinearity. A comparison of the resulting series of approximations of the models with the corresponding sets of randomized ensembles showed that the biological networks are on average relatively less nonlinear, suggesting that they may have been optimized for linearity by natural selection for the purpose of controllability. A further categorical analysis of the biological models revealed that the nonlinearity of cancer and disease networks could not only be sometimes higher than expected but are also relatively more variable, suggesting that the agents of disease may leverage the heterogeneity of regulatory nonlinearity to their advantage.
1 Introduction
How nonlinear are biological regulatory networks? That is, to what extent do the biochemical components of these networks non-independently interact in influencing downstream processes (Fig 1). Research on this front has hitherto focused on the various manifestations of nonlinearity in the dynamics of biological systems, such as chaos, bifurcation, multistability, synchronization, patterning, dissipation, etc.[1], but a characterization of nonlinearity in the underlying systems that give rise to those phenomena is lacking. A more complete understanding of biological nonlinearity would have theoretical implications ranging from canalization to control [2, 3] and practical implications for biomedical therapy, synthetic biology, etc. [1, 4]. A good example of this concerns the mapping between molecular or genetic information and the resulting system-level anatomical structure and function of an organism. Advances in regenerative medicine and synthetic morphology require rational control of physiological and anatomical outcomes [5], but progress in genetics and molecular biology produce methods and knowledge targeting the lowest-level cellular hardware. There is no one-to-one mapping from genetic information to tissue-and organ-level structure; similarly, ion channels open and close post-translationally, driving physiological dynamics that are not readily inferred from proteomic or transcriptomic data. System-level properties in biology are often highly emergent, with gene-regulatory or bioelectric circuit dynamics connecting initial state information and transition rules to large-scale structure and function. Thus, the difficult inverse problem [6] of inferring outcomes and desirable interventions across scales of biology illustrates some of the fundamental questions about the directness or nonlinearity of encodings of information, as well as the importance of this question for practical advances in biomedicine and bioengineering that exploit the plasticity and robustness of cellular collectives. Many deep questions remain about the potential limitations and best strategies to bridge scales for prediction and control in developmental, evolutionary, and cell biology. To that end, we introduce here a formal characterization of the nonlinearity of models of biological regulatory networks, such as those often used to describe relationships between regulatory genes. Specifically, we consider a class of discrete models of biological regulatory systems called “Boolean models” that are known for their relative simplicity and tractability compared to continuous ordinary differential equation-based (ODE) models [7].
A Boolean network is a discrete network model characterized by the following features. Each node in a Boolean network can only be in one of two states, ON or OFF, which represents the expression or activity of that node. The state of a node depends on the states of other input nodes which are represented as a Boolean rule of these input nodes. Many of the available Boolean network models were created via literature search of the regulatory mechanisms and subsequently validated via experiments [8]. Some of the publicly available models were generated via network inference methods from time course data [2].
Previous studies have found that certain characteristic features of the biological Boolean models, such as the mean in-degree, output bias, sensitivity and canalization, tend to assume an optimal range of values that support optimal function [9, 10]. Here we study a new but generic feature of complex systems, namely, nonlinearity. To characterize the nonlinearity of Boolean networks we formalize an approach to generalizing Boolean logic by casting it as a form of probability, which was originally proposed by George Boole himself [11]. We leverage the continuous nature of these polynomials to decompose a Boolean function using Taylor-series and reveal its distinct layers of nonlinearity (Fig 2). Various other methods, both discrete and continuous, of decomposing Boolean functions exist, such as Reed-Muller, Walsh spectrum, Fourier and discrete Taylor [12, 13, 14]. Our continuous Taylor decomposition method is distinct in that it offers a clear and systematic way to characterize nonlinearity.
By characterizing the nonlinearity of networks in this way, we answer the following questions: 1) how well could biological Boolean models be approximated, that is, faithfully represented with only partial information containing lower levels of nonlinearity relative to that of the original?; 2) is there an optimal level of nonlinearity, characterized by maximum approximability, that these models may have been selected for by evolution?; and 3) do different classes of biological networks show characteristically different optimal levels of nonlinearity? To answer these questions, we first approximate the biological models by systematically composing the various nonlinear layers resulting in a sequence of model-approximations with increasing levels of nonlinearity. We then estimate the accuracy of these approximations by comparing the outputs of their simulations with that of the original unapproximated model. We then construct an appropriate random ensemble for each biological model and compare their mean accuracies for fixed levels of approximation. The main idea is that a biological model that is more approximable than expected for a particular level of nonlinearity would mean that the network may have been optimized for that level nonlinearity. Finally, we classify the biological networks into various categories and compare their approximabilities to identify any category-dependent effects.
Methods
Probabilistic generalization of Boolean logic
Here we provide a continuous-variable formulation of a Boolean function by casting Boolean values as probabilities, thus transforming it into a pseudo-Boolean function [16]. Consider random variables Xi : {0, 1} → [0, 1], i = 1, …, n, with Bernoulli distributions. That is, pi = Pr(Xi = 1) = 1 − Pr(Xi = 0) = 1 − qi, for i = 1, …, n. Let X = X1 × · · · × Xn be the product of random variables and f : X → {0, 1} a Boolean function. Let and . Note that X is a disjoint union of and . Then, where if xi = 1 and if xi = 0. Let . Thus, is a continuous-variable function. The following theorem shows that is a generalization of f in the sense that for all x ∈ {0, 1}n; proof is provided in SI Appendix.
For discrete values of xi ∈ {0, 1}, i = 1, …, n, we have .
If pi = 1/2 for all i = 1, …, n, then is the output bias of f.
Consider the AND, OR, XOR, and NOT Boolean functions given in Table 1. The continuous-variable generalization of f1, f2, f3, and f4 are: , and .
Note that the above expressions have previously been derived via other (not probability-based) means [14].
Taylor Decomposition of Boolean functions
Since is a continuous-variable function, we can calculate its Taylor expansion. And since is a square-free polynomial, its Taylor expansion is finite and simplified (any term containing multiple derivatives of the same variable is zeroed out), as described in Proposition 1.4 using the standard multi-index notation. Let α = (α1, …, αn) where αi ∈ {0, 1}. We define and .
For p ∈ [0, 1]n, we have
Note that in Equation 1 is the output bias of f as was seen in Corollary 1.2. A natural choice for p is p = (1/2, …, 1/2) as it represents an unbiased selection for each variable and it also gives the output bias of the function. Being a natural generalization of the discrete Taylor decomposition, it thus offers certain unique advantages over the latter. The Taylor decomposition can be used to approximate a Boolean function by considering a subset of the terms. For example, a linear approximation consists of terms only up to |α| ≤ 1, a bilinear approximation up to |α| ≤ 2, etc., up until |α| ≤ n where it ceases to be an approximation and provides an exact decomposition of . A visual illustration is provided in Figure 2. The approximation order of a Boolean network could therefore vary between its minimum and maximum in-degrees (number of inputs per node).
Consider the continuous generalizations of the AND, OR, XOR and NOT functions given in Example 1.3 The corresponding Taylor expansions using Equation 1 and using the derivatives shown in Table 2 with p = (1/2, 1/2) are: , and .
Note that , and in the above equations are the output biases of the AND, OR, XOR, and NOT functions respectively. Also note that both the AND and OR functions contain the linear and the second order terms in their Taylor decomposition while the XOR function only contains the second order term. This difference is because both the AND and OR functions are monotone while XOR is not since it requires both inputs to be known.
Approximability of a model
We considered a suite of Boolean network models of biochemical regulation from two sources, namely the cell collective [2] and reference [2]. This suite consists of 137 networks with the number of nodes ranging from 5 to 321. The mean in-degree of these models ranges from 1.1818 to 4.9375 with the variances ranging between 0.1636 and 9.2941, while the mean output bias is limited to the range [0.1625,0.65625] with the variances between 0.0070 and 0.0933. For each biological model we generated an associated ensemble of 100 randomized models, where the connectivity and the output bias of the nodes of the original model were preserved and the logic rules were randomly chosen under the above constraints. This approach helps avoid confounding the causes of any observed effects with network structure or output bias, thereby narrowing the focus on the role of nonlinearity. We applied the Taylor decomposition to both the biological models and the associated ensembles and computed all possible nonlinear approximations. Both the biological models and the associated random ensembles were then simulated using a set of 1000 randomly chosen initial states iterated through 500 update steps for all orders of approximation; the same initial conditions were used for a given biological model and the associated random ensemble. The states of the variables were restricted to the interval [0,1] at every step in the simulations. We then computed the mean approximation error (MAE) of each model as the percentage mean squared error (MSE) between the exact Boolean states and the approximated probabilistic states at the end of the simulations; for the random ensembles we computed a single average MAE. Finally, we computed the “approximability” of each biological model as the difference between the MAE of the associated random ensemble and its MAE. Per these definitions, the MAE can range between 0.0 and 100.0, while the approximability can range between −100.0 and 100.0.
Classification of biological models
To identify any differences among the approximabilities of different types of biological networks we sought to classify them. Since there are multiple ways to classify biological networks, we chose two classifications so that: 1) they are as orthogonal as possible to each other; and 2) each classification has an appropriate number of (neither too few nor too many) categories. Classification 1 (C1) follows the “pathway ontology” (PW) [17] where the networks are grouped into five categories (Figure 4(a)), namely biochemical (n = 13), signaling (n = 22), disease (n = 55), metabolic (n = 14) and regulatory (n = 33). According the definitions used in the PW ontology, a “signaling” network comprises mainly of extracellular signal transduction components such as growth factors, kinases, etc. A “regulatory” network, on the other hand, comprises intracellular transcriptional components such as genes, transcription factors, etc. The term “biochemical” here refers to networks that comprises a mix of signaling and regulatory components. “Metabolic” networks consist of components involved in the synthesis and conversion of biomolecules such as enzymes and lipids. Finally, “disease” networks consist of components involved in diseases such as cancer, anemia, pathogenic ailments and disorders such as cell cycle malfunction. Classification 2 was suggested by in-house expertise, where the networks are grouped into four categories (Figure 4(b)), namely metazoan (n = 85), cancer (n = 24), primitive (n = 19) and plants (n = 9). The “metazonan” category refers to multicellular organisms and “primitive” refers to unicellular organisms. A given model could naturally belong in multiple categories within a classification but is assigned a unique category for the purpose of simplicity; we chose the categories according to the emphasis laid in the abstracts of the corresponding publications. More details are provided in the SI Appendix (Table S1).
Results and Discussion
Biological networks are less nonlinear than expected by chance
We found that the biological models are relatively more approximable for various degrees of nonlinearity when compared to a reference ensemble (Figure 3). The contrast is most prominent in the linear regime where the biological models are about 2% more approximable (p < 10−5) compared to their random counterparts. This suggests that the biological regulatory networks may have been optimized (presumably by evolution) for linearity in the nonlinearity of the Boolean rules, given that the reference ensemble preserves the network structure and the output biases of the corresponding biological models. This has implications not only for the feasibility of biomedical approaches to control emergent somatic complexity or guided self-assembly of novel forms [18], but also for models of anatomical homeostasis and evolvability: linearity implies easier control of its own complex processes by any biological system, and more efficient credit assignment during evolution.
The approximability of a biological network depends on its class, with the cancer family displaying the most variability
Even though the nonlinearity of biological networks is less than expected on average, individual and category-dependent variations were observed. In the following, we focus on the approximability corresponding to the linear order (“linear approximability”) of biological networks since it’s maximized at the linear regime (Figure 3). First, there are a few networks that are more nonlinear than expected as evidenced by the negative linear approximability (Fig 4). Second, the disease networks in C1 and the cancer networks in C2 are the ones with the most linear approximability (high positive values). In other words, the cancer or disease pathways tend to be more optimized for linearity compared to the other categories. This makes sense since a more linear pathway is more amenable to control, which presumably works in favour of the agents of disease. Moreover, the disease and cancer networks also display the highest variability in their linear approximability compared to other categories (p < 0.05 in all comparisons), with the corresponding values extending into the negative regime as well, that is they are sometimes more nonlinear than expected. Note that about 56% of the disease category comprises of non-cancer networks (31/55), which suggests that the effect is not significantly biased by cancer networks. Taken together, these observations suggest that regulatory nonlinearity may offer an effective “entry point” to the agents of disease by virtue of its natural heterogeneity that they could leverage to their advantage perhaps as a means to evade treatment since there’s no single level of nonlinearity to target. This heterogeneity may also have a connection to one of the hallmarks of cancer, namely genetic heterogeneity [19] where the cancerous cells within an individual display heterogeneous gene expression compared to the homogeneous expression in the healthy cells. In the case of nonlinearity, the heterogeneity manifests at the population level, raising the question of whether it may also be observed at the level of single cells within an individual. In other words, could the heterogeneity of nonlinearity be yet another hallmark of cancer?
The shape of the Taylor spectrum explains the extreme opposite characters of linear approximability of a pair of cancer models
Why are some models more linearly approximable and others less? The answer lies in the organization of the corresponding Taylor decompositions, as described above. To illustrate this in detail, we compared the Taylor decompositions of the least and the most linearly approximable models in our dataset (the two most extreme outliers in Fig 4). Those models respectively are the following: a model describing the role of mutations in the regulation of metastasis in lung cancer [20] and henceforth referred to as the ’Metastasis’ model ; and a model describing the role of the protein p53 in the regulation of cell-cycle arrest in breast cancer [21] and henceforth referred to as the ‘P53’ model. Both P53 and Metastasis are models of cancer, as may be expected from Fig 4. P53 has a linear approximability of about 8.02 and it consists of 16 nodes with a mean in-degree of 3.8±2.4 and a mean output bias of 0.38±0.14, while Metastasis has a linear approximability of about −7.28 and it consists of 32 nodes with a mean in-degree of 4.9±2.5 and a mean output bias of 0.27±0.26. Thus, while P53 is smaller and sparser than Metastasis, its nodes exhibit more output-uncertainty compared to Metastasis. According to the mean field theory of random Boolean networks [22], the opposite characters of the mean in-degree and the output bias of these models means that their dynamical behaviors could be expected to be similar (although with the caution that the theory was originally developed for infinite-sized and homogeneously connected networks, which is not the case here). However, we know that their linear approximabilities, which is another expression of dynamical behavior, are opposites. One explanation for this discrepancy lies in the distinct apportioning of nonlinearity in their respective Taylor decompositions (Fig 5). Specifically, while the magnitude of nonlinearity, defined as the mean absolute value of the Taylor derivatives for a given order and normalized appropriately (see text of Fig 5), tends to be clustered around the linear order for P53, they are relatively more spread out for Metastasis. Moreover, while the magnitude of the linear order for P53 is more than twice as large the next largest magnitude at lower orders the corresponding ratios for Metastasis are relatively smaller, thus explaining why P53 is more linearly approximable than Metastasis. This result is consistent with predictions based on a model of scaling of cellular control policies [23]. A more controllable (linear) network (P53) is optimal for cooperation with other cells towards collective (normal morphogenetic) goals. In contrast, a cell defecting from the collective and reverting to a more unicellular lifestyle (Metastasis) should exhibit a less predictable, controllable network due to pressures from parasites and competitors that independent unicellular organisms face. Methods for calculating controllability (e.g., linearity) are an important addition to recent efforts to solve the conundrum of interpretability of information structures in contexts ranging from machine learning to evolutionary developmental biology [24, 25, 26].
Broader implications
This paper introduces the concept of regulatory nonlinearity as a new measure of characterization for Boolean networks. There are several other related characterizations of Boolean networks such as canalization [27], effective connectivity [10], symmetry [28] and controllability [29]. It has been previously reported that the levels of canalization (a measure of the extent to which fewer inputs influence the outputs of a Boolean function) and the mean effective connectivity (a measure of collective canalization) are high in biological networks [2, 10]. It has also been found that biological networks need few inputs to reprogram [30] and are relatively easier to control [3]. Our formulation of regulatory nonlinearity is related to these other measures in that more linearity implies more apportioning of influence to individual inputs rather than collective sets of inputs (Figure 5). Hence, we hypothesize that regulatory nonlinearity may serve the purpose of controllability and epigenetic stability [31]. Our results further moot the possibility that regulatory nonlinearity may be a factor underlying more powerful dynamical phenomena such as memory [32] and computation, defined as the capacity for adaptive information-processing [33]. Even though there’s increasing consensus that biological systems contain memory and perform computation, clarity is lacking as to what features of those systems enable it and what general principles underlie it [33, 34]. Our framework of regulatory nonlinearity offers an approach to answering these questions. For example, one could consider a known dynamical model with a capacity for memory [32] or universal computation such as the elementary cellular automaton (ECA) driven by rule 110 [35] and ask if there are unique properties of its Taylor spectrum that confer their respective capabilities. Present approaches to answering this question typically consists of characterization of the dynamical behavior and not the rules [36, 37, 32]. A characterization of the rules especially makes sense for ECA [38] since the structure is always the same (lattice) and the only feature that distinguishes one ECA from the other is the rule. Looking at such questions from an even broader perspective it becomes evident that they are only instances of the ultimate puzzle of complex systems, namely what connects the structure and the function of a system. Even though recent work has attempted to answer this question from the perspective of the rules or dynamical laws that govern the system [39, 10, 38], more tools are needed [40]. In that regard, our framework of regulatory nonlinearity could be a novel addition to this burgeoning toolkit in that it could also be applied to continuous models of biological networks such as those based on differential equations.
Limitations
The main limitation of our formulation of approximability is that the approximation accuracy will necessarily increase with higher orders of approximation for arbitrary Boolean networks (the highest order of approximation is exact). However, this does not affect the falsifiability of our framework since it’s possible to construct networks, say with XOR-like functions, that are less linearly approximable than the associated ensembles. The Metastasis model is another example in that regard (Figure 5). Furthermore, the notion of nonlinearity is limited to the local level of the Boolean rules in our framework, whereas its possible to conceive network-level measures of nonlinearity where the role of the network structure is included. Lastly, our conclusions about the linearity of biological regulatory networks may be a reflection of a hidden bias built in the inference methods that produced the models in the first place. We leave it to future work to explore these realms.
Supplementary Information
The nonlinearity of regulation in biological networks
Santosh Manicka, Kathleen Johnson, Michael Levin and David Murrugarra
1. Probabilistic generalization of Boolean logic
Here we show that is a generalization of f in the sense that .
For discrete values of xi ∈ {0, 1}, i = 1, …, n, we have
Proof. Let z = (z1, …, zn) ∈ {0, 1}n. Since each zi is either 0 or 1, we have that pi = 1 if zi = 1 or pi = 0 if zi = 0 for i = 1, …, n. We want to show that . Since , we have that either or . If , then f (z) = 1 and . Moreover, for any other with x ≠ z we have that Pr(x) = 0. Thus, . Now if , then because Σ∅ = 0. Thus, for all .
2. Maximum absolute value of a Taylor derivative
Here we show that .
We begin with the definition of the derivative given by Since is a pseudo-Boolean function and hence a multilinear polynomial [1], we can rewrite it by setting h to 1, as a finite difference; the idea being that the derivative taken over any point on a line is the line itself: Since multilinear interpolation can be formulated as weighted averaging [2], we can further rewrite it as follows (the weights are equal here since the non-binary values are all set to 0.5): As can be seen, there are a total of 2k terms, with half of them positively signed and half negative. This form of expression generalizes to derivatives taken over two or more variables. For example, the derivative taken over two variables, xi and xj, looks as follows: Following rearrangement of terms it becomes evident that this expression also contains 2k−1 positive terms and 2k−1 negative terms, with the only difference in the power of the denominator term. It can thus be concluded that any derivative (in multi-index notation) has 2k−1 positive terms and 2k−1 negative terms.
A straightforward way to maximize the value of a derivative expressed in this form is by assigning as many instances of 1 as possible to the positive terms and as few instances of 1 as possible to the negative terms. For a Boolean function with k inputs and output bias p, this can be accomplished by assigning min(2k−1, p2k) ones and max(p2k − 2k−1, 0) ones respectively. Therefore, Note that this formula only applies to a specific order of nonlinearity |α| independent of the other orders within the same Boolean function. In actuality, there are dependencies between the various orders within a Boolean function. That is, if a Boolean function were to be constructed such that the derivative of a particular order |α1| is maximized then there’s no guarantee that the derivative of another order |α2| ≠ |α1| could be simultaneously maximized. This is one of the limitations of the normalization for which the above formula is used.
Footnotes
The authors have declared that no competing interests exist.
New results from analysis of the biological data and new figures (1 and 5).