Feedback Regulation in a Cancer Stem Cell Model can Cause an Allee Effect

Abstract

The exact mechanisms of spontaneous tumor remission or complete response to treatment are phenomena in oncology that are not completely understood. We use a concept from ecology, the Allee effect, to help explain tumor extinction in a model of tumor growth that incorporates feedback regulation of stem cell dynamics, which occurs in many tumor types where certain signaling molecules, such as Wnts, are upregulated. Due to feedback and the Allee effect, a tumor may become extinct spontaneously or after therapy even when the entire tumor has not been eradicated by the end of therapy. We quantify the Allee effect using an ‘Allee index’ that approximates the area of the basin of attraction for tumor extinction. We show that effectiveness of combination therapy in cancer treatment may occur due to the increased probability that the system will be in the Allee region after combination treatment versus monotherapy. We identify therapies that can attenuate stem cell self-renewal, alter the Allee region and increase its size. We also show that decreased response of tumor cells to growth inhibitors can reduce the size of the Allee region and increase stem cell densities, which may help to explain why this phenomenon is a hallmark of cancer.

Appendices

Appendix 1: Approximation of the Separatrix for System (9) using the Stable Manifold Theorem.

We use the Stable Manifold Theorem (SMT) to prove that the separatrix described in Theorem 1 near the equilibrium point $\mathbf {P_2}(S_2,A_2)$ of system (9) exists and to approximate it. We follow the technique presented in (Perko 2001). We recall that $\mathbf {P_2}$ occurs at the unique intersection of the curves $\{p(S,a) = 0.5\}$ and $\{F(S,a) = 0\}$. We state the SMT here for reference.

Theorem 2

(The Stable Manifold Theorem). Let E be an open subset of ${\mathbb {R}}^n$ containing the origin, let $\mathbf {f} \in C^1(E)$, and let $\phi _t$ be the flow of the nonlinear system $\dot{\mathbf {x}} = \mathbf {f(x)}$. Suppose that $\mathbf {f(0) = 0}$ and that $D\mathbf {f(0)}$ has k eigenvalues with negative real part and $n-k$ eigenvalues with positive real part. Then there exists a k-dimensional differentiable manifold M tangent to the stable subspace $E^m$ of the linear system $\dot{\mathbf {x}} = \mathbf {Ax}$ at $\mathbf {0}$ where $\varvec{A} = D \mathbf {f(0)}$, such that for all $t \ge 0$, $\phi _t(M) \subset M$ for all $\mathbf {x_0} \in M$ and

$$\begin{aligned} \lim _{t \rightarrow \infty } \phi _t (\mathbf {x_0}) = \mathbf {0}. \end{aligned}$$

In our case, we make an affine change of coordinates to system (9) which sends $\mathbf {P_2} \rightarrow {\mathbf {0}}$ and use the constructive proof of the SMT (see Perko 2001, p. 108) to find the separatrix M.

1.1 Affine Change of Coordinates

To apply the SMT, we need to first make the affine change of coordinates: $\mathbf {c}: (S,a) \rightarrow (S,a) - (S_2,A_2)$. We let $(S^*,a^*) = \mathbf {c}(S,a)$. Then, applying $\mathbf {c}$ to (9), and noting that $(S,a) = (S^*,a^*) + (S_2,A_2)$, and $\frac{\partial }{\partial t}(S,a) = \frac{\partial }{\partial t}((S^*,a^*)+(S_2,A_2)) = \frac{\partial }{\partial t}(S^*,a^*)$, we obtain

$$\begin{aligned} \begin{aligned} \dot{S^*}&= (2p^*(S^*, a^*) -1)k(S^* + S_2) = f^*_1(S^*,a^*), \\ \dot{a^*}&= (a^* + A_2)\left( \frac{\beta (S^* + S_2)(a^* + A_2)}{1+ \lambda (a^* +A_2)} - 1 \right) = f^*_2(S^*,a^*), \\ p^*(S^*, a^*)&=\frac{\xi _1(a^* + A_2)}{1+ \xi _1 (a^* + A_2)} \frac{1}{1+ \xi _2(S^* + S_2)}. \end{aligned} \end{aligned}$$

(16)

The Jacobian for (16) is

$$\begin{aligned} \varvec{J}^*(S^*,a^*) = \left( \begin{array} {cc} 2p^*_{S^*} k (S^* + S_2) + (2p^* -1)k &{} 2p^*_{a^*}k(S^* + S_2) \\ (f_1)_{S^*} &{} (f_1)_{a^*} \end{array} \right) , \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} p^*_{S^*}&= \frac{-\xi _1\xi _2(a^* + A_2)}{(1+ \xi _1 (a^* + A_2))(1+\xi _2(S^*+ S_2))^2}, \\ p^*_{a^*}&= \frac{\xi _1}{(1+\xi _2(S^* + S_2)) (1+ \xi _1(a^* + A_2))^2}, \\ (f^*_2)_{S^*}&= \frac{\beta (a^* + A_2)^2}{1+\lambda (a^* + A_2)}, \\ (f^*_2)_{a^*}&= \frac{\beta (S^* + S_2)(a^* + A_2)(2+ \lambda (a^* + A_2))}{(1+ \lambda (a^* + A_2))^2} - 1. \end{aligned} \end{aligned}$$

(17)

In this coordinate system, $(S^*,a^*) = (0,0)$ is an equilibrium point and $P^*(0,0)$ corresponds to $\mathbf {P_2}(S_2,A_2)$. To use the SMT, we need to first determine $\varvec{A} = D\mathbf {f}(\mathbf {0}) = \varvec{J}^*(0,0)$. We have

$$\begin{aligned} \varvec{A}=\varvec{J}^*(0,0) = \left( \begin{array}{cc} 2p^*_{S^*} (0,0) k S_2 &{} 2p^*_{a^*} (0,0) k S_2 \\ (f_2)_{S^*} (0,0) &{} (f_2)_{a^*} (0,0) \end{array} \right) . \end{aligned}$$

(18)

We first note, as in the original $\varvec{J}(S_1,S_2)$, that since $p^*_{S^*} <0$ and $(f^*_2)_{a^*}(0,0)>0$, $2p^*_{S^*} (0,0) k S_2(f_2)_{a^*} (0,0) <0$ and since $p^*_{a^*}>0$ and $(f_2^*)_{S^*}>0$, $2p^*_{a^*} (0,0) k S_2 (f_2)_{S^*} (0,0) >0$. Therefore,

$$\begin{aligned} \det \varvec{J}^*(0,0)=2p^*_{S^*} (0,0) k S_2)(f_2^*)_{a^*} (0,0) - 2p^*_{a^*} (0,0) k S_2 (f^*_2)_{S^*} (0,0) <0, \end{aligned}$$

and hence $\varvec{J}^*(0,0)$ has one positive and one negative eigenvalue, and $P^*$ is a saddlepoint. We also recall that $S_2$ and $A_2$ satisfy

$$\begin{aligned} {\left\{ \begin{array}{ll} \dfrac{\xi _1A_2}{1+\xi _1 A_2} \dfrac{1}{1+\xi _2 S_2} = 0.5 , \\ \dfrac{\beta S_2 A_2}{1+ \lambda A_2}=1. \end{array}\right. } \end{aligned}$$

(19)

Using (19), we simplify (17) to calculate the elements of ${\varvec{A}}$,

$$\begin{aligned} \begin{aligned} p^*_{S^*} (0,0)&= \frac{-A_2\xi _1\xi _2}{(1+ \xi _1 A_2)(1+\xi _2S_2)^2} = \frac{- \xi _2}{2 (1+\xi _2 S_2)},\\ p^*_{a^*} (0,0)&= \frac{1}{(1+\xi _2S_2) (1+ \xi _1A_2)^2} = \frac{1}{2 A_2 (1+\xi _1 A_2)}, \\ (f^*_2)_{S^*} (0,0)&= \frac{\beta A_2^2}{1+\lambda A_2} = \frac{A_2}{S_2}, \\ (f^*_2)_{a^*} (0,0)&= \frac{\beta S_2 A_2(2+ \lambda A_2)}{(1+ \lambda A_2)^2} - 1 = \frac{2+\lambda A_2}{1+ \lambda A_2} - 1 = \frac{1}{1+\lambda A_2}. \end{aligned} \end{aligned}$$

(20)

Substituting (20) into (18), we have the following expression for $\varvec{A}=\varvec{J}^*(0,0)$,

$$\begin{aligned} \varvec{A}=\left( \begin{array} {cc}\dfrac{- \xi _2kS_2}{1+\xi _2 S_2} &{} \dfrac{kS_2}{A_2 (1+\xi _1 A_2)} \\ \dfrac{A_2}{S_2} &{} \dfrac{1}{1+\lambda A_2} \end{array} \right) . \end{aligned}$$

(21)

1.2 Preliminary Calculations for the SMT

Following (Perko 2001) and taking $\mathbf {x}= (S^*,a^*)$, we can rewrite the system (16) as

$$\begin{aligned} \dot{\mathbf {x}} = \mathbf {Ax} + \mathbf {F(x)}, \end{aligned}$$

(22)

where $\varvec{A}=\varvec{J}^*(0,0)$ and $\mathbf {F(x)} = \mathbf {f^*(x)} - \mathbf {Ax}$. We next need to find an invertible matrix $\varvec{C}$ such that

$$\begin{aligned} \varvec{B} = \varvec{C}^{-\mathbf{1}}\varvec{AC}= \left( \begin{array} {cc} L_1 &{} 0 \\ 0 &{} L_2 \end{array} \right) , \end{aligned}$$

(23)

where $L_1$ and $L_2$ are the negative and positive eigenvalues, respectively, of $\varvec{A}=(A_{ij})$. We first calculate the trace, T, and determinant, D, of A,

$$\begin{aligned} T&= A_{11} + A_{22} = \dfrac{- \xi _2kS_2}{1+\xi _2 S_2} + \frac{1}{1+\lambda A_2},\\ D&= A_{11}A_{22} - A_{12}A_{21} =\dfrac{- \xi _2kS_2}{1+\xi _2 S_2} \dfrac{1}{1+\lambda A_2} - \dfrac{kS_2}{A_2 (1+\xi _1 A_2)} \dfrac{A_2}{S_2} \\&= \dfrac{-\xi _2 k S_2}{(1+\xi _2 S_2)(1+\lambda A_2)} - \dfrac{k}{1+\xi _1 A_2}. \end{aligned}$$

We note, from the calculations above, that $D<0$. We proceed to calculate $0 = \det (A - L I)$ to obtain the quadratic equation

$$\begin{aligned} 0=L^2 - (A_{11} + A_{22}) L + (A_{11}A_{22} - A_{12}A_{21}) = L^2 - TL + D. \end{aligned}$$

The quadratic formula gives us:

$$\begin{aligned} L_{1,2} = \frac{T \mp (T^2 - 4D)^{1/2}}{2} = T/2 \mp (T^2/4 - D)^{1/2}. \end{aligned}$$

Since $D<0$, we find that $L_{1,2}$ are both real and have opposite sign; hence, $L_1 < 0 <L_2$. It can be verified that $\mathbf {v}_1 = [(L_1 - A_{22}), A_{21}]^T$ and $\mathbf {v}_2 = [ (L_2 - A_{22}), A_{21}]^T$ are eigenvectors corresponding (respectively) to $L_1$ and $L_2$. Here the superscript T denotes the transpose. Therefore, we have

$$\begin{aligned}&\varvec{A} = \varvec{C B C}^{-1} \\&\quad = \dfrac{1}{A_{21}(L_1-L_2)}\left( \begin{array} {cc} L_1 - A_{22} &{} L_2 - A_{22} \\ A_{21} &{} A_{21} \end{array} \right) \left( \begin{array} {cc} L_1 &{} 0 \\ 0 &{} L_2 \end{array} \right) \left( \begin{array} {cc} A_{21} &{} -L_2 + A_{22} \\ - A_{21} &{} L_1 - A_{22} \end{array} \right) \end{aligned}$$

We make another change of variables, taking $\mathbf {y}=\varvec{C}^{-1}(\mathbf {x})$, and writing (22) as

$$\begin{aligned} \dot{\mathbf {y}} = \mathbf {By} +\mathbf {G(y)}, \end{aligned}$$

(24)

where $\varvec{B}$ is from (23) and $\mathbf {G(y)} = \mathbf {C^{-1}F(Cy)}$.

1.3 Applying the SMT

By the SMT (taking $\mathbf {a}=(a_1,a_2)$),

$$\begin{aligned} \mathbf {u}(t,\mathbf {a}) = \varvec{U}(t)\mathbf {a}+\int _0^t \varvec{U}(t-s) \mathbf {G}(\mathbf {u}(s,\mathbf {a}))ds - \int _t^\infty \varvec{V}(t-s)\mathbf {G}(\mathbf {u}(s,\mathbf {a}))ds \end{aligned}$$

(25)

is the solution to (24), where

$$\begin{aligned} \varvec{U}(t) = \left( \begin{array} {cc} e^{L_1 t} &{} 0 \\ 0 &{} 0 \end{array} \right) \text{ and } \varvec{V}(t) = \left( \begin{array} {cc} 0 &{} 0 \\ 0 &{} e^{L_2t} \end{array} \right) . \end{aligned}$$

We solve for $\mathbf {u}$ using the method of successive approximation. We let $\mathbf {u}^{(0)} (t,\mathbf {a}) = \mathbf {0}$ and

$$\begin{aligned} \mathbf {u}^{(j+1)} (t,a) = \varvec{U}(t)\mathbf {a} + \int _0^t \varvec{U}(t-s) \mathbf {G}(\mathbf {u}^{(j)} (s,\mathbf {a})) ds - \int _t^\infty \varvec{V}(t-s) \mathbf {G}(\mathbf {u}^{(j)} (s,\mathbf {a}) )ds. \end{aligned}$$

(26)

To solve for $j=1$, we note that $\mathbf {G(0)}= \mathbf {{C}^{-1} F(C\cdot 0)} = \mathbf {{C}^{-1}F(\mathbf {0}) }= \mathbf {0}$ since $f_1(0,0) = f_2(0,0) = 0$. Therefore,

$$\begin{aligned} \mathbf {u}^{(1)} (t,\mathbf {a}) = \left( \begin{array} {c} e^{L_1 t} a_1 \\ 0 \end{array} \right) . \end{aligned}$$

For the next approximation, we first calculate $\varvec{U}(t-s) \mathbf {G}(\mathbf {u}^{(1)}(s,\mathbf {a})) = \varvec{U}(t-s) \varvec{C}^{-1} \mathbf {F(Cw)} = \varvec{H}_1 \mathbf {F(Cw)}$, where $\varvec{H}_1 = \varvec{U}(t-s)\varvec{C}^{-1}$ and $\mathbf {w}= (e^{L_1 s} a_1, 0)^T$. Simplifying $\varvec{H}_1$ gives us:

$$\begin{aligned} \varvec{H}_1&= \varvec{U}(t-s) \varvec{C}^{-1} = \dfrac{1}{A_{21}(L_1 - L_2)} \left( \begin{array} {cc} e^{L_1(t-s)} &{} 0 \\ 0 &{} 0 \end{array} \right) \left( \begin{array} {cc} A_{21} &{} A_{22} - L_2 \\ -A_{21} &{} L_1 - A_{22}\end{array} \right) \\&= e^{L_1(t-s)} \left( \begin{array} {cc} \frac{1}{L_1-L_2} &{} \frac{A_{22} - L_2}{A_{21}(L_1 - L_2)} \\ 0 &{} 0 \end{array} \right) . \end{aligned}$$

Then,

$$\begin{aligned} \varvec{H}_1 \mathbf {F}(\varvec{C}\mathbf {w})= & {} e^{L_1(t-s)} \left( \begin{array} {cc} \frac{1}{L_1-L_2} &{} \frac{A_{22} - L_2}{A_{21}(L_1 - L_2)} \\ 0 &{} 0 \end{array} \right) \nonumber \\&\quad \times \left[ \left( \begin{array} {c} f_1 (\mathbf {Cw}) \\ f_2 (\mathbf {Cw}) \end{array} \right) - e^{L_1 s} a_1 \left( \begin{array} {cc} A_{11} &{} A_{12} \\ A_{21} &{} A_{22} \end{array} \right) \left( \begin{array} {c} L_1 - A_{22} \\ A_{21} \end{array} \right) \right] \nonumber \\= & {} e^{L_1(t-s)} \left( \begin{array}{c} \frac{f_1(\mathbf {Cw})}{(L_1-L_2)} + \frac{f_2(\mathbf {Cw}) A_{22} - L_2}{A_{21}(L_1 - L_2)} \\ 0 \end{array}\right) \nonumber \\&-\,e^{L_1 t}a_1 \left( \begin{array} {c}\frac{L_1(T-L_2) -D}{L_1 - L_2} \\ 0 \end{array} \right) . \end{aligned}$$

(27)

Hence,

$$\begin{aligned} \int _0^t \varvec{U}(t-s) \mathbf {G}(\mathbf {u}^{(1)}(s,\mathbf {a}))= & {} \int _0^t e^{L_1(t-s)} \left( \begin{array} {c} \frac{f_1(\mathbf {Cw})}{(L_1-L_2)} + \frac{f_2(\mathbf {Cw}) A_{22} - L_2}{A_{21}(L_1 - L_2)} \\ 0 \end{array} \right) \nonumber \\&ds - t \left[ e^{L_1 t}a_1 \left( \begin{array} {c} \frac{L_1(T-L_2) -D}{L_1 - L_2} \\ 0 \end{array} \right) \right] . \end{aligned}$$

(28)

We note that our stable manifold will be of the form $y_2= \psi ^{(2)}_2(y_1)$, where $\psi ^{(2)}_2(a_1) = u^{(2)}_2(0,a_1,0)$. Since $\mathbf {U}(t)\mathbf {a}$ and (28) only contribute trivially to $u^{(2)}_2$, we will not perform further calculations on them.

Next, we calculate $\varvec{V}(t-s) \mathbf {G}(\mathbf {u}^{(1)}(s,\mathbf {a})) = \varvec{V}(t-s) \varvec{C}^{-1} \mathbf {F}(\mathbf {Cw}) = \varvec{H}_2 \mathbf {F}(\mathbf {Cw})$. As before, we first calculate $\varvec{H}_2$:

$$\begin{aligned} \varvec{H}_2&= \varvec{V}(t-s) \varvec{C}^{-1} = \dfrac{1}{A_{21}(L_1 - L_2)} \left( \begin{array}{cc} 0 &{} 0 \\ 0 &{} e^{L_2(t-s)} \end{array} \right) \left( \begin{array}{cc} A_{21} &{} A_{22} - L_2 \\ -A_{21} &{} L_1 - A_{22}\end{array} \right) \nonumber \\&= e^{L_2(t-s)}\left( \begin{array} {cc} 0 &{} 0 \\ \frac{-1}{L_1 - L_2} &{} \frac{L_1 - A_{22}}{A_{21}(L_1-L_2)}\end{array} \right) . \end{aligned}$$

We thus have,

$$\begin{aligned} \varvec{H}_2\mathbf {F}(\mathbf {Cw})= & {} e^{L_2(t-s)} \left( \begin{array} {cc} 0 &{} 0 \\ \frac{-1}{L_1 - L_2} &{} \frac{L_1 - A_{22}}{A_{21}(L_1-L_2)}\end{array} \right) \nonumber \\&\times \left[ \left( \begin{array} {c} f_1 (\mathbf {Cw}) \\ f_2 (\mathbf {Cw}) \end{array} \right) - e^{L_1 s} a_1 \left( \begin{array} {cc} A_{11} &{} A_{12} \\ A_{21} &{} A_{22} \end{array} \right) \left( \begin{array} {c} L_1 - A_{22}\\ A_{21} \end{array} \right) \right] \nonumber \\= & {} e^{L_2(t-s)} \left( \begin{array} {c} 0 \\ \frac{-f_1(\mathbf {Cw})}{L_1 - L_2} + \frac{f_2(\mathbf {Cw})(L_1 - A_{22})}{A_{21} (L_1- L_2)}\end{array} \right) \nonumber \\&-\, e^{s(L_1-L_2)}e^{L_2t}\begin{bmatrix}0 \\ g(L_1,L_2,A_{21},A_{22})\end{bmatrix}. \end{aligned}$$

(29)

Taking the integral of the right-hand term on the domain $[t,\infty )$ gives us $\frac{e^{L_1 t}}{L_2-L_1}(0, g(\cdot ))^T$. We find that $g(\cdot ) = L_1^2 - TL_1 + D = 0$. Therefore, this term does not contribute to the stable manifold.

The SMT allows us to calculate the second approximation to the separatrix, $\hbox {M}^* = u_2^{(2)}(0,a_1,0)$, as

$$\begin{aligned} {\text {M}}^* = \frac{1}{L_1 - L_2} \left( \int _0^\infty - e^{-L_2s}f^*_1 (\mathbf {Cw}) ds + \frac{L_1 - A_{22}}{A_{21}} \int _0^\infty e^{-L_2s} f^*_2 (\mathbf {Cw}) ds \right) , \end{aligned}$$

(30)

where $\mathbf {Cw} = e^{L_1 s} a_1 (L_1 - A_{22}, A_{21})^T$, and by (16),

$$\begin{aligned} f^*_1(\mathbf {Cw})&=\left( \dfrac{2\xi _1(e^{L_1s}a_1A_{21} + A_2)}{(1+ \xi _1(e^{L_1s}a_1A_{21} +A_2))(1+\xi _2(e^{L_1s}a_1(L_1-A_{22})+S_2))} -1 \right) \nonumber \\&\quad k (e^{L_1s}a_1(L_1-A_{22}) +S_2), \end{aligned}$$

(31)

$$\begin{aligned} f^*_2(\mathbf {Cw})&= (e^{L_1s}a_1A_{21} + A_2)\left( \frac{\beta (e^{L_1s}a_1(L_1-A_{22}) \!+\! S_2)(e^{L_1s}a_1A_{21} + A_2)}{1+ \lambda (e^{L_1s}a_1A_{21} +A_2)} - 1 \right) . \end{aligned}$$

(32)

We now solve $- \int _0^{\infty } I_1 ds=\int _0^\infty e^{-L_2s}f^*_1 (\mathbf {Cw}) ds$ and $\int _0^\infty I_2ds=\int _0^\infty e^{-L_2s} f^*_2 (\mathbf {Cw}) ds$. Substituting (31) into $I_1$, we obtain

$$\begin{aligned} I_1= & {} \left( \frac{ 2\xi _1 e^{-L_2 s} (e^{L_1 s } a_1 A_{21} + A_2)}{(1+ \xi _1(e^{L_1s}a_1A_{21} +A_2))(1+\xi _2(e^{L_1s}a_1(L_1-A_{22})+S_2))} - e^{-L_2 s} \right) \nonumber \\&k (e^{L_1 s} a_1 (L_1 - A_{22} )+ S_2). \end{aligned}$$

(33)

We split $I_1$ into three parts:

$$\begin{aligned} I_1= & {} I_{11} + I_{12} + I_{13} \\= & {} \frac{ (2 \xi _1e^{-L_2 s} (e^{L_1 s } a_1 A_{21} + A_2)k(e^{L_1 s} a_1 (L_1 - A_{22} )+ S_2) }{(1+ \xi _1(e^{L_1s}a_1A_{21} +A_2))(1+\xi _2(e^{L_1s}a_1(L_1-A_{22})+S_2)}\\&-\,ke^{(L_1-L_2)s} a_1 (L_1-A_{22}) - e^{-L_2 s} k S_2. \end{aligned}$$

We can directly integrate $I_{12}$ and $I_{13}$ to obtain

$$\begin{aligned} -\int _0^\infty I_1ds = -\int _0^\infty I_{11} ds - \frac{ka_1 (L_1 - A_{22})}{L_1 - L_2}+\frac{k S_2}{L_2}. \end{aligned}$$

(34)

We now work to simplify $I_{11}$ by taking $u = e^{L_1 s}$. The change of variables gives us

$$\begin{aligned} \begin{aligned} -\int _0^\infty I_{11} ds&= -\frac{2k}{L_1}\int _{u_1}^{u_2} u^{-T/L_1} \\&\quad \times \frac{(ua_1 A_{21} + A_2) (u a_1(L_1 - A_{22}) +S_2)}{ (1+ \xi _1 (u a_1 A_{21} + A_2))(1+ \xi _2(u a_1 (L_1 - A_{22}) + S_2))}du \\&=\frac{2\xi _1k}{L_1}\int _0^1 u^{-T/L_1} \frac{(uc_1 + A_2) (u c_2 +S_2)}{ (1+ \xi _1 (u c_1 + A_2))(1+ \xi _2(u c_2 + S_2))}du, \end{aligned} \end{aligned}$$

(35)

where we find that $u_1 = 1$ and $u_2 = \lim _{s\rightarrow \infty } e^{L_1 s} = 0$ since $L_1 < 0$. We take $c_1 = a_1 A_{21}$ and $c_2 = a_1(L_1 - A_{22})$.

We would like to do a partial fraction decomposition for the integrand term in (35) not containing $u^{-T/L_1}$, $I_{11}^*$. Noting that both the numerator and denominator are of degree 2, we first perform long division to obtain a fraction p / q where deg $p<$ deg q. Fully multiplying the terms in the fraction, and setting $c_3 = A_2c_2 + S_2c_1$ and $c_4 = A_2 \xi _1 + S_2 \xi _2 + A_2S_2 \xi _1 \xi _2 +1$ gives us

$$\begin{aligned} \begin{aligned} I_{11}^*&= \frac{ u^2 c_1 c_2 + u c_3 + A_2 S_2}{u^2 \xi _1 \xi _2 c_1 c_2 + u(\xi _1 \xi _2 c_3 + \xi _2 c_2 + \xi _1 c_1) + c_4}\\&=\frac{1}{\xi _1 \xi _2} - \frac{u (c_2 / \xi _1 + c_1 / \xi _2)+ (A_2 / \xi _2 + S_2 / \xi _1 + 1/(\xi _1\xi _2))}{(1+ \xi _1 (u c_1 + A_2))(1+ \xi _2(u c_2 + S_2))}. \end{aligned} \end{aligned}$$

(36)

Setting $c_5 = c_2 / \xi _1 + c_1 / \xi _2$ and $c_6 = A_2 / \xi _2 + S_2/ \xi _1 + 1/(\xi _1 \xi _2)$, the second term of (36) becomes

$$\begin{aligned} \frac{u c_5 + c_6}{(1+ \xi _1 (u c_1 + A_2))(1+ \xi _2(u c_2 + S_2))} = \frac{N_1}{(1+ \xi _1 (u c_1 + A_2))} + \frac{N_2}{(1+ \xi _2(u c_2 + S_2))}, \end{aligned}$$

where

$$\begin{aligned} N_1= & {} \frac{-c_5(\xi _1 A_2 +1) + c_1 c_6\xi _1}{\xi _1\xi _2(c_1S_2 - A_2c_2)+c_1\xi _1 - c_2\xi _2},\\ N_2= & {} \frac{-c_5(\xi _2 S_2 +1) + c_2 c_6\xi _2}{\xi _1\xi _2(-c_1S_2 +A_2c_2)-c_1\xi _1 + c_2\xi _2}. \end{aligned}$$

Therefore, the integrand in (35), $u^{-T/L_1}I_{11}^*$, can be written as

$$\begin{aligned} u^{-T/L_1}I_{11}^*=u^{-T/L_1} \left( \frac{1}{\xi _1 \xi _2} - \frac{N_1}{(1+ \xi _1 (u c_1 + A_2))} - \frac{N_2}{(1+ \xi _2(u c_2 + S_2))}\right) . \end{aligned}$$

We note that the first term can be integrated,

$$\begin{aligned} \frac{2\xi _1k}{L_1\xi _1 \xi _2 }\int _0^1 u^{-T/L_1} du= \frac{-2k}{L_2 \xi _1 \xi _2} u^{-L_1/L_2}, \end{aligned}$$

where the first equality comes about from the observation that $-T/L_1 = -1 - L_2/L_1$. To summarize, if we set

$$\begin{aligned} I^{**}_{11} = u^{-T/L_1}\left( \frac{N_1}{(1+ \xi _1 (u c_1 + A_2))} + \frac{N_2}{(1+ \xi _2(u c_2 + S_2))}\right) , \end{aligned}$$

we can rewrite (34) as

$$\begin{aligned} -\int _0^\infty I_1 ds= \frac{k S_2}{L_2}-\frac{k C_2}{L_1 - L_2} -\frac{2\xi _1k}{L_2 \xi _1 \xi _2} - \frac{2\xi _1k}{L_1} \int _0^1 I^{**}_{11} du. \end{aligned}$$

(37)

To solve $\int _0^1 I^{**}_{11} du$, we will need to use a hypergeometric function and the beta function. Indeed, we have the formula

$$\begin{aligned} \int _0^1 t^{b-1} (1-t)^{c-b-1}(1-tx)^{-a}dt = B(b,c-b) _2F_1(a,b;c;x), \end{aligned}$$

where $B(a,b) = \int _0^a t^{a-1}(1-t)^{b-1} dt$ and $_2F_1(a_1,a_2;b_1;x) = \sum _{k=0}^\infty \frac{(a_1)_k(a_2)_k}{(b_1)_k} \frac{x^k}{k!}$. In our case, we split $I^{**}_{11}$ naturally as a sum of two terms, and for the first integral, we have $b-1 = -T/L_1$, hence $b = 1-T/L_1 = -L_2/L_1$, $0=c-b-1$, hence $c = b+1 = 2-T/L_1$, $a=1$, and $x = (-\xi _1 C_1)/(\xi _1 A_2 +1)$, where we have pulled $(\xi _1 A_2 +1)^{-1}$ from the denominator. We want to first find an explicit representation for $B(b,c-b)$,

$$\begin{aligned} B(b,c-b) = B(-T/L_1+1,1) = \int _0^1 t^{-T/L_1} dt = \frac{1}{-L_2/L_1} t^{-T/L_1 +1}|^1_0 = \frac{-L_1}{L_2}. \end{aligned}$$

Using the hypergeometric function and (37), our final formula for $-\int _0^\infty I_1 ds = \int _0^\infty -e^{L_2 s} f_1^* (\mathbf {Cw})ds$ is

$$\begin{aligned} -\int _0^\infty I_1 ds= & {} k\left( \frac{S_2}{L_2} - \frac{C_2}{L_1-L_2}-\frac{2}{L_2 \xi _2} + \frac{2\xi _1N_1}{L_2(\xi _1 A_2 +1)} {}_2F_1^1\right. \nonumber \\&\left. + \frac{2\xi _1 N_2}{L_2(\xi _2 S_2 +1)} {}_2F_1^2 \right) , \end{aligned}$$

(38)

where

$$\begin{aligned} _2F_1^1= 2F_1\left( 1,-L_2/L_1;1-L_2/L_1; \frac{-\xi _1 c_1}{\xi _1 A_2 +1}\right) \end{aligned}$$

and

$$\begin{aligned} _2F_1^2 = 2F_1\left( 1,-L_2/L_1;1-L_2/L_1; \frac{-\xi _2 c_2}{\xi _2 S_2 +1}\right) \end{aligned}$$

We now begin work to solve $\int _0^\infty I_2 ds = \int _0^\infty e^{-L_2 s} f_2(\mathbf {Cw}) ds$. Using the same substitution as earlier, i.e., $u = e^{L_1 s}$, and again taking $c_1 = a_1 A_{21}$ and $c_2 = a_1 (L_1 - A_{22})$, we obtain

$$\begin{aligned} \int _0^\infty I_2 ds= & {} \frac{-1}{L_1} \int _0^1 u^{-T/L_1} (uc_1 + A_2) \left( \frac{\beta (uc_2 + S_2)(uc_1 + A_2)}{1+ \lambda (uc_1 +A_2)} - 1 \right) , \nonumber \\= & {} \frac{-\beta }{L_1} \int _0^1 u^{-T/L_1} \frac{(uc_2 + S_2)(uc_1 + A_2)^2}{1+ \lambda (uc_1 + A_2)}du + \frac{1}{L_1}\int _0^1 u^{-T/L_1 +1} c_1\nonumber \\&+ u^{-T/L_1} A_2 du, \nonumber \\= & {} \frac{-\beta }{L_1} \int _0^1 u^{-T/L_1}\left( c_3^* u^2 + c_4^*u + c_5^* + \frac{c_6^*}{\lambda u c_1 + \lambda A_2 +1}\right) du\nonumber \\&+ \frac{c_1}{L_1 - L_2} + \frac{ -A_2}{L_2}, \end{aligned}$$

(39)

where the last equality comes from long division in the first integral and full integration of the second. The constants are as follows:

$$\begin{aligned} c_3^*= & {} \frac{c_1 c_2}{\lambda }\text{, } c_4^* = \frac{c_2 A_2 + c_1S_2}{\lambda } - \frac{c_2}{\lambda ^2}\text{, } c_5^* = \frac{c_2}{c_1 \lambda ^3} \\&+\, \frac{S_2(A_2-1)}{\lambda ^2} \text{, } c_6^* = \frac{-1}{\lambda ^3} + \frac{S_2-(A_2c_2/c_1)}{\lambda ^2}. \end{aligned}$$

We concentrate now on the first integral in (39). The first three terms multiplied by $u^{-T/L_1}$ can be integrated in a straightforward manner. The last one can be integrated using a hypergeometric function as described earlier. We thus obtain

$$\begin{aligned} \int _0^\infty I_2 ds = -\beta \left( \frac{c_3^*}{2 L_1 - L_2} + \frac{c_4^*}{L_1 - L_2} - \frac{c_5^*}{L_2} \right) + \frac{\beta c_6^*}{L_2(1+\lambda A_2)}{}_2F_1^3 + \frac{c_1}{L_1 - L_2} - \frac{A_2}{L_2}, \end{aligned}$$

(40)

where $_2F_1^3 = {}_2 F_1\left( 1, -L_2/L_1; 1-L_2/L_1; \frac{-\lambda c_1}{1+ \lambda A_2}\right)$. Therefore, using (30), (38) and (40) we obtain an explicit solution for $M^*$,

$$\begin{aligned} M^*= & {} \frac{k}{L_1-L_2}\left( \frac{S_2}{L_2} - \frac{c_2}{L_1-L_2}-\frac{2}{L_2 \xi _2} + \frac{2 N_1 \xi _1}{L_2(\xi _1 A_2 +1)} {}_2F_1^1+ \frac{2 N_2 \xi _1}{L_2(\xi _2 S_2 +1)} {}_2F_1^2 \right) \nonumber \\&+ \frac{L_1 - A_{22}}{(L_1 - L_2) A_{21}} \left( -\beta \left( \frac{c_3^*}{2 L_1 - L_2}+ \frac{c_4^*}{L_1 - L_2} - \frac{c_5^*}{L_2} \right) +\frac{\beta c_6^*}{L_2(1+\lambda A_2)}{}_2F_1^3\right. \nonumber \\&\left. + \frac{c_1}{L_1 - L_2} - \frac{A_2}{L_2} \right) , \end{aligned}$$

(41)

where the constants and hypergeometric functions were specified earlier.

1.4 Linear and Quadratic Approximation of $M^*$

We take a linear and quadratic portion of $M^*$ in order to more easily ascertain the effects of the system parameters. Using the expansion for the hypergeometric function, and noting that the untransformed stable manifold will intersect (0, 0), we remove all nonlinear terms and rewrite $M^*$ as $y_2 = m y_1$, where m is the slope of the line $y_2$ to obtain

$$\begin{aligned} y_2 = \frac{-y_1}{(L_1 - L_2)^2} c_7 , \end{aligned}$$

(42)

where $c_7$ is the constant

$$\begin{aligned} \begin{aligned} c_7 =&\frac{2 N_1 \xi _1^2 A_{21} k}{(\xi _1 A_2+1)^2} + \frac{2 N_2 \xi _1 \xi _2 k (L_1 -A_{22})}{(\xi _2 S_2 +1)^2} \\&- (L_1 - A_{22}) + \frac{L_1 -A_{22}}{A_{21}} \left( -\beta c_4^{**} + A_{21} + \frac{\beta c_{6}^{**} A_{21}\lambda }{ (1+\lambda A_2)^2 }\right) , \end{aligned} \end{aligned}$$

(43)

where $c_4^{**} = c_4^*/y_1$ and $c_6^{**} = c_6^*/y_1$. Next, since $(S^*, a^*) = \mathbf {x} = \mathbf {Cy}$, we can obtain

$$\begin{aligned} a^* = S^* \left( \frac{(L_1 -L_2)^2 - c_7}{(L_1 - A_{22})(L_1 - L_2)^2+ (-L_2 + A_{22}) c_7} \right) A_{21}. \end{aligned}$$

Keeping all quadratic terms, we obtain

$$\begin{aligned} y_2 = \frac{-(y_1)^2}{(L_1-L_2)(2L_1 - L_2)} c_8 - \frac{-y_1}{(L_1 - L_2)^2} c_7, \end{aligned}$$

(44)

where

$$\begin{aligned} c_8= & {} -2\xi _1\left( \frac{N_1 \xi _1^2kA_{21}^2}{(\xi _1 A_2 +1)^3} +\frac{ N_2 \xi _2^2 (L_1 - A_{22})^2}{(\xi _2 S_2+1)^3} \right) +\nonumber \\&\frac{L_1 - A_{22}}{A_{21}}\left( \frac{-\beta }{\lambda } A_{21} (L_1 - A_{22}) - \frac{\beta c_6^{**}A_{21}^2\lambda ^2}{(1+\lambda A_2)^3} \right) . \end{aligned}$$

The system can be solved for $x = (S^*,a^*)$ using the quadratic formula.

In Fig. 1, we plot $M^*$, $M^*_l$ and $M^*_q$ for specific parameter values in the original coordinate system (S, a). To plot $M^*$ and $M^*_q$ in the original coordinate system (S, a), we input a discreet set of values for $y_1$ and use Eqs. (41) or (44), respectively, to find $y_2$. We can then find $(S^*,a^*) = \mathbf {C^{-1}y}$ and hence $(S,a) = (S^*+S_2,a^*+A_2)$. To plot $M^*_l$ in the original coordinate system (S, a), we directly find $(S^*,a^*)$ using Eq. (42) and translate to (S, a). To simplify notation, when we refer to $M^*$, $M^*_l$ and $M^*_q$ in the main text, we are referring to these functions after coordinate transformation to (S, a).

Appendix 2: Calculation of the Coefficient of Determination, $R^2$

Here, we briefly describe the calculation of the coefficient of determination, or $R^2$, as a measure of the goodness of fit of a regression model to the data. One can find more in depth development and analysis of $R^2$ in Greene (2003).

The coefficient of determination is calculated as follows. Define the dependent variables as $\{y_i\}_{i=1}^n$ (in our case, the $y_i$ correspond to the values of $A_I$ for a given parameter input); taking $\bar{y} = \sum _{i} y_i / n$, we define the total variation in y as

$$\begin{aligned} SS_{t} = \sum _{i} (y_i - \bar{y})^2, \end{aligned}$$

which is just the sum of squared deviations. For each $y_i$, we associate a $b_i$, which is the value of the regression equation at independent variable $x_i$. We similarly define the residual sum of squares as

$$\begin{aligned} SS_{res} = \sum _{i} (y_i - b_i)^2. \end{aligned}$$

The unadjusted $R^2$ is calculated as

$$\begin{aligned} R^2 = 1- \frac{SS_{res}}{SS_{t}}. \end{aligned}$$

Note that if the data perfectly fit the model (i.e., $y_i = b_i$ $\forall i$), then $R^2=1$, indicating that a model with a perfect fit to the data has $R^2=1$, and decreases to 0 as the goodness of fit is reduced. The adjusted $R^2$, $\bar{R^2}$, corrects to an increase in $R^2$ that can occur due to incorporation of additional degrees of freedom into a model and thus should be used in lieu of $R^2$ when comparing goodness of fit between models with different degrees of freedom. It is defined as

$$\begin{aligned} \bar{R^2} = 1-\frac{SS_{res}/(n-d)}{SS_{t}/(n-1)}, \end{aligned}$$

where n is the total number of observations, d is the number of regression coefficients, $n-d$ is the degrees of freedom of $SS_{res}$, and $n-1$ is the degrees of freedom of $SS_{t}$. In the main text, the adjusted $R^2$ is presented without the bar notation.