Abstract
Model comparisons have been widely used to guide intervention strategies to control infectious diseases. Agreement between different models is crucial for providing robust evidence for policy-makers because differences in model properties can influence their predictions. In this study, we compared models generated with two individual-based model simulators for HIV epidemiology in a population with HSV-2. For each model simulator, we constructed four models, starting from a simplified model and stepwise including more model complexity. For the resulting eight models, the predictions of the impact of behavioural interventions on the HIV epidemic in Yaoundé (Cameroon) were compared. The results show that differences in model assumptions and model complexity can influence the size of the predicted impact of the intervention, as well as the predicted qualitative behaviour of the HIV epidemic after the intervention. Moreover, two models that agree in their predictions of the HIV epidemic in absence of an intervention can have different outputs when predicting the impact of interventions. These findings highlight the importance of making more data available for the calibration and validation of epidemiological models.
Introduction
Mathematical modelling has been widely applied to better understand the transmission, treatment and prevention of infectious diseases. The role of mathematical models in understanding the dynamics of the human immunodeficiency virus (HIV) epidemic has been recently reviewed by Geffen & Welte [1]. They present examples of HIV models that were used for estimating the size of the epidemic in specific subpopulations such as the Black population in South Africa and for estimating the impact of interventions such as the use of anti-retrovirals and condoms to reduce HIV and AIDS.
When considering mathematical models, two commonly used types of implementation can be distinguished: compartmental and individual-based models (IBMs). While compartmental models simulate population counts, IBMs (also called agent-based models or micro-simulation models) keep track of the history of each individual in the population separately.
Recent applications of compartmental HIV models include testing the effect of different assumptions for HIV dynamics on the predicted impact of anti-retroviral therapy (ART) in men-having-sex-with-men (MSM) [2], and the study of the influence of concurrent partnerships on HIV dynamics [3]. IBMs have been recently applied to assess the influence of pre-exposure prophylaxis (PrEP) in MSM [4], to evaluate the long-term effect of early ART initiation [5] and to understand the factors underlying the emergence of HIV in humans [6].
In contrast to compartmental models, IBMs allow a high degree of individual heterogeneity [7] (e.g. regarding (sexual) risk behaviour, age demography and individual response to treatment). This is an advantage when heterogeneity matters for the particular question/process that is studied. Because individual heterogeneity is inherent in transmission, prevention and treatment of HIV and other sexually transmitted infections (STI), IBMs are particularly suitable to estimate the most beneficial intervention for specific individuals. Furthermore, IBMs allow for an explicit modelling of the sexual relationships that jointly form the sexual network over which STI/HIV infections are transmitted [8].
Model comparisons are crucial for providing robust evidence for decision-making in public health and policy purposes [9]. To assess uncertainty, it is necessary to study how differences in model properties influence their prediction. Eaton et al. compared ten mathematical models aimed at studying HIV prevalence, incidence and ART [10]. However, eight of these models were compartmental, and the majority of these models did not include the co-factor effects of other STIs. The remaining two models were discrete time IBMs.
This study presents and compares eight models generated with two IBM frameworks for simulating HIV transmission dynamics with STI co-factor effects. The purpose of the comparison was to assess how differences in properties between the eight models influence the prediction of the impact of behavioural interventions on the HIV epidemic. For this purpose we chose the HIV epidemic in Yaoundé (Cameroon) during the period 1980-2005, and calibrated all models to HIV prevalence time series from Yaoundé.
Methods
Individual-based models
In this study, individual-based models developed with the Simpact Cyan 1.0 and StepSyn 1.0 modelling frameworks, were compared. A short description of each modelling framework is given below. More details about differences between the two modelling frameworks and the generation of the sexual networks are provided in the Supplementary Material, paragraph 1. For each modelling framework, four models were generated, starting from a simplified model and incorporating additional complexity.
Simpact Cyan 1.0 modelling framework
Simpact Cyan 1.0 (http://www.simpact.org/) [11] is a freely available framework for developing IBMs to simulate HIV transmission, progression and treatment. Models developed with Simpact Cyan 1.0 are event-driven, which means that the models are not updated at fixed time-intervals, but at every time an event happens, making Simpact Cyan 1.0 a continuous-time simulation modelling framework. Events occur as a result of stochastic event processes, described by hazard functions. The timing of an event is determined using the modified next reaction method (mNRM) [12].
To initialize a model, a number of individuals are generated. The age of each person is drawn from an age distribution based on a population pyramid, and when the age is larger or equal to the sexual debut age, a person is marked as sexually active.
In Simpact Cyan 1.0, the user can specify which events are possible in the simulation, depending on the research question he or she wants to answer. The possible events are: heterosexual and homosexual relationship formation and dissolution, conception and birth, HIV transmission, AIDS and non-AIDS mortality, HIV diagnosis and treatment, HIV treatment dropout, and STI transmission events. Furthermore, events describing the natural history of HIV are implemented in Simpact Cyan 1.0. For each event, the form and the parameters of the hazard function can be modified flexibly. The HIV transmission hazard can be described in terms of an individual’s viral load. In the simplified model in this study, this option as well as age-dependency is turned off. Simpact Cyan 1.0 is implemented in C++ with R [13] and Python [14] interfaces.
StepSyn 1.0 modelling framework
StepSyn 1.0 is an IBM modelling framework that simulates epidemics of STIs, including HIV. It focuses on the epidemiological synergy and interactions between HIV and other STIs. Time is divided into fixed one-week intervals. In this study, we use only weekly time steps to speed up computational time. Formation and dissolution of sexual relationships, and STI and HIV transmission are modelled as stochastic processes. For each STI, life history is explicitly modelled, including several stages, recurrences, genital ulcers, and urethral discharge symptoms. The effects of these symptoms on the HIV transmission probability are explicitly modelled. The parameters for the timing and duration of the STI stages and symptoms, and the co-factor effects of genital ulcers and discharge on HIV transmission are based on a literature review. In the present study, StepSyn 1.0 is parameterized to track only HIV and Herpes simplex virus type 2 (HSV-2), and in the simplified model the latter virus is modelled without explicit stages or recurrences, and with the probability of transmission and co-factor effect on HIV both constant and not dependent on recurrences. Heterosexual relationships can include marital and short-term links, and contacts between commercial sex workers (CSW) and clients. In the present study CSWs are not included. Individuals vary in their tendency to form short-term relationships, so that their number of non-marital partners within a year would follow a power law distribution. The parameters of the latter were derived from behavioural data gathered in the 4 Cities Study [15,16]. StepSyn is implemented in R [13]. StepSyn is not yet freely available but will become so upon publication of a manuscript that describes the modelling framework into more detail (manuscript in preparation).
Models used in this study
The following models were used in this study:
Simpact Cyan 1.0 basic model (Si_Ba). This simplified model assumes no population inflow and no outflow by other causes than AIDS. Furthermore, HIV transmission doesn’t depend on an individual’s viral load. HSV-2 is implemented as a generic STI co-factor.
Simpact Cyan 1.0 model with population inflow and outflow (Si_IO). Birth is implemented as an event related to relationship formation, followed by a conception event, using the default settings of Simpact Cyan 1.0. Non-AIDS mortality is age-dependent and is derived from the 1980 population pyramid of Yaoundé [17]. Other settings are the same as in the Si_Ba model.
Simpact Cyan 1.0 model implementing viral load (Si_VL). HIV transmission depends on an individual’s viral load. Other settings are the same as in the Si_Ba model.
Simpact Cyan 1.0 model implementing population inflow, outflow and viral load (Si_IO_VL). Birth and non-AIDS mortality are implemented in the same way as for the Si_IO model. HIV transmission depends on an individual’s viral load.
StepSyn 1.0 basic model (St_Ba). This simplified model assumes no population inflow (birth, immigration) and no outflow by other causes than AIDS. Furthermore, HSV-2 is modelled without explicit stages or recurrences, and with the probability of transmission and co-factor effect on HIV both constant and not dependent on recurrences.
StepSyn 1.0 model with population inflow and outflow (St_IO). This model implements birth, non-AIDS mortality, immigration and emigration. All of these variables are expressed as a rate. Other settings are the same as in the St_Ba model.
StepSyn 1.0 model with full set of HSV-2 co-factor assumptions (St_RG). HSV-2 life history is explicitly modelled, including several stages, recurrences and genital ulcers. The effects of these symptoms on the HIV transmission probability are explicitly modelled. Other settings are the same as in the St_Ba model.
StepSyn 1.0 model with population inflow, outflow and full set of HSV-2 co-factor assumptions (St_IO_RG). This model implements birth, immigration, non-AIDS mortality and emigration in the same way as the Si_IO model. For HSV-2, the same settings as in the St_RG model are used.
Calibration of the models
For the eight models, HIV transmission parameters were fitted to HIV prevalence data of Yaoundé from 1989 to 1998. A detailed description of the data is provided in the Supplementary Material, paragraph 2. The parameter estimation procedure is described in paragraph 3 of the Supplementary Material.
Furthermore, we adapted the parameters for the STI (HSV-2) co-factor. In major epidemiological reviews of HSV-2 [18–20] the only data about Yaoundé that is mentioned is the data collected by Buvé et al. [21] in the 4 Cities Study, in which the HSV-2 seroprevalence was 50% in females. Thus, we do not have information about the temporal trends of HSV-2 in Yaoundé. However, since the prevalence in Africa seems to have slightly declined between 2003 and 2012 [18] we assumed that the measured prevalence in Yaoundé, in 1997 [21], reflects an epidemic not far from its peak. Accordingly, we adapted the HSV-2 transmission parameters of the eight models so that the seroprevalence of this virus first increases and afterwards stabilizes at approximately 50% for females, corresponding to the HSV-2 prevalence in 1997 described by Buvé et al. [21].
The estimated parameters and other key parameters for each of the eight models are described in the Supplementary Material, paragraph 4.
All models could be calibrated so that they fit the HIV prevalence data between 1989 and 1998, and the HSV-2 prevalence first increases and afterwards stabilizes at approximately 50% for females (see Supplementary Material, paragraph 5).
Validation of the models against data not used for fitting
The eight models were validated against the HIV prevalence in 2004, which was not used for model calibration and has been reported to be 6.0% in males and 10.7% in females [22]. For each of the eight models we checked how well these literature values could be predicted.
Prediction of the impact of behavioural interventions
To investigate the impact of behavioural interventions on the HIV outbreak, the distribution of the number of partners was changed in all of the eight models to study the impact of behavioural change with respect to promiscuity. For the Simpact Cyan 1.0 models, the parameter describing the weight for the number of relationships a person already has was changed. For the StepSyn 1.0 models, we changed the parameter for the proportion of male pending short links (concurrent unstable relationships) that are fulfilled to reach the individual preferred degree (see Supplementary Material, paragraph 1.2.2. for more detail). In the remainder of the paper, we refer to these parameters as “promiscuity parameters”. Because sexual networks for Simpact Cyan 1.0 and StepSyn 1.0 are generated in different ways, the initial sexual network before applying interventions was different between Simpact Cyan 1.0 and StepSyn 1.0 models. Therefore we explored different values for the behavioural parameters for Simpact Cyan 1.0 and StepSyn 1.0 in order to determine a behavioural intervention that shows similar relative changes in the distribution of the number of partners. Details are shown in the Supplementary Material, paragraph 6. We found that changing the promiscuity parameter in Simpact Cyan 1.0 from 0 to −0.05, and changing the promiscuity parameter in StepSyn 1.0 from 1 to 0.7 has a similar effect on the distribution of the number of partners in both modeling frameworks:
the mean number of partners is reduced by 6%;
the 95% percentile is reduced with 1;
the median and 75% percentile are unchanged.
To investigate what would have happened to the outbreak if promiscuity would have been lower from day 1 onwards, we implemented the change of the promiscuity parameter described above in 1980. Second, the intervention was applied in 1990 to study its impact when it would have been implemented at some point in time during the outbreak.
When studying the impact of interventions, only the parameter(s) related to the interventions (in this study only the promiscuity parameter) are changed. This means that we assume that apart from the intervention, no other influences are present. Therefore, the impact of behavioural interventions was also explored during the study period 1980-2005.
Figures 2, 3 and 4 show that the interventions have a similar effect on HIV relative to the Simpact Cyan 1.0 and StepSyn 1.0 basic models (Si_Ba and St_Ba).
Apart from reducing HIV, the interventions also reduce HSV-2 (see Table S25), and the size of the reduction differs among the eight models and between interventions (intervention in 1990: reduction of the median HSV-2 prevalence in 1997: 0.3%-3.6% (females) and 0.2%-2% (males); lower promiscuity in 1980: reduction of 0.7%-5.3% (females) and 0.7%-2.9% (males)).
Results
Validation of the models against data not used for fitting
The median HIV prevalence from 1980 to 2005 (in %), together with the literature value for the HIV prevalence in 2004 [22] is shown in Fig. 1. All models simulating no population inflow and outflow, except the StepSyn 1.0 model with STI life history explicitly modeled (St_RG), largely overestimate the HIV prevalence in 2004 for both females and males. In contrast, the St-RG model underestimates the HIV prevalence in 2004 and the HIV prevalence stabilizes after 2000 and 2004 for females and males respectively. Implementing inflow and outflow considerably improves the prediction of the HIV prevalence for both females and males in 2004. For females the Simpact 1.0 model including inflow, outflow and a VL-dependent HIV transmission hazard (Si_IO_VL) results in the prediction closest to the literature value. For males, the StepSyn 1.0 model with inflow, outflow and STI life history explicitly modeled (St_IO_RG) results in the best prediction of the value in the literature. In general, the StepSyn 1.0 models showed less variation between simulations in predicted HIV prevalence in 2004 than the Simpact 1.0 models (Table 1).
The largest variability among predictions is observed for the Si_IO_VL model, while the St_IO_RG model has the lowest variability.
Prediction of the impact of behavioural interventions
Median HIV prevalence in 2004 for females and males (in %) and range of 100 simulations in case of no intervention, a behavioural intervention in 1990 and a lower promiscuity from 1980 onwards are shown in Table 1. For all models, implementation of a behavioural intervention reduced the median HIV prevalence and the range (Table 1 and Fig. 4). After implementing a behavioural intervention in 1990, a reduced but still increasing HIV epidemic was observed for all models except the StepSyn 1.0 model with STI life history explicitly modeled (St_RG) which shows a decreased epidemic after 1998 (Fig. 2). In case of a lower promiscuity from 1980 onwards (Fig. 3), similar trends are observed but the reduction in HIV prevalence is considerably larger than when applying the intervention in 1990. For the two models that resulted in the best predictions of the HIV prevalence in 2004 in case of no intervention (Si_IO_VL and St_IO_RG), the predicted effect of a behavioral intervention was larger for the St_IO_RG model than for the Si_IO_VL model (Table 1). While the Si_IO_VL model predicts a 2.2% and 1.5% reduction in median HIV prevalence in 2004 for females and males respectively, in case a behavioural intervention was implemented in 1990, the St_IO_RG model predicts a 4.7% and 2.6% reduction for females and males respectively. In case of a lower promiscuity from the start of the epidemic (1980), Si_IO_VL predicts a 4.1% and 2.6% reduction in HIV prevalence in 2004 for females and males respectively, while St_IO_RG model predicts a 7.1% and 4.2% reduction in females and males respectively.
Discussion
In this study, eight individual-based models, generated with the simulators Simpact Cyan 1.0 and StepSyn 1.0 were compared in terms of their prediction of the impact of behavioural interventions on the course of the HIV epidemic in Yaoundé, Cameroon.
All models could be fitted equally well to the calibration targets in Table S3 (see Figure S11), which shows that similar HIV prevalence curves can be simulated using different model assumptions and transmission parameters.
After calibration of the models, they were first validated against data not used for fitting. For each of the two modelling frameworks, the model that implements population inflow and outflow together with a detailed description of HIV transmission (i.e. the Si_IO_VL and St_IO_RG models) shows the best prediction of the HIV prevalence for males and females in 2004 (Fig. 1).
In general, a model implementing inflow and outflow better predicts HIV prevalence at a time point not used for fitting than its counterpart without inflow and outflow (Fig. 1). Remarkably, while models with no inflow and outflow implementing a generic STI co-factor effect largely overestimated the HIV prevalence in 2004, the model implementing HSV-2 life history and its effect on HIV in detail, and no inflow and outflow (the St_RG model) predicted a lower HIV prevalence for both males and females than what was reported in the literature. Moreover, the effect of implementing inflow and outflow on the model predictions was smaller for the model implementing HSV-2 in detail than for the other models.
The predictions of the impact of behavioural interventions for the eight models are similar during the first five years after the intervention, but show large differences on the long term (see Fig. 2 and 3). Similar conclusions could be drawn by Eaton et al [23] when comparing mathematical models predicting the impact of anti-retroviral therapy (ART).
Although the Si_IO_VL and St_IO_RG models predict the HIV prevalence in 2004 equally well, the two models behave differently when implementing the effect of a behavioural intervention. After an intervention in 1990 the median HIV prevalences for females and males in 2004 were reduced with 2.2% and 1.5% respectively for the Si_IO_VL model (see Table 1), while the St_IO_RG model predicts a reduction of 4.7% for females and 2.6% for males. In case of a lower promiscuity from 1980 onwards, a reduction in HIV prevalence of 4.1% for females and 2.6% for males were predicted by the Si_IO_VL model. For the St_IO_RG model, the corresponding values were 7.1% and 4.2%. More data has to become available to further explore which of these two models provides the most reliable predictions of behavioural interventions.
Seven of the eight models predicted that the HIV epidemic would still increase during the period 1980-2005, although at a lower rate, after applying a behavioural intervention (see Fig. 2 and 3). Only the St_RG model predicted that the HIV epidemic will decrease after reaching a peak in 1997.
This study shows that differences in model assumptions and model complexity can considerably influence their predictions of the impact of behavioural interventions. Hontelez et al. [24] reported similar conclusions after stepwise inclusion of model complexity in a model for predicting the impact of a universal test and treat (UTT) intervention, and concluded that sufficient detail is necessary to make accurate predictions.
Apart from differences in the size of the impact of a behavioural intervention on the HIV epidemic, we also detected differences in qualitative behaviour between simulations generated with different models.
Furthermore, we showed that even when there is agreement between two models in their prediction of a future time point not used for fitting, they can have different outputs when simulating the impact of interventions. These findings highlight the importance of making more data available for both the calibration and validation of epidemiological models that aim to inform decisions made by policy-makers.
Conflict of interest
The authors have declared that no competing interests exist.
Acknowledgements
The research conducted by DMH and NH in this study was funded by the Fonds Wetenschappelijk Onderzoek - Vlaanderen (Research Foundation – Flanders; FWO, http://www.fwo.be/en/) (Grant agreements G0E8416N and G0B2317N). The research done by JDS and AMV in this study has been supported in part by grant G.0692.14 and G0B2317N, funded by the FWO, Belgium. PL was supported by a PhD grant of the FWO (1S31916N). WD was supported by a postdoctoral fellowship from FWO (12L5816N). Research done by VM in this study has been funded by the ELTE Institutional Excellence Program supported by the National Research, Development and Innovation Office (NKFIH-1157-8/2019-DT). VM was also supported by the ÚNKP-18-4 New National Excellence Program of the Hungarian Ministry of Human Capacities and by a Bolyai János Research Fellowship of the Hungarian Academy of Sciences. The authors gratefully acknowledge support from the FWO Scientific Research Community on Network Statistics for Sexually Transmitted Diseases Epidemiology. The computational resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government – department EWI.
Appendix.
See supplementary_material.pdf