Abstract
Detecting the targets of drugs and other molecules in intact cellular contexts is a major objective in drug discovery and in biology more broadly. Thermal proteome profiling (TPP) pursues this aim at proteome-wide scale by inferring target engagement from its effects on temperature-dependent protein denaturation. However, a key challenge of TPP is the statistical analysis of the measured melting curves with controlled false discovery rates at high proteome coverage and detection power. We present non-parametric analysis of response curves (NPARC), a statistical method for TPP based on functional data analysis and nonlinear regression. We evaluate NPARC on five independent TPP datasets and observe that it is able to detect subtle changes in any region of the melting curves, reliably detects the known targets, and outperforms a melting point-centric, single-parameter fitting approach in terms of specificity and sensitivity. NPARC can be combined with established analysis of variance (ANOVA) statistics and enables flexible, factorial experimental designs and replication levels. To facilitate access to a wide range of users, a freely available software implementation of NPARC is provided.
Introduction
Determining the cellular interaction partners of drugs and other small molecules remains a key challenge [1, 2, 3, 4]. In drug research, better assays to detect targets (and off-targets) would provide valuable information on drugs’ mechanisms of action, reveal potential reasons for side effects, and elucidate avenues for drug repurposing. More broadly, in cell biology basic research, the dynamical landscape of binding partners of metabolites, messengers or chemical probes contains much uncharted territory. Thermal proteome profiling (TPP) addresses these needs by screening for protein targets of drugs or small molecules in living cells on a proteome-wide scale [5, 6]. TPP combines multiplexed quantitative mass spectrometry with the cellular thermal shift assay (CETSA) [7], which identifies binding events from shifts in protein thermostability (see Figure S1 for a detailed explanation). A typical TPP experiment generates temperature dependent abundance measurements for a large part of the cellular proteome. Drug binding proteins can then be inferred by comparing the melting curves of proteins between samples treated with drug and vehicle (negative control without drug).
Applications of TPP successfully identified previously unknown protein-ligand in-teractions [5], protein complexes [8] and downstream effects of drugs in signaling networks [6, 9, 10, 11] in human cells. Recently it has also been extended to study drug resistance in bacteria [12] and targets of antimalarial drugs in plasmodium [13]. There is urgent interest in further advancing its component technologies, including experimental and computational aspects, in order to maximize its biological discovery potential [14, 15, 16, 17, 18, 19].
The central computational task in TPP data analysis is the comparison of the temperature dependent abundance measurements—which can be visualized as melting curves—for each protein with and without (or with various concentrations of) drug. The aim is to detect changes in thermostability from statistically significant changes in the melting curves.
A naïve approach is to summarize each curve into a single parameter, such as the melting point (Tm), which is defined as the temperature of half-maximum relative abundance (horizontal line in Figure 1A). Its value is estimated by fitting a parametric model separately for the control and treatment conditions and comparing the estimates. Statistical significance is assessed using replicates and hypothesis testing, such as a t- or z-test. While the approach has delivered valid and important results [5, 6, 20, 21], we will see in the following that it tends to lead to needlessly high rates of false negatives. There are three main reasons for that: first, drug-induced effects on thermostability do not always imply significant shifts in Tm (Figure 1B-C). Second, the true Tm of a protein can lie outside of the measured temperature range, which impairs its estimation (Figure 1D). Both scenarios can result in important targets being missed in the analysis (Figure 1E). The third reason is a more subtle statistical one: hypothesis tests using only the point estimates of Tm do not take into account goodness-of-fit of the parametric model or the confidence range of the estimates. Thus, important information is ignored, which statistically leads to loss of power.
Here, we propose an alternative approach that compares whole curves instead of summary parameters and does not rely on Tm estimation. The method, non-parametric analysis of response curves (NPARC), is based on a branch of statistical data analysis that works on continuous functions rather than individual numbers, termed functional data analysis [22]. It considers the measured melting curves as samples from an underlying stochastic process with a smooth mean function—which can be modelled parametrically or non-parametrically [23]—and constructs its hypothesis tests directly on these samples. NPARC’s F-statistic uses a more exible model that makes fewer assumptions on the data than Tm-estimation, is computationally more stable, and it directly uses the information from replicates. As a consequence, reliable estimates of the null distribution of this statistic can be obtained, it shows higher sensitivity for small but reproducible effects, and failures due to model misspecification or outliers are reduced. This increases proteome coverage, which can make the difference between missing or detecting an important drug target.
We demonstrate NPARC on the five published datasets introduced in Table 1. We also compare its results to those of the Tm-based method used by [6]. Three of the experiments used the cancer drugs panobinostat or dasatinib in different concentrations, one investigated the effects of the high-affinity, ATP-competitive pan-kinase inhibitor staurosporine, one the cellular metabolite ATP. While the cancer drugs interact with limited sets of proteins, the two other compounds are promiscuous binders and affect the thermostability of a large fraction of the cellular proteome.
Experimental Procedures
Datasets and preprocessing
Five TPP datasets (Table 1) were obtained from the supplements of the respective publications. Each dataset contained relative abundance measurements per protein and temperature which had been scaled to the value measured at 37° C (the lowest of the ten temperatures assayed) and subjected to the global normalization procedure described by Savitski et al. [5]. Only proteins quantified with at least one unique peptide in each of two replicates of the vehicle and compound treated conditions were included in the analysis; the resulting proteome coverages are listed in Table 1.
Curation of lists of expected targets
Lists of expected protein targets for the pan-kinase inhibitor staurosporine and ATP were obtained from Gene Ontology Consortium annotations via the Bioconductor annotation packages AnnotationDbi (version 1.36.2), org.Hs.eg.db (version 3.4.0) and GO.db (version 3.4.0). Terms and numbers of annotated proteins are shown in Table 2.
Mathematical model
NPARC is based on fitting two competing models to the data, a null model and an alternative model. The null model states that the relative protein abundance at temperature t (given in °C) is explained by a single smooth function µ0 (t) irrespective of the treatment condition (Figure 2A). The alternative model posits two condition-specific functions: µT (t) for the treatment condition and µV (t) for the vehicle condition (Figure 2B). Deviations between observed data and fitted model are referred to as residuals, and the sum of squared residuals (RSS) serves as an indicator of each model’s goodness-of-fit. We then compute where xt,i,c is the measured value at temperature t for experimental replicate i and condition c ∈ {V, T}, and the summations extend over all temperatures, replicates, and conditions.
Choice of the mean function
The mean functions µ0 (t), µT (t) and µV (t) are each chosen from the same space of smooth functions f: ℝ+ → [0, 1] spanned by the three parameters a, b ∈ ℝ, f∞ ∈ [0, 1] and the prescription
The shape of these functions is sigmoid, and the functional form (3) can be motivated by simplifying protein thermodynamics considerations [5]. The mean functions and the RSS values are computed separately from the data for each protein. In order not to overburden the notation, we omit the protein indices.
Hypothesis test statistic
To discriminate between null and alternative models, we compute the F-statistic with d2 /1 > 0 defined as below. F quantifies the relative reduction in residuals from null to alternative model. While F is by definition always positive, it will be small for proteins not affected by the treatment, while a high value of F indicates a reproducible change in thermostability.
Null distribution
To compute a p-value from a value of the F-statistic (4), we need its null distribution, i.e., its statistical distribution if the data generating process is described by a common mean function µ0 (t). If the residuals were independent and identically normal distributed, this distribution would be given by an analytical formula, namely that of the F (d1, d2)-distribution with parameters d1, d2 > 0, and these parameters—sometimes called degrees of freedom—would be explicitly given from the number of measurements and number of model parameters that go into the computation of RSS0 and RSS1. In practice, this is not the case, since the residuals are heteroscedastic (i.e. have different variances at different temperatures) and correlated. However, the family of F (d1, d2)-distributions is quite flexible, and we can approximate the distribution of the F-statistic (4) on data occurring in practice with an F (d1, d2)-distribution with different “effective degrees of freedom” d1, d2. To this end, we separately approximate the numerator and denominator of F as and use the fact that the ratio of two χ2-distributed random variables χ2 (d1), χ2 (d2) has an F (d1, d2)-distribution [24]. The scale parameter and the effective degrees of freedom d1 and d2 are estimated from the the empirical distributions—across proteins—of RSS0 and RSS1. Thus, we assume that , are the same for all proteins. In particular, we estimate from the moments of RSS1 − RSS0 as where mean and variance are computed across proteins on the observed values of RSS0 – RSS1 (see Supplementary Methods for details). Then, d1 and d2 are obtained by numerical optimization of the likelihoods for models (5) and (6) using the fitdistr function of the R package MASS [25].
p-values
For each protein, a p-value is computed from its F-statistic and the cumulative F-distribution with parameters d1, d2 as described above. The multiset of p-values across all proteins is corrected for multiple testing with the method of Benjamini and Hochberg [26]. The outcome of such an analysis is exemplarily shown in Figure 2C.
Results
Application to panobinostat
We assessed the ability of NPARC to detect drug targets on a dataset on panobinostat (Table 1). Panobinostat is a broad-spectrum histone deacetylase (HDAC) inhibitor known to interact with HDAC1, HDAC2, HDAC6, HDAC8, HDAC10, and tetratricopeptide repeat protein 38 (TTC38) [6].
Out of 3649 proteins reproducibly quantified across both biological replicates in both treatment conditions, NPARC yielded 16 proteins with Benjamini-Hochberg adjusted p-values ≤ 0.01. They contained the expected HDAC targets (Figure 3A-as well as TTC38, the histone proteins H2AFV or H2AFZ (the two variants could not be distinguished by mass spectrometry), and zinc finger FYVE domain-containing protein 28 (ZFYVE28) (Figure 3F-H). These proteins were previously identified as direct or indirect targets of panobinostat [6, 11]. In addition, eight more proteins were detected for which no direct or indirect interactions with panobinostat have been described (Figure S3). They reached statistical significance because they either showed effect sizes comparable to known panobinostat targets, or more subtle but highly reproducible changes in a similar strength to those already described for dasatinib target BTK (Figure 1B). We reanalyzed the more recent 2D-TPP dataset of short-term (15 min) panobinostat-treatment of HepG2 cells [11] for these proteins. All of them were identified and quantified at sufficient peptide coverage, but none of them showed stabilization. We thus conclude that the additionally found proteins are likely not direct binders of panobinostat, but rather indirect effects, like altered protein-protein interactions or post-translational modifications. The longer (5 hours) incubation time of the assay used to generate the panobinostat dataset in Table 1 makes it more sensitive to such effects.
Beyond two-group comparisons
Since NPARC is based on analysis of variance (ANOVA), it admits experimental designs in which the covariate has multiple levels. An example is the dataset for the BCR-ABL inhibitor dasatinib, which comprises measurements on cells treated at two different concentrations as well as untreated cells. NPARC successfully identified known targets of dasatinib (Figure S4).
Replicate agreement and model fit diagnostics
Application of the Tm-based approach by [6] to the panobinostat data failed to detect HDAC1 and HDAC2. This was because the data for these proteins had relatively high variance in the drug-treated condition, as is visible in Figure 3A-B. This led to their exclusion according to one of the data quality filter criteria of that method (Table 3), namely the criterion that asks for sufficiently high coefficients of determination (R2). In contrast, a better and statistically sound trade-off between variability and effect size is an integral part of NPARC, and does not require an ad hoc filter criterion.
To further assess the price of the various filter criteria of the Tm-based approach by [6], we tabulated the numbers of proteins affected by them in each of the five datasets. These proteins would, in principle, not be detectable by that method, no matter how strong the effect. Their numbers amounted to 14-27% of the total numbers of proteins for which melting points could be determined in both replicates (Figure 1E and Table 4 and to 21-32% of all proteins irrespective of melting point availability (Figure S2). In contrast, the F-test of NPARC could be applied to all proteins irrespective of these or similar criteria, a fact which contributed to the increased protein coverage and sensitivity of NPARC.
Effects beyond those on the melting point
Many of the proteins detected by NPARC displayed reproducible changes in curve shape, while their Tm-shifts were small, and not considered significant by the Tm-based approach (Figure 4). An example is the effect of staurosporine on protein kinase C beta (PRKCB), shown in Figure 1C. PRKCB is part of the PKC family, whose members were the first reported staurosporine targets [27, 4] and also exhibit similar characteristics (Figure S5).
Further examples include the effects of staurosporine on RanGTP binding tRNA export receptor exportin-T (XPOT) and two members of the p38 MAPK signaling pathway: Mitogen-activated protein kinase 14 (MAPK14) and MAP kinase-activated protein kinase 2 (MAPKAPK2) (Figure 4); and the effect of dasatinib on Bruton tyrosine kinase (BTK), an important drug target in B-cell leukemia (Figure 1B).
Missing melting point estimates
For highly thermostable proteins, the Tm in one or more of the treatment conditions can be outside of the tested temperature range of a TPP experiment (Figure 1E). One example is NAD(P)H quinone dehydrogenase 2 (NQO2), a cytosolic flavoprotein and a common off-target of kinase inhibitors [28, 29, 30]. In concordance with previous CETSA studies that found NQO2 to be highly stable [31], we observed denaturation only beginning at 67 °C (Figure 1D). Staurosporine treatment further stabilized NQO2 to an extent that it showed no sign of melting in the tested temperature range. The Tm-based approach by [6] will discard such proteins, in order to avoid potential problems from extrapolation of the fit beyond the measured temperature range. In contrast, the functional data analysis approach of NPARC is able to detect changes in any part of the melting curves, without reference to a single point such as Tm.
Sensitivity and specificity
So far, we have described increased sensitivity of NPARC, i.e., its ability to detect more true targets. However, this is only useful if at the same time specificity is maintained, i.e., if false positive detection remains under control. To compare these performance characteristics between NPARC and the Tm-based approach, we computed pseudo receiver operator characteristic (ROC) curves for each of these methods on the staurosporine data and the ATP data, using as pseudo ground truth lists of expected targets from Gene Ontology annotation (Table 2). Here, the term pseudo refers to the fact that these target lists, and hence the ROC curves, are only approximations of the truth; however, the relative ranking of two methods in such a pseudo-ROC comparison is likely to be faithful even in the presence of such approximation error [32].
Figure 5 shows the results of NPARC on both datasets, as well as those of the z- test of the Tm-based approach by [6] applied to the individual replicates (displayed as a continuous lines parameterized by the z-cutoff), and those of the full procedure of [6] (shown by isolated points, due to its single, fixed cutoff). On the staurosporine data, the full procedure of [6] performs close to NPARC. For the individual z-tests, as well as overall on the ATP data, NPARC shows superior performance. Given that the decision rule set of [6] (listed in Table 5) and its cutoff parameters were developed and tuned partly on the staurosporine data, these results indicate that NPARC has fewer “fudge parameters” and is likely to be superior in applications to new datasets.
Discussion
Thermal proteome profiling offers the possibility to comprehensively characterize ligand-protein interactions on a proteome-wide scale in living cells. However, the method poses the analytical challenge of how to identify statistically significant shifts in thermostability amongst thousands of measurements.
To address this challenge, we introduced a functional data analysis approach to test for treatment effects by comparing competing models by their goodness-of-fit. This enables detection of treatment effects even if a (de-)stabilization of a protein is not captured by a single summary parameter like the Tm. The presented method is based on a sound statistical foundation and does not rely on hard-to-choose cutoff or tuning parameters. We showed that our method compares favourably to previous approaches with respect to sensitivity and specificity for several exemplary datasets, including ones with specific and ones with promiscuous binders.
The approach fits into the framework of analysis of variance (ANOVA) or linear models, and can thus be extended to experimental designs more complex than treatment-control comparisons, such as multiple levels (e.g. drug concentrations) per covariate, multiple covariates and interactions.
The suggested framework is flexible with regard to the mean function used to represent the melting behavior and can be adapted to the particular biological process of interest. To represent nonlinear relationships, approaches include locally linear regression [33], spline regression [34, 35] and nonlinear parametric regression. Here, we chose the latter as it incorporates a priori knowledge about the data and thus has favourable estimation efficiency. For example, sigmoid curves have horizontal asymptotes at both sides of the temperature range. In contrast, splines and local regression tend to overfit data near the boundaries of the observation range.
In a cellular environment we occasionally observe non-sigmoid melting curves for subsets of proteins. One possible reason is the presence of protein subpopulations each with distinct melting curves [16]. For example, the formation of protein complexes, the binding to other molecules, or the localization in cellular compartments can lead to deviations from the idealized sigmoid melting curve expected from the same protein in purified form. Our model currently does not account for such systematic and reproducible shape deviations. This could be adapted in future work by adding a low-parametric systematic modification to the sigmoid mean function. We have considered CETSA experimental designs, where the temperature is the major experimental variable and drug concentration is either zero or a chosen value. It appears relatively straightforward to extend NPARC to the isothermal dose response (ITDR) design [6, 7] where temperature is held constant and the drug concentration is varied across a range of values. A further extension of interest would be to 2D-TPP [11] where both factors are changed.
We employ the same “average” null distribution for all proteins, which we obtain by estimating its parameters (d1, d2, σ0) from the distributions of residuals across all proteins. It is conceivable that determining null distributions in a protein-dependent manner, for instance by stratification, could increase the overall power of the method.
The here presented approach is likely to increase the accuracy of profiling protein-ligand interactions in living cells. To facilitate access to a wide range of users we have implemented NPARC in a freely available R workflow [36].
Additional Files
The following Figures and Tables can be found in the Supplementary Material.
Additional file 1 — Supplementary Methods
Detailed description of the fitting procedures for the scaling parameter, melting points, and mean functions of the model.
Additional file 2 — Table S1
Spreadsheet containing the results of the NPARC approach and of the Tm-based approach for all datasets listed in Table 1.
Additional file 3 — Supplementary Figures S1-S5
PDF containing Supplementary Figures S1-S5.
Additional file 4 — Supplementary Figure S6
All proteins detected by the NPARC approach with Benjamini-Hochberg adjusted F-test p-values ≤ 0.01 in the staurosporine data.
Additional file 5 — Supplementary Figure S7
All proteins detected by the NPARC approach with Benjamini-Hochberg adjusted F-test p-values ≤ 0.01 in the ATP data.
Software and data availability
The TPP-TR experiments based on staurosporine-, panobinostat-, ATP-, and dasatinib treatments are included in the supplementary materials of references [5, 9, 6]. All results generated from this data are provided in the supplementary material attached to this work. A vignette describing the workflow and relevant code to reproduce the results is available for download [36]. An open-source R/Bioconductor package NPARC will soon be released.
Competing interests
HF, MS, DC and MB are employees and/or shareholders of GlaxoSmithKline.
Authors’ contributions
KB, DC and HF contributed equally. KB, DC, HF, SA, MS and WH developed the model. KB and DC implemented the software package and the analytical computations. KB, DC, HF, NK, MS, MB and WH interpreted the results. KB, DC, HF, NK, SA and WH wrote the manuscript. All authors read and approved the final manuscript.
Acknowledgments
We would like to thank Sindhuja Sridharan (EMBL Heidelberg) for help with Figure preparation. We thank EMBL and GlaxoSmithKline for supporting the work. KB is supported by a Cambridge Cancer Centre studentship. SA is funded by the Deutsche Forschungsgemeinschaft, SFB 1036. WH acknowledges funding from the European Commission’s H2020 Programme, Collaborative research project SOUND (Grant Agreement no 633974).
Footnotes
↵† Equal contributor
Refined descriptions of the results. Revised Figures S6 and S7.
- Abbreviations
- CETSA
- Cellular thermal shift assay
- FDR
- False discovery rate
- H0
- Null hypothesis
- H1
- Alternative hypothesis
- NPARC
- Non-parametric analysis of response curves
- ROC
- Receiver operating characteristic
- RSS
- Residual sum of squares
- Tm
- Melting point
- TPP
- Thermal proteome profiling