Improved antibody structure prediction by deep learning of side chain conformations

Antibody engineering is becoming increasingly popular in medicine for the development of diagnostics and immunotherapies. Antibody function relies largely on the recognition and binding of antigenic epitopes via the loops in the complementarity determining regions. Hence, accurate high-resolution modeling of these loops is essential for effective antibody engineering and design. Deep learning methods have previously been shown to effectively predict antibody backbone structures described as a set of inter-residue distances and orientations. However, antigen binding is also dependent on the specific conformations of surface side chains. To address this shortcoming, we created DeepSCAb: a deep learning method that predicts inter-residue geometries as well as side chain dihedrals of the antibody variable fragment. The network requires only sequence as input, rendering it particularly useful for antibodies without any known backbone conformations. Rotamer predictions use an interpretable self-attention layer, which learns to identify structurally conserved anchor positions across several species. We evaluate the performance of our model for discriminating near-native structures from sets of decoys and find that DeepSCAb outperforms similar methods lacking side chain context. When compared to alternative rotamer repacking methods, which require an input backbone structure, DeepSCAb predicts side chain conformations competitively. Our findings suggest that DeepSCAb improves antibody structure prediction with accurate side chain modeling and is adaptable to applications in docking of antibody-antigen complexes and design of new therapeutic antibody sequences.


Introduction
Antibodies are specialized proteins that play a crucial role in the detection and destruction of pathogens. The binding and specificity of antibodies are largely determined by the complementarity determining regions (CDRs) that consist of three loops in the light chain and three loops in the heavy chain [1]. Structural diversity is largely achieved by the third loop in the heavy chain, which determines many antigen binding properties. Additionally, CDR H3 does not have a canonical fold like the other loops [2], making it challenging to model [3][4][5]. Currently, engineering of new antibodies is hindered by accurate prediction of the CDR H3 loop, including the corresponding side chains for docking applications. Prediction of side chains is a critical component of structure prediction and protein design [6]. The surface of the antibody CDR loops including the side chains play an important role in antigen recognition [7].
There has been a growing interest in effective design of new antibodies since they are commonly used in biotherapeutics [8]. Antibody structure determination via techniques like X-ray crystallography and NMR is challenging and time-consuming. Machine learning methods improve overall structure prediction and docking [9]. Recently, highly accurate structure prediction models have been proposed for proteins in general [10][11][12] and for antibodies [13][14][15][16]. The performance of AlphaFold2 has been impressively accurate in the recent CASP14 experiment and surpassed most other protein structure prediction methods proposed to date [11]. Unlike the other deep learning-based methods, AlphaFold2 predicts all side chain rotamers in addition to the protein backbone. While current antibody prediction methods utilizing deep learning do not directly predict side chains, they are all able to predict the backbones with high accuracy. Hence, a next step towards the advancement of antibody modeling and engineering is the accurate prediction of side chains to improve overall structure prediction and docking.
Presently, there are successful methods for rotamer predictions that rely on calculating the probability of a χ angle as a function of backbone torsion angles. For instance, SCWRL4 uses backbone-dependent libraries to calculate rotamer frequencies based on kernel density estimates and kernel regressions [17]. Antibody-specific methods like PEARS capture rotameric preferences based on the immunogenetics numbering scheme to restrict possible side chain conformations in the sample space based on positional information [18]. Both SCWRL4 and PEARS require the antibody sequence and backbone structure to generate side chain predictions. They repack the side chains onto the provided backbone, and their performance generally declines when the input is not the crystal backbone. To address these limitations, we propose DeepSCAb (deep side-chain antibody), a deep neural network that predicts full F V structures, including side chain conformations from only the amino acid sequence.

Training dataset
The training dataset for our model was curated from SAbDab, a database of antibody structures from the Protein Data Bank [19]. We enforced a threshold of 99% sequence identity as well as a resolution cutoff of 3Å for high quality data. We removed targets belonging to the RosettaAntibody benchmark set [20] to evaluate model performance, which resulted in a total of 1433 antibody structures that we used to train and validate our network. module consists of a 3 block 1D ResNet and a 25 block 2D ResNet. The one-hot encoded antibody sequence of dimension L × 21 is taken as input to pass through the module via the 1D ResNet, where L represents the number of residues in the sequence and 21 represents each amino acid type with the addition of a chain delimiter. The 1D ResNet begins with a 1D convolution that projects the input features up to L × 64, followed by three 1D ResNet blocks (two 1D convolution with a kernel size of 17) that maintain dimensionality. The output of the 1D ResNet is then transformed to pairwise by redundantly expanding the L × 32 tensor to an L × L × 64 tensor. Next, this tensor passes through 25 blocks in the 2D ResNet that maintain dimensionality with two 2D convolutions and kernel size of 5 × 5. The resulting tensors are converted to pairwise probability distributions over C β distance, d, the orientation dihedrals ω and θ, and the planar angle φ. The inter-residue module is trained as described for DeepH3 [13].
The rotamer module takes as input the inter-residue features. The tensors of dimension L × L × 64 resulting from the 2D ResNet are transformed to sequential by stacking of rows and columns to obtain a final dimension of L × 128. The rotamer module contains a multi-head attention layer of 1 block with 8 parallel attention heads and a feedforward dimension of 512. The self-attention layer outputs L × 128 tensors, which then pass through a 1D convolution with kernel size of 5. The tensors are converted to rotamer probability distributions that are conditionally predicted for each χ dihedral using softmax. For example, χ 1 is an input to χ 2 , and χ 1 through χ 4 are inputs to χ 5 . The predicted rotamers are added back into the inter-residue module: the rotamer tensors are stacked onto the pairwise before the final 2D convolution to update the d, ω, θ, and φ outputs.
Distances are discretized into 36 equal-sized bins in the range of 0 to 18Å. All dihedral outputs of the network are discretized into 36 equal-sized bins in the range of −180°to 180°with the exception of χ 1 . The χ 1 dihedral is discretized into 36 non-uniform bins, with 6 bins of 30°and 30 bins of 6°. The small bins are centered around −60°, 60°, and 180°, consistent with observed conformational isomers. The planar angle φ is discretized into 36 equal-sized bins with range 0 to 180°. Pairwise dihedrals are not calculated for glycine residues due to the absence of a C β atom. Side chain dihedrals were not calculated for glycine and alanine residues due to the absence of a C γ atom and for proline residues due to its non-rotameric nature.
Categorical cross-entropy loss is calculated for each output, where the pairwise losses are summed with equal weight and the rotamer losses are scaled based on each dihedral's frequency of observation: i.e., χ 5 rotamers are much less frequent than χ 1 . The Adam optimizer is used with a learning rate of 0.001. We trained five models on random 95/5 training/validation splits and averaged over model predictions to generate potentials for downstream applications. DeepSCAb models were trained on one NVIDIA K80 GPU, which required approximately 100 hours for 120 epochs of training.

Side chain only predictions
To investigate the effect of inter-residual predictions on rotameric predictions, we designed a side chain only network as a control. The control network takes as input the one-hot encoded antibody sequence, which passes through a 3 block 1D ResNet. The remaining architecture of the control network as well as its training process is similar to the rotamer module of DeepSCAb (SFigure 1). However, there are differences in dimension due to the 1D ResNet returning an L × 32 tensor. The control network models were trained on one NVIDIA K80 GPU, which required 10 hours for 20 epochs of training. We adopted a shorter training process for the control network as the models tended to overfit after 20 epochs.

Transformer encoder attention layer
The rotamer module contains a transformer encoder layer that adds the capacity to aggregate information over the entire sequence (SFigure 2). We tuned the number of parallel attention heads, the feedforward dimension, and the number of blocks according to validation loss during training. We found that 8 attention heads outperformed 16, feedforward with a dimension of 512 outperformed 1,024 and 2,048, and one block of attention performed identically to two. We further experimented with adding a sinusoidal positional embedding prior to the self-attention layer and obtained identical results, implying that the convolutions in our network contain sufficient information on the order of input elements, rendering positional encoding unnecessary [21].

Interpreting the attention layer
In our interpretation of the rotamer attention, we take into consideration only one model out of the five that were trained on random training/validation splits. We do not report an average over the attention matrices from multiple models since they vary amongst themselves (SFigure 3). Nevertheless, the properties of attention are conserved across individual models.
We utilize a selected subset of the independent test set to display the most variation across the highly-attended positions as well as the corresponding residue types. This subset consists of the following human PDBs: 1JFQ, 1MFA, 2VXV, 3E8U, 3GIZ, 3HC4, 3LIZ, 3MXW, 3OZ9, and 4NZU.

Modeling side chains with DeepSCAb in Rosetta
DeepSCAb generates constraints that are utilized for the prediction of an antibody structure. Discrete potentials are converted to continuous function via the built-in Rosetta spline function. The constraints include all 9 geometries, namely d, ω, θ, φ, χ 1 , χ 2 , χ 3 , χ 4 , and χ 5 . The ConstraintSetMover in Rosetta applies these constraints onto the native pose and then the PackRotamersMover models side chain structures. We chose the standard ref2015 full-atom score function with a weight of 1.0 for all constraints. This protocol can repack side chains on any backbone structure with DeepSCAb predictions.

Side chain predictions using alternative methods
To assess the side chain prediction accuracy against relative solvent accessible surface area, we compare DeepSCAb to three alternative methods: PEARS, SCWRL4, and Rosetta. For each alternative method, we provide the backbones of the benchmark targets and their sequences. PEARS utilizes antibody-specific rotamer libraries and assigns rotamers based on the IMGT numbering scheme. We generated predictions from PEARS using the publicly available server [18]. SCWRL4 generates χ kernel density estimates based on backbone-dependent rotamer libraries by minimizing the conformational energies for each residue. We generated SCWRL4 predictions using the SCWRL4.0 algorithm [17]. Rosetta predictions were generated using default packing protocol (PackRotamersMover ) and the ref2015 energy function.

Data and Code Availability
The source code for DeepSCAb, as well as pretrained models, will be made available prior to publication. The structures predicted by DeepSCAb and alternative methods for benchmarking will be made available prior to publication.

Overview of the method
Our deep learning method for antibody structure prediction consists of inter-residue and rotamer modules. We trained DeepSCAb to predict antibody backbones as inter-residual distance and orientations. Then, we simultaneously trained the model to predict the side chain conformations using an attention layer. The pairwise and rotamer probability distributions from DeepSCAb predictions were used for structure realization and packing of the side chains using Rosetta.
DeepSCAb predicts inter-residue and side chain orientations from sequence.
DeepSCAb is a neural network that only requires an antibody sequence to predict full F V structures including side chain geometries ( Figure 1A). The combined sequences of the antibody heavy and light chains are inputted as a one-hot encoding to initially pass through the inter-residue module. The architecture of this module is similar to our previous method for CDR H3 loop structure prediction, DeepH3 ( Figure 1B) [13]. The network is pretrained to predict pairwise geometries, such as d, ω, θ, and φ. We then feed the penultimate outputs into the rotamer module for the prediction of side chain conformations, such as χ 1 , χ 2 , χ 3 , χ 4 , and χ 5 . Lastly, these tensors are used to update the inter-residue module to obtain the final outputs.
We calculate pairwise energies after passing the input sequence through multiple blocks of 1D and 2D ResNets as outlined in the Methods section. The network then splits into four output branches, following the collection of pairwise tensors being fed into the rotamer module. Here, we implemented an attention layer before predicting rotamers. We predicted every χ angle after χ 1 by conditioning on preceding χ angle(s). For χ 1 angles, we only inputted the inter-residue features.
We additionally experimented with predicting side chains independently using the same training regime (i.e., directly predicting from pairwise features as for χ 1 ). We observed that predicting side chains conditionally resulted in lower cross-entropy losses for all nine outputs, which is ideal.
Rotamer module attends to structurally-conserved anchor positions.
The rotamer module includes a self-attention layer that allows us to identify the positions that most significantly influence the side chain predictions. Rather than attending broadly across the entire antibody sequence, we observed that the model restricted attention to structurally-conserved residues, which we refer to as anchors.
We tested the conservation of anchor positions in various species and settings including ten human antibody targets, a bovine antibody, and mouse and rat sequences with unknown structures. We collected the human antibodies from the independent test set and selected a random bovine antibody (6E9G). Lastly, the mouse and rat antibody structures shown are predictions from DeepSCAb using the protocol described for DeepAb [14], for random paired sequences from OAS [22]. Across the aforementioned range of systems, we found that the anchor positions, as well as anchor residue types, are frequently conserved (Figure 2A).
When we analyzed how attention patterns changed throughout the course of training we observed a process that resembled a search for anchor residues. The initial epochs (before epoch 25)  Side chain predictions improve CDR H3 loop structure accuracy.
Training on backbone geometries improves side chain predictions.
To assess the side chain prediction accuracy of the model without any knowledge of backbone preferences, we designed a control network that consists of primarily the rotamer module. To understand the impact of including backbone conformation for side chain predictions, we evaluated the control network and DeepSCAb on a decoy discrimination task using the RosettaAntibody benchmark. For each target in the benchmark, we score 2800 decoys generated by RosettaAntibody and measure the RM SD of the top-1 and top-5 scoring structures referencing the native [23]. For the top-1 scoring decoys, DeepSCAb (RM SD = 3.2Å) outperformed the control network (RM SD = 5.0Å) by 1.8Å (32 better, 7 same, 10 worse). For the top-5 scoring decoys, DeepSCAb (RM SD = 2.5Å) outperformed the control network (RM SD = 3.3Å) by 0.8Å (23 better, 11 same, 15 worse) ( Table 1). Due to the considerable improvement observed in DeepSCAb over the control network, we conclude that the inter-residue predictions are necessary for side chain predictions. Training on side chain geometries improves inter-residual predictions in return.
Since DeepSCAb outperformed the control network, we next compared the performance of DeepSCAb to DeepH3 on the same decoy discrimination task. For the Top 1-scoring decoys, DeepSCAb modestly outperformed DeepH3 (10 better, 33 same, 7 worse; ∆RM SD = 0Å). For the Top 5-scoring decoys, DeepSCAb outperformed DeepH3 (10 better, 35 same, 4 worse; ∆RM SD = −0.1Å) ( Table 1). Using the independent test set, we plotted the structures chosen by DeepH3 against the ones chosen by DeepSCAb based on their RM SD (Å). DeepSCAb was more successful at distinguishing near-native structures in both Top 1 and Top 5 plots, though improvements over DeepH3 were most notable in the Top 5 comparison ( Figure  3A). We then analyzed two targets chosen from the Top 5 decoys for the three methods and ref2015. We show the structure scores against RM SD in the CDR H3 loop for the target 2FB4 (loop length of 19) ( Figure 3B). DeepSCAb outperformed the control network (∆RM SD = −10.8Å), ref2015 (∆RM SD = −10.2Å), and DeepH3 (∆RM SD = −1.2Å). Comparison of the structures identified by DeepSCAb and DeepH3 revealed that both models are able to place the CDR H3 loop in the correct orientation, however, the addition of side chain information in DeepSCAb results in a more accurate structure ( Figure 3C). We further plotted the funnel energies in the CDR H3 loop for the target 3MLR (loop length of 17) ( Figure 3D). DeepSCAb outperformed the control network (∆RM SD = −4.4Å), ref2015 (∆RM SD = −5.8Å), and DeepH3 (∆RM SD = −1.4Å). We displayed all structures and the native overlaid for the target 3MLR ( Figure 3E). DeepSCAb predicts the loop structure with the highest accuracy, given one of the longer and more difficult of the CDR H3 loops. Hence, the addition of side chain orientations is beneficial for accurately predicting pairwise geometries.
DeepSCAb is competitive with alternative rotamer packing methods.
The context of the predicted side chains is crucial in determining the accuracy and usefulness of a method. Side chains that are greatly exposed to a solvent play an active role in the binding of an antigen, yet are also inherently the most flexible. We evaluated the performance of our method and three alternative methods as a function of relative side chain solvent accessible surface area (SC SASA) using Rosetta. We compared the success of DeepSCAb in predicting side chain conformations to PEARS, SCWRL4, and Rosetta, where the native structure was used as a reference for all measurements. We omitted the target 3MLR from side chain packing and relative SC SASA comparisons as PEARS was unable to model this structure due to its long L3 loop.
In Figure 4, we illustrate that while DeepSCAb did not significantly outperform any method in the F V region, the CDR H3 loop or the non-H3 loops, it remained competitive in per-residue side chain prediction accuracy with the most successful alternative method, PEARS. However, we note that the alternative methods have and require access to the true backbone, while our method does not. We took a closer look at the average RM SD at various relative SC SASA values and found that DeepSCAb was able to uncover similar amounts of information from only the sequence, even for the challenging CDR H3 loop. As the methods, on average, perform very similar to one another (Table 1), DeepSCAb may be the most useful in antibody engineering applications, where the backbone conformation is flexible upon design requirements and restrictions.

Discussion
The results outlined in our work show that our method is a step towards accurate antibody docking and design via inclusion of side chain predictions. We demonstrated that DeepSCAb predictions remain competitively accurate at varying side chain surface exposure. Compared to alternative rotamer building methods, DeepSCAb is robust to non-canonical loops as it learns patterns pertaining to neighbor backbone, side chain geometry, and sequence, rather than relying on rotamer libraries. Using the rotamer module attention, we are able to identify the residue types and positions that are the most influential in the context of side chain predictions. While this analysis provides insight into rotamer prediction, it does not reveal local biophysical interactions that could be tied into the fundamentals of side chain conformation in complex energy landscapes. The anchor sites are consistently scattered throughout the sequence, in stark contrast with the local chemical environment typically considered by most side chain placement algorithms. Perhaps DeepSCAb is learning an internal, structurally conserved numbering scheme as a reference for side chain prediction similar to the well-performing PEARS algorithm [18], which was designed intentionally to use antibody-specific residue positioning to condition rotamer predictions. Alternatively, the model could be identifying global antibody features such as germline class or species, which have some side chains conserved.
This work demonstrated that inter-residue features improve side chain predictions. Also, inclusion of side chains improves overall F V structure prediction compared to machine learning models that only predict inter-residue geometries. Concurrent with this work, improved methods for antibody structure prediction have been developed [14,15]. DeepAb uses a similar architecture to predict inter-residue geometries, and ABlooper predicts CDR loop coordinates directly. Our work suggests that both of these methods might be improved by incorporating side chain context into predictions.
Most side chain repacking methods average sample conformations based on a backbone-dependent rotamer library [17], and accurate methods for antibody side chain repacking estimate χ angle densities based on a position-dependent rotamer library [18]. Since our method does not require structure as an input, DeepSCAb is unaffected by uncertainty in backbone structure [24] and allows for predictions on sequences without known backbone conformations. This feature is useful when there are multiple potential backbone conformations of interest, e.g., for the design of new therapeutic antibodies. With minimal modification, our network can aid antibody design. For instance, DeepSCAb can be used in parallel with RosettaAntibodyDesign [25] for rapid placement of side chains or to hallucinate new antibody sequences using the trRosetta architecture [10].

Conclusion
In this study, we investigated the effect of inter-residual predictions on the accuracy of side chain dihedrals as well as the effect of rotamer predictions on the overall antibody structure prediction accuracy. We found that DeepSCAb competitively predicts rotamers when compared to alternative methods that require true backbone coordinates. The performance of our method is robust to when the backbone is perturbed or deviates from the crystal structure. Since DeepSCAb predicts a probability distribution over the backbone and side chain geometries, we expect it will be adaptable to and useful for designing new antibodies.

Supporting information
Supp.   Figure 2. Interpretation of the attention matrices. L × L matrix is shown for a target chosen at random. The final attention is calculated by averaging over the rows to collapse the matrix to dimension L × 1.