Abstract
High-resolution electron microscopy (EM) of nervous systems enables the reconstruction of neural circuits at the level of individual synaptic connections. However, for invertebrates, such as Drosophila melanogaster, it has so far been unclear whether the phenotype of neurons or synapses alone is sufficient to predict specific functional properties such as neurotransmitter identity. Here, we show that in Drosophila melanogaster artificial convolutional neural networks can confidently predict the type of neurotransmitter released at a synaptic site from EM images alone. The network successfully discriminates between six types of neurotransmitters (GABA, glutamate, acetylcholine, serotonin, dopamine, and octopamine) with an average accuracy of 87% for individual synapses and 94% for entire neurons, assuming each neuron expresses only one neurotransmitter. This result is surprising as there are often no obvious cues in the EM images that human observers can use to predict neurotransmitter identity. We apply the proposed method to quantify whether, similar to the ventral nervous system (VNS), all hemilineages in the Drosophila melanogaster brain express only one fast acting transmitter within their neurons. To test this principle, we predict the neurotransmitter identity of all identified synapses in 89 hemilineages in the Drosophila melanogaster adult brain. While the majority of our predictions show homogeneity of fast-acting neurotransmitter identity within a single hemilineage, we identify a set of hemilineages that express two fast-acting neurotransmitters with high statistical significance. As a result, our predictions are inconsistent with the hypothesis that all neurons within a hemilineage express the same fast-acting neurotransmitter in the brain of Drosophila melanogaster.
1 Introduction
In recent years, advances in imaging technology enabled high resolution electron microscopy (EM) imaging of whole brain data sets (Zheng et al., 2018; Ryan et al., 2016; Cook et al., 2019; Ohyama et al., 2015), opening up the possibility of generating cellular level wiring diagrams (connectomes) of nervous systems. Generating a connectome entails identifying all neurons and the synapses that connect them. Due to the size of these data sets manual tracing of all neurons and synapses is not feasible even for comparitively small organisms such as Drosophila melanogaster. However, recent advances in automated methods for segmenting neurons (Funke et al., 2018; Januszewski et al., 2018; Lee et al., 2019) and detecting synapses (Kreshuk et al., 2015; Staffler et al., 2017; Buhmann et al., 2019) greatly reduce the time of human involvement in these tasks and have just recently been applied to generate the connectome for a large part of the Drosophila melanogaster brain (Xu et al., 2020).
However, EM data does not directly give us information about gene expression and as a result quantities such as neurotransmitter identity, while crucial to determine the function of any given synapse, are unknown for a majority of the cells in the connectome of Drosophila melanogaster. The action a neuron has on its downstream targets is determined by the neurotransmitters it releases into the extracellular space. Before release, neurotransmitters are packaged into different types of vesicles at synaptic sites. The so-called ‘classical’, fast-acting transmitters GABA, acetylcholine and glutamate are contained in small, clear vesicles, while monoamines such as dopamine, norepinephrine, octopamine and serotonin are packaged into pleomorphic clear-core or small dense-core vesicles (Goyal and Chaudhury, 2013). The large number of various neuropeptides such as cholecystokinin, galanin, neurokinin and oxytocin are contained in large dense-core vesicles. In mammals it is generally possible for humans to distinguish between different clear-core and dense-core vesicles and thus identify the neurotransmitter of any given synaptic site (Goyal and Chaudhury, 2013). In invertebrates such as Drosophila melanogaster it is so far unknown whether synaptic phenotype, as seen in EM, is sufficient to consistently determine neurotransmitter identity, especially different varieties of clear-core vesicles.
As a result, adding neurotransmitter identities to connectomic data requires light microscopy (LM) pipelines, in which gene transcripts or proteins involved in the pathway of interest have been made visible using fluorescent probes. Common methods for neurotransmitter detection include sequencing transcriptomics (RNAseq) (Henry et al., 2012; Konstantinides et al., 2015; Davie et al., 2018; Davis et al., 2020), immunolabeling (Hyatt and Wise, 2001) and fluorescent in situ hybridization (FISH) (Long et al., 2017; Meissner et al., 2019). Subsequent morphological matching of these neurons to reconstructed neurons in the EM data can then be performed using NBLAST (Costa et al., 2016), providing neurotransmitter identity to connectome data (Bates et al., 2019b,a).
However, this approach is very difficult to scale to an entire connectome of Drosophila melanogaster comprising ~150,000 cells. Although imaging expression patterns for multiple neurotransmitters on a brain scale can be done in a matter of minutes to days depending on the required resolution1 (Meissner et al., 2019), bridging the gap between LM and EM remains challenging: In addition to imaging expression patterns of neurotransmitter related proteins, it requires a well-characterized, sparse genetic driver line in order to perform accurate morphological matching to EM tracings using tools such as NBLAST (Costa et al., 2016). As a result, transmitter identity is known for only a few hundred types of neurons (Bates et al., 2019a).
Here, we show that it is possible to determine the primary neurotransmitter of a given neuron in the Drosophila melanogaster brain from the phenotype of its synaptic sites in EM alone. For that, we train a deep learning classifier to predict the neurotransmitter of a 640 × 640 × 640nm3 3D EM volume with a synaptic site at the center. We find that this method is able to classify the neurotransmitter of any given synapse with 87% accuracy on average. Furthermore, we show on a large test set that the classifier generalizes across neurons with different developmental histories (i.e., that derive from different ‘hemilineages’) and brain regions, indicating that the influence of the neurotransmitter on the phenotype of a synaptic site is largely conserved across cells. We use our method to predict the neurotransmitter identity of over 1000 neurons in 89 hemilineages with so far unknown neurotransmitter identities in the Drosophila melanogaster brain. In contrast to recent findings in the ventral nervous system (VNS) (Lacin et al., 2019), our results suggest that the neurotransmitter identity of neurons within hemilineages in the brain is not limited to one fast-acting transmitter.
Given that the relation of synaptic phenotype and neurotransmitter identity is not fully understood in Drosophila melanogaster, it is an exciting research question whether we are able to extract the set of rules and features the classifier uses to predict neurotransmitter identity. However, due to the blackbox nature of deep neural network classifiers and the complexity of synaptic sites, this is a challenging problem and beyond the scope of this paper. We plan to investigate this question in future work.
In summary, our method circumvents a major bottleneck in neurotransmitter identification, matching LM expression patterns to EM tracings, and is able to assign neurotransmitter identity to individually traced neurons in a matter of seconds. Combined with automated synapse detection methods (Buhmann et al., 2018; Kreshuk et al., 2015; Staffler et al., 2017; Buhmann et al., 2019), this opens up the possibility of generating a comprehensive neurotransmitter atlas for the connectome of Drosophila melanogaster.
2 Methods
We learn a mapping f: x → y, where x is a local 3D EM volume with a synaptic site at the center and y the neurotransmitter of the corresponding neuron (see Fig. 1). To this end, we need to generate a training dataset of pairs (x,y). This involves light microscopy of genetically tagged neurons to determine their neurotransmitter expression, neuron tracing and synapse annotation in the corresponding EM dataset and matching of the LM neuron morphology to a traced neuron in EM.
2.1 Data Acquisition
We acquire the majority of neurotransmitter to neuron assignments used for training and evaluation from published reconstructions in the full adult fly brain (FAFB) dataset (Bates et al., 2020; Dolan et al., 2018; Felsenberg et al., 2018; Frechter et al., 2019; Huoviala et al., 2018; Dolan et al., 2019; Marin et al., 2020; Sayin et al., 2019; Turner-Evans et al., 2019; Zheng et al., 2018), as well as unpublished but identified neuron reconstructions offered by the FAFB community (see acknowledgements). In these studies, the authors had already linked some of their reconstructed cell types to immunohistochemical data (Aso et al., 2014; Bräcker et al., 2013; Busch et al., 2009; Davis et al., 2018; Dolan et al., 2019; Ito et al., 2013; Lai et al., 2008; Okada et al., 2009; Shinomiya et al., 2015; Tanaka et al., 2012; Wilson and Laurent, 2005). Stainings were typically performed on neurons visualized by GFP expression in a GAL4/split-GAL4 line. Dissected brains are incubated with primary antibodies (e.g., anti-VGlut, anti-GABA, anti-ChAT), followed by secondary antibodies which have a fluorescent tag to visualise the primary antibody. The transcripts/proteins related to certain transmitter expressions are thus labelled across the brain and if they colocalize with the GFP signal for the GAL4/split-GAL4 line of interest, those neurons are considered to express that transmitter3. For RNA transcripts, usually the neuron’s soma is examined. Other methods involve RNA sequencing and include TAPIN-seq (Davis et al., 2018). Note that individual studies often only test single transmitters and do not show negative staining. As a result, there is limited data for cotransmission of multiple neurotransmitters in a single neuron and we therefore assume no cotransmission of neurotransmitters within one neuron.
We use manually reconstructed neuron annotations from the FAFB community CATMAID4 (Saalfeld et al., 2009; Schneider-Mizell et al., 2016) database5. Synapses were annotated at presynaptic sites, defined by T-bars, vesicles and a thick dark active zone by a synaptic cleft (Prokop and Meinertzhagen, 2006). In total, the assembled dataset contains 153,593 cholinergic synapses (679 neurons), 33881 GABAergic synapses (181 neurons), 7953 glutamatergic synapses (49 neurons), 9526 dopaminergic synapses (89 neurons), 2924 octopaminergic synapses (7 neurons) and 4732 serotonergic synapses (5 neurons).
2.2 Train and Test Datasets
For each neurotransmitter y ∈ {GABA, ACh, GLUT, SER, OCT, DOP}, we divide the data in test, train and validation set by randomly assigning entire neurons, each containing multiple synapses. We refer to this split as Neuron split in the following. We use 70% of neurons for training, 10% for validation and the remaining 20% for testing. Splitting the dataset by entire neurons, instead of randomly sampling synapses, mirrors the real world use case in which we typically know the neurotransmitter of an entire neuron and are interested in the neurotransmitter of an entirely different neuron.
In order to test how well the proposed methods generalizes across morphologically distinct cells and regions, and to exclude potentially confounding variables, we also generate two additional splits that separate the data by hemilineage (Hemilineage split - neurons in a hemilineage are lineally related, see section 4 for further details) and brain region (Brain Region split) respectively. To this end, we find the optimal split between entire hemilineages and brain regions, such that the fraction of synapses for every neurotransmitter in the train set is close to 80% of all synapses of that neurotransmitter. We further use randomly selected 12.5% of the training synapses (10% of the entire dataset) for validation.
2.3 Network Architecture and Training
We use a 3D VGG-style (Simonyan and Zisserman, 2014) network to predict the neurotransmitter identity from a 3D EM input cube of edge length 640 nm with a synaptic site at its center. The network consists of four functional blocks, each consisting of two 3D convolution operations, batch normalization, ReLU non-linearities and subsequent max pooling with a downsample factor of (z=1, y=2, x=2) for the first three blocks and (z=2, y=2, x=2) for the last block and is followed by three fully connected layers with dropout (p=0.5) applied after the first two. We train the network to minimize cross entropy loss over the six classes (GABA, ACh, GLUT, SER, OCT and DOP), using the Adam optimizer (Kingma and Ba, 2014). We train for a total of 500,000 iterations in batches of size eight and select the iteration with highest validation accuracy for testing. A full specification of the network architecture and training pipeline, including data augmentations, can be found in the appendix. For an illustration of the used network architecture see Fig. 1c.
3 Classifier accuracy
We tested the classifier on our held out test sets. For the Neuron split, the test set consists of a total of 40,104 synapses from 185 neurons that the network was not trained on. We achieve an average, per transmitter accuracy of 87% for the neurotransmitter prediction of individual synapses. Since we assume that each neuron expresses the same neurotransmitters at its synaptic sites we can additionally quantify the per neuron accuracy of the presented method. To this end we assign each neuron with more than 30 synapses in the test set a neurotransmitter by majority vote of its synapses, leading to an average accuracy of 94% for the neurotransmitter prediction per neuron. For the Hemilineage split, we find an accuracy of 75% for individual synapses and 92% for entire neurons. The Brain Region split evaluates to 88% synapse classification accuracy and 95% neuron classification accuracy. A per class overview can be seen in Fig. 3, for a summary of the results and data splits see Table 1.
4 Transmitter prediction for hemilineages
Similar to neurons, which release the same set of neurotransmitters at their synaptic sites (Eccles, 1976; Dale, 1934), it has been found that sets of lineally related neurons in the Drosophila melanogaster ventral nervous system (VNS), so-called hemilineages, also show homogeneous neurotransmitter expression patterns (Lacin et al., 2019). If a similar principle holds for the Drosophila melanogaster brain, it would enable us to assign neurotransmitter identity to large groups of neurons simultaneously. With the presented method we are able to verify to what extent such a principle holds.
The about 45,000 neurons of the central brain of Drosophila melanogaster (Croset et al., 2018) are generated by a set of stem cells known as neuroblasts. During division neuroblasts generate two cells, one additional stem cell and one cell that further divides into two sibling neurons. In only one of these siblings the so called Notch pathway is activated, leading to two different “hemilineages” of neurons within the lineage (Kumar et al., 2009; Sen, 2019; Lacin et al., 2019). Lacin et al. (2019) showed that each hemilineage in the VNS expresses just one of the fast-acting transmitters acetylcholine, glutamate and GABA, even though mRNA transcripts for combinations of these can appear in the nucleus (Lacin et al., 2019). This raises the question whether the same holds true in the adult brain. Using the presented classifier, we predict the neurotransmitter identity of all identified neurons within 89 out of a total of ~150 identified hemilineages in the Drosophila melanogaster brain. The majority of our predictions show homogeneity of neurotransmitter identity within a single hemilineage, in line with findings in the VNS. However, we identify a set of hemilineages which express two fast acting neurotransmitters with high statistical significance. We find no hemilineage that expresses all three. As a result, our predictions are inconsistent with the hypothesis that all hemilineages express the same fast-acting neurotransmitter.
4.1 Hemilineage assignments in Drosophila
Cell body fiber tracts for identified hemilineages had previously been identified using TrakEM2 (Cardona et al., 2012) in a lightlevel atlas for a Drosophila melanogaster brain, stained with an antibody against neurotactin (BP104) (Lovick et al., 2013). We extracted these expertly identified tracts and registered them into a common template brain, JFRC2, using CMTK (Rohlfing and Maurer, 2003), and then into FAFB space (Bates et al., 2019b). We could now identify cell body fibre tracts in this ssTEM dataset in the vicinity of the transformed hemilineage tracts.
4.2 Predictions
We retrain the classifier on 90% of the entire dataset and use the remaining 10% to select the best performing iteration. We predict the neurotransmitter identity of 180,675 synapses within 1,164 neurons with so far unknown neurotransmitter identity. These neurons come from a total of 89 hemilineages, of which 20 have more than one neuron with genetically identified neurotransmitter. Fig. 4 shows ground truth neurotransmitter annotations for the subset of neurons that have known neurotransmitters and our predictions for the remaining neurons in the hemilineage for five selected hemilineages. In the following, we analyse the results by quantifying how neurotransmitter predictions are distributed over neurons and synapses within hemilineages.
4.2.1 Neuron Level Entropy
In order to quantify multimodality of neurotransmitter predictions on neuron level within a hemilineage we calculate the entropy H of the neurotransmitter distribution over neurons in the following way: Let n ∈ Nh be a neuron in hemilineage h and the predicted neurotransmitter of neuron n. Then
A value of H(Nh) = 0 (minimal entropy) then means that all neurons within hemilineage h have the same predicted neurotransmitter, while a value of H(Nh) = 1 (maximal entropy) means that within hemilineage h all predicted neurotransmitters are equally common.
4.2.2 Synapse Level Entropy
Similarly we can quantify the average multimodality over synapses within neurons of a given hemilineage: Let s ∈ Sn be the synapses in neuron n ∈ Nh of hemilineage h and the predicted neurotransmitter. The entropy of predicted synaptic neurotransmitters H(sn) in neuron n is then given by:
With this, the average synaptic entropy over all neurons within hemilineage h is given by:
A value of H(Sh) = 0 (minimal entropy) then means that all synapses of all neurons in hemilineage h have the same predicted neurotransmitter, while a value of H(Sh) = 1 (maximal entropy) means that in all neurons within hemilineage h all synaptic neurotransmitter predictions are equally common. Fig. 5 shows the distribution of H(Nh) and H(Sh) of all predicted hemilineages with more than ten neurons that have more than 30 synapses each.
On the population level we find relatively lower values of H(Sh) (Synapse level entropy) than H(Nh) (Neuron level entropy). 75% of hemilineages show a synapse level entropy below q75(H(Sh)) = 0.19 as compared to q75(H(Nh)) = 0.34. This is reassuring as it suggests less variation of neurotransmitter identity predictions within individual neurons compared to variations of neurotransmitter identity of neurons within a hemilineage. However, we also find cases with a high level of synaptic entropy, such as hemilineage 16 and 30. For these hemilineages it is unclear whether neuron level multimodality is only an artifact of uncertain, multimodal predictions on synapse level of individual neurons. In contrast to 16 and 30 hemilineages 29, 27 and 42 show high neuron level entropy H(Nh) ≥ q75 but low synapse level entropy H(Sh) ≤ q25, suggesting clear neuron level segregation of predicted neurotransmitters within those hemilineages. Hemilineages such as 24 and 40 with H(Sh) < q25 and H(Sn) < q25 appear homogeneous within each neuron and within the entire hemilineage.
4.3 Number of distinct, fast-acting neurotransmitters in hemilineages of the Drosophila melanogaster adult brain
We can now ask the question how likely it is to observe a given prediction of neurotransmitters in a hemilineage under some error rate given by the confusion matrix on the test set, and the assumption that all neurons in the hemilineage have the same underlying neurotransmitter. We can then compare this likelihood to the alternative hypothesis that a hemilineage consists of neurons with more than one neurotransmitter. Out of 26 investigated hemilineages with a sufficient number of predicted neurotransmitters, up to five show strong evidence for expressing two distinct, fastacting neurotransmitters (Bayes factor K ≥ 102, decisive). We find none that expresses all three.
4.3.1 Probability to observe neurotransmitter predictions ŷ
Given a neuron has true neurotransmitter y ∈ Y, the probability that we predict neurotransmitter ŷ ∈ Y (assuming that each prediction is independent and identically distributed) is given by the categorical distribution where C is the neuron confusion matrix obtained on the test data set (see Fig. 3).
Let m be the number of different neurotransmitters in hemilineage h. We model the probability p(ŷ|m) of observing neurotransmitter predictions ŷ = {ŷ0, ŷ1,…,ŷn} under the assumption that hemilineage h contains m different neurotransmitters. Here, ŷj is the predicted neurotransmitter of neuron j in hemilineage h with n neurons total. Let be the set of subsets of true neurotransmitters Y with cardinality c, then: where p(ŷ|S) is the probability to observe predictions ŷ if the hemilineage has true underlying neurotransmitters y ∈ S and p(S|m) is the probability for the set of true neurotransmitters S given the hemilineage contains m different neurotransmitters. Since we assume i.i.d. predictions ŷ, p(ŷ|S) factorizes as follows: and marginalizing over y ∈ S yields:
Regarding p(S|m) and p(y|S) we assume a flat prior, i.e.:
With this, the probability of observing predictions ŷ given m different neurotransmitters becomes:
4.3.2 Bayes Factor
With this formalism in place, we can compare hypotheses about the number of true neurotransmitters m in a given hemilineage by using the Bayes Factor , where D is our observed data (predicted neurotransmitters) and M1, M2 are two models about the underlying true neurotransmitters that we wish to compare. The Bayes factor for a model M1 with m1 true neurotransmitters per hemilineage and model M2 with m2 different neurotransmitters is given by:
So far, we assumed that p(ŷj|y) = Cy,ŷj., i.e., we estimate this distribution on the test dataset. However, because our test set is finite we can not expect that the estimated error rates perfectly transfer to other datasets. In order to relax our assumptions about this distribution we simulate additional errors, by incorporating additive smoothing on the counts of neurons Ny,ŷ that have true neurotransmitter y and were predicted as neurotransmitter ŷ, i.e.: where is the smoothing parameter. With we then have and the count normalized smoothing parameter. In the limit of approaches the uniform distribution with probability 1/6 for each neurotransmitter, whereas a value of α = 0 means we recover the observed confusion matrix C. With this, our distributions are now parametrized by α and the bayes factor becomes: where is as defined in (13) but with Cy,ŷj. replaced with its expected value .
The prior distribution on α, p(α), allows us to encode our prior knowledge about α and use it to weight the likelihood of the corresponding model. Given the data, a value of α = ϵ with epsilon small (0 < ϵ ≪ 1), should be most probable, while the probability of values α > ϵ should monotonically decrease as we deviate more from the observed confusion matrix. Values of α < ϵ should have probability zero, because they correspond to the unsmoothed confusion matrix with zero entries, i.e., a probability of zero for missclassification of certain neurotransmitters. While these probabilities may be small, they are likely greater than zero and an artifact caused by the finite test set. Many distributions fulfill these criteria, in particular the family of exponential distributions with rate parameter λ:
Thus, λ controls the weight for smoothing parameter α in the integral . For λ → 0, the expected confusion matrix approaches the unweighted average of all C(α) in the integration range. For λ → ∞, the expected confusion matrix approaches the e-smoothed confusion matrix . The rate parameter λ can also be understood via its influence on the expected average accuracy . For values of λ → 0, the expected accuracy approaches chance level while for values of λ → ∞, the expected accuracy approaches the ϵ-smoothed, observed accuracy on the test set.
4.3.3 Evidence for two fast acting transmitters in a single hemilineage
We calculate the bayes factor and for the set of three classical neurotransmitters Ycl = {GABA, ACh, GLUT} for those hemilineages that have more than ten annotated neurons and 30 annotated synapses each with neurotransmitter predictions in the set Ycl. For this analysis, we ignore all other neurons with predicted neurotransmitter identity ŷ ∉ Ycl. Fig. 6 shows K2,1 and K3,2 for a range of rate parameters λ and corresponding expected average accuracy . For hemilineages 30, 16, 27, 20, and 42 there is decisive evidence (K2,1 ≥ 102) for the presence of two distinct fast acting neurotransmitters for a large range of expected accuracies . However, note that hemilineage 30 and 16 show high synaptic entropy H(Sh) (see Fig. 5), indicating that individual neurons within the hemilineage already show bimodal neurotransmitter predictions. As such, strong bimodality on neuron level is at least partially explained by uncertain predictions on individual synaptic level. This is in contrast to hemilineage 27 and 42, who show synaptic entropies below the 25% percentile. In these hemilineages, large Bayes factor values K2,1 directly stem from neuron level segregation of the predicted neurotransmitters within the hemilineage. See Fig. 7 for a rendering of the neurotransmitter predictions of these hemilineages and corresponding NBLAST dendrograms, indicating that the two fast acting neurotransmitters in some of these hemilineages are divided between morphologically distinct neurons. The remaining 13 hemilineages show no strong evidence for either hypothesis (K2,1 ≈ 1, n=5) or favor the hypothesis of expressing only one fast acting neurotransmitter (K2,3 ≤ 10−2, n=8). No hemilineage shows evidence for expressing all three fast acting neurotransmitters (K2,3 < 10−2).
5 Discussion
5.1 Results
We presented a classifier which is able to predict the neurotransmitter identity of a synapse from a local 3D EM volume with high accuracy. We showed that the method generalizes across neurons, brain regions and hemilineages. Furthermore we predicted the neurotransmitter identity of 180,675 synapses within 1,164 neurons from 89 hemilineages with so far unknown neurotransmitter identity. We analyzed the neurotransmitter distribution of 26 hemilineages that have a sufficient amount of annotated neurons and synapses and showed that most of them homogeneously express one fast acting neurotransmitter. However, we also identified a set of five hemilineages that, according to our predictions, express two distinct fast-acting neurotransmitters with high statistical signficance. Two of those five, 27 and 42, also show low synaptic entropy H(Sh), indicating that the observed effect is a result of neuron-level neurotransmitter segregation within the hemilineage.
5.2 Limitations
A potential source of neurotransmitter missclassification is the possibility that a given neuron releases more than one neurotransmitter at its synaptic sites. Due to a lack of known and annotated neurons with cotransmission of the considered neurotransmitters our current model ignores this possibility. However, single cell transcriptomic data of the Drosophila melanogaster brain shows that neurotransmitter gene expression is largely exclusive for the fast acting transmitters ACh, GABA and GLUT (Croset et al., 2018), excluding widespread cotransmission of these transmitters. For the considered monoamines (SER, OCT, and DOP), coexpression with another fast acting transmitter is more probable. In particular Croset et al. (2018) suggests that a large fraction of octopaminergic neurons likely corelease GLUT, while SER and DOP show less evidence for coexpression with fast acting transmitters. If a particular neuron in the dataset were to corelease a fast acting transmitter and a monoamine the presented classifier would predict only one of the two. However, this is not a fundamental limitation of the presented approach and could be remedied if coexpression training data becomes available.
Another current limitation is the fact that we only consider the set of six neurotransmitters {GABA, ACh, GLUT, DOP, OCT, SER} and due to our use of a softmax normalization at the network output layer, the model is forced to select one of the six classes, even if there is no evidence for either of them. As a result, the current model is not able to identify synapses or neurons that are not part of the considered neurotransmitters, noteably histamine (Nässel, 1999), tyramine and a vast number of neuropeptides (Croset et al., 2018). Similar to coexpression, we expect an extension to further neurotransmitters to be possible with available training data.
Regarding our analysis of the number of distinct fast-acting neurotransmitters in a hemilineage, a potential source of error is misassignments of neurons to hemilineages. If neurons are erroneously assigned to a particular hemilineage any observed effect of multimodal neurotransmitter distributions on neuron-level within a hemilineage could be an artifact. Furthermore, for hemilineages 16 and 30 the high synaptic entropy H(Sh) suggests that the phenotype of synapses is ambiguous in these hemilineages. Although coexpression of fast-acting neurotransmitters is unlikely (Croset et al., 2018), the neurotransmitter distribution would be consistent with cotransmission of GABA and glutamate in the neurons of these hemilineages.
5.3 Generalization
We showed that our network is able to generalize across brain regions and hemilineages. However, although the performance on the brain region split (88% average accuracy) outperforms even the baseline neuron split (87% average accuracy), the hemilineage split suffers a performance decrease of more than 10% (75% average accuracy), suggesting that the influence of the neurotransmitter on the phenotype of a synaptic site is influenced by its hemilineage. This is partially remedied when averaging over multiple synapses: Neuron-level neurotransmitter classification for the hemilineage split is robust with an average accuracy of 92%. Note that the presented data splits already exclude neuron identity, hemilineage identity and brain region as potential confounding variables for the prediction of neurotransmitter identity as performance is far above chance level.
5.4 Interpretability
Given that in invertebrates humans can not generally distinguish different neurotransmitter containing vesicles from EM alone, an interesting question is which features the classifier relies on. However, despite the fact that we have complete information about the artificial neural system that is able to assign images of synaptic sites to neurotransmitters, it is difficult to extract the rules under which it operates. It is, for example, possible to visualise areas of importance for the neural network via use of various attribution methods (for an overview see Ancona et al. (2018)), but often the masked input image still contains a large amount of complexity that varies for each image of a given class (data not shown). We aim to explore this question in more depth in future work.
6 Acknowledgements
This work was supported by a Wellcome Trust Collaborative Award (203261/Z/16/Z), an ERC Consolidator grant (649111) and core support from the MRC (MC-U105188491) to G.S.X.E.J; a Boehringer Ingelheim Fonds PhD Fellowship and a Herchel Smith Studentship to A.S.B. The synapses in this study were manually identified in the FAFB dataset by tracers working on different projects across 17 laboratories. For these annotations, we thank Ruairi Roberts, Philipp Schlegel, Fiona Love, Lisa Marin, Amelia Edmondson-Stait, Xincheng Zhao, Jawaid Ali, Johann Schor, Imaan Tamimi, Arian Jamasb, Marisa Dreher, Markus Pleijzier, Robert Turnbull, Nadiya Sharifi, Steven Calle, Andrew Dacks, Konrad Heinz, Kimberly Meechan, Aidan Smith, Najla Masood-panah, Serene Dhawan, Peter Gibb, Corey Fisher, Claire Peterson, Jason Polsky, Tansy Yang, Kathi Eichler, Joseph Hsu, Irene Varela, Lucia Kmecova, Istvan Taisz, Jacob Ratliff, Kaylynn Coates, Anna Li, Marta Costa, Tyler Paterson, Claire Managan, Adam Heath, Katie Stevens, Jack Mccarty, Nora Forknall, Laurin Bueld, Neha Rampally, Zane Mitrevica, Kelli Fairbanks, Stanley Tran, Shada Alghailani, Quinn Vanderbeck, Lauren Warner, Henrique Ludwig, Jeremy Johnson and Levi Helmick, each of whom have contributed over 1,000 synapses. We principally thank the Wellcome Trust UK and US Drosophila Connectomics, Jefferis, Janelia CATMAID annotation team, Bock, Preat, Wilson, Dacks, Hampel and Seeds groups for sharing their published and unpublished work in the FAFB dataset. Further, we thank Michael Reiser, Vivek Jayaraman, Arthur Zhao, Tatsuo Okubo, Jenny Lu and Kathi Eichler for identifying neuron matches which helped us build our ground truth.
Appendix
Footnotes
↵1 Throughput estimated at around one neuron per minute at sufficient spatial resolution for colocalization with single cell labeling - personal communication with authors.
↵2 See Footnote 5
↵3 A commonly used, full step-by-step protocol can be found at https://www.janelia.org/project-team/flylight/protocols.