Abstract
In chemical synapses of the central nervous system (CNS) an information unit is transmitted via the presynaptic release of a neurotransmitter vesicle, a quantum, eliciting a postsynaptic membrane electric response of a certain amplitude, a quantal size. This key determinant of neural computation has not been reliably measured due to the nature of synaptic communication and technical limitations of electrophysiological recordings in the CNS. Measuring amplitudes of spontaneous postsynaptic currents (sPSCs) or potentials (sPSPs) at the cell soma potentially offers a technically straight forward way to estimate the quantal size as these miniature responses (minis) are ellicited by the spontaneous release of a single neurotransmitter vesicle. However, somatically recorded minis are massively attenuated and, for the most part, are indistinguishable from background noise fluctuations. For them to be accurately estimated, a revised method would have to isolate the noise component. As the first step in devising such a method, we developed and described a novel sPSP/sPSC detection algorithm as part of our quantal analysis software called ‘minis’. We tested the performance of the algorithm in detecting real and simulated minis in rat cortical slice whole-cell recordings and compared it to other most commonly used similar detection algorithms in the field of the synaptic function research. This benchmarking analysis revealed superior detection performance by our algorithm. The release version of the algorithm also offers great flexibility as it can be controlled through graphical and programming interfaces making it suitable for the needs of most individual researchers studying the central synapse function.
Significance statement Being able to measure accurately the electrical size of synapses in the brain is critical for understanding neural computations. Offering a novel algorithm for measuring properties of spontaneous synaptic signalling is an important first step in doing so. By its qualities of being transparent and objectively evaluated, this algorithm sets an important methodological standard for comparative assessment of other popular algorithms in the field of the synaptic function research. Because of its superior performance and flexibility, the algorithm also offers a technological improvement greatly useful to individual researchers in the field. Finally, its evaluation process highlighted common methodological pitfalls surrounding the spontaneous synaptic signalling research that hopefully could reduce their future impact.
Introduction
Synaptic transmission in the CNS occurs in quanta (Kuno, 1964; Larkman et al., 1991; Kraszewski and Grantyn, 1992; Kullmann and Nicoll, 1992; Stern et al., 1992; Jonas et al., 1993; Isaacson and Walmsley, 1995; Forti et al., 1997; Wall and Usowicz, 1998; Sahara and Takahashi, 2001; Hardingham et al., 2010) corresponding to the release of a single neurotransmitter vesicle (Heuser et al., 1979). The amplitude of the postsynaptic response to a quantum, a quantal size, is a key parameter defining the efficacy of synaptic transmission (Vere-Jones, 1966; Katz, 1969) and, therefore, an important determinant of neural computation.
Quantal size measurements in the CNS proved to be notoriously difficult due to multi-synaptic contacts, limited accessibility, large quantal variance, and low signal-to-noise ratio of somatic electrophysiological recordings (Korn and Faber, 1991; Bekkers, 1994). For these reasons the classical quantal analysis largely failed. More recent methods did not fare better (e.g., noise deconvolution) (Edwards et al., 1976; Jack et al., 1981; Redman, 1990; Clamann et al., 1991) and introduced additional assumptions, requirements, and signal correspondence issues creating new sources of error (e.g., Variance-Mean Analysis and optical quantal analysis) (Reid and Clements, 1999; Yuste et al., 1999; Clements and Silver, 2000; Oertner et al., 2002; Clements, 2003; Silver, 2003; Biró et al., 2005; Enoki et al., 2009; Peled and Isacoff, 2011). A truly improved method based on direct electrophysiological measurements would be preferred.
A related method of assessing synaptic function is to measure properties of sPSCs. sPSC amplitude measurements may be directly taken as quantal size estimates as the sPSC or sPSP corresponds to the response to a spontaneous release of a single neurotransmitter vesicle (Fatt and Katz, 1952; del Castillo and Katz, 1954; Brown et al., 1979; Isaacson and Walmsley, 1995; Wall and Usowicz, 1998). Measurements of sPSC amplitudes are routine in the literature. However, they are likely to be overestimates as the reported sPSC frequency values are below the expected range (see Research practices of measuring minis in the CNS).
The use of the voltage clamp is one major reason why sPSCs “go missing”. Theoretical studies (Rall and Segev, 1985; Major, 1993; Spruston et al., 1993) predicted and experimental findings (Williams and Mitchell, 2008) confirmed that voltage clamp fails to clamp dendrites. Unclamped dendrites distort sPSCs. Voltage clamp could also cause massive amplitude attenuation down to half of its original value if series resistance is not compensated (Williams and Mitchell, 2008). These issues can be fixed by switching to current clamp. Yet most sPSCs are undetected not due to recording configuration but because of the physiology of the neuron itself: they are filtered by resistive and capacitive properties of its membrane. Distal minis leak charge via the dendritic membrane, but an even more significant amplitude attenuation, 40-fold and more, is inflicted by large dendritic trees charging the capacitance of their membranes (Stuart and Spruston, 1998; Williams and Stuart, 2002; Nevian et al., 2007; Williams and Mitchell, 2008; Larkum et al., 2009; Major et al., 2013). Such amplitudes are indistinguishable from noise fluctuations in somatic recordings. Therefore, a reliable quantal analysis method would require estimating noise component in addition to measuring putative sPSPs.
As an initial step in an attempt to create such a method, we developed a novel algorithm, called ‘minis’, to detect spontaneous postsynaptic events. A novel algorithm was needed for a few reasons.
First, existing algorithms were not published in peer-reviewed literature. Popular algorithms like Mini Analysis and Clampfit are proprietary, while custom ones are restricted to the originator lab. Second, because existing algorithms are not open, they are not modifiable. Thus, they cannot be automated and integrated with other software by individual researchers. They have to be curated manually which is a major practical obstacle limiting their use. Third, existing algorithms do not meet the objective performance assessment standard as they lack accompanying published benchmarking studies. In response, we developed an algorithm that was transparent, flexible, and objectively evaluated.
Materials and Methods
Literature search
We examined neuroscience literature over the period of January 1, 2020 – January 31, 2022 to get a representative sample of the most recently reported amplitude and frequency values of spontaneous excitatory postsynaptic currents (sEPSCs) and potentials (sEPSPs) in cortical pyramidal cells, as well as, of common data analysis practices. We also compared reported sEPSC frequency values to an expected range of values. We limited our search to publications in English only that reported research results on synaptic function in the CNS. We examined the MEDLINE database using the PubMed search engine with the following search keywords: (“spontaneous neurotransmission”[Title/Abstract] OR “spontaneous neurotransmitter release”[Title/Abstract] OR “spontaneous transmitter release”[Title/Abstract] OR “spontaneous postsynaptic”[Title/Abstract] OR “miniature postsynaptic”[Title/Abstract] OR “mEPSP”[Title/Abstract] OR “mEPSC”[Title/Abstract] OR “mPSP”[Title/Abstract] OR “mPSC”[Title/Abstract] OR “mIPSP”[Title/Abstract] OR “mIPSC”[Title/Abstract] OR “mEPP”[Title/Abstract] OR “mEPC”[Title/Abstract] OR “mIPP”[Title/Abstract] OR “mIPC”[Title/Abstract] OR “sEPSP”[Title/Abstract] OR “sEPSC”[Title/Abstract] OR “sPSP”[Title/Abstract] OR “sPSC”[Title/Abstract] OR “sIPSP”[Title/Abstract] OR “sIPSC”[Title/Abstract] OR “sEPP”[Title/Abstract] OR “sEPC”[Title/Abstract] OR “sIPP”[Title/Abstract] OR “sIPC”[Title/Abstract]) AND (“cortex”[Title/Abstract] OR “cortical”[Title/Abstract] OR “neocortex”[Title/Abstract] OR “neocortical”[Title/Abstract] OR “hippocampus”[Title/Abstract] OR “hippocampal”[Title/Abstract] OR “thalamus”[Title/Abstract] OR “thalamic”[Title/Abstract] OR “brain”[Title/Abstract] OR “cerebral”[Title/Abstract] OR “central nervous system”[Title/Abstract] OR “pyramidal”[Title/Abstract] OR “CNS”[Title/Abstract]).
Animals and electrophysiology
All experimental procedures were carried out in accordance with the UK Animals (Scientific Procedures) Act 1986 at the Cardiff University under Home Office personal and project licenses.
19- to 27-day old Wistar rats (RRID:RGD_13508588; n = 11) of either sex were anaesthetised with isoflurane and decapitated, and the brain removed under cold artificial cerebrospinal fluid (aCSF), composed of (in mM) 125 NaCl, 26 NaHCO3, 2.3 KCl, 1.26 KH2PO4, 1 MgSO4x7H2O, 10 glucose, and 2 CaCl2. Coronal somatosensory cortical slices 350 µm thick were cut and submerged for 1 to 3 hours into aCSF at room temperature (21-23°C). Individual slices were then placed in the recording chamber and held at 35-37°C with aCSF flowing over both surfaces. For recording purposes at 35- 37°C the aCSF solution contained (in mM) 125 NaCl, 24 NaHCO3, 2.3 KCl, 1.26 KH2PO4, 1 MgSO4x7H2O, 10 glucose, and 2 CaCl2. Brains of 9 animals were used to obtain a single recording slice from each. One other animal was used to obtain 2 recording slices, while another one was used for obtaining 3 recording slices.
Whole-cell recordings (n = 14) were obtained under visual control using infrared scanning gradient contrast imaging with the imaging beam and a Dodt tube. Recordings were made with 4.5-9 MΩ borosilicate glass pipettes (Harvard Apparatus, 1.5 mm outside diameter and 0.86 mm internal diameter filament; Sutter P97 puller), filled with (mM): 140 KGluconat, 10 HEPES, 2 MgCl2, 3 ATP- Na2, 0.3 GTP-Na, 0.1 Magnesium Green; 7.35 pH, 273 mOsm osmolarity. Signals were amplified, low- pass filtered (5 KHz) and digitised (Axopatch-200B, custom amplifier/filters, Digidata 1440, Clampex software; Molecular Devices). Pipettes and electrodes were positioned with Sutter MP265 and 285 manipulators with a diagonal (axial) mode, using parallelogram trajectories.
The electrophysiological recordings consisted of two phases. During the first phase we recorded sEPSCs and sEPSPs in the background of physiological noise (‘noise + minis’ condition) by applying aCSF solution added with an action potential and inhibitory postsynaptic potential blockers: (in mM) 125 NaCl, 24 NaHCO3, 2.3 KCl, 1.26 KH2PO4, 1 MgSO4x7H2O, 10 glucose, 2 CaCl2, 1 μM tetradotoxin (TTX; glutamate NMDA receptor antagonist), and 12.5 μM gabazine (GABAA receptor antagonist).
During the second phase we recorded noise with sEPSPs/sEPSCs being blocked (‘noise-alone’ condition). The recording was carried out after adding 40 μM NBQX (AMPA receptor antagonist) and 50 μM CPP (NMDA receptor antagonist) to the solution used in the ‘noise + minis’ condition.
Experimental design
The voltage and current clamp whole-cell patch clamp recordings obtained during the initial recording phase (‘noise + minis’ condition) were subjected to sEPSC/sEPSP detection analyses using ‘minis’, Mini Analysis (Bluecell; RRID:SCR_002184), and Clampfit (part of the pClamp software suite, Molecular Devices; RRID:SCR_011323). sEPSCs/sEPSPs were analysed in terms of their amplitude, 10- 90% rise time, 1/e decay time, and incidence frequency. Detection performance was compared in terms of these measures, as well as compared qualitatively by inspecting recording series data traces and noting various common putative errors committed by different detection algorithms.
The current clamp recordings obtained during the second recording phase (‘noise-alone’ condition) served as the background data in sEPSP computer simulations. Computationally simulated sEPSPs were superimposed onto these membrane potential noise recordings and the resultant hybrid recordings were then subjected to sEPSP detection analyses using the same three algorithms. Detection performance was quantified using signal detection theory measures and compared between different algorithms.
Simulations
Simulating distributions of spontaneous excitatory postsynaptic potentials of arbitrary amplitudes and rise times
All simulations were carried out within ‘minis’ software environment which is based on Matlab (Mathworks). Simulation of minis-like events was based on an analytical solution to the passive cable equation for a transient current (Rall, 1977) using lumped terms with a double exponential synaptic current (Major et al., 1993). We used a current impulse given by and the membrane voltage response given by where Q is the injected charge (C), τsy1 and τsy2 are synaptic rise and decay time constants (ms), l is the physical length of a dendrite (μm), x is the physical stimulation distance from the soma (μm), t is the time (ms), and τn and Bn are given by with τm and λ being the dendritic membrane time (ms) and length (μm) constants, respectively. In our simulations we used 100 terms (n).
An initial cable segment was constructed to have an electrotonic length l/λ = 0.6. Charge (Q) was injected at a location (x) along this cable and the resulting simulated postsynaptic membrane potential was measured at the unsealed end of this cable segment. The precise location of the injected charge was varied systematically depending on the desired shape of the simulated event. Increasing the distance of the injected charge along the cable away from the measurement site gradually increased the rise time of the simulated event. If the rise time of the simulated event was too fast with an initial segment, l/λ was gradually increased until the required rise time was obtained. Increasing the amount of injected charge increased the amplitude of the simulated event. Therefore, one was able to construct a simulated postsynaptic potential of any rise time or amplitude. A pool of simulated events could be created having any kind of distribution of amplitudes and rise times with individual events positioned pseudo-randomly along the temporal extent of the membrane potential recording.
Simulations used to construct a full receiver operating characteristic curve
All simulated sEPSPs were drawn from a bivariate normal distribution with the following amplitude dimension parameters: μ1 = 0.3 mV and σ1 = 0.05 mV. The following rise time dimension parameters were used: μ2 = 0.05 ms and σ2 = 2.5 ms. The rotation factor was ρ = 0. Simulated events were drawn pseudo-randomly from a distribution at the following frequencies (Hz): 640, 320, 160, 80, 40, 20, 10, 5, 2.5, 2.5, 2.5, 2.5, 2.5, and 2.5. The noise amplitude scale factors for these 14 frequency conditions were as follows: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1.2, 1.4, 1.8, 2.6, and 4.2. The drawn simulated events were then positioned at pseudo-randomly determined locations along the current-clamp noise recording obtained during the second recording phase (‘noise-alone’ condition). 4 different simulation traces per recording each of them having different event positioning were generated in every frequency/noise scale condition resulting in a total number of 14*4*14 = 784 traces. Sections of noise recordings of various durations were used. 2 of them were 200 seconds long while the remaining 12 were 100 seconds long.
Simulations using a realistic range of amplitudes and frequencies of spontaneous excitatory postsynaptic potentials
All simulated events were drawn from bivariate normal distributions with the following amplitude and rise time parameters: μ1 = varying, σ1 = 1*10-9 mV, μ2 = 0.05 ms, and σ2 = 2.5 ms. The rotation factor was ρ = 0. 10 distributions were used with different μ1 parameter (mV): 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, and 0.5. The same number of events was drawn from each distribution with pooled frequencies (Hz) of 53, 33, and 13. The drawn simulated events were positioned at pseudo-random locations along the noise recording using the same recording sections as the ones described in the previous subsection. 4 different simulation traces per recording each of them having different event positioning were generated in every frequency condition resulting in a total number of 3*4*14 = 168 traces.
Detection of spontaneous excitatory postsynaptic potentials and currents
‘minis’ detection algorithm
We developed a novel algorithm to detect postsynaptic potentials and currents that we have incorporated into a data analysis software called ‘minis’ which was fully written in Matlab (Mathworks) and distributed as an application programming interface in the form of a packaged Matlab app or a Python (Python Software Foundation) package or as a compiled standalone desktop application with a graphical user interface. In brief, the detection algorithm takes a filtered membrane potential or current trace and detects peaks and estimates their amplitudes and rise and decay times after applying certain processing steps. These steps are outlined below (with parameter values tailored for detecting simulated sEPSPs):
The data is band-stop filtered to remove mains noise (e.g., 50 Hz and 150 Hz frequency components; optional). Butterworth filter is used with a stopband attenuation of 10 Db and a passband ripple of 0.05 Db. The stopband size is 1 Hz and the passband being the entire frequency range except for the 6 Hz window surrounding the stopband frequency.
The recording trace is smoothed using a Gaussian window (optional). We used a 1.5- millisecond window. This step removes high frequency noise.
Peaks are identified in a filtered and smoothed recording trace.
Any peaks within the peak integration period are discarded if they are smaller than the peak of interest. Only the largest peak is kept. We used a peak integration period of 2.5 ms.
The baseline of the peak is positioned at the lowest value before the peak at a distance controlled by the maximum time to peak parameter. We used 10 ms maximum time to peak.
The temporal length of the baseline is controlled by the baseline duration parameter. The extent of the baseline falls 80% prior to the lowest value and the remaining 20% extending beyond it. We used a 2-ms baseline duration parameter.
If the baseline overlaps with the previous peak, its duration is shortened so that the baseline starts where the previous peak ends.
If the amplitude of the peak is outside of the range of acceptable amplitudes, the peak is discarded. We used a range of 0.1-10 mV to construct a full receiver operating characteristic (ROC) curve. In other conditions we used a range of 0.05-10 mV.
If the first half of the rise time (10-50% or 20-50%, depending on the rise time duration of choice) is larger than the second half (50-90% or 50-80%) by a factor of 5 or more, the maximum time to peak parameter is reduced to start at the end of the baseline and a new baseline is established. Correction can be repeated one more time.
The amplitude test is applied again (see point 8).
If the second half of the rise time is larger than the first half by a factor of 5 or more, the peak is shifted to a previously discarded smaller peak (if exists).
The amplitude test is applied again (see point 8).
If the end of the baseline deviates from 10% or 20% rise time mark (depending on whether 10-90% or 20-80% rise time is measured) by more than a half of the baseline duration, a new baseline is established by reducing the maximum time to peak period to start a half baseline duration before the 10% or 20% rise time mark.
Finally, the amplitude test is applied again (see point 8).
An sPSP/sPSC is established.
For detection of real sEPSPs we used the same parameters as above but with an exception of varying the lower bound on acceptable amplitudes from 0 mV to 0.3 mV. A single recording example of 100 seconds duration was used.
When detecting real sEPSCs, we used the same parameters as above but with an exception of having the Gaussian smoothing window set to 0.5 ms. The lower limit on acceptable amplitudes varied between 0 and -20 pA and is stated in the Results section describing particular voltage clamp detection instances. The upper limit was set to -50 pA. Raw data was also low-pass filtered at 4 kHz prior to the detection analysis. Two recording examples obtained in slices coming from two different animals were used with both of them being 200 seconds in duration.
Detection using Mini Analysis software
Mini Analysis software version 6.0.7 was used to detect sEPSPs and sEPSCs for a benchmarking purpose. We used the following detection parameters: An amplitude threshold of 0.1 mV (when constructing the full ROC curve) or 0.05 mV (all other conditions), a period to search a local maximum of 12500 μs, time before a peak for baseline of 10000 μs, a period to search a decay time of 30000 μs, a fraction of peak to find a decay time of 0.37, a period to average the baseline of 2000 μs, an area threshold of 0 mV*μs, a number of points to average for peak of 1, and using the detect complex peak option. The chosen detection parameters closely resembled those used in the ‘minis’ detection algorithm. They were also found to give the most optimal detection performance of the Mini Analysis software in comparison to other tested parameter sets. Event detection was carried out on the recording data that was filtered and smoothed in the same way as described in steps 1 and 2 of the previous subsection.
For the purpose of detecting real events, we used the same parameters as above but we varied the amplitude threshold between 0 mV and 0.3 mV in the case of membrane potential data and between 0 pA and -20 pA in the case of current data. The same recordings with the same data filtering and smoothing parameters were used as described in the previous subsection.
Detection using Clampfit
Clampfit software, part of pClamp 11.0.3.03 software suite, was used to detect real and simulated sEPSPs and sEPSCs for a benchmarking purpose. Recordings were band-stop filtered as described in step 1 of the ‘minis’ detection algorithm. Data was not smoothed. Template search algorithm was used with 9 templates. The template set was constructed based on simulated events superimposed on the noise recording. Events having 0.5 ms, 1.5 ms, 3.5 ms, and 6.5 ms 10-90% rise times were selected to produce 4 distinct templates. Additional 5 templates were constructed in a way that made them potentially a mix of multiple simulated waveforms. They were constructed to have extra- fast, fast, intermediate, slow, and extra-slow rise times corresponding to 2 ms, 2.75 ms, 6 ms, 7 ms, and 8 ms 0-100% rise times and 8 ms, 9.25 ms, 17 ms, 23 ms, and 37 ms duration full decays, respectively. Template match threshold was set to 4 with the rest of the detection parameters being set to their default values as well. The same set of templates was used to analyse all simulated current clamp data.
For the purpose of detecting real postsynaptic currents, 5 different templates were created based on the low-pass filtered at 4 kHz and unsmoothed data. The data was examined for postsynaptic currents that varied in their shape, particularly in the speed of their rise and decay times, and templates were made to capture these aspects in as much distinct manner as possible. Different sets of templates were created separately for each analysed recording.
Detection algorithm response classification
At the outset we identified prominent noise events in the membrane potential noise recordings (the 2nd recording phase: ‘noise-alone’ condition) that could be potentially confused by any of the algorithms for true simulated sEPSPs (Noise traces in Figure 1A and B). All noise recordings were filtered and smoothed like described in points 1 and 2 of the ‘minis’ detection algorithm. The data was further smoothed by a moving average window of 20 ms. Then all peaks larger than 0.01 mV and having a half-width of at least 0.5 ms were identified. These peaks were classed as noise events. Subsequently we simulated sEPSPs and positioned them at pseudo-random locations over an empty trace having the same temporal length as the noise recording of interest (Simulation traces in Figure 1A and B). The peaks of these events were classed as signal events. The simulated trace was then overlaid the noise recording (Simulation + noise traces in Figure 1A and B). Locations of signal and noise events formed the ground truth information.
Following the detection process, all detected events were associated with one of the signal and noise events depending on their temporal proximity to these predefined ground truth events. A detected event was classed as a true positive (or a hit) if it was the closest to the signal event out of all detection events associated with this signal event and no further from this signal event than 5 ms. If no detection event occurred within 10 ms window surrounding the signal event (-5 to 5 ms range), the signal event was classed as a false negative (or a miss). If no detection event occurred within the same duration window surrounding the noise event, the noise event was classed as a true negative (or a correct rejection). The remaining detection events were classed as false positives (or false alarms). Examples of classified detection events are shown in Figure 1 separately for all three detection algorithms.
Detection response distance to the nearest neighbour
The temporal distance to the peak of the nearest simulated sEPSP was available for every classified response. Detection performance was then evaluated as a function of the distance to the nearest neighbour in general and in the case when membrane potential was stable.
Membrane potential phase classification
Algorithm detection performance was assessed not only for full recordings but also separately for periods when membrane potential was rising, decaying, or remained stable. The decay phase of the purely simulated membrane potential was set to be a period between 0.0625 ms past the peak and until the purely simulated membrane potential trace dropped to the ratio of 0.3/e mV or until the membrane potential started rising again to reach a new peak that was higher than the decaying initial peak. If the new peak was not higher than the initial one, the entire new peak was classed as part of the ongoing decay phase. Peaks occurring on the decay phase of the membrane potential were assigned a membrane potential decay value that was present 0.25 ms prior to the appearance of membrane potential inflection points associated with the start of these peaks. The rise phase was essentially a decay phase in reverse. The only difference was that the ratio of 0.3/e mV was replaced by the ratio of 0.3/10 mV. Peaks occurring on the membrane potential rise phase were assigned a membrane potential rise value that was present 0.125 ms after the appearance of membrane potential inflection points following the decay of these peaks. Periods that were already classified as decay phases obviously could not be reclassified as rise phases. Periods of simulated membrane potential outside of the rise and decay phases were classed as stable (or a flat phase). The classification of purely simulated membrane potentials into three different phases was used to assign phases to the hybrid sections with noise recordings superimposed with simulated potentials. The detection performance during rise and decay phases was evaluated as a function of the membrane potential rate of change. Performance during stable periods was evaluated as a function of the distance to the nearest simulated sEPSP (nearest neighbour).
Data Analyses
Performance of the three tested algorithms was compared using measures of signal detection theory. All detection responses were classified as described in the Detection algorithm response classification subsection. True positive rate (TPR) was calculated as follows: Throughout this text we use TPR, detection probability, and sensitivity interchangeably. False positive rate (FPR) was calculated as follows: Throughout this text we use FPR and false alarm rate interchangeably.
Calculation of TPR and FPR allowed producing ROC curves and calculating area under the curve (AUC; area under the ROC curve with 0.5 indicating chance performance and 1 indicating perfect performance) as detection performance indicators. These are the most commonly used performance indicators for classifiers in addition to sensitivity (TPR), specificity (1 – FPR) and d’ (or sensitivity index) which we often use in parallel throughout this text. d’ statistic was calculated as follows: where Z is the Z-score.
The above detection performance indicators were calculated for full recordings, as well as separately for rising, decaying, and stable periods of membrane potential.
Statistical analyses
All inferential statistics were based on the assumption that distributions of all measures were normal. All mean distribution values are stated/depicted with 95% confidence limits.
Data accessibility
All data analysed in this study is publicly available (EBRAINS DOI). The available data include current and voltage clamp recordings containing sEPSCs/sEPSPs (‘noise + minis’ recording condition) that were analysed and used to produce figures. They also include ‘noise-alone’ current clamp recordings that were used in computer simulations. The hybrid membrane potential traces containing simulated sEPSPs superimposed on noise recordings are also available. All electrophysiological recordings were stored in Axon Binary File (ABF) format.
sEPSC and sEPSP templates used for detecting real and simulated events in Mini Analysis are also contained within the same dataset in the form of Axon Template Files (ATFs).
Code accessibility
All analyses were carried out in Matlab and the analysis code is publicly available on Github (available once published). The code is complete with instructions on how to reproduce all figures reported in this study.
Software accessibility
The present study reported the use of a novel sEPSP/sEPSC detection algorithm that is part of ‘minis’ software. It is a commercial software containing proprietary code with its user copies available to acquire for a price. A free trial version of the software is available on Github (Dervinis, 2022).
Results
Research practices of measuring minis in the central nervous system
There were 107 studies in total that matched our inclusion criteria. All of these studies used voltage clamp as their preferred electrophysiological method for recording spontaneous postsynaptic events. We looked at the kind of software they used to detect these events. Majority, 47.66%, reported using Mini Analysis. 26.17% reported using Clampfit for that purpose, while 13.08% stated that they were using a custom built algorithm. The remaining 13.08% did not specify which software they used for that purpose. It is, therefore, instructive to compare the performance of the most popular detection algorithms, namely Mini Analysis and Clampfit, relative to each other, as well as to ‘minis’.
35 studies (or 32.71% of all studies) reported using a current amplitude detection threshold (-6.7895 ± 0.43986 pA, range of -3 to -20 pA, n = 19). We believe this to be the lower limit on this proportion as only 2 studies reported explicitly of not using any detection thresholds. Most of the studies did not give enough details (65.42% of all studies) to judge one way or another. However, studies that used Mini Analysis for detection must have used a detection threshold as this software requires it.
Out of all studies, 29 carried out research in neocortical pyramidal cells. 25 studies reported a mean sEPSC amplitude value of -14.31 ± 0.44 pA (range of -6 to -27.5 pA). 26 studies reported a mean frequency value of 3.16 ± 0.14 Hz (range of 0.12-7.96 Hz).
Total putative excitatory synapse count in cortical pyramidal cells numbers in tens of thousands (Eyal et al., 2018). Layer 2/3 pyramidal cells are thought to have 20,000 to 30,000 putative excitatory synapses. Large tufted layer 5 pyramidal cells would have an even larger number of contacts (Larkman, 1991). Therefore, we can roughly assume that for most cortical pyramidal cells the number of putative excitatory synapses ranges between 20,000 and 60,000. As a precaution we could assume that only a minority, roughly 40%, of them are functional synapses with active zones (Motta et al., 2019; Holler et al., 2021). Given that a spontaneous vesicle release at a synapse occurs with an approximate frequency of 0.0021 Hz (Murthy and Stevens, 1999), the frequency of such events within a single cortical pyramidal neuron would be expected to range somewhere between 16.8 Hz and 50.4 Hz (potentially even 10-100 Hz in the extremes). Frequency values reported in the recent literature are below this expected frequency range by an order of magnitude.
Algorithm detection performance comparison for real excitatory spontaneous postsynaptic currents
We compared our novel ‘minis’ algorithm with two other widely used programs: Mini Analysis and Clampfit. We recorded sEPSCs using voltage-clamp (‘noise + minis’ condition) and carried out detection on the same recordings using all three algorithms in turn. Detection using ‘minis’ and Mini Analysis was carried out using amplitude threshold of -3 pA. The aim of this exercise was to compare visually the performance of these algorithms using voltage clamp data and to establish the set of putative errors commonly committed by these detection algorithms. Examples of current traces with detection performance are shown in Figure 2. One common striking error committed by both Mini Analysis and Clampfit but not ‘minis’ was to miss obvious sEPSCs occurring in close proximity to other sEPSCs. This error affected any size sEPSCs and examples include small-to-moderate (Figure 2D, J, and L), as well as large events (Figure 2A, B, G, and J). sEPSC-like events were also commonly missed on the membrane current rise phase with Mini Analysis and Clampfit being affected regularly. Examples of such missed events occurring on a slowly (Figure 2C, H, and L), moderately (Figure 2D), and rapidly (Figure 2A, G, H, K, and M) rising membrane current phases are abundant. Just like events were missed on the rise phase, other events could also be missed on the decay phase of the membrane potential due to these events being overshadowed by earlier events.
Examples of this happening are shown in Figure 2E, F, and I. Visual inspection suggested that these errors were more common in Mini Analysis and Clampfit compared to ‘minis’. Moreover, sEPSC-like events were often missed if they occurred on membrane current troughs (Figure 2B and M) with Mini Analysis and Clampfit being the biggest offenders of this type. Finally, detected peak misplacements were uniquely common in Mini Analysis (Figure 2K). Visual inspection of voltage- clamp data suggested strongly that ‘minis’ overall fared better compared to Mini Analysis and Clampfit. However, objective comparison was needed to establish this beyond doubt. The results of quantitative detection performance analyses are described in the following subsections.
Algorithm performance comparison for detecting moderately-sized simulated spontaneous excitatory postsynaptic potentials
Detection under a wide range of frequency conditions: Overall performance
In Results: Literature search subsection we reported that detection of spontaneous postsynaptic events is typically carried out using an amplitude threshold. This approach implies that the range of amplitudes of spontaneous postsynaptic events is somewhat known and that, on the smaller end, the amplitudes are moderately-sized and larger than pure noise fluctuations. Leaving aside the question of empirical validity of this dubious assumption, we evaluated automated detection of moderately-sized simulated sEPSPs in whole-cell patch clamp noise recordings (the 2nd recording phase: ‘noise-alone’ condition).
Like in the previous subsection, we compared our novel ‘minis’ algorithm with Mini Analysis and Clampfit. We created 14 simulation frequency conditions with a varying signal-to-noise ratio that mimicked the effects of varying the signal detection threshold. On the one hand, increasing the frequency of simulated sEPSPs tended to decrease the TPR as simulated events would start to overshadow each other (mimicking an increase in the signal detection threshold). On the other hand, increasing the noise amplitude tended primarily to increase the FPR before it would start having any effects on the TPR (mimicking a decrease in the signal detection threshold).
We plotted the detection performance in terms of TPRs and FPRs for all 14 conditions and all three algorithms in Figure 3A. We found that the ROC curve for ‘minis’ had no overlap with any other curve and was positioned furthest away from the diagonal than any other curve in the first 6 conditions (starting from left to right; 640, 320, 160, 80, 40, and 20 Hz with no noise scaling) indicative of a superior performance in this frequency range. The same observation was supported by another measure, d’, shown in Figure 3B. Detection performance in the remaining 8 conditions (10, 5, 2.5, 2.5, 2.5, 2.5, 2.5, and 2.5 Hz with noise scaling factors of 1, 1, 1, 1.2, 1.4, 1.8, 2.6, and 4.2, respectively) was very similar for both ‘minis’ and Mini Analysis as none of the two showed a clear superiority on both performance measures simultaneously: ROC and d’ curves overlapped. In contrast, the Clampfit software showed consistently the poorest performance across all 14 conditions with the performance severely deteriorating in high amplitude noise conditions.
When overall performance was assessed irrespective of individual frequency and signal-to-noise ratio conditions, we found that ‘minis’ showed by far the best performance with the mean AUC value of 0.96 ± 0.00233 (Figure 3C). A paired samples t-test comparing to Mini Analysis with the mean AUC value of 0.937 ± 0.00215 gave t(13) = 12.4 and p = 1.37*10-8. Clampfit showed the poorest overall performance with the mean AUC value of 0.706 ± 0.00428, t(13) = 33.1, and p = 6.14*10-14 when compared to Mini Analysis mean AUC value in a paired samples t-test. These differences in detection performance were further corroborated by the d’ measure averaged across all signal-to-noise ratio conditions (Figure 3D). A paired samples t-test comparing ‘minis’ mean d’ (2.45 ± 0.0447) and Mini Analysis mean d’ (2.11 ± 0.0402) gave t(13) = 14 and p = 3.24*10-9. The same test comparing Mini Analysis mean d’ and Clampfit mean d’ (1.23 ± 0.024) gave t(13) = 13.3 and p = 6.07*10-9. In summary, these findings clearly demonstrated that ‘minis’ detection algorithm had a superior performance in comparison to other algorithms at simulated event frequencies higher than 10 Hz while Clampfit showed consistently the poorest detection performance across the entire frequency and signal-to-noise range.
Detection under a wide range of frequency conditions: Common errors
Both ‘minis’ and Mini Analysis were fairly good at detecting moderately-sized simulated sEPSPs. Yet all algorithms, including ‘minis’, missed a sizeable proportion of them. Figure 4 shows examples of common mistakes made by all algorithms. All mistakes could be grouped into a limited number of types. One such type was missing simulated events that occurred simultaneously or in very close proximity to other events (Figures 4A, B, C, E, F, G, and H). Unfortunately, none of the three algorithms could deal with events that overlapped near-perfectly always lumping them together.
Detecting peaks in the data using thresholds or fitting templates to the data does not offer a way of identifying instances of overlaps – a different approach is needed. Another common error type was missing events with peaks coinciding with the rise phase of another sEPSP (Figures 4A, B, C, E, F, and G). Unlike perfect or near-perfect overlaps, these events had a clear but brief decay period that was quickly upended by a rise phase of a background potential. Because of the fact that these decays were brief, they often got lumped with a subsequent sEPSP or they could not be adequately fit with a template by Clampfit and, therefore, were discarded. ‘minis’ (but not Mini Analysis) does offer a way to minimise these errors by reducing the peak integration period but this comes at the expense of noise fluctuations on the rise phase increasingly passing as postsynaptic potentials. An effective solution to this problem would, thus, require isolating and measuring the noise component in whole- cell patch clamp recordings which currently is not implemented in any of the three algorithms.
An even more common error type was to miss sEPSPs that occurred on the decay phase of another, often larger simulated earlier sEPSP (Figure 4G and H). The amplitude of sEPSPs occurring on decays is effectively reduced and the steeper the decay, the more the amplitude is affected. Amplitude of such attenuated potentials may fall below an acceptable threshold or shape distortions may prevent fitting a template. As a result, such sEPSPs were occasionally discarded. Unfortunately, there is no easy way of correcting these errors, as it would require estimating the underlying decaying potential and subtracting it from the overall membrane potential. sEPSPs that neither occurred during the rise nor decay phases were also occasionally missed (Figures 4A and B). This could have happened due to a failure to fit a template (Clampfit) or the amplitude being reduced by background noise fluctuations beyond an acceptable threshold. In recordings of real sEPSPs these errors would mainly stem from there being no difference between small amplitude sEPSPs and background noise fluctuations. An effective solution to this problem would require isolating and measuring the noise component in whole-cell patch clamp recordings similarly to previously discussed errors occurring on the membrane potential rise phase.
A whole other class of detection errors appeared in the form of false alarms occurring on the rise (Figure 4E) or decay (Figures 4B, C, and D) phases of underlying sEPSPs or in the absence of any potentials in the vicinity whatsoever (Figures 4C and D). These mistakes were often caused by background noise fluctuations passing as real events and might have been affected by amplitude detection thresholds or an inability to discern shapes of real events from noise. Just like the previously discussed error type, a solution would require isolating and measuring properties of the noise component. In addition, there were errors that uniquely affected one algorithm or another, like the misplacement of detected events by Mini Analysis (Figure 4F). These errors might have been artificially introduced by conceptual errors in the algorithm. Overall, we saw that distinct detection errors could be made depending on how close they were to neighbouring sEPSPs and whether they occurred on the rise or decay phases of existing other sEPSPs in the vicinity or in the absence of any sEPSPs in the background. All three algorithms occasionally committed these errors but their incidence was not even across the board.
Detection under a wide range of frequency conditions: The membrane potential phase effect
In order to quantify the detection performance in different circumstances, we evaluated the probability of making errors in relation to the proximity to neighbouring sEPSPs and the phase of the background membrane potential: rising, decaying, or being stable. We looked at TPR as a function of distance to the nearest neighbour (Figure 5A) taken together for all frequency conditions but excluding conditions where noise was scaled: 640, 320, 160, 80, 40, 20, 10, 5, and 2.5 Hz conditions (9 in total). We found that detection was superior for ‘minis’ across the entire distance range. Mini Analysis came firmly the second with Clampfit consistently showing the worst performance. These differences in detection performance were also reflected in the cumulative TPR (Figure 5D). As for the FPR (Figure 5B), ‘minis’ performance was superior at detecting simulated sEPSPs that were within 10 ms distance away from the nearest neighbour. This performance edge over Mini Analysis was not maintained for larger distances while Clampfit again showed consistently the worst performance across the entire distance range. These performance differences were reflected in the cumulative FPR (Figure 5E). Taken together, true and false positive rates allowed as to calculate a combined performance indicator, d’, as a function of distance to the nearest neighbour (Figure 5C). As expected, the combined measure indicated that ‘minis’ had the superior detection performance within 10 ms distance to the nearest neighbour while Clampfit showed the worst performance across the entire distance range.
During the membrane potential rise phase performance differences were unequivocal. There were very few events and, therefore, positive detections when the background membrane potential was rising at 200 μV/s or faster (Figure 6D). Most of the detected events coinciding with the rise phase actually occurred when membrane potential was changing at the rate of 10 to 100 μV/s. In this range ‘minis’ showed the best detection performance with d’ between 1 and 2 (Figure 6C). Mini Analysis was the second best, while Clampfit showed the worst performance. This order of performance was maintained for rates below 10 μV/s with overall performance improving for all algorithms. The performance edge of ‘minis’ over the other algorithms came primarily due to low FPR (Figure 6B). The TPR of ‘minis’ was also superior to the other two algorithms but, in terms of sensitivity, Mini Analysis was not far behind (Figure 6A). In the range between 5 and 35 μV/s ‘minis’ actually fell slightly behind Mini Analysis. The cumulative rates clearly supported the observation that ‘minis’ was superior to the other two algorithms with Mini Analysis being the second while Clampfit having the worst performance (Figure 6D and E).
The same behaviour by the three algorithms was also observed for detecting simulated sEPSPs occurring on the decay phase of the background membrane potential. Most of the detected simulated events were associated with rates of membrane potential change in the range of -100 and -10 μV/s (Figure 7D). In this range and higher ‘minis’ demonstrated the best performance relative to the other two algorithms with d’ ranging between 1.5 and 2.5 (Figure 7C). Mini Analysis was again the second, while Clampfit showed a very poor performance. Clampfit often had d’ that was close to 0, meaning that its performance was often not different from the chance level when background membrane potential was decaying. The abysmal performance was mainly due to large FPR for this algorithm (Figure 7B). ‘minis’ had unequivocally the highest TPR and the lowest FPR (Figure 7A and B) with cumulative rates very clearly reflecting this fact (Figure 7D and E). The overall performance of the runner up Mini Analysis was substantially poorer.
In order to get the full scope of performance, we also evaluated detection when the membrane potential was deemed to be stable (Figure 8). Simulated sEPSPs that were separated from their neighbours by 2 ms and more were easily detected by all three algorithms with all of them showing d’ around 2 (Figure 8C). This is not surprising as well isolated simulated events were not overshadowed by their neighbours and, therefore, posed few problems to any algorithm irrespective of its nature. Simulated sEPSPs coming close to their neighbours (2 ms and less) posed significant problems to Clampfit and their detectability by this software fell sharply. In the distance range between 0.5 and 2 ms ‘minis’ showed better performance relative to the other two algorithms primarily driven by low FPR (Figure 8A, B, and C). At even closer distances Mini Analysis had a slight edge over ‘minis’ but both of them maintained good performance level with d’ prime ranging between 2.5 and 3. The cumulative rates indicated that ‘minis’ and Mini Analysis had a similar performance level with corresponding curves nearly overlapping each other (Figure 8D and E).
Clampfit cumulative rates were somewhat more problematic. However, as the majority of detected simulated sEPSPs during stable membrane potential conditions had neighbours at the distance of 10 ms and more (Figure 8D), any of the three algorithms could be trusted to carry out relatively good detection under these conditions. As we saw, this situation contrasts with previous observations that when the background membrane potential was not stable, either increasing or decreasing, ‘minis’ performed was substantially better than other algorithms with Clampfit showing very poor performance.
Detection under realistic frequency conditions
Data points used to construct the full ROC curve include frequency and noise conditions that were not necessarily realistic. Only a narrow range of sEPSP frequencies are expected to occur in real recordings and, based on our calculations, the range is likely to be between 16.8 Hz and 50.4 Hz (10- 100 Hz in the extremes). Frequency conditions that are within this range were marked by circles in Figures 3A and B. Just like across the entire ROC curve, the performance of ‘minis’ was unequivocally superior in this frequency range compared to the other two algorithms with Mini Analysis being the second and Clampfit coming confidently the last.
Other measures like true positive and false positive rates as functions of the distance to the nearest neighbour and membrane potential rate of change during the rise or decay phases pointed to a very similar conclusion made regarding the full ROC curve. First, we found that ‘minis’ performed better than other algorithms at detecting sEPSPs that were within 10 ms distance to other sEPSPs and that at larger distances Mini Analysis performed just as good (Figure 9). Meanwhile, Clampfit performed consistently the worst across the range of distances whether we looked at d’ (Figure 9C) or whether we looked at the individual rates (Figures 9A, B, D, and E). When it came to membrane potential rate of change, ‘minis’ consistently showed the best performance at detecting simulated sEPSPs occurring simultaneously with the background membrane potential rate of change ranging between -40 and 100 µV/s (Figures 10 and 11). This is the range where the vast majority of simulated events occurred (Figures 10D and 11D). In terms of individual TPR and FPR (Figures 10A and B and 11A and B) and cumulative rates (Figures 10D and E and 11D and E), the conclusions are very much identical to those made regarding the full ROC curve. Namely, that ‘minis’ had the largest TPR and the smallest FPR with an exception being the TPR in relation to the rising background membrane potential (Figure 10A and D) with there being a range (5-35 µV/s) where Mini Analysis had a slightly higher TPR but not overall performance measured as d’ (Figure 10C). Similar to conditions used to construct the full ROC curve, Clampfit showed consistently the worst performance in terms of all measures. It is worth mentioning however, that performance of all three algorithms was better in relation to realistic simulated sEPSP frequencies (Figures 10C and 11C) compared to performance in relation to the full ROC curve (Figures 6C and 7C). Finally, we also found that all three algorithms showed equally good detection performance when sEPSPs were well isolated and occurred in the presence of stable background potentials but ‘minis’ outperformed other algorithms at distances between 0.5 and 2 ms, while Clampfit performance sharply dropped in this range (Figure 12). At even shorter distances Mini Analysis had a narrow edge over ‘minis’. However, as the vast majority of simulated sEPSPs occurring during stable background membrane potential conditions had neighbours at 10 ms distance and further, all three algorithms were equally suitable under these conditions. ‘minis’ was irreplaceable, however, when it came to unstable background membrane potential conditions. In summary, evaluation of detection of sEPSPs simulated at realistic frequencies supported conclusions made regarding detection performance under a much broader range of simulation frequencies.
Algorithm performance comparison for detecting simulated excitatory postsynaptic potentials with a wide range of amplitudes
So far we have only tested performance of the three algorithms when detecting moderately-sized sEPSPs. The existing evidence does not indicate that either sPSPs or sPSCs form a distinct amplitude distribution that can be separated from the amplitude distribution of noise fluctuations. Therefore, here we aimed to test the performance of detecting simulated sEPSPs of a wide range of amplitudes with an intention to evaluate the three algorithms in more realistic settings.
We used three frequency conditions (Hz): 13, 33, and 53. Within each of these conditions simulated sEPSPs were drawn pseudo-randomly from a uniform distribution of the following amplitudes (mV): 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, and 0.6. Just like in previous simulations, we positioned these simulated sEPSPs pseudo-randomly over noise recordings and detected them using the three algorithms. The overall performance of these algorithms is shown in terms of true and false positive rates in Figure 13A and d’ in Figure 13B. A few things are prominent in these graphs.
First of all, the detection performance of all algorithms was considerably worse compared to performance detecting moderately-sized sEPSPs. Secondly, ‘minis’ again had the edge over two other algorithms, especially at faster simulation frequencies. Third, Clampfit performance was consistently the poorest of the three.
Performance was also assessed irrespective of individual frequency conditions. We found ‘minis’ to have by far the best performance with the mean AUC value of 0.827 ± 0.00312 (Figure 13C). A paired samples t-test comparing to Mini Analysis with the mean AUC value of 0.809 ± 0.00301 gave t(13) = 8.57 and p = 1.04*10-6. Clampfit showed the poorest performance with the mean AUC value of 0.665 ± 0.00424, t(13) = 17.4, and p = 2.16*10-10 when compared to the Mini Analysis mean AUC value in a paired samples t-test. These differences in detection performance were further corroborated by the d’ measure averaged across all frequency conditions (Figure 13D). A paired samples t-test comparing ‘minis’ d’ (1.5 ± 0.0484) and Mini Analysis d’ (1.33 ± 0.0398) gave t(13) = 7.02 and p = 9.03*10-6. The same test comparing Mini Analysis d’ and Clampfit d’ (0.61 ± 0.0215) gave t(13) = 8.06 and p = 3.48*10-6. The conclusion based on combined measures was in line with the assessment based on individual frequency conditions.
Other measures like true positive and false positive rates as functions of the distance to the nearest neighbour and membrane potential rate of change during the rise or decay phases pointed to a very similar conclusion regarding realistic settings as the one made regarding the full ROC curve. First, we found that ‘minis’ performed better than other algorithms at detecting sEPSPs that were within 10 ms distance to other sEPSPs and that at larger distances Mini Analysis performed just as good (Figure 14). Meanwhile, Clampfit performed consistently the worst across the range of distances whether in terms of d’ (Figure 14C) or whether in terms of individual rates (Figures 14A, B, D, and E). When it came to membrane potential rate of change, ‘minis’ performed better than Mini Analysis which in turn performed better than Clampfit when detecting simulated sEPSPs occurring during rise and decay phases of background membrane potential (Figures 15 and 16). One difference compared to the full ROC curve analysis was that the detection performance of ‘minis’ and Mini Analysis overlapped in the range of 5-30 µV/s when detecting simulated sEPSPs occurring on the rise phase of the background membrane potential (Figure 15C). This is largely caused by an increased FPR of ‘minis’ in this rate range (Figure 15B). The overall performance of all algorithms was better (Figures 15C and 16C) compared to the full ROC curve analysis (Figures 6C and 7C), but still worse when compared to detecting moderately-sized simulated sEPSPs with realistic frequencies (Figures 10C and 11C) as described in the previous subsection. Finally, compared to the results presented in the previous subsection, detection of simulated sEPSPs during stable background membrane potential conditions was somewhat worse for all algorithms with d’ ranging between 1 and 1.5 (Figure 17C). This is the direct consequence of increased FPRs (Figure 17B). Similarly to results reported earlier, ‘minis’ had a performance advantage in the range of 0.5-2 ms over other algorithms, but Mini Analysis had a slight edge at distances that were smaller. However, the vast majority of the simulated sEPSPs occurring during stable membrane conditions were separated from their neighbours by 10 ms or more (Figure17D). Moreover, the cumulative rates were balanced in the sense that a higher TPR tended to be associated with a higher FPR and vice versa which had an offsetting mutual effect (Figure 17D and E). As a result, none of the three algorithm had an overall advantage over others. In summary, all algorithms tended to perform comparatively even in detecting varying amplitude simulated sEPSPs when the background membrane potential was stable but ‘minis’ had a substantial advantage when it came to unstable background membrane potential with Clampfit severely underperforming under these conditions.
Amplitude distributions of real spontaneous excitatory postsynaptic potentials and noise fluctuations are not separated
In the final evaluation stages we ran detection on the real membrane potential data. We ran thresholded detections on sEPSPs with the inhibitory ones being blocked (the 1st recording phase: ‘noise + minis’ condition). As we varied the minimal amplitude threshold from 0 mV to 0.3 mV, we saw the mean amplitude of all detected events gradually linearly increased when using ‘minis’ or Mini Analysis (Figure 18A). ‘minis’ tended to give a lower amplitude estimate compared to Mini Analysis in the low amplitude threshold range. This did not apply to Clampfit as this software does not have an amplitude detection threshold parameter. A few things are worth noting here. First, ‘minis’ appears to be more sensitive at picking sEPSPs of a small amplitude. This may provide a less biased amplitude estimate when signal-to-noise ratio is high but would give an amplitude underestimate when the signal-to-noise ratio is low. Second, Clampfit mean amplitude estimate was rather high, most likely an overestimate. This result is in line with our simulation results showing small TPR for this algorithm meaning a large number of sEPSPs, especially small ones, go missing.
Third, the fact that mean amplitudes increased linearly meant that noise fluctuations and small sEPSPs had amplitudes that were indistinguishable from one another. If that was not the case, we would have observed a threshold range where mean amplitudes would have stopped increasing or slowed down increasing temporarily marking a range where the two amplitude distributions were separated.
As we increased the amplitude threshold, we saw the mean 10-90% rise time also changing for both ‘minis’ and Mini Analysis (Figure 18B). Initially it increased as expected due to small amplitude noise events that were being shed having rapid rise times. The rise time eventually started falling as the amplitude threshold increased, because presumably multicomponent sEPSPs that were not well aligned were being shed. These events would inevitably have smaller amplitudes but also have more extended rise times due to the misalignment. Finally, only the multicomponent large amplitude events that were well aligned remained. The dependency of estimated mean 10-90% rise times on the amplitude detection threshold stresses the point that, just like mean amplitudes, rise times cannot be adequately measured using thresholded detection. In addition, we also saw that mean 1/e decay time estimated by ‘minis’ and Mini Analysis increased as the amplitude detection threshold was increased (Figure 18C). This was presumably because small amplitude and fast rising noise events also have fast decays. An amplitude threshold value at which decay times stopped increasing presumably marks the upper limit on the amplitude of noise fluctuations. Finally, we plotted the frequency of detected sEPSPs as a function of amplitude detection threshold (Figure 18D). First of all, we noted that frequency values given by Clampfit were likely to be underestimates as we know this software to miss many events based on our simulation results. Second, estimates provided by ‘minis’ and Mini Analysis appeared to drop in value exponentially as the amplitude detection threshold was increased. This fact suggests that even the slightest shift in the amplitude detection threshold would result in a wildly different frequency of sPSPs estimate in real recordings. It also suggests that a change in the state of synaptic function that would result in the reduction in the mean amplitude of sPSPs is likely to be partly misinterpreted as a reduction in the frequency due to the amplitude threshold suddenly discarding a large proportion of small amplitude sPSPs as noise fluctuations.
Amplitude distributions of real spontaneous excitatory postsynaptic currents and noise fluctuations are not separated
Detection of sEPSPs may be uniquely flawed. Serious issues that we reported in the previous subsection may not afflict detection of sPSCs as severely. In this subsection we took a voltage clamp recording of sEPSCs with the inhibitory ones being pharmacologically suppressed (the 1st recording phase: ‘noise + minis’ condition) and demonstrated that the same issues held true to voltage clamp recordings. As we varied the amplitude detection threshold, the mean amplitude of detected sEPSCs tended to increase for both ‘minis’ and Mini Analysis output (Figure 19A). This increase in the amplitude was linear indicating that there was no amplitude range where the amplitude distributions of noise fluctuations and small sEPSCs could be separated. Moreover, we saw that both mean 10-90% rise times and decay times changed as the amplitude detection threshold increased (Figures 19B and C). Finally, the estimated frequency of detected sEPSCs dropped exponentially as the amplitude detection threshold was increased (Figure 19D), implying that the same amplitude threshold-related issues reported in the previous subsection held true to sPSCs.
Discussion
Here we have demonstrated that frequencies of sEPSCs, as reported in the literature on synaptic function in the CNS, are underestimated by an order of magnitude compared to an expected value for excitatory synaptic transmission in neocortical pyramidal cells. We noted that these currents go missing because of voltage clamp errors – namely, the space clamp error and failures to adequately compensate series resistance – and amplitude attenuation due to filtering by the resistive and capacitive properties of dendritic membranes (Rall and Segev, 1985; Major, 1993; Spruston et al., 1993; Stuart and Spruston, 1998; Williams and Stuart, 2002; Nevian et al., 2007; Williams and Mitchell, 2008). We showed that the mean absolute amplitude of detected sEPSCs increased linearly as the amplitude threshold had been gradually increased using sEPSC detection algorithms indicative of there being no separation between the amplitude distributions of real sEPSCs and membrane current noise fluctuations. Although replacing voltage clamp with current clamp recordings addresses voltage clamp errors, the amplitude of sPSPs is equally attenuated by cell dendritic trees (Williams and Stuart, 2002; Nevian et al., 2007; Larkum et al., 2009). Just like for voltage clamp recordings, analyses of detected sEPSPs using a varying detection amplitude threshold indicates there being no separation between amplitude distributions of real sEPSPs and membrane potential noise fluctuations. These observations essentially spell doom for using direct amplitude measurement of sPSPs and sPSCs alone as an estimate of the quantal size in the CNS.
The same method could, however, be used to measure the properties of noise fluctuations allowing to subsequently isolate signal and noise components. As the first step in addressing limitations of estimating amplitudes and frequencies of sPSPs stemming from neuronal membrane biology and developing a method to reliably estimate the quantal size in the CNS, we developed an effective sPSC and sPSP detection algorithm, called ‘minis’. We quantified the performance of ‘minis’ in a highly objective manner using measures of signal detection theory – namely, ROC curves and d’ alongside measures of sensitivity (TPR) and FPR (1 – specificity) – and demonstrated a superior performance of this algorithm relative to other two most popular algorithms in the field of synaptic function research: Mini Analysis and Clampfit. Our algorithm had a performance edge over the other two algorithms not only for detecting moderately-sized sEPSPs but across a wide range of realistic sEPSP amplitudes, as well as across a wide range of sEPSP incidence frequencies and especially within a realistic 13-80 Hz range of frequencies. As a necessary step in building a reliable quantal size estimation method, we succeeded in developing an algorithm for detecting postsynaptic events that is transparent, objectively evaluated, and flexible.
Notwithstanding an impressive performance of ‘minis’ algorithm, the algorithm has important limitations and there are caveats to the method of assessing its performance. First of all, ‘minis’ algorithm, just like the other two detection algorithms, cannot by itself alone be used to estimate the quantal size in the CNS. Due to amplitude attenuation of spontaneous postsynaptic events and the need to use amplitude detection thresholds, the estimated amplitude will always be only an upper limit on the actual amplitude. We have also demonstrated that detection of simulated sEPSPs is impaired on the rise and decay phases of membrane potential and that the severity of this impairment increases for increasing membrane potential rate of change, as well as with decreasing distance to the nearest neighbour. Therefore, mean amplitude estimate is not only affected by the presence of noise and amplitude attenuation but also by the presence of other spontaneous postsynaptic events. With regards to estimating the frequency of spontaneous postsynaptic events, all of the above limitations apply. All of these factors reduce event detectability and, therefore, result in an underestimated frequency value. Despite these limitations however, ‘minis’ fares better on this front compared to the other two algorithms. We saw that ‘minis’ performance is superior on rise and decay phases of membrane potential, as well as for detecting postsynaptic events in close proximity to other events. To our knowledge, this is the first time these problems are addressed, their impact is quantified in an objective manner, and their implications are discussed.
Other caveats concern the method for assessing the performance of detection algorithms. We did not use entirely realistic distributions for simulating sEPSPs in terms of their frequencies, amplitudes, and their rise times. However, the shapes of these distributions are not entirely known and this is an empirical question that remains unanswered to this day. Because of this reason we refrained for trying to address this question here and avoided making empirically dubious assumptions. Instead, we explored a range of scenarios where simulation frequencies ranged from more to less realistic numbers. We explored the effects of varying noise levels, as well as a range of plausible amplitude values. We believe that the large size of parameter space we explored does allow us to conclude that the three algorithms differ systematically in their performance with one particular algorithm – ‘minis’ – being superior to the other two.
We have not constructed ROC curves in a usual fashion by varying a detection threshold of each algorithm. Unfortunately, we were limited by the nature of algorithms, specifically Clampfit which does not use a detection threshold. However, our manipulations of varying the incidence frequency of simulated events, as well as varying the amplitude of noise used in our simulations allowed us to reliably mimic effects of varying the detection threshold. Moreover, we are aware of one particular issue regarding detection response classification as correct rejections. A proper definition of correct rejections would require to treat every series data sample as a potential noise event that could either be falsely detected as a signal event or correctly rejected as a noise event. Unfortunately, such definition would result in large numbers of correct rejections overwhelming the number of other detection measures like true positives, false alarms, and misses and, therefore, would make detection performance analysis not particularly meaningful. Alternatively, treating every 10 ms noise window as a basis for correct rejection would still produce a relatively large number of correct rejections. There is also no reason why a single noise event should be limited to a single 10 ms window. Therefore, we thought that a reasonable practical definition of correct rejections as rejected prominent noise fluctuations fared better than alternatives. Having explicit and transparent albeit imperfect methods to quantify the performance is better than relying on subjective examination of recording traces alone.
Finally we would like to address the question of whether we used optimal detection settings for each algorithm. Mini Analysis settings were similar to those of ‘minis’. The two algorithms are somewhat similar and chosen settings were observed to give the best performance for both algorithms when detecting simulated sPSPs after a considerable exploration. In case there is doubt about our method, we have made our recorded and simulated data and Matlab analysis code publicly available. With regards to using Clampfit to analyse simulated data, we used 9 templates constructed in a typical fashion by averaging a number of waveforms with similar shapes but separating qualitatively different shapes. These templates have also been made available publicly. We tested different detection regimes with templates being constructed based on smoothed traces or unsmoothed ones and comparing detection of smoothed and unsmoothed recording traces and chose the best regime. However, we did not construct new templates for every new recording or every new simulation condition, because this is extremely effort-consuming and is not an efficient way to conduct this type of analysis. Undoubtedly it would have improved Clampfit detection performance but the effect would definitely have been limited. It would actually be an improvement of the Clampfit algorithm to have a number of templates not limited to 9 and having a wide range of inbuilt templates available for users to choose from. This is, however, a separate matter outside the scope of this study. Given these considerations, we are confident in the validity of our approach.
In the discussion of errors made by different detection algorithms we alluded to the usefulness of being able to estimate properties of noise membrane potential fluctuations. Noise events could potentially be eliminated from combined signal and noise event distribution by direct subtraction. Or rather they could be used to more accurately reverse-engineer the signal distribution with the help of simulations. The future paper will explore this possibility and present a quantal analysis method based on ‘minis’ spontaneous postsynaptic event detection algorithm.
Acknowledgements
None
Footnotes
Conflict of interest statement: The authors declare no competing financial interests.