A pragmatic reevaluation of the efficacy of nonhuman primate optogenetics

A recent paper published in Neuron by Tremblay et al. (2020) introduces an openly available resource detailing published and unpublished studies using optogenetics to manipulate the nonhuman primate (NHP) brain. The open science efforts of the team are important and rare in NHP neuroscience, but the conclusions drawn about the success rate of optogenetics in the NHP brain are problematic for quantitative and theoretical reasons. Quantitively, the analyses in the paper are performed at a level relevant to the rodent but not NHP brain (single injections) and individual injections are clustered within a few monkeys and a few studies. Theoretically, the report makes strong claims about the importance of the technology for disease related functional outcomes, but behavior was not widely tested. The original article reports a 91% success rate for optogenetic experiments in NHPs based on the presence of any outcome (histological, physiological, or behavioral outcomes) after an injection of viral vectors. Reanalysis of the data clustered at the level of brain region and animal with a modified definition of success that included a behavioral and biological effect reveals that the rate of success was approximately 62.5%, and only 53.1% for strong outcomes, in experiments that attempted to measure a behavioral and a biological effect. Only 6% of the experiments in the total database successfully achieved histological, physiological, and behavioral endpoints. This calls into question the current efficacy of optogenetic techniques in the NHP brain and suggests that we are a long way from being able to leverage them in “the service of patients with neurological or psychiatric conditions”.


Abstract
A recent paper published in Neuron by Tremblay et al. (2020) introduces an openly available resource detailing published and unpublished studies using optogenetics to manipulate the nonhuman primate (NHP) brain. The open science efforts of the team are important and rare in NHP neuroscience, but the conclusions drawn about the success rate of optogenetics in the NHP brain are problematic for quantitative and theoretical reasons. Quantitively, the analyses in the paper are performed at a level relevant to the rodent but not NHP brain (single injections) and individual injections are clustered within a few monkeys and a few studies. Theoretically, the report makes strong claims about the importance of the technology for disease related functional outcomes, but behavior was not widely tested. The original article reports a 91% success rate for optogenetic experiments in NHPs based on the presence of any outcome (histological, physiological, or behavioral outcomes) after an injection of viral vectors. Reanalysis of the data clustered at the level of brain region and animal with a modified definition of success that included a behavioral and biological effect reveals that the rate of success was approximately 62.5%, and only 53.1% for strong outcomes, in experiments that attempted to measure a behavioral and a biological effect. Only 6% of the experiments in the total database successfully achieved histological, physiological, and behavioral endpoints. This calls into question the current efficacy of optogenetic techniques in the NHP brain and suggests that we are a long way from being able to leverage them in "the service of patients with neurological or psychiatric conditions".
Optogenetic approaches allow cellular physiology to be altered using light, opening the door to highly temporally and spatially precise manipulation of neural circuits. These approaches have led to an explosion of publications on circuit mechanisms of behavior, primarily in mice, but the implementation of these methods in the nonhuman primate (NHP) brain has lagged substantially. In a recent paper published in Neuron (Tremblay et al., 2020), a large team of neuroscientists introduced an openly available resource detailing published and unpublished studies that use optogenetic techniques to manipulate the NHP brain. The effort reflects contributions from 45 research laboratories, with 1042 individual data points included (552 previously unpublished) representing experiments in 198 monkeys.
The resource detailed in Tremblay et al. (2020) aggregates an enormous amount of technical expertise on neurosurgical placement of viral vectors in the NHP brain, a substantial contribution to the field. Moreover, meta-and open science efforts in NHP neuroscience are critical and until very recently, very rare. By metascience, we refer to a collection of approaches that include meta-analyses (analyses of existing analyses), pooling of data across laboratories, splitting data collection efforts strategically across laboratories to guard against protocols being laboratory specific, and gathering published and unpublished data in shared resources, among others (e.g., the "many labs" projects; e.g., Ebersole et al., 2016;Klein et al., 2014Klein et al., , 2018. These efforts are important for NHP neuroscience because our field typically uses small sample sizes and has had a historical bias against replicating experiments, constraints that are both ethical and practical. By open science, we mean the sharing of scientific resources including primary materials and data (for an NHP specific example see Milham et al., 2020;Milham et al., 2018). Combining meta-and open data practices, as the optogenetics database does, sets the stage for scientists to be able to evaluate the effectiveness of methods or experiments and to determine where efforts are needed to move science forward. This particular effort deserves significant recognition because of the large number of unpublished data sets and the number of "unsuccessful" studies included in it. These two types of data are critical for determining if an approach is truly effective. Without inclusion of those types of data, particularly in the context of a publication culture that focuses heavily on null hypothesis testing and rewards significant effects to a greater degree than those that are not significant (with regards to ease and visibility of publications), the success of an approach is likely to be overestimated. In that vein, efforts that gather data that is both published and unpublished, as well as successful and unsuccessful, on the efficacy of new tech is both important and laudable.
It is the strength of the authors efforts in compiling this resource that allows us to draw different conclusions from them about the usefulness of optogenetics in nonhuman primates. In turn this curtails the impact of this method in the service of understanding and developing treatment interventions for psychiatric and neurologic disease. Two issues are notable. First, the definition of experimental success was overly broad on two dimensions. Studies were considered successful if they generated histological, or neurophysiological, or behavioral effects, and what was counted as a successful behavioral effect included subjective evaluations of narrative accounts of animals' behaviors as well as effects qualitatively judged to be "weak". At a minimum, the successful use of optogenetics in behavioral or systems neuroscience (separate from its translational relevance) would require some evidence that neurons have been modulated either via histology of viral expression or altered neuronal activity measured in addition to an optogenetic mediated change in behavioral outcomes. This necessitates looking at conjunctions of successcases in which there are both behavioral and histological effects or cases in which there are both behavioral and neurophysiological effects.
A second issue is that the unit of analysis for descriptive statistics in the original report is the single injection. It is perhaps not surprising that this database is organized injection-byinjection, given the success of single injections in rodent experiments to generate functional outcomes (Deisseroth, 2012). But, it is unclear whether single injections would have measurable outcomes in primates, particularly meaningful behavioral outcomes that are clearly grounded in neurophysiology and/or histology. Given that multiple injections of a viral vector into brain region are often required to achieve adequate coverage in larger NHP brains, a more appropriate basis to judge the relative success of experiments would include an analysis at the level of brain region and monkey and taking into consideration multiple markers of success simultaneously.
The summary of the resources in the paper by Tremblay states that optogenetics in NHPs has a 91% success rate ( Figure 4D of Tremblay et al. 2020) -based on outcomes related to modulation in neural activity via histological outcomes measured in histology, or neurophysiology, or behavior, including injections where effects were weak or mixed. When only strong outcomes were considered, only 76% of injections were considered successful. In addition to being predicated on a broad definition of success, those rates are based on individual injections that may or may not have behavioral or physiological effects in NHP. Most importantly, the data points are clustered within animals, and it is animals, not neurons, which produce behaviors (Tinbergen, 1963).

Methods
We examined the publicly-available database described in the Neuron paper and then requested the unredacted spreadsheet from the first author so that we could unambiguously determine which experiments were performed in a single animal, something that was not possible using only the publicly posted database (https://osf.io/mknfu/). This allowed us to evaluate the number of successful experiments at the level of brain regions and animals, rather than single injections of viral vectors. In this way, we considered a single "experiment" to be one or more injections of the same viral vector into one brain area in one animal. Injections into two or more brain regions in the same monkey would be considered as two separate experiments in this analysis, even though they may be biologically constrained by being in the same animal. The choice of level of analysis in clustered data has a significant impact on outcomes, given that adjacent injections into the same area in the same animal could reasonably be considered to not be independent experiments (see also Aarts et al., 2014). It is also this level of analysis that is most pertinent to behavioral neuroscience investigations of how neural circuits control behavior.
The database we received included 1085 injections. This was increased from the 1042 injections reported in the original publication, reflecting the utility of the database as an open resource. We read this database into R and assigned experiment identifiers based on unique conjunctions of laboratory, animal identifier, and brain region, resulting in 383 unique experiments. We then read this database back out and carried out analyses to determine success rates at the experiment level. We categorized each injection as missing, failed, weak success, or strong success for histology, physiology, and behavior, attempting to follow the descriptions in Tremblay et al. 2020. Any experiment that had at least one injection classified as "strong success" was categorized as a strong success for that domain; experiments that had no strong success but at least one "weak success" were categorized as weak success, and so forth. Thus, this categorization may still overestimate the true success rate in cases where many injections were placed in a brain area but only a subset had successful effects. The redacted database with experiment identifiers and associated R analysis script are available at https://osf.io/j7rm2/.

Results and Discussion
When we analyzed success rates using monkey and brain region rather than injection as the unit of analysis a somewhat different picture emerged. First, very few experiments assessed all three outcomes. Only 39 out of the 383 cases we identified in the database (10.2%) evaluated the efficacy of an optogenetic manipulation in terms of histological, physiological, and behavioral effects. In this small subset of experiments there was a 59% (23/39) success rate on all three outcomes. The majority of these successes were in premotor cortex (7/9) and visual cortex (7/12).
Put another way, only 6% (23/383) of the experiments attempted and reported were successful in using optogenetics to bridge the three levels of analysis, from neurons to behavior.
When we examined each outcome domain separately, we found high success rates, as reported in the original paper, for histological (226/261, 86.6%) and neurophysiological (168/197, 85.3%) verifications of optogenetics. Success rates using behavioral outcomes (47/74, 63.5%) to validate optogenetic manipulations were lower. These success rates are lower than those reported and highlighted in the original paper, especially when we only considered strong effects: histology (72.4%), neurophysiology (75.1%), and behavior (52.7%). Moreover, these proportions consider only cases in which histology, physiology, or behavior was tested (Figure 1). The different success rates between histological and functional evaluations of optogenetics suggests the impediment in using optogenetics to study the primate brain is not merely viral transduction per se, but rather difficulties in modulating sufficient numbers of neurons in the larger primate brain (Gradinaru, 2020). Without question the further development of NHP specific genetic tools, rather than simply porting tools developed in rodent models to NHP, is necessary meet the lofty goal of using NHPs as a translational bridge between behavioral and systems neuroscience and the clinic.
Speaking practically, a successful optogenetic experiment involves evaluating success in one or more of the outcome domains mentioned above. With the end goal of using optogenetics in NHPs to study the neural control of behavior at the level of neurons and circuits, it is reasonable to expect that behavioral manipulations should be validated in combination with either histology or neurophysiology. Therefore, we considered a definition of success as yielding a combination of effects (either behavior ∩ neurophysiology or behavior ∩ histology). Using a broad definition that included both weak and strong effects of success, there was a 62.5% (45/72) success rate among experiments that reported optogenetics effects on behavior and neurophysiology and a 62.5% (25/40) success rate among experiments that reported optogenetic effects on both behavior and histology. When we again adopted Tremblay and colleagues' approach in considering only strong effects, success rates dropped to 48.6% (35/72) for studies that evaluated behavior in combination with neurophysiology and to 57.5% (23/40) for studies that evaluated behavior in combination with histology. Notably, these results are for experiments pooled across brain regions and laboratories. Perhaps more concerning for scholars interested in psychiatric Figure 1. Numbers of experiments with histological, physiological, or behavioral outcomes indicating experiments that did not have/report a particular outcome (grey), cases where the experiment failed to detect effects (orange), cases where weak effects were detected (yellow), and cases where strong effects were detected (blue). and neurological diseases, is that 66.3% (254/383) of the experiments evaluated optogenetics in primary sensory or motor areas. Cortex -particularly primate cortex -is heterogenous in its structure and success in one region does not guarantee success across the entire brain. Modulation of those areas should produce fairly stereotyped, clear functional (i.e., behavioral) outcomes, yet only 29 (11.4%; 29/254) of those experiments even evaluated behavior and physiology.
In our view, a 62.5% (all effects) or 53.1% (strong effects only) success rate (averaged across the two conjunctions that include behavior as a functional outcome) is meaningfully different from the 91% (all effects) and 76% (strong effects only) success rate suggested by Tremblay et al. (2020). The lower success rate that we identified suggests that optogenetics is still in development as a tool in NHPs, rather than ready to be deployed to study functional outcomes in primates or to intervene in clinical contexts. This may reflect its true efficacy in mammals, previously masked by the lower costs and higher throughput associated with rodent models. Or, it may reflect issues with deploying viral techniques in primates -a perspective supported by mixed results of experiments using DREADDs (Designer Receptors Exclusively Activated By Designer Drugs) to manipulate behavior in primates, despite histological validation (e.g., Eldridge et al., 2016;Upright et al., 2018). Recognizing that further research is needed to determine the origin of the low success rates, it is nevertheless the case that at the moment success in obtaining strong effects across multiple functional outcomes when pursuing optogenetics in NHPs amounts to flipping a coin. This warrants a very different conclusion from the one that a reader might reasonably draw after reading the survey by Tremblay et al. (2020).
Even if we were to imagine that optogenetics was ready to be deployed in the way described by Tremblay and colleagues, there are a number of other major challenges that exist when translating the technology from use in rodents to use in primates, that must be considered. For example, for optogenetics to work, a light probe must be proximate to the structure being manipulated and this represents a significant engineering challenge in deep areas of the primate brain. Accessing deep structures -for example the amygdala, hippocampus, and subgenual cingulate cortex -that are part of circuitry relevant to psychiatric and neurological disorders without damaging the areas through which probes are passed is a significant engineering challenge that has not yet been sufficiently addressed. We are a long way from conquering the engineering problems, much less interpretive issues, that will allow for the "development of clinical technologies relying on optogenetics to control neural populations and pathways with unprecedented precision" as the authors claim (p. 12-13). It is shortsighted to assume that translation will work when so many basic experiments targeting areas with known anatomical structure and functions fail. When optogenetics sufficiently matures to be used as a reliable tool in primate systems neuroscience, it is more likely to allow elegant circuit dissections of behavior that can inform existing treatments than to be a frontline treatment for psychiatric disorders (Lüscher & Pollak, 2016).
Given the data in this open resource, we propose that the state of the technology for use in NHPs may be oversold. This has significant consequences for what work is funded and can be carried out, and creates a tension between hypothesis-driven work that speeds scientific discovery and the need to employ methods that are unreliable in NHPs. Like much of the tool development in systems neuroscience, optogenetics has primarily been developed in rodent models, and so the translation of these approaches to NHPs is not necessarily straightforward or trivial. The drive for technological innovation from the BRAIN Initiative has created a primacy of technology such that optogenetics and related methods are often viewed by science funders as a better approach for systems neuroscience than "classical" experimental methods in spite of interpretive difficulties with transient manipulations of neural activity (e.g., Otchy et al., 2015). Fancy ideas that are executed reliably and cheaply are just as important as fancy tools in accelerating scientific discovery.
As the drive for new technology pushes forward, we propose that the ability to manipulate behavior using optogenetics (or other genetic tools, such as DREADDs) should remain the gold standard by which we evaluate its success. This view echoes recent calls to reexamine the utility of behavior itself in neuroscience (Krakauer et al., 2017;Niv, 2020). We would suggest that an approach with a current success rate of 53.1% or 62.5% (on average) should be evaluated differently than one with a success rate of 76% or 91% in terms of choosing a method for interrogating function of neural circuits. Given this, we propose that more effort needs to be devoted to validating and optimizing optogenetics and related technologies specifically in NHPs. Validation and optimization is needed before such tools can be efficiently deployed in experimental contexts, and certainly before claims about their translational relevance are made. Such basic science efforts may not have the glamour or appeal of new technological invention or the deployment of new technology in experimental contexts, but they are vital for moving the field forward and ultimately for generating new effective treatments and interventions of psychiatric and neurological disease.

Conclusion
The data provided by this open resource allow us to conclude, upon appropriate recounting, that optogenetic manipulation is not yet at a state of development where it can be reliably applied to investigate functional effects of neural circuit manipulations in NHPs. Nonetheless, the meta-and open science nature of the database, as well as the efforts of project leader and the individual scientific teams, should be celebrated because they allow for a gestalt evaluation of the efficacy of optogenetics in the NHP brain.