Background
Many areas of neuroscience are now critically dependent on computational tools to help understand the large volumes of data being created. Furthermore, computer models are increasingly being used to help predict and understand the function of the nervous system. Many of these computations are complex and often cannot be concisely reported in the methods section of a scientific article. In a few areas there are widely used software packages for analysis (e.g., SPM, FSL, AFNI, BrainVoyager, FreeSurfer in neuroimaging) or simulation (e.g. NEURON, NEST, Brian). However, we often write new computer programs to solve specific problems in the course of our research. Some of these programs may be relatively small scripts that help analyze all of our data, and these rarely get described in papers. As authors, how best can we maximize the chances that other scientists can reproduce our computations or reuse our methods on their data? Is our research reproducible1?
To date, the sharing of computer programs underlying neuroscience research has been the exception (see below for some examples), rather than the rule. However, there are many potential benefits to sharing these programs, including increased understanding and reuse of your work. Furthermore, open source programs can be scrutinized and improved, whereas the functioning of closed source programs remains forever unclear2. Funding agencies, research institutes and publishers are all gradually developing policies to reduce the withholding of computer programs relating to research3. The Nature family of journals has recently published opinion pieces in favor of sharing whatever code is available, in whatever form4,5. More recently, since October 2014, all Nature journals require papers to include a statement declaring whether the programs underlying central results in a paper are available. In April 2015 Nature Biotechnology offered recommendations for providing code with papers and began asking referees to give feedback on their ability to test code that accompanies submitted manuscripts6. In July 2015 F1000Research stated that "Software papers describing non-open software, code and/or web tools will be rejected” (http://f1000research.com/channels/f1000-faculty-reviews/for-authors/article-guidelines/software-tool-articles). Also in July 2015, BioMed Central introduced a minimum standards of reporting checklist for BMC Neuroscience and several other journals, requiring submissions to include a code availability statement and for code to be cited using a DOI or similar unique identifier7. We believe that all journals should adopt policies that highly encourage, or even mandate, the sharing of software relating to journal publications.
What should be shared?
It may not be obvious what to share, especially for complex projects with many collaborators. As advocated by Claerbout and Donoho, for computational sciences the scholarship is not the article; the ”scholarship is the complete software […]”8,9. So, ideally, you should share as much code and data as is needed to allow others to reproduce your work, but this may not be possible or practical. However, it is expected that you will share key parts of the work, e.g. implementations of novel algorithms or analyses. At a minimum, we suggest following the recommendation of submission of work to ModelDB10, i. e. to share enough code, data and documentation to allow at least one key figure from your manuscript to be reproduced. However, by adopting appropriate software tools, as mentioned in the next section, it is now relatively straightforward to share the materials required to regenerate all figures and tables. On the other hand, code that is not novel because it is already available, or that you feel that is unlikely to be of use to others need not be shared. This includes code that performs simple preprocessing or statistical tests, or code that deals with local computing issues such as hardware and software configurations. Finally, if your work is computationally intensive and requires a long time to run (e.g. many weeks), you may prefer to provide a small “toy” example to demonstrate your code.
By getting into the habit of sharing as much as possible, not only do you help others who wish to reproduce your work (which is a basic tenet of the scientific method), you will be helping other members of your laboratory, or even yourself in the future. By sharing your code publicly, you are more likely to write higherquality code11, and you will know where to find it after you’ve moved on from the project12, rather than the code disappearing on a colleague’s laptop when they leave your group. You will be part of a community and benefit from the code shared by others, thus contributing to a reduction in software development time for yourself and others.
Simple steps to help you share your code
Once you have decided what you plan to share, here are some simple guidelines for how to share your work. Ideally, these principles should be followed throughout the lifetime of your project, not just at the end when you wish to publish your results. Guidelines similar to these have been proposed recently in many areas of science13–15, suggesting that they are part of norms that are emerging across disciplines. In the ‘further reading’ section below, we list some specific proposals from other fields that expand on the guidelines we suggest here.
Version control
Use a version control system (such as Git) to develop the code16. The version control database can then be easily and freely shared with others using sites such as http://github.com17 or https://bitbucket.org. These sites allow you fine control over private versus public access to your code. This means that you can keep your code repository private during its development, and then publicly share the repository at a later stage e.g. at the time of publication. It also makes it easy for others to contribute to your code, and to adapt it for their own uses.
Persistent URLs
Generate stable URLs (such as a DOI) for key versions of your software. Unique identifiers are a key element in demonstrating the integrity and reproducibility of research18, and allow referencing of the exact version of your code used to produce figures. DOIs can be obtained freely and routinely with sites such as http://zenodo.org and http://figshare.com. If your work includes computer models of neural systems, you may wish to consider depositing these models in established repositories such as ModelDB10, Open Source Brain19, INCF Software Center20 or NITRC21. Some of these sites allow for private sharing of repositories with anonymous peer reviewers. Journal articles that include a persistent URL to code deposited in a trusted repository meet the requirements of level two of the ‘analytic methods (code) transparency’ standard of the TOP guidelines13.
License
Choose a suitable license for your code to assert how you wish others to reuse your code. For example, to maximize reuse, you may wish to use a permissive license such as MIT or BSD22. Licenses are also important to protect you from others misusing your code. Visit http://choosealicense.com/ to get a simple overview of which license to choose, or http://www.software.ac.uk/resources/guides/adopting-open-source-licence for a detailed guide.
Etiquette
When working with code written by others, observe Daniel Kahneman’s ’re-producibility etiquette’23 and have a discussion with the authors of the code to give them a chance to fix bugs or respond to issues you have identified before you make any public statements. Cite their code in an appropriate fashion.
Documentation
Contrary to popular expectations, you do not need to write extensive documentation or a user’s guide for the code still be to useful to others4. However, it is worth providing a minimal README file to give an introduction to what the code does, and how to run it. For example, you should provide instructions on how to regenerate a key result, or a particular figure from a paper. Literate programming methods, where code and narrative text are interwoven in the same document, make documentation semi-automatic and can save a lot of time when preparing code to accompany a publication24,25. However, these methods admittedly take more time to write in the first instance, and you should be prepared to rewrite documentation when rewriting code. In any cases, well-documented code allows for easier re-use and checking.
Tools
Consider using modern, widely used software tools that can help with making your computational research reproducible. Many of these tools have already been used in neuroscience and serve as good examples to follow, for example Org mode26, IPython/Jupyter27 and Knitr28. Virtualization environments, such as VirtualBox appliances and Docker containers, can also be used to encapsulate or preserve all of the computational environment so that other users can run your code without having to install numerous dependencies29.
Case studies
As well as the examples listed above in Tools26–28, there are many prior examples to follow when sharing your code. For example, some prominent examples of reproducible research in computational neuroscience include Vogels et al.30 and Waskom et al.31; see https://github.com/WagnerLabPapers for details. The ModelDB repository contains over 1000 computational models deposited with instructions for reproducing key figures to papers e.g. https://senselab.med.yale.edu/ModelDB/showModel.cshtml?model=93321 for a model of activity-dependent conductances32.
Data
Any experimental data collected alongside the software should also be released. For small datasets, this could be stored alongside the software, although it may be preferable to store experimental data separately in an appropriate repository. Both PLOS and Scientific Data maintain useful lists of subject-specific and general repositories for data storage, see http://journals.plos.org/plosbiology/s/data-availability#loc-recommended-repositories and http://www.nature.com/sdata/data-policies/repositories.
Standards
Use of community standards where appropriate should be encouraged. In computational neuroscience for example, PyNN33 and NeuroML34 are widely used formats for making models more accessible and portable across multiple simulators.
Tests
Testing the code has long been recognized as a critical step in software industry but the practice is not widely adopted yet by researchers. We recommend including test suites that demonstrate the code is producing the correct results35. These tests can be at a low level (testing each individual function, called unit testing) or at a higher level (e.g. testing that the program yields correct answers on simulated data)36. Linking tests to continuous integration services (such as Travis CI, https://travisci.org) allows these tests to be automatically run each time a change is made to the code, ensuring failing tests are immediately flagged and can be dealt with quickly.
Further reading (note to editor: please make this a box feature)
Varsha Khodiyar 2015 Code Sharing — read our tips and share your own. Scientific Data Blog, February 19, 2015. http://blogs.nature.com/scientificdata/2015/02/19/code-sharing-tips/
Leveque Randall 2013. Top ten reasons to not share your code (and why you should anyway). SIAM News, April 2013, http://sinews.siam.org/DetailsPage/tabid/607/ArticleID/386/Top-Ten-Reasons-To-Not-Share-Your-Code-and-why-you-should-anyway.aspx Stodden V., & Miguez, S., 2014. Best practices for computational science: software infrastructure and environments for reproducible and extensible research. Journal of Open Research Software. 2(1), p.e21. DOI: http://doi.org/10.5334/jors.ay Stodden, V., Leisch, F., & Peng, R. (Eds.). (2014). Implementing reproducible research. CRC press, Chapman and Hall.
Halchenko, Y. O. and Hanke, M. (2015). Four aspects to make science open “by design” and not as an after-thought. GigaScience, 4. DOI: http://doi.org/10.1186/s13742-015-0072-7
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig E (2013) Ten simple rules for reproducible computational research. PLoS Comput Biol 9:e1003285.
Online communities discussing code sharing (note to editor: please make this a box feature)
StackExchange and related projects
StackExchange is a network of free and highly active question-and-answer websites. Two members of the network are relevant to questions of code sharing: http://stackoverflow.com/ which is dedicated to questions about programming in any language in any context, and http://academia.stackexchange.com/questions/tagged/reproducible-research which is focused questions relating to reproducible research in academic context. A related project is https://neurostars.org/ which is a similar free public Q&A website focused on neuroinformatics questions, and with many questions on software packages, etc.
Scientists for Reproducible Research
This is an international multi-disciplinary email list that discusses a wide range of issues relating to code sharing: https://groups.google.com/forum/#!forum/reproducible-research
GitHub
GitHub is an online repository for computer code and programs that has a large community of researchers that develop and share their code openly on the site. GitHub is the largest and most active code sharing site (others include BitBucket and GitLab) and has convenient tools for facilitating efficient collaborative coding37. If you are using an open source program you may find a community of users and developers active on GitHub, where you can ask questions and report problems.
Closing remarks
Changing the behaviors of neuroscientists so that they make their code more available will likely be resisted by those who do not see the community benefits as outweighing the personal costs of the time and effort required to share code38. The community benefits, in our view, are obvious and substantial: we can demonstrate more robustly and transparently the reliability of our results, we can more easily adapt methods developed by others to our data, and the impact of our work increases as others can similarly reuse our methods on their data. Thus, we will endeavor to lead by example, and follow all these practices as part of our future work in all scientific publications. Even if the code we produce today will not run ten years from now, it will still be a more precise and complete expression of our analysis than the text of the methods section in our paper.
However, exhortations such as this editorial are only a small part of making code sharing a normal part of doing neuroscience; many other activities are important. All researchers should be trained in sound coding principles; such training is provided by organizations such as Software Carpentry36 and through national neuroinformatics nodes, e.g. http://python.g-node.org. Furthermore, we should request code and data when reviewing, and submit to and review for journals that support code sharing. Grant proposals should be checked for mentions of code availability, and we should encourage efforts toward openness in hiring, promotion, and reference letters39. Funding agencies and publishers should also consider mandating code sharing by default. This combination of efforts on a variety of fronts will increase the visibility of research accompanied by open source code, and demonstrate to others in the discipline that code sharing is a desirable activity that helps move the field forward.
We believe that the sociological barriers to code sharing are harder to overcome than the technical ones. Currently, academic success is strongly linked to publications and there is little recognition for producing and sharing code. Code may also be seen as providing a private competitive advantage to researchers. We challenge this view and propose that code be regarded as part of the research products which should be shared by default, and that there should be an obligation to share code for those conducting publicly funded research. We hope our code availability review will help establish such sharing as the norm. Moreover, we are advocating for code sharing as part of a broader culture change embracing transparency, reproducibility, and re-usability of research products.
Acknowledgments
This article is based upon discussions from a workshop to encourage sharing in neuroscience, held in Cambridge, December 2014. It was financially supported and organized by the International Neuroinformatics Coordinating Facility (http://www.incf.org), with additional support from the Software Sustainability institute (http://www.software.ac.uk).