RT Journal Article
SR Electronic
T1 Developing a modern data workflow for evolving data
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 344804
DO 10.1101/344804
A1 Yenni, Glenda M.
A1 Christensen, Erica M.
A1 Bledsoe, Ellen K.
A1 Supp, Sarah R.
A1 Diaz, Renata M.
A1 White, Ethan P.
A1 Ernest, S.K. Morgan
YR 2018
UL http://biorxiv.org/content/early/2018/07/24/344804.abstract
AB Data management and publication are core components of the research process. An emerging challenge that has received limited attention in biology is managing, working with, and providing access to data under continual active collection. “Evolving data” present unique challenges in quality assurance and control, data publication, archiving, and reproducibility. We developed a evolving data workflow for a long-term ecological study that addresses many of the challenges associated with managing this type of data. We do this by leveraging existing tools to: 1) perform quality assurance and control; 2) import, restructure, version, and archive data; 3) rapidly publish new data in ways that ensure appropriate credit to all contributors; and 4) automate most steps in the data pipeline to reduce the time and effort required by researchers. The workflow uses two tools from software development, version control and continuous integration, to create a modern data management system that automates the pipeline.