PT  - JOURNAL ARTICLE
AU  - Gary Kane
AU  - Gonçalo Lopes
AU  - Jonny L. Saunders
AU  - Alexander Mathis
AU  - Mackenzie W. Mathis
TI  - Real-time, low-latency closed-loop feedback using markerless posture tracking
AID  - 10.1101/2020.08.04.236422
DP  - 2020 Jan 01
TA  - bioRxiv
PG  - 2020.08.04.236422
4099  - http://biorxiv.org/content/early/2020/11/24/2020.08.04.236422.short
4100  - http://biorxiv.org/content/early/2020/11/24/2020.08.04.236422.full
AB  - The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI), and integration into (2) Bonsai and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.Competing Interest StatementThe authors have declared no competing interest.