Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

SLEAP: Multi-animal pose tracking

View ORCID ProfileTalmo D. Pereira, Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, Grace McKenzie-Smith, View ORCID ProfileSarah D. Kocher, View ORCID ProfileAnnegret L. Falkner, View ORCID ProfileJoshua W. Shaevitz, View ORCID ProfileMala Murthy
doi: https://doi.org/10.1101/2020.08.31.276246
Talmo D. Pereira
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Talmo D. Pereira
Nathaniel Tabris
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Junyu Li
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Shruthi Ravindranath
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eleni S. Papadoyannis
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Z. Yan Wang
3Dept. of Ecology and Evolutionary Biology, Princeton University
4Lewis-Sigler Institute for Integrative Genomics, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
David M. Turner
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Grace McKenzie-Smith
2Dept. of Physics, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sarah D. Kocher
3Dept. of Ecology and Evolutionary Biology, Princeton University
4Lewis-Sigler Institute for Integrative Genomics, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Sarah D. Kocher
Annegret L. Falkner
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Annegret L. Falkner
Joshua W. Shaevitz
1Princeton Neuroscience Institute, Princeton University
2Dept. of Physics, Princeton University
4Lewis-Sigler Institute for Integrative Genomics, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Joshua W. Shaevitz
  • For correspondence: shaevitz@princeton.edu mmurthy@princeton.edu
Mala Murthy
1Princeton Neuroscience Institute, Princeton University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mala Murthy
  • For correspondence: shaevitz@princeton.edu mmurthy@princeton.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation to quantify and model natural animal behavior. This has led to important advances in deep learning-based markerless pose estimation that have been enabled in part by the success of deep learning for computer vision applications. Here we present SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning. This system is capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. SLEAP implements several complementary approaches for dealing with the problems inherent in moving from single-to multi-animal pose tracking, including configurable neural network architectures, inference techniques, and tracking algorithms, enabling easy specialization and tuning for particular experimental conditions or performance requirements. We report results on multiple datasets of socially interacting animals (flies, bees, and mice) and describe how dataset-specific properties can be leveraged to determine the best configuration of SLEAP models. Using a high accuracy model (<2.8 px error on 95% of points), we were able to track two animals from full size 1024 × 1024 pixel frames at up to 320 FPS. The SLEAP framework comes with a sophisticated graphical user interface, multi-platform support, Colab-based GPU-free training and inference, and complete tutorials available, in addition to the datasets, at sleap.ai.

Competing Interest Statement

The authors have declared no competing interest.

Footnotes

  • https://sleap.ai

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC 4.0 International license.
Back to top
PreviousNext
Posted September 02, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
SLEAP: Multi-animal pose tracking
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
SLEAP: Multi-animal pose tracking
Talmo D. Pereira, Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, Grace McKenzie-Smith, Sarah D. Kocher, Annegret L. Falkner, Joshua W. Shaevitz, Mala Murthy
bioRxiv 2020.08.31.276246; doi: https://doi.org/10.1101/2020.08.31.276246
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
SLEAP: Multi-animal pose tracking
Talmo D. Pereira, Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, Grace McKenzie-Smith, Sarah D. Kocher, Annegret L. Falkner, Joshua W. Shaevitz, Mala Murthy
bioRxiv 2020.08.31.276246; doi: https://doi.org/10.1101/2020.08.31.276246

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Animal Behavior and Cognition
Subject Areas
All Articles
  • Animal Behavior and Cognition (4113)
  • Biochemistry (8815)
  • Bioengineering (6519)
  • Bioinformatics (23463)
  • Biophysics (11790)
  • Cancer Biology (9209)
  • Cell Biology (13323)
  • Clinical Trials (138)
  • Developmental Biology (7438)
  • Ecology (11410)
  • Epidemiology (2066)
  • Evolutionary Biology (15151)
  • Genetics (10436)
  • Genomics (14044)
  • Immunology (9171)
  • Microbiology (22155)
  • Molecular Biology (8812)
  • Neuroscience (47570)
  • Paleontology (350)
  • Pathology (1428)
  • Pharmacology and Toxicology (2491)
  • Physiology (3730)
  • Plant Biology (8080)
  • Scientific Communication and Education (1437)
  • Synthetic Biology (2221)
  • Systems Biology (6037)
  • Zoology (1253)