PT  - JOURNAL ARTICLE
AU  - Matthew R Whiteway
AU  - Evan S Schaffer
AU  - Anqi Wu
AU  - E Kelly Buchanan
AU  - Omer F Onder
AU  - Neeli Mishra
AU  - Liam Paninski
TI  - Semi-supervised sequence modeling for improved behavioral segmentation
AID  - 10.1101/2021.06.16.448685
DP  - 2021 Jan 01
TA  - bioRxiv
PG  - 2021.06.16.448685
4099  - http://biorxiv.org/content/early/2021/06/17/2021.06.16.448685.short
4100  - http://biorxiv.org/content/early/2021/06/17/2021.06.16.448685.full
AB  - A popular approach to quantifying animal behavior from video data is through discrete behavioral segmentation, wherein video frames are labeled as containing one or more behavior classes such as walking or grooming. Sequence models learn to map behavioral features extracted from video frames to discrete behaviors, and both supervised and unsupervised methods are common. However, each approach has its drawbacks: supervised models require a time-consuming annotation step where humans must hand label the desired behaviors; unsupervised models may fail to accurately segment particular behaviors of interest. We introduce a semi-supervised approach that addresses these challenges by constructing a sequence model loss function with (1) a standard supervised loss that classifies a sparse set of hand labels; (2) a weakly supervised loss that classifies a set of easy-to-compute heuristic labels; and (3) a self-supervised loss that predicts the evolution of the behavioral features. With this approach, we show that a large number of unlabeled frames can improve supervised segmentation in the regime of sparse hand labels and also show that a small number of hand labeled frames can increase the precision of unsupervised segmentation.Competing Interest StatementThe authors have declared no competing interest.