Video based phenotyping platform for the laboratory mouse

Abstract

Automated detection of complex animal behavior remains a challenge in behavioral neurogenetics. Developments in computer-vision have greatly advanced automated behavior detection and allow high-throughput pre-clinical studies. An integrated hardware and software solution is necessary to facilitate the adoption of these advances in the field of behavioral neurogenetics, particularly for non-computational labs. We have published a series of papers using an open field arena to annotate complex behaviors such as grooming, posture, and gait as well as higher level constructs such as frailty. Here, we describe in detail our integrated hardware and software solution for robust, high quality video monitoring of mice. We provide safety data for mice in long-term monitoring by recording environmental conditions in the arenas as well as health markers. The details provided here offer one potential solution for standardized high quality video data for computer vision tasks, and enables replication of our work.

1 Introduction

Behavioral neurogenetics seeks to link complex and dynamic behaviors with their underlying neural circuits and genetics [1]. The laboratory mouse has been at the forefront of these discoveries. Animal behavior quantification has rapidly advanced in the past few years with the application of machine learning to the problem of behavior annotation. These advances are mainly due to breakthroughs in the statistical learning field [2–7]. Our lab has previously used computer-vision methods to track visually diverse mice under different environmental conditions [8], infer pose for gait analysis and posture analysis [9], and detect complex behaviors like grooming [10]. We have also used computervision derived features to predict complex constructs, such as health and frailty [11].

Although major advances have been made in behavior annotation using computer vision, a major challenge exists in the democratization of this technology. Currently, a high level of expertise is needed for efficient use of these methods. For instance, large amounts of training data are needed to train a pose estimation or grooming detection network. Many labs collect data in disparate ways which lack standardized visual appearance. In order to apply existing models for detecting pose or behavior, each individual lab must train their own model at high cost of labeled data. This is a challenge to the field - many useful models exist, but they cannot be directly applied across data originating from different labs. If visual input is standardized across labs, trained models can be adopted across labs, decreasing barriers to entry and increasing reproducibility of data.

With this in mind, we describe our data acquisition system here. We have developed an integrated mouse phenotyping platform called JAX Animal Behavior System (JABS), which consists of video collection hardware and software, a behavior labeling and active learning app, and an online database for sharing classifiers. The JABS database integrates open-field data from diverse mouse strains and allows users to calculate heritability and conduct GWAS from their deposited behavior classifier. Here, we specifically detail the JABS data acquisition module, describe and release all design specifications and software. Additionally, we provide results on the health and safety of the mice in our long-term monitoring arena, which can be used for Institutional Animal Care and Use Committee applications. Our hardware and software solution collects high quality data for behavior analysis and has been used for several of our papers [8–11]. Adoption of the JABS data acquisition module will allow labs to use the trained machine learning models described in these manuscripts. In addition, we hope this standard system for the laboratory mouse will democratize new behavioral methods.

2 Methods

2.1 Data acquisition - Hardware and Software

We use a standardized hardware setup for high quality data collection and optimized storage (Figure 1). The end result is uniform video data across day and night. Complete details of the software and hardware, including 3D designs used for data collection, are available on our Github (https://github.com/KumarLabJax/JABS-data-pipeline/tree/main). We also provide a step-by-step assembly guide (https://github.com/KumarLabJax/JABS-data-pipeline/blob/main/Multi-day%20setup%20PowerPoint%20V3.pptx).

Figure 1:

JABS Data Acquisition Module. Image of the monitoring hardware, architecture of the real-time monitoring app, and screenshots from videos taken during daytime and nighttime.

We have organized the animal habitat design into three groups of specifications. The first group of specifications are requirements necessary for enabling compatibility with our machine learning algorithms. The second group describes components that can be modified as long as they produce data that adheres to the first group. The third group describes components that do not affect compatibility with our machine learning algorithms. While we distinguish between abstract requirements in group 1 and specific hardware in group 2 that meet those requirements, we recommend that users of our algorithms use our specific hardware in group 2 to ensure compatibility.

The design elements that are critical to match specifications in order to re-use machine learning algorithms include (1a) the camera viewpoint, (1b) minimum camera resolution and frame rate, (1c) field of view and imaging distance, (1d) image quality and (1e) the general appearance of the habitat (cage or enclosure). The design elements that are flexible but impact the compatibility are (2a) camera model, (2b) compute interface for capturing frames, (2c) lighting conditions, (2d) strains and ages of mice and (2e) animal bedding contents and quantity. Design elements that have no impact on compatibility are (3a) height of habitat walls to prevent mice from escaping, (3b) animal husbandry concerns, (3c) mounting hardware, (3d) technician ergonomic considerations and (3e) electrical connection hardware and management.

2.1.1 Group 1 specifications

Our system operates on a top-down camera viewpoint. This specification enables flexibility and allows more diverse downstream hardware and ease of construction. While other approaches such as imaging from the bottom through a clear floor are possible, they are achieved at the cost of limiting assay duration and construction complexity. The top-down viewpoint enables wider adoption due to construction simplicity and the ability to test more varied assays.

Our algorithms are trained using data originating from 800×800 pixel resolution image data and 30 frames per second temporal resolution. This resolution was selected to strike a balance between resolution of the data and size of data produced. While imaging at higher spatial and temporal resolution is possible and sometimes necessary for certain behaviors, these values were selected for general mouse behavior. We train and test our developed algorithms against the spatial resolution. We note that these are minimum requirements, and down-sampling higher resolution and frame rate data still allows our algorithms to be applied.

Similar to the pixel resolution, we also specify the field of view and imaging distance for the acquired images in real-world coordinates. These are necessary to achieve similar camera perspectives on imaged mice. Cameras must be mounted at a working distance of approximately 100cm above the floor of the arena. Additionally, the field of view of the arena should allow for between 5 – 15% of the pixels to view the walls (FoV between 55cm and 60cm). Having the camera a far distance away from the arena floor reduces the effect of both perspective distortion and barrel distortion. We selected values such that our custom camera calibrations are not necessary, as any error introduced by these distortions are typically less than 1%.

Additionally, image quality is important for meeting valid criteria for enabling the use of machine learning algorithms. Carefully adjusting a variety of parameters of hardware and software values in order to achieve similar sharpness and overall quality of the image is important. While we cannot provide an exact number or metric to meet this quality, users of our algorithms should strive for equal or better quality that exists within our training data. One of the most overlooked aspect of image quality in behavioral recordings is image compression. We recommend against using typical software-default video compression algorithms and instead recommend using either defaults outlined in the software we use or recording uncompressed video data. Using software-defaults will introduce compression artifacts into the video and will affect algorithm performance.

Finally, the general appearance of the cage should be visually similar to the variety of training data used in training the machine learning algorithms. Please read associated documentation on this for each individual algorithm for assessing the limitations. While our group strives for the most general visual diversities in mice behavioral assays, we still need to acknowledge that any machine learning algorithms should always be validated on new datasets that they are applied to. Generally our machine learning algorithms earlier in the entire processing pipeline, such as pose estimation, are trained on more diverse datasets than algorithms later in the pipeline, such as nocifensive and frailty predictions.

2.1.2 Group 2 specifications

In order to achieve compliant imaging data for use with our machine learning algorithms, we specify the hardware we use. While the hardware and software mentioned in this section is modifiable, we recommend that careful consideration is taken such that changes still produce complaint video data.

We modified a standard open field arena that has been used for high-throughput behavioral screens [12]. The animal environment floor is 52 cm square with 92 cm high walls to prevent animals escaping and to limit environmental effects. The floor was cut from a 6mm sheet of Celtec^® (Scranton, PA) Expanded PVC Sheet, Celtec® 700, White, Satin / Smooth, Digital Print Gradesquare and the walls from 6mm thick Celtec® Expanded PVC Sheet, Celtec® 700, Gray, (6 mm x 48 in x 96 in), Satin / Smooth, Digital Print Grade. All non-moving seams were bonded with adhesive from the same manufacturer. We used a Basler (Highland, IL) acA1300-75gm camera with a Tamron (Commack, NY) 12VM412ASIR 1/2” 4-12mm F/1.2 Infrared Manual C-Mount Lens. Additionally, to control for lighting conditions, we mounted a Hoya (Wattana, Bangkok) IR-80 (800nm), 50.8mm Sq., 2.5mm Thick, Colored Glass Longpass Filter in front of the lens using a 3D printed mount. Our cameras are mounted 105 +/− 5 cm above the habitat floor and powered the camera using the power over ethernet (PoE) option with a TRENDnet (Torrance, CA) Gigabit Power Over Ethernet Plus Injector. For IR lighting, we used 6 10 inch segments of LED infrared light strips (LightingWill DC12V SMD5050 300LEDs IR InfraRed 940nm Tri-chip White PCB Flexible LED Strips 60LEDs 14.4W Per Meter) mounted on 16-inch plastic around the camera. We used 940nm LED after testing 850nm LED which produced a marked red hue. The light sections were coupled with the manufactured connectors and powered from an 120vac:12vdc power adapter.

For image capture, we connected the camera to an nVidia (Santa Clara, CA) Jetson AGX Xavier development kit embedded computer. To store the images, we connected a local four-terabyte (4TB) USB connected hard drive (Toshiba (Tokyo, Japan) Canvio Basics 4TB Portable External Hard Drive USB 3.0) to the embedded device. When writing compressed videos to disk, our software both applies optimized de-noising filters as well as selecting low compression settings for the codec. While most other systems rely on the codec for compression, we rely on applying more specific de-noising to remove unwanted information instead of risking visual artifacts in important areas of the image. We utilize the free ffmpeg library for handling this filtering and compression steps with the specific settings available in our shared C++ recording software. Complete parts list and assembly steps are described in (https://github.com/KumarLabJax/JABS-data-pipeline)

2.1.3 Group 3 specifications

Finally, here we present hardware and software that can be modified without risk of affecting video compliance.

For natural light, we used a F&V (Netherlands) fully dimmable R-300SE Daylight LED ring light powered by a 120vac:12vdc power adapter. These lights are adjustable to meet the visible lighting needs of specific assays without affecting the visual appearance of the data. To keep the animals nourished, we installed water bottles and a food hopper external to the animal environment. These were placed on the outside of the arena on a removable panel. The panel can be customized as needed for experiments without the need to replace/modify the entire arena. To suspend the camera and lights, we used a wire shelf from our solution for technician ergonomics.

To raise the animal cage to an ergonomic height, we used the 24-inch by 24-inch option of the Metro (Wilkes-Barre, PA) Super Erecta wire shelving system with three shelves. As mentioned in the earlier paragraph, the topmost shelf was used to suspend the camera and lights. We also hinged one wall, turning it into a door, to allow easier animal access. Communication between the electronic devices was interconnected with CAT5 cables and a network switch and a powered USB hub was used between the USB connected hard drive and the nVidia compute device. We used a digital timer for the visible LED light, a 120v power strip to consolidate the power, and a universal power source (battery backup) between the chamber and facility power.

For ease of use and reduction of environmental noise, we also include a software for remote monitoring and welfare check. The software consists of three main components: a recording client implemented in C++, a control server implemented with the Flask Python framework, and a web-based user interface implemented with Angular (Figure 1). The recording client runs locally on each Nvidia Jetson Xavier computer and communicates with the server using the Microsoft C++ REST SDK to provide centralized monitoring and control of distributed recording devices. The recording client captures raw frames from the camera and encodes video using the ffmpeg library. In addition to saving encoded video on the local hard drive, the recording client can optionally send video over the RTMP protocol to a NGINX server configured with the nginx-rtmp plug-in. The web interface communicates with the control server, which relays recording start and stop commands to individual recording devices, enabling the user to remotely control various aspects of recording in addition to viewing the live stream from the NGINX streaming server using the HTTP Live Streaming (HLS) protocol (Figure 2).

Figure 2:

JABS web-based control system for recording and monitoring experiments. Screenshots from Angular web client that allows monitoring of multiple JABS Acquisition units in multiple physical locations. Dashboard view allows monitoring of all JABS units and their status, Device Status provided detailed data on individual devices, recording session dashboard allows initiation of new experiments, and remote welfare view allows live video to be observed from each unit.

3 Results and Discussion

We have previously used versions of the JABS data acquisition module to collect short (1-hr) and long (4 - 30 days) of continuous video data [8–11]. The modifiable front panel allows customization, such as use of single or multiple water bottles to conduct 2-bottle choice experiments with ethanol or methamphetamine. We have observed a wide range of data from JABS acquisition, varying from 10-20 GB/hr depending on visual content. For instance, large number of mice with high motion lead to larger files. Large data management is an important aspect of JABS.

3.1 Environment checks

Since the JABS acquisition arena was not previously validated for long-term housing, we carried out a series of experiments to confirm health and welfare of animals over time in these apparatus. These data can be used for Institutional ACUC protocols. We compare our data with established guidelines from the Guide for the Care and Use of Laboratory Animals (the Guide) [13]. Our experiments were performed in one room at The Jackson Laboratory, Bar Harbor, Maine (JAX) with temperature and humidity set to 70-74°F (^~21-23°C) and 40-50%, respectively.

One concern related to use of the JABS arena in long-term experiments was that the 90 cm height of the walls without lower air openings might result in inadequate air flow and buildup of toxic gases. To address this, we compared environmental parameters in JABS arenas with that of a standard JAX housing cage. Two JABS arenas were observed with 12 male C57BL/6J mice 12-16 weeks old in each for a 14-day period. At the same time, one wean cage containing 10 male C57BL/6J age-matched mice was observed on a conventional rack for matching air flow in the same room. We used a #2 Wean Cage (30.80 x 30.80 x 14.29 cm) from Thoren (Hazleton, Pennsylvania) with 727.8 cm² floor space, which is a common housing container for mice and is approved at JAX to house 10 animals. This commercial cage has a floor area that is ^~1/4 that of the JABS arena. The ceiling height in the wean cage ranges 5-14 cm due to the sloped metal wire cover that contains food and water. The JABS arena, by contrast, has no ceiling.

Food, water, bedding type and depth and light level were all matched in the arenas and wean cage. Bedding (1:1 ratio of aspen chip/shavings mix (chip from P.J. Murphy Forest Products, Montville, New Jersey and shavings from Northeast Forest Products Corp., Warrenburg, New York) and Alpha-Dri (Shepherd Specialty Papers)) was left unchanged for the full two week period in all pens, as we wish to minimize interaction with long term monitoring mice in JABS arenas as much as possible. To determine if forced air flow was needed for an acceptable arena environment, one of two arenas and the wean cage were exposed to normal room air flow, while the second arena had a 6” quiet electric fan mounted above. The fan was pointed so as to blow air up, potentially drawing air out of the arena while not actively blowing air over the mice. The choice of the fan blowing upwards was based on a previous experiment with the fan blowing down into the arena starting day 7. In that experiment, we observed that mice in the arena with a fan lost more weight than mice in the arena with no fan, suggesting the fan blowing down introduced a confounding negative environmental factor.

We checked for CO₂ and ammonia, common gases that can build up in housing [13]. CO₂ was measured using an Amprobe (Everett, Washington) CO₂-100 meter with levels checked daily except weekends and holidays throughout the experiment in both arenas and the wean cage. CO₂ measurements were taken in the middle of the room before and after each arena and wean cage measurement as a control. For higher levels, CO₂ is shown as a range, as it was observed to oscillate between high and low values over reading periods lasting several minutes. Ammonia was tested using Sensidyne (St. Petersburg, Florida) Ammonia Gas Detector Tubes (5-260 ppm) in the arena without a fan and the wean cage on days 0, 2, 4, 7 and 14, with air samples taken near floor level and areas of greatest waste accumulation in both. MadgeTech (Warner, New Hampshire) RHTEMP1000IS temperature and humidity data loggers were placed on the floor in each arena and the wean cage and left for the duration of the experiment. An environment monitor (Hobo, U12-012, temperature/humidity/light monitor, Onset, Bourne, Massachusetts) was mounted on the wall of the room to provide background levels. Body weight measurements were taken daily except weekends and holidays. Grain and water were weighed at the start and end of each experiment to check consumption.

We observed daily room background CO₂ levels of 454 to 502 ppm throughout the 14-day experiment. These are very close to expected outdoors levels and indicative of a high air exchange rate [14]. JABS arena CO₂ levels varied from a low of 511 ppm on day 1 to an oscillating range of 630 to 1565 ppm on day 14. The JAX standard wean cage experienced an oscillating range of 2320 to 2830 on day 0 climbing to an oscillating range of 3650 to 4370 ppm on day 14. The wean cage CO₂ values approximately match those from another published study of maximum grouped mice in similar static housing [15]. Indoor CO₂ is often evaluated as level above background [14]. We observe a maximum JABS arena CO₂ level above background of 1082 ppm. This is ^~4 fold lower than the maximum observed maximum CO₂ level above background observed in the wean cage (4121 ppm) (Figure 3A, arena with fan excluded from graph for clarity).

Figure 3:

(A) Carbon dioxide concentrations and (B) ammonia concentrations were both much higher in the standard wean cage than in the JABS arena. Carbon dioxide was also compared to room background levels. (C) Temperature and (D) humidity measured at floor level in JABS arenas and a standard wean cage compared to room background across a 14 day period. (E) Average body weight as percent of start weight in each JABS arena and wean cage across the 14 day period. (F) Food and (G) water consumption shown as grams per mouse per day for one JABS arena and one wean cage for a 14 day period.

Ammonia levels in the JABS arena were below the detection threshold of 5 ppm on days 0, 2, 4 and 7, and rose to 18 ppm on day 14. Ammonia levels in the wean cage were <5 ppm on days 0 and 2, rose to 52 ppm on day 4 and reached levels of ^~230ppm on both days 7 and 14. An early concern was that the high walls of the JABS arena may inhibit air flow and lead to a buildup of toxic gases. CO₂ and ammonia levels both indicate that the JABS arena has greater air exchange than standard conventional housing. The National Institute for Occupational Safety and Health (NIOSH) recommended maximum ammonia exposure for humans is 25 ppm over an 8 hour period. There is no mouse standard, but a similar recommended maximum continuous exposure of 25 ppm for mice is sometimes used [13, 16]. Ammonia levels are influenced by several variables, with the most prominent being air changes per hour (ACH) [17, 18]. JAX animal rooms have ^~10 ACH and PIV cages in those rooms have ^~55-65 ACH. We find that ammonia levels remained 10-50 fold lower in the JABS arena than our control standard static wean cage and well within the strictest recommended range (Figure 3B). One consideration in use of JABS arenas for observations will be the impact of ammonia levels on behavior [19]. Currently all mice moved into JABS arena experiments in our facility are coming from PIV housing. Although we did not measure ammonia levels in PIV housing, we anticipate from Ferrecchia et al. [18] that ammonia levels will be similar in the PIV home cage the mice come from and the JABS arena they go into, and this would have minimal impact upon observed behavior.

Temperatures in all locations (room background, two JABS arenas and one wean cage) remained in a range of 22-26°C throughout the experiment. Variance in room background readings suggest temperature fluctuations are more due to innate room conditions (such as environmental controls) than anything else. We find that arena structure does not adversely affect control of the temperature to which mice are exposed (Figure 3C).

The probes which measured temperature also measured humidity. The room probe, mounted on one wall of the 8’x8’ room at ^~6’ above the floor, measured consistent background humidity of 45+/−5% throughout the experiment (Figure 3D, green line). Housing probes were placed in the bedding on the floor of each chamber - near the center in the JABS arenas and along one wall in the much more space limited wean cage. Humidity in the JABS arenas was 55-60% throughout the 14-day experiment, aside from notable sporadic spikes (Figure 3D, blue and black lines). The spikes do not correlate with background humidity changes and are interpreted as occasions when the mice urinated on or very near to the probe, which evaporated quickly. By contrast, humidity levels within the wean cage started in the same 55-60% range but quickly climbed above 75% (within 12 hours) and did not come back down as humidity in the JABS arenas had done. Wean cage humidity then continued a gradual climb, reaching 97.5% by day 14 (Figure 3D, red line). While both the JABS arenas and the standard wean cage can be anticipated to have higher humidity in their micro-environments compared to the room macro-environment due to the mice contributing moisture via urination and the housing structures somewhat inhibiting air flow (per the Guide [13]), the JABS arenas were found to maintain a much drier environment. This is likely partly due to a greater bedding to mouse ratio (the arena has 1.25 times as many mice but 4 times as much bedding material in this test, so 3.2 times as much bedding per mouse in JABS arena compared to the wean cage) but also due to superior air circulation in the arena compared to the wean cage. (Figure 3D). [conclusion here]

Weight is often used as a marker for health though body condition score is used as a more reliable indicator of serious concerns [20, 21]. Mice in arenas lost weight compared to those in the wean cage and this was initially a cause of concern. However, mice in JABS arenas maintained a healthy appearance and normal body condition score throughout the experiment. Other measurements demonstrating normal parameters and other control experiments not shown additionally led us to believe the weight differences are because JABS arena mice are active while wean cage mice, with more limited movement available, are sedentary. Mice started the experiment at 25-33 grams body weight. The lowest average recorded during the experiment was 95.6

Per mouse grain usage was comparable between the JABS arena and the wean cage and in an expected range [22] (Figure 3F). Per mouse water usage was comparable between the JABS arena and the wean cage and in the expected range [23]. Somewhat higher water use in the arena could be indicative of higher activity requiring more hydration (Figure 3G). Since only one JABS arena and one wean cage were tested, error bars are not available to aid in interpretation.

Three mice from one arena and three from a wean cage were necropsied immediately following 14 days in the JABS arena or control wean cage to determine if any environmental conditions, such as possible low air flow in arenas potentially leading to a buildup of heavy ‘unhealthy’ gases like ammonia or CO₂, were detrimental to mouse health. Nasal cavity, eyes, trachea and lungs were collected from each mouse. They were H&E stained and analyzed by a qualifed pathologist. No signs of pathology were observed in any of the tissue samples collected (Figure 4).

Figure 4:

Representative hematoxylin and eosin (H&E) stained tissue sections from mice after spending 14 days in the JABS arena or control wean cage. Tissues selected for examination (eye, lung, trachea and nasal passages) are those expected to be most affected if the mice lived in a space with inadequate air flow. All tissues appeared normal.

Based on these environmental and histological analysis, we conclude that the JABS arena is comparable and in many respects better than a standard wean cage. Lack of holes near the floor do not create a buildup of ammonia or CO₂. Mice seem to eat and drink at normal levels. We observe a slight decrease in body weight initially, which is gained in the next few days. We hypothesize that this could be due to the novel environment and increased space for movement, leading to more active mice.

Recently developed methods of automated behavior detection have great potential for use in behavioral genetics research. Community value and adoption of these methods requires a commitment to develop and support a new ecosystem of behavior analysis, especially for use by behavioral researchers without computational expertise. A standardized data collection apparatus would benefit the community by easing the sharing of techniques, data, and classifiers. Here we have shown that our apparatus provides robust high quality video data under differing experimental conditions, mice, and lengths of time.

4 Acknowledgments

We thank members of the Kumar Lab for helpful advice, Michelle Foskett of JAX PQC for the loan of Sensidyne and MadgeTech equipment, Rosalinda Doty for histological analysis. This work was funded by The Jackson Laboratory Directors Innovation Fund, National Institute of Health DA041668 (NIDA) and DA048634 (NIDA). All code and training data will be available at Kumarlab.org and Kumar Lab Github (https://github.com/KumarLabJax/JABS-data-pipeline).

Footnotes

• https://github.com/KumarLabJax/JABS-data-pipeline/tree/main