Supervised learning

Supervised learning is a widely used paradigm in animal behavior analysis. These methods have enabled researchers to automate the process of identifying and categorizing behaviors from vast amounts of data. This approach relies on large, human-annotated datasets to train models. During training, the model is exposed to many examples (like a student studying many cases) and learns to perform a specific task or group of tasks, such as identifying different behaviors from video recordings. The model learns the task by adjusting its internal parameters to improve its performance.

Once trained, these models can be applied to new, unseen data, similar to how a veterinarian applies their knowledge to diagnose new patients. However, the model’s performance on new data depends on its ability to generalize: to apply what it has learned to different situations. For example, it would be difficult for an AI model trained only on videos of mice to accurately analyze the behavior of birds, or even for models trained only on videos of mice from one laboratory to generalize to mice from a different laboratory. In this case, the AI model may require fine-tuning to adjust to data that differ from its original training set. This fine-tuning involves further refining the model’s internal parameters to better match the characteristics of the new data.

The ability to generalize effectively often depends on the size and diversity of the original training dataset. A model trained on a wide range of behaviors, species, and experimental setups is more likely to generalize better to new and varied data. Therefore, before using an AI model in your workflow, it’s crucial to understand the type of data on which it was trained and evaluate it on your own data. This will help you assess whether the model is suitable for your specific needs or requires further adaptation.

One example of supervised learning in animal behavior analysis is the Mouse Action Recognition System (MARS),³ a collaborative effort between behavioral neuroscientists and computer scientists. The MARS tackles the challenge of automating behavior analysis by first requiring human experts to annotate video data. This process involves labeling anatomical landmarks (also called keypoints) on the animal’s body, such as the nose, tail, and paws. A model is then trained with the use of human annotations to detect these keypoints on new images; this task is called pose estimation, in which pose refers to the overall configuration of the body defined by keypoints. The next step requires users to assign behavioral labels to different video segments, like sniffing, grooming, or mounting. The model then learns to recognize these behaviors by matching the visual information (keypoints and movement patterns) to the human-provided labels. This workflow of first performing pose estimation, then behavior classification is common across supervised learning setups and has also been reviewed in multiple works.^4,11,17 Notably, tools for pose estimation include DeepLabCut¹⁸ and SLEAP,¹⁹ which has been applied to label keypoints on species including flies and gerbils, as well as to track multiple animals in videos. Other tools for behavior classification include SimBA²⁰ and DeepEthogram,²¹ in which the goal is to map each input video frame to a human-defined behavior label.

While supervised learning has significantly advanced the field of animal behavior analysis, several key challenges remain. (1) Annotation bottleneck: these models require a labeled training set to learn new tasks. Creating large, accurately annotated datasets is time-consuming and expensive, often requiring significant manual effort from experts, which limits the scalability of supervised learning approaches. For example, MARS is developed with approximately 14 hours of frame-level behavior annotations, and 15,000 frames of pose annotations across training and evaluation data. Researchers are exploring various techniques to address this challenge, including active learning, transfer learning, and semi-supervised learning. (2) Generalization to new data: supervised learning models trained on a single dataset may not generalize well to new datasets with different experimental setups, animal strains, or behavioral definitions. This is because the model might have learned features that are specific to the training dataset that don’t transfer well to new situations. For example, the MARS pose estimators were trained with images from a top-view camera, and retraining with new data was needed for camera angles from the side. Additionally, the model was initially trained with black and white mice and might not generalize well to other colors or species. This highlights a common challenge in supervised learning: models often need to be adapted to new settings or types of data. Techniques to tackle this include transfer learning via fine-tuning, as well as data augmentation, which artificially increases the diversity of training data. (3) Interpretability: many powerful models are complex ‘black boxes’ due to their architectures and large numbers of parameters.^17,22 This can make it difficult for researchers to trust the model’s output and gain insights into the underlying factors driving animal behavior. Researchers are actively developing techniques to make AI models more interpretable, such as highlighting important features that influence decisions or identifying representative examples that activate specific model components.

Self-supervised learning

While supervised learning has been impactful,^4,11,17 it often faces limitations due to the reliance on labeled data. Self-supervised learning²³ offers an alternative approach by leveraging the inherent structure of the data itself to generate supervisory signals, eliminating the need for explicit labels. In self-supervised learning,²⁴ pretext tasks are designed to encourage the model to learn relevant representations from unlabeled data without relying on explicit labels. These pretext tasks include predicting missing parts of the data or reconstructing the original data from a scrambled version. These tasks enable the model to learn important features and relationships within the data without labels (eg, training the model to predict future trajectories⁵), which can then be used for other downstream tasks (eg, using the learned representation as input for behavior classification). The performance of a self-supervised model is often judged by how well it performs on a separate task, like accurately classifying different behaviors, after it has learned from the unlabeled data.

One example of a self-supervised framework for studying behavior is task programming,⁵ which uses pretext tasks designed by experts to learn representations of behavior in a self-supervised way, in addition to predicting future trajectories. These tasks represent attributes important for studying behavior, such as the distance between animals or speed of each animal. This encourages the model to learn about spatial and temporal relationships in trajectory data, as well as important features of animal behavior defined by experts. Once trained, these representations can be applied to learn human-defined behaviors with significant reduction in annotation requirements. Other frameworks for self-supervised learning of behavior are reviewed in Luxem et al⁴ and notably includes VAME,²⁵ which uses a variational autoencoder to map trajectory data to a lower dimensional space and perform clustering for behavior discovery. Instead of behavior, another line of work^15,26 uses self-supervised learning to discover keypoints from behavioral videos. B-KinD trains a model to reconstruct the movements in the original video (eg, difference across 2 video frames) in order to enable the model to learn meaningful representations of the animal’s body and movement, even without explicit keypoint annotations. Finally, there are also frameworks that use unsupervised learning,²⁷ which learns structure from data without a pretext task.

One of the key advantages of self-supervised learning is its ability to leverage large amounts of unlabeled data,²⁴ which is more readily available compared to expert-labeled data. This makes these techniques especially valuable in veterinary medicine, in which labeled data for diverse animal species, complex patient records, and varied imaging modalities (eg, radiographs, ultrasounds, histopathology) can be scarce or costly to obtain. By leveraging unlabeled datasets, training with self-supervised learning compared to only supervised data can lead to more robust and generalizable models. However, as with supervised learning, if a self-supervised model is only trained on a limited set of unlabeled data (e.g., videos all from a single strain of mice), it is also unlikely to generalize to new settings. This can be seen with B-KinD, which has been successfully used to discover keypoints on animals like mice and fruit flies. While a model trained on black and white mice might not directly transfer to videos with two black mice, it demonstrates efficient performance when applied to videos with similar characteristics, requiring only a few minutes of training video for such cases. Furthermore, designing effective pretext tasks is crucial in self-supervised learning.²⁴ The task should be challenging enough to encourage the model to learn meaningful representations but not so difficult that it becomes impossible to solve. Collaborations between AI researchers and veterinarians could ensure that the learned representations are relevant to clinically relevant downstream tasks.

Foundation models

Building on advances in self-supervised learning, foundation models represent a recent paradigm shift in AI.²⁸ These models are trained on massive, diverse datasets, often encompassing internet-scale data. During training, foundation models use self-supervised objectives that enable them to learn representations without explicit labels across massive datasets. This allows them to be adapted to a wide range of downstream tasks with minimal or no fine-tuning. A prominent example²⁹ is large language models like GPT-4, which has demonstrated a wide range of capabilities in natural language understanding and generation after being trained on vast amounts of text data. This represents a departure from self-supervised methods discussed in the previous section in terms of training data scale, general-purpose nature of the learned model, and emergent capabilities.

In the domain of video analysis, foundation models are also emerging as a promising tool.^30–32 One example, which has been applied to animal behavior analysis, is VideoPrism,³³ a video foundation model trained on a massive dataset of approximately 600 million videos. VideoPrism is a video encoder that maps input videos to semantically meaningful representations, which then empower a range of downstream tasks in computer vision, including captioning, video question answering, and video-text retrieval. In addition, this model, with minimal adaptation, is applicable to a variety of animal behavior analysis tasks,³⁴ including localization, behavior classification, and retrieval of videos with similar behaviors.

Training and developing foundation models is a computationally intensive process that requires significant resources and expertise (eg, estimation for a single training run of a 1.5 billion–parameter model is at least $80,000, depending on factors such as hardware, dataset size, and training time).³⁵ For example, these models often contain a large number of parameters (eg, a 70 billion–parameter open-sourced language model³⁶) to fit the massive training datasets. However, once trained, these models can be readily used by researchers and practitioners without the need for extensive model training or fine-tuning.²⁸ This provides researchers in animal behavior with access to powerful AI systems that could potentially work out of the box on their data. These general-purpose capabilities also hold promise for applications in veterinary medicine, such as analyzing patient videos, predicting disease risk, and assisting with clinical decision-making.

Furthermore, foundation models provide a strong starting point for specialization. Domain-specific models can be derived from foundation models through fine-tuning on smaller, targeted datasets.²⁸ In the future, a foundation model trained on diverse animal videos could be further fine-tuned on a dataset of videos specifically focused on canine behavior to improve its performance on canine-specific tasks. Alternatively, a decoder module, which translates the general representations learned by the foundation model into task-specific outputs, can be added to adapt it to specific tasks or datasets. VideoPrism, originally trained on internet videos, learns a general-purpose video representation that is compatible with various decoders (eg, classification or localization) for animal behavior analysis.³⁴ In a similar fashion, it is also possible to train decoders to identify specific diseases or conditions from general-purpose video foundation model representations.

Data

For modern AI models to generalize to unseen settings (eg, new clinics, new patients, new tasks), they require large, diverse training datasets. There is an opportunity to pool datasets containing heterogeneous data modalities, such as electronic health records, medical images (x-rays, ultrasounds, etc), pathology reports, genomic sequences, and even sensor data from wearable devices, across various sources, such as veterinary hospitals, research institutions, and diagnostic laboratories. Assuming a sufficiently large and representative dataset based on current trends in foundation model research,^28,37 this aggregated dataset has the potential to enable the development of more robust models that work out of the box without fine-tuning to tackle tasks such as more accurate diagnoses, personalized treatments, and improved disease prediction.

However, data sharing and aggregation raise important questions about data ownership, privacy, security, and usage rights. Clear agreements and ethical guidelines are needed to ensure responsible data handling and to address potential biases in the data, such as underrepresentation of certain breeds or species, overrepresentation of specific clinical settings, or variations in data collection practices. Such biases could lead to inaccurate predictions or limited generalizability to diverse populations. Furthermore, the ownership, control, and potential liability of the trained AI models, especially when developed with the use of collaborative efforts or sensitive data, raise complex questions about intellectual property rights, data usage agreements, and potential conflicts of interest. Artificial intelligence developers also need to ensure the privacy of clinical data, as training data can be extracted from trained generative models such as large language models.³⁸ Transparent agreements and ethical guidelines are needed to ensure fair and responsible use of AI models and data.⁷

Finally, because the performance of AI systems is highly dependent on their training data, practitioners and users should be aware of the data on which the model they are using is trained (or have a way to evaluate the model carefully in their setting) before deploying it in clinical practice.

Model

The diverse range of data modalities and tasks in veterinary medicine—from radiographs and bloodwork to clinical notes and genomic data—presents a unique opportunity to push the capabilities of modern AI systems. However, this also presents a challenge: effectively integrating these diverse data types into a cohesive, interpretable system. For example, imagine a computational system that can analyze a radiograph; cross-reference it with the patient’s breed, age, and clinical history; and then suggest a diagnosis while also providing the reasoning behind its conclusion, drawing on established veterinary knowledge and best practices. To achieve this, we need models that not only leverage the power of deep learning on large datasets but also incorporate decades of accumulated veterinary expertise. This human-in-the-loop approach, in which AI learns from and collaborates with veterinarians, is crucial not only for improving animal care but also for building trust in these systems.

One step toward this approach is to develop models that can explain their reasoning in a way that is meaningful to veterinarians. Veterinary medicine encompasses a wide range of specialties, each with its own unique knowledge base and clinical focus. For instance, an explanation of patient conditions and diagnosis might vary for a radiologist, oncologist, or cardiologist. To achieve this, close collaboration between AI researchers and veterinary professionals is crucial. By working together, we can ensure that these models are developed and deployed in a way that addresses the specific needs, concerns, and ethical considerations of the veterinary community. This collaboration can help bridge the gap between computational expertise and domain-specific knowledge, leading to more effective and trustworthy AI solutions for veterinary medicine.

Finally, there is an opportunity to enhance the human-animal bond by providing tools to continuously monitor and analyze animal behavior, communication, and emotional states from videos, wearables, or other devices. This can lead to improved response to the needs of animals (eg, improving livestock health^39,40) as well as a strengthened connection between humans and their animal companions. For example, AI-powered apps could provide personalized recommendations for pet care and training, tailored to the specific needs and personality of the animal. However, it’s important to consider how the technology is used in a way that complements, not replaces, human-animal interactions.

Evaluation

Evaluating the impact of AI on animal well-being is crucial but complex, requiring a multifaceted approach that goes beyond traditional metrics like accuracy.⁴¹ While AI offers promising applications in continuous monitoring, early disease detection, and personalized interventions, assessing its true impact on animal welfare can be challenging. To tackle this challenge, we need to develop robust evaluation metrics that capture the multifaceted aspects of animal well-being, including the following: (1) physical health indicators such as recovery time, longevity, and disease rate and (2) behavioral and emotional state indicators such as social interactions, vocalizations, and signs of stress. In addition, we also need metrics to assess how AI impacts the veterinary practice itself, such as client and veterinarian satisfaction, diagnostic accuracy, and efficiency.

Furthermore, establishing standardized protocols for data collection and evaluation facilitates comparability across different studies, enabling us to better measure the progress of model development in the field. This is particularly crucial as the field moves toward more collaborative and open research. Notable examples of standardization effort from other fields include data sheets for documenting data collection and usage from machine learning⁴²; the Neurodata Without Borders format, introduced for organizing and sharing neurophysiology data⁴³; and guidelines for using and developing open-source tools for behavioral video analysis (such as the use of large-scale standardized benchmarks).⁴ In human medicine, large-scale datasets with standardized formats have enabled the development of powerful models for language (eg, Med-PaLM⁴⁴) and medical image segmentation (eg, Med-SAM⁴⁵).

Ethical considerations must be at the forefront of any evaluation process. Collaborations between AI developers and veterinarians are needed to ensure that these models prioritize the best interests of stakeholders and avoid any unintended consequences. Ideally, new systems or modifications to existing ones should undergo comprehensive evaluations on existing datasets or simulated environments before deployment. This allows for controlled testing and refinement, minimizing risks to real animals. By combining robust evaluation metrics, ethical considerations, and simulated testing, we can pave the way for the responsible and effective use of emerging technologies to improve animal well-being.