Humanoid animal behavior labeler: An interactive annotation agent to accelerate ethological applications in precision management of animals using large vision-language models

Animal behavior annotation was critical for precision livestock management but was challenged by inconsistency of behavioral labeling, limited dataset quality, and expensive annotation processes. To overcome these issues, this study introduced the Humanoid Animal Behavior Labeler (HABLer), integrated computer vision models, multimodal behavior recognition models, quantitative behavior metrics, and expert-driven corrections that formed a novel AI-assistant annotation workflow to save time and improve result consistency in animal behavior labeling task. To demonstrate the capability of HABLer, three short surveillance video samples featuring pigs and cattle were discussed in this study. The results indicated that an animal behavior expert with HABLer saved up to an average of 65.7% manual time compared to traditional labeling workflow. HABLer achieved an average Mean over Frames (MoF) of 0.8720 and Intersection over Union (IoU) of 0.9351 at the initial round of annotation across three short posture annotation experiments, costing only 11% to 22% of the manual annotation. In a 125-minute pig posture labeling task, HABLer helped save 60% of annotation time and achieved an overall average of 94% accuracy on the initial prediction of pigs’ postures. This research demonstrated HABLer's general capabilities and extensive potentials in performing complex behavioral labeling tasks with respect to the limited examples in this research and opened the HABLer for all practitioners for promoting generative-AI applications in precision livestock farming (http://www.ai4as.cn/Tool/HABLER). HABLer will serve as a public, continuously updated tool demonstrating the limits of cutting-edge computer vision and VLMs in animal behavior recognition, with expanded coverage across species and behavior types in the future. This pioneering approach represented a significant advancement of creating consistent and reliable animal behavioral dataset, supporting scalable ethological applications for enlightening livestock welfare and management in the livestock production industry.