Robust piglet nursing behavior monitoring through multi-modal fusion of computer vision and ambient floor vibration

Nursing is a critical activity during the lactation period of swine farming. Continuous monitoring of piglet nursing behavior during the lactation period is essential to informing animal caretakers about the health status of piglets to reduce the mortality rate, maximize lactational growth, and improve animal welfare. Traditional approaches rely on manual observation and wearable devices, which are labor-intensive and can cause discomfort to the animals. Recent advancement in computer vision and ambient vibration sensing enables non-contact piglet nursing monitoring: The computer vision approach captures piglet location but has limited observation of their detailed movement due to lighting, resolution, and visual obstruction constraints; the ambient vibration sensing approach captures piglet movement patterns but has limited location information. In this study, a novel approach to integrate these two complementary sensing modalities is developed for robust piglet nursing behavior monitoring during the lactation period. This study leverages the state-of-the-art Segment Anything Model (SAM) to first convert images into sparse representations of piglet behaviors and then combine with ambient vibration to collaboratively infer piglet nursing pattern and intensity. This new approach enables piglet nursing monitoring with much lower computing and storage requirements than conventional computer vision methods, making it more practical for farm settings. Real-world experiments were conducted at a pig farm for continuous vision and vibration monitoring of 8 pens over 3 farrowing cycles. This study has a 97% accuracy in classifying 5 nursing stages, representing a significant 3× and 3.8× error reduction compared to the baseline method using only vision or vibration data, respectively. The multi-modal fusion approach leads to an efficient, robust, and accurate piglet nursing model that can immediately inform caretakers of issues that arise during this crucial time point of a piglet's life.