TY - JOUR
T1 - Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network
AU - Ravan, Aniket
AU - Feng, Ruopei
AU - Gruebele, Martin
AU - Chemla, Yann R.
N1 - This work was supported by the National Science Foundation (NSF) Physics Frontiers Center (PFC): Center for Physics of Living Cells (CPLC) PHY-1430124 (to M.G. and Y.R.C.). This work also utilizes resources supported by the National Science Foundation’s Major Research Instrumentation program, grant #1725729. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
PY - 2023/10
Y1 - 2023/10
N2 - Quantitative ethology requires an accurate estimation of an organism’s postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.
AB - Quantitative ethology requires an accurate estimation of an organism’s postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.
UR - http://www.scopus.com/inward/record.url?scp=85175348220&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85175348220&partnerID=8YFLogxK
U2 - 10.1371/journal.pcbi.1011566
DO - 10.1371/journal.pcbi.1011566
M3 - Article
C2 - 37871114
SN - 1553-734X
VL - 19
JO - PLoS computational biology
JF - PLoS computational biology
IS - 10
M1 - e1011566
ER -