TY - GEN
T1 - 4D Human Body Capture from Egocentric Video via 3D Scene Grounding
AU - Liu, Miao
AU - Yang, Dexin
AU - Zhang, Yan
AU - Cui, Zhaopeng
AU - Rehg, James M.
AU - Tang, Siyu
N1 - Funding Information:
Weintroduceanoveltaskofestimatingatimeseriesof 3D human body models for the second-person in an ego-centricvideo,whicharetemporally-coherentandgrounded onthe3Dscene. Toaddressthechallengesofegocentric video, we propose aneffective optimization-based method that exploits the 2D observations of the entire video se-quenceandhuman-scenecontactforhumanmotioncapture. We conduct detailed experiments on our EgoMoCap dataset to demonstrate the benefits of our approach. We believe our workpointstoexcitingresearchdirectionsinegocentricso-cialinteractionanalysisand4Dhumanbodyreconstruction. Acknowledgments. Portionsofthisresearchweresup-ported in part by National Science Foundation Award 2033413 and a gift from Facebook.
Publisher Copyright:
© 2021 IEEE.
PY - 2021
Y1 - 2021
N2 - We introduce a novel task of reconstructing a time series of second-person1 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges,we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses,shapes,and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover,we compare our method with the previous state-of-the-art method on human motion capture from monocular video,and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition,we demonstrate that our approach produces more realistic human-scene interaction.
AB - We introduce a novel task of reconstructing a time series of second-person1 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges,we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses,shapes,and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover,we compare our method with the previous state-of-the-art method on human motion capture from monocular video,and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition,we demonstrate that our approach produces more realistic human-scene interaction.
KW - 3D Human Reconstruction
KW - Egocnetric Vision
KW - Human Scene Interaction
UR - http://www.scopus.com/inward/record.url?scp=85121011309&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85121011309&partnerID=8YFLogxK
U2 - 10.1109/3DV53792.2021.00101
DO - 10.1109/3DV53792.2021.00101
M3 - Conference contribution
AN - SCOPUS:85121011309
T3 - Proceedings - 2021 International Conference on 3D Vision, 3DV 2021
SP - 930
EP - 939
BT - Proceedings - 2021 International Conference on 3D Vision, 3DV 2021
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 9th International Conference on 3D Vision, 3DV 2021
Y2 - 1 December 2021 through 3 December 2021
ER -