TY - GEN
T1 - ShapeClipper
T2 - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
AU - Huang, Zixuan
AU - Jampani, Varun
AU - Thai, Anh
AU - Li, Yuanzhen
AU - Stojanov, Stefan
AU - Rehg, James M.
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images. Instead of relying on laborious 3D, multi-view or camera pose annotation, ShapeClipper learns shape reconstruction from a set of single-view segmented images. The key idea is to facilitate shape learning via CLIP-based shape consistency, where we encourage objects with similar CLIP encodings to share similar shapes. We also leverage off-the-shelf normals as an additional geometric constraint so the model can learn better bottom-up reasoning of detailed surface geometry. These two novel consistency constraints, when used to regularize our model, improve its ability to learn both global shape structure and local geometric details. We evaluate our method over three challenging real-world datasets, Pix3D, Pascal3D+, and Open-Images, where we achieve superior performance over state-of-the-art methods.11project website at: https://zixuanh.com/projects/shapeclipper.html
AB - We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images. Instead of relying on laborious 3D, multi-view or camera pose annotation, ShapeClipper learns shape reconstruction from a set of single-view segmented images. The key idea is to facilitate shape learning via CLIP-based shape consistency, where we encourage objects with similar CLIP encodings to share similar shapes. We also leverage off-the-shelf normals as an additional geometric constraint so the model can learn better bottom-up reasoning of detailed surface geometry. These two novel consistency constraints, when used to regularize our model, improve its ability to learn both global shape structure and local geometric details. We evaluate our method over three challenging real-world datasets, Pix3D, Pascal3D+, and Open-Images, where we achieve superior performance over state-of-the-art methods.11project website at: https://zixuanh.com/projects/shapeclipper.html
KW - 3D from single images
UR - http://www.scopus.com/inward/record.url?scp=85166207113&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85166207113&partnerID=8YFLogxK
U2 - 10.1109/CVPR52729.2023.01241
DO - 10.1109/CVPR52729.2023.01241
M3 - Conference contribution
AN - SCOPUS:85166207113
T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
SP - 12912
EP - 12922
BT - Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
PB - IEEE Computer Society
Y2 - 18 June 2023 through 22 June 2023
ER -