TY - GEN
T1 - Controllable Visual-Tactile Synthesis
AU - Gao, Ruihan
AU - Yuan, Wenzhen
AU - Zhu, Jun-Yan
PY - 2023/10
Y1 - 2023/10
N2 - Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. In this work, we leverage deep generative models to create a multi-sensory experience where users can touch and see the synthesized object when sliding their fingers on a haptic surface. The main challenges lie in the significant scale discrepancy between vision and touch sensing and the lack of explicit mapping from touch sensing data to a haptic rendering device. To bridge this gap, we collect high-resolution tactile data with a GelSight sensor and create a new visuotactile clothing dataset. We then develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We evaluate our method regarding image quality and tactile rendering accuracy. Finally, we introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device for an immersive experience, allowing for challenging materials and editable sketch inputs.
AB - Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. In this work, we leverage deep generative models to create a multi-sensory experience where users can touch and see the synthesized object when sliding their fingers on a haptic surface. The main challenges lie in the significant scale discrepancy between vision and touch sensing and the lack of explicit mapping from touch sensing data to a haptic rendering device. To bridge this gap, we collect high-resolution tactile data with a GelSight sensor and create a new visuotactile clothing dataset. We then develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We evaluate our method regarding image quality and tactile rendering accuracy. Finally, we introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device for an immersive experience, allowing for challenging materials and editable sketch inputs.
UR - http://www.scopus.com/inward/record.url?scp=85185487672&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85185487672&partnerID=8YFLogxK
U2 - 10.1109/ICCV51070.2023.00648
DO - 10.1109/ICCV51070.2023.00648
M3 - Conference contribution
T3 - Proceedings of the IEEE International Conference on Computer Vision
SP - 7017
EP - 7029
BT - Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
ER -