RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

Zhenggang Tang; Balakumar Sundaralingam; Jonathan Tremblay; Bowen Wen; Ye Yuan; Stephen Tyree; Charles Loop; Alexander Schwing; Stan Birchfield

doi:10.1109/ICRA48891.2023.10160247

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

Zhenggang Tang, Balakumar Sundaralingam, Jonathan Tremblay, Bowen Wen, Ye Yuan, Stephen Tyree, Charles Loop, Alexander Schwing, Stan Birchfield

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab. Our results are also available at https://ngp-mpc.github.io/.

Original language	English (US)
Title of host publication	Proceedings - ICRA 2023
Subtitle of host publication	IEEE International Conference on Robotics and Automation
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1778-1785
Number of pages	8
ISBN (Electronic)	9798350323658
DOIs	https://doi.org/10.1109/ICRA48891.2023.10160247
State	Published - 2023
Event	2023 IEEE International Conference on Robotics and Automation, ICRA 2023 - London, United Kingdom Duration: May 29 2023 → Jun 2 2023

Publication series

Name	Proceedings - IEEE International Conference on Robotics and Automation
Volume	2023-May
ISSN (Print)	1050-4729

Conference

Conference	2023 IEEE International Conference on Robotics and Automation, ICRA 2023
Country/Territory	United Kingdom
City	London
Period	5/29/23 → 6/2/23

ASJC Scopus subject areas

Software
Control and Systems Engineering
Electrical and Electronic Engineering
Artificial Intelligence

Online availability

10.1109/ICRA48891.2023.10160247

Library availability

Discover UIUC Full Text

Cite this

Tang, Z., Sundaralingam, B., Tremblay, J., Wen, B., Yuan, Y., Tyree, S., Loop, C., Schwing, A., & Birchfield, S. (2023). RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control. In Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation (pp. 1778-1785). (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2023-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA48891.2023.10160247

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control. / Tang, Zhenggang; Sundaralingam, Balakumar; Tremblay, Jonathan et al.
Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1778-1785 (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2023-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Tang, Z, Sundaralingam, B, Tremblay, J, Wen, B, Yuan, Y, Tyree, S, Loop, C, Schwing, A & Birchfield, S 2023, RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control. in Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Proceedings - IEEE International Conference on Robotics and Automation, vol. 2023-May, Institute of Electrical and Electronics Engineers Inc., pp. 1778-1785, 2023 IEEE International Conference on Robotics and Automation, ICRA 2023, London, United Kingdom, 5/29/23. https://doi.org/10.1109/ICRA48891.2023.10160247

Tang Z, Sundaralingam B, Tremblay J, Wen B, Yuan Y, Tyree S et al. RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control. In Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1778-1785. (Proceedings - IEEE International Conference on Robotics and Automation). doi: 10.1109/ICRA48891.2023.10160247

Tang, Zhenggang ; Sundaralingam, Balakumar ; Tremblay, Jonathan et al. / RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control. Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 1778-1785 (Proceedings - IEEE International Conference on Robotics and Automation).

@inproceedings{d92d4e08a6da486aaa30a6a3159dfd15,

title = "RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control",

abstract = "We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab. Our results are also available at https://ngp-mpc.github.io/.",

author = "Zhenggang Tang and Balakumar Sundaralingam and Jonathan Tremblay and Bowen Wen and Ye Yuan and Stephen Tyree and Charles Loop and Alexander Schwing and Stan Birchfield",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Robotics and Automation, ICRA 2023 ; Conference date: 29-05-2023 Through 02-06-2023",

year = "2023",

doi = "10.1109/ICRA48891.2023.10160247",

language = "English (US)",

series = "Proceedings - IEEE International Conference on Robotics and Automation",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1778--1785",

booktitle = "Proceedings - ICRA 2023",

address = "United States",

}

TY - GEN

T1 - RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

AU - Tang, Zhenggang

AU - Sundaralingam, Balakumar

AU - Tremblay, Jonathan

AU - Wen, Bowen

AU - Yuan, Ye

AU - Tyree, Stephen

AU - Loop, Charles

AU - Schwing, Alexander

AU - Birchfield, Stan

PY - 2023

Y1 - 2023

N2 - We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab. Our results are also available at https://ngp-mpc.github.io/.

AB - We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab. Our results are also available at https://ngp-mpc.github.io/.

UR - http://www.scopus.com/inward/record.url?scp=85168694346&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85168694346&partnerID=8YFLogxK

U2 - 10.1109/ICRA48891.2023.10160247

DO - 10.1109/ICRA48891.2023.10160247

M3 - Conference contribution

AN - SCOPUS:85168694346

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 1778

EP - 1785

BT - Proceedings - ICRA 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE International Conference on Robotics and Automation, ICRA 2023

Y2 - 29 May 2023 through 2 June 2023

ER -

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this