Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze

Yu Chen Chang, Nitish Gandi, Kazuki Shin, Ye Ji Mun, Katherine Driggs-Campbell, Joohyung Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Current approaches in robot teleoperation often require significant mental and physical effort. In this study, we propose a new intent detection framework to teleoperate robotic arms based on human speech and natural eye gaze. Our framework applies instance segmentation on the robot's camera image and predicts the human's intended object through matching eye-gaze data, instance masks, instance classes, and transcribed words. Our experiment results show a prediction accuracy between 90.7% and 98.6%, including cases when the target objects are duplicated or occluded. The prediction accuracy of the combination of eye gaze and speech inputs outperformed the prediction accuracy of eye gaze input only, between 79.9% and 89.2%, and speech input only, between 25.3% and 71.6%. Moreover, we observe that eye gaze input has a greater importance than speech input in improving prediction accuracy when two duplicated target objects are present in the scene. Our results from NASA TLX questionnaires show that teleoperating the robotic arms with our proposed framework requires little effort including cases when the target objects are duplicated or occluded.

Original languageEnglish (US)
Title of host publication2023 IEEE-RAS 22nd International Conference on Humanoid Robots, Humanoids 2023
PublisherIEEE Computer Society
ISBN (Electronic)9798350303278
DOIs
StatePublished - 2023
Event22nd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2023 - Austin, United States
Duration: Dec 12 2023Dec 14 2023

Publication series

NameIEEE-RAS International Conference on Humanoid Robots
ISSN (Print)2164-0572
ISSN (Electronic)2164-0580

Conference

Conference22nd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2023
Country/TerritoryUnited States
CityAustin
Period12/12/2312/14/23

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze'. Together they form a unique fingerprint.

Cite this