Back2Future-SIM: Creating Real-Time Interactable Immersive Virtual World For Robot Teleoperation
Sait Akturk
Supervisors : Martin Jagersand, Li Cheng
completed_ai_compressed.mp4thesis.pdf
Abstract
In the context of human-robot interaction (HRI), robot autonomy addresses how environmental input influences a robot's actions, spanning a spectrum from full human control to independent robot motion. The quality of HRI relies on information exchange, evaluated through factors like interaction duration, mental workload, and spatial awareness. Sensory feedback is crucial, with a growing interest in combining communication methods for more natural interactions. Challenges arise in distant proximity interactions with communication delays, where predictive sensory feedback emerges as a solution to enhance interaction efficiency by providing predictive information from the robot's environment.
In this thesis, our focus is on providing predictive haptic feedback and immersive visuals from the virtual replica of the remote scene in a physics simulator. Our system acts as a bridge between the operator and the follower robot, considering real-world constraints. We create a cyber-physical system using a real-time 3D surface mesh reconstruction algorithm and a digital twin of the Barrett WAM arm robot. The Gazebo physics simulator integrates the digital twin and an incremental surface mesh to create a real-time virtual replica of the remote scene. This virtual replica is used to provide haptic surface interaction feedback through collision detection from the physics simulator. Additionally, we address the operator's spatial awareness by using an immersive display for predictive visualization.
System Diagram
Surface Reconstruction
We used the incremental surface mesh reconstruction algorithm not just to provide 3D information but to address the delay issue of robot teleoperation. The visual feedback from the follower robot is obtained by the motion and the requirement for visual feedback often requires low latency and high bandwidth network. Independent intermediate representation that is incrementally updated using remote scenes eliminates the delay problem using predictive immersive display. Additionally, We used the incremental surface mesh for creating collision mesh in the physics simulator. Thus, the surface mesh reconstruction is not just important for providing predictive visual, but also predictive haptic feedback through interaction in the virtual replica.
The usage of the physics simulator in our proposed system is multifaceted. We use a physics simulator not only to integrate the incremental surface mesh and the digital twin of the WAM arm robot for solving delay issues but also to improve the operator's agility and understanding of the remote scene and the follower robot state. While the interaction of the modalities provides the predictive haptic feedback, the utilization of the sensors provided by the physics simulator provides immersive predictive visuals including interaction and the follower robot state. The intermediate virtual representation gives full control of the remote environment and the follower robot by not requiring direct feedback.
Physics Simulator
Immersive Display
We use the immersive display to provide predictive visuals based on the virtual replica of the follower robot environments. We control the camera pose and view using the operator's head motions and the hand controller, which improve spatial awareness and reduce delay-related cognitive load.
Experiments & Results
Precision:the accuracy of the estimated mesh surface compared to the ground truth scan.
Completeness: measures how well the estimated 3D points align with the ground truth points within a 25 mm threshold.
RTF (Real-Time Factor) : is a measure of how a computer simulation's speed compares to real-world time, with 1 being real-time, above 1 faster, and below 1 slower.
