RoboTwin: Robotic Teleoperation Using Digital Twins

Low-latency, safe teleoperation through synchronized digital twins

RoboTwin is a dual-digital-twin framework for teleoperating a remote robotic manipulator. Instead of relying only on delayed video, the operator controls a local digital twin and receives immediate visual and haptic feedback. A second twin at the remote site synchronizes the physical robot, acts as a safety layer, and communicates changes in the real environment back to the operator.

I designed and implemented the intercity teleoperation system for controlling a UR3 manipulator over approximately 400 km, developed the real-time digital twin, and designed discrepancy-detection and state-correction methods for synchronizing the simulated and physical environments. This work was carried out at the Indian Institute of Science, Bengaluru, with Diwakar Kumar Singh, Nithish Krishnabharathi Gnani, Prof. T. V. Prabhakar, and Prof. Chandramani Singh.

Paper  ·  Demo videos

Full system demo

End-to-end RoboTwin teleoperation with a haptic device, synchronized digital twins, and a physical UR3 manipulator.

Dual-digital-twin architecture

The operator-side twin follows commands from a Geomagic Touch haptic device with negligible local delay. End-effector poses are sent to the remote-side twin, where RMPflow and ROS 2 generate commands for the physical UR3. Contact forces from the simulated environment are returned through the haptic device, while motion scaling supports precise micro- and macro-movements.

Two coupled feedback loops connect the operator, simulated environments, and physical robot.

Environment synchronization

A RealSense RGB-D camera observes the remote workspace, while a matching virtual camera renders the expected scene in Isaac Sim. The system compares the real and synthetic observations to detect discrepancies. Known objects are localized with YOLOv8 and rendered in the twin; unexpected objects are extracted as point clouds from RGB-D differences and transmitted to update the operator-side environment.

Compact pose and environment updates flow between the haptic interface, both twins, and the real robot.
Known objects detected in the real workspace and rendered in the digital twin.
Previously unknown objects transferred to the digital twin as discrepancy point clouds.

Evaluation

We evaluated RoboTwin through a spiral-tracing study with 17 participants under 1 ms and 100 ms round-trip network delays. Compared with conventional video feedback, the digital-twin interface reduced reported workload across the NASA-TLX dimensions and enabled more stable tracing under high latency. Transmitting compact object updates instead of a continuous video stream also reduced the data rate for operator-side object identification by approximately 25×.