Open-Vocabulary Mobile Manipulation

This project explores how a mobile manipulator can turn an underspecified natural-language request into a grounded, executable plan. Given a command such as “Bring me something sweet,” the system reasons about what the user might mean, where a suitable object could be found, and which navigation and manipulation skills are required to complete the task.

The work is being developed at the Humanoid Robots Lab, University of Bonn, with Sagar and Tanya, under the supervision of Rohit Menon. A project overview is available in our presentation.

Full system demo

End-to-end execution of a language-guided mobile-manipulation task with the Toyota HSR. If the embedded player is unavailable, open the demo in Google Drive.

Planning and execution

The system combines an LLM planner with a semantic scene graph and a description of the robot’s available capabilities. The planner produces a structured sequence of grounded actions, which are passed to an execution loop built around navigation, manipulation, and sensor feedback. If grounding or execution fails, the resulting context is returned to the planner so that it can revise the plan.

High-level planning, action grounding, and closed-loop execution using the scene graph and robot capabilities.

Scene-graph grounding

My work focuses on constructing scene-graph representations of indoor environments and using them for language-guided task planning. We investigate two complementary pipelines: reconstructing an environment from iPad RGB-D scans and extracting objects with Mask3D, and building the scene graph directly from the Toyota HSR’s RGB-D observations using BOXER.

The graph organizes rooms, furniture, and objects into a representation that the planner can query. Overlaying this information on the map connects semantic reasoning with locations the robot can physically navigate to.

Semantic room, furniture, and object information overlaid on the robot's map.

Exploration and replanning

When the requested object is not already represented in the scene graph, the planner identifies plausible search locations and orders them for exploration. The robot navigates to each location and inspects the current view. A vision-language model reasons about whether a suitable object is visible, while SAM3 provides object-level verification and segmentation.

Once a valid target is discovered, the system adds the new context to the action history and asks the planner for an updated plan.

Open-vocabulary object inspection with the Toyota HSR

The robot inspects a candidate location, identifies relevant objects, and verifies the selected target before replanning.

Manipulation and clarification

After selecting an object, the revised plan invokes segmentation, grasp-pose estimation, and pickup. Navigation and manipulation are executed through Nav2 and MoveIt. If the original request does not specify where the object should be delivered, the robot asks the user for clarification, incorporates the response, and completes the delivery.

This separation between high-level reasoning and grounded robot skills allows the system to respond flexibly to new objects, incomplete requests, and changes observed during execution.