Human-Guided Robot Task and Motion Adaptation through Demonstrations, Language, and Physical Corrections
Speaker
About this event
Robots are increasingly deployed beyond traditional industrial settings, extending their use to homes, hospitals, agriculture, and other everyday environments. In these applications, robots must perform diverse tasks under varying conditions while accounting for user preferences and expectations. Traditional robot programming is effective for repetitive tasks, but it may be insufficient when robot behavior must adapt to end-user needs and safety concerns. This motivates intuitive human-robot interaction methods that allow users to guide, correct, and personalize robot behavior without programming expertise. The first part of this thesis addresses this need through learning from demonstration (LfD), which enables non-expert users to teach robot tasks without explicit programming. While many LfD approaches focus on reproducing demonstrated motions or task sequences, demonstrations can also implicitly convey why particular objects are selected. To capture this intention, we propose a keyframe-based method that infers task goals from object-attribute constraints. During demonstrations, the system records the robot state, object poses, and object attributes, and identifies object-centric keyframes based on meaningful environmental changes. It then extracts constraints over discrete and continuous object attributes across and within demonstrations. These constraints represent the semantic rationale behind the user’s object choices and are used during reproduction to select suitable objects in new environments. In this way, the robot can generalize demonstrated tasks to novel scenes and unseen objects while preserving the user’s intended object-level semantics. Although demonstrations can teach the robot what task to perform, the resulting motion may still conflict with the user’s preferences or safety expectations. Natural language offers an intuitive way to correct robot motion, such as asking the robot to move slower or keep a larger distance from a fragile item. However, existing low-level correction methods often treat feedback as environment-specific and mainly focus on unconditional corrections that specify only the desired motion change. Some corrections are valid only under specific conditions; ignoring these conditions may cause unnecessary detours or overly conservative behavior. Conditional corrections therefore introduce an additional challenge because they specify both what the robot should do and when the correction should be applied, as in “stay away from the open book while carrying a full cup.” The second part of this thesis therefore represents such corrections as reusable user preferences. A fine-tuned language model converts each correction into a structured preference, which is stored in memory and later selected based on the current scene and robot state. The selected preferences are activated over relevant trajectory segments and converted into constraints to generate corrected motion in new environments. While natural language corrections are semantically informative, they may lack precise physical grounding about the desired motion change. In contrast, physical corrections provide direct geometric information but can be semantically ambiguous, as the same intervention may reflect different intentions. The third part of this thesis exploits these complementary modalities to construct and refine structured task-and-motion plans. Verbal cues clarify the semantic target of the correction, while physical interaction grounds the correction in the robot’s motion, allowing the model to infer the corresponding update to the task-and-motion plan. The plan is represented as a sequence of task-level objectives, such as grasping or placing an object, together with motion-level preferences between consecutive objectives. A transformer-based model conditions the current multimodal feedback on the previous plan to support both plan construction and incremental revision. We compare two architectures that differ in how the previous plan is incorporated.
Host
