Fig. 1: Two types of task structures: (i) a geometric task structure (Fig. 1A) that we use; (ii) a semantic task structure[3] (Fig. 1B) that extract task semantic meanings by a tree or graph, which has been intensively studied [13, 14] in the literature. 

This thesis studies how to enable a real-world robot to efficiently learn a new task by watching human demonstration videos. Learning by watching provides a more human-intuitive task teaching interface than methods requiring coordinates programming, reward/cost design, kinesthetic teaching, or teleoperation.  However, challenges regarding massive human demonstrations, tedious data annotations or heavy training of a robot controller impede its acceptance in real-world applications. To overcome these challenges, we introduce a geometric task structure to the problem solution. 

What is a geometric task structure?

A geometric task structure uses geometric features observed in image planes[1, 2] to specify a task by either explicitly forming geometric constraints or implicitly extracting task-relevant keypoints. For example, in figure 1A, a screwing task can be specified by points and lines constraints. 

What new insights can it bring to us in robot learning?

• It enables unsupervised learning from a few human demonstrations without tedious annotations. We propose Visual Geometric Skill Imitation Learning (VGS-IL) (our publication [4,5]), a method that uses a task function to learn the geometric task structure by extracting task-relevant image features to compose geometric constraints, thus forming an efficient and interpretable representation. The task function is optimized by Incremental Maximum Entropy Inverse Reinforcement Learning (InMaxEnt-IRL) (our publication [6]) based on “temporal-frame-orders” in human demonstration video. 

• It enables efficient transfer to a robot controller. Unlike prevalent methods that require intensive training on the robot when mapping the learned representation to robot actions, our proposed task function selects out geometric constraints with adjoint geometric errors that can be directly used in a conventional visual servoing controller or a UVS controller[7]. 

• Furthermore, we show that introducing geometric task structure to the representation learning of “what is the task” enables task generalization. We devise Categorical Object Generalizable VGS-IL (CoVGS-IL)  that uses a graph-structured task function to select out task-relevant image features on categorical objects with the same task functionality, thus fulfilling task generalization. The insight is  “task specification correspondence” as shown in Fig. 2 below.

Fig. 2: The same geometric constraint from categorical objects defines the same task, which is called task specification correspondence. The affordance of object parts alone does not define any task. Beyond affordance, the interconnections between multiple object parts provide task specification, thus represents the task and can be used to build task specification correspondence.

How does this work relate to robot learning literature?

In addition to the "learning by watching" approaches, a.k.a., "third-person visual imitation learning"[12],  our work also closely relates to methods that use task-relevant geometric features to boost up sample-efficient robot learning and task generalization (Fig. 3).