A Framework and Algorithm for Human-Robot Collaboration Based on Multimodal Reinforcement Learning

Figure 7

A typical human-robot collaboration process. (a) The user makes an act of pointing to an unplaced block and speaks the speech keyword “put the block on”; (b) the multimodal reinforcement learning intention understanding algorithm derives the user's intention: the robot actively picks up the blocks; (c) the user gives the speech evaluation: “That is it” NLP evaluates this feedback to be positive; (d) the robot is reset, and the reinforcement learning learns the user's habit and the round of collaboration ends.

(a)

(b)

(c)

(d)