We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA outperforms the state-of-the-art imitation learning methods by 45.8 percents in success rate. It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making.

PDF

In Proceedings of the 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand, January 2022.

@inproceedings{corl2022-zhu, title={VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors}, author={Yifeng Zhu and Abhishek Joshi and Peter Stone and Yuke Zhu}, booktitle={Proceedings of the 6th Conference on Robot Learning (CoRL 2022)}, month={January}, address={Auckland, New Zealand}, url="http://nn.cs.utexas.edu/?corl2022-zhu", year={2022} }