Action sequencing using visual permutations

Michael Burke, Subramanian Ramamoorthy, Kartic Subr

Research output: Contribution to journalArticleResearchpeer-review


Humans can easily reason about the sequence of high level actions needed to complete tasks, but it is particularly difficult to instil this ability in robots trained from relatively few examples. This work considers the task of neural action sequencing conditioned on a single reference visual state. This task is extremely challenging as it is not only subject to the significant combinatorial complexity that arises from large action sets, but also requires a model that can perform some form of symbol grounding, mapping high dimensional input data to actions, while reasoning about action relationships. Drawing on human cognitive abilities to rearrange objects in scenes to create new configurations, we take a permutation perspective and argue that action sequencing benefits from the ability to reason about both permutations and ordering concepts. Empirical analysis shows that neural models trained with latent permutations outperform standard neural architectures in constrained action sequencing tasks. Results also show that action sequencing using visual permutations is an effective mechanism to initialise and speed up traditional planning techniques and successfully scales to far greater action set sizes than models considered previously.

Original languageEnglish
Pages (from-to)1745-1752
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number2
Publication statusPublished - Apr 2021


  • Cloning
  • Deep Learning Methods
  • Learning from Demonstration
  • Modeling
  • Planning
  • Representation Learning
  • Robots
  • Sequential analysis
  • Task analysis
  • Visualization

Cite this