A well-known challenge with symbolic systems is that they are constrained by their domain: that is, they require a prede- termined set of facts to compute over and struggle to handle novelty. Machine learning techniques offer a potential resolution to this problem by allowing exploration that expands the symbolic domain. To this end, we describe the “real–sim–real” problem, where an agent must construct its own simulation environment for training and then deploy the learned policy (correcting the simulation as necessary). We then present an approach which addresses this problem, and demonstrate that an agent can use sensor data to construct its own simulation environment, train in it, and then use this newly-obtained knowledge to solve a task planning problem. Further, we show that when there is a mismatch between the generated simulation environment and the real environment, this mismatch can be resolved by the agent through exploration in the real world, and then be again applied to solve the better-understood task planning problem. We integrate this work with prior work on object-centric policies, where the agent trains directly on the environment to understand how the environment should be manipulated, which improves sim-to-real transfer and allows training to occur in real time.
@inproceedings{thierauf2025icdl,
title={Robots that Learn to Solve Symbolic Novelties with Self-Generated RL Simulations},
author={Christopher Thierauf and Matthias Scheutz},
year={2025},
booktitle={International Conference on Development and Learning},
url={https://hrilab.tufts.edu/publications/thierauf2025icdl.pdf}
}