1 Département d'informatique et de recherche opérationnelle, Université de Montréal 2 Mila - Quebec Artificial Intelligence Institute 3 Robotics Institute, Carnegie Mellon University 4 Department of Psychology, University of California, Berkeley 5 CIFAR Artificial Intelligence Chair
* Corresponding author: tabitha [hyphen] edith [dot] lee [at] mila [dot] quebec † Equal contribution
Abstract: Due to fundamental exploration challenges without informed priors or specialized algorithms, agents may be unable to consistently receive informative rewards, leading to inefficient or intractable learning. To address these challenges, we introduce CURATE, an automatic curriculum learning algorithm for reinforcement learning agents in structured task spaces of monotonic difficulty. Through "exploration by exploitation," CURATE dynamically scales the task difficulty to match the agent's current competence. By exploiting its current capabilities that were learned in easier tasks, the agent improves its exploration in more difficult tasks. Our key insight is that the learning improvement in tasks that are close to those used for training is inversely proportional to their difficulty, and an agent that chooses a nearby distribution of the easiest unsolved tasks at any given time can automatically induce an easiest-to-hardest curriculum in these task spaces. To achieve this, CURATE conducts policy search in the task space to learn the best task distribution for training. As the agent's mastery grows, the learned curriculum adapts in an approximately easiest-to-hardest and task-directed fashion, efficiently culminating in a performant agent. Our experiments across three diverse domains (MiniGrid, Procgen, BipedalWalker) demonstrate that CURATE learns effective curricula for sample efficiency and proficiency with the potential for yielding broadly capable agents, matching or exceeding prior curriculum methods that do not require informed initialization or predefined schedules.
Summary
CURATE is an automatic curriculum learning algorithm for reinforcement learning agents that adapts the difficulty of training tasks to the agent's current competence.
By prioritizing the easiest (i.e., highest return) tasks that have not yet been solved, CURATE induces an approximately easy-to-hard curricula.
Designed for difficult tasks, CURATE learns effective curricula for sample efficiency and proficiency with the potential for yielding broadly capable agents.
Overview of Results
Experimental domains studied in the CURATE work.
In our work, we examine CURATE's performance across three domains that are diverse in task parameterization, dimensionality, actions, and observations.
These include MiniGrid MultiRoom, all 16 games of the Procgen Curriculum Suite, and BipedalWalker.
We find that CURATE learns effective curricula for sample efficiency and proficiency with the potential for yielding broadly capable agents, matching or exceeding prior curriculum methods that do not require informed initialization or predefined schedules.
We note that our investigation focused on the subclass of curriculum learning methods for which neither informed initializations nor predefined schedules are available (with the exception of ACCEL, which typically starts from simple tasks, and the Hand Curriculum for MultiRoom, which represents a hand-crafted, expert curriculum).
Limiting our investigation to such a subclass is valuable, as specifying an informed initialization or predefined schedule may require human expertise for best performance.
Sample Efficiency and Proficiency
CURATE is effective at solving difficult target tasks with sample efficiency and proficiency.
MiniGrid MultiRoom (10 trials)
Procgen Curriculum Suite (16 games, 6 trials per game)
Potential for Generalization and Broad Test Performance
Under some conditions, CURATE offers generalization across the task space, training broadly capable agents.
Procgen Curriculum Suite: CoinRun (aggregate of 15 tasks, 6 trials)
BipedalWalker (aggregate of 7 tasks, 10 trials)
Visualizations of CURATE Curricula
Below are visualizations of curricula that are learned by CURATE.
These visualizations comprise all of the one- and two-dimensional task spaces that were investigated in this work.
Certain properties are evident, including:
an approximately easiest-to-hardest progression, which is achieved without a priori schedules;
maintaining relatively smaller training distributions at a given time, which focuses training to boost learning progress;
task directedness (for multidimensional task spaces), which keeps the curricula steered towards the target tasks;
diversity in starting task distribution, which allows CURATE to bypass large sections of the task space that the agent can already solve.
Videos of Agents Trained using CURATE Curricula
Below are videos of successful trajectories for trained agents that use CURATE curricula to solve the most difficult tasks in their respective domain.
In these videos, MiniGrid uses 5 episodes, Procgen Curriculum Suite uses 3 episodes, and BipedalWalker uses 1 episode.
The Procgen Curriculum Suite is our adaptation of the classic Procgen benchmark by Cobbe et al., ICML 2020.
We introduce the Procgen Curriculum Suite as a part of the CURATE work to support benchmarking in curriculum learning with a human-specified task space for each game.
The BipedalWalker domain used in our work is from J. Parker-Holder and M. Jiang et al., ICML 2022.
The BipedalWalker domain was first introduced by Brockman et al., arXiv 2016.
The BipedalWalker Max task is introduced in our work and represents tasks at maximum difficulty: none of the methods we tested did well in this domain.
BipedalWalker-v3
BipedalWalkerHardcore-v3
BipedalWalker-Max-v0
BipedalWalker-Med-Stairs-v0
BipedalWalker-Med-PitGap-v0
BipedalWalker-Med-StumpHeight-v0
BipedalWalker-Med-Roughness-v0
Citation
Please consider citing our work if you find it useful or related to your research. Thank you.
@article{lee2026curate,
title={CURATE: Automatic Curriculum Learning for Reinforcement Learning Agents through Competence-Based Curriculum Policy Search in Structured Task Spaces},
author={Lee, Tabitha Edith and Ke, Nan Rosemary and Patil, Sarvesh and Dahmani, Annya and Yiu, Eunice and Saleh, Esra'a and Gopnik, Alison and Kroemer, Oliver and Berseth, Glen},
journal={Transactions on Machine Learning Research (TMLR)},
month={August},
year={2026},
organization={TMLR},
url={https://openreview.net/forum?id=DlnvWfoIgv},
}
Acknowledgments
We gratefully acknowledge the following funding support for our research: IVADO, including the Postdoctoral Research Funding Program; the Natural Sciences and Engineering Research Council of Canada (NSERC); the Canadian Institute for Advanced Research (CIFAR), including the Next Generation Trainee Fellowship for the Learning in Machines & Brain program; the National Institute of Standards and Technology of the United States (NIST, award no. 70NANB23H178); Google-BAIR Commons; the DARPA Machine Common Sense Program; the ONR MURI program; and the NSF ARIA AI Institute.
We also gratefully acknowledge compute resources and support from the Digital Research Alliance of Canada (DRAC), Mila, and NVIDIA.
We thank the Mila IDT team, including Fabrice Normandin and Olexa Bilaniuk, for their help and support.
We also thank Minqi Jiang, Michael Dennis, Roger Creus Castanyer, and Florence Cloutier for their insight, feedback, and support with our work.
We also thank our reviewers for their helpful feedback that strengthened our work.
|
|
