CURATE: Automatic Curriculum Learning for Reinforcement Learning Agents through Competence-Based Curriculum Policy Search in Structured Task Spaces
Transactions on Machine Learning Research (TMLR), August 2026
Tabitha Edith Lee,1,2*
Nan Rosemary Ke,2†
Sarvesh Patil,3†
Annya Dahmani,4
Eunice Yiu,4
Esra'a Saleh,1,2
Alison Gopnik,4
Oliver Kroemer,3
and Glen Berseth1,2,5
1 Département d'informatique et de recherche opérationnelle, Université de Montréal
2 Mila - Quebec Artificial Intelligence Institute
3 Robotics Institute, Carnegie Mellon University
4 Department of Psychology, University of California, Berkeley
5 CIFAR Artificial Intelligence Chair
* Corresponding author: tabitha [hyphen] edith [dot] lee [at] mila [dot] quebec
† Equal contribution
Abstract: Due to fundamental exploration challenges without informed priors or specialized algorithms, agents may be unable to consistently receive informative rewards, leading to inefficient or intractable learning. To address these challenges, we introduce CURATE, an automatic curriculum learning algorithm for reinforcement learning agents in structured task spaces of monotonic difficulty. Through "exploration by exploitation," CURATE dynamically scales the task difficulty to match the agent's current competence. By exploiting its current capabilities that were learned in easier tasks, the agent improves its exploration in more difficult tasks. Our key insight is that the learning improvement in tasks that are close to those used for training is inversely proportional to their difficulty, and an agent that chooses a nearby distribution of the easiest unsolved tasks at any given time can automatically induce an easiest-to-hardest curriculum in these task spaces. To achieve this, CURATE conducts policy search in the task space to learn the best task distribution for training. As the agent's mastery grows, the learned curriculum adapts in an approximately easiest-to-hardest and task-directed fashion, efficiently culminating in a performant agent. Our experiments across three diverse domains (MiniGrid, Procgen, BipedalWalker) demonstrate that CURATE learns effective curricula for sample efficiency and proficiency with the potential for yielding broadly capable agents, matching or exceeding prior curriculum methods that do not require informed initialization or predefined schedules.
Videos of Agents Trained using CURATE Curricula
Below are videos of successful trajectories for trained agents that use CURATE curricula to solve the most difficult tasks in their respective domain.
In these videos, MiniGrid uses 5 episodes, Procgen Curriculum Suite uses 3 episodes, and BipedalWalker uses 1 episode.
MiniGrid MultiRoom
We use a reimplementation of the MultiRoom-N4-Random domain that first appeared in M. Jiang et al., ICML 2021.
This task is defined within the MiniGrid benchmark by Chevalier-Boisvert et al., NeurIPS 2023.
MiniGrid-MultiRoom-N4to4-S4to7-G13-T80-v0
Procgen Curriculum Suite
The Procgen Curriculum Suite is our adaptation of the classic Procgen benchmark by Cobbe et al., ICML 2020.
We introduce the Procgen Curriculum Suite as a part of the CURATE work to support benchmarking in curriculum learning with a human-specified task space for each game.
Procgen-BigFish-Easy-F10to10-Terminal-v0
Procgen-BossFight-Easy-H2to2-R2to2-I3to3-Terminal-v0
Procgen-CaveFlyer-Easy-O3to3-Terminal-v0
Procgen-Chaser-Easy-E3to3-O75to75-Terminal-v0
Procgen-Climber-Easy-P5to5-E20to20-Terminal-v0
Procgen-CoinRun-Easy-D3to3-S5to5-v0
Procgen-Dodgeball-Easy-E3to3-Terminal-v0
Procgen-FruitBot-Easy-W5to5-G60to60-B10to10-Terminal-v0
Procgen-Heist-Easy-D2to2-K3to3-v0
Procgen-Jumper-Easy-S20to20-v0
Procgen-Leaper-Easy-R3to3-W3to3-v0
Procgen-Maze-Easy-D6to6-v0
Procgen-Miner-Easy-D3to3-B20to20-Terminal-v0
Procgen-Ninja-Easy-D3to3-S5to5-v0
Plunder-Easy-T8to8-J10to10-Terminal-v0
Procgen-StarPilot-Easy-W250to250-T10to10-G3to3-F45to45-Terminal-v0
BipedalWalker
The BipedalWalker domain used in our work is from J. Parker-Holder and M. Jiang et al., ICML 2022.
The BipedalWalker domain was first introduced by Brockman et al., arXiv 2016.
The BipedalWalker Max task is introduced in our work and represents tasks at maximum difficulty: none of the methods we tested did well in this domain.
BipedalWalker-v3
BipedalWalkerHardcore-v3
BipedalWalker-Max-v0
BipedalWalker-Med-Stairs-v0
BipedalWalker-Med-PitGap-v0
BipedalWalker-Med-StumpHeight-v0
BipedalWalker-Med-Roughness-v0
Citation
Please consider citing our work if you find it useful or related to your research. Thank you.
@article{lee2026curate,
title={CURATE: Automatic Curriculum Learning for Reinforcement Learning Agents through Competence-Based Curriculum Policy Search in Structured Task Spaces},
author={Lee, Tabitha Edith and Ke, Nan Rosemary and Patil, Sarvesh and Dahmani, Annya and Yiu, Eunice and Saleh, Esra'a and Gopnik, Alison and Kroemer, Oliver and Berseth, Glen},
journal={Transactions on Machine Learning Research (TMLR)},
month={August},
year={2026},
organization={TMLR},
url={https://openreview.net/forum?id=DlnvWfoIgv},
}
|
|
MiniGrid-MultiRoom-N4to4-S4to7-G13-T80-v0
@article{lee2026curate,
title={CURATE: Automatic Curriculum Learning for Reinforcement Learning Agents through Competence-Based Curriculum Policy Search in Structured Task Spaces},
author={Lee, Tabitha Edith and Ke, Nan Rosemary and Patil, Sarvesh and Dahmani, Annya and Yiu, Eunice and Saleh, Esra'a and Gopnik, Alison and Kroemer, Oliver and Berseth, Glen},
journal={Transactions on Machine Learning Research (TMLR)},
month={August},
year={2026},
organization={TMLR},
url={https://openreview.net/forum?id=DlnvWfoIgv},
}
|
|