Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning
CoRR(2024)
摘要
Supervised learning is often computationally easy in practice. But to what
extent does this mean that other modes of learning, such as reinforcement
learning (RL), ought to be computationally easy by extension? In this work we
show the first cryptographic separation between RL and supervised learning, by
exhibiting a class of block MDPs and associated decoding functions where
reward-free exploration is provably computationally harder than the associated
regression problem. We also show that there is no computationally efficient
algorithm for reward-directed RL in block MDPs, even when given access to an
oracle for this regression problem.
It is known that being able to perform regression in block MDPs is necessary
for finding a good policy; our results suggest that it is not sufficient. Our
separation lower bound uses a new robustness property of the Learning Parities
with Noise (LPN) hardness assumption, which is crucial in handling the
dependent nature of RL data. We argue that separations and oracle lower bounds,
such as ours, are a more meaningful way to prove hardness of learning because
the constructions better reflect the practical reality that supervised learning
by itself is often not the computational bottleneck.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要