HyperAgent: A Simple, Scalable, Efficient and Provable Reinforcement Learning Framework for Complex Environments
CoRR(2024)
摘要
To solve complex tasks under resource constraints, reinforcement learning
(RL) agents need to be simple, efficient, and scalable with (1) large state
space and (2) increasingly accumulated data of interactions. We propose the
HyperAgent, a RL framework with hypermodel, index sampling schemes and
incremental update mechanism, enabling computation-efficient sequential
posterior approximation and data-efficient action selection under general value
function approximation beyond conjugacy. The implementation of is
simple as it only adds one module and one line of code additional to DDQN.
Practically, HyperAgent demonstrates its robust performance in large-scale deep
RL benchmarks with significant efficiency gain in terms of both data and
computation. Theoretically, among the practically scalable algorithms,
HyperAgent is the first method to achieve provably scalable per-step
computational complexity as well as sublinear regret under tabular RL. The core
of our theoretical analysis is the sequential posterior approximation argument,
made possible by the first analytical tool for sequential random projection, a
non-trivial martingale extension of the Johnson-Lindenstrauss lemma. This work
bridges the theoretical and practical realms of RL, establishing a new
benchmark for RL algorithm design.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要