Asymptotics of Language Model Alignment
arxiv(2024)
摘要
Let p denote a generative language model. Let r denote a reward model
that returns a scalar that captures the degree at which a draw from p is
preferred. The goal of language model alignment is to alter p to a new
distribution ϕ that results in a higher expected reward while keeping
ϕ close to p. A popular alignment method is the KL-constrained
reinforcement learning (RL), which chooses a distribution ϕ_Δ that
maximizes E_ϕ_Δ r(y) subject to a relative entropy constraint
KL(ϕ_Δ || p) ≤Δ. Another simple alignment method is
best-of-N, where N samples are drawn from p and one with highest reward
is selected. In this paper, we offer a closed-form characterization of the
optimal KL-constrained RL solution. We demonstrate that any alignment method
that achieves a comparable trade-off between KL divergence and reward must
approximate the optimal KL-constrained RL solution in terms of relative
entropy. To further analyze the properties of alignment methods, we introduce
two simplifying assumptions: we let the language model be memoryless, and the
reward model be linear. Although these assumptions may not reflect complex
real-world scenarios, they enable a precise characterization of the asymptotic
behavior of both the best-of-N alignment, and the KL-constrained RL method,
in terms of information-theoretic quantities. We prove that the reward of the
optimal KL-constrained RL solution satisfies a large deviation principle, and
we fully characterize its rate function. We also show that the rate of growth
of the scaled cumulants of the reward is characterized by a proper Renyi cross
entropy. Finally, we show that best-of-N is asymptotically equivalent to
KL-constrained RL solution by proving that their expected rewards are
asymptotically equal, and concluding that the two distributions must be close
in KL divergence.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要