The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates
arxiv(2024)
摘要
Journals and conferences worry that peer reviews assisted by artificial
intelligence (AI), in particular, large language models (LLMs), may negatively
influence the validity and fairness of the peer-review system, a cornerstone of
modern science. In this work, we address this concern with a quasi-experimental
study of the prevalence and impact of AI-assisted peer reviews in the context
of the 2024 International Conference on Learning Representations (ICLR), a
large and prestigious machine-learning conference. Our contributions are
threefold. Firstly, we obtain a lower bound for the prevalence of AI-assisted
reviews at ICLR 2024 using the GPTZero LLM detector, estimating that at least
15.8% of reviews were written with AI assistance. Secondly, we estimate the
impact of AI-assisted reviews on submission scores. Considering pairs of
reviews with different scores assigned to the same paper, we find that in
53.4% of pairs the AI-assisted review scores higher than the human review
(p = 0.002; relative difference in probability of scoring higher: +14.4%
in favor of AI-assisted reviews). Thirdly, we assess the impact of receiving an
AI-assisted peer review on submission acceptance. In a matched study,
submissions near the acceptance threshold that received an AI-assisted peer
review were 4.9 percentage points (p = 0.024) more likely to be accepted
than submissions that did not. Overall, we show that AI-assisted reviews are
consequential to the peer-review process and offer a discussion on future
implications of current trends
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要