A System-Level Analysis of Conference Peer Review

BSTRACTWe undertake a system-level analysis of the conference peer review process. The process involves three constituencies with different objectives: authors want their papers accepted at prestigious venues (and quickly), conferences want to present a program with many high-quality and few low-quality papers, and reviewers want to avoid being overburdened by reviews. These objectives are far from aligned; the key obstacle is that the evaluation of the merits of a submission (both by the authors and the reviewers) is inherently noisy. Over the years, conferences have experimented with numerous policies and innovations to navigate the tradeoffs. These experiments include setting various bars for acceptance, varying the number of reviews per submission, requiring prior reviews to be included with resubmissions, and others. The purpose of the present work is to investigate, both analytically and using agent-based simulations, how well various policies work, and more importantly, why they do or do not work. We model the conference-author interactions as a Stackelberg game in which a prestigious conference commits to a threshold acceptance policy which will be applied to the (noisy) reviews of each submitted paper; the authors best-respond by submitting or not submitting to the conference, the alternative being a "sure accept" (such as arXiv or a lightly refereed venue). Our findings include: observing that the conference should typically set a higher acceptance threshold than the actual desired quality, which we call the resubmission gap and quantify in terms of various parameters; observing that the reviewing load is heavily driven by resubmissions of borderline papers --- therefore, a judicious choice of acceptance threshold may lead to fewer reviews while incurring an acceptable loss in quality; observing that depending on the paper quality distribution, stricter reviewing may lead to higher or lower acceptance rates --- the former is the result of self selection by the authors. As a rule of thumb, a relatively small number of reviews per paper, coupled with a strict acceptance policy, tends to do well in trading off these two objectives; finding that a relatively small increase in review quality or in self assessment by the authors is much more effective for conference quality control (without a large increase in review burden) than increases in the quantity of reviews per paper.; showing that keeping track of past reviews of papers can help reduce the review burden without a decrease in conference quality. For robustness, we consider different models of paper quality and learn some of the parameters from real data.