Benchmarking Educational Program Repair
arxiv(2024)
摘要
The emergence of large language models (LLMs) has sparked enormous interest
due to their potential application across a range of educational tasks. For
example, recent work in programming education has used LLMs to generate
learning resources, improve error messages, and provide feedback on code.
However, one factor that limits progress within the field is that much of the
research uses bespoke datasets and different evaluation metrics, making direct
comparisons between results unreliable. Thus, there is a pressing need for
standardization and benchmarks that facilitate the equitable comparison of
competing approaches. One task where LLMs show great promise is program repair,
which can be used to provide debugging support and next-step hints to students.
In this article, we propose a novel educational program repair benchmark. We
curate two high-quality publicly available programming datasets, present a
unified evaluation procedure introducing a novel evaluation metric rouge@k for
approximating the quality of repairs, and evaluate a set of five recent models
to establish baseline performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要