Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs
CoRR(2024)
摘要
The advent of Large Language Models (LLMs) has significantly transformed the
AI landscape, enhancing machine learning and AI capabilities. Factuality issue
is a critical concern for LLMs, as they may generate factually incorrect
responses. In this paper, we propose GraphEval to evaluate an LLM's performance
using a substantially large test dataset. Specifically, the test dataset is
retrieved from a large knowledge graph with more than 10 million facts without
expensive human efforts. Unlike conventional methods that evaluate LLMs based
on generated responses, GraphEval streamlines the evaluation process by
creating a judge model to estimate the correctness of the answers given by the
LLM. Our experiments demonstrate that the judge model's factuality assessment
aligns closely with the correctness of the LLM's generated outputs, while also
substantially reducing evaluation costs. Besides, our findings offer valuable
insights into LLM performance across different metrics and highlight the
potential for future improvements in ensuring the factual integrity of LLM
outputs. The code is publicly available at https://github.com/xz-liu/GraphEval.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要