Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels
CoRR(2024)
摘要
The rapid advancement of natural language processing, information retrieval
(IR), computer vision, and other technologies has presented significant
challenges in evaluating the performance of these systems. One of the main
challenges is the scarcity of human-labeled data, which hinders the fair and
accurate assessment of these systems. In this work, we specifically focus on
evaluating IR systems with sparse labels, borrowing from recent research on
evaluating computer vision tasks. taking inspiration from the success of using
Fréchet Inception Distance (FID) in assessing text-to-image generation
systems. We propose leveraging the Fréchet Distance to measure the distance
between the distributions of relevant judged items and retrieved results. Our
experimental results on MS MARCO V1 dataset and TREC Deep Learning Tracks query
sets demonstrate the effectiveness of the Fréchet Distance as a metric for
evaluating IR systems, particularly in settings where a few labels are
available. This approach contributes to the advancement of evaluation
methodologies in real-world scenarios such as the assessment of generative IR
systems.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要