Retrieval-Augmented Generation: Is Dense Passage Retrieval Retrieving?
CoRR(2024)
摘要
Dense passage retrieval (DPR) is the first step in the retrieval augmented
generation (RAG) paradigm for improving the performance of large language
models (LLM). DPR fine-tunes pre-trained networks to enhance the alignment of
the embeddings between queries and relevant textual data. A deeper
understanding of DPR fine-tuning will be required to fundamentally unlock the
full potential of this approach. In this work, we explore DPR-trained models
mechanistically by using a combination of probing, layer activation analysis,
and model editing. Our experiments show that DPR training decentralizes how
knowledge is stored in the network, creating multiple access pathways to the
same information. We also uncover a limitation in this training style: the
internal knowledge of the pre-trained model bounds what the retrieval model can
retrieve. These findings suggest a few possible directions for dense retrieval:
(1) expose the DPR training process to more knowledge so more can be
decentralized, (2) inject facts as decentralized representations, (3) model and
incorporate knowledge uncertainty in the retrieval process, and (4) directly
map internal model knowledge to a knowledge base.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要