Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents
CoRR(2024)
摘要
Interactive Data Analysis, the collaboration between humans and LLM agents,
enables real-time data exploration for informed decision-making. The challenges
and costs of collecting realistic interactive logs for data analysis hinder the
quantitative evaluation of Large Language Model (LLM) agents in this task. To
mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate
LLM agents on interactive data analysis. Tapilot-Crossing contains 1024
interactions, covering 4 practical scenarios: Normal, Action, Private, and
Private Action. Notably, Tapilot-Crossing is constructed by an economical
multi-agent environment, Decision Company, with few human efforts. We evaluate
popular and advanced LLM agents in Tapilot-Crossing, which underscores the
challenges of interactive data analysis. Furthermore, we propose Adaptive
Interaction Reflection (AIR), a self-generated reflection strategy that guides
LLM agents to learn from successful history. Experiments demonstrate that Air
can evolve LLMs into effective interactive data analysis agents, achieving a
relative performance improvement of up to 44.5
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要