Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement
CoRR(2024)
摘要
The advent of Large Language Models (LLMs) has propelled dialogue generation
into new realms, particularly in the field of role-playing systems (RPSs).
While enhanced with ordinary role-relevant training dialogues, existing
LLM-based RPSs still struggle to align with roles when handling intricate and
trapped queries in boundary scenarios. In this paper, we design the Modular
ORchestrated Trap-setting Interaction SystEm (MORTISE) to benchmark and improve
the role-playing LLMs' performance. MORTISE can produce highly role-relevant
aggressive queries through the collaborative effort of multiple LLM-based
modules, and formulate corresponding responses to create an adversarial
training dataset via a consistent response generator. We select 190 Chinese and
English roles to construct aggressive queries to benchmark existing
role-playing LLMs. Through comprehensive evaluation, we find that existing
models exhibit a general deficiency in role alignment capabilities. We further
select 180 of the roles to collect an adversarial training dataset (named
RoleAD) and retain the other 10 roles for testing. Experiments on models
improved by RoleAD indicate that our adversarial dataset ameliorates this
deficiency, with the improvements demonstrating a degree of generalizability in
ordinary scenarios.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要