Context-aware Talking Face Video Generation
CoRR(2024)
摘要
In this paper, we consider a novel and practical case for talking face video
generation. Specifically, we focus on the scenarios involving multi-people
interactions, where the talking context, such as audience or surroundings, is
present. In these situations, the video generation should take the context into
consideration in order to generate video content naturally aligned with driving
audios and spatially coherent to the context. To achieve this, we provide a
two-stage and cross-modal controllable video generation pipeline, taking facial
landmarks as an explicit and compact control signal to bridge the driving
audio, talking context and generated videos. Inside this pipeline, we devise a
3D video diffusion model, allowing for efficient contort of both spatial
conditions (landmarks and context video), as well as audio condition for
temporally coherent generation. The experimental results verify the advantage
of the proposed method over other baselines in terms of audio-video
synchronization, video fidelity and frame consistency.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要