OmniVid: A Generative Framework for Universal Video Understanding
CVPR 2024(2024)
摘要
The core of video understanding tasks, such as recognition, captioning, and
tracking, is to automatically detect objects or actions in a video and analyze
their temporal evolution. Despite sharing a common goal, different tasks often
rely on distinct model architectures and annotation formats. In contrast,
natural language processing benefits from a unified output space, i.e., text
sequences, which simplifies the training of powerful foundational language
models, such as GPT-3, with extensive training corpora. Inspired by this, we
seek to unify the output space of video understanding tasks by using languages
as labels and additionally introducing time and box tokens. In this way, a
variety of video tasks could be formulated as video-grounded token generation.
This enables us to address various types of video tasks, including
classification (such as action recognition), captioning (covering clip
captioning, video question answering, and dense video captioning), and
localization tasks (such as visual object tracking) within a fully shared
encoder-decoder architecture, following a generative framework. Through
comprehensive experiments, we demonstrate such a simple and straightforward
idea is quite effective and can achieve state-of-the-art or competitive results
on seven video benchmarks, providing a novel perspective for more universal
video understanding. Code is available at https://github.com/wangjk666/OmniVid.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要