Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
arxiv(2023)
摘要
Vision language models (VLM) have demonstrated remarkable performance across
various downstream tasks. However, understanding fine-grained visual-linguistic
concepts, such as attributes and inter-object relationships, remains a
significant challenge. While several benchmarks aim to evaluate VLMs in finer
granularity, their primary focus remains on the linguistic aspect, neglecting
the visual dimension. Here, we highlight the importance of evaluating VLMs from
both a textual and visual perspective. We introduce a progressive pipeline to
synthesize images that vary in a specific attribute while ensuring consistency
in all other aspects. Utilizing this data engine, we carefully design a
benchmark, SPEC, to diagnose the comprehension of object size, position,
existence, and count. Subsequently, we conduct a thorough evaluation of four
leading VLMs on SPEC. Surprisingly, their performance is close to random guess,
revealing significant limitations. With this in mind, we propose a simple yet
effective approach to optimize VLMs in fine-grained understanding, achieving
significant improvements on SPEC without compromising the zero-shot
performance. Results on two additional fine-grained benchmarks also show
consistent improvements, further validating the transferability of our
approach. Code and data are available at https://github.com/wjpoom/SPEC.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要