Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset
CoRR(2024)
摘要
Existing work has observed that current text-to-image systems do not
accurately reflect explicit spatial relations between objects such as 'left of'
or 'below'. We hypothesize that this is because explicit spatial relations
rarely appear in the image captions used to train these models. We propose an
automatic method that, given existing images, generates synthetic captions that
contain 14 explicit spatial relations. We introduce the Spatial Relation for
Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for
training, and more than 60 thousand captions for evaluation. In order to test
generalization we also provide an 'unseen' split, where the set of objects in
the train and test captions are disjoint. SR4G is the first dataset that can be
used to spatially fine-tune text-to-image systems. We show that fine-tuning two
different Stable Diffusion models (denoted as SD_SR4G) yields up to 9
points improvements in the VISOR metric. The improvement holds in the 'unseen'
split, showing that SD_SR4G is able to generalize to unseen objects.
SD_SR4G improves the state-of-the-art with fewer parameters, and avoids
complex architectures. Our analysis shows that improvement is consistent for
all relations. The dataset and the code will be publicly available.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要