On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning
arxiv(2024)
摘要
Self-supervised learning (SSL) has developed rapidly in recent years.
However, most of the mainstream methods are computationally expensive and rely
on two (or more) augmentations for each image to construct positive pairs.
Moreover, they mainly focus on large models and large-scale datasets, which
lack flexibility and feasibility in many practical applications. In this paper,
we propose an efficient single-branch SSL method based on non-parametric
instance discrimination, aiming to improve the algorithm, model, and data
efficiency of SSL. By analyzing the gradient formula, we correct the update
rule of the memory bank with improved performance. We further propose a novel
self-distillation loss that minimizes the KL divergence between the probability
distribution and its square root version. We show that this alleviates the
infrequent updating problem in instance discrimination and greatly accelerates
convergence. We systematically compare the training overhead and performance of
different methods in different scales of data, and under different backbones.
Experimental results show that our method outperforms various baselines with
significantly less overhead, and is especially effective for limited amounts of
data and small models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要