Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training
CoRR(2024)
摘要
In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel
and cost-effective approach that significantly enhances the pre-training
performance of Masked Image Modeling (MIM) approaches by prioritizing token
salience. Our method provides robustness against variations in masking ratios,
effectively mitigating the performance instability issues common in existing
methods. This relaxes the sensitivity of MIM-based pre-training to masking
ratios, which in turn allows us to propose an adaptive strategy for `tailored'
masking ratios for each data sample, which no existing method can provide.
Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that
dynamically adjusts the proportion of masking for the unique content of each
image based on token salience. We show that our method significantly improves
over the state-of-the-art in mask-based pre-training on the ImageNet-1K
dataset.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要