WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
CoRR(2024)
摘要
A common phenomena confining the representation quality in Self-Supervised
Learning (SSL) is dimensional collapse (also known as rank degeneration), where
the learned representations are mapped to a low dimensional subspace of the
representation space. The State-of-the-Art SSL methods have shown to suffer
from dimensional collapse and fall behind maintaining full rank. Recent
approaches to prevent this problem have proposed using contrastive losses,
regularization techniques, or architectural tricks. We propose WERank, a new
regularizer on the weight parameters of the network to prevent rank
degeneration at different layers of the network. We provide empirical evidence
and mathematical justification to demonstrate the effectiveness of the proposed
regularization method in preventing dimensional collapse. We verify the impact
of WERank on graph SSL where dimensional collapse is more pronounced due to the
lack of proper data augmentation. We empirically demonstrate that WERank is
effective in helping BYOL to achieve higher rank during SSL pre-training and
consequently downstream accuracy during evaluation probing. Ablation studies
and experimental analysis shed lights on the underlying factors behind the
performance gains of the proposed approach.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要