The Entropy Enigma: Success and Failure of Entropy Minimization
arxiv(2024)
摘要
Entropy minimization (EM) is frequently used to increase the accuracy of
classification models when they're faced with new data at test time. EM is a
self-supervised learning method that optimizes classifiers to assign even
higher probabilities to their top predicted classes. In this paper, we analyze
why EM works when adapting a model for a few steps and why it eventually fails
after adapting for many steps. We show that, at first, EM causes the model to
embed test images close to training images, thereby increasing model accuracy.
After many steps of optimization, EM makes the model embed test images far away
from the embeddings of training images, which results in a degradation of
accuracy. Building upon our insights, we present a method for solving a
practical problem: estimating a model's accuracy on a given arbitrary dataset
without having access to its labels. Our method estimates accuracy by looking
at how the embeddings of input images change as the model is optimized to
minimize entropy. Experiments on 23 challenging datasets show that our method
sets the SoTA with a mean absolute error of 5.75%, an improvement of
29.62% over the previous SoTA on this task. Our code is available at
https://github.com/oripress/EntropyEnigma
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要