To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
arxiv(2024)
摘要
The temperature parameter plays a profound role during training and/or
inference with large foundation models (LFMs) such as large language models
(LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax
function in LLMs, which is crucial for next token generation, and it scales the
similarities in the contrastive loss for training CLIP models. A significant
question remains: Is it viable to learn a neural network to predict a
personalized temperature of any input data for enhancing LFMs"? In this paper,
we present a principled framework for learning a small yet generalizable
temperature prediction network (TempNet) to improve LFMs. Our solution is
composed of a novel learning framework with a robust loss underpinned by
constrained distributionally robust optimization (DRO), and a properly designed
TempNet with theoretical inspiration. TempNet can be trained together with a
large foundation model from scratch or learned separately given a pretrained
foundation model. It is not only useful for predicting personalized temperature
to promote the training of LFMs but also generalizable and transferable to new
tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly
improves the performance of existing solutions or models, e.g. Table 1. The
code to reproduce the experimental results in this paper can be found at
https://github.com/zhqiu/TempNet.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要