Hot PATE: Private Aggregation of Distributions for Diverse Task
arxiv(2023)
摘要
The Private Aggregation of Teacher Ensembles (PATE) framework is a versatile
approach to privacy-preserving machine learning. In PATE, teacher models that
are not privacy-preserving are trained on distinct portions of sensitive data.
Privacy-preserving knowledge transfer to a student model is then facilitated by
privately aggregating teachers' predictions on new examples. Employing PATE
with generative auto-regressive models presents both challenges and
opportunities. These models excel in open ended diverse (aka hot) tasks
with multiple valid responses. Moreover, the knowledge of models is often
encapsulated in the response distribution itself and preserving this diversity
is critical for fluid and effective knowledge transfer from teachers to
student. In all prior designs, higher diversity resulted in lower teacher
agreement and thus – a tradeoff between diversity and privacy. Prior works
with PATE thus focused on non-diverse settings or limiting diversity to improve
utility.
We propose hot PATE, a design tailored for the diverse setting. In hot
PATE, each teacher model produces a response distribution that can be highly
diverse. We mathematically model the notion of preserving diversity and
propose an aggregation method, coordinated ensembles, that preserves
privacy and transfers diversity with no penalty to privacy or
efficiency. We demonstrate empirically the benefits of hot PATE for in-context
learning via prompts and potential to unleash more of the capabilities of
generative models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要