Communication-Efficient Federated Learning via Regularized Sparse Random Networks
IEEE Communications Letters(2023)
摘要
This work presents a new method for enhancing communication efficiency in
stochastic Federated Learning that trains over-parameterized random networks.
In this setting, a binary mask is optimized instead of the model weights, which
are kept fixed. The mask characterizes a sparse sub-network that is able to
generalize as good as a smaller target network. Importantly, sparse binary
masks are exchanged rather than the floating point weights in traditional
federated learning, reducing communication cost to at most 1 bit per parameter
(Bpp). We show that previous state of the art stochastic methods fail to find
sparse networks that can reduce the communication and storage overhead using
consistent loss objectives. To address this, we propose adding a regularization
term to local objectives that acts as a proxy of the transmitted masks entropy,
therefore encouraging sparser solutions by eliminating redundant features
across sub-networks. Extensive empirical experiments demonstrate significant
improvements in communication and memory efficiency of up to five magnitudes
compared to the literature, with minimal performance degradation in validation
accuracy in some instances
更多查看译文
关键词
Federated Learning,Sparse random Networks,Unstructured Sparsity
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要