Pairwise Margin Maximization for Deep Neural Networks

The weight decay regularization term is widely used during training to constrain expressivity, avoid overfitting, and improve generalization. Historically, this concept was borrowed from the SVM maximum margin principle and extended to multiclass deep networks. Carefully inspecting this principle reveals that it is not optimal for multi-class classification in general, and in particular when using...