Margin-based regularization for deep neural networks

Abstract

Deep Neural Networks (DNNs) have achieved significant success in various tasks, yet understanding the factors that enhance their generalization remains a challenge. This study aims to explore whether recent advances in post hoc margin-based generalization prediction measures can be applied to effectively enhance the generalization ability of modern DNNs. Specifically, we explore the use of constrained margins as a regularization technique to improve generalization, focusing on their application within robustness-enhancing methods.

Large margins – the minimum distance between data points and decision boundaries – in the input space are known to correlate with increased robustness. In this study, we focus on two margin-maximizing approaches: a simple approach which modifies the loss function to increase an approximation of the margin (Large margin loss), and a more complex state-of-the-art method (Dynamics-Aware Robust Training) which builds upon this approach.

We first study standard margin maximization methods aimed at improving the robustness of DNNs. Our experiments, conducted on the CIFAR-10 dataset, identify and assess the effectiveness of the components introduced in the large margin loss and DyART in improving robustness. Model performance is evaluated under well-known adversarial attacks, including AutoAttack and Projected Gradient Descent (PGD). We find that while more precise margin approximations do not improve the generalization ability of DNNs, calculating the gradient of the margin with respect to model parameters leads to substantial robustness improvements.

Following this, we adapt constrained margins, a metric which has shown to be more correlated with generalization than standard margins, to both the large margin loss function and DyART. Our aim is to investigate how maximizing these constrained margins impact generalization during training in approaches traditionally designed to enhance robustness using standard margins. We found that maximizing constrained margins during training neither improves generalization performance nor improves adversarial robustness of DNNs.