TY - JOUR
T1 - Adversarially Robust Models may not Transfer Better
T2 - 39th International Conference on Machine Learning, ICML 2022
AU - Xu, Xiaojun
AU - Zhang, Jacky Yibo
AU - Ma, Evelyn
AU - Son, Danny
AU - Koyejo, Oluwasanmi
AU - Li, Bo
N1 - Funding Information:
This work is partially supported by NSF 1910100, NSF 2046795, NSF 1909577, NSF 1934986, NSF CNS 2046726, NIFA award 2020-67021-32799, C3 AI, and the Alfred P. Sloan Foundation.
Publisher Copyright:
Copyright © 2022 by the author(s)
PY - 2022
Y1 - 2022
N2 - Machine learning (ML) robustness and domain generalization are fundamentally correlated: they essentially concern data distribution shifts under adversarial and natural settings, respectively. On one hand, recent studies show that more robust (adversarially trained) models are more generalizable. On the other hand, there is a lack of theoretical understanding of their fundamental connections. In this paper, we explore the relationship between regularization and domain transferability considering different factors such as norm regularization and data augmentations (DA). We propose a general theoretical framework proving that factors involving the model function class regularization are sufficient conditions for relative domain transferability. Our analysis implies that “robustness” is neither necessary nor sufficient for transferability; rather, regularization is a more fundamental perspective for understanding domain transferability. We then discuss popular DA protocols (including adversarial training) and show when they can be viewed as the function class regularization under certain conditions and therefore improve generalization. We conduct extensive experiments to verify our theoretical findings and show several counterexamples where robustness and generalization are negatively correlated on different datasets.
AB - Machine learning (ML) robustness and domain generalization are fundamentally correlated: they essentially concern data distribution shifts under adversarial and natural settings, respectively. On one hand, recent studies show that more robust (adversarially trained) models are more generalizable. On the other hand, there is a lack of theoretical understanding of their fundamental connections. In this paper, we explore the relationship between regularization and domain transferability considering different factors such as norm regularization and data augmentations (DA). We propose a general theoretical framework proving that factors involving the model function class regularization are sufficient conditions for relative domain transferability. Our analysis implies that “robustness” is neither necessary nor sufficient for transferability; rather, regularization is a more fundamental perspective for understanding domain transferability. We then discuss popular DA protocols (including adversarial training) and show when they can be viewed as the function class regularization under certain conditions and therefore improve generalization. We conduct extensive experiments to verify our theoretical findings and show several counterexamples where robustness and generalization are negatively correlated on different datasets.
UR - http://www.scopus.com/inward/record.url?scp=85148572795&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85148572795&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85148572795
SN - 2640-3498
VL - 162
SP - 24770
EP - 24802
JO - Proceedings of Machine Learning Research
JF - Proceedings of Machine Learning Research
Y2 - 17 July 2022 through 23 July 2022
ER -