TY - GEN
T1 - Unrolling SGD
T2 - 7th IEEE European Symposium on Security and Privacy, Euro S and P 2022
AU - Thudi, Anvith
AU - Deza, Gabriel
AU - Chandrasekaran, Varun
AU - Papernot, Nicolas
N1 - Funding Information:
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: CIFAR through the Canada CIFAR AI Chair program, DARPA through the GARD program, Intel, Meta, NFRF through an Exploration grant, and NSERC through the Discovery Grant and COHESA Strategic Alliance. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Machine unlearning is the process through which a deployed machine learning model is made to forget about some of its training data points. While naively retraining the model from scratch is an option, it is almost always associated with large computational overheads for deep learning models. Thus, several approaches to approximately unlearn have been proposed along with corresponding metrics that formalize what it means for a model to forget about a data point. In this work, we first taxonomize approaches and metrics of approximate unlearning. As a result, we identify verification error, i.e., the \ell_2 difference between the weights of an approximately unlearned and a naively retrained model, as an approximate unlearning metric that should be optimized for as it subsumes a large class of other metrics. We theoretically analyze the canonical training algorithm, stochastic gradient descent (SGD), to surface the variables which are relevant to reducing the verification error of approximate unlearning for SGD. From this analysis, we first derive an easy-to-compute proxy for verification error (termed unlearning error). The analysis also informs the design of a new training objective penalty that limits the overall change in weights during SGD and as a result facilitates approximate unlearning with lower verification error. We validate our theoretical work through an empirical evaluation on learning with CIFAR-10, CIFAR-100, and IMDB sentiment analysis.
AB - Machine unlearning is the process through which a deployed machine learning model is made to forget about some of its training data points. While naively retraining the model from scratch is an option, it is almost always associated with large computational overheads for deep learning models. Thus, several approaches to approximately unlearn have been proposed along with corresponding metrics that formalize what it means for a model to forget about a data point. In this work, we first taxonomize approaches and metrics of approximate unlearning. As a result, we identify verification error, i.e., the \ell_2 difference between the weights of an approximately unlearned and a naively retrained model, as an approximate unlearning metric that should be optimized for as it subsumes a large class of other metrics. We theoretically analyze the canonical training algorithm, stochastic gradient descent (SGD), to surface the variables which are relevant to reducing the verification error of approximate unlearning for SGD. From this analysis, we first derive an easy-to-compute proxy for verification error (termed unlearning error). The analysis also informs the design of a new training objective penalty that limits the overall change in weights during SGD and as a result facilitates approximate unlearning with lower verification error. We validate our theoretical work through an empirical evaluation on learning with CIFAR-10, CIFAR-100, and IMDB sentiment analysis.
UR - http://www.scopus.com/inward/record.url?scp=85133317187&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85133317187&partnerID=8YFLogxK
U2 - 10.1109/EuroSP53844.2022.00027
DO - 10.1109/EuroSP53844.2022.00027
M3 - Conference contribution
AN - SCOPUS:85133317187
T3 - Proceedings - 7th IEEE European Symposium on Security and Privacy, Euro S and P 2022
SP - 303
EP - 319
BT - Proceedings - 7th IEEE European Symposium on Security and Privacy, Euro S and P 2022
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 6 June 2022 through 10 June 2022
ER -