Self-learning for Annotating Website Privacy Policies at Scale

With the increasing importance of user data privacy, it is crucial for individuals to understand how companies handle their information. While considerable research has been conducted on automatically identifying privacy-related information in policies, the lack of high-quality annotated training data in this domain remains a significant challenge. Manual annotation of privacy policies is a demanding and time-consuming task that requires domain knowledge. To address this issue, we propose a semi-supervised method, specifically an iterative self-learning approach, to augment the limited training dataset and improve classification performance. Our approach leverages two state-of-the-art models, BERT and XLNet, and involves automatic labelling of data and model retraining with pseudo-labels. We evaluated our approach on the OPP-115 corpora and observed a 10% improvement in the macro F-1 score for BERT, demonstrating the effectiveness of self-learning. This is the first attempt to automatically annotate privacy policies using a self-learning method without requiring additional annotations, offering a promising solution to the challenge of training data scarcity in this domain.