Colloquium Computer Science - Dr. Benjamin Zhao, University of Macquarie

Title: Security and Privacy Attacks against Machine Learning

Abstract:

We shall present high-level overviews of 3 attacks against machine learning systems. We start with the presentation of a low skill attack exploiting models applied directly in making security decisions. Specifically, we inspect the setting of Biometric Authentication that under standard performance metrics should be secure but are vulnerable to random inputs. Our evaluation of this attack reveals that simply tuning the threshold is incapable of rejecting these random inputs and propose a training data augmentation defense to mitigate such attackers.

Next, we shall discuss the tradeoffs of providing utility and resistance to real world privacy attacks of membership and attribute inference when leveraging Differentially Private Machine Learning techniques. There exists multiple opportunities within the machine learning process in which to introduce differentially private noise, while providing the same guaranteed protections. Specifically, before the data is seen by the machine learning model, during the learning of the model parameters, and directly to model parameters after training. We empirically measure this tradeoff in multiple membership inference and attribute inference attacks, across multiple privacy budgets.

Finally, we shall present our latest work on Unintended Memorization and Timing Attacks against Named Entity Recognition Models. Named entity recognition models often find employ in sensitive applications such as text redaction. We demonstrate that machine learning approaches to recognizing entities for redaction may unintentionally leak sensitive information it was trained to redact. This is achieved with black-box privacy attacks capitalizing on unintended memorization within the model, and a novel timing side channel to determine if a given redacted text was contained within the training dataset. Our experiments include the redaction of both password and health data, presenting both security risks and a privacy issues, further exacerbated by results indicating memorization after only a single phrase.