How someone with bad intentions could employ AI

‘I sometimes wonder what world we’re leaving to our children,’ says Fatih Turkmen, researcher of Privacy and Security with and of AI at the University of Groningen. Current AI developments are going very fast, so fast that various countries are calling for better regulations and possibly an international institution that oversees things. Turkmen’s research is, amongst other things, about making chatbots more privacy-friendly, secure against malicious attacks and about how we could use AI safely for managing important and perhaps critical infrastructures.

FSE Science Newsroom | Charlotte Vlek 

Regarding AI, there are many things to worry about. The enormous use of energy and water, built-in biases of AI and the AI bubble... why should we worry about privacy?

‘Large language models such as ChatGPT or Claude are much like a sponge. They absorb everything, and when you squeeze them in the right way, they spit everything out. Someone with bad intentions could get all kinds of sensitive information out of such a model, when using the right prompts for instance. You can compare it to the autocomplete function on your laptop or phone, that sometimes suddenly adds your credit card number at the end of a sentence. But now it’s available to the whole internet.’

But an AI always needs information to train on. Is it even possible to train in a privacy-friendly way?

‘In our research we are developing ways to make sure sensitive information cannot be extracted, and we also think about encrypting information before it goes into an AI. And there are techniques that make sure that sensitive training data cannot simply be spit out by an AI in a chat conversation. In our research, we are working, for instance, on various so-called privacy enhancing techniques (PET).’

You currently have a research project on privacy and security of chatbots. How does such research work?

‘We run experiments, and we devise our own attacks to try to elicit sensitive information from a bot or manipulate its behaviour. That may sound strange, but if we don’t do it, someone else might do it with malicious intentions. Our goal is to be ahead of them, identify the weak spots in a system and fix them before they are exploited.’

‘One example is memorization. AI models are trained on enormous amounts of data, which may include sensitive or personal information. We might expect the model to learn general patterns from this data rather than remember individual examples. But the reality is that AI models can sometimes memorize specific parts of their training data. This could potentially expose information that was never intended to be revealed. We investigate whether and under what conditions such memorized information can be extracted from a model and how we can quantify the memorized content.’

Your research is not just about privacy and security of AI, but also about how we can use AI safely, right?

‘Yes. Because there are good ways to use AI while keeping your privacy. For instance, by keeping the data where it is generated without any sharing, and distributing AI computations to multiple servers. That way, the information cannot be ‘stolen’ from one central place. And besides that, we want to help companies who want to adopt the current developments in AI, by showing them how to do it in a secure way.’

Like what companies?

‘Consider infrastructure like electricity, water, telecommunication... these all rely on networks that are maintained by one company, but it’s often another company that distributes to consumers. These companies need to exchange information for their processes, or share cyber threat intelligence. How can they do this securely, while preserving customer’s privacy? And for some companies it will be tempting to use AI for locating malfunctions in their network. A power plant could have sensors, for instance, to detect when there is a fire, or another physical problem.’

‘But when a sensor indicates a problem, you need to determine whether there is a real physical problem, or perhaps a cyberattack. That may sound paranoid, but it’s the reality! You don’t want a blackout in the whole country just because a sensor went off for no reason. That’s why it’s very important to have a secure AI-system but also human oversight in the loop. We want to help companies by developing an AI-system that is specific to their application, and local. If companies would start using AI software from OpenAI or Anthropic, they would essentially leave the security of our infrastructure up for grabs.’ 

Are you worried?

‘I do worry about the fact that only a few large companies now have all the power. This power should not be with company, but rather with a country. Or even better: with an independent institution. And it makes total sense to decentralize: there are ways to train AI in multiple layers or make predictions collaboratively, in which everyone involved only manages their own layer of the larger thing. That way, you can make sure that no single person or institution has all the power.’

Read more: