How AI can help people with language impairments find their speech

Ordering a coffee at that nice new corner shop: most of us do this without thinking about it. For people with language impairments (such as aphasia), however, it can pose a serious challenge. What if AI-driven models could help these people – for example, with a smartphone app? Computational linguistics researcher Frank Tsiwah is exploring various ways to make this happen. ‘If it can make a single life better, then it will be worth it.’

_{Text: Thomas Vos, Corporate Communication UG / Photos: Henk Veenstra}

The rise of AI-driven large language models (such as ChatGPT) triggered something in Tsiwah. ‘I asked myself: why can’t we exploit the architecture of large language models to help people with conditions like aphasia? At their core, these models are word-predicting models. That is exactly what patients with aphasia struggle with: finding the right words in a certain context –such as when ordering a coffee at a cafe,’ Tsiwah explains.

Background

Tsiwah, a computational linguistics researcher in the Faculty of Arts, completed a Bachelor’s degree in linguistics in Ghana, where he is originally from. During his studies, he encountered an NGO focusing on children with speech and language impairments, and he signed up as a volunteer. ‘That’s when I started getting interested in the clinical aspect of linguistics.’ A Master’s degree in clinical linguistics in Groningen followed, and when a PhD position opened up in the Faculty of Arts, he knew it was right for him.

Aphasia

In his research thus far, Tsiwah has devoted a lot of attention to aphasia. ‘The classic definition of aphasia is that it is a language impairment which results from a focal brain injury, most commonly caused by a stroke. Thirty per cent of people experiencing a stroke will get aphasia. In the Netherlands alone, there are 10,000 new cases of aphasia each year. With aphasia, you experience a breakdown of your ability to use language. One of the main symptoms is that you have difficulty finding the right words, in a way that can also be observed in people with dementia or similar conditions.’

Fluent and non-fluent

Aphasia can be broadly categorized as either fluent or non-fluent. Tsiwah continues: ‘With fluent aphasia, particularly the Wernicke type, you speak fluently, but what you’re saying doesn’t make much sense. For example, a person with fluent aphasia might start talking about their vacation if you ask how their morning was. Furthermore, they make up words that don’t exist, and they try to circumvent not having to say the words they are struggling to find. In contrast, patients with the non-fluent type often have considerable comprehension, but their sentence production is very fragmented. Instead of saying ‘I want a cup of coffee,’ they will say something like ‘Want cup coffee.’

Pipeline for a tool

In his current research, for which he received an NWO Veni Grant in 2025, Tsiwah is trying to build an AI-driven large language model to help patients with aphasia. ‘Right now, regular treatment consists of helping patients find the words they’re looking for, restricted to a specific context, such as picture naming. But how does this translate to day-to-day conversation, where speech is more fluent and spontaneous? I want to create the pipeline for a tool – such as an app on your mobile phone – that can help with normal conversation within real-life contexts. This tool must be able to recognize and understand speech, transcribe it and, if there is a significantly long pause in the conversation, predict possible words that could follow and suggest those through an earpiece or as text on your smartphone, for example. If such a tool is created, it will obviously be important for the data to be stored locally and for there to be an option for users to turn it on and off in certain contexts.’

Large language models

Is it possible to fine-tune an AI-driven large language model that can predict missing words? Tsiwah believes it is, but it will not be easy. ‘The core of large language models is word prediction. They can do this with hundreds of thousands of word combinations. However, current models such as ChatGPT are trained with fluent language from books, the internet, and such. As a result, they break down when they encounter fragmented speech. My idea is to go all the way back and teach a large language model to understand fragmented, aphasic language. This could teach it how best to predict next words for a person with aphasia.’

Pilot projects

Tsiwah has already run promising pilot projects using open-source large language models that had already been trained with billions of texts. He then fine-tuned those models to understand aphasic language and make it specific to a task at hand. ‘A sentence such as “I want a cup of coffee” has been said millions of times. If a patient with aphasia says, “I want a cup…,” the model could predict the word “coffee.” But it is obviously important for the model to know the context within which this sentence is said.’

Helping to train language systems

Tsiwah ‘s goal with this research is clear. ‘For the Veni grant, I am not promising to deliver a finished product. That will be the next step. Right now, I want to do the research and create the pipeline for such a tool. It also shouldn’t become a tool that will always predict everything for patients. It is meant to help them train their language systems in the early stages of their condition.’

More directions

Fine-tuning large language models is not the only direction Tsiwah is exploring. He is also in the development phase of using electromyography (EMG), which measures muscle activity by using sensors. ‘In recent years, there has been more interest in the potential applications of this technology for people with speech impairments – for example, for people who have lost their voice box (their larynx) due to cancer. What if we put sensors on their speech-related muscles that are still intact, and while they are speaking – with no voicing – we can use a model already trained to recognize words from muscle information to figure out what this person is trying to say. This could then be turned into an audible voice through a speaker.’

Converging technologies

Tsiwah hopes that these different research paths will converge in the future. He continues, ‘I don’t yet know exactly how these lines of work will come together, but I can imagine the possibility of combining an AI model that uses context to suggest next words with another model that reads speech‑muscle activity to decode what people intended to say without voicing it. Together, this will help give a voice back to help people with speech and language impairments caused by either brain damage or laryngeal cancer. That is my vision for the future. It’s hard work, but the people I’m serving keep me going. If it can make a single life better, it will be worth it.’

More information

Frank Tsiwah