Francesco Possemato is fascinated by the ‘messiness’ of interactions

Communication devices are indispensable for people with speech disorders. However, current technology often falls short during interactions. Artificial intelligence may offer a solution, says Francesco Possemato, a researcher at the University of Groningen and a JTS Scholar. In this episode of our series about the JTS Scholars, he talks about his research and the development of new communication tools.

Text: Jelle Posthuma

In his work, Possemato focuses on the ‘messiness’ of conversations. This contrasts sharply with the ‘terrible’ grammars he read whilst at university, which centred around correct language usage. ‘These books prescribe how things should be done. Whereas in our daily lives, it’s all about messy interactions. A lot goes wrong, but we often manage to solve problems flexibly as the interaction unfolds. It is precisely that messiness and the process of problem-solving that I find fascinating.’

His field of expertise is conversation analysis. In practice, this means he spends many hours recording conversations, which he then transcribes and analyses. Possemato also incorporates non-verbal communication, such as gestures and posture, into his analyses. ‘It’s actually a kind of forensic investigation. I look in particular at problems in interactions. How do people identify problems in a conversation and how do they resolve them?’

What makes us human?

In his research he also looks for universal aspects of human language. He is currently involved in an Australian Research Council funded project focusing on a wide variety of languages, ranging from Japanese to Australian Aboriginal languages According to the researcher, humans’ problem-solving ability is one such universal trait. ‘There are other primates that solve ‘problems’ during interactions, such as a baby bonobo that uses gestures to attract its mother’s attention, but humans have the unique ability to reflect on and fix their own language.’

It makes human language adaptive, he continues. ‘For example, we can say: “I didn’t mean Robert, but James.” This is how interactions are repaired. There are, of course, major cultural differences between languages, but if we strip language of all their specific grammatical structures, we find that there are actually strong similarities. The way in which we organize, understand and repair interactions is remarkably consistent. These mechanisms are universal,’ emphasises Possemato. ‘And that is what makes us human.’

Support for speech disorders

A striking commonality in people’s interactions is the brief pause they leave between turns in a conversation. ‘In virtually every language, there is about half a second of silence between turns.’ People who need to use tools due to a speech disorder often leave much longer pauses, because, for example, they have to select letters or images in order to communicate. Possemato analyses the problems caused by these communication systems during interactions.

When using an augmented communicationtool, the pauses between turns can range from 20 seconds to several minutes, he continues. A conversation partner may start to fill in the gaps, which disrupts the sequential nature of an interaction. ‘This affects the understanding and common ground of an interaction; it becomes difficult to understand each other.’ And the consequences are far-reaching: ‘In the most extreme cases, people are excluded from fundamental social activities, from everyday interaction to access to basic services, for example education.’

In his research, Possemato focuses on people with dysarthria and ALS. The use of AI can help to develop new tools that better capture the essence of human interactions for these groups. Together with Prof. Jeff Higginbotham (SUNY at Buffalo) and his lab, Possemato is working on new technology that uses AI. ‘I’m not an engineer, of course, but I can inform Augmentative and Alternative Communication (AAC) systems based on conversation analytic research. Engineers, for their part, focus more on the design of the device.’

The use of AI can reduce delays and personalise the options for the speaker, he continues. This is of great importance, as it concerns a person’s identity. Possemato points to the large language models that have taken off enormously since 2023. ‘We’ve been able to make great strides in the field of prosody – think of rhythm and intonation.’ Just before the introduction of the well-known AI models, researchers mostly used actors, who, for example, had to pronounce the word “okay” in forty different ways. ‘That takes a lot of time and money, whereas we can now tweak it in five seconds using AI.’

Interdisciplinary

Possemato argues that collaboration between engineers, computer scientists, and linguists is crucial when developing new communication aids. ‘Linguists are trained to recognise the subtle elements of language and interaction. For users, this may be far more important than, for example, grammatical correctness or the number of words a device has at its disposal. Above all, they want to be able to convey their message effectively.’ In turn, Possemato learns a great deal from the engineers. ‘In our field, we’re good at describing things, whereas engineers mainly ask themselves how to solve a problem. I’ve learnt this problem-solving approach, and that’s what makes this kind of collaboration so exciting.’

The interdisciplinary approach and the theme of AI fit seamlessly with the Jantina Tammes School’s Scholarship. Possemato hopes that, through his Scholarship, he can further expand the links between disciplines. ‘I’m not yet part of a network in Groningen. In particular, I hope to build up a local network in the North in the field of AI.’

According to the researcher, the time is now to capitalize on the potential of AI for new communication tools, whilst taking ethical values into account. ‘If we don’t do it, Big Tech will run with it.’ This would have consequences for the end user, he explains. Large companies are unlikely to adopt an open-source approach. ‘We do, though, which means the technology remains accessible for everyone.’