Meaning-making by AI: ‘It remains a grey box’

Digital communication is not just about text. Non-verbal expressions also play a major role. Janina Wildfeuer, Associate Professor at the Faculty of Arts, researches this ‘multimodality’ of digital communication. In the third episode of our series on the JTS Scholars, she explains how knowledge of this field can, among other things, help in detecting online fake news.

Text: Jelle Posthuma

Wildfeuer studies multimodality: the interaction between verbal and non-verbal forms of expression, such as images, videos, gestures, and facial expressions. The Groningen researcher focuses mainly on digital communication. ‘I look, for example, at social media, where pictures, gifs, and animations play an important role.’ Wildfeuer refers to these expressions as ‘artifacts’. ‘My interest lies in fundamental questions about how meaning is created through such artifacts: how is meaning constructed when language is not the only factor – or when it’s completely missing?’

‘Grey box’

Artificial intelligence (AI) plays an increasingly important role in this research. The Associate Professor investigates how AI systems deal with multimodality. ‘Language models such as ChatGPT primarily work with written text. But we can also ask the system to interpret images or even generate non-verbal artifacts.’ Yet, much remains unclear about how these interpretations and AI-generated images come about. ‘Multimodality research is still relatively young. Compared to language, there is very little annotated data available on how images function, that is, how meaning is constructed in and by them.’

Annotating data on non-verbal artifacts is a complex and time-consuming process, Wildfeuer emphasizes. Because of this limited knowledge, we do not yet fully understand how AI systems assign meaning. ‘Images can mean many different things; the range of interpretation is much broader than with verbal data – and not all these interpretations are yet coded. Moreover, context and perspective play a crucial role.’ For this reason, AI systems remain, for now, a kind of ‘grey box’, according to the researcher. ‘We don’t know exactly how the interpretation is formed and whether it is the right one in the particular situation. This can lead to undesirable effects, such as discrimination or bias.’

To expand fundamental knowledge about multimodality in AI systems, Wildfeuer wants to collaborate with researchers from other faculties. The Scholarship at the Jantina Tammes School offers an excellent opportunity to do so. ‘With our research group, we are firmly rooted in the Faculty of Arts. But we want to broaden our perspective and connect with colleagues from other disciplines. Within the UG, for instance, there is still little collaboration with researchers working on generating or interpreting images with AI.’

Fake news detection

In addition to fundamental insights that Wildfeuer’s research group mainly aims for, there is also always an applied perspective. A concrete example is the detection of fake news, Wildfeuer explains. She refers to a fake news incident in Chinese media: ‘A few years ago, images circulated online of a large fire in an imposing building in France. The story was that the national library in Marseille had been set on fire by rioters. But that turned out not to be true at all: the images showed a building on the other side of the world. Nevertheless, several major news sites in China picked up the story, thereby contributing to the spread of fake news.’

Together with her colleagues, Wildfeuer is working on a project on fake news on TikTok or Douyin, the Chinese version of TikTok. With a large collection of videos spreading fake news, the team wants to explore whether it is possible to develop an AI tool that can automatically detect fake news. ‘To do that, it is crucial to understand exactly how (moving) images work: which inserts are used, which voice-overs are applied, and whether, for example, certain sound effects play a role. This interplay determines the meaning of an artifact. We are currently analyzing the videos manually and using our results to train an AI tool, so that it can eventually recognize and interpret these elements independently.’

The Associate Professor also wants to involve people outside the university in the project. ‘To understand how these videos are received, we must pay attention to the user. So far, little research has been done on this aspect. We want to address this by organizing workshops and conducting short surveys. This way, we can connect user experiences with our analytical results and assess the extent to which our interpretations correspond with what viewers actually perceive.’

Humanity

Wildfeuer stresses that meaning-making ultimately remains a uniquely human trait. ‘AI is developing at lightning speed. But assigning meaning is something specific to us as humans. A tool consults a database, while we are capable of creating new interpretations. Ultimately, AI systems are based on existing data: in a sense, it is a continuous copy and paste process. That is a very limited form of meaning-making. Humans can probe further, make connections, and be creative.’

Making connections and exploring meaning-making is also the aim of the 12^th International Conference on Multimodality (ICOM) that will come to Groningen this year. Wildfeuer and her group will host the bi-annual event from October 29 to 31 at the House of Connections in Groningen. The conference will bring together speakers from all over the world, ‘some of the big names and leading research groups in multimodality research will join us for exciting talks, workshops, and pitches on the most recent work done in the field’, Wildfeuer highlights. More information is available on the ICOM website. The conference is open for everyone, registration is possible until 15 October.