Emerging Language heroes on Lowlands science

In August 2019 computational linguists Gregory Mills, Martijn Wieling and their team went to the music festival Lowlands to do research. Right next to the main stage they had 180 couples playing a game, similar to guitar hero , as part of an experiment that could tell us a lot about the origins of language. At the same time this research could give computers information on how to solve miscommunication for the future. How is it possible to do serious research in a chaotic and fun environment like Lowlands and what drives a researcher to try? Gregory Mills tells us about his experiences and how his research could have impact.

Between shouted instructions and unexpected results

Mills and Wieling designed a game in which couples have to compose an assigned piece of music together. They both have part of the sounds, but just one of them knows how the combination should turn out; he or she is the composer. The difficulty is that they can't use natural language to communicate, they are only permitted to use two nonsense sounds. Mills knew it could work from a controlled pilot, but was surprised to see it work in full swing: “It was a great experience. After a short – often shouted - instruction, most of the participants thought it was impossible to do this successfully. But the surprising thing was, that they did well! Even slightly intoxicated people, with a lot of loud noise and distraction around them could do this. It shows that we humans have a great capacity to communicate and negotiate meaning in a very effective way.”

Why was Gregory Mills interested in doing this experiment on the Lowlands festival? “My colleague Martijn Wieling did an experiment on how alcohol effects your articulation on Lowlands last year. He was very enthusiastic about this opportunity and the professional way the Lowlands organisation worked. In his research so far he used monologues, but he became interested in how people adapt the way they speak to their conversational partner. My subject of research is turn taking, but so far I mainly studied this in the virtual world of text chats. So I was interested in expanding to real life speech, he was interested in expanding to dialogues. In this design we could both explore new territory.” Normally Mills does these kind of experiments in a lab, where you have a controlled setting and more time to do a full introduction and debriefing. “The Lowlands data are more noisy, so they are harder to work with. But on the other hand, I think they are more natural and we have a better cross section of a young population.”

Towards resolving miscommunication in digital interfaces

The preliminary results of the experiment are promising. “We see that miscommunication triggers. People that figure out how to deal with it, are the most successful in the game. Unexpected maybe is that being in a relationship with your gaming partner does not help. Sometimes these pairs even perform worse, maybe they are not focused on solving miscommunication anymore. Finding out how people communicate without natural language shows us something about the origins of language. Participants who only copy each others sounds are not successful, you have to add something and take complementary roles to make it work.”

Fascinating as it is, what is the use of this knowledge for our modern world? “Well, you are also talking to Siri or Alexa, using WhatsApp or Zoom. So Artificial Intelligence and digital devices are often used as an interface to communicate with other humans, and we would like these conversations to run as smoothly as possible. But there are barriers of course. For example, when we are talking to each other in a video-call, we cannot really establish eye contact. You have to look into a web cam to appear to be looking at your partner, but by doing so, you cannot look into his/her eyes yourself. So we will miss signals, which we try to compensate for. When we know how it works, the interface could help and develop ways to prevent or repair miscommunication. For example in WhatsApp a lot of miscommunication can occur when people are typing simultaneously. The notification “is typing” was added to signal something you cannot see and make you more patient in turn taking.”

Citizen Science for the public good

Resolving miscommunication in language technologies is beneficiary for all of us. But Mills thinks there is more to it. “Participants had great fun playing the game, but thought it was also an eye-opening experience into the nature of language and meaning. So I am planning to make it open for everyone as a game to play and to learn from. I think the more people are scientifically literate, the better. That way we can all have a say in technical developments.”