Impact | Automatic recognition for whispered speech

In the coming weeks the nominees for the Ben Feringa Impact Award 2025 will introduce themselves and their impactful research or project. The winners will be announced on 13 May. This week: Igor Marchenko with the research on improving automatic recognition for whispered speech.

Who are you?

I am Igor Marchenko, originally from Kaliningrad - the westernmost city of Russia. I hold a master's in Speech Technology and am now a PhD researcher at Campus Fryslân working on Frisian speech technologies as part of the "Melodies of the North" project.

Growing up in a linguistically unique region sparked my interest in how language shapes identity. This led me to the Netherlands' multilingual setting, and particularly to Friesland, where I combine technical interest with meaningful language preservation work. My research on whispered speech showed me how technology can adapt to linguistic diversity - rather than requiring speakers to conform.

Can you explain what your research was about?

My research explored how we can improve automatic recognition for whispered speech - something that current systems generally don't handle well. Whispering changes the way speech sounds quite dramatically, and since most systems are only trained on normal speech, they often fail when someone whispers. I wanted to find a way to make these systems more flexible, without needing to collect large amounts of new whispered recordings. To do this, I designed a training method that helps the system get used to the kinds of differences whispering introduces.

By carefully teaching the system which parts of speech are most affected by whispering, the system learns to rely more on the stable cues that remain - and to predict what's missing. This makes it better at understanding whispered speech, even with very little data.

What made the research impactful?

This research contributes to the growing understanding that speech technologies need to accommodate a wider variety of users and communication situations. Whispered speech is more than a curiosity - it plays a crucial role in environments where privacy, health, or social factors restrict vocal expression. People recovering from voice loss or living with speech impairments may depend on whispering as their main mode of communication, yet existing systems often struggle to process it effectively.

My work shows that with more targeted training of models, we can support these users even with limited data. This approach also holds potential for other low-resource contexts, such as regional languages or unconventional speech patterns. By questioning the assumption that "standard speech" is the default, my research pushes for more human-centric technology.

What was your personal motivation to conduct this research?

I've always been interested in speech in all of its variations, be it dialects, accents, or different ways of speaking like whispering. Whispered speech stood out to me as an interesting challenge: it highlights how much we take certain vocal cues for granted, and how quickly systems fall apart when those cues disappear. Working on this topic taught me how valuable a linguistic perspective can be in technology, especially when dealing with non-standard input. It reinforced the idea that instead of expecting people to adapt their speech to be better understood by machines, we should focus on improving technology to better recognize and process the natural diversity of human voices. This principle continues to guide my research on Frisian today.