Toward pragmatic inference in machines: relation-aware multimodal modeling of sarcasm in conversation
Human communication often means more than words alone. A sentence such as "Thank you for being on time” can express gratitude, but with a flat tone or an eye roll, it means the opposite. People understand such hidden meanings by combining words with tone of voice, facial expression, body language, and social context. Current language technologies, however, still rely heavily on words and often struggle when meaning depends on how something is said.
This dissertation investigates how machines can better understand intended meaning in spoken interaction. It uses sarcasm as a test case because linguistic research shows that sarcasm often depends on how words, voice, and facial expressions work together. This allows the models to be guided by how people communicate, rather than by trial and error alone.
The findings show that machines better understand sarcasm when they capture how signals work together, such as the mismatch between positive words like “Great job” and a flat, negative-sounding delivery. In contrast to today’s data-driven trend, this dissertation shows that more data helps when it preserves the clues people use to infer intended meaning. It further introduces a Mandarin dataset, showing that systems trained on English need adjustment before they can reliably interpret meaning in other languages.
The goal is to support reliable, accessible language technology that can manage uncertainty and reduce misunderstandings between people and machines. This research contributes to systems that process not only what people say, but also what they mean.