Multimodal Communication

Our research is aimed at theory- and method-building for multimodal communication based on fine-grained qualitative and quantitative analyses of diverse multimodal artefacts as well as exploratory and experimental studies with readers and users. Our work is inspired by and made useful mainly for multimodal communication in professional contexts, but we also have experience in analyzing narrative and entertainment artefacts such as films, comics, and video games.

The main focus of our current work is on first-aid instructions. In collaboration with the Oranje Kruis and the Koninklijke Nederlandse Vereniging EHBO afdeling Groningen, we have analyzed a large corpus of multimodal instruction documents and videos in the PAT project and tested the effects of design features such as layout and the use of arrows on comprehension, recall and task execution. We apply our analysis and evaluation methods also to other media used in the first-aid instruction domain such as mobile applications and video. More recently, we have started to collect a corpus of first-aid comics, which we are analyzing for layout and functional features. The effects of certain (combinations of) design features on user appreciation and usability will again be investigated in experimental studies. Future work is aimed at analyzing and comparing the multimodal complexity of diverse first-aid instructions in the form of videos, comics, serious games, etc.

We regularly organize international events such as the Bremen-Groningen Online Workshops on Multimodality (#BreGroMM) and the AREA workshop series on annotation, recognition and evaluation of actions.