The Pictures and Text (PAT) Workbench: Toward Automated Evaluation of Multimodal First-Aid Instructions
Project members: Ielka van der Sluis (PI), Gisela Redeker, Jonas Bulthuis, Herbert Kruitbosch, Annet Onnes, Renate Vergeer
Funded by: the Groningen Centre for Digital Humanities in collaboration with the Research and Innovation Support (RIS) department of the Center for Information Technology (CIT)
The PAT Workbench is part of the PAT project in which the use, effects and optimisation of instructions that contain pictures and text is investigated. The workbench will support novel and methodical investigations of multimodal instructions (MIs) for first-aid procedures like operating an AED, producing an arm sling, removing ticks, and so forth. The huge variation in pictorial and verbal means employed in these instructions demonstrates the urgency to obtain validated guidelines based on empirical evidence collected from readers and users.
- an online application in which multimodal documents can be annotated in terms of the text and pictures as well as the relations between them,
- a multimodal annotation model to annotate multimodal documents, specially first-aid instructions,
- an annotated corpus of multimodal first-aid instructions.
- taggers to automatically annotate linguistic, pictorial, and layout features,
- algorithm and interface for the automatic evaluation of MIs based on crowdsourced reader assessments of MIs,
- algorithms to automatically generate effective text-picture combinations,
- authoring guidelines to produce good quality instructions.
|Last modified:||30 October 2018 2.23 p.m.|