Projects

The Pictures and Text (PAT) Workbench: Toward Automated Evaluation of Multimodal First-Aid Instructions

Project members: Ielka van der Sluis (PI), Gisela Redeker, Jonas Bulthuis, Herbert Kruitbosch, Annet Onnes, Renate Vergeer

Funded by: the Groningen Centre for Digital Humanities in collaboration with the Research and Innovation Support (RIS) department of the Center for Information Technology (CIT)

The PAT Workbench is part of the PAT project in which the use, effects and optimisation of instructions that contain pictures and text is investigated. The workbench will support novel and methodical investigations of multimodal instructions (MIs) for first-aid procedures like operating an AED, producing an arm sling, removing ticks, and so forth. The huge variation in pictorial and verbal means employed in these instructions demonstrates the urgency to obtain validated guidelines based on empirical evidence collected from readers and users.

The PAT Workbench project delivers:

• an online application in which multimodal documents can be annotated in terms of the text and pictures as well as the relations between them,
• a multimodal annotation model to annotate multimodal documents, specially first-aid instructions,
• an annotated corpus of multimodal first-aid instructions.
  

In the PAT project, these results will later be used to develop:

• taggers to automatically annotate linguistic, pictorial, and layout features,
• algorithm and interface for the automatic evaluation of MIs based on crowdsourced reader assessments of MIs,
• algorithms to automatically generate effective text-picture combinations,
• authoring guidelines to produce good quality instructions.
  

Links:

Center for Digital Humanities Groningen