Publication

Text-Picture Relations in Multimodal Instructions

van der Sluis, I., Eppinga, A. N. & Redeker, G., 19-Sep-2017, Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication. Montpellier, France., 6 p.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,
recall and the instruction’s attractiveness. Results show that users
first read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.
Original languageEnglish
Title of host publicationProceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication
Place of PublicationMontpellier, France.
Number of pages6
Publication statusPublished - 19-Sep-2017
EventFMSC workshop on Foundations of Situated or Multimodal Communication - Montpellier , Montpellier , France
Duration: 19-Sep-201719-Sep-2018

Workshop

WorkshopFMSC workshop on Foundations of Situated or Multimodal Communication
CountryFrance
CityMontpellier
Period19/09/201719/09/2018

Event

FMSC workshop on Foundations of Situated or Multimodal Communication

19/09/201719/09/2018

Montpellier , France

Event: Workshop

    Keywords

  • multimodal, instructions, text-picture relations

Download statistics

No data available

ID: 73838891