Publication

Action Categorisation in Multimodal Instructions

van der Sluis, I., Vergeer, R. & Redeker, G., 7-May-2018, Proceedings of the 11th edition of the Language Resources and Evaluation Conference (LREC 2018): Proceedings of the 1st International Workshop on Annotation, Recognition and Evaluation of Actions (AREA 2018). Pustejovsky, J. & van der Sluis, I. (eds.). Miyazaki, p. 31-36 6 p.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

We present an explorative study for the (semi-)automatic categorisation of actions in Dutch multimodal first aid instructions,
where the actions needed to successfully execute the procedure in question are presented verbally and in pictures. We start
with the categorisation of verbalised actions and expect that this will later facilitate the identification of those actions in the
pictures, which is known to be hard. Comparisons of and user-based experimentation with the verbal and visual representations
will allow us to determine the effectiveness of picture-text combinations and will eventually support the automatic generation
of multimodal documents. We used Natural Language Processing tools to identify and categorise 2,388 verbs in a corpus of 78
multimodal instructions. We show that the main action structure of an instruction can be retrieved through verb identification
using the Alpino parser followed by a manual selection operation. The selected main action verbs were subsequently
generalised and categorised with the use of Cornetto, a lexical resource that combines a Dutch Wordnet and a Dutch Reference
Lexicon. Results show that these tools are useful but also have limitations which make human intervention essential to guide
an accurate categorisation of actions in multimodal instructions.
Original languageEnglish
Title of host publicationProceedings of the 11th edition of the Language Resources and Evaluation Conference (LREC 2018): Proceedings of the 1st International Workshop on Annotation, Recognition and Evaluation of Actions (AREA 2018)
EditorsJames Pustejovsky, Ielka van der Sluis
Place of PublicationMiyazaki
Pages31-36
Number of pages6
Publication statusPublished - 7-May-2018
EventAREA - Annotation, Recognition and Evaluation of Actions: in conjunction with the 11th edition of the Language Resources and Evaluation Conference (LREC 2018) - Phoenix Seagaia Resort, Miyazaki, Japan
Duration: 7-May-2018 → …
http://www.areaworkshop.org/

Workshop

WorkshopAREA - Annotation, Recognition and Evaluation of Actions
Abbreviated titleAREA
CountryJapan
CityMiyazaki
Period07/05/2018 → …
Internet address

Event

AREA - Annotation, Recognition and Evaluation of Actions: in conjunction with the 11th edition of the Language Resources and Evaluation Conference (LREC 2018)

07/05/2018 → …

Miyazaki, Japan

Event: Workshop

    Keywords

  • instructions, actions, verbs, categorisation, task structure

Download statistics

No data available

ID: 65103448