Publication

Evaluating Scoped Meaning Representations

van Noord, R., Abzianidze, L., Haagsma, H. & Bos, J., 8-May-2018, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Paris: European Language Resources Association (ELRA), p. 1685-1693 9 p.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

APA

van Noord, R., Abzianidze, L., Haagsma, H., & Bos, J. (2018). Evaluating Scoped Meaning Representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (pp. 1685-1693). European Language Resources Association (ELRA).

Author

van Noord, Rik ; Abzianidze, Lasha ; Haagsma, Hessel ; Bos, Johan. / Evaluating Scoped Meaning Representations. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Paris : European Language Resources Association (ELRA), 2018. pp. 1685-1693

Harvard

van Noord, R, Abzianidze, L, Haagsma, H & Bos, J 2018, Evaluating Scoped Meaning Representations. in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Paris, pp. 1685-1693, Eleventh International Conference on Language Resources and Evaluation , Miyazaki , Japan, 07/05/2018.

Standard

Evaluating Scoped Meaning Representations. / van Noord, Rik; Abzianidze, Lasha; Haagsma, Hessel; Bos, Johan.

Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Paris : European Language Resources Association (ELRA), 2018. p. 1685-1693.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Vancouver

van Noord R, Abzianidze L, Haagsma H, Bos J. Evaluating Scoped Meaning Representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Paris: European Language Resources Association (ELRA). 2018. p. 1685-1693


BibTeX

@inproceedings{b5646e39618545d499d8b255568fa35f,
title = "Evaluating Scoped Meaning Representations",
abstract = "Semantic parsing offers many opportunities to improve natural language understanding. We present a semantically annotated parallel corpus for English, German, Italian, and Dutch where sentences are aligned with scoped meaning representations in order to capture the semantics of negation, modals, quantification, and presupposition triggers. The semantic formalism is based on Discourse Representation Theory, but concepts are represented by WordNet synsets and thematic roles by VerbNet relations. Translating scoped meaning representations to sets of clauses enables us to compare them for the purpose of semantic parser evaluation and checking translations. This is done by computing precision and recall on matching clauses, in a similar way as is done for Abstract Meaning Representations. We show that our matching tool for evaluating scoped meaning representations is both accurate and efficient. Applying this matching tool to three baseline semantic parsers yields F-scores between 43% and 54%. A pilot study is performed to automatically find changes in meaning by comparing meaning representations of translations. This comparison turns out to be an additional way of (i) finding annotation mistakes and (ii) finding instances where our semantic analysis needs to be improved.",
keywords = "meaning representations, semantic parsing, discourse representation structures, semantic evaluation",
author = "{van Noord}, Rik and Lasha Abzianidze and Hessel Haagsma and Johan Bos",
year = "2018",
month = may,
day = "8",
language = "English",
isbn = "979-10-95546-00-9",
pages = "1685--1693",
booktitle = "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)",
publisher = "European Language Resources Association (ELRA)",
note = "Eleventh International Conference on Language Resources and Evaluation , LREC 2018 ; Conference date: 07-05-2018 Through 12-05-2018",
url = "http://lrec2018.lrec-conf.org/en/",

}

RIS

TY - GEN

T1 - Evaluating Scoped Meaning Representations

AU - van Noord, Rik

AU - Abzianidze, Lasha

AU - Haagsma, Hessel

AU - Bos, Johan

PY - 2018/5/8

Y1 - 2018/5/8

N2 - Semantic parsing offers many opportunities to improve natural language understanding. We present a semantically annotated parallel corpus for English, German, Italian, and Dutch where sentences are aligned with scoped meaning representations in order to capture the semantics of negation, modals, quantification, and presupposition triggers. The semantic formalism is based on Discourse Representation Theory, but concepts are represented by WordNet synsets and thematic roles by VerbNet relations. Translating scoped meaning representations to sets of clauses enables us to compare them for the purpose of semantic parser evaluation and checking translations. This is done by computing precision and recall on matching clauses, in a similar way as is done for Abstract Meaning Representations. We show that our matching tool for evaluating scoped meaning representations is both accurate and efficient. Applying this matching tool to three baseline semantic parsers yields F-scores between 43% and 54%. A pilot study is performed to automatically find changes in meaning by comparing meaning representations of translations. This comparison turns out to be an additional way of (i) finding annotation mistakes and (ii) finding instances where our semantic analysis needs to be improved.

AB - Semantic parsing offers many opportunities to improve natural language understanding. We present a semantically annotated parallel corpus for English, German, Italian, and Dutch where sentences are aligned with scoped meaning representations in order to capture the semantics of negation, modals, quantification, and presupposition triggers. The semantic formalism is based on Discourse Representation Theory, but concepts are represented by WordNet synsets and thematic roles by VerbNet relations. Translating scoped meaning representations to sets of clauses enables us to compare them for the purpose of semantic parser evaluation and checking translations. This is done by computing precision and recall on matching clauses, in a similar way as is done for Abstract Meaning Representations. We show that our matching tool for evaluating scoped meaning representations is both accurate and efficient. Applying this matching tool to three baseline semantic parsers yields F-scores between 43% and 54%. A pilot study is performed to automatically find changes in meaning by comparing meaning representations of translations. This comparison turns out to be an additional way of (i) finding annotation mistakes and (ii) finding instances where our semantic analysis needs to be improved.

KW - meaning representations

KW - semantic parsing

KW - discourse representation structures

KW - semantic evaluation

M3 - Conference contribution

SN - 979-10-95546-00-9

SP - 1685

EP - 1693

BT - Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

PB - European Language Resources Association (ELRA)

CY - Paris

T2 - Eleventh International Conference on Language Resources and Evaluation

Y2 - 7 May 2018 through 12 May 2018

ER -

ID: 61402262