To test machine comprehension, start by defining comprehension

Jesse Dunietz; Gregory Burnham; Akash Bharadwaj; Owen Rambow; Jennifer Chu-Carroll; David Ferrucci

To test machine comprehension, start by defining comprehension

Jesse Dunietz
, Gregory Burnham
, Akash Bharadwaj
, Owen Rambow
, Jennifer Chu-Carroll
, David Ferrucci

Linguistics

Elemental Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

49 Scopus citations

Abstract

Many tasks aim to measure MACHINE READING COMPREHENSION (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension-a TEMPLATE OF UNDERSTANDING-for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.

Original language	English
Title of host publication	ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	7839-7859
Number of pages	21
ISBN (Electronic)	9781952148255
State	Published - 2020
Event	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States Duration: Jul 5 2020 → Jul 10 2020

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
Country/Territory	United States
City	Virtual, Online
Period	07/5/20 → 07/10/20

Cite this

Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu-Carroll, J., & Ferrucci, D. (2020). To test machine comprehension, start by defining comprehension. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (pp. 7839-7859). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL).

Dunietz, Jesse ; Burnham, Gregory ; Bharadwaj, Akash et al. / To test machine comprehension, start by defining comprehension. ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. pp. 7839-7859 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{df0bbf8eb2ec45138ff923d1b21996d0,

title = "To test machine comprehension, start by defining comprehension",

abstract = "Many tasks aim to measure MACHINE READING COMPREHENSION (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension-a TEMPLATE OF UNDERSTANDING-for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.",

author = "Jesse Dunietz and Gregory Burnham and Akash Bharadwaj and Owen Rambow and Jennifer Chu-Carroll and David Ferrucci",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 ; Conference date: 05-07-2020 Through 10-07-2020",

year = "2020",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "7839--7859",

booktitle = "ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference",

}

Dunietz, J, Burnham, G, Bharadwaj, A, Rambow, O, Chu-Carroll, J & Ferrucci, D 2020, To test machine comprehension, start by defining comprehension. in ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 7839-7859, 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Virtual, Online, United States, 07/5/20.

To test machine comprehension, start by defining comprehension. / Dunietz, Jesse; Burnham, Gregory; Bharadwaj, Akash et al.
ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. p. 7839-7859 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - To test machine comprehension, start by defining comprehension

AU - Dunietz, Jesse

AU - Burnham, Gregory

AU - Bharadwaj, Akash

AU - Rambow, Owen

AU - Chu-Carroll, Jennifer

AU - Ferrucci, David

PY - 2020

Y1 - 2020

N2 - Many tasks aim to measure MACHINE READING COMPREHENSION (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension-a TEMPLATE OF UNDERSTANDING-for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.

AB - Many tasks aim to measure MACHINE READING COMPREHENSION (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension-a TEMPLATE OF UNDERSTANDING-for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.

UR - https://www.scopus.com/pages/publications/85098787360

M3 - Conference contribution

AN - SCOPUS:85098787360

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 7839

EP - 7859

BT - ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020

Y2 - 5 July 2020 through 10 July 2020

ER -

Dunietz J, Burnham G, Bharadwaj A, Rambow O, Chu-Carroll J, Ferrucci D. To test machine comprehension, start by defining comprehension. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2020. p. 7839-7859. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

To test machine comprehension, start by defining comprehension

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this