TY - GEN
T1 - Views Are My Own, but Also Yours
T2 - Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
AU - Soubki, Adil
AU - Murzaku, John
AU - Jordehi, Arash Yousefi
AU - Zeng, Peter
AU - Markowska, Magdalena
AU - Mirroshandel, Seyed Abolghasem
AU - Rambow, Owen
N1 - Publisher Copyright:
© 2024 Association for Computational Linguistics.
PY - 2024
Y1 - 2024
N2 - Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, COMMON-TOM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on COMMON-TOM.
AB - Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, COMMON-TOM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on COMMON-TOM.
UR - https://www.scopus.com/pages/publications/85205286140
U2 - 10.18653/v1/2024.findings-acl.880
DO - 10.18653/v1/2024.findings-acl.880
M3 - Conference contribution
AN - SCOPUS:85205286140
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 14815
EP - 14823
BT - The 62nd Annual Meeting of the Association for Computational Linguistics
A2 - Ku, Lun-Wei
A2 - Martins, Andre
A2 - Srikumar, Vivek
PB - Association for Computational Linguistics (ACL)
Y2 - 11 August 2024 through 16 August 2024
ER -