TAG parser evaluation using textual entailments

Parser Evaluation using Textual Entailments (PETE, Yuret et al. (2013)) is a restricted textual entailment task designed to evaluate in a uniform manner parsers that produce different representations of syntactic structure. In PETE, entailments can be resolved using syntactic relations alone, and do not implicate lexical semantics or world knowledge. We evaluate TAG parsers on the PETE task, and compare our results to the state-of-the-art. Our TAG parser combined with structural transformations to compute entailments outperforms the CCG-based results on the development set, though it falls behind these results on the test set. The CCG parser makes use of a number of heuristics for entailment comparison, however. Adding such heuristics to our best TAG parser yields state-of-the-art results on the test set when using accuracy as a metric. This sensitivity to heuristics suggests that the PETE task may suffer from an unrepresentative development set, and that we need to improve upon formalism-independent parsing evaluation methods.