Lemmas Matter, But Not Like That: Predictors of Lemma-Based Generalization in Morphological Inflection

Recent work has shown that overlap - whether a given lemma or feature set is attested independently in train - drives model performance on morphological inflection tasks. The impact of lemma overlap, however, is debated, with accuracy drops from 0% to 30% reported between seen and unseen test lemmas. In this paper, we introduce a novel splitting algorithm designed to investigate predictors of accuracy on seen and unseen lemmas. We find only an 11% average drop from seen to unseen test lemmas but show that the number of lemmas in train has a much stronger effect on accuracy on unseen than seen lemmas. We also show that the previously reported 30% drop is inflated due to the introduction of a near-30% drop in the number of training lemmas from the original splits to the novel splits. These results help us better understand the factors affecting morphological generalization by neural models.