Linguistically rich vector representations of supertags for TAG parsing

In this paper, we explore several techniques for producing vector representations of TAG supertags that can be used as inputs to a neural network-based TAG parser. In the simplest case, the supertag is encoded as a 1-hot vector that is projected to a dense vector. Secondly, we use a tree-recursive neural network that is given as input the structure of the elementary tree. Thirdly, we use hand-crafted feature vectors that describe the syntactic features of each supertag, and project these to a dense vector. These three representations are learned during the training of a neural network TAG parser with a layer that embeds supertags in a low-dimensional space. Finally, we consider an embedding that is trained only on patterns of linear co-occurrence among supertags. By testing the resulting vector representations on the task of completing syntactic analogies, we show that these vector representations capture syntactically relevant information. While our linguistically-informed embeddings outperform atomic embeddings on the syntactic analogy task, we find that the same embeddings lead to only a slight improvement on the task of TAG parsing, indicating that the neural parser is able to induce useful representations of supertags from the data alone.