Impact of quality and quantity of corpora on stochastic generation

Recently, there has been some interest in using stochastic approaches in generation. However, there has been little research so far on the question of how the quality, size, and genre of training corpora influence the quality of stochastic generation components. In this paper, we investigate these issues using the FERGUS system. FERGUS uses two distinct stochastic models, a tree model which refers to a grammar, and a linear language model. We use automatic grammar extraction techniques to extract grammars from different-sized tree banks, and then use these extracted grammars to train the tree models. We also investigate the impact of the quality of the annotated corpus, by using a hand-annotated corpus as well as an automatically annotated corpus. Our results show that automatic grammar extraction is a viable alternative to hand crafted grammars for generation; furthermore, as expected, both quality and size of the training corpus matter.