A cognitive inspired method for assessing novelty of short-text ideas

In creativity research a typical problem is that of assessing the novelty of ideas or solutions generated by many people to open ended problems. For datasets larger than a few hundreds, human assessment of novelty becomes time consuming and error prone. Existing novelty detection methods such as: distance based text similarity or language model approaches do not work well for small datasets. Moreover, when compared to human novelty ratings, these approaches fail to capture the same cognitive processes or biases. We are proposing a novel cognitive model inspired by a leaky accumulator decision making models for detecting novel ideas from short text. The model is applied on a collection of ideas generated in a group brainstorming experiment. It evaluates an idea term by term and it accumulates surprise and relevance. The final novelty decision is taken at the end of each idea by means of a threshold. An important component of the model is a small domain dataset which is used to evaluate the surprise of a term's context compared to common domain knowledge. The model is compared with other methods: feature based classifiers, tf-idf similarity distance, and pretrained language models (ULMFIT).