Expanding a national network for automated analysis of constructed response assessments to reveal student thinking in STEM

Improving STEM education requires valid and reliable instruments for providing insight into student thinking. Constructed response (CR) assessments reveal more about student thinking and the persistence of misconceptions than do multiple-choice questions, but require more analysis on the part of educators. In the Automated Analysis of Constructed Response (AACR) Research Group (www.msu.edu/~aacr) we have developed constructed response versions of well-established conceptual assessment inventories and created computer automated analysis resources that predict human ratings of student writing about these topics in introductory STEM courses. The research uses a two-stage, feature-based approach to automated analysis of constructed response assessments. First, we design items to identify important disciplinary constructs based on prior research. The items are administered via online course management systems where students enter responses. We use lexical analysis software to extract key terms and scientific concepts from the students' writing. These terms and concepts are used as variables for statistical classification techniques to predict expert ratings of student responses. The inter-rater reliability (IRR) between automated predictions and expert human raters is as high as IRR between human experts. We recently received another round of funding to extend our work to provide an online community where instructors may obtain, score and contribute to the library of items and resources necessary for their analyses. We provide an overview of the goals of the project and introduce the opportunities to participate in the development of a national network of faculty using these techniques.