Use of competency-based metrics to determine effectiveness of a postgraduate thoracoscopy course

Background: Despite the paradigm shift from process to competency-based education, no study has explored how competency-based metrics might be used to assess short-term effectiveness of thoracoscopy-related postgraduate medical education. Objectives: To assess the use of a single-group, pre-/post-test model comprised of multiple-choice questions (MCQ) and psychomotor skill measures to ascertain the effectiveness of a postgraduate thoracoscopy program. Methods: A 37-item MCQ test of cognitive knowledge was administered to 17 chest physicians before and after a 2-day continued medical education-approved program. Pre-and post-course technical skills were assessed using rigid videothoracoscopy simulation stations. Competency-based metrics (mean relative gain, mean absolute gain, and class-average normalized gain <g>) were calculated. A <g> >30% was used to determine curricular effectiveness. Results: Mean cognitive knowledge score improved significantly from 20.9 to 28.7 (7.8 ± 1.3 points, p < 0.001), representing a relative gain of 37% and an absolute gain of 21%. Mean technical skill score improved significantly from 5.20 to 7.82 (2.62 ± 0.33 points, p < 0.001), representing a relative gain of 50% and an absolute gain of 33%. Non-parametric testing confirmed t test results (p < 0.001). Class-average normalized gains were 48 and 92%, respectively. Conclusion: Competency-based metrics, including class-average normalized gain, can be used to assess course effectiveness and to determine if a program meets predesignated objectives of knowledge acquisition and psychomotor technical skill.