Publication

On Natural Variation in Grades in Higher Education, and Its Implications for Assessing Effectiveness of Educational Innovations

Boevé, A. J., Meijer, R. R., Beldhuis, H. J. A., Bosker, R. J. & Albers, C. J., 2019, In : Educational Measurement: Issues and Practice. 38, 4, p. 55-66 12 p.

Research output: Contribution to journalArticleAcademicpeer-review

Copy link to clipboard

Documents

  • On Natural Variation in Grades in Higher Education,and Its Implications for Assessing Effectivenessof Educational Innovations

    Final publisher's version, 1 MB, PDF document

    Embargo ends: 01/02/2020

    Request copy

DOI

To investigate the effect of innovations in the teaching?learning environment, researchers often compare study results from different cohorts across years. However, variance in scores can be attributed to both random fluctuation and systematic changes due to the innovation, complicating cohort comparisons. In the present study, we illustrate how using information about the variation in course grades over time can help researchers and practitioners better compare the grades and pass rates of different cohorts of students. To this end, all 375,093 grades from all 40,087 first-year students at a Dutch university during a period of six consecutive years were examined. Overall, about 17% of the variation in grades could be attributed to random variation between years and courses. With respect to passing courses, this percentage was almost 40%. Nonsignificant improvements in grades could be flagged as highly significant when this is ignored, thus leading to an overrepresentation of significant effects in educational literature. As a consequence, too many educational innovations are claimed to be effective.
Original languageEnglish
Pages (from-to)55-66
Number of pages12
JournalEducational Measurement: Issues and Practice
Volume38
Issue number4
Early online date31-Jul-2019
Publication statusPublished - 2019

    Keywords

  • educational innovations, effectiveness, pass rates, reporting bias, statistical modeling, FLIPPED CLASSROOM, EQUIVALENCE, PERFORMANCE, TESTS

View graph of relations

ID: 94193394