“Current data-based methods of analysis and predictive models are insufficient – big data is able to remedy this. ”
– Dr. Alexander Lenk
Pass/fail decisions by faculty are mostly based on the ability of students to pass or fail prior to (Angoff) or post standard setting (Borderline Regression Analysis) criteria developed by faculty. Standard setting is a critical part of educational, licensing and certification testing. But outside of the cadre of practitioners, this aspect of test development is not well understood. Standard setting is the methodology used to define levels of achievement or proficiency and the cut-scores corresponding to those levels. A cut-score is simply the score that serves to classify the students whose score is below the cut-score into one level and the students whose score is at or above the cut-score into the next and higher level.
A few words about this paper…
A historically significant paper I have to say, and although not yet cited it formed the basis of what I eventually pursued for the last 10 years in the spin-off company, Qpercom. According to the Irish Times, we are “dragging exam assessment out of the dark ages“ (Oct. 2016). This suppressed paper actually forms the basis of what Qpercom has worked to achieve since 2008 with client partners worldwide. From being a PT clinician by training, I moved into medical education. As clinical researchers, we put a lot of effort into developing the Smallest Detectable Difference (SDD), to be detected using a ‘ruler’. Measuring maximal mouth opening with a metal ruler is one of the outcome variables in patients with maxillofacial pain. With the newly acquired evidence that you had to measure at least 12 mm difference in mouth opening before and after an intervention to be successful in patients with temporomandibular joint disc displacement, I changed jobs and moved into medical/dental education. I was immediately challenged to look into comparable measurements used in oral hygiene training. Probing depth measurements were used as an example to demonstrate the use of generalisability and decision studies in educational decision-making. Fourteen years after this publication, we are comparing 10 different European Universities on their quality assurance outcome of OSCEs. Have a read, use the evidence, and I hope this will help students and staff measuring any kind of assessment outcome, plus this historically significant paper needs citations!