Duration: 4 hours
Description: The Regulator has placed increased focus on reliability in recent years, and not only for general qualifications. But how do you measure the reliability of your assessments?
And how do you interpret the results from those measurements in order to decide whether action is required? Would an assessment candidate get the same results if they took the assessment for a second time? Or if they had a different marker? What if they faced slightly different tasks/questions? How much error is in the scores from a particular assessment and how does this affect the likelihood that a candidate is given the wrong outcome? All these questions and others are within the domain of reliability.
This session will introduce some of the key aspects of this topic, seemingly complex but so key to good assessment, and give attendees key messages to take away for their professional work.
Rather than focusing on complex statistics, we start from practical considerations and look at qualitative interpretation before moving to consider numerical approaches. The programme also looks at practical implementations of reliability measurement in areas where collecting data can be difficult (for example in evidence-based vocational qualifications).
This unit is often delivered alongside the validity unit.
- rating the reliability of assessment types
- reliability Indices and their practical application
- dealing with key reliability concepts – internal reliability, inter-form reliability and intermarker reliability
- concepts of reliability and validity taken together
- how practical reliability measures can be used to introduce transparency and increase confidence in assessment performance.
Typical Task - Working with own assessments to gauge and use appropriate measures of reliability.