Web site logical path: [www.psy.gla.ac.uk] [~steve] [talks] [this page]
??
Presenter
Steve Draper,
School of Psychology,
University of Glasgow.
Slides:
In APR, instead of reading each student script once and deciding its mark,
markers see pairs of scripts and decide which should rank above the other on a
single complex criterion. Software assembles these pairwise judgements
into a quantitative interval scale (based on Thurstone's "law of comparative
judgement"). Finally, grade boundaries are superimposed on the rank
order. Controlled studies using professional markers employed by school
exam boards have shown that marking in this way gives much higher (NOT lower)
reliability, and that for large numbers of scripts, the total time taken is
less. The statistics can directly identify when sufficient consensus has
been reached, which scripts generate most disagreement (send them for a second
and third opinion), and which markers agree least with other markers.
Originally designed for higher reliability (fairness) of marks and
reduced costs, it can also produce feedback comments. The talk will also
discuss how it applies to different disciplines.
The most interesting underlying issue is that APR is in complete contrast to
the approach which is currently, by default, the dominant one of breaking a
judgement down into separate judgements against multiple explicit criteria,
which at least has the virtue of supporting usefully explicit and diagnostic
feedback to learners. Instead, APR uses a single complex criterion for
marking. However in many ways, real academic values have a large
implicit aspect; and furthermore, are holistic instead of being always
and simply reductionist. APR is particularly appropriate for portfolios
of work, and for work where different students may choose to submit in
different media (e.g. printed, web pages, audio tapes).
In order to book online and obtain further information about the conference,
please visit
6th Annual University of Glasgow Learning and Teaching Conference.
Web site logical path:
[www.psy.gla.ac.uk]
[~steve]
[talks]
[this page]
Handout:
Related material:
Abstract
Assessment by pairwise ranking (APR), also referred to by various other terms,
has recently emerged from the school sector as a radical departure in
assessment, only recently made feasible by technology. This talk
introduces it, and discusses whether and how it could be beneficial and
practical to introduce it in Higher Education.
[Top of this page]