Last changed 9 Nov 2017 ............... Length about 5,000 words (43,000 bytes).
(Document started on 25 Dec 2009.) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/rap/index.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [rap] [this page]

Assessment and Feedback (A&F) in HE

By Steve Draper, Department of Psychology, University of Glasgow.

This is an entry page into pages on A&F (assessment and feedback) in HE. This began with my involvement with the REAP project (April 2005 - July 2007); and with followup work.

Contents (click to jump to a section)

Design Principles
Interventions (some learning designs for feedback)
Links to other websites on A&F
5 important websites on A&F
Conflicting criteria for ASSESSMENT design I.e. these are principles for assessment design, NOT feedback.
Key issues
Transformation: How to achieve change in practice
QEE website

Design Principles

Rival sets of design principles for feedback and assessment
Some proposed new principles
My heresies and anti-principles on A&F
Eight tips for enhancing a proactive approach in students (by David Nicol): "Make feedback work for you".
- A webpage: the panel on its right hand side shortlists the tips, the main text expands on each one.
- Leaflet summary (PDF) (Print as 2-sided colour A4; then fold once to give 4 X A5 panels.)
- Local copy of leaflet
- Workshop for students based on the 8 tips

Do students actually want or need feedback?

The rest of this page goes with the conventional literature that presupposes without even discussion, much less evidence, that students want feedback in order to improve their learning. Although this may be untrue:

Many publications have reported that many students do not read or use feedback.
See my paper on multiple feedback loops which argues that improving technical skill or understanding is only one use of feedback, and not necessarily the important one for learners.
"... like most normally constituted writers Martin had no use for any candid opinion that was not wholly favourable."
"He had shown his letter to Maturin partly as a mark of confidence and esteem ... and partly so that Maturin might praise it, possibly adding a few well-turned phrases; for like most normally constituted writers Martin had no use for any candid opinion that was not wholly favourable." -- P.O'Brian (1984 / 2003) The far side of the world p.131
N.B. this may be an entirely normal learner attitude; and so is important for good teaching to grasp. See Mitra's use of standin "grandmothers" to achieve wildly impressive learning results, precisely by supplying uncritical positive feedback: [ Mitra's TED talk; Youtube piece about it The project ] (I.e. "grandmothers" supply NO informative but demotivating teacherly "feedback" i.e. negative criticism, which makes teachers self-important, but is essentially about telling learners, and so is anti-constructivist). Thus it is rational, and supported by considerable evidence, to consider the whole usual approach to feedback as counter-productive.

Oh well, back to the bad old assumptions ...

Interventions (some learning designs for feedback)

My pages on particular A&F interventions (learning designs, methods)

Reciprocal Peer Critiquing
Feedback calendars
One Minute Papers
Elective feedback
Sue Bloxham, from whom I myself first learned about elective feedback.
Self-assessment. A simple step is to require, for all work, that the student makes a serious attempt to estimate a mark for their work, and submits that (perhaps in a sealed envelope). This both gives them practice at making the judgement (to improve future self-regulation); and can be used by the tutor to guide formative comments (is the student misguided about the standard required; or judges accurately but just doesn't know how to improve their work).
Note that when I got my students to do both formative self-assessment and elective feedback, they said it didn't add anything. Requiring a mark might add something; but for formative purposes, perhaps elective feedback and self-assessment may cover the same ground w.r.t. intrinsic benefit to the learner. (Getting comments from others however adds some value.)
Feedback vivas. Two teachers, one student, 10 minutes: dialogue about a piece of submitted student work and the written feedback by tutors on it returned at least two days before the viva. This achieves staff-student dialogue about feedback, though at a cost.
Prompted student processing of feedback. I've held meetings with students to discuss written feedback I've given them. Not a lot of discussion happened. Better has been to get them to write brief answers to questions on it, and what they will do with it. This achieved staff-student (and peer) dialogue about feedback.

Oral feedback

MP3
Work flow support.  Beryl Plimmer.
Camtasia: Sound plus markup
Two lessons?:  techno support that eases the marker's load pays immediate
	dividends in delivered value to students.
Markup may be important partly because of that, but also intrinsically when
pointing is important to effective (and economical) communication about the
issues with a piece of work.

Speed tutorials. asdf
EVS mediated class tests. Class tests mediated by electronic voting systems (EVS), provided they can be expressed in multiple choice question format, have the feedback advantage of on the spot turnaround (within the hour, not 14 days: 2-3 orders of magnitude faster), per-item feedback on personal right/wrong and the group's performance, plus explanations as required, and of dialogue about the feedback with both teacher and peers. Here students might work for half an hour on paper on a set of questions; then they key in the answers they have calculated; and the lecturer responds to the displayed, aggregated results: moving swiftly over unproblematic questions, and explaining those questions where substantial numbers got the wrong answer. Besides providing instant feedback to students without staff labour, I have seen students question staff for clarification at this stage, thus essentially making the feedback interactive and responsive, while also shared by the whole class. This, even though much cheaper, is qualitatively superior to the usual written feedback we give students, which must often miss the mark because it has to assume some level of understanding which we cannot check. (See also here.)
Ipsative feedback ("ipsative" means compared to the person's own performance). The idea is, when commenting on a piece of work, to link the comments to the student's previous work and the previous feedback given on it: for each issue, have they improved, stayed the same, got worse? This is best supported by using a) a structured feedback form (e.g. one section per marking criterion); b) digital records so the marker can summon up the previous feedback that student got (on the structured form).
Jo Royle had good responses from this technique on Access courses, and at GIC.
C.O'Siochru at Liverpool Hope reports feedback on a year 2 project to developing the year project.

Gwyneth Hughes had a JISC project on 'assessment careers' i.e. on ipsative feedback. She now has a book on it:
Hughes,G. (2014) Ipsative Assessment: Motivation through marking progress (Basingstoke: Palgrave Macmillan); doi:10.1057/9781137267221 publisher's details.
(Golf handicaps apply the ipsative principle, not to feedback but to assessment i.e. to assigning scores or marks based on that person's past performances.) The patchwork text technique achieves significant ipsative feedback from peers, rather than from teachers.
2-D feedback: letting the learner know how they are doing both relative to their previous performance (ipsative) and relative the rest of the class (e.g. their rank in class, or a grade if they know what the grades mean in terms of other students' performance). (Note that neither of these two useful dimensions is the rather unhelpful one mostly used in HEIs of some absolute scale with grade descriptors, which however doesn't give the student any usable comparisons for the mark they receive. Like giving a volume in minims, a weight in scruples, or a temperature in degrees Réaumur: numbers actually only are useful to people who already remember the numbers of some cases measured on the same scale as comparison points.)
Hanscomb's virtues.
When teachers mark a piece of student work, we often involuntarily perceive or attribute characteristics to the student to do with the way they did it: procrastination, showing contempt for the marker, showing undue conscientiousness, .... The educational issue is: should (HE) teachers pay any attention to this? On the one hand, if we want to focus on students' learning we should pay attention to results not to their personal habits such as the clothes they wear, whether they work at night or day, whether they are tall, or bald. On the other hand, employers explicitly ask us to comment in letters of reference on some of these attributes (e.g. diligence, self-starting, timeliness, sickness absences, ...); and most programmes in fact give directive study skill advice. If we were to give feedback based on these attributions, it would be a new dimension to feedback on student work, but a logical continuation of the role of study advisor.
Feedback on work related attributes.
Lorna Morrow is developing a feedback sheet and procedure for doing a version of Hanscomb's idea, but focussed on work related attributes, in this department.
Plus/minus marking
John Cowan's technique of plus/minus marking
Sequenced assignments. The idea is that feedback use by students works much better if they have an immediate opportunity / requirement to use it. Some versions of this are:
- Students resubmit the same work with corrections suggested by the feedback, or retake the same test (e.g. MCQs with feedback automatically provided).
- Patchwork text where each week each student writes a few hundred words, to a different format but always on the same subject (their reflections on their own professional practice).
- Gunn's "body and belief" course, where there was a sequence of assignments, always different in their format (e.g. short talk, long essay, ..) but always the same in their topic (a particular period of history and a particular theoretical "lens" both selected by each student).
Reader-Based Feedback This refers to expressing feedback not as judgements about the student, but as the personal responses and feelings of the reader.
Peter Elbow points out that we have not one but two critical modes or voices: in one (judgemental, authoritative, "constructive") we tell the learner what they should have done; in the other, we tell them what we felt when we read their work (describing our personal feelings and interpretations). It phrases nothing as an attribute, presupposed true, of the student; and everything as a feeling or problem of the reader/tutor e.g. "your emailing back revisions in response to every bit of feedback given makes me think you are conscientious"; or "I was excited by your introduction, but felt lost by the end of section 2".
The latter "Reader-Based Feedback" mode a) is forced on us when we don't know what the author intended to communicate, and so cannot be constructive and concrete; (b) is much less affronting for authors (students) sensitive to criticism. For teaching creative writing, most students identify their writing with their core identity and are ultra-sensitive; in hard sciences, there is usually an external objective standard of correctness, so comments are mostly seen as impersonal and checkable by the student. Academic essay writing is in between. I also have a short discussion of my view of Elbow's "Reader-Based Feedback" within this page.
The reference for Reader-Based Feedback is probably
Elbow, Peter (1973/ 1998) Writing without Teachers (New York: Oxford UP)
An impure, but "constructive", variation on this insists on giving a clear judgement instead of only an observation for the recipient to process, and instruction instead of leaving the recipent to decide on action; while still having part of the quality of RBF. One schema for such a response is: "When A happens, I feel B because C. What I imagine is D [show you understand what is behind their behaviour]. What I would prefer is E."

Links to other websites on A&F

GU "LEAF" A&F toolkit

REAP project website (Page on my part of the REAP project)

REAP website home page
Case studies from within REAP
Case studies from outside REAP (papers from REAP online conference)
Published papers (mostly on theory or the project overall, rather than on cases).

New Strathclyde website(s) on A&F

For staff: webpage leaflet
For students: webpage leaflet

Students

NUS principles of good feedback
Feedback amnesty campaign

5+ important websites on A&F

See also a 2-part report on practical advice on giving learners feedback by Thalheimer: Part 1 Part 2

Conflicting criteria for assessment design

In this draft, I'm writing this section egocentrically, referring to practices in this psychology dept., which is an essay-based discipline. I believe the points are general, but here I'm not writing to bring this out.

The idea here is NOT to offer techniques for assessment BUT to provide a clear statement about the conflicting criteria which any assessment must satisfy or compromise over. This is necessary to have any rational thought, let alone discussion, about choices in decideing on assessment design. Most of the literature is lacking this.

a) Criteria / requirements / dimensions of merit / aims / constraints: all of which independently apply to any assessment design.
List EXPLICITLY the key criteria that have to be considered, both the naively aspirational educational slogans, and also the unspeakable but real constraints. What is hard about redesigning assessment is that there isn't one thing you want to improve, but how to optimise, or at least satisfice (reach an acceptibility threshold), multiple requirements that often conflict. This is made much harder by some not being written down in public, and so not discussed rationally by staff. (There is a provisional list below.)

b) Metrics: For each of these criteria give a measurement scale that shows teachers the degree to which it is satisfied. E.g. if you want to raise the NSS score, then the NSS subscale is the measure (and could be administered every semester by a course team). If you want to improve learning, then you must show (for instance) grade rises year on year to demonstrate whether or not you succeeded.

c) Marks: I will also occasionally mention the marks or grades given to students as the result of an assessment activity, to point out what they would (logically) mean if they were to represent that educational aim (criterion) for the assessment design.

Draft list of assessment constraints / dimensions

Learning from doing. The single biggest use of assessment at the moment, which however is never mentioned in most literature on assessment, (is not to measure student knowledge at all, but) is to mount an activity which is powerfully "mathemagenic" (productive of learning). We learn mostly by doing; it often doesn't matter whether it succeeds or fails, it often requires NO feedback by staff (contrary to what Laurillard says), just the internal changes that happen when we plan and attempt something new.
Reword? At a simple level, the whole of the Maths presentation at the workshop was about the large demonstrated learning benefits from persuading students to actually do some maths work every week. Conversely, students generally report learning a lot from doing their final year project, although we don't measure this. Their whole redesign addresses this criterion.

Metric: The metric for satisfying this design criterion/aim is how much the student learns from the activity, pre-to-post.
Mark: essentially this measures attendance (engaging in the learning activity with reasonable sincerity), if it aligns with this aim.
Produces information that is useful to the student.
1. The information might be used either formatively or summatively.
2. It may be based on:
  1. A human judgement
  2. A fact (right answer), independently confirmable elsewhere
  3. Or most powerfully, the degree of success of a construction (a bridge you built, a cake you baked).
3. It may be in the form of:
  1. a mark/grade,
  2. written comments,
  3. It may be only an internal effect of changing the learner's degree of certainty / confidence in knowing something. Here are three kinds of assessment to do with this:
    - "Catalytic assessment" (Draper, 2009b) and peer discussion in general is one way of problematising confidence: the learner wonders if they have got it right, and is likely to work later to resolve it.
    - Formative tests: typically these have many items, and which items a learner fails shows which topics they need to direct further effort to.
    - Reassurance quizzes: essentially these are like formative tests in that any missed items show something that needs further work, but students may mostly use the overall score to tell then whether they are on the right track or have mistaken a lot and need to do a major redirection of effort.
[2.2] But the often neglected further issue is: to which use is it put? As argued in Draper (2009a), egocentric academics hold whole conferences on A&F while presupposing that the only use is to improve the technical knowledge of the learner. Each type of learner use of assessment and feedback is in fact an independent criterion for designing an assessment, so that it produces that information. Thus this one sub-criterion of providing informaton useful to the learner in fact produces six alternative independent criteria, all desirable.
Draper,S.W. (2009a) "What are learners actually regulating when given feedback?" British Journal of Educational Technology vol.40 no.2 pp.306-315 doi:10.1111/j.1467-8535.2008.00930.x
Draper,S.W. (2009b) "Catalytic assessment: understanding how MCQs and EVS can foster deep learning" British Journal of Educational Technology vol.40 no.2 pp.285-293 doi:10.1111/j.1467-8535.2008.00920.x
One list of learner uses follows.
1. Self-regulate and allocate the learner's limited time and effort: if I got a B grade, then I needn't think about this topic any more. Spend less time on what I'm good at, more on what I am struggling with. As used in "mastery learning", and its use of formative testing to focus remedial learning each week, this brings large gains.
  Another form of this is "catalytic assessment" (Draper, 2009b): designed, like a brain-teaser, to signal to the learner that this is something they don't understand yet but want to.
2. Decide future courses, based on what I did well on in the past. Spend more time on what I'm good at, drop what I struggle with. Our educational system requires students to make choices, but we fail to design assessments to support that choice optimally.
3. Decide on the quality of the marker. Seek out other opinions.
4. Improve the learner's technical knowledge.
5. Decide whether and how to adjust my learning method.
6. The mark may be interpreted by a learner as feedback on their learning, revision, and exam technique as a whole process.
Metric: measures of pre/post change in information picked up by the learner.
Cost to staff (in time, mostly).
Metric: Staff-hours on the assessment.
Defensiveness against student complaints, which cost both school and senate office staff a lot of time and trouble. This criterion has always been the main problem obstructing useful feedback from exams.
Metric: Staff-hours / money spent on complaints and appeals.
A measure for employers to use to discriminate amongst job applicants.
Metric: (Validity, reliability, and ..) One metric is variance. E.g. Coursework not only has a higher mean mark, it has a lower standard deviation which makes it of considerably less use in discriminating capability.
A measure of competence: (if you want this, use senate schedule C for pass/fail course marks; if you don't then don't moan about competence assessment as an aim).
Metric: (Validity, reliability)
Mark: Pass / fail.
A measure of how much specific knowledge a student has. Our level 3 stats exam does this well; our other level 3 exams (1-hour essays) do not, because they offer a choice of questions each of which requires only a small proportion of the course's content knowledge.
Metric: (Validity, reliability)
A measure of generic discipline skill. Exam essays are our instrument for this, and quite good at it. The main criterion is: to what extent is the essay written like a psychologist? There are low level skills we teach but don't use much assessment on measuring. Then (mid-level) we assess specific content knowledge (facts and concepts rather than skills). We, like most departments, focus most on the ultimate, deep, high level skill of thinking and writing like a professional in the discipline. It is why essays are fundamentally confusing to level 1 students because essays mean different things in each discipline: for a very deep reason. The metric for this criterion is whether a given assessment measures, usually tacitly, how well the candidate exhibits disciplinary thinking (rather than reproduction of specific facts, names, etc.).
Metric: (Validity, reliability)
Student enjoyment of the activity: Do students like doing it? Giving students a choice of topic in an essay or project is motivated by this. On all other criteria, a fixed topic would be better. (Students may learn more if they enjoy it: that would be a positive secondary effect. Equally, they may choose a topic that is least work to them: a negative secondary effect on how much they learn.) In choosing a topic for an assessment, students are in fact choosing part of their curriculum: another deep educational issue disguised as an assessment design choice by teachers.
Metric: student self-reports on enjoyment. More sophisticated versions of this might ask for self-reports on how much they feel they learned, and separately how much it corresponded to their intrinsic learning goals (as opposed to required curriculum learning goals).
Raise NSS scores for the A&F subscale. There is generally little correlation between scores on the A&F subscale, and on the overall course (programme) satisfaction, so there is no reason to think that A&F contributes to learning nor to student satisfaction.
Metric: The NSS subscale: how much does it increase?

Key issues

NSS: A&F scores don't affect the overall student rating of a course

Perhaps feedback doesn't make a difference to the amount of learning. Teachers should have communicated it in advance, so feedback not necessary; learners should know how to check and remediate their own learning, and not rely on being told this.

F-Prompting seems to be SO important, transformative of whether students learn from feedback. The main problem seems to be that our students mostly do not have any concept of learing from our written feedback: it doesn't occur to them to actively use it.

Transformation: How to achieve change in practice

Reflecting back on the success of REAP gave us some ideas on what does (and does not) go into making a project effective at actually changing learning and teaching in practice, and making it measurably better. These papers are about this, and so effectively on ideas about how to design and run large projects that bring about significant, large scale changes (in areas such as A&F).

Transformation in e-learning
Draper,S.W. and Nicol,D. (2006) The content of a talk given at ALT-C, Sept 2006 Local copy (PDF)
Understanding the prospects for transformation
Nicol,D. and Draper,S.W. (2006?) Local copy (PDF) REAP website copy (PDF)
A blueprint for transformational organisational change in higher education: REAP as a case study
Nicol,D. and Draper,S.W. (2009) Local copy (PDF)
A shorter version of this is in: Transforming Higher Education through Technology-Enhanced Learning ed. Terry Mayes, Derek Morrison, Harvey Mellar, Peter Bullen and Martin Oliver (2009) (York: Higher Education Academy) ch.14 pp.191-207 Local copy (PDF) REAP website copy (PDF)
Achieving transformational or sustainable educational Change
Draper,S.W. & Nicol,D.J. (2013) "Achieving transformational or sustainable educational change" ch.16 pp.190-203 in Reconceptualising feedback in higher education: Developing dialogue with students S.Merry, M.Price, D.Carless & M.Taras (eds.) (London: Routledge) Local copy (PDF)

References and links to Carol Twigg's work on transformation Resources for doing it her way

Transforming Higher Education through Technology-Enhanced Learning ed. Terry Mayes, Derek Morrison, Harvey Mellar, Peter Bullen and Martin Oliver (2009) (York: Higher Education Academy). A book, available online.

QEE website

QEE / QET Integrative Assessment

sharepoint s.draper 4i8x7t1m

HEAcademy

http://jisctechdis.ac.uk/

Web site logical path: [www.psy.gla.ac.uk] [~steve] [rap] [this page]
[Top of this page]