Last changed 19 Sept 2008 ............... Length about 1,000 words (9,000 bytes).
(Document started on 28 Dec 2007.) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/studdesign.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
Question design: [one question] [purposes] [Student design] [contingent sessions] [feedback] [whole sessions]

Having students design the questions

(written by Steve Draper, as part of the Interactive Lectures website)

Another use of MCQs, and hence perhaps of EVS, is to get learners not just to answer questions, but to design (write) them. I myself only realised this because Andy Sharp (then at Glasgow Caledonian University) did it: but with hindsight, this is obviously a powerful idea. A number of people have done it and published about it, but it is still a very uncommon piece of good practice.

It can work well without EVS, but may work better with EVS since that makes if faster and cheaper to administer the questions to the whole class in one way or another.

Designing questions is likely to be strongly productive of (better) learning because:

You have to give reasons, as part of the design process, why each answer option is right or wrong. Giving reasons rather than just choosing an option, leads to better learning.
You have to pick a question topic: so you have to start attending to what the curriculum is, what the important topics are, and what you are supposed to know about each one. Doing this is good; starting on this earlier in a course is better.
Picking or designing the type of question, and the type of knowledge it asks about e.g. trivia, simple facts, concepts, reasons. This first makes students think about the type of learning desired, and then how this can be expressed in questions and answers. (This is typically supported by briefing students on Bloom's taxonomy, which can be linked to types of questions.) The thrust of this is to move students away from simple fact reproduction to deeper types of learning. This is furthermore going to make them more reflective about their own learning, and the aims of the course.
Considering how to generate "distractors": incorrect response options. The best ones, both for question design and for generating learning in the designer as well as in those answering it, are tempting misconceptions. A learner who can give the right answer and say exactly why a misconception is tempting yet wrong, understands more.

Support for such an exercise

Group work: when the activity is new to the students, then doing it in groups is generally a comfort and support for them. If this became a familiar and frequent activity, then they might do it faster and get more from it by doing it solo.

Train/brief them on Bloom's taxonomy.

Have the questions tried out on the whole class with EVS: their peer's responses are powerful feedback on the design process.

If the questions produced are "marked", then give more credit for:

Higher levels in Bloom's taxonomy
Specifying which response option is correct, and giving clear reasons (in their documentation) for the correctness or wrongness of each answer option.
Questions that discriminate within the class i.e. that some other students get right and some get wrong.
Strong relationship to the learning aims and objectives of the course.

Motivations and contexts for student question design

Require students (or groups of students) to write and deliver a talk to the whole class, and to include some EVS questions as part of their presentation (Sharp & Sutherland).

Examine the whole course by MCQs. Tell the students that the final exam will be composed of the teacher's selection from the set of questions composed by the students during the course.

One useful activity for student study groups, especially during revision time, is to test each other on questions. Designing questions for use in this context is a further improvement.

Weekly quizzes for the class (e.g. using EVS) with students in turn providing questions for these. Thus students coooperate in creating a question bank for the class.

Having students create the topics rather than the wording of questions

Nick Bowskill has developed an application of EVS that both has students provide the content of questions, and do so for the purpose of an in depth course feedback exercise.

The elicitation session he has developed is really a form of the pyramid evaluation technique for course evaluation in depth: a mixture of solo and small group and plenary to identify issues, then identify which are common to many learners. (Actually instead of applying it to a single module, he applied it to the whole first year of a programme of course makes it a great attack on improving induction, and making it responsive to individual student cohorts and departments.)

It is also a kind of halfway case of student-generated EVS questions: they provide the material for the subject of the questions, which staff then actually edit and deliver. You could argue that this is actually better (more student-active) than students inventing MCQs to test staff-specified subject matter.

A rough recipe for such a session is to begin by asking each student to write down the one or two most important issues or problems for them with the course on a slip of paper. Then having them discuss in groups of 5 what one or two issues the group as a whole would suggest as most important. Then as a plenary of the whole class, have each group shout out their issue which is typed into the EVS software as a question. Then display the list of issues as options to the EVS question "Which for you is the most important issue or problem?". (If you are using software that allows the EVS handsets to be used to express rankings, then you could collect these instead and get more information.)

References

Here's a list of papers I know about, dealing with students designing questions (not necessarily with EVS).

Arthur, N. (2006) "Using student-generated assessment items to enhance teamwork, feedback and the learning process" Synergy: Supporting the Scholarship of Teaching and Learning at the University of Sydney Issue 24, pp.21-23

Bali, M. & Keaney, H. (2007) "Collaborative Assessment Using Clickers" From the REAP International Online Conference on Assessment Design for Learner Responsibility, 29th-31st May, 2007. Available at: http://ewds.strath.ac.uk/REAP07 Fellenz, M.R. (2004) "Using assessment to support higher level learning: the multiple choice item development assignment" Assessment and Evaluation in Higher Education vol.29 no.6 pp.703-719

Sharp, A. & Sutherland, A. (2007). "Learning Gains...My (ARS)S - The impact of student empowerment using Audience Response Systems Technology on Knowledge Construction, Student Engagement and Assessment" From the REAP International Online Conference on Assessment Design for Learner Responsibility, 29th-31st May, 2007. Available at: http://ewds.strath.ac.uk/REAP07

Bloom

Bloom, B. S. (1956) Taxonomy of educational objectives: the classification of educational goals (London, Longmans)