Web site logical path:
[www.psy.gla.ac.uk]
[~steve]
[EVSmain]
[this page]
Question design:
[one question]
[purposes]
[contingent sessions]
[feedback]
[whole sessions]
There is a whole art to designing MCQs (multiple choice questions). Much of the literature on this is for assessment. In this context however we don't much care (as that literature does) about fairness, or discriminatory power, but instead will concentrate on what will maximise learning.
Here I just discuss possible formats for a question, without varying the
purpose or difficulty. I was in part inspired by
A common type of MCQ concerns one relationship
e.g. (using school chemistry as an example domain)
"What is the chemical symbol for gold: Ag, Al, Au, Ar ?"
Applied to statistics this might be:
The idea is to require students to access knowledge of a topic from several
different starting points. Here I exercised three kinds of link, and each
kind in both directions. Exercising these different types and directions of
link is not only important in itself (because understanding requires
understanding all of these) but keeps the type of mental demand on the
students fresh, even if you are in fact sticking on one topic.
The first is that of linking ideas or concepts to particular examples or
instances of them e.g. is a whale a fish or a mammal? Another form of this
is linking (engineering or maths) problems with the principle or rule that is
likely to be used to solve it.
However both concepts and instances are represented in more than one way, and
practice at these alternative representations and their equivalences is
usually an essential aspect of learning a subject. Thus concepts usually have
both a technical name, and a definition or description, and testing this
relationship is important. Similarly instances usually have more than one
standard method of description and, although these are specific to each
subject, learners need to master them all, and questions testing these
equivalences are important. In teaching French language, both the spelling,
the pronounciation, and the meaning of a word need to be learned.
In statistics, an example
data set should be represented by a graph, a table of values, as well as a
description such as "bell shaped curve with long tails". In chemistry, the
name "copper sulfate" should be linked to "CuSO4" and a photograph of blue
crystals, and questions should test these links. (See Johnstone, A.H. (1991)
"Why is science difficult to learn? Things are seldom what they seem"
Journal of computer assisted learning vol.7 no.2 pp.75-83 for an
argument related to this based in teaching Chemistry.
See also Roy Tasker's group:
http://visualizingchemistry.com/research.)
These relationships are all bidirectional, so questions can (and should) be
asked in both directions e.g. both "which of these is a mammal" and "to which
of these categories do dolphins belong?". Thus a subject with three standard
representations for instances plus concept names and concept definitions will
have five representations, and so 20 types of question (pick one of five for
the question, and one of the remaining four for the response categories).
Additional variations come from allowing more than one item as an answer, or
asking the question in the negative e.g. "which of these is not a mammal?:
mouse, platypus, porpoise?".
The problem of technical vocabulary is a general
one, and suggests that the concept name-definition link should be treated
especially carefully. If you ask questions that are problems (real-world
cases) and ask which concept applies but use only the technical names of the
concepts, then students must understand perfectly both concept and the
vocabulary; and if they get it wrong you don't know which aspect they got
wrong. Asking concept-case questions using not technical vocabulary but
paraphrased descriptions of the concepts can separate these; and separate
questions to test name-definition (i.e. concept vocabulary).
It may or may not be a good idea to include null responses as an option.
Against offering them is the idea that you want to force students to commit to
an answer rather than do nothing, and also the observation that when provided
usually few take the null option, given the anonymity of entering a guess.
Furthermore, a respondent could simply not press any button; although that,
for the presenter, is ambiguous between a decision rejecting all the
alternatives, the equipment giving trouble to some of the audience, or the
audience getting bored or disengaged. However if you do include them as
standard, it may give you better, quicker feedback about problems. In fact
there are at least three usually applicable distinct null options to use:
An extension of this are:
Assertion-reason questions.
Russell, Mark (2008) "Using an electronic voting system to enhance learning
and teaching" Engineering Education vol.3 no.2 pp.58-65
doi:10.11120/ened.2008.03020058
Web site logical path:
[www.psy.gla.ac.uk]
[~steve]
[EVSmain]
[this page]
Reversing the relationship
You can equally, and additionally, ask about the same relationship in reverse:
"Which metal is represented by the symbol 'Au'? Gold, silver, platinum, copper?"
Multiple types of relationship
When you have several relationships, the alternative question types multiply.
Consider these 3 linked pieces of information: a photo of a gold nugget or
ring; the word (name) "Gold"; and the symbol "Au". These 3 pieces of
information each have a relationship with the other 2, giving 3 types of
relationship; and each has 2 directions, giving 6 question types in all:
Types of relationship to exercise / test
In the abstract there are three different classes of relationship to test:
Further Response Options
The handsets do not directly allow the audience to specify more than one
answer per question. However you can offer at least some combinations
yourself e.g.
"Is a Black Widow:
Assertion-reason questions
I particularly commend asking MCQs that, instead of asking which fact is true,
ask which reason for a given fact is the right one. Covertly related questions:
Using 3 questions to make a strong test of understanding one concept
Mark Russell suggests using 3 (say) alternative questions all testing the same
key concept. With MCQs with 4 response options, 25% of students will get a
question right by accident if they answer at random: not a strong test.
He suggests having 3 alternative questions testing exactly the same concept,
and only students who get all 3 of these correct should be regarded as having
learned the concept.
The questions are tacitly linked (by being about the same concept), but not
listed adjacently and not using similar structure. He found that students who
did not have a sound understanding of the concept did not even recognise that
the 3 questions were linked: the disguise does not need to be elaborate
(contrary to expert / staff perceptions, who naturally see the 3 questions as
"about the same thing" exactly because they grasp the concept). Some references on MCQ design
[Top of this page]