Last changed
16 May 2009 ............... Length about 400 words (6,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/lobby.html.
You may copy it.
How to refer to it.
This is the entrance point for my web pages on Electronic Voting Systems (EVS) for use in lectures; or more generally for interactive lectures (ILIG = Interactive Lecture Interest Group); or more specifically for the PRS equipment which we mainly use, and for local Glasgow University arrangements.
If you want a quick look at what it's all about, to see if it might interest you, then try
To see all the things available on this site you should read over the main website index page; or print off all the pages to study: they are available as a single web page ready for printing: compilation on designing lectures (that use an EVS) and a comprehensive compilation.
Some of the most popular parts are:
Last changed
3 Oct 2017 ............... Length about 4,000 words (43,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/main.html.
You may copy it.
How to refer to it.
This is the home page for some web pages, now largely out of date, about interactive lecture methods in general, and using classroom electronic voting systems (EVS) in particular. (EVS are also sometimes referred to as "clickers", PRS, GRS, CCS: for a discussion, see this list of terms used.)
(If you find this site useful, other major sets of pages on a similar topic are at: Peer instruction for computer science; Stanford / tomorrow's professor; Vanderbilt; Amherst; or this Simpson & Oliver 2002 report (33 pages); pages on EVS use in maths from Loughborough and from Surrey. or a page at Colorado.)
If you are located in the UK, particularly, then you might want to join the special interest group on EVS "ESTICT": "Engaging Students Through In-Class Technology". (Other, desperate enquiries about it to Sian Cox [sian.cox.1 AT city.ac.uk].) You might also want to join this email list: http://www.jiscmail.ac.uk/lists/ELECTRONIC-VOTING-SYSTEMS.html (electronic-voting-systems AT jiscmail.ac.uk).
You can access the pages on this website in alternative ways:
The websites for versions of the YACRS server are:
https://classresponse.gla.ac.uk (most stable)
https://learn.gla.ac.uk/yacrs/ (More advanced)
https://learn.gla.ac.uk/yalis/ (being developed
further)
Basic teacher guide
A talk about it was given at the 2015 internal learning and teaching
conference. The abstract is at page 12 of
the proceedings.
Wifi coverage: the expert seems to be Drew McConnell; and the online web pages about coverage are not accurate (seem to under-report its extent). I have the impression that there are just a very few rooms with no usable coverage. Newer / larger lecture theatres may have multiple points. A test of YACRS in the Joseph Black LT with 190 students had no trouble for mass student access to wifi, even though it had only one access point in the room itself.
A server is being commissioned for this. Currently ask
Niall Barr
to set permissions for you to use this; and to get more information.
You may want to consider these elements (you can find links for these from the link above):
In that bibliography, a very few outstanding things are starred: if you want to do some reading, you could do much worse than start with them. See also below.
Other important Glasgow University contacts:
Other people who use EVS (and PRS users) in the UK are listed here.
Last changed
7 Dec 2015 ............... Length about 1300 words (10,000 bytes).
(Document started on 29 Jan 2005.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/terms.html.
You may copy it.
How to refer to it.
There are various terms or names used to refer to the equipment in question:
I don't like them. On the other hand, some others do. I've debated this most with Michael McCabe. Here are my views (with which he disagrees quite strongly) on what the names should say, and what is wrong with the ones being used.
Kay reports finding (in 2008) 26 terms in the literature.
[System vs. equipment, technology]
Saying "system" to refer to a small bit of equipment which is not a system that
stands alone (without human operators it does nothing), but a wholly dependent
adjunct on the real system of, say, teacher, students and discussion is
inaccurate and self-inflating: "equipment" might be more exact. The real
"system" using EVS in, say, education is something like the plan for the whole
lecture or session. There are a number of quite different alternatives that
do use EVS (e.g. Mazur's "Peer instruction", or contingent teaching); and also
still others (e.g. MacManaway's use of "lecture scripts") that do not, but are
equally revolutionary and promising.
[voting, polling vs. texting vs. other shared data types]
The equipment I'm usually referring to is for giving one of a small number of
pre-determined alternative choices i.e. responding only to MCQs (multiple
choice questions): hence the direct term would be "voting" or "polling".
This also contrasts it to some other technologies that support free-text
open-ended input from the audience (like mobile phone SMS texting).
Note, however, that although this too certainly could be useful in some ways,
many types of meeting cannot handle this: imagine a hundred people all sending
in text responses: no-one (neither audience nor presenter) can scan a hundred
different text messages and summarise or cluster them usefully. A feature of
voting (i.e. of MCQs) is that summarising is easy: just count the votes for each
alternative and present these five or so numbers. This is a fundamental
advantage for large groups of more than about six people (say). So voting is a
feature not a limitation for such groups. Of course other kinds of interaction
are organised round free-text: email, blogs, discussion fora, etc. So we need
a term for these that contrasts with voting, but covers all the free-text
group electronic communication systems -- perhaps "texting". A third
alternative is passing around other material e.g. software, as in a classroom
or lab with networked computers.
[1 vs. 2 way]
To technologists, a huge difference is equipment that offers 1-way vs. 2-way
communication (e.g. feedback lights or a little screen on each handset).
However to users, this is about as unimportant as whether the person you are
talking to says "yes" (2-way) or nods (1-way for a sonic technologist, but
2-way in terms of human communication).
All the equipment relies on fast feedback, but some do this by projecting
information on a big screen for all to read together.
[Decision support vs. establishing mutual knowledge of the spread
of opinions]
Furthermore the applications are less about making group decisions (at least
with the voting technology) and more about coordinating group thinking and
understanding by giving everyone an overview of what and how strong the
consensus or disagreement is. These distinguish it from formal voting for
political candidates or in shareholder meetings: more synchronous than
asynchronous; more about establishing mutual knowledge of the varieties of
opinion than reaching a final decision.
[personal vs. subgroup voting]
Another issue is whether every audience member has their own handset and vote,
or whether they agree a group vote i.e. one vote per small group.
[Face to face vs. online, "virtual"]
The main application I'm interested in is face to face, but actually it could
perfectly well be done online (but synchronously) (though the equipment might
be different). And one of the areas we are exploring at Glasgow is moving
MCQs and associated discussion between the web out of class, and EVS in class
as seamlessly as possible.
[Education vs. other applications]
The applications I am interested in are educational, but many sets of the same
technology are sold to business for meetings for planning, brain-storming etc.
That's what is wrong for some audiences in saying "classroom EVS".
"Group decision support system" is a term sometimes used for the business, not
educational, applications.
Technological distinctions that can matter are:
I've started to standardise on the term "EVS", although perhaps "synchronous electronic polling equipment (SEPE)" would really be even more exact.
Last changed
15 Feb 2005 ............... Length about 800 words (7,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/il.html.
A summary or introductory page on interactive lectures.
Because there is no point in having lectures or class meetings UNLESS they are interactive. Lectures may have originated before printing, when reading a book to a class addressed what was then the bottleneck in learning and teaching: the number of available books. Nowadays, if one-way monologue transmission is what's needed, then books, emails, tapes will do that, and do it better because they are self-paced for the learner. [The negative way of putting it.]
In fact it is not enough to be different: it should be better than the alternatives. Learners are routinely much more interactive with the material when using books (or handouts) than they can be with lectures: they read at their own pace, re-read anything they can't understand, can see the spelling of peculiar names and terms, ask other students what a piece means, and carry on until they understand it rather than until a fixed time has passed. All of these ordinary interactive and active learning actions are impossible or strongly discouraged in lectures.
So for a lecture to be interactive in a worthwhile sense, what occurs must depend on the actions of the participants (not merely on a fixed agenda), and benefit learning in ways not achieved by, say, reading a comparable textbook.
Another method is to use a voting system: put up a multiple choice question, have all the audience give an anonymous answer, and immediately display the aggregated results.
Another method is "Just in time teaching", where students are required both to read the material and to submit questions on it in advance, thus allowing the contact time to be spent on what they cannot learn for themselves.
In fact there are many methods.
The general benefits, and specific pedagogic issues, are very similar regardless of the technique used. I have written about them in a number of different places including:
The key underlying issues, roughly glossed by the broad term "interactivity", probably are:
Last changed
20 Feb 2005 ............... Length about 300 words (3,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/why.html.
What are the pedagogical benefits / aims?
To "engage" the students i.e. not only to wake them up and cheer them up,
but to get their minds working on the subject matter, and so to prompt
learning.
How specifically?:
But above all, realise from the start that there are powerful benefits not just for learners but also for teachers. Both need feedback, and both do much better if that feedback is fast and frequent -- every few minutes rather than once a year. So the other great benefit of using EVS is the feedback it gives to the lecturer, whether you think of that as like course feedback, or as allowing "contingent teaching" i.e. adapting how the time is spent on rather than sticking to a rigid plan that pays no attention to how this particular audience is responding.
Last changed
31 Jan 2005 ............... Length about 500 words (5,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/handsetintro.html.
This is a brief introduction to the technique of using EVS (electronic voting systems) for interaction in lectures. (A complementary technique is the one minute paper which uses open-ended audience input. An introduction to interactive lectures and why attempt them is here.)
The technique is much as in the "Ask the audience" lifeline in the TV show "Who wants to be a millionaire?". A multiple choice question (MCQ) is displayed with up to 10 alternative response options, the handsets (using infrared like domestic TV remote controls) distributed to each audience member as they arrive allow everyone to contribute their opinion anonymously, and after the specified time (e.g. 60 seconds) elapses the aggregated results are displayed as a barchart. Thus everybody sees the consensus or spread of opinion, knows how their own relates to that, and contributes while remaining anonymous. It is thus like a show of hands, but with privacy for individuals, more accurate and automatic counting, and more convenient for multiple-choice rather than yes/no questions.
It can be used for any purpose that MCQs can serve, including:
At Glasgow University we currently use the PRS equipment: small handheld transmitters for each audience member, some receivers connected to a laptop up front, itself connected to a data projector and running the PRS software. This equipment is portable, and there is enough for our largest lecture theatres (300 seats). Given advance organisation, setting up and packing up can be quick. We can accommodate those who normally use OHPs, powerpoint, ad hoc oral questions, or a mixture.
More practical details are offered here, and more details of how to design and use the questions are available through the main page, e.g. here.
Fig.1 Infrared handset transmitter
Fig.2 A receiver
Fig.3 The projected feedback during collection, showing handset ID numbers
Fig.4 Display of aggregated responses
Last changed
9 Sept 2007; 2 Sept 2023
............... Length about 2000 words (17,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/resources/minute.html.
The basic idea is that at the end of your session (e.g. lecture) you ask students to spend one minute (60 seconds) writing down their personal (and anonymous) answer to one or two questions, such as "What was least clear in this lecture?". Then you collect the scraps of paper and brood over them afterwards, possibly responding in the next session. It's wonderful because it takes only a minute of the students' time (each), requires no technology or preparation, but gives you immediate insight into how your class is doing. There are probably other benefits too.
That is the short version, which is all you really need to give it a try out. Trying it out is probably, if it is at all possible, the best second step in understanding the technique. However when you want more information, theory, and examples, then the rest of this document offers some.
Credit might go to: The "minute paper" has long been ascribed to Wilson as he was apparently the first to describe it in the literature: R.C.Wilson "Improving faculty teaching: Effective use of student evaluations and consultants" J. Higher Educ. vol.57 pp.192-211 (1986). More recently, it has been acknowledged that the original source of the idea was Berkeley physicist C. Schwartz. See Barbara Gross Davis, Lynn Wood, and Robert C. Wilson, A Berkeley Compendium of Suggestions for Teaching with Excellence (University of California, Berkeley) (1983) available at http://teaching.berkeley.edu/compendium/suggestions/file95.html See equally http://www-writing.berkeley.edu:16080/wab/2-2-gone.htm
I am addressing this note to teachers like myself: what they might do, and why. However a student could usefully read this, and carry it out privately. They could then use what they write for these one minute papers a) as a useful study habit; b) as a procedure for generating a question to ask as part of their good practice in being a student.
Although your first uses are likely to be generic, if you use it regularly you can focus it to your particular concerns that day for that class, by designing questions with respect to the learning objectives, or important disciplinary skills, or the sequence of development important for that course.
Many are best announced at the start but written at the end i.e. "At the end I am going to ask you to write for a minute on ....". This should promote more thinking during the class.
In asking each question, don't forget to specify the "rubric" i.e. state what kind of response is required e.g.
Many questions can be fitted under both of two contrasting types e.g. asked either as MCQs or as one-minute open ended papers; or be both reflective and about testing content retention.
You can get, if you wish, still more precise information by focussing the question you ask e.g. on a learning objective from the course, on a specific skill you think important to the discipline, etc. In other words, as an evaluation technique, it can be sensitive to context, to the discipline, to the course, to a particular (perhaps unusual) session. But also, it can be completely open-ended, and detect the surprises the teacher would never have thought to ask about (e.g. "I had no idea my graphs were not self-explanatory").
Above all, they can be used to get learners to:
Aspects of this, and of how this technique contributes and can succeed at this, are:
The handsets give an immediate shared group response, and so can move the dialogue forward faster (every 5 minutes rather than once per session). However one-minute papers are better at uncovering complete surprises (students saying things it didn't occur to the teacher to put as an optional response in an MCQ); and at giving you a chance to think about each answer even if it does take you by surprise.
Last changed
24 Feb 2005 ............... Length about 4,000 words (29,000 bytes).
(Document started on 15 Feb 2005.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/td.html.
You may copy it.
How to refer to it.
By Steve Draper, Department of Psychology, University of Glasgow.
Yes. Here are several methods of replacing exposition and using the face to face large group "lecture" periods for something else.
Should we expect to believe the reports of success with these methods, and should we expect them to generalise to many subjects and contexts? Again the answer is yes, which I'll arrive at by considering various types of theoretical analysis in turn.
Re-expression by learners (Laurillard activity 2) is achieved in peer discussion in the MacManaway and Interactive Engagement schemes, and by the quizzes in the OU and JITT schemes. Feedback on correctness (Laurillard activity 3) is provided by peer responses in the IE schemes and by the quiz in the JITT and IE schemes. Remediation more specifically targeted at student problems by the teacher (a fuller instantiation of Laurillard activity 3) is provided in the JITT scheme (because class time is given to questions sent in in advance), and often in the IE schemes in response to the voting results.
Thus in terms of the Laurillard model, instead of only covering activity 1 as a strictly expository lecture does, these schemes offer some substantial provision of activities 2,3 and 4 in quantities and frequency approaching that allocated to activity 1, while using only large group occasions and without extra staff time.
The schemes discussed here (apart from the OU) do not neglect this aspect, so again we can expect them to succeed on these grounds. They do not abolish classes, so management and administrative functions can be covered there as before. In fact the quizzes and to some extent the peer discussion offer better information than either standard lectures, a textbook or lecture script about how a student is doing both in relation to the teacher's expectations and to the rest of the class. They also do this not just absolutely (do you understand X which you need to know before the exam) but in terms of the timeline (you should have understood this by today).
In addition to this, these schemes also give much superior feedback to the teacher about how the whole course is going for this particular class of students. This equally is part of the management layer. However standard lectures are never very good for this. While a new, nervous, or uncaring lecturer may pick up nothing about a classes' understanding, even a highly skilled one has difficulty since at best the only information is a few facial expressions and how the self-selected one student answers each question from the lecturer. In contrast most of the above methods get feedback from every student, and formative feedback for the teacher is crucial to good teaching and learning. What I have found in interviewing adopters of EVS is that while many introduced it in order to increase student engagement, the heaviest users now most value the way it keeps them in much better touch with each particular class than they ever had without it.
This formative feedback to teachers is important for debugging an exposition they have authored, but is also important for adapting the course for each class, dwelling on the points that this particlar set find difficult.
Despite Hake, we should not ignore the fact that "inspiration" is indeed an important factor: a pervasive characteristic of humans is to pay attention to what others are paying attention to, and if one person is enthusiastic that influences us; conversely, we are less likely to buy from someone who can't show any enthusiasm for what they are selling (whether material goods or intellectual ones). This is clearly important in religion, in commerce, in entertainment, in science. But it is not clear that face to face contact is particularly special as a medium for passing on this information about enthusiasm: on the contrary, we know the written word has been and is important for this in all those spheres. We should also consider reports of being inspired by individuals e.g. Ghandi, Mother Teresa. Frequently this is in fact done not by meeting the person but reading about them. Personal inspiration can be by written medium, not only by face to face contact. When I think of whom I most admire, it is people I have read about, not met.
In any case, the schemes for transforming lectures discussed here still have face to face classes. In other words, inspiration is no argument for lectures: firstly, inspiration is much less important than effective teaching methods; secondly, for millennia it has been transmitted by other media as well; and thirdly in any case it can be transmitted in repurposed face to face classes that are not devoted to exposition.
Thus we can replace some or all exposition in lectures. Furthermore, we can re-purpose these large group meetings to cover other learning activities significantly better than usual. We can feel some confidence in this by a careful analysis of the functions covered by traditional lectures, and the ones thought important in general, and show how these are each covered in proposed new teaching schemes. This in turn leads to two further issues to address.
Firstly: which functions can in fact be effectively covered in large group teaching with the economies of scale that allows, and which others must be covered in other ways? Besides exposition, and the way the schemes above address Laurillard's activities 1 to 4, other functions that can be addressed in large groups in lecture theatres include:
Secondly, some aspects of a course can use large group teaching (see above), but all the rest must be done in smaller groups. How small, and how to organise them? One of the most interesting functions to notice is that many of the schemes above use peer discussion, coordinated by the teacher but otherwise not supervised or facilitated by staff. For this the effective size is no more than 5 learners, and 2 or 4 may often be best. Both our experience and published research on group dynamics and conversation structures support this. Instead of clinging to group sizes dictated either by current resources or by what staff are used to (which often leads to "tutorial" group sizes of 6, 10, or 20), we should consider what is effective. When the learning benefit is in the student generating an utterance, then 2 is the best size, since then at any given moment half the students are generating utterances. Where spontaneous and flowing group interaction is required, then 5 is the maximum number. For creating and coordinating a community, then it can be as large as you like provided an appropriate method is used e.g. using EVS to show everyone the degree of agreement and diversity on a question, or having the lecturer summarise written responses submitted earlier.
However forming groups simply by dividing the number of students by the number of staff is a foolish administrative response, not a pedagogic one. What is the point of groups of 10 or 20? Not much. If the model is for a series of short one to one interactions (which may be relevant for pastoral and counselling functions), then consider how to organise this. Putting a group of students in the same room is obviously inappropriate for this, and ICT makes this less and less necessary. If the model is for more personalised topics e.g. all the students with trouble over subtopic X go to one group, then we need NOT to assign permanent groups, but should organise ad hoc ones based on that subtopic. In general, what the schemes above suggest for the future is to consider a course as involving groups of all sizes, not necessarily permanent, not necessarily supervised; and organised in a variety of ways, including possibly pyramids and unsupervised groups. This is after all only an extension of the eternal expectation that learners will do some work alone: the ultimate small unsupervised group.
In the end, we should consider:
Dufresne, R.J., Gerace, W.J., Leonard, W.J., Mestre, J.P., & Wenk, L. (1996) Classtalk: A Classroom Communication System for Active Learning Journal of Computing in Higher Education vol.7 pp.3-47 http://umperg.physics.umass.edu/projects/ASKIT/classtalkPaper
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand student survey of mechanics data for introductory physics courses. American Journal of Physics, 66, 64-74.
R.R. Hake (1991) "My Conversion To The Arons-Advocated Method Of Science Education" Teaching Education vol.3 no.2 pp.109-111 online pdf copy
Hunt, D. (1982) "Effects of human self-assessment responding on learning" Journal of Applied Psychology vol.67 pp.75-82.
Laurillard, D. (1993), Rethinking university teaching (London: Routledge)
MacManaway,M.A. (1968) "Using lecture scripts" Universities Quarterly vol.22 no.June pp.327-336
MacManaway,M.A. (1970) "Teaching methods in HE -- innovation and research" Universities Quarterly vol.24 no.3 pp.321-329
Mazur, E. (1997). Peer Instruction: A User’s Manual. Upper Saddle River, NJ:Prentice-Hall.
Meltzer,D.E. & Manivannan,K. (1996) "Promoting interactivity in physics lecture classes" The physics teacher vol.34 no.2 p.72-76 especially p.74
Novak,G.M., Gavrin,A.D., Christian,W. & Patterson,E.T. (1999) Just-in-time teaching: Blending Active Learning and Web Technology (Upper Saddle River, NJ: Prentice- Hall)
Novak,G.M., Gavrin,A.D., Christian,W. & Patterson,E.T. (1999) http://www.jitt.org/ Just in Time Teaching (visited 20 Feb 2005)
Resnick,L.B. (1989) "Introduction" ch.1 pp.1-24 in L.B.Resnick (Ed.) Knowing, learning and instruction: Essays in honor of Robert Glaser (Hillsdale, NJ: Lawrence Erlbaum Associates).
Last changed
15 Oct 2009 ............... Length about 1700 words (13,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/local.html.
This page is about the use of EVS (electronic voting systems) in lectures at Glasgow University. It was written a few years ago, and assumes the use of the old IR equipment; though most of the rest of the advice is still reasonable. More up to date advice about use of the current equipment here.
To date, student response, and lecturers' perceptions of that, have been almost entirely favourable in an expanding range of trials here at the University of Glasgow (to say nothing of those elsewhere) already involving students in levels 1,2,3 and 4, and diverse subjects (psychology, medicine, philosophy, computer science, ...), and in sequences from one-off to every lecture for a term.
The equipment is mobile, and so can be used anywhere with a few minutes setup. It additionally requires a PC (laptops are also mobile, and we can supply one if necessary), and a data projector (the machine for projecting a computer's displayed output on to a big screen).
In principle, the equipment is available for anyone at the university to use, and there is enough for the two largest lecture theatres to be using it simultaneously. In practice, the human and equipment resources are not unlimited, and advance arrangements are necessary. We can accommodate any size audience, but there is a slight chance of too many bookings coinciding for the equipment, and a considerable chance of us not having enough experienced student assistants available at the right time: that is the currently scarcest resource.
My current view is that there are three main kinds of educational gain available here:
If it's one of mine you needn't ask, just turn up; and probably other users feel the same. We are none of us expert, yet we all seem to be getting good effects and needn't feel defensive about it. It usually isn't practicable to get 200 students to provide an audience for a realistic demonstration: so seeing a real use is the best option.
One way of introducing a new audience to the EVS is described here.
There are several alternative modes you could use this in.
[ Long past bookings Past workshops for prospective users (Past uses) Interim evaluation report ]
Last changed
3 Nov 2010 ............... Length about 1600 words (10,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/start.html.
Here is one possible way of introducing the EVS (electronic voting system), and in particular the PRS IR (infra red) equipment) to a new audience. Below is the script for you, the presenter, to act on; and below that, a slide to use.
Assuming the handsets have been distributed, and the time (not necessarily the start of the session) has now come to use or comment on them.
[Make the next points while the voting is going on.]
A. Check handset is turned on -- green light on?
B. Turn it over and read the 3 digit ID number
C. Point at a receiver (small box with red light on)
D. Press the number of your choice
-- see your ID come up on the screen
In almost every group we have run, about 1 in 50 of the audience fail to get it to work for them despite considerable effort. However we have failed to identify a pattern, either of the type of person or the type of problem. Furthermore hardly anyone ever asks for help (they are seeing hundreds around them succeed without effort) until they have been explicitly asked several times. Even though it feels like it's holding up the whole session, it is really only a few more minutes. Just keep asking until the total distinct handset IDs counted on the screen display matches your count of the people/handsets handed out. Keep asking, search the audience with your eyes, run up and down the aisles (carrying a spare handset or two) to attend to whoever lets slip they have a problem. It may be anything, or even something you can't fix: but usually it's turning the handset on, a handset battery being flat, not pointing the handset at a receiver (but at the screen, or into the head of the person in front of them); not being able to recognise their ID number on the screen.
Last changed
25 Jan 2003 ............... Length about 300 words (3,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/question.html.
What is involved in presenting each question?
Last changed
6 June 2004 ............... Length about 300 words (2500 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/length.html.
How many questions? How long do they take?
A rule of thumb for a 50 minute lecture is to use only 3 EVS questions.
In a "tutorial" session organised entirely around questions, you could at most use about 12 if there were no discussion: 60 secs to express a question, 90 secs to collect votes, 90 secs to comment briefly on the responses gives 4 minutes per question if there is no discussion or detailed explanation, and so 12 questions in a lecture.
Allowing 5 mins (still very short) for discussion by audience and presenter of issues that are not well understood would mean only 5 such questions in a session.
It is also possible, especially with a large collection of questions ready, to "use up" some by just asking someone to shout out the answer to warm up the audience, and then vote on a few to make sure the whole audience is keeping up with the noisy few. It would only take 20 seconds rather than 4 minutes for each such informal use of a question. Never let the EVS become too central or important: it is only one aid among others.
Thus for various reasons you may want to prepare a large number of questions from which you select only a few, depending on how the session unfolds.
Last changed
13 April 2022 ............... Length about 1500 words (12,000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/qdesign.html.
There is a whole art to designing MCQs (multiple choice questions). Much of the literature on this is for assessment. In this context however we don't much care (as that literature does) about fairness, or discriminatory power, but instead will concentrate on what will maximise learning.
Here I just discuss possible formats for a question, without varying the
purpose or difficulty. I was in part inspired by
A common type of MCQ concerns one relationship
e.g. (using school chemistry as an example domain)
"What is the chemical symbol for gold: Ag, Al, Au, Ar ?"
Applied to statistics this might be:
The idea is to require students to access knowledge of a topic from several
different starting points. Here I exercised three kinds of link, and each
kind in both directions. Exercising these different types and directions of
link is not only important in itself (because understanding requires
understanding all of these) but keeps the type of mental demand on the
students fresh, even if you are in fact sticking on one topic.
The first is that of linking ideas or concepts to particular examples or
instances of them e.g. is a whale a fish or a mammal? Another form of this
is linking (engineering or maths) problems with the principle or rule that is
likely to be used to solve it.
However both concepts and instances are represented in more than one way, and
practice at these alternative representations and their equivalences is
usually an essential aspect of learning a subject. Thus concepts usually have
both a technical name, and a definition or description, and testing this
relationship is important. Similarly instances usually have more than one
standard method of description and, although these are specific to each
subject, learners need to master them all, and questions testing these
equivalences are important. In teaching French language, both the spelling,
the pronounciation, and the meaning of a word need to be learned.
In statistics, an example
data set should be represented by a graph, a table of values, as well as a
description such as "bell shaped curve with long tails". In chemistry, the
name "copper sulfate" should be linked to "CuSO4" and a photograph of blue
crystals, and questions should test these links. (See Johnstone, A.H. (1991)
"Why is science difficult to learn? Things are seldom what they seem"
Journal of computer assisted learning vol.7 no.2 pp.75-83 for an
argument related to this based in teaching Chemistry.
See also Roy Tasker's group:
http://visualizingchemistry.com/research.)
These relationships are all bidirectional, so questions can (and should) be
asked in both directions e.g. both "which of these is a mammal" and "to which
of these categories do dolphins belong?". Thus a subject with three standard
representations for instances plus concept names and concept definitions will
have five representations, and so 20 types of question (pick one of five for
the question, and one of the remaining four for the response categories).
Additional variations come from allowing more than one item as an answer, or
asking the question in the negative e.g. "which of these is not a mammal?:
mouse, platypus, porpoise?".
The problem of technical vocabulary is a general
one, and suggests that the concept name-definition link should be treated
especially carefully. If you ask questions that are problems (real-world
cases) and ask which concept applies but use only the technical names of the
concepts, then students must understand perfectly both concept and the
vocabulary; and if they get it wrong you don't know which aspect they got
wrong. Asking concept-case questions using not technical vocabulary but
paraphrased descriptions of the concepts can separate these; and separate
questions to test name-definition (i.e. concept vocabulary).
It may or may not be a good idea to include null responses as an option.
Against offering them is the idea that you want to force students to commit to
an answer rather than do nothing, and also the observation that when provided
usually few take the null option, given the anonymity of entering a guess.
Furthermore, a respondent could simply not press any button; although that,
for the presenter, is ambiguous between a decision rejecting all the
alternatives, the equipment giving trouble to some of the audience, or the
audience getting bored or disengaged. However if you do include them as
standard, it may give you better, quicker feedback about problems. In fact
there are at least three usually applicable distinct null options to use:
An extension of this are:
Assertion-reason questions.
Russell, Mark (2008) "Using an electronic voting system to enhance learning
and teaching" Engineering Education vol.3 no.2 pp.58-65
doi:10.11120/ened.2008.03020058
Last changed
13 April 2022 ............... Length about 4,000 words (29,000 bytes).
EVS questions may be used for many pedagogic purposes.
These can be classified in an abstract way: discussed at length
elsewhere and summarised here:
In theory, I might bet that using CBM would work as well for (deep) learning
INSTEAD of Mazur's PI.
I believe both work the same way in learners: forcing them to think about
whether they are sure of their answer, and then self-correcting by thinking up
reasons for and against it. See:
However pedagogic uses are probably labelled rather differently by practising
lecturers, under phrases like "adding a quiz", "revision lectures",
"tutorial sessions", "establishing pre-requisites at the start",
"launching a class discussion". This kind of category is more apparent in the
following sections and groupings of ways to use EVS.
Putting up arguments or descriptions for criticism may be motivating as well
as useful (e.g. describe a proposed experiment and ask what is faulty about
it). It allows students to practise criticism which is useful; and criticism
is easier than constructive proposals which, in effect, is what they are
exclusively asked for in most "problem solving" questions, and so questions
asking for critiques may be a better starting point.
Thus in extending beyond a few SAQs, presenters may like to vary their
question types with a view to encouraging a better atmosphere and more
light hearted interaction.
This approach could be used, for instance, in:
The general benefit is that peer discussion requires not just deciding on an
answer or position (which voting requires) but also generating reasons for and
against the alternatives, and also perhaps dealing with reasons and objections
and opinions voiced by others. That is, although the MCQ posed only directly
asks for an answer, discussion implicitly requires reasons and reasoning, and
this is the real pedagogical aim. Furthermore, if the discussion is done in
small groups of, say, four, then at any moment one in four not only one in the
whole room is engaged in such generation activity.
There are two classes of question for this: those that really do have a right
answer, and those that really don't. (Or, to use Willie Dunn's phrase, those
that concern objects of mastery and those that are a focus for speculation.)
In the former case, the question may be a "brain teaser" i.e. optimised to
provoke uncertainty and dispute (see below). In the latter case, the issue to
be discussed simply has to be posed as if it had a fixed answer, even though
it is generally agreed it does not: for instance as in the classic debate
format ("This house believes that women are dangerous."). Do not assume that a
given discipline necessarily only uses one or the other kind of question. GPs
(doctors), for instance, according to Willie Dunn in a personal note, "came to
distinguish between topics which were a focus for speculation and those which
were an object of mastery. In the latter the GPs were interested in what the
expert had to say because he was the master, but with the other topics there
was no scientifically-determined correct answer and GPs were interested in
what their peers had to say as much as the opinion of the expert, and such
systems [i.e. like PRS] allowed us to do this."
Slight differences in format for discussion sessions have been studied:
Nicol, D. J. & Boyle, J. T. (2003)
"Peer Instruction versus Class-wide Discussion in large classes:
a comparison of two interaction methods in the wired classroom"
Studies in Higher Education.
In practice, most presenters might use a mixture and other variations.
The main variables are in the number of (re)votes, and the choice or mixture
of individual thought, small group peer discussion, and plenary or whole-class
discussion. While small group discussion may maximise student cognitive
activity and so learning, plenary discussion gives better (perhaps vital)
feedback to the teacher by revealing reasons entertained by various learners,
and so may maximise teacher adaptation to the audience.
The two leading alternatives are summarised in this table (adapted from Nicol
& Boyle, 2003).
The difference is only that in the SAQ case the presenter may be focussing on
finding weak spots and achieving remediation up to a basic standard whether the discussion is done by the presenter or class as a whole, while in the
discussion case, the focus may be on the way that peer discussion is engaging
and brings benefits in better understanding and more solid retention
regardless of whether understanding was already adequate.
Nevertheless optimising a question for diagnosing what the learners know
(self-assessment questions), and optimising it for fooling a large proportion
and for initiating discussion are not quite the same thing. There are
benefits from initiating discussion independently of whether this is the
most urgent topic for the class (e.g. promoting the practice of peer
interaction, generating arguments for an answer probably improves the
learner's grasp even if they had selected the right answer, and is more
related to deep learning, and promotes their learning of reasons as well as of
answers, etc.).
Some questions seem interesting but hard to get right if you
haven't seen that particular question before. Designing a really good brain
teaser is not just about a good question, but about creating distractors i.e.
wrong but very tempting answers. In fact, they are really paradoxes: where
there seem to be excellent reasons for each contradictory alternative. Such
questions are ideal for starting discussions, but perhaps less than optimal
for simply being a fair diagnosis of knowledge.
In fact ideally, the alternative answers should be created to match common
learner misconceptions for the topic. An idea is to use the method of
phenomenography to collect these misconceptions: the idea here would be to
then express the findings as alternative responses to an MCQ.
Great brain teasers are very hard to design, but may be collected or borrowed,
or generated by research.
Here's an example that enraged me in primary school, but which you can
probably "see through".
Here is one from Papert's Mindstorms p.131 ch.5.
Another example on the topic of Newtonian mechanics can be paraphrased as
follows.
Another one (taken from the book "The Tipping Point") can be expressed:
Brain teasers seem to relate the teaching
to students' prior conceptions, since tempting answers are most often those
suggested by earlier but incorrect or incomplete ways of thinking.
Whereas with most questions it is enough to give (eventually) the right answer
and explain why it is right, with a good brain teaser it may be important in
addition to explain why exactly each tempting wrong answer is wrong.
This extra requirement on the feedback a presenter should produce is discussed
further here.
Finally, here is an example of a failed brain teaser.
"Isn't it amazing that our legs are exactly the right length to reach the
ground?" (This is analogous to some specious arguments that have appeared in
cosomology / evolution.) At the meta-level, the brain teaser or puzzle here is
to analyse why that is tempting to anyone; something to do with starting the
analysis from your seat of consciousness in your head (several feet above the
ground) and then noticing what a good fit from this egocentric viewpoint your
legs make between this viewpoint and the ground.
May need a link here on to the page seq.html about designing sequences with/of
questions. And on from there to lecture.html.
If this can be made to work pedagogically, socially, and technically then it
would be a unique exploitation of e-learning with the advantages of face to
face campus teaching; and would be expected to enhance learning because so
much is simply proportional to the time spent by the learner thinking: so any
minutes spent on real discussion outside class is a step in the right direction.
Simply give the prediction in the question, and ask which of the offered
reasons are the right or best one(s); or which of the offered bits of evidence
actually support or disconfirm the prediction.
For instance, in teaching the part of perception dealing with visual
illusions, the presenter could put up the illusion together with a question
about how it is seen, and the audience will then see the proportion of the
audience that "saw" the illusory percept, and compare what they are told,
their own personal perceptual experience, and the spread of responses in the
audience.
In a practical module in psychology supported by lectures, Paddy O'Donnell and I have had
the class design and pilot questionnaire items (questions) in small groups
on a topic such as the introduction and use of mobile phones, for which the
class is itself a suitable population. Each group then submited their items to
us, and we then picked a set drawing on many people's contributions to form a
larger questionnaire. We then used a session to administer that
questionnaire to the class, with them responding using the voting equipment.
But the end of that session we had responses from a class of about 100 to a
sizeable questionnaire. We could then make that data set available almost
immediately to the class, and have them analyse the data and write a report.
A final year research project has also been run, using this as the data
collection mechanism: it allowed a large number of subjects to be "run"
simultaneously, which is the advantage for the researcher.
In a class on the public communication of science, Steve Brindley has surveyed
the class on some aspects of the demonstrations and materials he used, since
they are a themselves a relevant target for such communciation and their
preferences for different modes (e.g. active vs. passive presentations) are
indicative of the subject of the course: what methods of presentation of
science are effective, and how do people vary in their preferences.
He would then begin the next lecture by re-presenting
and commenting on the data collected last time.
Last changed
6 Aug 2003 ............... Length about 1,600 words (10,000 bytes).
Besides the different purposes for questions (practising exam questions, collecting data for
a psychological study, launching discussion on topics without a right or
wrong answer), an independent issue is whether the session as a whole has a
fixed plan, or is designed to vary contingent (depending) on audience responses.
The obvious example of this is to use questions to discover any points where
understanding is lacking, and then to address those points. (While direct
self-assessment questions are the obvious choice for this diagnosis function,
in fact other question types can probably be used.) This is to act
contingently. By contingency I mean having the presenter NOT have a fixed
sequence of stuff to present, but a flexible branching plan, where which
branches actually get presented depends on how the audience answers questions
or otherwise shows their needs. There are degrees of this.
An example of this can be found in the box on p.74 of
Meltzer,D.E. & Manivannan,K. (1996) "Promoting interactivity in physics
lecture classes" The physics teacher vol.34 no.2 p.72-76.
It's a sample problem for a basic physics class at university, where a
simple problem is broken down into 10 MCQ steps.
Another way of looking at this is that of training on the parts of a skill
or piece of knowledge separately, then again on fitting them together into a
whole. Diagnostically, if a learner passes the test for the whole thing, we
can usually take it they know it all. But if not, then learning may be much
more effective if the pieces are learned separately before being put together.
Not only is there less to learn at a time, but more importantly feedback is
much clearer, less ambiguous if it is feedback on a single thing at a time.
When a question is answered wrongly by everyone, it may be a sign that too
much has been put together at once.
In terms of the lesson/lecture plan, though, there is a single fixed course of
events, although learners contribute answers at many steps, with the questions
being used to help all the learners converge on the right action at each step.
One case is when a question links instances (only) to technical terms e.g.
(in psychology) "which of these would be the most reliable measure?"
If learners get this wrong, you won't know if that is because they don't
understand the issues, or this problem, or have just forgotten the special
technical meaning of "reliable". In other words, a question may require
understanding of both the problem case, and the concepts, and the special
technical vocabulary. If very few get it right, it could be unpacked by
asking about the vocabulary separately from the other issues e.g. "which of
these measures would give the greatest test-retest consistency?".
This is one aspect of the problem of technical
vocabulary.
Another case of this was about the top level problem decomposition in
introductory programming. The presenter had a set of problems (each of which
requiring a program to be designed) {P1, P2, P3}. He had a set of standard
top level structures {S1,S2, ... e.g. sequential, conditional, iteration} and
the problem the students "should" be able to do is to select the right
structure for each given problem. To justify/argue about this means to
generate a set of reasons for {F1,F2, ...} and against {A1,A2...} each
structure for each problem. I suggest having a bank of questions to select
from here. If there are 3 problems and 5 top level structures then 2*3*5=30
questions. An example of one of these 30 would be a set of alternative
reasons FOR using structure 3 (iteration) on problem 2, and the question asks
the audience which (subset) of these are good reasons.
The general notion is, that if a question turns out to go too far over the
audience's head, we could use these "lower" questions to structure the
discussion that is needed about reasons for each answer. (While if everyone
gets it right, you speed on without explanation. If half get it right, you go
for (audience) discussion because the reasons are there among the audience.
But if all get it wrong, support is needed; and these further questions could
keep the interaction going instead of crashing out into didactic
monologue.)
Last changed
27 May 2003 ............... Length about 900 words (6000 bytes).
While the presenter may be focussing on finding the most important topics for
discussion and on whether the audience seems "engaged", part of what each
learner is doing is seeking feedback. Feedback not only in the sense of "how
am I doing?", though that is vital for regulating the direction and amount of
effort any rational learner puts in, but also in the sense of diagnosing and
fixing errors in their performance and understanding. So "feedback" includes,
in general, information about the subject matter, not just about indicators of
the learner's performance.
This can be thought about as levels of detail, discussed at length in
another paper,
but summarised here. A key point is that, while our image of ideal feedback
may be individually judged and personalised information, in fact it can be
mass produced for a large class to a surprising extent, so handset sessions
may be able to deliver more in this way than expected.
The last (5) is a separate item because the previous one (4) concerned only
correct principles, but this one (5) concerns misconceptions, and in general
negative reasons why apparent connections of this activity with other
principles are mistaken. Thus (4) is self-contained, and context-free; while
(5) is open-ended and depends on the learner's prior knowledge. This is only
needed when the learner has not just made a slip or mistake but is in the grip
of a rooted misconception -- but is crucial when that is the case. Well
designed "brain teasers" are of this kind: eliciting wrong answers that may be
held with conviction. Thus with mass questions that are forced choice, i.e.
MCQ, one can identify in advance what the wrong answers are going to be and
have canned explanations ready.
Here are two rough tries, applying to actual handset questions posed to an
introductory statistics class, at describing the kind of extra explanation
that might be desirable here. Their feature is explaining why the wrong
options are attractive, but also why they are wrong despite that.
Example1. A question on sample vs. population medians.
Example2. Regression Analysis: Reading versus Motivation
Which of the following statements are correct?
For more examples, see some of the examples of
brain
teasers, which in essence are questions especially designed to need this extra
explanation.
Last changed
21 Feb 2003 ............... Length about 700 words (5,000 bytes).
Any session or lecture can be thought of as having 3 aspects, all of which
ideally will be well managed. If you are designing a new kind of session
(e.g. with handsets) you may want to think about these aspects explicitly.
They are:
Last changed
21 Dec 2007 ............... Length about 200 words (3,000 bytes).
This page is to collect a few pointers to sets of questions that might be used
with EVS that are available on the web. Further suggestions and pointers are
welcome.
For first year physics at University of Sydney:
their webpage
and
a local copy to print off as one document.
The Galileo project
has some examples if you regester online with them.
The SDI
(Socratic dialog Inducing) lab has some examples.
?Roy Tasker
Last changed
15 Feb 2005 ............... Length about 900 words (6000 bytes).
There are basically three classes of evidence to consider:
ABRAHAMSON, A.L. (1998) An Overview of Teaching and Learning
Research with Classroom Communication Systems. Paper presented at the
International Conference of the Teaching of Mathematics,
Samos, Greece.
Last changed
23 Jul 2004 ............... Length about 900 words (6000 bytes).
This page is a place for other questions frequently asked by inquirers, but
not answered elsewhere in these web pages.
Last changed
25 June 2010 ............... Length about 7,000 words (49,000 bytes).
What are the alternative methods and technologies to the PRS handsets we've
bought? In fact this is all part of a wider set of choices.
Our own approach at Glasgow university adopted the position:
The ideal system for this would allow huge groups of students to register a
response to a MCQ (multiple choice question) with privacy (secret ballot), and
have the results immediately summarised, and the summary displayed.
Feedback to individual students (e.g. by LCDs on handsets) could be nice.
Best of all would be to escape the MCQ strategy and have open-ended input from
each audience member.
Besides these web pages containing our views, there are some other reports on
what technology to adopt:
For another view you could look at this
5 Aug 2005 news article by news.com, and
3 ads.
The article also reports that
"U.K market research firm DTC Worldwide, which tracks the global market for
education technology, expects that 8 million clickers ... will be sold
annually by 2008".
Thus there are now several alternative pieces of software you might use with the
same EVS handset hardware.
"WordPads work with the proprietary WordWall interactive whiteboard
interface, although you can run it as a Powerpoint overlay. It
supports a much wider range of interactions than other systems.
Multiple-choice tests can be run through the handset screen, so
students can answer at their own pace. It's also the only EVS to
support text contribution using phone-style predictive texting and
offers a multilingual interface in 12 different languages. Students
can also offer opinions via Lickert scales, definition match-up,
odd-one-outs and category quizzes. Students can use joypad controls to
indicate preference and there are also a variety of different game
modes.
Pads cost £28 each [as of May 2009], although discounts are available for
large sets. Although initially aimed at school classrooms, they have recently
added functionality to support larger, lecture-sized sets."
For that reason, I long assumed that EVS couldn't usefully do open ended text
input, because the presenter and audience couldn't do anything with it.
However Promethean have gone a considerable way to proving me wrong. Their
handsets allow text input similar to mobile phones, and crucially their
software supports the presenter in using it. One mode provides a set of 8?
blank boxes on the screen and as the words (or possibly phrases) come in, the
presenter uses the mouse to sort these into the boxes (thus grouping variant
spellings, synonyms etc. together); then a followup MCQ vote could be taken
directly from that screen with each box being an optional response. With an
audience of 30 this only takes a few seconds and works very well.
This allows a 2 phase student feedback quiz to be done very fast: you first
ask (free text mode) "what is the thing you struggle with most on the
course?", quickly group similar answers, then re-vote to check which really
is the top issue for the group.
Even with 300 something can be done: it provides a list of the received words
with a frequency count against each word. With big numbers it doesn't matter
losing a few percent to deviant spellings etc., or ignoring words that only
one person put in: you probably only want the popular (high frequency) ones
anyway.
However if you do want this function, then the most obvious method is to teach
in rooms or labs with a computer per student (or at least per group of
students); and use the network instead of infrared to interact. If the
computers use wireless networking, then the system could be mobile and
flexible. (See this discussion.)
Other specialised equipment however allows some of this.
The service provided seems to be:
First, the attractions are large: a speaker need only arrange and pay for the
service in advance, and have a live internet (WWW) connection in the lecture
theatre (actually still quite difficult and rare in my university), but they
can reasonably assume most of the audience will come with their "handset" i.e.
mobile phone, and no other equipment or setup is necessary on the spot.
PollEverywhere also says their software is integrated with PowerPoint.
However there are several issues, in fact drawbacks, with this.
Even if coverage could be obtained, SMS messages, unlike voice calls but like
emails, are not guaranteed to be instantaneous or to be delivered within any
specific time. Texts can be and in practice often are delayed, and there are
no prospects for an archictecture to allow text delivery for a vote within a
guaranteed time (e.g. 2 minutes; 2 seconds for the best EVS equipment). But
not only is this prompt delivery not guaranteed, it is frequently not
delivered as the only published trial (see below) showed.
Matt Jones (always@acm.org) has a paper on trying SMS mobile phone
text messaging in this way:
The paradox, or rather cleft stick, is that:
Mark Finn has
a journal paper reviewing projects to date that have used PDAs in teaching.
A comment suggested by John Cowan and in different words by Willie Dunn, is
that these systems represented a first effort towards engaging with learning
rather than teaching. Their importance was perhaps more that shift: and when the
equipment, or more generally feedback classrooms, were abandoned, it was as much
to take the underlying change in attitude further into new forms than a step
backwards.
They have had a string of other papers, mostly not in English, e.g.:
The system was used in four places, the largest room having 300 student
places:
The choice of one button was deliberate: "the psychologist Prof. Dr. Daniels
stressed the point that not more than one button per student should be used".
At least in its heyday, student enthusiasm brought official support for it.
However after 2000, it ceased to be used.
My information on this comes from the publications and personal communications
from the authors.
The original theoretical notion here was essentially control engineering: it
was to get better (and faster) feedback to the teacher. This turned out to be,
for him, the start of a line of research on learning and understanding e.g.
Hunt, D. (1982) "Effects of human self-assessment responding on learning"
Journal of Applied Psychology vol.67 pp.75-82.).
Bridgman, C. F. (1965) "Innovations in the Teaching of Anatomy."
American Journal of Veterinary Research vol.26 no.115 pp.1552-1561
The idea was that 4-choice MCQs would be posed via the OHP at 4-6 moments
within a lecture. Ideally each followed on from material presented in the
preceding 5-10 minutes, to test whether the listeners had sufficiently
absorbed the principles to be able to select the right answer to a deductive
follow-up question. As was standard with such MCQs, one wrong answer offered
was reasonable, the other two seriously off beam. At least five or six
lecturers used it quite extensively, and it was demonstrated at a meeting of
the Physiological Society.
The system was built out of second hand Post Office relays, and suffered
scuffing and other wear in the six or so years after the installers left,
and was eventually condemned on Health and Safety grounds, I guess a little
before 1980, and stripped out by contractors -- who, I believe, charged
more than the funds required by the two installers, who had merely covered
their S/H purchases. In fact unreliability was a problem (often a few of
the student sets would not be working on a given day.
Here's what Willie Dunn, who they say inspired their implementation, has
to say now about the rationale:
If the GPs are presented with a series of cases and make decisions about
actions, if the expert agrees with the decision made then he can move on
quickly until he finds an area of need, and his contributions become
problem-oriented as far as the GPs are concerned. And there is plenty
evidence supporting problem-oriented learning.
We had systems the operated with small groups: a 10-student version was
constructed and used before the lecture theatre was equipped (later with
computers), and the physiology adventure was an attempt to bring some of the
best in small group situations into a large group. I used to teach a class of
200 in the lecture theatre.
Later on we came to distinguish between topics which were a focus for
speculation and those which were an object of mastery. In the latter the GPs
were interested in what the expert had to say because he was the master, but
with the other topics there was no scientifically-determined correct answer
and GPs were interested in what their peers had to say as much as the
opinion of the expert, and such systems allowed us to do this.
A number of variations on this theme were developed. However we found that
the technology of the time was not reliable enough or cheap enough to continue
the hardware development at that time, and we switched to cheaper and more
reliable (but less flexible systems). But along came computer systems from
the mid-seventies and many of these ideas were incorporated into our work in
this field.
The lecture room, not theatre, was capable of accommodating about 50 students
and was fitted out to enable it to be used as a feedback classroom. This was
done by bolting electrical trunking, about 80mm x 40 mm, to the head of long
flat desks, large enough to take drawing boards when the room was used as a
drawing office. At each student position there was a hole in the trunking,
facing the student, fashioned with a porthole from a shop supplying parts for
model ships. On top of the trunking was a knob with pointer, which could be
turned to one of six or seven positions. The positions were labelled
alphabetically. Four were for MCQ responses, the others for things like "I
didn't understand this at all, and don't want to respond"; and there was also
an "Off" position, for an unused seat, or for use when someone could not
answer the question posed.
All of this was connected electronically to a simple control panel at the
front of the class, which enabled the lecturer to switch between options and
find out how many of the students were responding under each heading. The
lecturer clicked on the correct answer, and those who had returned it were
rewarded with a reasonably confidential light showing through the porthole.
It was (deliberately) not possible to identify which student was giving which
response. There was no public display, but public reporting by lecturer.
The apparatus was used to enable a lecturer to pose
multiple choice questions in class, give time for the students to choose
an answer, and obtain feedback about assimilation - and also the
attractiveness of the various distractors. The lecturer could then
respond accordingly - or as he thought likely to be suitable.
The main effects, which were not inconsiderable, were to reveal lack of
understanding beyond what the lecturer had anticipated, and enable
something to be done about it right away; to identify relatively common
misunderstandings; and to open up the dialogue so that the "lecture"
moved a little towards being a learning activity, in which students
reported understanding and puzzlement.
It was used mainly by the initiator, John Cowan, but 2-3 others explored it as
an option, rather than building it in. It had two possible disadvantages for
the "explorers" - it called for additional preparation, and it showed up
weaknesses in one's teaching of which one might otherwise be unaware.
It was used for about 5 years and then abandoned partly because the initiator
changed jobs and site, but mainly because
"I wasn't really into lecturing to classes any more".
John Cowan: "It originated in a visit to Jim Cowan (no relation) at Watsons
College (for schoolboys), through a chance contact arising from boys' club
work and a conversation about innovations in teaching and learning. I had
mentioned my desire to know what was being assimilated, rather than what I
delivered. He told me he had this commercial feedback classroom, and invited
me to go and see it in use. The theoretical idea was the first stirring of
wanting to know what learning was happening in activities for which I was
responsible."
It is mentioned in John Cowan's PhD ("The Feasibility of Resource-Based
Learning in Civil Engineering Education" Heriot-Watt University, 1975), but
otherwise unpublished.
Last changed
18 May 2003 ............... Length about 900 words (6000 bytes).
This page is about details that most lecturers using the PRS handset system
will never need or want to know. For the rest of you, ....
High/low confidence buttons
Last changed
8 Nov 2010 ............... Length about 2,000 words (23,000 bytes).
This page lists some sites and people I know of in the UK mainly in Higher
Education who are interested in classroom handsets, PRS, or similar
approaches to interactive teaching.
(To see why we are interested in this technique, and other information about
why you might be interested, look at the parent page
to this one.)
I suggest that if you are looking for a person or place, that you use your
browser's "Find" command to search this page for (part of) the name you
are interested in.
This page is organised firstly by (university) site with just a few key people
mentioned: it would not be practical to mention them all. The order is
idiosyncratic: expect to search using the Find command, not by scanning by
eye. This page just contains people I happen to know about: it is not likely
to be complete. If you would like to be added or removed, or if you can
suggest someone else who should be listed here, please email me
(s.draper@psy.gla.ac.uk) and I will
act promptly on your request. People have found it useful to discover who is
already interested in EVS or PRS near them, and conversely to advertise here
their interest to others in their institution or city. Also, any pointers to
papers and web documents on this would be gratefully received.
(PRS is used widely in some places outside the UK, including
Hong Kong University of Science and Technology,
UMass/Amherst, Rutgers, University of British Columbia, North Dakota,
and UC Berkeley. See also
for
mainly USA sites using PRS.)
Besides a central pool of mobile EVS equipment, available for any department
or teacher to use, Physics have bought their own set of kit for first year
lectures, Statistics have about 150 for their use, and Steve Brindley has a
set for sessions aimed at those outside the university. Modern Languages use
a set of 50 in smallish language teaching groups regularly; and a larger set
for level 1 and 2 classes.
Charles Higgins
in Education is acquiring 300 partly for external use.
Wolverhampton is also the home of the large
REVEAL project
involving EVS use largely in schools: using Promethean equipment;
and have reports available. Contacts there:
Andrew Hutchinson and Diana Bannister, who are part of a
learning technologies team.
An unconfirmed report says the
psychology group
also bought PRS equipment.
The
ExPERT centre has bought about 80 IML handsets. For more
information, contact
Lesley-Jane Reynolds.
Ray d'Inverno
in Maths has begun using PRS, and has
an early report and pedagogic rationale for PRS use.
Chemistry and Psychology have each purchased sets of 130+ TurningPoint RF
handsets, and the university a further 240
for wider experimentation.
David
Read is using them in schools outreach.
Gordon Campbell, professor of Renaissance Studies, uses them.
Last changed
13 Nov 2009.............Length about 3821 words (36,000 bytes).
This page is an ad hoc bibliography of papers about EVS. I expect to paste in
lists of references I come across without checking them: you use these at your
own risk, but they could be a useful starting point.
Please send in suggestions: great papers, useful papers, papers you have
written, corrections to entries already here. Attached word documents or HTML
is preferred (or I may not do the formatting). I will probably only include
either published journal articles and books or reports on the web;
and I may exclude anything without explanation.
A very few I have starred '**'. I think these are worth special attention.
See also
Will Thalheimer's annotated bibliography.
While mine is an uncritical heap to use as a starting point, his selects often
cited ones AND provides some critical appraisal.
ABRAHAMSON, A.L. (1998) An Overview of Teaching and Learning
Research with Classroom Communication Systems. Paper presented at the
International Conference of the Teaching of Mathematics,
Samos, Greece.
ANDERSON, T., HOWE, C., SODEN. R., HALLIDAY, J. & LOW, J. (2001).
Peer interaction and the learning of critical thinking skills in
further education students. Instructional Science, 29, 1-32.
Angelo, T.A., and K.P. Cross (1993). Minute Paper. In Classroom
Assessment Techniques: A Handbook for College Teachers, San
Francisco: Jossey-Bass, 148-153.
Annett, D. (1969): Feedback and Human Behaviour. New York,
Penguin.
** Banks,D.A. (ed.) (2006) Audience response systems in higher
education: Applications and cases (Information Science Publishing)
Barak, M., Lipson, A., & Lerman, S. (2006). Wireless laptops as
means for promoting active learning in large lecture halls.
Journal of Research on Technology in Education, 38(3), 245-263.
Beatty, I. D., Gerace, W. J., Leonard, W. J., & Dufresne, R. J.
(2006). Designing effective questions for classroom response system
teaching. American Journal of Physics, 74(1), 31-39.
N Beatty, I.D., Gerace, W.J., Leonard, W.KJ., Dufresne, R.J.,
Am. J. Phys xxx xxxx (2006)
Bjork, R. A., & Richardson-Klavehn, A. (1989). On the puzzling
relationship between environmental context and human memory. In C.
Izawa (Ed.) Current Issues in Cognitive Processes: The Tulane
Floweree Symposium on Cognition (pp. 313-344). Hillsdale, NJ:
Erlbaum.
Bligh, D. (2000), What’s the use of lectures? San
Francisco: Jossey-Bass.
Bloom’s Taxonomy : http://www.officeport.com/edu/blooms.htm
Bloom, B. S. (1956). Taxonomy of Educational Objectives, the
Classification of Educational Goals — Handbook I: Cognitive
Domain New York: McKay.
Boyle, J.T. & Nicol, D.J. (2003) "Using classroom communication
systems to support interaction and discussion in large class settings"
Association for Learning Technology Journal vol.11 no.3
pp.43-57
Bransford J. , Brophy S., Williams S. (2000) "When Computer
Technologies Meet the Learning Sciences - Issues and Opportunities"
Journal of Applied Developmental Psychology Volume 21,
Number 1, January 2000, pp. 59-84
Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How
people learn: Brain, mind, experience, and school. Washington, D.
C.: National Academy Press.
Brookfield, S. D. (1995), Becoming a critically reflective
teacher, San Francisco: Jossey-Bass.
Brown, J.S., Collins, A. & Duguid, P. (1989). Situated cognition
and the culture of learning. Educational Researcher. 18, 32-42.
Brumby, M.N. (1984), ‘Misconceptions about the concept of
natural selection by medical biology students’, Science
Education, 68(4), 493-503.
Bruner, J. (1985), ‘Vygotsky: a historical and conceptual
perspective’ in Wertsch, J.V. (ed), Culture, communication
and cognition. Cambridge: Cambridge University Press.
Burnstein, R. A. & Lederman, L. M. (2001). Using wireless keypads
in lecture classes. The Physics Teacher, vol.39 no.1 pp.8-11.
N CAL-laborate, UniServe Science International
Newsletter, 14 (2005) available online http://science.uniserve.edu.au/pubs/callab/vol14/cal14_bates.pdf
Carnaghan, C., & Webb, A. (2005). Investigating the effects of
group response systems on learning outcomes and satisfaction in
accounting education. Downloaded January 2007 from
www.learning.uwaterloo.ca/LIF/responsepad_june20051.pdf .
Carnaghan, C., & Webb, A. (2005). Investigating the effects of
group response systems on learning outcomes and satisfaction in
accounting education. Retrieved on January 1, 2007 from
www.learning.uwaterloo.ca/LIF/ responsepad_june20051.pdf
Champagne, A. B., Klopfer, L. E. and Anderson, J. H. (1980). Factors
influencing the learning of classical mechanics. American Journal
of Physics, 48, 1074-1079.
Chickering, Arthur W. and Gamson, Zelda F, Seven Principles for Good
Practice in Undergraduate Education -
http://aahebulletin.com/public/archive/sevenprinciples1987.asp
Christianson, R. G. & Fisher, K. M. (1999). Comparison of student
learning about diffusion and osmosis in constructivist and traditional
classrooms. International Journal of Science Education, 21,
687-698.
Cohen, E.G. (1994), ‘Restructuring the classroom: Conditions for
productive small groups’, Review of Educational Research,
64(1), 3-35.
Comlekci, T., Boyle, J.T., King, W., Dempster, W., Lee, C.K.,
Hamilton, R. and Wheel, M.A. (1999), ‘New approaches in
mechanical engineering education at the University of Strathclyde in
Scotland: I — Use of Technology for interactive teaching’,
in Saglamer, G.(ed), Engineering Education in the Third
Millenium, Leuchtturm- Verlag.
Cristle, C. A., & Schuster, J. W. (2003). The effects of using
response cards on student participation, academic achievement, and
on-task behavior during whole-class math instruction. Journal of
Behavioral Education, 12(3), 147-165.
Crombie, W. J. (2006). Harvard launches wireless classroom.
Harvard University Gazette (February 23, 2006), Downloaded
February 2007 from
www.news.harvard.edu/gazette/2006/02.23/05-eclassroom.html .
** Crouch, C.H. and Mazur, E. (2001), ‘Peer Instruction: Ten years
of experience and results’, American Journal of Physics,
69, 970-977
Crouch,C., A. Fagen, P. Callan & E. Mazur (2004) "Classroom
demonstrations: learning tools or entertainment" Am. J. Physics
vol.72 no.6 pp.835-838
Cutts,Q. Carbone,A., & van Haaster,K. (2004) "Using an Electronic
Voting System to Promote Active Reflection on Coursework Feedback" To
appear in Proc. of the Intnl. Conf. on Computers in Education
2004, Melbourne, Australia, Nov. 30th — Dec 3rd 2004.
Cutts,Q. Kennedy,G., Mitchell,C., & Draper,S.W. (2004)
"Maximising dialogue in lectures using group response systems"
Accepted for 7th IASTED Internat. Conf. on Computers and Advanced
Technology in Education, Hawaii, 16-18th August 2004
Cutts,Q.I. & Kennedy, G.E. (2005) "Connecting Learning
Environments Using Electronic Voting Systems" Seventh Australasian
Computer Education Conference, Newcastle, Australia. Conferences in
Research and Practice in Information Technology, Vol 42, Alison
Young and Denise Tolhurst (Eds)
Cutts,Q.I. & Kennedy,G.E. (2005) "The association between
students’ use of an electronic voting system and their learning
outcomes" Journal of Computer Assisted learning vol.21
pp.260–268
Davies, G. (1986). Context effects in episodic memory: A review.
Cahiers de Psychologie Cognitive, 6, 157-174.
Reversing the relationship
You can equally, and additionally, ask about the same relationship in reverse:
"Which metal is represented by the symbol 'Au'? Gold, silver, platinum, copper?"
Multiple types of relationship
When you have several relationships, the alternative question types multiply.
Consider these 3 linked pieces of information: a photo of a gold nugget or
ring; the word (name) "Gold"; and the symbol "Au". These 3 pieces of
information each have a relationship with the other 2, giving 3 types of
relationship; and each has 2 directions, giving 6 question types in all:
Types of relationship to exercise / test
In the abstract there are three different classes of relationship to test:
Further Response Options
The handsets do not directly allow the audience to specify more than one
answer per question. However you can offer at least some combinations
yourself e.g.
"Is a Black Widow:
Assertion-reason questions
I particularly commend asking MCQs that, instead of asking which fact is true,
ask which reason for a given fact is the right one. Covertly related questions:
Using 3 questions to make a strong test of understanding one concept
Mark Russell suggests using 3 (say) alternative questions all testing the same
key concept. With MCQs with 4 response options, 25% of students will get a
question right by accident if they answer at random: not a strong test.
He suggests having 3 alternative questions testing exactly the same concept,
and only students who get all 3 of these correct should be regarded as having
learned the concept.
The questions are tacitly linked (by being about the same concept), but not
listed adjacently and not using similar structure. He found that students who
did not have a sound understanding of the concept did not even recognise that
the 3 questions were linked: the disguise does not need to be elaborate
(contrary to expert / staff perceptions, who naturally see the 3 questions as
"about the same thing" exactly because they grasp the concept). Some references on MCQ design
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/qpurpose.html.
Pedagogical formats for using questions and voting
(written by
Steve Draper,
as part of the Interactive Lectures website)
Draper,S.W. (2009a) "Catalytic assessment: understanding how MCQs and EVS
can foster deep learning" British Journal of Educational Technology
vol.40 no.2 pp.285-293
doi: 10.1111/j.1467-8535.2008.00920.x
SAQs and creating feedback for both learner and teacher
Asking test questions, or "self-assessment questions" (SAQs) since only the
student knows what answer they gave individually, is useful in more than one
way.
A first cautious use of EVS
The simplest way to introduce some EVS use into otherwise conventional
lectures is to add some SAQs at the end so students can check if they have
understood the material. This is simplest for the presenter: just add
two or three simple questions near the end without otherwise changing the
lecture plan. Students who get them wrong now know what they need to work on.
If the average performance is worse than the lecturer likes, she or he can
address this at the start of the next lecture. Even doing this in a simple,
uninspired way has in fact consistently been viewed positively by students in
our developing experience, as they welcome being able to check their
understanding.
Extending this use: Emotional connotations of questions
If you put up an exam question, its importance and relevance is clear to
everyone and leads to serious treatment. However, it may reduce discussion
even while increasing attention, since to get it wrong is to "fail" in the
terms of the course. Asking brain teasers is a way of exercising the same
knowledge, but without the threatening overtones, and so may be more effective
for purposes such as encouraging discussion.Contingent teaching: Extending the role of questions in a session
Test questions can soon lead to trying a more contingent approach, where a
session plan is no longer for a fixed lecture sequence of material, but is
prepared to vary depending upon audience response. This may mean preparing a
large set of questions, those actually used depending upon the audience: this
is discussed in "designing a set of questions for a
contingent session".
Designing for discussion
Another important purpose for questions is to promote discussion, especially
peer discussion. A general format might be: pose a question and take an
initial vote (this gets each person to commit privately to a definite initial
position, and shows everyone what the spread of opinion on it is). Then,
without expressing an opinion or revealing what the right answer if any is,
tell the audience to discuss it. Finally, you might take a new vote, and see
if opinions have shifted.
Discussion recipes
"Peer Instruction":
Mazur Sequence "Class-wide Discussion":
Dufresne (PERG) Sequence
[voting]
Students provide individual responses (revised answer).
Questions to discuss, not resolve
Examples of questions to launch discussion in topics that don't have clear
right and wrong answers are familiar from debates and exam questions.
The point, remember, is to use a question as an occasion first to remind the
group there really are differences of view on it, but mainly to exercise
giving and evaluating reasons for and against. The MCQ, like a debate, is
simply a conventional provocation for this.
"Brain teasers"
Using questions with right and wrong answers to launch discussion is, in
practice, less showing a different kind of question to the audience and
more a different emphasis in the presenter's purpose. Both look like (and are)
tests of knowledge; in both cases if (but only if) the audience is fairly
split in their responses then it is a good idea to ask them to discuss the
question with their neighbours and then re-voting, rather than telling them
the right answer; in both cases the session will become more contingent: what
happens will depend partly on how the discussion goes not just on the
presenter's prepared plan; in both cases the presenter may need to bring a
larger set of questions than can be used, and proceed until one turns out to
produce the right level of divisiveness in initial responses."If a bottle of beer and a glass cost one pound fifty, and the beer
costs a pound more than the glass, how much does the glass cost?"
The trap seems to lie in matching the beer to one pound, the glass to fifty
pence, and being satisfied that a "more" relation holds.
"A monkey and a rock are attached to opposite ends of a rope that is hung over
a pulley. The monkey and the rock are of equal weight and balance one
another. The monkey begins to climb the rope. What happens to the rock?"
His analysis of why this is hard (but not complex) is: students don't have
the category of "laws-of-motion problem" like conservation of energy problem.
I.e. we have mostly learned Newton without having really learned the
pre-requisite concept of what IS a law of motion. Another view is that it
requires you to think of Newtons 3rd law (reaction), and most people can
repeat the law without having exercised it much.
Remember the old logo or advert for Levi's jeans that showed a pair
of jeans being pulled apart by two teams of mules pulling in opposite
directions. If one of the mule teams was sent away, and their leg of the jeans
tied to a big tree instead, would the force (tension) in the jeans be: half,
the same, or twice what it was with two mule teams?
The trouble here is how can two mule teams produce no more force than one team,
when one team clearly produces more than no teams; on the other hand, one mule
pulling one leg (while the other is tied to the tree) clearly produces force,
so a second mule team isn't necessary.
Take a large piece of paper, fold it over, then do that again and again a
total of 50 times. How tall do you think the final stack is going to be?
Somehow even those who have been taught better, tend think it will be about 50
times the thickness of a piece of paper, whereas really it is doubled 50 times
i.e. it will be 2 to the 50th power thicknesses, which is a huge number; and
probably comes out as about the distance from here to the sun. Extending discussion beyond the lecture theatre
An idea which Quintin is committed to trying out (again, better) from Sept.
2004 is extending discussion, using the web, beyond the classroom. The
pedagogical and technical idea is to create software to make it easy for a
presenter to ship a question (for instance the last one used in a lecture, but
it could be all of them), perhaps complete with initial voting pattern, to the
web where the class may continue the discussion with both text discussion and
voting. Just before the next lecture, the presenter may equally freeze the
discussion there and export it (the question, new voting pattern, perhaps
discussion text) back into powerpoint for presentation in the first part of
their next lecture.Direct tests of reasons
One of the main reasons that discussion leads to learning, is that it gets
learners to produce reasons for a belief or prediction (or answer to a
question), and requires judgements about which reasons to accept and which to
reject. This can also be done directly by questions about reasons. Collecting experimental data
A voting system can obviously be used to collect survey data from an audience.
Besides being useful in evaluating the equipment itself, or the course in
which it is used (course feedback), this is particularly useful when that data
is itself the subject of the course as it may be in psychology, physiology,
parts of medical teaching, etc.
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/contingent.html.
Degrees of contingency
(written by
Steve Draper,
as part of the Interactive Lectures website)Contents (click to jump to a section)
Implicit contingency
First are simple self-assessment questions, where little changes in the
session itself depending on how the audience answers, but the implicit hope
is that learners will (contingently i.e. depending on whether they got a
question right) later address the gaps in their knowledge which the questions
exposed, or that the teacher will address them later.Whole/part training
Secondly, we might present a case or problem with many questions in it; but the
sequence is fixed. A complete example of a problem being solved might be
prepared, with questions at each intermediate step, giving the audience
practice and self-assessment at each, and also showing the teacher where to
speed up and where to slow down in going over the method.Contingent path through a case study
Thirdly, we could have a prepared case study (e.g. a case presented to
physicians), with a fixed start and end point; but where the audience votes on
what actions and tests to do next, and the presenter provides the information
the audience decided to ask for next. Thus the sequence of items depends (is
contingent) on the audience's responses to the questions; and the presenter
has to have created slides, perhaps with overlays, that allows them to jump and
branch in the way required, rather than trudging through a fixed sequence
regardless of the audience's responses.Diagnosing audience need
Fourthly, a fully contingent session might be conducted, where the audience's
needs are diagnosed, and the time is spent on the topics shown to be needing
attention. The plan for such a session is no longer a straight line, but a
tree branching at each question posed.
The kinds of question you can use for this include:
(You can either pick the most popular on the first vote; or else operate a
single transferable vote, by deleting the less popular half of the topics
after the first vote and re-voting.)
Designing a bank of diagnostic questions
If you want to take diagnosis from test questions seriously, you need to come
with a large set, selecting each one depending on the response to the last
one. A fuller scheme for designing such a bank might be:
Responding to the answer distribution
When the audience's answers are in, the presenter must
a) state which answer (if any) was right, and b) decide what to do next:
Selecting the next question
Decomposing a topic the audience was lost with
While handset questions are MCQs, the real aim is (when required) to bring out
the reasons for and against each alternative answer. When it turns out that
most of the audience gets it wrong, how best to decompose the issue? My
suggestion is to generate a set of associated part questions.
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/feedback.html.
Feedback to students
(written by
Steve Draper,
as part of the Interactive Lectures website)Levels of feedback (in order of increasing informativeness)
The null-hypothesis for a Wilcoxon test could be:
Why is it that this vocabulary difference is seductively misleading to half
the class? Perhaps because both are artificial views of the same real people:
the technical terms don't refer to any real property (like age, sex, or
height), just a stance taken by the analyst. And everyone who is in the
sample is in the population. It's like arguing about whether to call someone
a woman or a female, where the measure is the average blood type of a woman or
of a female. And furthermore because of this, most investigators don't have a
fixed idea about either sample or population. They would like their ideas to
apply the population of all possible people alive and unborn; but know it is
likely that it only applies to a limited population; but that they will only
discuss this in the last paragraph of their report, long after getting the
data and doing the stats. Similarly, they are continually reviewing whom to
use as a sample. So not only are these unreal properties that exist only in
the mind of the analyst, but they are continually shifting there in most
cases. (None of this is about casting doubt on the utility of the concepts,
just about why they may stay fuzzy in learners' minds for longer than you
might expect.)
There was something cunning in the question on whether a correlation
was significant or not, with a p value of 0.085. Firstly because it isn't
instantly easy to convert 0.085 to 8.5% to 1 in 12. 0.085 looks like a
negligible number to me at first glance. And secondly, the explanation didn't
mention the wholly arbitrary and conventional nature of picking 0.05 as the
threshold of "significance".
The regression equation is Reading = 2.07 + 0.659 MotivationPredictor Coef SE Coef T P Constant 2.074 1.980 1.05 0.309 Motivati 0.6588 0.3616 1.82 0.085
S = 2.782 R-Sq = 15.6% R-Sq(adj) = 10.9%
a. There seems to be a negative relationship between Motivation and Reading
ability.
b. Motivation is a significant predictor of reading ability.
c. About 11% of the variability in the Reading score is explained by the
Motivation score.
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/manage.html.
Designing and managing a teaching session
(written by
Steve Draper,
as part of the Interactive Lectures website)
Techniques include having an agenda, relating each part, or each
question from the audience, to the overall purpose, ending with a summing up,
etc.
If you want people to participate in this way then you could: ask them
questions to elicit oral responses, start with a question that is easy and
unstressful to answer, etc. If questions or answers are helpful, always say
so; if not, say why not (while in many cases also saying thank you that they
were prepared to volunteer something at all), ...
A technique used entirely to create the right precedent is to ask everyone to
stand up; then only those to sit down who think the answer to this question
is X ... Feedback to the presenter
In running a session, the presenter has to make various judgements on the fly,
because they must make decisions on:
(Document started on 6 Jan 2005.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/qbanks.html.
You may copy it.
How to refer to it.
Question banks available on the web
By
Steve Draper,
Department of Psychology,
University of Glasgow.
(Document started on 15 Feb 2005.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/evidence.html.
You may copy it.
How to refer to it.
Kinds of evidence about the effectiveness of EVS
By
Steve Draper,
Department of Psychology,
University of Glasgow.
.
(Document started on 23 Jul 2004.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/faq.html.
You may copy it.
How to refer to it.
Other frequent questions
By
Steve Draper,
Department of Psychology,
University of Glasgow.How many handsets do you lose?
xx
xx
xx
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/tech.html.
EVS technologies, alternatives, vendors
(written by
Steve Draper,
as part of the Interactive Lectures website)Contents (click to jump to a section)
Non-MCQ
There are many other interactive techniques than MCQs. See for example:
Non-electronic, but MCQ
Given the use of MCQs as the way to involve student interaction, there
are other ways that are possible and in fact have been heavily used in the
recent past and present.
Electric but not wireless (MCQ)
There have been, and perhaps still are, cases where particular rooms
have had systems installed based on wiring rather than wireless technology.
Some examples of this are described in the History section at the end of this
page. Nowadays the installation or even the cable costs of such systems would
outweigh those of wireless ones, besides being tied to a single room.
Electronic voting (MCQ) technologies
This section lists non-computer special electronic handset voting systems that
support MCQs. Non-MCQ systems, that allow open-ended responses from the
audience, are discussed in a later section.
And don't forget the alternative of using computers: one PC per student
(discussed in this paper) and in a section
below.
Open-ended audience input
The key function that MCQ-oriented technology cannot cover is allowing
open-ended (e.g. free text) input from each audience member, rather than just
indicating a selection from a small, fixed set of alternatives.
However it is important to think, not just that MCQs are a limited way of
asking questions, but what on earth a presenter in front of an audience of
several hundred could possibly do with hundreds of free text inputs. The
great virtue of MCQs is that great numbers of answers can be summarised in a
single, simple summary (e.g. a bar or pie chart), whereas it would take a
human not a computer, and considerable time, to group free text answers into
"similar" points.
Voting by SMS (mobile phone) texting
For some years, in nearly every talk or workshop on EVS, someone would
suggest that it could all be done using texting on the mobile phones very
nearly all students carry. In 2008, an MIT startup company
Poll Everywhere created by Jeff
Vyduna (jvyduna AT sloan.mit.edu) offered this service (which is not yet
available in the UK). Here is a discussion about the apparent prospects and
problems with this approach.
Jones,M. & Marsden,G. (2004)
"'Please turn ON your mobile phone'
-- First impressions of text-messaging in lectures"
(University of Waikato, Computer Science Tech Report (07/2004))
In that study:
Nevertheless, the students were favourable to this. So it is feasible if you
don't mind only a minority voting successfully.Summary: how to decide if to adopt this technology
Other near future
In 2002 near future solutions look like including using text messaging from
mobile phones (see previous section), or with all audience members having a
PDA (personal digital assistant i.e. palm-top computer) with wireless
networking. See also this system for voting from pocket PCs:
class in hand. Historical cases of classroom voting technology
I'm interested in having a history of classroom voting technology; and putting
it here. I repeatedly hear rumours about a "system just like that": these (so
far) turn out to be wired (as opposed to wireless) handset-equipped lecture
theatres. Such systems are of course not mobile but tied to particular rooms,
and support only MCQs, not open-ended responses from the audience.
Mentions a machine in use at the time at the Veterinary College in London
where each student's desk in a lecture theatre has a 5-position dial for
responding to MCQs presented on an OHP. A panel on the podium gave totals
for each response. Based on an American idea, it says.
This paper both describes the equipment and the pedagogical rationale. It was
applied to a class of 120 Veterinary students learning anatomy at UCLA.
Students had a 5-way choice (MCQs with up to 5 alternative responses).
Results only visible initially to the lecturer.
Totals for each response, plus in fact all responses (so possible to go back to
individual students). Equipment obviously tied to one room.
The paper gives views of the advantages of this approach that are very similar
to ours: alert students due to active participation, feedback to students,
feedback to lecturers on how well the material is getting across, which
allows them "to branch into new explanations".
[Information from Neil Spurway, John Womersley, and Sheila Jennett; and
Willie Dunn]
The basic thinking was to improve the efficiency of the lecture in
situations where the students have partial knowledge. For example, suppose we
have an expert meeting with a group of GPs as part of a continuing
professional education programme. He could give a pre-prepared lecture, but
would that meet the needs of the GPs? We might have identified a general need,
but what of more specific needs, say about treatments.
(Document started on 18 May 2003.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/techtech.html.
You may copy it.
How to refer to it.
Some technical details on PRS
By
Steve Draper,
Department of Psychology,
University of Glasgow.
ID numbers
Time stamps
Log files
Range
Angles
How to install it; magic password; obscure error messges
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/people.html.
UK handset users and sites
(written by
Steve Draper,
as part of the Interactive Lectures website)Strathclyde University
Prof. Jim Boyle, in Mechanical Engineering, may be the longest standing
practitioner in the UK, starting with non-technologically supported "peer
instruction" in 1997, then using an earlier equipment system (Classtalk), and
then being the first UK purchaser of PRS in 1998. He has by now modified not
just his teaching practice but the timetable and the architecture of his
teaching rooms, and been involved in
University of Glasgow
We have tried out the use of handsets since October 2001 in a variety of
departments including Philosophy, Psychology, Computing Science, Biological
sciences, Medicine, Vet School and the Dental School (with GPs), with
audience sizes from 20 to 300, and with students in level 1 to level 4.
See our interim report on this.
See also the various papers published, listed on
the main web page.
Three contacts are
Steve Draper (Psychology),
Quintin Cutts
(Computing Science), and
Susan Stuart in Philosophy.University of Edinburgh
Physics and Biology have embarked on
serious PRS use in first year classes:
Simon Bates
is the person to talk to
about this,
and also see this site
for other educational initiatives there.
This followed preparatory work by
University of Surrey
Vicki Simpson and Paul Burt (E-learning unit)
(or try
here or
here)
have been advising on adoption of the handsets. They had an Interactive
Presenter system with 250 handsets that they used to pilot for the delivery of
lectures in the School of Biomedical and Molecular Sciences. They have now
bought 1,000 RF handsets using TurningPoint.
University of Bath
Now has a particularly well integrated and organised system of supporting new
EVS users, with multiple elements to the support including collecting
evaluation data on the success of the new uses.
They currently have Turningpoint kit.
See here.
Contact
Nitin Parmar.
J.Smith (2001) Dialogue and interaction in a classroom environment
(Final year research project, School of Mechanical Engineering, University of
Bath).
Summary.
University of Aberdeen
Phil Marston of the
learning technology unit has
bought a small set, and is evaluating them.
http://www.abdn.ac.uk/diss/ltu/pmarston/prs/.
(or if you have already registered, then
here).
University of Wolverhampton
Apparently have used PRS to teach computing; but now have Turningpoint kit.
Contact ILE.
Glasgow Caledonian University
A large set (many hundreds) of PRS-RF handsets have been bought and used in
the business school
for first year students.
University of Central England (in Birmingham)
Bill Madill (Bill.Madill@uce.ac.uk) of the School of Property and
Construction wrote an MEd thesis on Peer Instruction.
He has a case study of using PRS available on the web:
synopsis and
full
case study.
University of Wales, college of medicine (Cardiff)
They bought a (non-PRS) system and used it for a while from 1997, but it may
have fallen into disuse: see
this paper by Joe Nicholls.
Wendy Sadler
in Physics and Astronomy is buying a set for school liaison as
well as for students.
University of Portsmouth
Michael McCabe
(michael.mccabe@port.ac.uk) was awarded a
3-year HEFCE National Teaching Fellowship for Project LOLA (Live and
On-Line Assessment -- the proposal is
available). The live part of the assessment relates to the use of
interactive classrooms in face-to-face teaching, which includes PRS handsets
as one approach. Other papers are listed on the main page.
University of Lancaster
Caroline Elliott (Economics dept.) has done
featured
work on using handsets in 2000/1.
The dept. of Accounting and Finance also uses them regularly
(see here).
There is a set of about 150 PRS handsets: contact
Sue Armitage.
University of Southampton
Su White
and Hugh Davis
and others in Electronics and Computer Science have acquired some equipment
and begun exploring its use in teaching from 2002.University of Nottingham
Liz Sockett
in Genetics is a big fan, and uses them extensively.
Science Museum (London)
Deborah Scopes (d.scopes@nmsi.ac.uk) has been exploring the use of handsets
as an enhancement to public debates and lectures on science.
University of Ulster
Edwin Curran says PRS was installed ready for Sept 2003 in a 170 seat lecture
theatre in Engineering, plus a portable system.
University of Liverpool
Both CPS and PRS used there. Doug Moffat (Mechanical Engineering).
Liverpool John Moores University
Laura Bishop (Palaeoantropologist) and Clare Milsom (Geologist)
are considering introducing EVS use there.
University of Salford
PRS used there.
Elizabeth Laws, Engineering.
Kingston University
George Masikunas,
Andreas Panayiotidis,
and others at the Kingston University (Kingston Upon Thames)
business school
introduced PRS in 2003-4 and use it for first year classes of about 250
students, where small groups are required to discuss and agree answers to the
questions posed.
They now (Sept. 2005) have and use a set of 60 handsets of the PPVote system.
University of Central Lancashire at Preston
Mick Wood
is leading the introduction of EVS (using IML not PRS kit) there,
with a first application in Sports Psychology.
University College London
Martin Oliver produced a report on whether using handsets might be worthwhile
at UCL.
University of Keele
Stephen
Bostock in the Staff Development Centre got interested in using PRS,
meantime introduced the use of coloured cubes as a substitute, but now has
Promethean radio-connected voting handsets working with interactive
whiteboards, around campus. University of Northumbria
Chris Turnock
(or here)
is currently co-ordinating staff development in the use of the system before
undertaking further evaluation of its use within the university.
Paul Barlow (paul.barlow@unn.ac.uk) in the School of Humanities is considering
applying EVS in the Arts area.
University of Leeds
Leeds now has over 100 PRS handsets and
a simple introductory website for EVS.
Contact
Tony Lowe.
Robert Gordon University (in Aberdeen)
Roger McDermott
in the school of computing started to use them in various classes
from October 2004, and there are more than 300 PRS handsets now. Contact
Garry Brindley.
The faculty of health and social care has also taken up
their use.
Kings College London
Ann Wilkinson is looking into EVS use. Professor Simon Howell in
Biomedical Sciences is believed to have used an EVS.
Bangor University
Paul Wood
(r.p.wood@bangor.ac.uk) is installing a 120 set PRS system in a lecture
theatre in November 2004 and plans to start trials with enthusiastic
academics.
Coventry University
Anne
Dickinson has been investigating possible use, particularly of
the Discourse equipment,
and has written Lewisham College, London
Raja Habib and Raul Saraiva are looking into using PRS on behalf of their
college.
Brooklands College, Weybridge, Surrey
Theresa Willis
National University of Ireland, Galway
Ann Torres
has started (October 2004) using PRS for teaching Marketing Principles to a
class of over 300.
Army Foundation College in Harrogate
Lesley Harburn (lesley.pete@tiscali.co.uk)
is using Promethean portable pods as EVS in teaching
16 year-old Junior Soldiers as part of an Apprenticeship in IT at the
Army Foundation College in Harrogate.
Essex University
Caroline Angus
in Sports Science is a current big user (with PRS-RF kit).
Bournemouth University
Kimberley Norman
in the Learning Design Studio is interested.
Brighton University
Gareth Reast
is looking into purchasing a set of CPS (eInstruction); perhaps
for use in the business school and applied social sciences.
University of Hertfordshire
They have some PRS and some Promethean equipment for audiences of 250.
Contact: Andy Oliver.
Roehampton University
Andy Curtis
is going to purchase some kit, and get it used.
University of East Anglia
Believed to have bought some kit for the medical school. Contact
Ann Barrett.
University of Bristol
Have got the loan of 400 Promethean RF handsets. Contact
Nic Earle.
University of Durham
Have 50-100 Quizdom handsets for Maths. Contact
James Blowey.
Also interest from Stuart Jones, Geology dept.
St. George's, University of London
Have 50-100 Quizdom handsets (medical students). Contact
Philip Harvey.
Queen's University Belfast
Have 500 TurningPoint handsets. See here for
the project. Contact David Robinson, or
Prof. Brian Whalley,
Geomorphology, for the original push.
University of Newcastle
Have 100 TurningPoint handsets as part of
central support.
Contact Az Mohammed (Az.Mohammed AT newcastle.ac.uk).
Leicester University
Has a set of RF KEEpad handsets plus TurningPoint software for use with first
year biology students from Oct. 2007.
Contact Jo Badge.
Another set at the university has been used for school outreach. Loughborough University
Has now got 300 handsets plus TurningPoint software. These are held
centrally by media services and booked out. Currently most used in teaching
maths to engineers by the Math Education Centre (contact
Carol Robinson).
City University
Since about Easter 2008, they have 1000 RF handsets using TurningPoint
software. See
this page.
Key contact: Sian Cox.
Reading University
Gan Niyadurupola (d.g.niyadurupola AT reading.ac.uk)
has used them successfully in chemistry.
Queen Margaret University, Edinburgh
They are triallng Turning Point in class sizes ranging from 10 to 180.
Contact
Graeme Ferris.
Warwick University
Library and medical school have kit and TurningPoint software since 2006.
Abertay University
Rebecca Ross is in the process of purchasing.
Have made enquiries
Mike Watkinson (m.watkinson@qmul.ac.uk), Chemistry, Queen Mary, University of
London
(Document started on 6 Jan 2005.)
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/bib.html.
You may copy it.
How to refer to it.
Ad hoc bibliography on EVS
By
Steve Draper,
Department of Psychology,
University of Glasgow.