Last changed 5 Dec 2004 ............... Length about 2,000 words (14,000 bytes).
(Document started on 18 Apr 2004.) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/video1/rationale.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [ilig] [video1] [this page]

Rationale for video

By Steve Draper, Department of Psychology, University of Glasgow.

This page is to document the design rationale for a video about EVS (Electronic Voting System) use.

Aims

Our purposes in making the video "An example of using the PRS voting equipment" showing the statistics class tutorial were:

To show the reality of us using EVS for people who can't be there when a class is actually running.
To do something my website doesn't for prospective clients: give them some sense of what it all feels like in practice. Hence
- Show the equipment
- and the procedure e.g. show handsets being given out and retrieved, the assistants, ...
- Real time not edited, so clients can see how long it takes per question (I.e. not edited for impatient viewers to give an unrealistic picture that's amusing to watch.)
- Show the kind of introduction of each question and discussion of the responses that is typically involved.
Sneak in some direct evidence from interviews as well as just showing it in use. I.e. some evaluation, not only demonstration.
Bring out the special features of this class, rather than of EVS use in general. These were:
- Extremely quiet class that eschews all overt interaction with the lecturer. (This was the motivation for introducing an EVS to this class.)
- The contrast between what this looks like to an observer (almost dead), and what individual students will say about it afterwards: hence the importance of the interviews.
- Contingent teaching: an example of bringing a bank of questions and replanning the session on the fly, depending on the audience responses.
- A feature, as it turned out, of the interview with the lecturer was a clear articulation of an important general point: that a big part of the value of using EVS is the much better feedback they give, not to the students, but to the lecturer about the state of understanding of the class. This is also said by other experienced users, but is different from what our starting point usually is: increasing interactivity for the students. So I was happy to make this salient as it is general, important, and non-obvious for new and prospective adopters of EVS.

The main issue, as brought out in Quintin's comments below, is that being realistic, which is good for clients who are already serious about seeing what is involved in practice, is in tension with making a video that is both interesting to watch, a general introduction to PRS, and perhaps also communicates as many points as possible about the subject as a whole. This edit of this video went for the realism, but really would need to be re-edited to be a good (standalone) introduction.

Critique / possible re-edits

Comments from Quintin

Here are some comments from Quintin.

Useful/Best

The quality of the presentation was very good - by this I mean the video work, editing etc. I liked the fades and the text over at the start, all of it really.

It clearly shows how the system works and one typical way in which it might be used - your major purpose in making it.

I loved the clips at 27:12 and 27:25 of two blokes and two girls respectively thinking, discussing and then answering - I thought these showed well how the system promoted processing/discussion - shame these weren't in the trailer.

As I said, the trailer was extremely useful in a seminar I gave for showing the major aspects of the system in a very short space of time.

Worst
I wouldn't say worst, rather, the bits that struck me or puzzled me. The main one of these undoubtedly is how little interaction is portrayed in the film. It doesn't appear that PRS has promoted discussion, instead there seem to be many long silences, and many shots of quiet students. I guess they do all look attentive though. This is why I liked the two shots above so much - they really showed engagement through discussion. There probably are other instances, yet these ones just stood out to me.

I wasn't convinced the third question was necessary, since it was very similar in nature to the second, apart from that it demonstrated that the students were doing well, keying in John's harder question. But as a viewer I was a little bored by it.

Programme notes and discussion
The notes were a useful intro on what to expect, but I didn't find them too easy to use alongside the DVD as it was running, and also not very necessary.

I think the question here is one of familiarity. I of course didn't really need the notes, as I knew about the context already. But they'll certainly be necessary for others.

However, I didn't know the stats material being covered because I've never taken a stats course of any kind. This made the film harder to watch, interestingly, or cut me out a bit, because I couldn't play along with the questions - make my own guess, see how I did etc. Bringing me into the experience in that way might improve the viewer's experience (why do folk enjoy Who Wants To Be a Millionaire?). There's a problem here of course since all films of real use are going to be subject specific (one assumes), and so non-subject folk will find the questions hard/meaningless. My unfamiliarity with the material made it hard to hold the question in my mind while the response options were revealed. It frustrated me a bit, and oddly made me think John wasn't doing a very good job (although I'm sure in context he is doing just fine!).

I wondered whether you could make more of the quiet parts of the film - there are a number - by including comments in the form of text. For me, this would be embedding the programme notes into the film itself. An easy example is to make a comment about the waiting time - one to two minutes - while students answer, by saying that lecturers have found it useful thinking time, or time to talk to one or two students or, or, ... Another would be a comment about student feedback that the breaks are good for concentration. What I'm saying is that much of what is we now know about handset use could be incorporated into the presentation itself, since I perceive a few slow bits.

The use of peer discussion/instruction in Q4/5 is against most of the literature I've read on this kind of use - Mazur, Novak, Boyle and others. Most of them indicate that it's only worth using peer discussion when the split between right/wrong is no greater than 30/70 - ie at least 30% of students need to have got it right. In this use, only about 5-10% of the class got the question right. In fact, this was an even more subtle interaction, because John actually asked them to consider a different question in their discussion from that on the board, and to use the result to influence the result to their first - he asked them to talk about "What value does Cramer's V statistic take?" rather than directly about the question. I'm not sure if the students would have picked that up, and might rather have just attempted to justify their answer. Don't know what this means exactly - but it's an interesting observation! John's comment/question was leading and helpful, but I'm concerned that they may not have really picked it up. This all came over to me in the relatively quiet class discussion. Well it seemed a little quiet to me. You only had this one opportunity I know, but in a future version, it might be nice to get a more intense discussion.

That's another example of a use of captions in the film - one could emphasise the point that Stats 1C is a quiet class who don't interact much, and that the feedback (you could give stats) shows that the class do gain from having the system there. And that in other class, students are more forthcoming with student-lecturer discussion after a vote.

I see the captions as a cheap way of increasing the information content in the film for viewers. It doesn't require any further filming, just some text box overlays on the existing edited work, in the quieter parts of the presentation.

The examples of use in the film really amplify what the students sometimes say - that the lecturer gets the most out of it - whilst minimising what the students get. The students don't look enthralled in the film, and John says in the interview that whilst "I hope the students get something too...", he finds that the information for lecturers is the most useful. This is mitigated only to some extent for me with the student interviews - in that they do state that the handsets are useful, but not very articulately, apart from one girl. She's good.

So, in summary:

excellent presentation and well worth having
could you put programme notes into the film in the form of captions - would give more information/commentary about what was happening, and also provide useful background/contextual material, and alternative options at particular points, and experiences in other classes. These could also be voice-overs, I guess.
seems to favour the view that staff are the main beneficiary
stats is known to be a quiet class. Would be nice to have a future version with a livelier class.

Possible re-edits

Include a new index point for jumping straight to the best examples of students discussing. I added this to the programme notes as a new listed time; and it isn't far beyond an existing chapter point in the DVD version.
Use these clips in the trailer. We had considerable trouble keeping to 30 seconds in the trailer as it is: including each phase of a vote, etc. But actually there is one audience shot in the middle of the trailer that could be replaced by this.
Using subtitles and/or voiceover to a) make more points b) stop it being boring for a general audience. In a way this is a different purpose for the video: not to show someone already very interested what it feels like, but as an efficient general introduction. But couldn't we do both? i.e. keep the visuals and so the timing; but add parallel channels. We could. Although it would then kill off what I actually think is an important part of the actual emotional feel of using an EVS to the lecturer: learning to keep quiet, allow for the time it needs, and particularly time when the lecturer has nothing to do but let the audience (and equipment) do its thing.
Put the programme notes on a PDF file on the DVD?

Alternative types of EVS video

This one: give a realistic feel, assuming the audience is already seriously interested in using an EVS.
A general introduction as an alternative to text. This would need to be more efficient with the audience's time; but would also probably show bits from a much wider range of classes.
Presenting different pedagogic ways of using an EVS. This would again use a much wider range, and examples chosen for being particularly good of their kind and/or critiqued for their method; and include explanations about the idea behind each.

Future plans

We'd like to have about three contrasting uses on film. Probably they would all fit on a single DVD.

So ideally the set would range over:

A very big class, and a small one
A definitely non-science subject, as well as science related ones.
A noisy and lively class, as well as this very quiet one.
Contrasting pedagogic uses and techniques.
- This one illustrated contingent teaching
- Jim Boyle's class would illustrate the Mazur approach centered on discussion.
- Simple self-assessment questions tacked on a lecture
- Serious class test?
- Progressive steps through a single problem being solved.
- Mass data gathering; experimental demo with audience as subjects.
- Contrasting question presentation techniques: show PRS integrated with powerpoint, perhaps with photomicrographs, etc.
Have a sequence showing setup and re-collection at end in a bit of detail.

More realistically, probably plan to film Jim Boyle's class, and one of Quintin's; and search for a non-science class.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [ilig] [video1] [this page]
[Top of this page]