Last changed 27 Mar 2002 ............... Length about 2,000 words (18,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/interim.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]

Use of the PRS (Personal Response System) handsets at Glasgow University

INTERIM EVALUATION REPORT: MARCH 2002

by
Stephen W. Draper and Margaret I. Brown
Department of Psychology
University of Glasgow

Contents (click to jump to a section)

Overview

In the current academic year (Oct 2001- March 2002), interactive handsets have been trialled at the University of Glasgow by lecturers in Philosophy Psychology, Computing Science, IBLS, Medicine, Vet School and the Dental School (with GPs), with audience sizes from 20 to 300, and with students in level 1 to level 4. Handsets have been used in lectures and formative assessment sessions. They have been well received by students in all but one case, as judged by responses to our key evaluation question about whether, in each student's view, there was a net gain in benefits over disadvantages. The lecturers who used them have also been asked about their views, and again in all but one (different) case, felt the benefits outweighed the difficulties. Our evaluations, conducted by Margaret Brown, have also amassed a list of benefits and of disadvantages mainly from the student view, which we will be writing up soon.

As we begin the task of writing up the evaluation studies (March 2002), our initial impression is thus that the handsets do indeed support learning gains in the ways discussed in Draper et al. (2001), but that benefits depend, not directly on the mere technology, but on how well the particular way they are used on each occasion elicits thought and reflection in the learners.

As only limited time was available [2 contracts (GRUMPS; TLC), each of 10%, but since 12/3/02 only one contract (GRUMPS) of 10%] the main focus of the evaluation so far has therefore been on observation and collection of data, and not on the analysis of data and the production of written reports. Immediate feedback (including comments from the students) after a session involving handset use was given to the lecturers either verbally or in a written report.

Collation and analysis of all the data and information we have collected, together with further evaluation is necessary in order to ensure the effective use of the PRS handsets in Glasgow University. The amount of work which can be carried out will depend on the funding available.

Methods used in the evaluation

The following methods have been employed in the evaluation to date.

After each session or series of sessions we now ask students to answer the following core question using their handsets. In addition they are sometimes asked to give written comments, and sometimes to answer further questions via the handsets.

What was, for you, the balance of benefit vs. disadvantage from the use of the handsets in your lectures?

  1. Definitely benefited
  2. Benefits outweigh any disadvantages
  3. Neutral
  4. More disadvantage than benefit
  5. Definite negative net value

The main problem identified by students is that too much time sometimes is involved in setting up the equipment in lecture theatres other than the Boyd Orr where the receivers are permanently installed. The setting up time has been reduced as we gain more experience. The single most frequent type of setup problem has been with the data projectors. Factors like these can affect the views of students and lecturers.

Further work has now to be done to look more closely at the information we have collected: voting figures from the lecturers' questions and the evaluation questions; comments from students and lecturers etc. In addition it should be possible to look at the data in the PRS files from questions asked in a specific lecture and identify if it is the same students that are experiencing problems with every question.

Feedback from six lecturers who had used the PRS handsets in their lectures

  1. The essential feature of the use of this equipment is that both students and lecturer get to know the distribution of responses and, in confidence, how their own response relates to that. The element of anonymity encourages everyone to contribute and, unlike in face to face groups, each individual can express the choice they incline to, rather than the choice they would feel able to explain and justify to others.

    I have been using this equipment in an Introductory Logic course with a class of about one hundred students, and intend to use it in the forthcoming term with a Introductory Philosophy of Mind course. There have been two noticeable results so far. The first is that, if the students are to answer the questions in a way that will be helpful to them, they have to reflect more on what they have learnt and how they are learning. The second is that my teaching is being directed more by what the students need, or at least, say they need, rather than what I think they need. This means that I am not second-guessing or making unwarranted assumptions about their progress.

  2. I found the handsets very beneficial in my lecture and speaking with some students afterwards they also appreciated it. In the 3rd year I have asked questions by way of a written test, and they hand it in to me at the end. They mark it during the lecture, so get to see where they have gone wrong, but I don't until later - so I can't modify the lecture instantly, only for the next year. With the handsets I could see exactly which points I had not conveyed clearly and could rectify it straight away, the major example being when I asked the students what I thought was a simple question - identifying the FCoV carrier cat! Although most (68%) got it right, an astonishing number chose one of the other cats. I could see that they hadn't fully understood that many antibody positive cats are not infected. It was great, because the students who got the wrong answer are very likely the same ones who never utter a word in interactive lectures and it gave them a chance to participate anonymously.

    I wish I could use handsets at all my lectures - is that ever a possibility?

  3. The feeling was that the idea worked well, but that the time it took with a large group was too long. This meant that students lost the thread. The group that we had were generally very good, plite and responsive and some of those that we have lectured to in previous years might have been more difficult to keep in order.

    Our general conclusion was that the system would work well for groups up to about 50 in number, but for a group of this size [200] a set up with buttons that responded instantaneously would be required. When it starts to be installed as a feature of this type in lecture rooms we will use it. We will think about using it for some update courses that we give that have about 50 participants.

  4. I think (and the results also showed this) that the students liked both the experience and the fact that they could test their understanding of the topics as they went along.

    The results of their tests gave me some idea of how they had understood the concepts, and if it had been obvious that they were not following what was going on it would have allowed me to reprise the previous section (as it was I didn't have to do this). It also gave me some information that will help me to plump up the slides on the web to include extra, helpful information.

    As far as the technical side is concerned, I found it extremely easy to use, especially the PRS interface with Powerpoint. The Chemistry Lecture Theatre is certainly not ideal for testing new technology but I think the system stood up to it very well. Even when we were delayed in getting the equipment to the hall, the set up time did not encroach too much on the lecture.

    In total I think it was a worthwhile experience, both for me and for the students. I would recommend it to others, and I would use it again.

  5. I used the handsets in a level 4 option class in social psychology. The class size is about 50. The use of the sets is easy. A slight problem is the time it takes to register the students answers and you have to time this into the lecture. The students on the other hand do not mind the
    delay. The consumer report indicates (informally) that they enjoy the use of the device. From the staff point of view getting the level of the questions right takes time and experience. My questions were too easy (or else I explain things very well). What I notice as a social psychologist is that there is a level of group effect to be seen as the scores come up. People do not feel individually
    exposed because the replies are anonymous but they do watch the distribution of answers as it appears on the screen. That by itself may be a learning experience as they then consider other possible answers. I would need then to decide what do do when the students are having difficulties. I would need a plan B which would involve a fuller explanation. So it would affect the way I plan lectures. But why not?

  6. We used the handsets for a prelab tutorial session with about about 100 students in each sitting. Slides of photomicrographs were displayed using a slide projector. Multiple choice questions were displayed using an overhead projector. The students were asked questions on each of the photomicrographs and then we displayed their responses and Rob went through the correct answers. I felt the session went well although we definitely needed two people to cope with the the slides, overheads and the computer. It was also a bit hectic handing out the handsets and a handout at the same time. All the students that I have been able to ask, enjoyed the session and several commented on how they felt it was useful in finding out how much they knew (without me prompting such a response). Yes I think we would consider using it again, perhaps for a revision session when we could go over their class test. I think the system also has potential for monitoring lecture attendances which seems to be getting more and more of a problem.

Feedback from students

Data from an evaluation in the Dept. of Philosophy

Some of the evaluation was carried out within an additional relevant contract in the Philosophy Dept. (15%) and the following data was gained from the Questionnaire in Level 2 (Logic).

61 students reported using handsets in Logic lectures (level 2) and rated their usefulness.
The percentage of students who rated them in each category is shown below:
Extremely useful 18.0%
Very useful 21.3%
Useful 37.7%
Not very useful 21.3%
Not at all useful 0.0%
No rating 1.6%

Comments from students who rated handsets "Extremely useful"

The anonymity allows the student to show he/she is unsure of the subject without embarrassing themselves. Lecturer can gauge their method of instruction and ensure all students are absorbing the subject matter. Philosophy courses/lectures could use this method. As they explain and discuss various theories they could confirm that the subject matter is getting across and how well/badly it is going across.

Allows us to see where we are in relation to class mates - lecturer knows what to cover. Takes up time.

Fun. Lack of reliability disrupts lecture.

Comments from students who rated handsets "Very useful"

To see how students are coping with what's being taught. Performed in a discreet way.

Students: encourages us to participate; more likely we will be forced to listen this way. Lecturer: Let's her know what we do and don't understand. Can be a bit time consuming setting it all up. Are definitely useful. Would be good if system was inbuilt.

Good to know if on right track. Takes time. Would be better if set up in advance of lecture.

Comments from students who rated handsets "Useful"

Students: Know how I am doing compared to other students. Interesting. Lecturer: Indication of students' gaps, not what lecturer thinks might be gaps. Students: Can distract from the learning point entirely. Lecturer: Has to be able to give clear instructions on what I am voting for. 3 options are too many for voting if comparing 1 and 2 to be used only and vary the 1/2 . Encourage speed in giving vote to avoid lengthy intervals between "lecture" and voting. The intention is not to give a diversion from the lecture, welcome though it may be.

Good fun. Quite good for gauging how many others are as lost as I am ! Time consuming at first, but getting better. Keep using them please!

Let's you see if you're on the same level as the rest of the class. It takes time to organise it, which could be used for lecturing. I think the benefits outweigh the disadvantages and therefore a good idea.

Allows lecturer to know if they are making it clear enough but only useful to students if they follow up.

Comments from students who rated handsets "Not very useful"

Find out what sections are difficult and how I am doing in relation to rest of class. Very time consuming. Could sometimes be substituted for a show of hands as confidence on material increases.

Students - it's fun, like "millionaire"! Lecturer - can see how class are coping. More useful for lecturer but the colours are nice

Instant idea of understanding. Amusing distraction. V. fiddly! Clearer labelling on display

Some benefits and problems collected from students

From the comments from students on different courses (including those studying Logic), we identified the following suggested benefits and problems of using handsets in lectures. Students in some classes have helped us identify which of these are important (this data is still to be processed), and these should be addressed by lecturers using, or intending to use handsets.

Benefits

  1. Using handsets is fun and breaks up the lecture.
  2. Makes lectures more interactive/ interesting and involves the whole class.
  3. I like the ability to contribute opinion to the lecture and it lets me see what others think about it too.
  4. The anonymity allows students to answer without embarrassing themselves.
  5. Gives me an idea of how I am doing in relation to rest of class.
  6. Checks whether you are understanding it as well as you think you are.
  7. Allows problem areas to be identified.
  8. Lecturers can change what they do depending on what students are finding difficult.
  9. Gives a measure of how well the lecturer is putting the ideas across.

Problems

  1. Setting up and use of handsets takes up too much time in lectures.
  2. Can distract from the learning point entirely.
  3. Sometimes it is not clear what I am supposed to be voting for.
  4. Main focus of lecture seems to be on handset use and not on course content.
  5. The questions sometimes seem to be for the benefit of the lecturer and future students and not us.
  6. Annoying students who persist in pressing their buttons and cause problems for people trying to make an initial vote.
  7. Not completely anonymous in some situations.
  8. Some students could vote randomly and mislead the lecturer.
  9. Sometimes the lecturer seems to be asking questions just for the sake of it.

Questions currently used in evaluations

Here are the current versions of questions used in evaluating the handsets by displaying the questions and having students (the audience) use the handsets themselves to indicate their answers. In addition, we would normally hand out a small piece of paper to each person asking for any further comments on the handsets or the questions.

Here we indicate the text of the questions between horizontal lines. It would be displayed on an overhead projector slide or in Powerpoint. Participants respond by pressing the corresponding digit on their handsets.

Core question

What was, for you, the balance of benefit vs. disadvantage from the use of the handsets in your lectures?
  1. Definitely benefited
  2. Benefits outweigh any disadvantages
  3. Neutral
  4. More disadvantage than benefit
  5. Definite negative net value

Variant question

  1. Extremely useful
  2. Very useful
  3. Useful
  4. Not very useful
  5. Not at all useful
  6. No rating

Questions about particular pros and cons

These questions were done by displaying two slides: one with a numbered list of advantages or disadvantages, and a second asking questions about the list.

Benefits

  1. Using handsets is fun and breaks up the lecture.
  2. Makes lectures more interactive/ interesting and involves the whole class.
  3. I like the ability to contribute opinion to the lecture and it lets me see what others think about it too.
  4. The anonymity allows students to answer without embarrassing themselves.
  5. Gives me an idea of how I am doing in relation to rest of class.
  6. Checks whether you are understanding it as well as you think you are.
  7. Allows problem areas to be identified.
  8. Lecturers can change what they do depending on what students are finding difficult.
  9. Gives a measure of how well the lecturer is putting the ideas across.

Problems

  1. Setting up and use of handsets takes up too much time in lectures.
  2. Can distract from the learning point entirely.
  3. Sometimes it is not clear what I am supposed to be voting for.
  4. Main focus of lecture seems to be on handset use and not on course content.
  5. The questions sometimes seem to be for the benefit of the lecturer and future students and not us.
  6. Annoying students who persist in pressing their buttons and cause problems for people trying to make an initial vote.
  7. Not completely anonymous in some situations.
  8. Some students could vote randomly and mislead the lecturer.
  9. Sometimes the lecturer seems to be asking questions just for the sake of it.

Acknowledgements

This evaluation was supported in part by the EPSRC funded grant to GRUMPS (GR/N38114). It was also supported by a grant from the Philosophy LTSN to Susan Stuart.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
[Top of this page]