Last modified 9 Feb 1998
This is a WWW document by Martin Rutherford, Anne Tawse and Diane Strachan
The content of this page is the responsibility of Martin Rutherford.
This is version 3 of the page
This page contains our lecture notes for Lecture 3 of the HCI course on the MSc(IT) at Glasgow University. For more information see the handout on Practical Methods for Measuring Human-Computer Interaction, particularly pages 20-21.
A questionnaire is one of the quickest and simplest ways of collecting the users' responses to something that they have experienced. However, they are only of use if designed well and used appropriately. For example, ambiguous questions may be misleading.
In particular, fixed response questionnaires allow us to compare answers more easily than if the user had been allowed a free choice with their response. In class, the statement
was used as an example to show the importance of regulating the range of user responses, since the question would provide a very different range of responses from those who had never used a computer before, right up to Computer Scientists.It is difficult to learn Pascal
Questionnaires are the most useful way of collecting data from large numbers of people. This means, for example, that one investigator can have their questions answered by 20,000 people.
As mentioned above, questionnaires are only effective if designed properly. This applies both to the questions themselves and the responses.
The main issue is to avoid ambiguity. If the question is not clear then the results may not be usable. Thus, it is important to test your questionnaire with a prototype and debugging. For example, it would be a good idea to ask someone to try it out, thinking aloud while they answer.
A classic way of clouding the issue is to use conjunctions in a question. For example, in the question,
some of those responding may feel that although the letter may be impressive, it may not be especially business-like and vice-versa.How confident are you that you could write a letter using ClarisWorks with an impressive and business-like appearance?
One way of solving this problem is to ask two related questions instead of one. If they are worded in a similar manner then it will not take the user too much extra time as in the first question the user will take more time to understand the responses so they will be quicker on the second. This means that you can get more information. An example raised in class was
It was suggested that it would be better to ask two questions, Do you go to lectures? and Do you do any follow-up reading after the lectures?Do you find HCI lectures stimulate an interest in the subject?
Finally, it is important to offer definitions of what your anchor points mean. For example, in a questionanaire that asks
the anchor points could be defined as 0 = No never drink it while 5 = Drink it every day.Do you like orange juice?
We started a class email discussion by sending this message:
The question we made up in today's lecture was:
How often do you find the Think Pascal error messages helpful?
None of the time 0 - 1 - 2 - 3 - 4 - 5 - 6 All of the time some of the time Please feel free to critise and send your comments to the class.
Here are a selection of the contributions made to this discussion:
Catherine Lawson said
My answer to the Question is 3The error messages are less helpful when you are learning to use the package. But once you get to know them they can be quite helpful. Near the beginning of term I spent an hour and a half pulling my hair out at a message that said "this statement does not make sense" This persisted until the next day when someone said, "yeh I had that too for ages, check that there isn't an end or a semi colon missing on your last procedure before the hand." Two seconds later this problem was solved. The placement of the hand can be very misleading.
Heather Parton said
Question is clear, unambiguous and appropriate.Responses ok, helpful anchor points especially some of the time.
Michael Gallacher said
Having used Pascal the error messages acan be obtuse at times and possiblly the range should be negative to positiveeg -2 all the way up to 2
John Paul Nelson said
I reckon it depends what the error message is. "semicolon or end expected" always tells me exactly the problem, but "incorrect use of readstring, read array or read variable." tells me precious little. Maybe your question should be more specific. The question from my friend (whose name I embarrassingly don't know) was "do you find the lectures stimulate an interest in the subject" - our question was read out in the class, so it's alredy been covered. I think the best way to create questionaires is to make the questions totally unambiguous. But then, what's unambiguous to one man is as clear as mud to another, so I don't know that it's ever possible to design a totally clear questionaire for everyone. It would be interesting to hear everyone else's ideas on this.
Steve Draper (HCI lecturer) said
Anne Tawse sent a message asking for comments on her draft question.
"How often do you find the Think Pascal error messages helpful?"
My own reaction is: good first draft
Perhaps eventually there would be difficulty over "helpful": the obvious meaning is "helpful enough for me to fix the problem"; but then some users may start to wonder about messages that seem helpful but later I find they didn't get me to think about the underlying problem, messages that were actually very unhelpful but just giving the right line number was enough for me to fix it, .....
If all you really care about is whether the user succeeds, given the message, then perhaps "How often do you find that the THink Pascal error message is enough for you to solve the problem?"
Other opinions?
Page designed and formatted by Martin Rutherford, Diane Strachan and Anne Tawse.
Thanks to all those who contributed to the discussion. Please feel free to continue it.