Summary of Lecture 6

The Use of Incident Diaries in HCI

The

Ultimate

Incident...

Date page last modified 04.02.98
Who maintains the web page: Edward Niescior.
Who has editorial responsibility: Clare Mc Govern, Lucinda Bailey and Edward Niescior.
Lecture 6 (week 3).
Topics: Incident Diaries, with an email discussion, and a Review of Psychological Instruments used in HCI.

The Use of Incident Diaries in HCI

Key questions:

1) Why use an incident diary?

• To catch infrequent or unexpected interface problems that have been missed by other instruments, or cannot be simulated in a lab environment,

  eg network problems like printer queues.

• As an alternative to think aloud protocols if this is uneconomical (too many subjects) or impractical (investigator can't be there).
• To record problems on the spot allowing accurate recording.

2) How do you define an incident?

• The incident is the problem you are attempting to discover and observe.

During the iterative process of diary design, the problem may begin as fairly undefined, but gradually crystallises, as successive diaries allow you to hone in on the exact problem.

eg. To study Netscape problems:

First draft - "Fill in every time Netscape doesn't work."
Second draft - "Fill in every time Netscape says 'The server does not have a DNS entry' ".

3) Problems with Incident Diaries

• Getting participants to remember to fill it in.
• Ensuring honest entries (See e-mail discussion).
  (Not metioned in lecture)
• The unreliability of retrospective entries.

4) Design Hints

A good incident diary should include:

• A clear title, stating the function of the diary,

  eg. (From example given in lecture) "Diary of using e-mail"



• A clear definition of the incident to be recorded,

  eg. "The diary is designed to record each time you use e-mail in any way".



• How often the diary is to be filled in,

  eg. "Please fill it in EVERY time you look at your e-mail, even if you only check it but do not read or send messages."



• A cover sheet for once-off data,

  eg participants name, previous experience, if any, with the package/task.



• Unambiguous column headings, so participants know exactly what to fill in.


• A concluding section asking:

how many incidents were recorded later and if so, how much later.
This allows the investigator to estimate accuracy of entries.

how many incidents were not recorded at all.
This gives the investigator an idea of how representative the results are.

The Key to Success

Persuading participants to fill in the diary is crucial in obtaining meaningful results.

This can be achieved through :

• gaining participants' support, by stressing the value of their contribution towards meaningful results.
• keeping their work to a minimum by setting a realistic limit to the number of entries.
• establishing a habit, by being present when the diary is introduced to the test environment.
• good design, so that the diary is quick and easy to use.

We asked the class for feedback on various issues, and were inundated with copious replies for which we are most grateful.

Is honesty an issue?

We felt that under some circumstances, honesty could be an issue, although it was not mentioned in the lectures.
We asked:
"How great an issue do you feel honesty is in completing diaries?
For example, would you be too embarrassed to reveal that you hadn't received any mail in 3 days?! "

Wendy Thomas said:
"I would not be too embarrassed, but it's a good point - people will lie about things where it's a tad embarrassing to be different from the supposed norm. Especially happens when the diary is named - for example anonymous incident diaries might be more likely to get truthful answers than personally distributed ones."

Katrin Hartmann said:
"It's a great challenge for the evaluator to make the subjects aware of the importance of being honest...a few right sentences at the beginning can do a lot."

Roisin McGrath said:
"I don't think filling in the diary honestly or dishonestly is any big deal, folk tell you what they like!"

Mel Terras said:
"no, not embarrassed......but there is the chance to lie, after all......"

Memory Clouding

How accurate did those who filled in the email diary feel their answers could be if they completed their entries from memory?

We asked:
"Did you fill the diary in after the event, and if so, do you think this affects the accuracy of your response?"

Martin Rutherford said:
"One of the instances I filled in after the event, and I found I couldn't remember how many emails I had received - so yes, it did affect the accuracy."

Katrin Hartmann said:
"I find it a good idea to put a field on the diary in which the subject can state when it was filled in so that the evaluator can work with this info."

Part two of this lecture
Back to Lecture Five
Forward to Lecture Seven
Index
Main Page

If you would like to add to this discussion, or if you have any other comments to make regarding the material on this web page, please mail us: