Last changed 26 Jul 1998 ............... Length about 1000 words (7,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/overview.html. You may copy it. How to refer to it.

[ Teacher overview    Teacher details    Learner main page    Course home page (sample) ]

CBL evaluation: Overview of the ATOM

Preface: introduction to the notion of an ATOM

Follow this link for a brief introduction to the notion of an ATOM, and terms such as trail, remote expert, delivery, etc.

Summary

State the topic and type of ATOM.
This ATOM is a practical introduction to evaluating CBL material. It was intended as one week's work on a CBL course, but is probably rather more than that: one of the bigger exercises on a course. It spreads over several weeks in elapsed time because of the need to plan, execute, and analyse an evaluation. (The sequence of activities is given on the learners' main page.) Timetabling may be constrained to fit around the delivery time of the CBL chosen to be evaluated.

How the concept fits into the course

Relationships and constraints on this topic in relation to the surrounding course
This ATOM was designed for use in a module on CBL (computer based learning). Evaluation, though often neglected, is a crucial topic. Conceptually, it could be taught at any point: you could begin with it to set a standard, or end with it. The kind of evaluation presented here is empirical, classroom based evaluation i.e. it is about observing and measuring student reactions and learning outcomes of CBL (not about software features, or teacher opinions).
Student prerequisite knowledge
None, although it is assumed that the students will have some knowledge of HCI evaluation or else they may be unready to accept the importance of testing real users.
Student prerequisite skills
It is assumed, although perhaps not wholly essential, that the students will have some experience of HCI evaluation methods, some experience of designing and using questionnaires, and some experience of receiving CBL material themselves.
Course level
This ATOM has been used at M.Sc. level but there is no obvious reason not to use it at any level, except perhaps for issues of independence and study habits. These often depend more upon discipline (arts vs. science) than course level.

However planning and executing an evaluation as an autonomous activity, without simpler warmup activities, probably does require previous experience that may not be available except at this level.

Timetabling: how the activity fits into the course

Amount of student work
Intended as one week's work for a module (say 8 hours), but probably more.
Amount of contact hours
We scheduled 2 2-hour class meetings (using video conference), where this ATOM was a main item of business.
Group work
The plan is for students to work in groups. Any size is conceivable. Because an evaluation of this kind usually involves several different instruments (questionnaires), it is natural to have several (more than two) students per group. Having several groups in the class allows different designs to be contrasted.
Relationship to rest of course
The key constraint is that this exercise is about evaluating some (other) piece of CBL. When this has been decided on, then the work for this ATOM is scheduled around it: with data gathering during that delivery, analysis and writeup after, and planning before. Thus this ATOM can probably not be the last activity in a course, and the CBL delivery used as a target cannot be the first thing in a course. More details on the sequence of activities is given elsewhere.
Learner activity plan
Follow this link for details.
Assessment
The reports of their group's evaluation written by each student should probably be marked. In the one delivery so far, this was done by the local deliverers, but after (taking into account) the verbal comments given to the students by the remote expert.

Resources needed

Special
The ATOM requires that the students have access to either themselves or another group of students while they are receiving some CBL material.
Remote expert
State how much work is required of the remote expert, and how essential it is
Used for seminar/tutorial (by video conference) twice: to discuss the reading and/or preliminary design of the evaluation, then to discuss preliminary findings by the student groups. Only 2-4 hours of expert's time. Such an expert is only necessary to the extent that expert discussion of the (always diverse) methods and findings by the various students is wanted by the deliverers.
Software
None (other than that CBL material).
Hardware
None for computing.
Video conferencing for the whole class, if the remote expert is used.
Reading
Papers are available on the web and in journals.

History and admin.

Authored and maintained by Steve Draper at the University of Glasgow.

Number of times delivered: 1
Evaluation reports available: 1
Number of Trails (examples of past student work): none.

Structure of the ATOM's web pages

There are 4 kinds of web page associated with this, perhaps with any, ATOM.
  1. The teacher overview page [this page], intended for teachers browsing for ATOMs they might choose to use in their courses.
  2. Teacher details: fuller notes for teachers, with instructions for them, and rationales for why the exercise is the way it is.
  3. The learner details page: the instructions for learners. This is likely to have the basic explanation and description of the ATOM, and also acts like a handout for students.
  4. The course home page (for learners) for a particular delivery of an ATOM, special to the institution, course, and circumstances. This should have all the links needed for those students, and in turn should act as the entrance lobby, and be pointed to by the course pages, links from the teachers' home pages etc. A sample is included here, but its content and format may vary widely depending on the course within which the ATOM is being used.

[ Teacher overview    Teacher details    Learner main page    Course home page (sample) ]