07 Apr 1997 ............... Length about 1000 words.
This is a WWW version of a document. You may copy it. How to refer to it.

The role of open ended observation in evaluation


S.W.Draper
Department of Psychology
University of Glasgow
Glasgow G12 8QQ U.K.
email: steve@psy.gla.ac.uk

Preface

This is a summary of a talk, not a proper paper. The original talk was for an audience of evaluators of Computer Assisted Learning (CAL). However the issue, I believe, equally applies to evaluating IR, although my examples will come from CAL.

Abstract

We have recently published a paper on our method of evaluating CAL which we call "Integrative Evaluation". However various questions arise that are not clearly discussed there, not least because we have not done much work on them. We will discuss one of these in this short talk, and illustrate it with relevant cases.

This issue is the role of open ended observation (OEO) in evaluation (as opposed to fixed quizzes and questionnaires), which we believe to be vital and is always an important part of our studies. We have, however, been asked how we could know when we have done enough, or whether we have noticed the most important issues in each given case? We will discuss and attempt to justify our apparently unstructured use of OEO by giving example cases of four kinds: ones where our preconceptions were confirmed by OEO, ones where they were disproved, ones where OEO directed how we then analysed data from structured measures, and ones where OEO alerted us to completely new issues affecting the whole study. Such cases convince us of the importance of OEO, but in the end we may not be able to know whether we have done enough.

1. Introduction

Any evaluation of a piece of CAL faces both the need to answer systematically questions we are interested in in advance e.g. did all students learn the material up to some criterion, and the need to detect unexpected problems and issues. An analogy with visual perception may be useful. One thing that perception does is support specific tasks such as checking whether a particular friend's car drives past you: you scan all cars, make sure you don't miss any, and without bothering about irrelevant attributes of the cars e.g. how dirty they are, whether hub caps are missing, look at the identifying features (perhaps the registration number, or the colour and size). Another thing perception does however is allow you to notice completely unexpected things, such as a tiger walking down the street towards you, someone's umbrella which is just about to poke your eye out, or a street vendor offering vension which would do nicely for your dinner. It will do these things even though you did not plan to do them, and could not say that, for instance, you noticed everything on sale by street vendors.

Similarly with evaluation: it is important to cover both functions. Methods such as exam-type tests and questionnaires with fixed response categories will never warn you that something you did not anticipate is in fact important in the situation you are studying. Hence it is vital always to have some open-ended questions and preferably personal observation by the evaluator. However, open-ended questions and observations are not a substitute for fixed questions: only by putting the same question or task to each learner and requiring the answers to be expressed using the same categories (or marked using the same coding or marking scheme) can you get comparative results that allow you to discover and report results such as what proportion of learners were affected by an issue.

Any evaluation study, then, should have both open-ended measures for detecting surprises, and fixed measures for generating comparative data that can answer specific questions. Without fixed measures you may not be able to say anything definite about the courseware: only an unstructured set of observations and opinions from individuals, which may or may not be shared by the other learners. Without open-ended measures you have no chance of detecting problems or anything you did not think of in advance, and it is from the unexpected that most important improvements stem.

Open-ended observation is a descendent of the approach of "Illuminative evaluation", a term introduced by Parlett & Hamilton (1972) to denote an observational approach inspired by ethnographic rather than experimental traditions and methods. (See also Parlett & Dearden (1977).) We have used various kinds of open-ended observation in our studies, such as personal observation, some video recording, interviews and questionnaires that include open-ended questions, and focus groups. However we have typically spent much less effort than an ethnographer would, spending hours, days and months "hanging out" in the relevant situation as well as interviewing "informants".

How then can we judge that what we do is enough? How can we know if we have identified all the issues, or at least the important ones? This paper addresses this question, mainly by looking at a variety of specific instances in which we have learned something from open-ended observation.

It is true that what you perceive is quite strongly affected by what you want to see: if you look out for someone you are much more likely to see them. But we also know that we can be surprised, that we can notice the unexpected (a tiger walking down the street). It is this familiar possibility that makes open-ended observation worthwhile. [But can we say anything more specific about it?]

This applies to both open-ended observation and to comparable measures. That is, we can be surprised by comparable measures such as the outcome of an experiment; but can also be surprised by quite other things. However it is not talked about in discussions of experimental methodology: experiments are designed to achieve comparable measures, even though often they are occasioned by unexpected observations; and also, they are often contrived only by close attention to informal observation that tells you how to structure things, to control variables ....

References

Draper,S.W., Brown,M.I., Edgerton,E., Henderson,F.P., McAteer,E., Smith,E.D., & Watt,H.D. (1994) Observing and measuring the performance of educational technology TILT project report no.1, Robert Clark Centre, University of Glasgow

Draper,S.W., Brown, M.I., Henderson,F.P. & McAteer,E. (1996) "Integrative evaluation: an emerging role for classroom studies of CAL" Computers and Education vol.26 no.1-3, pp.17-32 and Computer assisted learning: selected contributions from the CAL 95 symposium Kibby,M.R. & Hartley,J.R. (eds.) (Pergamon: Oxford) pp.17-32

Parlett, M.R. & Hamilton,D. (1972/77/87) "Evaluation as illumination: a new approach to the study of innovatory programmes".
(1972) workshop at Cambridge, and unpublished report Occasional paper 9, Centre for research in the educational sciences, University of Edinburgh.
(1977) D.Hamilton, D.Jenkins, C.King, B.MacDonald & M.Parlett (eds.) Beyond the numbers game: a reader in educational evaluation (Basingstoke: Macmillan) ch.1.1 pp.6-22.
(1987) R.Murphy & H.Torrance (eds.) Evaluating education: issues and methods (Milton Keynes: Open University Press) ch.1.4 pp.57-73

Parlett, M. & Dearden,G. (1977) Introduction to illuminative evaluation: studies in higher education (Pacific soundings press)