Last changed 20 July 1999 ............... Length about 6,000 words (46,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/resq.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [this page]

MEASURING LEARNING RESOURCE USE

M.I.Brown, G.F.Doughty, S.W.Draper, F.P.Henderson and E.McAteer
Dept. of Psychology, University of Glasgow, Glasgow G12 8QQ, UK.

A version of this paper appeared in Computers and Education 1996 vol.27, pp. 103-113.

This paper describes work by the evaluation group within the TILT project (Teaching with Independent Learning Technologies). Enquiries about TILT generally should be sent to the project director: g.doughty@elec.gla.ac.uk or G.F.Doughty, Robert Clark Centre, 66 Oakfield Avenue, University of Glasgow, Glasgow G12 8LS, U.K.

Contents (click to jump to a section)

Abstract

Contact time is only part of a student's learning, especially in higher education, and teachers and lab classes are only two of the resources students draw upon. Despite this, there is a prevalence of studies which appear to be evaluating a piece of courseware in isolation. Learning gains from a CAL package are important, but since acquisition and retention of knowledge is really what is ultimately important and will depend on other learning experiences in the course, an additional question in any learning situation therefore is: what resources is a student using? And, following up on that, which are most useful, are some better than others, or do they complement each other in essential ways?

This paper describes the design and application of the Resource Questionnaire, the instrument we are developing in an attempt to gather information on the learning resources used by students. The resources asked about may include not only lectures, tutorials and courseware, but books, handouts, notes, and discussions with other students.

Some preliminary results are described and the importance of this information to teaching staff in assessing and increasing the value of the resources to students by ensuring their effective integration into a course, is discussed.

INTRODUCTION

This paper reports the development of a questionnaire instrument designed to measure how a piece of CAL (Computer Assisted Learning) operates as part of a larger set of resources available to students. We have used this as part of a larger package of instruments in what we term "Integrative Evaluation" [1]. Here we focus on how a piece of CAL relates to other aspects of a course. Despite the fact that acquisition and retention of knowledge is what is ultimately important to a student, not whether the source was CAL, and that in most cases learners have multiple resources to draw upon, many studies appear to be evaluating a piece of courseware (i.e. a CAL package) in isolation. A few authors have identified this: Laurillard [2] regards the integration of the CAL package into the course as a whole to be very important, and Blondel et al. [3] discussing computer-assisted language learning (CALL), state " since computers are not used in isolation, CALL should not be evaluated in isolation".

Learning gains from a package are important. In a controlled situation (e.g. in scheduled classes) pre- and post-tests may show learning gains. However, in an open access situation (which is how a lot of CAL material is used), if students don't use it, or can't use it, then however good the package was shown to be in controlled trials, in these situations it will have no positive effect on student learning. Students may not have been able to access a CAL package due to a shortage of computers or computer lab hours, or they may lack the time or the motivation to attempt to use it, or they may just not have been told about it. As Rowntree [4] emphasised in 1974, "Innovation may appear ineffective but may in fact not be properly implemented". Thus effective integration of CAL or any other innovation into a course is essential otherwise no matter how good it is, it will not be of value to students.

The CAL package is in general just one of many resources that is available to students and like other resources may or may not be used, and may or may not be the one that matters for an individual student, just as some may doze through a lecture but then learn the material from a textbook. Differences between students are probably as important as those between teaching interventions: no intervention will ever suit all students confronting it, but a good intervention will have only a small number who do not feel it is of use to them [5]. In fact this point applies much more widely than to recent attempts to introduce CAL in higher education. Firstly, it applies in principle to all educational interventions and resources: all of them operate not in a vacuum but as part of an ensemble of resources, where the main aim is that the ensemble as a whole should be effective. This applies particularly in higher education, where students have, and as an educational principle are meant to have, considerable responsibility and control over how, when, and what they learn. Nevertheless most evaluation tends to assume that one intervention (typically the lecture) is the primary resource. For instance many standard course feedback questionnaires [6] do not have questions addressing the relative value of resources.

On the other hand in distance learning and resource based learning course designers are much more conscious of providing multiple, partly redundant, resources and are more likely to seek information about their relative roles and effectiveness, although rather little seems to have been published on this. This may be because it seems to those working on distance learning too obvious to mention, even though as noted it is not to those working on evaluation in campus institutions. In this paper we take the view that CAL should be evaluated more in the way that distance learning would be than in the way that lecture based courses usually are.

This is partly because CAL cannot replace all the features of a typical lecture, but still more because in reality no one resource in Higher Education is important by itself. The issue for all courses in fact is of how students use and benefit from each of the resources available to them. For instance, textbooks are important on most courses, but students do not use them while in the classroom. In other words, a learning activity thought by teachers as well as students to be central, goes on out of sight of the teacher who in fact has almost no information about its quantity, quality, or importance. Lecturers' "consultation" hours, questions in lectures, borrowing other student's notes, and peer interaction with classmates are other largely unmonitored resources. Any study of learning in natural conditions, at least in Higher Education, needs to address this question of the pattern of resource use.

A question in any learning situation therefore is: what resources is each student using? And, following up on that, which are most useful, are some better than others, do they complement each other in essential ways? This is of interest for at least the following reasons:

We have developed a questionnaire, the Resource Questionnaire, in order to measure the use and value of the different learning resources available to students. This instrument has undergone significant changes since its inception, and it will undergo more, but it promises to address the need for measuring where any particular resource being evaluated (e.g. courseware) fits into the learning situation as a whole and whether it is being successfully integrated into the course. From this we hope to find out the value of CAL (and other resources) to students, and to increase the value of CAL by determining ways to ensure its effective integration into courses in the future.

THE RESOURCE QUESTIONNAIRE

The Resource questionnaire itself

Downloadable versions of the resource questionnaire may be found here.

An example resource questionnaire

Page 2 of the Resource Questionnaire given to students studying Portuguese.

Usefulness of Resources

In the following table, tick each resource you have used during the course, tick how useful you consider each was to you in learning/understanding Portuguese. Please give reasons for your answers. Extra comments can be written on the front of the form.

Resource Use tick if used Usefulness of Resource Reason
not at all useful not very useful useful very useful extremely useful
classes with natural language speaker              
computer class using De Tudo Um Pouco              
course book              
reference books              
handouts              
material from the Audio Visual Library             videos? books? tapes?
Portuguese course from Audio Visual Library             which?
on-line material (eg data-bases, dictionaries etc.)             what?
computer open access De Tudo Um Pouco              
weekly class assignment              
term exam              
sample exam questions              
past exam papers              
own class notes              
own notes from computer session              
discussions with students on course              
discussions with students or friends, not on course              
discussions with class teacher              
discussions with course tutor              
discussions with demonstrator              
on-line message box              
E-mail communication             who with?
Other resources             please specify

Aim of the Resource Questionnaire

The questionnaire was originally developed in order to determine what resources students were using to learn or understand a topic or course and how useful they considered them to be, but also now includes additional questions relating to the value of the resources to the student and their accessibility. Learning resources are defined as anything that a student uses to learn and understand a topic. The questionnaire lists the resources that the teacher managing the course considered were available to the students studying a particular topic or a whole course. The resources asked about include not only lectures, tutorials and courseware, but books, handouts, notes, and discussions with other students. The questionnaire can be used at the end of or during a course where CAL use was scheduled, and also where CAL was optional on open access. It can also be used in courses where no CAL package is available in order to assess the use and value to the students of other resources.

The questionnaire asks students to report their views on the following:

On-line monitoring of CAL package use

Packages may be being used by teachers with little computing experience who may not have the ability or the computing support to set up a system to monitor student use of a package. A simple questionnaire is therefore often easier to administer in order to determine if the package is being used and if students consider it to be useful to them. In any case with computer monitoring, unless students are tracked through their use of a package, it is not possible to determine if they are actually using that package or just logged into it whilst engaged in some other pursuit. In any case, such on-line monitoring will not give information about other resources.

Creating a particular Resource Questionnaire

Although the general design of the Resource Questionnaire is re-used across studies, it must always be customised for each new application. These choices include: creating a list of the resources available and significant in the particular course, tailoring prompts in the "Reason" column, and varying the design to cover any particular issues the teacher is interested in. This is done by consulting the teacher in charge.

Use of resources by students and perceived usefulness of each resource

Figure 1 shows a page from a Resource Questionnaire used in order to determine the use of resources by students learning Portuguese. It allows students to indicate their use, and the perceived usefulness of each resource. In each case the students can indicate their response on a 5 point scale ("not at all useful" to "extremely useful") and can add reasons for their answer. A completed page of a questionnaire given to students studying Unix is shown in Figure 2. It is of interest to look at the percentage of those students on a course who actually used a resource and then at the percentage of these students who found it "useful", "very useful" or "extremely useful", as even if few students use a resource, if they react positively to it then it can obviously be of value, and the reasons for other students not using it should be examined. Examination of the resources used by individual students and their reported degree of usefulness should enable staff to build up learning profiles of successful students which might enable them to support less able students in developing good study skills.

Amount of time available to students to use the resources

Boyd and Mitchell [8] note the importance to the student of knowing the amount of time available for using a package as this can affect their perception of appropriate pacing and/or rate of achievement. It is therefore important to determine whether students felt that there was enough time available to use the CAL packages. This is important when time is scheduled for their use and also if it is available on open access as, if students do not believe that there is enough time to use a CAL package or other resource, they may well not even attempt to use it. We hope that by asking students to report on a five point scale ("far too little" -"far too much" time available) how much time they considered was available to them to use the resource then any real problems may be identified. The percentage of students who report that there was "far too little time" / "not enough time" to use a resource is important as this should identify if there is an overload at that part of the course. In addition, looking at the perceived usefulness of a resource and the perceived availability of time to use it could identify whether a re-allocation of time within a course would result in more effective use of student and staff time (e.g. replacing some lectures with tutorials).

Resources which were difficult to access

" What matters most to the young is the effective access to the new educational goods and the presence, at his or her side of a competent tutor." [9], and Scanlon et al. [10] similarly emphasised the need for careful planning and costing to ensure easy access to CAL packages. We agree that this is very important as problems of access to CAL packages which are meant to be available for repeated use could easily discourage students. In our studies we must determine whether students do experience difficulties in gaining access. If a large percentage of students are identified as "not knowing about a resource" or "having difficulty accessing a resource" then there may be a real problem in the presentation of that resource to the students and /or the availability of that resource to the students. Answers to this question should also determine if disabled students are having problems of access, as these problems would be specified in the comments.

Resources which increased/ decreased a student's interest in the subject

Kenning and Kenning [11], considering computer assisted language learning (CALL), suggest that CALL may well find its justification, not in improved test scores, but in positive reactions from learners and teachers and that the value of CALL lies in the fostering of motivation and the enhancement of language learning. The Resource Questionnaire therefore attempts to further detect positive/negative reactions from students by asking them about learning resources which increased or decreased their interest in a topic.

Re-use of resources by students

If students report that they will use a resource again, it suggests that they value it and therefore if they had also rated it positively in terms of usefulness, this could reinforce their answers on usefulness. However it should be borne in mind that reporting that they will use it again does not mean that they will, and a further questionnaire or follow-up interviews would be necessary to confirm any re-use of the resources.

Resources considered by students to be essential or non essential

By asking students which resources they could or could not have managed without in learning and understanding a topic or course, the questionnaire further attempts to identify the usefulness of the resources available to the students. This question is only related to the resources that they actually used, and does not therefore give information on resources that they actively did not use because they considered that they could manage without them. This should be monitored in future questionnaires as students tend to ignore anything not felt to be essential [12], and if this is found to be a reason for non-use of CAL it would need to be addressed.

SOME PRELIMINARY RESULTS FROM USE OF THE QUESTIONNAIRE

Versions of the Questionnaire have been used in eight studies so far. From these we shall concentrate on some of the results obtained which illustrate the value of the questionnaire to academic staff.

  1. In a course on 16th Century Musicianship Skills, a CAL package was available on optional open access, and training in the use of the package was also optional. There was no scheduled use. This situation was partly because the package had been produced by a developer in the TILT Project who considered there was a need for such a package to aid students, and not by the lecturer who taught the course. As on-line logging was unavailable, the Resource Questionnaire was administered at two points during the six week module to assess actual use, rather than relying on predicted use which has been found by us to be unreliable. Delivery of the Resource Questionnaire to these students was pressured - only five minutes for questionnaire completion at the start of each of two lectures was available. This prevented the necessary full introduction of the questionnaire, and few students were able to complete it in the time allocated.

    Some interesting findings emerged despite these problems. For example, to a CAL-naive audience, wording is essential. Stating "Have you used the package" was not enough, as many students had opened the package during the training session and felt this counted as package use, confounding the results. However conclusions were drawn from the results which guided the subsequent evaluation. The package did appear under-used, though most students (including those who had only opened it up during the training session) felt it would be "useful" to "extremely useful". In subsequent interviews and focus groups prompted by the Resource Questionnaire results, reasons given for non-use included time constraints and inadequate study skills. As the version of the Resource Questionnaire used in this study was one of the earliest, the specific question about the amount of time available for resource use was unfortunately not asked. It was concluded that the lack of use was a reflection of a student problem or difficulty, and the fact that students were not reminded of the existence of the package and its potential benefits. External constraints, such as problems in gaining access to the CAL material, were not reported. Positive presentation of the CAL to the students, and the possibility of training the students in study skills (for CAL and other resources) and time management, are under discussion in light of these results. More recently the lecturer in charge of the course has realised the students' need for the training that the CAL package offers and has asked for it to be made available to the students. Since he has initiated the request the chances of it being positively presented to the students are much greater.

  2. Second year B.Tech. Ed. students studying electronic circuits could access CAL packages during three scheduled labs and also on open access. Only 27% of those completing the questionnaire (three out of eleven students) used the CAL package. Two of these students found the sections of the package they accessed "useful" or "very useful". The other found them "not very useful". A further questionnaire later in the term indicated that a total of six students had used the package. Of the initial three, the two students who rated it positively in questionnaire 1 said they would refer to it again. Other students found it difficult to understand. Information from the lecturer may partly explain the negative reactions from the students. The scheduled sessions were not compulsory and due to other commitments, illness etc. neither the lecturer, technician, nor demonstrator were present at the second class. Changes in the method of presentation of the CAL to the students with more directed help may be necessary.

  3. Twenty one students studying Portuguese all used a CAL package during scheduled classes where help was available, and all rated it positively, but 33% of the class reported that there was "not enough" time available to use it. 43% of the class also used CAL on open access and of these, 44% reported that there was "not enough" or "far too little" time available to use it. [Open access: 9am-9.15pm (Mon.-Thurs.) and 9am-4.45pm (Fri.), outwith scheduled use by other classes, and usually 9am-5pm in vacations]. Reasons given were mainly concerned with the computers not being available due to other classes using the lab. When asked about resources that increased or decreased their interest in a topic, the CAL package scored highly (43% reported that it increased their interest). Similarly the classes with a native speaker were rated highly by all the students, but again 33% of the class considered that there was "not enough"/"far too little" time available for this resource. This was partly the result of administrative problems in ensuring the presence of a native speaker. 23% of the class reported that these classes increased their interest in the subject and 19% of the class noted that the audio visual material increased their interest. In this situation the students are using the CAL package and rating it positively, but staff will address the issue of increasing the access to it and the time available to use it. In addition, due to the feedback on the classes with a native speaker, staff are working to ensure the continuation and reinforce the role of these natural language classes.

  4. A CAL package was developed to aid Social Science and Accounting and Finance students study statistics. In one study, twenty eight Sociology students (Year 3) used the package during a timetabled session and of these only two (7%) reported that they would not use the package again. This question which is normally included in the Resource Questionnaire was in this case included in a Post-Task Questionnaire. As further use of the CAL package was being logged, this allowed us to compare predicted re-use with actual re-use of CAL. Of the twenty six students who reported that they would re-use it only four (14%) did so, in the following four months, two for 5 minutes or less, one for 11.5 minutes to complete all the sections, and one for 42.5 minutes apparently for revision. This indicated that predicted re-use of CAL is not reliable. However, the two students who did not want to re-use it also did not appear to value it as a resource, as they reported that they would not recommend it to others, and had not learnt anything from it.

  5. Dental students used CAL packages during scheduled classes with help available. The Resource Questionnaire indicated that 95% of the class of forty two students used the packages, and of these 78% rated them positively (35% of those who used them rated them "very" or "extremely useful"). Only 26% of the class used the video tapes which were available, but of these 82% rated them positively. 73% of those who used them considered that there was "not enough" or "far too little" time available for their use. Other resources that there was not enough /far too little time available for were textbooks, teaching clinic, discussions with academics. Resources that were reported as being difficult to access by the class of Dental students were: video tapes, 24%; textbooks, 17%; CAL, 10%. Some students commented "What videos?" Staff considered the videos a very useful resource and had not been aware of the infrequent use of them. In this Dental course it was therefore decided that the students should be made more aware of the presence and means of accessing the videos, as those who did use them did find them useful.

  6. Forty-one students who were learning to use Unix also completed a Resource Questionnaire. All resources were rated positively by the majority of students using them. 49% of those who attended the scheduled labs and 27% of those who attended unscheduled labs, considered that there was not enough time available for using these two resources. This problem relating to computers is further identified by the fact that 27% of the class reported the computers as a resource difficult to access. This was reported as being due to too few computers being available during scheduled classes and the lab being too crowded at all times. In addition, 86% of those who used Exercises as a resource rated them positively and of these 38% did not consider that there was enough time available to spend on them.

The Resource Questionnaire can highlight problems in all courses not just where CAL is in use (e.g. in the course on Unix). Also additional questions can be included in the questionnaire by lecturers or evaluators concerning any aspect of a course. For instance, in the Unix course, lecturers used the questionnaire to quantify problems of which they were already aware. Students were asked if they considered there was sufficient time to work on Unix and whether there was sufficient time between a lecture and the related lab. (There was a one hour lunch break). 32% reported that there was not sufficient time to work on Unix, and 32% that there was not sufficient time between a lecture and the related lab. The Resource Questionnaire itself then identified where students required more time (i.e. in the scheduled and unscheduled labs and in doing exercises). Results obtained from the same students learning C programming showed that 79% of them considered that there was not sufficient time to work on C, and again the main problems were identified as not enough time in the labs and on the exercises. We hope to question these students after a year in order to determine their opinion of the courses in light of their experience since then. In the meantime changes are being made in the courses with re-timetabling and the introduction of recommended summer reading.

From these examples and others, it is apparent that when a new resource (in this case CAL) is introduced into a course, it is useful to know whether it is being used and considered useful by the students for whom it was intended. In addition, in any course it is useful for teachers to know which resources students are using and their opinion of them. In the examples above, the CAL packages which were used by students in scheduled classes were used and rated positively by the majority of students. However, where the CAL packages were optional in open access or scheduled situations only a small percentage of students used them, although many of them did rate them positively. That this could also be true for resources other than CAL was illustrated by the infrequent student use of videos in Dentistry.

IMPROVING THE VALUE OF CAL PACKAGES AND OTHER RESOURCES BY ENSURING THE EFFECTIVE INTEGRATION OF THEM INTO DEGREE COURSES

Many people consider that the use of independent learning technologies will bring benefits to education. What these benefits will be, we believe will depend to a great extent not only on the quality of the packages but also on how these packages are delivered to the students. This in turn depends not only on staff integrating them into courses in such a way that students are motivated to make full use of them, but also on the provision of a sufficient number of computers and support to allow students easy access to the packages. If this is not done effectively then, as we have found, packages may not be used. This is similar to what Jones & O'Shea [12] found, which was that students may ignore anything not felt to be essential and that perceived educational benefits often have little to do with the amount of use of CAL. Use often is related to the time available to use it, and the problems associated with doing so.

We consider that effective integration should increase the value of a package to students.Staff know that some students "don't listen", or "don't try to access resources till the last minute if at all". How much credence should be given to the percentage of students reporting "not enough time available", "difficulty in accessing a resource", etc. is debatable as some students may use "lack of time" as an excuse rather than a reason for not using a resource. However if analysis of the results shows that students who used a resource, even if it was a small percentage of the class, found it useful but considered that there was too little time available or that it was difficult to access then there may be a real problem and further methods such as focus groups may be necessary to identify it. Similarly if students report that they did not know about a resource or did not know how to access it then there may be a problem, and therefore the way students are told about the resource and the method of accessing it should be improved. Access problems may include shortage of available computers (too few altogether, or lack of access to computer labs in the evening etc.).

Brudenelle and Carpenter [13] emphasise that lack of interaction between faculty and students and faulty orientation to the CAI (computer assisted instruction) as two of the reasons for students' negative attitudes towards CAI. Presentation of the package to the students by the lecturer will be important as it will affect their perception of task and their motivation. We have stressed this in [14], where we emphasise that students should be made fully aware of the resource's value in helping them fulfil the objectives of a course, and be clear about the contribution the resource will make in helping them successfully complete the course's assessments. Similarly in Case Studies from MIT Project Athena [15] it is stated "In education tools can be used with positive or negative results. Education is, after all, one of the subtlest of human activities. And whether the potential benefits that the computer seems to offer are actually achieved depends largely on the teachers who use it."

EFFECTIVE USE OF THE QUESTIONNAIRE

Not only must the courseware or other resource be properly integrated into the course, but also the evaluation itself should be planned and implemented with care. The Resource Questionnaire can take up to 20 minutes for students to complete carefully, allowing time for instruction as to its completion, and for its return to the researcher. Failure to plan the administration time carefully can result in confused students and even more confusing results. If time is limited, the number of resources must also be limited as, if too many are listed, students cannot be expected to concentrate fully on all their responses.

If the questionnaire is administered during the use of one of the resources, for example a lecture, then students who do not attend lectures will be omitted and their reasons for this and their methods of compensation not detected.

Finally predictive questioning, in this case asking students if they will use a resource again, was not a reliable method of estimating re-use as subsequent logging of use showed. It did, however, give additional information on the perceived value of the resource to the student.

CONCLUSIONS

Considerably more work could usefully be done on this questionnaire and its use, and currently studies are underway to validate it by comparing results using the questionnaire with results obtained by logging the open access use of more of the courseware we are evaluating. It is in principle possible that students seriously misreport their use of resources (e.g. because they do not remember such activities accurately) and so more checks on validity are appropriate. However comparable work on whether people can report accurately from memory on their use of and need for commands in user-computer interfaces found accuracies above 80% [16] so confidence in our instrument's validity seems reasonable.

The effective integration of CAL into a course does not take place automatically, but requires effort on the part of staff involved and may involve new study skills on the part of students. It is apparent that many people realise the importance of integrating CAL into the curriculum and have identified where problems may occur, and like Scanlon et al. [10] we recognise the need for careful planning before introducing a CAL package into a course [5]. However, once a new resource has been introduced, it is important to be able to determine if the integration of it into the course is effective, and we consider the Resource Questionnaire to be a simple method of achieving this. In addition it may also:

As regards the effective integration of a resource such as CAL into a course, which is what we are concentrating on at present, the questionnaire should enable the lecturer to obtain the answers to these important questions about a resource:

If the answers to any of these are negative, then it may be necessary to adjust the situation in which CAL is used in order to maximise the benefits to the students as only then can effective learning take place. By examining the comments made by the students in the questionnaire and if necessary in follow-up interviews, it should be possible to determine which of the following needs to be actively increased:

Certainly our experience shows that the questionnaire can enable lecturers to identify problems associated with the use by students of the different learning resources available to them, and consequently to improve the value of these resources to students by improving their integration within the course. Although we have most often used it in evaluations of the use of CAL, we have also used it in courses without any CAL component. In principle it is applicable to any course, is of greater interest the more a course revolves around multiple resources of comparable importance, and should be applicable to distance learning courses although we have not yet had an opportunity to test this. Together with our other instruments [1], it provides a method of evaluating the integration of resources (including CAL) within a course.

ACKNOWLEDGEMENTS

A version of this paper was originally presented at CAL'95, and we thank the audience for their feedback. We are furthermore grateful to an anonymous referee for pointing out the relationship of this work to a longer tradition of evaluating distance learning.

The work was done as part of the TILT (Teaching with Independent Learning Technologies) project, funded through the TLTP programme (Teaching and Learning Technology Programme) by the UK university funding councils (DENI, HEFCE, HEFCW, SHEFC) and by the University of Glasgow. The studies discussed here could not have been done without the active participation of all the TILT Research Assistants and many members of the university teaching staff to whom we are grateful, particularly: Stephen Arnold, Peter Dickman, Mike Harland and Denis Kinane.

REFERENCES

1. Draper, S.W., Brown, M.I., Henderson, F.P., and McAteer, E., Integrative evaluation: An emerging role for classroom studies of CAL, Computers & Education. 26, 17-32 (1996)

2. Laurillard, D., Rethinking university teaching: A framework for the effective use of educational technology . Routledge: London. (1993).

3. Blondel, M., Brick, N., Horrocks, G. and McBride, N., Theory and Reality of Evaluation; Evaluation and Text Jumbler. Computer Assisted Language Learning, 3, pp.55-67. (1990).

4. Rowntree, D., Educational Technology in Curriculum Development . Harper & Row. (1974).

5. Creanor, L., Durndell, H., Henderson, F.P., Primrose, C., Brown, M.I., Draper, S.W. and McAteer, E., The Hypertext Approach to Information Skills: Development and Evaluation. Technical report no.4, TILT project, Robert Clark Centre, University of Glasgow (1995).

6. Marsh, H.W., "Student's evaluations of university teaching: research findings, methodological issues, and directions for future research" Int. Journal of Educational Research, 11, 3, pp.253-388. (1987).

7. Draper, S.W., Brown, M.I., Edgerton, E,. Henderson, F.P., McAteer, E., Smith, E.D., and Watt, H.D., Observing and Measuring the Performance of Educational Technology. Technical report no.1, TILT project, Robert Clark Centre, University of Glasgow (1994).

8. Boyd, G. McI., and Mitchell, P.D., How can Intelligent CAL better adapt to Learners? CAL91 Symposium, Lancaster. Computers & Education 18, pp. 23-28. (1992).

9. OECD Report, New Information Technologies, A Challenge for Education.. New Information Technologies Centre for Educational Research and Innovation. (CERI). OECD. Paris. (1986).

10. Scanlon, E., Jones, A., O'Shea, T., Murphy, P., Whitelegg, E. and Vincent, T., Computer Assisted Learning. Institutional Research Review, 1, 59-79. Open University Press. (1982).

11. Kenning, M-M., & Kenning, M.J., Computers and Language Learning. Current Theory and Practice. Ellis Horwood Series in Computers and their Applications. (1990).

12. Jones, A.C. and O'Shea, T., Barriers to the Use of Computer Assisted Learning. British Journal of Educational Technology, 3, pp. 207-217. (1982).

13. Brudenell, I., Carpenter, C.S., Adult Learning Styles and Attitudes Toward Computer Assisted Instruction. Journal of Nursing Education, 29, pp.79-83. (1990).

14. Shaw, R.(Editor) (25 authors), Using Learning Technologies. Interim Conclusions from the TILT Project.. Technical report no.3, TILT project, Robert Clark Centre, University of Glasgow. (1995).

15. Hodges, M.E. & Sasnet, R.M., (Eds.) Multimedia Computing: Case Studies from MIT Project Athena. Addison-Wesley Publishing Company, Inc. (1993).

16. Edgerton, E.A., Draper, S.W., & Barton, S.B. Feature checklists in HCI: some basic results. Interchi '93 Adjunct proceedings, pp.189-190. Ashlund, S., Mullet, K., Henderson, A., Hollnagel, E. & White,T. (Eds.). (ACM). (1993).

Web site logical path: [www.psy.gla.ac.uk] [~steve] [this page]
[Top of this page]