17 Jun 1997 ............... Length about 5400 words (34000 bytes).
This is a WWW version of a document by Steve Draper. You may copy it.

DRAFT

Teaching collaboration: how we made it work

Contents (click to jump to a section)


A Technical Report
by
Stephen W. Draper
Department of Psychology
University of Glasgow
Glasgow G12 8QQ U.K.
email: steve@psy.gla.ac.uk
WWW URL: http://www.psy.gla.ac.uk/~steve

Preface

This is a note (second in series) on how to do teaching collaboration, based on what we did and what worked in MANTCHI.

This note stems from the project meeting 27 May 1997.

Acronyms

Introduction 1: nature/status of this document

This is a "case study" of how academics from 4 different HEIs made agreements to collaborate in their teaching at a project meeting on 27 May 1997. Because it is only one episode, it is rather uncertain what the key features were that would generalise to future episodes. Nevertheless it seems to me worth trying to record the episode while it is still fresh, as how to make collaborations work is not entirely obvious yet important.

This report is based firstly on my own views and perceptions as a project participant; secondly my on talking to other project members (PDG, ACK, Terry Mayes, David Benyon, Julian); and thirdly on Margaret Brown talking to three other participants at the workshop who were HCI teachers but not MANTCHI members.

Introduction 2: Context of the MANTCHI project

Jan 1997 - July 1998.
4 HEIs
Use of MANs
Tutorial"material"
Answer garden
Learning by overhearing

Introduction 3: ped-led issue

The MANTCHI project proposal was geared to doing real teaching applications and so was less nakedly technology-based than earlier projects that simply developed hardware and software. Nevertheless its specific promises were expressed as the use of a software technology and development of material, rather than as the achievement of any pedagogical needs, tasks, or achievements.

However the distinctive feature and strength of the project is the assembly of a collaboration of many teaching academics: cross HEI, all teaching one subject (HCI).

To satisfy the project promises we have to: a) collaborate across HEIs in our teaching, b) use AnswerGarden (AG) software somehow in that teaching.

The problem, as I now see it, is that:

You almost certainly won't get one of these by aiming and planning for another. Our problem was that we had promised in terms of technology use, but actually needed collaboration.

How could we have done it?

Tech led: spec and impl. the software. Fit it into some course without much forethought.

Ask "what is AG good for?". This didn't seem to spark much enthusiasm or action. After all, our courses as now running didn't need any new discussion mechanism. That is, as teachers there wasn't any obvious specific need for it or place to use it.

How did we do it?

1. Offline attempts

The project as a whole decided that everyone should post offers of collaboration. This was done by email early in 1997. [and posted in BCSCW? and collated by Sandra?].

This did not take off and immediately lead to multiple collaborative teaching instances. However three such cases were done (see next subsection). And furthermore, these offers may have contributed to agreements made at the meeting (subsection 3). I am aware of the following case. I made a general suggestion in an email that teachback between students is a good idea on general pedagogical grounds, and would probably work particularly well across HEIs. Tom Buggy posted suggestions for collaborations he would like to be part of, one of which was such a teachback situation. Before the meeting, I re-read these old emails, and during the meeting as a result suggested teachback to Phil Gray as a worthwhile offer; and when he agreed, suggested to Julian Newman that he phone Tom Buggy (who wasn't present at the meeting) to see if he would like to be the partner on this; and a collaboration on this was agreed.

2. Actual collaborations before the meeting

Three cases of actual collaboration were both arranged and performed before the May meeting. These are described in another memo. However they could be seen as of different character, which explains why they happened and others didn't.

Two of the cases were of actual student experience of CSCW technologies in courses whose subject matter included CSCW: they were then in part lab. exercises, and to experience CSCW you more or less need collaborators at a remote site. These cases were: using BCSCW software; and using Superjanet video conferencing.

The third case was guest teaching by David Benyon of Napier for Alistair Kilgour of Heriot Watt. This did not really involve any special technology; and it was in a long tradition of occasional guest lecturing.

3. The meeting (Project meeting/workshop of 27 May 1997)

The meeting was proposed at the previous project meeting as a 1 day workshop, and gained more or less unanimous agreement and determination. An agenda for the day was planned in the interim, but in the event this was not strictly adhered to.

What was actually done was:

  1. Presentations, which however do not seem to have been important for the organisation of collaborations. However they will have served as a status report for the project. In particular, the presentation about software basically took the decision to rewrite from scratch, leaving the feeling that decisions could be taken apart from that: specify the software later, rather than trying to decide the software and plan the teaching around it. This may have been important in allowing the technology to be backgrounded.

  2. Morning small groups, people from different HEIs carefully mixed in groups, each group had a topic-to-be-taught area. They were supposed to discuss their assigned topic area and how to collaborate.
    No-one seems to have felt their time was wasted, so all the groups were probably useful to the individuals involved. But from the viewpoint of what was productive for the meeting as a whole, the important outcomes were a) the idea of changing the afternoon's agenda to one of proposing and bidding for units; b) 2 ideas on what the units of collaboration should be. [Was it significant that most of this came from the group with Phil and Terry in: the most critical and the most enthusiastic w.r.t. collaborating in teaching: maximised the tension, and got the biggest contribution to the solution.?]
    b1) An ATOM: a 1 week chunk of material to offer. [Although only briefly discussed, this seems to mean about 10 student hours of work, expository resources plus a task for the students, the idea of using an AG-like tool to supplement this, and suggestions for assessment by the author which would however be the responsibility of the local deliverer.]
    b2) Student projects, where one HEI's students do the design, and the other's do the evaluation.

  3. Plenary. Making public the different ideas, agreeing on a new agenda for the afternoon.

  4. Afternoon groups divided by HEI: 30 mins to come up with a set of offers of ATOMs that could be delivered.

  5. Some visiting of other groups to strike initial deals i.e. accepting others' ATOMs.

  6. Public listing of
    a) ATOMs on offer
    b) deals already agreed
    c) ATOMs wanted (none of these actually declared at this point, but there were some indications that more of this would have been good).
    d) Then people "voted": declared which ATOMs they would be interested in receiving.

How was it?

  1. It felt good to everyone. We'll have to see what actual action outcomes there are, but currently I think we will now proceed to a considerable number of teaching collaborations.

  2. What was different or significant? Basically:
  3. What is and isn't this?
    *It will bring about HEI collaboration over teaching (one project aim).
    *It may involve AG: but this is not yet explored or planned beyond a few phrases.
    *It isn't designed in order to improve learning (though this could be a consequence). I.e. we didn't ask ourselves what we could do that would most improve learning and teaching, though no doubt we did pass suggestions through a personal filter of what would be OK by us on quality grounds.
    *It isn't designed to support learning by overhearing (tertiary courseware?), although this could be a consequence.

  4. Weaknesses of how we did it.
  5. Is this a ped-led way of designing project activities? See later section.

  6. How does this help the project aims? See later section.

  7. Agenda shift. Previous project meetings had tended to focus on technology issues, and not get down to specifics of teaching. The agenda for this meeting was subject-topic (what was to be taught in collaborations) first, then technology (already an important shift). Then the agenda was again changed during the day, so that the focus became a) format (i.e. learning activity) first e.g. ATOMs b) subject topic to be taught c) no time left for other details of the teaching e.g. format d) no time left for discussing technology.

Possible types of cross-HEI collaboration

  1. ATOMs: common chunks of course material. About 10 student hours of work, expository resources plus a task for the students, the idea of using an AG-like tool to supplement this, and suggestions for assessment which would however be the responsibility of the local deliverer.

  2. Project work student collaboration: one class evaluates the project products of another class. E.g. each designs a thing; and other class evaluates it.

  3. Teachback across HEI classes. E.g. 2 week 2 topic chunk. In week 1 each class prepares a topic (different for each class). At changeover, each class teaches the other about their topic (probably one on one student work; could be 1 teaches 3, or 3 teach 1). Then during the rest of week 2 students have to learn topic 2 as best they can, starting from that session. Both topics assessed for credit.
    For receivers: if no gain, they only lost an hour, but potential is to get a useful start. For transmitters: we know it will benefit them a LOT. Just get them to comply. Marks: mainly assess topic knowledge but could also mark the presentations; and use receiver exam marks and grading of presentation too?
    Evaluation: Ask students at end of week 2 how much help giving the teachback presentation was to them; and how much help receiving it was. Look at their exam marks; particularly try to compare benefits of giving and receiving for each student.
    More complex design: each class split in half for topics, so can do within-class comparisons on exams: but this would remove the need for cross-HEI collaboration, while being a better test of teachback alone.

  4. As before, use of CSCW technology between HEIs. In effect, a "lab exercise", although the exercise could also attempt real work at the conceptual level at the same time as the content of the exercise.

  5. Seminars across HEIs (cf. Netsem). That is, do topics in which students prepare and present by turns; but the presentations are to students at other HEIs as well as their own; and might be by email or video conference.

  6. Ditto enhanced by AG. THe idea here is to have an AG of past discussions, from which students will pick a topic for their seminar. In Erica's phrase: this would be a question garden i.e. a collection that shows what good questions are. The garden might be the archive of some email discussion group. My sense is that in many such groups, a few topics recur regularly (besides many others that are dealt with just once), and often this shows which issues remain debateable as well as important to the field. This is on important case of learning by overhearing: eavesdropping (lurking) does actually instruct lurkers about what the issues (not answers) are. In this context, these are the issues that are a good bet for promoting student discussion.

Pedagogy-led vs. technology-led planning

My own theoretical slant

I believe (Draper, 1997)that a main lesson from TLTP as a whole is that if you want achieve and demonstrate benefits to learning, you have to take an approach that has that as a goal: is pedagogically led and NOT technology led. The vast majority of projects and funding is technology led. In almost all cases this has not led to demonstrable learning benefits (unsurprising if these were not the goal being worked on).

The key difference in deciding what to do is whether you start by asking about how to develop the technology, or you start by asking how to improve teaching. On a smaller scale, the question is where to fit changes into our courses, and how to justify this (to students, to TQA etc.). In medicine, the difference would be between saying "I'm going to try out this new chemical on patients to see what it does" versus asking "What disease is there where we need improvement so much that a new unproven drug is likely to do more good than harm?".

It does NOT follow the route (which early MANTCHI meetings in effect did) of asking "what existing software is available?" then "How can we install it" then "You ought to use it (for the sake of the project, not the students)".

Is this a fair view?

I held and hold this paritcular theoretical view, which I have expressed in papers, and expressed prior to the meeting to the project. On this view, the project struggled because while it didn't adopt this view, and succeeded when it finally came round to acting on it. Is this the blinkered view of a prior theory, or does it describe what really underlay our experience?

Talking to other project members suggests it does get at an important and central issue, but also suggests important qualifications.

Aversion to educational prescriptive pronouncements

Firstly, most project members like most academics in general feel (even if they don't care to say this) that teaching is what they do, and educational theorists make them feel uncomfortable: disapproved of rather than assisted. So just saying something like "it should be pedagogy-led" wasn't very helpful or directly inspiring. At least as important in the meeting, was getting the social dynamics right, so people actually discussed and considered the right things. It was also probably the case that everyone basically felt they knew that teaching changes should be designed from an educational viewpoint, and it was a case of getting round to doing that thinking: it wasn't that such an approach seemed original or peculiar, but it did take something to make us actually carry it out.

Tacit expressions of pedagogical concern

But also: resistance from some participants (especially, say, me and Phil Gray) on the basis of ped-type issues to urgings to form collaborations. For instance, TQA requirements form a lot of pressure not to change courses, not to make any short term changes that affect the published curriculum. These pressures (TQA, and also any personal pride or concern about our current course design) act to maintain the current topics; so although trying out novel or improved methods is OK, keeping the topics tends to mean changing one's own methods rather than being able to use someone else. This tends against accepting other people's bids for the topics they can uniquely provide.

As things turned out, it would seem that project members show a considerable spread on this issue. Many reacted positively to the bidding process, seeing others' offers often as opportunities to replace topics in their courses they felt least good about.

Spending meeting time discussing technical (software) issues

What was true, was that project meetings had tended to devote considerable time to discussing the technology e.g. the different software available of the answer garden type, but that such discussions seemed relevant given our grant proposal, yet didn't actually help participants to see how it would be relevant to their teaching. In fact a lot more such discussion was scheduled for the meeting, but it didn't happen much and this was probably crucial.

Urgency

Another factor was a growing feeling of urgency: that we had passed through almost half the project, we only had one teaching year left, and we really had to get collaborations agreed if we were to get anything much done. Thus the subconscious reasoning may have shifted from "well first you choose or design the software, then you implement it, then you find a way to test it in teaching; so first thing is to discuss the software" to "If teaching collaborations across HEIs is going to happen we are approaching the last point where they can be planned; so organise them, fit in the software around that".

"Social dynamics": how the meeting was organised

Small groups were important to get more people actively thinking. Also, getting people together in one place may have been important in getting us to really generate bids, and to have this shaped and helped by instant reactions from colleagues and from those in other HEIs. Colleagues (small groups organised by HEI) are good for discussing to overcome objections; "customers" (the bidding process) are good for quickly seeing which ATOMS are in demand, and which not: this saves working up a careful proposal that would not be of interest to anyone.

Was it really ped-led?

Success came by focussing on actual specific teaching rather than technology. And in fact, within "teaching" it came by focussing on specific student activity rather than either topic or current student needs. That is, we had to decide on a unit or format for collaboration first, topics second. The different kinds of unit are distinguished by the kind of student activity or task to be organised: task-led rather than pedagogical-need-led.
  1. Yes: not technology-led but teaching&learning-led.

    However, it was NOT done in the way I argue for in my Niche paper: not by analysing what the biggest need or lack is in a current course, and then looking for something to address that bottleneck. Instead, in our meeting it was to do with a) units we could conceive of making changes around (not too big, related to the curriculum units we tend to think in terms of, made specific by specifying student activities). b) some self-interest: what would reduce our teaching burden and/or be interesting.

  2. No: the ATOM idea is about units of curriculum i.e. content.
  3. No: Self-interest: how it would reduce teachers' work, not improve student learning.
  4. No: not greatest pedagogical need (although that would be one criterion used for accepting others' offers i.e. drop topics we didn't feel we were delivering well).
  5. Yes: where would the input of collaborators fit in and carry a real part of the course.

Overall our decision process reflected how HEI teachers usually plan courses, not how they would maximise quality for students. But the net effect will probably be to raise quality a) because the things currently done least well will be selected for replacement by collaboration, and b) because at least some of the methods (student activities) will be better quality than what they replace e.g. contain more discussion, require students to exercise what they know more, expose students to more than one teacher's views.

How I'd like to spend meetings

Myself I would happily spend days discussing teaching HCI: far more time than we had. I'm sure I would learn a lot from other people's views, practices, experiences. This seldom happens in academic life, and probably won't happen much more on this project. This seems to also have been felt by those workshop participants who taught HCI but weren't MANTCHI members: feedback from them was enthusiastic about the opportunity to meet other HCI teachers and to discuss teaching with them.

What's all this got to do with the project aims?

Part of the issue that has to be faced if the ped-led view is adopted, is how to reconcile spending major project meetings and time discussing teaching instead of discussing the explicit project objectives.

Problem 1 People begin by feeling that project meetings should do project business; and that software use should go from requirements, to implementation, installation, use. Instead, we should have realised that teaching redesign and collaboration are major activities in their own right, that take an expensive amount of time: needing organisation and planning. They are a major project action in their own right, requiring significant resources.

Problem 2 Our project mainly promised to apply and try out AG software. But the collaborations we discussed were designed to be teaching and be collaborations across HEIs. They were not designed to use AG, nor to use the MANs (although these may well get used as a side effect). So on the face of it, collaboration is not addressing the main aim of the project, so how can we justify a focus on it.

Answer 1 The proposal promised use of AG software across the MANs. This can't be done without collaboration. The mistake was not to recognise that collaboration is a big thing in its own right, and must be addressed as a major activity in and of itself. It was as if we had promised to try out a drug in clinical practice without having made any plans about whether that was ethically acceptable, who the subjects would be, how to get a control group, etc. This planning wasn't done in the project proposal, so it must be done as a project action: even though it wasn't listed as a project objective, milestone, or activity.

Answer 2 If we achieve real and useful collaboration, that will actually be an important deliverable in its own right, and SHEFC will probably be interested. Secondly, we probably will get to use AG as part of it, and if so, then it will be in a sensible and meaningful context. Thirdly, the MANs will get used to some extent: how much is part of what we will discover; and again this will be in a meaningful context where other alternatives have been considered, rather than as a mere demonstration divorced from serious application.

Conclusion on the ped-led issue

Planning is quite hard, and usually only one aim can be focussed on at a time.
Such planning might often be done during the writing of a grant proposal. If not, it has to be scheduled as an activity that will take resources: a lot of time by the main project members. It would be reasonable to make such planning a major project activity and outcome.

What is in fact required to achieve politically and scientifically major results is to show that technology leads to improved learning, in this case using collaboration. This means a much more difficult design aim, of finding a way of simultaneously organising collaboration, technology use, and improved learning outcomes. Neither we nor other projects even attempt this.

Dropping topics to make room for collaboration

Often, collaboration means dropping existing topics on a course to make room for those offered in collaborations. This can be a problem.

    Against change

  1. In the short run, external TQA, internal academic committees, and giving students proper advance notification of course content all militate against course changes of either content or method (types of student task). Two years advance notice may be necessary for major changes (e.g. adding or dropping a topic like CSCW from the curriculum), at least in principle.

  2. HEI teachers often design courses to teach what they are interested in. Dropping this for something else is less attractive.

    In favour of taking help (collaboration)

  3. On the other hand, when you have to design a new course, or redesign an old one, you are open to help: in fact probably keen on getting it. This is an opportunity, but typically on a 4 year cycle i.e. once in 4 years.

  4. On a smaller scale, a teacher may have to take over part of a course they haven't taught before: the topics may be fairly fixed, but they will be keen to accept material on it from elsewhere.

  5. There may be topics in many courses that the teacher feels are least well done or least interesting to them (but taught because they seem necessary to the subject): favourites for dropping or getting them done by outsiders. This means at any time there may be 10% they would be most ready to change.

  6. But collaboration might not involve a change of topic, but additional and supplementary activities for students (e.g. teachback). This would favour units of collaboration other than ATOMs or expositions.

My reluctance

I was dead reluctant, and didn't agree to give up anything or to accept any collaboration on to my own HCI course, though I did offer two ATOMs for others. Does this mean I'm good or bad? Generous or defensive and uncooperative?

Proposed future recipes

This has given us hindsight. Given that hindsight, what would we now recommend to any group attempting the same i.e. wishing to organise teaching collaboration across HEIs?

Recipe 1

Organise a workshop for all participants: do not try to do it by email, but take the face to face aspect as crucial.

Begin by circulating (1) a written agenda (this recipe), together with (2) some ideas on units of collaboration, and suggest that participants come with proposals.

Plenaries, small groups, and face to face bidding are all essential ingredients.

  1. Short introductory plenary talk to warm people up: present the state of the project, and why it is important and urgent to get collaborations organised.
  2. Divide into small groups to discuss the units of collaboration, mixed across HEIs. Start by discussing the ones suggested here, and decide whether to accept, modify, or reject them. Propose new ones.
  3. Plenary: pool the results so all know the main ideas on units of collaboration.
  4. Small groups divided by HEI. Each HEI decides
    a) what topic+unit types to offer;
    b) what they would particularly like to accept if anyone will offer it;
    c) what bits of their current course they will drop in order to make room for receiving others' units;
    d) what the most important problems in their current courses are that they would like to improve. 30 mins should be enough.
  5. Plenary to share these: write them up on a board, but also have each briefly explained by its author. Get preliminary bids/ expressions of interest on the spot.
  6. Divide up again somehow by possible collaborations (need an idea how to schedule this as each person will often be part of several potential collaborations). Refine each such bid.
  7. Back to plenary: start to worry about balance of trade deficits.

Recipe 2 (Wider scope)

Decide to redesign a whole course. Then focus on how this gives particular scope for collaboration:
  1. New deliverers who may not be expert in various necessary topics: they would particularly welcome material from outside.
  2. Redesign asking what topics would be best for students, as opposed to most interesting or convenient for the teachers: this may throw up things the teachers would welcome help on.
  3. Redesign courses with collaboration in mind as a basic factor.
  4. Consider activity type independently of curriculum units: collaboration may be in units like teachback or email seminars that are content-independent. Invent new alternative types.

Possible project top level conclusions

Possible outcome views:

AG is still only an intermediate instrument to help debugging course exposition and delivery. But this could be powerful in letting collaborating academics a) develop only small bits of course (others are doing the rest); b) try it out on several classes per year: faster development; c) software assistance in gathering this feedback to the author is more likely to be justified in this mass approach, and in cross-institution work.

Cost savings (if any) will be in author time: sharing materials development across HEIs. No savings in delivery (each local course manager has to be able to deliver it and examine it).

Separately, student cross-HEI collaboration may have a big value.

The development and dissemination of our method of setting up teaching collaborations across HEIs.

Summary

References

Draper, S.W. (1997) "Niche-based success in CAL"
Proc. CAL97 Davis,N. & Milne,W. (eds.)
URL http://www.psy.gla.ac.uk/~steve/niche.html
Submitted to: Computers and Education