Last changed 23 May 2021 ............... Length about 1,000 words (10,000 bytes).
(Document started on 25 Jan 2018.) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/educ/meth.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [educ] [Home] [this page]
Other related pages: [Effect size] [Invite to seminar] [Educ. refs]
Linked circle of topics relevant to CCSE: [Adey&Shayer] [Papert's gears] [Early play] [spatial] [curricula] [methodology]

Methodological issues

By Steve Draper,   Department of Psychology,   University of Glasgow.

This page overlaps with another /~steve/best/rct.html and I'm not sure whether to merge them.


  1. Ideas about different learning trajectories

    2. The different relationships between the intervention and the measured effect.

    Cognitive acceleration

    Delayed post-tests

    ZOPD

    MBarr: they DO catch up naturally

    They don't catch up naturally

    Assymmetric learning effects: why counter-balancing doesn't work on humans

    Expectation effects

  2. Learning to learn

    3. Jigsaw; Ann Brown.
A&S;  Papert;  diSessa; = CogAcc.
How in fact in most disciplines, learning how to learn more is a central outcome.
E.g. compSci students leave knowing how to learn another programming language
(without teachers).

Revision strategies.

    title

    Not learning the subject matter, but learning to learn. A&S is a pure example of "cognitive acceleration": of training pupils on materials that for 2 years had no demonstrable effect, but after the training, then they learned STEM material in the year following much faster than control group kids did. The intellectual equivalent of spinach for Popeye; morning coffee on you. [See (b, c) of the five main papers, and Adey (2004).]

  3. Multi-stage research

    4. Much research, especially applied research, goes through multiple stages; and criticisms that are important at one stage may be largely irrelevant at another. The design of an experiment needs to be appropriate to the stage of the overall research programme, not dictated by a single ideal standard for one-off experiments that are imagined to stand by themselves.

    MRC stage model for health interventions

    The MRC offers this model on p.4 of: MRC (2000) A Framework for development and evaluation of RCTs for complex interventions to improve health

    1. Theory - Pre-clinical.
      Explore relevant theory to ensure best choice of interention and hypothesis and to predict major confounders and strategic design issues.
    2. Modelling - Phase 1.
      Identify the components of the intervention, and the underlying mechanisms by which they will influence outcomes to provide evidence that you can predict how they relate to and interact with each other.
    3. Exploratory trial - Phase 2.
      Describe the constant and variable components of a replicable intervention AND a feasible protocol for comparing the intervention to an appropriate alternative.
    4. Definitive RCT - Phase 3.
      Compare a fully-defined intervention to an appropriate alternative using a protocol that is theoretically defensible, reproducible and adequately controlled, in a study with appropriate statistical power.
    5. Long-term implementation - Phase 4.
      Determine whether others can reliably replicate your intervention and results in uncontrolled settings over the long term.

    The MRC model is for medical research, but education is likelwise an applied field partly to do with both theory and controlled experiments: and so it is interesting to look for comparisons and transfer of methods.

    The MRC "phase 3" stage uses RCTs (Randomised Controlled Trials). In another web page I comment on RCTs and on issues and variations in study design related to them.

    Shayer model for educational research

    1. Primary effect study: what effects, and how large, can be achieved using the intervention?
    2. Replication study: Can the effect be transferred from the researchers to any other teacher(s)? (Or is it only achievable by the originator and therefore may depend or be wholly due to some unconsicous skill on their part?). The test at this point is probably still using a highly self-selected and unusually enthusiastic and able teacher; but at least it has been transferable.
    3. Generalizability study: Can a teacher training course be created for transferring the intervention, and is it successful in achieving the effects for pupil attainment even with perhaps reluctant teachers? This is obviously essential for the research to have any significant effect nationally.

    Shayer gives this model for applied educational research on pp.112-3 in Shayer(1992). Shayer,M. (1992) "Problems and issues in intervention studies" in Demetriou,A., Shayer,M. & Efklides,A. (eds.) Neo-Piagetian theories of cognitive development: implications and applications for education ch. 6 pp.107-121 (London : Routledge) GoogleBook

    Less generally, one might see Adey & Shayer's own CASE work as having an extended number of stages.

    1. Theoretical work.
    2. Development of materials derived from the theory, to use in the next stage.
    3. Primary effect study.
    4. Replication study: transfer to any other teacher(s)?
    5. Generalizability: create, develop, and test a teacher training course.
    6. Roll-out in multiple schools (test it generalises over institutions as well as teachers)
    7. Follow-up studies on long term effects.

    The Louis Pasteur "model" of types of experiment

    Rather different is a model, or rather reported practice, of types or phases or uses for, experiments offered by one of Louis Pasteur's associates. In a biography, he said that Pasteur's central activity in life was experimentation, and they could be classified into three types.
    1. On entering a new field (which Pasteur did quite a number of times) he would do numerous experiments recorded only in his lab notebooks, by the end of which he had arrived at his new theories in this field; and remarkably, seldom had to change them later. Here he was experimenting to learn, not to communicate to others.
    2. Doing roughly what most researchers would do today: to develop experiments that would convince other scientists as long as they weren't prejudiced by their own theories. These were often published.
    3. Doing experiments of such monumental precision, that anyone who tried to object to their conclusions would simply look hopelessly irrational. Experiments to utterly crush any opposition. Pasteur did some of these: often as public demonstrations.

  4. Individual differences in effects

    5. I.e. does a given intervention have the same effect on everyone, or does it work on some but not on others? See /~steve/best/effect.html#idiff for an elaboration of this small, but rather general and important, point. The point is: in most educational interventions, you should not only study the group means, but also the individuals e.g. report what percent of subjects do and don't show the effect.

    Shayer & Adey (1992,3) showed such differences in effect of their (CASE) interventions. Despite a large-ish overall group effect size: a) Some individual kids show a really big effect, some none at all. b) About twice as many boys as girls show the effect.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [educ] [this page]
[Top of this page]