Three cases of reasoning with effect sizes in pedagogical research: the good, the bad, and the downright disgraceful

Provisional commentary

Practical uses of effect sizes are relative: which one is bigger when trying to choose between two "treatments" you might use or do research on. Bloom understood this, and exemplifies this, even though his paper was published before the term "effect size" came into use. (In his title, "sigma" means "standard deviation", "2 sigma" would today be called a Cohen's 'd' of 2: a pretty big effect.) Bloom exemplifies practical reasoning about research.

The recent seminar here by Daniel Lakens seemed rather depressing to me, because even though it was about improving the choices journal editors must make, it still reduced them to choosing different magic numbers, when the rational view is that there are no magic numbers, just practical decisions to make. The Sievertsen paper seems to show a researcher who in the end thinks a big number of participants (for him, 100% of the children in the Danish public school system over 4 years, giving over 0.5 million data points) must somehow make his analysis valuable; when in reality it shows that the effect size is so small, that if he picked any other published effect in the literature and acted on it, it would do more good than acting on his findings. The Perkins paper may give us a bit of sympathy for that, as it is an example of the irrational tug many of us feel when an effect is demonstrated in an RCT, but is really too small to be worth taking action on, yet we just want to know how it could be possible that the seats students have in a lecture theatre change their course grades measurably.

Bloom's paper is exemplary because:

• He uses effect sizes to make informed decisions about practical questions.
• Comparisons between concrete cases, not abstract scores anchored by cases far different from the ones in question.
• Use of an existence proof, that is technically achievable but not affordable if rolled out, to set a realistic target on what is possible and so to be aimed for by finding new methods.

Partly related material

• (I have) A web page with some examples of, and points about, effect size
• Rough notes on the lecture theatre seating position literature
• Mayer,R.E. (2009) (2nd edition essential) Multimedia learning (CUP) [the principle of "personalisation"]
  I now think that the LT seating position effect might be explicable by eye contact between lecturer and learner; and that that is taken as a token of a personal relationship between them; and that that is part of what Mayer's principle of "personalisation" might be extended to include. Mayer looked at what aspects of video make the viewer take a monologue as a part of a personal relationship.
• The (≈ 5) best papers I've ever found on education

  What I should take to the session

  Printout of the NewSci table.
  Printout of my esize page
  Slides, or PDF of slides, from CERE course on LTseating; on Bloom's paper.