Metascience: the one idea that could revolutionise educational research?

by

18th December 2020

The most controversial research paper of all time?

There have been some pretty provocative article titles in the history of science. But few come close to a paper I chanced upon during my undergraduate studies in philosophy: John Ioannidis’ 2005 essay “Why Most Published Research Findings are False”.

The content of the article is even more provocative than the title. Over the course of six pages, Ioannidis argues that three everyday features of scientific practice undermine its ability to produce accurate and useful findings:

 

  1. Questionable Research Practices – Like any one in any job, scientists cut corners to hit their targets. And the one metric that matters for researchers is ‘number of articles published in high impact journals’. In order to produce findings that get published in these journals, scientists engage in all kinds of methodological and statistical gymnastics to contort their results into a form that makes them publishable.

 

  1. Publication Bias – It takes two to tango and scientific journals are as much part of the problem as individual scientists. The publication policies of scientific journals tend to favour statistically significant findings with large effect sizes. That means that they are full of claims that particular social programmes or medical treatments are effective, but wholly devoid of any countervailing evidence to show why they might be ineffective. The net result is a scientific literature that is, at best, incomplete, and at worst, utterly biased.

 

  1. Replication Bias – Journals also prefer to publish exciting, novel findings rather than replications of studies that have already been published. This disincentivises scientists from replicating previous studies. With the vast majority of studies in several disciplines (from molecular genetics to psychology) never having been replicated, entire academic literatures could be composed of lucky one-off findings and the spawn of scientists’ questionable research practices.

Summed together, these three features of scientific research undermine the very enterprise of science itself. The whole cathedral is built on an edifice of misrepresentations and mythology.

In order to produce findings that get published in these journals, scientists engage in all kinds of methodological and statistical gymnastics to contort their results into a form that makes them publishable. Click To Tweet

But at the same time, Ioannidis’ work is more than mere iconoclasm. It shows how governments, scientific journals and researchers can work together to produce more reliable insights. As we discuss below, this has enormous relevance to educational research.

What is metascience?

Naturally the reaction to Ioannidis’ paper was one of fury and suspicion. However, in subsequent years it has entered the canon as one of the great pieces of research in the 21st century.

Ioannidis employed a canny combination of probability theory and statistical modelling to evidence his charges against science. In effect, he used the tools of science to study and critique science itself.

Metascience helped re-engineer systems of reporting, publication and evaluation within some scientific disciplines to ensure that quality scientific research is produced Click To Tweet

This reflexive practice of science is known as ‘metascience’ and Ioannidis’ 2005 paper is considered its founding document. Over the course of the last 15 years metascience has fortified itself into a scientific discipline in its own right, going on to expose even more problematic research and reporting practices. It has also helped re-engineer systems of reporting, publication and evaluation within some scientific disciplines to ensure that quality scientific research is produced.

But has metascience reached educational research?

What can metascience tell us about educational research?

There’s no formal metascience programme within educational research. But there have been a few recent studies that fall within its domain. This raft of new papers highlight systemic problems with educational research that are almost identical to the ones picked out by Ioannidis:

  1. Questionable research practices – A recent survey study of nearly 1,500 educational researchers concluded that the majority of academics in the field have engaged in research and reporting processes that misrepresent the validity of their results.

 

  1. Publication bias – As I discussed in a Schools Week article last year, a sequence of studies over the course of the 2010s have shown that the major educational research journals are skewed towards publishing statistically significant results with large effect sizes while rejecting those that do not meet these criteria.

 

  1. Replication bias – A 2014 systematic review found that only 0.13% of articles published in the top 100 educational research journals are replications. To put that into context, the field of psychology was plunged into a “Replication Crisis” when its reproducibility index was found to be 1.1%. What’s more, the 2014 study found that of the few replication studies that were conducted within education, nearly half are unsuccessful.
A recent survey study of nearly 1,500 educational researchers concluded that the majority of academics in the field have engaged in research and reporting processes that misrepresent the validity of their results Click To Tweet

These are sobering and slightly disturbing findings. But while the diagnosis may be damning, I think there are good reasons to believe that the prognosis is promising.

How can metascience help us improve educational research?

It’s probably too early to ape Ioannidis and declare most published educational research findings as false. Nonetheless, we should take the above findings seriously. Doing so presents several exciting opportunities to repair and reform the cathedral of scientific research.

  1. Stopping Questionable Research Practices

An international group of educational researchers should form an initial working party on educational research and metascience, with the long-term aim of establishing a formal body. This body, modelled on metascience institutions in medicine, should run its own journal publishing research into the depth and extent of problematic scientific practices in educational research.

The body can also lead on providing training materials for researchers on optimal research practices, as well as establishing funding channels for further metascientific research in the field.

Academic journals in education can also disincentivise questionable research practices by having researchers pre-register their methods for studies. This limits researchers’ scope to contort their results to produce ‘publishable’ findings, while still offering scientists the opportunity to get their work published.

Academic journals in education can also disincentivise questionable research practices by having researchers pre-register their methods for studies Click To Tweet
  1. Stopping Publication Bias

There is good news in educational research on this front. For example, the EEF has committed to publishing negative results from its studies as well as positive ones. But more action is needed.

Journals need to follow suit and commit to the publication of studies that show negative results or results with small effect sizes. Some fields have established separate journals that have this as their exclusive remit. Equally, some journals now practice ‘results-blind’ peer-review, where reviewers only see the method of a study and not the outcome of the research. This has helped drive higher rates of publication of negative results.

Educational research journals should practice ‘results-blind’ peer-review, where reviewers only see the method of a study and not the outcome of the research. This can help drive higher rates of publication of negative results. Click To Tweet
  1. Stopping Replication Bias

Journals have to ditch their tacit anti-replication policy. But, to go even further, they need to place a premium on replication studies, publishing them independent of their outcome.

A special issue of one of the major educational research journals devoted to replications of some classic studies in education could be a powerful way of placing the issue of replication high up on educational researchers’ agenda. This model has already proved successful in social and personality psychology.

Concluding Thoughts

Metascience was born into controversy. But properly viewed, it is a research programme that has generated far more light than heat. Indeed, it has already been used to improve disciplines such as medicine, physics and psychology. Educational research can reap these same benefits. We just need to see beyond the provocations and embrace the potential of metascience.