Measuring and evaluating what you value: Why go horse riding?
10th January 2017
News of funding for an EEF evaluation of cultural learning prompted debate today about the intrinsic versus instrumental value of such activities.
I'm sure the trips will have a positive impact, on literacy in particular. But some things are simply worth doing for their own sake. https://t.co/VEuexRyhjm
— Russell Hobby (@russellhobby) January 10, 2017
As Simon Burgess pointed, out, the trial will in fact look at other benefits too, but the problem of attempting to shoe-horn activities’ benefits into a narrow academic framework remains a common one.
Indeed at LKMco we have several times had to turn down (or reshape) potential projects because an organisation wanted to make the case for something by measuring impact on something that was not what they actually cared about. It’s important to avoid this trap for a number of reasons.
We work on lots of small scale evaluation projects and the first step usually involves clarifying the organistion’s “theory of change“. In other words, what are they trying to achieve and how do they think their activities do that. Unfortunately, but quite understandably, they often want to test whether their program improves something they think the ‘market’ (usually schools or policy makers) values (usually attainment – specifically maths and or literacy) – regardless of whether that’s actually what they get out of bed for in the morning.
Passionate horse riders keen to share their love of the activity might therefore recognise that schools are more likely to fund horse riding lessons if is ‘proven to improve maths’.
The problem is:
A) This simply reinforces some schools’ / the education system’s tendency to value and pursue a narrow set of outcomes.
B) It’s unrealistic. If you want to improve average attainment in maths there are more plausible ways of doing so. A study of horse riding is fairly unlikely to reveal a striking impact.
C) As Laura McInerney pointed out, when such a study concludes that there was a negligible/non-significant impact (particularly compared to interventions that are actually focused on this outcome), it leaves the door open to those who want to discredit the approach.
What if we find out that theatre trips ruin pupil attainment? WHAT THEN, EH? https://t.co/JoXrAYWBXo
— Laura McInerney (@miss_mcinerney) January 10, 2017
D) It detracts from the more genuine case that should be built around the real reasons for valuing and giving pupils access to such activities.
Advocates of particular interventions would do better to make an authentic and evidence based case for the things they actually value rather than pursuing a market driven attempt to prove instrumental value, based on (at best) tangentially related outcomes.