When I read a review of Jerry Muller’s book, The Tyranny of Metrics, I immediately ordered a copy from my library. I went to collect it the other day.
Risking the charge of confirmation bias, I nodded approvingly to myself when I read on page 3, ‘And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction.’ As Muller points out ‘The things that get measured may draw effort away from the things that we really care about.’
Jonathan Harris, in his wonderful artpiece, Data Can Help Us, (image for this blog) also points this out in a series of provocative sentences, for example: Data ‘will help educators make excellent standardised tests, but will it help us embrace different standards of excellence?’ (Here I had a brief pause to check if data is/are singular or plural) Harris warns us of the dangers of abandoning timeless decision-making tools like wisdom, morality, and personal experience in favour of data converted into numbers.
Similarly, Muller notes that ‘The most characteristic feature of metric fixation is the aspiration to replace judgement based on experience with standardized measurement’.
It’s not that numbers are ‘wrong’, it’s that converting data into meaningful numbers that are useful and could produce good outcomes is not easily done, as a good example in an HBR article Know the Difference Between Your Data and Your Metrics illustrates. The authors say ‘As we learned, there is a difference between numbers and numbers that matter.’
I was drawn to Muller’s book because I work in a field, organisation design and development, where it is almost impossible to identify what numbers might matter – assuming that it is sensible to even try to reduce organisation development or organisation design to a metric, or set of metrics. If I am asked to suggest meaningful metrics that might show that what OD & D practitioners do produces value (and for whom) I find it hard to offer anything with confidence.
One reason for this is that the metric required is often a monetary return-on-investment one (dollars or pounds). However, a monetary unit is a different unit of measurement from what are sought as outcomes of the piece of work which could be, for example, increased collaboration, aligned leadership, a ‘one-team’ culture, etc). We expect that if the input is in money – cost of time to design and deliver the work, that the outcome will also be in money.
In any instance where the input unit is not the same as the output unit we are not comparing like units. Sticking with money as an input unit, we could try and convert the outcome to units of money, but these would be proxy measures. They distort how we see the outcomes. There are several factors contributing to the distortion, among them:
- we don’t know that there is a direct cause/effect relationship between the OD & D work what we are looking for in results
- the conversion to like units is not easy to do – what are the proxy metrics of ‘aligned leadership’, for example, that would give us a monetary value?
- interventions don’t create outcomes – they may create a different context that enable something to change – but it may not be what is expected.
Toby Lowe, a researcher on managing performance at the UK’s Newcastle University is clear that, ‘The simplification required to measure and attribute ‘outcomes’ turns the organisation and delivery of social interventions into a game, the rules of which promote gamesmanship, distorting the behaviour of organisations, managers and practitioners who undertake it.’
I collected Muller’s book a day or two after coming back from the ODNE conference where I spoke on the sacred cows of organisation development: a sacred cow being an organisational or societal norm that is accepted without question. (Until it is questioned – for example votes for women came about after challenging the sacred cow that only men could vote).
One of the ‘sacred cows’ I offered for discussion was ‘Organisation development activity can deliver tangible business results’, asking participants to consider what would happen if we did not give up this sacred cow, and what would happen if we did give it up. There was some debate on what ‘tangible’ results are, and a lot of debate on the issues around ‘proving’ the ROI of the OD & D work.
Too late to share in the conference, I find that Gerry Muller offers a thoughtful 10-point checklist on how to approach the use of performance metrics. He notes, before presenting the list, that ‘measurement demands judgement: judgement about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available.’ (Judgement is not measurable.) Here are the checklist items with a summary quote from the fuller explanation that Muller gives:
- What kind of information are you thinking of measuring – ‘measurements attached to human activity are not reliable unless the humans agree with the goals of the measurement’.
- How useful is the information? ‘the ease of measuring may be inversely proportional to the significance of what is measured’.
- How useful are more metrics? ‘the fact that metrics is helpful doesn’t mean that more metrics is more helpful’.
- What are the costs of not relying on upon standardized measurement? ‘Are there sources of information about performance, based on the judgement and experience of clients?’
- To what purposes will the measurement be put? ‘Here a key distinction is between data to be used for purposes of internal monitoring by the practitioners themselves versus data to be used by external parties for reward and punishment’.
- What are the costs of acquiring the metrics? ‘Information is never free, and often it is expensive in ways that rarely occur to those who demand more of it’.
- Ask why people at the top of the organisation are demanding performance metrics. ‘Sometimes [this] flows from the ignorance of executives about the institutions they’ve been hired to manage.’
- How and by whom are the measures of performance developed? ‘Measurements are more likely to be meaningful when they are developed from the bottom up … from direct experience’.
- Remember that even the best measures are subject to corruption or goal diversion. ‘There are inevitable drawbacks to all schemes of measured reward’.
- Recognising the limits of the possible is the beginning of wisdom. ‘Not all problems are solvable, and even fewer are solvable by metrics.’
Do you think that the outcomes of OD & D work can be identified and then converted into useful proxy measures to show ROI? Let me know.
Naomi Stanford 