In our latest blog Director General for Regulation, Ed Humpherson, reflects on why communicating uncertainty is a constant challenge for statisticians…
A recurring motif of OSR reports is the need to communicate uncertainty better. This was the theme of an article I recently published in the Journal of Risk Communication, which explores how uncertainty is presented in official statistics.
What is the issue?
Start with GDP. When we reviewed ONS’s process for revising GDP, uncertainty and its communication was one of our main requirements for improvement. We asked ONS to improve the presentation of early estimates of GDP and all supporting information it provides on the uncertainty surrounding these estimates. ONS should ensure that the uncertainty surrounding GDP statistics is in itself a key message.
I also chaired a discussion of uncertainty with Chris Giles of the FT, Johnny Runge of Kings College London, Marianthi Dunn (OSR’s lead author on GDP revisions), and Sumit Dey-Chowdrey of ONS.
It wasn’t just GDP where this was an issue. We highlighted communicating uncertainty for labour market statistics (see the section and requirement on “volatility” on page 4) and research and development statistics (see requirement 2).
So it comes up a lot in our work.
Why is it so difficult?
Well, first of all, as I argue in the journal article, it’s just hard. I think there may be an innate tendency for people to see a number and assume it has a certainty, an absolute rightness. And this in turn may be deeply embedded because of our experience of learning maths as children: we see a subject in which answers are definitively right or wrong.
And I also think this is exacerbated by an assumption that statistics are counts of fixed things – the economy, population, crime. It’s only when you spend time understanding the process of compiling official statistics that you realise that the numbers are often more complicated than a simple count. They are driven by assumptions, definitions and methodological choices.
This psychological anchoring on ‘hard’ numbers is exacerbated by the way statistics are used in public debate. Numbers are slotted into political arguments – perhaps weaponised, even – without nuance or context. The percentage increase in GDP. The unemployment rate. Crime going up.
These factors reinforce each other. They mean that people communicating official statistics have a big challenge. They must convey what they know about their statistics – that is, that they are best estimates, not certain numbers – to audiences that often are expecting single, fixed results.
In the face of these phenomena it can be tempting for statistics producers to fall back on generic wording. As we found when we did a review of communicating uncertainty, producers can often say generic things like “users should exercise caution”.
I do not underestimate the challenge facing producers. But is there a way to think about uncertainty that might help?
I think that it’s not always recognised that there are two distinct categories of uncertainty. My article puts forward two concepts:
- Specific uncertainty: This type of uncertainty is bound up with the process of collecting and estimating that is at the heart of statistics. It involves the recognition that the estimate is only an approximation, and a range of other estimates could also be plausible. This is perhaps the more familiar notion of uncertainty that arises in surveys – that the survey result represents a central estimate around which there is a confidence interval.
- Contextual uncertainty: This is less about the process of production, and more about the inferences that users may draw, for policy and other decision purposes, from statistics. Numbers may not mean what users think they mean; and this may lead them to place unwarranted weight on them for decision making purposes.
As an analogy, consider maps. To quote my article: “There is a difference between the map-maker’s perspective, and the map-reader’s perspective. The map maker may successfully communicate the conventions that go into the production of the map – what different icons mean, how contour lines are measured, and so on. That is useful. But it is quite distinct from the map reader’s desire to use the map to determine how to undertake a journey, which depends on their inferences about the best route, the paths to avoid, the steeper hills to navigate around.”
My simple proposition is that producers may struggle because they don’t separate these two concepts. And they often focus on the first – the technical communication of uncertainty – without paying sufficient attention to the second – how people may use the numbers.
Of course, it might be said that the standard boilerplate phrasing “users should treat the number with caution” is a nod towards contextual uncertainty. But it’s not a helpful one, and wherever it arises, producers should provide some more useful steers to the users.
So, to save you the trouble of reading the whole article, here’s a summary of what it says about communicating statistics:
- It’s hard
- It’s not always done well
- It’s helpful to distinguish these two types: specific and contextual
And one thing is clear – generic wording like “users should employ caution” should be avoided. If my article and OSR’s overall body of recommendations achieves anything, I hope it consigns this generic wording to history.