Over the past year we have been hosting online seminars for analysts in government, covering a range of topics including how to communicate uncertainty in statistics. Following the publication of our insight report, Approaches to presenting uncertainty in the statistical system, Mark Pont explores the themes discussed in the report and answers some of the questions from the event.
To stay informed of future events and our work in this area, sign up for the OSR monthly newsletter.
Why should you communicate uncertainty in statistics?
Uncertainty in statistics is important to acknowledge, but it’s not always easy to communicate, and not always presented well (or at all!) There is some uncertainty in most statistics and acknowledging it shouldn’t be viewed as negative. Openness helps build trust (as found by recent work by the ONS Economic Statistics Centre of Excellence), and being clear on uncertainty helps users. If there was one overall improvement I’d suggest it would simply be for those statistical outputs that don’t mention uncertainty at all, to do so. This applies to the range of different outputs produced by the analytical community, including internal analysis that is provided internally within organisations, for example as part of policy development.
How to communicate uncertainty effectively
There are different ways of communicating uncertainty and the approach you take depends on the context – of the data, of the statistics, and most importantly an understanding of how the statistics might be used, and by whom.
We have found mixed practice of presenting uncertainty in statistics across government. We found some good examples. But also, a range of cases where uncertainty isn’t mentioned at all, and the statistics therefore appear to be precise facts. This is particularly so in detailed tables – which many users will access directly to get their data, therefore bypassing any information about uncertainty presented in statistical bulletins or separate quality documents. An important part of the statistician’s role is to enable all data to be used appropriately, and in particular not used beyond the weight they can bear. This means that the help for users around data tables needs to be readily available.
Our Approaches to presenting uncertainty in the statistical system publication highlighted some examples of good practice in communicating uncertainty effectively:
- Department for Environment, Food and Rural Affairs (DEFRA) England Biodiversity Indicators bulletin
- Office for National Statistics: Covid ad-hoc analysis
- Welsh Government: Welsh Index of Multiple Deprivation
- Office for National Statistics: Population Projections
What are the issues with releasing detailed breakdowns of data?
One of the questions that we most often consider is the conflict between wanting to provide as much data as possible to users, particularly the kinds of breakdowns provided in detailed tables that may be based on very small samples, while recognising that (particularly for detailed breakdowns) they may not be very reliable. Even with appropriate contextual information about uncertainty, analysts should consider whether such detailed data are sufficiently good to support the uses to which they may be put.
There is always a balance to be struck. Our view is that using words like “estimate” and “around” are a simple way to help show users that the statistics aren’t perfect. As mentioned earlier, this humility from analysts helps breed trust, and is a positive thing. On the other hand, we would not want the valuable messages in the statistics to be overwhelmed by information about their quality.
Factors for success when communicating uncertainty
Good communication of uncertainty relies on various factors. At the forefront of this is that statisticians need to know their users and understand how the stats might be used by others. They also need to understand the impact on decision making of users over-interpreting the precision of any estimates. They also need to understand how well equipped their users are to understand uncertainty and to use it in their particular context. An effective approach to communicating uncertainty can then be determined.
Preventing deliberate misuse is a particular problem, however well uncertainty is communicated. But prominent advice on what the statistics do and do not say makes misuse more open to challenge. The Code of Practice for Statistics requires statistical producers to challenge inappropriate use. OSR may also be able to intervene, so do please engage with us if you need support.
The media play a vital role in onward communication and interpretation of statistics. There’s clearly a role for statisticians and their comms teams to work together to help the media understand and reflect uncertainty appropriately in their reporting. Comms teams can help bring new insights into the user experience. Ensuring that uncertainty (through the use of words like “estimate”) is communicated through media briefings as well as statistical bulletins, is also important.
Again, clear lines on what the statistics do and do not say are helpful and can be directly lifted into onward reporting. However, we’re still very conscious that small apparent differences in outcomes (for example, whether GDP growth is slightly positive or slightly negative) can lead to qualitatively different headlines. We plan to think more about what we can do in this space.
Some uncertainty can be quantified, for example through sampling errors or by modelling errors. But much uncertainty (for example non-sampling biases in surveys or estimates resulting from complex statistical methods) can be difficult or impossible to quantify accurately or at all. In those situations, to help users decide on the usefulness of your estimates, an approximation of their uncertainty would be sufficient.
It’s worth remembering that some users may struggle to understand confidence intervals or need additional guidance to help with their interpretation. Describing uncertainty in a narrative form or visually can help to ensure accessibility to a wide range of users. This also gives a great opportunity to bring together information about confidence intervals alongside factors such as your underlying confidence in the evidence base itself.
What other resources are available?
- The Data Quality Hub’s guidance on presenting uncertainty is still the go-to resource and highly recommended for all to read and apply.
- In 2020, the Winton Centre wrote about ‘The effects of communicating uncertainty on public trust in facts and numbers’, which explored and compared whether different ways of communication uncertainty made a difference to the public’s trust in the numbers.
- In 2021, FullFact, referencing this earlier work, produced a very useful review on presenting uncertainty including a list of key recommendations.
- ESCOE published research in 2021 concluding that the way that uncertainty information is communicated around productivity measures matters and that by being clear and directly communicating uncertainty was the best approach in setting the public’s expectations around future data revisions. ESCOE took this work further and looked at testing different visual representations of uncertainty with the public when comparing international estimates of productivity.
- OSR’s own Approaches to presenting uncertainty in the statistical system is the first part of an ongoing series of outputs that consider uncertainty in statistics.
To end, it is probably worth reiterating the common thread that holds this blog together. Being able to communicate uncertainty well requires an understanding of the use and potential use of the statistics, and the potential harms of their use without an appropriate understanding of uncertainty. The question of uncertainty in data applies across the Analysis Function to all who publish information, whether in numerical form or not, or provide such information to colleagues within government.
Related links
Communicating uncertainty in statistics
Approaches to presenting uncertainty in the statistical system