Mark Pont

How to communicate uncertainty in statistics

Over the past year we have been hosting online seminars for analysts in government, covering a range of topics including how to communicate uncertainty in statistics. Following the publication of our insight report, Approaches to presenting uncertainty in the statistical system, Mark Pont explores the themes discussed in the report and answers some of the questions from the event.

To stay informed of future events and our work in this area, sign up for the OSR monthly newsletter.

Why should you communicate uncertainty in statistics?

Uncertainty in statistics is important to acknowledge, but it’s not always easy to communicate, and not always presented well (or at all!) There is some uncertainty in most statistics and acknowledging it shouldn’t be viewed as negative. Openness helps build trust (as found by recent work by the ONS Economic Statistics Centre of Excellence), and being clear on uncertainty helps users. If there was one overall improvement I’d suggest it would simply be for those statistical outputs that don’t mention uncertainty at all, to do so. This applies to the range of different outputs produced by the analytical community, including internal analysis that is provided internally within organisations, for example as part of policy development.

How to communicate uncertainty effectively

There are different ways of communicating uncertainty and the approach you take depends on the context – of the data, of the statistics, and most importantly an understanding of how the statistics might be used, and by whom.

We have found mixed practice of presenting uncertainty in statistics across government. We found some good examples. But also, a range of cases where uncertainty isn’t mentioned at all, and the statistics therefore appear to be precise facts. This is particularly so in detailed tables – which many users will access directly to get their data, therefore bypassing any information about uncertainty presented in statistical bulletins or separate quality documents. An important part of the statistician’s role is to enable all data to be used appropriately, and in particular not used beyond the weight they can bear. This means that the help for users around data tables needs to be readily available.

Our Approaches to presenting uncertainty in the statistical system publication highlighted some examples of good practice in communicating uncertainty effectively:

What are the issues with releasing detailed breakdowns of data?

One of the questions that we most often consider is the conflict between wanting to provide as much data as possible to users, particularly the kinds of breakdowns provided in detailed tables that may be based on very small samples, while recognising that (particularly for detailed breakdowns) they may not be very reliable. Even with appropriate contextual information about uncertainty, analysts should consider whether such detailed data are sufficiently good to support the uses to which they may be put.

There is always a balance to be struck. Our view is that using words like “estimate” and “around” are a simple way to help show users that the statistics aren’t perfect. As mentioned earlier, this humility from analysts helps breed trust, and is a positive thing. On the other hand, we would not want the valuable messages in the statistics to be overwhelmed by information about their quality.

Factors for success when communicating uncertainty

Good communication of uncertainty relies on various factors. At the forefront of this is that statisticians need to know their users and understand how the stats might be used by others. They also need to understand the impact on decision making of users over-interpreting the precision of any estimates. They also need to understand how well equipped their users are to understand uncertainty and to use it in their particular context. An effective approach to communicating uncertainty can then be determined.

Preventing deliberate misuse is a particular problem, however well uncertainty is communicated. But prominent advice on what the statistics do and do not say makes misuse more open to challenge. The Code of Practice for Statistics requires statistical producers to challenge inappropriate use. OSR may also be able to intervene, so do please engage with us if you need support.

The media play a vital role in onward communication and interpretation of statistics. There’s clearly a role for statisticians and their comms teams to work together to help the media understand and reflect uncertainty appropriately in their reporting. Comms teams can help bring new insights into the user experience. Ensuring that uncertainty (through the use of words like “estimate”) is communicated through media briefings as well as statistical bulletins, is also important.

Again, clear lines on what the statistics do and do not say are helpful and can be directly lifted into onward reporting. However, we’re still very conscious that small apparent differences in outcomes (for example, whether GDP growth is slightly positive or slightly negative) can lead to qualitatively different headlines. We plan to think more about what we can do in this space.

Some uncertainty can be quantified, for example through sampling errors or by modelling errors. But much uncertainty (for example non-sampling biases in surveys or estimates resulting from complex statistical methods) can be difficult or impossible to quantify accurately or at all. In those situations, to help users decide on the usefulness of your estimates, an approximation of their uncertainty would be sufficient.

It’s worth remembering that some users may struggle to understand confidence intervals or need additional guidance to help with their interpretation. Describing uncertainty in a narrative form or visually can help to ensure accessibility to a wide range of users. This also gives a great opportunity to bring together information about confidence intervals alongside factors such as your underlying confidence in the evidence base itself.

What other resources are available?

The Data Quality Hub’s guidance on presenting uncertainty is still the go-to resource and highly recommended for all to read and apply.
In 2020, the Winton Centre wrote about ‘The effects of communicating uncertainty on public trust in facts and numbers’, which explored and compared whether different ways of communication uncertainty made a difference to the public’s trust in the numbers.
In 2021, FullFact, referencing this earlier work, produced a very useful review on presenting uncertainty including a list of key recommendations.
ESCOE published research in 2021 concluding that the way that uncertainty information is communicated around productivity measures matters and that by being clear and directly communicating uncertainty was the best approach in setting the public’s expectations around future data revisions. ESCOE took this work further and looked at testing different visual representations of uncertainty with the public when comparing international estimates of productivity.
OSR’s own Approaches to presenting uncertainty in the statistical system is the first part of an ongoing series of outputs that consider uncertainty in statistics.

To end, it is probably worth reiterating the common thread that holds this blog together. Being able to communicate uncertainty well requires an understanding of the use and potential use of the statistics, and the potential harms of their use without an appropriate understanding of uncertainty. The question of uncertainty in data applies across the Analysis Function to all who publish information, whether in numerical form or not, or provide such information to colleagues within government.

Communicating uncertainty in statistics

Assessment Programme Lead, Mark Pont, discusses the importance of statisticians understanding and communicating uncertainty in their statistics, in light of our recent report exploring approaches to presenting uncertainty in the statistical system.

Uncertainty exists in so many aspects of life and taking it into account is an important part of the decisions we make.

I recently had a hernia repaired at a hospital about an hour from where I live. Ahead of admission the hospital gave me an approximate discharge time. I needed to make plans to get home, which revolved around whether it made sense for my wife to drop me off then spend the afternoon mooching around art galleries, parks and shops. So, I needed to understand how accurate the estimate was, and what factors and assumptions might affect its accuracy. It turned out that it depended on things like where in the order for that day’s surgery I ended up, and how the surgery and my immediate recovery went. All of this was useful intel for our planning.

Later (after the op) I needed a more accurate estimate. My wife was (as planned!) mooching around art galleries, parks and shops, and we needed to try to coordinate her getting back to the hospital close to my discharge so that neither of us was left waiting around too long.

Taking uncertainty into account is also necessary when using official statistics. People who make decisions based on statistics need to factor the uncertainties around the statistics into their decision making. It’s not great to develop policy based on an assumption about the accuracy of the statistics that turns out not to be true. Statistics are rarely, if ever, absolute facts. There will always be some inherent uncertainty – from partial data collection in samples, delays in updating administrative databases, and so on. And different users may want different information about uncertainty depending on the nature of the decisions they’re faced with making and their level of expertise.

Our first Insight project considers the way that uncertainty in official statistics is communicated. We found a mixed bag of practice.

There are many cases where uncertainty is presented in some form in statistical bulletins – in the narrative, charts and infographics. Good examples include using words like “estimate” within the narrative, inclusion of error bounds in charts and clear lists of ways that the statistics can and can’t be used. Projections too often include variants, which gives a neat way of showing that the effect of different assumptions.

There are occasions though where estimates are presented as though they are absolute facts. Not acknowledging that uncertainty could exist within them could lead users to false conclusions. There are also times where better descriptions are needed to help users take uncertainty and its effects into account appropriately. Phrases like “care needs to be taken” and “caution is needed” are widely used, but they could be more specifically helpful in guiding appropriate use of the statistics.

We also found that the communication of uncertainty in detailed data tables (particularly where they are user-specified) is less well-developed, not least because describing uncertainty succinctly in these situations isn’t easy.

There is, however, an abundance of guidance and support available for analysts to help them think through uncertainty and how best to present it to users. We at OSR will continue to help improve and socialise that guidance. We will also develop our regulatory work to understand more about what effective communication of uncertainty looks like, and to encourage the spreading of good practice across government data outputs. We especially expect to develop our thinking on how the change in quality and uncertainty over time can be most helpfully communicated.

And for those who have made it this far, the surgery went well, I’m fully recovered and my wife enjoyed her afternoon out.

Compliant toddlers and minor train delays

Have you ever grumbled to yourself when a train that has been subject to a “minor” delay has had a “major” impact on your day? Perhaps you missed a connection as a result, which meant you missed an important meeting. Or did you consider that “hot” curry on Friday night quite mild?

Language is tricky. And especially adjectives. The role of an adjective is to modify a noun, to describe it more fully or more specifically. A “red” bus tells us more about a bus, and a “compliant” toddler tells us something more about the particular toddler. Although whether you believe a compliant toddler really exists is another question.

Adjectives often offer a value judgement because of the subjectivity of the writer’s point of view. Some adjectives are completely objective – whether a “metal” box is metal or not can be proven. But many are subjective and depend on context. They reflect something of the writer’s personal beliefs, opinions or the way they judge the world – a “fast” car, for example. The reader too will also have their own context against which they interpret each word.

It’s important that messages from statistics are clear and unambiguous. But they also need to be presented in an interesting and helpful way to encourage the reader to engage with them. We don’t want the narrative around statistics to be dry and lifeless. But we need to be careful that in bringing the statistics to life, we don’t choose words that could mislead or imply there’s something in the statistics that isn’t really there. Or vice versa.

We’ve recently responded to some correspondence about the use of “vast” and “overwhelming” when describing statistics about the sex of people who are victims of domestic abuse. Without going too far into the details of the cases (feel free to follow the links), we concluded that these adjectives could give a misleading picture and did not fairly reflect the underlying statistics. We mention it here to make a general point: we won’t sit on the fence if we see a use of language that runs the risk of misdescribing the statistics.

As well as these, there are other words that need careful use. “Most” is another awkward word that can mean different things to different people. One use is synonymous with “more than a half”. But its use to describe proportions only just over 50% can be unhelpful. Many (or perhaps some, or even most!) people would regard it as describing a more noteworthy proportion than that. Likewise, one person’s “small” change could make a world of difference to someone else and a decision they’re faced with. It’s therefore best to avoid these kinds of word where possible or clarify them where they are essential to use.

The call here, then, is for those describing statistics – whether the statistician or a user – to take a step back and review the words they’re using.

do they reflect the statistics fairly and objectively, or could they be seen to be adding something that isn’t really there?
might they minimise something that is an important point?
do they mean the same to all readers?
are there cultural or linguistic differences relating to certain groups that you need to consider to make sure that the data are correctly understood?
and for those of us in Wales, would their translation into Welsh convey the same message?

I think it’s fair to say, though, that in many cases of UK official statistics, we see generally good practice. We will continue to work to further improve clarity and coherence of communication, as outlined in our five-year strategy.

Innovation and Experimental Statistics

Recently we saw the announcement of the development of a fast-charging carbon-ion battery. The ongoing testing of driverless cars is seldom out of the news. And a child car seat with built-in airbags has just gone on sale.

What do these have to do with official statistics?

Well, each is a product undergoing development or recently developed. They’re being developed and tested in response to a changing world and the advent of new technologies. Many of us have been caught short by dead batteries. The idea of being permanently chaufferred around must appeal to many. And making car travel safer for kids is a no-brainer.

The world of official statistics is no different. The world changes, so needs new statistics to bring new insights. Changing technologies allow new analyses and new ways of making statistics and data accessible. And data themselves change. Innovation should be at the forefront of a statistical system that meets its users’ needs. Collaboration and partnership with experts, and engagement with users all contribute to effective innovation.

Like the examples given above, there are different stages of the development of statistics. Some won’t make it past the first hurdle. There’s no shame in hitting an idea on the head if it’s not working. As Thomas Edison, perhaps one of the world’s greatest inventors, said: “I have not failed. I’ve just found 10,000 ways that won’t work”. Others will become well-developed before proving to be unsuccessful (anyone remember the Delorean sports car?) But others will make it, following user testing, and become embedded as normal parts of our lives.

“Experimental statistics” is a label that the statistical Code of Practice suggests could be used to flag statistics under development. These are a subset of official statistics. The experimental label indicates that they’re under development, or being prototyped or tested. Testing involves more engagement with users and stakeholders than usual, and may include for example deep methodological review. These activities can be used to verify the validity of the product and to input into any further development needed.

As with any other product under development, the experimental tag should be brief and temporary. At some point, a decision will be needed about whether to abandon the endeavour, take the development in a different direction, or to mainstream the statistics. Those statistics that do become established and demonstrate the highest levels of trustworthiness, quality and value can be assessed for compliance with the Code of Practice and achieve National Statistics status.

Watch out for more about innovation and experimental statistics in the New Year when we publish the new Code of Practice for Statistics.