Why misleading statistics should never become a catchphrase

In our latest blog, Head of Casework Elise Rohan, talks about the problem with the repeated use of misleading statistics and how you can combat this.

I have always been a fan of the show Catchphrase. The joy of being able to recall expressions and idioms without always understanding what they mean or where they come from. They are just phrases I have heard repeated elsewhere.

Our ability to remember things we have heard repeated isn’t limited to words. Have you ever found yourself quoting a statistic and struggling to remember where you first heard it? Such as; two-thirds of lottery winners end up broke or that half of all marriages end in divorce.

As our world becomes more abundant with data, statistics are increasingly used to persuade and provoke discussion – from daytime television to debates in Parliament. In many cases, statistics are seen as a tool to strengthen weak arguments.

At OSR, our vision is that statistics should serve the public good. A big part of that is encouraging their use in wider debate, but it also involves combating and safeguarding against misleading statistics.

What do we mean by misleading statistics?

Misleading statistics are those that misrepresent data either intentionally or not. We have developed a definition of misleadingness in the context of our work as statistics regulator which is:

“We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave a reasonable person believing something which the full statistical evidence would not support”

Repetition of incorrect or unsupported statistics has the potential to harm our vision and public confidence in statistics. The repeated use of misleading statistics creates a validity through reuse. This is known as the ‘illusory truth effect’ or repetition bias. The more you say something, the more confident you become at saying it. Research on this phenomenon has found that we have a cognitive bias to perceive confidence and fluency as characteristics of truthfulness. I’m sure we can all think of public figures who have been accused spreading ‘fake news’ in this way.

We also see this type of misleadingness in the types of casework we receive in OSR. For example, we recently commented on the repeated use of an unsupported claim concerning sex-based differences in online harassment in the Houses of Parliament. And of course one of our most high-profile interventions concerned repeated claims made about the UK’s contribution to the European Union.

 

So, what can you do to combat this?

• Develop the skills to critically challenge what you see.
  • Is a source provided for the figure? If so, is the source reliable?
  • Can you access the underlying information to check and understand the figures for yourself?
  • Is the figure presented in the right context? Is it clear why the time frame has been chosen or why any comparisons have been made?

• The accuracy of claims is often nuanced rather than a binary true or false – as explained in Tim Harford’s guide to statistics in a misleading age. The House of Commons Library has published guidance on How to spot spin and inappropriate use of statistics.
• If you see a statistic that feels questionable, try and fact-check it when you first hear it to reduce the influence of the illusory truth effect. Make use of search engines or organisations such as Full Fact to see if the claim has already been checked and commented on.
• If you’re using statistics to make a claim or support an argument, make sure you help people understand what you’re saying and prevent misinterpretation by following our principles for intelligent transparency. How easily can someone verify what you have said? Is the context for your claim and any limitations clear?
• Finally, if you see misleading statistics being repeated, get in touch with us. Every year, we receive hundreds of queries, many of which are about misleading statistics. In 2022/23, we dealt with 372 cases, in line with our interventions policy.