A couple of months ago this tweet from the Governor of Utah caught my eye:
Welp. We screwed up. Because of a reporting error we have not yet hit 70% on our adult vaccinations. I promised to admit our mistakes and hold us accountable. I hope you will forgive us—and know we have made changes to ensure it won’t happen again. pic.twitter.com/NZK8A8gabt
— Spencer Cox (@SpencerJCox) July 12, 2021
The background was that the Utah Department of Health had identified an error in its published vaccinations figures – fewer people had been vaccinated against coronavirus than had been reported. In a public letter to his fellow Utahns, Governor Cox admitted the mistake.
Here at the Office for Statistics Regulation we love this kind of openness and action. Five things stood out to us from Governor Cox’s letter, which we think all statistics producers in the UK should be thinking about when it comes to data errors.
- Identify your mistake
You can’t fix what you don’t know is broken. This is why rigorous quality assurance is so important. Statisticians need to regularly look under the hood of their analysis to assure themselves and others that it is correct. In Utah, a healthy dose of scepticism about an unexpectedly high vaccination rate made the data team double-, triple- and quadruple-check their figures, until they found the mistake. So, as a statistics producer ask yourself: how do I assure myself that my analysis is correct? Do the numbers look as I expect, and why or why not? What are the risk points for errors? If the root cause of an error isn’t obvious then it can help to ask the five whys until you reach it.
- Be open
One of the things that impressed us most about this example was how direct and open the communication of the error was. There was clear ownership of the mistake and a willingness to correct the figures quickly, publicly and with humility. In the UK, our Code of Practice for Statistics requires government statistics producers to handle corrections transparently. It’s also important that government officials and minsters who use statistics are open about mistakes.
- Fix it
Of course, once you have identified the mistake, it needs to be fixed. As well as being transparent about corrections, the Code of Practice asks that they are made as quickly as is practical.
- Explain
In Utah, Governor Cox explained that while they had reported that 70% of adults had received at least one dose of a coronavirus vaccine, the actual figure was 67%. In a separate statement, the Utah Department of Health went into more detail about the numbers and the mistake. Statistics and data should help people understand the world around them. So, when admitting a data error, it’s important to clearly explain the impact of it – what has changed and what does this mean?
- Improve
The last, but perhaps the most important, step is to learn from the mistake – so that you can avoid it, or something similar, happening again. In Utah, the data team re-examined their processes to prevent this general type of error from being repeated. Statistics producers should reflect on why a mistake was made and what can be done to avoid it in future – and then share what they have learned, and what action they are taking, with their users.
Statistics and data produced by governments serve a vital purpose. They keep us informed, help us make decisions and allow us to hold our governments to account – so we must have confidence in them and the people who produce them. As Governor Cox said, “trust consists of two things: competence and ethical behaviours”. We agree. The Code of Practice places a strong emphasis on trustworthiness. We see that trustworthiness is demonstrated by organisations which are open, and where people who work on statistics are truthful, impartial, and appropriately skilled. We are all human, we mess up and we make mistakes – but we can retain trust by actively looking for our mistakes, being open when we find them and by learning from them.