Office for Statistics Regulation

Securing public confidence in algorithms – lessons from the 2020 exam awards

Last March, schools and colleges across the UK were closed and the qualification regulators (Ofqual, Qualification Wales, SQA and CCEA) were directed by the respective governments of England, Wales, Scotland and Northern Ireland to oversee the development of an approach to awarding grades in the absence of exams. While each regulator developed a different approach, all approaches involved statistical models or algorithms.

When grades were awarded to students last summer there was intense media coverage across the UK. We read headlines such as, “Futures sacrificed for the sake of statistics” and statements implying that a “mutant algorithm” was to blame. The decisions that were made based on the calculated grades had a significant impact on many children’s lives. Up until last summer, most people had probably never felt personally affected by a government algorithm before.

Statistical models and algorithms though are increasingly becoming a part of normal life. As technology and the availability of data increases, developing these types of models in the public sector can play a significant role in improving services for society.

We are concerned that public confidence in the use of statistical models has been damaged by the exam processes last year. As Ed Humpherson, our Director General, said recently, “Public debate about algorithms veers between blind optimism in their potential and complete rejection. Both extremes have been very evident, at different times, in the discussion of the use of models to award exam grades.”

Our review

In the post 2020 exams world, the risk of a public backlash when public sector decisions are supported with an algorithm feels much more real now – regardless of how ‘right’ the algorithm may be.

As the UK regulator of official statistics, it is our role to uphold public confidence in statistics. We believe that statistical models have an important role to play in supporting decision making and that they can also command public confidence. It is because of this that we reviewed the approaches taken last year to award grades, to get a better understanding of what happened and, most importantly, what others can learn from the experience. To us it was striking that, though the approaches and models in England, Wales, Scotland and Northern Ireland had similarities and differences, all four failed to command public confidence.

Our recently published report explores what we found. We acknowledge the unique situation that the regulators were working in which was far removed from their normal roles. We conclude that many of the decisions made supported public confidence. To support learning for others, we have also clearly highlighted in the report the areas where we feel different choices could have been made.

We found that achieving public confidence is about much more than just the technical design of the model. It is also not just about doing one or two key things really well. It stems from considering public confidence as part of an end-to-end process, from deciding to use a statistical model through to deploying it. It is influenced by a number of factors including the confidence placed in models, the extent of public understanding of the models, the limitations of the models and the process they were replacing, and the approach to quality assuring the results at an individual level.

Lessons for those developing statistical models

We’ve identified 40 lessons for model developers which support public confidence. These fall under the following high-level principles:

Be open and trustworthy – ensure transparency about the aims of the model and the model itself (including limitations), be open to and act on feedback and ensure the use of the model is ethical and legal.
Be rigorous and ensure quality throughout – establish clear governance and accountability, involve the full range of subject matter and technical experts when developing the model and ensure the data and outputs of the model are fully quality assured.
Meet the need and provide public value – engage with commissioners of the model throughout, fully consider whether a model is the right approach, test acceptability of the model with all affected groups and be clear on the timing and grounds for appeal against decisions supported by the model.

Lessons for policy makers who commission statistical models

Model developers should not work in isolation. They should collaborate with others throughout the end-to-end development process. It is important that policy makers who commission models are part of this wider collaboration.

A statistical model might not always be the best approach to meet your need. Model commissioners need to be clear what the model is aiming to achieve and whether it meets the intended need, understanding the model’s strengths and limitations and being open to alternative approaches.

Statistical models used to support decisions are more than just automated processes. All models are built on a set of assumptions. Commissioners of models should ensure that they understand these and provide advice on the acceptability of the assumptions and other key decisions made in the model development.

The development of a statistical model should be regarded as more than just a technical exercise. Commissioners need to work with model developers throughout the end to end process and have regular reviews to check the model will meet the policy objective.

Lessons and recommendations for the centre of government

We also looked at the big picture, at the wider infrastructure that is in place to support public bodies working with statistical models. Looking at this, we found that public bodies developing models need more guidance and support and that this should be easier to access.

There are lots of different organisations in the statistical modelling, algorithm and AI space. As a result, it is not always clear what guidance is relevant to whom and where public bodies can go to for support. Some of this is down to inconsistencies in the terminology used to describe models, and some is due to it simply being hard to find out who is doing what in this very fast-moving area.

At the highest level, we feel that clearer leadership is needed from government. We are calling on the Analysis Function and Digital Function, working with the administrations in Scotland, Wales and Northern Ireland, to ensure that they provide consistent and joined up leadership on the use of models.

To support this, we recommend that those working in this area collaborate and develop a central directory of guidance. We see the Centre for Data Ethics and Innovation having a key role in this space. In addition, we are recommending that more guidance is developed, particularly for those wanting to test the public acceptability of their models.

And lastly, we recommend that any public body developing advanced statistical models with a high public value should consult the National Statistician for advice and guidance.

Our role going forward

This review supports the work of our Automation and Technology work program. Through this programme we will be clarifying our regulatory role when statistical models and algorithms are used by public bodies. We are working on guidance about models and AI in the context of the Code of Practice for Statistics and advocating for the use of automation within statistical production. We will continue to work with other organisations to support the implementation of the review findings.

We are happy to discuss the findings of our review further. Please get in touch with us.

Listen to your enthusiasts: Implementing RAP at Public Health Scotland

This is a guest blog from Scott Heald following the launch of our new report: Reproducible Analytical Pipelines (RAP): Overcoming barriers to adoption.

Firstly, let me introduce myself. I’m Scott, the Head of Profession for Statistics at Public Health Scotland (PHS). PHS is a new body, formed in April 2020 (to lead on tackling the challenges of Scotland’s public health but dominated by the COVID-19 pandemic in our first year). Our RAP journey started when we were in NHS Scotland’s Information Services Division (ISD), the health and care statistics body which now forms part of PHS.

PHS, and ISD before it, has been a big fan of RAP from the beginning. I wanted to share our story, from a bunch of enthusiastic statisticians who convinced me it was the right thing to do (I didn’t need much convincing!), to embedding it within our organisation in our reporting on COVID-19.

Our RAP journey began with a programme of work to transform how we published our statistics. It quickly became clear that the programme had to be as much about our processes for producing statistics, not just the final published output. More automation was key – to speed up processes, eliminate manual errors, and to release capacity to add value to our statistics. Greater job satisfaction for our statisticians was a welcome impact too.

“We don’t use that here”

Up till this point our software of choice was propriety standards. More and more graduates were joining our organisation and wondering why we weren’t using open-source software like R, having been taught it at university. I guess, in those early days, I was probably part of the “we don’t use that here” (partly out of fear as I had not personally used any of the new software they were talking about).

However, I was persuaded (and willing) for a group of our statisticians to show us what could be done using R. Long story, cut short, is that our group of enthusiasts showed the power of what could be done and PHS is now making the strategic shift to follow RAP principles as a way of working.

The art of persuasion

I’d describe our RAP journey as “bottom up”, with support from the “top down”. When we started seeing the results, we didn’t need much convincing. OSR’s report showcases our work on Hospital Standardised Mortality Ratios – a quarterly publication which used to take five days to run (lots of detached processes, lots of room for error). I remember vividly the first time the team ran it the RAP way. Five minutes after the process started, the finished report was in my inbox. We couldn’t believe it! And, to be sure, spent the next five days running it the old way to make sure we got the same answers (we more or less did; the RAP way was more accurate, highlighting a few errors we fixed along the way!).

Our learning is that it’s a relatively easy shift for more recent graduates because they already know R. The focus for our training had to be on members of staff who have been with us for longer and weren’t familiar with it. And that can take some persuading – team leaders finding themselves managing teams who are using software they have never used themselves. We had to support and train at all levels (with tasters for managers who themselves may not need to delve into the finer details of R, but know they would know how to do it if they were doing their time again).

So, what have we learnt?

Be prepared to try new ways of working
Listen to your staff, who have a different perspective and fresh take on ways of working
Start small – it’s easy to make the case when you can showcase the benefits of the RAP approach
RAP improves the quality of what we do and eliminates errors
Be prepared to invest in training – and recognise your staff will be in different places
Use buddies – our central transformation team certainly helped with that, creating capacity in teams to RAP their processes
Be open and share your code – we publish our code on GitHub, a great community to share ideas and approaches

Listen to your enthusiasts

OSR’s report highlights that we have a small central transformation team to support teams with their RAP work. This is crucially important as the initial work to RAP your processes can take time, so the additional capacity to support our teams to enable this to happen is a must. This initial investment is worth it for the longer-term gains. It’s not all about making efficiencies either. It’s about streamlining processes, reducing error, and giving our analysts job satisfaction. They are now able to add more value because they have more time to delve into the data and help users understand what the data are telling them.

Our focus on transforming the processes for our existing publications has stalled due to many of our staff being redirected to supporting PHS’s key role in Scotland’s response to the COVID-19 pandemic. However, a success of our RAP work is that many of our new processes required to produce our daily COVID-19 statistics are done using RAP principles – they had to be as we’re producing more statistics, more frequently than ever before. Our earlier use of RAP meant we were in a good place to apply the techniques to our new ways of working.

And my final bit of advice? Listen to your enthusiasts – I’m glad I did.

Transparency is fundamental to trust – the government must learn from mistakes during the pandemic

As part of our work on statistical leadership, we are hosting a series of guest blogs. This blog is from Will Moy, Chief Executive of Full Fact.

This time last year, the UK was about to head into what would be the first of three national lockdowns. Already, the news was dominated by the coronavirus pandemic, shining a spotlight on data that shows no sign of fading any time soon.

In the past year, in-depth coverage and debate around the merits of statistical releases have become commonplace, while the nation has watched numerous public briefings filled with data and statistics from politicians and scientists.

As an organisation dedicated to fighting bad information, Full Fact knows just how valuable good information, used responsibly and communicated well, is to public life and public debate. We support the OSR’s vision that statistics should serve the public good.

It has, therefore, been heartening to see the new prominence given to discussions about the importance of data, statistics and evidence during the pandemic, whether among members of the public, in newspaper columns or in public statements from the government.

At the same time, the government needs to work to maintain public confidence in both its decisions and the data it is using to make them, which is a key part of ensuring it can tackle the crisis: this shapes what people are willing to do and how long they are willing to do it for.

We have all been asked to trust that the government is acting in our best interests, as our political leaders point to data and scientific evidence as justification for decisions that have had real and significant impacts on the way we live our lives.

In doing so, the need for a clear commitment to transparency, accountability and honesty in public life could not have been stronger. However, this bar was not always met.

Ministers or officials too often referred to data or statistics in statements to the press, the public or Parliament without publishing the full data so the public and intermediaries could check for themselves. A failure to be transparent on the data and evidence informing the government risks undermining public confidence in fundamental parts of the government’s response.

As fact checkers, we rely on publicly available information to scrutinise such statements. We can and do question departments but time is of the essence when trying to challenge false or misleading claims. By and large, a fact check is at its most valuable when it is published soon after the claim has been made, reducing the time any inaccuracy has to spread, and coming when the media or public are still interested in that topic or debate.

For instance, we were unable to assess claims about the turnaround times of tests made by Prime Minister Boris Johnson on 3 June 2020 until the necessary data was published a month later, on 2 July, and raised this issue with the OSR. Full Fact also found it impossible to assess a government target set out in July 2020 – to “test 150,000 at-risk people without symptoms per day by September” – because the right data wasn’t available at the start of October.

We discuss these examples, and more, in our Full Fact Report 2021, which is based on our experience fact checking the pandemic. In it we make recommendations for action from the government, including that when government departments, ministers or officials refer to data or information when making statements to the public, the media or Parliament, the full data must be made publicly available.

Throughout the pandemic Full Fact has been pleased to work alongside the OSR to help secure publication of missing or unpublished data, as well as challenging misuse of information – and both organisations are committed to continuing this work.

But publication of information in a timely manner should not require our interventions. The government must make a consistent commitment to doing so as soon as is possible, and before being asked.

It is undeniable that there are still many changes – some cultural, some technical, and many long-term – that need to be made within the statistical system. This will require efforts from statisticians at all levels and across all departments, and the OSR’s report into Statistical Leadership proposes a comprehensive set of recommendations towards this.

Crucially, the report highlights the pivotal role of committed leaders inside and outside the Government Statistical Service in ensuring government and society get the most trustworthy, high quality, and high value statistics. The report is in itself important and welcome leadership from the OSR: the recommendations should be promoted and acted on with progress monitored and evaluated.

Full Fact strongly agrees that efforts from statistical leaders, and the wider system, must be equalled by political leaders. It is of critical importance they lead by example, whether that is ensuring that statistical thinking is seen as essential within the civil service, or respecting the principles set out in the Code of Practice for Statistics.

Demonstrating respect for these responsibilities is essential if the public is to have confidence in the production and use of statistics in public life. The government must learn lessons from the pandemic and prove it is worthy of the public’s trust.

Talking numbers and making them count

As a communications professional, I use insights and ideas to implement and deliver impactful communications. Using statistics is a great way to use evidence and explain complicated information, but comms people are not always well-known for their statistical literacy and this can sometimes cause problems.

During a busy day in the comms team, any use of numbers in a press release, tweet or presentation should align with the Code, ensuring messages are clear, measured and appropriately tell the story. It is essential that production and use of statistics by governments command confidence in the statistics and organisations using them and help those listening understand the key messages.

At the Office for Statistics Regulation we are interested in how numbers can be used powerfully and collectively across government, to convey important messages and information. Statistical leadership by government is essential to ensure the right data and analysis exist; to ensure they are used at the right time to inform decisions; and to ensure they are communicated clearly and transparently in a way which will support confidence in the data and decisions made on the basis of it.

Statistical leadership is not just about having good leadership of the statistics profession. While this is important, we want to make sure individuals inside and outside the statistics profession show leadership. This should happen right through from the most junior analysts producing statistics to the most senior Ministers quoting statistics in parliament and media. It is relevant to all professions including policy and communications specialists.

Communications teams should work in close partnership with their department’s analysts, to ensure that any use of statistics does not distract from your key communications messages, or itself become the story. The winning situation is using statistics in a helpful way, to convey the right impact, help tell the story, gain understanding and enhance the organisation’s reputation in the process.

The Code of Practice for Statistics and its principles and practices of ‘trustworthiness, quality and value’ provides an excellent guide to ensure this is done as effectively as possible, to ensure users can confidently make decisions about the statistics that are presented to them, using them without question to access what they require and need.

Statistics can really add to public debate as we have seen during the events of COVID-19, when the nation has used numbers to understand the pandemic and its impacts on society, the economy and wider. But it is essential that anyone using numbers and speaking on behalf of government can communicate statistics effectively, in a way that commands confidence and helps those listening understand the key messages. The simplest way to achieve these outcomes and empower your message is to ask the right questions about statistics before you use them. And, if you still feel unsure then find another way to evidence your point.

However, comms people don’t need to know the Code inside out and should always work closely with Heads of Profession for Statistics for advice, support on using numbers and understanding of guiding principles.

If you are interested in finding out more about using statistics, the Code or Statistics Leadership please get in touch with me or visit our website.

Here are some tips…

Does it look right? Is that an implausible number? If it’s unusual, it could be wrong… what’s behind the surprise?
What exactly are we measuring and why? Is the source reputable and did they show their working?
Where do the data come from? What is the backstory and always be curious about someone else’s data. What do we discover if we go another click?
Only compare the comparable. Watch out for changes in definitions and different ways to measure the same thing . What’s the historical trend?
Presentation is key. Good use of pictures and graphics help convey meaning and should never cause confusion or misrepresentation
Remember to ask your Head of Profession for statistics, or a statistician who has worked to produce the data, for advice on how best to present numbers in communications.

I didn’t think I’d ever be interested in population statistics, but then I came to my Census

Ok so that’s not quite true. I’ve been working with a focus on population and society statistics for the past three years here at the Office for Statistics Regulation (OSR) and I love it. Working on areas such as migration, culture, wellbeing… it is fascinating. These statistics form such an important role in people, government and decision makers understanding and planning for our society in this country.

But even with my work hat off for a second, this is such an exciting time for statistics as we near Census day in England, Wales and Northern Ireland and work continues to prepare for Scotland’s Census in 2022. The Census is so important and a unique source of data. It will be used by many different people for many different purposes. For me, I think Census data can be most valuable when it is used to support how people understand their local communities, whether it be by local councils, community groups or even school students. I am so eager to play my part and fill in my Census return – online this time of course.

Work hat back on and I have been leading OSR’s assessment of the Censuses in the UK. Undertaking this assessment is the role OSR has when it comes to Census data and statistics. We aren’t collecting the returns or producing the data but we are working closely with the Census offices to ensure what they are doing, ultimately delivering Census data and statistics to the public, is in line with the Code of Practice for Statistics.

We collect evidence from Census offices, speak with users of Census data, make our judgements on compliance with the Code; reporting on this through our formal assessment reports and public correspondence with Census offices. That is the nuts and bolts of the assessment process. The reality of the assessment and OSR’s involvement is that we are continuously engaging with Census offices as they are developing and delivering their Census plans to support the best possible outcomes. We meet with Census offices regularly to discuss their work, share our views on their developments, and talk through how they have taken forward our findings. It has kept me busy since 2018 and will continue to do so until well after the data and statistics are published.

This ongoing conversation as part of the assessment is overlaid with a more formal reporting structure. We have completed phase 1 of the assessment and are kicking off phase 2 for England and Wales and Northern Ireland. For each phase of the assessment, the Office for National Statistics (ONS) and the Northern Ireland Statistics and Research Agency (NISRA) publish an accreditation report. These reports provide an update to Census users on how they consider the practices, processes and procedures for Census 2021 meet the standards of the Code. This provides OSR with evidence for our assessment and more importantly provides useful information on progress on the Census for all who are interested. You should definitely take a read!

We are always keen to hear from users of Census data and I have had some extremely valuable conversations with individuals and organisations to date – a big thank you if you have submitted feedback or if we have spoken in the past. As part of this second phase, we have opened up our user consultation once more . Your views and opinions are so important to help us understand what Census offices are doing well and how things could be improved. So please, do find out more and get in touch.

I hope you all are as excited as I am as we get closer to 21 March!

Next stop: National Statistics status

In 2020 we assessed estimates of station usage produced by the Office of Rail and Road (ORR) and designated them as National Statistics in December. In this blog, Lyndsey Melbourne, Head of Profession for Statistics at ORR, and Anna Price, the lead regulator for the assessment, talk about their experience and why assessments of official statistics are so valuable.

Where it all began…

Lyndsey: Most of our statistics were designated as National Statistics in 2012. In 2019 OSR carried out a compliance check and confirmed they continued to uphold the high standards expected. At the time we also discussed future assessments – in particular our most popular set of statistics, estimates of station usage, had never been assessed. These statistics provide unique information about each of the 2,500+ mainline rail stations in Great Britain. The granularity of data is one of the main reasons that these statistics are of interest to a very broad range to users: they are relevant to anyone no matter where they live. We were keen to further promote the quality and value of the statistics by gaining National Statistics status.

Anna: This assessment was a bit different to others. Usually we do our review, publish our findings and requirements, and then give producers a few months to meet these requirements. But when we first met with Lyndsey and Jay, the lead statistician for estimates of station usage, in April 2020 they told us they were keen to get National Statistics status in time for the next statistical release in December.

The assessment process

Anna: To support ORR to achieve this ambition, we adapted our usual process, for example sharing our findings and requirements as we developed them. This let the statisticians at ORR start on improvements while we worked on more complex findings and wrote our report, instead of waiting until the end. We had lots of meetings with the team during the project and were really impressed with the ideas they came up with each time we raised an area for improvement. I think the flexibility and enthusiasm of both teams was the reason that the project was so successful.

Lyndsey: Throughout the assessment, OSR were flexible and happy to work with us to agree timescales to fit in with our publication plans and around our day jobs. We were keen to work towards achieving National Statistics designation of the statistics in time for our next annual publication planned for December 2020. Otherwise, it would be up to 20 months before we could publish designated statistics!

OSR were very accommodating to this request and we worked closely during the following eight months to review and improve our statistics. OSR’s flexible approach allowed emerging requirements from their assessment to be addressed during the production process of the next set of statistics.

It’s fair to say that producing the annual publication at the same time as addressing OSR requirements was a challenge, but being able to confirm to users that our statistics had been successfully designated as National Statistics on publication day was very satisfying.

The value of this assessment

Lyndsey: During the assessment OSR spoke to a range of users and the feedback they obtained was extremely valuable. We have continued to speak to these users to understand their use of our statistics and how they could be improved further.

The improvement plan we developed to address OSR requirements and other feedback from users was a really useful tool for us. Sharing ideas and drafts with OSR along the way and getting their feedback was another valuable part of the process. We published this improvement plan on the user engagement page of our data portal to keep users up to date on the changes we were making.

Anna: The users of these statistics are passionate about them. So it was a lot of fun to hear about how they use the statistics, what they like and what would make using them even better. Seeing the variety of people who use the statistics, for a variety of purposes, was really motivating – it made it even more satisfying when we saw changes to the statistics in December which met the user needs we had identified.

At OSR we like to champion good practice, as well as areas for improvement. So it was nice to highlight the great work that ORR were already doing on these statistics – like the Twitter Q&A that ORR host on publication day, this year accompanied by a launch video and a live YouTube Q&A. It’s great to see statisticians putting themselves out there to talk about their statistics directly with users.

To keep up to date with our latest work, you can follow us on Twitter and sign up to our monthly newsletter.

Upskilling Government leaders in data and statistical literacy

As part of our work on statistical leadership, the Office for Statistics Regulation (OSR) is hosting a series of guest blogs. The first in the series is from James Kuht, from the Number 10 Data Science Team, who has been leading development of the Data Masterclass for Senior Leaders.

How data-literate do Senior Leaders in Government need to be?

There was a time when it would be inconceivable to imagine senior Government Ministers and officials addressing the nation accompanied by a series of charts and statistics, but over the course of the pandemic it has become relatively normal. Our Leaders are more reliant on data and statistical literacy than ever before and through the Code of Practice for Statistics have a commitment to demonstrating good use of data.

There has been rising use of data and statistics across Government over recent years; the increasing appetite for evidence-based policy-making (the ‘Nudge Unit’ perhaps being the most public example), compelling visual communications of data (as evidenced by departmental statistical releases awash with colourful charts), and new data science techniques such as machine learning increasing in their use. This shift is deliberate and strongly supported from the centre of Government, perhaps best summarised in Rt. Hon Michael Gove’s Ditchley Lecture, in which he calls on Government “to evaluate data more rigorously […] to evaluate policy successes and delivery failures”, with the help of “data analytics specialists”. His reasons are clear – “it is imperative that we learn the hugely valuable lessons that lie buried in our data”.

But Gove also highlights a hurdle we must first overcome to achieve this – “Government must also ask itself if its people have the skills necessary for the challenges that I have set out”. With many Senior Civil Servants being Humanities Graduates, & the largest source of Graduate-entry Civil Servants continuing to be so – there are many of our Senior Leaders who may not have had much formal training on data or statistics since GCSE’s or O-levels. Given this, we need to ensure that there is suitably high-quality training on data & statistics accessible to our current and future Senior Leaders.

What does a Senior Leader in Government need to know?

The first challenge is is to communicate that statistics are accessible and to inspire Senior Leaders with concrete examples of their value. This is no mean feat(!) – data and statistical training can often be intimidating. If it only caters for those who are ‘data-curious’ then it likely alienates the target audience. Focussing on making training relevant and of direct value to the learner is crucial.

The second challenge is to turn Senior Leaders into more effective consumers and commissioners of statistics – since, as Senior Leaders, this is largely their role when it comes to statistics. Effective consumers meaning that they are able to scrutinise the data they are presented, and aren’t hoodwinked by misleading headlines which might misuse a relative risk in place of an absolute risk, to give one common example. Effect commissioners meaning that they ask for the right analysis to support the decisions they make and understand the importance of transparency – making the analysis behind the decisions open when possible, and choosing the right visualisations to communicate their messages.

Crucially, third we need to improve the interface between Senior Leaders & analytical professionals. Senior Leaders don’t need to be able to produce complex analysis nor deeply critique a scientific paper, but they do need to be able to ask the right questions and request the correct support from Government Analysts who can.

The fourth and final objective is to give Senior Leaders an awareness of the opportunities and risks presented by new data science techniques, such as machine learning, which are undoubtedly having an increased impact on work across Government.

How do you teach Senior Leaders these skills?

Recently we have worked with stakeholders across Government, such as ONS’s Data Science Campus, the Government Digital Service, the Government Statistical Service and the Behavioural Insights Team to create an offering which addresses the challenges outlined above, a ‘Data Masterclass for Senior Leaders’.

We chose to make it available to Senior Leaders within Government as a 5-10 hour online course so that we could achieve maximum scale, whilst also protecting the health of participants who are largely, of course, working from home at the moment.

The Masterclass comprises a mix of 10-15 minute engaging talks from some of the UK’s leading experts on data & statistics such as Sir David Spiegelhalter, Dr Hannah Fry, & Sir Ian Diamond, with compelling data-exemplary case studies from across Government, to really bring the concepts to life. Feedback so far has been strongly positive from the 76 Senior Civil Servants who have completed the course, with an average rating of 95%, and comments like the below reassuring us that we’ve at least succeeded in achieving objective one of the above!

“A really fabulous course. Accessible and relevant – you managed to bring what could have been quite a dry issue to life, and make it not too complicated for the data-challenged”

Going forward, we’re looking forward to scaling the Data Masterclass in partnership with the ONS Data Science Campus and making it available to every Senior Civil Servant. We hope this upskilling supports the brilliant work of many stakeholders in Government’s use of data and statistics, so that we can derive the “hugely valuable lessons” that lie within our data, in a way that is responsible and respects the privacy of UK citizens.

Can we make trust go viral? Increasing public trust in a digital age

It’s roughly a year ago that I gave evidence to the House of Lords Democracy and Digital Technologies Committee.

A lot has happened in a year. I did things at the evidence session that don’t happen now. The Committee session took place in person, not over Zoom. I shook hands with the other witnesses. Afterwards I chatted with several people, including the Committee Chair, Lord Puttnam, and gave no thought as to whether we were two metres apart.

But looking back on the Lords report, published in June last year, it’s clear that the report remains highly relevant.

The Committee was grappling with one of the challenges of our age: whether or not norms of democracy and democratic debate are under threat from digital technologies, social media and so on. In particular, if it is true that false information can circulate more freely in social media than they did in the past, does that erode one of the bedrocks of democracy? Lord Puttnam’s foreword vividly describes this as another kind of virus:

“Our Committee is delivering this Report to Parliament in the middle of an unprecedented health and consequential economic crisis. But our Report focuses on a different form of crisis, one with roots that extend far deeper, and are likely to last far longer than COVID-19. This is a virus that affects all of us in the UK – a pandemic of ‘misinformation’ and ‘disinformation’. If allowed to flourish these counterfeit truths will result in the collapse of public trust, and without trust democracy as we know it will simply decline into irrelevance.” Foreword by Lord Puttnam, Chair of the Committee

Empowering citizens

The Committee’s report made a range of thoughtful recommendations. A lot of them cover legislative and regulatory change, around how digital platforms operate and electoral laws should adapt to the digital era.

And they align well with the principles that underpin our work at the Office for Statistics Regulation (OSR) and the Code of Practice for Statistics.

This is because the report highlights the importance of information that serves the public good. This includes empowering citizens:

Alongside establishing rules in the online world, we must also empower citizens, young and old, to take part as critical users of information. We need to create a programme of lifelong education that will equip people with the skills they need to be active citizens.

There is growing interest in this notion of digital or statistical literacy. The recent ESCoE report on understanding economic concepts has grown awareness of issues around public understanding and engagement in economic matters. Meanwhile, OSR’s own review of literature on the public good of statistics found that a lack of statistical literacy can lead to misunderstanding or misuse of statistics, and susceptibility to misinformation.

The report also touches on the public’s need for high quality information. The report places journalism at the centre of this: “The public needs to have access to high quality public interest journalism to help inform them about current events.”

We agree with the crucial role of journalism. But journalism is often playing the role of intermediary between Government and citizens. And if citizens should be equipped to interpret information, so too the Government has a crucial role to play in providing clear, accessible, high quality information.

Statistics that serve the public good

The pandemic has shown this repeatedly. The UK’s four governments have put an enormous emphasis on providing accessible data on cases, testing, deaths, hospitalisations, and now vaccines. All four governments have improved the range and depth of data available, and all have sought to comply with the Code of Practice for Statistics. There have been occasions where we’ve pointed out ways in which the information can be better presented, or more information made available. But that’s against the backdrop of commitments to inform the public. And it’s clear that this information is used and valued by the public.

Data and evidence has remained prominent for Parliament during the pandemic. The Commons Science and Technology Committee report recently published a strong report on The UK response to covid-19: use of scientific advice. And the Public Administration and Constitutional Affairs Committee is currently undertaking an inquiry into the transparency and accountability of Covid-19. Both of these Committees have drawn on our work at OSR to emphasise the importance of presenting data well. The Science and Technology Committee summarised the issues thus:

As the Office for Statistics Regulation advised, in order to maintain high levels of confidence, data and statistics should be presented in ways that align with high standards of clarity and rigour—especially when they are used to support measures of great public impact.” Key finding 6, Science and Technology Committee

The Government’s role in informing the public

My reflection, then, looking back over the last year and at the Lords report, is how important the Government’s role is in informing the public.

As I said to the Lords Committee:

What keeps me awake at night is that so many official statistics have the potential to be valuable assets, not just for the policy elites, but for the public more broadly. But that potential is unrealised, because they are presented and communicated in a dry and almost mechanical way: we did this survey and got these results. They are not presented in a way that engages and links into the things that people are concerned about. I worry much more about that than I do about the more pervasive concerns people have about misinformation. I worry about the good information not getting out there. Q93, Select Committee on Democracy and Digital Technologies

The enduring lesson of the last 12 months is this: providing information that is trustworthy, high quality and high value to the public is a crucial function of Government.

Which COVID-19 deaths figures should I be looking at?

Every day we see figures for number of COVID-19 deaths in the UK quoted in the media, but what do these mean, and which figures should we pay most attention to?

With the rising death rate, and the complexity and potential confusion surrounding this seemingly straightforward measure of the impact of COVID-19, we are increasingly being asked our view on which data should be regarded as the best measure of COVID-19 deaths.

Of course, whichever way the numbers are presented, each individual death is a sad event. But it is really important to understand the strengths and limitations of the data being considered in order to understand the pandemic and learn from what the UK has experienced.

There are many official sources of data and each has a place in helping understand the impact of COVID-19. Our blog from August goes in to more detail about the difference sources, their uses and limitations. Here we outline some of the key issues to consider when thinking about which figures to use.

What is the difference between figures by date of death and figures based on date reported? Which should I use?

A commonly used headline is the number of deaths reported each day in the UK Government’s coronavirus dashboard, based on deaths which occurred within 28 days of a positive COVID-19 test. This has the advantage of capturing all newly reported deaths each day. It is understandable that this figure makes headlines as it is the timeliest data published, and captures all the additional deaths (within 28 days of a positive COVID-19 test) which government has been made aware of within the previous 24 hour reporting period. However, it has limitations and it is really important that in the reporting of these figures the implications of these limitations are clear.

As well as data by date reported the UK government coronavirus dashboard includes data on deaths within 28 days of a positive COVID-19 test by date of death on the deaths page of the dashboard. These are usually considered to be reasonably complete from about five days after the reference date. Looking at data by date reported shows large fluctuations in numbers, particularly after weekends and bank holidays. Data on date of death will give a better sense of the development of the pandemic and the changing rate of deaths.

This difference between figures for date reported and date of death has been particularly notable in the period following Christmas and New Year given bank holidays and the higher rates of deaths seen over the period. For example, looking at data published on 21 January for deaths within 28 days of a positive COVID-19 test:

Deaths by date of death have a current peak on 12 January with 1,117 deaths (compared with a peak of 1,073 on the 8 April).
Deaths by date reported have a peak of 1,820 deaths on 20 January (compared with 1,224 on 21 April).

Data by date of death should always be used if possible.

How can I best understand if COVID-19 was the cause of death?

The data outlined on the coronavirus dashboard, highlighted above, are based on deaths within 28 days of a positive test. There will be occasions within these cases where an individual had a positive COVID-19 test, but this was unrelated to the subsequent death. There will also be cases where a death was due to COVID-19 but occurred more than 28 days after a positive test result. PHE has published information in a technical note which looks at the impact of the 28 day cut off compared with alternative measures.

A more reliable measure is based on data drawn directly from the system of death registrations and includes data where COVID-19 is mentioned on the death certificate. The Office for National Statistics (ONS) publishes weekly figures, including a UK figure drawing on data from National Records Scotland (NRS) and Northern Ireland Statistics and Research Agency (NISRA).

ONS data are based on information from death certificates and include cases where COVID-19 is likely to have contributed to death (either confirmed or suspected) in the opinion of the certifying doctor. The provisional count is published weekly, 11 days after the end of the time period it covers. These data have many strengths, but provisional figures first published will not capture all deaths due to registration delays.

How can I best understand the impact of the pandemic on deaths?

The measures outlined above all aim to give counts of deaths where COVID-19 infection was a factor in the death. A broader measure which looks at the change in deaths because of the pandemic, whether or not due to a COVID-19 infection, is “excess deaths”. This is the difference between the number of deaths we would expect to have observed and the number of deaths we have seen. This is generally considered to be the best way to estimate the impact of a pandemic or other major event on the death rate.

ONS published a blog alongside its latest publication of excess deaths, which highlights the complexities in this measure. For example, a single figure of how many deaths there have been one year compared with a previous year may not be helpful, due to changes in the population. For this reason, in addition to providing the counts of total deaths, ONS produces estimates for excess deaths in a number of different ways. In its weekly statistics it compares numbers and rates to a five-year average, so that is comparing a similar period in terms of life expectancy, advances in healthcare, population size and shape. It also publishes Age Standardised Mortality Rates for England and Wales so that rates taking into account changes to the population size and structure can be compared.

OSR in 2021

2020 was a year in which statistics and data were at the centre of public life. We became a nation of armchair epidemiologists.

There’s no sign that this will change in 2021. We expect statistics to remain in the public eye with the pandemic continuing to effect all aspects of life, a national COVID-19 vaccination programme, EU departure, the decennial Census, and Scottish and Welsh elections.

Statistics and evidence remain crucial to managing the current pandemic and understanding its impacts on the economy and society. This meant that 2020 was very busy for us at the Office for Statistics Regulation.

Our Annual Casework report gave some indication of the noticeably higher volume of ‘use of statistics issues’ being reported to us, as questions and queries about numbers and their uses increased in what proved to be our busiest year to date. In the light of this experience, at the end of 2020 we updated our Interventions Policy and supporting FAQs which clarifies the context for interventions that we have made and will make going forward.

At OSR we cover a huge range of issues. Our 2020 assessments and reviews illustrate this. We highlighted the need for significant improvements in the statistics, including adult social care, employment, poverty, and business demography to name a few.

OSR in 2021

We will continue to review the use of statistics in the pandemic, and have some high profile reports due for publication, including our review on developing statistical models to calculate exam grades, one of the most high profile examples of statistics in the public eye in recent times. We will also publish our report on statistical leadership in the public sector.

Our work in 2021 will include:

Data gaps and the public good

These are ongoing issues facing the statistical system, and we will continue to focus on data-gaps, public good of statistics, automation modelling and much more.

Granularity of statistics

We will clarify our expectations of producers to provide more granular data, including data broken down by key characteristics and geographies. It is important that statistics reflect society and that people can see themselves in the statistics available.

Our role in data

We will be developing our thinking on our role in regulation of data. This will build on projects underway, including the exam grade review, and help us determine how far and how regularly we want to focus our regulatory work beyond official statistics. We will also seek to maintain trust in National Statistics and continue to focus on this core aspect of our regulatory work programme.

Quality

The pandemic has led to significant changes to how data are produced and what is available, and the departure from the EU may have significant impacts too. We will consider the implications of some of the decisions being made and what this means for our judgement of quality of statistics. One area we will consider for an early review following EU departure is ONS decisions on the classification of organisations to the public and private sector. There are new governance arrangements covering these decisions and it is important that the arrangements are effective and command user confidence.

We will continue to support producers of statistics

Our role during the pandemic has been to support producers, look at issues that have arisen and promote trust and transparency in statistics and data that support decision making. This work has seen us produce guidance, publish statements and get involved in various issues such as impacts on the care sector, local area data, test and trace, rough sleeping, comparisons and prevalence rates, measurement related to deaths, and hospital capacity and occupancy.

And we will continue to grow the use of the Code of Practice. As the boundaries between official statistics and other forms of data come down, we have also seen many more committed and engaged organisations adopt the Code of Practice for Statistics and successfully created a new award in partnership with the Royal Statistical Society to recognise outstanding examples of how well this can work within our community of practice.

To conclude, there is much to learn from how the pandemic has impacted on statistics and data. In 2021 we will continue to look at the statistics system to build on the dynamism and innovation it has displayed during the pandemic as new areas of demand for insight emerge.

To keep up to date with our latest work, you can follow us on Twitter and sign up to our monthly newsletter.