Mark Pont to Ruth Studley: COVID-19 Infection Survey Statistics

Dear Ruth,

COVID-19 Infection Survey Statistics

As you are aware, we recently undertook a review of statistics from the ONS COVID-19 Infection Survey (CIS). Given the cessation of the REACT study and changes in testing regimes across the UK, these statistics are more important than ever, and are now the most up-to-date, reliable estimate of the number of COVID-19 infections in the community. They contribute to scientific advice provided to governments, informing decisions on the ongoing management of the pandemic. And there is a high level of public interest in the survey – people really value the statistics and many use them to make day-to-day decisions, including potentially serious decisions for those vulnerable to COVID-19.

Since I last wrote to ONS about these statistics, it is great to see how much work you have done to develop them. ONS demonstrates a strong commitment to the Code of Practice for Statistics in its work on the CIS. The background to this review and our main findings and recommendations are set out in more detail below.

Background

In May 2020 we carried out a rapid review of the first statistics from the CIS, endorsing the agile approach taken by ONS to deliver timely statistics about a rapidly evolving national emergency. Following this, the CIS expanded from a survey of approximately 28,000 people in England, to over 150,000 people across the UK. Given the expansion of the survey, we undertook a more in-depth review published in March 2021. We considered that ONS was providing clear and relevant insights for users and were impressed by the strong working relationships ONS had with its survey provider and academic partners. We identified several areas to support further development of the statistics, including improvements to user engagement and the communication of methods.

In February 2022, the UK Government published its Living with Covid plan. This included plans to maintain a scaled back version of the CIS. In June 2022, ONS announced changes being made to the survey, introducing a digital questionnaire and sending swab and blood sample kits by post. In light of these changes, we agreed with your team that we would undertake a further review of the statistics against the Code of Practice. Since the move to digital is underway and the implications won’t be fully understood until the autumn, our findings reflect current practices along with planned changes. We have also highlighted our expectations for what ONS should consider and communicate about these changes in future.

Findings

Trustworthiness

Over the past year, ONS has demonstrated the ability to adapt to changing needs for timely information. For example, at the start of the Omicron wave there were emerging concerns that the new variant could be more severe. In response, ONS introduced an early additional headline release. When infection levels became more stable and decreased, ONS returned to the usual weekly schedule. This showed excellent consideration of user needs and ONS’s ability to be flexible. Subsequently, ONS has developed a scenario-based publishing plan to determine whether more-frequent releases are required and how regularly additional analysis is published.

We welcome the blogs which ONS published this year informing users about plans for the survey (May 2022 and July 2022), as well as its press release in June 2022 regarding the change in mode to digital data collection. However, these came quite late in the process and after information about the changes had been provided to participants and published in the study protocol, causing some confusion for users. While we appreciate that ONS is working in a fast-moving environment and that decisions about the survey programme may sit with other partners, we consider that it could have done more to keep users informed in a clear and timely way about planned or potential changes to the survey. ONS should ensure it keeps users informed about development plans, including the future of the survey as the financial year ends, even if these plans are tentative and subject to change.

Quality

We consider that the involvement of experts in various fields, use of appropriate methods, thorough quality assurance, and ONS’s efforts to ensure representativeness provide sufficient assurance that the statistics from the CIS continue to be of high quality and fit their intended uses. The production of the statistics involves experts in surveys, public health, and statistical methods from across government and academia. Analyses are based on sound methods and ONS carries out extensive quality assurance, including triangulation with other COVID-19 data sources when these have been available. One important aspect of quality is representativeness. ONS acknowledges that the survey over and underrepresents certain groups and has taken steps to address this, including adjustments to sample sizes to account for differing response rates and by using post-stratification to adjust the estimates to be more representative of the population.

Since our last review, ONS has published a lot more information about methods and quality. This includes a Quality and Methodology Information document, technical articles to communicate complex analyses and methodologies (such as this article on Cumulative Incidence), and the continued development of the methods article (for example, developing a flowchart to explain the processes involved in producing the statistics). However, sometimes there is a delay in updating the methods information and expert users are keen to see this updated in a timelier way.

It is great to hear about how the principles of Reproducible Analytical Pipelines have been introduced to the data processing, analysis, and the dissemination of outputs. This has involved using code to streamline processes and version control for code management. The benefits you have realised from this, such as improved quality assurance, minimising the risk of errors and increased collaboration, enhance the trustworthiness, quality, and value of the statistics for users.

It is also very encouraging to see your plans to understand and publish information about the change in mode. This includes information about:

the changes involved in the move to digital
the results of the parallel run (where some data were collected in-person and some digitally)

any impact on the response rates and sample, and therefore the representativeness of the survey
any effects on the survey estimates, for all measures currently published, and on the time series

Your recently published initial findings on the effects of the change of mode offer a first insight into many of these aspects. It will be important for ONS to continue to inform users about the impacts of the change in mode on the statistics to support appropriate use. We look forward to seeing the results of further work in due course.

Value

We are particularly impressed by the work ONS has done since our last review to understand and engage with users. The additional resource dedicated to user engagement, including the creation of the CIS strategic development hub, has clearly been instrumental in this. ONS has shown a willingness to talk with users, including offering one-to-one conversations to understand their requirements. ONS has also built good relationships with key government stakeholders, including good working-level relationships with the devolved administrations. However, devolved administrations would further benefit from increased engagement at a senior level. ONS should ensure that the devolved administrations have appropriate input at the programme level, for example on decisions relating to changes to the survey.

As a result of this user engagement, ONS has developed a good understanding of the wide-ranging audience these statistics have. ONS has responded to this by producing a large range of outputs, each aimed at meeting different user needs. The range includes bulletins with key findings, data tables, technical articles, blogs and methodology and quality documents. However, we found that it can be challenging for users to find the information they need – for example, it can be difficult to find additional analysis, technical information about methods and blogs about the CIS. Given the huge amount of valuable information available from this survey it is a shame that it is not always easily findable.

For the statistics to remain as valuable as possible, it is important for many users, particularly those in the devolved administrations, that granular breakdowns are still available following changes to the survey mode. Users also told us that they are particularly interested in understanding more about long COVID. We welcome ONS’s plans to carry out further analyses using data from the survey, including on antibody levels and looking back to understand in more detail what happened over the course of the pandemic. As highlighted in our review of lessons learned during the pandemic last year, ONS should also consider how the CIS can be adapted to play a role in understanding public health in future.

We would like to thank all the teams in ONS for their positive engagement with us throughout the review process. The dedication of ONS staff to the production and continual improvement of these statistics is commendable. We look forward to hearing from you as you continue to develop these statistics.

Yours sincerely

Mark Pont
Assessment Programme Lead