Dear James
Compliance review of statistics in development from the Winter Coronavirus (COVID-19) Infection Study, England and Scotland
Thank you very much for inviting us to independently review the trustworthiness, quality and value of the statistics in development from the Winter Coronavirus (COVID-19) Infection Study (Winter CIS). Information about coronavirus continues to have high interest for expert users, such as public health and NHS officials, health researchers and the media.
The Winter CIS was launched by ONS in November 2023, covering households in England and Scotland. Your team published data from the study for transparency every two weeks from 7 December 2023 to 14 March 2024. Raw results from the study were also sent securely to the UK Health Security Agency for onward analysis to allow that organisation to estimate the incidence and prevalence of coronavirus SARS-CoV-2, which can cause COVID-19.
I welcome the innovation and agility shown by your team. This has been demonstrated through transparent joint and collaborative working with the UK Health Security Agency, with continuous learning, allowing your teams to iterate your working processes. We note from your recent Blog that statistics to capture information about respiratory viruses will continue to evolve and we consider that retaining the label ‘official statistics in development’ seems appropriate.
Explaining study representativeness
As you know, in random sample surveys it is important that the achieved sample adequately represents the proportions of various characteristics (age, ethnicity, locality) shown by the community under scrutiny. This will reduce biases to the survey results and provide a closer estimate to the true value. The design of the Winter CIS as a longitudinal panel survey, since it is based on a cohort of people who responded to the Coronavirus (COVID-19) Infection Survey (CIS), means that it is not a random sample survey.
Even with an imbalanced sample, appropriate weighting helps to ensure representative estimates. You have gone some way to providing transparency about the accuracy of the Winter CIS estimates, by publishing the survey results data, the response rates for various survey waves and the population basis that was used to calculate the estimates. You have also published a Quality and Methodology Information report (QMI) which makes clear that the survey might over-represent or under-represent certain groups and sets out information about what the survey data can and cannot be used for. The QMI provides some detail on the data collection procedures, quality assurance processes, response rates and representativeness. Please ensure it is updated to include more detail to fully explain how you have reduced biases in the design and weighting of the study, as outlined in the paragraph below.
To assist further development of these and other statistics based on the Winter CIS sample frame, (such as the ad hoc release titled Experiences of GP practice access), it would be helpful for researchers to fully understand the various recruitment processes, the study design, the methodological procedures and the weighting choices that have been taken. For both the Winter CIS and other releases that use the Winter CIS as a sample frame, you should publish information about:
- The distinction between the processes of participant recruitment and engagement, giving consideration to digital inclusion and the impact of incentives, or their removal.
- The maturation of the dataset over time, to understand the personal characteristics of the respondents over subsequent waves of the study.
- The statistical inferences made during the course of calculating the estimates, including the weighting decisions.
We are pleased to hear that you have agreed to publish updated metadata by the end of June and place the final survey results and weighting data into a repository to further academic research on it, such as ONS’s Integrated Data Service.
Coherence across the UK
Both UKHSA and yourselves have been very clear to link to other statistics about the coronavirus and COVID-19. We welcome this approach to provide coherence with existing coronavirus data from the rest of the UK.
Users of the Winter CIS have identified the need for comparable statistics across the UK, particularly to understand the effects of coronavirus on the lives of individuals. During any future planning for a study of this nature, we consider that time should be taken by statistics producers to fully consider the possibilities for the development of coherent UK-wide statistics.
I would like to thank your team for its positive engagement with us during this review. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter.
Yours sincerely
Mark Pont