Ed Humpherson to Dr Sarah Rasmussen: ONS COVID-19 Infection Survey ad hoc analysis

Dear Sarah

ONS COVID-19 Infection Survey ad hoc analysis

Thank you for raising your concerns about the Office for National Statistics’ (ONS) ad hoc analysis of the number of school workers, key workers and other professions in England who had COVID-19, which was included in the 6 November Coronavirus (COVID-19) Infection Survey bulletin. It is important that data are shared in a trustworthy way and that government can draw on reliable data to make important decisions.

We have considered the concerns you raised and respond to them below. We also want to make you aware that we are undertaking a review of the COVID-19 Infection Survey as a follow up to the Rapid Review of the survey we completed in May 2020. We will consider some of the points you raised in more detail through this review, which we expect to publish in early 2021.

Presentation of statistics in the Office for National Statistics bulletin

You highlighted concerns about the ad hoc analysis published by ONS as part of the COVID-19 Infection Survey results on 6 November. We do not think that ONS intentionally presented the analysis in a misleading way. However, there are some changes which could have been made to the analysis and the accompanying text to support those reading the bulletin in understanding the analysis.

ONS has published a statement in response to some of the concerns raised about the analysis, which addresses some of the issues you raised. We have also considered the concerns:

End date of the analysis: ONS explained that the ad hoc analysis only covered the time period up to 16 October, not 31 October like other analysis in the bulletin, because the occupation data requires further processing and is only available around two weeks after the initial headline results. This meant that outputs could not be produced as quickly for this analysis as other parts of the publication. It would have been helpful if ONS had given a rationale for the discrepancy when the data were first published.

Evidence of a change: We agree that ONS could have done more to explain the uncertainty around the estimates for education staff categories, and the implications of this. It could have been clearer that no evidence of a difference is not the same as evidence of no difference. It should also have explained the impact of small numbers of cases in some categories on the ability to determine any differences. ONS has now updated the analysis to include “all teachers” in one category.

Use of data by the Chief Medical Officer (CMO) prior to publication

My team has also looked into the timeline for the sharing of information between ONS and the UK Government and SAGE. In our view the CMO was not quoting the ad hoc analysis published by ONS on 6 November when he addressed the Science and Technology Committee on 3 November. ONS shared the results of the analysis with SAGE on 4 November, which was then referenced in the report that was made public on 13 November. We have been told that the evidence which informed the CMO’s response on 3 November included other published ONS data, data on COVID-related deaths by occupation. His response was consistent with the Statement from the UK Chief Medical Officers on schools and childcare reopening made in August 2020, though we would have liked to see clearer references to the evidence quoted in this statement.

The Coronavirus (COVID-19) Infection Survey is a critical source of information on the pandemic, so it is important that the statistics meets the highest standards of the Code of Practice for Statistics. We will investigate further the issues you have raised with us as part of our in-depth review of the trustworthiness, quality and value of the Infection Survey statistics.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

Response on Death-Registration Statistics

Dear Professor Bird

Thank you for your letter in March 2019 asking us to consider removing National Statistics designation from death-registration in England, Wales and Northern Ireland due to the delays that occur when a death is referred for a coroner’s investigation in those countries.

We designate statistics and as part of that process we consider the quality of the data on which they are based. Over twenty sets of mortality statistics based on registration data in England, Wales and Northern Ireland have been designated as National Statistics, following assessments of their adherence to the Code of Practice for Statistics[1]. We are assured that the steps statistics producers take to handle the impact of late registration meet the high standards expected of National Statistics and do not consider that de-designation is necessary.

This letter:

  • sets out the rationale for our judgement;
  • highlights some improvements that the Office for National Statistics (ONS) and the Northern Ireland Statistics and Research Agency (NISRA) are making to ensure that users fully understand the impact of late registration;
  • notifies you that I have written to the Chief Coroner for England and Wales and the head of the Coroners’ Service in Northern Ireland.

Our expectations of statistics producers

Concerns about data quality can result in de-designation (or a failure to gain designation) if there is evidence that statistics are materially misleading or do not provide the best available estimate as a consequence. The Code of Practice sets out our expectations of what statistics producers and data providers should do in situations where a data quality issue cannot be addressed at source – providing there are no concerns about intentional manipulation. We expect statistics producers and data providers to:

  • provide users with clear and accessible information about the extent and impact of data
    quality issues, and any associated limitations;
  • consult with users about how best to address data quality issues and meet their needs. In
    this instance, that includes whether to prioritise timeliness by using registrations data, or
    prioritise providing information about deaths within specific periods, by using occurrences
    data;
  • ensure that data limitations do not result in the statistics or data being materially misleading.

We used these criteria to review the information provided about late registrations by ONS, NISRA, NHS Digital, and the Clinical Practice Research Datalink (CPRD). We have also looked in detail at the statistics most affected by late registrations – suicides and drug-related deaths – around half of which occur outside their registration year.

Do statistics producers meet these expectations?

Around 4% of deaths are registered after the year in which they occur. We agree with ONS’s assessment[2] that registration data therefore provide an accurate and timely estimate of overall annual death occurrences and their trends. The majority of estimates relating to specific causes of death are also only minimally affected by late registration. Statistics based on causes of death for which there are longer registration delays require additional steps. For example, by delaying the statistical release to allow more time for registrations to occur, as happens with infant mortality. Where users need more timely estimates, for example for deaths by suicide, additional information
is provided to highlight how the estimates have been affected by late registration.

Considerable efforts have been made in recent years to improve the information about late registrations provided to users, largely due to your and the RSS’s efforts to highlight the importance of this issue. ONS conducts user consultations regularly and they recognise that choosing to prioritise the timeliness of some mortality statistics does not meet all user needs. The ONS and NISRA bulletins and tables make clear that the statistics refer to death registrations rather than occurrences.

A wide range of information about the extent and impact of late registrations on the statistics is available. ONS is now planning to improve this information by bringing it together in one place to give users a more comprehensive picture of the impact of late registration across all mortality statistics. NISRA also has plans to develop the information provided to users in the coming year. We encourage both organisations to work together to share relevant information on this topic. ONS and NISRA have also been engaging with the Chief Coroner for England and Wales, and the Northern Ireland Coroners Service, respectively, to highlight this issue.

Our Review identified some additional steps that ONS and NISRA have agreed to take to enhance users’ understanding of the impact of late registrations, particularly relating to suicides and drugpoisoning.

Do research data providers meet these expectations?

ONS, via its NOMIS platform, provides clear information for users about late registrations, at multiple stages in the data access process. CPRD’s new information resource for researchers about late registration, and their processes for supporting users who apply for these data, are helpful and comprehensive. We agree that it would be valuable for the CPRD website to highlight the issue of late registrations on their website. We will speak to NHS Digital about the information they provide for users and suggest they provide a link to CPRD’s new resource.

Next steps

Thank you for raising this important issue. We note that late registration prevalence is increasing so we will keep this matter under review. I have also today written to the Chief Coroner for England and Wales and the head of the Coroners’ Service in Northern Ireland to outline our concerns about this issue. I am copying this letter to Iain Bell, Deputy National Statistician for Population and Public Policy, Office for National Statistics, and Siobhan Carey, Chief Executive, NISRA and Registrar General, Northern Ireland.

Yours sincerely

Ed Humpherson

[1] https://osr.statisticsauthority.gov.uk/code-of-practice/

[2] https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/methodologies/impactofregistrationdelaysonmortalitystatistics2016

Related links:

Ed Humpherson to His Honour Judge Mark Lucraft QC

Ed Humpherson to The Honourable Mrs Justice Keegan DBE QC