Dear Emma,
Deaths involving COVID-19 by vaccination status publication
I am writing to you about the Office for National Statistics’ (ONS) Deaths involving COVID-19 by vaccination status, England: deaths occurring between 2 January and 24 September 2021 release, which was published on 1 November.
It is good that the ONS is looking to produce additional analysis to provide insight into key issues. However, the very high public interest in data related to vaccinations means it is critical that data are clearly explained, and that users are supported to interpret the data appropriately.
There has been criticism of two aspects of the deaths involving COVID-19 by vaccination status publication: first, the headline finding giving the age-adjusted risk of deaths involving COVID-19 between 2 January and 24 September 2021; and second, the presentation of deaths (all deaths) by age group (Table 4).
Thank you to your team for its positive engagement with the Office for Statistics Regulation and the proactive approach it is taking to address concerns. Your team clearly has a good understanding of the issues and has thought about them a lot. The additions to the publication and blog are helpful in supporting understanding and interpretation. However, controversy remains.
Risk of death involving COVID-19
The headline in the publication is the age adjusted risk of deaths involving COVID-19 for vaccinated and unvaccinated groups for the period 2 January to 24 September. This was also the key message in the main tweet associated with the publication. Focusing on the headline figure has been unhelpful and has undermined the more helpful analysis provided later in the report. The headline figure is based on a time period driven by data availability. While the age-standardised mortality rates for deaths involving COVID-19 are consistently lower for people who have received two vaccinations, the size of the difference varies enormously depending on the time frame chosen. The data cover a period when very few people had two doses of vaccination, to a period when the majority of the adult population had two doses (data taken from gov.uk on 24 November 2021 show second dose uptake for age 12 and over in England was 0.8% on 10 January 2021 and 77.4% by 24 September 2021). It also covers a period when case rates varied significantly as well as the levels of natural immunity in the population.
It is clear from Figure 1, mortality rates by week, that there is huge variation in age standardised mortality rates over time, and that there is considerable variation in the difference between the rates for vaccinated and unvaccinated. Figure 1 also benefits from a visual representation of the uncertainty associated with the data. The Comparison to other studies section also provides helpful context. The problems might have been avoided if, instead of considering the whole period, age-standardised mortality rates and their ratios had been calculated over shorter intervals, such as monthly or bi-monthly, thus complementing Figure 1.
Given the analysis carried out, more should have been done to highlight the uncertainty associated with the headline figure. The publication itself did go some way towards putting the headline figure in context, but this was not the case in the more condensed description in the tweet.
The blog published by ONS on Friday 19 November provides further explanation of the figures. It is helpful in explaining the approach used to calculate the figures and some of the limitations with the headline figure. The blog is clear up front that “Due to various factors described in this blog this figure changes over time. It is also important to understand that it is not a measure of vaccine effectiveness.” The blog also outlines plans for further analysis, including data covering more recent time periods. These are positive steps in supporting the public to interpret the data, but I would urge you to take the focus off the headline figure in any future publications.
All deaths by age band
The publication from 1 November also includes additional data in the accompanying dataset. It is good that ONS expanded the data available in response to user feedback. However, more should have been done to address the risk that the additional data could mislead people, specifically arising from the wide 10 – 59 age band. Of particular concern are the data in Table 4 which show weekly mortality rates by vaccination status (for all deaths, not just those involving COVID-19) by age group. Your team explained that the higher mortality rates for vaccinated people are likely to be a consequence of the wide age band. Older and more vulnerable individuals are more likely to be vaccinated and also have higher mortality rates, so therefore the groups with higher mortality rates are more likely to be in the vaccinated group (NHS England and Improvement vaccination statistics show vaccine uptake is higher in older age groups. The youngest groups are not eligible for a vaccine (under 12) or second dose (under 16)).
The footnote you have now added to Table 4 helps explain this issue. However, the risk of the data being used to support incorrect claims remains. It is good that you have recognised these concerns. Your team explained to us that where future analysis covers wide age bands it will apply age standardisation within the bands, the team also told us when the analysis is updated it will include more breakdowns by smaller age bands. It is important that this is done promptly.
It is commendable that ONS is responding to user interest in data about vaccinations and seeking to shed light on important issues in a succinct and accessible way. However, this has increased the risk of misinterpretation. ONS must do all it can to support understanding of the data it publishes and seek to minimise the risk of misinterpretation or misleading use of the data.
Yours sincerely
Ed Humpherson
Director General for Regulation