Jess McGregor and Julie Billett to Ed Humpherson: Public Health England Care Home Management Information

Dear UK Statistics Authority

RE: Public Health England (PHE) Care Home Management Information

We are writing on behalf of Camden Council and in relation to the PHE management information on care homes.

Camden Council have had serious concerns about how the data on outbreaks within care homes for Camden have been represented. The data first came to our attention a few weeks ago when the percentage of outbreaks was over 100% and subsequently rose to 136% when the local evidence shows the percentage affected is actually much lower. The discrepancy has caused a lot of work locally in explaining to political leaders and the media why the figure is incorrect and it is of particular concern to us that it was being reported in this way from a national agency. It has caused substantial issues around credibility and trust of statistics. On further review of the outputs from the data, we are equally concerned that the presentation of this management information is generally misleading and below the standards we normally expect to see.

While we appreciate that PHE has now corrected the Camden data (by removing homeless hostels from figures and a care home which on testing had negative results) and has made an amendment in the meta data to make it clearer that the numerator and denominator are not equivalent, we still think more should be done to ensure that the data that is being published by a national agency is not misleading. We also understand that PHE are looking into making further amendments to this data, but we are unclear about what specific action they will be taking to rectify issues.

Our specific concerns are:
1. Discrepancy between the numerator and the denominator

The reason that Camden has been reported >100% is that the numerator and denominator are not equivalent, with non CQC registered care settings included in the numerator but not the denominator. We have requested that PHE reviews whether it is appropriate to publish data with this type of discrepancy when it should be possible to limit the numerator to CQC registered care homes and make the numerator and denominator data equivalent.

We have also requested that PHE considers whether combining settings into a ‘care home’ category like this is meaningful to social care. Within social care, the term care home does not incorporate these wider settings.

2. Presentation within the infographic

There is no explanation of the major limitations of these statistics on the infographic provided by PHE and we think that some of the wording and presentation of the data needs to be amended so it is not misleading. Given the way it has been calculated, it will become more misleading as the pandemic progresses and less useful.

It would be easy for someone looking at these statistics (bottom graph) to come to the conclusion that there were 111 outbreaks in care homes during the week of the 25th May 2020. When in fact, the actual figure is somewhere between 111 and 6,225 if there are some ongoing in some care settings and there are new outbreaks in care homes that have already had one.

We think that the wording about the ‘weekly suspected or confirmed COVID-19 outbreaks in care homes’ needs to be amended to show they are new outbreaks and in homes that have previously not had a reported outbreak (or may still be having one) and may include other care settings too.

We have discussed with PHE that they are looking at ways to start calculating the beginning and the end of an outbreak using the data, and this would clearly be a better way of describing the impact of COVID-19 on care homes or settings, and enabling areas to identify whether they have care homes or  settings with repeated outbreaks which may require further interventions and support. We would be supportive of this but only if PHE can provide accurate and meaningful data.

3. Meta data and data tables

In line with comments above on the infographic, we think the meta data and data tables also need to be clearer on all of these points.

As we have said above, the production of this management information has caused us a lot of work locally because in our view, there has been insufficient thought about how the data are presented, its limitations, and understanding of social care settings. We have been in dialogue with PHE about this for three weeks, and while we are pleased that they have now corrected the Camden figures, we are not clear whether they accept our points above and will be modifying their outputs in line with this. Given the amount of mistrust this has all caused in statistics locally, we would be grateful if the regulator could look into this and take a view to prevent this happening again in the future.

Yours faithfully

 

Jess McGregor, Director of Adult Social Care

Julie Billett, Director of Public Health

 

Related Links

Ed Humpherson response to Jess McGregor and Julie Billett

Ed Humpherson response to Jess McGregor and Julie Billett: COVID-19: number of outbreaks in care homes – management information published by Public Health England

Dear Ms Billett and Ms McGregor 

COVID-19: number of outbreaks in care homes – management information published by Public Health England 1 

Thank you very much for your letter of 24th June to the UK Statistics Authority. My team at the Office for Statistics Regulation have considered the matter in detail and talked extensively with the team responsible for the statistics at Public Health England (PHE). 

We agree with your concerns and, whilst the information presented has been compiled at speed with the intention of transparency, there are a lot of important details about the types of setting that have been lost.  

The dataset is released with granular information at the local authority level, which would be of great use to stakeholders; however, the misleading use of the term “care home” in this context gives the data less utility than it should have. The data is sourced by PHE from all supported living facilities and it should be described as such, to make most sense for the experienced users. We have suggested that PHE amend the title of the release to better reflect this. 

We agree there is an erroneous calculation of proportions, using mismatching numerator and denominator data. The numerator and denominator are not comparable, because non-Care Quality Commission (CQC) registered care settings are included in the numerator, but not the denominator. This fact is noted in the metadata on the landing page and in the data tables. The use of a denominator that underestimates the total number of facilities at risk of suspected or confirmed outbreaks, leads to over-inflation of the proportion of facilities with a suspected or confirmed outbreak. My team has explored with PHE the use of the small denominator and they are considering how to improve the calculation with the data they currently collect. 

There is no explanation of the major limitations of these statistics on the infographic provided by PHE. The infographic includes the proportion of outbreaks and the number of care homes for each PHE centre but, as noted above, this denominator does not include all settings at risk captured in the numerator. PHE is considering removing the proportion numerical and replacing it with an indicator calculated using comparable numerator and denominator. 

We agree with your concern that the presentation as it stands makes it difficult to identify whether a region has supported living facilities with repeated outbreaks which may require further interventions and support. My team has explored with PHE what improvements are possible to the data and presentation to make the information more useful for a range of stakeholders. PHE assures us that it has identified more data that it can use to present a more comprehensive publication and that this will be released later in 2020. In the interim, we have asked PHE to engage publicly with stakeholders to let them know about proposals to change the current publication and obtain stakeholders views on the content, timing and presentation of the new material. 

We agree with your third concern and have asked PHE to improve the information about the publication itself (the metadata), by publishing details of the data sources, the methodology, the quality assurance processes and the strengths and limitations of the data. This will help a range of stakeholders to decide if the publication is the right material for their needs. 

Thank you very much for drawing this to our attention and will closely follow PHE’s progress with this work. 

I am copying this letter to Clare Griffiths, Head of Profession for Statistics at PHE and Julia Verne, Head of Clinical Epidemiology at PHE. 

Yours sincerely 

 

 Ed Humpherson 

Director General for Regulation 

 

1 COVID-19: number of outbreaks in care homes – management information

 

Related Links

Jess McGregor and Julie Billett to Ed Humpherson: Public Health England Care Home Management Information