Scott Heald response to Ed Humpherson: Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

Dear Ed

Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

 Thank you for your letter of 14 January 2021 relating to the above report.  As you know, Public Health Scotland is building on the good practice in the publication of statistics from ISD Scotland, one of its constituent parts, and I’m pleased that you have recognised the team’s efforts to present the information in the report in line with the Code of Practice for Statistics.   This report was produced by experts from within Public Health Scotland and Universities of Edinburgh and Glasgow.

Your specific pointers on the complex statistics presented in tables 10 and 11 are helpful.  We recognise that the analysis undertaken in section 2 of the report is complex and therefore may be confusing for some readers.  Given this complexity and the high profile nature of this report, the report team were involved in a number of briefings for key stakeholders, including the media, to walk them through our results and to give the opportunity to ask questions.  We have also received direct feedback from users of the report too.  Our aim is to be as open and transparent about the analysis and conclusions as possible, and your comments and separate feedback from users (through the briefing sessions and direct correspondence) will enable us to provide even greater clarity for readers in interpreting the statistics in subsequent iterations of the report.

The team are currently working on further analyses which we intend to publish as an update to the existing report – we will pre-announce publication dates in due course.  At the same time, we will take the opportunity to update the existing sections in the report to take on board your comments and ensure all aspects of the analyses undertaken are explained in an understandable way for all users.

The team would also be happy to engage directly with the individuals who have contacted you so that we can talk through their points.

If you have any further questions about this report, please don’t hesitate to contact me.

Many thanks,

Scott

Scott Heald

Head of Profession for Statistics, Public Health Scotland.

 

Cc   Dr Jennifer Burton, University of Glasgow

Professor Bruce Guthrie, University of Edinburgh

Fiona Mackenzie, Public Health Scotland.

 

Related links:

Ed Humpherson to Scott Heald: Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

Ed Humpherson to Scott Heald: Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

Dear Scott

Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

On 28 October, Public Health Scotland (PHS) published the report ‘Discharges from NHSScotland hospitals to care homes between 1 March and 31 May 2020’. This was commissioned by the Cabinet Secretary for Health and Sport on 18 August and you collaborated with experts from the University of Glasgow and University of Edinburgh in the production of this report.

As you are already aware, we have received correspondence which includes a query around the presentation of the findings in the report. I would like to thank you and your team for your prompt and positive engagement with us on this.

This is a management information publication and we are encouraged to see that many of the principles of the Code of Practice for Statistics have been adopted by you in the publication of this report. For example, the publication date of the report was pre-announced via the PHS statistical release calendar, and a pre-release access list was published. A comprehensive methodology note was published which includes the data sources used, the level of quality assurance and data completeness. This is of particular importance, given the data challenges you faced for the research.

We understand that the methods used were based upon a similar study by Public Health Wales. We have reviewed the communication of findings in consultation with method experts from the Office for National Statistics (ONS). We consider the report is clear, thorough and transparent about the limitations and quality of the dataset and we recognise that the data and analyses undertaken were complex.

When reporting statistical analyses, the uncertainty of estimates and confidence intervals should be communicated in a measured and clear way. Whilst we understand that communicating complex statistical messaging can be challenging, we consider there are lessons to be learnt from this case and expect the improvement points (Annex) to be addressed in the presentation of any future statistical analysis of the care home data.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

Communicating uncertainty

Often in analyses and reports of this nature, the main challenge is how best to express the strength of evidence and how to communicate statistical uncertainty. Whilst we recognise the difficulties in expressing this clearly, we consider there are some sections of the report that could be confusing to readers.

The hazard ratios in tables 10 and 11 show that while there is a statistical relationship between hospital discharge and care home outbreak in the univariate analyses, this relationship ceases to be significant in the adjusted models. This is clearly explained in the report, but some of the discussion of the uncertainty around the estimates led to the report feeling a little inconsistent in its messaging. For example, on page 39, where the report outlines that the estimated risk of hospital discharge is not statistically significant, and then proceeds to detail a best estimate of risk figure. Greater clarity and consistency with explanations would assist the reader to understand the findings of the statistical modelling.

Specific feedback on analysis of associations between any hospital discharge and outbreak (table 10)

The adjusted hazard ratio when looking at discharge compared to no discharge is 1.21 with confidence intervals of 0.94-1.54. Although this is not statistically significant, the fact that the lower confidence interval is close to 1 means that this is marginal for this level of confidence. The section on interpreting table 10 states clearly and in bold that “hospital discharge was not statistically significantly associated with care home outbreaks (adjusted HR 1.21)”.

The conclusion section for this table then goes on to acknowledge that “the best estimate of the hazard ratio for hospital discharge is >1 and the confidence interval in the adjusted analysis is relatively wide. We therefore cannot statistically exclude the presence of a small risk from hospital discharge”. 

While it is good to see this discussion of uncertainty, this sentence feels quite technical and perhaps harder for a less experienced user to understand. It might also have been helpful to include this point in the section on interpreting table 10, alongside the statement of the non-significant finding, rather than in the conclusion. The way the information is presented in the report gives too much emphasis to the non-significant finding and not enough to the uncertainty. Presenting all of the information together (rather than under separate “interpreting table 10” and “conclusion” headers) would allow for a more balanced overall discussion of the statistical finding and the uncertainty around it.

Specific feedback on analysis of associations between different types of hospital discharge and outbreaks (table 11)

When looking at the different types of discharge, we see adjusted hazard ratios of 1.00 for tested negative, 1.27 for untested and 1.45 for tested positive. Although the confidence intervals again suggest these findings are not significant, the observed ‘dose-response’ pattern in the adjusted hazard ratios is consistent with a causal relationship between positivity and outbreak. Given the sensitivity of the care home setting during this pandemic, and the likely uses of the evidence from this analysis, some users may have benefited from additional discussion of this in the report.

Similar to table 10, the statement made in the section on interpreting the findings for table 11 is that none of the differences are statistically significant. The conclusion for table 11 then goes on to state: “The analysis does not find statistical evidence that hospital discharges of any kind were associated with care home outbreaks. However, our certainty about the three types of hospital discharge defined by testing status varies.”

The first sentence could be read as not finding evidence for each and all of the types of discharge, but that is inconsistent with the second sentence.

The conclusion section for table 11 then goes on to provide some explanation of the uncertainty by stating that there may be a small risk where a person was untested and a moderate to large risk where they tested positive. This explanation was clearer and easier to understand than that for table 10. Again though, it would have been clearer if both the non-significant finding and the uncertainty around it were discussed in the section on interpreting the findings, rather than under separate headings.

Confidence intervals

Some of the confidence intervals around the estimates were very wide. In the report, there is some explanation around the uncertainty of wide confidence intervals and how caution is needed in interpreting them. In the section on table 11, however, you draw attention to the upper bound of a confidence interval of 374% and point to this as evidence of an association between a positive test and an outbreak. This seems more of a reflection of the sample size and the lack of statistical power – having less data would tend to make the upper bound of the confidence interval higher, but would not mean that the evidence was stronger. Given the importance of communicating uncertainty clearly, we encourage you to be careful in your discussion of confidence intervals and be clear where these indicate that we should be less confident overall in the robustness of the findings.

Impact of methodological choice on findings

A clear rationale is provided in the report as to why the estimates need to be adjusted for care home size: “Larger homes will receive more discharges, and are also more likely to be services for older people, provide nursing care and be privately owned. These relationships between different care home characteristics means that simple comparison by single characteristics may be misleading.”

However, it is possible that there may be collinearity between hospital discharges and care home size, and the degree to which the model estimates may be affected by this was not covered in the report. It would be helpful for users to have more detailed explanation of how the possible relationship between discharges and care home size might have impacted on the findings.

Use of charts

The report makes good use of tables to display both univariate and adjusted hazard ratios along with their associated confidence intervals. This is very helpful and clear but of more use to a more experienced reader. Using charts in addition to the tables would enable you to present the information in a more visual way that might be easier for less expert users to understand.

Links to further helpful information

The following resources available on the GSS website may be of interest for any future presentation of the care home data.

The Good Practice Team will also carry out reviews of statistical publications. Their page on communicating statistics on the GSS website provides further information on this.

 

Related links:

Scott Heald response to Ed Humpherson: Presentation of findings from the Discharges from NHSScotland hospitals to care homes report

 

Scott Heald and Roger Halliday response to Ed Humpherson: The use of unpublished statistics in Scottish Government news release

Dear Ed,

The use of unpublished statistics in Scottish Government press release

Thank you for your letter of 8 July relating to the use of an unpublished statistics on routine serology (antibody) testing in a Scottish Government news release on 23rd June. This was an oversight, for which we apologise. As you are aware, there is very high demand for statistics relating to COVID-19 and the statistical teams in Public Health Scotland and Scottish Government are producing a large volume of statistics every day. On this occasion, the fact this statistic had not been published was missed by the teams in both organisations.

Scottish Government and Public Health Scotland have been working closely during the COVID19 period to ensure we are presenting as complete and coherent a picture on COVID-19 in Scotland as early as possible. We do recognise the importance of the statistics on antibody testing and the need to get these into the public domain. These data are currently still in development but, given the public focus on these statistics, Public Health Scotland will release an initial adhoc report on antibody testing on Wednesday 15th July. This will be published alongside the Public Health Scotland weekly COVID-19 report.

If you have any further questions, please don’t hesitate to contact us.

Scott Heald & Roger Halliday

 

Related links

Ed Humpherson to Scott Heald and Roger Halliday: The use of unpublished statistics in Scottish Government news release

Ed Humpherson to Scott Heald and Roger Halliday: The use of unpublished statistics in Scottish Government news release

Dear Scott and Roger

The use of unpublished statistics in Scottish Government news release

On 23 June, a news release ‘COVID-19 antibody testing’ was published on the Scottish Government website detailing a letter written to NHS boards from the Chief Medical Officer Dr Gregor Smith about the use of serology testing for COVID-19 in Scotland. Within the news release, it reported that ‘so far, 4,431 antibody tests, for surveillance purposes, have been completed’. There is no reference to the data source of the figure quoted in the news release.

To date, there has been no publication of routine serology (antibody) testing in Scotland. As such, this figure cannot be verified. This is unacceptable for a figure of such importance used in a government news release.

We understand that wider plans are underway around the data collection and publication of serology information in Scotland. Nevertheless, our expectation, as set out in our public statement of 18 May, is that any data used publicly by Government should be published in an accessible form, with appropriate explanations of context and sources. Antibody testing data are of particular interest to the public and to ensure public confidence and equality of access, we urge you to publish the data quoted in the Scottish Government news release.

Yours sincerely

Ed Humpherson

Director General for Regulation

 

Related Links

Scott Heald and Roger Halliday response to Ed Humpherson: The use of unpublished statistics in Scottish Government news release

Scott Heald to Ed Humpherson: Publication of the first COVID-19 statistics weekly report by Public Health Scotland

Dear Ed

Publication of the first COVID-19 statistics weekly report by Public Health Scotland

Further to my letter of 23 April 2020, I wanted to update you on progress with our new weekly report on COVID-19 statistics in Scotland.  The first report was released on 6 May 2020.  This is a week later than originally intended to allow the team to undertake further quality assurance on the data presented and to ensure, where appropriate, that we maintained consistency with the figures published each day by Scottish Government.

The new report, accompanying dashboard and open data can be accessed via the Public Health Scotland website at:

https://www.publichealthscotland.scot/our-areas-of-work/sharing-our-data-and-intelligence/coronavirus-covid-19-data/

The content of the report and dashboard will be enhanced over the coming weeks with a range of additional statistics, some of which will be published weekly; others will be published as one-off “spotlights”.  We will outline our plans for this in advance so that users can understand our forward timetable.

In order to present a coherent picture on COVID-19 statistics in Scotland, we have aligned the timing of our weekly report with the weekly report on deaths which is published by National Records for Scotland (NRS).  I originally thought the NRS report was published at 2pm but that was a misunderstanding on my part – their report is published at 12 noon each Wednesday and I made the decision on Tuesday (5 May) to align the timing of the Public Health Scotland report, in the spirit of ensuring that both are published at the same time – we pre-announced this timing change on our website and notified key users of the change.

The Public Health Scotland report contains a link to the NRS website so that users can quickly access the weekly deaths report, and to the Scottish Government website so that users can access the daily data.

I’d be very happy to speak with you, or one of your team, about our new report.  Please let me know if you have any questions or comments.

Best wishes,

Scott Heald
Head of Profession for Statistics

Response from Ed Humpherson to Scott Heald: Publication of COVID-19 related statistics by Public Health Scotland

Dear Scott,

Thank you for your letter today. I am happy to confirm the exemption from the Code of Practice for Statistics’ standard publication time of 9.30am, to permit a later release time of 2.00pm each Wednesday for the PHS’s weekly COVID-19 report, starting on Wednesday 29 April.

Given the ongoing COVID-19 pandemic I welcome your endeavours to publish this release in line with the NRS weekly death release. The exemption from 9.30am publication, and publication at 2.00pm, aligns with other COVID-19 related statistics in Scotland. Your collaborative effort with other statistics producers in Scotland in providing a coherent picture of COVID-19 data to users is commendable and I appreciate your transparency around your forthcoming plans.

I also welcome your commitment to review and keep users informed publicly of any changes to your official statistics outputs during this period.

Yours sincerely

Ed Humpherson

Director General for Regulation

Related links:

Scott Heald to Ed Humpherson: Publication of COVID-19 related statistics by Public Health Scotland

Scott Heald to Ed Humpherson: Publication of COVID-19 related statistics by Public Health Scotland

Dear Ed

Publication of COVID-19 related statistics by Public Health Scotland

Thank you for your letter of 1st April 2020 welcoming the launch of Public Health Scotland.  As you indicated, we have launched in unprecedented times and Public Health Scotland is continuing the work being led by Health Protection Scotland, ISD Scotland and NHS Health Scotland to support the COVID-19 effort in Scotland.

We are currently working with key stakeholders, including Scottish Government and NHS Boards, to review our official statistics outputs during this period and we plan to publicise this on the Public Health Scotland next week (week commencing 27 April 2020).  Key statistical series will continue where we can but some will be paused due to a combination of data and staff availability to undertake the work in NHS Boards and Public Health Scotland.  We will keep users informed of our plans through-out the COVID-19 period.

In addition, we plan to release a weekly report on COVID-19 (starting on Wednesday 29 April), bringing together a range of statistics to provide a weekly summary for users.  We will be working with other statistical producers in Scotland, in particular Scottish Government and National Records for Scotland, to ensure we collectively present a coherent picture on COVID-19 for users. We would therefore like to publish this weekly bulletin at 2.00pm each Wednesday, at the same time NRS publishes its weekly report on death statistics. Accordingly, I would be grateful if you could indicate your support for Public Health Scotland moving the release time for the weekly report to 2.00pm.  I appreciate that this represents a departure from the requirements of the Code of Practice for Statistics, but consider it prudent to align with other COVID-19 related statistics in Scotland.

Public Health Scotland also provides a range of management information to Scottish Government for its daily reporting on COVID-19 statistics.  To complement the figures released by Scottish Government, Public Health Scotland has developed an interactive dashboard which gives a more visual presentation of the statistics.  The dashboard is currently in beta format (and can also be accessed via the Scottish Government’s daily reporting website) and we will enhance the content over time based on user feedback.   We will pre-announce any changes to the dashboard content so that users are aware of our plans in advance.

Please let me know if you have any questions about our plans.  Thank you.

Best wishes,

Scott Heald
Head of Profession for Statistics

Public Health Scotland Launch

Dear Scott

LAUNCH OF PUBLIC HEALTH SCOTLAND, APRIL 2020

I am writing to mark and welcome the official launch of Public Health Scotland today. As the largest producer of official statistics about health services and population health in Scotland, you have been faced with preparing for this day under the most challenging set of circumstances imaginable. You will no doubt have difficult decisions to make in the coming weeks and months as the impact of the pandemic on your statistical releases unfolds. We have published guidance to support all producers in these circumstances and we aim to be very flexible in our regulatory approach in light of these exceptional and difficult challenges.

We met with you and Angela Leitch, Public Health Scotland’s Chief Executive, last month to discuss your long-term plans for statistics production. Public Health Scotland’s role as an official statistics producer is clearly understood and well supported at a senior level in the organisation. The governance arrangements that were in place in ISD to ensure the trustworthiness, quality and value of statistics continue in the new organisation and your role and responsibilities as Head of Profession are well respected.

I would like to extend my best wishes to you and all your colleagues, old and new as you adapt to both your new organisation and the situation you are now facing. We look forward to working with you all.

Yours sincerely

Ed Humpherson

Director General for Regulation