Dear Scott
Rapid Review of weekly COVID-19 and Winter Statistical Report
As you are aware, we recently undertook a rapid review of the weekly Public Health Scotland COVID-19 and Winter Statistical Report. Data on COVID-19 are of high media and public interest and it is important for the information to be easily found and understood, to minimise misuse. Given the evolving nature of this report, any specific comments about presentation are based on a review of the publication on 19 January 2022. This letter summarises our findings against compliance with the Code of Practice for Statistics.
Value
Being agile in your response to users’ needs throughout the pandemic has been beneficial to users and something we commend. Your statisticians have been flexible and have updated the content of the report over time to meet changing user needs during the pandemic – for example, the addition of Test and Protect statistics and the more recent addition of hospitalisations ‘because of’ or ‘with’ COVID-19.
The main report is quite lengthy, but ensures that all the data are in one place, making it easy for users to know where they need to go for information. It is good that you present the data in a range of different ways to suit a wide range of audience needs – an interactive open data dashboard, an HTML webpage and a PDF summary along with the main report. It is also good that you use other means to make key messages more accessible to users, such as by writing your own blog, as well as linking to other countries’ data and research.
It was good that you made users aware in advance of changes to the reports by updates on Twitter and the website. For example, you tweeted on 8 December when changes were being made to the Weekly COVID-19 report to make it a Winter Weekly report. Analysts explained to users what changes were occurring and this was an engaging use of social media to inform the public.
You told us that you intend to carry out more engagement with stakeholders to understand their needs for data in the future. With an increasing amount of possible data to publish, it will be important to keep on top of users’ needs and find out how users access the information you present and whether there are any changes to the timing, format or layout of the information that would be helpful for them, and for you to take appropriate action in response.
Quality
We noted good use of 95% confidence intervals in Tables 14, 15 and 16 as well as error bands in Figures 13 and 15. Explanations around smaller populations having a wider confidence interval than larger populations are particularly useful to non-expert users. Renaming and providing a link to the excellent metadata spreadsheet would be helpful in this regard, as it would clarify that it is full of helpful information about the quality of the data.
Examples of potential biases and issues that need to be considered when using the data are explained and help to limit the risk of misinterpretation. The report contains several well-explained caveats – for example, on the peak in figure 15. The paragraphs headed “Interpretation of data” are very well highlighted in the “COVID-19 Cases, Hospitalisations, and Deaths by Vaccine Status” section of the report. This helps to not only reduce the risk of misuse of statistics, but also provides users extra insight into the data.
However, the summary report, webpage and dashboard do not mention any potential uncertainties in the data, which carries the risk of misuse, especially over-interpretation. We consider that more needs to be done in these outputs to make the uncertainty clearer. This could involve including prominent explanations of uncertainty and caveats around the statistics. You have told us that the statistics team will provide more explanation of data uncertainties in future publications.
Whilst the methodologies used to calculate the statistics are explained in each section, nothing about the quality assurance arrangements is published. You should publish an explanation of your assurances on data quality, taking into account the Authority’s Quality Assurance of Administrative Data (QAAD) toolkit.
Trustworthiness
Public confidence in the statistics is being promoted by the fact that these comprehensive reports are released in an open and transparent manner every Wednesday and that revisions are clearly explained.
We appreciate the efforts you and your team have put into this publication and the adaptability shown throughout the course of the pandemic. I am copying this letter to Duncan Buchanan, Head of Service, Public Health Scotland.
Please do not hesitate to get in touch if you would like to discuss any aspects of this letter.
Yours sincerely
Ed Humpherson
Director General for Regulation