Dear Stephen
NHS TEST AND TRACE STATISTICS (ENGLAND)
Thank you for inviting my team to review your NHS Test and Trace statistics (England) publication. The pace of development of the programme and associated management information has been unprecedented. It is clear that providing these timely statistics is the result of a huge amount of work by individuals across a range of teams and organisations. We welcome the publication of these important data in an orderly release which demonstrates a commitment to the Code of Practice for Statistics.
Our rapid review has focused on the extent to which the statistics are produced and published in line with the expectations set out in the Code of Practice for Statistics. It is based primarily on publicly available information, supported by discussion with you and your team. It has not included a detailed investigation of the statistical methods or quality assurance processes supporting production of these statistics.
We appreciate the openness with which you and your team have engaged with the Office for Statistics Regulation (OSR) and your clear desire to constantly make improvements to the information available. Our review identifies areas we consider priorities for improvement over coming months.
Statistics on people tested for COVID-19 and subsequent contact tracing for those who test positive are essential for several purposes:
- to support understanding of the pandemic;
- to manage the test and trace programme;
- to inform the public about the implementation and effectiveness of the programme and enable them to hold government to account.
Without greater clarity on the purposes of this publication and clearer information on how data in this publication fit with other statistics or research outputs, this publication may not serve any of these three aims as well as it should. There are some key questions which the publication is not yet able to answer, such as the impact the programme has on reducing the spread of COVID-19. It is likely that the questions identified cannot all be answered solely through management information from the NHS Test and Trace Programme, but it is important that government seeks to better understand the effectiveness of the programme and its impact on the pandemic outcomes.
A summary of our findings is set out below, with further information provided as Annex A to this letter. We have also provided more detailed feedback to you directly.
Value
The rapid development of these statistics and the improvements that have already been made in the five weeks since the first release should be commended. This first step in publishing information about testing and tracing is an approach that other countries within the UK can look to as they develop their own statistics.
The timeliness of statistics on the NHS Test and Trace Programme is an important part of their value but comes with inevitable trade-offs. More context and greater clarity on the purpose of the publication would enhance the value of these statistics.
A key purpose of the NHS Test and Trace (England) statistics must be to understand how effective the Test and Trace programme is. There are several important questions about the effectiveness of the programme that cannot be answered with the information currently available. In seeking to improve the publication we consider you should prioritise the provision of information which would help to answer these questions. For example: What proportion of those with COVID-19 are not covered by these statistics, perhaps because they are asymptomatic or have symptoms but do not choose to book a test? What is the journey time for an individual from experiencing symptoms to having their contacts advised to isolate? What is the impact of this time lag on the ability to reduce transmission of the virus? And what proportion of people asked go on to self-isolate? Further important questions we think the publication should seek to answer are outlined in Annex A.
There is a range of data on COVID-19 which has been rapidly developed and published. This leads to a confusing landscape. There needs to be greater clarity on the purpose of each publication and how they fit together. The Department of Health and Social Care should take a lead in helping individuals to navigate these data. Signposting readers to outputs that are likely to be of interest would also support navigation across different datasets. For example, the work you have done to improve the data on testing in the wider population (Pillar 2 data) has recently enabled extremely valuable data on the number of positive cases resulting from Pillar 1 and 2 tests by local authority to be added to the Coronavirus Beta Dashboard. It would be helpful to link to these data in the publication (as well as the methodology note which currently includes a link). It would also be helpful to link to the gov.uk Coronavirus Statistics and Analysis page.
You explained your approach to iteratively developing the publication in order to support timeliness and transparency, taking into account data availability and your assessment of the quality of these data. This means the content of the publication is constantly evolving. To maximise the value of the data in the bulletin and ensure it can be readily understood by the public, you should review the language used and continue your development of the ‘Main points’ section. A visualisation of the numbers as they flow through the system would support understanding of the end to end process. Two examples of ways other organisations have achieved this are provided in Annex B to this letter.
Quality
You have published a methodology note containing definitions for the many technical terms used throughout the bulletin, as well as a clear explanation of the complex test and trace process. This aids understanding of the statistics.
It would be helpful to provide further information to ensure data are used appropriately and limitations are well understood. For example, an indication of the scale of duplicates in the cases transferred to the contact tracing system would support users in understanding if the data are fit for their particular purpose.
To reassure users about the quality of these statistics, we encourage you to publish more information about your approach to quality assurance in line with the Government Statistical Service guidance on urgent quality assurance of data. This should cover quality assurance arrangements and how they reduce the risk of errors.
Trustworthiness
The publication of these statistics is well supported by publication of a Statement of compliance with the Code of Practice for Statistics. For example, this document clearly states that decisions about which data are published are taken independently and based on statistical judgments.
We are pleased to see that the statistics are being published on a regular weekly basis and would encourage the department to more formally pre-announce future publication dates.
You have also demonstrated good practice in your handling of revisions, which are clearly signposted and explained. In future releases we would like to see an indication of the size of revisions, so that users can understand their impact.
As set out in our rapid review guidance you can include a statement in your methodology note such as “These statistics have been produced quickly in response to developing world events. The Office for Statistics Regulation, on behalf of the UK Statistics Authority, has reviewed them against several key aspects of the Code of Practice for Statistics and regards them as consistent with the Code’s pillars of Trustworthiness, Quality and Value.”
We look forward to seeing these statistics continue to develop.
Yours sincerely
Ed Humpherson
Director General for Regulation