Dear Lucy
Review of NHS Test and Trace (England) and NHS COVID-19 app statistics
As you are aware, we recently undertook a review of the NHS Test and Trace (England) statistics and NHS COVID-19 app statistics published by the Department of Health and Social Care (DHSC). A summary of our findings and recommendations is provided below and set out in detail in the annexes to this letter. More-detailed feedback has also been provided to your teams directly.
Test and trace statistics are vital to support public understanding of the pandemic and the government response to it. It is clear that your teams have worked hard to develop these statistics since our last review. The improvements we have seen help to reflect the issues of most importance to the public.
However, we remain concerned that there are several important questions which cannot be answered by the statistics. DHSC currently publishes two outputs about test and trace, one using data from the NHS Test and Trace system and one using data from the NHS COVID-19 app. Both of these provide useful metrics about operational delivery, but they leave a gap in answering broader questions – particularly those about the effectiveness of the programme. It is important that DHSC prioritises filling these gaps, whether in these publications or in a separate release. Where it is not possible to do this using data from the Test and Trace system or the app, DHSC should work with other organisations to ensure key questions can be answered as soon as possible.
Background
In July 2020 the Office for Statistic Regulation (OSR) carried out a rapid review of the early statistics published about the NHS Test and Trace programme in England. We welcomed the pace of development and your commitment to continually improving the statistics in line with the Code of Practice for Statistics.
The review was clear that it is essential to understand how effective the Test and Trace programme is. It identified important questions which could not be answered with the information available at that time, such as: What is the end-to-end time from an individual first experiencing symptoms to having their contacts advised to isolate? And how many people who are asked complete self-isolation? We asked DHSC to prioritise the provision of information which would help users understand the effectiveness of the programme in future.
Since our initial review, DHSC has made great progress in expanding its Test and Trace statistics to include additional data and has started to publish statistics about the NHS COVID-19 app. Given these developments and the high level of public interest in these data due to the current and planned changes to self-isolation rules in England, we agreed with your team that we would review the Test and Trace statistics again, this time including the new app statistics.
Summary of our findings
There have been many positive developments in the Test and Trace (England) statistics since our last review. These include:
- the addition of new data, for example on rapid asymptomatic testing, testing of overseas arrivals to the UK and the managed quarantine service
- developing the main reports, for example improving the charts and tables, and adding links to supporting documents and other relevant data sources
- expanding the published quality and methodology information, for example adding data limitations to the methodology note
- sharing improvement plans with users
We are pleased to see that some of the questions about the programme can now be answered, such as the time taken to reach close contacts following a positive case first observing symptoms. However, there are still gaps in measuring effectiveness. For example, there is little information about how many people who are asked to self-isolate complete it. There are also no statistics to allow users to understand what the vaccination status is of those who test positive or of close contacts. While Public Health England (PHE) publishes research on positive tests among contacts of a case identified with a variant of concern or interest, it is not clear how many close contacts identified through the tracing system go on to test positive in total. We are pleased to hear that DHSC is currently assessing the quality of data on vaccination status, with a view to adding this information to the statistics in future, and that it is working on other analysis which will contribute to understanding the effectiveness of the programme. Where it is not possible to answer some of these questions using operational data, DHSC should look at alternative sources of information and work with other organisations to ensure key questions can be answered.
The publication of management information about the NHS COVID-19 app supports transparency. These data are used by DHSC to understand use of the app and support its development. However, there are significant limitations in the quality of the data, which impact how the data can be used. We consider that the published information about quality needs to be improved so that users are clear about the caveats and understand how to use the data appropriately. For example, due to anonymity for users of the app, it is not possible to combine the figures with those from the Test and Trace programme to obtain the total number of people self-isolating – this is an important question which should be answered. Improving the coherence between the Test and Trace statistics and the app statistics will help users understand how they relate to each other and how they should be used. There are also some improvements to be made in terms of orderly release, publishing development plans and providing contact details so users know who to speak to if they have questions or feedback. To aid public confidence in the statistics, we ask that you support the team responsible for them to apply relevant principles of the Code of Practice, as you have done for the Test and Trace statistics.
Given that it may require collaboration with other organisations to answer some of the key questions we have highlighted, I have copied this letter to: Clare Griffiths, Head of Profession for Statistics, PHE; Mark Svenson, Head of Operational Information and Head of Profession for Statistics, NHS England and Improvement; and Emma Rourke, Director of Health Analysis and Pandemic Insight, Office for National Statistics.
We would like to thank your teams for their positive engagement with us throughout this review. We look forward to hearing from you as you continue to develop these statistics.
Yours sincerely
Ed Humpherson
Director General for Regulation
Trustworthiness
Strengths
DHSC publishes a statement of compliance, committing to the principles and practices within the Code of Practice for Statistics. This helps users understand how the data are used in advance of publication, for example by ministers and officials to support operational management.
It is also good to see information about planned developments included in the main reports.
Since our previous review, DHSC has moved the release of the statistics to better align with policy decisions on international travel. This helps to support the public’s understanding of data informing such decisions.
Weaknesses
–
Recommendations
DHSC should continue to publish further information about ambitions and plans for the statistics, even if these plans are tentative and later change or if timescales are unknown.
Quality
Strengths
The methodology note accompanying the statistics contains a wealth of useful information for users. This includes explanations about:
- why some definitions and methods have changed over time (for example due to changes in policy)
- why some data cannot be published (for example the positivity rate for lateral flow device tests due to potential bias)
- how the figures can and cannot be used and how they differ to other data sources, both within the publication and elsewhere, including helpful links where relevant
It is helpful that important limitations which users should be aware of are highlighted in footnotes in the main reports, for example information about duplicate records of close contacts. The main Test and Trace report also contains clear information about revisions. This includes the reasons behind revisions and the impact of this on how the statistics should be used.
Weaknesses
There is limited information in the methodology note about data sources and arrangements with data suppliers.
It could be clearer how the data in the Test and Trace statistics relate to the NHS COVID-19 app data. While links to similar statistics published in other UK countries are included, there is limited guidance on whether these can be compared.
While the main reports do contain some information about specific limitations, the general information about strengths and limitations advises users to treat the data “with caution” due to its developmental nature. This does not advise users on what action to take as a result of the limitations.
There is limited information about the quality assurance undertaken by DHSC. This means that it is hard for users to understand the level of quality of the statistics and how DHSC assures itself that it is sufficient for its intended purposes.
Recommendations
DHSC should publish information about data sources and arrangements with data suppliers. DHSC should ensure that it is clear to users how the statistics relate to the NHS COVID-19 app data and improve guidance about how the statistics can be compared with similar statistics in other UK countries.
DHSC should support people to use the statistics appropriately, by providing advice in the main reports on what can and cannot be done with the data given its quality and limitations.
To reassure users about the quality of these statistics, DHSC should publish more information about its approach to quality assurance. This should cover quality assurance arrangements and how they reduce the risk of errors.
Value
Strengths
DHSC undertakes a range of user engagement activities to continually develop its understanding of user needs. This includes reviewing user requests, fortnightly meetings with an expert group from the Royal Statistical Society and a user survey carried out in March 2021.
We welcome the additional data and insights in the main reports and data tables since our last review – for example, information about:
- rapid asymptomatic testing
- international travel
- managed quarantine service
- breakdowns by geographies and demographic characteristics
We also welcome the improvements to the presentation of the statistics in the main reports, which includes:
- information about updates to the publication in the Main Points section
- improved charts and tables
- the use of clearer, more-accessible language, supported by a comprehensive glossary
- contextual information about relevant policy changes, for example for the managed quarantine service
The main reports are long, so we are pleased to hear that you are currently reviewing the content, with a view to removing less relevant information as well as adding new data. It is important that areas of focus for the bulletins in future are those which are most relevant to current policy decisions and public debate.
Weaknesses
There are many important questions which the statistics are not yet able to answer. For example:
- How many people who test positive complete self-isolation as requested?
- How many positive tests are reinfections?
- What is the vaccination status of people who test positive?
- How many close contacts go on to test positive?
- What is the rate of infection among close contacts?
- How many close contacts in total are asked to self-isolate (from both the tracing system and the app)?
Recommendations
DHSC should consider publishing the results of its user engagement survey, and any future surveys. This would aid transparency and provide an opportunity to let users know how their feedback is addressed.
During its review of the main reports, DHSC should ensure that the information included is the most relevant based on current policy decisions and public debate. Questions which support understanding of the effectiveness of the Test and Trace programme should be prioritised – particularly considering upcoming changes in policy. For example, it will be important to understand the results from daily tests used instead of self-isolation for some groups. Where some of these questions cannot be answered using data from the Test and Trace programme, DHSC should be open about this with users and work with others to address these gaps where possible.
As well as reviewing whether some information included in the main report is no longer as relevant, navigation of the long bulletins would be aided by the inclusion of a contents page.
Trustworthiness
Strengths
Publication of the NHS COVID-19 app data enhances transparency about the app and its use with the public. We are pleased to see the strong collaboration between policy and analytical colleagues to produce and develop these statistics.
The figures are updated weekly. This appears to be in line with the DHSC weekly Test and Trace publication.
Weaknesses
A publication schedule of the data is not available.
There are no details about whether the data are subject to revisions or corrections.
There is no contact information for the lead statistician or analyst responsible for production of the statistics.
Recommendations
With support from the Head of Profession for Statistics, the team should seek to improve public confidence in the statistics by applying relevant principles of the Code of Practice, as has been done for the Test and Trace statistics. This should include clearer publication of a release schedule, information about revisions and corrections, and contact details for the lead statistician or analyst.
Quality
Strengths
The ‘About the data’ and ‘Notes on the data’ sections provide useful information about the data, some of its limitations and the broader context that the data fit into.
Weaknesses
The information from the app relies on self-reporting, is voluntary and all data are anonymous, therefore the reliability and robustness of the data is questionable. Data quality information should be more-detailed to support users to understand this fully – for example, explaining the reason for ‘orphan’ contacts (users identified as a close contact for whom a ‘parent’ case is not included in the data).
There is no information about quality assurance and how DHSC verify and check the data from the data suppliers (Google and Apple).
Recommendations
DHSC should improve its understanding of the app data quality and its communication of this information to users. This should include guidance about what the data can and cannot be used for.
Value
Strengths
The data are equally available to all and include some statistics at local authority level for users to interrogate. The data are also available to download in other formats.
We welcome the plans that the team told us about to develop the statistics further by publishing additional metrics in the future, such as the number of users who still have the app installed and how many have contact tracing enabled.
Weaknesses
There is a lack of contextual information accompanying the statistics. For example, it is unclear what is categorised as a ‘venue’, whether all venues are expected to have a check-in poster and whether there are other ways that visitors can check-in at a venue that are not captured in these data.
The publication does not include any statistical commentary and there is limited insight about what the data mean. Whilst it is helpful to have the link to the NHS Test and Trace publication, there could be more information about how the two sources relate to one another and where there are overlaps.
Recommendations
DHSC should review the presentation of the statistics, focusing on:
- explaining the purpose of the data and what they are used for
- expanding the contextual information and definitions to improve user understanding
- providing insights for users about what the data tell us
- making clear how the app data relate to the NHS Test and Trace publication
DHSC should also publish its plans for future development of these statistics.