Dear Chris
We have independently reviewed the actions that NHS England has taken to address the six requirements outlined in our assessment report. On behalf of the Board of the UK Statistics Authority, I am pleased to confirm that they comply with the standards of Trustworthiness, Quality and Value in the Code of Practice for Statistics and should be labelled as accredited official statistics. The detail supporting our judgement is included in the annex to this letter.
The performance of the National Health Service in England is a hugely topical issue, often covered in public and political discourse. Individuals and organisations want to hold the UK Government to account against its commitments, understand what level of service can be expected in local areas and understand how performance compares against other areas and countries.
We welcome the actions you have taken to improve the value of these statistics for users. The publication of a development plan for the statistics and a dedicated data quality page increases transparency, supports users to understand and use the statistics appropriately, and allows users to feed into development plans. We also welcome the hard work the statistics team has put into expanding its user engagement and responding to user requests for more-granular data. These activities demonstrate your willingness to listen to a range of different users and address their needs. Your ongoing development work, for example exploring the feasibility of publishing site-level data, demonstrates your commitment to meeting user needs and upholding the standards of the Code of Practice for Statistics. We look forward to hearing about the outcome from this work.
I would like to thank your team for its open and constructive engagement throughout our assessment and for your ongoing commitment to improving these statistics.
Yours sincerely
Ed Humpherson
Director General for OSR
Related links:
Review of actions taken in response to Assessment Report 382, Accident and Emergency Statistics produced by NHS England
Requirement 1: NHS England should review and reduce its pre-release access lists, bearing in mind the principles of good pre-release practice, to support the principle of equality of access and reduce the risk of data leaks. As part of meeting this requirement, NHSE should share its justification for each requirement with OSR.
Actions taken by NHS England: The NHS England team reviews the pre-release access list on a regular basis and reduces the list when possible. The team provided OSR with justifications for all those included on the list. The statistics teams and Heads of Profession for both NHS England and the Department for Health and Social Care (DHSC) work closely with their respective communications teams and private offices to ensure that colleagues in these teams understand their obligations under the Code of Practice and pre-release access. Clear wording is included in the title and content of emails sharing pre-release materials to ensure appropriate use.
OSR’s evaluation of evidence: We consider this requirement to be met. While the overall length of the pre-release access list has not been reduced, we consider the justification provided for those included on the list to be valid. We are satisfied that both NHS England and DHSC have sufficient mitigations in place to manage the risks associated with a long pre-release access list.
Requirement 2: NHS England should publish a development plan for its A&E statistics to enhance transparency and allow users to input to developments. It should share this plan both with its known users and more generally on its website, to ensure it reaches as wide a range of individuals and organisations as possible.
Actions taken by NHS England: NHS England has published a development page for its A&E statistics. This page invites feedback from users on recent changes to the statistics and shares plans with users regarding the ongoing development of the statistics. The page also includes information about how NHS England carries out engagement about the statistics with internal and external users.
OSR’s evaluation of evidence: We consider this requirement to be met. The publication of a page which details ongoing and future developments, and NHS England’s user engagement activities, improves transparency. Importantly, it also provides opportunities for users to feed into this work.
Requirement 3: NHS England should expand and improve its published information about quality and methods to help users understand the strengths and limitations of the statistics and support appropriate use of them. This should include explanations about the level of uncertainty associated with the statistics, including confidence in the accuracy of data submissions, and what this means for their use, as well as the quality assurance processes carried out. It would also be helpful to provide a summary of the data journey, from data submission to publication.
Actions taken by NHS England: NHS England has published a data quality page for its A&E statistics. This page provides summary information on data completeness and quality assurance for both data sources used to produce the statistics. It includes several links to more-technical information about data quality as well as a flow chart explaining the flow of data from trusts to final publication.
OSR’s evaluation of evidence: We consider this requirement to be met. The introduction of a dedicated data quality page is an important addition for users of the statistics. During the assessment, we found that published information on quality and methods did not consistently provide enough detail to fully reassure users about the quality of the statistics. We also found that users had to work quite hard to find relevant quality information, as it was spread across several parts of the website. The new page brings together key information about the data sources and their quality. It provides users with guidance on the implications of any quality issues on the use of the statistics and allows users who require more detail to easily access more-technical documentation.
Requirement 4: NHS England should regularly and proactively engage with a wider range of users to understand their needs and implement improvements to the statistics which enhance their public value. It might be helpful to NHS England to publish a short user engagement plan setting out its intentions in relation to meeting this requirement.
Action taken by NHS England: NHS England has proactively contacted several organisations which use its A&E statistics, such as professional groups, health think tanks and research organisations, the House of Commons Library and media outlets, with a view to seeking their feedback. On its new development page, NHS England has published information about its user engagement activities and plans.
OSR’s evaluation of evidence: We consider this requirement to be met. We are pleased to see NHS England expand its user engagement activities beyond its core stakeholders. This will broaden the team’s understanding of the range of users of the statistics and their needs.
Requirement 5: NHS England should combine the 12-hour waits information with other monthly statistics about A&E and improve signposting between outputs on this topic to improve the overall accessibility of A&E statistics and insights that they provide.
Actions taken by NHS England: NHS England has improved signposting between its monthly A&E statistics and information about 12-hour waits. The statistics team considered merging these two outputs, but given the statistics come from two different data sources, the decision was made to keep the analyses separate.
OSR’s evaluation of evidence: We consider this requirement to be met. While NHS England has not combined the 12-hour waits information with its other monthly statistics, we consider that the reasoning for this is valid and that the improved signposting between the two outputs is sufficient to address the requirement.
Requirement 6: Building on its current progress in responding to users’ requirements, NHS England should explore the feasibility of publishing more-granular information in its monthly A&E publication to enhance the insights provided by the statistics. Based on user feedback, the priority for this work should be providing site-level breakdowns.
Actions taken by NHS England: In response to user requests, NHS has published more-granular breakdowns of its A&E statistics, including breakdowns by age, gender, ethnicity, chief complaint and frailty. NHS England has used the Emergency Care Services Dataset (ECDS), which, at record level, enables the production of summaries for specific care areas and patient characteristics.
On its new development page, NHS England welcomes user feedback on other areas of interest from ECDS data. On this page, NHS England also explains that it is currently exploring whether publishing site-level data would be beneficial for users and in the best interest of patients and their outcomes.
OSR’s evaluation of evidence: We consider this requirement to be met. We are pleased to see NHS England responding to user requests by publishing more-granular breakdowns of the statistics. This was a key user need we identified during our assessment. Although NHS England has not yet been able to include site-level breakdowns in the statistics, it has publicly committed to exploring this addition on its new development page. The team has also agreed to keep users updated on the outcome of this work.