Value

User engagement

1.29 Users of statistics and data should be at the heart of statistical production – their needs should be understood, their views sought and acted on where practicable, and their use of statistics supported. This requires producers of statistics to maintain an open dialogue using proactive formal and informal engagement to listen to the views of new and established contacts.

1.30 We found some evidence of user engagement by NHS England. For example, the statisticians at NHS England engage with analysts at the Department for Health and Social Care and NHS England responded to user feedback by providing new statistics on 12-hour waits. In 2022, prior to the merger with NHS England, NHS Digital carried out discovery research which involved engagement with some users of its statistics (though this project was not specific to A&E statistics) and more recently carried out a consultation inviting feedback on proposed improvements to the quality indicator and annual A&E publications. However, beyond government colleagues, we found that the statisticians have very limited knowledge about who uses their statistics and do not carry out regular, proactive user engagement. Engaging more effectively with a broader range of users would enable NHS England to better understand user needs and help it to maximise the public value of the statistics, while at the same time prioritising limited resources. These users could include, for example, health think thanks, academic researchers and institutions, parliamentary researchers, journalists, and members of the public.

Accessibility and insight

1.31 To support society’s needs for information, statistics and data should be easily accessible, presented clearly and explained meaningfully. Statistics and data should also be released at the greatest level of detail possible to meet user needs.

1.32 Users we spoke to reported using the monthly A&E activity publication most often as it provides the timeliest information against the 4-hour target. Although used less frequently, users also highlighted the value of the additional measures and breakdowns provided in the quality indicator and annual A&E publications. We found that most direct users of the statistics are expert users who have a good understanding of the topic area and need to carry out their own analysis of the figures. As a result, most users we spoke to go straight to the data tables to access the statistics, though a couple said they also find the monthly commentary document useful for obtaining a quick overview of the main messages. Users of the commentary document appreciate its simplicity but highlighted that the addition of charts, particularly showing trends over time, would enhance its value. Users of the data tables have well-established processes for obtaining and analysing the data they need and find this relatively easy to do. However, some noted that unannounced changes have occasionally been made to the table design which causes their established processes to fail and require updating. While the current format of the data tables is largely suitable for these users, many would prefer to access the statistics in a more reliable and automated way, such as via an API or open data platform. We appreciate that there will not be one format which will suit all users. Therefore, as part of the user engagement requirement above, NHS England should make improvements to enhance the accessibility of its outputs in line with users’ priority requirements.

1.33 In February 2023, NHS England started publishing statistics on 12-hour waits from arrival to admission, transfer or discharge in response to user feedback and our own intervention. Although this is an important addition to the monthly A&E statistics, it is currently published in a separate document. Several users commented that this makes it harder for them to find and use the new 12-hour waits information. They were not sure of the reason for the new information being separate and felt that it makes it harder for them to fully understand the overall picture in relation to A&E waits. Combining the new information into the existing monthly output would help to increase the overall insights offered by the statistics. We also consider that, in the short term, the accessibility of A&E statistics in England would be improved by more prominent signposting between the various outputs. In the longer term, accessibility may be maximised by publishing the outputs on the same website.

1.34 NHS England currently presents the monthly A&E statistics at NHS Trust level. This is the level of legal accountability for the 4-hour target and there is strong user interest in these figures. However, many users also want the statistics to be broken down by site (i.e. hospital). NHS Trusts can comprise several sites with different characteristics, so figures presented at Trust level can mask big differences between individual site performance. There was also some interest in more-frequent breakdowns in the statistics by factors such as age, gender, and ethnicity. Currently demographic breakdowns such as these are available only in the annual publication.

1.35 NHS England explained to us that the aggregate data submitted by NHS Trusts each month do not allow for site or demographic breakdowns. To provide more-granular breakdowns in the monthly statistics, NHS England would need to change the source of the statistics to make increasing use of the record-level data provided by the ECDS. It is good that NHS England understands the user need for additional breakdowns and would like to address this. We are aware that it will take a considerable amount of work to implement this improvement and that this may take some time. It will be important to keep users updated on progress towards this improvement, even if timelines are tentative or change.

Comparability

1.36 We know from our previous research that users of health statistics often have a strong interest in comparing NHS performance between the four nations of the UK. We also know that direct comparisons are not always appropriate, or even possible, due to operational or policy differences between the four nations. Therefore, we expect statistics producers to provide comparable statistics where possible and, where this is not possible, clearly notify users that this is the case and provide guidance which supports appropriate use of statistics.

1.37 It is good that the annual A&E publication includes a section on four-nation comparisons of A&E attendances and 4 and 12-hour waiting times. The publication explains that, given the different models of service provision across the four nations, to allow the most like-for-like comparisons, the statistics used for comparisons are restricted to data for ‘Type 1’ or ‘Major’ A&E departments. The annual publication also provides a link to useful information produced by the UK Comparative Waiting Times Group which outlines some of differences between A&E services and policies across the four countries which impact the statistics.

1.38 We strongly support the ongoing work by the UK Comparative Waiting Times Group and ONS’s Health Coherence Team to understand and explain differences in NHS performance statistics across the four nations. The group plans to publish a series of summaries in 2024 which will bring together published data from across the four nations and explain how the data fit within different policy frameworks and infrastructures. These summaries will cover several topics, including A&E waiting times. We encourage NHS England to continue to engage with these efforts to ensure that users are provided with clear information about comparability.

Reproducible analytical pipelines

1.39 The Reproducible Analytical Pipeline (RAP) is a set of principles and good practices which ensures that analyses are reproducible, auditable, efficient, and high quality. RAP principles support the highest standards of trustworthiness, quality and value, so we welcome the Analysis Function’s RAP strategy, which aims to embed RAP as the default approach to analysis in government. Prior to the merger, NHS Digital had already made good progress implementing RAP principles into its statistical production, including the use of automation to reduce the manual steps in the data processing pipelines and moving towards open-source code. The team has told us that next steps in this area include improving the current pipelines to increase the levels of automation and publishing code and supporting information to improve transparency. Team members have worked closely with the organisation’s RAP community and we support the plans the team has in place to extend this to other publications. Following the merger of the two organisations, we would like to see teams share learning and support each other so that RAP principles can be adopted more widely across NHS England in the long term.