As the UK’s regulator of statistics, we have a unique perspective on statistics that report on NHS performance. So, when changes are proposed to those standards, we are keen to highlight the key factors that must be considered throughout the process. These factors will be crucial for public confidence in the statistics about A&E.
The background is that NHS England has published an interim report detailing a review of NHS access standards. The report includes recommendations for updating and supplementing some of the older targets currently in use. And in particular, they propose moving away from the 4-hour waiting time target for urgent care.
Last November, we published a report that summarised our recent interventions about statistics on performance against the existing Accident and Emergency standards. We described problems we had seen with changes to the way A&E performance was recorded; with the reporting of data by one individual trust; with timeliness; and with comparability. And we showed how those problems had been addressed.
The findings of our report remain relevant to the changes that NHS England are proposing, and we’d encourage NHS England to take on board the following considerations.
Public importance
These statistics are among the most prominent official statistics produced in the UK. They help inform public understanding of the performance of the health service and provide the basis both for political debate and for the NHS’s operational decision-making in all four administrations of the UK. It is not our role to determine clinical standards. But we do care a lot about the statistics that report performance against the standards.
Clarity of purpose
Our report noted that the underlying issue with these performance standards is their purpose: Is it to hold hospitals to account? Is it a barometer of the way the whole health system is performing (so problems elsewhere, including in social care, mean that patients get ‘stuck’ in A&E because there is nowhere to go)? Or is it to measure individual patient experience of urgent care? Resolving this issue is a crucial first step in creating meaningful measures.
Involve statisticians and other analysts in the pilots
Statisticians and other analysts should be involved throughout. NHS England should ensure that statisticians and analysts are involved in any re-development of the data capture systems at an early stage, to provide assurance that these systems will support the proposed standards. Statisticians will also be able to ensure that the quality of new data is documented, that current data series are maintained for as long as feasible and that users are informed about data changes promptly.
Engage clinical experts
And it’s not just the statisticians and analysts who should be engaged. For a change of this scale, it is very important that the clinical teams who lead the provision of services in hospitals are also involved. From the interim report, it is hard to tell how far the proposals are the result of extensive engagement with expert clinical staff. We expect – under our regulatory standard for administrative data – that the operational experts are engaged with any process of change in how performance is measured.
Publish clear evaluations of the planned testing programme
It is good that the changes to access standards are going to be trialled in a range of different hospitals. Publishing details of the planned testing programme, followed by its independent evaluation, will help enhance transparency and confidence in what the pilots are indicating about the changes. One motivation to move away from the 4-hour target is to remove the tendency for admissions to cluster around the 3 hours 50 minutes mark. The evaluation should consider whether there are new, different unintended outcomes from the new average waiting times approach.
Consider the risk that performance standards could be misleading
The NHS England report acknowledges that the current standards can be misleading to patients. This demonstrates an appetite to improve the public value of the performance measures. NHS England should undertake user testing of how the new standards will be understood – whether they are meaningful to the public and what the public understand by them.
Consider the opportunity for better statistics on patient outcomes
There are good opportunities here. Some of the proposed changes, such as the ability to track the whole wait experienced by every patient, should enable the production of more detailed statistics to compare performance across the UK. We also expect that the potential to join up data across organisational boundaries – such as between ambulance and hospital or from hospital to home care – could lead to new ways of presenting detail about the pathways of care experienced by patients.
Trustworthiness, quality and value
Underlying all this, we argue that for the public to have confidence in statistics, they should meet the highest standards of trustworthiness, quality and value. Changing targets that are not being met will always raise suspicions that goal posts are being moved for presentational reasons. NHS England need to make sure that this impression is addressed head-on. By a transparent approach, including independent evaluation, NHS England can demonstrate trustworthiness. If the evaluation shows that these standards improve patient experiences, they will enhance value. If they can involve both analyst and clinical expertise, they should secure quality. After all, it would be a huge pity to develop performance standards that mean little to staff, patients and the wider public, even if – or especially if – they are achieving better outcomes.
See also: