Quality

Data sources and quality assurance

1.19 The two data sources used in production of these statistics are the Monthly Trust Situation Reports (MSitAE) and the Emergency Care Data Set (ECDS). Both datasets are administrative data sources and the data are appropriate for producing A&E statistics.

1.20 MSitAE is a collection of aggregate data covering A&E attendances and emergency admissions from NHS Trusts in England and is used by NHS England to monitor activity and performance levels. The data are also used to produce the monthly A&E statistics. Data are submitted to NHS England via the Strategic Data Collection Service (SDCS). SDCS is a secure data collection system used by health and social care organisations to submit data. Before submission, data are quality-assured and signed off by providers. NHS England also performs central validation checks aiming to ensure good data quality.

1.21 ECDS is a collection of patient-level data on A&E attendances from NHS Trusts in England. ECDS replaced the Hospital Episode Statistics dataset, which had previously been used to produce the quality indicators and annual A&E statistics, in April 2020. Trusts (or ‘providers’) collect administrative and clinical information locally to support the care of patients; these data are then transferred as a fully automated daily feed to the Secondary Uses Service (SUS) at NHS England and the ECDS dataset is derived. Data are submitted using an xml schema, which returns an error if mistakes have been made. The use of the schema aims to ensure some standardisation of the data received and means that the data have to meet certain validation rules before being submitted to SUS. Each month, NHS England creates data quality dashboards to show NHS providers the completeness and validity of their data submissions. This helps to highlight any issues in the provisional data, allowing time for corrections to be made before the annual data cut is taken and the databases for the year are frozen. NHS England has commissioned support from a central team to help providers improve the quality of their submissions.

1.22 There are some differences between the ECDS data and MSitAE data. The main difference is that, as discussed above, the ECDS data are available at a more granular level. Another difference between the datasets is that MSitAE data do not include attendances where the A&E appointment has been pre-arranged. Therefore, when comparing ECDS directly with MSitAE, NHS England excludes planned follow-up attendances from the analysis.

1.23 Data collection in relation to performance monitoring is often at risk of misleading practices such as gaming and misreporting because those providing the data may be incentivised to submit inaccurate figures. NHS England told us that it is increasing use of record-level data allows for comparisons with aggregate returns, and that it expects that these would highlight discrepancies arising from such practices. While users did not raise misleading practices as a concern and we have seen no evidence that they are occurring, we consider that NHS England could do more to assure itself and users that data quality is not being impacted by such issues.

Communicating quality and methods to users

1.24 The three statistical releases are each accompanied by their own individual information on quality and methods. Briefly, the monthly A&E activity publication provides comprehensive information on comparability both between the ECDS and MSitAE and with data for the other UK countries. The release also contains details of coverage and completeness for the ECDS, and some limited methodology information.

1.25 The annual A&E release for 2022/23 includes a link to Hospital Accident and Emergency Care Activity supporting information. This contains quality information for the ECDS, as well as details of the methodology used to suppress confidential data. NHS England also publishes a separate ECDS data quality page, which includes a link to a data quality dashboard. The monthly provisional release provides high-level quality and methodology information via a separate supporting information document.

1.26 Despite this, information on quality and methods overall does not consistently provide enough detail to fully reassure users about the quality of the statistics. For example:

• From the published information it is difficult to understand the methods involved for data submission and the overall data journey, as well as the statistical methods and processes. Some information is available across the publications and in the wider ECDS webpages, but overall this does not provide a complete picture.
• Similarly, there is limited description of quality assurance processes across all three releases. NHS England have described the approach to quality assurance for both data sources to us, and this includes several steps from the data collection and submission stage to the point at which the data reaches the statisticians. However, this information is not clearly communicated for users to read and understand.
• There is very limited information published about quality for MSitAE for the monthly A&E statistics.
• There is very little explanation of limitations arising from the methodology or of uncertainty across all the releases.

1.27 Users we spoke to were generally happy with the quality of the statistics but raised some issues around the communication of data quality issues. Examples included a lack of information on missing data, very limited quality information around demographic variables, and lack of explanation around whether NHS trusts apply the definitions for the A&E unit types consistently.

1.28 Between May 2019 and May 2023, 14 NHS Trusts undertook field testing of new urgent and emergency care performance metrics and stopped reporting information on 4-hour performance. From June 2023 4-hour performance data from these 14 trusts has been reintroduced. Some of the users we spoke to requested a back-series including the missing data. NHS England has informed us that it is unable to produce this due to agreements it made with the 14 providers to not assess them by the 4-hour standard throughout the field test period. The team has published information, however, on the impact of the clinical review of standards on the national A&E timeseries.