Quality

Introduction

Quality means that the statistics and numerical information represent the best available estimate of what they aim to measure at a particular point in time and are not materially misleading.

Quality is analytical in nature and is a product of the professional judgements made in the specification, collection, aggregation, processing, analysis, and dissemination of data.

Findings

Suitable data sources

The statistics are based on data extracted from administrative or management systems, with the majority being drawn from NHSBSA’s own systems. The Statement of Administrative Sources for Statistics at NHSBSA outlines clearly that use of administrative or management data has advantages over survey data of being timelier and providing greater coverage, while at the same time reducing burden and minimising costs.

NHSBSA receives a small proportion of the data from NHS Resolution. This subset of data relates to appeals and applications. NHSBSA told us that the current process for obtaining data from NHS Resolution is fairly informal. The two organisations are therefore working together to establish a more formal data flow. NHSBSA has told us it is in the process of agreeing a Statement of Requirements with NHS Resolution, along with a Quality Assurance of Administrative Data (QAAD) toolkit to help it perform an assessment of the data source.

Currently NHSBSA receives data from NHS Resolution in Excel files via email. NHSBSA acknowledges that this process leads to a higher risk of the data being corrupted and so it is working to improve both how it obtains the data and the format it receives it in. The full process is still being finalised but NHSBSA has advised us that in the first instance it is planning for the data to be provided as a raw extract from NHS Resolution’s system in CSV format, which will then be deposited into a secure a file sharing mechanism.

NHSBSA has been unable to obtain pharmacy data that were previously collated by NHS Digital and published as the Pharmacy Collection (PHS1) when it had responsibility for the statistics. This includes data on pharmacy openings and closures, pharmacy exemption categories, pharmacy consolidations, pharmacies commissioned to provide enhanced services, and decisions on applications for new and existing pharmacy premises. NHSBSA has been unable to obtain these data due to the fact that they are held by a third-party contractor whom NHSBSA has to negotiate with through NHSE&I. NHSBSA pre-announced this to users prior to the first publication since it took over the statistics, and is continuing to collaborate with NHSE&I to try to obtain these data from an alternative source. Currently NHSBSA is struggling to make progress in this area and has acknowledged there is a risk that it may ultimately not be able to obtain these data. NHSBSA has told us that it is working on mitigation plans for if this turns out to be the case, with the intention of presenting some data on this topic in its forthcoming publication. It has committed to being transparent with users and describing any changes in methods clearly to them, as well as explaining any impacts on the figures that these changes may have.

Sound methods

NHSBSA has made changes to some of the methods since taking the statistics over from NHS Digital. It has explained the changes clearly and comprehensively in the background information and methodology note, but not all users access this and some told us it was too detailed for their requirements. This led to some users questioning the figures and whether they were accurate. To provide clarity on the methodological changes for as wide a range of users as possible, NHSBSA should provide a brief explanation of the changes in the main report. This could include linking to the background information and methodology note for those users who require more detailed information.

Since taking over the statistics, NHSBSA is also no longer able to produce 10 years’ worth of figures, as NHS Digital did. This means that users are unable to assess the impact of the new methodological approach on older data. To address this, NHSBSA has committed to publishing information for users outlining the likely impact on older figures if the new approach were to be applied to them.

Assured quality

NHSBSA outlines clearly the accuracy and reliability of the statistics in the background information and methodology note and the quality assurance of administrative data (QAAD) assessment. Although the data are drawn from administrative systems, there are potential sources of error that could affect the accuracy of the figures. These are largely due to there being a manual element to data entry and prescription processing. There is also a risk of data corruption during the transfer process for the small amount of data that NHSBSA receives from NHS Resolution. Currently this uncertainty is not reflected in the messaging and figures within the publication. This could lead to users concluding, incorrectly, that the figures are precise. NHSBSA should ensure that the degree of uncertainty in the figures is clearly explained within the publication to ensure appropriate use.

In the background information and methodology note, NHSBSA explains that contractors submit data to NHSBSA relating to advanced services that they provide to the public on behalf of the NHS. As part of the prescriptions processing procedure, a separate team in NHSBSA carries out a series of manual and automated assurance checks to ensure the accuracy of these data. Following these checks, adjustments may be made to the reimbursements or fees paid to contractors for their services. Errors identified after payment are not corrected in the statistical data; however, information on some of the checks and the accuracy of the data is published on NHSBSA’s website. NHSBSA also publishes separate information related to post payment verification undertaken by provider assurance. The statistical team at NHSBSA has not yet fully explored the potential impact of these checks on the accuracy of the statistics, although it has plans in place to do so, and it is believed to be negligible. NHSBSA should explore the impact of checks made as part of the prescriptions processing procedure on the accuracy of the statistics, and determine whether any adjustments to the statistics are needed. Depending on the findings of these investigations, NHSBSA should either make any necessary changes to the statistics and explain these changes and the reasons for them clearly to the users, or communicate clearly to users the accuracy of the statistics and the sources of any uncertainty.