The Statistics Authority’s quality assurance of administrative data standard has three levels of assurance: basic, enhanced and comprehensive.
This case example illustrates comprehensive quality assurance of administrative data sources used to derive PHE’s MRSA healthcare-associated infection statistics.
Contents:
Background to the statistics
The data sources and suppliers
Approach to the quality assurance of the admin data
Outline of PHE’s quality assurance approach
- Data checks by NHS acute trusts with senior officer sign-off
- Validation of record during data capture
- Monitoring levels of trust item response, data entry and processing errors
- Corroboration against independent sources – laboratory records and deaths
- Independent cross-checks through double running of statistical tables
Background to the statistics
Healthcare-associated infections (HCAIs) are infections that can arise either through medical or surgical interventions in a healthcare setting or simply from being in contact with a healthcare setting. These include infections that may be picked up outside of the hospital setting, although still within the healthcare environment, such as in a community-based health establishment.
Most strains of Staphyloccus aureus respond to the more commonly used antibiotics, but some strains are more resistant. Those resistant to the antibiotic meticillin (i.e. ‘Meticillin-resistant Staphylococcus aureus’ or MRSA) often require different types of antibiotic to treat them. Mandatory surveillance is required for the infections of greatest public concern including MRSA.
These infections pose a serious health risk to patients, staff and visitors in a healthcare setting. In addition to the morbidity risks and in some cases mortality risks, they can also result in significant extra costs to the NHS. NHS England currently operates a ‘zero tolerance’ approach to MRSA infections.
The incidence of HCAIs can be volatile, with underlying patterns changing to reflect new or different treatments and the implementation of interventions and enhanced precautions. For example, the counts of MRSA infections originating from skin and soft tissue sources have increased in recent years while those associated with intravenous lines have declined. It is therefore particularly important to ensure that relevant data are collected that will allow professionals to make the correct judgments about where healthcare resources should be placed.
The major users and uses of these statistics are:
- hospitals and care homes – to reduce preventable mortality and morbidity within the NHS
- NHS acute trust hospitals and Clinical Commissioning Groups (CCG) – to monitor against performance targets
- public health officials – to detect and manage any outbreaks or indications of problem areas
- DH – to inform government about performance against the NHS outcome frameworks; to make comparisons either across England, across the UK or across Europe; to input into future policies and to provide information about the impact of key interventions and precautionary strategies
Excerpt from PHE’s MRSA infographic
Source: PHE MRSA 2016
The producers
PHE, which is an executive agency of the Department of Health (DH), was formed in April 2013 following the amalgamation of the Health Protection Agency (HPA), the National Treatment Agency (NTA) and various Public Health Observatories in England. PHE, and previously HPA, has been monitoring the surveillance of MRSA since April 2001, although it was not mandatory until April 2004.
The data sources and suppliers
NHS acute trusts report cases of healthcare-associated infections, including MRSA, to PHE using a web-based data capture system (HCAI DCS). This diagram illustrates the current data flows:
Figure: HCAI data collection flow
Source: PHE
MRSA is treated differently from other infections as screening takes place on admission or pre-admission to hospital and a Post Infection Review (PIR) is completed after the infection is confirmed. This process outlines how a case becomes included in the monthly statistical summaries:
- a case is detected, either through screening or by clinical symptoms
- the blood isolates are sent to the designated pathological laboratory
- the trust infection control team decides to attribute the cause of the infection
- all confirmed cases are entered into the data capture system by a member of the trust infection control team
- the data are signed-off by the trust by the 15th day of the month following the incident
- PHE staff extract the data shortly after the 16th day of each month
- PHE staff carry out quality assurance processes, including cross-referencing cases recorded on separate laboratory surveillance schemes
- PHE publishes the monthly HCAI statistical around 4 to 6 weeks after extraction
Approach to the quality assurance of the admin data
PHE has documented its comprehensive assurance approach in its Data Quality Statement.
Trust verification
NHS acute trusts report infection cases. The verification process relies on a senior officer of the trust (as delegated by the Trust’s Chief Medical Officer) signing-off their data on a monthly basis. The sign-off provides PHE with formal assurance that the data are accurate and complete, and that the published statistics include details of all cases for the reported time-period.
Some trusts may need to amend their data in situations in which sign-off occurs prior to full laboratory results being available. PHE monitors the pattern of amendments requested by trusts. It supports trusts in identifying duplicate records. PHE permits suppliers to request subsequent changes to the data and revises the published statistics up to three months after provisional release. It indicates to users which data have been revised and includes quality information about any reported issues from trusts alongside the statistics. It also highlights any item non-response and refers them to NHS England for follow up and resolution.
PHE identifies the risk of processing errors made by trusts through investigating the level of errors in date of birth and NHS number, as an overall indicator of bias for the full system. PHE assesses the percentage of cases where it is not able to match using these two variables – in 2016 less than 4.5% of all cases were unmatched. It concluded that this was satisfactory for its purposes.
Corroboration of coverage
PHE compares the data from NHS acute trusts with data from laboratories which also report cases of microbial infections identified in various samples eg blood, urine etc. This enables PHE to assess the completeness and coverage of cases captured in the two sources. PHE monitors the differences each year, for example, identifying data quality issues for laboratories.
As well as comparing the overall totals of cases identified in the two schemes, PHE conducts data matching checks. In April 2016 it said that 66% of MRSA cases reported to the mandatory scheme (NHS trusts) were identified in the voluntary (laboratory) surveillance scheme. It also checked the matching with records held in its HCAI DCS database and matched 92% eligible records.
Part of its routine data checks involve contacting laboratories with the cases reported to the voluntary surveillance scheme but not identified on the HCAI DCS, to obtain feedback on the cases. It then adds the unmatched cases to the database where appropriate.
PHE estimates the expected number of cases which are eligible for mandatory reporting but not captured by the HCAI DCS. By using this method, PHE has demonstrated that the HCAI DCS has extremely high level of coverage of 93% for MRSA (as well as for other HCAIs).
Additionally, PHE can analyse a separate data set, the Quarterly Mandatory Laboratory Returns, to confirm the accuracy of coverage by NHS trusts. For example, PHE notes the potential for the introduction of bias into the statistics due to the use of the statistics for performance monitoring by trusts. It conducted an investigation of data between 2008 and 2013 of Clostridium difficile toxin tests, to determine whether there were changes in testing over the period. It also examined the pattern in numbers of deaths (a source not related to the surveillance scheme). These checks supported the patterns shown by the HCAI statistics.
Other checks
PHE describes in its Data Quality Statement its own QA approach with two scientists independently running the data processing to produce its statistics. Checks are carried out by third parties; for example, PHE notes that NHS acute trusts and the Care Quality Commission also perform audits of local infection data, which can result in requests to add infection episodes that had not previously been entered in the data capture system.