Quality means that the statistics and numerical information represent the best available estimate of what they aim to measure at a particular point in time and are not materially misleading.
Quality is analytical in nature and is a product of the professional judgements made in the specification, collection, aggregation, processing, analysis, and dissemination of data.
The statistics are based on sound data and methods
The data for the PIP statistics are sourced from the PIP Computer System and represent a 100% population of PIP claimants with a postcode in NI on the reference date. The PIP Computer System is also the source for PIP statistics for GB and therefore the PIP statistics produced by DWP are comparable with those for NI. Users told us that they had no concerns with the quality or methodology of the statistics and thought that the frequency of the statistics was good.
The statistics team has a strong relationship with the PIP operations team which enables it to investigate and respond to issues in a timely manner. The data-sharing agreement was updated at the beginning of the COVID-19 pandemic to ensure that the statistics and operations teams could more easily investigate claims that appeared to have errors, which helps assure the quality of the statistics.
The bulletin and tables include a methodology and quality note for the statistics but the information provided is brief and does not cover any limitations of the statistics. The team has carried out a self-assessment against our Quality Assurance of Administrative Data (QAAD) framework, as part of a NISRA-wide initiative, to assure itself of the level of quality at all stages of the production process. The statistics team should build on its methodology and quality notes in the bulletin and tables to bring out the key information from its QAAD self-assessment, to give users the necessary information to use the statistics appropriately.
Reproducible analytical pipelines are being developed to reduce errors
PSU is currently exploring the use of Reproducible Analytical Pipelines (RAP) in the production of its statistics. RAP involves introducing methods to make analysis easily reproducible. The statistics team told us that the aim of introducing RAP is to improve the quality of the statistics by reducing the risk of human errors associated with its current software packages. The team also hopes the adoption of RAP will enhance the accessibility of the statistics and data, by introducing the capability for filtering and drop-down options in the charts within the publication. We welcome the plans to introduce RAP into the production of the statistics as it shows commitment to quality and innovation.
The roll-out of RAP across PSU’s statistics was intended to start with one statistical release being piloted, before being adopted more widely across PSU. However, the COVID-19 pandemic has slowed progress on the pilot and therefore the team told us that a steering group has been set up to develop RAP across publications rather than waiting for the pilot to be completed. It is hoped that once RAP has been fully implemented, it will free up resources within the team that can be reallocated to address developments requested by users.
Greater collaboration with DWP would improve coherence and quality
A Service Level Agreement is in place between PSU and DWP to facilitate data sharing. The statistics team has access to a selection of 73 tables monthly relating to NI PIP data, sourced from the same PIP Computer System. However, the statistics team told us that, although it has access to the data, the PIP statistics require complex analytical coding to match the right datasets and to apply the correct rules to the data to produce statistics that are comparable with DWP. This has been a barrier to replicating the breakdowns available in DWP’s PIP statistics as the statistics team do not have access to the code which DWP uses to produce the PIP statistics.
More generally, it is imperative that the team is aware of issues or amendments in the PIP Computer System. DWP communicates with DfC on such changes or issues. However, PSU would benefit from strengthening this relationship and these communications. The statistics team in DfC acknowledged this challenge and said that whilst it had good relationships with DWP, the relative size of DWP compared to DfC and staff changes can make it difficult for the statistics team to foster the same links that the operational and policy colleagues have formed.
The statistics team needs to form a stronger and more collaborative relationship with DWP to enhance the coherence of PIP statistics across the UK. The statistics team should establish regular engagement with the statisticians working on PIP in DWP and use existing user groups or forums to keep up to date on changes made to the PIP Computer System and developments with the DWP publication. A more joined up approach will also ensure a common understanding of the quality and priorities of PIP statistics.
DfC publishes a data quality statement but it does not detail its understanding of the quality of administrative data.
- The methodology notes in the bulletin are brief and do not cover any limitations of the statistics.
- The team has reviewed the statistics against our QAAD framework.
To reassure itself and users of the quality of administrative data, DfC should expand the information on data quality within the bulletin and tables, highlighting any limitations or issues that were identified through the QAAD framework.
The relationship between DfC and DWP needs to be stronger to facilitate the development of DfC’s PIP statistics and to improve coherence between outputs on PIP.
The statistics team recognises the importance of strengthening its relationship with DWP for access to code to replicate DWP’s statistics and being made aware of crucial developments to the operational system from which the data are sourced.
Users highlighted a lack of coherence in outputs between DWP and DfC.
To enhance the coherence and quality of PIP statistics, DfC must build strong links with the responsible statisticians in DWP and establish regular engagement with DWP, to discuss issues and priorities for developments.