Quality

Introduction

Quality means that the statistics and numerical information represent the best available estimate of what they aim to measure at a particular point in time and are not materially misleading.

Quality is analytical in nature and is a product of the professional judgements made in the specification, collection, aggregation, processing, analysis, and dissemination of data.

Findings

The statistics are based on sound data and methods

The data for the PIP statistics are sourced from the PIP Computer System and represent a 100% population of PIP claimants with a postcode in NI on the reference date. The PIP Computer System is also the source for PIP statistics for GB and therefore the PIP statistics produced by DWP are comparable with those for NI. Users told us that they had no concerns with the quality or methodology of the statistics and thought that the frequency of the statistics was good.

The statistics team has a strong relationship with the PIP operations team which enables it to investigate and respond to issues in a timely manner. The data-sharing agreement was updated at the beginning of the COVID-19 pandemic to ensure that the statistics and operations teams could more easily investigate claims that appeared to have errors, which helps assure the quality of the statistics.

The bulletin and tables include a methodology and quality note for the statistics but the information provided is brief and does not cover any limitations of the statistics. The team has carried out a self-assessment against our Quality Assurance of Administrative Data (QAAD) framework, as part of a NISRA-wide initiative, to assure itself of the level of quality at all stages of the production process. The statistics team should build on its methodology and quality notes in the bulletin and tables to bring out the key information from its QAAD self-assessment, to give users the necessary information to use the statistics appropriately.

Reproducible analytical pipelines are being developed to reduce errors

PSU is currently exploring the use of Reproducible Analytical Pipelines (RAP) in the production of its statistics. RAP involves introducing methods to make analysis easily reproducible. The statistics team told us that the aim of introducing RAP is to improve the quality of the statistics by reducing the risk of human errors associated with its current software packages. The team also hopes the adoption of RAP will enhance the accessibility of the statistics and data, by introducing the capability for filtering and drop-down options in the charts within the publication. We welcome the plans to introduce RAP into the production of the statistics as it shows commitment to quality and innovation.

The roll-out of RAP across PSU’s statistics was intended to start with one statistical release being piloted, before being adopted more widely across PSU. However, the COVID-19 pandemic has slowed progress on the pilot and therefore the team told us that a steering group has been set up to develop RAP across publications rather than waiting for the pilot to be completed. It is hoped that once RAP has been fully implemented, it will free up resources within the team that can be reallocated to address developments requested by users.

Greater collaboration with DWP would improve coherence and quality

A Service Level Agreement is in place between PSU and DWP to facilitate data sharing. The statistics team has access to a selection of 73 tables monthly relating to NI PIP data, sourced from the same PIP Computer System. However, the statistics team told us that, although it has access to the data, the PIP statistics require complex analytical coding to match the right datasets and to apply the correct rules to the data to produce statistics that are comparable with DWP. This has been a barrier to replicating the breakdowns available in DWP’s PIP statistics as the statistics team do not have access to the code which DWP uses to produce the PIP statistics.

More generally, it is imperative that the team is aware of issues or amendments in the PIP Computer System. DWP communicates with DfC on such changes or issues. However, PSU would benefit from strengthening this relationship and these communications. The statistics team in DfC acknowledged this challenge and said that whilst it had good relationships with DWP, the relative size of DWP compared to DfC and staff changes can make it difficult for the statistics team to foster the same links that the operational and policy colleagues have formed.

The statistics team needs to form a stronger and more collaborative relationship with DWP to enhance the coherence of PIP statistics across the UK. The statistics team should establish regular engagement with the statisticians working on PIP in DWP and use existing user groups or forums to keep up to date on changes made to the PIP Computer System and developments with the DWP publication. A more joined up approach will also ensure a common understanding of the quality and priorities of PIP statistics.