This Quality Framework
Introduction
The vision of the Office for Statistics Regulation is simple: statistics that serve the public good. Public good is ensured by statistics that have public value, are high quality and are produced by people and organisations that are trustworthy.
Statistics being high quality means that the statistics fit their intended uses, are based on appropriate data and methods and are not materially misleading. Ensuring quality requires skilled professional judgement about collecting, preparing, analysing and publishing statistics and data in ways that meet the needs of people who want to use the statistics. It is important to recognise that no data sources are perfect: there are always strengths and limitations with any data.
We regulate the production of statistics using assessments against our Code of Practice. The need for enhanced scrutiny of the quality of economic statistics has driven us to develop a quality-focused framework of indicators against which to assess quality. This framework does not replace the Code of Practice but supplements it. ‘Standard’ assessments and assessments of new statistics will continue to be carried out using our Code of Practice framework of Trustworthiness, Quality and Value. Assessments where the focus of interest is on the quality of the statistics, or changes in it, will use a quality-focused approach. This framework of quality-focused indicators will be used to drive our quality-focused assessments and ensure that data and methods produce assured statistics. We will report on the indicators that are most relevant to explaining our judgements and requirements for the statistics that we assess. Not all indicators will be relevant for all sets of statistics, and we do not intend to score and rate sets of statistics against them.
The framework was developed from the practices in the Quality pillar of the Code of Practice for Statistics[1] (the Code), the International Monetary Fund’s Data Quality Assessment Framework (IMF DQAF) and the Quality Assurance Framework of the European Statistical System (ESS QAF)[2]. These frameworks include indicators at a range of levels that are relevant to ensuring the quality of statistics.
We have carried out pilot assessments on Producer Price Inflation statistics and the Profitability of UK Companies’ and Gross Operating Surplus of non-financial corporations statistics in order to test and further develop the framework. These pilot assessments proved successful and demonstrated that the framework was able to be applied in practice. They also led to some improvements to the indicators in this framework. The framework is now being used for subsequent Spotlight on Quality assessments[3].
We are publishing this framework to provide transparency around our Spotlight on Quality assessments. We want to ensure that our framework will provide users and stakeholders of UK economic statistics with continued assurance around quality. We also want producers of economic statistics to understand the framework that we will be using to assess the quality of their statistics and strive to make sure that their statistics are meeting these standards. We welcome feedback on this framework and will review, refresh and re-publish it as appropriate.
The framework is being developed and first applied to economic statistics, due to the changing context of the regulation of these statistics. We intend to test, and then widen the use of, this framework on the regulation of statistics beyond economic statistics and will consider where the framework may need to be adapted for that use.
[1] A review of the Code of Practice for Statistics was published in March 2024. A key action of that review was for the Code to be refreshed. Changes as a result of that refresh will be reflected in this framework at a subsequent version.
[2] The OCED also has a Data Quality Framework (https://www.oecd.org/sdd/qualityframeworkforoecdstatisticalactivities.htm) which was not explicitly used in the generation of this quality framework but which aligns with the frameworks used.
[3] The latest information on Spotlight on Quality assessments that we have carried out or are carrying out can be found on the Spotlight on Quality pages of our website.
Overview of the framework
The framework is structured around four principles. The first captures foundational factors that affect quality, such as resources, development plans and prioritisation, and is based largely on practices from our Trustworthiness and Value pillars. The latter three are based on the three principles in the Quality pillar of the Code. Each of the four principles of the Spotlight on Quality framework has been designed to ensure the statistics fit their intended uses, are based on appropriate data and methods and are not materially misleading. This includes using appropriate systems and resources to produce statistics and data in ways that facilitate quality assurance and enhance trust in the statistics.
1. Resources, plans and prioritisation
This principle covers the factors that enable the production of high-quality statistics, such as the availability and allocation of resources, the development and implementation of plans, and the prioritisation of user needs. It includes indicators such as there being sufficient human and financial resources; suitable systems; an established development work plan; user involvement in developing plans; and transparency around progress and prioritisation decisions.
2. Suitable data sources
This principle covers the factors that relate to the appropriateness and quality of the data sources used to produce statistics, such as the coverage, accuracy, timeliness and coherence of the data. It includes indicators such as definitions and concepts within data sources; relationships with data suppliers; source metadata; coherence of source data; explanation of data sources, and their quality and limitations, to users; innovation in sourcing data; and collaboration to maximise data use.
3. Sound methods
This principle covers the factors that relate to the validity and reliability of the methods used to produce statistics, such as the design, testing, documentation and review of the methods. It includes indicators such as the use of appropriate methods and recognised standards, classifications and definitions; explanation of reasons for deviations from standards; transparency of methods and their limitations; advance notice and user feedback on changes to methods; production of consistent time series; collaboration to improve methods; and the use of independent internal and external reviews.
4. Assured quality
This principle covers the factors that relate to how quality of statistics is assured, such as the organisational culture, the quality dimensions of the output data and provision of information about the quality of the statistics. It includes indicators such as that quality meets users’ needs; proactive user engagement around quality; transparency of output quality; proportionate quality assurance and risk minimisation; quality of provisional data; and understanding of revisions.
For each indicator in the framework, we explain in more detail what it seeks to measure and the reasons for its inclusion in this framework, including where a similar indicator is included in international quality assessment frameworks. We also provide examples of some of the questions that we will be asking when assessing statistics against the framework.
There are various ways in which the indicators could be grouped into a framework. For this version of the framework, we have chosen to broadly follow the structure in the Code. This structure is familiar to producers and users of statistics and to our regulator team and makes clear the links between the Code and the framework. Alternative structures could include using the Generic Statistical Business Process Model or grouping the indicators into those which relate to the quality of estimates, those that relate to communicating the quality and those which relate to current or potential risks to quality. Based on learning from future Spotlight on Quality assessments, changes to the Code during the upcoming refresh and feedback on the framework, we will consider the appropriateness of the structure at our next review.
The framework has been peer reviewed ahead of publication by colleagues in some producer and user organisations in the UK, and by colleagues in international organisations with experience in assessing the quality of a range of statistics.
In the final section of this document, we discuss the indicators that are included in other quality assessment frameworks but which we have chosen not to include in this framework. We explain the reasons for those decisions.
Annex A contains the framework of indicators.