Evidence from Previous OSR Assessments

Since 2015, OSR has conducted 26 assessments and reviews of ONS’s economic statistics. A full list of these reviews can be found in Annex 2.

Successes in economic statistics

Our regulatory engagement with ONS on economic statistics has identified a range of areas where ONS has made significant progress in recent years. These successes have included methodological improvements to the National Accounts, particularly the introduction of double deflation, and a range of improvements to price statistics, including positive developments on the measurement of private sector rents for dwellings (as described in our recent assessment) and on car prices.

Issues around response rates, administrative data and user engagement

Some issues regularly come up across our suite of reviews and assessments. These common issues include:

• Falling response rates to household surveys, including the Labour Force Survey and the Living Costs and Food Survey. Declining response to surveys has negatively affected data quality. While exacerbated by the pandemic, issues with response rates long pre-dated this crisis.
• Challenges in making progress on the use of administrative data obtained from other government departments. These challenges were noted both in our own review of GDP revisions and the Independent Review of the UK Statistics Authority 2023-24 conducted by Professor Denise Lievesley. Analysis by ONS and OSR in connection with the challenges faced by the Labour Force Survey, in both its original and transformed form, has confirmed that PAYE data can provide relatively reliable data on some aspects of the labour market. These data have been available for several years but not been fully exploited. OSR’s assessment of the Annual Business Survey also pointed to the unexploited potential of administrative sources to improve survey coverage and quality.
• The potential to redesign surveys to complement administrative data, where they are available. Our review of GDP revisions revealed scope for further exploiting VAT data on sales, and potentially data on business purchases. Doing so could facilitate the redesigning of the relevant surveys to increase the focus on larger firms and thereby improve the quality of results. Our assessment of the Annual Business Survey made similar points.
• Challenges with the performance of legacy IT systems, including the “black box” nature of many processes. These challenges include facilitating the use of administrative data, updating survey questions and methods to stay relevant and reducing risks to quality. The performance of supporting IT systems was a key source of concern in OSR’s reviews of the Producer Price Indices (PPIs), business investment statistics and the Living Costs and Food Survey.
• Issues of coherence across data sources. This is particularly, of course, in respect of the labour market, but was also revealed in OSR’s assessment of the Purchases Inquiry, which highlighted inconsistencies with data on purchases from the Annual Business Survey.
• Scope for better engagement with users, despite improvements. Enhanced engagement is critical to improve users’ understanding of the features and limitations of statistics, including associated uncertainty, and to better understand users’ needs. A range of OSR assessments, including those undertaken on profitability and gross operation surplus, research and development statistics, construction statistics, the PPIs, business investment statistics, the Annual Business Survey and the Living Costs and Food Survey, have highlighted the need for better communication with users on such matters.
• The fact that samples and other data sources are not always regularly reviewed, updated and reprioritised, despite the range of measures that ONS has taken to maintain and improve quality. Small businesses may not be adequately covered. Relevant examples of concerns about sample representativeness are provided by the need for a radical revision of research and development statistics and the requirement for ONS to review samples for the PPIs and business investment statistics.
• Under-prioritisation of outputs which do not directly have a high profile, but which nevertheless contribute to the overall quality of the National Accounts. The deterioration of the sample for PPIs identified in OSR’s review on these statistics provides such an example.

ONS has responded to issues identified in OSR assessments by making improvements, for example revising the sample for the Research and Development Survey to improve coverage, particularly of smaller businesses; putting in place a programme to introduce a new business register; and improving communication on the quality of GDP statistics and the associated uncertainty. However, several of these issues are present across a range of statistics – and are therefore systemic.

In addition, ONS has sought to address problems with household surveys. Steps it has taken to this end include initiating the Social Surveys Quality Recovery Project (SSQRP), which is increasing the number of field interviewers and sample sizes to boost response numbers; holding communication campaigns to increase response rates; increasing incentives to improve response rates; and investigating pay and benefits options to improve interviewer retention. ONS has also engaged a “recruit, train & deploy” employment agency to substantially further increase field interviewers. These steps have led to an increase in responses for the Labour Force Survey. However, given the long-running challenges with social surveys, these activities should have been undertaken earlier and at greater pace, with the necessary funding put in place, and communicated publicly as a coherent recovery plan.