Ensuring quality

Survey response rates

Good statistics need good data inputs, so statisticians want data collections with sufficient coverage, whether these are surveys or administrative sources. In the UK, and in other countries across the world, we have seen a steady decline in response rates to household surveys, which can reduce the accuracy of estimates and is a potential cause of bias in the survey data. Surveys will remain essential in some areas, but it has become more challenging and expensive to maintain quality.

We saw further challenges during the COVID-19 pandemic as many producers had to rapidly adapt their approach in light of the suddenly limited opportunities to interview people face to face.

Household surveys provide data for some of the most important statistics used to inform decisions in significant areas, including government spending and setting interest rates. Increased uncertainty and gaps in these data present challenges for decision-makers, so it is essential that any gaps and quality issues be robustly addressed; this will inevitably lead to the need for additional investment.

Great Britain, excluding imputed households, January to March 2014, to January to March 2024

The Labour Force Survey (LFS) is one of the most significant social survey collections in official statistics in the UK. In October 2023, response issues in the ONS Labour Force Survey, which covers England, Scotland and Wales, became so acute that ONS had to suspend LFS-based labour market estimates for the UK and introduce an experimental method using administrative sources. We undertook a rapid review and provided recommendations to ONS on how it could maximise the value of these data.

ONS has since taken steps to improve the number of interviews conducted and reinstated LFS-based estimates, for which we carried out a further review. As part of a longer-term response, ONS is already planning to introduce the Transformed Labour Force Survey (TLFS) in England, Scotland and Wales, which will make greater use of online data collection. This work has so far shown a promising increase in the response rates. Northern Ireland is also transforming the NI LFS to an online first survey called the NI Labour Market Survey.

It will be important for producers to share best practice both in and outside the official statistical system to address these challenges and consider how the most suitable administrative data, different collection approaches and nudge techniques can be used to mitigate declines in response rates.

Last year, the National Records of Scotland (NRS) faced unexpected challenges with data quality, as the overall response rate for Scotland’s Census 2022 was lower than the designed rate. NRS put measures in place to ensure that the best-quality estimates were produced and to provide assurance for users. These measures included oversampling its Census Coverage Survey in areas where it expected a lower response, and establishing an International Steering Group to deliver external expertise and advice.

The International Steering Group was a group of experts from different countries and organisations that advised NRS on its statistical methods and use of administrative data. The group ensured that NRS was aware of best practices and experiences from other censuses and surveys. Its guidance helped NRS ensure that its methods were robust, transparent and consistent with international good practice. In our accreditation of Scotland’s Census 2022, we found that NRS had taken a rigorous and methodical approach to the quality challenges that had arisen. Members of the statistical system who face similar challenges can learn from NRS’s experience.

Furthermore, business survey response rates have recovered from the challenges faced in the pandemic. ONS and NISRA have been undertaking good work to consider how they can optimise their survey operations to ensure they align with the needs of respondents and to increase the volume and quality of data received through business surveys.

Challenges of measuring the economy

Economic statistics are not compiled in isolation as individual data series. They are derived within a complex and organic framework of economic variables and models, which come together to build a comprehensive picture of the UK’s economy. This starts at the goods and services produced, transformed and consumed, taking into account public sector finances and our relationship with the rest of the world, and ends with our balance sheet, which shows how the transactions in our economy have affected our net worth. These statistics also consider the prices we pay and the jobs we do.

Economic statistics are vital for understanding the economy and a key source of information and intelligence for those making important decisions on tax, spending and interest rates. These statistics need to be of good quality, use data sources that reflect the modern economy and clearly communicate any uncertainties.

Through our Spotlight on Quality programme, we have developed a new model for assessing statistics that allows us to respond to the complexity of the economic framework used to compile important economic indicators. This approach has allowed us to scrutinise the elements of quality that are most relevant to each set of statistics and dive deeper into the sources, methods and quality assurance than possible in our standard assessments.

Our initial reviews on Producer Price Inflation and the Profitability of UK companies and Gross Operating Surplus of private non-financial corporations statistics, alongside the rapid reviews we carried out in response to challenges with Labour Market Statistics and GDP, identified common themes around data source quality and communication. We plan to undertake a systemic review to consider these common threads and track ONS’s progress on improving economic statistics since Professor Sir Charles Bean conducted his independent review of UK economic statistics.

We are glad to see ONS taking a more prominent role in work to develop the next round of international standards for economic statistics. Our review will also consider ONS’s preparedness for implementing these new standards, which we see as an important step for improving measurement to better capture the modern economy. ONS has also made significant progress this year on the Beyond GDP agenda, which looks beyond traditional approaches to take a holistic look at economic, environmental and social progress, and public sector measurement, which has been welcomed by users and strengthened ONS’s international standing.

Our recommendations