Findings

Data sources, methods and QA

The suitability of the revised methodology and quality assurance approach to produce headline labour market estimates.

Suitable data sources

In response to ONS concerns about the quality of the LFS-based estimates for the three months to August 2023, ONS quickly developed experimental headline estimates of the UK labour market using alternative data sources. The new method uses LFS data for April -June 2023 and applies growth rates calculated from PAYE RTI data for employment and Claimant Count data for unemployment. This is partly explained in the blog that ONS published alongside labour market estimates. ONS describes these sources as robust. We appreciate that these new estimates were produced at speed. However, to help users understand their suitability ONS should now publish information detailing how these data sources were selected and outlining any potential sources of bias and uncertainty in these data, given neither series are accredited official statistics (‘National Statistics’).

To help users’ understanding of the decision to not publish the labour market estimates in their usual way, ONS should publish additional information on the specific quality concerns that led to these decisions. Users we spoke to as part of this review have also raised concerns about the quality of the LFS data for May-July 2023, which some users thought was used as the base to apply the growth rates calculated from the PAYE RTI and Claimant Count data, given that response rates in the LFS were already declining. ONS has confirmed to us that April – June 2023 data are used as the base. Some users also commented that if there were growing concerns about the quality of the LFS, ONS was not communicating this clearly and so users were not aware that ONS was building up to its current position of having to suspend using the LFS data in its current form. We also heard from users that they felt the tone of the communications from ONS downplayed the seriousness of the issues with the quality of the LFS.

ONS was only able to provide users with UK headline estimates in October 2023. The update published on 2 November 2023 by ONS explained that for November’s publication, it would again publish a set of experimental adjusted headline estimates based on growth rates from PAYE RTI and the Claimant Count for May to July 2023 onwards, with the latest quarter being July to September 2023. As in October, ONS also plans to produce the November experimental adjusted headline estimates by age group and, additionally, by region and country. Not publishing geographic and characteristic breakdowns in October caused significant issues for some users, particularly for the Welsh and Scottish Governments who rely on the country-level LFS data for monitoring their own economic policies. We welcome the ONS’s plans for the November release which will go some way to address these issues.

Sound methods and quality assurance

ONS has sought expert methodological advice from academic experts when developing the new experimental method; we welcome this given the tight time constraints.

While ONS announced that it would be publishing new experimental estimates using an alternative method, users were not given advance notice of the methodological change to headline labour market estimates. We recognise that this was complicated by a desire not to jeopardise orderly release as some of the data used in the method was already publicly available. Although we recognise ONS faced challenging circumstances, in future we would expect ONS to provide users with more advance notice and detail around changes to methods, explaining why the changes are being made. This would help users to understand how the estimates are calculated.

ONS could have done more to fully explain the method that was used. The most detailed description is in a blog post, but this is not linked from the main published estimates. Some users we spoke to found that the limited information and the messaging around it led to some confusion as to how the latest labour market estimates were derived and whether it was a permanent improvement to the LFS.

The lack of a back series using the new approach has complicated the ability of users to fully understand the impact of the changes. Given that LFS-based estimates of employment have been growing more slowly than PAYE RTI, interpretation of a series that combines data from both sources is challenging in the absence of additional guidance. It is also not clear whether the method used meets international standards and whether the new estimates can be compared internationally.

Clarity of communication

The clarity of communication with users on what the new series can and cannot be used for

We support ONS’s approach to label the new series as experimental statistics. This approach should continue unless ONS seeks accreditation for these new statistics.

The labels are clear in the visual graphs and data tables. However, more clarity is needed on what the series can and cannot be used for. In ONS’s blog, the series are described as the ‘best available’, but key is whether the uncertainty around these estimates are clearly explained to allow appropriate use and meet user needs. We consider that commenting on this is an important focus of further ONS communications.

Whilst we recognise that the alternative experimental measure was compiled in a very short period of time, users would have benefited from being provided with information on how this alternative measure should be used and the strengths and limitations.

We also note that ONS published a user request for headline labour market estimates based on the TLFS on its website. This publication means that there are two estimates of the employment rate, unemployment rate and inactivity rate for GB available. The TLFS is currently in development and has not yet been confirmed by ONS as meeting its statistical quality criteria. We are concerned that, although these estimates are published as indicative only, ONS has not published any quality information alongside the figures to support appropriate use. This data is also not accessible from the labour market release.

Meeting users' needs

The extent to which these statistics meet users’ needs

As part of this review, we have engaged with key policy users including the Devolved Governments, the Bank of England (BoE), HM Treasury (HMT), the Office for Budget Responsibility (OBR), and the Department for Work and Pensions (DWP). Given the speed of this review, we have not been able to proactively engage with a wider group of users, but have received feedback from other groups, which has been considered for this report.

ONS’s labour market estimates compiled using the LFS data are the official UK headline indicators. Typically, the estimates are provided at a UK, sub-national, regional, and parliamentary constituency levels to help inform national and local-level decision making. The new experimental series provide labour market estimates at a UK level only, meaning that a large amount of insight at sub-national and disaggregated levels (such as ethnicity and disability data) is missing, which is having a detrimental impact on users.

In addition to its use in the compilation of official estimates of the UK labour market, LFS data also plays a vital role in understanding changing dynamics and is a rich source of demographic data. The Annual Population Survey (APS) uses data collected from the LFS combined with local area boosts to allow for more detailed analysis. Data derived from the LFS and APS are used extensively and are highly valued by a wide range of users. ONS has told us that it is exploring the extent, scale and impact of the low response rates on the APS, which is next due for publication in January 2024.

The Welsh Government and Scottish Government typically use the APS for analysis of labour market trends at a disaggregated level. Given the APS is subject to a time lag, the devolved administrations rely on the results from the LFS as the timelier official measure of headline labour market trends. The data gaps are not limited to the devolved governments. Other government agencies are lacking breakdowns by disability, ethnicity and other characteristics.

Users have expressed concern over the uncertainty around the Labour Market statistics due to be published in the coming months, such as whether these will be based on the LFS data if response rates and quality improve sufficiently or whether the experimental method will continue to be used and what the implications are for the availability of regional and disaggregated data. We welcome that ONS has published its plans for the release of Labour Market data for the forthcoming months, and committed to a further publication in early December, therefore giving users time to consider the implications for their own needs and so reducing some of the uncertainty.

Transition to Transformed Labour Force Survey (TLFS)

Further actions planned to improve the quality of the LFS during the transition period to the Transformed Labour Force Survey (TLFS)

We were pleased to see more information provided by ONS in the action plan it published on 2 November 2023.

We recognise that ONS faces significant and continued challenges around survey response rates and the ability to meet user needs with the continued uncertainty around the transition to the TLFS. The action plan published by ONS provides some reassurance to users around the response rates of the TLFS. It also provides a structured plan for improving the quality of LFS estimates in the transition period. The outcomes of this work will need to be monitored to understand whether the quality issues are addressed and any impact on the appropriate use of these statistics. We encourage ONS to continue to share its plans and any updates to these plans throughout the transition period.

Managing quality

Learning lessons in managing quality during transformation

During our review users have highlighted concerns about the decisions that have contributed to current response issues in the LFS. Whilst we recognise that ONS made its best effort to produce a headline estimate for users after the decision was made that the LFS data was not of sufficient quality to publish, ONS could have done more to prevent these circumstances from arising. However, we also acknowledge these decisions require the balancing of resources across outputs. Putting more resource into maintaining response rates for the LFS, in the context of a constrained budget, would require less investment in another area. Some users, as noted previously, were not aware of the quality issues and would have appreciated greater communication about it. However, other users have been raising concerns about the quality of the LFS for several months due to falling response rates and biases in the sample. ONS should have been more proactive in evaluating the impact of its decision making ahead of time to prevent the decline in LFS data quality.

In addition, users also questioned whether ONS reached the right judgement in balancing investment decisions between developing the strategic TLFS and mitigating the quality of the LFS in a way that gives value for money. We will consider this theme further in our regulatory work on the TLFS.