Findings

Transparency is a cross-cutting theme of the Code of Practice for Statistics; it encompasses clarity and openness around statistics to ensure trust in them. A key theme that has emerged from this review is the need for improved, clear and open communication. Users welcomed such qualities of communication in the methodology article which accompanied the recent reintroduction of the reweighted LFS. ONS should continue to build on this clarity in its communications approach and embed it across all its communications, from plans for reweighting to explaining data issues and improvements, to its approach to transitioning from the LFS to the TLFS. This approach builds trust and confidence in the quality of numbers being produced and ensures users can use them appropriately and with proportionate confidence.

Meeting user need: communication of plans and priorities

Users we spoke to as part of this review generally welcomed the re-introduction of the LFS-based estimates and indicative datasets. These estimates are viewed as an improvement compared with those based on the experimental approach used previously, as these data provide a wider set of labour market metrics such as economic inactivity by reason. However, it was noted that the small sample size is still affecting the quality of the estimates as the results for some of the breakdowns have large sampling variability, so it is difficult to determine if changes reflect real changes in the labour market or appear because of sampling variability, which introduced volatility in the statistics. Falling response rates in surveys such as the LFS and the resulting detrimental impact on the quality of outputs are also  highlighted in the recently published Independent Review of the UK Statistics Authority by Professor Denise Lievesley CBE.

Users were also positive about the improved transparency provided in the reweighting methods article released in advance of the February 2024 labour market publication, particularly ONS’s clarity on the need to take account of wider labour market data sources.

However, users noted that while some UK headline series have been modelled back to 2011 to remove the discontinuity introduced by reweighting, not all series have been modelled due to time constraints.  This limits users’ ability to carry out meaningful time-series analysis for key variables such as job-to-job moves and regional employment and unemployment rates. This discontinuity in the data has had a particularly adverse effect on Scottish and Welsh estimates. The decision to not reweight the Annual Population Survey (APS) data that the Scottish Government and Welsh Government financially contribute towards and use to estimate lower-level regional breakdowns of the labour market data has also led to incomparability between the APS and LFS data.

ONS told us it recognises the user need for a fully reweighted data set but that this would take a significant amount of resource and a further 6 months to complete, which is why ONS prioritised the reweighting of LFS estimates for the period of July to September 2022 onwards. Only UK headline estimates were modelled and scaled back to 2011. ONS told us it is considering the next steps in the coming months.

We recognise that having a reweighted time series is very important for users. It is crucial to enable comparisons over time. However, we also appreciate that ONS has a finite set of resource and that this reweighting is time consuming and so work needs to be prioritised against the broader TLFS and LFS work programme. We heard from users that they would like some clarity on whether ONS plans to carry out any further modelling of the LFS data back to 2011 below the UK-level estimates of employment, unemployment and inactivity; reweighting of APS data; and reintroduction of the household LFS. More certainty would allow users time to determine any appropriate alternative data sources to meet their needs in this interim period before the switch to the TLFS. To enable users to plan their analysis and work, ONS should be clearer to users about its plans and priorities for the LFS in the coming months. This should include how user needs will be met.

Users in Scotland also highlighted that while the 2021 Census results for England, Wales and Northern Ireland have been published, results from the 2022 Census for Scotland have not been. It is not clear from the information ONS has published what impact the absence of Scottish census data has on the reweighted LFS data and what this means for their use. This should be clearer.

Accessibility of updates and communications

ONS has recently published several updates on the LFS, such as its plans for improvement and reintroduction in November 2023 and a further update to complement the Impact of reweighting on LFS key indicators: 2024 article in early February 2024 ahead of the reintroduction of the LFS-based estimates later that month. However, these outputs are published across the ONS website with no clear signposting, and so users may not have the full narrative as regards changes to the LFS.

Information about the trade-offs from the switch from the experimental estimates back to the LFS is covered in the reweighting article, but as some users may not read the article, ONS needs to consider how this information can be more clearly communicated alongside its results. Users told us they would have liked further detail on the choice of timing for switching back to using the LFS, and while ONS told us that the decision was in part due to the usefulness of the experimental estimates decreasing over time, this does not appear to have been communicated to users very clearly.

ONS has committed to considering how it can present a more joined-up narrative around the LFS. We recognise that exact timings may not always be known but consider that improved communication around the challenges being faced and the steps ONS is taking to overcome these difficulties will help to improve trust in the LFS improvement process and transition to the TLFS.

Explaining how the data should be used

In its reweighting methods article, ONS helpfully outlines some of the trade-offs it made when deciding to reintroduce reweighted LFS data. In this article, and the labour market bulletins and data tables, ONS clearly provides warnings for users about the ongoing volatility of the LFS, advising caution and for users to refer to a range of sources to piece together the full picture, such as Workforce Jobs, claimant count data and Pay As You Earn Real Time Information (PAYE RTI) estimates; we heard from some users that this advice is welcome. The increase in volatility in the estimates is due to increased sampling variability. We recognise the greater transparency provided by the reweighting methods article but think it could be further improved for the less-experienced users. For example, the comparability and strengths and limitations of these different sources could be better outlined to help guide users on the most appropriate use; this could include better signposting to existing documents such as ONS’s Comparison of labour market data sources. More generally, the non-specific use of words like ‘volatility’ and ONS advising ‘caution’ do not provide users with enough information to support appropriate use.

To help users to understand the uncertainty around these estimates, ONS publishes the sampling variability (95% confidence intervals) of the headline LFS estimates in the sampling variability section (Dataset A11) of its Labour market overview statistical bulletin. However, sampling variability data are not provided by age group, which we understand is one of the key areas that has seen increased sampling variability over the past five months.

The reweighting of the LFS and its impact, including the discontinuity introduced to some series, are explained in the methods article and in the Employment in the UK bulletin. However, some users pointed out that the methods article and the published improvement plan are focussed at the UK level, and so they would like to have seen more information on how improvements to the LFS data at a country and regional level have been implemented and what this means for using the data.

Communication of data quality issues and improvements

ONS’s response to our letter setting out our intention to carry out this short review demonstrates its commitment to data quality and describes the impact of the improvements already in place. The team told us that while the improvement work is ongoing and that it is starting to see improvements to the data quality, the full impact of the improvement work has yet to be seen.

ONS has been carrying out thorough quality assurance of the LFS data. It uses a range of techniques, including looking at confidence intervals, sampling variation and QA input from key stakeholders such as the devolved nations, and compares the LFS data with similar administrative sources to understand the strengths and limitations of the estimates so that this information can be shared with users. We recognise, though, that despite having robust procedures in place, errors may occur, for example, the recent error in ONS’s processing of the Northern Ireland LFS data which resulted in NISRA not being able to publish NI LFS-based estimates on 12 March 2024. This error was discovered during the final quality assuring of the data and before publication, which demonstrates the importance of good-quality assurance. We recognise ONS is investigating what led to the error and intends to take steps to ensure that this does not happen again.

Information advising users of the recent NI LFS data processing error and its implications for the data could be more consistent. This information was included on the ONS release calendar, the operational note that was circulated to users, and while the latest Employment in the UK release published 12 March highlights the Northern Ireland data issue and its impact prominently, this is not replicated in the UK labour market overview release, and so this key information could be missed by some users. Additionally, ONS communicated in its March 2024 release that it would not be publishing regional LFS data for Scotland and Wales, however ad hoc regional LFS data for Scotland and Wales were published by ONS. We recognise that this ad hoc data was to respond to Scottish Government and Welsh Government needs. However, the availability of data for Scotland and Wales should have been signposted from the releases. ONS has told us that it is looking at how to best signpost to this data from the release and we advise that this should be completed as soon as practicable.

Transition to the TLFS

Users have welcomed the reintroduction of the LFS-based estimates, but as the LFS continues to go through a period of instability, users of the data would also like greater clarity about the process of transition to the TLFS and how this links with improvement plans for the LFS as the two are increasingly interlinked. This includes the criteria and evidence that ONS is using to guide the transition and any further improvements to the LFS. Providing a better understanding of the criteria ONS is using will help assure users’ trust in labour market data.

ONS has previously been transparent with users on its governance of the TLFS by publishing its governance process in response to one of the recommendations from our ongoing review of the TLFS.

While information in the Employment bulletin on planned improvements to the LFS signposts users to the TLFS update published in November 2023 and the Impact of reweighting on LFS key indicators: 2024 article also focusses on the TLFS as the long-term solution, this could lead some users to believe that there are no plans to make any further improvements to the existing LFS, which is not the case.

We recognise that ONS intends to publish a plan setting out further improvements and developments for the LFS data shortly as well as its plans for the TLFS development work. These publications should give users an indication of what they can expect and when.

We also appreciate that ONS intends to make the publishing of high-quality employment data one of its priorities for 2024/25 and plans to align resources with these priorities. We recognise that given the LFS-based estimates are published monthly, the teams involved work within tight timescales and that the monthly labour market production timetable does not allow much contingency should an issue arise. The teams working on the production of the LFS-based estimates are also those working on the TLFS, and so extra work required on one can impact the other. Therefore, we hope that this priority focus and resource realignment can help to bolster and further support the transformation and production of these important labour market statistics.