Dear Alex, Darren,
ONS Labour Force Survey (LFS) transformation
As you are aware, we recently started a review of ONS’ transformation of the Labour Force Survey (LFS). As we have discussed with the team, this letter presents the first in a series covering our emerging findings and will lead to a decision about the continuing National Statistics status of ONS’s labour market statistics produced from the LFS. This letter provides a summary of our findings and recommendations, which are set out in more detail in the annex.
Background
The LFS is the main household survey that is used in the compilation of official estimates of the UK labour market. It plays a vital role in understanding changing dynamics and is a rich source of demographic data used to inform estimates of population change. The Annual Population Survey (APS) uses data collected from the LFS combined with local area boosts to allow for more detailed analysis. Data derived from the LFS and APS are used extensively, and are therefore highly valued, by a wide range of users. ONS has been developing a different version of the LFS using an online-first multimode approach. This version has been running live since March 2020, in parallel with the existing survey, with plans to start incorporating transformed LFS data into regular labour market statistical outputs from September 2023. Given the complexity and scale of the transformation of the LFS, we agreed to engage early and undertake a review against the Code of Practice for Statistics using a phased approach.
Summary of our findings
We recognise and fully support ONS’ ambitions to improve the quality, coverage, adaptability and responsiveness of the LFS to maximise its relevance and meet changing user needs. Following a steady decline in response rates and challenges faced in survey data collection during the Covid-19 pandemic, action was needed to improve the quality and robustness of these important estimates. It is really encouraging to see that ONS is addressing these concerns through the transformation of the LFS, in addition to exploring opportunities to maximise greater use of administrative data to innovate and develop its labour market statistics.
The LFS transformation is also taking place in the context of wider transformation of population and migration statistics. Therefore, we are pleased to see close working between the LFS and other relevant teams in ONS to understand and manage the impact of these transformations on each other.
ONS has an ambitious timetable, with plans to decommission the current LFS in June 2023. There is lots of excellent internal development work ongoing at present to implement ‘knock to nudge’ fieldwork in November 2022, finalise content requirements, develop and finalise statistical methodology and evaluate early transformed LFS data results. ONS’ ambitions are well supported across the user community, with a recognition that with declining response rates, action needed to be taken to improve the quality of estimates and coverage.
As with any transformation project, maintaining public trust and confidence in the changes being made is really important – and transparency is key to this. Given the extensive public value that the LFS offers, users are keen to understand more about how the transformation will affect their use, including what variables will continue to be available and its approach for dealing with survey non-response.
There has been little information available publicly about the LFS transformation project. Therefore, many of our detailed recommendations point to a need for ONS to be increasingly frequent and transparent in communicating progress (especially on development of methods) and plans via its website to a broad range of users. By engaging well with a broad range of users, ONS can harness the insight to maximise the public value of the transformed LFS, alongside demonstrating its trustworthiness. We hope that ONS can build on the regularity and detail of information published in September to further support public trust and confidence.
We would like to thank all the teams in ONS who have positively engaged with us so far. The efforts of staff to innovate, develop and transform the LFS is commendable. We look forward to continuing our review in parallel to your plans to develop and transform the LFS.
Yours sincerely
Mark Pont
Assessment Programme Lead
Annex: OSR review findings and recommendations – ONS Labour Force Survey (LFS) transformation
Findings |
Recommendations |
|
|---|---|---|
Value |
ONS is engaging well with key stakeholders, such as the devolved administrations, through a steering group and hosting focused conversations with established users. ONS also carried out a wider public user engagement exercise in March-June 2022. Apart from these activities, users that we spoke to were less confident in ONS’ long-term plans to keep them informed of the details. Most users knew about the transformation of the LFS, although less detail was known amongst the academic community which also raises questions about the extent of ONS’ user reach. To date, ONS has published little information about the LFS transformation project, for example users found it difficult to fully respond to the public user engagement exercise in the absence of more-detailed information about the proposals. We recognise that ONS is working through the detail of the changes in incremental stages, with many answers yet to be finalised. But supporting users through the changes, for example by providing updates on progress and plans, is crucial with open and transparent communication and feedback channels playing a key role. As a first step, we were pleased to see that in September 2022, ONS published the first in a series of planned quarterly updates to better inform the public on its plans and progress. |
To enhance public confidence and maximise the public value of the transformed LFS, ONS should:
|
|
The LFS is a rich source of demographic data and users really value the range of variables available to support cross-topic and longitudinal analysis according to their individual needs. The transformation has provided a good opportunity for ONS to review the relevance of the LFS survey content. The availability of some variables will change at both an aggregate and micro level and understandably users are concerned about this. We are pleased to hear that in November, ONS intends to publish a suite of user guidance documents including a first version of mapping variables from the transformed LFS to the current LFS. This is a great step forward in supporting users through the changes and it will be important that as part of the communication plans, ONS clearly explains any data discontinuities and the plans to mitigate these. |
Upon publishing finalised variables for the transformed LFS, ONS should explain any likely data discontinuities, any subsequent impact on time series and its plans to mitigate these. | |
|
The Annual Population Survey (APS) uses data combined from two waves of the main LFS (Waves 1 and 5) with data collected on a local sample boost. The larger sample size for the APS dataset due to its design, allows for more-detailed analysis, especially at a local level. Users rely on both LFS and APS datasets interchangeably. ONS is currently working on how the transformation will affect the APS but has not yet communicated with users about the details of its plans. Population data are used in the LFS and APS to provide contextual comparisons of labour markets for different areas and for use as a denominator for calculating rates within the datasets. Conversely, LFS and APS data are also used alongside other sources to feed into estimates of population change. For example, data from the APS are used to measure the stock of migrants in the UK. It is good to see that the LFS transformation team is working closely with other teams in ONS involved in the wider transformation of population and migration statistics and this should continue. |
To support ongoing user needs, ONS needs to be open in its communication with users about how the redesign of the LFS will affect the availability and granularity of APS. For example, what, if any, datasets will be available in the interim to support longitudinal analysis. | |
Trustworthiness |
There is a strong governance structure in place across two divisions in the ONS: Social Survey Transformation (SST) and the Economic, Social and Environmental Statistics Group (ESEG), with shared objectives for the LFS transformation project. The Labour Market Statistics Steering Group sits between the two divisions to maximise collaboration and senior leaders are closely and actively involved in decision-making. Additional governance boards have been introduced for the duration of the parallel run to further aid decision making, with reporting through existing project and programme boards. ONS is comparing estimates produced from the current LFS and the transformed LFS through a parallel run. ONS should also triangulate early results with other relevant data sources. The parallel run is planned to be complete in Spring 2023 and ONS cites this as a key decision point to assess if the transformed LFS has met ONS’ statistical quality criteria to start decommissioning the existing survey. However, in the event of the transformed LFS data not meeting the statistical quality criteria, ONS’ contingency plans are currently unclear. At present, there is no information in the public domain about:
|
To support public confidence in the transformation process, ONS should publish information about its;
|
Quality |
ONS has developed an ‘online-first’ multi-mode collection approach for the new survey. Following successful early testing in 2018 and 2019, the online survey went live in March 2020. Face-to-face interviewing was suspended during the pandemic, and data collection for the LFS moved to interviews by telephone only. It is not clear what analysis has been done retrospectively to look at the mode effects of switching from a face-to-face to a telephone-only survey and from telephone-only to an online-first approach. The online-first approach is supported by telephone collection aimed at those with limited/no online access (introduced in February 2022) and ‘knock to nudge’ fieldwork (planned introduction in November 2022) to target under-represented areas. We understand that ONS will also consider targeted face-to-face interviewing if needed. The Northern Ireland Statistics and Research Agency (NISRA) is responsible for LFS fieldwork operations in Northern Ireland with finalised plans for the transformed LFS yet to be finalised. As ONS continues to work through its plans to transform the LFS, much of the detailed information is still to be published. ONS has committed to publish more of a detailed overview of the survey design and methodology between October and December 2022. |
To reassure users about the changes made to survey approach, ONS should publish sufficient detail about the survey design, sampling frame and any analysis exploring mode effects |
|
Methods for the transformed survey are yet to be finalised. The change in surveying methods during the pandemic affected non-response bias and the quality of the LFS estimates. ONS undertook two reweighting exercises to address this. In October 2020, housing tenure-based weights were introduced and in May 2021, HM Revenue and Customs (HMRC) Real Time Information (RTI) data were used to improve population weights to mitigate the increase in non-response from those with a non-UK country of birth or nationality. ONS worked with external experts to develop the current weighting methodology with reweighted LFS estimates published in June 2022 using up-to date HMRC RTI data. Early 2021 Census data were used to provide further quality assurance. |
No recommendation | |
|
ONS has applied an adaptive, responsive design to improve quality, respondent experience and operational efficiency. It has developed its survey design principles through engagement with external academic experts and international counterparts such as Statistics Netherlands. ONS has a well-established design assurance process, including input from academic experts. Internal technical assurance is sought via the ONS Methods and Research Assurance Group MaRAG and quality assured with input from ONS Fellows. The Methodological Assurance Review Panel (MARP) also provides external, independent assurance on the overall statistical design and methods being developed for the transformed LFS. The current LFS is designed to provide accurate quarter-on-quarter estimates of change due to a rotation pattern. Households stay in the sample for five consecutive quarters (or waves), with a fifth of the sample replaced each quarter – with an 80% overlap in respondents in consecutive quarters. Published monthly bulletins use rolling quarterly data. ONS is taking forward plans to produce monthly estimates in response to user demand for more granular and timely estimates. The ONS methodology team is in consultation with various experts to use a modelling approach such as state space models, with more detail published in 2019. Maintaining consistency and comparability over time will be challenging. The ONS methodology team is working on how to maintain the back-series and are liaising with users to understand estimates that are essential for a continuing time series. |
To help support understanding of the changes being made, ONS should ensure that all relevant papers submitted to the Methodological Assurance Review Panel (MARP) are published, accessible and well signposted on the UKSA website. In particular, those covering imputation, weighting, maintaining time series, dealing with non-response bias and modelled monthly estimates. |