Section A - Actions taken against initial recommendations
Recommendation 1 - Value
To enhance public confidence and maximise the public value of the transformed Labour Force Survey (TLFS), ONS should:
- communicate its long-term user engagement plans, detailing wider planned activities to engage with both established and new users on an ongoing basis, for example, publication of a user engagement strategy.
- regularly publish its updated plans, progress made and what users can expect to see through the quarterly transformation updates.
- seek further additional ways to engage with a wide range of users, considering communication style for both expert and non-expert users, for example, using social media, blogs.
Actions taken and commitments made by ONS to meet the recommendation
ONS has developed a stakeholder engagement plan outlining planned engagement activities with internal and external users of the current Labour Force Survey (LFS) and Annual Population Survey (APS), but not published it.
ONS continues to engage regularly through the Central Local Information Partnership (CLIP) sub-national labour market meeting, LFS/TLFS working group and bilateral conversations with representatives of government departments. In February 2023 ONS engaged with a wide user base via the annual LFS/APS conference.
From February 2023 ONS added a note to NOMIS and to the labour market monthly overview bulletin directing users to the latest transformation update. ONS has committed to continue to promote the transformation updates through these channels, with the latest transformation update published on 17 May 2023.
Recent messaging around the extension to the TLFS timetable is a good example where different communication methods were used to maximise reach, for example via established groups, a published blog post, and email circulation lists.
OSR’s evaluation of evidence
We are assured by the comprehensive stakeholder engagement plan that ONS has shared with us. To help users understand when and how they will have the opportunity to feed into ongoing developments the plan should be published.
It is good to see that ONS has continued to publish high-level updates via the transformation updates, promoting these via regular labour market outputs and considering additional ways to engage with users.
Actions remaining
To meet this recommendation, in addition to fulfilling its commitment to publish regular updates on the transformation process and continue to widen user engagement, ONS should publish a version of its stakeholder engagement plan as soon as possible, and before end August 2023 at the latest.
Recommendation 2 - Value
Upon publishing finalised variables for the transformed LFS, ONS should explain any likely data discontinuities, any subsequent impact on time series and its plans to mitigate these.
Actions taken and commitments made by ONS to meet the recommendation
ONS published a suite of user guides in November 2022 and as part of that a mapping document was included comparing questionnaire variables between the TLFS and LFS. ONS published an updated version on 3 July 2023, which included additional TLFS variables and relevant metadata for TLFS microdata. The user guides are iterative, with further plans to update as the survey content continues to be finalised.
ONS told us that work exploring data discontinuities will be complete around October 2023 and that it plans to publish any analysis.
OSR’s evaluation of evidence
The first and latest iterations of the published TLFS user guides provide a good level of initial information on the survey design, methodology and plans to disseminate data. The mapping document compares all the questionnaire variables in the TLFS with those of the existing LFS highlighting the changes and similarities between them. It also provides users with a list of additional topics with questions still in development that ONS plans to include in the survey in October 2023. The guides are still being developed, subject to further revision and there are some important information gaps to fill, such as the degree of TLFS question harmonisation with existing surveys and the frequency and coverage of TLFS variables.
ONS expects the work looking at data discontinuities to be complete around October 2023. Publishing information on discontinuities, explaining the differences and any impact on time series, will be essential to help users during the transition from LFS to TLFS.
Actions remaining
This recommendation will have been fulfilled once ONS has published adequate information on:
- finalised variables and questions (including harmonisation and frequency and coverage of TLFS variables) for the TLFS in the next iteration of the user guides in the autumn 2023.
- its analysis of data discontinuities explaining any impact on time series in autumn 2023 and before the decommission decision is made.
Recommendation 3 - Value
To support ongoing user needs, ONS needs to be open in its communication with users about how the redesign of the LFS will affect the availability and granularity of APS. For example, what, if any, datasets will be available in the interim to support longitudinal analysis.
Actions taken and commitments made by ONS to meet the recommendation
A section in the January 2023 transformation update informed users that the APS will change and expected timescales for future APS datasets – this has not been updated since the extension of the timetable.
As stated in its latest background user guide, ONS plans to develop TLFS microdata to replace all current LFS and APS datasets, including two-quarter (person and household basis) and five-quarter longitudinal datasets (person basis) and will assess the need for pooled datasets as the volume of TLFS data increases.
ONS continues to liaise with the Scottish Government and Welsh Government to ensure the transformed data for the APS continues to meet their needs, including recent redistribution of the TLFS sample in Scotland and Wales to improve representativeness. ONS published its issued sample distribution for wave 1 of the TLFS for each local authority in Great Britain on 3 July.
OSR’s evaluation of evidence
The latest user guide provides a list of the expected datasets planned to replace all current LFS and APS datasets. It also provides further detail on how the TLFS will be disseminated using other channels such as via NOMIS, the Secure Research Service (SRS) and the UK data Service (UKDS).
To help users understand how the transition to the TLFS will affect their use, ONS should publish expected timescales on the availability of these replacement datasets.
Actions remaining
To meet this recommendation, ONS should ensure that the transition from the current APS to its new equivalent is seamless and continues to support ongoing user need. As part of this, ONS should:
- publish expected timescales on the future availability of datasets, replacing current APS datasets now that the timetable has been extended in the next iteration of the user guides in the Autumn 2023.
Recommendation 4 - Trustworthiness
To support public confidence in the transformation process, ONS should publish information about its:
- governance arrangements
- evaluation and assurance process including decision-making checkpoints for the parallel run, and the statistical quality criteria it proposes to use in determining whether the transformation has been successful
- contingency plans in the event of the transformed LFS data not meeting the statistical quality criteria.
Early insights and the results of any comparative analysis should also be made available publicly.
Actions taken and commitments made by ONS to meet the recommendation
ONS published summary information about TLFS governance arrangements as part of its transformation update on progress and plans in January 2023.
ONS announced in April 2023 that it has extended the parallel run of the LFS and TLFS until the end of 2023. The latest transformation update (published in May 2023) outlines the statistical quality criteria that ONS intends to use to inform its decision-making for the parallel run.
On current plans, October 2023 is expected to be the start of the last quarter of data collected by the current LFS (Oct-Dec 2023) with the first series of labour market and productivity outputs using TLFS published in March 2024.
ONS has started to share TLFS data with key users to enable external scrutiny and quality assurance, via data sharing agreements. ONS told us that is plans to publish indicative results using data from the TLFS in autumn 2023.
OSR’s evaluation of evidence
ONS has publicly communicated its governance arrangements, statistical quality criteria and promoted its decision to extend the parallel run. This is a good step forward to maintain public confidence in its decision making. However, at present, it is not clear how ONS is planning to triangulate TLFS data with other data sources as part of its quality assurance.
The extension of the parallel run now provides ONS with the opportunity to involve users as part of the quality assurance process by sharing early insight and analysis of the TLFS data before the decision is made to decommission the LFS. Engagement with these users should be supported by regular, open, and transparent communication. The increased cost of the extended parallel run provides an incentive for ONS to ensure that all actions have been taken to ensure effective delivery of a high quality TLFS. In autumn 2023, we expect the results of the analysis to be published before the decision to decommission the current LFS is made.
Actions remaining
To meet this recommendation, and support public confidence in the transformation process, before the decommission decision is made in the autumn 2023, ONS should:
- explain how it is triangulating the TLFS data with other data sources, such as Census, to sense-check the reliability and accuracy as part of the quality assurance process
- publish planned activity that will be undertaken and decisions that will be made between October 2023 and March 2024.
Recommendation 5 - Quality
To reassure users about the changes made to the survey approach, ONS should publish sufficient detail about the survey design, sampling frame and any analysis exploring mode effects.
Actions taken and commitments made by ONS to meet the recommendation
In November 2022 and July 2023, ONS published a suite of user guidance which covered a background user guide, a variables mapping document and a dummy dataset.
As detailed in the latest transformation update, ONS intends to analyse mode effects and share some early insights with users in summer 2023.
OSR’s evaluation of evidence
It is good to see that high-level information on the survey design has been published as part of the background user guide, including estimated response rates in November 2022.
The recently published issued sample size for Wave 1 of the TLFS helps users to understand the distribution across local authorities in England, Scotland and Wales. With the final ‘Knock to Nudge’ mode of the TLFS introduced in November 2022 and the adaptive survey design now fully in place, ONS should also publish achieved response rates.
Analysis of mode effects is incomplete and is an important part of reassuring users on the changes made to the overall survey approach.
Actions remaining
To meet this recommendation, ONS should:
- include achieved TLFS response rates in the next iteration of the user guides planned for November 2023.
- publish its analysis of mode effects in the autumn 2023.
Recommendation 6 – Quality
To help support understanding of the changes being made, ONS should ensure that all relevant papers submitted to the Methodological Assurance Review Panel (MARP) are published, accessible and well signposted on the UKSA website. In particular, those covering imputation, weighting, maintaining time series, dealing with non-response bias and modelled monthly estimates.
Actions taken and commitments made by ONS to meet the recommendation
ONS is continuing to develop and finalise its statistical methodology. ONS told us it plans to seek assurance on its methods via its Methods and Research Assurance Group (MaRAG) with papers to be submitted in June 2023.
In October 2022, ONS published an article exploring the impact of using an alternative imputation method on main labour market estimates.
ONS has told us that it plans to publish a methodology paper in summer 2023 that will cover TLFS methods including imputation and weighting.
OSR’s evaluation of evidence
ONS’s final methodological approaches for the TLFS are still under development.
Some high-level information has been shared publicly via the transformation updates; however, detailed technical papers are not yet available.
Actions remaining
To meet this recommendation and support understanding of the changes being made, ONS should ensure that all relevant technical papers describing the methods used in the TLFS are published, accessible and well signposted before the end of December 2023. As part of this, ONS should:
- engage with expert users as part of ONS’s data sharing plans to help quality assure the methods used on TLFS
- take account of the results of the current exploratory work on current population weights used in the LFS to help inform the decision about methods used.