Findings

Transparency

NISRA has generally been clear and transparent in its communication about the transformation of its labour market statistics. Users are satisfied with NISRA’s communication on most aspects of the transformation of labour market statistics. NISRA has published transformation updates on its website throughout the transformation. The most recent update provides users with information on NISRA’s progress to date, next steps and early analysis on the sample from the new survey, which shows positive results both in terms of the sample size and its representativeness.

NISRA’s ad hoc approach to providing updates means that key updates are provided in a timely way, but some users told us this approach did not fully meet their needs as they did not know when to expect updates. Additionally, some users told us that they were unclear on the content of the survey and the timeline for the project. NISRA told us that it plans to publish a transition roadmap for users as part of wider user engagement strategy for delivering the new survey to make the transition period clearer for users.

Labour Market Survey

In response to ONS’s plans to transform the LFS, NISRA considered its options for producing labour market statistics in Northern Ireland, reviewing the existing LFS survey questions and engaging with users to understand how the LFS data are used. It found that some data collected from the LFS are not used and/or are available from other sources and shared high-level information on survey content with users in its Transformation User Update.

At this time ONS was planning to move forward with its Transformed Labour Force Survey (TLFS) which was intended to be a relatively extensive online survey. However, social survey response rates were declining and NISRA’s view was that the breadth of data collected in the ONS TLFS may lead to response rate and data quality challenges, so it decided to move away from ONS’s approach and develop its own labour market survey.

NISRA has since developed its own online labour market survey which focuses on key labour market topics. NISRA told us that it has worked closely with ONS to develop a ‘must have’ list of key labour market questions to ensure that consistent UK-level labour market estimates can be produced alongside ensuring that NI users’ needs are met and removing content which is not used or available from other sources to shorten the survey. This approach was taken to maximise response rates and the quality of responses.

Whilst we are satisfied NISRA has made an informed decision to use a shorter survey, balancing response rate and data quality considerations with user needs, some users remain concerned about the impact of the changes to the survey and whether the new survey will meet their needs. NISRA has published a question-mapping document on its website and some high level comparisons between the LFS and the LMS, but some users are still not clear which content from the old survey will not be included in the new survey or if removed content will be available from other sources.

User engagement

NISRA regularly engages with users through a range of channels. NISRA has its own user group for labour market statistics. Minutes from this group are published on the website alongside copies of its labour market statistics newsletter. NISRA also engages with users one-to-one and has attended ONS’s user groups on multiple occasions to provide updates for UK-level data users. This engagement includes government departments, academics, the Central Statistics Office (CSO), research institutes and think tanks.

Despite some concerns about project timelines and the content of the survey, users spoke highly of their engagement with NISRA and are confident that NISRA will deliver a high-quality output. Users told us that they were not clear on NISRA’s plans for transitioning to the new survey, both in terms of the timing and the impact that this will have. They told us that they are concerned about the impact this will have on their own analysis and how they can use estimates from the new survey. Some users commented that whilst they had good engagement generally with NISRA, on occasions where they had shared with NISRA their views on what they would like to see from the new labour market survey, users were unsure how their views were used to inform decision making or, if they were not used, why.

Methodology and Data Quality

NISRA’s transformed LMS has had a higher response rate and is representative of the population, with low numbers of incomplete responses. The latest transformation update informs users that the response rate is 60.2%, and 95% of responses are complete responses. Further analysis demonstrates to users that the weighted sample represents the population when broken down by age, region and deprivation level.

NISRA is still performing quality assurance tests on the data it has collected from the new survey. NISRA’s achieved sample from the new survey has exceeded its targets in terms of the response rate, the level of missingness and the representativeness. NISRA is still analysing the differences between the old and the new survey to ensure that it can explain these differences to users when it transitions to the new survey as its main data source. NISRA plans to publish this analysis on completion.

NISRA is still developing the methods it will use for the LMS but expects this to be ready soon. NISRA has shared information with users in its latest transformation update, including how the weighting methods it uses ensures that the achieved sample is representative of the population. NISRA also told us that it is working closely with ONS to ensure that the methods used deliver UK-consistent estimates.

Whilst we recognise that this work is still ongoing, we expect NISRA to publish information for users as soon as it is ready to support the use of the data and statistics.

Quality Criteria

We would have expected NISRA to have communicated the quality criteria it is using to assess the readiness of the new survey and a timeline of when it plans to make key decisions at this stage of the transformation. Users are uncertain about NISRA’s plans for transitioning to the new survey and what criteria NISRA will use to assess the quality of the new survey. NISRA told us it is still developing its measures of success and quality criteria which is disappointing. We expect NISRA to publish this information as soon as possible.

UK-level estimates

NISRA’s redevelopment of its LFS alongside ONS’s redevelopment has led to challenges to ensure that the surveys are designed and processed to produce UK-level estimates. In collaboration with users, NISRA and ONS are working closely together to ensure that cohesive UK-level estimates can be delivered for users both during and after the transformation. NISRA and ONS agreed a ‘must have’ list for aligned UK labour market estimates, which informed the development of the new surveys in Great Britian and Northern Ireland. They are also collaborating on the methods that they will use to generate the estimates.

Transitioning to two new surveys (the LMS and the TLFS) with different content and timelines has presented a range of challenges for both producers. It is likely that NISRA’s transition will occur at a different time to ONS’s, which presents additional challenges in delivering UK-level estimates. Whilst this work is still ongoing, and we have been told that ONS and NISRA have worked together to align outputs and to identify a clear path forward to maintain current publication timeliness, this has not yet been published.

Users told us that UK-level estimates are a priority for most users of labour market statistics in Northern Ireland and across the rest of the UK. Some users need UK-level estimates, and some need to compare Northern Ireland against the rest of the UK. It is vital that NISRA and ONS deliver cohesive UK-level estimates.