UK Statistical System UK Statistics Authority Code of Practice
Articles by: Siobhan

Quality under challenge: Regulating statistics and data from the Labour Force Survey

In our latest blog, Head of Assessment, Siobhan, discusses the challenges of regulating statistics and data from the Labour Force Survey.

Many concerns have been raised about the quality of the Labour Force Survey (LFS) produced by the Office for National Statistics (ONS), the statistics produced from it and the challenges ONS faces in delivering the online Transformed Labour Force Survey (TLFS) that will replace it.  

So, what are we, as the Office for Statistics Regulation (OSR), doing about it? Our key interventions include: 

  • removing the LFS’s status as accredited official statistics to signal quality concerns to users 
  • where there are quality concerns, removing the accreditation of statistics based on LFS data and associated Annual Population Survey (APS) data. To date, we have removed the accreditation from 14 statistical outputs 
  • setting requirements for ONS to improve its communication and engagement, and to consider the lessons that can be learnt from the LFS 
  • reviewing ONS’s work to develop an online replacement for the LFS, the TLFS 

What is the Labour Force Survey?

The LFS is the main household survey providing labour market information in the UK. It is ONS’s largest regular household survey, outside of the census, and has been running since the 1970s.  

Statistics calculated using the LFS (and the related Annual Population Survey, also produced by ONS) are vital to understanding the world we live in. These statistics are used to estimate the number of people employed, unemployed and economically inactive across the UK, in different parts of the UK, and for different groups, such as young people, older people and those with disabilities. These statistics also inform key economic decisions, such as interest rates set by the Bank of England and government tax and spending. 

De-accrediting statistics derived from the Labour Force Survey 

OSR’s primary regulatory role is to independently review statistics to ensure that they comply with the standards of Trustworthiness, Quality and Value in the Code of Practice for Statistics. When they meet these standards, we give them ‘accredited official statistics’ status, and when statistics fall short, we can also remove the accreditation. We may decide on this course of action for several reasons. These could be related to concerns around the quality of the data sources used to produce the statistics, where user need is not being met or where substantial changes to the data sources and methods require us to conduct a review to ensure the quality of the data is such that they continue to be applicable for their intended use.  

As we highlighted in our 2020 assessment, the response rate for the LFS has been steadily declining, and we called on ONS to address this issue and share any relevant information with users. ONS continued to develop its plans for the TLFS, which aims to address the shortcomings of the LFS. 

The long-standing response rate challenges facing the LFS were exacerbated by the COVID-19 pandemic. These issues then became acute when the boost to enable pandemic operations was removed in July 2023. Following this, ONS had to suspend publication of its estimates of UK employment, unemployment and economic inactivity based on LFS data. 

We removed the accreditation from LFS-based estimates and datasets in November 2023. We have also removed the accreditation from other outputs based on data from the APS. The APS is based on responses to wave 1 and wave 5 of the LFS plus a boost sample. 

Monitoring ONS’s work to improve the LFS

When ONS reintroduced LFS-based labour market statistics in February 2024, we carried out a short review of these statistics. In August 2024, we carried out a follow-up review to check the progress made against the requirements set out in our initial report.  

We identified four outstanding requirements, which focus on: 

  • communicating updates on both the LFS and TLFS in one place that users can easily access 
  • improved communication around the uncertainty in the data and what this means for the use of these data 
  • the publication of more-detailed information about the principles and quality criteria that ONS will consider in making further LFS improvements and the transition to the TLFS 
  • the publication of more-detailed information about ONS’s plans for improving the LFS from now until the transition to the TLFS and for transitioning to the TLFS  

We will continue to closely monitor ONS’s work to improve the LFS and plan to report on progress against these requirements next year. 

Reviewing ONS’s work to transform the LFS 

Over the last few years, ONS has been developing a different version of the LFS using an online-first multimode approach (that is, online first, followed by telephone and use of face-to-face interviewers to encourage households to participate). 

Recognising the significance of these statistics for government and economic decision-making, rather than wait to review the final statistical outputs, we have carried out regulatory work throughout their development. The aim of this work is to share early regulatory insights to help ONS in ensuring the new survey meets the standards of Trustworthiness, Quality and Value, in line with the Code of Practice for Statistics.  

We have carried out our review in phases, with each focused on the most relevant elements of the Code. The aim is to assess the statistics produced from the survey against all parts of the Code once the transformed survey is fully implemented.  

The first phase (which started in April 2022) focused on the design and development work ONS had planned before transitioning to the new survey approach. We published our initial findings in November 2022. In July 2023, we published an updated letter and progress report following phase two of our review. We are now entering the third phase of this work, which focuses on ONS’s engagement with users, its communication about its planned work and how it is assessing the quality and readiness of its transformed survey. 

We plan to publish the outcome of phase three of our review of ONS’s LFS transformation in early 2025. 

What’s next? 

A key theme throughout our regulatory work on the LFS and TLFS has been the need for improved communication and clarity, specifically around plans, sources of uncertainty and quality criteria for the transformed LFS. We also called on ONS to identify what lessons can be learnt from the LFS to more effectively and transparently manage and pre-empt quality issues in the future. So, we were pleased to see the comprehensive communication of ONS’s plans and activities in its letter to the Treasury Select Committee and in the update it published on 3 December. We also welcome its engagement through the stakeholder panel and the external methodological input it has sought from Professor Ray Chambers and Professor James Brown.  

We recognise there is still more to do. We will continue urging ONS to provide regular, clear, and comprehensive updates for users, as well as to seek challenge and input from key users and experts to ensure the future production of statistics that meet user needs. 

We are also working to understand to what extent response issues are impacting other household surveys used across the statistics landscape. We have asked ONS to consider whether the issues, concerns, and lessons it has learnt from the LFS apply to its other surveys. 

We will also carry out our own lessons learnt review, focusing on how we can best apply the Code of Practice as a tool to support transformation activities.

What does it mean to be an accredited official statistic?

In our latest blog one of our Head Statistics Regulators discusses what it means to be an accredited official statistic and how official statistics accreditation can help users and producers of statistics…

Here at the OSR we are responsible for carrying out assessments to determine whether a set of statistics can be confirmed as accredited official statistics – a designation previously known as badged as National Statistics.

I often get asked what does it mean to be an accredited official statistic?

The OSR defines accredited official statistics as official statistics that we have independently reviewed and confirmed as complying with the standards of trustworthiness, quality and value in the Code of Practice for Statistics (the Code).

For me, accreditation is a shortcut. It’s a quick way of signalling to users of the statistics that the standards of the Code have been met. It’s similar to a quality mark, but what quality is being assessed: that of the processes to produce the statistics or that of the statistics themselves?

“It’s more than just quality”

The Code encompasses much more than quality. Accreditation implies not only the good quality of the statistics themselves but also that they are presented, quality assured and disseminated according to set standards. It is about the value of the statistics for users and whether they are robust enough to bear the weight of decision-making required of them. Thus, accreditation considers both the processes and the statistics themselves and the structures, people and organisations, that support statistics planning, production and communication.

“Context and use matter”

When we assess the quality of the statistics themselves we are not looking for them to meet a ‘gold standard’.  We recognise that any statistic is only ever a best estimate at a particular point in time. It also depends on context and whether the statistics are good enough for their intended use, which will vary according to user and societal need. For example more-timely but less-accurate data (due to gaps in data sources to ensure timeliness) may be acceptable in one context but would not be in another. It can take two years after the reference period for a relatively comprehensive picture of the economy and so GDP, to emerge. However, more-timely GDP statistics are needed to inform policy, budgeting, investment and employment decisions in the public and private sectors.

However, what we do require is for producers of statistics to ensure that they are producing the most appropriate estimate available by ensuring suitable data sources are used, methods are robust and estimates are quality assured. This work should be carried out with the context and use of the statistics in mind and in an open, professional and transparent way, by making clear any limitations to the data, inherent uncertainty due to the timeliness of the data, or planned revisions so users can use the statistics appropriately for their needs. More detail on our approach to quality is provided in our publication Quality and statistics: An OSR perspective.

What if the statistics are not accredited?

If statistics are not accredited, it doesn’t mean that they aren’t trustworthy, of a high quality or valuable. It also doesn’t mean that they are. What it does mean though is that we haven’t independently checked that they comply with the standards of trustworthiness, quality and value in the Code. Being a chartered naval architect or having a plumbing and heating qualification signals that someone external has checked you can do something to a particular standard, e.g. build and design ships or install and maintain boilers and heating appliances. However, that doesn’t mean someone without official qualifications can’t also do the same things. You would just want to look for more evidence that they can, e.g. references, evidence they understand the relevant standards and rules they should be following etc.

We encourage all users of statistics, to ask themselves some questions to ensure the data are fit for their purposes. These include considering where the data has come from, why has it been collected, how well the data fits with the concept you are trying to measure, what checks have been carried out to assure the data, how can you access the data. Some more detail on things to consider are set out in our guidance on questions for data users.

What if the accreditation is removed?

Legally only we (the UK Statistics Authority) can remove the badge, i.e. the accreditation, from a set of statistics. We may decide on this course of action for a number of reasons. These could be related to concerns around the quality of the data sources used to produce the statistics, where user need is not being met or where substantial changes to the data sources and methods require us to conduct a review to ensure the quality of the data is such that they continue to be applicable for their intended use. The reason(s) should be included in the release and/or the announcement explaining why the accreditation has been removed.

What if some of the input statistics are not accredited?

Different sets of statistics often feed through into others. For example, migration data are used to inform population estimates and projections. Data from the labour force survey feed into productivity estimates. Producers of data and statistics should always quality-assure their data sources and be aware of any changes to them. The extent of quality assurance required and the weight placed on different data sources will vary depending on factors such as how much they affect the overall calculation, or whether there are any alternatives. We would expect this information to be communicated to users so they can understand the quality-assurance processes carried out and why the producer has decided that the data sources are fit for use. This is the case regardless of whether the source data are accredited or non-accredited.

How do I get my statistics accredited?

If you are a government department of official body that produces official statistics, and you have a set of statistics that you consider meet the standards of trustworthiness, quality and value in the Code then you can ask us to assess them. The benefits of doing so include:

  • An independent assessment of the processes, methods and outputs used to produce the statistics against the recognised standards of the Code.
  • Public demonstration of your organisation’s commitment to trustworthiness, quality and public value.

Your first step towards assessment is to talk with your Head of Profession for Statistics who can provide guidance and points to consider.

If you are producer of data, statistics and analysis which are not official statistics, whether inside government or beyond you can contact us to discuss voluntarily application of the Code. While this approach will not lead to accredited official statistics it is a public demonstration of your commitment to the standards of the Code, which many organisations find beneficial for their work.

How do I find out more?

Related reading:

Futureproofing the Code of Practice for Statistics: findings and next steps from our review

National Statistics designation review

What is intelligent transparency and how you can help?

Statistics Regulator Siobhan Tuohy-Smith discusses what we mean by intelligent transparency and how you can be an advocate for intelligent transparency across government and official data, statistics and wider analysis.

So what is intelligent transparency?

Everyone, I think, has a fairly clear idea of what transparency means. It means being open or clear – getting the data or statistics out there. But what do we mean when we talk about intelligent transparency?  

At its heart intelligent transparency is about proactively taking an open, clear and accessible approach to the release and use of data, statistics and wider analysis. As set out in our regulatory guidance on transparency, intelligent transparency is informed by three core principles: equality of access, enhancing understanding and analytical leadership. It’s about more than just getting the data out there. Intelligent transparency is about thinking about transparency from the outset of policy development, getting data and statistics out at the right time to support thinking and decisions on an issue, supporting the wider public need for information and presenting the data and statistics in a way that aids understanding and prevents misinterpretation. For example, the Welsh Government Chief Statistician posted a blog on understanding COVID-19 infection rates in Wales on 11 January 2022. This post went beyond just getting the data out there, by also aiding user understanding of the data to avoid misinterpretation. 

Why is transparency important?

For me, transparency is a key part of what we do at the Office for Statistics Regulation (OSR). It’s a theme the runs throughout the Code of Practice for Statistics, from practice Q2.3 about transparency about the methods used, to V2.1 about ensuring free and equal access to regular and ad hoc official statistics, to principle T3 about orderly release, to name but a few.  

Transparency is also a core part of ensuring data, statistics and wider analysis serve the public good. Only by seeing the numbers, and understanding where they came from, can we really understand what they mean and how to use them to best inform individual decisions and understanding of an issue. Individual decisions about where and when to buy a new house, mortgage decisions, and what school to send your child to. Or public understanding about COVID-19 infection rates or a new policy around climate change.   

We highlighted the need for intelligent transparency as a key theme in our recent State of the Statistics System report and it continues to recur as a theme in our casework.

What can you do to support intelligent transparency?

In OSR, we continue to champion intelligent transparency and equal access to data, statistics and wider analysis. We: 

  • Are building our evidence base, highlighting good examples and understanding more about barriers to transparency to help support those working across government to implement intelligent transparency. Today we have published some FAQs about intelligent transparency to help support this work. 
  • Engage with analysts, policy-makers and the communications function across government, and interested parties outside of government to advocate intelligent transparency and develop networks committed to intelligent transparency. 

But we recognise that this is not something we can do alone. We need your help! 

So what can you do: 

You can be an advocate for intelligent transparency across government and official data, statistics and wider analysis: 

  • As a user of this data, you can continue to question what you see and ask yourself does it make sense? Do you know where it comes from? Is it being used appropriately?  
  • If you are based in a department or a public body, you can champion intelligent transparency in your team, your department and your individual work. Build networks to promote our intelligent transparency guidance across all colleagues and senior leaders in your organisation. You can engage with users to understand what information it is they need to inform their work to inform the case for publishing it; get in touch with your Head of Profession or OSR if you experience issues publishing statistics, data or wider analysis of significant public value or interest 

You can get in touch with us via regulation@statistics.gov.uk if you are interested in working with us on intelligent transparency. You can also keep up to date with our work via our newsletter.  

You can raise concerns with us via regulation@statistics.gov.uk – our FAQs about how to raise an issue set out what to expect if you raise a concern with us.    

Looking ahead

We will continue to champion intelligent transparency and with your help, together, we can help intelligent transparency become the default for all government data, statistics and wider analysis.

 

Related Links:

Regulatory guidance for the transparent release and use of statistics

Intelligent Transparency FAQs

Why migration statistics matter

Like many out there I wake up every morning hoping that a way, that protects life, has been found to bring peace in Ukraine. As it says in one of my young daughter’s books, “The world’s already far too full of cuts and burns and bumps”[1].

Unfortunately, the conflict continues and people from Ukraine are fleeing. On 10 March 2022, the UN Refugee Agency (UNHCR) estimates that just over 2.3 million people have fled Ukraine since 24 February 2022. Across the UK there has been an outpouring of public sympathy for Ukrainian people forced to flee. The Government has introduced some new visa routes for Ukrainians; and debate continues among the public, in the media and in Parliament about whether the UK is doing enough to help.

As with any crisis, lots of decisions will need to be made. Decisions by individuals, by governments and by agencies and by organisations helping to support people flee Ukraine and build new lives. Data and statistics are a key part of this decision-making process. For example, to inform local and national emergency response planning, the Office for National Statistics (ONS) has published new data about the number of Ukrainian nationals by local authority and the Home Office has published the number of people applying for these new Ukrainian visa routes.

Here at the Office for Statistics Regulation (OSR) I lead on OSR’s work on migration. At the heart of my role is ensuring that data and statistics serve the public good. What does this mean in this context? It means ensuring that the best possible data are available to inform decision-making. And it also means ensuring data are publicly available to help the public understand the impact of decisions made, for example to evaluate the impact of new visa routes for Ukrainians and the impact this has on the make-up of society in the UK.

Earlier this month, we published the first in a series of reports looking at how the Office for National Statistics (ONS) is transforming the way it measures international migration. These statistics provide estimates of how many people are flowing into and out of the country from across the world and what the impact is on the number of migrants in the UK. The previous methods, based on the International Passenger Survey (IPS), had limitations so it’s great to see new robust methods being developed in a credible way and in discussion with expert users. Our report welcomes the ambitious and collaborative approach being taken by the ONS to transform the way it measures international migration and recommends some ways ONS can build on this good work. I would like to thank all those we have engaged with us as part of this work for their openness and time. I look forward to continuing this work to ensure that the transformed migration statistics are trustworthy, high quality and support society’s needs for information.

More widely we also engage with other government bodies responsible for the production and publication of statistics and data on migration. For example, we regularly engage with the Home Office, which is responsible for publishing a wide range of statistics about migrants. We have recently written to the Home Office about the publication of data on migrants arriving in Small Boats. In our letter we welcomed the Department’s plans to regularly publish additional data about this topic.

At the OSR we want our work to have an impact. That means ensuring that data and statistics are there to inform decision-making across society, the public, private and third sectors and to help hold organisations to account. This is at the heart of what I do as a statistical regulator at the OSR and at the core of our migration work. I just hope in a small way this can have a positive impact on what is happening out there in the world today.

If you would like to feed into any of our work on migration statistics please get in touch with Siobhan Tuohy-Smith.

[1] Donaldson J & Scheffler A, 2010, Zog, Published in the UK by Alison Green Books.