State of the Statistical System 2022/23

28 June 2023
Last updated:
28 June 2023

Continuing Innovation and Responsiveness


The statistical system is continuing to innovate and respond to emerging issues; however, this pace is difficult to maintain given the current pressure on resources and funding

Our view

  • There continues to be a significant shift in government and public demand for statistics and data from COVID-19 to other key issues, such as the war in Ukraine and the rising cost of living.
  • The statistics and data landscape is becoming increasingly complex, with greater amounts of data being made available by government departments. The statistical system has demonstrated its responsiveness to meet many of these data needs in new and innovative ways.
  • In a wider landscape of technological advances, statistics need to remain relevant, accurate and reliable. Producers are continuing to innovate and are looking to new data sources such as administrative data to support them in this.
  • We are seeing good examples of innovation, efficiency and collaboration across the statistical system, but this is not consistent.
  • Against a backdrop of financial, resourcing and capability pressures, the statistical system is facing continued challenges to keep pace with the demand for statistics and data. The direct impact on the statistical system of these pressures is currently unknown. We caution that this pace is likely not sustainable with current resources and funding, especially if additional priorities emerge.

The production of COVID-19 statistics has continued to reduce throughout the past year. The COVID-19 Infection Survey, first published in May 2020 by the ONS, transitioned from the use of in-person data collection, to a digital questionnaire and postal kits in June 2022, and was paused in 31 March 2023. In May 2023, ONS launched a new UK-wide interim survey to monitor COVID-19 and other respiratory infections. In addition, we have seen management information first published during the pandemic being halted or reduced in frequency, such as the domestic transport use by mode statistics produced by the Department for Transport which moved from a weekly to a monthly publication in December 2022, and the COVID-19 statistics dashboard for Northern Ireland which was stopped in May 2022.

Since Russia’s invasion of Ukraine in February 2022, the statistical system has continued to respond to the need for statistics on the social and economic impact in the UK. Examples of statistics published to meet new user needs include the number of visa holders entering the UK under the Ukraine humanitarian schemes, experiences of Homes for Ukraine scheme sponsors, and school placements for children from outside of the UK.

The rising cost of living is a continued area of focus for governments and the wider public. In April 2022, the Royal Statistical Society (RSS) wrote to the National Statistician Sir Ian Diamond setting out the need for inflation measures to “capture the true costs for all households”. The ONS has developed experimental measures of inflation using new data sources, including scanner and web-scraped data, to better reflect the lived experience of households. In addition, the ONS has released new analysis on the impact of the rising cost of living, including the introduction of the Cost of Living latest insights tool and analysis on the rising costs of everyday food.

The statistical landscape is becoming increasingly complex with a greater amount of data being made available by government departments, including alternative sources such as management information being more commonly available.

In a wider landscape of technological advances, statistics need to remain relevant, accurate and reliable. Over the last year, producers are continuing to innovate and improve statistics and looking to new data sources to support them in this.

Administrative data are increasingly being used as a data source for statistics and are helping to provide new insights and improve the quality of statistics. An example of this is the Department for Work and Pensions who are exploring the integration of administrative data into the Family Resources Survey (FRS) and related outputs through its FRS Administrative Data Transformation Project.

Statistical methods are becoming increasingly complex, with more use of data science and statistical models in the production of official statistics. As highlighted in our Guidance for Models, it is crucial that producers ensure that any development of models is explainable, and interpretable to meet the transparency requirements of the Code of Practice for Statistics (the Code).

An example of the use of statistical modelling is ONS’s development of a dynamic population model (DPM) to transform its population statistics. The DPM uses statistical modelling techniques to combine a range of administrative and survey data sources to estimate the population and population change. The aim of the DPM is to produce more timely, coherent, and higher quality population estimates than the current mid-year estimation approach. To date ONS has published provisional estimates for England and Wales at local authority level. To support public confidence and trust in the provisional estimates, ONS is publishing technical papers as it develops the model. The transformation of population statistics is a priority area for the Office for Statistics Regulation (OSR), and we will continue to monitor the development of the DPM and carry out appropriate regulatory work in this area this year.

The increasing use of new and alternative data sources and advances in technology are opportunities for the statistical system to embrace. Advancement and use of technology currently varies across the system. The role of a statistician is evolving and needs to keep pace with the increasing use of data science techniques. There are increasingly skills overlaps with the role of a data scientists, and producers need to keep pace with these skill changes.

There is increasing awareness and implementation of Reproducible Analytical Pipeline (RAP) principles. These encourage greater automation of end-to-end processes and support the efficient and sustainable production of statistics and the highest standards of the Code of Practice. Removing manual elements of statistical production enables producers to free up considerable resources, as well as enhance the quality of data processing and reduce the risk of human error.

We continue to advocate for RAP principles to be the default in statistical production. We recognise that embedding RAP principles within an organisation or government department requires access to the right tools and training and statisticians having the time and support to carry out development work.

The Government Analysis Function Reproducible Analytical Pipeline (RAP) strategy, outlines the ambition across government to improve efficiency, use digital technology, increase trust and improve business continuity. Some government departments have published plans setting out their commitment to produce better quality and more efficient outputs through the adoption of RAP principles, with varying degrees of maturity across different organisations. For example, the quarterly Welsh language statistics from the Annual Population Survey are now produced using RAP which has introduced a well-defined QA process, clearer methodology and removed unnecessary steps. This has reduced the time taken to complete the process from three to four days to one hour. Social Security Scotland are also seeking ways to implement RAP, and as a first step has automated the production of statistical tables, saving time and reducing the risk of any manual errors.

The UK statistical system continues to explore ways to use data to improve insight. This involves collaboration across government departments, but also with organisations outside of government such as the RSS, Research Data Scotland, Administrative Data Research UK (ADR UK), private companies, and academics. These partnerships are a beneficial way to maximise the availability of resources as well as make use of expert knowledge.

There have been significant improvements in data sharing and linkage over the past year, enabling outputs such as the ONS’s statistics on sociodemographic inequalities in suicides which show estimates for rates of suicide across different demographic groups for the first time. In addition, there are strong examples of collaboration with external organisations including the ONS’s publication of UK spending on credit and debit cards, the independent Ulster University Economic Policy Centre in Northern Ireland, which is partially funded by the Department for the Economy and the Department of Finance, and the Ministry of Justice (MoJ) Data First programme which is funded by ADR UK. MoJ published its areas of research interest in 2020 providing a summary of departmental evidence priorities over a three-to-five-year period to strengthen and maximise collaboration with academic experts and research funders. Proactive engagement meant that MoJ was able to create research partnerships and secure funding to extend its data-linking project, DataFirst.

However, data sharing and linkage are still not embedded consistently across government. There are many areas that would benefit, for example energy support policies, which have been highlighted by the Institute for Government. We understand that significant barriers still remain for data sharing and linkage including prioritisation by leaders, skills and retention, data protection issues, and legal processes. Our upcoming Data Sharing and Linkage report, due to be published in July, will explore these issues in more detail. While the benefits of data sharing and linkage may seem relatively small from an individual department perspective, the benefits for the system as a whole are wide reaching.

A number of producers told us that they are facing continued challenges with resourcing. We heard about the challenges of high vacancy rates, often as a result of recruitment freezes and falling retention rates. We also heard of the difficulties of getting the right skilled people in post, especially as technology evolves and statisticians are required to have more technical skills. It is important to note that these challenges are not being consistently felt across the whole UK statistical system.

The Autumn Statement of 2022 confirmed that UK government departments need to maintain committed budgets for the remaining two years of the Spending Review. With rising costs, it is likely that funding pressures will continue in the year ahead and the statistical system in the UK therefore needs to be efficient.

The ONS is the national statistics institute and much of the Government Statistical System (GSS) support sits within the organisation. We heard mixed views from statistics producers about the role of the GSS now that the profession is part of the broader Government Analysis Function.

Some producers felt that it was a positive development that some of the central support, previously provided exclusively to the GSS by ONS, has now been expanded under the Analysis Function to support data and analysis professionals more widely. An example shared with us is the GSS Quality Centre which has been replaced by the Data Quality Hub, which provides guidance and support on data more broadly than statistics. The Analysis Function told us that the broadening of some of these support mechanisms is intended to improve statistical quality across professions and ensure a holistic offer to all analysts. We also heard opposing views which voiced concern about the potential devaluing of the GSS, and hence the specialised skills of statisticians, and more general negative impacts arising from the loss of the GSS website brand.  Some of these views reflected that, as many of these changes are relatively new, the impact is not yet widely understood.


Why is this important and what is the impact?

  • The increasing availability of data and the growing use of artificial intelligence should be seen as an opportunity for the statistical system. The role of statisticians and the value of official statistics outputs needs to keep pace in the landscape of new artificial intelligence tools such as ChatGPT and other large language models.
  • Statistical methods are becoming increasingly complex, with more use of data science and statistical models in the production of official statistics. To maintain public trust and confidence, producers must be transparent as they develop new methods.
  • The innovative approaches are inconsistent across government departments. Greater collaboration across government and between government and external organisations will support consistency.
  • There is a concerning risk that continued financial and resource pressures will hinder future progress and evolution of the system to keep pace with increasing demand. A successful statistical system depends on having a workforce that is sufficiently resourced and skilled to deliver.
  • Producers are pointing to the need for more prioritisation and redeployment of people in the immediate term, in addition to trialling a greater use of apprenticeships and analysts who are not aligned to a specific profession. While these are good short-term solutions, the difficulties around recruitment and retention, which have been building for a number of years, will likely have considerable knock-on effects in the years to come if they cannot be addressed in a systematic way across government.

What do we want to see from the statistical system and what are we doing to support this?

  • Our guidance for models, which builds on our 2020 review of the statistical models developed for awarding grades, can help producers in the design, development and use of models. It also highlights the importance of establishing public confidence and trust in the development of new methods and models.
  • The role of a statistician is evolving with more use of complex methodologies and data science techniques. Statistics producers need to keep pace with these skill changes. It is important that statisticians are supported in their roles and in professional development. The GSS and the Government Analysis Function have an important role in this.
  • The direct impact on the statistical system of resourcing and funding pressures is currently unknown. We have not been able to fully quantify and validate the challenges we heard about. As an example, staffing data are not publicly available for the whole UK statistical system, as highlighted in our Statistical Leadership report in 2021, this is a data gap. It is important to collect and share this information so that the scale of the challenges can be assessed and effectively resolved.
  • In the short term, government departments should be exploring alternative recruitment options such as apprenticeships. In the long term, the system must find ways to train and retain high quality staff. We are exploring how, as a regulator, we can assist producers who are facing resource pressures and we set out some initial views on this in our November 2022 blog. We are currently working on additional guidance to help producers prioritise and make these decisions.
  • We encourage increased collaboration across the UK statistical system to make the most efficient use of data and people in a joined-up way and to share learning across the system on improving efficiency.
  • Producers should look to make more use of cross-government capability by offering more opportunities for statisticians to apply for loans, secondments or joint project work.
Back to top
Download PDF version (285.81 KB)