The production of COVID-19 statistics has continued to reduce throughout the past year. The COVID-19 Infection Survey, first published in May 2020 by the ONS, transitioned from the use of in-person data collection, to a digital questionnaire and postal kits in June 2022, and was paused in 31 March 2023. In May 2023, ONS launched a new UK-wide interim survey to monitor COVID-19 and other respiratory infections. In addition, we have seen management information first published during the pandemic being halted or reduced in frequency, such as the domestic transport use by mode statistics produced by the Department for Transport which moved from a weekly to a monthly publication in December 2022, and the COVID-19 statistics dashboard for Northern Ireland which was stopped in May 2022.

Since Russia’s invasion of Ukraine in February 2022, the statistical system has continued to respond to the need for statistics on the social and economic impact in the UK. Examples of statistics published to meet new user needs include the number of visa holders entering the UK under the Ukraine humanitarian schemes, experiences of Homes for Ukraine scheme sponsors, and school placements for children from outside of the UK.

The rising cost of living is a continued area of focus for governments and the wider public. In April 2022, the Royal Statistical Society (RSS) wrote to the National Statistician Sir Ian Diamond setting out the need for inflation measures to “capture the true costs for all households”. The ONS has developed experimental measures of inflation using new data sources, including scanner and web-scraped data, to better reflect the lived experience of households. In addition, the ONS has released new analysis on the impact of the rising cost of living, including the introduction of the Cost of Living latest insights tool and analysis on the rising costs of everyday food.

The statistical landscape is becoming increasingly complex with a greater amount of data being made available by government departments, including alternative sources such as management information being more commonly available.

In a wider landscape of technological advances, statistics need to remain relevant, accurate and reliable. Over the last year, producers are continuing to innovate and improve statistics and looking to new data sources to support them in this.

Administrative data are increasingly being used as a data source for statistics and are helping to provide new insights and improve the quality of statistics. An example of this is the Department for Work and Pensions who are exploring the integration of administrative data into the Family Resources Survey (FRS) and related outputs through its FRS Administrative Data Transformation Project.

Statistical methods are becoming increasingly complex, with more use of data science and statistical models in the production of official statistics. As highlighted in our Guidance for Models, it is crucial that producers ensure that any development of models is explainable, and interpretable to meet the transparency requirements of the Code of Practice for Statistics (the Code).

An example of the use of statistical modelling is ONS’s development of a dynamic population model (DPM) to transform its population statistics. The DPM uses statistical modelling techniques to combine a range of administrative and survey data sources to estimate the population and population change. The aim of the DPM is to produce more timely, coherent, and higher quality population estimates than the current mid-year estimation approach. To date ONS has published provisional estimates for England and Wales at local authority level. To support public confidence and trust in the provisional estimates, ONS is publishing technical papers as it develops the model. The transformation of population statistics is a priority area for the Office for Statistics Regulation (OSR), and we will continue to monitor the development of the DPM and carry out appropriate regulatory work in this area this year.

The increasing use of new and alternative data sources and advances in technology are opportunities for the statistical system to embrace. Advancement and use of technology currently varies across the system. The role of a statistician is evolving and needs to keep pace with the increasing use of data science techniques. There are increasingly skills overlaps with the role of a data scientists, and producers need to keep pace with these skill changes.

There is increasing awareness and implementation of Reproducible Analytical Pipeline (RAP) principles. These encourage greater automation of end-to-end processes and support the efficient and sustainable production of statistics and the highest standards of the Code of Practice. Removing manual elements of statistical production enables producers to free up considerable resources, as well as enhance the quality of data processing and reduce the risk of human error.

We continue to advocate for RAP principles to be the default in statistical production. We recognise that embedding RAP principles within an organisation or government department requires access to the right tools and training and statisticians having the time and support to carry out development work.

The Government Analysis Function Reproducible Analytical Pipeline (RAP) strategy, outlines the ambition across government to improve efficiency, use digital technology, increase trust and improve business continuity. Some government departments have published plans setting out their commitment to produce better quality and more efficient outputs through the adoption of RAP principles, with varying degrees of maturity across different organisations. For example, the quarterly Welsh language statistics from the Annual Population Survey are now produced using RAP which has introduced a well-defined QA process, clearer methodology and removed unnecessary steps. This has reduced the time taken to complete the process from three to four days to one hour. Social Security Scotland are also seeking ways to implement RAP, and as a first step has automated the production of statistical tables, saving time and reducing the risk of any manual errors.

The UK statistical system continues to explore ways to use data to improve insight. This involves collaboration across government departments, but also with organisations outside of government such as the RSS, Research Data Scotland, Administrative Data Research UK (ADR UK), private companies, and academics. These partnerships are a beneficial way to maximise the availability of resources as well as make use of expert knowledge.

There have been significant improvements in data sharing and linkage over the past year, enabling outputs such as the ONS’s statistics on sociodemographic inequalities in suicides which show estimates for rates of suicide across different demographic groups for the first time. In addition, there are strong examples of collaboration with external organisations including the ONS’s publication of UK spending on credit and debit cards, the independent Ulster University Economic Policy Centre in Northern Ireland, which is partially funded by the Department for the Economy and the Department of Finance, and the Ministry of Justice (MoJ) Data First programme which is funded by ADR UK. MoJ published its areas of research interest in 2020 providing a summary of departmental evidence priorities over a three-to-five-year period to strengthen and maximise collaboration with academic experts and research funders. Proactive engagement meant that MoJ was able to create research partnerships and secure funding to extend its data-linking project, DataFirst.

However, data sharing and linkage are still not embedded consistently across government. There are many areas that would benefit, for example energy support policies, which have been highlighted by the Institute for Government. We understand that significant barriers still remain for data sharing and linkage including prioritisation by leaders, skills and retention, data protection issues, and legal processes. Our upcoming Data Sharing and Linkage report, due to be published in July, will explore these issues in more detail. While the benefits of data sharing and linkage may seem relatively small from an individual department perspective, the benefits for the system as a whole are wide reaching.

A number of producers told us that they are facing continued challenges with resourcing. We heard about the challenges of high vacancy rates, often as a result of recruitment freezes and falling retention rates. We also heard of the difficulties of getting the right skilled people in post, especially as technology evolves and statisticians are required to have more technical skills. It is important to note that these challenges are not being consistently felt across the whole UK statistical system.

The Autumn Statement of 2022 confirmed that UK government departments need to maintain committed budgets for the remaining two years of the Spending Review. With rising costs, it is likely that funding pressures will continue in the year ahead and the statistical system in the UK therefore needs to be efficient.

The ONS is the national statistics institute and much of the Government Statistical System (GSS) support sits within the organisation. We heard mixed views from statistics producers about the role of the GSS now that the profession is part of the broader Government Analysis Function.

Some producers felt that it was a positive development that some of the central support, previously provided exclusively to the GSS by ONS, has now been expanded under the Analysis Function to support data and analysis professionals more widely. An example shared with us is the GSS Quality Centre which has been replaced by the Data Quality Hub, which provides guidance and support on data more broadly than statistics. The Analysis Function told us that the broadening of some of these support mechanisms is intended to improve statistical quality across professions and ensure a holistic offer to all analysts. We also heard opposing views which voiced concern about the potential devaluing of the GSS, and hence the specialised skills of statisticians, and more general negative impacts arising from the loss of the GSS website brand.  Some of these views reflected that, as many of these changes are relatively new, the impact is not yet widely understood.