Other system-wide issues

This section examines the system-wide topics of data sharing and linkage, artificial intelligence and UK-wide comparability.

Data sharing and data linkage

Last year’s report highlighted the need for overcoming cultural and technical barriers to data sharing and linkage. The data sharing, access and linkage landscape continues to be marked with welcome pockets of innovation and progress. Nonetheless, the overall story of data sharing and linkage remains one of individual successes, rather than a coherent and consistent delivery of improvements in access to, and joining up of, government data. This, alongside an ongoing lack of strategic direction and resource allocation, is leading to a failure to realise data’s full potential to fuel policy innovation and socio-economic development. Nonetheless, there are many parts of the statistical system with ambition for this important area of innovation and development. We recognise that comprehensive, safe and sustained data sharing and linkage for the public good cannot be achieved by statistics producers alone and requires other partners.

Many of the challenges and barriers which have limited recent progress in this area were highlighted in our follow-up report, Data Sharing and Linkage for the Public Good, which was published in July 2024. A year on, the issues remain largely the same.

At the heart of these challenges are the perceived risks of sharing data for data owners, which can seem greater than the perceived benefits which may accrue to other departments or policy makers beyond the statistical system. As a result of this imbalance, despite positive statements about the importance of sharing data from ministers and senior civil servants, there can be inertia and a lack of strategic leadership, inhibiting further progress. The inactivity in this area is compounded by the practical challenges around the process, technical and funding sides of sharing. The positive examples that we do see are often resource-intensive to achieve.

These challenges may have been a factor in the ONS’s Integrated Data Service (IDS) not being able to fully deliver on its ambition to dramatically increase the number of linked datasets available to analysts across government and accredited researchers outside of government. In the ONS Strategic Business Plan: April 2025 to March 2026, ONS set out its prioritisation of resources with a greater emphasis on core economic and population statistics. This change has an impact on resources for the IDS. Our understanding is that the IDS will no longer support government-wide analysis, but will continue to support ONS’s production of statistics. As these changes take place, we will focus our attention on ensuring continued access for external researchers through the Secure Research Service. Making linked data available in a secure way to external researchers is one of the most significant ways in which the ONS can continue to support improved data sharing and linkage.

Despite this mixed strategic landscape, the potential for significant progress remains, and ambitious examples of data sharing and powerful use cases have provided important insight over the last year:

• The Ministry of Justice (MoJ)’s cross-government Better Outcomes through Linked Data (BOLD) programme continues to demonstrate how system-level developments can directly improve outcomes for the public, with recent advances in sharing data for linked analysis to inform evidence in multiple departments. This includes the Department of Health and Social Care beginning to publish new drug and alcohol treatments for victims and suspects of homicide statistics in collaboration with the Home Office.
• The ONS Research Excellence Awards 2025 offer examples of impactful work made possible by sharing and linking data, including through the ONS’s Integrated Data Service (IDS).
• In March 2024 the Department for Work and Pensions (DWP) published initial illustrative data in its Family Resources Survey Transformation: integrating administrative data into the FRS publication to demonstrate the value of linking to DWP benefits administrative data.
• The DWP, the Department for Health and Social Care, NHS England and ONS have also been bringing together labour market, census and health data to understand the impact of health on the economy; resulting publications include The impact of bariatric surgery on monthly employee pay and employee status, England and The impact of NHS Talking Therapies on monthly employee pay and employment status, England.
• The Home Office is making continued progress in data sharing with ONS to collaborate on new long-term migration statistics.
• Some departments are also making welcome improvements to their data sharing services, such as by reducing the time it takes to access their data. The Department for Education (DfE) reduced the average turnaround time for data requests by 32% over 2024, despite receiving and processing a 36% increase in applications via the DfE Data Sharing Service.

More broadly, there is growing recognition of the need for the coordination of processes and sharing best practice. The Data Sharing Network of Experts (DSNE) and the Data Linkage Champion Network continue to be positive initiatives for knowledge sharing and problem solving, demonstrating the importance of a community-wide approach to overcoming barriers. The new GSS strategic vision has also set out the case for “increased leadership to influence data sharing”, one outcome of which has already been a knowledge sharing network for methodological expertise among Heads of Profession and their departments. The Scottish Government has used its experience of data sharing partnerships with UK Government departments to share best practice and build new technical solutions to transfer data securely from DWP and HMRC, with a focus on eradicating child poverty. The case for optimism was made in OSR’s post on the growing bottom-up drive to make better use of datasets in specific policy areas.

Beyond central government, the number of use cases showing the real-world impact of access to data and linked datasets has notably increased. The 2023/24 ADR UK impact report, published in September 2024, shows how the analysis of linked data has fed directly into policy development across multiple thematic areas, demonstrating the benefits of extending the reach of data beyond government to academic research communities.

Of note, ADR Wales led the development of a reproducible analytical pipeline for transferring data from the Welsh Government into the SAIL Databank. This innovation has streamlined data preparation and transfer processes, resulting in the availability of higher-quality administrative data for research purposes. In Northern Ireland, the Longitudinal Education Outcomes initiative with ADR NI continues to make new data available for research through its linked, de-identified database.

The research community’s approach to data sharing initiatives continues to benefit from the work of the Public Engagement in Data Research Initiative on social licence. This increasing maturity and UK-wide facilitation has been reflected in UK Government programmes, such as the MoJ’s Data First.

Looking to the future, the Department for Science, Innovation and Technology is working on a National Data Library, which it hopes can provide a central coordinating mechanism for data sharing and access, while the government’s missions and the Plan for Change provide a powerful user need for data that are joined up across departments.

Recent events like OSR’s May 2025 seminar for government analysts on data linkage demonstrated the ongoing need for education and positive use case proliferation to spread understanding about data sharing, access and linkage beyond the statistical system and across all areas of government. In addition to the statisticians, there needs to be more widespread recognition that analysts and chief data officers in data-holding organisations have crucial roles to play in facilitating and embedding effective and safe sharing. The UK Statistics Assembly identified the need to accelerate and improve the capacity for the linkage of data as a common theme of feedback from the statistics community.

Insightful analysis can be undertaken when cultural and technical barriers to data sharing, access and linkage are overcome, driving significant and impactful improvements to people’s lives and society from both within and outside government. Sharing and linking data provides greater insights into society and the economy, enabling policy development and stimulating innovation, and is vital for socio-economic development.

A key ambition of NISRA’s Corporate Plan 2025-2029 is to work with data owners to mobilise more administrative data and develop new linked datasets that support joined-up policy making. A new cross-departmental data forum is being established to advance cross-government analysis, consider a new Data Strategy for the Northern Ireland Civil Service which outlines future data mobilisation needs, and recommend the legislative changes needed to deliver this vision.

We want to see partners, both within the statistical system and beyond, continuing to review and implement the recommendations set out in our 2024 Data Sharing and Linkage for the Public Good: Follow-Up Report. We recognise that comprehensive, safe and sustained data sharing and linkage for the public good cannot be achieved by statistics producers alone but requires strategic leadership and the buy-in of the wider data ecosystem. Through our regulatory and systemic work, we will continue to advocate for the six ‘enablers for change’ outlined in our publication How government can make more data available for research and statistics in 2025. Our commitment is also reflected in the proposals for our revised Code of Practice for Statistics, which contains a new practice relating to social acceptability for data and statistics, alongside practices relating to enabling data re-use and linkage. OSR will continue to seek opportunities to influence, contribute to and collaborate with partners within and outside government to enhance the public good of data and statistics through enhanced data sharing, access and linkage. We want to minimise missed opportunities for data use and ensure the benefits are fully realised, while upholding public confidence in secure and ethical data sharing.

Artificial intelligence (AI)

There continues to be considerable discussion about the potential and realised use of AI in producing and communicating official statistics. This is especially true at the international level. The recent UNECE Generative AI and Official Statistics Workshop showcased a wide range of examples of national statistics offices using AI, including to make official statistics more discoverable and accessible, to improve the efficiency of statistical report writing, to improve survey data quality and for automatic coding. Within our own statistical system, OSR is starting to hear of producers exploring specific use cases of AI, such as to enhance the quality of survey response classification, and to support summarising and drafting text.

It is positive to see AI starting to be used in official statistics; however, there are still risks to using AI, which could impact on the trustworthiness, quality and value of official statistics. If not effectively mitigated, these risks (as explored in our 2024 blog) will impact the ability of official statistics to deliver public good and to inspire public confidence.

Many organisations have developed guidance on how AI should be used, which statistical producers should use to guide their own use of AI. Of particular significance are the AI Playbook for the UK Government, which offers guidance on using AI safely, effectively and securely for people working in government organisations; a white paper, Large Language Models in Official Statistics, published by the UNECE; and the Guidance on the Impact Evaluation of AI Interventions, published by the Evaluation Task Force. Many government organisations also have their own internal AI principles and policies.

What has not yet emerged is a single overall view of how much AI is currently being used, or could be used, across the statistical system, or clear leadership to translate and implement existing AI guidance for use within the UK statistical system. It is encouraging to see the ambition to implement a focus on the use of AI in statistics in the new GSS strategic vision, which was published in 2024. Having a stronger network across the GSS for discussing methodological expertise, which has already been established as part of the new vision, should help improve collective knowledge and share best practice.

OSR has a role to play here too. The Code of Practice for Statistics can be used to help ensure the trustworthiness, quality and value of statistics produced using AI. Our Guidance for Models explains how the framework of the Code helps in designing, developing and using statistical models. Given the recent rapid developments of AI and feedback from our recent consultation on the Code, we are planning more-specific guidance on how to use AI in line with the Code.

The use of AI brings opportunities but also risks. The understanding of how AI can be used in a reliable, transparent way which supports public confidence is still evolving. The statistical system needs to be equipped to respond to the needs and questions that arise in this area to adopt AI in a responsible way that enhances the trustworthiness, quality and value of official statistics.

The potential future benefits of AI use in official statistics are significant, and we expect this to be a growing area of interest for the system. We would like to see producers exploring the possibility of using AI in official statistics in a transparent, responsible and collaborative way, drawing on existing guidance.

We will engage with producers to support innovation and best practice in this area. We intend to develop guidance on using AI in line with the Code, working collaboratively with producers and learning from other regulators in different sectors.

UK-wide comparability

In June 2025 we published a systemic review which looked at the adequacy of UK-wide comparable statistics and data. Though statistical producers have done good work in this area, the comparability of statistics across the UK has long been a challenge in the UK statistical system. In many ways this challenge is a consequence of the UK political and statistical system, where responsibility for policy, data collection, analysis and evaluation is devolved to different levels. We want our review to help the UK statistical system to address the persistent challenges in delivering UK-wide comparable statistics on priority topics, such as on health and education outcomes.

Our review responds to recommendations from the Public Administration and Constitutional Affairs Committee in 2024 and the Independent Review of the UK Statistics Authority, which both highlighted the difficulties in comparing the experiences and outcomes of citizens across different parts of the UK. Additionally, the UK Statistics Assembly held in January 2025 highlighted the importance of the statistical system recognising the need for UK-wide statistics and recorded this as a high priority.

To address these challenges, we have proposed a comparability framework to help the UK statistical system better understand where data comparability issues exist and prioritise resources to develop statistics where comparability should be improved. In addition, we have made four key recommendations for the Government Statistical Service (GSS) to facilitate a step change in delivering UK-wide comparable statistics on priority topics. The recommendations focus on reviewing legal frameworks for data sharing, engaging with users, reviewing governance and seeking cross-UK political commitments to adequately finance meaningfully comparable statistics at national, regional and local levels.

Given the long-running and complex nature of this issue, our recommendations are challenging and will require careful thought and oversight to successfully deliver. But in our view, they are all necessary to drive the changes needed to address the issues highlighted in recent reviews and, importantly, meet the needs of statistics users.

Recommendations