Overview

In July 2023, the Office for Statistics Regulation (OSR) published a report on data sharing and linkage for the public good. The report reviewed the data sharing and linkage landscape across government and identified steps needed to move towards a brighter future for research and statistics, where data sharing and linkage is a priority across government and barriers have been successfully tackled. In the report, we made 16 recommendations which, if implemented, will enable greater data sharing and linkage for research and statistics for the public good.

Since the previous report was published, OSR has continued to engage with key stakeholders across government and beyond, aiming to generate and maintain momentum on our recommendations. Our report has had positive impacts on several projects relating to data sharing and linkage, as well as the culture in government around data use. These impacts include influencing the strategic approach taken by the Department for Science, Innovation and Technology (DSIT) to reviewing cross-government data sharing policy; developments in the Data Marketplace led by the Central Digital and Data Office (CDDO); the implementation of Wave 2 of the Public Engagement in Data Research Initiative (PEDRI); and technical innovation by the ONS Data Science Campus in developing new privacy-enhancing technologies (PETs). These initiatives reflect wider progress in several areas of data sharing and linkage, such as in public engagement activity and the support resources available to researchers and data users. Much of this positive activity has been led by National Institutes and programmes such as ADR UK, DARE UK and HDR UK, which help to bridge the gap between data owners and analysts in government and beyond. Nonetheless, there is still significant progress to be made in overcoming many of the remaining barriers to data sharing and linkage.

In our previous report, we committed to providing an update on progress against our recommendations within one year of publication. To assess progress, we have spoken to many of the same stakeholders, and some new ones, asking them for updates on work relevant to the recommendations. This report summarises our findings and our refreshed recommendations to government based on these findings. While some recommendations remain unchanged, others have been updated to reflect developments over the last year and to make them more useful to data owners and users.

As the regulatory arm of the UK Statistics Authority (UKSA), our interest in data sharing and linkage – and the focus of this report and its recommendations – primarily relates to research and statistics. We recognise that effective data sharing and linkage happens between some government departments to enable operational delivery of public services, and that there are indeed pockets of effective data sharing and linkage ongoing between departments for use in statistics. However, there is still much to be done across government to reach a place where sharing and linking datasets for research, statistics and evaluation – both across government and for external researchers – is the norm across the UK statistical system, rather than the exception.

Key findings

There continues to be a failure to deliver on data sharing and linkage across government. Leadership is needed to ensure data sharing and linkage between departments and with external researchers is prioritised.

We are disappointed that progress on data sharing and linkage has been so slow across many areas highlighted in our previous report. The benefits of effective data sharing and linkage for the public good are clear. Sharing and linking data provides greater insights into society and the economy, enabling policy development and stimulating innovation, and is vital for socio-economic development. Without leadership across government, these benefits will not be realised and the UK will lag behind other comparable countries.

In March 2024, an independent review of the UKSA led by Professor Denise Lievesley CBE was published. We strongly endorse the recognition that “an effective system of responsible data sharing is critically important” to realise the potential of data held by government. The independent review also recognised the significant systemic, often cultural, barriers limiting data sharing across government. It recommended that the highest levels of government tackle these barriers and ensure data sharing for research and statistics purposes is prioritised. The cultural barriers were also highlighted in the May 2024 report Transforming the UK’s Evidence Base by the Public Administration and Constitutional Affairs Committee (PACAC). For sharing and linking datasets to be the norm across the UK statistical system, rather than the exception, there is still much to be done across government. To move towards this aim, we need to see leadership from central government and the statistical system, concerted efforts from the wider data community and government working in partnership with external researchers and organisations.

There are promising signs that some parts of government are willing to take leadership around data sharing and linkage, but stronger commitments are needed.

The Five Safes Framework underpins the government’s approach to enabling researchers to access data for research and statistical purposes. Thus, it is vital to ensure that the framework continues to effectively enable data sharing and linkage in a secure way. The UK Statistics Authority (UKSA) and Research Accreditation Panel (RAP) have committed to engaging with our recommendation to review the Five Safes Framework. Progress here includes discussing the potential to broaden use cases for data by considering a change in focus from ‘safe projects’ to ‘safe programmes’. The current strategic role of the Department for Science, Innovation and Technology (DSIT) in developing and cohering policy towards data access for the benefit of all of society and across the economy, has the potential to drive systemic change across government. In the government’s formal response to the Lievesley independent review of the UKSA, the Cabinet Office has committed to push forward data sharing in government.

As a new government forms, we also recognise the role of politicians in enabling data sharing and linkage. Politicians should set the expectation that departments must work collaboratively on the issues facing society and the economy. The Cabinet Office should work with the new government to ensure that effective data sharing and linkage forms part of the new government’s long-term strategy to tackle issues facing the UK. We look forward to the Cabinet Office’s more detailed response to the Lievesley review later this year.

Senior leaders and all in government need more awareness of and exposure to data issues, but data literacy is not being prioritised.

We previously pointed to the Data Masterclass – delivered by the ONS Data Science Campus in partnership with the 10 Downing Street Data Science Team – as a way for senior leaders to improve their data literacy. We suggested that this course be expanded to cover more topics, including the benefits of data sharing and linkage. So, it is disappointing that the Data Masterclass is currently on hold, with new cohort sign-up and course development paused. The importance of fostering a stronger data-driven culture in government was reflected in OSR’s recent report, Analytical leadership: Achieving better outcomes for citizens.

The conversation around data sharing and linkage continues to focus on the risks – from the individual privacy risks to the reputational risks to data owners and government departments.

Yet effective data sharing and linkage across government for research and statistics offers considerable benefits for the public good. It remains important to acknowledge the potential risks associated with sharing data and linked datasets, and to ensure both security and privacy. However, we would also like to see an emphasis on potential benefits for the public good. A UK-wide public dialogue report, produced by OSR and ADR UK, showed that the public support the use of government data in research and statistics, as long as they’re informed of their use and what safeguarding is in place. Government should start by thinking about what it wants to achieve and focus on the potential positive outcomes from effective data sharing and linkage. Leadership, including political leadership, is needed to promote these benefits, and we see gaps across government in leadership on data sharing and linkage. This report also highlights how social licence and retaining public confidence remain fundamental for progress. More can be done to work with members of the public and raise public awareness of data as an asset which can benefit government as a whole.

There are pockets of innovative and ambitious work happening in government around data sharing and linkage and highly successful data sharing and linkage projects.

The cross-government Better Outcomes through Shared Data (BOLD) programme continues to demonstrate how system-level developments can directly improve outcomes for the public. HMRC and ONS are expanding a long-standing data sharing arrangement for research and statistical purposes. In Northern Ireland, NISRA (Northern Ireland Statistics and Research Agency) has supported the growing interest in cross-departmental data insights, such as with the creation of the Educational Outcomes Linkage initiative. Meanwhile, new supporting materials have been introduced or progressed. The CDDO is developing a data marketplace to improve the discoverability of data within government and provides a central place for those within government to find out about what data are held and how data can be accessed. This departmental work is complemented, and often enabled, by the support of organisations such as ADR UK, which has launched a comprehensive Learning Hub bringing together information on skills and resources for those using administrative data and data linkage, as well as a searchable public metadata catalogue for datasets held across the ADR UK partnership. As for technical developments, there has been significant progress in the development and trial application of privacy-enhancing technologies (PETs) over the last year. DARE UK Phase 1 driver programmes are developing standards and new technical solutions to meet the needs of TRE users in relation to areas such as data analysis and disclosure control. All these initiatives are positive and have the potential to support data sharing and linkage by government.

Processes for data sharing and linkage continue to create barriers to effective and efficient data sharing.

Through discussions with stakeholders for this follow up, we have been made aware of additional process challenges. These include delays to output approvals by trusted research environments (TREs), barriers created by the requirement to publish all analyses enabled by the DEA (2017) Research Power and lengthy legal review of complex and non-standard data sharing agreements. The time taken to access data remains a significant barrier in some cases. Researchers using one secure research environment have reported significant increases in wait times for data access and output approval, and one analytical organisation beyond government has told us this has deterred them from applying for data from that environment in the future. The value of government data to provide evidence to help tackle the issues of the day cannot be realised if data sharing processes fail to enable timely access and publication of analyses.

The Integrated Data Service (IDS) continues to experience challenges.

The independent review of UKSA recommends that government departments support the development of programmes like the Integrated Data Service (IDS)The review recognises the IDS as critical to enabling greater use of administrative data and improving the efficiency of statistical analysis across government and academia. It recommends that “the ONS [take] action to ensure that the purpose, scope and requirements of the IDS are clearly communicated and that the needs and concerns of departmental data owners are sufficiently understood.” In recognition of the pivotal role the IDS could take in facilitating the sharing of data across government, OSR is currently considering what role it can play in supporting the development and success of the IDS. Many of the recommendations in this report are relevant to the work of the IDS, and we refer to the IDS specifically in discussion around recommendations 4, 8, 10, 11, 14 and 15.

Summary of progress against recommendations

The following expandable boxes give a list of the recommendations from last year’s report and OSR’s overall assessment of whether there has been progress towards each recommendation being met. The table also shows where we have made minor changes to the original recommendations or altered the recommendations more significantly to change their focus (revised), and whether we have retained them. While most recommendations have been retained, some have been removed.

Next steps

OSR has an ongoing role in driving forward data sharing and linkage across government, beyond this follow-up report. As such, we intend to report again on the state of progress in coming years, using our refined recommendations from this review as a guide. In the meantime, OSR will convene, support, advocate and warn on specific areas where we think we can have a useful impact. In doing so, our goal is to help government, researchers and the private sector move towards a future where data sharing and linkage is widely and safely employed for the public good.