Data sharing and linkage for the public good: breaking down barriers

Our findings from our report on enabling greater data sharing and linkage are clear: sharing and linking of datasets is still the exception, rather than the norm.

Barriers to progress persist. These include a lack of leadership in championing a cross-government approach to data sharing, uncertainty around funding and resourcing, a lack of clarity about data access processes, and nervousness about the safety of data sharing and social licence for data sharing and linkage. We call for leadership, including political leadership, and an approach that starts with thinking about the benefits of data sharing and linkage to society, then supporting the removal of barriers to deliver these benefits, in a secure way. We have reiterated this sentiment more recently, in our statement on how Government can support the statistics system to be at its best now and in the future.

Examples of initiatives enabling data sharing and linkage

Among all these firm messages, the thing I want to make sure isn’t lost is that many teams and organisations are spearheading impressive initiatives that are helping to overcome barriers. These will benefit analysts and organisations within and beyond government who want to share, access or link data. Below I highlight initiatives we heard about while researching our report, grouped under the themes of our findings.

Public engagement and social license

It is important to obtain social acceptability for data sharing and linkage. Engaging the public to track attitudes towards data sharing and linkage remains vital, so that the social licence is understood and maintained. But there can be a lack of understanding about how to carry out public engagement in a meaningful way.

• The Public Engagement in Data Research Initiative (PEDRI) is developing good practice guidelines to help researchers and other analysts conduct public engagement in data research and statistics.
• The Public attitudes to data and AI tracker survey published by the Department for Science, Innovation and Technology (DSIT), provides insight into how society views data use. The 2024 survey, due to be published in December, will include specific questions on public attitudes to data sharing and linkage.
• The Administrative Data Research UK (ADR UK) Learning Hub contains useful resources on public engagement in practice and brings together information on skills and resources for those using administrative data and data linkage.

The public’s key concern regarding data use remains data security. It is likely that many people’s attitudes towards data sharing and linkage continue to be influenced by concerns around data security.

• The Five Safes Framework is voluntarily adopted by data services to help provide safe research access to data. Research Data Scotland has introduced effective guidance, to help understand and implement the framework.
• DSIT and the Information Commissioner’s Office (ICO) are supporting organisations to assess the costs and benefits associated with using Privacy Enhancing Technologies (PETs). Wider adoption of PETs could enable greater use of sensitive data in a privacy-preserving manner.

Good leadership, and the skills and availability of staff

At every step of the pathway to share and link data, the people involved, and their skills and expertise, are key to projects’ success or failure. A lack of awareness, priorities and capability of people involved in decision making and development, including senior leaders, analysts and those in data governance roles, can impose barriers.

• Programs like the Digital Excellence Programme, which trains civil servants in data literacy and AI can strengthen data literacy among leaders in government.
• The Data Science Campus, part of the Office for National Statistics, has launched One Big Thing – a new learning initiative, which in 2023 was focused on strengthening civil servants’ data skills. This course is now available for all civil servants on Civil Service Learning.
• The Data Linkage Champion Network provides a forum for civil servants of all grades and from across government to discuss and promote data linkage.

Hesitancy around data sharing and linkage due to potential overcaution remains a major barrier. The issue is not always that government departments disagree over whether to share data. It can also be that everyone agrees to a data share in principle, but it does not advance because of the complexities of the process.

• The Central Digital Data Office (CDDO) has established the Data Sharing Network of Experts (DSNE) to help departments deal with questions or differences of opinion around data sharing.

Non-technical processes that govern how data sharing and linkage happens across government

When an external researcher or government analyst wishes to access data there are several steps they follow. Firstly, they must know the data they wish to access and where they are held. Secondly, they must establish the legal route to the data and, finally, they must gain access to that data. Each of these steps can pose barriers.

• The UK Statistics Authority (UKSA) has published an online resource answering frequently asked questions about the Digital Economy Act (DEA, 2017) Research power and the Statistics and Registration Service Act (SRSA, 2007). It covers what data can be accessed via the Research power and SRSA, accredited processing environments and who can access data and for what purposes.
• ADR UK’s online Learning Hub includes information on the DEA (2017) for researchers wishing to access administrative data under this legislation. It has a slide deck produced by the UKSA that explains what the DEA 2017 Research power allows for and contains a visual map of the data access journey.
• The CDDO is developing a data marketplace to improve the discoverability of data within government. The marketplace allows those within government to find out what data are held and how data can be accessed.
• ADR UK has created a searchable public metadata cataloguethat contains information about the datasets held across the ADR UK partnership. It includes a webpage for each dataset with links to information on how to access the data, as well as a description of the dataset.
• HDR UK has created a searchable public metadata catalogue which contains information from over 850 different health-related datasets across the UK.
• The Pan-UK Data Governance Steering Group was established by the UK Health Data Research Alliance, convened by Health Data Research UK (HDR UK), to simplify and streamline data access governance processes. The Steering Group co-developed and published Transparency Standards with HDR UK’s Public Advisory Board (PAB) to guide good practice.
• Research Data Scotland runs a Researcher Access Service for those wishing to access public data in Scotland. It publishes a data access overview describing the stages of data access – from discovering data to receiving access.

Technical specifics of datasets, and the infrastructure to support data sharing and linkage

Technical challenges can pose significant barriers to effective and efficient data sharing and linkage, including problems presented by the recording of accurate metadata.

• CDDO is leading efforts to help departments identify their Essential Shared Data Assets (ESDAs) – data assets that are critical from a cross-government perspective. This includes the creation and maintenance of accurate metadata, which will improve the documentation of data held by government.

Alongside initiatives that could have widespread benefits for eroding barriers, our report highlights examples of specific analytical projects and programmes that have successfully used linked data to deliver impactful analysis. These case studies offer a window on what can be achieved when data are shared or made accessible.