Technical Challenges: Update on recommendations

In our previous report, we highlighted that technical challenges can pose significant barriers to effective and efficient data sharing and linkage. These include problems presented by variation in data standards and definitions, the effectiveness of data linkage methodologies and the recording of accurate metadata. As such, we made recommendations advocating the allocation of sufficient resources to allow the development of quality metadata and documentation, and in support of government doing more to standardise data and definitions. We also advised that the development and deployment of data linkage methodologies should be done in the spirit of transparency, openness and collaboration.

During engagement with stakeholders for this report, we heard that there has been welcome progress against our recommendations. Leadership by CDDO, particularly in creating the Data Marketplace and guiding departments in identifying and recording their Essential Shared Data Assets (ESDAs), is helping to create conditions for a government-wide approach to overcoming technical challenges. The Department for Science, Innovation and Technology (DSIT) has continued its work to map current data standards across government and has engaged those outside government to discover what types of data they would like to access and in what form. Nonetheless, significant technical challenges remain for data sharing, and consistent buy-in and proactivity by all government departments will be required to overcome them. This includes the allocation of sufficient resources by senior leaders, as this would ultimately result in more efficient and cost-effective data sharing and linkage across government.

Recommendation 15: Sufficient Resources

Original recommendation:

To enable effective, efficient, and good quality data linking across government, senior leaders should ensure there are sufficient resources allocated to developing quality metadata and documentation for data held within their organisations.

Key findings

• The CDDO is leading efforts to help departments identify their Essential Shared Data Assets (ESDAs), which will be crucial to improving the documentation of data held by government.
• More needs to be done within departments to record accurate metadata for ESDAs to ensure efficient and effective data sharing is possible.

Summary of findings

Since our previous report, we have not identified any specific examples where extra resource has been allocated to the maintenance of data documentation and metadata. However, we would like to acknowledge the CDDO’s systemic work in this space, as we believe it is one of the biggest drivers for prioritising data sharing and linkage resource in government departments. The CDDO’s work helping government departments identify their Essential Shared Data Assets (ESDAs) and its development and trial of the Data Ownership Model have been successful in increasing the identification and documentation of data in government. This includes the creation and maintenance of accurate metadata. These two initiatives underpin the development of the Data Marketplace, outlined in CDDO’s roadmap for 2022 to 2025. The CDDO is working with analysts contributing to the IDS to use this initiative to drive the discoverability of these ESDAs and their potential availability on the IDS platform.

The CDDO defines ESDAs as “data assets that are critical from a cross-government perspective”. The CDDO aims to help government departments identify these data assets within their organisations and via the Data Marketplace improve their discoverability and further potential sharing across government to improve public service delivery, analyse the effectiveness of policies and programmes and ensure the effective use of resources. In tandem with the identification of ESDAs, the CDDO is also trialling the data ownership model that is applied to ESDAs. The Data Ownership Model defines specific roles in the management and care of ESDAs. The key objectives for the Data Ownership Model include considering “where data assets may have value to wider government, society and the economy, and the protection and exploitation approaches required to realise it” and ensuring “every Essential Shared Data Asset has accurate metadata”. It is our hope that the CDDO’s work will both educate senior leaders on the importance of data documentation and metadata and give them the provision to recruit more analysts to take on these responsibilities.

Recommendation 16: Standardisation

Original recommendation:

Many departments are looking to standardise government data and definitions, but it is unclear whether or how these initiatives are working together. Those working to standardise the adoption of consistent data standards across government should come together to agree, in as much as is possible for the data in question, one approach to standardisation which is clear and transparent. Given the work done by the Data Standards Authority, led by the Central Digital and Data Office (CDDO), the CDDO may be best placed to bring this work together.

Key findings

• The importance of standardisation is being increasingly recognised across government, with recent initiatives by the CDDO, ONS and DSIT taking welcome action.
• Variation in data standards and definitions remains, and departments can do more to support government-wide standardisation work to improve the potential for high-quality linked outputs.

Summary of findings

Much progress has been made in line with this recommendation since our last report. The CDDO, through the Data Standards Authority, has begun working with ONS and other departments to define data primitives, which it is defining as common data attributes shared between different departments. This work is closely linked to ongoing work with representatives from local government around data sharing needs around the identification and provision of services and support to vulnerable people. However, there is limited resource within the CDDO to carry out this work, and getting buy-in from some departments is difficult.

The standardisation of data systems is still an issue, with multiple stakeholders saying that legacy systems and unique software are still causing issues. This was raised as a problem mainly when sharing data between academia and government. Academics can often use software that requires subscriptions, whereas government favours open-source software. This creates issues as there is little desire within government to go back to using licenced software.

DSIT been working with the CDDO on addressing coherence in data sharing and linkage. The CDDO is focused on the technical aspects, whereas DSIT focuses on policy. DSIT has engaged with businesses and researchers to find out what types of data they would like to access, and in what form. Similarly to the CDDO, DSIT also wants to help government understand what data it holds and whether they are in an accessible form. DSIT told us that coherence of policy approach is needed to enable data sharing and linkage, but that consistency is not necessary for each sharing scenario. Each one will be different, and individual considerations will be needed for different datasets and approaches. However, consistency for interoperability purposes, such as through standards, remain important. DSIT is also helping the CDDO develop the Data Marketplace.

We believe this recommendation is fit for purpose.