6. Annex A - OSR comparability framework tool proposal

By comparable UK-wide statistics and data, we mean statistics, data and analysis which can, as a minimum, be meaningfully compared across England, Scotland, Wales and Northern Ireland and at local levels of geography. These statistics and data lend themselves to better analysis, can be used to draw policy lessons on what works, and enable citizens to understand outcomes, draw firm conclusions and hold respective governments to account.

The Fraser of Allander Institute (as part of the research programme of the Economic Statistics Centre of Excellence) published research on comparability: Supporting Comparative Regional Analysis Across the UK: Evaluating the availability, comparability, and dissemination of Northern Ireland’s socioeconomic data, March 2024. 

This research builds on internal guidance issued by the Department for Levelling Up Housing and Communities (DLUHC) in 2023, and proposes that comparability has six tiers, which we have summarised below:

1. Full comparability: Data is collected and analysed on a UK-wide basis, including surveys administered by one producer on behalf of another. Importantly, sample sizes must be sufficiently large to support reliable regional inferences. An example of fully comparable data are indicators on personal wellbeing produced using the Annual Population Survey (APS), a UK-wide survey.
2. Direct comparability: Data is collected separately across the four nations, but with definitions and methodologies that are closely aligned, allowing for a direct comparison. An example of this cited by DLUHC (2023) is court case timelines published by His Majesty’s Courts and Tribunals Service, Scottish Government and the Northern Ireland Department for Justice.
3. Meaningful comparability: Data is collected separately across the four nations and has, for instance, definitional or methodological differences. An analysis of the extent of comparability has taken place which may uncover that ‘meaningful’ comparisons can still be made. However, these must be considered on a case-by-case basis to determine whether they are “good enough” to fulfil a specific user or purpose. Producers must therefore sufficiently understand uses and users’ needs to determine that making comparisons is appropriate and truly meaningful. Meaningful comparability also provides a useful starting point where it has been agreed between different statistical bodies that direct comparability is desirable. An example of this type of data cited by DLUHC (2023) is data on pupil attainment.
4. Conceptual comparability and coherence: Data is collected separately across the four nations and has, for instance, definitional or methodological differences. An analysis of the extent of comparability will not allow for meaningful comparison across the four nations, but indicators are available measuring similar concepts. These differences have been explored and clearly explained to users, for example, through guidance published by the GSS coherence team. Data on ambulance response times are an example of this.
5. Conceptual comparability but not yet coherent: Data is collected separately across the four nations and has, for instance, definitional or methodological differences. An analysis of the extent of comparability will not allow for meaningful comparison and these differences have not been fully articulated to users. For example, the Scottish Government, Welsh Government, NISRA and UK government all produce input-output tables. However, some differences in methodology will occur across producers. These differing approaches have not been fully documented by the GSS but have been discussed by researchers.
6. Not comparable and a data gap exists: Data cannot be compared because an indicator does not exist for all four nations or there is a data gap in one nation. ONS publications which are GB only also fall into this category. In the UK, examples of two data gaps affecting all four nations are estimates of interregional trade and regional prices.

OSR comparability framework tool proposal

As part of our work on comparability, we propose the use and further development of a comparability framework tool to enhance the current understanding of the extent of comparability of UK statistics and data on agreed priority topics.

Combined with an enhanced understanding of users’ needs for UK-wide comparable data, this framework should be used by the UK statistical system and users to better understand where data comparability issues exist and inform priorities for the resourcing and development of statistics where the extent of comparability across the UK should be improved.

We propose that the GSS, led by the National and Chief Statisticians:

1. draw on the enhanced understanding of users’ needs for UK-wide comparable statistics and data on priority topics obtained from cross-UK user engagement
2. use the comparability framework tool to assess and score the extent of comparability of priority statistics identified by users and publish these findings
3. publish a strategy to produce and maintain comparable UK-wide statistics and data on selected priority measures at national, regional and local levels, where the National Statistician and Chief Statisticians all agree that meaningful comparable statistics (as a minimum) should be provided
4. monitor progress in the development and maintenance of comparable UK-wide statistics for agreed priority measures on an ongoing basis, refreshing the understanding of users’ needs for comparable UK-wide statistics at least every three years, potentially as part of the triennial UK Statistics Assembly process
5. seek finance and resourcing commitments from UK and devolved government settlements to develop and maintain a priority suite of meaningfully comparable UK-wide statistics on agreed priority measures, as required

OSR will also trial this framework in some of its upcoming assessments and consider the case for embedding its use.

An example of how the concept of a comparability framework could be applied to official statistics is set out below.

Comparability framework tool - concept illustration