What is Levelling Up and how will its success be measured?

‘Levelling Up’ is now a term used daily in the media – but what does it mean and how are we, as the statistics watchdog, going to monitor its impact? Statistics Regulator Ben Bohane discusses…

Not long before I joined OSR as a Regulator, the Conservative Party made ‘Levelling Up’ a key part of their 2019 election manifesto. It focused on challenging and changing the geographical inequality in the UK, through investment and new infrastructure to allow ‘everyone the opportunity to flourish’.

Prior to working at OSR, I taught Economics to young people as a Secondary School teacher. I found teaching young people how changes in the economy and government spending might impact on their lives – really rewarding. I think back to those young people now – do they understand what Levelling Up is amongst the media hype? How will the proposals that are outlined in the Levelling Up White Paper impact their lives and futures?

If I were to explain it to my students now, I would describe Levelling Up as a plan to eradicate regional disparities in the UK, raise living standards and provide greater opportunities to people, in communities and areas that have so far not had the success of more prosperous parts of the country.

But with confusion over the concept really means – how can we measure something for which the success means different things to different people? Back in March, OSR’s Director Ed Humpherson wrote about ‘Why I Love Evaluation’ stating that evaluation “provides evidence of what works; it supports good policy; it builds the skills and reputation of analysts; it helps scrutiny.” Ongoing evaluation of Levelling Up will be key to its success.

In OSR our focus is on ensuring existing statistics that can be used to measure the success of government policy are of sufficient trustworthiness, quality and provide public value. But also, that statistics are available in the first place. As the government highlights in the White Paper, many of the metrics that will be used to measure the success of Levelling Up are either not yet available or of insufficient quality. The clarity of what’s being measured is important if people want to track progress through data.

In OSR we’ve already been working on public interest in regional disparities and fighting for a statistical system that is more responsive to regional and local demands. We have:

Reviewed key ONS surveys that feed into economics statistics (Annual Business Survey, Business Demography, Annual Purchases Survey)
Focused on driving improvements in sub-UK economic statistics (ONS Small Area Income Estimates, NISRA BESES statistics, ONS FDI statistics, ONS Regional HFCE statistics, HMT CRA statistics, ONS GVA statistics)
Provided evidence to the Treasury Committee’s inquiry on regional imbalances in the UK economy

Our Business Plan highlights that we have seen a growing public expectation that decisions affecting all aspects of our lives will be evidenced by trustworthy and accessible statistics. Over the coming months and years, we will continue to review new statistics and data sources from the Department for Levelling Up, Housing and Communities, Office for National Statistics and other data providers as they are developed to ensure that evidence and evaluation is at the forefront of pushing the plans forward.

Our regulatory programme for this year focuses on projects that will improve public understanding of the issues, current and emerging, that people want to be sighted on. As the statistics regulator, reviewing the statistics used in Levelling Up, we will be tracking the implementation of the GSS Sub National Data strategy and new tools such as the ONS Sub National Indicators Explorer, ensuring statistics are the best quality they can be and clearly focussed on measuring the outlined Levelling Up missions.

Statistics supported by clear analysis and evaluation will provide the evidence to measure the impacts, successes and failures of Levelling Up – and any future government policies to address regional disparities and improve people’s lives. As the government implements policies to address regional inequalities – and businesses and households respond – we will focus on ensuring that the statistics both accurately measure and live up to this ambitious long-term strategy. It is important to me as a statistics regulator that we do this. After all, the vision of Levelling Up is so important to the futures of those young people I used to teach.

Appendix 1: Background foundation work on surveys that are used to produce economic statistics

ONS Purchases Survey statistics (December 2019):

ONS reintroduce the survey following the National Statistics Quality Review of National Accounts to provide better information about purchasing patterns by business.

Our review found that the quality of outputs from the survey is still being improved, which reflected ONS’s own narrative that it would normally be several years before a new survey was producing statistics that could be used with confidence. It also reflected that the Annual Purchases Survey aims to collect variables that do not naturally fit with many businesses’ operational models. Our report noted the discrepancy between estimates of intermediate consumption derived from the Annual Purchases Survey and the Annual Business Survey. We said that it is an essential part of demonstrating that the quality of the statistics meets users’ needs that these differences are understood, explained well, and are used to further improve the statistics.

ONS UK Business Demography statistics (October 2020)

We reviewed ONS Business Demography statistics because we felt they should be considered key economic indicators. They are not regarded as such because they are not as good or as useful as they should be. The ONS’s business register – the Inter-Departmental Business Register (IDBR) – holds a wealth of data on the UK’s business population, some of which are used to produce business demography statistics. The remainder of which remain a largely untapped resource. In response to the COVID pandemic, ONS introduced a weekly indicator of business births and deaths and introduced a quarterly series of experimental business demography statistics. These innovations presented a platform for further development of the statistics. However some required improvements to the statistics rely on significant investment and we said that work to develop ONS’s business register should urgently be re-introduced to ensure that users’ needs for business population statistics are met. In our review we made several short-term recommendations for ONS:

demonstrate progress in understanding the access difficulties users are experiencing when using and linking IDBR data with data
publish its plans for publishing more timely business demography statistics, and its plans for developing the recently introduced quarterly experimental statistics
publish a narrative covering what ONS already knows about the range of key data quality issues, building on the supporting quality information provided with the new quarterly experimental statistics
publish its plans to restart and resource work to develop its business register

We also said in the longer term, ONS should publish a plan which includes specific actions, deliverables and a timetable that explains how it will address the improvements identified in the report, including plans for reviewing the funding of the Statistical Business Register.

ONS Annual Business Survey statistics (September 2021)

We reviewed ONS ABS and found that the significant time delay on the publication of ABS data means that the data are not always used to measure the ongoing impacts of structural and cyclical changes to the UK economy. As a result, ABS data are not fully meeting users’ needs for timely and detailed data on business performance.

We found ONS focus and priority on transforming short-term surveys means there has been a lack of investment in finance, staff, and systems and so ABS data have been unable to keep up with changing demands on their use. The lack of investment has curtailed ONS’s efforts to improve the detail and timeliness of ABS data.

We found a lack of investment has been a common theme of OSR’s recent assessments of ONS’s structural economic surveys and statistics. We strongly urged ONS to revisit the investment needs of these outputs, to ensure structural economic data are available to assess, for example, the ongoing impact of the economic shocks of Brexit and the pandemic.

Appendix 2: OSR work on regional statistics and Levelling Up

ONS Statistics on Regional Gross Value Added (August 2017)

“Many of the R-GVA users that we spoke to cited poor timeliness as a limitation of these statistics” and “that unless the R-GVA statisticians find new sources that provide the same level of detailed information more quickly than the current sources (which they indicated to us is unlikely in the short term), the timeliness of these statistics is unlikely to change significantly”.

“ONS might do more to bring out the differences between the regions through the proportions of people in the region who are economically inactive, which can affect the GVA per head statistics and the impact of commuting on the statistics” and requested ONS “to work with its national and regional stakeholders to bolster the statistical services such as information, advice and guidance available to provide even greater insight in sub-regions (particularly new city-regions) and in preparing contextual information to aid regional and country media in interpreting the statistics”.

ONS Statistics on Regional Gross Value Added (Phase Two) (June 2018)

We asked ONS to make further improvements, for example, “investigate whether improvements in the quality of deflators by adopting regional price statistics could be achieved technically and cost-effectively taking account of expected use of the statistics and user need”. We also asked ONS to “review the best way of making quality metrics both more useable to a less expert audience and more accessible generally”.

HM Treasury Statistics on Government Spending: Country and Regional Analysis (May 2019)

We asked HM Treasury to:

collaborate with producers of other public finance statistics and with analysts in the countries and regions to seek views, update their understanding of users’ needs to better support the use of these statistics
communicate effectively with the widest possible audience to increase awareness of the statistics and data
present CRA data in a more engaging way that supports and promotes use by all types of users and those with interests in spending at programme and service levels (sub functional levels)
test the strength of user need for CRA on a ‘where-benefits’ basis, examine the feasibility of collecting data on this basis and the trade-off between enhanced functionality and increased burden on data suppliers
provide a clear and comprehensive account in each annual CRA publication to allocation methods, including the inclusion of links to published documents about allocation methods in respect to all ongoing major project spending
ensure that users are provided with appropriate insights about changes in the data. This should include helping users understand impacts on the CRA data and provide links, when applicable, to other output areas where information on Brexit impacts has already been published
establish a development programme for these statistics and periodically review that programme; be open about progress towards meeting priorities and objectives; and arrange for users and other stakeholders to be involved in prioritising statistical plans
strengthen its arrangements for reviewing requests to allow pre-release access to new people; review the current list of those with pre-release access for CRA, with a view to minimising the numbers of individuals included and inform the Authority of the justification for each inclusion

ONS Experimental statistics on Regional Household Final Consumption Expenditure (HFCE) (January 2021)

We highlighted the potential of HFCE estimates as a highly important component in fully understanding regional economies. Prior to this there were no regional estimates of the expenditure measure of GDP, except in Scotland, a topic we previously highlighted in our 2020 submission to the Treasury Select Committee’s inquiry into Regional Imbalances.

DLUHC Levelling Up Fund prospectus (March 2021)

The Levelling Up Fund prospectus included a list of local authorities by priority category.

However, initially no description of the methodology used was attached to enable users to understand how local authorities were allocated to priorities areas. A week later a DLUHC published a methodology document, but it was still not possible to recreate the full dataset used to allocate local authorities to priorities areas.

We wrote to DLUHC publicly highlighting our concerns about the transparency of data related to the Levelling Up Fund and we requested DLUHC publish data that supported the allocation of priorities areas to enhance public confidence in the decisions that were being made.

As a result, DLUHC published the Levelling Up Fund: prioritisation of places model, which showed all the steps that were taken when using data to assign Local Authorities in England, Scotland and Wales to categories 1, 2 and 3. The spreadsheet included a “data and input construction” tab which included links to the source data with explanations of the source and why it was chosen.

ONS Foreign Direct Investment (FDI) Statistics and DIT Inward Investment Statistics (April 2021)

As a result of our review new questions were added to the quarterly and annual FDI surveys, to collect more-granular data on sub-national FDI and ONS is now publishing experimental UK sub-national FDI statistics.

NISRA BESES statistics (December 2021)

As a result of our review, NISRA will be publishing more timely imports data and has developed an interactive dashboard that provides more-granular monthly international trade data on products.

ONS Income Estimates for Small Areas statistics (January 2022)

We suggested how further value could be added by ONS understanding the needs of current non-users who require income estimates at lower levels of geography.

Users’ want to be able to aggregate estimates for lower-level super output areas into bespoke geographies, the estimates are given for middle-layer super output areas which are too large for users’ needs.

DLUHC planning applications in England statistics and at the same time Homes England Housing Statistics (March 2022)

We felt at the time it was likely that planning performance and planning reforms will in some part be included in new Levelling up legislation, given its assumed focus on local area development.

LA planning application performance at the time had also been identified as a priority departmental outcomes metric in the 2021 Spending Review.

We advised further developments to the statistics. One of these developments included sub-national commentary, which should be introduced to help explore, for example, trends in planning to support regeneration in the 20 English towns and cities prioritised in the Levelling Up white paper.

We found the statistics could be further enhanced if Homes England were to publish information about aspects of quality, for example, limitations of data sources, quality assurance checks carried out by data suppliers, and the team’s assessment of data quality against our quality assurance of administrative data (QAAD) matrix

We also asked Homes England to consider how any uncertainty in the statistics might be more clearly communicated to users, as the latest data are provisional and subject to revision.

Finally, we suggested further insight and context should be added by enhancing the narrative and analysis provided for users who wish to explore the topic further.

Appendix 3: Treasury Committee evidence

2019 response

Key point

There is a range of official statistics on regional economic performance. They should be considered alongside other forms of data published by Government and others.

What we said

All data, whether classified as official statistics or not, should seek to adhere to high standards of trustworthiness, quality and value (which we describe as voluntary adoption of the Code of Practice’s pillars).

Key point

There are some limitations to the current data sources, both in terms of data gaps and in terms of quality.

What we said

In our written evidence referring to regional economic data, we highlighted “the quality of regional data is affected by the granularity that the data sources can provide, and/or the timeliness of the data provision”. Regional data is more volatile than national estimates and there are significant challenges in forming regional estimates of GDP.

We said “In arriving at aggregate estimates, statisticians often combine both administrative and survey data sources….and then disaggregate to provide regional breakdowns (a top-down approach). Survey data is often limited in its depth: for example, the data used to compile R-GVA can become stretched at lower geographies, becoming increasingly volatile as it is disaggregated further.”

Key point

There is a significant use of modelled data, which apportions national data to regions using formulae, rather than directly observed data, which would be gathered at the local level.

What we said

“During our regulatory work, we received feedback from users of regional and sub-regional economic data expressing concern that they can’t tell whether the data they are using is based on observed economic behaviour or come from modelled estimates. They view data based on observed estimates as more reliable than modelled estimates”.

At our request, the ONS conducted research into how much data measuring economic growth are directly observed at a regional level and collected in a way that can be immediately and wholly assigned to a single region, and how much data are modelled to provide regional estimates.

Key point

It may be worth considering a network of regional statistical observatories, akin to the Bank of England’s regional agents, that can help provide ONS and others with better insight into regional economic issues.

What we said

We wanted to highlight the benefits of a presence outside the offices of London, Newport and Titchfield – both to better understand the dynamic of regional economies, and to be closer to users with a regional focus (like combined mayoral authorities).

2020 response

“Developments [in regional statistics] will be enabled by better access to administrative data, where ONS can provide enhanced (ideally flexible) geographies with more use of direct estimation”.

“Regional performance information published by the UK Government can be found in some departmental annual reports and accounts but is not summarised in any compendium”.

We laid out several important conditions for publishing regional economic forecasts that could be adopted to help people make judgements about the UK and regional economy.

One of these conditions was: “It would be important to communicate the uncertainties associated with any regional GVA forecasts. For example, there are deficiencies in historical GVA data. Forecasts will only be as good as the data they rely on”.