Ed Humpherson to Darren Morgan: OSR Assessment of ONS GDP, UK regions and countries statistics – Interim Findings

Dear Darren

OSR Assessment of ONS GDP, UK regions and countries statistics – Interim Findings

As you know, over the last six months OSR has been assessing ONS’s experimental quarterly regional GDP statistics against the Code of Practice for Statistics, with a view to considering National Statistics for them.

It is good that ONS introduced these statistics in 2019, in response to one of the strategic findings in Sir Charlie Bean’s Independent Review of UK Economic Statistics, that the availability of regional statistics was “inadequate”, and has continued to develop them since.

Our assessment has found areas of good practice in the production of these statistics. In particular, ONS has addressed one of the key themes in Bean’s report that there is a need to improve the coverage and timeliness of subnational estimates, whilst exploring the potential for using administrative data to fill some of the gaps in the availability of regional statistics. ONS has also engaged well with users of quarterly regional GDP statistics, keeping users well informed on the developments planned and made to the statistics.

As part of the assessment, users also raised concerns about the economic coherence of quarterly regional GDP statistics with other local information, along with other queries around automatic balancing and benchmarking. We recognise that this will reflect that the methods are under development, and that some of the analysis will relate to the Covid-19 pandemic where usual relationships between economic variables may not hold.

In September 2022 ONS published updated quarterly regional GDP statistics based on a revised methodology. As the estimates from this new methodology become established and users become familiar with them, an important part of demonstrating the value of your statistics as they improve will be to bring together information about methods and uncertainty in the data so that users can understand what the ONS statistics show alongside their own statistics. In particular, we welcome the intention to publish an article for users covering the structure of the new methodology and the role the automatic balancing tool plays in producing quarterly regional GDP statistics. It is good to see that ONS has recently improved the timeliness of the statistics by publishing them around seven months after the reference period and to note your intention to improve timeliness further.

In the light of the ongoing developments, we have paused the assessment and offer further details on these improvement points in the annex to this letter.

My team will keep in touch with yours as the statistics develop, including deciding when we should restart the assessment. I am copying this letter to Heather Bovill, Deputy Director for Surveys and Economic Indicators, and to Jon Gough, Assistant Deputy Director for Surveys and Economic Indicators, both at ONS.

Yours sincerely

Ed Humpherson
Director General for Regulation

Annex: ONS GDP, UK regions and countries statistics – OSR Interim Findings

	Findings	Recommendations
Quality	As part of the assessment process, users raised with us the question of economic coherence between quarterly regional GDP statistics and other local information. OSR’s analysis suggested that there doesn’t appear to be a strong visual relationship between quarterly regional GDP statistics and other local economic information, including business confidence indicators, unemployment and HMRC Real Time Payroll Information. Users told us a similar story on their own triangulation work, comparing the quarterly regional GDP statistics published with HMRC Real Time Payroll Information, credit card spending and vacancy data, noting that in most cases, their analysis suggested that quarterly regional GDP statistics acted as the outlier in the comparisons. ONS in its introductory article from 2019 Introducing GDP for the countries of the UK and the regions of England notes that “Regional data is more volatile than national measures and so quarterly regional GDP statistics need to be carefully interpreted alongside economic trends both in the regions and in the UK”. This caution reflects the need to apportion data to local levels and then constrain to other totals, each of which can cause more uncertainty around the regional data. The apportionment uses data held on the Inter-Departmental Business Register, which provides information on the employment and structure of businesses in the UK. While being a long-established method of apportioning national data to countries and regions, it adds another layer of complexity to the production of these statistics. The time period under consideration also largely covered the height of the Covid-19 pandemic, where relationships between economic variables may have been unusual. One specific aspect of the methodology for producing the quarterly regional GDP statistics is that they are benchmarked against the most recent year’s annual estimates of regional GDP, which are themselves constrained to national estimates of GDP. This process of benchmarking ensures that estimates of quarterly regional GDP are also broadly in line with the quarterly national estimates. Users told us that they would like to understand more about the benchmarking adjustments. ONS uses an automatic balancing tool (called FICO Xpress) to benchmark estimates of quarterly regional GDP with estimates of national GDP. ONS responded well to users’ concerns regarding the use and impact of FICO Xpress and introduced a new approach to constraining totals in its September 2022 GDP, UK regions and countries: October to December 2021 statistical release. The new approach constrains quarterly estimates of regional GDP to quarterly UK GDP at current prices for two-digit SIC industries. ONS told us that it plans to provide users with more-detailed documentation on quarterly regional GDP methods and to continue working closely with key stakeholders to ensure they are well sighted on these methods.	Being open about the methods used to compile quarterly regional GDP statistics (as ONS plans to do), and the statistical uncertainty of the estimates will be an important step towards establishing the quality and credibility of these statistics. Documenting the reasons for selecting its constraining method and the impact that constraining has on the quality of quarterly regional GDP statistics would help users in their understanding of the statistics. To further aid users’ understanding of the impact of constraining, ONS should consider publishing the adjustments it makes using its automatic balancing tool to quarterly estimates of regional GDP to constrain to the annual regional measure of GDP and national estimates of GDP. It will also be important for ONS to reassure users about the coherence between ONS’s statistics and other available data, including accounting for the uncertainty in the statistics. As part of this it may be helpful for ONS to consider publishing analyses that provide users with insights into the economic coherence of quarterly regional GDP statistics with other relevant data.
Value	Since September 2022, ONS has published quarterly regional GDP statistics around seven months after the reference period which represents an improvement of around six weeks compared with previous outputs. The timeliness of these estimates compares favourably with similar estimates of regional GDP statistics published by other National Statistical Institutes. However, users told us that quarterly regional GDP statistics are not sufficiently timely to enable analysts to keep policymakers and local economic development bodies appraised of the current picture of regional economic activity. ONS told us that it has plans to improve timeliness further over the coming quarters.	Once ONS has embedded a set of robust methods to produce these statistics, ONS should consider and plan, in conjunction with users in what ways timeliness could be further improved.