Context 

The Office for Statistics Regulation (OSR) has recently undertaken several quality assessments of specific economic statistics produced by the Office for National Statistics (ONS), including those published on Business Profits and Gross Operating Surplus, GDP revisions, Producer Price Indices and  Business Enterprise Research and Development statistics. These assessments have revealed specific areas with potential for improvement.  

Some common themes (for example, survey sample size and design) have emerged, and OSR considers it would now be helpful to look at these themes in a more systemic way.  

Furthermore, it has been around eight years since Professor Sir Charles Bean undertook a comprehensive review of ONS’s economic statistics and the National Accounts. The Bean Review, as it is known, presented a set of recommendations for improvement. ONS was provided additional resources to fund associated developments and set out a strategy for delivering better outputs. The strategy included methodological improvements, greatly expanding the use of administrative data, and steps to better measure aspects of the modern economy. It is appropriate now to review the progress ONS has made in these areas. 

The economic and social context has of course evolved since the Bean Review. Response rates to surveys have fallen, at least in part due to the pandemic. The economy has continued to change, and the needs of users have shifted as the policy context has changed (with, for example, a greater focus on spatial inequality and industrial policy). OSR considers that a systemic review should also look at how ONS has responded to this changing context. 

OSR has undertaken initial engagement with some key stakeholders to inform the scope of the review. This has revealed strong support for many ONS outputs but also some common concerns, including:  

• the adequacy of source data – both administrative and survey sources 

• the priority given to defining, maintaining and improving ONS’s core economic statistics relative to new activities and outputs 

It is for these reasons that OSR intends to undertake a systemic review of the quality and value of the economic statistics produced by ONS. 

Scope

The systemic review will examine how effectively and efficiently the economic statistics produced by ONS meet existing and emerging stakeholder needs.   

The statistics considered will be those covering economic output, expenditure, trade, prices, the business sector and the labour market. However, the review will avoid duplicating other ongoing assessments, for example those in connection with the transformation of the Labour Force Survey. 

Approach 

As the scope of the review is potentially very broad, OSR considers that a phased approach would be most efficient and could provide earlier insight. This may be helpful for ONS’s strategic and business planning in the context of the development of the new UK Government’s spending plans.  

OSR proposes that the review have three phases, focusing sequentially on various aspects of ONS’s performance. The sequencing reflects evidence gathered from the previous assessments undertaken by OSR and from the initial stakeholder engagement:   

1. The adequacy of data sources, including:
   • progress in the use of administrative and private sector data since the Bean Review and barriers to the greater use of such data 
   • the resourcing, prioritisation and design of surveys to complement administrative data, including the way ONS is addressing reductions in response rates and quality concerns over both period since the pandemic and the longer term.
     
2. The extent to which ONS outputs meet user needs, including:
   • clarity around what constitutes ONS’s core outputs and its associated approach to resource allocation 
   • progress in measuring the “modern” economy 
   • progress in understanding and responding to new and emerging customer needs (for example during the pandemic and in respect of industrial strategy and regional inequality) 
   • how effectively ONS has contextualised and quality-assured its outputs to promote informed use by stakeholders
     
3. ONS’s organisational context, including:
   • whether the appropriate structures are in place both within ONS and across government to maximise the efficient sharing of relevant data 
   • the effectiveness of ONS’s strategic approach to IT system choices 
   • whether ONS staff resources and IT systems are adequately and appropriately funded, appropriately deployed and effectively prioritised to maximise the quality of outputs 
   • the performance of ONS relative to other National Statistical Institutes (NSIs) regarding resource levels and outputs 

While the review will discuss these phases sequentially, the underlying issues are inter-related, and the reporting will reflect these inter-relationships. 

As noted, the review will draw on lessons learned in the quality-related work already undertaken and in progress by OSR. It will complement this by drawing on material on methodology and quality published by ONS and by undertaking additional case studies that focus on specific ONS outputs.   

The timing of the reporting will reflect the availability of resources within OSR.  

Work Plan

Reflecting the sequential approach to be adopted by the review, initial work will be focused on the first phase of the review, covering the adequacy of data sources.  

The work plan for the second and third phases of the review will be developed in the light of evidence gathered during the first phases.    

This document will be updated as the review progresses. 

To assess progress in using administrative and private sector data, we will review ONS’s published documents. Progress will be considered in the context of the Bean Review recommendations, ONS’s own objectives and timelines as set out in the various iterations of its strategy documents and user needs. Gaps in the available documentation will be addressed through roundtable discussions with relevant ONS staff (including ex-ONS employees). Engagement with key users will inform the assessment of how well the use of administrative and private sector data is meeting their needs.

Barriers to the greater use of administrative data will also be investigated through discussion with ONS staff and GSS (Government Statistical Service) staff in other government departments. In particular, we will focus on the existence and effectiveness of cross-government processes in place to promote and incentives data sharing.  

To the extent possible based on available material, ONS’s progress in using administrative and private sector data will be compared with that of other NSIs. Such comparisons may be challenging as they will need to take account of differences in the legal and institutional context (for example, the role of Identity Cards). 

Regarding the role of surveys, ONS’s published documents will be used to identify and describe the key features of the main source surveys for the National Accounts and related economic statistics. The features considered will include the sources used for weighting.

We will particularly focus on assessing achieved sample sizes and survey design in the context of surveys’ relative importance for ONS’s key outputs. This work will also draw on OSR’s previous assessments of individual economic statistics, and additional case studies if appropriate.  

Changes in response rates over time will be investigated. To the extent possible, we will assess the experience of other NSIs in this area to provide context. 

We will also consider changes to sample size or survey design to reflect the increasing use of administrative data. Again, any gaps that may result from limitations in the availability of published material will be addressed through discussion with ONS staff. 

Finally, we will assess the transparency of ONS’s messaging around its achieved sample sizes and the resulting implications for quality and uncertainty.