In our latest blog, Statistics Regulator Emily Carless discusses the importance of quality in economic statistics…
It’s an exciting time to be a regulator of economic statistics in the UK. The economic statistics landscape is changing, with more new and innovative data available than ever before. The challenges presented by increases in the cost of living have also put greater attention on economic data. The regulatory landscape has also changed: the UK’s departure from the EU means the European statistical office will no longer have a role in verifying the quality of UK statistics. This has created an opportunity for us, as the independent regulator of official statistics in the UK, to develop a new programme to provide continued assurance to users on the quality of economic statistics.
Since March this year I have been leading this programme, which we are calling the ‘Spotlight on Quality: Assuring Confidence in Economic Statistics’. As part of this programme, we are carrying out a series of quality-focused assessments. To help us do this, we are developing a quality assessment framework. We started with the Quality pillar of our Code of Practice for Statistics, and have adapted it by taking relevant practices from other international frameworks to enable a deeper dive on the many elements of quality in economic statistics.
Earlier this summer we published the report of the first pilot assessment on Producer Price Inflation statistics, which are produced by the Office for National Statistics (ONS). We gathered feedback on the assessment and the programme more generally to get insight into how we can maximise the benefits of the programme. My colleague Job led the first pilot assessment and has this to say:
“ONS’s Producer Price Indices (PPIs) were an excellent candidate for the first quality-focused assessment. The new quality assessment framework that we developed helped us investigate all aspects of quality in greater detail than we normally would in an assessment. We’re continuing to refine the framework to make it an even more effective tool. One thing we trialled with this assessment, which worked particularly well, is speaking to the international statistical community about the UK’s statistics. Conversations with price statistics experts in other National Statistical Institutes gave us an insight into the extent to which ONS is following international best practice in producing PPIs. We’ll look to replicate this approach in other quality-focused assessments.”
We received positive feedback about our quality focused assessments from the experts we spoke to in other NSIs. Espen Kristiansen of Statistics Norway commented on the benefits to the international community of our programme:
“It is very valuable to have an external review of statistics. It reminds us how we can frame a discussion about quality, which for statistics is a surprisingly complex and many-faceted concept. Not only can the review detect flaws, it also makes us better at continuously improving our statistics in the future.”
Users of economic statistics in the UK have also found our quality-focused assessment on the PPIs useful. Cheryl Blake from the Ministry of Defence said:
“We welcome the findings of the OSR report on the quality of the PPIs and the opportunity to provide user feedback, particularly the recommendations to publish weights and standard errors which will greatly help our team better understand the underlying drivers of the indices and their quality. This is of growing importance as there is growing interest across Government in using industry-specific PPIs for indexing contracts to better track prices and provide a fairer return to industry. Further information on sample size and resultant quality will help inform index selection and we hope to work with ONS regarding the rationalising on index publication to ensure the provision of critical indices for indexing Government contracts.”
In carrying out these assessments we aim to be supportive of the producers of economic statistics, to champion where they have improved the quality of their statistics as well as to challenge where further improvement is need. Chris Jenkins, Assistant Deputy Director for Prices division in ONS, highlighted the benefits of these assessments to the producer team:
“The regular churn of our monthly statistics sometimes means we don’t get the opportunity to review the methods and metadata that supports our production process as frequently as we would like. This assessment provided us with the perfect opportunity to take stock of what we do, and the constructive support from OSR highlighted areas where we are doing really well, but also areas where we need to make quality improvements. Having the assessment now gives us a clear set of actions we can take to improve the quality of PPI, which is a key economic indicator.”
The outcomes from our quality-focused assessments also provide useful information on where the quality of other statistics could be improved. We will work with statistics producers and the Statistical Heads of Professions to share good practice and shine a light on where the quality of statistics more broadly can be improved. Rachel Skentelbery, ONS Deputy Head of Profession for Statistics, said:
“We welcome OSR’s new quality-focused assessment programme which complements work we are leading to support ONS colleagues in assessing and improving the quality of statistics. Many of the findings identified in these assessments will be relevant beyond the specific output being reviewed, for example in areas such as RAP [reproducible analytical pipelines], quality assurance, user needs and transparency, and we look forward to working together to share lessons and promote best practice to drive improvements to quality across the organisation.”
Over the next few months we will be publishing the report of our second pilot quality-focused assessment on the Profitability of UK companies and Gross Operating Surplus of private non-financial corporations, and publishing the quality assessment framework that we have developed for use in these assessments.