Experimental statistics – official statistics in development

About this guide

This guide sets out the Office for Statistics Regulation’s expectations regarding the production and handling of experimental statistics, a subset of official statistics going through development and evaluation, in line with the Code of Practice for Statistics.

The Code of Practice encourages innovation and improvement and highlights the need for National Statistics and other official statistics to remain relevant for use, to provide a dynamic public service.

The GSS Good Practice Team has published guidance on the GSS Policy and Guidance Hub on best practice in relation to experimental statistics for statistics producers.

Innovation and engagement

The Code of Practice encourages innovation and improvement and highlights the need for National Statistics and other official statistics to remain relevant for use, to provide a dynamic public service.

This is not a call for change for change’s sake. Instead producers should keep alert to the developments and opportunities in evolving technologies and methods, to improve estimates, to better meet the public good.

It requires a growing understanding of emerging user needs, to support producers in recognising when existing statistics are no longer relevant and can be replaced by new or improved statistics.

Experimental statistics status provides a clear statement of the nature of the official statistics going through development, with a potentially wider degree of uncertainty in the resulting estimates as the methods and processes are established and verified.

Testing of the experimental statistics allows producers to gain a good understanding of their quality, including their accuracy and reliability, and their value. Users are central to this process – without their involvement, producers will have an incomplete understanding of the suitability of the statistics.

Development approach

The overall goal should be to ensure that the experimental statistics are used to develop statistics that can go onto meet the standards of the Code of Practice for Statistics. The evaluation should determine whether the statistics are of sufficient quality and value to be published as official statistics. If they are not, the development and  production should stop. However, where the statistics are useful and there is the potential to improve the quality, the development should continue to try and realise it.

Openness and transparency about the development, the decision-making process and the nature of the evaluation is important. It is essential for the effective and timely completion of the statistics development that a plan is used that makes clear the points at which evaluation will be conducted.

When producing experimental statistics, it may be helpful to describe the development in terms of who, why, what, and when.

Who: Producers should be clear about who is carrying out the development and testing. They should also be clear about how they will involve users and method or topic experts in the process, and work to get their active involvement.

Why: Producers should help users understand the breadth of the development, why it is needed and important – the potential value that the new or re-developed statistics may offer.

What: Producers should be specific in setting out the scope and nature of the development. They should also be clear about the criteria being used to decide whether the statistics are of sufficient quality and how they will determine that the statistics meet users’ needs. They should be clear on what can be achieved with the available resources.

When: An essential element of developing experimental statistics is that the process is timebound. This means that producers should set out the timeframe that they expect the development to run, giving a clear idea of expected milestones. It may be that the development runs for months or even several years – it is important that the likely period of development is set out.

As evaluation proceeds, it may be the case that the plan needs to be revised. Again, being open about changes to the plan is important and helps maintain user confidence in the statistics and the producers.

The conclusion of the development will be the producer’s judgement of one of three outcomes:

1. The statistics are of sufficient quality and value to be used in a meaningful way and so be published as official statistics.
2. The statistics are of insufficient quality or do not meet the required need and so production will be stopped.
3. There is insufficient evidence to reach a conclusion or further refinement is requirement – producers would continue the development and be clear about extent of further testing.

When the producer, under the guidance of the Head of Profession, decides that the statistics are useful and of sufficient quality, they should publish the statistics as official statistics, making clear the basis of their decision.

The label ‘experimental statistics’ should not be used indefinitely. Some producers have used the label to indicate that the statistics are of poor quality – that is also not appropriate. It is worth statistics producers reviewing their existing experimental statistics to be certain that a development and evaluation should continue. Where a development has concluded and the production of the statistics is continuing, then the label of ‘experimental statistics’ should be removed; the statistics should be published as ‘official statistics’, making sure to describe their strengths and limitations.

National statistics

Experimental statistics development is applicable to and valuable for National Statistics. To fully meet the standards of the Code of Practice, it is important that producers of National Statistics remain vigilant to identify the opportunities to improve the statistics where appropriate. The world and society continually evolve so it is essential that our most important statistics keep up – a message underlined in the Independent Review of Economic Statistics by Sir Charles Bean.

We strongly encourage producers to maintain the relevance of their National Statistics. The use of experimental statistics provides an excellent means of demonstrating innovation and improvement, while at the same time showing how the statistics can be appropriately used and understood, with producers then using that knowledge to feed back into the development.

The particular National Statistics being redeveloped would be relabelled as ‘experimental statistics’, but only after receiving the agreement of the Office for Statistics  Regulation (OSR), since under the Statistics and Registration Service Act 2007 the NS designation can only be removed with the approval of the regulator. At the end of the project, OSR would conduct either a compliance check or an assessment, to determine that the designation as National Statistics can be reapplied.