Annex A – The framework of indicators for ‘Spotlight on Quality’ assessments

1. Resources, plans and prioritisation

1.1 Sufficient resources

Indicator 1.1: Sufficient human and financial resources are provided to produce statistics that meet users’ needs.

1.2 Good business practices

Indicator 1.2: Good business practices are maintained in the use of resources. Where appropriate, statistics producers take opportunities to share resources and collaborate to achieve common goals and produce coherent statistics.

1.3 Clarity of responsibilities

Indicator 1.3: The responsibility for collecting, processing, quality-assuring and disseminating the statistics is clearly specified.

1.4 Suitable systems

Indicator 1.4: Sustainable, robust and flexible systems are used to produce statistics that meet current user needs and enable innovation and improvement.

1.5 Established development work programme

Indicator 1.5: A development work programme is established, published and regularly reviewed and includes planned improvements to quality.

1.6 User involvement in developing plans

Indicator 1.6: Users and other stakeholders help develop and prioritise statistical plans.

1.7 Transparency of progress towards plans

Indicator 1.7: Statistics producers are open about progress towards meeting development priorities and objectives.

1.8 Transparency of prioritisation decisions

Indicator 1.8 Producers are transparent about prioritisation and how decisions on priorities affect quality.

2. Suitable data sources

2.1 Appropriateness and quality of source data

Indicator 2.1 Statistics are based on data sources that are appropriate for the intended uses. Producers evaluate appropriate quality dimensions in relation to data sources to ensure that statistics are suitable for the intended uses.

2.2 Definitions and concepts of data sources

Indicator 2.2 Data sources are based on definitions and concepts that are suitable approximations of what the statistics aim to measure, or that can be processed to become suitable for producing the statistics.

2.3 Coherence of source data

Indicator 2.3 Source data are coherent across different levels of aggregation, consistent over time, and comparable between geographical areas, whenever possible. Internal coherence of source data is regularly monitored.

2.4 Explanation of data sources

Indicator 2.4 The nature of data sources used, how and why they were selected, and any adjustments applied to them are explained to users.

2.5 Explanation of the quality of source data

Indicator 2.5 Quality of the source data, including potential bias, uncertainty and possible distortive effects, is explained to users and the extent of any impact on the statistics clearly reported.

2.6 Limitations of data sources

Indicator 2.6 The limitations of data sources are identified and addressed where possible. Statistics producers are open about the extent to which limitations can be overcome and the effect on the statistics.

2.7 Relationships with data suppliers

Indicator 2.7 Producers establish and maintain constructive relationships with those involved in the collection, recording, supplying, linking and quality assurance of data.

2.8 Statement of data requirements

Indicator 2.8 Producers share a clear statement of data requirements with the organisations that provide that data, setting out decisions on timing, definitions and format of data supply, and explaining how and why the data will be used.

2.9 Source metadata

Indicator 2.9 Producers specify and receive appropriate metadata with each data delivery to ensure the quality of the data is understood.

2.10 Regular review of source data

Indicator 2.10 Producers regularly review data sources to ensure that they continue to be suitable.

2.11 Innovation in sourcing data

Indicator 2.11 Producers are innovative with their approach to sourcing data and consider alternative data sources to facilitate better-quality or timelier statistics, where appropriate

2.12 Explanation of changes to data sources to users

Indicator 2.12: The effect of changes in the circumstances and context of a data source on the statistics over time should be evaluated. Reasons for any lack of consistency and related implications for use should be clearly explained to users.

2.13 Monitor and minimise burden

Indicator 2.13: Statistics producers are transparent in their approach to monitoring and reducing the burden on those providing their information, and on those involved in collecting, recording and supplying data. The burden imposed should be proportionate to the benefits arising from the use of the statistics.

2.14 Collaborate to maximise use of data

Indicator 2.14: Statistics producers communicate and collaborate with others to maximise their use of administrative data, data sharing, cross analysis of sources and the re-use of data to avoid duplicating requests for information.

3. Sound Methods

3.1 Use of appropriate methods and processes

Indicator 3.1: Methods and processes are appropriate and based on national and international good practice, scientific principles or established professional consensus.

3.2 Use of recognised standards, classifications and definitions

Indicator 3.2: Statistics, data and metadata are compiled using national and international recognised standards, classifications and definitions which are harmonised to be consistent and coherent with related statistics and data where possible.

3.3 Explanation of reasons for deviations from standards

Indicator 3.3: Reasons for deviations from standards, classifications and definitions are clearly explained, including any implications for use of the statistics and data.

3.4 Transparency of methods

Indicator 3.4: Producers are transparent about the methods, standards, classifications and definitions used, giving the reasons for their selection. The level of detail is proportionate to the complexity of the methods chosen and reflects the needs of different types of users and uses. Published methods information is reviewed and updated whenever needed.

3.5 Explanation of limitations of methods

Indicator 3.5: Limitations of the methods and their application are identified and explained to users, including the effect on the statistics and their use.

3.6 Advance notice of changes to methods

Indicator 3.6: Producers of statistics and data provide users with advance notice about changes to methods, explaining why the changes are being made. Users are made aware of the nature, extent and effect of the changes.

3.7 User feedback on changes to methods

3.7: Producers seek and implement, where appropriate, feedback from users about changes to methods.

3.8 Consistent time series

Indicator 3.8: Where a change in methods leads to a break in the time series, a consistent time series is produced, with back series provided where possible.

3.9 Collaboration to improve methods

Indicator 3.9: Statistics producers collaborate with topic and methods experts, the scientific and international community and producers of related statistics and data to improve methods wherever possible.

3.10 Up-to-date knowledge of developments

Indicator 3.10: Producers keep up to date with developments that might improve methods and quality. They assess the added value of potential improvements and evaluate the likely impact on the statistics, including in relation to comparability and coherence.

3.11 Independent internal and external review

Indicator 3.11: Producers seek independent internal and external review of their statistical methods and are open to identified areas for improvement.

4. Assured Quality

4.1 The environment and organisational culture prioritise quality in statistics

Indicator 4.1 Organisations are open about their commitment to quality, make clear their approach to quality management and create an environment which prioritises quality in statistics. They ensure that the organisational culture, structure and tools are in place to manage quality effectively and promote and adopt appropriate quality standards. Individual sets of statistics are produced in line with the organisations approach to quality management.

4.2 Quality meets users’ needs

Indicator 4.2: Statistics are produced to a level of quality that meets users’ needs. The strengths and limitations of the statistics and data are evaluated in relation to different uses and trade-offs between dimensions of quality are fully understood.

4.3 Proactive user engagement around quality

Indicator 4.3: The producer actively seeks, and acts on, input from users about all dimensions of quality of the statistics and data through proactive user engagement.

4.4 Accuracy and communication of uncertainty and bias

Indicator 4.4. User needs around the accuracy of the statistics are considered and the nature and scale of any uncertainty and bias in the estimates are understood and clearly explained.

4.5 Timeliness

Indicator 4.5: Statistics and data are released on a timely basis and at intervals that meet the needs of users as far as practicable. The statistics are released as soon as they are considered ready.

4.6 Granularity

Indicator 4.6: Statistics are published to a level of detail that meets users’ needs whilst protecting confidentiality. Information about quality should be provided alongside granular estimates to support their appropriate use.

4.7 Transparency of output quality

Indicator 4.7: The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, is monitored and reported regularly.

4.8 Provision of metadata

Indicator 4.8 Up-to-date and relevant metadata are accessible alongside the statistics and data.

4.9 Proportionate quality assurance

Indicator 4.9 Quality assurance arrangements are proportionate to the nature of the quality issues and the importance of the statistics in serving the public good.

4.10 Risk minimisation

Indicator 4.10. The risk quality issues pose to statistics and data and their impact are minimised to an acceptable level for the intended uses, taking users’ needs of quality and uncertainty into account.

4.11 Application of Reproducible Analytical Pipelines (RAP) principles

Indicator 4.11: Wherever possible, Reproducible Analytical Pipelines (RAP) principles are implemented to embed robust quality management, improve transparency of the process and reduce the risk of errors.

4.12 Validation with other data sources

Indicator 4.12: Statistics are validated through comparison with other relevant statistics and data sources. The validation process is clearly communicated to users.

4.13 Transparency of quality assurance

Indicator 4.13. Statistics producers are transparent about the quality assurance approach taken throughout the preparation of the statistics. This includes the aspects of quality assurance carried out by other teams or organisations.

4.14 Quality of provisional estimates

Indicator 4.14: Data accuracy and reliability are considered before the publication of preliminary estimates. When preliminary estimates are released, appropriate information is provided to the user about the quality of the published estimates.

4.15 Explanation of revisions and corrections

Indicator 4.15 Scheduled revisions, or unscheduled corrections that result from errors, are released as soon as possible and explained alongside the statistics, being clear on the scale, nature, cause and impact.

4.16 Revisions analysis

Indicator 4.16: Revisions analysis is conducted and published on a regular basis. The analysis examines differences between preliminary and revised estimates where applicable.

4.17 Reduction in revisions

Indicator 4.17:  Revisions analysis is used to reduce future revisions by informing improvements to sources, methods, processes and outputs, as appropriate.