4. Assured Quality

4.1 The environment and organisational culture prioritise quality in statistics

Indicator 4.1 Organisations are open about their commitment to quality, make clear their approach to quality management and create an environment that prioritises quality in statistics. They ensure that the organisational culture, structure and tools are in place to manage quality effectively and promote and adopt appropriate quality standards. Individual sets of statistics are produced in line with the organisation’s approach to quality management.

Quality is ensured not just through the approaches and actions of the producer teams responsible for the individual sets of statistics but through an organisational culture and procedures that place a high value on quality. By being open about their commitment to quality and their approach to quality management, both within the organisation and publicly, organisations can ensure they have the right culture and procedures in place. Ensuring that the environment within which statistics are produced makes quality is a priority and that the tools and processes are in place to manage quality effectively can help ensure that quality is prioritised throughout the organisation. The promotion and adoption of appropriate quality standards can further support producer teams in producing quality outcomes.

This indicator is derived from the Code practice T4.5. Similar indicators also exist in the international frameworks. For example, the IMF DQAF includes indicator 0.4.1, ‘Processes are in place to focus on quality.’ The ESS QAF includes indicator 4.1, ‘Quality policy is defined and made available to the public. An organisational structure and tools are in place to deal with quality management.’ The requirement for the statistics  being assessed to be produced in line with the organisational approach has been added.

4.2 Quality meets users’ needs

Indicator 4.2: Statistics are produced to a level of quality that meets users’ needs. The strengths and limitations of the statistics and data are evaluated in relation to different uses, and trade-offs between dimensions of quality are fully understood.

Statistics will often have several different uses and different strengths and limitations in relation to each. Producers should evaluate the strengths and limitations of their statistics and data in relation to the known and anticipated uses of key users but also proactively explore other potential uses of the data. Where there are competing priorities between different dimensions of quality, these should be fully understood and the implications of decisions on the different uses of the statistics fully explored to ensure that the quality meets users’ needs.

This indicator is derived from the Code practice Q3.1 with references to trade-offs between dimensions of quality added to reflect wording in the ESS QAF indicator 4.3, ‘Output quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics’. The part of the Code practice around the strengths and limitations being clearly explained to users has been moved to a later indicator on transparency of output quality.

4.3 Proactive user engagement around quality

Indicator 4.3: The producer actively seeks, and acts on, input from users about all dimensions of quality of the statistics and data through proactive user engagement.

Users of statistics and data should be at the centre of decisions about statistics. Their needs should be understood, and their views sought and acted on. As a key aspect of quality is that statistics fit their intended uses, it is important that producers actively seek user input through proactive user engagement. This feedback should include satisfaction with, and emerging needs, around all dimensions of quality. This will enable producers to understand the competing priorities between dimensions of quality for different uses and where user needs are not being met, facilitating discussion on the highest priority improvements to quality that are required.

This indicator is derived from the Code practice V1.3, ‘User satisfaction with the relevance and usefulness of the statistics and data should be reviewed routinely. This should consider the timeliness, accessibility, clarity and accuracy of the statistics and data.’ The ESS QAF has a similar indicator, 11.3, ‘User satisfaction is monitored on a regular basis and is systematically followed up’, and the IMF DQAF includes an indicator about monitoring the relevance and practical utility of existing statistics in meeting user needs.

4.4 Accuracy and communication of uncertainty and bias

Indicator 4.4. User needs around the accuracy of the statistics are considered and the nature and scale of any uncertainty and bias in the estimates are understood and clearly explained.

A key determinant of quality of statistics is their accuracy, that is, the difference between the estimate and the true value. Uncertainty and bias are inherent in all statistics to a varying degree and should be understood by the producer team as part of understanding the quality of their statistics and data. Different users may have different requirements around the accuracy of the statistics depending on their use. Producers should understand user needs for accuracy and factor this into their methods and processes. The nature and scale of uncertainty and bias should be explored and the effect on the statistics understood and communicated. As set out in our report Approaches to presenting uncertainty in the statistical system, ensuring that uncertainty around estimates is conveyed well is critical to the appropriate use and interpretation of statistics. Our report also found that different users may want different information about uncertainty depending on the nature of the decisions they’re faced with making and their level of expertise. Guidance on communicating uncertainty has been published by the Government Analysis Function to support analysts.

This indicator is derived from the last part of Code practice Q3.3 with explicit reference to accuracy and to understanding and communicating bias added.

4.5 Timeliness

Indicator 4.5: Statistics and data are released on a timely basis and at intervals that meet the needs of users as far as practicable. The statistics are released as soon as they are considered ready.

A key dimension of quality for users is the timeliness of the statistics. This relates both to how long after the end of the reference period the statistics are published and the frequency of the publication. As we have commented on in our State of the Statistical System 2022/23 report, there is continued demand for more-timely statistics. Increased timeliness creates tension with other dimensions of quality as more-timely statistics may be based on fewer data, and so have reduced accuracy, or on data that do not match the concept of interest as closely. Producers should have a clear understanding of users’ needs around the timeliness of statistics and data and be transparent about the impact of meeting those needs on other dimensions of quality.

The Code discusses timeliness in all three pillars – Trustworthiness, Quality and Value. The wording of this indicator has come from the practice T3.5 in the Trustworthiness pillar. In the Quality pillar, timeliness is mentioned in practice Q3.3 around monitoring and reporting on various quality dimensions. In the Value pillar, timeliness is mentioned in a practice around regularly reviewing user satisfaction. The need for regular reviews of user satisfaction is included in this framework in a later indicator. Both the IMF DQAF and ESS QAF include indicators specifically around periodicity and timeliness meeting dissemination standards in line with their role in assuring the quality of statistics provided to them.

4.6 Granularity

Indicator 4.6: Statistics are published to a level of detail that meets users’ needs whilst protecting confidentiality. Information about quality should be provided alongside granular estimates to support their appropriate use.

Many users of statistics and data have an interest in statistics below the headline national figures. This can be due to a desire to understand the headline statistics better or an interest in sub-populations. It is important that more-granular statistics are produced to meet user needs, whilst protecting confidentiality, to ensure the relevance of the statistics. The quality of these more-granular estimates may be different to that of the headline statistics due to smaller sample sizes or different sources or methods being used to produce the estimates. In addition, statistical disclosure control methods may affect the quality of the statistics, for example, due to rounding techniques affecting accuracy. Producers should seek to understand the quality of granular estimates and communicate it to users alongside the estimates to support their appropriate use.

This indicator is derived from Code practice V2.4. Elements around protecting confidentiality and providing information about the quality of granular estimates have been added. The indicator was included in this framework following stakeholder feedback on the importance of granularity including during our pilot assessments, which found that the availability and quality of more-granular statistics was a key concern of users. The IMF DQAF mentions granularity in a practice on data users being consulted or kept informed on specific aspects of current data, including the usefulness in terms of detail. The ESS QAF includes indicator 3.2, ‘The scope, detail and cost of statistics are commensurate with needs. ‘

4.7 Transparency of output quality

Indicator 4.7: The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, is monitored and reported regularly.

In order for users to use statistics effectively and appropriately, they need to understand the quality of the statistics and data. Understanding the accuracy and reliability of the statistics and data can ensure that they do not place too much weight on the statistics or make decisions based on small movements in trends or differences between groups where this is not appropriate. Understanding the coherence and comparability of the statistics and data enables users to only make appropriate comparisons between time periods, geographic areas or data sources. Understanding the timeliness and punctuality of statistics and data enables users to understand to which time period the statistics refer and whether that is suitable for their needs. Providing this information clearly alongside the statistics and data ensures that users have the information they need to not misuse the data. Producer teams should regularly monitor all these dimensions of quality, and any other suitable quality dimensions, and report on them to ensure that users of the statistics and data understand the fitness for their use and can use them appropriately.

This indicator is derived from the first part of the Code practice Q3.3. The ESS QAF includes a similar indicator, 15.7: ‘Users are kept informed about the quality of statistical outputs with respect to the quality criteria for European Statistics.’

4.8 Provision of metadata

Indicator 4.8 Up-to-date and relevant metadata are accessible alongside the statistics and data.

Whilst quality dimensions, such as the ESS quality dimensions, provide information on the quality of a set of statistics, metadata provide information on a particular release of that data. The metadata can include a range of indicators, such as the response rates for a survey source or the coverage of an administrative source. These things can change over time, so producer teams should ensure that the metadata are kept up to date and are accessible alongside the statistics and data. Where these metadata are not kept up to date, users can get a misleading impression of the quality of the data.

This indicator is derived from IMF DQAF indicator 5.2, ‘Up-to-date and pertinent metadata are made available’. The ESS QAF also includes several indicators on metadata, such as indicator 15.1: ‘Statistics and the corresponding metadata are presented, and archived, in a form that facilitates proper interpretation and meaningful comparisons.’ It is also reflected in Code practice V1.3.

4.9 Proportionate quality assurance

Indicator 4.9 Quality assurance arrangements are proportionate to the nature of the quality issues and the importance of the statistics in serving the public good.

Producers should be curious about the statistics that they are producing and explore any unexpected results effectively. The nature of the data sources and methods on which statistics are based varies. Therefore, different sets of statistics and data will require different quality assurance arrangements. Producers should ensure the arrangements are proportionate to the nature of the quality issues and the importance of the statistics in serving the public good. For example, the quality assurance arrangements for statistics that are based on a simple summing of administrative data records with a small user base will be different to the arrangements for a set of statistics that are based on multiple data sources and which are used for a variety of major policy decisions.

This indicator is derived from the first part of the Code practice Q3.2. Similar indicators are not included in the international frameworks. The ESS QAF includes a principle around cost effectiveness, and the IMF DQAF includes indicators around the efficient use of resources.

4.10 Risk minimisation

Indicator 4.10. The risk quality issues pose to statistics and data and their impact are minimised to an acceptable level for the intended uses, taking users’ needs of quality and uncertainty into account.

The risk and impact of quality issues will vary depending on the statistics that are being produced. For some sets of statistics, quality issues will have a larger impact than for others. Producers should aim to understand the intended uses and users’ needs regarding quality and uncertainty and take these into account in identifying the impact of quality issues. Users’ views on the acceptable risk of quality issues should also be taken into account in developing processes. For example, the processes used for statistics that need a high degree of accuracy and for which there would be a high impact from errors should provide a higher degree of risk minimisation with additional quality assurance processes to ensure that errors do not occur.

This indicator is derived from the last part of the Code practice Q3.2. Risk management is also referred to in the institutional methods underpinning indicator 4.1 in the ESS QAF.

4.11 Application of Reproducible Analytical Pipelines (RAP) principles

Indicator 4.11: Wherever possible, Reproducible Analytical Pipelines (RAP) principles are implemented to embed robust quality management, improve transparency of the process and reduce the risk of errors.

In 2021, OSR published a review called Reproducible, Analytical Pipelines: Overcoming Barriers to adoption. The reproducible analytical pipeline, also known as RAP, is a set of principles and good practice for data analysis and presentation. RAP was developed as a solution to several problems, including time-consuming, error-prone manual processes. RAP combines modern statistical tools with good practice in software development  to allow all the steps of statistical production, from input data to the final high-quality output , to be carried out in a sustainable and transparent way. The use of RAP helps ensure the quality of the output through embedding robust quality management in the statistical production process. It improves the transparency of the process and reduces the risk of errors.

This indicator builds on the practices in the Code, such as T4.3 and T4.5, and reflects our 2021 report and our subsequent focus on supporting producers in using RAP principles to ensure the quality, sustainability and transparency of their processes.

4.12 Validation with other data sources

Indicator 4.12: Statistics are validated through comparison with other relevant statistics and data sources. The validation process is clearly communicated to users.

Other statistics and data sources can provide useful information to validate statistics and assure their quality. Other sources may provide data on related concepts or on concepts that vary in a similar way. By using these sources to validate the statistics, producers can be more confident of their data. This validation process should also be communicated to users so that they can understand how the statistics have been validated and how they relate to other statistics and data.

This indicator is derived from the second Code practice, Q3.3, with the addition of communication of the validation process to users. Similar indicators are included in the international frameworks, for example, the IMF DQAF indicator 4.2.3: ‘Statistics are consistent or reconcilable with those obtained through other data sources and/or statistical frameworks.

4.13 Transparency of quality assurance

Indicator 4.13. Statistics producers are transparent about the quality assurance approach taken throughout the preparation of the statistics. This includes the aspects of quality assurance carried out by other teams or organisations.

Transparency around the quality assurance approach taken can help users to understand more about the quality of the data. Producers being open about what they have found through being curious in their quality assurance will help users understand the reasons for any unexpected results. It can also help in conversations with stakeholders around the level of quality assurance that is needed. Being transparent around quality assurance can also enable teams to fully understand their own quality assurance approach more fully. It can help improve quality by facilitating collaboration over quality assurance approaches between teams. This transparency should include aspects of quality assurance undertaken by other teams and organisations. This allows for a more rounded understanding of the quality assurance and reduces the potential for either duplication of effort or some quality assurance steps to be missed.

This indicator is derived from Code practice Q3.2. The ESS QAF also includes indicators around information on processes being publicly available.

4.14 Quality of provisional estimates

Indicator 4.14: Data accuracy and reliability are considered before the publication of preliminary estimates. When preliminary estimates are released, appropriate information is provided to the user about the quality of the published estimates.

For some sets of statistics, there is a user need for more-timely estimates. Producers need to consider whether the accuracy and reliability of preliminary estimates are sufficient for them to be published. These preliminary estimates might be based on a lower response to a survey, prior to a quality assurance-checking exercise on an administrative data source or based on different, more-timely data sources. Producers and users need to understand the quality of these preliminary estimates in relation to subsequent estimates, and appropriate information should be published alongside them. Any additional uncertainty or bias in preliminary estimates should be clearly communicated. In the example of estimating quarterly Gross Domestic Product in the UK, provisional estimates are based on the output approach; subsequent estimates then include income and expenditure approaches. These estimates will then be revised in future years when annual data have been included and supply use balancing is applied. Quarterly estimates of GDP may be revised for many years.

This indicator is derived from practices in the ESS QAF under indicator 13.5, ‘Preliminary results of acceptable aggregate accuracy and reliability can be released when considered useful.’ There are also practices around the dissemination of preliminary results taking account of data accuracy and reliability and appropriate information being provided to the user. The IMF DQAF also includes practices around the ‘Assessment and validation of intermediate data and statistical outputs’ and ‘Preliminary and/or revised/updated data are clearly identified’. Although provisional estimates are not specifically mentioned in the Quality pillar of the Code, this indicator relates to practices on understanding and communicating quality more generally which are widely included in the Code.

4.15 Explanation of revisions and corrections

Indicator 4.15 Scheduled revisions, or unscheduled corrections that result from errors, are released as soon as possible and explained alongside the statistics, being clear on the scale, nature, cause and impact.

Scheduled revisions and unscheduled corrections should be released as soon as possible to ensure that users have the best-quality estimates available. Schedules for revisions should be transparent. Users should be notified clearly when corrections are made. The revisions and corrections should be clearly explained alongside the statistics so that users can understand the impact on their use of the statistics. These explanations should include clear information on the scale, nature, cause and impact of the revisions and corrections. Where revisions or corrections alter the narrative that the statistics provide, this change in narrative should be made clear to users.

This indicator is adapted from the Code practices Q3.4 and T3.9. Similar practices are included in the international frameworks, such as ESS QAF indicators 6.3, ‘Errors discovered in published statistics are corrected at the earliest possible date and publicised’, and 8.5, ‘Revisions follow standard, well-established and transparent procedures’.

4.16 Revisions analysis

Indicator 4.16: Revisions analysis is conducted and published on a regular basis. The analysis examines differences between preliminary and revised estimates where applicable.

Where revisions are routinely made to estimates, revisions analysis provides useful insight into the quality of preliminary estimates, including the amount of uncertainty and any bias in the estimates. It is helpful for producers to understand whether any changes to sources, methods or processes might help improve the quality of preliminary estimates. It is helpful for users in understanding the likely scale of revisions and whether they have historically been in the same direction. Therefore, the analyses should be both conducted and published transparently on a regular basis. There is a variety of measures of the scale of revisions, each of which will bring different insight into the revisions, and producers should consider and engage with users to discuss which are the most useful.

This indicator is derived from practices in the IMF DQAF. These are 3.5.1 (first part), ‘Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes’, and 4.3.3, ‘Studies and analyses of revisions are made public’. The ESS QAF also includes a practice on revision studies. Revision studies or analysis are not explicitly mentioned in the Code but will be part of regular reviews included in Code practice Q3.5 on the strengths and limitations in the data and methods.

4.17 Reduction in revisions

Indicator 4.17:  Revisions analysis is used to reduce future revisions by informing improvements to sources, methods, processes and outputs, as appropriate.

Producers should ensure that they use the results of any revisions analysis to inform improvements to sources, methods, processes and outputs. Having an understanding of the scale, nature and direction of revisions can help inform improvements through identifying any systemic issues in preliminary estimates. These improvements can help to increase the accuracy of preliminary estimates and reduce future revisions.

This indicator is derived from ESS QAF indicator 12.3, ‘Revisions are regularly analysed in order to improve source data, statistical processes and outputs’, and the second part of IMF DQAF indicator 3.5.1, ‘Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes’. Acting on revision analyses is not explicitly mentioned in the Code but will be part of being open in addressing issues identified and transparent about decisions as a result of reviews in line with Code practice Q3.5.