Annex 3: Quality grading of statistics [prototype]

Quality means that data and methods produce assured statistics – the statistics fit their intended uses, are based on appropriate data and methods and are not materially misleading.

3* Good quality = Meets intended use or uses well and likely to be suitable for other uses

2* Fair quality = Meets intended use or uses reasonably well and somewhat likely to be suitable for other uses

1* Limited quality = May be useful as indicative statistics and unlikely to be suitable for other uses

No star = not graded

Note: All official statistics are regarded as of usable quality for their intended uses or would not be published. If they are below par, check if you should be publishing the statistics.

How do quality issues impact the statistics?

Start from a quality score of 10. Consider whether there are quality issues associated with each dimension and decide whether to drop a point where there is a negative impact (worse = -1), remain the same (no impact = 0), or add a point where strengths or mitigations overcome earlier identified problems (better = +1).

Consider these quality dimensions to inform your rating of the statistics:

• Relevance: How well does the source match the concept being measured; how well do the statistics meet the needs of users; have you been able to mitigate any limitations that help to support use?
  If you identify issues that affect the statistics, drop one point; otherwise remain at 10.
• Completeness: Are the statistics based on data that have sufficient coverage for the population being measured; are data records missing important information; are you able to minimise the impact of missing data?
  If you identify issues that affect the statistics, drop one point; otherwise remain level.
• Accuracy: How well do the data reflect the real-world values; are there any biases in the data or the statistics; what is the level and nature of sampling variability?
  If you identify issues that affect the statistics, drop one point; otherwise remain level.
• Consistency: Are the statistics sufficiently consistent over time; are there any recent changes to data or methods that affect the statistics; are the data internally coherent; are they comparable with other related sources and show credible patterns?
  If you identify issues that affect the statistics, drop one point; otherwise remain level.
• Timeliness: Does the lag between the reference period and publication impact the currency of the statistics; could a lag or a focus on recent data lead users to be misled?
  If you identify issues that affect the statistics, drop one point; otherwise remain level.
• Adjustments: What adjustments have been made to respond to any quality dimensions; how far have you been able to address known quality issues?
  If you identify a means to overcome issues, add a point; otherwise remain level.
• Uncertainty: What is the likely size and direction of uncertainty in the statistics; what is the implication of the level of uncertainty for the interpretation and use of the estimates; what is the risk of users being misled by the statistics?
  If you regard uncertainty to be large, drop one point; if you regard uncertainty to be small, add one point; otherwise remain level.

Scoring

If you score 9-12, the rating is 3* Good quality

If you score 5-8, the rating is 2* Fair quality

If you score 1-4, the rating is 1* Limited quality

This rating of the statistics allows for the trade-offs in quality dimensions and the scale of uncertainty in relation to the intended uses made of the statistics. It should reflect the confidence that users can have in the statistics and the level of risk that the statistics may mislead users.

Now write a brief pen picture setting out your grading decision and rationale, highlighting the key strengths and limitations of the statistics

How well does the grade match your initial expectation? If there are unusual circumstances that lead to a score of 8, but the statistics are widely accepted to meet their primary purposes, you might want to consider whether they should be scored 3*. Similarly, if there is a score of 9 with the statistics affected by a single high impact issue, consider whether 2* is more appropriate.

Other quality indicators

Decide if other quality dimensions are more relevant, for example:

• Uniqueness: Have the data been cleaned sufficiently, for example, to remove duplicates; have the weaknesses in the data set been addressed to improve the usefulness of the statistics?
• Validity: How far have you been able to overcome invalid records to enable the data to be useful?

Take stock

How would other quality indicators change your view of the quality grade?

Speak with colleagues about your quality grading and see if they reach a similar conclusion.

Look to benchmark your quality grading judgement in your area and organisation.

Revisit your scoring at reasonable intervals to consider how quality has changed – remember quality is not fixed but dynamic.