Adding value for users – focusing on Fitness for Purpose and Value
Fitness for purpose
We heard from some users during our exploratory review that they were keen to have information upfront about whether the statistics were suitable for their own use. They said that the NS designation showed that the producers were being held accountable which was somewhat reassuring. However, they felt that the designation itself did not convey the information they needed about the quality of the statistics and data.
In our consultation paper, we highlighted the importance of clear descriptions of uncertainty to aid user understanding, building on the guidance of the Best Practice and Impact (BPI) Division of the Office for National Statistics and the work of the Winton Centre for Risk and Evidence Communication at the University of Cambridge.
We asked: how can statistics producers provide users with clear current information about the quality of the data/statistics to aid users’ choice and interpretation?
- Do you have any examples of measuring and displaying the uncertainty in estimates?
- How can you summarise the strength of the evidence about fitness for purpose, for example, giving a professional judgement about the evidence compiled in methods and quality information about the five quality dimensions (relevance, accuracy and reliability, timeliness and punctuality, accessibility and clarity, comparability and coherence)?
- How might the fitness for purpose statement account for different known uses?
- Are there other ways you can suggest, to summarise the professional judgement to assist users?
Producers told us
Producers were strongly supportive of having effective summaries of quality to support users. They recognised that only expert users were likely to access the detailed method and quality guidance material that they provide, but that these are important statements about the statistics, nonetheless.
Producers highlighted some ways in which they give summary quality information about their statistics. While most quality information is provided in background quality notes, producers said that they also put important information about the meaning or quality issues alongside the statistics or in glossaries. Infographic-style summaries in statistical outputs, clearly showing what users can and cannot do with the statistics, were highlighted as being effective ways of giving prominent guidance. Confidence intervals are used to show uncertainty around estimates, and descriptions of findings are limited to statistically significant results. Defra highlighted the use of a traffic lights system in the Natural England MENE dashboard in relation to sample size, guiding the user on the degree of confidence they could have in the results for local areas.
Another means of showing the standard of quality management was suggested that could be adopted by producers is seeking ISO 9001 certification of quality management systems. ONS has gained the ISO 9001: 2015 for its quality management of the consumer prices index. ISO accreditation is used in some other countries by national statistical institutes, such as by Statistics Sweden obtaining the ISO 20252 (international standards for the production of marketing, opinion and social research) for all of its official statistics.
Producers suggested a range of ways in which aspects of quality could be summarised for users using quality metrics, including in dashboards, spider plots, and rating scales (such as RAG ratings, traffic lights, five- or three-star ratings). There were other suggestions that focused more on a narrative description about quality, such as the producer providing one or two lines about quality, feeding health warnings with headlines, using a Trip advisor-type presentation (giving user ratings on individual characteristics of a product), and for the GSS adoption of a standard template or form presented upfront about strengths and limitations of the statistics, e.g. a table with ‘can’ in one column and ‘can’t’ in the other.
What next?
The Winton Centre has highlighted both the benefits of explaining uncertainty well and the risks associated with poor communication on the understanding about some statistics. Setting out plainly the merits of data and statistics will support appropriate use, guard against inappropriate use and minimise the risk of users failing to use statistics that would be of help to them through a misunderstanding of their nature and quality. This information is a practical demonstration of meeting Code standards under the Quality and Value pillars.
We will further research ways to prominently explain about the quality of the statistics to support user choice, drawing on the ideas suggested in this review, working with the Data Quality Hub and Quality Champions, and with the advice of the Winton Centre. We will also review research commissioned on other regulators to learn from their approaches. The solutions proposed will undergo development and testing involving stakeholders.
Value
We raised in the consultation paper the concept of National Statistics being used to tell the national life story, and whether there was benefit in bringing together the statistics in a portal for wider access by users. The ideas were based on the premise that National Statistics are the most important statistics for the producer bodies.
We asked: how can we tell the national life story using National Statistics?
- What is the purpose of the statistics you produce? Where does your organisation fit into telling the national life story?
- What is in your collection of National Statistics?
- Are they the statistics that represent the key statistics for telling your part of the national life story?
- Are there important statistics missing from your National Statistics collection?
- Are there any you think are less relevant to the national life story?
- Do you see value in establishing a National Statistics collection/portal?
- Is there value in including some official statistics and non-official statistics, including from outside government e.g. Understanding Society, British Social Attitudes Survey, in the collection of statistics to tell the national life story?
Producers told us
There was an unequivocal response to the notion of the National Statistics being the important statistics for a producer body – they are not. As noted in the section on Provenance, the choice of which official statistics were National Statistics was a legacy decision and did not reflect current importance in the view of heads of profession. Furthermore, the impact of the coronavirus pandemic has shifted the focus onto rapidly developed statistics required to meet the moment.
The idea of bringing together official statistics to tell the national life story was also regarded as problematic – at least it would need to reflect ‘life stories’, as producers strongly felt that there could not be a single representative life story across the UK. Producers though thought it is important to bring together statistics and provide a narrative. This was felt to be helpful for topic areas, rather than seeking to give one overall commentary.
There was interest in the suggestion of a portal, and in fact the GSS have a programme underway, the Integrated Data Programme, which is developing tools to bring together official statistics from across OS producer bodies. One aspect of the development includes a platform to enable users to better access the statistics. This is an encouraging development with great promise.
Producers identified other important aspects of delivering valuable statistics for users, such as, by providing granular data, speedy and timely information, curated topics, and a greater use of dashboards for bringing together statistics.
What next?
Delivering valuable statistics was seen as essential but not best achieved through considering a national life story. Rather than pursue bringing together statistics to form a national life story, we will consider other ways that can add value to users, through producer collaboration and sharing insight and clear communication about the provenance and fitness for purpose of official statistics. We will work further with the Good Practice Team in the Best Practice and Impact Division in ONS and with the Presentation and Dissemination Champions. Proposals will be developed and tested with stakeholders.
Thank you
We are very grateful to all the heads of profession and analysts in OS producer bodies across the UK that participated in the discussions through this phase of the review (see Annex) and look forward to the next phase in shaping the National Statistics designation to meet today’s needs.