Appendix 1: Primary Research: Building Trust in Official Statistics
Introduction
This short primary research project complements the adjacent literature review, “Trust and Official Statistics”.
The literature review provides insight into the current levels of trust in bodies, actors and platforms involved in the production and communication of official statistics, as well as providing a broad indication of the sorts of strategies that organisations and academics recommend for increasing trust levels. The literature review identified that further primary research to understand whether members of the public think these strategies would have a meaningful impact on their own trust levels is needed.
This piece of work is specifically tailored to start to fill this gap by analysing data gathered as part of a previous Office for Statistics Regulation (OSR) research project, where members of the public were asked what they think would help increase their trust in official statistics. This supports a user-focused ambition and provides an initial picture of the types of strategies that members of the public would consider to be effective to build trustworthiness, and consequently levels of trust.
Methodology
The analysis presented here is based on the open-text responses to a survey question asked as part of the OSR research project Statistics in Personal Decision Making. Given the source of these data, any occasions where the themes identified here echo those presented in the project Statistics in Personal Decision Making will be signposted.
This research focuses on analysis of free-text answers to the question: ‘what might increase your trust in official statistics?’ This question was asked as part of a broadly nationally representative online survey conducted in October 2024. More details about the survey can be found in the Appendix of OSR’s published Statistics in Personal Decision Making report. The full survey can be found in Appendix 2. Appendix 5 provides more detail about how the survey was designed, and Appendix 7 includes details of the survey sample composition.
The survey responses (n=1405) underwent an inductive qualitative data analysis (QDA) process where classification was based directly on the data as opposed to a predetermined list of codes. NVivo software was used to structure the organisation of codes and to facilitate the identification of patterns and themes. This approach adhered to Braun and Clarke’s (2006) six steps, following the systematic process of: familiarisation; preliminary coding; collation and theme generation; reviewing themes; defining and naming themes; discussion and write up.
During the familiarisation phase, some responses were removed (n=173). As a consequence of this being the final question in OSR’s online survey, some responses made references to earlier answers (n=4). In addition to this, responses which stated the respondents already trusted official statistics (n=41), did not propose a solution or dismissed the prospect that anything could improve their trust (n=97), or declared that they do not use them (n=11), were also removed. The final category removed during this phase was irrelevant or nonsensical responses (n=20). Following this, the remaining 1232 responses were preliminary coded.
Continuing the process, during the collation and review phases, seven themes were identified. These are: quality product; verification and reviews; integrity and transparency; user relevance; communication; trust in producers; and finally, data management. In addition to this, 32 sub-themes were also created. The inclusion of sub-themes enriches the analysis, as it reveals what aspects of each broad theme respondents considered to be necessary and presents this in a more coherent manner than isolated codes would allow.
Within the analysis, responses which reflected more than one theme were coded accordingly, with duplicates treated as examples of both themes. However, a different approach was taken for duplications within the respective themes. Duplications at this level were manually reviewed, and a subjective decision was taken as to which sub-theme best captured the survey response.
Results
This small-scale thematic research analysed responses members of the public gave when asked, ‘what might increase your trust in official statistics?’ To present this thematic analysis, each sub-theme is provided in an expandable box. These boxes present the exemplar answers from survey respondents, alongside the sub-theme with which they correspond. Verbatim quotations of the specific answers given within the survey are indicated by quotation marks.
Confidence in the Output
This sub-theme includes recommendations of accuracy, credibility, quality and reliability, alongside requests for ‘evidence to support them’ and ‘proof they are true’.
Methods
This sub-theme includes pleas to dedicate further effort to explicitly outlining the data collection process and citing sources. Further to this, clear articulations of methodologies, including ‘paper trails [users] could follow’ and requests for the inclusion of more detail are also included. The final element incorporated within this sub-theme refers to sample sizes, with the importance of ‘a well-established research sample size’ and efforts to get ‘more people in the surveys’ requested by respondents.
Useful Output
This sub-theme incorporates suggestions relating to the importance of access, use and context. Access relates to ease of access to all the necessary material, as a precursor to effective use. This includes being ‘fully open to the public’, as well as ‘access to raw data’, ‘more open-source statistics’ and ‘the ability to view all the data not just the results’. Use incorporates ‘consistency and comparability over time’ with pleas for ‘uniformity’ and ‘not changing the method of [how] they get the data’ proposed as a route to ‘help build trust in their reliability’. Finally, respondents also suggested providing background information, context and more detail to help users situate the analysis.
Amendments
This sub-theme includes responses where modifications to the statistics themselves were proposed. Specifically, this includes suggestions to use more-innovative methods and to ‘integrate new technologies’. Furthermore, respondents also requested that the type of statistic be expanded to include ‘a wide range of basic topics’.
Endorsements and Approvals
This sub-theme includes any suggestions of endorsements. Respondents proposed celebrity endorsements including by ‘Carol Vorderman’ or advocated for them being checked by ‘Martin Lewis, Preston etc.’. Within this sub-theme, ‘official documentation’ being ‘backed by [an] official body’ and a ‘government guarantee that it is real’ were all mentioned. In addition to this, recommendations for ‘more signposts to official status’, ‘gov.uk on the paperwork’ or a ‘logo to show verification from an approved source’ were also included. Given that there is already an accredited official statistics badge, this suggests that the status of official statistics, in particular accredited official statistics, could be communicated more overtly to the public.
Mentoring and Verification
This sub-theme incorporates recommendations relating to audits, cross-referencing outputs, peer review and verification procedures. Within this sub-theme, ‘watchdogs’, ‘pilot reviews’ and ‘peer review and scrutiny’ were all recommended. ‘More accountability’, ‘information in the media about wrongdoing’ and ‘statisticians that are willing to answer all questions put to them’ were directly requested. There was also a suggestion that ‘data auditors be named’ to improve transparency and accountability.
Recommendations and Reviews
This sub-theme exemplifies the processes respondents suggested they would find reassuring before assigning their trust. In this respect, familiarity and ‘hearing others use them’, as well as showcasing ‘good experiences from friends and family’ are included in this sub-theme. Alongside this, a wider pool of user reviews where ‘others recommended the service’ was also mentioned as a route to heighten trust.
Self-verification
This sub-theme captures responses which emphasise verification through self-use. Suggestions highlight the importance of ‘personal experience’ and point to the belief that ‘if [they] used them and found they were correct’, they would have confidence in them. These suggestions do not relate to specific strategies that can be adopted per se but do reiterate the importance of communicating the value of statistics and supporting use across a wider user base.
Avoid Manipulation and Deceit
This involves honesty, avoiding manipulation and not breaking promises. Within this sub-theme, honesty and truth were often cited explicitly as important characteristics. Respondents also requested a ‘uniform analysis, not always changing the goalposts to suit the perceived need’, and pleas to ensure statistics were ‘plain and couldn’t be twisted to suit’, or ‘adapted to look good’ also featured. Concerns relating to statistics being collected ‘to make the government look better’ were also expressed.
Free from Interference
This sub-theme incorporates impartiality, independence and no profitability. It includes recommendations such as ‘be sure that [there are] no profitable businesses’, with ‘the knowledge that they are truly impartial’ and ‘confirmation of no political interference’ suggested. In addition to the need for statistics to actually be independent, respondents also signposted the importance of this independence being communicated, exemplified and showcased to the public. Only then can trust be built on the basis of ‘true independence’.
Open and Balanced
This sub-theme reflects respondents’ pleas that statistical production and dissemination be open, transparent and unbiased. Requests for ‘stats without judgement’, ‘accurate and unbiased data’ and ‘reflect[ing] a balanced view’ are included in this sub-theme. To quote one respondent, ‘transparency: clearly communicating the methodologies, data collection processes and any limitations of the statistics can help users understand how the data is produced.’
Professional and Courteous
This sub-theme includes responses where requests for alignment with standards of behaviour and conduct are proposed. Alongside this, being ‘less intrusive’, ‘more responsible’ and ‘more approachable’ are also reflected here.
Providing Examples
The recommendations proposed here reflect respondents’ wishes to see more examples and ‘real life experiences’ reflected in statistical outputs. Suggestions include ‘highlighting success stories: showcasing instances where official statistics have led to positive outcomes can reinforce their value and reliability’ as well as showing ‘real world examples where they have been used and the benefits derived’.
Outcomes and Action
These responses capture respondents’ desire to see ‘things happen instead of relying on statistics’. Within this, respondents appeared to value ‘outcomes’ and ‘when solutions are provided to problems’. These suggestions indicate that translating statistical insights into policies and taking action to remedy some of the negative situations that official statistics report is seen as a necessary component of their value to members of the public.
Reflecting Personal Experiences
This incorporates responses where ‘more information related to [the individual]’ was requested. Specifically, this involves providing ‘more statistics relevant to [the individual’s] local area’ alongside more generalised requests to showcase ‘their relevance on a day-to-day basis’ as well as dedicating efforts to ensuring they ‘correlate with personal experience’. The importance of personal experience, and ensuring that official statistics resonate with individuals, can be achieved by providing more-varied statistical outputs, with bespoke personalisation capacities.
Representative
These suggestions include presenting the whole story alongside requests to increase representation and inclusivity. Requests include ‘seeing results from a wide range of areas and people’, ‘wider demographics’ and ensuring ‘the whole story was presented’.
Responsive
This sub-theme incorporates responses relating to ‘public engagement and consultation’, ‘more open discussion’ and ‘being involved’. In addition to this, a respondent also noted that when ‘actively seeking feedback from users’, it is important to ‘demonstrate that the producers of statistics are responsive to the needs and concerns of the public’. This points to the importance of effective public dialogue, moving beyond superficial engagement, and genuinely reflecting public concerns. In addition to this, the feedback loop must be completed, and participants should be informed of any outcomes stemming from the engagement exercise.
Supportive
This sub-theme captures any suggestions that producers provide ‘more help’ to assist users with the usability of statistics. This may involve supporting members of the public in developing the critical skills needed to scrutinise statistics as well as providing ‘information on how to fact check the figures provided’.
Timely
This sub-theme includes suggestions to ensure that the ‘information [is] supplied at the right time’ and that ‘regular updates and refreshers’ are provided. Alongside this, respondents also requested ‘more real time data or frequent updates [to] make statistics more reflective of the current situation.’ In addition to this, ‘fresher, dynamic content’ was also requested.
User-Friendly Publication
This sub-theme reflects responses related to ease of use. This includes suggestions of a ‘simple format’ and ‘making them less complicated’. Furthermore, respondents also requested that statistics be ‘easy to find’, with access and a ‘user friendly page where they can all be displayed in easy-to-understand ways by categories or searching’ provided.
Communicating Uncertainty
This sub-theme captures responses which suggest that the ‘clear communication of margins of error’ would positively contribute towards higher levels of trust. Alongside this, responses which proposed that measures to ensure that users are made aware of any limitations in the process of data collection and the methods of analysis were also included.
Education and Support
Suggestions relating to education and providing users with the information they need to be able to effectively use statistical products are included in this sub-theme. For instance, responses include ‘enhanced public understanding’, requests for statistics to be ‘explained better’ and pleas for ‘more information’ to be provided. In addition, respondents also proposed ‘training and resources’ and reflected on how ‘increasing [their] skills’ and providing ‘a grounding in statistics [would allow them to] appreciate them more’ and help to ‘build confidence’.
Incentivisation
This sub-theme proposes the distribution of ‘vouchers to people who take part’, suggesting that ‘if people were incentivised to believe them’, trust may increase.
Increased Exposure Efforts
This sub-theme captures recommendations to invest in the profile of statistical outputs. Suggestions include ‘more exposure’, ‘more publicity and creating more awareness’, alongside statements such as ‘I [am] just not aware of what good they are.’ Furthermore, enhanced media coverage also featured within this sub-theme. For example, ‘if they were talked about on tv’, ‘quoted more often in social media’ and ‘if they were made public on the news’. In addition to this, being ‘widely available’ and ‘knowing where to look for them’ was also recommended by respondents. This suggests that well-signposted publication, and thus exposure, is seen by the members of the public as an effective route to improved trust.
Statistics Giving Positive News
This relates to the way official statistics are presented and discussed within the public discourse, with recommendations of ‘more good news’ and ‘higher positive results’ included in this sub-theme. This may be a problematic driver of trust, as accurate statistics cannot, by definition, always reflect a positive experience.
Consistency in Messaging
This sub-theme captures issues relating to statistics being undermined by ‘other bod[ies] saying they are not correct’. This is identified as a negative contributor, with conflicting messaging by others seen to lower trust.
Simplicity
This includes responses which relate to simplicity in communication, reducing the amount of technical jargon, and the use of visuals. Within this, ‘charts’, ‘bullet points’ and ‘more use of graphics’ are suggested. Respondents also suggested that statistics be ‘clearer and easier to understand’, ‘easier to follow’ and ‘explained simply’. Overall, suggestions included within this sub-theme promote a ‘simplistic approach [which uses] easy-to-understand language’ as well as the use of ‘less jargon’ as a possible strategy to heighten trust.
Trust in Politicians, Government and Civil Servants
This sub-theme is inherently more politically charged. Statements such as ‘a more honest government’, ‘better trust in the government’ and ‘my trust will increase when the government becomes more trustworthy’ exemplify this point. This is included as a separate sub-theme as the spillover effects between official statistics producers and the government/civil service was explicitly reported in the survey. For instance: ‘this is difficult since trust in statistics means trust in the government’ and ‘anything produced by the government is suspect.’ As such, it is important to acknowledge the wider environment of trust, pay attention to spillover effects and consider networks of trust building (or falling).
Trust in the Competence of Statistical Producers
This sub-theme is explicitly tied to official statistics. It incorporates responses which highlight expertise and competence, reputation and finally, confidence in the statistical producer. ‘Pro-advice’, being ‘thorough’ and knowing that ‘a lot of thought has been put into it’ are considered important from a competence perspective, with producers having ‘the necessary experience’ cited as an important asset. In addition, these suggestions point to a ‘better reputation’, with brand recognition as a ‘reliable, reputable source’ also included here.
Automation Concerns
This sub-theme relates to apprehensions surrounding AI. For example, ‘having humans obtain the information’ [all capitalised] and ‘if they were based on actual evidence and not just calculated data from a computer’. This reflects the impersonal nature of AI. However, it is worth noting that AI usage was not always viewed as negative, with some respondents expressing apathy rather than concern.
Data Governance
This shows the importance of ‘strong data-driven governance’, with ‘good infrastructure’, ‘using public data’ and ‘sharing data’ reflecting examples of this sub-theme. Moreover, responses which endorse collaboration surrounding data are also included.
Data Security
This sub-theme includes responses which reflect data security and privacy concerns. ‘Secure and trusted’, ‘assur[ance] that my data is safe’ and ‘robust data security and protection’ were all proposed by respondents. This suggests that reassurances around data security may be a precondition of trust in official statistics. As such, wider dialogue to articulate the precautions and procedures involved in data collection and data storage process could help moderate concerns.
Recommendations based on the analysis
Based on the answers given by respondents, the following section provides a synopsis of the key takeaways and recommendations from the survey analysis. They are structured thematically, and the intended audience for each recommendation is made clear.
Theme 1: Quality Product
- Producers should commit efforts to ensuring that explanatory information about data collection methods and (where suitable) analytical approaches is easily accessible to the public. Producers may wish to dedicate attention to considering how this information gets into the public domain. Encouraging intermediaries to signpost explanatory information alongside headline figures could be helpful here.
- Anyone involved in the communication of official statistics should dedicate efforts towards demonstrating that the statistical output is a quality product. Actions to display the quality of the product and help build public confidence in the accuracy and authenticity of official statistics should be embraced.
- Anyone involved in statistical production or communication should ‘show the [positive] track record’ of the statistical products they produce. ‘Ongoing reliability’ and a ‘past history of being correct’ are regarded as an important part of having confidence in the product.
Theme 2: Verification and Reviews
- OSR, as the independent UK regulator of official statistics, should ensure statistics are aligning to the Code of Practice. In addition, producers of statistics should implement robust audit and quality checks on a regular basis. It is important that this process, and the outcome – including any recommendations – be communicated to the public, and that they are aware that the audit and monitoring procedure has occurred.
- Regulators should ensure that retrospective accountability, or waiting for issues of misuse to occur, is not relied upon in place of active and ongoing monitoring.
- Respondents reported that hearing recommendations and reviews from people who had used official statistics in the past could be a possible route to building trust. As such, anyone involved in the statistical system could encourage the wider use of official statistics and contribute towards publicising efforts. As mentioned previously, wider exposure may support this route to trust building.
Theme 3: Integrity and Transparency
- Producers, and the regulator should take concerted steps to exemplify impartiality and ‘true independence’ to the public.
- Anyone involved in the statistical system should be aware that honesty was one of the core areas respondents signposted for improvements. With this in mind, producers should continue to ensure their outputs are objective, truthful and honest, and invest in ensuring this is clearly exemplified to the public.
- Producers should be open and transparent about statistical production, and those communicating official statistics must ensure they are presented in a balanced manner.
- It is paramount that the manipulation of statistics is avoided. Statistics must not be distorted or skewed in order to present a more favourable account of the situation. In addition, producers should ensure that this professionalism is clear in the way statistics are presented, emphasising the political independence of statistical production.
Theme 4: User Relevance
- Anyone involved in the statistical system has a responsibility to ensure that statistics are useful to the public. One way of achieving this is to reflect the users’ current context, whether from the perspective of time or experience. Another is to provide personalised statistics and highlight how each individual statistic relates to user experiences.
- Statistics should be circulated in a way which ensures members of the public can access them, and thus benefit from the value they provide.
- Producers should ensure official statistics are published in a timely manner and are up to date. This is seen as important to ensure relevance to users.
Theme 5: Communication
- When communicating official statistics, producers should explicitly state any limitations and uncertainties within the data and the statistical output.
- Anyone producing, disseminating or citing official statistics should explain them simply and clearly, using non-technical language, and where possible, visual strategies should be included to support comprehension. Alongside this, guidance and support on how to use – and interpret – official statistics should accompany any publication (whether written or verbal).
- Those involved in the statistical sphere should dedicate efforts towards clearly explaining the value of statistics.
- Official statistics should be published via a range of communication channels, and they should be clearly signposted as official (and as accredited where applicable). Increased exposure for the purpose of generating ‘more publicity and creating more awareness’ was suggested as a positive route to building trust.
Theme 6: Trust in Producers
- Anyone involved in statistical production or communication should explain who is producing the statistical outputs. Their expertise, and the processes they followed, should be also laid out.
Theme 7: Data Management
- Producers should be clear about the amount, and nature, of AI involvement in statistical outputs.
- Producers should reassure the public that ‘privacy [is] a priority’, and public assurances that they are adhering to appropriate data management protocols and that all data are kept securely should be made.
Further Research
Recognising the limits of the analysis presented here, further research to establish the prioritisation of measures would be useful. This would help provide a fuller picture of which responses are the most significant, from the perspective of the public, and therefore help focus efforts in the most effective areas.
Back to top