5 Annex 1: Methods

In the task of evaluating levels of trust in official statistics, a literature review was established as an appropriate first stage. This enabled the project to meet the stated objectives of both enhancing understanding and providing practical recommendations. This was then followed by primary research which was based on analysis of free-text answers to the question: ‘what might increase your trust in official statistics?’ The responses were analysed using NVivo software, and the identified themes, along with a more detailed methodology, are reported in Appendix 1: Primary Research: Building trust in official statistics.

For the literature review, a dual process was undertaken when selecting literature for inclusion. Firstly, stakeholders, namely those with a particular interest in understanding public levels of confidence in official statistics or evidenced-based information more broadly, were invited to suggest resources to be considered in the review. This included submissions from those working on trust in data, as well as members of ONS who had been involved in surveys dedicated to this theme. Each of the recommended resources was screened and considered for inclusion in the review. This process of expert suggestions was particularly helpful in uncovering reports and surveys, and meaningfully contributed to the evidence base this review draws upon.

The second stage followed a more systematic and structured approach. This involved conducting a Boolean search for academic contributions which featured “official” and “statistics” in the title and/or abstract. Once duplicated results were removed (n=18), the remaining articles were screened (n=83). As part of the screening process, 50 records were excluded due to an irrelevant title, and a further 21 were removed on the basis of the abstract. To account for this, oftentimes, these contributions appeared to use the search terms in a different capacity, which meant they were not suitable for inclusion in this exercise. For instance, to generally refer to “statistics” (i.e., poll data, or a product of the authors own analysis) or using “official” to refer to an established, approved, or recognised policy or position maintained by a government or organisation.

At the end of this process, given the specificity of the topic area, it became clear that traditional approaches would only generate a few studies of relevance (n=16). To provide a breakdown, this included 12 articles which were identified via this process and a further 4 which were identified using snowballing. Consequently, a more pragmatic approach was taken.

Over and above source limitation, the pragmatic approach which was adopted within this review is beneficial as it recognises the wider context within which official statistics are produced and communicated to the public. With this in mind, the inclusion criteria were expanded to accommodate studies which considered other relevant areas. The relevant areas were established based on conversations with those working in the official statistics sphere and with the intent to reflect the journey official statistics take. This approach of centralising the journey adheres to the premise that trust can be broken at any stage of this process, and reflects the idea that low levels of trust do not simply depict a hesitancy to trust official statistics (as a product). The approach taken in this review reflects the premise that the dynamics involved in trust mechanisms are far wider.

Having settled on this pragmatic approach, it was identified that initial searches on these broadened fields were far beyond what was manageable for an exhaustive review within the time frame of this fixed-term appointment (3 months). Moreover, they were often unrelated, or tangentially related, in such a way that efforts to systematically filter searches did not produce helpful material. With this in mind, it is acknowledged that this review is not exhaustive, and that relevant material may have been missed.

As part of this pragmatic approach, articles and documents were inspected. Those which met the criteria of 1) being focused on trust in experts and/or government practitioners, and 2) concentrated on the production and/or dissemination of evidence and/or data, were deemed relevant. Alongside this, any contributions which discussed trustworthy protocols of evidence production, or promoted trustworthy communication practices, were examined.

From this, a process of snowballing was undertaken, and a further literature search was conducted focusing on key terms uncovered in the initial stage. The key terms included the respective definitions (trust as: a personality trait, a reciprocal process, a rational choice, a socially desirable objective, and vulnerability in trust), alongside a dedicated search which ensured the entire journey of statistical production, dissemination and reception was accounted for and that the relevant factors which relate to trust in media, trust in government and trust in evidence/data were not missed. This funnelled snowball approach was particularly useful as it produced a series of literature reviews and dedicated book chapters which were based in different disciplines and concentrated on these specific themes. These resources were valuable in the more theoretically oriented aspects of the review. In addition, they signposted to further points of exploration, particularly with regard to the models of trust building which are frequently cited within the literature.

This review is not intended to be exhaustive, and is informed by the literature searches, key terms and stakeholders engaged in the selection process. It does, however, provide an overview of key themes related to trust in official statistics, as well as signposting to some practical recommendations which those based in the area of official statistics may want to consider.