7. Appendix A: Methods

The aim of this literature review was to understand how official statistics inform public policy, including at what points of the policy cycle statistics are used and what facilitates or prevents this use. Our review included three stages: 

• identifying relevant literature 
• summarising literature 
• drafting the review 

Artificial Intelligence (AI) was used to support these three stages, as shown in Section 7.4. While these uses contributed to the development of the literature review, the review itself was written by the OSR research function. This ensured human oversight of all AI contributions. 

7.1 Identifying relevant literature 

Literature was identified by searching for key terms on Google Scholar and a variety of AI tools (including Semantic Scholar, Research Rabbit and Perplexity AI). Through existing knowledge of papers, through snowballing (using citations within papers to find related evidence) and through applying a range of Artificial Intelligence (AI) tools, a pool of potentially relevant literature was identified. The use of AI is described in Section 7.4.  

OSR does not have extensive memberships to journals and journal access sites (such as Web of Science and PsychInfo), so only those papers that we could access online for free were reviewed. Any papers for which an abstract only was available were excluded from consideration within this review. This decision to exclude papers where there was only access to an abstract aimed to ensure that the full paper could be scrutinised. Limited access to articles means that while this review has aimed to provide a thorough survey of the literature, it could not be described as exhaustive. Despite this limitation, we are confident in the conclusions drawn throughout the review. 

During the literature search, it became evident that ‘grey literature’ had potential to contribute to the research aims. Grey literature refers to information produced outside of traditional publishing and distribution channels (such as reports, working papers, policy documents, newsletters, government documents, white papers and so on). As such, we widened our search beyond academic literature by including documents published by government departments explaining their evidence base for policy.  

The relevance of literature was ascertained by reading abstracts or summaries and searching for key words within documents. If the researcher judged that there was little apparent potential to contribute to the research aims, documents were discounted at this stage. 

7.2 Summarising literature 

Literature was summarised both manually and with the assistance of AI tools (as described in section 7.4). The manual summarising process involved reading the paper, highlighting relevant parts and transforming these into a summary. The AI-assisted summarising process involved ’asking’ the tool to summarise the paper in 1000 words or fewer and then ‘asking’ detailed questions about various contexts for more focused mini-summaries.  

The prompts always included the phrase ‘if you cannot find anything related to this, that is fine, please just let me know’. This was done to minimise the occurrence of AI hallucinations. To further mitigate this issue, the tools were also ‘asked’ to provide article page numbers for all summary points, and these were then checked manually against the paper. 

7.3 Drafting the review 

This review has gone through several drafts. The first draft focused specifically on the ways in which official statistics are used to inform, develop, monitor and evaluate UK public policy. However, upon subsequent re-drafts, an important narrative emerged concerning facilitators on using official statistics within policy and conversely, barriers to this. Each draft was reviewed by two additional authors with comments provided and acted upon. AI tools were used for some initial drafts of the introduction and conclusion sections, though these sections went on to undergo significant human review and redrafting. 

7.4 Overview of the use of AI
Use of AI in the literature review

Finding articles:

• Semantic scholar
  • Used in addition to Google Scholar to locate papers with different wording but similar meaning to the search parameters
• Research rabbit
  • Used to track similar papers using various network analyses (such as citations)
• Perplexity AI
  • Used for natural language searching of research articles
• Open AI ChatGPT 3.5 (accessed via the chat function of the Microsoft Edge internet browser)
  • Used to search for articles

Summarising articles

• Anthropic Claude 2 
  • Used to summarise articles using specific prompts by asking it to draw out specific mentions of interest
• Anthropic Claude 3
  • Used to summarise articles with specific prompts by asking it to draw out specific mentions of interest 
• Open AI ChatGPT 3.5  (accessed via the chat function of the Microsoft Edge internet browser) 
  • Used to summarise articles with specific prompts by asking it to draw out specific mentions of interest 

Re-writing/restructuring introduction and conclusion sections 

• Microsoft Word Co-pilot (accessed via Microsoft Word co-pilot package) 
  • Used to change bullet point lists into text in first draft of the literature review