Chapter 2: Design and methodology

Chapter summary

This mixed-methods study explored whether and how statistics inform personal decision-making by combining qualitative and quantitative research. We conducted 42 semi-structured interviews, which then informed the design of a nationally representative survey of 2,118 UK adults. Both methods were used to address the research questions. This chapter details the study’s design and methodology, including the semi-structured interviews, the online survey and the limitations of the research.

2.1. Semi-structured interviews

In June and July 2024, we conducted 42 online, semi-structured interviews with public participants. Each interview lasted approximately 45–60 minutes. Participants were selected based on having made a recent “life decision”, which was then explored during the interview. These decisions were:

• choosing a primary school
• choosing a baby name
• moving to a new area
• buying a property
• choosing to work after retirement
• seeking a job in a new sector

The interviews were divided into three parts, as described below. Additional information on the interview structure can be found in Appendix 1 (topic guide), while Appendix 8 outlines how decisions and statistics were presented to participants. Appendix 4 provides a breakdown of the number of interviews covering each topic across Parts 1, 2 and 3.

• Part 1: Real-life decisions. Participants discussed at least one of the personal decisions they had made, exploring the factors involved in making the decision, the sources and data used (if any) and what might have improved their decision-making process. Using a narrative interviewing technique, participants were guided chronologically through their decision process. Finally, we presented relevant data from official statistics to assess whether participants were aware of them, their prior use of these data and their perceived usefulness, and whether they could have influenced their decision.
• Part 2: Hypothetical decisions. Participants considered a hypothetical decision through a scenario or vignette, which helped us assess how they might use data and statistics in decision-making. In cases where participants had faced similar decisions in real life, they were invited to discuss those instead. Similar to Part 1, participants were finally shown a potentially relevant data from official statistics to assess whether they were aware of them, and whether they found them useful for making the hypothetical decision.
• Part 3: Presentation of official statistics. We presented a selection of official statistics both in the form they were published and in simplified formats, asking participants whether they found them interesting, understandable and potentially useful for personal decision-making. In some cases, participants explored a secondary source, via a hyperlink, with the interviewer observing their navigation process on a shared screen.

In addition, for participants who, during their recruitment to the research, had agreed with a statement that statistics had helped them make a decision about their life in the past month, we included an additional question between Part 1 and Part 2. In this question, participants were asked to elaborate on their response, specifying which statistics they had used and what decisions they had made.

2.1.1. Interview sampling and recruitment

Public participants were recruited through the professional recruitment agency Criteria, and received a £35 incentive to take part. To ensure that we gathered insights from a general, non-expert audience, we only recruited participants who did not work, and had not worked, in a job that mostly involves data analysis. To ensure a diverse range of perspectives were included, we used purposive sampling – a non-probability sampling method – to achieve diversity across demographics such as sex, age, socioeconomic status, employment, ethnicity and location. However, given that several of the life decisions that we chose focused on parenting, the sample included a larger proportion of participants aged 35–44 than participants in other age groups. Appendix 3 includes detailed information about the interview sample.

2.1.2. Interview analysis

Interviews were recorded and transcribed in full by a professional transcription service. Transcripts were then analysed by three researchers using a case-and-theme-based framework approach, enabling us to identify commonalities and differences within the diverse qualitative data and to develop descriptive categories and explanatory concepts.

2.2. Online survey

We conducted an online survey in October 2024 using quotas from the 2021 Censuses in England and Wales and in Northern Ireland, and the 2022 Census in Scotland.

The survey was developed following the interview research, and included 20 closed- and open-ended questions exploring a range of factors related to official statistics, including awareness, use for decisions, barriers, value compared to other information sources and trust. The full survey can be found in Appendix 2, and Appendix 5 provides more detail about how the survey was designed.

2.2.1. Survey sampling and recruitment

We recruited a sample of 2,118 UK adults in October 2024 to participate in the survey. Participants were recruited to be broadly nationally representative on sex, age, region, ethnicity, education and socioeconomic status (see Appendix 7). The survey was hosted on the Behavioural Insight Team (BIT)’s online survey and experimental platform, Predictiv. Participants were recruited via BIT’s trusted recruitment partner, Cint. Participant demographic data were collected using our recruitment partner’s standard screening questions. These were reviewed and compared to the relevant Government Statistical Service (GSS) Harmonised Standards.

2.2.2. Survey analysis

Several different analytical approaches were taken in analysing survey data, including descriptive statistics, logistical regressions, subgroup analysis and, for open-ended questions, exploratory data analysis and topic modelling. These methods are described in more detail in Appendix 6.

2.3. Identifying and selecting decisions and statistics

During the first stage of the research, we developed a longlist of real-life decisions (Part 1), hypothetical scenarios (Part 2) and official statistics (Part 3) to be included in the research. The full longlist can be found in Appendix 9.

The selection process involved reviewing accredited official statistics, consulting OSR regulators, circulating an online call for evidence, exploring statistics news summaries to identify indications of statistics use, and consulting with members of our expert and public advisory groups (see Section 2.3). We then collaborated with OSR to refine the final selection of decisions and statistics for inclusion, considering factors such as the feasibility of recruiting participants who had made specific decisions, the sensitivity of asking participants about certain decisions, the likelihood of official statistics playing a role in those decisions, and ensuring that a variety of topic areas were chosen from each of the OSR domains.

The process was exploratory rather than systematic, due to constraints around budget and timelines, as well as the need for flexibility in addressing the diverse range of research objectives and practical constraints of participant recruitment and topic sensitivity. Unlike systematic processes that rely on predefined inclusion and exclusion criteria, our exploratory approach allowed us to respond flexibly to the insights gained during the process. However, we acknowledge that this approach introduced certain limitations. The absence of strict inclusion and exclusion criteria may have led to some decisions being included or excluded based on subjective judgements, though we believe this risk was mitigated by incorporating a range of perspectives throughout the process.

When deciding how to present selected official statistics in the survey and interviews, we chose on many occasions to adapt how the statistics were presented rather than to display the statistics as they were published. This is because the present project explored the potential value and uses of statistics, rather than their current usability. It also did not aim to quality-test the service that statistics producers provide. Based on the researchers’ previous experience, we decided to simplify the presentation of many of the statistics so that they could be easily discussed in an interview or understood in a survey. This allowed for a more in-depth discussion of how official statistics could be used, while acknowledging that it comes at the cost of knowing how they are currently used.

2.4. Role of the project advisory group and public contributors

This research sought advice from a project advisory group comprising five members from both within and outside of OSR. These members each represented a distinct area of expertise seen as important for the project, including regulatory practices, official statistics, ethical research practices, public engagement and methodology.

To ensure that the voices of members of the public also shaped the research, a small group of public contributors were recruited through existing public engagement networks and word of mouth. This group did not aim to be representative of the UK population. Its purpose instead was to allow members of the public to feed into our work, and to identify potential sensitivities or improvements.

2.5. Strengths and limitations of the data and design

We believe that this project is an important starting point to understand how official statistics inform the public’s decision-making; however, it is exploratory in nature, and its design does not allow for causal inferences or the establishment of definitive relationships between the use of official statistics and decision-making outcomes. Instead, the findings should be viewed as indicative and exploratory, providing insights into potential patterns and areas for further research. Future studies could address these limitations by employing different methodologies to generate causal evidence (such as randomised controlled trials), using a more systematic approach to selecting decisions and statistics and asking similar questions about a larger number of official statistics for each topic to explore how characteristics of official statistics may impact their use in decision-making.

When interpreting findings from this research, it should be noted that official statistics are situated within a wide marketplace of other evidence that individuals may use to make decisions. In our interviews, we saw that participants did not necessarily recall the specific source the information they used came from; therefore, it is possible that in some instances experiences which were attributed by participants and respondents to official statistics, or interpreted by the researchers to be official statistics, may be tied to other data instead. To address this, where we are aware that data other than official statistics were being referred to, we have clarified this in our reporting. However, it is possible that there may be instances where we refer to official statistics, but participants and respondents had used other data that they mistook for official statistics, or that the researchers mistakenly interpreted to have been official statistics.

2.5.1. Qualitative interviews

The qualitative interviews involved three key components: real-life decisions, hypothetical scenarios and the presentation of statistics. Each has inherent strengths and limitations that must be considered. When asking participants about how they made previous real-world decisions and what information they used, memory constraints and recall bias can lead to inaccuracies, as people may unintentionally distort or selectively recall events or decisions (Schacter, 1999). In addition, decision-making processes frequently involve subconscious influences that individuals may not fully recognise or articulate even at the time (Kahneman, 2011). For instance, a participant may not have realised that an official statistic influenced their decision or, conversely, mistakenly attribute a decision to official statistics. Post-decision rationalisation is also known to bias self-reporting of decisions, as individuals retrospectively align their explanations with norms (Festinger, 1957). This risk is particularly important to keep in mind, because participants were made aware of the objectives of the research, potentially increasing the likelihood of rationalising that official statistics influenced their decisions.

To mitigate these effects, we used a narrative interviewing technique, where we asked participants to freely explain their decision before any probing. The approach provided insights into real-life decisions made by real people in real-life contexts, and therefore improved validity, but it was inherently still retrospective, so participants may have omitted details or recalled reasons that did not influence their decision at the time. We chose not to explore ongoing decisions, as it was deemed unethical to interfere with those. Overall, any findings about people’s recall of personal decisions, as relied on in this study, should be interpreted with those limitations in mind.

Using hypothetical scenarios is a widely accepted method in decision-making and behavioural research (Aguinis and Bradley, 2014). This approach removes some of the noise and complexity of real-life contexts, allowing researchers to isolate and examine key variables without the confounding factors that naturally arise in real-world situations (Evans, 2008). Humans are also highly capable of imagining and reasoning through hypothetical scenarios, particularly when these are designed to be realistic, feasible and relatable to their circumstances (Evans and Stanovich, 2013; Kahneman and Tversky, 1982).

While hypothetical scenarios are inherently artificial, they can still yield rich and nuanced insights when appropriately designed and analysed with their limitations in mind (Hughes and Huby, 2004). We aimed to ensure that the scenarios were plausible and contained enough context for participants to understand the situation while remaining sufficiently vague to encourage participants to introduce their own interpretations and additional factors influencing their decisions. The ways participants filled in these gaps and defined the situations provide critical information about what they perceive as important in decision-making.

Official statistics were rarely presented in the exact format found in statistical bulletins online. This reduced external validity, but changes to the way statistics were presented made their relevance to decisions clearer and their format easier to understand in an interview setting, particularly when shared on screen. Because of this limitation, the findings about individual statistics should not be taken to represent true views (or direct reflections) about those statistics themselves. Rather, they provide broader insights into how participants engage with and use statistics in decision-making more generally.

Finally, the interview evidence in this report is based on qualitative research that is not – and does not set out to be – representative of the wider population. Instead, the research focused on generating in-depth insights from general, non-expert audiences who did not work, and had not worked, in a job that mostly involves data analysis. We used purposive sampling – a non-probability sampling method – to achieve diversity across demographics such as sex, age, socioeconomic status, employment, ethnicity and location. This sampling strategy allowed us to explore a breadth of views and experiences, and through conducting semi-structured interviews, we were able to generate detailed and textured data on the personal decision-making and use of official statistics among our interviewees.

However, even those in-depth insights might have been biased in various ways. For instance, both interview participants and survey respondents were inevitably primed to think about how official statistics play a role in decision-making as the interview and survey progressed, which might have biased their responses. The presentation of official statistics in interviews was also designed to specifically explore the use of official statistics in personal decision-making, and therefore opinions may have differed if we had used statistics in their original form, or asked about their use for other purposes.

2.5.2. Survey

There are also strengths and limitations related to the online survey, including related to data collection and analysis.

The survey was online, which means that digitally excluded people are less likely to participate. However, this design choice allowed us to reach people at wider geographies and more people with short timeframes and at lower cost than if we had conducted the survey in person.

Some survey questions referred to statistics but did not show the statistics themselves. As such, we cannot assume that respondents would express the same views if they saw the statistics or data in their original form. However, this approach was best suited to an online survey context, as question design best practice is to use short, clear and simple questions. This is especially important when no researcher is present to explain the resources, as they were in the interview phase.

Analysis of the free-text survey data used data science techniques rather than detailed thematic analysis. This may mean that there is more richness in the data that has yet to be explored. Despite the potential loss of richness, this approach was best suited to this project due to time constraints, and given that it had human oversight, we are still confident in the results. It also leaves opportunity for future, in-depth interrogation of the free-text data.

Finally, the survey intended to look at high-level views from a wide range of people, to complement the in-depth but smaller number of interviews. As the interview data were analysed in detail, the research findings still offer rich insights.

2.6. Ethics

OSR completed an ethics self-assessment, where it considered public good, confidentiality and data security, methods and quality, legal compliance, public views and transparency. This assessment was evaluated by the UK Statistics Authority Data Ethics Team. Similarly, the researchers assessed the ethical risks and how to mitigate them in a minimal-risk application to the Social Science and Humanities Research Ethics Subcommittee at King’s College London (MRA-23/24-41889).