Method

We selected a content analysis as the most appropriate methodology for this study as it can involve quantitative and qualitative measurements, offering a level of broad analysis as well as deep analysis of the material. We began by selecting a sample of applications to the NSDEC and the RAP.

Sampling

Sampling criteria 

There were 50 applications in total to the NSDEC, dated from January 2016 until February 2020, and 90 applications in total to the RAP applications (which started after the NSDEC) dated from October 2019 until July 2020 (see Table 1).  We used stratified sampling to ensure that a range of applications were examined. To create the sample, all of the applications were first categorised according to the theme of the research.

Table 1: Count of themes represented in all NSDEC and RAP applications

The applications were categorised based on key words in the title of the research project. As RAP applicants are asked to choose from a list of research themes provided, their categorisation was double-checked. However, as the NSDEC application does not involve allocation to research themes this categorisation could not be checked for accuracy.

The applications were then categorised according to the type of applicant. Applications came from either academia (affiliation with a university) (RAP=50, NSDEC=3), government departments (affiliation with the civil service) (RAP=11, NSDEC=27) or other organisations (private sector businesses or third sector organisations) (RAP=29, NSDEC=20).

Once the categorisation was completed, applications were randomly selected within each type of applicant and each theme as much as possible. This meant that 12 RAP applications and 18 NSDEC applications were chosen for analysis (see Table 2).

Table 2: Sampling of applications

Scoping

The focus of our analysis was the response to one question in the applications which asked how the research would serve the public good.

However the applications are structured in different ways[2].In the NSDEC, the question is:

The NSDEC applications provide one large free text box to complete in response to the above question.

In the RAP, the application states:

This is followed by seven free text boxes each containing one public benefit as seen in Table 3. Abbreviations for each public benefit are provided in the table to facilitate discussion about them later in the report.

[2] The applications also have guidance documents which elaborate more on the questions above (NSDEC Guidelines on completing the Application for Ethical Review Form, 2017; RAP Application Guidance, 2019)

Table 3: List of the public benefits provided in RAP applications

Analysis

Quantitative

Stage one

The seven public benefits listed in Table 3 were used as a framework to analyse references to public benefits in both NSDEC and RAP applications. This was done so that comparisons could be drawn between RAP and NSDEC applications. The first stage of the analysis involved a count of which of these public benefits were most frequently mentioned in each application form.

• In the RAP applications, the first stage was to count which of the seven free text boxes were completed, to provide insight into which public benefits were fulfilled by the research. The applications asked applicants to complete at least one of the boxes
• In the NSDEC applications, the researcher read the free text sections about the public good and, using the seven public benefits from Table 3 as a guide, coded the texts according to which of the seven public benefits were referred to

Intercoder Reliability

We carried out reliability testing in order to ensure that the NSDEC was coded consistently. We recruited an independent researcher to code a sub-sample of the NSDEC texts according to the seven public benefits listed in the RAP applications.

Following guidance by O’Connor and Joffe (2020), 25% of the sample was coded by the independent researcher, which equated to four applications. Analyses demonstrated agreement between the two researchers on 21 out of 28 (75%) of the judgements of the codes in the sub-sample (see Table 4). The disagreements were distributed over five of the seven public benefits therefore there was no indication of bias in the judgements. According to guidance from Landis and Koch (1977), 75% agreement can be classed as ‘substantial’, therefore it can be considered an acceptable amount of agreement which implies reliability in the coding framework.

Stage Two

The second stage of the analysis involved comparing NSDEC and RAP applications to identify if there were differences in the types of public benefits referred to. This stage also involved counting which public benefits were mentioned according to the different type of applicants and the different themes of research.

Qualitative

In this stage of the analysis, the applications were coded for references to the public good (including synonyms). The texts were then examined to understand if they referred to any public benefits which did not fall into the seven pre-defined public benefits. This was done to understand if the applications contained other conceptualisations of the public good, or public benefits.