Findings

Data sources and methods

2.1 We commend ONS for expanding the sample size from 4,000 businesses to approximately 37,000 and for including a wider range of businesses from the Inter-Departmental Business Register (IDBR). It is also good to see ONS decide to increase the number of long-form questionnaires distributed from 400 to 8,000, increasing the amount of detailed data received and reducing the number of statistical assumptions required in the production of BERD estimates.

2.2 While the increased sample size improved quality, we note that this would have significantly increased the respondent burden for this survey. ONS’s quick transition from collecting data through paper questionnaires to the use of electronic questionnaires (EQs) will have partly offset the increased burden, and we welcome ONS’s further work to identify a better balance. ONS used the UK BERD: 2022 data to reoptimise the sample size, finding a sample of 20,000 businesses to be sufficient to optimise coverage and quality, while reducing respondent and processing burdens.

2.3 The transformation to EQs has also enabled faster data collection and processing. The EQ also provides respondents with additional guidance when completing the survey, including clearer definitions, explanations, examples and contextual information alongside questions. These improvements minimise misinterpretation of the questions and ensure a more robust and valid data collection process. However, this guidance was not publicly available to users, and some users told us that they felt that having access to this guidance would help them better understand what information is included in the BERD data, and therefore better understand these statistics. Before the move to EQs, such materials were available as part of the metadata on the SRS but were made available upon request but not published.

2.4 ONS responded proactively to address the undercoverage in the sample population of the BERD survey and engaged with domestic and international users. Furthermore, we consider the phased implementation of BERD transformation that ONS has adopted to be a helpful approach. The phased approach has meant that users have experienced a smaller interruption than a large transformation would have likely caused, ultimately preparing users for improvements over time and allowing for a smoother transition. ONS has balanced users’ needs for timely BERD data and implemented the necessary methodological improvements to ensure greater population coverage.

2.5 While many users expressed a need for a detailed back series for BERD, we recognise the barriers that ONS has faced in providing this. We appreciate ONS’s dedication here and look forward to the time series being released, which ONS is hoping to complete in June 2024, as announced in its May 2024 Update on Transformation of R&D Statistics. To further manage users’ expectations, we would encourage ONS to publish more details, including how far back the time series will go.

2.6 Many ONS business statistics are published only in current prices. The February 2024 release of BERD statistics discusses current-price values, though the dataset also includes constant-price statistics, which show volume movements of R&D once price changes have been removed. Figure 1, using data from the February 2024 release, shows the growth rates for these two measures in 2022. Domestic and international users told us that they prefer using the constant-price data to inform their policy and economic analysis. However, users felt it was difficult to explain the real reduction in R&D activity between 2021 and 2022 to policymakers because ONS hadn’t adequately explained the opposite growth between the current- and constant-price series. ONS should work with users to better understand how constant-price estimates are used and enhance their presentation and communication in its BERD statistics in the light of users’ needs.

2.7 Both ONS and NISRA use the GDP deflator to convert their respective current-price BERD statistics into constant-price statistics. However, prices in R&D may not fluctuate consistently with the GDP price deflator, as the GDP deflator also includes non-market output activity, which played a more significant role during the COVID-19 pandemic period than usual. ONS produces a price deflator for R&D, which is used in the production of Business Investment statistics, though growth in this deflator has remained unchanged since 2015. We encourage ONS to review, and update where necessary, the deflator used in deriving these constant-price statistics, taking into account the variables that contribute to changes in the price of R&D activity. This will enhance users’ trust in the quality of the constant-price BERD statistics, ensuring a representative price movement for the R&D activity is reflected.

Figure 1: The growth rates of UK BERD between 2015 and 2022, in both current and constant prices

Source: ONS, UK BERD 2022 data

Systems, resources and prioritisation

2.8 It is good that ONS moved to EQs, which introduced electronic forms of data capture and processing. It is also good that ONS introduced a Reproducible Analytical Pipeline (RAP) for the BERD statistics. The RAP has increased the efficiency of data collection and processing, as well as the quality of the BERD statistics.

2.9 Despite the resource constraints and pressure on the BERD statistical teams, the BERD transformation is a good example of how ONS drew together a multi-disciplinary team dedicated to responding to a methodological issue and rectifying it. The BERD statistics team has collaborated with other teams across ONS and sought opportunities to engage with stakeholders. Additionally, ONS collaborated with NISRA to produce coherent methodologies to further enhance the quality of UK R&D statistics. Users and data suppliers recognise and praise the efforts made. Stakeholders also shared their appreciation of the BERD statistics team’s willingness to take responsibility and ownership of the methodological issues.

2.10 ONS and NISRA have cooperated to ensure that ONS’s BERD statistics transformation is implemented smoothly. The organisations hold monthly meetings to share updates on progress and to ensure respective methodologies are consistent and coherent, and that estimates are comparable. It is good that ONS has committed to share its approach of collecting and processing BERD data, with NISRA. We also commend ONS for its collaboration with NISRA to discuss the potential of introducing a back series of NI BERD estimates.

2.11 ONS explained to us that the undercoverage of SMEs was the result of the previous sample design being unable to account for the rapid uptake in SMEs carrying out R&D following the introduction of tax incentives to boost investment in R&D. Considering the wider economic environment that the statistics are reporting on is of vital importance for ensuring valid methodology is implemented. Both users and producers indicated that limited resources inhibit ONS’s ability to respond to such changes quickly, and it is therefore regrettable that there wasn’t resource to anticipate and deal with this sooner. We plan to return to the wider point of funding methodological development for economic statistics in our review of economic statistics produced by ONS.

Quality assurance

2.12 Annual publication suited all the users we spoke to, but some said that the latest time lag (14 months) was unwelcome. However, we note that the methodological improvements that ONS incorporated during this time were necessary to ensure the quality of the statistics. ONS’s intention to explore further accelerating the timetable is commendable.

2.13 ONS used two governance routes for the methodological assurance of the BERD statistics, namely the Methods and Research Assurance Group (MaRAG), supported by the University of Southampton, and the Methodological Assurance for Statistics Transformation (MAST) group. The groups reviewed and provided advice on methodological improvements to UK BERD statistics. ONS also established various user groups, including the R&D user and producer group and the R&D expert user group, which it used to provide progress updates, answer users’ questions and explain the transformation plans.

2.14 While we appreciate that ONS cross-validates BERD statistics with HMRC tax credit data, investigating opportunities for wider cross-validation may lead to further improvements to the statistics. For example, users suggested that ONS should consider private sector R&D statistics and company balance sheets within its cross-validation process. However, we recognise that R&D activity is very competitive and market sensitive, potentially causing businesses to be reluctant to publish detailed R&D accounts. Alternatively, engaging with specialist expert users, who have strong knowledge of R&D activity within specific industries, would give ONS the opportunity to identify ways that it can further develop and enhance these statistics. We also encourage ONS to maintain and strengthen transparent data sharing with HMRC to better understand how tax policy changes can impact R&D activity and in doing so, enhance the quality-assurance process of producing BERD statistics. Such understanding should enable ONS to keep abreast of policy changes, updating methodologies when required.

2.15 As part of the quality-assurance process for the UK BERD: 2022 data, ONS used its expert user group to review the statistics, following its procedures and guidance for sharing the statistics prior to publication under a quality-assurance agreement. Following a verbal explanation, expert users were emailed a request to provide quality-assurance feedback and were required to sign a declaration form to request access to the BERD statistics before publication if they wanted to take part. Some of these users provided extensive and valuable feedback, and engaged with ONS in conversation afterwards. However, a handful of members explained that they received little contextual information about the request and were unsure what was expected of them when carrying out the quality assurance. Consequently, some members declined to participate, citing a lack of confidence in their ability to undertake the task with the information provided. Those that did accept remained unsure of the expectation of their contributions within the ONS quality-assurance process. Some users also told us that they found it difficult to quality-assure a single year’s data and would have preferred to contextualise it with a time series trend over time. To ensure greater coherence and transparency in the quality-assurance process, we recommend that ONS provide better guidance for experts to follow in future methods developments.

2.16 BERD statistics, at their current level of granularity, contain some caveats. Some users who advised policymakers felt it wasn’t clear that BERD activity is not distributed homogeneously within the International Territory Level 1 (ITL 1) regions, meaning policymakers could erroneously overlook the dispersion within regions. *For more information about ITL classification, please see ONS’s article on International geographies. ITL are used to classify regional breakdowns of data. ITL 1 is the largest classification, and ITL 2 is the next granular classification. Engaging with users to understand what further information they would find helpful, may contribute to a deeper understanding of the BERD concept and statistics.

2.17 Similarly, some users expressed a desire for an indicator of the data’s reliability and accuracy, such as a coefficient of variation, to be provided alongside the publications to strengthen ONS’s communication of uncertainty within BERD statistics.

2.18 Information about the quality of the BERD statistics can be found in the BERD QMI, which was last revised in February 2024 and reflects the recent improvements. ONS has also published several informative and explanatory articles regarding the recent transformation, including the uplift approach: Comparison of ONS BERD with HMRC R&D tax credit statistics; Options for Transformation of BERD statistics; Update on transformation of R&D statistics; UK BERD: 2022; and the Impact of new BERD data on Business Investment statistics. Users told us they welcomed the informative articles, which explained the methodological improvements to the BERD statistics. However, they felt it was often difficult to locate previous explanatory articles with relevant information, as these were published in different places rather than one central location.

User engagement

2.19 Users recognise the difficulties within transforming BERD statistics and were content with the phased approach that ONS had taken in adopting the new methodology for the release of the 2022 statistics, which better reflect R&D activity in the UK, incorporating sectors previously under covered and the improved sampling method. Both domestic and international users praised the explanatory presentations delivered by the BERD team throughout the development of these statistics. For example, ONS conducted an R&D statistics webinar in February 2024, explaining the methodological improvements and answering users’ questions. The participants commended ONS for its openness during this webinar. Users praised the various forms of engagement conducted by ONS believing it to have facilitated a well-managed and transparent introduction of the 2022 BERD data.

2.20 We commend ONS for establishing a technical user group and an R&D expert user group to inform users about methodological improvements and to gather feedback. Despite attempts to communicate the changes in advance, in line with the Code of Practice for Statistics, several users recollected their surprise at hearing that the uplift approach was being implemented for the BERD UK: 2021 release. This led users to ask for clarification on the changes in the BERD data. ONS should consider how it can strengthen its communication of methodological updates enhancing transparency and clarity on the changes it is introducing. Similarly, users outside the main statistical community, such as academics and research professionals, felt less informed and involved in the BERD methodological improvements and production process, despite the vital importance of the data to these users. It may lead to further strengthening of the statistics if ONS were to explore ways to expand such a formal user group. Such expansion could include proactively reaching out and inviting those from the secondary statistical community, including, but not limited to, academics and independent research organisations.