Dear Jon
Annual Statistics of Scientific Procedures on Living Animals, Great Britain
We recently completed our compliance check of your Annual Statistics of Scientific Procedures on Living Animals, Great Britain, against the Code of Practice for Statistics. These statistics are important to meet the legislative requirements set out in the Animals (Scientific Procedures) Act 1986 and provide users with transparency about the use of animals in research.
The compliance check was initiated in response to concerns raised with us on several occasions in 2022 via our casework function. These concerns focussed on errors that had been identified in the published statistics by the complainants, as well as the scope of these statistics.
Quality assurance
Your team outlined steps that have been put in place following recent errors and discrepancies in data collection, and shared with us its detailed, internal quality assurance procedures and documentation. We understand that, as part of this quality assurance, the team works with officials in the Animals in Science Regulation Unit (ASRU) to review the data and publication at various stages in the statistical process. This engagement is good, but the Code emphasises that statistics producers should establish and maintain constructive relationships with those involved in the collection, recording, supply, and quality assurance of data. In order to understand more about the operational context in which the data are collected, the team should consider how to approach building these relationships and instigate contact with establishments. We understand that there has previously been some interaction between the statistical team and establishments, and we encourage you to revisit and expand on this relationship.
We anticipate that the conditions of the project licence held by each establishment, and the audit process that establishments are subject to, are likely to provide some assurance regarding data quality management procedures that data suppliers will need to have in place. However, it is essential for you to investigate and gain a full understanding of the processes and quality assurance that the data are subject to prior to them being entered into the online data collection system. This should help to minimise the risk of errors, similar to those that have been brought to our attention, happening in future. You need to establish clarity, transparency and a clear understanding of the roles of, responsibilities of, and interactions between the statistical team, the establishments and officials in the ASRU with regard to data collection and quality assurance. This will ensure the appropriate arrangements are in place and understood by all parties, and when this information is documented it can then be shared with users to provide reassurance that robust processes are in place.
After gaining a more detailed understanding of the data collection and assurance processes, further information about the strengths and limitations of these statistics should then be published. Whilst some information is currently provided within the user guide, more detail needs to be provided regarding the data sources, methods, uncertainty and quality assurance processes. Making more-detailed quality assurance information available to users would help to reassure them of the various steps that take place and enhance the trustworthiness of the statistics. Detailed information about the importance of investigating, managing and communicating quality assurance procedures is detailed in the Quality Assurance of Administrative Data guidance. Whilst we welcome the reference to this within the release, the team needs to revisit this guidance and use it to evaluate and strengthen their current approach to quality assurance.
Revisions and corrections
We welcome that the release includes a revisions table that explicitly notes what has changed following the original publication (this appears on p33 of the November 2022 publication). However, it is not clear exactly why these changes have been made and, in some instances, hundreds of thousands of procedures are affected. Also, some revisions appear to be corrections (such as the reference to Xenopus and sheep previously not being included in the total for table 9.3) and should therefore be treated as such. The Code states that scheduled revisions or unscheduled corrections to the statistics and data should be released as soon as practicable, and ensuring all corrections are handled transparently, and in a timely manner, will ultimately help promote confidence in the statistics.
User engagement
The sensitive nature of the subject matter means that user engagement might be challenging. However, engaging with a wider range of users (which we understand was previously planned but postponed due to limited resources) may enable the statistics to add further public value. We have received several queries regarding the remit and scope of the statistics; for example, concerning the rehoming of protected animals and also animals that were bred for scientific procedures but were killed or died without being used in regulated procedures. These queries highlight the demand for further discussion with users about the relevance of these statistics, and also transparency around recent decisions to stop collecting certain data (such as establishment type). Our user engagement guidance provides a useful framework to help plan these activities. This wider engagement can then inform any future decisions made regarding these statistics that will help to ensure that they remain useful, relevant, and continue to support understanding of this important topic.
Clarity and insight
Whilst we found that the statistics are presented impartially and objectively within the publication, the commentary itself is sometimes too limited to enable helpful interpretation. For example, p14 of the November 2022 publication mentions a 96% increase in the number of experimental procedures using birds compared to the previous year, but provides no further information to help users understand the potential reasons for this. We acknowledge the effort that has been made to aid accessibility and user understanding, such as providing definitions of technical terms and a clear structure. However, given the technical language used throughout the release, the team should ensure that all necessary definitions are provided, and should also consider whether including a glossary would further aid understanding.
Innovations and improvements
Against these necessary improvements, we recognise that there has been a number of positive developments for these statistics that are listed within the release itself, for example:
- the transition to a new online data collection system, with automatic data validation to prevent invalid combinations of data being entered. This should ensure data are entered more accurately leading to statistics that are more trustworthy, better quality and therefore more valuable;
- the team has shown innovation via the creation of a reproducible analytical pipeline (RAP) to automate quality assurance processes and the production of statistical outputs, in line with RAP principles. Along with the new data collection system, this should reduce the likelihood of errors in the statistics and so enhance their trustworthiness. The use of RAP has also reduced the time taken to produce the report, meaning that the team are now considering bringing forward the release date;
- the report is now published in HTML format on gov.uk to improve accessibility;
- breakdowns by country are now included for particular tables in the release. This increase in granularity should allow more nuanced use and add to the value of the statistics.
It is good that the team has taken these proactive steps to continually improve how the statistics are produced and published.
Next steps
Within this letter, we have outlined some specific findings that it will be important for your team to reflect and act on in order to continue complying with the Code, and we ask that you report back to us with your plans for addressing these in April 2023. I would like to thank your team for their open and positive engagement on this review. Our Transport, Environment and Climate Change Lead will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspect of this letter further. I am copying this letter to Will Reynolds, Head of the Animals in Science Policy and Coordination Unit, and Amy Baxter, Deputy Head of Profession for Statistics.
Yours sincerely
Mark Pont
Assessment Programme Lead