Dear Ian
Statistics about Butterflies, England and UK: Confirmation of accredited official statistics status
We have independently reviewed the actions your team has taken to address the four requirements outlined in our assessment report.
On behalf of the Board of the UK Statistics Authority, I am pleased to confirm that they comply with the standards of trustworthiness, quality and value in the Code of Practice for Statistics and should be labelled accredited official statistics. We have included the detail around our judgement in the annex to this letter.
Defra has implemented various improvements to these statistics including to the information on the quality of the statistics and the methods used. We were particularly impressed at how quickly Defra was able to implement some changes for its October 2023 release, soon after the publication of our report.
The next release will see further improvements realised for example, combining reporting at UK level and for England into one release. In doing so understanding differences in data or trends in England from the UK as a whole will be easier to highlight for users. The bulletin will also include additional breakdowns of habitats and generalist or specialist species. These improvements should aid in bringing greater insights on an important area which helps users understand the state of our environment.
I would like to thank your team for its open and constructive engagement throughout and for your ongoing commitment to improving these statistics.
I am copying this letter to the team at Defra – Clare Betts, Karen Thomas and Madeleine May.
Yours sincerely
Ed Humpherson
Director General for OSR
Review of actions taken in response to Assessment Report 380: Statistics about butterflies, England and UK, produced by the Department for Environment, Food and Rural Affairs (Defra)
Requirement 1: Understanding the context and strengths and limitations of the statistics
To help users understand the context and strengths and limitations of the statistics, Defra should expand on the information it provides to its users about the methods used to create the statistics, the reasons for those choices, how limitations like the representativeness or spatial bias of the survey are accounted for and what uncertainties in the statistics arise as a result. This should include information on level of surveying, spatial bias, and any species-specific challenges in surveys. Spatial metadata on the survey could be accessibly provided using maps or heat maps or simply via text description that should allow users to assess how representative the latest year’s sample is and compare representativity over time. It would provide helpful reassurances to users for methods and decisions that align with a standard approach used internationally or in academia to be flagged.
Actions taken by Defra
Working with Butterfly Conservation as data supplier, Defra has expanded the information provided to users on methods and the strengths and limitations of the statistics. In particular, for its October 2023 release, Defra:
- reviewed the information within the Technical Annex for the statistics and sought to improve the readability of the information by removing less-relevant content for example. As part of this Defra has refreshed the information about the models used and the information on spatial bias and species-specific challenges.
- updated the information on assessment of change in the main bulletin to better explain how it reports on long and short-term changes over time, more clearly signposting to information on uncertainty. Within the bulletin Defra has also included a map of sampling sites to help users understand how representative the sample data are.
Defra intends to implement further improvements for the next annual statistical publication, scheduled for October 2024, which include:
- a time-series of the number of sites monitored since 1976. Defra has untaken this analysis of data provided by Butterfly Conservation which shows an increase in the number of sites monitored and that over time the indices have better spatial coverage.
- incorporating or signposting to other measures and information on sampling intensity and bias, for example, research currently being undertaken by UK Centre for Ecology & Hydrology (UKCEH) on how to improve the data about habitat and regional biases.
OSR’s evaluation of evidence
Within the short turn-around time between the assessment report and the October 2023 statistical output, Defra made positive improvements within the release and technical document to provide users with more information on the strengths and limitations of the data and methods.
Since then, Defra has continued to plan for further improvements in its next release as described. Defra detailed these in its development plan, published in February 2024, which also includes its longer term work to review methods and reporting.
Defra has shown a clear commitment to improve the information it provides to users on methods and the strengths and limitations of the statistics. Defra has shared a draft of some of this new information planned for its October 2024 release with us and we considered this will benefit users in understanding and interpreting these data.
Requirement 2: Understanding how the statistics can be used
To help users understand how the statistics can be used, Defra should provide appropriate and accessible information about long-term trends for butterflies, including consideration of where alternative trends might be considered and why a particular definition was appropriate.
Actions taken by Defra to meet the requirement
Defra’s commentary in the October 2023 release better explained the use of short-term and long-term trends.
To help explain and justify the selection of 1976 as the baseline year for long-term trends, Defra included explanation and trend data pre-1976 which shows users that the 1976 data are representative of the trend up to that point.
Defra consulted with both Butterfly Conservation and UKCEH regarding the short-term trend time period reported in the bulletin. Defra heard recommendations that using a 10-year, rather than 5-year period would be more representative of butterfly trends. Defra is acting on this feedback and plans to include this additional trend for its October 2024 release.
OSR’s evaluation of evidence
Defra has addressed a concern we had at the time of the assessment regarding the use of 1976 as the baseline year for long-term trends. It has also taken forward our recommendation of considering the use of alternative trends time periods which we are pleased to see should be implemented for the next release.
Requirement 3: Publishing development plans and outcomes of user queries review
Defra should publish its development plans and the outcomes of the review of user queries should be published alongside the statistics. Users should be made aware of any possible changes to the statistics and be given the opportunity to feed in their views.
Actions taken by Defra to meet the requirement
Defra published a development plan for its butterflies statistics in February 2024, which set out its short and longer term plans for these statistics. These plans were informed by a user engagement survey for the Biodiversity Indicators publications as well as the recommendations from this assessment.
Defra has been actively progressing its development work since then and will realise some of these improvements for its next release; of particular note is adding additional breakdowns of habitats and generalist or specialist species to the bulletin.
Defra has committed to ensuring that it updates its development plan annually.
OSR’s evaluation of evidence
Defra has been more transparent with its users in publishing its developments plans which we welcome. We consider this will aid and encourage engagement with users as well as demonstrate Defra’s clear commitment to continuous improvement for these data.
Requirement 4: Enhancing the public value of the statistics
To enhance the public value of these statistics, Defra should develop a strategy for user engagement in relation to its two sets of butterfly statistics. Where appropriate this should build on the wider strategy relating to biodiversity indicators to ensure a coherent approach to user engagement and optimise the resources devoted to engagement.
Actions taken by Defra to meet the requirement
Defra has developed and shared with us its user engagement project plan for its range of wildlife and biodiversity statistics. This included various planned activities using existing engagement routes but also plans for how it can explore different methods of engagement.
Over the summer and autumn Defra will continue to develop its research on improvements to engagement methods and seek to publish a finalised strategy by the end of the year.
Since the assessment Defra has undertaken various new engagement activities, these include:
- Adding a snap survey on data requirements and request for comments on how best to engage with users to the statistics landing page
- increasing the visibility of the Defrastats X account to help engage with its broader user base. The team is reposting from this account to tie in with relevant citizen science data collection exercises, for example the team helped promote the RSPB’s Big Garden Birdwatch in January
Defra has also utilised existing user and stakeholder events to discuss and promote its butterfly and biodiversity indicators including at the annual Biological Recording Verification and Interpretation meeting.
OSR’s evaluation of evidence
Defra has invested effort in enhancing its user engagement.
The Butterflies team is drawing on good practice from other parts of Defra and across government to ensure its engagements activities are more effective and align with broader departmental plans.
Defra has also thought creatively about how it may be able to tap into secondary users of the statistics – i.e. the many volunteers who are involved in data recording but who may not necessarily access the Defra statistics themselves.
At the time of the assessment, while Defra had strong ties with its key stakeholders – such as Butterfly Conservation – we felt Defra could do more to engage with its wider user base. We consider Defra has made good progress in addressing this concern.
Related Link:
Ed Humpherson to Ian Lonsdale: Assessment of Statistics about butterflies, England and UK