Dear Ross
Scottish Fire and Rescue Incident and Organisational Statistics
We recently completed our compliance check of your Scottish Fire and Rescue Incident and Organisational Statistics against the Code of Practice for Statistics. We last provided an informal review of these statistics in 2018, before Scottish Fire and Rescue Service (SFRS) became an official statistics producer. The recent change in the law in Scotland for every home to have interlinked fire alarms has increased public interest in these statistics.
During our review, we found a number of positives in the way that these statistics are produced and presented, which demonstrate their quality, value and trustworthiness:
- The statistics are well-presented, are clear and easy to follow, and offer a good depth of insight and analysis. The guidance notes provided alongside the statistics are detailed and accessible, providing users with information on comparability to previous releases, quality assurance and definitions. Users can now see a pre-release access list and any corrections or revisions made to the statistics via the statistical news publication.
- The statistics team has been responsive to changing user demands. For example, the statistics now provide more detail on the workforce and include new tables for both fires and casualties occurring in purpose-built multi-storey flats (including high-rise flats), houses in multiple occupation, and other multiple occupancy property types. The statistics team plans to develop new statistical tables on topics related to emergency incident communications.
- A ‘Command and Control’ Management System for recording incidents by fire services is being developed that makes it easier to provide data to the statistics team. Its new reporting features will allow statisticians to better assure the quality of data via more automated checks and the removal of manual checks.
We also identified some ways in which the trustworthiness, quality and value of the statistics could be enhanced:
- Further staffing resource in the data team and investment in streamlining statistical processing, analysis and quality assurance would allow the potential value of SFRS statistics to be fully realised. For example, the statistics team has concerns about the quality of data from a new system that records the type of, number of and stock value of smoke alarms but has little resource to investigate problems with inaccurate records of installation or stock counts.
- To enhance its understanding of data quality more widely, SFRS should further review the quality of data drawn from different administrative sources, using our Quality Assurance of Administrative Data (QAAD) framework – to better understand for example, types of checks carried out by data suppliers, and the operational circumstances in which the data are produced. The statistics team is drawing on administrative data provided by third parties, and for some of these sources the statisticians do not have direct access to systems and data. There are also currently no service-wide agreed standards for data quality, no processes in place to scrutinise data, or systems to report concerns with data quality. The team told us that this had resulted in poor quality or misleading statistics being presented to Boards and Committees, which creates a risk for effective decision making and for the reputation of the Service.
- While we recognise SFRS has improved the quality information that it publishes, this could be further strengthened by including additional insights obtained from carrying out further investigations into data quality, as described above. This should include, for example, details of any uncertainties with or risks to the statistics from known or potential data quality issues and how risk is minimised to prevent misinterpretation or misuse. The team should also communicate its approach to overall quality assurance management. As an example of good practice, Home Office has published a transparent assessment of the quality of sources for its Fire and rescue incident statistics for England.
- SFRS has worked hard to address some data gaps that users identified, but more extensive user engagement may help to further improve the public value of the statistics. Setting up an active user group to discuss developing the statistics on a more regular basis may be beneficial. We recently published guidance on user engagement which the team may find useful to plan future engagement activities.
- The statistics team told us there are compatibility issues with the SFRS Fire Recording System and the Home Office live online IRS. The National Fire Chiefs Council is leading a project with Home Office to build a new version of IRS, that will in the long-term look to capture more than just operational data. The IRS has the potential to greatly benefit all three nations; we encourage SFRS to work closely with Home Office and Welsh Government statisticians to overcome the compatibility issues and ensure it accurately captures SFRS data.
- The statistics team told us that it increasingly has responsibilities for broader SFRS data and analysis. We would strongly advocate voluntary application of the three pillars from the Code of Practice – Trustworthiness, Quality and Value – more broadly across the SFRS in order to provide the public with confidence in the data.
I would like to thank you and your team for your positive engagement throughout this review. After your next round of outputs, planned for Autumn 2022, our Housing Planning and Local Services domain lead will be in touch to discuss the progress made against our recommendations in this letter. Please do not hesitate to contact us if you would like to discuss any aspects of this letter further.
I am copying this letter to Mark McAteer, Director for Strategic Planning, Performance and Communication, Chris Fitzpatrick Business Intelligence and Data Services Manager and Rebecca Cameron, Statistical Analyst at the SFRS.
Yours sincerely
Mark Pont
Assessment Programme Lead