Dear Mr Macdonald,
Concerns raised about the Scottish Intensive Care Society Audit of Critical Care
We have investigated the issues raised by an anonymous complaint about the Scottish Intensive Care Society Audit of Critical Care (SICSAG). As requested by the complainants, I am reporting our findings to you. We have focused our investigation on the key points relevant to the Code of Practice for Statistics published in February 2018 and its three pillars of trustworthiness, quality and value (see the subheadings below). We would be happy to discuss any of the issues raised in more detail with you or the authors of the letter.
Due to the importance of the issues raised, and the technical nature of some of the points, we commissioned an external review from an expert based at the Nuffield Trust to help us reach our judgement. His report (attached as Annex A) supported us in reaching our views.
Quality: sound methods
- Following the external review of methods (provided at Annex A), we are reassured that the statistical methodology applied by SICSAG in the reporting of intensive care mortality is appropriate, and that the approach taken for handling possible outliers follows international good practice in this area. The external review addressed the key methodological issues raised by the complainant. We also asked the reviewer to comment on the methods used to avoid large numbers of units being inappropriately classified as abnormal (over-dispersion) following advice from an international expert in this field.
Quality: assured quality
- Data ‘gaming’ is a very serious breach that would undermine the audit process and invalidate related statistics. The data validation process that Information Services Division Scotland (ISD) apply should identify data entry inconsistencies, whether accidental or deliberate, but we recommend that additional checks should be carried out to provide further reassurances. This is also supported by the external review.
Trustworthiness: orderly release
- Publication timeliness and orderly release dates are important markers of trustworthiness. We are confident that ISD’s processes in this area are compliant with the Code of Practice for Statistics. It was noted that in 2016 and 2017 the release date of the SICSAG report coincided with reporting of School Higher Examination Results for Scotland. ISD have released the SICSAG report on a consistent date since 2012 (2nd Tuesday in August). It is only in 2016 and 2017, due to a change in the week Examination Results were published (from 1st Tuesday in August to 2nd Tuesday in August), that the two coincided. We would not expect ISD to change its release date in the event of the release clashing with the release of school exam results.
Value: clarity and insight
- We consider that some aspects of the interpretation of the results could have been set out more clearly in the report. For example, as the external review highlights, more details of the relationship between outliers and the review process that results from their identification would have provided valuable context and additional insight to these statistics. It could also have provided reassurance that the statistics were being used appropriately.
- We welcome the changes made to the report to improve the clarity of the interpretation of outliers as a result of ISD’s own review following the complaints raised. We also encourage ISD to reflect on the external review’s additional feedback about areas to improve when they next review the report’s commentary.
Value: accessibility
- Between the 2015 and 2016 reports the volume of additional information available to download as tables and figures reduced. These changes should be informed by user needs and reasons for changes should be explained clearly. We recommend that ISD review the information provided alongside the published reports to ensure that user needs are being met. ISD should also explain the reasoning for decisions taken – including where information ceases to be published – more clearly to users.
We understand that the complainants wish to remain anonymous. ISD are keen to have a meeting between all interested parties to discuss the concerns raised. We support that and would welcome your assistance to facilitate such a meeting. If this is not possible, we would still welcome a meeting with yourself, ISD and SICSAG.
Please thank the complainants for bringing these matters to my attention.
Ed Humpherson
Director General for Regulation
Related Links: