Dear Siobhan
STATISTICS FROM THE NORTHERN IRELAND SAFE COMMUNITY SURVEY (NISCS)
We have completed our review of the compliance of statistics from the Northern Ireland Safe Community Survey, produced by the Department of Justice (DoJ), with the Code of Practice for Statistics. This was the final part of our three-part review, which began in May 2018. This letter sets out our key findings from the review. I am pleased to confirm the statistics from the survey should continue to be designated as National Statistics: in this letter we have highlighted several areas for improvement, which will ensure the statistics continue to meet the highest standards of the Code.
As we completed our review, the COVID-19 pandemic meant the Safe Community Survey, which was a face-to-face survey, had to pause while arrangements were made to collect data for the 2020/21 statistics by phone instead. We support this approach, which will ensure the safety of those collecting data for, or contributing data to, the statistics. We look forward to seeing how your plans develop for the 2020/21 statistics and encourage the team to keep users of the statistics informed about the changes.
The rest of this letter briefly summarises our approach to the recent review and our findings, which will apply to the 2019/20 statistics. We illustrate areas where progress has been made since the first part of the review, as well as areas where we consider improvements should still be made to enhance the value of the statistics.
Our review of these statistics has been in three parts; this part focusses on the presentation and reporting of the statistics. As part of our research, we spoke to a range of users of the statistics to understand the impact of recent changes to the statistical bulletins and the extent to which users feel that their needs are being met.
In each part of our review, we considered the trustworthiness, quality and value – all of which are fundamental to supporting public confidence in statistics and, together, form the basis of the Code – of the survey and the statistics. Taken together, our findings across the review are as follows.
Trustworthiness
Ensuring sufficient resource
The team has taken on two new statisticians during the course of our review, allowing improvements to the current statistics and to address the improvements suggested in the earlier phases of our work. A new member of staff has now joined the statistics team and the team has published a Future Programme of Work, which provides a high-level outline of developments that the team aims to address in relation to the survey and statistics. These steps demonstrate DoJ’s prioritisation of, and commitment to, the statistics, while offering greater transparency about upcoming changes to the statistics for users. Users we spoke to said that engagement on the Future Work Programme had been good and they were happy with the information around the proposed statistical priorities. We note that dates for different pieces work on the Future Programme are still to be confirmed. In order to enhance trustworthiness, the team should include more-explicit timescales in the Programme and we encourage DoJ and NISRA to continue to ensure enough resources in the team to fulfil this work. Including this information will also allow users of the statistics to more easily engage with DoJ on the order and prioritisation of developments.
Quality
Improving data sources
Throughout this review, the statistics team has demonstrated ongoing commitment to improving the quality of the statistics from the Safe Community Survey. In general, the information about quality provided alongside the survey and statistics in the Quality Report and User Guide is extensive, and provides helpful information that allows users to understand the strengths and limitations of the data and statistics.
The survey sample size has increased from 2,000 to 3,500 from April 2018. This was achieved through additional financial investment in the survey, in response to feedback that statistics from the survey were no longer adequate to track progress against outcomes in the Northern Ireland Executive’s Programme for Government Framework. Given the change in the sample size, it was good to see that the team had done some work to ensure the integrity of the long-term crime trend estimates, and included confidence intervals around published survey estimates, so that users can more easily understand the quality of the statistics.
Regional Data
Despite the increase in survey sample size since 2018, some users of the statistics we spoke to still feel that estimates below country and population level need to be more robust. For example, when looking at disaggregated data for different counties in NI, users feel they are unable to make evidenced-based decisions because the confidence intervals around the estimates are so large. This issue arises because the sample sizes for geographic areas and demographic groups below country and population level respectively are relatively small. The team is aware of the user demand for robust statistics at a more disaggregated level, especially for analysis at Local Government District level, and is currently considering how this might be met. In addition to considering how the user need at low geographic levels can be met, the statistics team should consider how it can most effectively explain uncertainty around the survey estimates, so that users understand why the confidence intervals around disaggregated data are variable and can be large. The team should also be clear about how the statistics can and cannot be used.
Impact of COVID-19
It is good that the statistics team was able to move from face-to-face survey data collection to phone data collection. The team has been working well with policy colleagues to discuss changes to the questions and informing users of the changes happening to the survey on its website. We understand that there are some areas that will not be able to be covered in a phone survey. However, it is good know that important topics have been retained, such as:
- experience of crime
- perceptions of crime and anti-social behaviour
- confidence in the police and police accountability arrangements
It is also good that the team has contacted both ONS and Scottish Government to discuss their experiences of changing to a phone survey design, this type of cross-nation engagement will further the improvement of the statistics, particularly during the current challenges. The team now has quarterly meetings with statisticians from the ONS and the Scottish Government, which will enable shared learning and provide an opportunity to discuss current issues and future development.
The change of data collection method will have an impact on the type of data that can be collected and the questions that can be asked. As this work develops, we encourage the team to be clear to users in advance about the impact this may have on future statistics, for example on trends in domestic violence, which could be difficult to collect by phone.
Value
Continued relevance to users
We have been impressed by the statistics team’s collaborative and user-focused approach to developing these statistics. This approach continues to help the statistics team through the current difficulties; the communication between the team, the Scottish Government and ONS will ensure the continued value of the statistics despite a more reduced output in 2020/21.
In 2018, the statistics team completed a consultation process and had extensive dialogue with users and potential users in government about their needs from, and proposed changes to, the survey and statistics. The team took account of these needs, by accommodating demand from policy teams to collect data that reflect important issues not previously covered by the survey. Several of the users we spoke to had been able to input into the questions asked in the survey. To maintain an understanding of the use and potential use of the statistics and data, the team now holds a biannual working group meeting with key policy and statistical stakeholders. In addition, the team has recently completed a branch-wide customer survey which sought comments on individual branch publications including NISCS reports. We look forward to seeing the results of this survey. Having this dialogue will help ensure that the maximum value is achieved from the survey and the statistics.
One of the key messages from the DoJ consultation in 2018 was that the presentation and reporting of the statistics needed to be improved. In response, the statistics team has introduced a new layout for its statistical bulletins, with changes to the content and presentation of the data and supporting narrative. Many of the users we spoke to now find the statistics to be useful and the narrative easy to follow: in particular, the use of annotations on charts has proved effective, as this helps users to better understand trends in the data.
Greater insight for users
As part of the improvements to the statistical bulletins, the inclusion of tables of police recorded crime data alongside the survey data in the Experience of crime publication gives users a fuller picture of crime in Northern Ireland. Our user engagement highlighted that, while the addition of these data is welcomed, there is still more that could be done to maximise the insight users get from them: the statistics team should include commentary that highlights which source of data is best for answering the different questions users might have about different crime types, for example. The crime statistics produced by ONS do this well and provide a helpful example. We anticipate the positive relationship the team already has with statisticians in ONS will facilitate greater insights.
Among the planned developments to the statistics, an issue worth highlighting is the clear demand to understand child victimisation. It has been positive to see questions being included in the Young Persons’ Behaviour and Attitudes Survey as part of an effort to meet this need, it also good that the Northern Ireland Safe Community Survey: Assessment of the Feasibility to Include Children Aged 10 to 15 and Communal Establishments was published on 17 April 2020. To maintain transparency, we encourage the team to continue to communicate its plans effectively to users in this area. This is an important issue and it is vital that the team is transparent with users about whether and when the data and statistics could be improved, remaining clear about the possible effects of moving to a phone survey.
I would like to thank your team for working closely with us throughout the review process and for its commitment to improving these statistics. Our Crime and Security team will continue to liaise with you and your team and, if you wish to discuss any aspects of this letter, we are happy to do so. I look forward to seeing the trustworthiness, quality and value of the statistics continue to improve. I am copying this letter to Ruth Fulton, Head of NISRA Statistical Support Branch; and to Joan Ritchie, the crime statistician at the Department of Justice.
Yours sincerely
Mark Pont
Assessment Programme Lead
Related Links
Compliance check of Northern Ireland Safe Community Survey