Northern Ireland Safe Community Telephone Survey
I am writing to endorse the approach taken by the Department of Justice (DoJ) in adapting the Northern Ireland Safe Community Survey (NISCS) in light of the data collection restrictions during the COVID-19 pandemic. I congratulate your team for ensuring that data on people’s experiences of crimes and perceptions of crime and policing continue to be collected.
Statistics from the NISCS are vital for understanding the prevalence, type and impact of crime experienced in Northern Ireland. Since the start of the COVID-19 pandemic, your team has worked hard to establish a telephone survey (the Northern Ireland Safe Community Telephone Survey (NISCTS)), to replace the face-to-face survey, which was suspended in March 2020. Given the changes in survey methodology, I support your decision to publish statistics from the NISCTS as experimental statistics.
My team conducted a rapid regulatory review of these statistics. We have reviewed the extent to which they have been produced in accordance with the Code of Practice for Statistics’ Trustworthiness, Quality and Value pillars. Our findings are based on discussions with your team and a review of the first statistics from the survey, published in March 2022. I appreciate the time your team has spent talking through the development of the survey with my team. A summary of our findings, including recommendations, is set out below.
Value
- Your team worked closely with policy colleagues to determine which questions to include in the questionnaire. The headline estimate of crime prevalence was identified as the main priority as this was a Programme for Government indicator, but other areas of departmental interest were also prioritised.
- The statistical bulletin is informative and engaging. The commentary provides clear guidance for users on how to interpret the new statistics, in particular on comparing NISCTS estimates with those from the face-to-face NISCS. The break in time series is clearly indicated on the visualisations.
- There are some important gaps in the data and statistics from the NISCTS. For ethical and safeguarding reasons, which we support, the self-completion modules on domestic violence and abuse, and sexual violence and abuse were not included in the questionnaire. It is good to hear that the team is exploring the potential of developing an online version of the domestic abuse self-completion module so that data on this type of crime can be collected again.
- We welcome DoJ’s engagement with statisticians in the Office for National Statistics (ONS) and Scottish Government, who established similar telephone-operated crime surveys. We are pleased that all three teams talked about their respective surveys at the first Annual Crime and Justice Forum, hosted by ONS in February 2022. This user event facilitated collaboration and added insight for users on the similarities and differences between the three surveys.
Quality
- Unlike the telephone-operated crime surveys in England and Wales, and Scotland, the NISCTS did not re-contact participants from the previous face-to-face survey. A new survey sample was drawn, in line with guidance from the Northern Ireland Statistics and Research Agency, which ensured that there is consistency across household surveys. DoJ tried to keep the design of the NISCTS as close as possible to the NISCS. Data have been collected on a continuous basis during the pandemic since June 2020.
- The main limitation of the survey is the low response rate (15% compared to 56% for the NISCS), which led to a small achieved sample size (1,100, compared with 3,500 for the NISCS). This has affected the precision of the estimates and the level of disaggregation possible. The low response rate is a concern. NISRA introduced incentives in 2020 before discontinuing the NISCS and adopting a knock to nudge approach in summer 2021 in order to boost the response rate for the new survey. It is good to hear that these measures are having a positive impact. We encourage you be transparent about the changes to the response rate, to help users understand the impact on data quality and re-enforce DoJ’s trustworthiness.
- Methodological changes to the survey are well-explained. The technical annex describes all aspects of the survey design and methods, including sampling and fieldwork, and the availability of demographic breakdowns. The Things users need to know document published alongside the statistical bulletin is excellent. It supports user interpretation of the statistics by outlining the factors users should take into consideration when comparing 2020/21 data with data from previous years.
- We encourage you to carry out a study of the comparability of the estimates from the NISCTS and NISCS, like ONS did for the Telephone-operated Crime Survey for England and Wales. As a result, ONS produced an adjusted dataset for the face-to-face survey that allowed it to make direct comparisons with telephone-operated survey estimates. Producing comparable datasets for Northern Ireland would add insight on changes in crime victimisation before and after the pandemic.
- The information about uncertainty around the estimates is good. Confidence intervals are presented alongside the headline estimates for each crime type in the data tables, and the bulletin explains the impact of the response rate on the size of the confidence interval.
Trustworthiness
- Your team has been transparent with users about changes to the survey throughout its development. An August 2021 update explained the transition to a telephone survey and which questions and modules were shortened or removed. It was open about the low response rate and how this might affect reporting. A further update in February 2022 outlined the reporting plans, including how the findings will be presented and how the report will be structured. These updates have helped users understand the impact of the changes and the limitations of the new statistics. We welcome that, in response to our feedback, you have made these updates more visible by improving signposting on the main survey landing page.
- Similarly, the November 2020 Future Programme of Work update was open about the likely impact of the pandemic on timescales for development work.
Thank you for team’s positive engagement throughout this review. We look forward to seeing you continue to develop the telephone-operated crime survey. My crime and security statistics team will continue to liaise with your team over the coming months.
Yours sincerely,
Ed Humpherson
Director General for Regulation