Introduction

Background to the statistics

1.1 The Justice Data Lab (JDL) is a small analytical team within the Ministry of Justice (MoJ) that supports organisations that provide offender services in England and Wales. It was set up in 2013 with the aim of giving organisations access to reoffending data to help them estimate the effectiveness of interventions to reduce reoffending.

1.2 Organisations supply data to JDL on the offenders who received the intervention. JDL uses these data to match offenders to records in the Police National Computer (PNC) database. It then uses a statistical technique called propensity score matching (PSM) to create a comparison group of offenders who did not receive the intervention. Offenders are matched on a range of variables, including demographic characteristics and offending history, so that the comparison group is as similar as possible to the intervention group (treatment group). Once matched, JDL conducts statistical significance testing to compare the reoffending outcomes of the treatment and comparison groups.

1.3 Analyses include headline results on reoffending outcomes (whether individuals have reoffended and the time and frequency of reoffending), typically over a one-year period after their release from prison. If the sample size is large enough and the matching quality is high enough, a series of sub-analyses are also conducted to investigate the impact of other factors, such as attendance levels, method of delivery of intervention programmes and participant demographics, on those reoffending outcomes.

1.4 JDL support is used by a wide range of organisations. These include both ‘internal’ programme providers, for example, His Majesty’s Prison and Probation Service (HMPPS), and ‘external’ programme providers, encompassing mainly organisations in the voluntary and community sector, such as charities, educators and public and private bodies. As JDL is an analysis service, it is not involved in the design or delivery of programmes.

1.5 MoJ publishes a separate report for each programme it analyses. It has published over 250 JDL reports, all of which are published as official statistics. MoJ is one of the few UK Government departments that routinely publishes impact evaluation analysis as official statistics.

1.6 JDL’s work contributes to MoJ’s evidence base on reducing reoffending.

Why we did this review

1.7 Between December 2024 and February 2025, a concern was raised with us about a statistical method that MoJ uses to produce the JDL statistics, specifically its approach to statistical significance testing. The concern focused on whether MoJ should be applying multiple comparison correction (MCC) methods when carrying out many significance tests on the same dataset to account for family-wise error – the probability of incurring at least one Type I error (a false positive) when performing multiple statistical hypothesis tests. MCC mitigates the increased risk of false positives (the likelihood of finding a statistically significant result by chance) and ‘p-hacking’ (persistent statistical testing until a statistically significance result is found). It does this by adjusting the statistical significance threshold, making it more difficult to achieve statistical significance.

1.8 We investigated some aspects of the statistical methods at the time the concern was raised. Although we were broadly satisfied with MoJ’s approach to statistical significance testing, we committed to carrying out a review of the statistics’ compliance with the standards of Trustworthiness, Quality and Value in the Code of Practice for Statistics. Accordingly, this review has considered, in more detail, MoJ’s approach to statistical significance testing, as well as aspects of the data collection, the aims of the analyses and the presentation of the statistics.

1.9 For this report, we initially reviewed a sample of JDL reports and all methodology reports and user guidance documents (see Annex for full list). The JDL reports we reviewed were published from 2013 to 2025 and cover both internal and external programmes. We examined the consistency of the presentation of the statistics, methods descriptions and terminology between reports.