Compliance review of Justice Data Lab statistics

Overview

At the time of this report, the Ministry of Justice (MoJ)’s Justice Data Lab (JDL) statistics are published as official statistics.

JDL is an analytical team within MoJ that provides free analysis to organisations offering interventions to reduce reoffending in England and Wales, to help them understand their impact. Each JDL report presents the analysis results for a specific intervention programme. JDL’s work also contributes to MoJ’s evidence base on reducing reoffending.

This review examined the extent to which the JDL statistics meet the standards of Trustworthiness, Quality and Value in the Code of Practice for Statistics (the Code), with a focus on statistical methods. It aims to make recommendations for improving the statistics.

Why we did this review

Between December 2024 and February 2025, a concern was raised with us about the statistical methods that MoJ uses to produce the JDL statistics, specifically its approach to statistical significance testing when comparing the difference in headline reoffending measures between the intervention group (the treatment group) and the comparison group.

We investigated some aspects of MoJ’s statistical methods at the time. Although we were broadly satisfied with MoJ’s approach to statistical significance testing, we committed to carrying out a review of the statistics’ compliance with the standards of the Code. Accordingly, this review has considered, in more detail, MoJ’s approach to statistical significance testing, as well as aspects of the data collection, the aims of the analyses and the presentation of the statistics.

Highlighted findings

We are assured about MoJ’s approach to statistical significance testing. The JDL team has sought expert advice on the appropriateness of applying multiple comparison correction (a technique used to mitigate the increased risk of false positive when carrying out many significance tests on the same dataset) and all measures tested for statistical significance are specified in advance of the tests being carried out.

MoJ uses a consistent and rigorous methodology for JDL analyses, and peer review has helped MoJ refine JDL processes. The published information on JDL’s methodology is comprehensive. To improve the accessibility of methods information, MoJ should consolidate existing information into a central document that covers all stages of the JDL process from start to finish.

JDL reports contain clear and helpful advice for users on interpreting the statistics. Overall, uncertainty in the statistics and caveats and limitations are communicated well. To aid user interpretation, it would be helpful if more detail on key caveats and limitations was included in each report.

All JDL reports contain some content written by the intervention programme provider, including a summary of the programme and their views on the analysis results. MoJ told us that including this material helps build trust with programme providers, and providers value its inclusion. Nevertheless, this content is not impartial or objective and should therefore be published separately from the main report.

The summary spreadsheet of JDL publications helpfully brings together the findings of all JDL analyses to date. It also includes comparisons of findings across programmes, although one chart gives the impression that findings are directly comparable between programmes, when they are not.

We welcome MoJ’s renewed focus on user engagement. We encourage MoJ to monitor wider uses of the JDL statistics and consider how it can promote the statistics to a wider audience to maximise their value.

Our judgement

We have identified several recommendations that MoJ should address to enhance the Trustworthiness, Quality and Value of the statistics. Our recommendations ask MoJ to, among other things:

publish content written by intervention programme providers separately from the main report while continuing to make the content easily accessible to users
produce a central methodology document that outlines all stages of the JDL analysis process, including quality assurance arrangements
strengthen its rationale for not applying multiple comparison correction if this is deemed appropriate and include an explanation in the report
provide more detail on the most important general caveats and limitations in each report

Next steps

We expect MoJ to immediately implement the recommendation on the impartial and objective presentation of the statistics as this is critical to ensuring Code compliance. We will continue to engage with the JDL team as it implements the remaining recommendations.