Data quality

1.25 Several users told us that the quality of the official statistics meets their needs and that they trust Scottish Government analysts to produce high quality statistics. However, some users queried with us aspects of the PR2 system. They had concerns about outdated IT infrastructure, the system’s inability to measure certain features of the prison population (for example, due to the way information is recorded it cannot be used to track prisoners’ journeys at an aggregate level), and the level of, and lack of information about, quality assurance of data.

1.26 Scottish Government is confident in the quality of the data used to produce the statistics. It told us that PR2 serves the functions it was designed for – understanding and managing the prison population – and that it accurately records information. The system was not designed for analytical purposes, which presents challenges for extracting and analysing data. In particular, PR2 does not easily allow longitudinal population analysis (because information on the system is overwritten) or the interrogation of data for certain purposes, such as producing a collective measure of time spent of remand. To overcome these limitations, Scottish Government built a new dataset based on PR2 but constructed in a different way (the Cellwise method). The Cellwise dataset is longitudinal, which allows the measurement of flows into and out of prison, changes in prisoner demographics, and changes in prisoners’ custodial ‘journeys’.

1.27 The bulletin and technical manual are clear about the nature and limitations of PR2. They cover general limitations of the system, such as instability of data, as well as limitations relating to specific aspects of the data, such as missing information about warrants or sentences. Most users we spoke to appreciate the transparency about limitations and think it provides helpful advice for interpreting the statistics. We consider that Scottish Government should go further in assuring users about the suitability of PR2 for producing statistics by explaining the strengths of the system and why it is confident in the quality of the data.

1.28 The technical manual contains a brief overview of Scottish Government’s quality assurance (QA) arrangements, describing the types of errors that arise and how they are resolved. However, it does not explain SPS’s role in processing and quality assuring data, for example, how it checks the accuracy and reliability of information recorded by prison officers. Given users’ concerns about quality assurance, it is important to reassure them about the level of quality assurance applied at all stages of data collection and statistics production.

1.29 To ensure the statistics continue to deliver the insight that users need, Scottish Government and SPS should continue to invest resources in PR2. This applies to the people using the system: skilled analysts are vital for producing trustworthy, high-quality statistics. The new SPS Head of Data and Analysis provides a much-needed boost to SPS’s analytical capacity and capability and plays an important role in coordinating and facilitating knowledge sharing between SPS and Scottish Government. This should lead to a better understanding of data quality across both organisations and support more detailed and insightful analysis. We encourage Scottish Government to make the most of this relationship, and where possible, to share any knowledge with users by updating the quality information.

1.30 The technical manual outlines quality issues for demographic characteristics data and explains why certain characteristics (such as prisoner religion) are excluded from analysis. It explains how information is recorded for most demographic characteristics apart from gender. Currently, it states that “the collection of prisoner gender is determined by SPS recording policy”, with a link to the policy. To support understanding of the nature of the data, it is essential for Scottish Government to add a description of the approach to recording gender in prisons. This should clarify whether sex or gender identity is being recorded, how it is being recorded (for example, if responses are recorded after the prisoner has been asked or if it is assumed by the data collector), and the extent to which recording practice varies between prison establishments. It is important that the terms sex and gender are not conflated. The team may want to consult our draft guidance on collecting and reporting data about sex in official statistics. It would also be helpful to signpost SPS’s quarterly performance reports as these provide more information on the gender of prisoners. If relevant, the technical manual should highlight any impact on the statistics of the ongoing SPS review of the policy.

1.31 The Cellwise method used to construct the statistics is sound. The team worked with a colleague with operational expertise to ensure that the method replicates the picture of the custodial cell experience of each person. It engaged with policy colleagues and other stakeholders as it developed the method, to assure them of its robustness.

1.32 The technical manual gives a step-by-step account of the construction of the Cellwise dataset. It covers the assumptions made and the analytical factors and measurements, with illustrated examples to aid understanding. While we welcome this level of detail and transparency, less technical users told us the information can be difficult to follow. To enhance the accessibility of quality and methods information for all users, Scottish Government could add a high-level summary of the key strengths and limitations of the methods and data source.

1.33 The bulletin outlines the differences between the Cellwise dataset and other sources of information about the prison population, including the earlier official statistics published by Scottish Government and the aggregate figures published by SPS. This helps users understand the comparability and coherence of the statistics. Despite these differences, the Cellwise method produces an estimate of the overall prison population which is very closely aligned to the other sources, giving Scottish Government and the public confidence in the statistics.

1.34 Due to the way information is recorded on PR2 and the way the Cellwise dataset is constructed, there is uncertainty in the estimates. The language in the 2020-21 bulletin gives some indication of this uncertainty (for example, a figure being “around X”), but it should be more explicit about the fact that the statistics are estimates, and the language used throughout the bulletin should be consistent. It is good that the team has added a clear statement to the introduction of the 2021-22 bulletin about the figures representing estimates. There is also uncertainty in the population estimates and rates used to make comparisons with the prison population, as these are derived from household surveys. Although the technical manual mentions that ethnic group sizes from the Scottish Survey Core Questions (SSCQ) have a margin of error, it does not explain how this is calculated or how users should interpret the intervals.

1.35 The team applies Reproducible Analytical Pipeline (RAP) principles to the production of the statistics, which supports robust quality management. Data extraction from SPS systems is automated and an open-source programming language (R) is used to process data. It is good that the team is exploring further opportunities to embed RAP principles, including the use of version control software. The team may also like to consider publishing code online, to demonstrate public transparency.

1.36 Our main concern about to the current setup is that there is only one statistician in the team with access to the data and sufficient knowledge to run the code.