UK Statistical System UK Statistics Authority Code of Practice
Compliance check

Mark Pont to Lucy Vickers: Adult Substance Misuse Treatment Statistics for England

We recently completed our compliance check of the Department for Health and Social Care’s (DHSC) Adult Substance Misuse Treatment statistics against the Code of Practice for Statistics. The statistics are published in a provisional monthly release supplemented with an annual report the following year. The statistics were designated as National Statistics in 2015 and provide valuable insight into adults receiving treatment for substance misuse and the characteristics of different substance users. I am pleased to confirm that these statistics should continue to be designated as National Statistics. 

We carried out this review because the responsibility for the production and publication of the statistics was transferred from Public Health England (PHE) to the Office for Health Improvement and Disparities (OHID) within DHSC in October 2021. 

Our compliance check found many positive features that demonstrate the trustworthiness, quality and value of the statistics. For example: 

  • These statistics are important for policy makers and service providers, at the local level and for England as a whole. The value of the statistics is enhanced by the commentary and context provided throughout the annual report – for example, a section is included in the statistics about the policy context, there is discussion of the impact of the coronavirus pandemic on the statistics and trends in characteristics since 2005/06 are presented. Furthermore, the statistics are published in a variety of formats to suit different users’ needs such as charts, data tables and the interactive ViewIt tool. 
  • The value of the statistics is also enhanced by the level of granularity of the different regions. The provisional monthly release provides a breakdown of trends at a local authority level, allowing users to understand their local area. 
  • It is great to hear that the team has a good understanding of users of its provisional monthly data. These users include treatment providers, clinicians, local authorities and the general public. It is also good to hear that the team consults with users every few years on planned changes to the underlying dataset. Engagement is an essential part of maintaining the relevance and usefulness of the statistics for users. 

We identified some areas that would further enhance the trustworthiness, quality, and value of these statistics, some of which the team was already considering: 

  • It is good to hear of the team’s plans to update the Quality and Methodology report published on the National Drug Treatment Monitoring System this year. The report is exceptionally detailed and provides good assurance to users regarding the quality of statistics. However, the report was last updated in 2020 so it is not clear whether there have been changes to quality or methodology that users should be aware of. We consider it would also help users’ understanding if you were to add a short summary of the essential aspects of the statistics’ quality and methods, such as any limitations or what is and is not included in the statistics, within the annual report. This would help support appropriate use of the statistics without having to access a different website.  
  • While the team has a good understanding of users of its monthly statistics, we support its plans to expand user engagement activities relating to the annual report. Gaining a better understanding of what users require from the annual publication over and above the monthly statistics and what users want regarding the presentation of the statistics would be helpful as you continue to develop them. 
  • Finally, it is good that the team has committed to make some small changes that will enhance the demonstration of the trustworthiness of the statistics. This includes adding contact details to the annual release, adding information about roles of individuals to the pre-release access list and providing greater clarity regarding the move from PHE to OHID.   

I would like to thank your team for their positive engagement with us during this review. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter.  

Yours sincerely 

Mark Pont 

Assessment Programme Lead  

Mark Pont to Sarah Crofts: Compliance review of ONS’s Health Index for England

Dear Sarah 

Compliance review of ONS’s Health Index for England

We recently completed our compliance check of the Health Index for England against the  Code of Practice for Statistics. The Health Index is an important new development that provides an overall picture of health both for England as a whole and for regions and local authorities. The index is comprised of 56 indicators across a broad range of aspects of health within the three main domains of Healthy People, Healthy Lives and Healthy Places. This allows users to understand how health and its components has been declining or improving over time, and how it varies from area to area. The most recent release, published in November 2022, provides important insights into health in the earlier stages of the coronavirus (COVID-19) pandemic at national, regional and local authority level. 

Overall, our review found that the statistics demonstrate compliance with many areas of the Code of Practice. It is clear both from the published outputs and from talking to your team how much thought and hard work has gone into the development of the Index to date. We found a range of positive features, as well as some areas for improvement that we consider would enhance the trustworthiness, quality and value of the statistics. Addressing these areas will be beneficial as you work towards obtaining National Statistics designation for the Health Index. 

  • Both the main release and the local authority tool provide a good insight into how aspects of health have changed (or not) since the Health Index started in 2015. The explanations of the findings are clear, as well as possible reasons for some of the findings being provided. The improvements to the release for 2020 data reflect a lot of work from your team. 
  • It is good that you have collaborated with such a wide range of experts in the form of the Expert Advisory Group (EAG) in constructing the Health Index. We are also pleased that you have worked with The Alan Turing Institute to quality assure the Health Index methods, as well as having plans in place to develop the methodology further. This demonstrates a clear commitment both to ensuring that the methodology is sound, and to ongoing refinement of the methodology as you continue to develop the Index. We note that planned future developments include a health projections model and enabling local authorities to produce their own health indices. Your ongoing commitment to further developing the scope and utility of the Health Index is likely to benefit a wide range of users. 
  • It is evident from our conversations with your team how much consideration it has put into the various methodological aspects of the Index and how these might be taken forward, yet it is sometimes hard for users to see or appreciate this. Sharing details of both activities to date and planned work would help users to understand the amount of effort that is going into making the Index as useful and useable as possible. When sharing your plans, you should also provide details of how you plan to prioritise work going forward so it is clear to users when developments may occur. 
  • You should take into account uncertainty when presenting the statistics to help avoid over or under-interpreting. For example, you could explain changes more carefully, as there are some occasions where small differences are presented as though they are statistically significant, and some users might interpret these as being more meaningful than they really are. Your team told us that it has done a lot of work to try and understand how best to understand uncertainty in relation to the Index, and that it plans to continue to do so. It is fundamental to be clear to users about sources of uncertainty to help them in interpreting the findings appropriately. It would be helpful if you could bring together your work so far and publish, separately, some information on this and your plans for your users.  
  • Your team told us about its user engagement activities, and particularly the work that it has carried out at the local level to understand who uses the Index, what for, and what decisions it might inform. A notable example that the team told us about is specific NHS Trusts using the Health Index framework to develop a version at a lower geography to better understand and target health issues in their authorities. Expanding this excellent work to understand more about users and uses of the Index at national level and in the third sector will help to add further value. We welcome that your team is starting to develop its plans in this area.  
  • Although you have improved the timeliness of the most recent release compared to previous ones, it is still relatively out of date. It is good that you are looking to further improve the timeliness of the Health Index, potentially with an early provisional release, as well as exploring the trade-offs between timeliness and accuracy. Seeking users’ views on this possible provisional release as part of the planned user engagement work outlined above, would help you understand whether a more timely but possibly less accurate early release will be of benefit to them and make the Health Index more useful overall.  
  • Your team told us that it follows Reproducible Analytical Pipeline (RAP) principles in its work and gave strong examples of this. There are some areas where future developments could still be possible, such as automation of your quality assurance procedures or investigation of ways to semi-automate the factor analysis. We encourage you to keep this on your work programme and to continue to investigate where further improvements might be possible. 

 I would like to thank your team for their positive engagement on this review. Our Health and Social care domain will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.   

 Yours sincerely 

Mark Pont 

Assessment Programme Lead 

Mark Pont to John Marais: Statistics on modern slavery in the UK

Dear John 

Statistics on modern slavery in the UK

We recently completed our compliance check of your Modern slavery in the UK and Child victims of modern slavery in the UK compendium articles against the Code of Practice for Statistics. We carried out a complementary compliance check of Home Office’s National Referral Mechanism (NRM) statistics, as the articles identify the NRM statistics as the best available source of the number of potential modern slavery victims in the UK.  

Modern slavery is a term for all forms of human trafficking, slavery, servitude and forced labour. Measuring the prevalence of modern slavery is challenging because there is no single source or method that can accurately quantify the number of victims. Our review focused primarily on the value and coherence of the articles, looking at how they bring together and explain statistics and data from a range of sources.  

Both articles are standalone releases, and there are currently no plans to update them. Given the importance and public profile of modern slavery, and the insight generated by the articles, we encourage you to put in place plans to update them, particularly the modern slavery in the UK article. We identified a number of strengths as well as a couple of ways in which the value and quality of the articles could be enhanced.  

Both articles draw on a wide range of data sources to present a detailed picture of the scale and nature of modern slavery. They bring together official statistics from UK Government departments and the Devolved Administrations, with data which are not official statistics, from the Crown Prosecution Service, charities and support services. The statistical commentary is clear and accessible, with explanations of key trends in the data and reasons for changes over time. The modern slavery in the UK article contains a wealth of background information and context, including an excellent flow chart of the NRM process, through which potential victims are referred.   

We found a range of other positive features that demonstrate the value and quality of the statistics: 

  • The collaborative approach used to produce the articles is excellent. Your team worked with stakeholders across academia, government and the third sector to understand information needs, source data, and review data quality. The guest blog post by Professor Sir Bernard Silverman, published on the same day as the modern slavery in the UK article, is a good endorsement of the collaboration and the value of the articles. 
  • The articles are some of the few statistics outputs that bring together crime and justice data from England and Wales, Scotland and Northern Ireland. The driver for creating a UK view is the UN’s Sustainable Development Goals (SDG) indicators, several of which measure the UK’s progress towards combatting modern slavery. For some sources, data from UK countries cannot be combined due to differences in definitions and recording processes – for example, for modern slavery offences recorded by the police. Nonetheless, the articles provide the most comprehensive and coherent picture of modern slavery in the UK.  
  • The articles demonstrate a good awareness of the sensitive nature of modern slavery. Both articles contain a ‘Finding help’ section, with contact details of victim support services. The child victims article intersperses statistical commentary with quotes from victims. This is a powerful way of bringing the statistics to life and highlighting the impact of modern slavery on victims.   
  • Both articles are clear about the challenges of measuring modern slavery; in particular, the lack of a definitive source of data or suitable method to quantify the number of modern slavery victims in the UK. The articles are transparent about the nature of all data sources and their limitations, including the lack of comparability of data sources due to differences in coverage or data collection methods.  
  • The framework used to evaluate and categorise relevant data sources – each data source has been badged as a ‘legal’, ‘support’ or ‘awareness’ indicator – is sensible and helps users understand the complexity of the modern slavery data landscape. It is good that this framework was developed and tested with stakeholders.  
  • The accompanying article on Sourcing data on child victims of modern slavery gives a clear overview of how suitable data sources were identified and reviewed for the SDG indicators relating to modern slavery and children in the UK. 

We also identified two ways in which the quality and value of the statistics could be enhanced: 

  • To aid user understanding of the quality of all data sources, information should be added on the strengths and limitations of data which are not official statistics, including data from victim support services, helplines, and applications, and how data from these sources were reviewed and quality assured. We encourage the team to continue exploring other data and data sources that could add insight on modern slavery. 
  • Signposting between the two articles could be improved, particularly as the child victims’ article does not contain the same level of detail as the modern slavery in the UK article. We recognise that the child victims’ article was reduced in size to make it more accessible and user-friendly.   

Thank you to your team for its positive engagement during this review. We look forward to continuing to engage with your team. Please get in touch if you would like to discuss any aspects of this letter further or if we can offer further assistance as these statistics continue to develop. 

I am copying this letter to Meghan Elkin and Peter Jones, heads of the Centre for Crime and Justice at ONS; Tom Bucke, Head of Unit for Drugs, Exploitation and Abuse Analysis at Home Office; and Jon Simmons, Head of Profession for Statistics at Home Office. 

Yours sincerely 

Mark Pont 

Assessment Programme Lead 

 

Related Links:

Mark Pont to Jon Simmons: National Referral Mechanism and Duty to Notify statistics

Mark Pont to Jon Simmons: National Referral Mechanism and Duty to Notify statistics

Dear Jon 

National Referral Mechanism and Duty to Notify statistics 

We recently completed our compliance check of your National Referral Mechanism (NRM) and Duty to Notify statistics against the Code of Practice for Statistics. We carried out a complementary compliance check of the Office for National Statistics’ (ONS) Modern slavery in the UK and Child victims of modern slavery in the UK compendium articles, to help us understand the modern slavery data landscape and how the National Referral Mechanism (NRM) statistics fit into this landscape. 

Modern slavery is a term for all forms of human trafficking, slavery, servitude and forced labour. Measuring the prevalence of modern slavery is challenging because there is no single source or method that can accurately quantify the number of victims. The NRM currently provides the best available estimate of the number of potential victims in the UK. It is therefore a highly valuable data source on modern slavery. Potential victims that come to the attention of authorised ‘first responder’ organisations are referred to the NRM. NRM figures are known to be an undercount because many cases of modern slavery remain hidden or unreported.   

We found a range of positive features that demonstrate the value and quality of the statistics: 

  • Since taking over production of the NRM statistics from the National Crime Agency in 2019, Home Office has improved the presentation and accessibility of the statistics, for example, by adding commentary to the statistical bulletin and releasing the data tables in a reusable format. Information has been added on different aspects of the NRM process, such as the number of ‘conclusive grounds’ decisions, which adds insight on the proportion of referrals deemed by Home Office to be genuine cases of modern slavery. A user told us that they value the data on the Duty to Notify process, which was added to the bulletin in 2021. It is good that the statistics team has used internal and external peer review to drive improvements.  
  • The users we spoke to, and the letter we received from academic and sector users of modern slavery data, highlighted the importance of, and need for, more data on key areas of modern slavery, including immigration outcomes, criminal justice outcomes and support access. We therefore welcome the work your team is doing to link NRM data with immigration data. Data linkage is the only way to fill certain information gaps on modern slavery and immigration, such as the number of modern slavery victims that have applied for asylum, and the number of people coming to the UK via different routes. To maximise the value of the NRM data and generate further insight, we encourage your team to be ambitious in its data linkage plans, by exploring opportunities to link NRM data with a range of other data sources.  
  • The statistical bulletin is transparent about methods changes, such as the 2019 change to the recording of exploitation types. A caveat about the lack of comparability with previous data gives helpful advice to users. Making the caveat more prominent would further support appropriate use of the statistics.  
  • It is good that anonymised disaggregated NRM data are freely available to academic researchers through the UK Data Service. Home Office has also given some non-governmental organisations and members of the public access to the anonymised data. 

We also identified several ways in which the value and quality of the statistics could be enhanced: 

  • The insight of the statistical bulletin could be improved by adding a clearer narrative of victims’ journey through the NRM process. To help users visualise and better understand the NRM process, we encourage you to add a flow chart like the one in ONS’s Modern slavery in the UK article. While it is good that the bulletin discusses possible drivers of changes in the number NRM referrals over time, it should acknowledge that recent increases in the number of referrals may reflect changes in awareness and identification of potential victims by ‘first responder’ organisations, as argued in the ONS article. It should also explain why the NRM and Duty to Notify figures are likely to be undercounts, and that the true number of modern slavery victims may be higher.  
  • The statistics are not granular enough to meet some users’ needs. The users we spoke to said that they would welcome further breakdowns of the NRM referral data in the bulletin and data tables, including breakdowns by exploitation sub-type, age of victims, and location of exploitation. They also told us they would like to see outcome decisions broken down by UK and non-UK nationals. Expanding the datasets released via the UK Data Service would enhance their value too. We recommend that you engage directly with users to understand their needs regarding the release and presentation of the statistics. 
  • We heard about the challenges that academic users have faced in terms of permissions and access required for research involving the NRM. Difficulties with accessing modern slavery data were also highlighted by the former Independent Anti-Slavery Commissioner in her Annual Report 2021-2022. Improving data access for research would support better public understanding of modern slavery and allow greater scrutiny of the response to modern slavery. 
  • The bulletin contains limited information about the quality of the statistics. To assure users about the quality of data and data standards, and aid interpretation of the statistics, the quality and methods information should be significantly expanded. The information should cover the NRM live management information system, how data are recorded, the system’s strengths and limitations, and sources of bias in the data. It should discuss data accuracy and reliability, comparability and coherence, and other aspects of data quality. Given the variety of authorised ‘first responder’ organisations referring potential victims and sharing data, it should also explain how data are quality assured at all stages of the data collection and statistics production process. Our Quality Assurance of Administrative Data (QAAD) framework might be helpful for reviewing and reporting quality. 
  • The bulletin states that the statistics may differ from those in previous or future bulletins, due to updates to case files on the live management information system. The scale, nature and impact of revisions should be explained, to assure users about the consistency and comparability of the statistics.  
  • Modern slavery is a relatively data-rich area, as evidenced by ONS’s modern slavery in the UK compendium article. The NRM bulletin should signpost other data sources on modern slavery to help users easily find related statistics.  

Lastly, it is important that the integrity and credibility of the NRM statistics are not undermined. In a recent letter to Professor Jennifer Rubin, we highlighted the need for clarity and transparency around the evidence used to support public statements about modern slavery. We welcome the commitment in Professor Rubin’s response to ensuring that public statements are sourced from published statistics or other reliable evidence. 

Thank you to your team for its positive engagement during this review. We look forward to continuing to engage with your team. Please get in touch if you would like to discuss any aspects of this letter further or if we can offer further assistance as these statistics continue to develop. 

I am copying this letter to Tom Bucke, Head of Unit for Drugs, Exploitation and Abuse Analysis at Home Office; John Marais, Deputy Director Crime, Income and Wealth Division at ONS; and Meghan Elkin and Peter Jones, heads of the Centre for Crime and Justice at ONS. 

Yours sincerely 

Mark Pont 

Assessment Programme Lead 

 

Related Links:

Mark Pont to John Marais: Statistics on modern slavery in the UK (March 2023)

Maya Esslemont and Anna Powell-Smith to Ed Humpherson: Modern slavery data (October 2022)

Ed Humpherson to Jennifer Rubin: use of National Referral Mechanism statistics (December 2022)

Jennifer Rubin to Ed Humpherson: use of National Referral Mechanism statistics (January 2023)

Mark Pont to Stephanie Howarth: School Workforce Annual Census statistics

Dear Steph

School Workforce Annual Census statistics

We recently completed our Compliance Check against the Code of Practice for Statistics of the School Workforce Annual Census (SWAC) statistics. Our review primarily focused on the Value pillar and identified several strengths of the SWAC statistics as well as a number of ways in which we consider the quality and value of the statistics could be further enhanced:

  • The SWAC statistics provide valuable information on teachers and support staff in local authority maintained schools in Wales and are the only comprehensive source of information on teachers’ pay, meeting a vital policy need. We understand that you are planning on exploring the possibility of developing a teacher pay data series that is coherent with one or more of the other UK nations. We consider this to be a positive step that will enable users to build a cross-UK picture in this policy area and we look forward to the outcomes of this work.
  • The SWAC data are available in the Secure Anonymised Information Linkage (SAIL) Databank. This means that they can be linked to other data sources to provide valuable insights into the school workforce staff in Wales, greatly increasing the value of the statistics. One example of such work was the linking of the SWAC data to the shielded patient list (SPL) in 2020 which enabled analysis of the number of teachers and teaching assistants on the SPL by local authority as well as by school phase and medium. This analysis contributed to a strong evidence base for policy decisions relating to the reopening of schools during the COVID-19 pandemic. The Administrative Data Research (ADR) Wales Planned Programme of Work for 2022-2026 outlines several planned research areas using the SWAC data including: the role of Welsh language skills and usage in the teaching workforce, pay and progression differences in the teaching workforce, and the impact of school leadership on pupil outcomes. This work by ADR Wales will help inform policy, operational decisions and the legislative programme.
  • The team has a good understanding of the key users of the SWAC data and has engaged with them throughout the development of the statistics. We understand that you are currently working on extending your engagement beyond your key users. This will enable you to gather a wider range of views on the statistics which will help to maximise public value by ensuring that the statistics continue to meet the needs of users and potential users.
  • There are other similar data in this space, for example, statistics on the school workforce published by the Education Workforce Council (EWC). We encourage you to continue to investigate and explain any differences between the SWAC statistics and alternative data sources. We consider that a close working relationship with such organisations would benefit both the statistics and the users.
  • We welcome the use of technology and automated processes that are already in use in the collection and production of the SWAC statistics. We recommend that you consider incorporating principles of the Government Statistical Service’s Reproducible Analytical Pipelines (RAP) which will help you to ensure that your processes are of high quality, transparent and efficient.

I would like to thank you and your colleagues for their positive engagement on this review and hope the points outlined above are beneficial in the further development of the SWAC statistics. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Gareth Thomas, Head of School Workforce Statistics, Welsh Government.

Yours sincerely

Mark Pont
Assessment Programme Lead

Mark Pont to Emma Hickman: Housing, private rental, and housing purchase affordability statistics

Dear Emma

Housing, private rental, and housing purchase affordability statistics

We recently completed our compliance check of your housing affordability, private rental affordability and housing purchase affordability statistics against the Code of Practice for Statistics. These are important official statistics that present affordability ratios based on housing costs and incomes at various geographies. They are important to a range of users to monitor housing trends and inform policy and other decisions. Such uses include the design of schemes to support first-time buyers onto the property ladder; to inform local authority house building targets; and commercial decisions that require an understanding of the demand for different types of housing. The statistics should also provide valuable insights to users in the context of the current cost of living crisis. This letter presents our findings, including feedback to consider alongside the planned development of these statistics.

Our review found a range of positive features that demonstrate the trustworthiness, quality, and value of the statistics:

  • The headline housing affordability statistics are well-established and used for local planning and housing policy. The statistics are produced to high standards and are well-presented, accessible and supported by a range of insightful visualisations. The new private rental and housing purchase affordability outputs provide further valuable insights into different aspects of housing affordability. The team should be commended for its efforts in using household income data to overcome known limitations of the headline series.
  • The team has established an effective quality management approach. This approach includes, for example, collaboration with analysts in the devolved administrations to obtain their expertise which provides additional reassurance around data quality ahead of the statistics publication. The approach is documented extensively and accessibly in the Housing affordability and Additional measures of affordability Quality and Methodology Information (QMI) reports.
  • Extensive engagement with producer teams across the UK to gain buy in and expertise while developing new outputs. The engagement includes that with the Department for Levelling Up, Housing and Communities and the devolved administrations, facilitated though the Government Statistical Service (GSS) cross-government housing and planning working group. The engagement has drawn on cross-government work on the coherence of income data sources to determine the most appropriate sources for developing private rental and housing purchase outputs. This has ensured that consistent income data sources have been used to measure housing affordability in the different UK countries and regions, wherever possible.
  • The team has a range of plans to further enhance the quality and value of the statistics:
  • to enhance value by developing new mortgage affordability estimates from administrative data based on the proportion of people’s income spent on mortgage repayments, and estimates housing affordability for small geographical areas. This will enable a greater understanding of the extent of mortgage affordability than has been available to date, and on variations in affordability at lower levels of geography.
  • to expand the coverage and the data sources used for the forthcoming private rental affordability statistics to include estimates for Wales and Northern Ireland for the first time this year, and potentially for Scotland in 2023. This will help to better inform relevant policies relating to private rented sector provision in these countries.
  • to automate statistical production processes to enhance quality and efficiency, starting with private rental affordability. This will speed up the production process by three months, improving the statistics’ relevance and value, and enable policy and decision makers to understand the latest trends sooner. We published areview into the use of Reproducible Analytical Pipelines (RAP) principles and overcoming barriers, which may be useful to the team as it considers a more automated approach.

We also identified areas that may enhance the trustworthiness, quality, and value of these statistics:

  • With recent increases in mortgage interest rates there is growing interest in mortgage affordability. The team has identified a potential mortgage data source produced by the Financial Conduct Authority (FCA), but has not yet been able to gain access to the data. We support the team’s ambitions to access these data which may enable the production of robust measures of mortgage affordability. These statistics will be of significant value to policymakers and others looking to understand the extent of recent mortgage interest rates changes on the affordability of mortgages for homeowners across the UK.
  • References and links between the three affordability outputs should be clear enough to enable users to be aware of the related estimates and navigate easily between them. There is also potential to provide more insight as to how these statistics fit into ONS’s work on the cost of living, to provide a coherent overall picture. This will enable users to understand the relevance of these statistics in this broader context.
  • Summary information about the team’s planned developments for these statistics is included in the GSS cross-government Housing and planning statistics work programme and the QMI reports. Users can provide feedback via a generic email link. However, more-proactive engagement with users to seek their feedback on the most recent developments would help to maximise the public value and usefulness of these statistics. The team told us that it is considering writing a blog on its forthcoming development work.
  • We note that not all future planned releases are on the ONS release calendar, for example, planned outputs on mortgage affordability or housing affordability for small areas. The team should ensure that these releases are added to support trustworthiness; to provide users with as much advance notice as possible; and to promote the potential value of the new statistics for answering users’ key questions.
  • The descriptions of limitations of the statistics in the QMI documents are comprehensive and helpful. A range of uses for the statistics is provided in the QMI reports. However, the bulletins could be clearer on the exact and potential uses for each series, so that users know when to use one measure over another. While it’s good to highlight the main caveats in the bulletins, we note that sometimes the volume of caveats presented might be seen to play down the usefulness of the statistics. We encourage you to consider how to best balance openness about limitations with clarity around the statistics value for answering the key questions.
  • There is potential to enhance the value of the published quality information in relation to the administrative data used, for example by reviewing data suppliers’ assurance processes for the sources provided in relation to the team’s use of these data, and communicating this to users. This will demonstrate transparency and further enable users’ appropriate interpretation of statistical quality. You may find our Quality Assurance of Administrative Data (QAAD) framework helpful.

I would like to thank your team for their positive engagement on this review. My team will continue to engage with your team as it develops these statistics. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further. I am copying this letter to Rachel Skentelbery, Deputy Head of Profession for Statistics at ONS, and to Tony Wilkins, Tim Pateman and Nick Richardson, the responsible statisticians.

Yours sincerely

Mark Pont
Assessment Programme Lead

Ed Humpherson to Tony Dent: Better Statistics CIC

Dear Tony

Thank you for your letter of 28 October. I am pleased to hear about the positive engagement that Better Statistics has had with Savanta and the Centre for Data Ethics and Innovation (CDEI) on the CDEI tracker survey. There are four further issues you raise which I would like to address in this letter: the Office for National Statistics (ONS) COVID-19 Infection Survey (CIS) and our recent review of it; communicating uncertainty; our review of population estimates; and the Code of Practice for Statistics.

Before taking these issues in turn, I wanted to reassure you that, even where we do not agree, your questions, observations and challenges are useful for us – and we do in fact agree with you on several points as you will see below. However, I do feel that our engagement would be enhanced through face-to-face conversations. With that in mind, I would like to suggest that we meet to discuss how we work with you and your Better Statistics colleagues, and how we can engage in open discussions about the UK’s statistical system.

Office for National Statistics COVID-19 Infection Survey

The main purpose of our review of the CIS was to consider whether, and to what extent, the statistics continued to meet the standards of the Code of Practice for Statistics in light of plans to scale the survey back. Our review gathered a range of evidence, including documentation from ONS, interviews with ONS and feedback from users of the statistics. We concluded that the statistics from the CIS continue to be of sufficient quality and fit for their intended uses. This conclusion was based on a combination of factors, such as user needs, use of appropriate methods, thorough quality assurance processes, efforts taken to improve representativeness and the involvement of experts in various fields.

While the conclusion in this review remains our current position on the CIS, I take very seriously the concerns you raise both about the quality of the statistics and our review of them. As such, I am in the process of commissioning an external expert to provide an independent assessment on both issues. We will keep you informed about the independent assessment, and of course publish the outcome of this review in due course.

We have also continued to engage with ONS on the more specific points you raise regarding the CIS. I have provided an update on these below:

  • We consider that the Methods article published on 5 August gives a good account of methods and quality information for the CIS and provided a fuller explanation of some of the points that you had raised with us previously, such as whether household size is included in all analyses and the reason in some cases that it is not. ONS has informed us that an updated Methods article following the move to digital data collection will be published in the coming weeks.
  • ONS told us that it remains committed to publishing more information on response rates. ONS has been carrying out this analysis and hopes to publish it soon.
  • You raise a further point that ONS should consider alternative methods to the MRP model which may be able to better account for household size. The modelling approach taken by ONS is considered by many statisticians to be a state-of-the-art technique and appropriate for use in this context. While it’s possible that other models could have been used, we consider that this method is appropriate for the survey. However, this is an issue which we will ask an external expert to consider as part of the independent review.
  • Finally, with regards to value for money of the survey, you mention a request to investigate the clauses of the contract between ONS and its survey provider IQVIA. We are not aware of having received such a request. In any case, examining a contract would be beyond our remit and more relevant for financial auditors. Separately, we understand that although there was previously some confusion on ONS’s behalf about documentation which you had requested from IQVIA, this will be shared with you imminently.

Communicating uncertainty

I agree with you that it is essential that uncertainty around statistics is communicated well to support appropriate use and interpretation. I also agree that the fortnightly ONS business survey (BICS) is an example of statistical reporting which does not adequately convey the uncertainty around the estimates presented. My team has raised this with ONS and asked it to make improvements to the publication at the soonest opportunity. We will review these statistics as part of a planned compliance check in the last quarter of the 2022/23 financial year. More broadly, we continue to support and challenge the statistical system to improve how it communicates uncertainty, as we follow up our recent report on this topic.

OSR’s review of population estimates

The purpose of our review was to consider whether ONS’s estimates and projections could bear the weight that is put on them, reviewing the methods and data that underpin these statistics in line with the Code. The review was broad in its nature, as was our intended audience with such a range of users of these data. As part of our review, we spoke with various stakeholders including other government departments, representatives from several different local areas, and expert demographers and population specialists in academia. While our intention was not to focus only on issues experienced by users in areas such as Coventry, we did consider user perspectives during our review. Indeed, one of the conclusions of our review was that ONS did not adequately consider the concerns raised by some users. As you suggest, this may be an issue where we will need to agree to disagree.

Code of Practice for Statistics

As set out in my letter to you of 28 April, we do not make judgments on value for money of government spending. Nevertheless, we consider that the Code sufficiently addresses key aspects of the effective use of data and other resources in relation to statistical practice.

However, we always welcome challenge to our work and to potential improvements to the Code, and we like your suggestion of a more detailed discussion about the Code. To that end, we propose that we should convene an open discussion with stakeholders, including Better Statistics, next year to examine the purpose of the Code, how it helps and where it could help more.

If you would like to discuss any of these issues, please let me know and my team will be happy to arrange a meeting.

Yours sincerely

Ed Humpherson
Director General for Regulation

Related links

Ed Humpherson to Darren Morgan: Business Enterprise Research and Development (BERD) statistics

Dear Darren

Business Enterprise Research and Development (BERD) statistics

Thank you for your letter of 4 November setting out your development plans for UK Business Enterprise Research and Development (BERD) statistics. These plans align with the Code of Practice for Statistics, particularly relating to innovation and improvement.

ONS has been considering for some time the opportunities from linking its BERD data to HMRC’s Research and Development Tax Credit statistics as a means of improving the estimates. As statistics regulator, we first proposed that ONS investigate the potential of data linkage between ONS and HMRC in June 2012, as a requirement in our assessment report of these statistics. In our Compliance Check in August 2020 we again raised the issue, highlighting the need for ONS to publish information explaining to users the differences between the ONS and HMRC data on research and development in the UK.

Now that we can see the impacts of the joint endeavour between ONS and HMRC, it is clear that there are considerable benefits from collaborative working and data linkage for better statistics.

Linking to HMRC’s data has also allowed ONS to recognise that very small businesses have not previously featured in the BERD population but have actually been playing a much more prominent role in research and development than previously thought. The data linkage has also allowed ONS to be confident in the uplift factors that it has applied to deal with the under-coverage of mainly small businesses in the latest BERD survey. All in all, the ability to triangulate the BERD statistics with HMRC’s administrative tax data has given robust reassurance to ONS around the latest estimates of BERD spending.

I welcome your commitment to make further improvements to the methodology behind these statistics. While the interim method for producing BERD estimates yields much higher figures than those published in previous years they are far closer to the levels of BERD that other measures, especially HMRC’s R&D Tax Credit statistics, have indicated. The work that ONS and HMRC have conducted together has provided sufficient reassurance that the new method has produced the better estimates of total UK BERD. We therefore agree with you that total UK BERD statistics being published on 22 November 2022 should retain National Statistics status.

I recognise that there is greater uncertainty around the more-detailed breakdowns of BERD and you are prudently restricting the publication of data in those tables. I agree that the current National Statistics of such detailed sub-national and sectoral statistics should be suspended temporarily until after the second stage of your development plans are completed in 2023. We will add information to our webpages about the temporary suspension following this letter.

In the methodology article published at the end of September 2022, ONS is commendably open around the uncertainty in the latest estimates. Users and stakeholders in the statistics may have residual questions about the quality of the total aggregate level of UK BERD. It is important that you continue to be clear in communications of the newest statistics and data concerning your judgements around strengths and limitations of the latest estimates. You should particularly think about the advice you signpost to users of historical UK BERD time series around the uncertainty they should bear in mind when using that data series.

I look forward to engaging further with you and your team as you carry out these developments and around the eventual reconfirmation of National Statistics status of disaggregated BERD data when there is more certainty around those low-level statistics.

Yours sincerely

Ed Humpherson
Director General for Regulation

Mark Pont to Ian Lonsdale: United Kingdom Food Security Report

Dear Ian 

United Kingdom Food Security Report

We recently completed our compliance check of your United Kingdom Food Security Report against the Code of Practice for Statistics. This will be a valuable source for the UK Government in building upon and monitoring progress towards the aims set out in the Government food strategy. With the recent increases to cost of living generally there will be increasing focus on food security.  

This is the first government statistical publication focussed on food security in over a decade and is a comprehensive and generally well-structured report that gives a good overview of all aspects of food security. Given the likely increasing public interest in food security, and that it is a potentially sensitive issue, means it is all the more important to have independently produced official statistics that are developed by analysts across government. Our review found a range of positive features demonstrating the trustworthiness, quality, and value of the statistics as they are currently published. 

  • The breadth of content means it should address the needs of a large number of users whose interests might focus on different aspects of food security. Splitting the report into five themes improves the accessibility by allowing users to more easily find the sections relevant to their needs. 
  • In relation to this, the consistent structure and section headings used throughout the report for each indicator and within each theme further supports the accessibility of the report and aids users in finding the information they seek. 
  • Listing all staff members and external academics who were involved in the publication and having chapters written by the most relevant government department highlights the collaboration involved in producing the report as well as demonstrating trustworthiness by showing the professional capabilities employed in the production.  

In addition to identifying these strengths, we also identified ways in which the value of the statistics could be enhanced. 

  • While there is a consistent approach used for structuring of indicators throughout the report, there are some gaps in referencing sources for figures external to the report. Some indicators provide sources for all figures used but others make reference to figures with no source provided. A full approach to referencing will enhance trustworthiness and add value by allowing users to investigate further as required. 
  • Given current concerns about cost of living the report’s production was timely. However, there have already been major changes in the UK food security situation since the report was published. Consequently, it will be important to be responsive to these developments and transparent with users regarding how and when the report will be updated. To ensure the continued value of this report it is essential that users know what the future plans are, in relation to the data included as well as the report itself. 
  • While we acknowledge that there were time constraints in producing the first publication to meet the publication date set out in legislation it is unfortunate that wider user engagement was not carried out. As part of the future plans for this publication we would expect wider user engagement to be factored in. This wider user engagement could inform the best way to provide updated information whether this is via updating sections of the reports, producing interim publications or directing users to other publications and also whether more spatially granular data is needed. 
  • While the comprehensiveness of the report will be helpful to many, it is of a length that might put off less-expert users. Given the importance of the subject and the effort invested in producing it, to maximise value it would be worth exploring the possibility of producing a shorter more accessible summary. 

I would like to thank you and your team for their positive engagement on this review and hope the above helps in the development of this publication. Our Transport, Environment and Climate Change Lead will continue to engage with you and your team on progress in the coming year. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further. 

Yours sincerely

Mark Pont
Assessment Programme Lead  

Mark Pont to Alastair McAlpine: Planning performance statistics

Dear Alastair

Planning Performance Statistics for Scotland

We recently completed our compliance check of your Planning Performance Statistics against our Code of Practice for Statistics. These are important Official Statistics providing insights into the performance of local authorities (LAs) in making planning decisions across Scotland. In addition to our findings, this letter also provides feedback for you and your team to consider alongside the planned developments for these statistics.

Our review found a range of positive features that demonstrate the trustworthiness, quality, and value of the statistics:

  • The outputs have been streamlined to meet accessibility requirements, allowing more users with different needs to access them. We heard from your team that feedback from planning policy colleagues and LAs to ensure their continued relevance helped guide the changes.
  • Insight has been added through the addition to the statistical release of a section providing context to the Scottish Government’s planning policy, to help users understand the need for reporting on planning performance. Further clarity has also been added to the statistical release around the statutory time targets for determining local and national development applications where a processing agreement is not in place.
  • In line with our expectations around trustworthiness, the team is proactive in reviewing its pre-release access list, keeping it relevant to only those who need early access to the statistics, recently reducing the list from 25 recipients to 20.
  • Following our previous compliance check of your Scottish Vacant and Derelict Land Survey (SVDLS) statistics, we welcome that the team has reviewed and refreshed the previously outdated information on users and uses of the statistics, for both the SVDLS and the Planning Performance Statistics.

Your team told us of its plans for the further development of the statistical outputs to enhance their value and insight, including:

  • Further refining the statistical bulletin over the forthcoming publications, informed by user need, to ensure it focuses on key topics and messages, avoids any unnecessary repetition, presents information in formats that are helpful to users and further enhances accessibility.
  • Proposed engagement with users of planning statistics later this year to better understand the audience these statistics have. Using a sub-set of harmonised user engagement questions which were developed through collaboration between the GSS Coherence team and colleagues in the GSS-cross government planning statistics sub-group, should enable some cross-UK analysis of user views and allow a degree of comparability when looking at key user issues for each country, and should go some way to provide coherence across the planning statistics landscape.
  • Aims to introduce Reproducible Analytical Pipelines (RAP) into the statistical production process to make it more efficient, in line with what is happening across other Scottish Government statistics teams. We have published a review into the use of RAP principles and overcoming barriers which may be useful to your team as you consider a more automated approach
  • Adding further insight through working with LAs who supply planning performance data, to understand the range of circumstances where LAs apply the ‘stop the clock’ procedure and therefore explain within the statistical release differences in where ‘stop the clock’ is used. The team has agreed to also include more explanation of the planning of National Developments, which do not form part of these statistics, to enable users to access this information more easily.
  • The team told us that it is feeding into some of the Digital Planning in Scotland developments to oversee that any proposals meet the needs, or enhance the collection, of the PPS data. We recommend that the team communicate any forthcoming changes to users at the earliest opportunity and set out its views on its level of assurance around data quality, potential impacts on data quality or methods related to the changes, and how these will be mitigated. The planning activity statistics produced by the Department for Infrastructure (DfI) in Northern Ireland have recently gone through a similar process and we suggest contacting them to learn from their experiences.
  • Plans to further enhance the value of the recently published quality assurance information by using our Quality Assurance of Administrative Data (QAAD) framework – for example, by setting out the quality assurance that the team undertakes, reviewing its assurance processes for the source data supplied, and communicating its assurances around data quality. This will demonstrate transparency and enable users’ appropriate interpretation of statistical quality.

To maximise the benefits from these developments we recommend sharing your plans with users including the key findings from your planned user engagement events and explaining how these will shape the development of the statistics.

I would like to thank your team for their positive engagement on this review. My team will continue to engage with them as the developments set out above are implemented. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further. I am copying this letter to Adam Krawczyk, Head of Housing, Homelessness and Regeneration Analysis at Scottish Government, and Charles Brown and Karren Friel, the responsible statisticians.

Yours sincerely

Mark Pont
Assessment Programme Lead