Mark Pont to Sandra Tudor: Land Use Change statistics and Land Use in England

Dear Sandra

Land Use Change statistics and Land Use in England

As you are aware, we recently completed our review of the compliance of the Land Use Change statistics against the Code of Practice for Statistics. I am pleased to confirm that these should continue to be designated as National Statistics. As part of this work, we also reviewed your official statistics on Land Use in England. We found several positive examples in the way that both sets of statistics are produced and presented by your team:

  • The statistics are clearly presented with the main points easily digestible. The team makes good use of tables, charts and maps to improve the clarity and ease of data interpretation.
  • The underlying data sets and sources are explained, and background information on the methodology and supporting quality assurance statement are published alongside the statistics to assist users in fully understanding the data. We welcome the team’s plans to publish an updated version of the methodology and quality assurance guidance to reflect the latest developments, including proposed changes to the team’s approach to measuring hectarage.
  • Demonstrating transparency through publishing a flow chart outlining the production and quality assurance processes, and publishing details of your assurances around the quality of the data.
  • The team has taken an innovative approach to increasing usability and adding to the user experience by using Power BI to produce an interactive report on Land Use and introducing topic factsheets. We welcome plans to explore the use of social media to broaden your reach to users and promote the statistics.
  • The positive relationship the team has with its data supplier – Ordnance Survey – which means any data queries identified during quality assurance can be dealt with effectively and your team is able to draw on Ordnance Survey’s technical expertise.
  • Plans to add further value to the Land Use statistics by publishing more detailed breakdowns of land use categories in response to user need.

Our review also identified several ways in which we consider that you could further enhance the trustworthiness, quality and value of the statistics:

  • In order to support transparency and ensure that all users are informed about delays to Land Use Change statistics due to delays gaining access to the latest Ordnance Survey data, publicly inform users through the appropriate channels, including the Land Use Change statistics webpage, the reasons for the delay and plans for forthcoming releases as soon as possible. Use these channels to communicate planned developments in advance and for users to feed in their views on proposals.
  • As definitions of land use have, and can, still be developed further, it is important to communicate clearly to users where updated or improved definitions used by Ordnance Survey and your team have led to changes in the statistics or their interpretation over time, and draw users’ attention to any potential discontinuities within the statistics and data tables.
  • Some of the commentary within the releases could be improved to:
    • add further insight to give a broader picture across topical aspects of land use where possible, by enhancing the statistical commentary to include material of relevant policies or related statistics on relevant topics (for example, Green Belt statistics or the National Planning Policy Framework).
    • aid users’ interpretation and understanding of the flow of land use between the different land use categories, for example by explaining the Sankey diagram used within the Land Use Change release more clearly.
    • communicate definitional differences in the term ‘developed land’ used in Land Use Change 2017-18 which is based on 2011 Census geographies, and that used in Land Use stock 2018-19, to avoid any confusion between the two definitions.
  • Your team explained that due to the highly visual nature of the releases there are several constraints in place preventing the publication of the statistics in the more accessible html format compared to pdf. We suggest speaking with the Government Statistical Service Good Best Practice team as it may be able to offer some advice.
  • Draw users’ attention to details of the Central Local Information Partnership (CLIP) meetings held on the Knowledge Hub website and consider publishing a summary of the relevant minutes and actions from the CLIP Planning Statistics Subgroup that your team attends, to demonstrate transparency about your approach to engaging with users and to help foster wider user engagement.

We appreciate the clear commitment shown to the continued development of these statistics through the public user consultation on proposed changes to the Land Use Change statistics and proposed additional statistics on Land Use stock. We suggest the team consider with users how greater insights could be drawn from using the two sets of statistics together, including exploring whether bringing them together into a single publication would add value. The team shared with us some of its future plans for getting the new releases peer reviewed within the wider directorate and with subject experts such as the Office for National Statistics geography team, as well as engaging with users about the new products. We welcome the news that work is progressing well with regards to procuring the new data from Ordnance Survey and particularly commend the team’s willingness to share knowledge and expertise with the other UK countries should they wish to develop their own land use statistics.

Your team told us that it would like Land Use statistics to be considered for a future National Statistics Assessment, which could include a potential reassessment of the Land Use Change statistics. Undertaking the steps outlined in this letter will go some way in preparing the statistics for this. We welcome the plans from your team to continue engaging with our Housing, Planning and Local Services domain as the statistics are developed over the coming months.

Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further. I am copying this letter to Andrew Presland and Alexander Reynolds, the responsible statisticians, and Richard Field, head of housing and planning statistics at MHCLG.

Yours sincerely

 

Mark Pont

Assessment Programme Lead

Mark Pont to Sandra Tudor: Compliance Check of the English Housing Survey

Dear Sandra

English Housing Survey statistics including Estimates of leasehold dwellings As you are aware, we recently completed our review of the compliance of the English Housing Survey (EHS) statistics against the Code of Practice for Statistics. I am pleased to confirm that EHS statistics should continue to be designated as National Statistics. As part of this work, we also reviewed your experimental official statistics on the number of leasehold dwellings in England, as the EHS is the main data source for these estimates, and have offered some suggested developments in the annex to this letter.

We initiated this review following the public commitment we made in our 2020/21 Regulatory Work Programme to focus on statistics about key issues within Housing. We appreciate the positive and constructive way that the team engaged with us during the review, especially as we continue through these challenging times. We found a range of positive features that demonstrate the trustworthiness, quality, and value of the statistics:

  • the EHS statistics are comprehensive, accessible and professionally compiled. The EHS headline report presents an authoritative account of the main trends in household numbers, tenure and housing conditions, supported by visuals to aid interpretation (for example, the movers by tenure diagram). The separate topic reports and factsheets add additional insight on a range of themes
  • extensive technical and quality information is clearly and accessibly set out. This includes an assessment of the EHS against the European Statistical System quality dimensions, standard errors and confidence intervals published routinely alongside the results, and a technical report setting the overall methods approach in detail
  • quickly moving the EHS data collection to telephone interviewing this year and adapting the physical survey to take place outside of the property, with plans to draw on Energy Performance Certificate (EPC) and Valuation Office Agency (VOA) data to inform the adapted physical survey approach
  • introducing an innovative follow-up panel survey this year to explore how the circumstances of 5000 households have changed since the start of the pandemic
  • plans to improve the timeliness of EHS headline results by one-month from this year onwards by publishing in December rather than January, including the panel survey wave one results this year
  • regular engagement with the survey contractors – NatCen, the Building Research Establishment (BRE) and CADS Housing Surveys – through the year, and with policy users to inform decisions around EHS topic modules and thematic reports
  • developing experimental leasehold dwellings statistics to fill a longstanding data gap of policy and user interest. Further details are included in the annex to this letter · EHS data being a long-established source for wider reuse via the UK Data Service
  • plans to consider improvements to the accessibility of some EHS outputs where feasible, such as publishing in html and using tools, such as Power BI.

We identified some areas where the public value of the statistics could be improved, in order to continue to meet the high standards required of National Statistics:

  • To support transparency regarding planned developments of EHS statistics, including innovations to overcome the data collection challenges posed by COVID-19, publish details about future development plans for the EHS statistics and the team’s overall approach to user engagement, so that users are aware of planned changes in advance and are clear about the available channels to feed in their views
  • Adding further insight and value to the statistical commentary where possible, for example by including wider context such as referring to relevant policies and drawing on related third-party data sources (for example, additional insights from MHCLG Council Taxbase statistics on second homes)
  • The ability to compare housing conditions across the UK nations continues to be of interest for users. It is great the BRE has produced, on behalf of the four UK nations, “A snapshot of housing conditions throughout the UK”. It would be helpful to provide more-prominent links to this work to assist such users
  • Reduce the number of individuals granted pre-release access, wherever possible
  • Although the EHS outputs are primarily based on survey results, where additional administrative data are used (for example, EPC data), we expect producers to have their own assurance of the quality of each source and its suitability for their use individually, building on the assurances provided by the data suppliers. Producers should do this by engaging with data suppliers in a way that is proportionate to the materiality of each source in the production of the final statistics. Our administrative data quality assurance guidance may be useful to refer to when doing this.

The suspension of all face-to-face surveys due to COVID-19 has created challenges for the EHS data collection. We discussed this in detail with your statisticians and heard of their innovative work and plans in this context, such as moving to panel, telephone and online data collection, and trailing the use of an app for respondents to video record evidence inside the property for the physical survey. We welcome that EHS statisticians share knowledge of these planned developments with other UK and Republic of Ireland statisticians through the Five Nations House Conditions Surveys group. Given the uncertainty and changing nature of events, we welcome that the team has agreed to keep in contact with us as their plans progress.

Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further. I am copying this letter to Reannan Rottier, EHS Lead Statistician.

Yours sincerely

Mark Pont

Assessment Programme Lead

 

Annex 1: Suggested developments for leasehold dwellings statistics

The published experimental leasehold dwellings statistics are a welcome innovation, and were developed in 2014-15 to fill a longstanding data gap of policy and user interest. The leasehold dwellings statistics can be used to inform the development and monitoring housing policies at a regional, and national level, including monitoring the impact of regulating the leasehold market. We welcome the improved timeliness of the 2018-19 leasehold dwelling statistics and support the team’s ambitions to publish them as part of the main EHS outputs in 2021. Our review has identified some areas that we consider it would be good for MHCLG to consider as part of its own developments, to enhance the trustworthiness, quality and value of the leasehold dwellings statistics:

  • Develop the statistical commentary to highlight the key questions that the statistics are able to answer, for example in relation to pertinent debates, relevant government policies, initiatives or targets
  • Better insight could be drawn from the statistics by describing the various approaches that have been taken to estimating the number of leasehold dwellings, and their strengths and weaknesses. For example:
    • explain more prominently in the bulletin the difficulties in accurately counting the number of leasehold dwellings, building on material already included on how the EHS leasehold measure alone warranted a new approach
    • given the slightly confusing landscape of leasehold situations, explain more clearly the differences between sets of estimates such as:
      • why numbers for social sector leasehold dwellings are low at 234 thousand compared to 4.2 million the private sector
      • why social and private sector estimates are not comparable (i.e. that leasehold properties in the social sector, the local authority or housing association is the leaseholder, not the social tenant).
    • Update the linked 2014 Residential leasehold dwellings in England Technical paper, to provide clarity around some aspects of methods and quality, including:
      • the statisticians’ assurances around the suitability of the individual administrative sources used. A low level of data quality concern is reported for the statistics overall, but the rationale for this rating is not clear and a limited assessment of assurance for the individual administrative sources is provided. Assurance should be informed by an understanding of the data collection operational context and quality assurance for each source, and the nature of the arrangements and relationships held with third-party data suppliers. Our administrative data quality assurance guidance may be useful to refer to when doing this
      • explanations around the statistics evaluation in reference to additional sources available triangulate the results or enhance insights (for example, the Regulator for Social Housing is due to publish new statistics on the number of units held by private registered providers on a leasehold and freehold basis in 2021)
      • details on the new calibration methods and the rationale for their use over the previous imputation approach
      • the strengths and limitations of the EHS sample design for measuring the extent of the leasehold population, given the skewed distribution of leasehold properties (with around two thirds of leasehold dwellings accounted for by flats in London and houses in the North West)
      • details of the matching methods used and the characteristics of unmatched cases
      • confidence intervals for lower-level estimate breakdowns to provide greater clarity around potential uncertainty in

Mark Pont to Sandra Tudor: English Indices of Deprivation 2019 statistics

Dear Sandra

ENGLISH INDICES OF DEPRIVATION 2019 STATISTICS

I am writing to you following our review of the Ministry of Housing, Communities and Local Government’s (MHCLG) English Indices of Deprivation 2019 (IoD) statistics against the Code of Practice for Statistics. The statistics were reviewed against the three pillars of the Code: trustworthiness, quality and value.

IoD statistics are an important tool for identifying the most disadvantaged areas and for supporting decisions about addressing local needs. They are widely used by central and local government and community organisations to target their services. The statistics have been considered as part of a wider review of the indices of deprivation statistics in Great Britain, alongside our compliance checks of the statistics produced by the Scottish Government and the Welsh Government.

I am pleased to confirm that these statistics should continue to be designated as National Statistics. We found several positive examples in the way that MHCLG produces and presents these statistics:

  • The suite of products has been tailored to meet a wide range of uses and users. The statistical bulletin is detailed but easy to follow. The technical report is comprehensive and sets out the process for constructing the index of multiple deprivation in a clear and accessible way. The FAQs document complements the technical report by bringing out the key messages for less-technical users. The statistics team told us these products have reduced the level of requests they receive as users are now able to address their own queries.
  • The team is committed to developing the IoD and has identified opportunities to use the statistics to support MHCLG policies. For example, several policies have come into action around Levelling Up[1] and Stronger Towns[2], which will benefit from regional analysis linked to deprivation. More recently, the team has responded positively to the challenges of the pandemic by providing guidance and assurance of outputs produced by the Office for National Statistics to allow it to explore the relationship between deprivation and COVID-19.
  • MHCLG has worked with the University of Sheffield and MySociety to develop a new mapping tool which allows users to visualise the statistics at different geographical scales including new geographies that the statistics have not been presented by before – Westminster Parliamentary Constituencies and Travel to Work areas. These new breakdowns are designed to meet the user need emerging from the MHCLG polices highlighted above.
  • The team spoke highly of its relationship with the devolved nations. The ‘four nations group’ meets regularly and works collaboratively to make guidance and presentation across the deprivation statistics more consistent. For IoD 2019, MHCLG extended its coverage for income and employment data to cover Wales which has enabled new comparable analysis for these two domains. The analysis has been carried out in partnership with the Welsh Government.

We have identified several areas where we consider that improvements could further enhance the public value of the statistics:

  • The ability to combine and compare indices of deprivation across the devolved nations continues to be an area of interest for some users. Each of the producers we spoke to said they deal with queries relating to this on a regular basis, despite their joint effort to set out in the statistical releases how the statistics can and can’t be used. MHCLG, as part of the ‘four nations group’, should look to ensure that appropriate resource is devoted to developing updated UK-wide guidance and insight.
  • The construction of IoD is carried out by contractors and the team told us that it faced delays in 2019 as a result of needing to establish data sharing agreements with some data suppliers. To build resilience in the team, MHCLG should look to document the process of how access to the data was granted for the contractors so that this can be referred to in the next iteration of IoD.
  • The statistical release contains some analysis of areas that have been ‘persistently most deprived’ across historic iterations of the IoD. To improve harmonisation and consistency across the nations’ indices of deprivation, MHCLG could look to discuss and agree a common definition of ‘deep-rooted deprivation’, in line with the Welsh and Scottish Governments.

Our Labour Market and Welfare team will continue to engage with you and your team in the coming months to follow up on areas that have been highlighted for improvement. We would like to thank the team for its engagement and cooperation throughout the review process.

I am copying this letter to Bowie Penney, the responsible analyst.

 

Yours sincerely

Mark Pont

Assessment Programme Lead

[1] The Levelling Up agenda is aimed at tackling regional inequalities and was announced following the December 2019 general election.

[2] The Stronger Towns Fund was announced in March 2019 and is targeted at towns to create new jobs, help train local people and boost growth.

 

Related Links

Mark Pont to Siobhan Carey: Northern Ireland Multiple Deprivation Measure

Mark Pont to Roger Halliday: Scottish Index of Multiple Deprivation 2020 statistics

Mark Pont to Glyn Jones: Welsh Index of Multiple Deprivation 2019 statistics

Mark Pont to Sandra Tudor: Compliance Check of Rough sleeping snapshot in England

Dear Sandra

STATISTICS ON ROUGH SLEEPING SNAPSHOT IN ENGLAND

We have recently conducted our review of the compliance of Ministry of Housing, Communities and Local Government’s (MHCLG) Rough Sleeping Snapshot official statistics against the Code of Practice for Statistics

While these statistics are not National Statistics, they are important official statistics valued by users and so we have made some recommendations to support their continued development. We considered the Trustworthiness, Quality and Value of these statistics in relation to the Code and appreciate the positive and constructive way that the team has engaged with us during our review, especially at this particularly challenging time.

We welcome the news that new management information is being collected on the numbers of homeless and rough sleepers in England being assisted into emergency accommodation since the start of the COVID-19 pandemic. We would like to recognise the positive steps being taken by MHCLG statisticians working with other MHCLG officials to determine how this management information can be best used to complement the existing statistics. We welcome the recent release of an early cut of this management information which support transparency and provides a valuable insight into nature of the current homeless landscape in England.

Within the Rough Sleeping Snapshot, we found a range of positive features that demonstrate the quality and value of the statistics:

  • Developing the statistics by inviting an independent review by the Government Statistical Service’s (GSS) Good Practice Team and peer review by other GSS colleagues;
  • Introducing a range of valuable innovations this year, including a Reproducible Analytical Pipeline approach, a new accessible html publication on GOV.UK; a new accessible infographic; regional analysis through a new Power BI dashboard, and new insightful commentary informed by local authority (LA) contextual feedback;
  • Changing the name of the bulletin to provide clarity about the ‘snapshot’ method used, and active engagement with national media in advance of the publication of the figures, to help ensure their appropriate interpretation;
  • Working with Homeless Link to strengthen LA guidance around when it is appropriate to change the rough sleeping measurement approach used between years, and producing a new video explaining Homeless Link’s role in overseeing and verifying LA estimates;
  • Publication of a new technical report with accessible information about the methods and quality assurance approaches employed, an analysis of the impact of changes in LAs’ measurement approaches back to 2010, and comparisons with other administrative sources in London;
  • Plans to develop the demographic information collected to better understand the length of time and frequency that individuals sleep rough, and to use UK harmonised age categories;
  • Plans to include new insights from other sources such as figures on rough sleepers in emergency accommodation due to COVID-19 as part of the narrative this autumn, to support emerging policy questions, and help establish whether rough sleeping has ended by the end of the current parliament in 2024.

We identified some areas for improvement that would enhance the value, quality and trustworthiness of the rough sleeping statistics:

  • Enhance the value of the statistics through engagement with a broader range of users and transparently set out plans for the further development of the statistics, so that users have clear opportunities to feed in their views about how the statistics could be further improved;
  • Draw on a range of published statistics, management information and other research on rough sleepers, to produce new analytical insights to more fully convey the lived experience of rough sleepers in England;
  • Enhance quality, by presenting the information included in the published infographic about how the statistics can and can’t be used, more upfront in the statistical bulletin;
  • Review the team’s QAAD data quality risk rating of ‘low’ due to the large number of LA data suppliers and partners representing an increased risk to data quality, and differences between some London LA’s snapshot submissions and their Combined Homelessness and Information Network (CHAIN) data, and consider how CHAIN and other data sources, while having their own limitations, might be used to improve the verification of LA submissions;
  • Support trustworthiness, by exploring the need for data suppression to protect rough sleepers from being identified, in light of plans to collect additional demographic data;
  • Specify the number of special advisers granted pre-release access to the figures and the provisional month of the forthcoming publication on the GOV.UK release calendar;
  • Establish a new process for the timely publication of ad-hoc data requests for rough sleeping analyses and management information used in public statements.

Thank you for engaging effectively with us during this review. We welcome the updates on your continued collaborative work as part of the Cross-Government Homelessness Statistics Working Group. With management information now being collected across the UK on the numbers of rough sleepers and homeless helped into emergency accommodation since the start of the pandemic, alongside other administrative sources, we look forward to seeing GSS statisticians work together to more fully illustrate the complexity of the overall UK rough sleeping and homelessness picture.

Our Housing, Planning and Local Services Domain Lead will continue to engage with your team on progress in the coming months and we would welcome a progress update from you upon the next publication of these statistics.

Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Jon White and Anthony Ash, the Responsible Analysts and to Ricky Taylor, Team Leader.

Yours sincerely

Mark Pont

Assessment Programme Lead

Response from Ed Humpherson to Sandra Tudor: Temporary suspension of National Statistics designation for Local authority revenue expenditure and financing

Dear Sandra

TEMPORARY SUSPENSION OF NATIONAL STATISTICS DESIGNATION: LOCAL AUTHORITY REVENUE EXPENDITURE AND FINANCING, ENGLAND: 2020 TO 2021 BUDGET

Thank you for your letter of 26 June, regarding your annual local authority revenue expenditure and financing, England: 2020 to 2021 budget statistics. Given the data inconsistencies identified by your statisticians for this year’s statistics and the associated challenges that they will have to quality assure the figures due to COVID-19, I support your decision to temporarily suspend the National Statistics designation.

I am pleased that your statisticians have engaged with key users of the statistics about their plans to publish a reduced range of statistics for this year which will include additional guidance and caveats to help ensure their appropriate interpretation.

We will continue to engage with your statisticians, via our Housing, Planning and Local Services team, and look forward to reinstating the designation once you feel assured that local authorities are once again reporting their budget information on a comparable basis.

Yours sincerely

Ed Humpherson

Director General for Regulation

 

Related links:

Letter from Sandra Tudor

Sandra Tudor, MHCLG to Ed Humpherson: Temporary suspension of National Statistics designation for Local authority revenue expenditure and financing

Dear Ed,

TEMPORARY SUSPENSION OF NATIONAL STATISTICS DESIGNATION FOR LOCAL AUTHORITY REVENUE EXPENDITURE AND FINANCING, ENGLAND: 2020 TO 2021 BUDGET

I am writing to ask you to consider a temporary suspension of the National Statistics badging for the Ministry of Housing, Communities and Local Government’s statistics on Local authority revenue expenditure and financing, England: 2020 to 2021 budget.

The Department collects and publishes revenue account budget data from local authorities on an annual basis. Local authorities are required by the Secretary of State to complete and return these forms under section 139A of the 1988 Local Government Act (as inserted by paragraph 68 of Schedule 5 to the Local Government and Housing Act 1989). Budgets are set ahead of the financial year and usually approved in February.

This year forms were sent out 14th February via the Department’s online data collection platform, with returns due back 10th April. In response to the Coronavirus outbreak, we wrote to local authorities on 6th April, extending the deadline by 2 weeks and adding a question to the form asking whether figures included the first tranche of COVID-19 grant that was allocated to authorities at the end of March.

The vast majority of authorities have returned their original budgets, without making any adjustment as a result of COVID-19. However some authorities were not content to send out of date budgets and have made adjustments to varying degrees. For example, some have only included additional grants, whilst others have considered potential impacts on income and reserve levels. This presents a particular challenge when compiling aggregate England figures. Additionally whatever steps we take, we know these budget figures are likely to differ greatly from actual revenue expenditure over the year, even for those authorities who have made some adjustments.

Our priority is to set out the data as clearly as possible and with all appropriate caveats. For this reason we intend to remove some of the tables and commentary which are less meaningful with the differences in reporting. This also allows us to focus on adding more detail to the tables we do include and to be as helpful as possible to our key users. These are the Office for National Statics, HM Treasury and Office for Budget Responsibility, who use these as starting points for national totals for forecasting and reporting. We have consulted them on the most useful approach for this year’s publication.

These statistics are currently designated as National Statistics, but it is my assessment that, given these challenges, the statistics cannot be subject to the normal level of quality assurance. Having consulted your published COVID-19 guidance on the suspension of National Statistics status, I feel the most appropriate course of action would be to seek temporary de-designation for the 2020 to 2021 statistics.

I am confident that these issues are specific to this year only and will be resolved going forward and I anticipate writing a further letter asking for the temporary suspension of the National Statistics badging to end ahead of next year’s release.

Yours sincerely,

Sandra Tudor,

Head of Profession for Statistics, MHCLG

 

Related links:

Response from Ed Humpherson to Sandra Tudor

Rough Sleeping Initiative evaluation

Dear Sandra,

Further to my previous correspondence in July I am writing following the publication of the Ministry for Housing, Communities and Local Government’s (MHCLG) Rough Sleeping Initiative (RSI) 2018 impact evaluation.

I welcome transparent presentation of the evaluation’s methodology and its findings, which indicate that the RSI has had a statistically significant impact on reducing rough sleeping between 2017 and 2018. It is clear that the analysts involved have done a lot of good work to rigorously evaluate the RSI and their efforts to undertake this should be commended.

I would also like to thank MHCLG analysts for bringing the evaluation to the attention of my team and providing them with an overview of the findings. My team have identified some additional analyses that might be explored further. These include examining the possible effects of local authorities’ changing their estimation approaches on the rough sleeping levels reported in earlier years and to explore relative, rather than absolute changes between RSI and comparator authority areas.

I recognise that the accurate estimation of rough sleeping in England is challenging and that while the snap-shot methodology is limited, it represents the most comprehensive approach currently available in England. Nevertheless, where further improvements can be made this is desirable. I therefore welcome Homeless Link’s enhanced guidance for local authorities this year to help further ensure that the estimation approaches chosen in 2019 are those that will most accurately measure rough sleeping, regardless of other non-official approaches being used in RSI areas to collate RSI management information throughout the year.

I would also like to take the opportunity to recognise MHCLG’s contribution to the GSS’s recent work on UK homelessness. This is important work providing valuable insights into the UK statistical picture on homelessness. I also look forward to MHCLG’s separate RSI process evaluation later this year which will explore what has worked and why from the perspective of RSI local authorities. I encourage you to consider where findings from this work might be used to enhance the value of the annual rough sleeping statistics, the broader GSS work and how you might demonstrate that the principles of the Code of Practice can also apply to such analyses.

In relation to the publication of further MHCLG analyses, we are aware that the way MHCLG summarised the RSI 2018 impact evaluation in some of its communications was not completely consistent with the underlying work. I encourage you to consider how you might reaffirm the importance of analysts having final sign off on such communications within the department to help ensure that confidence in important evidence isn’t undermined once published.

Yours sincerely

Ed Humpherson
Director General for Regulation

 

Related Links:

Ed Humpherson to Sandra Tudor (July 2019)

Sandra Tudor to Ed Humpherson (May 2019)

Comparability of rough sleeping statistics

Dear Sandra,

Thank you for your letter dated 20 May in response to our concerns about the use of the Ministry for Housing, Communities and Local Government’s (MHCLG) most recent rough sleeping figures in public debate. I am writing to set out our thinking on the apparent reductions in the number of rough sleepers in areas receiving Rough Sleeping Initiative (RSI) funding.

I appreciate the inherent difficulties in the accurate counting of rough sleepers in England and the limitations of the snap-shot methodology. These were of central concern to the users we spoke with during our 2015 assessment. Users also highlighted that the statistics were insufficient for assessing policies designed to reduce rough sleeping at the time.

I welcome your update that MHCLG plans to improve its broader rough sleeping evidence base and local authorities year-round recording of rough sleepers. I hope that this leads to a broader statistical view of rough sleeping in England beyond the snap-shot count alone, in the same way that MHCLG statisticians are currently working to provide a fuller statistical picture of homelessness more generally.

My team recently met with you to discuss some of our concerns about the recording of rough sleepers in RSI areas – the 83 areas with the largest numbers of rough sleepers in England. While we recognise that the methodology allows local authorities to decide whether a ‘count’ or an ‘estimate’ is the best collection approach in their areas, between 2017 and 2018, 19 RSI areas changed their data collection approach from an estimates approach to a count. One local authority changed their approach from a count to an estimate. We are concerned that these changes may be material for the comparability of the statistical series.

I therefore welcome your plans to publish an evaluation of the Rough Sleeping Initiative this summer. In order to establish the comparability and overall accuracy of the statistics, it seems particularly important that the evaluation can determine the relative differences between the levels and trends recorded by the counts and estimates approaches. Clarity here is essential to ensure that these statistics can be considered a reliable source for monitoring rough sleeping policy. The inclusion of many new additional RSI areas in this autumn’s rough sleeping count only increases the urgency with which this clarity is needed.

Until the evaluation confirms otherwise, I am not confident that changes in the number of rough sleepers are reliable. They may in part be a result of changes in measurement approach. In the light of this concern, I understand that you have improved the ministerial guidance to avoid too much weight being placed on these statistics before the evaluation is published, and I welcome this guidance. I also appreciate your continued professional scepticism, as well as your openness to review the snap-shot methodology should the evaluation reveal significant limitations for the counts/estimates approach for effectively evaluating the Rough Sleeping Initiative.

Yours sincerely

Ed Humpherson
Director General for Regulation

 

Related Links:

Sandra Tudor to Ed Humpherson (May 2019)

Sir David Norgrove to Lord Bourne of Aberystwyth (April 2019)

Sir David Norgrove to John Healey MP (March 2019)

John Healey MP to Sir David Norgrove (February 2019)

MHCLG response to letters on rough sleeping

Dear Ed,

I am writing to you following the recent letter from Sir David Norgrove, Chair, UK Statistics Authority to Lord Bourne about the use of statistics on the impact of the Rough Sleeping Initiative[1], and also Sir David’s recent reply to John Healey MP, Shadow Secretary of State for Housing who expressed concern about the quality of the rough sleeping statistics[2]. In these letters, Sir David raised a number of matters including improvements to the rough sleeping statistics, consideration of using a CHAIN (Combined Homelessness and Information Network) approach beyond London, the impact of local authorities switching from an estimate to a count (or vice versa) and conclusions about the impact of the Rough Sleeping Initiative drawn from the Rough Sleeping statistics. We are also aware of your letter to Clive Betts MP, chair of the Housing, Communities and Local Government Select Committee this week. This letter reiterates the UKSA’s concerns in the light of the Committee’s recent evidence session.

By way of response to the recent letters from Sir David, I would like to set out the improvements we have made to the annual Rough Sleeping statistics following the UKSA Assessment in 2015 and also our plans for improving the evidence base on rough sleeping, which is broader than the annual statistics.

Accurately counting or estimating the number of people sleeping rough within a local authority area is inherently difficult given the fluctuating nature of rough sleeping, with individuals regularly moving into and out of rough sleeping, and its hidden nature. The latest statistical release[3] makes clear there are a range of factors that have an impact on the number of people seen or estimated to be sleeping rough on any given night. The methodology we use has been in place since 2010 when it was developed after consultation with local authorities and the voluntary sector with the objective of strengthening the accuracy of the figures. The annual snapshot has never claimed to be a total figure of all those sleeping rough in the country and is a pragmatic approach that provides a representation of those sleeping rough, alongside a reliable and consistent measure of change.  The methodology used in England aligns with the approach which is now standard in many parts of the world including Canada, the United States and a number of other European countries including France, Ireland, Italy and Spain.

All counts and estimates are independently verified by Homeless Link who actively challenge areas in situations where the veracity of the count or estimate is questionable. They oversee the whole process and issue clear guidance on how to conduct a count and where it may be appropriate to move from a count to an estimate, or vice versa.  Hundreds of local partners and voluntary sector organisations actively engage in the count and producing the estimate which also gives us confidence in the reliability of the statistics.

Since the UKSA Assessment of the Homelessness and Rough Sleeping Statistics in 2015, we have addressed the range of requirements set out in the report, as detailed in Annex A and summarised in my letter[4] to you of 26 June 2018. We have expanded the statistical release to include demographic information about young people, gender and nationality, as well as providing a time series table. The release has much clearer information about the methodological issues which affect the data and the role played by the lead statistician in their production. We held a user event in November 2016 supported by the UKSA.  Further information about all the latest developments in both homelessness and rough sleeping can be found on the homelessness statistics user forum page[5].  Whilst we have a programme of work underway to improve the broader evidence base for homelessness and rough sleeping, we are not planning any major changes to the single night snapshot methodology but continue to encourage users to provide feedback on how these statistics are used and how well they meet their needs. We remain confident in the comparability of the time series to provide a way of assessing the extent of rough sleeping across local areas on a single night and measuring change. It has always been the case that each area choses the most appropriate approach to assess the extent of rough sleeping in their local area each year.

We routinely meet with Homeless Link to review the count and estimate process so that we can be confident that these statistics are a consistent measurement of change.  We will keep users updated on any new developments via the user forum and in future statistical publications.

As recommended, we have included CHAIN (Combined Homelessness and Information Network) statistics within the Rough Sleeping Statistics Release, but, at this stage, have concluded that it would not be practical to extend a CHAIN multi-agency database approach beyond London and those other cities which have started to adopt similar approaches. CHAIN is a multi-agency database which records information about people seen rough sleeping by outreach teams in London across the year whereas the single night snapshot provides a way of estimating the number of people sleeping rough across all local authorities in England on a single night and is a way of assessing change over time. Although, CHAIN offers more information on the flows and characteristics of people rough sleeping, data quality is influenced by the number of outreach workers active on any one night. Thus a local authority which does not have a commissioned outreach team may appear to have fewer rough sleepers than an adjacent borough which does have an active outreach service. MHCLG is actively encouraging local authorities with significant numbers of rough sleepers to improve their year-round data on rough sleeping and is working collaboratively with local authorities to develop improved homelessness data and outcomes that can measure progress in reducing rough sleeping and homelessness.  We have also funded a significant number of additional outreach posts through the RSI which increases an area’s capacity and understanding of rough sleeping and rough sleepers in their area.

Our broader programme of work will improve the evidence available for developing, implementing and monitoring policies in this area. Alongside our recent overhaul of homelessness statistics, introducing case level collections with far greater details, we are also establishing local data pilots which will seek to improve the data collected by a wide range of services which people who sleep rough access. These data pilots will not provide or contribute to a national measure of rough sleeping, but they will help us to develop and test a multi-agency outcomes framework, looking across a range of services, including key health services.

The data pilots, our programme of evaluations and new research studies as detailed in Annex B, are all part of an ongoing effort to ensure that our interventions are evidenced based and that we build up an evidence base on ‘what works’ to reduce rough sleeping. This was set out in our Rough Sleeping Strategy and we made a commitment that “…over the next nine years, we will plug evidence gaps and pilot, test and evaluate new approaches which will inform our plans for wider roll-out”. It is important to note that this relates to the broader range of evidence in this area, rather than the Rough Sleeping Statistics specifically – while Sir David’s reply to John Healey MP, conflates the two.

The latest published annual rough sleeping statistics included specific reference to the areas supported by the government’s Rough Sleeping Initiative (RSI). We will be publishing an evaluation of the initiative in the summer which will help us to better understand the impact of the initiative.

The evaluation is in two parts: an impact evaluation and a process evaluation.  The impact evaluation, led by MHCLG analysts with independent peer review, will look at the impact of the range of activities in RSI areas on the overall numbers of people sleeping rough. This will include consideration of the impact of the change in choice of approach for assessing the extent of rough sleeping, as well as other factors such as the weather, levels of funding and, the types of activity within the areas who are part of the initiative. The process evaluation, carried out by Ipsos MORI will share learning about how the Rough Sleeping Initiative has worked, good practice and key challenges, including findings from a survey of initiative areas and case studies.  We are intending to publish these findings in the summer and would welcome a meeting with the UKSA to discuss the findings and provide further advice, once this work is complete.

The UKSA have highlighted concerns about an ‘apparent methodology shift’ impacting the annual rough sleeping statistics.  As stated above, the impact evaluation will provide information on this.  However, we provide additional information in the annual statistical release on areas that have changed their approach to assessing the extent of rough sleeping in their local area this year compared to previous years. This is an established part of the methodology – local authorities, not central government, decide in conjunction with their local partners and based on advice from Homeless Link (who verify the rough sleeping statistics collection) to use the approach that they believe will return the most accurate figure to assess the extent of people sleeping rough in their local area. This decision is not linked to any requirement under RSI funding. There are a number of reasons for areas to change from an estimate to a count year to year as set out in guidance provided by Homeless Link. This includes:

  • A significant change in the number, population or location of people sleeping rough where sites are visible
  • High numbers of people sleeping rough in the area with no ongoing data collection e.g. no/limited outreach
  • An increase or fluctuations in numbers of people sleeping rough where sleep sites are accessible/visible
  • Difficulties forming an estimate on the basis of the information available
  • Significant disagreement about the numbers between agencies

Overall, 74 areas used a count this year compared to 52 last year, and 252 used an estimate compared to 274 in 2017.  In RSI funded areas, 46 areas used a count this year compared to 28 last year, and 37 used an estimate compared to 55 in 2017. Over three quarters of RSI areas used the same approach as last year. Given the number of new posts which have been funded through the RSI, there is also greater capacity to conduct street counts, which may explain some of the increase in areas doing street counts this year.  As outlined above, in our view this does not affect the comparability of the time series as the methodology has always been that local areas decide the approach to use to assess the extent of rough sleeping in their local area to provide the most accurate figure. However, the RSI Impact evaluation will consider this issue in more detail.

We are in regular contact with your officials about both the homelessness and the rough sleeping statistics and we will discuss further once the RSI evaluations have been published.  We will also be in contact to discuss reviewing the assessment of the homelessness and rough sleeping statistics following the overhaul of the homelessness statistics.  As outlined in my letter of 26 June 2018 we expect it to take time for the new collection arrangements to become established and data quality to be assured and as yet we are just a few months on from our initial publication of experimental statistics in December 2018.

Yours sincerely,

Sandra Tudor
Head of Profession for Statistics, MHCLG

 

[1] UKSA letter to Lord Bourne, 18 April 2019

[2] UKSA letter to John Healey, 25 March 2019

[3] Rough sleeping in England: autumn 2018

[4] Letter from Sandra Tudor, Head of Profession for Statistics, MHCLG to Ed Humpherson, Director General for Regulation, Office for Statistics Regulation, 26 June 2018 

[5] MHCLG homelessness statistics user forum page

 

Related Links:

Sir David Norgrove to Lord Bourne (April 2019)

Sir David Norgrove to Rt Hon John Healey MP (March 2019)