Dear all
Household estimates and projections for Wales, Scotland and Northern Ireland
I am writing to you following our recent compliance check against the Code of Practice for Statistics of the household estimates and projections published by the Welsh Government, National Records of Scotland (NRS), and Northern Ireland Statistics and Research Agency (NISRA). As part of this work, we also explored with your teams the main themes highlighted in another recent review of ours, on the population statistics produced by the Office for National Statistics (ONS), to which ONS has recently responded.
I am pleased to confirm that these statistics for the three nations should continue to be designated as National Statistics.
Household estimates and projections are a key source used to inform local housing need and are therefore of high user interest. In reviewing the household estimates and projections, we found several examples of good practice which support the trustworthiness, quality, and value of the statistics. Further detail is included separately for each country in an annex to this letter. Some specific examples that we feel are important aspects that help realise cross-UK benefits include:
- A planned transparent public commitment by the statistics teams in all four UK nations to coordinate the timing of the next sets of household projections (and subnational population projections) using results from the 2021 England and Wales, and Northern Ireland Censuses, and using a common 2021 base year, where possible.
- The existing strong communication links between statisticians and housing need analytical teams in Wales and Scotland. Within Wales this has led in particular to timely additional housing need analyses and the publication of a comprehensive housing need publication for Wales, which directly reflects the considerable uncertainty and potential variants in the estimation of future housing need, as reflected in the variant household projection official statistics.
- Sensitivity analyses carried out by NISRA and NRS to enhance understanding of the quality and robustness of its projections. We also welcome NISRA’s proactive collection and analysis of students’ home and university addresses as part of the 2021 Census.
- Direct local engagement with Welsh local authorities as part of subnational projections quality assurance and regular fora for Scottish local authorities to contribute their views on data quality and developments, to ensure projections reflect local knowledge. We also welcome that while NISRA is commissioned to produce household projections specifically to inform housing need analyses, further steps are taken to ensure the projections have value for other users.
We have also identified some key areas for improvement across the producer teams, in order to strengthen the quality and public value of these statistics across the UK:
- The need for enhanced collaboration between the UK producers to share and use the latest data, methods, and improvements at the subnational level. While there are well-established arrangements for collaboration around national projections and Censuses, collaboration at the subnational household projections level is much less established. Strengthening opportunities for working together at this level will enable further sharing of knowledge that would enhance the value of the projections for users, particularly in the context of coordinated UK household projections outputs in 2023.
- Investigate potential data quality issues in local-level data, particularly in university towns and cities with large student populations, which could lead to misleading population estimates and poor local planning or service provision decisions. We encourage the producer teams to work together with ONS to inform each other’s understanding of source data quality, and address any identified local data quality issues, ahead of future household estimates and projections being produced.
- The use of fan-charts means that more-intelligent conversations can be had by decision makers about possible future trends. Producers should enhance presentation of uncertainty in household estimates and projections, including at a local level. For example, exploring options for interactive variants projections, fan-charts or other visuals or interactive content, to help support the appropriate interpretation of data and uncertainty, across different levels of geography.
- Enhance relationships between statisticians and housing need analysts and provide appropriate guidance to ensure that the relative strengths, limitations, degree of uncertainty, and range of household projections variants, are suitably accounted for in the housing needs analyses, while encouraging their publication in line with the principles of the Code of Practice for Statistics.
- Collaborate to publish a current overview of the methods used to produce estimates of households and dwellings in the four UK countries. This will help to improve coherence and accessibility by creating a single access point for users to understand the different nations’ approaches.
I would like to thank your teams for their positive engagement during this review. While the next round of household projections is some way off, we recommend that you initiate collaborative cross-team discussions around these findings, and to explore opportunities to share best practice sooner rather than later. We also recommend that you proactively seek out opportunities to engage users about your forthcoming timetables and developments, so that they are aware and have opportunities to contribute their views.
I am copying this letter to Martin Parry (Welsh Government), Sandy Taylor (NRS), Jonathan Harvey and Dr Jos IJpelaar (NISRA), the lead statisticians. We will also be writing to ONS and MHCLG to highlight these findings.
Yours sincerely
Mark Pont
Assessment Programme Lead
Annex: Summary of compliance check findings for the household estimates and projections produced by the Welsh Government, National Records of Scotland (NRS), and the Northern Ireland Statistics and Research Agency (NISRA)
The Welsh Government’s household estimates and projections are well-presented statistics with helpful commentary and guidance. A selection of additional variant projections is also published to help illustrate the range of different projections that might be expected in different migration scenarios. The bulletins contain useful information to help communicate uncertainty and ensure their appropriate interpretation, for example, although household projections are produced for a 25-year period, only the first 10 years are reported on.
Comparisons with previous household estimates and projections are also helpfully provided to highlight the extent of revisions over time. The general summary of quality and methods in the bulletins is both accessible and informative. However, the more detailed supporting quality and technical reports have not been updated since 2016 and 2017. These need updating to reflect more-recent assessments of administrative data quality, and technical and methods changes. More signposting and linking between the different subnational projections outputs would help to enhance accessibility and make navigation easier for users.
The statisticians have a broad range of communication channels to inform users about developments and promote the appropriate interpretation and use of the statistics. The team has been particularly strong on the transparent communication of decisions around changes, such as to amend methods and the timing of previous publications in response to quality issues with the source data provided by ONS. The team communicates its plans and decisions well to users through various channels, though any summary information of users’ views on the projections or developments is not very accessible.
The team also works closely with analytical research colleagues on the additional housing need, and provides advice on how the projections should be interpreted and reflected in housing need analyses, including the presentation of variant housing need projections. The statisticians also run quality assurance workshops with local authorities, which are able to provide challenge on the subnational projections before their publication and helping to account for local knowledge. The statisticians have identified what they believe may be potential issues with source data quality in some local areas containing universities, which they have committed to explore further with ONS. The Welsh Government is also looking for suitable alternatives to the POPGROUP software that it uses to produce its projections, as it will soon no longer be supported. We recommend that the Welsh Government collaborates with other producers, to share alternative ideas and best practice for producing projections, in line with established statistical practice.
NRS’s household estimates and projections releases include a concise summary of the key findings and a mixture of charts and visualisations. The main messages are clearly and objectively set out, and make good use of charts and visualisations, with the methods and sources information clearly set out. There are clear channels for engagement with a wide range of users and stakeholders through various groups and forums, allowing for users’ views to be fed in, and enabling quality assurance of the data with the councils and the local government finance team.
There is good communication with the Centre for Housing Market Analysis, which produces guidance for local councils for determining local housing need and demand, of which the household projections are a key input. The releases clearly highlight to users the limitations of the projections, for example, that projections become more uncertain the greater the time elapsed since the previous Census, and how they should and should not be used. The public value of the household estimates has been enhanced by adding further insights, and drawing on other sources such as Scotland’s Census, the Scottish Household Survey, Scottish Housing statistics and the Scottish Index of Multiple Deprivation.
The team has also shown innovation by introducing interactive data visualisations; making use of Reproducible Analytical Pipelines (RAP) for quality assurance of the household estimates; moving more if its coding from SAS to R; and plans to introduce more automation.
NRS should also review and update its published information on the quality assurance of the projections and estimates and make it easier for users to find, and to demonstrate the teams’ assurances around source data quality for users. More generally, we suggest that NRS identify an appropriate channel to communicate its road map for its next household estimates and projections, and how these will be coordinated with those for other UK nations given Scotland’s Census will occur a year later.
The team told us that it has developed a tool to run sensitivity analyses on the household projections to model the impact of changes to the underlying assumptions used. NRS believes this could be developed for public use. Making these supplementary tools available for wider use and analysis with appropriate guidance, would help to communicate uncertainty and inform how the analyses should or should not be interpreted. This would help support users’ needs and increase the value of the statistics. We recommend that NRS discusses these plans with other producers, so that they might also consider new interactive methods of communicating variation and uncertainty in their future household projection outputs.
NISRA’s household projections for areas within Northern Ireland are comprehensive and accessible. As NISRA’s household projections are commissioned, they do not follow a regular schedule of release as those produced by other countries do. Also, unlike other UK countries, NISRA does not produce household estimates. While the primary use of the household projections is to inform housing need analyses in the commissioning department, the statisticians undertake additional engagement to further ensure the projections also have value for other users.
NISRA’s supporting material on methods and data quality provides a comprehensive account of the quality of data that feeds into the household projections publication. NISRA produces a detailed account of the quality assurance undertaken on the administrative data that feed into the population estimates and household projections, including the strengths and weaknesses of the sources. Clearer signposting to this information from the household projections bulletin would be helpful.
NISRA statisticians are aware of the keen interest in local-level estimates by local authorities and updated Census estimates will bring some clarity – NISRA’s work on students’ home and university addresses will help to assure the quality of these statistics.
NISRA also carries out sensitivity analyses to enhance understanding of the quality and robustness of its projections. We recommend that NISRA shares information about the sensitivity analyses that it carries out with other UK producers, so that they might benefit from its approach. Unlike other UK producers, NISRA does not currently publish any variant household projections. Given that variants can be useful for communicating uncertainty in projections, we recommend that, as part of its engagement around commissioning the next round of household projections, NISRA explores with the relevant commissioning bodies, and users more generally, the options and extent of interest there may be for variant household projections, such as the options outlined above for example using fan-charts or other visuals or interactive content. We recommend collaborating with other nations in this process.
The statisticians also made us aware of their plans to improve the accessibility of household projections, utilising developments which they will first implement for their mid-year estimates. These include a move towards html and content that is more interactive, using R-coding shared with them by NRS, and the potential to realise benefits from advances associated with the dissemination of 2021 Census outputs. We encourage NISRA to share details of its plans with other producers too, and seek feedback from users, so that they are developed in ways that best meet users’ needs.