1. Overview
Background
To inform our judgements around the suitability and quality assurance of the data and methods used in the Dynamic Population Model, we commissioned an independent review from Professor Arkadiusz Wiśniowski, University of Manchester.
1.1 The Office for National Statistics (ONS) has traditionally produced its population estimates for England and Wales primarily using information from a survey it carries out every 10 years – the census. However, to improve statistics on population to better reflect changes in society and technology and meet user needs, ONS has developed Admin-Based Population Estimates (ABPEs) for England and Wales using a ground-breaking and innovative new method – the Dynamic Population Model (DPM).
1.2 The DPM uses a Bayesian statistical model to produce a coherent estimate of population counts (stock) and changes (flows) using births, deaths and migration data. The work that ONS is doing to improve estimates of the population in England and Wales is at the forefront of harnessing technological advancements for statistics production. The results will influence how population data are compiled and used in the production of official statistics and wider research across the UK.
Aims and approach of our assessment
1.3 ONS requested this assessment to reflect new developments as part of UK Government’s broader ambition to move away from undertaking a census every 10 years and make more use of administrative data. The production of the ABPEs using the DPM represents a significant development in the way that the population of England and Wales will be estimated. In this report, we set out our findings and requirements to support ONS in its continued development of the ABPEs as it works towards achieving accredited official statistics status for these statistics.
1.4 Our phased assessment reflects the scale and ambition of the development of these statistics and the wider transformation context. It will consider whether the statistics meet the professional standards set out in the statutory Code of Practice for Statistics. This phase of our assessment, the first, aims to provide reassurance to users on the new approach and therefore focused on three areas:
a) the extent to which ONS is benchmarking the state of readiness of population estimates to replace the cohort component method currently used.
b) the suitability and quality assurance of the data and methods used in the DPM to produce population estimates for England and Wales.
c) an initial investigation into user understanding of, and confidence in, the proposed new method, and ONS’s communications on the method.
1.5 We conducted a desk-based review of the statistical publications, associated methodology papers and relevant documentation that ONS has shared. We also spoke with some key users, including demographers and local authorities, to gauge user confidence in, and wider understanding of, the new approach.
1.6 To help inform our judgements, specifically in relation to (b) above, we sought independent expertise from Arkadiusz Wiśniowski, Professor of social statistics and demography at the University of Manchester, who specialises in population estimation, data integration and Bayesian methods. A copy of Professor Wiśniowski’s report is available here.
Summary of findings and requirements
1.7 We support ONS’s ambition to make use of technological advances in developing innovative new methods for the production of population statistics to increase the accuracy of these statistics and better meet the needs of their users. As it improves its population estimation methods, ONS can also share its learning with other statistics producers, both more widely in the UK and internationally.
1.8 Population statistics not only provide insight about the size and composition of the population in a society, but also serve as the bedrock of many other important statistics that are used to underpin decisions affecting people’s lives. Maintaining statistical quality, user confidence and trust in its population statistics is crucial as ONS transitions to using the new method to produce ABPEs. Our review and findings are focused on three areas: the readiness of the ABPEs to replace the current mid-year estimates (MYEs), the suitability and quality assurance of the data and methods used in the DPM and user understanding of and confidence in the new approach.
1.9 Demonstrating the moving landscape and pace at which the ABPEs are being developed, ONS has already made several commitments and plans to improve these statistics, which are dependent on whether it can secure appropriate resourcing over the coming months. We support these commitments and so our requirements for the first phase of this assessment are focused on areas beyond these commitments.
1.10 We have identified eleven requirements for ONS to address to ensure that these statistics are on course to meet the high standards of public value, quality and trustworthiness associated with accredited official statistics status. These requirements cover areas such as governance, data quality, methods, revisions, user engagement, and communication.
Benchmarking the state of readiness of ABPEs to replace MYEs
1.11 ONS faces challenges in governance, decision-making and communication in the transition from the MYEs to the ABPEs. There are dependencies between the current MYEs and the ABPEs, particularly in the way that census data are used. To explore these dependencies further, ONS has presented the ABPEs in different ways, with varying reliance on census data. However, the evidence around the quality of the ABPEs is currently not conclusive, as the ABPEs are dependent on the input data used. It is essential that ONS understand and communicate these dependencies between the two estimates. In considering when the ABPEs will replace the MYEs, ONS needs to develop and publish criteria to support its decision; there is a significant risk that without defined criteria ONS’s decision- making will undermine user confidence.
1.12 The ONS team working to produce and develop these estimates is relatively small, and as ONS reorganises its team structure, there are challenges in terms of staff capability and financial position. As with any major transformation programme, it is important that appropriate senior oversight and governance is in place to manage change, support decision-making, and to identify and mitigate any risks.
1.13 Requirement 1:
To maintain public confidence in its population statistics, ONS needs to understand the current dependencies between the ABPEs and MYEs. Together with key stakeholders, such as the Welsh Government, ONS should also develop and publish criteria to support its decision about when the ABPEs will replace the MYEs. The criteria should include statistical quality, operational readiness, planned evaluation and assurance processes and contingency plans, and be usefully applied to the ABPEs and MYEs.
1.14 Requirement 2:
To ensure that there is sufficient oversight and leadership of the production of ABPEs in a way that is joined-up across ONS, and support the ongoing development of ABPEs, ONS should strengthen its governance structure. Work here should include establishing clearly defined decision-making responsibilities to manage any risks associated with funding, capability and prioritisation across the ABPEs production process.
Understanding the suitability and quality assurance of the data and methods
1.15 Given the complexity of the methods used in the DPM and in formulating our view against the Code, we commissioned an independent review from Professor Arkadiusz Wiśniowski which contains additional recommendations for ONS to address.
1.16 Requirement 3:
To improve and quality assure the methods used in the DPM in a way that supports public confidence in the ABPEs, ONS should publish a response by October 2024 detailing how it plans to address the recommendations and suggestions in Professor Wiśniowski’s report, and, in particular, the essential recommendations (R1-R7). Any recommendations that ONS decides not to take forward should be clearly explained within the response, setting out how it has considered the recommendation.
R1.
To provide a comprehensive and detailed methods guide that will ensure that the Dynamic Population Model (DPM) is reproducible. The guide should describe in detail:
a. data inputs,
b. modelling framework,
c. assumptions regarding population components,
d. computational methods,
e. model testing, and
f. analysis of the outputs.
The methods guide should contain versioning similar to the versioning of the Statistical Population Dataset (SPD).
R2.
To provide in the documentation (R1) a clear differentiation between bias and accuracy (or precision) of the data inputs and assess each data input in terms of bias and accuracy. The assessment should inform the DPM. Such a distinction is essential for the DPM to produce reliable (i.e. unbiased and accurate) population estimates.
R3.
To quantify in the documentation (R1) the assumptions in the model, e.g., for precision this could be done by providing coefficients of variation around the mean, rather than stating that one source is more precise than the other. The current version of the DPM relies on informative priors and such quantification is required as an input to the model. It will ensure that the various assumptions can be tested and their impact on ABPEs assessed.
R4.
To test and document the impact of using a coverage benchmark in the DPM (Option 1: correct in the data inputs, Option 2: Correct in the DPM via model parameters). The documentation should contain a description which option has been implemented.
R5.
To analyse the sensitivity of the ABPEs to a variety of prior distributions assumed for the accuracy (precision) of each of the data inputs. Special attention should be paid to precision of migration (currently internal, cross-border and international migration being jointly modelled as in- and out-flows to and from LAs). Sensitivity analysis should be carried out for the prior distributions for the coverage adjustment parameters. These analyses will inform if the ABPEs are robust to the assumptions about data quality and help identify extreme situations where the DPM may require further research.
R6.
To continue developing a quality assurance processes at each stage of producing ABPEs, i.e. starting with producing data inputs, assessment of their bias and accuracy, quantification in terms of data-corrections and/or model parameters, as well as robustness and sensitivity analyses of the DPM and ABPEs. This is to ensure the sustainability of the DPM if data inputs change or new sources are introduced in the future.
R7.
To provide a statement that accompanies the DPM-based ABPEs on the potential sources of uncertainty or bias that are unaccounted for and, where possible, an assessment of their importance in a given situation, e.g. when considering estimates for age groups or LAs.
1.17 The DPM is in development, and ONS is continuing to learn about the model’s capability, both in terms of its strengths and limitations. The model relies primarily on administrative data supplied by other government departments, with one of the benefits being that it can incorporate new data sources over time. This is dependent on overcoming practical challenges and continuing to build strong relationships with data suppliers. ONS’s Data Pipeline Maturity framework and strong governance helps ONS to manage any risks associated with its data supply.
1.18 Whilst in theory the DPM can flex and overcome data supply issues, understanding the quality and associated uncertainty of the data inputs is crucial to the model being able to produce unbiased population estimates. ONS should therefore prioritise further work to document and understand more about the sensitivity of the model and how the data inputs and technical assumptions in the DPM affect the quality of the ABPEs.
1.19 Requirement 4:
To maximise the capability of the DPM and the quality of the ABPEs, ONS should:
- address the practical implications of incorporating new data sources into the DPM over time and ensure it is appropriately resourced.
- continue with its plans to conduct sensitivity analysis to explore how the model’s performance is affected by the availability and quality of different data sources.
- review and test the capability of the DPM, at suitable intervals, to account for the integration of any novel and volatile/changeable data sources that are included in the model over time.
- implement regular fully audited assumption checking and validation to support reproducibility and to help keep the model sustainable.
1.20 The DPM uses the Statistical Population Dataset (SPD), which is based on linked administrative data and derived from the Demographic Index (DI), as its main stock measure (the count of the population on a given day). The SPD is subject to over-coverage and under-coverage, due to the nature of the data collected for administrative purposes. Therefore, methods are applied to adjust for coverage issues using census-based data. The SPD estimates, particularly at a granular level, vary when compared to Census-based 2021 estimates by different population group characteristics, therefore pointing to a need for ONS to understand those differences and any potential sources of bias that may exist in the SPD.
1.21 Requirement 5:
To maximise appropriate use of the ABPEs, and avoid inappropriate use of these statistics, ONS should:
- better understand the source of any bias in the SPD and introduce documented quality metrics for the DI that quantify errors (in particular, linkage errors) and any associated uncertainty that may propagate into the SPD and subsequently the DPM.
- publish information on the DI, including on how it is created, reviewed, updated and quality-assured.
- communicate and present, in a simple way, how the stock data (and other data inputs to the model) change over time, as this may affect the quality of the ABPEs and how it compares to that of other population estimates, such as the MYEs.
- ensure its published quality information includes explanation of any strengths and limitations, and reflects the latest data inputs used, for example, updating to the latest version of the SPD.
1.22 Quality assurance arrangements covering the end-to-end process are not yet well established across the production of ABPEs in ONS. A well-documented approach is necessary to support the reproducibility and user understanding of the ABPEs’ strengths and limitations.
1.23 Requirement 6:
To audit the ABPEs production process, understand the impact of data issues and support confidence in its approach, ONS needs to build on the principles set out in its published data quality strategy and implement an end-to-end process that will:
- fully audit and document the process and methods applied at each stage to support cross-production knowledge and capability, and ensure that mechanisms are in place for various teams to discuss, log and audit any decisions or fixes that take place.
- oversee and assess the quality of the data inputs separately and in stages. This should help ONS develop the quality assurance information published alongside the statistics and support users’ understanding of the strengths and limitations of the ABPEs.
- ensure compliance with Reproducible Analytical Pipelines (RAP) standards.
1.24 Revisions will form a routine part of the production of ABPEs, in particular as ONS seeks to produce more-frequent population estimates using the DPM. To manage user expectations, ONS is developing a revisions policy specific to the ABPEs.
1.25 Requirement 7:
To help users understand how to use the ABPEs, ONS should implement and publish a revisions policy, and as part of its development:
- carry out and publish a revisions analysis of the ABPEs to date, including how data input and methods differences may impact the scale of any revisions.
- clarify how the model will be able to take account of any changes in methods over time as part of producing an ABPEs back series.
- seek feedback and input from users and key stakeholders about its proposals and involve them in its decision-making.
1.26 ONS is at the forefront of developing a population estimation method for official statistics using a Bayesian statistical model. Given the innovative nature of this work, continued collaboration with Bayesian experts will be crucial for the ONS to develop the DPM.
1.27 Requirement 8:
To instil confidence in the ABPEs and ensure that the DPM methods are sound and subject to sufficient independent and external challenge, ONS should:
- continue with its plans to create a sub-group of its Methodological Assurance Review Panel (MARP; the independent panel used by ONS to provide advice and assurance on methods used to produce official statistics).
- create and implement an expert user group.
- make it easier for users to find relevant MARP papers to support technical user understanding of the methods used in the DPM.
User understanding and confidence in the proposed new method
1.28 Developing the ABPEs is a significant step towards improving population estimates, and users are confident that increased use of administrative data is the right approach. However, users have reported some concern about the differences between the ABPEs and MYEs, the accuracy of the new estimates and how the new method will work without a An engagement strategy, specific to the ABPEs, will help ONS to better understand user needs and use this feedback to drive developments. Transparent and open communication with users, particularly about its long-term plans, will help ONS manage user expectations, quality-assure the ABPEs, support public confidence and demonstrate that ONS is a trustworthy organisation that actively listens and responds to users’ views.
1.29 Requirement 9:
To maintain public confidence and help shape the future development of the ABPEs and manage user expectations, ONS should:
- develop and implement a user engagement strategy specific to the ABPEs. This strategy should detail specific activities and how users will be involved at various stages of the process. The approaches that have been implemented elsewhere in ONS, for example migration statistics, can serve as a good model for this.
- use feedback from users to drive developments to the ABPEs whilst also being transparent about where user needs cannot be met, (for example, the availability of breakdowns).
- publish regular updates on its plans for the ABPEs, including how the ABPEs form part of the wider population and migration statistics transformation, including timelines and any interdependencies.
1.30 Requirement 10:
To quality-assure the ABPEs at a local level, and strengthen its relationships with users, ONS should be open to scrutiny from key stakeholders, such as local authorities, and users and respond to any feedback appropriately.
1.31 Requirement 11:
To build trust in the new approach, ONS needs to improve the way that it communicates quality and methodology information and tailor its communication to the differing technical expertise of users of population statistics, including by:
- seeking feedback on its current published quality and methodology information with a broad range of users and working together with other stakeholders across the Analysis Function.
- helping users navigate to the various publications on the ONS website, for example by implementing a landing page.
Overall judgement
1.32 We acknowledge ONS’s level of ambition and the progress it has made, as well as the complexity of its work and the valuable staff expertise that has helped to shape and steer the work’s development to date. We commend the considerable efforts of ONS staff to deliver this ground-breaking work and support ONS’s overall aims, but there is significant work to be done before these statistics fully meet the standards of the Code of Practice for Statistics.
1.33 In our view, ONS should improve its governance and develop criteria to be able to benchmark the state of readiness of ABPEs to replace MYEs. More development work is needed to fully test the capabilities of the DPM, quantify the quality of data inputs going into the model and quality-assure the ABPEs to support appropriate use. And this should be done in collaboration with users and methods experts, with ONS encouraging external scrutiny and assurance through improved communication and engagement channels. ONS taking these steps will ensure that users have more confidence in the new method, and therefore the ABPEs.
1.34 ONS is working at pace and there is clear ambition to push the capability of the DPM and develop it further to meet more specific user needs (such as small area estimation). Whilst the DPM is an exciting new development for estimating the size of the population, we urge ONS to take time to address our requirements and embed strong foundations for further development to build on.
Next steps
1.35 We expect ONS to publish an action plan by October 2024, setting out how it intends to address the eleven requirements, and report back to us publicly every three months on its progress.
1.36 We will also consider the scope of the second phase of our assessment. We expect this to focus on the Value pillar of the Code of Practice for Statistics and to cover the coherence and comparability of population estimates across the UK.
Back to top