Ed Humpherson to Sir Ian Diamond: Assessment (phase one) of Admin Based Population Estimates for England and Wales

Dear Sir Ian,

Assessment (phase one): Office for National Statistics’ Admin-Based Population Estimates for England and Wales

Thank you for inviting us to assess the Office for National Statistics’ (ONS) Admin-Based Population Estimates (ABPEs) for England and Wales against the Code of Practice of Statistics. Today we have published the assessment report covering the first phase of our work, which is focused on the quality pillar of the Code. I would like to recognise and thank your teams for their positive engagement throughout the course of the assessment.

ABPEs have been developed using a ground-breaking and innovative new method, the Dynamic Population Model (DPM), and I support your ambition to improve methods and make use of technological advances in the production of population estimates. I acknowledge the progress made by ONS and the valuable staff expertise within your teams that has helped to shape and steer the work’s development.

The development and use of the DPM to produce population estimates in England and Wales is a significant milestone, which is likely to have an enduring influence on the way that population statistics are compiled in the future. At times of change and evolution of statistical methods, such as with the ABPEs, it is essential to be open and transparent in your approach to help users understand and accept the changes. The change in method will also need to demonstrate that it yields appropriate data quality. Therefore, our findings for this first phase of our assessment have focused on three areas: the readiness of the ABPEs to replace the current mid-year estimates; the suitability and quality assurance of the data and methods used in the DPM; and user understanding and confidence in the new approach.

We have identified eleven requirements (see Annex) for ONS to address that will help to enhance the public value, quality and trustworthiness of the ABPEs. These requirements cover areas such as governance, data quality, methods, revisions, user engagement, and communication.  ONS should improve its governance and develop criteria to be able to benchmark the state of readiness of ABPEs to replace MYEs. More development work is needed to fully test the capabilities of the DPM, quantify the quality of data inputs going into the model and quality-assure the ABPEs to support appropriate use. And this should be done in collaboration with users and methods experts, with ONS encouraging external scrutiny and assurance through improved communication and engagement channels. ONS taking these steps, building on the publication of ONS communications today, will ensure that users have more confidence in the new method, and therefore the ABPEs. We expect ONS to publish an action plan setting out how it intends to meet our requirements by October 2024 and publicly report back to us on progress every three months.

Professor Arkadiusz Wiśniowski, professor of social statistics and demography at the University of Manchester, has contributed to our assessment with an independent review of the data and methods used in the DPM. As part of its ongoing development of the DPM and in meeting the requirements in our assessment report, we expect ONS to address the essential recommendations in Professor Wiśniowski’s report and publish a plan that also considers his further recommendations and suggestions.

My team will continue to liaise with ONS to support progress and discuss the scope and timings of the second phase of our assessment in due course. I look forward to seeing the outcomes of the continued development of the ABPEs.

In conclusion, the team that is leading the APBE work is impressive, and the approach promises to deliver significant benefits for users. Our requirements will support the successful development of this crucial programme.

I am copying this letter to Mary Gregory, Director of Population Statistics at ONS, Rachel Skentelbery, Deputy Head of Profession for Statistics at ONS, Stephanie Howarth, Chief Statistician at Welsh Government and Professor Arkadiusz Wiśniowski at the University of Manchester.

Yours sincerely,

Ed Humpherson
Director General for Regulation

Annex: Requirements to enhance ONS’s Admin-Based Population Estimates (ABPEs)

Requirement 1: To maintain public confidence in its population statistics, ONS needs to understand the current dependencies between the ABPEs and MYEs. Together with key stakeholders, such as the Welsh Government, ONS should also develop and publish criteria to support its decision about when the ABPEs will replace the MYEs. The criteria should include statistical quality, operational readiness, planned evaluation and assurance processes and contingency plans, and be usefully applied to the ABPEs and MYEs.

Requirement 2: To ensure that there is sufficient oversight and leadership of the production of ABPEs in a way that is joined-up across ONS, and support the ongoing development of ABPEs, ONS should strengthen its governance structure. Work here should include establishing clearly defined decision-making responsibilities to manage any risks associated with funding, capability and prioritisation across the ABPEs production process.

Requirement 3: To improve and quality assure the methods used in the DPM in a way that supports public confidence in the ABPEs, ONS should publish a response by October 2024 detailing how it plans to address the recommendations and suggestions in Professor Wiśniowski’s report. In particular, the essential recommendations (R1-R7). Any recommendations that ONS decides not to take forward should be clearly explained within the response, setting out how it has considered the recommendation.

Requirement 4: To maximise the capability of the DPM and the quality of the ABPEs, ONS should:

address the practical implications of incorporating new data sources into the DPM over time and ensure it is appropriately resourced.
continue with its plans to conduct sensitivity analysis to explore how the model’s performance is affected by the availability and quality of different data sources.
review and test the capability of the DPM, at suitable intervals, to account for the integration of any novel and volatile/changeable data sources that are included in the model over time.
implement regular fully audited assumption checking and validation to support reproducibility and to help keep the model sustainable.

Requirement 5: To maximise appropriate use of the ABPEs, and avoid inappropriate use of these statistics, ONS should:

better understand the source of any bias in the SPD and introduce documented quality metrics for the DI that quantify errors (in particular, linkage errors) and any associated uncertainty that may propagate into the SPD and subsequently the DPM.
publish information on the DI, including on how it is created, reviewed, updated and quality-assured.
communicate and present, in a simple way, how the stock data (and other data inputs to the model) change over time, as this may affect the quality of the ABPEs and how it compares to that of other population estimates, such as the MYEs.
ensure its published quality information includes explanation of any strengths and limitations, and reflects the latest data inputs used, for example, updating to the latest version of the SPD.

Requirement 6: To audit the ABPEs production process, understand the impact of data issues and support confidence in its approach, ONS needs to build on the principles set out in its published data quality strategy and implement an end-to-end process that will:

fully audit and document the process and methods applied at each stage to support cross-production knowledge and capability, and ensure that mechanisms are in place for various teams to discuss, log and audit any decisions or fixes that take place.
oversee and assess the quality of the data inputs separately and in stages. This should help ONS develop the quality assurance information published alongside the statistics and support users’ understanding of the strengths and limitations of the ABPEs.
ensure compliance with Reproducible Analytical Pipelines (RAP) standards.

Requirement 7: To help users understand how to use the ABPEs, ONS should implement and publish a revisions policy, and as part of its development:

carry out and publish a revisions analysis of the ABPEs to date, including how data input and methods differences may impact the scale of any revisions.
clarify how the model will be able to take account of any changes in methods over time as part of producing an ABPEs back series.
seek feedback and input from users and key stakeholders about its proposals and involve them in its decision-making.

Requirement 8: To instil confidence in the ABPEs and ensure that the DPM methods are sound and subject to sufficient independent and external challenge, ONS should:

continue with its plans to create a sub-group of its Methodological Assurance Review Panel (MARP; the independent panel used by ONS to provide advice and assurance on methods used to produce official statistics).
create and implement an expert user group.
make it easier for users to find relevant MARP papers to support technical user understanding of the methods used in the DPM.

Requirement 9: To maintain public confidence and help shape the future development of the ABPEs and manage user expectations, ONS should:

develop and implement a user engagement strategy specific to the ABPEs. This strategy should detail specific activities and how users will be involved at various stages of the process. The approaches that have been implemented elsewhere in ONS, for example migration statistics, can serve as a good model for this.
use feedback from users to drive developments to the ABPEs whilst also being transparent about where user needs cannot be met, (for example, the availability of breakdowns).
publish regular updates on its plans for the ABPEs, including how the ABPEs form part of the wider population and migration statistics transformation, including timelines and any interdependencies.

Requirement 10: To quality-assure the ABPEs at a local level, and strengthen its relationships with users, ONS should be open to scrutiny from key stakeholders, such as local authorities, and users and respond to any feedback appropriately.

Requirement 11: To build trust in the new approach, ONS needs to improve the way that it communicates quality and methodology information and tailor its communication to the differing technical expertise of users of population statistics, including by:

seeking feedback on its current published quality and methodology information with a broad range of users and working together with other stakeholders across the Analysis Function.
helping users navigate to the various publications on the ONS website, for example by implementing a landing page.