3. Findings

Benchmarking the state of readiness of ABPEs to replace MYEs

This section outlines our findings related to the Trustworthiness pillar of the Code of Practice, namely the confidence in the people and organisations that produce statistics and data (in this case, the Office for National Statistics).

We consider how ONS plans to transition from the current MYEs to the ABPEs and how well this is supported through its governance and capability.

3.1 To demonstrate the evolution and improvement of the methods used in the DPM, ONS has published a series of outputs comparing the ABPEs with the MYEs in an open and transparent way. Census data are used in the production of ABPEs in several ways, including as a population stock for 2021 and 2011 to provide the coverage adjustment for the Statistical Population Dataset (SPD), and in the production of rate denominators for birth, death and migration rates. As census data are also a key input for the MYEs, this creates a challenge for understanding how comparable the ABPEs are with the MYEs in the absence of census data.

3.2 The work to develop the ABPEs forms part of the wider population transformation taking place within ONS as it considers the future of the census. In February 2023, ONS presented the ABPEs, with a reference date of 30 June 2021, in four different ways:

• an ABPE best estimate (this represents ONS’s best estimate derived from the DPM and uses the best available data, including Census 2021)
• an ABPE future estimate (this is indicative of what can be expected from population estimates derived from the DPM in the future, assuming there is no longer a census. It includes a proxy for a coverage adjustment method to adjust for bias in the SPD and does not directly include the Census 2021 estimate)
• an ABPE basic estimate (this does not include Census 2021 or a robust coverage adjustment method)
• the unadjusted SPD (before it is included in the DPM and without a coverage adjustment method)

3.3 As expected, where census data are used (such as for the ABPE best estimate), the total population estimate for England and Wales is much closer to the Census 2021-based MYEs, whereas both the ABPE basic and unadjusted SPD estimates are less comparable to the Census 2021-based MYEs. Therefore, the evidence around the quality of the ABPEs is not conclusive, as the ABPEs are dependent on the input data used. The analysis uses a quality standard set out in the evaluation criteria of the Beyond 2011 programme, a project looking at the alternatives to running a UK Census in 2021 to measure the quality of ABPEs at a local authority level (the quality standard is that estimates at local authority level are within 3.8% positive or negative of 2011 Census estimates). ONS has told us that it is updating its quality standards in light of Census 2021, starting with a minimum accuracy standard for ABPEs of age and sex by local authority.

3.4 The work to develop and deliver ABPEs is complex and progressing at pace, with many different teams across ONS working hard in their roles. While each individual team recognises its specific contribution, there is no clear governance to bring the aims of the work together to ensure consistency and working in a joined-up way.

3.5 As mentioned previously, the development of the ABPEs forms part of the wider population transformation taking place within ONS. The FPMS programme is ambitious, and like many major programmes, faces practical delivery constraints. As the programme spans multiple ONS teams, decision-making responsibilities need to be clear to ensure that appropriate oversight and governance are in place for the ABPEs. Such governance should appropriately consider how any developments are prioritised and resourced to meet programme deliverables, in line with any intended timescales. ONS should build on its recently published plans for population and migration statistics, taking into account any findings from the FPMS consultation. Alongside this, ONS should help users understand how the ABPEs, as the proposed method for measuring the population in the future, fit within these plans.

3.6 The ABPEs are currently published as official statistics in development. Final decisions around the ABPEs are made by the National Statistician. However, there is some uncertainty around how the decision to make the ABPEs the official population estimates will be made for England and Wales, and how this decision will be communicated both within and outside of ONS and to key stakeholders including the Welsh Government. ONS has developed success criteria for the FPMS programme as a whole, but no criteria for determining when ABPEs will become the official estimates of the population. Without clearly defined criteria, there is a risk that user confidence will be undermined.

3.8 ONS’s National Accounts are overseen by a coordination division, which plays a central role in coordinating timelines for system and data deliveries and statistical release timetables. Implementing a similar infrastructure for the ABPE teams would support the sustainable delivery of these estimates, so it is good to hear that ONS is reorganising its team structure with the aim of better supporting the production of the ABPEs.

3.9 The responsibility of producing ABPEs within ONS currently sits within the Population and Migration Research Division (PMRD). The PMRD team continues to research and develop the new method for population estimation. In the long term, ONS is planning to hand over the responsibility of the ABPEs to its Demography Division, which is responsible for producing population statistical outputs, including the MYEs. The plans for this transition, including timings, are crucial, as ONS needs to ensure that staff have a suitable level of expertise, knowledge and skills to embed a sustainable production process and have increased organisation resilience beyond the current handful of people who have expert knowledge. These plans remain under development.

3.10 As ONS develops these plans, it will be important that it ensures that they cover the end-to-end process, considering a range of scenarios that are at risk of change, including technical or data issues, staff turnover/capability issues, sustainability or improvements that are made to the methods and also the outputs. Given the computational challenges that ONS experienced at the start of the project and the complexity of Bayesian computational methods, it needs to ensure that there is capacity to support the DPM work. Therefore, we expect ONS to have an approved detailed plan that can be implemented at an appropriate time and before this transition is made to support this work operationally, drawing on principles outlined in the National Audit Office good practice guide to overcome challenges in managing risks.

3.11 The DPM production and development work is a complex, challenging, and pioneering effort in UK official statistics. It is being undertaken by teams who are working incredibly hard, and under considerable pressure. In addition to the complexity of this work, ONS also faces a challenge in terms of its financial position. Staff working on the ABPEs and developing the DPM are in roles that are funded by the wider FPMS programme. The programme is dependent on funding decisions made in the Spending Review and there is some uncertainty around its funding for this year and future financial years. Consequently, retaining and recruiting staff in the long term has been difficult. Any changes in the levels of funding could also impact the future delivery of the ABPEs.

3.12 The ONS team producing and developing the ABPEs is relatively small, with only a handful of people in ONS having a comprehensive understanding of how the DPM works. Estimating the population using a Bayesian demographic account, an internally consistent set of demographic stocks and flows, is challenging work. As far as we are aware, no other National Statistical Institutes have implemented this method, which limits the opportunities for ONS to learn from others’ experience.

Understanding the suitability and quality assurance of the data and methods

This section outlines our findings related to the Quality pillar of the Code of Practice; namely the data and methods that produce assured statistics.

Given the complexity of the methods used in the DPM, we commissioned an independent review from Professor Arkadiusz Wiśniowski which contains additional recommendations and suggestions for further work.

3.14 The DPM relies on access to data sources (as outlined in Annex A) that are supplied by other government departments and public agencies, with securely governed and sustainable data sharing representing a fundamental part of compiling the ABPEs. Our 2023 Data Sharing and Linkage for the Public Good report highlighted remaining challenges that hinder the benefits and value of data sharing and linkage. Government departments regularly use population data and therefore should recognise their role in facilitating data sharing to support ONS’s ambition to maximise the value and insight that administrative data can offer.

3.15 The recent Public Administration and Constitutional Affairs Committee report on the UK’s evidence base highlights the long-standing barriers to sharing data for statistical and research purposes. In the context of ABPEs, it is vital that ONS have continued and increased access to relevant data sources in order to produce these statistics, meet user needs and support the efficient use of public funds. We would welcome any cross-government commitment to support ONS with this.

3.16 ONS has many mature relationships with data suppliers and associated data sharing agreements that exist as part of the current production of MYEs and meeting the requirements of the wider FPMS programme. These strong relationships help ONS to regularly provide feedback on any data quality issues to data suppliers so that improvements can be made and any obstacles in data supply overcome. Strong governance is in place around the data supply for ABPEs to support escalation of any issues, as needed.

3.17 Recent delays with the Higher Education Student Record data provided by Jisc (under the Higher Education Statistics Agency brand), used within the SPD and migration datasets, have led to unexpected further work and pressure to the ABPEs production timetable. This delay highlights the interdependent nature of wider population and migration statistics, and the potential system-wide implications of a single delay or issue. ONS thus needs to consider the dependencies of data sources across the population statistics landscape and continue to use its data supplier relationships and its established governance mechanisms to escalate and deal with any future issues.

3.18 The ONS team that oversees and manages the data supply for the production of ABPEs is confident in the ability of its recently developed Data Pipeline Maturity Framework to assess the maturity of the data sources supplied to ONS. This is a good approach that will help ONS manage risks and put contingencies in place for any issues with data deliveries specific to the ABPEs.

3.19 The DPM is designed to be able to adapt to changes in data inputs. To support the sustainability of population estimates production and make use of a changing and evolving data landscape, newly identified and existing data sources can be incorporated into the model. For example, data from the Driver and Vehicle Licensing Agency (DVLA) have recently been identified as a potential source, and so ONS is seeking to collaborate with DVLA to understand how these data could potentially be used in the DPM. It is not clear, however, whether ONS has enough resource (both human and computational) to incorporate new data sources.

3.20 Whilst the DPM can adapt to changes in data availability, the quality of the data sources will affect its performance – highlighting the crucial need for ONS to fully understand the quality of the input data and embed knowledge around any uncertainties into the model itself. As Professor Wiśniowski mentioned in his report, without benchmarks to correct for biases in administrative data, the DPM may produce biased and/or uncertain ABPEs. We saw a positive example of the DPM‘s ability to overcome input data issues in 2023, when the annual MYEs were delayed from June to November. This delay was due to data quality issues with the internal migration component. Despite this set-back for the MYEs, in June 2023, the ABPEs were updated for all local authorities in England and Wales and the quality issues were accounted for within the credible intervals.

3.21 Coverage of special population groups, such as those that frequently move (for example students and foreign armed forces), or those that have atypical interactions with government services, such as those residing in communal establishments (prisons, care homes etc) is particularly challenging to measure. ONS makes adjustments to the MYEs to account for this. At present, ONS has published little information about how special populations are accounted for within the DPM. ONS has told us that with the planned inclusion of Ministry of Justice (MoJ) prison data in the SPD coverage of prison populations for the next release will be improved. However, ONS recognises it needs to do more to improve coverage of special populations more generally. Until these improvements have been made, ONS should update the published quality information that sits alongside the ABPEs to help users understand any coverage limitations.

3.22 It is unclear how ONS plans to evaluate the possible impact of changes in government policy on the quality of admin data sources that are used within the DPM. For example, the number of new GP registrations increased during the COVID-19 pandemic as part of the vaccination roll out, and the place in which some people registered differed from their permanent area of residence (as in the case of many students, for example).

3.23 In Bayesian demographic accounts, knowledge on the plausibility of demographic trends and the data models is dependent on having consistent and high-quality data. Therefore, how the DPM deals with volatile data (particularly those collected before and after the pandemic) and what impact this has on embedded assumptions are unclear. It is also unclear how and when ONS will review and update the model’s assumptions – this poses a risk to data quality as the accuracy of data sources changes over time. ONS should test the DPM in extreme situations where sudden changes in data quality may be encountered, as recommended by Professor Wiśniowski.

3.25 The SPD, a linked dataset, is the main stock measure used in the DPM. The SPD is created from the Demographic Index (DI), a composite data source built from a range of admin data sources using cuts of data. Methods for assuring the quality of the DI are not yet in place, with little publicly available information explaining the data transfer process and any risk of errors associated with this. ONS identified and proposed metrics to measure the quality of the DI in 2022. Our understanding is that the metrics have not yet been implemented, with insufficient programme funding cited as one of the reasons why this has not been done. Quantifying error arising in the DI (from linkage error or other data source issues) is an important metric to inform how SPD quality is measured.

3.26 To better understand the quality of the SPD, ONS conducted an analysis comparing both aggregate and individual records from the SPD version 4.0 and Census 2021. Whilst at an aggregate level the SPD is broadly similar to Census 2021 estimates, there are patterns of over-coverage (where SPD estimates are higher than Census 2021 estimates) and under-coverage (where SPD estimates are lower than Census 2021 estimates). Differences are by age, sex and geographically at national levels (with under-coverage for Wales – the SPD estimates are 5.2% lower than the Census 2021 estimates) and local authority levels. The differences between the estimates for Wales and England may be linked to the inclusion of Hospital Episode Statistics (HES), the Emergency Care Data Set (ECDS), and the Individualised Learner Record (ILR), which only cover England. ONS has told us that it plans to include Welsh equivalents of these data sources in the next iteration of the SPD.

3.27 Differences between the Census 2021 and the SPD counts likely reflect a combination of true population change and the differing features of the census and SPD. Census 2021 was carried out when England and Wales were under more COVID-19 restrictions than at the reference date for the SPD (June 2021). The way that people interacted with public services during the pandemic was atypical, and therefore reliance on administrative data sources collected during the pandemic needs to be quality-assured against survey data sources. The variability of estimates derived from the SPD compared to Census 2021 records on a disaggregated level, for populations with varying characteristics in different areas, points to a need for ONS to benchmark the SPD and capture any issues that may affect the quality of the ABPEs, including bias from over-coverage and under-coverage. Currently coverage ratios are derived from Census-based MYEs 2011 and 2021 data. Considering the UK Statistics Authority’s wider recommendation on the future of population and migration statistics, and building on its work to date, ONS needs to develop an alternative approach for coverage adjustment of the SPD in the long term. We expect ONS to implement the recommendations highlighted in Professor Wiśniowski’s report to work towards overcoming this issue.

3.28 Based on published material to date, it is difficult to understand how the stock data (and other data sources) change over time and how this affects the quality between ‘provisional’ and ‘updated’ ABPEs. Provisional mid-2023 ABPEs use SPD version 4.1 and population stock data from the Personal Demographics Service (PDS) yet there is no detailed quality information specific to SPD version 4.1, beyond a description of what data sources are used in its compilation. ONS should support the use of data by clearly communicating what data inputs are used over time, as this may affect the quality of the ABPEs.

3.29 The SPD is produced by linking data from various administrative data sources, meaning that data will come from different sources for different, and occasionally the same, people. ONS has not fully evaluated the impact of different approaches and definitions being used in the collection of data across all admin sources, or communicated this to users to ensure appropriate interpretation. Our published guidance on collecting and reporting data about sex and gender identity in official statistics highlights the need for producers to have a strong understanding of their data and statistics to be able to explain their strengths and limitations.

3.30 The complexities of creating a population stock measure using linked administrative data sources highlight how important it is that ONS accurately measure and report relevant quality information to help users understand any potential bias (for example, definitions, coverage issues and linkage errors) in the source data, and the extent of any impact on the statistics. ONS needs to address and prioritise improving its communication of quality with users.

3.32 As described previously in the ‘Introduction to the proposed new methods using the Dynamic Population Model’ section, migration data account for the largest proportion of population change and are the component with the most uncertainty. Four of the recommendations from OSR’s 2022 Review of Migration Statistics remain outstanding and these are relevant to the development of ABPEs. In particular, ONS’s current work to investigate all sources of uncertainty within LTIM estimates and the results of this should be published and considered in the wider context of ABPEs.

3.33 In order to understand the uncertainty of the ABPEs, ONS needs to investigate all sources of uncertainty and develop reliable measures for LTIM data, as recommended by Professor Wiśniowski. We expect ONS to prioritise this as part of addressing the outstanding requirements from our review of its migration statistics.

3.34 ONS is working on plans to improve estimation of internal migration and is working with the Scottish Government, NRS and NISRA to determine how it should approach cross-border flows to improve measuring internal migration. Data quality issues in some of the data used for the internal migration components caused a delay in publication, with the official MYEs for 2022 being published in November 2023. Whilst the flexibility of the DPM allowed the 2022 ABPEs to be published as planned in June 2023, these data quality issues point to a broader issue with the quality of migration data and the importance for ONS to ensure the DPM assumptions are appropriately updated.

3.35 ONS carries out a series of quality assurance checks on the input data for the DPM, and a real-time dashboard is in place to help identify any data anomalies or issues with the supply of administrative data. ONS uses its strong relationships with data suppliers and its internal forum for data quality management to understand any identified data issues and take action.

3.36 Given the variety and number of teams involved in producing these statistics, there is ambiguity in what the needs are for ABPEs and how errors are managed. There is no detailed process map nor fully documented quality assurance approach covering the end-to-end process. Appropriate research and quality metrics for clustering, linkage error, coverage and managing errors are not yet well established across the whole process, and there is siloed working across some of the teams. More clarity and documentation around how quality is described and assessed at each stage of the process is needed to fully audit the data flow and to help individual teams understand what impact any data issues may lead to as part of the wider production process.

3.37 ONS points to various published Quality Assurance of Administrative Data (QAAD) reports to support its level of assurance and confidence in the quality of individual data inputs, for example long-term international migration, births and deaths. Most data quality reports exist as part of the production of the MYEs and were last updated in 2016 or 2017. More recently, ONS developed Quantitative Quality Indicators (QQI) to describe its assessment of the quality of inputs used to compile the SPD version 4.0. Quantifying the quality of administrative data is crucial for the outputs of the DPM. Also, improving the definitions used to differentiate between bias and accuracy, in line with Professor Wiśniowski’s findings will help to ensure that the DPM produces reliable population estimates. We have encouraged the team to seek advice and expertise from the Government Data Quality Hub in considering what quality information, including that captured by the QQI, is needed to support appropriate ongoing use of the statistics.

3.38 Reproducible Analytical Pipelines (RAP) are automated statistical and analytical processes that increase the efficiency, reproducibility, auditability and quality assurance of the statistics. The ABPEs production pipeline does not yet meet RAP standards. The publication of the programming code using an open software platform such as GitHub would support transparency, as well as encourage peer scrutiny and challenge. We are encouraged to hear that once the code has undergone further internal development, ONS intends to publish it in an open and transparent way. The ONS Methodology and Quality team is also supporting the ABPEs team to building capability and ensuring that the production code is produced in a reproducible way that is safe and sustainable.

3.39 We welcome a review that ONS has started which will cover coding, quality assurance and capability, as well as a review of the methods, with the overall aim being to help the DPM move from a research phase into a production phase. Given the relevance of this review to the scope of our assessment and the areas where we are seeking further assurance, we look forward to seeing its outcome later this year.

3.41 Given the specialist knowledge of Bayesian statistical analysis needed to develop the DPM, ONS has been collaborating with experts in this field from the University of Southampton as well as a Bayesian demographer from New Zealand. There is no national or international consensus on the methods used in the DPM. Therefore, external collaboration with academic experts continues to be crucial to putting the Bayesian theory into practice. On paper, the theory underpinning the DPM is well grounded. TheMethodological Assurance Review Panel (MARP), which was formed by the National Statistician to provide advice and assurance on methods used to produce official statistics, provides independent assurance on the overall statistical design and methods being developed for the ABPEs. The panel comprises mainly academics whose expertise covers various aspects of statistics. Papers submitted to MARP are published on the UKSA website. However, they are not clearly linked to the publications, which would help technical users better understand the methods.

3.42 Recognising the niche area of expertise required to fully understand the practical application of Bayesian methods in the DPM, we support ONS’s plans to introduce an additional sub-group of MARP. Members of the sub-group should be independent of the development of the methods to introduce the extra scrutiny and challenge needed to appropriately interrogate the methods. A rigorous approach will ensure that users have confidence in how ONS develops the methods going forward.

3.43 Since the introduction of the DPM in 2022, ONS has continued to develop and make changes to the methods used in the DPM; for example, to improve the speed of the estimation process, the method was changed in December 2023. However, to understand at a high level how the methods have evolved over time, users would need to read five statistical publications and three accompanying methodology reports. Moreover, these publications are hard to navigate and understand, given the volume and changeability of the data sources used and the intricacies of the methods applied.

3.44 ONS has shared with us a list of priorities and developments for the methods (for example hierarchical models). There are no specific timescales for these developments or information about their impact on the model’s performance. Without more transparent documentation it is difficult to follow the model’s development and performance.

3.46 One of the strengths of the DPM is its ability to produce provisional timely estimates of the population. With revisions expected to be a normal part of publishing the ABPEs in the future, it is good to hear that ONS is planning to develop and implement a revisions policy for these statistics. This is a significant shift away from rebasing current population estimates every 10 years using the census, to a more frequent revision schedule.

3.47 The methods used to produce provisional estimates are dependent on the availability of data sources. For example, due to a lag with HESA data, the methods used to estimate internal migration differ from those used for the updated estimates. As shown in the table below, the only comparison between provisional and updated ABPEs that ONS has made to date was published in 2021 and 2022. Relatively small revisions were made for those two years.

Comparison of updated ABPEs with provisional estimates

Source: ONS’s ABPEs published June 2023

3.48 Changes to the methods used in the input data, such as with LTIM, will affect the consistency of time series. Within the DPM, it is unclear how Bayesian modelling and estimation will be able to identify and distinguish any changes in methods applied to input data to help users make sense of and understand the reasons underpinning changes in population trends.

3.49 In order to develop a population statistics revisions policy, ONS must understand its data and how they are used by different users. To meet user needs, ONS will need to consider the demands for both timeliness and consistency as well as the potential impact of revisions on other statistical publications where population estimates are used as a denominator or in survey weights. At the moment, the lack of a revisions policy for population estimates represents a significant gap. A revisions policy would support transparency and help users understand what will be published and when.

3.50 We recognise that there may be practical challenges to implementing a comprehensive revisions policy, and so would encourage the ONS team to apply the practices used in producing estimates of Gross Domestic Product (GDP) to ensure best-practice principles are followed. The team should also consider the findings outlined in our review on revisions of GDP estimates.

3.51 During our engagement, users welcomed the move to ABPEs highlighting the potential for timelier estimates, as well as the possibility to address the issue of drift in population estimates between censuses. However, others raised concerns about the clarity of revisions and the importance of understanding at what point the estimates are fit to use to support decision-making.

User understanding and confidence in the proposed new method

This section outlines our findings related to the Value pillar of the Code of Practice, namely statistics that support society’s need for information.

We highlight the importance of ONS’s engagement with users to understand their needs, manage their expectations and be responsive to feedback to help shape the development of ABPEs going forward. Tailored communication would support user understanding and build trust in the new method.

3.53 The ABPE team points to the FPMS consultation as its primary public engagement with users and stakeholders, in addition to a number of DPM-specific events that targeted expert users. The team told us it is awaiting feedback from the consultation, which will help it to understand users’ needs, particularly around quality. The team acknowledges that the consultation was very broad and asked users about their needs for all population estimates, whereas the DPM is focused on population estimates by age and sex.

3.54 In working to meet user needs, ONS also needs to understand the similarities and differences between the ABPEs and MYEs. As an example, the DPM does not currently split migration into internal and international components for publication. At present, ABPEs provide breakdowns of births, deaths and net migration only (which includes international migration and migration between England and Wales and other parts of the UK), whereas the MYEs are able to provide these same breakdowns and are the only source of information on international migration in Wales. We understand that ONS intends to provide this further breakdown in due course, which is fundamental to meeting Welsh users’ needs, in addition to those of users requiring estimates of internal migration.

3.55 Having an open and effective dialogue with users and stakeholders is vital in demonstrating that ONS is a trustworthy organisation that actively listens and responds to users’ views. Therefore, it is good to see, as highlighted in its recent update article on transformation of population and migrations statistics, that ONS intends to gather feedback from users, in particular local authorities, on the ABPEs in autumn 2024. We encourage the team to build on these plans to develop a broader strategy to understand users’ needs, including their need for breakdowns of components such as internal migration. This strategy will help to shape and steer the development of the ABPEs going forward.

3.56 As the outcome of the FPMS consultation is yet to be published, it is not clear to users what ONS’s long-term intention is for ABPEs as a statistical output, and the ability of the DPM to produce outputs in line with this intention. This sentiment was echoed in our conversations with users, with some seeing ABPEs as a replacement to the census while others see it as a way of measuring the population. For example, ABPEs currently cover the population by age, sex and local authority (in essence, they replicate the current annual MYEs), but it is unclear whether ABPEs will be developed in future to provide more-granular breakdowns to meet users’ needs for information such as data on small area estimation (geographical breakdowns lower than local authority in England and Wales) and characteristics such as ethnicity.

3.58 The ABPE are estimates and therefore subject to uncertainty. ONS uses credible intervals for breakdowns by age, sex and local authority to quantify the degree of uncertainty. At an aggregate level, the December 2023 ABPEs are very similar to official MYEs, at 0.1% lower than the Census 2021-based MYEs. Differences do vary by local authority, with the largest differences generally found in cities and more-urban local authorities. Aggregate comparisons can mask a range of differences at more-granular breakdowns, for example for particular age groups, highlighting the benefit of engaging with users to scrutinise and test evidence at a local level, especially for those estimates of specific population groups.

3.59 Measuring student populations is challenging in terms of capturing both internal and international student migration patterns. In 2021, as part of our review of population estimates and projections produced by ONS, we found that in some smaller cities with a large student population, the estimates appeared to be potentially higher than local evidence suggested. Many of our recommendations within the report pointed to a need for ONS to collaborate with others to learn from best practice, for example expert demographers who produce their own estimates on a more localised level. These findings remain relevant for the ABPEs. Any insights offered or queries raised by population demographers and other expert users such as local authorities need to be fully investigated with curiosity and rigour to fully understand the causes of any potential discrepancies with other data insights or sources. Accurately capturing difficult to measure populations, such as students, also highlights the need for ONS to reconcile its population estimates with other available data sources (for example, the Annual Population Survey) that may provide more assurance at lower-level geographies.

3.60 The users we spoke to reported confidence that the ABPEs are likely – in the long run – to meet their needs and would be higher quality than the current method of producing MYEs. However, some users reported that whilst ONS was open to receiving feedback, they were not informed about how ONS responded to their concerns about the accuracy of the ABPEs. ONS should be open to this kind of scrutiny from users to benefit the ongoing development of the model itself, and to have appropriate processes in place to ensure that these queries are responded to (even if it was decided that no further action was necessary).

3.61 Shared best practice and learning with the ONS National Accounts production teams may be helpful here, given its quality assurance through ‘curiosity meetings’ which involve internal and external stakeholders. We expect that ONS’s plans to engage with local authorities in the future, and the support from communications teams, mean that ONS can be more proactive with this feedback in future.

3.63 Recognising the complexity of the methods used in the DPM, ONS published a short video explaining to users at a high-level how the DPM works and why it is being developed. For technical users, ONS published a detailed paper and hosted a series of workshops and webinars to aid understanding of the methods used. The navigation between the research and documentation surrounding the development of ABPEs, published to date, could be improved. Users explained to us that a centralised landing page with links to all of the documentation in chronological order would help them to follow and understand the developments that are taking place.

3.64 Our engagement with users indicates that they appreciate the increased use of administrative data to produce these estimates, but do not understand the new method. We heard this from a wide range of users, including methodological and demography experts and local authorities’. Users told us that some of the published material was not understood and did not provide enough detail on the DPM. Experts who had spent considerable time trying to understand the methods also reported struggling to fully grasp how the DPM works, including stakeholders in devolved administrations. Academics told us there was a middle ground that wasn’t being met in terms of communicating information about the DPM to non-technical, but engaged, users. For example, there was a gap between the high-level information and the more-detailed methodological and academic studies documentation. Users were also concerned about how the new approach could work without a census and felt that ONS has not provided sufficient assurance on this.

3.65 ONS acknowledges that the DPM is complex and uses innovative methods, and as such, it is challenging to explain the method to audiences with varying technical expertise. It also recognises that this is an area where it needs to do further work. Further, ONS accepts that there has been some inconsistency in the language and naming of developments, which has confused some users. ONS has stated that it plans to address some of these issues, for example by publishing a plain-English technical volume and journal articles. It has also introduced a new role with oversight of the narrative and statistical design so that it can improve coherence across its outputs. We would encourage ONS to collaborate more widely across the Analysis Function to seek best practice and advice, for example with the Government Data Quality Hub and Data Science Campus, on how to communicate complex statistical methods in a way that supports understanding.

3.66 Our findings that ONS needs to engage more with users, be more responsive to scrutiny, and better communicate its methods, resonate strongly with the findings from our Review of Population Estimates and Projections, published in 2021. Given we have stated very similar recommendations before to ONS, it is disappointing that lessons do not appear to have been learnt from this previous work.