Modelling framework

Demographic accounts framework

19. The DPM is a modelling framework for population accounts that relies on solid theoretical foundations of the Bayesian demographic accounting approach developed by Bryant & Graham (2013, 2015) and Bryant & Zhang (2018). The framework permits the estimation of coherent population accounts that satisfy the population balancing equation, including the assumption that internal migration does not change the size of the population (i.e., internal net migration is always zero). Coherency is achieved by explicitly modelling the unobservable accounts with their constraints such as population balancing equation, and it permits incorporating various dimensions (i.e., the characteristics of the population), such as age, sex, region of residence, ethnicity, and other.

20. The framework relies on Bayesian inference which provides a natural method for producing measures of uncertainty of the ABPEs and the assessment of their reliability. The use of population stocks based on administrative sources compiled every year rather than relying on updating of the census, should lead, in principle, to the removal of the uncertainty (and bias) in population estimates due to the UPC or census drift (e.g. ONS 27/06/2023a Figures 1 & 2), which is a key advantage over the current cohort-component-based approach of producing MYE.

21. There is evidence of a strong engagement of the ONS with leading academics working on the methods implemented in the DPM, including organising workshops in academic conferences. The DPM development also learnt from experiences of other statistical offices, such as Australia, New Zealand, and Italy (Blackwell et al. 2022), which helped identify key strengths and limitations of the methods. Model development would also benefit from a systematic engagement with the ABPEs users, especially those working with population characteristics for which the potential errors can be largest, such as at local authorities. An example of such engagement was a case study presented in ONS (23/11/2022).

22. The development of the DPM encountered computational challenges early in the project. This led the team to develop a novel, cutting-edge approach to estimating model parameters and producing the ABPEs. This, however, was achieved at a cost of simplifications in the demographic accounting model, which are not estimated jointly as originally proposed in Bryant & Graham (2013, 2015) and Bryant & Zhang (2018). Briefly, the current procedure is to produce (i) approximate components of the accounts that do not have to be consistent, (ii) estimate model for demographic rates by using proxy estimates from (i); (iii) re-estimate demographic accounts individually for each local authority and use them to (iv) derive combined accounts and coherent estimates of internal migration.

23. The justification and rationale for using this approach are reasonable and especially important during the development of the modelling framework, when many models need to be tested. The rationale is also articulated in the technical documentation (Elliott & Blackwell 2023, Blackwell et al. 2022). However, in my opinion, the computational aspects should not be a major detractor from the foundations of the population accounts, such as fully coherent demographic accounts (as originally proposed in Bryant & Graham 2013), multiregional (Raymer et al. 2020) or bi-regional models (Wilson 2016). Research on foundations should go in line with the developments of the specific aspects of the model, such as new ways of smoothing data, hierarchical structure (currently not present in the DPM) and incorporation of other data sources. If indeed the computation of the joint model is not deemed possible, even with the aid of high-performance computing, the justification could be enriched by providing a comparison of the full model implemented on a small scale with the simplified approach followed by an analysis of errors, similarly to how an assessment of the computational methods and their impacts on the ABPEs has been demonstrated by ONS (18/12/2023).

The need for high-quality data

24. In general, the DPM is advertised as a flexible and adjustable platform that can accommodate a variety of population-related estimates to satisfy a variety of demands (Bryant & Zhang 2018; Elliott & Blackwell 2023). For instance, the ONS considers using the model for estimating breakdown by ethnicity, labour force status; it is also possible to provide monthly population estimates (Elliott & Blackwell 2023). However, there are at least two difficulties with this approach: the need for high-quality inputs and scalability.

25. Firstly, as has been demonstrated by the analyses carried out by the ONS, the model-based ABPEs can be sensitive to the inputs in the data (e.g., the sensitivity to the assumed precision of the MYE and SPD inputs, ONS 14/07/2022; comparison of three versions of ABPEs that utilise or not utilise 2021 Census, ONS 28/02/2023a, 28/02/2023b). As discussed later (Point 34), the inputs can be modified before being used in the model, or the model parameters can correct for data inadequacies. However, both approaches require deep understanding of the data generating processes of all DPM inputs (which the ONS generally demonstrate through their reports, e.g. ONS 2022 but also acknowledges the need for more work in this area, see e.g. Points 10 & 16) and a thorough testing within the DPM, which I understand from the communications with the ONS is work-in-progress. The ONS are well-aware of the need for a coverage benchmark and several options have been proposed and evaluated through simulation studies (ONS 2022, Elliott & Blackwell 2023, ONS 27/06/2023a). These evaluations and the transparency of the data quality assessments internally and externally to the ONS will be crucial to future deployments of the DPM and trust in the model-based ABPEs. The DPM is indeed a flexible and modifiable approach but without benchmarks to correct for biases in admin data, it may produce biased or uncertain ABPEs (acknowledged in ONS 2022). Thus, a robust quality assurance should be in place starting from data production and ending with the testing of the ABPEs.

26. The above issue is exemplified by the sensitivity analyses and comparisons of the model-based ABPEs with the official 2021 Census-based MYE (e.g., 14/07/2022, 28/02/2023a, 27/06/2023a, 27/06/2023b). ONS (28/02/2023) showed that when Census was used as input to the model and adjustment for coverage, the differences between model-based ABPEs and 2021-Census-MYE were minimal. However, when 2021 Census was used only as a coverage benchmark (and it is the best coverage benchmark available), the ABPEs for selected LAs can differ by 1-2% from the 2021-Census-MYE and much larger differences were found for detailed breakdowns by age. These analyses demonstrated that the choice of the coverage benchmark and the model inputs can modify the model-based ABPEs, especially at the local level and for detailed characteristics.

27. These findings and sensitivity of the model-based population estimates to the assumptions have been corroborated by a study carried out on Italian national accounts (Taglioni 2019), who tested the same Bayesian demographic accounting approach of Bryant and Graham (2013). The study examined models that incorporated the hierarchical structure and were estimated jointly, unlike the models currently proposed by the ONS. This study showed that the models for population components (current Step (ii) in the DPM, Point 22) can be sensitive to the choice of the informative prior distributions for the model parameters and their choice can be crucial to the resulting population estimates. However, the accounts models (current Step (iii) in the DPM) were less sensitive to the choice of the model. Further, the study revealed that in a situation where there are differences in population size in two data sources deemed to be of high-quality (Italian 2011 census and population register), fine-tuning of the model to produce reasonable results can be difficult, especially in the context of comparing multiple hierarchical models, which, based on the documentation, has not yet been tested by the ONS. I consider it important that the ONS provides evidence of testing the DPM in extreme situations that might be encountered in the future in terms of sudden changes in data-generating processes (e.g. through changes in the legal frameworks governing administrative systems and how people interact with them) that may affect the quality of the inputs. The ONS currently provides thorough comparisons of the ABPEs based on various iterations of the DPM (e.g., 28/02/2023b, 18/12/2023b). I am also aware that the ONS has been testing various assumptions of the model, e.g. about the distributional assumptions for the data (based on unpublished documents or those still in-preparation)*. I advise that comparisons could be presented in one document or a website, where various estimates can be compared with the 2021-Census-MYE but also between each other, ideally accompanied by the measures of errors (bias, e.g., via Mean Percentage Error, and precision, via e.g. Mean Absolute Percentage Error) across various dimensions (cf. Dańko et al. 2024).

*The other important tests could include goodness of fit measures via prior and posterior predictive checks, which are a typical component of a Bayesian workflow (e.g. Gabry et al. 2019). These checks include predicting the data inputs by using a model with only prior distributions, or estimated by using data inputs, potentially perturbated by random or systematic removals of portions of the data. I have been advised that such tests are being carried out by the ONS team working on the DPM development. Further, the current framework does not include hierarchical structure that permits borrowing of information across various model dimensions (as was tested by Taglioni 2019). I recommend research on the DPM is extended to test the robustness of the current modelling framework when hierarchical models are implemented for population components.

28. The second challenge to realising the potential of the DPM and its flexibility might be the scalability due to the above-mentioned computational difficulties (also corroborated by Taglioni 2019). Bayesian computational methods are complex and may require updating, as has already been demonstrated by the ONS (18/12/2023). Introduction of new dimensions and data sources may bring additional computing cost that will be prohibitive, even in the simplified framework. Model estimation also depends on a variety of packages in open software R. While this is in principle an approach I endorse and recommend, it is also susceptible to risks such as packages not being available/maintained/compatible with other packages in the future. Therefore, in my opinion, it is important to maintain or develop within the ONS the capacity for implementing computational methods used in and around the DPM.

Migration component

29. The last point of the procedure of reconciling internal migration (Step (iv) in Point 22) is potentially risky for the quality of the final estimates because internal migration can be the largest source of uncertainty for the characteristics of the population such as age, sex and local authority (Bryant & Graham 2013; Taglioni 2019). The currently used approach utilises an iterative proportional fitting algorithm (Elliott & Blackwell 2023), which is a reliable method for reconciling demographic accounts amongst LAs. The method uses as a starting point internal origin-destination migration data derived from the PDS (Point 18), but these data do not necessarily satisfy the demographic identity. The reconciliation is carried out to the known margins (Elliott & Blackwell 2023). However, the uncertainty in the internal migration estimates is derived purely from the margins and not the estimation procedure. Of concern might also be the bias in the data inputs as exemplified by the mismatch of the PDS (via DI) and 2021-Census-CSS (ONS 01/03/2023, see also Point 18).

30. Moreover, in the current approach, migration to and from each of the LAs is modelled jointly, as a single in- or out-flow to or from the local authority. Since the estimates of international migration are derived from a variety of sources (Point 14) and the DPM requires a careful setting of informative priors, it might thus be challenging to formulate a prior for the migration component as the uncertainty of the estimates may depend on the prior. I suggest that a thorough testing of the sensitivity of the ABPEs to the choice of this prior is performed and documented.

31. Further, a lack of a high-quality benchmark for the internal migration estimates to adjust for coverage issues and delays in reporting, together with any potential biases in the international migration data may lead to errors in the ABPEs that could potentially be similar to the intercensal drift, for example, if the benchmark based on the 2021 Census is continued to be used in the future. This may affect selected local authorities or sub-populations (e.g. mobile working-age persons, those working from home), see Point 35.

Documentation and reproducibility

32. As part of the assessment, I analysed the documentation of a part of the computer code used in producing the DPM population estimates that was made available by the ONS. The documentation is transparent with examples of how to use the R packages developed by the ONS team. I am aware of a detailed log file where issues with the code and the model are being reported. While certain aspects of the code seem to be indeed in development, the provided documentation assures that the model can be tested internally and potentially also by external stakeholders. As mentioned before, my opinion is that the ONS should develop and maintain the capacity to implement the future changes in the DPM methodology, computational methods and their implementation. This will ensure the new admin-based population estimation system is sustainable in the future and the ONS is on track to achieving this with the computer code they are developing.

33. A development by the ONS team of the R packages that implement the DPM and let users produce their own ABPEs are crucial for ensuring the reproducibility of the results and, thus, further testing of the DPM and ABPEs robustness. The current versions of the packages permit the estimation for selected local authorities and production of publishable documents with the analyses of the results that are generated by the package. A potential issue with reproducibility, especially if the packages are to be shared with partners external to the ONS, is that the exact data inputs may not be directly accessible due to data sharing agreements in place. These issues can be mitigated by an interactive R package that I understand is being developed by the ONS. This package could contain a toy model that would demonstrate the workflow of the DPM and permit testing of (some of) the model assumptions, perhaps by using synthetic (not real) data. This would also benefit the communications of the estimates especially to stakeholders interested in a more detailed understanding of the model (such as academics, local authorities, government departments).