Part I: Planning and designing a model that serves the public good

Models present many opportunities for developing and improving the production of statistics, and assisting decision making and predictive modelling. However, there are risks associated with introducing any model, or changes to a current model. Before developing, using, or changing a model, you should take the necessary steps in order to appropriately plan and design your model. In line with the quality pillar of the Code of Practice, it is important that you carefully consider whether your planned approach is right for the statistics or data you work with. Is the approach best suited to answer the question you want it to answer? For example, for some models, outputs may be statistically robust at the aggregate level but not at the individual level. Such models would be unethical if used for automated decision making at the individual level, even if model and data errors were well known and understood.  

If you wish to change a current, established model, you should not do so in haste. Many of the requirements needed to successfully manage, run and implement models are related to the transparency cross-cutting theme of the Code of Practice. 

There are two main factors which should guide your decision to create or use a model even before referring to specific elements of the Code of Practice.  

Back to top

Purpose

 

  • why do you want to continue to use or change your current approach? 
  • what are your motivations?  
  • can your intended approach directly meet the aim of the work? 
  • are there alternatives that also need to be considered?

Context

 

  • are you confident you can meet the quality of existing statistics or decision-making processes with the chosen approach? If not, what are the alternative benefits over a loss of quality? 
  • what is the lifespan of these statistics or the project? Is the investment in time required to deploy the model proportionate to the benefits it adds? 
  • are you adding value to the topic area? 
  • are you meeting the UK Statistics Authority’s strategy: statistics for the public good 
  • how will you measure model performance? How will you know if the model is working as expected? 
  • is learning capability sufficiently prioritised in your organisation?  
  • would the model be considered ethically appropriate in relation to ethical frameworks such as the UK Statistics Authority’s six ethical principles 

Checklist 

  • After considering the purpose and context of this work, the chosen approach is appropriate and warranted. 

Back to top

Ethical considerations

This section has been written in collaboration with our colleagues in the UK Statistics Authority’s (UKSA) Centre for Applied Data Ethics. Ethical implications and considerations for the use of models to create statistics or inform decisions are implicit throughout this guidance. Issues of data quality, methodological limitations, transparency, bias, and fairness all relate to ethical issues that should be carefully considered during project design. This section is to signpost readers to relevant additional material.  

The UK Statistics Authority ethical principles identify six key areas that should be explicitly considered to enable the ethical use of data. Models are no exception. These principles include public good, transparency, methods and quality, confidentiality and data security, legal compliance, and public views and engagement. When considering the use of a model, these ethical principles should be considered to ensure that the use of such techniques is ethically appropriate. An ethics self-assessment form can also be completed to assist in this process. 

Checklist 

  • The provenance of data is known (how it was collected, why, where, when and from whom).
  • Data and model bias are known and have been communicated sufficiently for others to understand them as well.
  • A diverse group of individuals have been included in the design and creation of the model.
  • The use of the model has clear benefits for users and serves the public good.
  • For models using personal data. The data subject’s identity (whether person or organisation) is protected, information is kept confidential and secure, and the issue of consent is considered appropriately. No data subject should be unfairly disadvantaged by the model.
  • The risks and limits of changes are considered and there is sufficient human oversight so that methods employed are consistent with recognised standards of integrity and quality.
  • The model and data used in the model are consistent with legal requirements such as Data Protection Legislation, the Human Rights Act 1998, the Statistics Registration and Service Act 2007 and the common law duty of confidence. 

Back to top

Professional capability

An important consideration in adopting any model in the production of statistics or decision making is to assess whether the responsible team is sufficiently skilled. Some data science modelling techniques, in particular machine learning and artificial intelligence (AI), require a different skillset from traditional statistical modelling techniques such as linear regressions. Likewise, knowledge of the skills required for more traditional statistical modelling techniques are needed to avoid use of complex techniques if not necessary. It is important to consider whether there is existing experience of modelling techniques in your team or an ability to train the team. If not, specialist resource may need to be brought in from outside your team. It is important to note that the team must have the appropriate knowledge and skills to manage both the implementation and maintenance of the model. Where a model is developed externally, you should explore what opportunities there are to upskill the team. This is to ensure that they can monitor and maintain the model, especially where data or context to the data may change. 

Checklist 

  • If the model is to be developed by the team, the team has the knowledge and skills needed to develop the model.
  • Regardless of who develops the model, the team has the knowledge and skills needed to manage and maintain the model.
  • The team has the knowledge and skills needed to successfully communicate the model and its outputs. 

Innovation and Improvement

Staying up to date with the latest analytical techniques presents an opportunity for team members to develop their skills and keep pace with developments in the wider field. Encouraging the learning of a wide range of techniques and providing team members with the tools and training to put learning into practice, contributes to career paths. As well as benefiting the development of individuals, it also provides an opportunity to push for the continuous development of statistics. New techniques may also provide a solution to existing data gaps or methodological issues that were not previously apparent. It is important to note; however, that newer techniques and methods should never be implemented without evaluation against other well established, and understood, techniques.  

Checklist

  • The benefits of building capability beyond the current work have been considered.
  • The team has access to continuous development and continuous learning. 

Transparent processes and management

The latest techniques present an opportunity to address user needs or achieve development goals. However, you must assess where these skills sit in your team and organisation’s overall development plan. You must assess whether these skills are sufficiently prioritised. Successful application of certain techniques, in particular machine learning and deep learning, relies on sufficient human and technological resource. It should be considered whether such techniques are the best use of available resource. The prioritisation of developments should be informed by users and any decision to reprioritise a project should be communicated openly with users. 

Checklist

  • The amount of resource required to implement the chosen techniques has been estimated.
  • The required resource can be allocated to meet the project aims.
  • Any reprioritisation required to meet this resource has been communicated. 

Back to top

Efficiency and proportionality

Any resource required to develop and maintain a model should be proportionate to the benefits and value arising from its use. For example, there may be an initial resource investment which leads to efficiencies being made in the longer term. These efficiencies then allow resource to be freed up to focus on development of the statistics. Contrary to this, models may require significant time to develop and maintain, which could limit resource elsewhere in the future. You must assess any potential impacts that a change in process might have on other statistics that use output from your model in their production. Please consult the Aqua Book chapter ‘Verification and validation’ for more information. 

Checklist

  • The benefit of the model outweighs the investment in resources to set it up.
  • The impact of this change in methodology on other statistics that rely on this model has been assessed and communicated. 

Back to top

Independent decision making and leadership (model accountability)

The Code of Practice states that the Chief Statistician or Head of Profession, or those with equivalent responsibility, should have sole authority for deciding on methods used for published statistics in their organisation. This requirement should be extended to models that generate experimental statistics and are used in decision making processes. This is particularly important for models where accountability of processes and outputs is necessary for data protection and gaining trust in the statistics.  

An appropriately chosen individual (Chief Statistician, Head of Profession, or other senior leader) must be responsible for any statistics, or data that are used in decision making, created by the model. 

This means the responsible individual is aware of the methodological choices that have been taken in designing the model. They should also be aware of the methods for validating the approach and any potential risks of the model. For models used in production of statistics or used in decision making, a clear chain of model accountability should be in place. A chain of model accountability allows anyone involved in the project to know who to go to if something goes wrong or an error is identified. Should any harm be caused by the outcome of the model, the chain of model accountability helps the accountable officer identify the source of the issue and who is legally responsible for the consequences of such harm. For more information on roles and responsibilities in a project lifecycle please see HM Treasury’s Aqua Book. 

Even when a model is developed by an external partner, the chain of model accountability must be established before the model is used. This is to ensure it is clear who is responsible for the decisions made by the model. The individual is responsible for ensuring that roles and responsibilities are clearly defined should an error occur. In addition, they must also be accountable for the handover of the model to or from the external partner. They must ensure that there are sufficient and appropriate skills in the team to maintain and monitor the model once the external relationship has ended. 

The ability to manage the model within the team should be assessed before commissioning the development of any model. Contingency should be built into the team’s development plan in case a team member leaves. This is to avoid a situation where the team does not have the relevant skills to continue using the model to produce the statistics. This could have a detrimental impact on the trustworthiness of the statistics and should be a priority when planning and designing a model. 

Checklist

  • A clear chain of model accountability has been established with clear roles and responsibilities.
  • The chosen responsible, senior leader knows that they are accountable for the development and outcome of the model. This includes any harms caused by the decisions made by the model.
  • The team has sufficient skills to manage the ongoing monitoring and development of the model. 

or  

  • The team has access to the necessary training to ensure they can manage and maintain the model once it is handed over from an external partner. 

Back to top
Download PDF version (163.24 KB)