My favourite number is 6. It was the number Roberto Carlos, my footballing idol, wore when he played for Brazil. As a kid I remember trying to recreate his infamous banana free kick versus France (1997) over, and over again. Practice makes perfect they say. So here, in honour of Senhor Carlos, I am going to provide 6 pieces of advice from our latest publication Guidance for Models so you can all practice turning your models, into a role models. If you just want to know why we made this guidance, skip to the end of this blog.
Every year we throw away 2.5 billion takeaway cups in the UK
Don’t create your model using the same criteria as a takeaway disposable cup: use once and throw away. A model, or components of it, should be reusable. Reusable not just by you, but by others too. It should also be robust (see testing) and adaptable (see modular). Do not make a disposable cup just for coffee, if someone else is also just for tea, when you can work together to make a sturdy cup for both tea and coffee.
Weed your borders
Like beautiful garden borders, models need maintenance. Borders may need mulching in the winter, pruning in the spring and lots of watering in the summer. Likewise, model dependencies may need updating, bugs resolving, and code changes made as a result of changes to data schemas. Teams need to plan their time accordingly to make sure the model still works and remains fit for purpose, especially as others may rely on your model’s outputs for their own model input.
Don’t be like a tragic episode of Grand Designs
We’ve all seen a Grand Designs episode when the keen, wannabe architect wants to design, build and project manage on their own dream home. It often ends in disaster: over budget, late, relationship breakups, and a lot of stress. Likewise, you need the correct people involved at the correct time to make your model a success. Model design plans should be scrutinised and checked. Experts should be consulted early on, instead of after things go wrong. Lastly, like all house builds, the model should be verified and signed off by someone you trust to make sure it is safe and secure.
It took Jean-François Champollion years to decipher the Rosetta Stone
Model documentation should be accessible and understandable to a wide range of audiences. You should not just publish detailed, technical documentation as not everyone will be able to understand the purpose or scope of your model. This may lead to your model being misused. You should explain your model as best as you can. If the nature of the model means it is hard to explain, you should describe how users can interpret your model and its outputs at the very least. Your should open source your code if possible, and provide installation guides, code comments and examples as well.
Be as analytical in choices as you are on Netflix
Aim to be as detailed in your model decisions as you are when choosing your next TV show to binge watch. Like choosing between a horror and a rom-com, you must understand what kind of model you need based on your scope and aims. You should seek advice and guidance from experts. Like the impossibilities of trying to stay up to date with the latest shows using a 90’s copy of the Radio Times, your decisions should be based on relevant and up to date information. Lastly, don’t overcommit and carry on if things don’t seem to be going well. Use regular checkpoints to reassess against your original needs. No one wants to force themselves through another experience like season eight of Game of Thrones.
Be as ethically minded as you are when switching your energy supplier
You read an article last week about the environmental benefits of veganism and want to give it a go? Great. You switched your electricity supplier to a 100% renewable electricity supplier? Way to go. You stopped going to that pub that treated its workers poorly? Power to the people! Now also understand that data, design choices and model selection all can have ethical implications. Power can be given, or taken, from certain groups based on the models we create and use. Ethics should not just be a tick box exercise; it should be the cornerstone of your model design and development.
Sure, nice analogies, but why did you actually create this guidance?
Last year the global pandemic thrust us into the limelight following a series of high profile uses of statistics for decision making. One of the biggest pieces of work we did last year was our review of the approach to developing statistical models to award 2020 exam results. “Algorithms” were blamed, with one headline stating “Dreams ruined by an algorithm” (BBC NI website). As such, we have been concerned about the threat of undermining public confidence in statistical models more broadly.
That exam review work took us into new frontiers by commenting on the use of statistical models that influence public life, not just those that produce official statistics. But statistical models are just one range of tools used by government analysts, data scientists and statisticians. Increasingly, newer techniques such as machine learning (ML) are being tested and deployed in the production of statistics and used to inform decisions. Furthermore, with the creation of the Office for Artificial Intelligence and the Government’s National AI Strategy, we are likely to see an increased use of more advanced Artificial Intelligence (AI) techniques going forward.
As a result, we identified this as a crucial time to provide guidance for the use of models, regardless of whether they are statistical models, machine learning models or AI models. There have been a number of publications for ethical guidance for models (Ethics, Transparency and Accountability Framework for Automated Decision-Making, Data Ethics framework) as well as the creation of the UK Statistics Authority’s Centre for Applied Data Ethics. There are also a number of technical guides on how to develop models (Aqua Book). However, we saw that there was no current guidance that suitably brought together social, ethical and technical aspects for all elements of model creation: data, design, development, delivery and deployment.
We believe our role as a regulator, and our experience of the exam review from last year, puts us in a prime position to provide this socio-technical guidance for models. As a result, we have published our alpha release version of our model guidance with the aim to obtain feedback and comments from a wide range of users.
If you have any feedback, please get in touch! We aim to release an updated version of the guidance in early 2022.