Quality

Quality assurance arrangements should be enhanced through widening data access

The quality assurance arrangements that the LCF team has in place are generally appropriate for the data. There is a quarterly round for ongoing quality assurance, in which the team extracts the data which will be delivered to National Accounts and carries out a range of checks to compare against previous quarters and totals. Additional checks are carried out on the annual round, which include looking at partial cases and running imputations for missing diaries. Detailed audit trail spreadsheets are produced to keep a record of the checks that have been carried out at each stage.

Lots of variables that are available in the final LCF data tables are derived through the processing system. As highlighted in the previous section, this processing system is error prone and a substantial amount of time can be lost to resolving these errors. If an error in a variable is detected, the team has to rerun the whole code to check that the fix hasn’t inadvertently caused an error elsewhere. Each time the system is run, it produces 20 datasets which need to be individually checked and can therefore take a long time to resolve. This has led to a heavy mistrust in the data by teams in ONS.

As highlighted earlier, the sample size of LCF can make it difficult to determine whether changes between periods are genuine or not. The volatility of the data makes it difficult to pick up outliers when there is constant change. The LCF statistics team does look to alternative sources to try to verify counterintuitive results where possible, but it is not always clear which sources to take into consideration.

We heard from users of LCF data in ONS that widening access to the LCF data for the purposes of quality assurance could support the LCF statistics team in identifying errors and would allow it to focus on the pre-processing stages. These teams felt that if LCF data aggregation was moved within the Economic Statistics Group (ESG), it would allow more investigations to be carried out in one place and each of the teams using the LCF data could bring their perspective to assessing counter intuitive results to determine whether the changes are genuine. However, it should be noted that not all users of LCF data in ONS are in ESG and the main publication using LCF data, Family Spending in the UK, is published by a different area in ONS. ONS should provide a mechanism and relevant access for other teams in ONS who make use of LCF data to be able to contribute to the quality assurance of the data.

ONS should explore creative solutions to improve the robustness of LCF data

In 2016, a National Statistics Quality Review (NSQR) was carried out for the LCF. It sought to highlight areas that have not kept up to date with international best practice, require some improvement or that could impact on ONS’s reputation. The review set out 30 recommendations for LCF to be fit for purpose.

The team has taken forward most of the recommendations with no additional headcount being allocated to progress them, despite one of the NSQR recommendations being for ONS to ‘allocate additional resources to the LCF research team to develop and implement a more robust questionnaire and testing process and ensure the questionnaire design keeps pace with ongoing changes in consumer spending/behaviour’. The remaining recommendations remain relevant to improving LCF. Work to address six of these was paused over the last year due to the COVID-19 pandemic.

The 2016 NSQR highlighted that many of ONS’s international partners have been actively using administrative data to enhance the accuracy, quality and analytical capability of their household expenditure data. The LCF statistics team has previously engaged with the Eurostat Innovative Tools and Sources Taskforce and hopes to continue to engage with them and other international organisations, despite the UK having left the European Union.

While funding is a limitation to increasing the sample size, the NSQR recommended several alternatives that could be explored to improve the response rate and thus the achieved sample size. ONS needs to invest time and resource to pursuing initiatives to improve the quality and robustness of LCF data. ONS should be open to creative solutions to improve the response rate, such as continuing exploring the use of different short and long form questionnaires/diary, alternative sampling strategies and linking with other data sources, rather than focusing only on increasing the existing sample.