Systems and sample size

The statistics team is capable but working with unfit systems

The Living Costs and Food Survey (LCF) collects information on spending patterns and the cost of living that reflect household budgets. It is conducted throughout the year, across the whole of the UK. The completed diaries and interview data are coded and edited electronically by a team in ONS to ensure internal consistency for each case received. Once complete, the LCF statistics team carries out further quality assurance checks before processing the data to produce the derived variables and the production of the LCF datasets. The data are then weighted and final datasets produced on a quarterly and annual basis.

We heard positive feedback from users of LCF about the professional capability of the LCF statistics team. The team keeps its key stakeholders updated when issues arise and is helpful in responding to queries. We found that despite the recent errors highlighted in the summary of this report, the LCF has largely continued to provide business-critical information.

The statistics team has endured challenges with the systems and resource it is working with to run the LCF, and it has done remarkably well to keep the LCF afloat despite these challenges. However, this has been at the expense of being able to dedicate time to develop the LCF. The team was reduced from two senior research officers to one at the time that Q-Stat tools (a processing package used in the creation of tables from LCF) were being replaced with an Excel and SAS-based system, as the new system was supposed to free up resource. However, the new system did not create as much of an efficiency saving as envisaged.

The current resourcing pressures mean the statistics team struggles to get an overview of the whole process of LCF – from writing the questions to producing the microdata sets. For the process to work effectively, there needs to be more than one person understanding the different stages of running LCF to be able to identify the impact of errors when they occur. The statistics team told us that maintaining two senior research officers would have alleviated some of the pressure faced by the team and allowed for capacity to analyse the data, progress the development work and take a step back to see the bigger picture.

The LCF data processing system is not fit for purpose and external stakeholders raised concerns to us about the longevity of LCF in its current form. The system is unstable, often producing inconsistent results between processing runs of data and requiring extensive manual checks following processing of annual (more than 100 manual checks) and quarterly (approximately 30 checks) data. This has led to a heavy mistrust in the data by teams in ONS.

While some ONS systems have been heavily invested in, such as those which produce trade-in-goods and regional Gross Domestic Product statistics, others like LCF have remained on legacy technology. Around nine years ago, a project was set up to explore how much it would cost to move LCF onto a new system. The initial discovery piece considered that doing so would be too costly, so some investment was put in to rewrite the aggregation in SAS and Excel. Opportunities to move LCF onto a new system are complicated by the nature of the diary tool being unique to LCF and the fact that LCF data have so many different dependencies. These issues appear to have restricted ONS’s appetite to develop the LCF.

Successful day-to-day management and development of the existing legacy system is impeded by the lack of staff trained in the system’s coding language (Manipula), which is unique to the LCF. The code is seen as overly complex, having been developed on an ad-hoc basis over time, and has resulted in a lack of understanding of how it works and therefore how to make adjustments. Central support to maintain parts of this system is not available. While, on occasions, support has been provided by the Blaise Development and Standards team, this is not a sustainable model. The natural turnover of staff means that as someone begins to understand the system, they tend to move on and the problem persists. This skill risk, coupled with an unstable and inconsistent system, are clear contributory factors to the recent errors in LCF data. ONS needs to take remedial action to improve the stability of the existing LCF processing system and to develop a new system which meets the needs of users and the staff running the systems.

The sample size of the LCF is a limitation of its use

The COVID-19 pandemic has seen the landscape of user need change with increased demand for timely and granular data. The LCF has been underperforming in terms of sample size and response rate in recent years and this was highlighted as the main limitation of the LCF by users. In the last two years, the LCF has achieved a sample of around 5,000 households out of a set sample of close to 13,000 households. The response rate has fallen from 51 per cent to 43 per cent since 2008. Due to the paper diary nature of LCF, and its burden on respondents, the LCF has been particularly hit in a wider pattern of falling response rates in social surveys. The LCF statistics team told us that the sample size was cut in 2006, due to a promise of a new allocation tool which would allow for a new un-clustered sample but the tool was never delivered. Prior to the COVID-19 pandemic, the LCF statistics team carried out a review of incentives. Incentives are used to encourage response rates and often take the form of money or a gift voucher which is provided to the respondent in exchange for completing a survey. The team tested a range of monetary incentives as well as unconditional incentives. From May 2021, the LCF has increased its conditional incentive to a £50 voucher for each adult who completes the questionnaire and diary.

The small achieved sample size makes it difficult for users to draw useful and robust conclusions from the data, particularly for individual categories and geographies. Some users told us that the sample size is too small to ensure accuracy when the spending categories are so detailed, and that even data for total spending are volatile over time, which limits their usability. As some users need to analyse the LCF data to such a precise level, the small sample size can create issues for understanding the validity of changes within individual categories where they relate to a small sample. For example, relatively few households buy a car each year which means the reported data on purchases of cars are based on a small sample of households. There is a risk of reputational damage for ONS where unusual changes in the results are assumed to be genuine and put down to small sample sizes and not identified as genuine errors, which can lead to a large impact on the price indices which use the LCF data for their weighting.

The LCF data are reported in the Family Spending statistics using Classification of individual consumption by purpose (COICOP) categories, an internationally recognised classification system consistent with that used by UK National Accounts. LCF data are currently reported at a 4-digit level of COICOP 1999 but the consumer trends team in ONS told us that there is a longer-term requirement to move to the updated COICOP 2018 which LCF would struggle to meet in its current form.

Statistics producers in the devolved administrations that we spoke to also expressed interest in wanting to replicate analysis that is carried out by HM Treasury for England, but which currently cannot be done for the other nations due to the small sample size. Without an increase to the sample size, the devolved administrations are unable to make use of some of the categories available in the LCF data. This prevents them from being able to draw comparisons with data for England, which impacts on budgetary analysis for the devolved administrations. ONS needs to develop a solution to address user need for more granular breakdowns of data, so that the devolved administrations and other key users can use the statistics in the ways that they need to for the public good.